September 19, 2014

Honest Networker 3613: Nov 20 07:20:50.927 UTC: %ENV_MON-2-TEMP: Hotpoint temp sensor(slot 18) temperature has reached WARNING level at 61(C) 3613: Nov 20 07:20:50.927 UTC: %ENV_MON-2-TEMP: Hotpoint temp sensor(slot 18) temperature has reached WARNING level at 61(C)

by ohseuch4aeji4xar at September 19, 2014 05:12 PM

Packet Pushers Blog/Podcast

Network Break 17

Take a stroll through the Intel IDF 2014 conference which was all about the Software Defined Network/Storage/Infrastructure/Architecture ......

by Packet Pushers Podcast at September 19, 2014 04:25 PM

Honest Networker
Packet Pushers Blog/Podcast

IPv6 Networking Detection Case #141 – Part 1: The Facts and Clues

Part #1 – I give you the facts and the clues. Part #2 – I give you what the problem ended up being. Ready to play? This is the IPv6 troubleshooting blog that started off as something else entirely.   I was going to do a post on IPv6 Multicasting, so I grabbed 3 ASR1K and […]

Author information


Denise "Fish" Fishburne
CPOC Engineer at Cisco Systems

Denise "Fish" Fishburne, (CCIE #2639, CCDE #2009:0014, Cisco Champion) is a team lead with Cisco's Customer Proof of Concept Lab in Research Triangle Park, N.C. Fish loves playing in the lab, troubleshooting, learning, and passing it on.

The post IPv6 Networking Detection Case #141 – Part 1: The Facts and Clues appeared first on Packet Pushers Podcast and was written by Denise "Fish" Fishburne.

by Denise "Fish" Fishburne at September 19, 2014 01:00 PM

Cisco IOS Hints and Tricks

Virtual Networking in CloudStack

If you mention open-source cloud orchestration tools these days, everyone immediately thinks about OpenStack (including the people who spent months or years trying to make it ready for production use). In the meantime, there are at least two other comparable open-source products (CloudStack and Eucalyptus) that nobody talks about. Obviously having a working product is not as sexy as having 50+ vendors and analysts producing press releases.

Read more ...

by Ivan Pepelnjak ( at September 19, 2014 01:39 PM


Some New Old Books

The Lean Startup approach is for companies that want to be more capital efficient and that leverage human creativity more effectively.  It relies on “validated learning,” rapid scientific experimentation, as well as a number of counter-intuitive practices that shorten product development cycles, measure actual progress without resorting to vanity metrics, and learn what customers really want. It enables a company to shift directions with agility, altering plans inch by inch, minute by minute.

The products lean startup builds are really experiments: the learning about how to build a sustainable business is the outcome of those experiments. That information is much more important, because it can influence and reshape the next set of ideas. And it uses the Build-Measure-Learn feedback loop at the core of Lean Startup model.

In short: build Minimum Viable Product (MVP) or Minimum Viable Service (MVS), launch, collect data to learn in order to perfect the ideas. Or if it doesn't work, fail fast, and pivot to another ideas.

Rather than wasting time creating elaborate plans for new product, it's better to launch quickly and find a way to test the idea continuously, to adapt and adjust before it’s too late.

How to decide which ideas to develop first?

In Four Steps to the Epiphany, Steven Blank offers the practical and proven four-step Customer Development process for search and offers insight into what makes some startups successful.

Rather than blindly execute a plan, The Four Steps helps uncover flaws in product and business plans and correct them before they become costly. Rapid iteration, customer feedback, testing your assumptions are all explained in this book.

What will be the next step if we want to use lean principle to launch new product/service?
We still need to create a plan. But don't waste time in creating plan.

Business Model Generation explains the most common Business Model patterns, based on concepts from leading business thinkers, and can help us reinterpret them for our own context. We can learn how to systematically understand, design, and implement a game-changing business model--or analyze and renovate an old one. Along the way, it will make us understand at a much deeper level our customers, distribution channels, partners, revenue streams, costs, and our core value proposition.

Business Model Generation advises us to use "Business Model Canvas" to analyze quickly any product/service we want to build. We can even use a modified model which is the "Lean Canvas" that is closer to the lean startup principle.

Each of us just need to spend few minutes to learn about the Lean Canvas model in order to use it.

Ideas are cheap. Execution is everything.
Think, build, launch. Get feedback, iterate, update. Repeat.

by (Himawan Nugroho) at September 19, 2014 12:18 AM

XKCD Comics

September 18, 2014

Honest Networker
Renesys Blog

Why Far-Flung Parts of the Internet Broke Today

VolumeDrive is a Pennsylvania-based hosting company that uses Cogent and (since late May of this year) Atrato for Internet transit. A routing leak this morning by VolumeDrive was passed on to the global Internet by Atrato causing disruptions to traffic in places as far-flung from the USA as Pakistan and Bulgaria.


The way Internet transit is supposed to work in BGP is that a provider announces the global routing table to its customers (i.e., a large number of routes). Then, in turn, the customers announce local routes to their respective providers (generally a small number of routes). Each customer selects the routes it prefers from the options it receives. When a transit customer accidentally announces the global routing table back to one of its providers, things get messy. This is what happened earlier today and it had far-reaching consequences.

At 06:49 UTC this morning (18-September), VolumeDrive (AS46664) began announcing to Atrato (AS5580) nearly all the BGP routes it learned from Cogent (AS174). The resulting AS paths were of the following format:

    … 5580 46664 174 …

Normally, VolumeDrive announces 39 prefixes (networks) to Atrato: 27 it originates itself and 12 it transits for two of its downstream customers, Visperad Networks (AS15351) and DataWagon (AS27176). However, during this leak, Atrato propagated over 400,000 routes learned from VolumeDrive or nearly the entire global routing table. A full table is currently hovering at around 500,000 routes — a figure on the minds of many due to the 512k limit on many older routers. (Note that this particular routing leak resulted in no new routes and therefore didn’t increase the size of the global routing table.)

The following graphic shows how the Internet has reached VolumeDrive over the past couple of days, using either Atrato or Cogent. VolumeDrive experienced a brief outage earlier in the week. Then just before midnight UTC, Atrato dropped out entirely as a VolumeDrive provider.


When Atrato briefly returned as a provider at 06:49 UTC, VolumeDrive passed them nearly all the routes it learned from Cogent. Evidently Atrato did not have the circuit breaker on the quantity of routes it would accept from VolumeDrive (MAXPREF), because it in turn announced these routes to the rest of the world. ISPs will often use such a limit to avoid propagating an erroneous flood of routes, electing instead to shutdown the link to contain the potential damage. That’s a good engineering practice as there are probably no legitimate circumstances where Atrato should receive 400,000 BGP routes from one of its customers. In fact, for a link that normally transits 39 prefixes, seeing a few hundred routes should be enough to trigger an automated response.

Global Impacts

To recap, a major routing leak occurred, one that was entirely preventable with some common-sense limits. So what? How much impact could this small Pennsylvania-hosting company have on the global Internet? Well, quite a lot in fact — such is the nature of our trust-based Internet routing. Pretty much anyone can mess it up.

According to Dyn’s IP Transit Intelligence tool (shown below), Atrato transits around 5,000 prefixes and has over 600 peering connections (not simply BGP adjacencies, but “peering” as opposed to “transit”) — at least they did at the start of the day. That much peering can act as a very loud amplifier for leaked routes.


Peering relationships are established between providers that wish to exchange traffic, often to avoid paying their upstream providers to carry their customers’ traffic. Routes learned from peers are generally prioritized over routes learned from a transit provider because peers often exchange traffic for free (settlement-free). Thus, when bad routes are propagated by a provider with a lot of peers, those routes can travel far and wide.

The following graphics depict the number of our traceroute measurements completing from both Islamabad, Pakistan and Los Angeles to China Unicom, China’s second largest ISP after China Telecom. The dips in completion rate begin at 06:49 UTC when, instead of going through Singtel (in the case of
Islamabad) or Telia (in the China Unicom), traffic was diverted to Atrato. Many of these traces never reached their intended destinations.

isb.4837-3 lax.4837

These graphics are generated from thousands of measurements; however, examining individual traceroutes reveals the exact details of the path changes during this incident. The traceroute shown below was performed yesterday and takes a path from Islamabad to Karachi where it then boards a submarine cable en-route to Singapore before finally reaching Zhengzhou, China. Geographically, it’s a reasonable route even if the recorded latencies are quite high.

trace from Islamabad, Pakistan to China Unicom Henan Province at 09:08 Sep 17, 2014
1  *
2      (PTCL, Islamabad, PK)              0.66ms
3       0.523ms
4    (ITI, Rawalpindi, PK)              3.907ms
5    (ITI, Karachi, PK)                 30.205ms
6   (PTCL, Karachi, PK)                26.221ms
7    (SingTel IX, Singapore)            226.13ms
8   (Singtel, Singapore)               521.436ms
9    (China Unicom, China)              522.932ms
10    (China Unicom, China)              472.701ms
11    (Backbone of China Unicom)         519.328ms
12    (China Unicom Henan province)      500.175ms
13     (China Unicom, Zhengzhou)          484.009ms
14     (China Unicom, Zhengzhou)          494.929ms

Next is a traceroute from the same server to the same IP in Zhengzhou during the routing leak. Instead of passing through Singapore en-route to China, the path goes first to Atrato in Amsterdam (Atrato peers with PTCL at AMSIX) and then onto Telia who takes it to San Jose, California before finally arriving in China. By exporting the leaked routes from VolumeDrive, Atrato, in effect, inserted itself into the path between Pakistan and China! Atrato could have easily overwhelmed its own capacity at this time, as many of our measurement did not reach their intended destinations.

trace from Islamabad, Pakistan to China Unicom Henan Province at 06:59 Sep 18, 2014
1  *
2      (PTCL, Islamabad, PK)                      0.484ms
3               0.567ms
4    (ITI, Rawalpindi, PK)                      2.838ms
5    (ITI, Karachi, PK)                         27.867ms
6   (PTCL, Karachi, PK)                        29.401ms
7                           164.01ms
8              165.093ms
9            170.78ms
10            172.437ms
11     (Atrato, Amsterdam, NL)                    159.247ms
12                      252.254ms
13                     237.843ms
14                     243.867ms
15                     246.865ms
16                     316.894ms
17  356.87ms
18   (China Unicom, China)                      355.425ms
19     (China Unicom, China)                      350.21ms
20    (Backbone of China Unicom)                 356.632ms
21    (Backbone of China Unicom)                 644.495ms
22    (China Unicom Henan province)              590.476ms
23     (China Unicom, Zhengzhou)                  537.887ms
24     (China Unicom, Zhengzhou)                  543.934ms

In the newly released Dyn Internet Intelligence tool, such impairments in the flow of traffic show up as gaps in completed latency measurements, illustrated below:


As an example of good Internet hygiene, SK Broadband (formerly Hanaro) apparently shutdown its connection to Atrato during the leak. Traceroutes from locations around the world (ranging from Denver to Bulgaria) reveal that traffic suddenly shifted from Atrato to NTT en route to SK Broadband. It appears that SK Broadband automatically turned down its connection to Atrato when faced with a flood of bogus routes, as illustrated below.

var.9318_vps01.var1 9318_vps01.den1


Not all routing leaks are origination leaks like in the Indosat leak earlier this year or the China Telecom leak of 2010. In that scenario, a provider announces the global routing table claiming that it is the "origin" (and therefore the destination) for every single routed network in the Internet. Routing leaks can also occur when routes are simply passed in the wrong direction between providers.

While basic route hygiene (e.g. using MAXPREF to limit the number of routes accepted from a customer or peer) could have prevented this incident, it underscores a larger point: we're all in this together. The Internet is our electronic commons and its proper functioning depends on everyone in control of an Internet router.

Routing goof-ups like this don't need to involve your IP address space to impact you. If it impacts the routes of someone you are trying to communicate with or one of the ISPs along the way, then it's your problem too. By understanding how the Internet works and gathering real-time Internet Intelligence on your assets and those of your customers or suppliers, you can work with your providers to mitigate the damaged caused by the inevitable mistakes, even when the source is on the other side of the world.

The post Why Far-Flung Parts of the Internet Broke Today appeared first on Renesys.

by Doug Madory at September 18, 2014 08:55 PM


ThousandEyes Network Monitoring Use Cases

ThousandEyes is a network monitoring company who’s shining a light on the darkened portion of the network path you don’t own. Rather than the Internet appearing to an enterprise as a generic cloud where magic happens, ThousandEyes looks inside the cloud, revealing details about how your enterprise gets to remote services. For example, […]

by Ethan Banks at September 18, 2014 07:35 PM

Security to the Core | Arbor Networks Security

Let’s Talk About NewPosThings

by Dennis Schwarz and Dave Loftus

NewPosThings is a point of sale (PoS) malware family that ASERT has been tracking for a few weeks. It operates similarly to other PoS malware by memory scraping processes looking for credit card track data and then exfiltrating the spoils to a command and control (C2) server. Based on compilation times, it has been in active development since at least October 20, 2013—with the latest timestamp being August 12, 2014. Since we haven’t come across any public details of this family, we’re releasing our malware analysis for posterity and to get ahead of the threat.

The analyzed sample has an MD5 of 4196c67648003a18f61573a77b6d3be6.


Its name comes from an embedded PDB pathname string from the analyzed sample:

C:\Users\Tom\documents\visual studio 2012\Projects\NewPosThings\Release\NewPosThings.pdb


The malware initializes itself as follows:

  • Sets some insecure file flags in the Registry:
    • “LowRiskFileTypes” in “HKCU\Software\Microsoft\Windows\CurrentVersion\Policies\Associations”
    • “1806” in “HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Zones\0”
  • Copies itself to “%APPDATA%\Java\JavaUpdate.exe”
  • Checks whether it is running as 64-bit and if so, exits with a MessageBox of “Use 64bit version.”
  • Kills any existing “JavaUpdate.exe” processes
  • Sets up Registry Run persistence (HKCU\Software\Microsoft\Windows\CurrentVersion\Run) under “Java Update Manager”
  • Executes copied executable passing the original executable’s pathname and “RM” as command line arguments
  • Original process exits

Second copy continues:

  • Deletes original executable
  • Phones home to the C2
  • Searches the Registry for VNC passwords. The following keys and values are checked:
    • HKLM\SOFTWARE\RealVNC\vncserver[Password]
    • HKLM\SOFTWARE\RealVNC\WinVNC4[Password]
    • HKCU\SOFTWARE\RealVNC\WinVNC4[Password]
    • HKCU\Software\TightVNC\Server\[Password]
    • HKCU\Software\TightVNC\Server\[PasswordViewOnly]
    • HKCU\Software\TigerVNC\WinVNC4\[Password]
    • HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\Ultravnc2_is1[InstallLocation] (then opens ultravnc.ini looking for “passwd=/passwd2=”)
  • Sets “SeDebugPrivilege” privilege
  • Starts memory scraping thread
  • Starts C2 communications thread
  • Starts key logging thread

C2 Communications

The initial C2 phone home is HTTP POST based and looks like:



Header-wise: there is a bug with Accept as, per the code, it is supposed to be “*/*” and the User-Agent is hardcoded to “Mozilla/4.0(compatible; MSIE 7.0b; Windows NT 6.0)”. The POST parameters break down into:

  • cs – is in base64 and decodes to “insert”
    • The base64 charset used in this malware is “ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_”
    • p – contains Windows version, “+32” is hardcoded (most likely a placeholder for architecture type), and computer name
    • m – the volume serial number of the root drive (used as an identifier)

An example response from the C2 is:



If any VNC passwords were found in the Registry they are exfiltrated to the C2:



The parameters are:

  • cs – base64, decodes to “log”
  • m – the volume serial number of the root drive (used as an identifier)
  • ls – base64, example below
>>> myb64.b64decode("dm5jX3Bhc3N3ZAoK")

The response from the C2 can contain a couple of other commands:

  • “update => url” – update self
  • “install => url” – download and execute
  • “not found” – remove self

Update and install both download to and execute from “%APPDATA%\Java\Explorer.exe”.

Credit Card Track 1 and 2 Memory Scraping Thread

Running in a loop, this thread takes a snapshot of all running processes it has access to on the system. For each process it:

  • Skips if it is a system process, one of:
    • svchost.exe
    • System
    • smss.exe
    • csrss.exe
    • winlogon.exe
    • lsass.exe
    • spoolsv.exe
    • alg.exe
    • wuauclt.exe
  • Skips if the process is the malware itself
  • Reads all accessible read/write memory
  • Parses each byte looking for “^” or “=” – field separators used in credit card track 1 and track 2 data respectfully
  • If there’s a match, further checks are done (track 1 and 2 are checked similarly):
    • Working backwards from the separator looks for 13-20 digits that could make up the Primary Account Number (PAN). Ends at a non-digit start sentinel
    • Checks the possible PAN with the Luhn algorithm—used to validate credit card numbers
    • Checks that the expiration date is reasonable: year is between [20]13 and [20]19 and month starts with a 0 or 1
    • Checks whether rest of the characters after the separator are digits
  • If the checks pan out, the track data is RSA encrypted with an embedded PUBLICKEYBLOB, see IDA screenshot below
  • It is then base64 encoded
    • If RSA encryption fails for whatever reason, the track data is just base64 encoded
  • Encrypted data is stored for later exfiltration to the C2



Credit card track data is exfiltrated very similarly to VNC passwords:



The “ls” parameter decodes to newline-separated entries:

 >>> ls = myb64.b64decode("U2Nhbm5pbmcgc3RhcnRlZAo0MUR1ekN5NF9oOGJiOVpUSDV0blRpWXhUbG5INnB4V0g2OXhZcVBLckJ2SjN0RXlvT1ZaanJnei1EOVpsOURPVTlsYVBnejVpWndkZ3FxUkw3U3RzeEdwVUs1YWwzRHR3aTZvTW9hNjRXWF9LSm9Da0QxRENmTHRwQ1NTTUhvYkNKY3lJVWJJanBBR2hfUWlaMDNYOHRPZUpZTmJrbDRzclptU05qMjlPUmM9CmV2NzA2QXYyZTlKTUpKRmlWZzZOaGMwSE85dlFvNy15RFJ2ZlJ5TmpFVVA0SlVTQU8wYUh3eUFERTVDWWRQVWNYWWhmRThqX0Q0b1I2NnNxejZ1ZzlyVmR2N0c4ZTRPLWdQYW9aVjdVSmJaaS1pN2NWUDJlb3hxb0ZCZTdkZ2pJeWZUNTRNelRDamlqanF4VzV1a080ek1HaThvVUdkUGpEamVoN0RiMEtYQT0K")
 >>> ls.split("\n")
['Scanning started', '41DuzCy4_h8bb9ZTH5tnTiYxTlnH6pxWH69xYqPKrBvJ3tEyoOVZjrgz-D9Zl9DOU9laPgz5iZwdgqqRL7StsxGpUK5al3Dtwi6oMoa64WX_KJoCkD1DCfLtpCSSMHobCJcyIUbIjpAGh_QiZ03X8tOeJYNbkl4srZmSNj29ORc=', 'ev706Av2e9JMJJFiVg6Nhc0HO9vQo7-yDRvfRyNjEUP4JUSAO0aHwyADE5CYdPUcXYhfE8j_D4oR66sqz6ug9rVdv7G8e4O-gPaoZV7UJbZi-i7cVP2eoxqoFBe7dgjIyfT54MzTCjijjqxW5ukO4zMGi8oUGdPjDjeh7Db0KXA=', '']

After the first entry, the rest are base64 and RSA encrypted credit card track data:

 >>> myb64.b64decode("41DuzCy4_h8bb9ZTH5tnTiYxTlnH6pxWH69xYqPKrBvJ3tEyoOVZjrgz-D9Zl9DOU9laPgz5iZwdgqqRL7StsxGpUK5al3Dtwi6oMoa64WX_KJoCkD1DCfLtpCSSMHobCJcyIUbIjpAGh_QiZ03X8tOeJYNbkl4srZmSNj29ORc=")

Credit Card Track 2 Key Logging Thread

The last thread starts by extracting a DLL from an executable resource and writes it to “%APPDATA%\Java\DLLx64.dll”. The filename is hardcoded even though the DLL is 32-bit. The “InstallHooks” export is executed which installs a keyboard hook via the SetWindowsHookEx function. After hook installation, the thread loops looking for Windows messages with an ID of 2023 (sent by the hook function). On receiving the message, a buffer of 400 keystrokes is collected and is searched for track 2 data as above—not particularly sure why the malware is looking for credit card track data in typed input though.

Command and Control Login


Samples and Campaigns

The following are the samples and campaigns known to ASERT:

MD5: 87f6385a4cb0520e19782350c30826bc

C2: hXXp://

Compilation date: 2013-10-20 23:15:49

Notes: I believe this is an earlier development version that differs somewhat from the above analysis. Copies itself to “%APPDATA% \Java\javaj.exe”.


MD5: 3cee6591a0ec2e1e1bdd317ec8777c58

C2: hXXp://

Compilation date: 2013-10-22 00:40:30

Notes: Also an earlier development version using “javaj.exe”.


MD5: ec0e8edbab6575e167689cca533f75f0

C2: hXXp://

Compilation date: 2013-12-21 20:34:39

Notes: Also an earlier development version using “javaj.exe”. This IP has hosted a Citadel C2 at hXXp:// in the past.


MD5: fefeb6a27f34b35a6a43c65c188bcde7

C2: hXXp://

Compilation date: 2014-05-11 10:31:02

Notes: Per Google Translate, “eklemek” is the Turkish word for “adding”.


MD5: 68dbce1053450a4395368835367d20b5

C2: hXXp://

Compilation date: 2014-05-12 19:00:35


MD5: 2576bc49e3c796b5b94695241d0d4359

C2: hXXp://

Compilation date: 2014-05-12 19:03:08


MD5: 3d58e0b2b9303e0bc4bb282c1ee2dd18

C2: hXXp://

Compilation date: 2014-05-15 12:30:47


MD5: 40e556c77948037497b9205932e69b97

C2: hXXp://

Compilation date: 2014-05-15 12:32:05


MD5: 4196c67648003a18f61573a77b6d3be6

C2: hXXp://

Compilation date: 2014-05-20 19:32:42

Notes: Live C2 panel at time of writing. hXXp:// is also a live C2 at the time of writing, but I don’t have the associated sample.


MD5: ae9899722707fc2c9716138580787026

C2: hXXp://

Compilation date: 2014-08-12 15:43:20

Notes: This sample returns to copying to “%APPDATA% \Java\JavaJ.exe”, just using camel-case. The PDB pathname has also changed to “C:\Users\Tom\documents\visual studio 2012\Projects\jsd_12.2\Release\jsd_12.2.pdb” so the author may have rebranded it from “NewPosThings” to “jsd”. At the time of writing, the C2 panel is live and the domain has hosted an Andromeda C2 at hXXp:// in the past.

Yara Rule

We’ve been using the following Yara rule for tagging and hunting:

// newposthings, Dennis Schwarz, Arbor Networks ASERT, September 2014
rule newposthings
    $pdb1 = "C:\\Users\\Tom\\documents\\visual studio 2012\\Projects\\NewPosThings\\Release\\NewPosThings.pdb" nocase
    $pdb2 = "C:\\Final32\\Release\\Final.pdb" nocase
    $pdb3 = "C:\\Users\\Tom\\documents\\visual studio 2012\\Projects\\jsd_12.2\\Release\\jsd_12.2.pdb" nocase
    $str1 = "install =>"
    $str2 = "update =>"
    $str3 = "cs=bG9n&m="
    $str4 = "cs=aW5zZXJ0&p="
    (any of ($pdb*)) or (all of ($str*))


This post has been an analysis of a point of sale malware known as NewPosThings. Its modus operandi is similar to that of other PoS malware where it memory scrapes for credit card track data, does some basic verification with the Luhn algorithm, and then exfiltrates to a command and control server. While it appears to be in active development, the scope and distribution of this threat is currently unknown. ASERT continues to monitor NewPosThings and other PoS malware threats.

by Dennis Schwarz at September 18, 2014 05:40 PM

My Etherealmind

Response: CloudFlare Keyless SSL Means The End of Big Pipes and Load Balancers in The Data Centre

This blog post from Cloudflare should mean the end of big Internet pipes in Enterprise Data Centre with massive reductions in load balancers, IDS units and web servers.

The post Response: CloudFlare Keyless SSL Means The End of Big Pipes and Load Balancers in The Data Centre appeared first on EtherealMind.

by Greg Ferro at September 18, 2014 05:16 PM

Cisco IOS Hints and Tricks

Dynamic FCoE – Sparse-Mode FCoE Strikes Again

A while ago Cisco added dynamic FCoE support to Nexus 5000 switches. It sounded interesting and I wanted to talk about it in my Data Center Fabrics update session, but I couldn’t find any documentation at that time.

In the meantime, the Configuring Dynamic FCoE Using FabricPath configuration guide appeared on Cisco’s web site and J Metz wrote a lengthly blog post explaining how it all works, triggering a severe attack of déjà vu.

Read more ...

by Ivan Pepelnjak ( at September 18, 2014 08:47 AM

September 17, 2014

Honest Networker
Packet Pushers Blog/Podcast

Community Show – Killing the Spanning Tree with SPB,TRILL,Fabricpath from Anthony Sequeira and Orhan Ergun

No country for old men !!. This week in the Orhan Show,  Anthony Sequeira and Orhan Ergun are talking about spanning tree , its drawbacks, spanning tree modes, technologies which can eliminate the spanning tree's drawbacks or completely do not use it. Orhan recommends all audience to read  this blog post about spanning tree which has been written by Greg Ferro recently.   Did you know that Shortest Path Bridging version of Avaya had been used in the core of a network at the Sochi  2014 Winter Olympics.   Below technologies are discussed in the podcast: Spanning Tree 802.1d, Rapid Spanning Tree 802.1w , Multiple Instance Spanning Tree 802.1s EtherChannel - Link Aggregation Group Multichassis EtherChannel The Virtual Switching System (VSS) Virtual Port Channel (VPC) Ethernet Host Virtualizer (EHV) Shortest Path Bridging (SPB) TRILL Fabricpath

by Packet Pushers Podcast at September 17, 2014 06:25 PM

Honest Networker

Trying to figure out what the latest “Cloud SDN NFV Orchestration service” really is

Trying to figure out what the latest “Cloud SDN NFV Orchestration service” really is

by ohseuch4aeji4xar at September 17, 2014 01:54 PM

Cisco IOS Hints and Tricks

You’ve been doing the same thing for the last 20 years

When we were discussing my autumn travel plans, my lovely wife asked me “What are you going to talk about in Bern?” She has a technical background, but I didn’t feel like going into the intricacies of SDN, SDDC and NetOps, so I told her the essence of my keynote speech:

Read more ...

by Ivan Pepelnjak ( at September 17, 2014 10:06 AM

XKCD Comics

September 16, 2014

Cisco IOS Hints and Tricks

Fate Sharing in IP Networks

My good friend Tiziano complained about the fact that BGP considers next hop reachable if there’s an entry in the IP routing table even though the router cannot even ping the next hop.

That behavior is one of the fundamental aspects of IP networks: networks built with IP routing protocols rely on fate sharing between control and data planes instead of path liveliness checks.

Read more ...

by Ivan Pepelnjak ( at September 16, 2014 09:06 PM

My Etherealmind

OpenDaylight in the Enterprise: OpenFlow and NETCONF in the right places

When using Open Daylight (ODL), two open standards for configuration are OpenFlow & NETCONF. Which is the better choice ? Is there an option for both ? A use case on when to use OpenFlow and NETCONF protocols in the Enterprise by using the best features of each protocol.

The post OpenDaylight in the Enterprise: OpenFlow and NETCONF in the right places appeared first on EtherealMind.

by Greg Ferro at September 16, 2014 08:03 PM

Cisco IOS Hints and Tricks

The Four Paths to SDN

After the initial onslaught of SDN washing, four distinct approaches to SDN have started to emerge, from centralized control plane architectures to smart reuse of existing protocols.

As always, each approach has its benefits and drawbacks, and there’s no universally best solution. You just got four more (somewhat immature) tools in your toolbox. And now for the details.

Read more ...

by Ivan Pepelnjak ( at September 16, 2014 08:57 PM


How We Filter Information About IT Products

The amount of information to be found out about tech products is astonishing. Anything you’d like to know about virtually any product is a Google search away. The hits you’ll get back are loaded with information, some useful and some…less useful. I am awash in data to ingest each and every day. I […]

by Ethan Banks at September 16, 2014 05:38 PM

Honest Networker
Packet Pushers Blog/Podcast

Show 205 – Open Source Network Monitoring with

Network monitoring is one of our most requested topics on Packet Pushers, and this week we take on open source monitoring solutions. Why open source? Because commercial NMS solutions are all over the place in functionality and price. So, if it's possible to put a solid NMS in place based on open source, then it's worth a try. Many of us are familiar with a variety of open source NMS projects, including Zabbix and Nagios. The trick with any of these packages is that they need a lot of configuration massaging before they are useful. Out of the gate, they just aren't all that easy to use. Enter, the central point of discussion on this show. OMD, somewhat like Security Onion, takes several different open source NMS tools, bundles them together, and adds a configuration layer that makes them relatively easy to deploy. Plus, the end resulting package is reasonably powerful. While you'll still need to do some tweaking to get precisely what you want, OMD takes much of the pain out of the initial setup. We have this discussion with Dominik, a Packet Pushers listener from Germany who needs to remain somewhat anonymous due to who he works with on a day to day basis. He's done quite a bit of interesting work with OMD, and talks through it with us. Links (Thanks, Dominik!) Check_MK Project Page. A powerful open source monitoring web front-end. Visualization add-on for the Nagios system. Monitoring map from the US Department of energy monitoring solution. OpManTek. Greg's favorite monitoring solution. OMD packages for Debian, Ubuntu, CentOS, RHEL and SuSE. Network flow visibility with NSX. (

by Packet Pushers Podcast at September 16, 2014 03:50 AM

September 15, 2014

The Networking Nerd

Why is Lync The Killer SDN Application?


The key to showing the promise of SDN is to find a real-world application to showcase capabilities.  I recently wrote about using SDN to slice education networks.  But this is just one idea.  When it comes to real promise, you have to shelve the approach and trot out a name.  People have to know that SDN will help them fix something on their network or optimize an troublesome program.  And it appears that application is Microsoft Lync.

MIssing Lync

Microsoft Lync (neè Microsoft Office Communicator) is a software application designed to facilitate communications.  It includes voice calling capability, instant messaging, and collaboration tools.  The voice part is particularly appealing to small businesses.  With a Microsoft Office 365 for Business subscription, you gain access to Lync.  That means introducing a voice soft client to your users.  And if it’s available, people are going to use it.

As a former voice engineer, I can tell you that soft clients are a bit of a pain to configure.  They have their own way of doing things.  Especially when Quality of Service (QoS) is involved.  In the past, tagging soft client voice packets with Cisco Jabber required setting cluster-wide parameters for all clients.  It was all-or-nothing.  There were also plans to use things like Cisco MediaNet to tag Jabber packets, but this appears to be an old method.  It was much easier to use physical phones and set their QoS value and leave the soft phones relegated to curiosities.

Lync doesn’t use a physical phone.  It’s all software based.  And as usage has grown, the need to categorize all that traffic for optimal network transmission has become important.  But configuring QoS for Lync is problematic at best.  Microsoft guidelines say to configure the Lync servers with QoS policies.  Some enterprising users have found ways to configure clients with Group Policy settings based on port numbers.  But it’s all still messy.

A Lync To The Future

That’s where SDN comes into play.  Dynamic QoS policies can be pushed into switches on the fly to recognize Lync traffic coming from hosts and adjust the network to suit high traffic volumes.  Video calls can be separated from audio calls and given different handling based on a variety of dynamically detected settings.  We can even guarantee end-to-end QoS and see that guarantee through the visibility that protocols like OpenFlow enable in a software defined network.

SDN QoS is very critical to the performance of soft clients.  Separating the user traffic from the critical communication traffic requires higher-order thinking and not group policy hacking.  Ensuring delivery end-to-end is only possible with SDN because of overall visibility.  Cisco has tried that with MediaNet and Cisco Prime, but it’s totally opt-in.  If there’s a device that Prime doesn’t know about inline, it will be a black hole.  SDN gives visibility into the entire network.

The Weakest Lync

That’s not to say that Lync doesn’t have it’s issues.  Cisco Jabber was an application built by a company with a strong networking background.  It reports information to the underlying infrastructure that allows QoS policies to work correctly.  The QoS marking method isn’t perfect, but at least it’s available.

Lync packets don’t respect the network.  Lync always assumes there will be adequate bandwidth.  Why else would it not allow for QoS tagging?  It’s also apparent when you realize that some vendors are marking packets with non-standard CoS/DSCP markings.  Lync will happily take override priority on the network.  Why doesn’t Lync listen to the traffic conditions around it?  Why does it exist in a vacuum?

Lync is an application written by application people.  It’s agnostic of networks.  It doesn’t know if it’s running on a high-speed LAN or across a slow WAN connection.  It can be ignorant of the network because that part just gets figured out.  It’s a classic example of a top-down program.  That’s why SDN holds such promise for Lync.  Because the app itself is unaware of the networks, SDN allows it to keep chugging along in bliss while the controllers and forwarding tables do all the heavy lifting.  And that’s why the tie between Lync and SDN is so strong.  Because SDN makes Lync work better without the need to actually do anything about Lync, or your server infrastructure in general.

Tom’s Take

Lync is the poster child for bad applications that can be fixed with SDN.  And when I say poster child, I mean it.  Extreme Networks, Aruba Networks, and Meru are all talking about using SDN in concert with Lync.  Some are using OpenFlow, others are using proprietary methods.  The end result is making a smarter network to handle an application living in a silo.  Cisco Jabber is easy to program for QoS because it was made by networking folks.  Lync is a pain because it lives in the same world as Office and SQL Server.  It’s only when networks become contentious that we have to find novel ways of solving problems.  Lync is the use case for SDN for small and medium enterprises focused primarily on wireless connectivity.  Because making Lync behave in that environment is indistinguishable from magic, at least without SDN.

If you want to see some interesting conversations about Lync and SDN, especially with OpenFlow, tune into SDN Connect Live on September 18th.  Meru Networks and Tech Field Day will have a roundtable discussion about Lync, featuring Lync experts and SDN practitioners.

by networkingnerd at September 15, 2014 04:37 PM

Packet Pushers Blog/Podcast

Viewing HTTP Headers Using Browser Developer Tools

I often find myself viewing HTTP headers (request and response) at the ‘client side’, which are  often much quicker (and safer) than decrypting SSL/TLS traffic on a load balancer/ADC. With the use of SSL/TLS growing rapidly even within private networks and the inability to decrypt PFS/DHE protected traffic, this can often be the only way to troubleshoot. The reasons I […]

Author information

Steven Iveson

Steven Iveson

Steven Iveson, the last of four children of the seventies, was born in London and has never been too far from a shooting, bombing or riot. He's now grateful to live in a small town in East Yorkshire in the north east of England with his wife Sam and their four children.

He's worked in the IT industry for over 15 years in a variety of roles, predominantly in data centre environments. Working with switches and routers pretty much from the start he now also has a thirst for application delivery, SDN, virtualisation and related products and technologies. He's published a number of F5 Networks related books and is a regular contributor at DevCentral.

The post Viewing HTTP Headers Using Browser Developer Tools appeared first on Packet Pushers Podcast and was written by Steven Iveson.

by Steven Iveson at September 15, 2014 01:02 PM

Cisco IOS Hints and Tricks

SIGS & Carrier’s Lunch DC Day: An Event Definitely worth Visiting

I spent last Tuesday in Bern attending the SIGS DC Day Event, and came back home extremely pleasantly surprised. The conference was nice and cozy, giving everyone plenty of opportunities to chat about data center technical challenges (thanks for all the wonderful conversations we had – you know who you are!).

Having the opportunity to meet fellow networking engineers and compare notes is great, but it’s even better to combine that with new knowledge, and that’s where the event really excelled.

Read more ...

by Ivan Pepelnjak ( at September 15, 2014 08:12 AM

XKCD Comics

September 12, 2014

Honest Networker
Networking Now (Juniper Blog)

Recap of VMworld 2014 USA - Juniper Style

This was the first year that I got to attend VMworld as a member of the Juniper family ( this was my fourth VMworld ). It was a great experience, we had our first lab in the Hands-on Lab which I personally think was a success and of course we had a booth. We received a lot of complements on the documentation for the lab and how it explained all the facets of the product. I had people fighting ( well not literally ) for the long sleeve shirts that we distributed to everyone who took the lab ( check it out below )



juniper lab shirt.JPG


It gave us a lot of visibility into our virtual security solutions and how they play in your VMware environment. The great thing, the fun isn't over…

A) VMware will make the labs available online in approximately 2 - 3 weeks so you can take them from the comfort of whatever you find comfortable, whenever you want to take it. The link is .
In the meantime, if you are interested in reading the lab that I wrote, it is available in PDF format and html format
2) We will be at VMworld 2014 Europe in Barcelona. Sadly we won't have shirts but the lab will be there and I promise to give you a hug if you take the lab. Hugs are better anyway.
The lab hours this year are : 
Monday / October 13 : 8:00 - 18:00
Tuesday / October 14 : 10:30 - 18:30
Wednesday / October 15 : 8:00 - 18:00
Thursday / October 16 : 8:00 - 18:00
I look forward to seeing you there!

by banksek at September 12, 2014 06:46 PM

Packet Pushers Blog/Podcast

Network Break 16

This week, EVO:RAIL & Converged thingies, Cisco's multiple SDN strategies, Don't be a precious snowflake and Congress open source project.

by Packet Pushers Podcast at September 12, 2014 05:22 PM

Renesys Blog

Sprint, Windstream: Latest ISPs to hijack foreign networks

Last year my colleague Jim Cowie broke a story about routing hijacks that resulted in Internet traffic being redirected through Iceland and Belarus. Unfortunately, little has changed since then and the phenomenon of BGP route hijacking continues unabated and on an almost daily basis.

In the past three days, the situation has gone from bad to downright strange as we have observed a flurry of this activity. Now, for the first time, we’re seeing major US carriers, Sprint and Windstream, involved in hijacking, along with the return of an operation out of Poland targeting Brazilian networks. We see router misconfigurations regularly in BGP data – could these incidents also be explained by simple command-line typos?

Route hijacking continues

In May my colleague, Earl Zmijewski gave a presentation about routing hijacks at the LINX 85 meeting, describing a comprehensive system that can be used to identify suspicious hijacks on a global basis and without any prior knowledge about the networks involved. While we now detect suspicious routing events on an almost daily basis, in the last couple of days we have witnessed a flurry of hijacks that really make you scratch your head.

To mention a few recent events, last month we tweeted about the Korea Meteorological Administration mistakenly hijacking a prefix from the US National Climatic Data Center, which hosts websites like and, thereby making these sites and many others unreachable to most of the Internet. And then we saw a company in South America trying to hijack back its address space from another entity that was squatting on it.

US carriers hijacking foreign networks

From 13:56 UTC on Tuesday (9-September) to 15:56 UTC on Wednesday (10-September), US wireless carrier Sprint started hijacking a prefix ( from Telesmart, an ISP in Macedonia. About 65% of our peers believed that Sprint was the origin of this prefix and so redirected Macedonian traffic for it to Sprint.

What was interesting was that once traffic arrived at Sprint, it continued onto Cogent and finally onto its intended destination at Telesmart in Skopje. Was this an accidental man-in-the-middle or something else? That is, probes from most of our servers around the world suddenly started going through Sprint to reach Internet resources hosted at Telesmart, such as news and municipal websites. Below is one of the more striking examples of the impact on traffic paths. Before the hijack, traffic from Sofia would reach Skopje in about 6ms, as these cites are fairly close.

trace from Sofia, Bulgaria to Telesmart, Skopje, Macedonia at 03:15 Sep 02, 2014
1 *
2    (Neterra, Sofia, Bulgaria)      0.339ms
3 (NTT Europe, Bulgaria Facility) 9.446ms
4 (NTT Europe, Bulgaria Facility) 6.883ms
5  (Telesmart, Skopje, Macedonia)  6.216ms

During the hijack, traffic took the scenic route through Frankfurt, Paris, Frankfurt (again), Munich, Vienna, Sofia (again), then on to Skopje and consequently took about 10 times longer to reach its intended destination.

trace from Sofia, Bulgaria to Telesmart, Skopje, Macedonia at 21:31 Sep 09, 2014
1 * 
2        (Neterra, Sofia, BG)                        10.682ms
3                   0.512ms
4                  34.056ms
5     (GTT, Frankfurt, DE)                        53.026ms
6          49.62ms
7         50.087ms
8      46.84ms
9   48.43ms
10       48.859ms
11       52.757ms
12       59.008ms
13       77.427ms
14        80.141ms
15     (Telesmart, Skopje, Macedonia)              48.916ms

For 26 hours, traffic to this Telesmart network in Skopje entered the Sprint network at various global locations (from Frankfurt to San Francisco) and was then handed off to Cogent in Paris en-route to Skopje. As we often are in these cases, we’re left wondering, what the heck was that? Sprint is a big network, but AS1239 doesn’t originate many prefixes outside the US and doesn’t originate anything starting with 95.x.x.x or 59.x.x.x or anything else that is typographically close to this prefix. Thus, this doesn’t look like an obvious and easily explained typo, something that we see all too often. Not only did traffic to this prefix get much slower, such diversion subjected a small amount of regional traffic to the risk of foreign interception — something on the minds of nearly everybody in the post-Snowden era.

But Sprint wasn’t the only US carrier to begin an unusual hijack on 9 September: US provider Windstream (AS7029) began announcing (SaudiNet), which is normally announced by Saudi Arabian incumbent, Saudi Telecom. Unlike the previous Sprint example, traceroutes to this prefix along the Windstream route died within Windstream, effectively knocking this network off the Internet for anyone accepting the bogus route. Then on Wednesday, Windstream announced a handful of strange routes for about 10 hours including one from Gaza, one from Iceland, and three from China — all more-specifics of existing routes, ensuring their global propagation and acceptance. These are listed below.

Hijack route    Organization label and location    Victim Route   Victim AS    Hadara Gaza BSA - 3ed subnet, PS    Hadara Technologies, AS15975 Advania hf., Reykjavik, IS Thor Data Center, AS50613 CHINANET Guangxi province network   China Telecom, AS4134 CHINANET Jiangxi province network  China Telecom, AS4134  CHINANET Huzhou node network  China Telecom, AS4134

Of the four examples above, only Advania of Iceland (AS30818) tried to get their traffic back by announcing the same more-specific as Windstream. Advania’s attempt to reclaim their network started at 15:06 UTC, at which point Windstream pulled their errant announcement. In all other cases, the hijacks continued until 15:49 UTC. Unlike the previous Sprint example, any traffic headed to computers in these IP address ranges would have been directed to Windstream during this time and then dropped, effectively knocking these networks off the Internet.

There is a potentially innocent explanation to this example. Perhaps, these address ranges were ones that Windstream deemed to be sources of bad traffic and so was “blackholing” them internally, a relatively common practice. In this scenario, we could have simply witnessed Windstream inadvertently leaking internal routes to the global Internet for 10 hours.

Polish hijacking of Brazilian IP address space

In another recent hijacking incident, starting at 03:16 UTC on 6 August, two different ISPs in Poland each started hijacking a prefix each from two different ISPs in Brazil. These hijacked routes were only circulated to a small number of other ISPs in Eastern Europe.

The two prefixes, and, were originated by INEA S.A. (AS33868) and INOTEL S.A. (AS44514), respectively. Once originated, they were both routed through the same upstream provider, another INEA S.A. ASN (AS13110) before going to Hurricane Electric (AS6939) and then onto a handful of providers in Eastern Europe. Routes took the following forms.

Normal route Hijacked route     … 3549 52770     … 6939 13110 33868 … 4230 28573 … 6939 13110 44514

Traceroutes from Eastern Europe show traffic paths redirected into INEA S.A. before continuing onto their proper destination.


trace from Warsaw, Poland to Revo Telecomunicações at 00:35 Aug 05, 2014
1 *
2   (GTS Poland Sp. z o.o., Warsaw, PL)             0.952
3     (GTS Czech s.r.o.,Prague, CZ)                   17.065
4 * 
5             17.222
6           25.666
7    (Revo Telecomunicações, Porto Alegre, BR)       248.358
8     (Revo Telecomunicações, Guarulhos, BR)          251.24
9      (Gm Telecom Ltda, Palmitinho, BR)               251.609

During (note the new INEA S.A. hops through Poznań, Poland):

trace from Warsaw, Poland to Revo Telecomunicações at 17:10 Aug 06, 2014
1  *
2  (GTS Poland Sp. z o.o., Warsaw)                 1.119ms
3   (GTS Poland Sp. z o.o., Warsaw)                 0.397ms
4                              5.407ms
5, Poznań, PL) 5.407ms
6, Poznań, PL)     5.608ms
7   (Level 3, Frankfurt, DE)                        27.76ms
8               27.74ms
9             27.65ms
10             27.743ms
11           27.854ms
12   (Revo Telecomunicações, Porto Alegre, BR)       249.385ms
13   (Revo Telecomunicações, Porto Alegre, BR)       248.138ms
14 *

At this year’s Defcon conference, Luca ‘kaeso’ Bruno and Mariano ‘emdel’ Graziano gave a talk about the vulnerabilities of some ISPs’ public looking glass utilities that would allow an attacker to remotely modify router configurations. INEA S.A. (AS33868) was on their list of vulnerable ISPs.

The hijacks described above ceased at the end of August. However, on Wednesday we saw some new suspicious activity from AS13110. At 00:14 UTC on 10 September, AS13110 hijacked a prefix from the US Department of Defense (, normally announced by AS27064). At 00:31 UTC the hijack had stopped, but then a few minutes later an ASN downstream of AS13110 began hijacking a new Brazilian prefix (, TecleNet Solucoes Tecnologicas) at 00:35 UTC. Was the DoD hijack an initial test before the real target was selected?

The above graph shows the percent of our peers accepting the bogus origin (AS57807) compared to the legitimate Brazilian origin (AS57807) over a day and a half of this activity. Then this hijack disappeared yesterday morning at 14:13 UTC while I was writing up this blog post.

And there is IP squatting to worry about

On 31 August, responding to an inquiry on the NANOG mail list, I wrote a description of a rather unique IP squatting operation out of Russia that briefly announces (mostly unused, sometimes not) IP address space assigned to others and originates these via two unused AS numbers. To be sure, IP squatting is not a new phenomenon. Nick Feamster and Anirudh Ramachandran from Georgia Tech presented on the use of IP squatting for spam operations at NANOG 36 in 2006.

Starting in late June, we have observed dozens of prefixes announced along AS paths of the following forms (origins are the rightmost ASN) each for no more than a couple hours at a time.

    … 39792 57756 {197329 or 43239}    prefix

    … 44050 57756 {197329 or 43239}    prefix

Since 30 August, these routes have taken the following form:

    … 44050 197598 {49121 197794 or 201910}    prefix

Below is a graph of the bogus route originations of this IP squatting operation on a recent day. Prefixes and origin ASNs are only up for a few hours at a time. AS201910 is the newest formerly unused ASN being used as a bogus origin. On Wednesday of this week, it announced three prefixes of unused IP address space:  Bundesministerium fuer Land- und Forstwirtschaft  AT  DuoDecad IT Services Luxembourg S.a r.l.  Luxembourg  Amadeus Data Processing GmbH  Bayern DE


While this IP squatting operation is rather unique in its methodology, there are far more examples of IP squatting that don’t involve brief prefix announcements from odd ASNs. Perhaps, an amusing on-going example is Bitcanal (AS197426) of Portugal, which originates unused IP address space from a number of locations ranging from Syria (, AYA Internet Service Provider) to Texas (, Office of the Attorney General, State of Texas). (Would someone please alert Texas Governor Rick Perry about that last one?)

The above graphs illustrate that the Syrian prefix has been globally routed out of Portugal since 25 August, while the new home of the Texas prefix has been seen by about 40% of our peers since 19 June.


So what is the story with all of this?

Well, for one, sloppiness abounds. Humans are at the controls and are always capable of mucking things up. For example, British Telecom Latin America (BT Latam, AS7908) has been globally announcing since April 2012. The route is technically a hijack of, which is announced by China Telecom but looks like a leaked internal route. Or take Beltelecom of Belarus, who has been globally announcing 13 prefixes within RFC6598 address space (such as and intended for internal carrier grade NAT operations since January of this year.

It is interesting to note that the Bitcoin hijack from earlier this year originated its bogus routes using two formerly unused ASNs with a common upstream AS path. The Russian IP squatting operation also typically uses two unused ASNs for its origination. Finally, the Polish hijacking of the Brazilian routes mentioned above also employed two different ASNs, although not unused this time. Perhaps it is a coincidence, or perhaps it is a known tactic: when hijacking multiple routes, it is better to spread out one’s bogus routes across multiple origins to lower the profile of any single misbehaving ASN.

Regardless of the cause of each of these incidents, the problem is a very real and growing one. Perhaps documenting these incidents will promote a greater understanding of the extent and nature of the problems around the trust-based Internet routing system in global use today.

The post Sprint, Windstream: Latest ISPs to hijack foreign networks appeared first on Renesys.

by Doug Madory at September 12, 2014 03:15 PM

Aaron's Worthless Words

FEMA and Your Business Continuity Plan

I passed the ROUTE exam a few days/weeks/months/something ago and decided to pursue certifications of another sort for a while. The wife and I are trying our best to help the community through our ham radio training, so I decided to go down that path a bit further. One thing I was interested in doing is to do EmComm during declared emergencies. That meant I had to take two FEMA courses online to be allowed in the EOC. I thought they would be terribly boring, but I found them to be quite familiar.

The first course was on the Incident Command System (ICS). The main idea is that, in the event of an emergency of any kind or size, an Incident Commander (IC) is assigned to be responsible for the recovery effort.  This mean analysis of the incident, generating an action plan (a very key component), and execution of said plan. If the IC can complete the action plan by himself, then off he goes. If he or she needs some additional resources like people or equipment, then he or she is empowered to draft that help from any entity that’s involved.

Another one of the big points of ICS is that a disaster response is distanced from day-to-day work.  That is, when you’re working in a response capacity, your day-job boss isn’t your boss (unless he happens to be your manager in the ICS).  ICS also gets rids of the whole dotted-line organizational structure.  This is called unity of command and supports a single objective work model.  No more being yelled at by 7 managers for not have a cover page on your TPS report.

Does this not sound like a great system reacting to a DR event? The IC in this case could be a product owner and that person would evaluate the situation, create a detailed action plan, draft personnel as needed, etc. It’s the IC’s job to figure out what needs to be done and what resources are need. If he or she calls you, you report to him or her today. Your day-to-day boss can’t pull you off to work on other things; this is a response situation that the business has blessed and is deemed a higher priority than your daily work. Very powerful stuff.

The second course I took was and introduction to the National Incident Management System (NIMS). The whole point of NIMS is to create a system that supports intra- and inter-departmental communication. At a very high level, this means using plain language and putting things in writing beforehand. We’ve all been told to “talk to Bill about that” and Bill tells you to call Mary.  NIMS dictates that we should have a document that says exactly what each is responsible for during a response. It tells us what Bill and Mary are supposed to be doing so we don’t have to guess. It’s better to know what everyone is doing than to guess or, even worse, assume what everyone is doing.

What if your product owner says we need to roll to the alternate data center? What does that really mean? Will most of your time be spent flopping around like a fish because you don’t know what’s going on? Do you have a list of outside entities (like your ISP or VAR) whose help will be needed during a DR incident? How about a definition of success? Do you have a plan to roll back when your recover from the disaster? Do you have a list of of tasks to get operations rolling at the alternate data center? Do you need to get your marketing department to send out messages to customers or update your website? Do you even have the criteria for declaring a disaster? Can I ask any more questions? NIMS tells us that we need to answer all those questions (and more) BEFORE the disaster happens. Let me say that again — BEFORE the disaster happens.

I recommend you take the ICS and NIMS training from FEMA. It’s obviously going to be focused on physical disasters, but the ideas apply pretty broadly. Each class takes 3 or 4 hours to finish and is broken up into sections, so you don’t have to do it all at once. There’s an exam at the end, but that’s obviously not a requirement (unless you need the cert like I did). The training is absolutely free and doesn’t require you to log in or anything.  And get your ham radio license.  :)

Send any new HTs questions my way.

by Aaron Conaway at September 12, 2014 02:04 PM

Cisco IOS Hints and Tricks

Tech Talks: The Essence of MPLS

Seamus Gilchrist sent me a fantastic list of MPLS- and MPLS-TE-related questions. Instead of starting an email exchange we agreed on something that should benefit a wider community: a lengthy whiteboard session discussing the basics of MPLS, MPLS-TE, load balancing and QoS in MPLS networks…

The first part of our conversation is already online: The Essence of MPLS.

by Ivan Pepelnjak ( at September 12, 2014 08:59 AM

XKCD Comics

September 11, 2014

Honest Networker
SNOsoft Research Team

How we breach retail networks…

We recently delivered an Advanced Persistent Threat  (APT) Penetration Test to one of our customers. People who know us know that when we say APT we’re not just using buzz words.  Our APT services maintain a 98% success rate at compromise while our unrestricted methodology maintains a 100% success at compromise to date.  (In fact we offer a challenge to back up our stats.  If we don’t penetrate with our unrestricted methodology then your test is free. If we do get in then you pay us an extra 10%.)  Lets begin the story about a large retail customer that wanted our APT services.

When we deliver covert engagements we don’t use the everyday and largely ineffective low and slow methodology.  Instead, we use a realistic offensive methodology that incorporates distributed scanning, the use of custom tools, zero-day malware (RADON) among other things.  We call this methodology Real Time Dynamic Testing™ because it’s delivered in real time and is dynamic.  At the core of our methodology are components normally reserved for vulnerability research and exploit development.  Needless to say, our methodology has teeth.

Our customer (the target) wanted a single /23 attacked during the engagement. The first thing that we did was to perform reconnaissance against the /23 so that we knew what we were up against.  Reconnaissance in this case involved distributed scanning and revealed a large number of http and https services running on 149 live targets.  The majority of the pages were uninteresting and provided static content while a few provided dynamic content.

While evaluating the dynamic pages we came across one that was called Make Boss. The application was appeared to be custom built for the purpose of managing software builds. What really snagged our attention was that this application didn’t support any sort of authentication.  Instead anyone who visited the page had access to use the application.

We quickly noticed that the application allowed us to generate new projects.  Then we noticed that we could point those new projects at any SVN or GIT repo local or remote.  We also identified a hidden questionable page named “list-dir.php” that enabled us to list the contents of any directory that the web server had permission to access.

We used “list-dir.php” to enumerate local users by guessing the contents of “C:\document~1” (Documents and Settings folder). In doing so we identified useful directories like “C:\MakeBoss\Source” and “C:\MakeBoss\Compiled”.  The existence of these directories told us that projects were built on and fetched from same server.

The next step was to see if in fact we could get the Make Boss application to establish a connection with a repository that we controlled.  To do this we setup an external listener using netcat at .  Then we configured a new project called “_Netragard” in Make Boss in such a way that it would connect to our listener. The test was a success as is shown by the redacted output below.

[titon@netragard:~]$ nc -l -p 8888 -v
listening on [any] 8888 …
xx.xx.xx.xx: inverse host lookup failed: Unknown server error : Connection timed out
connect to [xx.xx.xx.xx] from (UNKNOWN) [xx.xx.xx.xx] 1028
User-Agent: SVN/1.6.4 (r38063) neon/0.28.2
Connection: TE, Keep-Alive
TE: trailers
Content-Type: text/xml
Accept-Encoding: gzip
DAV: 104
Accept-Encoding: gzip
<?xml version=”1.0″ encoding=”utf-8″?><D:options xmlns:D=”DAV:”><D:activity-collection-set/></D:options>


With communications verified we setup a real instance of SVN and created a weaponized build.bat file.  We selected the build.bat because we knew that Make Boss would execute the build.bat server-side and if done right we could use it to infect the system. (A good reference for setting up subversion can be found here  Our initial attempts at getting execution failed due to file system permissions.  We managed to get successful execution of our build.bat by changing our target directory to “C:\TEMP” rather than working from the standard webserver directories.

With execution capabilities verified we modified our build.bat file so that it would deploy RADON (our home-grown 0-day pseudo-malware).  We used Make Boss to fetch and run our weaponized build.bat, which in turn infected the server running the Make Boss application.  Within seconds of infection our Command & Control server received a connection from the Make Boss server.  This represented our first point of penetration.

A note about RADON…

RADON is “safe” as far as malware goes because each strand is built with a pre-defined expiration date.  During this engagement RADON was set to expire 5 days after strand generation.  When RADON expires it quietly and cleanly self-destructs leaving the infected system in its original state which is more than what can be said for other “whitehat” frameworks (like Metasploit, etc).

RADON is also unique in that it’s designed for our highest-threat engagements (nation-state style).  By design RADON will communicate over both known and unknown covert channels.  Known channels are used for normal operation while covert channels are used for more specialized engagements.  All variants of RADON can be switched from known to covert and visa-versa from the Command & Control server.

Finally, it’s almost impossible to disrupt communication between RADON and its Command & Control center.  This is in part because of the way that RADON leverages key protocols that all networks depend on to operate.  Because of this, disrupting RADON’s covert channels would also disrupt all network functionality.

Back to the hack…

With the system infected by RADON we were able to take administrative control of the Make Boss server.  From there we identified domain administrator credentials that the server was happy to relinquish. We used those credentials to authenticate to and extract all current and historical passwords from the domain controller.   Then we used one of our specialized GPU password cracking machines to process the hashes and deliver us the keys to the kingdom.

With that accomplished we had established dominant network position. From this position we were able to propagate RADON to all endpoints and affect an irrecoverable network compromise.  Irrecoverable if we were the bad guys of course, but luckily we’re the good guys and our customer recovered just fine.  Never the less we had access to everything including but not limited to desktops, points of sale, web servers, databases, network devices, etc.

Not surprisingly our customers managed security service provider didn’t detect any of our activity, not even the mass infection.  They did however detect what we did next…

As a last step and to satisfy our customer we ran two different popular vulnerability scanners.  These are the same scanners that most penetration testing vendors rely on to deliver their services.  One of the scanners is more network centric and the other combines network and web application scanning.  Neither of the scanners identified a single viable vulnerability despite the existence of the (blatantly obvious) one that we exploited above.  The only things that were reported were informational findings like “port 80 open”, “deprecated SSL”, etc.

It’s really important to consider this when thinking about the breaches suffered by businesses like Hannaford, Sony, Target, Home Depot and so many.  If the penetration tests that you receive are based on the product of vulnerability scanners and those scanners fail to detect the most obvious vulnerabilities then where does that leave you?  Don’t be fooled by testers that promise to deliver “manual penetration tests” either.  In most cases they just vet scan reports and call the process of vetting “manual testing” which it isn’t.

by simon at September 11, 2014 09:03 PM

Honest Networker
My Etherealmind
Honest Networker
Cisco IOS Hints and Tricks

Open-Source Hybrid Cloud Reference Architecture on Software Gone Wild

A while ago Rick Parker told me about his amazing project: he started a meetup group that will build a reference private/hybrid cloud heavily relying on virtualized network services, and publish all documentation related to their effort, from high-level architecture to device and software configurations, and wiring plans.

In Episode 8 of Software Gone Wild Rick told us more about his project, and we simply couldn’t avoid a long list of topics including:

Read more ...

by Ivan Pepelnjak ( at September 11, 2014 08:19 AM

September 10, 2014

Cisco IOS Hints and Tricks

IPv6 Neighbor Discovery (ND) and Multicast Listener Discovery (MLD) Challenges

A few days ago Garrett Wollman published his exasperating experience running IPv6 on large L2 subnets with Juniper Ex4200 switches, concluding that “… much in IPv6 design and implementation has been botched by protocol designers and vendors …” (some of us would forcefully agree) making IPv6 “…simply unsafe to run on a production network…

The resulting debate on Hacker News is quite interesting (and Andrew Yourtchenko is trying hard to keep it close to facts) and definitely worth reading… but is ND/MLD really as broken as some people claim it is?

Read more ...

by Ivan Pepelnjak ( at September 10, 2014 08:13 PM

Honest Networker
Networking Now (Juniper Blog)

September 2014 Microsoft Patch Tuesday Summary

Welcome to the September edition of Microsoft Patch Tuesday Summary. In this edition there are 4 updates; one is marked "Critical" and three are rated "Important". A total of 42 vulnerabilities were fixed over 4 bulletins this month. One of the Critical update MS14-052 is an all version Internet Explorer (IE 6 to 11) patch. This single update resolves 37 CVE's (Common Vulnerability and Exposure) including the publicly disclosed CVE-2013-7331


Here is a list of Security bulletins which were rolled out in today's Patch Tuesday release.

by prashantk at September 10, 2014 04:58 AM

XKCD Comics

September 09, 2014

Peter's CCIE Musings and Rants

vSwitch Updates

vSwitch Updates:

vMotion support for networks with less than 100ms RTT, vMotion between "Data Centers" (not literal data centers, just the VMWare construct called "Data center") and "officially" supporting vMotion over routed networks (which has been happening for a while but was never officially supported.)

by peter_revill ( at September 09, 2014 09:34 PM

Packet Pushers Blog/Podcast

Show 204 – Reducing Your Attack Surface with Avaya Stealth Networks – Sponsored

"The problem with ‘covering your tracks’ in network security is that your ‘covering’ becomes more conspicuous than your ‘tracks’," says Ed Koehler, Distinguished Engineer for Avaya’s Networking Division. Ed joins Greg Ferro and Ethan Banks for a ninja nerd-fest outlining a set of technologies that not only offer some innovative ways to set up your security architecture, but also simplifies the way that you do it. The Packet Pushers team specifically discusses with Ed how to segment for greater anomaly identification, how to streamline your firewall strategy, and how an ISID-VLAN-VRF combination can create truly independent stealth networks. Ed also outlines how customers are using this technology in healthcare, government, retail, and transportation to comply with industry regulations such as PCI and HIPAA. This show gets fairly nerdy, digging into L2 and L3 segmentation and exactly how that gets done with Shortest Path Bridging in the Avaya implementation. Ed is an unapologetic details guy, and the conversation gets into the weeds (in the best possible way) regarding just how frames traverse an SPB fabric in the context of "stealth." Links 802.1aq (Wikipedia) Ed's blog Ed's YouTube channel Avaya - Stealth Networks Overview (PPT - right-click and "Save Link As...") Avaya Technology Forum - Stealth PCI Networking Presentation (PPT - right-click and "Save Link As...")

by Packet Pushers Podcast at September 09, 2014 03:52 PM

Networking Now (Juniper Blog)

Security for the Cloud Data Center with Juniper Spotlight Secure Threat Intelligence Platform

In an earlier blog, I posed a few questions on security challenges that some Cloud Builders are facing today. Here, I offer some ideas for you to consider.

by skathuria at September 09, 2014 12:00 PM

Cisco IOS Hints and Tricks

vMotion Enhancements in vSphere 6

VMware announced several vMotion enhancements in vSphere 6, ranging from “finally” to “interesting”.

vMotion across virtual switches. Finally. The tricks you had to use previous were absolutely bizarre.

Read more ...

by Ivan Pepelnjak ( at September 09, 2014 08:40 AM