Dealing with Cisco Router Vulnerabilities

This past week, Cisco posted three security advisories about IOS-based devices (almost all Cisco routers, most of their switches, and a few other types of devices). Each of these advisories basically noted that with a single specially "crafted" packet, one could at least crash their devices, and possibly execute arbitrary code on them. Presumably the issue is buffer overflows in infrequently-used parts of the packet parsing logic. One of the advisories is about TCP packets, another about IPv6 routing headers, and another about IP options. This seems to be exactly the kind of problems that Mike Lynn tried to point out in 2005. Since we have many IOS-based devices (more than 100), we have to upgrade them to non-vulnerable versions as soon as possible.

In order to find out which of our devices need to be upgraded, I first started to write a Perl script that gets the installed IOS version of each device from our RANCID repository, and then checks whether the version is vulnerable. When I wrote the function that checks an IOS version for vulnerability (starting from the table of vulnerable/fixed IOS versions in one of the advisories), I noticed that this was a lot of work, and would have to be repeated for each of the three advisories.

So I decided to go one step further in automation, and write another Perl script that parses a Cisco security advisory in HTML format, and extracts the IOS-version vulnerability information from that. Being a Lisp programmer, I thought that the most natural output for this parser would be a function which would take an IOS version at the input and give either "this version is safe" or an upgrade recommendation as output.

Then the original Perl script would call out to this generated function for each router. It took me almost two days (with lots of interrupts from other work) to write this until I was happy with the result. I now have a crontab entry that runs the script several times per working day and e-mails me a report with a summary (N routers vulnerable, M routers should be upgraded) and a detailed entry for each router. Current status: 31 routers are safe, 80 should be upgraded. Oh well...

The scripts work OK, but aren't documented and have to be adapted for other RANCID installations. If you want them anyway, just send me mail (simon "at" switch "dot" ch).

The advisory-parsing script was the more interesting to write. I decided to use HTML::TreeBuilder to turn the HTML page into a DOM-like tree. That turned out to be a good decision, because that module makes it easy to grovel around in those trees. Outputting the Perl code was quite ugly, and made me want to use Lisp, where this is much more natural. I incrementally added parsers for all cases of vulnerable/fixed version information, and found that this worked quite well, with very few exceptions remaining unhandled. I found a few bugs (inconsistencies or missing information about specific versions) in the advisories as well, but I'm too lazy to find out where to send the reports.

Now we have to decide which IOS versions to upgrade to, schedule maintenance for rebooting the routers, open tickets, and perform the actual upgrades. Each of these tasks could benefit from some automation as well.

One difficulty is to decide which IOS image is suitable for a device (that has to be upgraded). This can depend on many factors, including device type, installed line cards, chassis type, available memory (RAM, flash/disk). This cannot be completely automated, because the set of usable images depends on factors that are hard to formalize, such as experience with these versions in testing and operations, weighted against the amount of risk one can take on a given router, depending on whether it's a redundant router or a single point of failure for a customer, etc. It would be great if one could extract an image's installation requirements from Web pages, but I don't even know how to get from image version to CCO Web page location.

Another part of the job to actually copy the desired IOS image to a suitable flash/disk partition on the router, and chance the boot configuration so that the new version comes up. This is quite a bit of work (especially if you have to do it 80 times), and should be easy enough to automate.

Scheduling good times for the reboots is something where I'm not sure whether automation makes a lot of sense. Many of our routers are completely protected by redundant routers, so they can basically be rebooted at any time. On the other hand, there are a few routers that lack out-of-band access, and one might want to defer upgrading those routers until one has collected experience with upgrades on similar routers that do have out-of-band, just to minimize the risk that something goes wrong.

Opening the tickets will be a lot of work too. A colleague from our team recently rewrote the ticket system in PHP, and I should ask her whether we can somehow write scripts to create tickets in bulk.

Van Jacobson on Content-Centric Networking

I somehow found a Google Tech Talk by Van Jacobson on Google Video, with the modest title of "A New Way to look at Networking". Van Jacobson is a personal hero of mine, to the point that I have created a "fan page" on a Web site that I help maintain.
In the talk, Van quickly reviews the evolution of thinking about networks from the circuit-oriented view of the telephone world to the endpoint-to-endpoint-conversation-oriented view of the packet networking and Internet world. He argues that it's time to change the perspective once again, and focus on documents (content) and their dissemination.
After sending a pointer to the TF-NGN mailing list, I received positive feedback, as well as some points for discussion. (Someone also gave me the hint that Bill St. Arnaud had mentioned the video in his blog a week earlier!)
Interesting stuff!

Blogger and "Google Apps for Your Domain"

The new features in Blogger look nice, although not mind-bogglingly so.
One thing I miss (or maybe I just missed it) is the possibility to publish Blogs under an arbitrary domain name, i.e. http://blog.leinen.ch/. This could be easily achieved by integrating Blogger into Google Apps for Your Domain, which can already host Web pages, GMail accounts, and personalized "intranet" sites under arbitrary domain names.

UPDATE: Hey, Google is reading my blog! It only took'em until January 5 to announce "Blogger Custom Domains". So as of today, this blog is available under my own domain, blog.simon.leinen.ch.

WLAN attenuation (in hotel rooms)

I stumbled across an article "Redecorating Your Guestrooms May Spell Disaster for Wireless Signals"
by Bill Halpin, on the site of Core Communications, a company that builds WLANs for hotels and conventions. It includes a nice table of various kinds of stuff that is used for hotel rooms, along with their respective expected attenuation factors (in %) for WLAN signals. Silvered mirrors and other chrome or metal objects are worst. Some of this information might be useful for people other than hotel managers.
UPDATE: Core Communications was acquired and has become Swisscom Hospitality Services. To match the Swisscom look & feel, useful information such as the above article probably had to be removed.

The Korean Mini-Hompy

Everybody is blogging, so I finally decided to start a public weblog of my own when I stumbled over the Blogger icon on Google. Maybe I'll move this to another server soon, possibly my own.

A news item that fascinated me last week was this:

http://times.hankooki.com/lpage/exclusive/200411/kt2004112518452663570.htm

In Korea, eleven million people (as of November 2004) subscribe to a service called "Cyworld", which allows them to post personal "Mini Hompys", or mini-homepages. It seems to be a convincing combination of a homepage/blog hosting service, social software (personal homepages can somehow be interlinked), with the possibility of uploading images from camera phones.

But the fact that 11 million South Koreans use this service is absolutely stunning to me - this is more than a quarter of the country's total population, and I assume that in many subclasses of the population (age and/or social groups), penetration must be so close to 100% that it will be considered weird if you're not a subscriber.