Partial Internet outage 11/12/08 4:24pm to 4:45pm

We noticed a brief loss of connectivity to some destinations on the Internet this afternoon.   The problem occured in a portion of the Verizon network and affected traffic to some popular destinations such as CNN, MySpace, and Facebook.     The problem cleared while we were analyzing the situation and deciding on a course of action.

Numerous network operators are reporting the problem on outage mailing lists.   Verizon has not issued a statement at this time.   The rumor mill is pointing the finger at Level3 claiming bad announcements from Level3 (another very large network).

So how does all this work you ask?  (or the really short intoduction to BGP).

The Internet is not a single entity but rather a collection of independent networks connected together.  The networks connect to each other at gateway routers.   The gateway routers speak a language (actually a protocol) called BGP where they announce to each other what networks (and destinations) are available by sending traffic through the gateway.

Amplex maintains connections to two large networks (Verizon and Cogent) and we recieve information from both telling our router the fastest way to deliver traffic to it’s destination.   Should a network cease to be able to carry traffic to a particular destination (say MySpace) the neighbor router is supposed to ‘withdraw’ it’s offer to carry traffic to that destination.    When that happens, if we still have a route to the requested destination via our other connection, we will send data out the working connection.   Sometimes the route is withdrawn by both providers at the same time – this likely indicates that the destination network itself is no longer online.

In today’s outage Verizon continued to tell our router that the best path to MySpace, CNN, and other sites was to deliver the traffic to Verizon.   Unfortunatly Verizon was not keeping that promise but rather dropping the traffic inside it’s own network.    While that situation is not supposed to happen it does on fairly rare occasions.

Verizon will likely issue a ‘root cause analysis’ regarding the outage at a later date to explain to the routing engineers at other companies how and why this happened and how to prevent it in the future.

How could Amplex work around this problem?

We would shut down the connection to Verizon which then routes all traffic to Cogent.  Unfortunately this is not a decision to be made lightly since shutting down an upstream carrier causes our own announcements to the rest of the Internet to change.   There can be fairly long waits (and disconnections of existing VPN, Video, and other sessions) while the Internet determines the new best path to reach us.

Once we had established that the problem was at Verizon we were preparing to shut down the connection when the problem in Verizon’s network was resolved.