We have had reports of difficulty or slow response from Google today. This is related to maintenance performed by one of our upstream providers (Cogent) last night. In checking with Cogent they are telling us that they are continuing to have difficulty with the new software load in the Toledo router. While Cogent is resolving the issue we have shut down our connection to Cogent. Shutting down the connection to Cogent provides a work-around while they resolve the issue.
The problem is unusual in that Cogent (at least until we shut down to them) was advertising (telling us) that they have the best path to reach Google yet they are not delivering the traffic. This is the “Mortal Sin” of networking – advertising a route to a destination you can’t deliver to.
Unfortunately this is one of those issues that is hard for us to automatically detect. Until we hear from customers or notice it ourselves we don’t know about it. We apologize for the issue and thank the customers who notified us about it.
We are waiting for Cogent to notify us that the issue has been resolved before bringing the connection to them back up. Assuming the issue is resolved there should be no further downtime as the connection returns.
Update as of 4:00pm: Bringing Cogent back online resulted in the same issues we were seeing this morning. At this point further troubleshooting is going to need to wait until early AM hours. The inability to use the Cogent circuit tonight is going to lead to congestion on our other circuits. Service will work but will be somewhat slower this evening.