There are quite a few sensational stories in the media claiming that the government is gathering large amounts of data from network providers without warrants and/or abusing privacy. Some of the fuss is legitimate but for the most part it’s being blown out of proportion.
Lert’s start with how Amplex handles requests for data by government agencies:
- Any request for data of a transactional nature made by a governmental agency requires a valid subpoena. The request can specify that no notification be made to the target of the investigation for a certain amount of time. These requests are rare and very specific. There really isn’t anything to object to in these type of requests – it’s local or state law enforcement doing exactly what they are supposed to be doing. Amplex reviews these requests with our legal council to verify the authenticity of the request and complies as needed.
- Requests for stored data (email or other content) or any real time capture of data requires a search warrant, signed by a judge. Search warrants may specify that no notice be given to the affected party. Amplex reviews these requests with our legal council for authenticity and to narrow the scope of the request if the request is unreasonable, excessively broad, or technically unfeasible.
- National Security Letters (NSL). This is a special type of request for data from the federal government. This type of request contains a gag order prohibiting the disclosure of the contents of the letter, the requested data, etc. The Electronic Frontier Foundation has a good writeup of NSL’s. Amplex has never been presented with a National Security Letter (at least as of 6/8/2013).
On several occasions Amplex has been presented with a request from law enforcement for information outside the above processes when there was imminent danger. Amplex will cooperate with law enforcement in these situations when we believe the request is legitimate. Specific examples of this have included a bomb threat to a local school, and a person posting very specific threats online. Please note that our terms of service allow Amplex to cooperate with law enforcement agencies (LEA) in these situations. Is there a potential for abuse in this? Yes, but there are many legal options for Amplex to take if we discover that the process was abused or LEA intentionally deceived us.
So what are all the sensational stories about? What is the real story? We have a few ideas but first we need to discuss CALEA:
CALEA is a federal regulation that specifies a series of technical, legal, and management procedures by which LEA can gather needed data. The CALEA implementation in a private network is NOT controlled by the government. There are a considerable number of safeguards in the CALEA system to prevent unauthorized use and to limit the data captured to only the records specified in the accompanying search order.
PRISM is the system that hit the news this week. The system reportedly gives the government access to large amounts of data. The story doesn’t really add up. Why?
- The reported cost of the program is $20 million dollars. The federal government can’t fund a giant data gathering program with that money. $20 million barely pays for the consultant to design the program, much less implement it.
- The shear volume of data would be extremely difficult to capture and transport. Capturing all the data from a tiny network like Amplex would consume somewhere around 40TB of data per day. Getting that data transported off the network would be very expensive. There is no central point on the network where all data can be collected in any case.
- Large networks like Google, MSN, Facebook, are very decentralized. Large scale monitoring would require a massive investment in additional transport capacity. The networks have a hard enough time keeping up with their own growth, much less building a parallel network for the government.
- Most data transported on the Internet is asymmetric, meaning that the data path to the end user is not the same as the return path. Capturing data at mid-points in the network usually only gets you 1/2 the information. Trying to capture the data from multiple mid-points and putting it back together is very difficult. Capturing data only works well at the source or destination.
So what is PRISM really about? My best guess is that it’s a electronic system where the government can present CALEA requests along with the relevant legal search warrants, national security letters, etc. to the participating companies by electronic means rather than faxing, scanning, and/or using overnight delivery services to move paper around. That is about the only thing that $20M by the federal government will buy you. It also makes sense. Much of the legal system really is 15 years behind the rest of private industry and they still push mountains of paper around the country. The fed’s did something smart and built a secure communication system to deliver legal documents? Good for them. Perhaps they should have sent out a press releases instead of hiding it.
Are the feds overreaching with NSF letters and the PATRIOT act? Yes – and the Department of Justice has previously found major problems with the program. Are other government agencies abusing the program as well? I suspect we are about to find out.
Amplex experienced a network wide problem today at 11:25am. While we are still analyzing the logs we have a good idea of what caused the issue. The network experienced a broadcast storm and loop due to the failure of the mechanisms designed to prevent network loops.
We have seen this same issue twice in the past, approximately one month ago. In those cases the problem occurred late at night and was not noticed by most customers. Following the earlier occurrences we made several changes to the network to remove the lower bandwidth backup paths which caused a significant amount of instability. I can go into much more detail but it’s probably not worth discussing since the important part is…
What are we going to do about keeping it from happening again?
There are several steps we are taking to prevent the issue from occurring in the future:
- Installation of routers at tower sites. We are outgrowing the existing network layout (which has worked well for many years) and will be installing routers at the individual tower sites. This will significantly reduce the broadcast load on the network. We have avoided placing routers at tower sites in the past for reliability reasons. The advantages of individual tower routers now outweighs the risks. Installing routers is low risk and can be done with minimal impact on the network and customers. The first one will be installed at Luckey today.
- Splitting the network into 2 logical parts. The network consists of 2 rings that share a common path between Perrysburg and Lemoyne. The north ring primarily serves sites in Ottawa county, the south ring serves Wood county. We are adding an additional link between Perrysburg and Lemoyne and will use that to isolate the north and south rings. This will reduce the effective size of the network while also helping to isolate issues.
- Evaluating Performant Networks Software Defined Networking gear. Performant has designed a network appliance that promises to improve the stability and recovery time for Ethernet networks by incorporating ITU’s G.8032 “Ethernet Ring Protection Switching”. This standard and equipment allows for sub 50mSec failover in the event of breaks in an Ethernet ring. The Performant equipment adds an additional feature by continuously measuring the actual performance of the links so that it can make intelligent decisions based on the capacity of the individual links. Evaluating and installing this equipment is a long term project as the equipment is new and relatively untested. While it shows great promise we want to run it in a test environment for several weeks before attempting to deploy it.
We understand that a reliable network connection is very important to you and sincerely apologize for the issues today. If you have further questions please do not hesitate to contact us.
Mark Radabaugh, VP Amplex
We have had reports of difficulty or slow response from Google today. This is related to maintenance performed by one of our upstream providers (Cogent) last night. In checking with Cogent they are telling us that they are continuing to have difficulty with the new software load in the Toledo router. While Cogent is resolving the issue we have shut down our connection to Cogent. Shutting down the connection to Cogent provides a work-around while they resolve the issue.
The problem is unusual in that Cogent (at least until we shut down to them) was advertising (telling us) that they have the best path to reach Google yet they are not delivering the traffic. This is the “Mortal Sin” of networking – advertising a route to a destination you can’t deliver to.
Unfortunately this is one of those issues that is hard for us to automatically detect. Until we hear from customers or notice it ourselves we don’t know about it. We apologize for the issue and thank the customers who notified us about it.
We are waiting for Cogent to notify us that the issue has been resolved before bringing the connection to them back up. Assuming the issue is resolved there should be no further downtime as the connection returns.
Update as of 4:00pm: Bringing Cogent back online resulted in the same issues we were seeing this morning. At this point further troubleshooting is going to need to wait until early AM hours. The inability to use the Cogent circuit tonight is going to lead to congestion on our other circuits. Service will work but will be somewhat slower this evening.
A few customers have asked about the availability of ESPN3. ESPN3 is the sports network’s online video streaming service. To watch ESPN3 your service provider (in this case Amplex) has to pay ESPN for access. Why don’t we do that?
The simple answer is that we feel it is poor business model when applied to the Internet, and an incredibly slippery slope that will end badly for everyone.
ESPN’s plan is to recreate the Cable TV business model on the Internet. The Cable TV model is this:
- The networks (HBO, ESPN, ABC, etc.) negotiate deals with the cable company to carry the networks channels. The deal requires the cable company to pay X dollars per customer per month for the cable company to carry the networks programming.
- The contract specifies that all of the network’s channels must be carried, not just the popular ones. Why there are 50 junk channels? Because the contract says if you want to have Oprah you also have to pay for and carry our 12 other channels. Combine the junk channels from a dozen networks and you have Cable TV. 150 channels that you pay for, 3 that you actually watch.
Why does cable TV costs so much? It’s not because the cable companies are greedy. Ok, they probably are, but the bigger reason is that they are forced to pay ever increasing fees to the networks for content.
The ‘provider pays’ model is the way cable TV works. There are a lot of reasons that it should not be applied to the Internet:
- It raises our costs (and the cost of your Internet service) to pay for something the majority of customers do not watch.
- There are a huge number of sources of content on the Internet. The service provider negotiating with every content provider on the Internet is unworkable.
- Fees will escalate over time.
If ESPN’s model succeeds there is nothing to stop Netflix, Hulu, YouTube, or even Google from demanding the same type of business model. The day that happens the cost of Internet service is going to skyrocket. I can easily see content costs adding hundreds of dollars a month to the cost of Internet service. Cable TV is a naturally limited model in that the network can only carry a few dozen networks and a few hundred channels. The Internet is unlimited in the amount of content it can carry.
Do we really want to recreate a service where your content choices depend on the networks your service provider subscribes to?
ESPN refuses to sell a subscription directly to an end user.
If you would like ESPN to change this feel free to tell them about it: http://broadband.espn.go.com/espn360/watchespn/feedback
Amplex has used a couple of different radio frequencies to bring Internet service to our customers.Â The microwave frequencies around 5.7Ghz are used by the majority of customers who have line-of-sight to our towers.Â Â The 5.7Ghz equipment provides the highest speeds and capacity service.Â For customers that do not have line-of-sight we have used 900Mhz or 2.4Ghz to provide service.Â Â The 900Mhz equipment has not been as reliable, or as fast as we would like and we are working to replace as much of it as we can.
As Amplex has grown we have added additional tower locations.Â We are currently revisiting existing customers that we feel we can now serve from new tower locations and replacing 900Mhz radios with either 2.4 or 5.7Ghz equipment.
In addition to replacing 900Mhz radios we are also upgrading some existing 5.7Ghz customers.Â The new 5.7Ghz Access Points (AP’s) have far higher capacity than the existing equipment.Â We are currently retrofitting existing towers with new AP’s.Â Unfortunately using these new AP’s requires replacing theÂ radios at the customers houses.
Towers with the new high capacity AP’s include Luckey, Oak Harbor (west side), and Perrysburg (south side). Â All of the Perrysburg customers have been changed and the majority of the customers on the east side of Oak Harbor have the new radios.
We will be installing the high capacity AP’s at Graytown in May and visiting customers in the area to swap radios.Â The radio swap does not require access to the inside of the house and the appearance of the equipment is unchanged.