Usage Graphs, Plan caps or limits

usage graphUsage Graphs

We recently added usage graphs to the customer invoices.   We have added the graphs for a couple of reasons:

  • We want our customers to know how much data they are actually using.
  • We want you to be able to tell when your usage has changed significantly.
  • It makes it easier to compare services with other technologies such as WiFi hotspots.
  • New service plans may have data limits.

It’s very difficult for most customers to understand how much data they are using.  The units vary and there is considerable confusion concerning speed (how fast you can transfer data) in Megabits per second and usage (how much data in a time period) in Mega Bytes.

Cell phone and Cellular data plans normally have limited data usage .   Typical data plans have 5 to 10GB per month of data before additional costs are incurred.

Our typical home Internet customer uses 41GB of data a month.  15% of the users exceed 250Gb, and 1.5% are over 500Gb.

Are usage limits coming?

Not on the existing plans.  We anticipate that any new service plans will have usage limits.

Why is Amplex considering usage limits?

We want to offer higher speed plans to our customers.   With our newest equipment we can offer much higher speeds, but we need some type of usage limit before we can do so.   Our current highest speed plan is 9Mb/1Mb.  We would like to increase the speed on our existing plans, and offer higher speed plans up to 25Mbps.

With our current plans, a customer maxing out the service for an entire month generates a economic loss – the data consumed exceeds our costs.  Our current plans at 3.5 or 5Mb limit the losses on any given customer, simply because the customer can’t download enough data to cause a major problem. This changes if we offer higher speed plans with no usage limits.

We have not decided on what the new plan speeds or limits are going to be, though the FCC has established a minimum usage of 100Gb/mo for a plan to be considered broadband.   Our service plans will be at least 100Gb, and probably far higher.

Can I keep my existing plan?

Yes (seems like there is a Obamacare joke here somewhere….).   We may discontinue the existing plans for new customers, but will continue to offer the current plans if you want to stay with them.

Government data gathering and the PRISM program

There are quite a few sensational stories in the media claiming that the government is gathering large amounts of data from network providers without warrants and/or abusing privacy.   Some of the fuss is legitimate but for the most part it’s being blown out of proportion.

Lert’s start with how Amplex handles requests for data by government agencies:

  • Any request for data of a transactional nature made by a governmental agency requires a valid subpoena.  The request can specify that no notification be made to the target of the investigation for a certain amount of time.  These requests are rare and very specific.  There really isn’t anything to object to in these type of requests – it’s local or state law enforcement doing exactly what they are supposed to be doing.  Amplex reviews these requests with our legal council to verify the authenticity of the request and complies as needed.
  • Requests for stored data (email or other content) or any real time capture of data requires a search warrant, signed by a judge.   Search warrants may specify that no notice be given to the affected party.   Amplex reviews these requests with our legal council for authenticity and to narrow the scope of the request if the request is unreasonable, excessively broad, or technically unfeasible.
  • National Security Letters (NSL).  This is a special type of request for data from the federal government.  This type of request contains a gag order prohibiting the disclosure of the contents of the letter, the requested data, etc.   The Electronic Frontier Foundation has a good writeup of NSL’s.  Amplex has never been presented with a National Security Letter (at least as of 6/8/2013).

On several occasions Amplex has been presented with a request from law enforcement for information outside the above processes when there was imminent danger.  Amplex will cooperate with law enforcement in these situations when we believe the request is legitimate.  Specific examples of this have included a bomb threat to a local school, and a person posting very specific threats online.  Please note that our terms of service allow Amplex to cooperate with law enforcement agencies (LEA) in these situations.  Is there a potential for abuse in this?  Yes, but there are many legal options for Amplex to take if we discover that the process was abused or LEA intentionally deceived us.

So what are all the sensational stories about?   What is the real story?   We have a few ideas but first we need to discuss CALEA:

CALEA  is a federal regulation that specifies a series of technical, legal, and management procedures by which LEA can gather needed data.   The CALEA implementation in a private network is NOT controlled by the government.  There are a considerable number of safeguards in the CALEA system to prevent unauthorized use and to limit the data captured to only the records specified in the accompanying search order.

PRISM is the system that hit the news this week.  The system reportedly gives the government access to large amounts of data.   The story doesn’t really add up.  Why?

  • The reported cost of the program is $20 million dollars.    The federal government can’t fund a giant data gathering program with that money.   $20 million barely pays for the consultant to design the program, much less implement it.
  • The shear volume of data would be extremely difficult to capture and transport.  Capturing all the data from a tiny network like Amplex would consume somewhere around 40TB of data per day.  Getting that data transported off the network would be very expensive.   There is no central point on the network where all data can be collected in any case.
  • Large networks like  Google, MSN, Facebook, are very decentralized.   Large scale monitoring would require a massive investment in additional transport capacity.  The networks have a hard enough time keeping up with their own growth, much less building a parallel network for the government.
  • Most data transported on the Internet is asymmetric, meaning that the data path to the end user is not the same as the return path.   Capturing data at mid-points in the network usually only gets you 1/2 the information.  Trying to capture the data from multiple mid-points and putting it back together is very difficult.  Capturing data only works well at the source or destination.

So what is PRISM really about?   My best guess is that it’s a electronic system where the government can present CALEA requests along with the relevant legal search warrants,  national security letters, etc. to the participating companies by electronic means rather than faxing, scanning, and/or using overnight delivery services to move paper around.  That is about the only thing that $20M by the federal government will buy you.   It also makes sense.  Much of the legal system really is 15 years behind the rest of private industry and they still push mountains of paper around the country.   The fed’s did something smart and built a secure communication system to deliver legal documents?   Good for them.  Perhaps they should have sent out a press releases instead of hiding it.

Are the feds overreaching with NSF letters and the PATRIOT act?   Yes – and the Department of Justice has previously found major problems with the program.  Are other government agencies abusing the program as well?    I suspect we are about to find out.


Network downtime – June 3rd 11:24 to 12:31

Amplex experienced a network wide problem today at 11:25am.   While we are still analyzing the logs we have a good idea of what caused the issue.    The network experienced a broadcast storm and loop due to the failure of the mechanisms designed to prevent network loops.

We have seen this same issue twice in the past, approximately one month ago.  In those cases the problem occurred late at night and was not noticed by most customers.  Following the earlier occurrences we made several changes to the network to remove the lower bandwidth backup paths which caused a significant amount of instability.  I can go into much more detail but it’s probably not worth discussing since the important part is…

What are we going to do about keeping it from happening again?

There are several steps we are taking to prevent the issue from occurring in the future:

  • Installation of routers at tower sites.  We are outgrowing the existing network layout (which has worked well for many years) and will be installing routers at the individual tower sites.  This will significantly reduce the broadcast load on the network.  We have avoided placing routers at tower sites in the past for reliability reasons.   The advantages of individual tower routers now outweighs the risks.   Installing routers is low risk and can be done with minimal impact on the network and customers.   The first one will be installed at Luckey today.
  • Splitting the network into 2 logical parts.  The network consists of 2 rings that share a common path between Perrysburg and Lemoyne.  The north ring primarily serves sites in Ottawa county, the south ring serves Wood county.  We are adding an additional link between Perrysburg and Lemoyne and will use that to isolate the north and south rings.  This will reduce the effective size of the network while also helping to isolate issues.
  • Evaluating Performant Networks Software Defined Networking gear.  Performant has designed a network appliance that promises to improve the stability and recovery time for Ethernet networks by incorporating ITU’s G.8032 “Ethernet Ring Protection Switching”.   This standard and equipment allows for sub 50mSec failover in the event of breaks in an Ethernet ring.  The Performant equipment adds an additional feature by continuously measuring the actual performance of the links so that it can make intelligent decisions based on the capacity of the individual links.  Evaluating and installing this equipment is a long term project as the equipment is new and relatively untested.  While it shows great promise we want to run it in a test environment for several weeks before attempting to deploy it.

We understand that a reliable network connection is very important to you and sincerely apologize for the issues today.  If you have further questions please do not hesitate to contact us.

Mark Radabaugh, VP Amplex

Network issues 11/15/2011

We have had reports of difficulty or slow response from Google today.   This is related to maintenance performed by one of our upstream providers (Cogent) last night.   In checking with Cogent they are telling us that they are continuing to have difficulty with the new software load in the Toledo router.    While Cogent is resolving the issue we have shut down our connection to Cogent.  Shutting down the connection to Cogent provides a work-around while they resolve the issue.

The problem is unusual in that Cogent (at least until we shut down to them) was advertising (telling us) that they have the best path to reach Google yet they are not delivering the traffic.    This is the “Mortal Sin” of networking – advertising a route to a destination you can’t deliver to.

Unfortunately this is one of those issues that is hard for us to automatically detect.  Until we hear from customers or notice it ourselves we don’t know about it.    We apologize for the issue and thank the customers who notified us about it.

We are waiting for Cogent to notify us that the issue has been resolved before bringing the connection to them back up.  Assuming the issue is resolved there should be no further downtime as the connection returns.

Update as of 4:00pm:  Bringing Cogent back online resulted in the same issues we were seeing this morning.  At this point further troubleshooting is going to need to wait until early AM hours.   The inability to use the Cogent circuit tonight is going to lead to congestion on our other circuits.   Service will work but will be somewhat slower this evening.


Why can we not watch ESPN3 online?

A few customers have asked about the availability of ESPN3.   ESPN3 is the sports network’s online video streaming service.  To watch ESPN3 your service provider (in this case Amplex) has to pay ESPN for access.   Why don’t we do that?

The simple answer is that we feel it is poor business model when applied to the Internet, and an incredibly slippery slope that will end badly for everyone.

ESPN’s plan is to recreate the Cable TV business model on the Internet.  The Cable TV model is this:

  • The networks (HBO, ESPN, ABC, etc.) negotiate deals with the cable company to carry the networks channels.   The deal requires the cable company to pay X dollars per customer per month for the cable company to carry the networks programming.
  • The contract  specifies that all of the network’s channels must be carried, not just the popular ones.   Why there are 50 junk channels?   Because the contract says if you want to have Oprah you also have to pay for and carry our 12 other channels.  Combine the junk channels from a dozen networks and you have Cable TV.   150 channels that you pay for, 3 that you actually watch.

Why does cable TV costs so much?  It’s not because the cable companies are greedy.  Ok, they probably are, but the bigger reason is that they are forced to pay ever increasing fees to the networks for content.

The ‘provider pays’ model is the way cable TV works.   There are a lot of reasons that it should not be applied to the Internet:

  • It raises our costs (and the cost of your Internet service) to pay for something the majority of customers do not watch.
  • There are a huge number of sources of content on the Internet.   The service provider negotiating with every content provider on the Internet is unworkable.
  • Fees will escalate over time.

If ESPN’s model succeeds there is nothing to stop Netflix, Hulu, YouTube, or even Google from demanding the same type of business model.   The day that happens the cost of Internet service is going to skyrocket.   I can easily see content costs adding hundreds of dollars a month to the cost of Internet service.  Cable TV is a naturally limited model in that the network can only carry a few dozen networks and a few hundred channels.  The Internet is unlimited in the amount of content it can carry.

Do we really want to recreate a service where your content choices depend on the networks your service provider subscribes to?

ESPN refuses to sell a subscription directly to an end user.

If you would like ESPN to change this feel free to tell them about it: