A number of folk have asked me why Level 3 and Comcast have been generating so much media attention over the past few days, and ultimately what this means for their hosting operations.
So, what’s going on? Unfortunately, in a sea of rubbish, with authors ranging from Wall Street analysts to K Street lobbyists, it’s difficult to find an accurate and objective accounting of what actually took place. For starters, I’d refer straight to the sources, which touch on why Level 3 and Comcast are unhappy with one another:
Level 3 Press Release: https://www.level3.com/index.cfm?pageID=491&PR=962
Level 3 Clarification: https://www.level3.com/index.cfm?pageID=491&PR=963
Comcast’s rebuttals: https://blog.comcast.com/
Another good read is Dan Golding’s GigaOM post (https://gigaom.com/2010/12/01/comcast-level-3-battle/). (While we disagree on several key points, it is nonetheless refreshing to read an analysis written by an author who’s served his time in the trenches, and is a subject-matter expert on both the economics and technology in play.)
Game-Changers
In the absence of any real facts, one thing which is clear is that both Level 3 and Comcast grew their businesses in bold new directions, taking shortcuts and ignoring best-practices along the way.
In one corner, we have Comcast, freed from the shackles of AT&T as its sole provider, now aggressively attempting to establish itself as not merely an access provider, but a wholesale ISP which content hosters and smaller backbones might buy from. In doing so, Comcast took several key missteps, including treating “peering” as a profit center from practically day one. Where other access providers would be content with merely the “settlement free” exchange of traffic – an arrangement where both content originators and recipients exchange traffic at no cost, both avoiding having to pay a middleman to carry their bits – Comcast has made it clear it wants to collect money anywhere and everywhere possible. It would seem Comcast has again upped the ante, this time attempting the ultimate chutzpah of charging back its vendors for the “privilege” of servicing them. (Indeed, I’m a bit jealous I can’t do this right now, though I’ll certainly try with our next round of renewals.)
In the other corner, we have the financially-challenged Level 3, who’s re-invented itself once again (as seems to happen once a year), shifting a lot of sales focus from wholesale IP transit and infrastructure to CDN. Though they arrived late to the game, they’re coming on strong, targeting major content generators with cut-rate pricing. This is significant as Level 3 holds many large cable and DSL providers as its customers, and is effectively billing them for this new broadband-subscriber-bound traffic, as normal provider-customer relationships dictate. I can certainly sympathize with access providers trying to capacity plan around this influx of traffic. On a purely contractual basis, I’m sure what Level 3 is doing is on the up-and-up. No evidence points to traffic being “stolen”; likewise “peering contracts” were not “broken”, as they didn’t exist in the first place. Nonetheless, they certainly could have provided the operational community some additional forewarning, or perhaps built their CDN as a network autonomous to the Level 3 backbone, maintaining an open peering policy and seeking out access networks to exchange traffic with at no cost.
It’s important to realize that, with the exception of Comcast, every major US cable operator maintains settlement-freepeering relations with regional service providers, CDNs, and large websites. (This is not to say they’ll peer with any network off the street, however peering policies exist to set a fair bar – for instance to make sure that the party they’re dealing with is professionally staffed and can route to them intelligently – not to discriminate against certain business models outright.)
It’s all about the “Benjamins”
What’s ultimately under fire are simple economics, and how a jilted Comcast is trying to recover lost revenue. And who can blame them? A backbone provider sent me a diagram used to educate its costumers on the real issue:
Fairness
In their filings, Comcast states repeatedly that major CDNs, specifically Limelight and Akamai, are paying Comcast for access to its customers. What they fail to state is that both companies lead the industry with their fair peering policies and massive exchange presences:
Akamai peering: https://www.akamai.com/peering/
LLNW peering: https://login.llnw.net/noauth/peering.cgi
Why, then, are they ponying up cash? Given a choice, I have little doubt both organizations would have entered into a settlement-free relationship with Comcast, as precedent dictates. I’m sure they don’t consider it right that they pay Comcast to deliver traffic – rather, they agreed to it under protest, as it’s the only viable way to serve their content at scale. Captive eyeballs and extortion or not (more on this point later!), CDN is a competitive market: Akamai and Limelight are paid by their customers to deliver traffic; if these bits get discarded, customers will move their business elsewhere. If they had the luxury of time, these organizations might have gone to the regulators; unfortunately content has a habit of moving rapidly between providers and contracts.
As additional facts are revealed, I’d be very curious to see the term sheets and sales orders companies like Akamai and Limelight signed with Comcast.
It’s not about the ratios!
In an attempt to explain the issues, Comcast released a video of networking head honcho John Schanz discussing traffic ratios, disingenuously:
https://blog.comcast.com/2010/11/how-internet-peering-works.html
Traffic ratios date back to the days of “tier 1” telecom behemoths as major traffic sources, route miles, and hauling bits around the country to get between points “A” and “B”. On a modern-day Internet , they are often cited as an excuse for denying peering where it might actually make business or technical sense. This is especially poignant in the frame of content-access negotiations – access providers are collocated at major carrier hotel locations; whether they absorb content from a free peer or paid-for transit connection, the routes and costs for hauling this traffic back to their broadband subscribers remain the same.
The Tata Problem
Amidst all the talk of foul play with Level 3, little attention has surrounded Comcast’s relationship with Tata Communications, which I consider to be a far more egregious violation of their stated principles on Net Neutrality. As was the case with Level 3, Comcast purchases commodity IP transit service from Tata, as a means of reaching networks it doesn’t maintain direct peering relationships with. Unlike Level 3 though, Comcast runs its ports to Tata at capacity, deliberately, as a means of degrading connectivity to networks which won’t peer with them or pay them money.
Speaking off the record and respecting customer confidentiality, a Tata executive confirms, succinctly:
“[our] San Jose and New York links with Comcast are running full.”
One might explain the situation with Tata to their customers using a diagram like this:
Indeed, testing reachability to Comcast eyeballs over Tata, we see loss statistics like this:
Given the high packet loss at certain times (likely corresponding to when these interfaces are saturated), serving even simple HTTP content is clearly not sustainable. In contrast, entering the Comcast network by way of Level 3, things look a lot better:Perhaps Mr. Schanz could explain on the whiteboard why one need not worry, however a better course of action would be to simply purchase more capacity, through Tata and/or another provider. No stranger to Internet peering, Richard Steenbergen (CTO, nLayer) explains further:
“The true power of Comcast isn’t in the size or scope of its network, it’s in the captivity of its customer base.
If Level 3 turned off Comcast for refusing to pay their contractually obligated transit bills, the traffic would be forced through massively congested Tata transit ports, and a huge number of Level 3′s customers would take their business elsewhere as a result. If Comcast intentionally congests its transit providers and provides terrible service to its end users, which it has been doing for several months now, most of those users have no real alternatives to switch to.
In other words, content is mobile, eyeballs are not. Comcast realizes that they can (ab)use their captive eyeballs to force content to pay them for access, without having to create a “100% down” partition like Cogent has done in the past. For a Netflix customer, 20% packet loss is effectively just as down as a hard partition.
[…] while it should be every network’s right to choose who they do and don’t peer with, or buy transit from, things start to get murky when one network is abusing their franchise agreements and near monopoly or duopoly status in many markets. If users had an actual choice, and could get comparable broadband access elsewhere, then Comcast would be free to congest their network however they see fit. But that isn’t the case, and this is where government involvement and Net Neutrality start to have legitimate grievances with Comcast’s actions.”
Regulation and Disclosure
So, how should the FCC and other facets of the Federal Government intervene?
Perhaps at odds with my above criticisms, I believe in the power of the free market, and that networks should be left to negotiate (or not negotiate!) these issues without oversight. Telling network operators where they must or must not connect, and on what terms, is surely a recipe for disaster.
On the other hand, protecting the rights of broadband subscribers is of the utmost importance, given the scarcity of real competition on the last mile. If many Americans don’t like the rates their cable providers charge, or the quality of service delivered, they’re left to pound sand or downgrade to slow DSL. Just as the FCC is looking to codify “network management” practices on the last mile, I think it’s fair to penalize monopoly/duopoly providers who fail to adequately manage their backbone and external capacity.
More immediately, I’m hoping for full disclosure of any commercial proposals and agreements between Level 3 and Comcast. Absent this data in its rawest form, it’s impossible to form any intelligent opinions on the specific issues in play. For all we know, Comcast’s commercial proposal might be fully reasonable, and limited to some basic cost-recovery of capex costs in router ports or Level3 transit install fees it must now incur on short notice. We just don’t know!
Voxel Customer Impact
What does this all mean to your day-to-day hosting operations, as a customer of Voxel? Absolutely nothing, hopefully! The above commentary should serve only to shed light on a hot topic, not to present any grounds for immediate concern.
Voxel purchases a number of Ten Gigabit transit ports from Level 3, which provide a clear path for sending traffic to Comcast and other networks. When these circuits hit high “water marks” for utilization, we order more. Likewise, we carefully monitor for the quality of connectivity between our network and large broadband destinations; the above screenshots are only the tip of the iceberg for analytics. When we encounter a bad route, we engineer around it right away.
Holding our customers hostage, or deliberately saturating connections to prove a point, is at odds with how we do business, and is simply not in the cards.
(Disclaimer: Adam is Voxel’s VP of Network Architecture; among his responsibilities are peering and transit strategy. No yellow journalists or Washington lobbyists were harmed in the making of this post.)
