Despite Comcast-Netflix deal, settlement-free peering is alive and well

Over the past week or two, the Internet has been abuzz with coverage of the deal between Netflix and Comcast. The various themes and technical concepts discussed were hardly news to me; I first covered this subject a little over three years ago as a member of the founding technical team at Voxel, which was purchased by Internap in 2011.

Though the names and congestion points have changed somewhat, a lot remains the same, including the media’s inability to comprehend or explain how traffic is exchanged at scale on the global Internet: “peering,” in the vernacular. This is not to call any journalists lazy or irresponsible; rather, peering is a highly specialized pocket of network engineering knowledge, and if you’re not working in its trenches, it’s easy to fall behind. That most engineers “in the know” are bound by strict NDAs with their employers and/or connectivity partners doesn’t help the situation; fortunately, my employer is transparent enough to allow me to share a few words of wisdom.

Settlement-free peering is dead! Long live settlement-free peering!
Before continuing further, it is important to define terminology. As explained in our technical data sheet on peering:

Peering is a connectivity method where two networks establish a direct IP connection between their networks, bypassing any third-party networks or middlemen. The most popular form of peering is known as “settlement-free peering”, where two networks agree to exchange traffic with one another directly without any form of compensation. Another form is “settlement-based peering” or “paid peering”, where two networks agree to exchange traffic directly, and compensation is involved.

Many news articles and blogs incorrectly state that settlement-free interconnection has gone the way of the dinosaur, and now content providers must pay access providers to reach customers directly. Though there are certainly well-documented examples of money changing hands, it is not fair to describe any recent announcements as a change in precedent, or even the norm.

Pretty much since inception, Verizon, AT&T, Time Warner Cable and Comcast have always required money for exchanging traffic with Internet content producers and their bandwidth suppliers (there were regionalized and M&A-related exceptions: for example, SBC peering more pervasively before the M&A which made them the “new” AT&T; likewise Adelphia prior to the fall of the Rigas family empire). This trend has continued given the expanding monopoly or oligopoly positions in the access market.

At the same time, it is important to consider a tier of networks below the “big four”. Many thousands of networks, representing both big access and big content alike, agree to exchange traffic in a settlement-free manner. Peering policies exist not to restrict data flow, but to ensure that a base set of technical standards are maintained; for example, to make sure that networks maintain a professionally staffed Network Operations Center (NOC) responding to security and operational issues, or that sufficient traffic is exchanged to justify the administrative burden of configuration or scaling router ports. I submit that these exchanges comprise the norm, not the exception. They have garnered little fanfare in the media because they continue to scale in a reliable and customer-friendly manner.

While financial terms of settlement-free peering agreements are usually confidential, there are exceptions. The Netflix Open Connect Program is a settlement-free arrangement of sorts: access providers agree to house Netflix servers in their datacenters, or connect to Netflix at carrier-neutral exchange points; in turn, their customers are granted direct access to Netflix’s content library, bypassing third-party intermediary networks. Notable participants include SuddenLink, Cablevision, RCN, and Sonic.net. Why would these access providers allow Netflix, who reportedly represents over 30% of the Internet’s traffic during peak hours, to abuse their pipes for free, when others have charged tolls? It’s really quite simple: their customers benefit.

I live in a pocket of northern New Jersey where Cablevision and Verizon are options for broadband Internet. Though Verizon has the upper hand in the “speeds and feeds” arms race, their congested peering makes for a sub-par user experience: with congestion on a large portion of their peering connections with other ISPs, Netflix and YouTube are hardly usable between 6:00 PM and midnight. In contrast, though limping along with its slower DOCSIS 3.0 access technologies, Cablevision runs its edge connections free of congestion, allowing me to stream cat videos or “House of Cards” (Season 2, Episode 1? Just wow!) in beautiful 1080p. Cablevision has successfully proven the value of a more open peering policy as a competitive differentiator.

Ratios make the world go round!
All the same, telco die-hards and their sympathizers will continue to argue that settlement-free peering doesn’t add up. To them, large content producers are abusing access networks by lumping traffic on their pipes, whether or not this is the same content requested of their revenue-generating customers. Care and feeding of large-scale networks does not come cheap, and everyone needs to pay their fair share in network upgrades. It is also believed that traffic ratios should factor into economic settlement: if a network is generating traffic predominantly towards a network, rather than receiving traffic from it, then it should bear the majority of the cost burden.

Some additional food for thought:

• Counter to the above wisdom, it is believed that scaling a subscriber’s upstream bandwidth on a DOCSIS-based cable network costs approximately five times as much as scaling a user’s downstream bandwidth. Shouldn’t backup operators, and other services with “pull heavy” traffic profiles, be the ones truly viewed as abusers?
• Paid peering is said to level the playing field by allowing a content provider to offset an access provider’s costs of servicing heavy traffic flows. Do transit/peering costs at carrier-neutral datacenters, where ISPs and content providers connect, have a bearing on costs of scaling the last mile? Has there been any open disclosure of what these costs even are?
• Relating Internet traffic to conventional telecom terms (as has been common over the past week or two), why should IP traffic be viewed as “sender pays” when “Bill and Keep” is a more apt analogy?
• Customer support is a major expense for broadband ISPs, and it is said that a single truck roll (service call) can move a subscriber into the red for a full year. I don’t believe any ISPs maintain statistics on truck rolls incorrectly ordered for congested peering; if they did, they might find that their yearly OpEx in misdiagnosed support incidents dwarfs any possible revenues in paid peering.

A rose by any other name
A key tenet of the FCC’s Open Internet Order (December 2010) is “no blocking” summarized as:

ii. No blocking. Fixed broadband providers may not block lawful content, applications, services, or non-harmful devices; mobile broadband providers may not block lawful websites, or block applications that compete with their voice or video telephony services;

Leading up to last week’s announcement, Comcast and Verizon both issued public statements confirming that they did not block or rate-limit Netflix traffic. These statements were technically accurate; it was highly doubtful either party logged into its routers or deployed any Deep Packet Inspection to specifically alter traffic to or from Netflix’s infrastructure. Nonetheless, what was the intent behind delaying certain capacity upgrades at the edges of their networks? The data presents an interesting picture:

The above graph illustrates the quality of connectivity between Comcast and Tata (a transit provider Comcast uses for reachability to large swaths of the Internet, including the backbones operated by mega-telcos NTT, TeliaSonera, and XO, who provide Netflix with bandwidth) in New York. The evening packet loss displayed in blue is sufficient to degrade video quality to where rabbit ears look like a viable alternative, even with the most advanced of adaptive codecs/players.

Similarly, this graph illustrates connectivity between Comcast and Cogent Communications, a large supplier to Netflix, in New York.

In contrast, this graph illustrates one of few uncongested paths to Comcast in New York, identified and used by our Managed Internet Route Optimizer^TM (MIRO) technology.

Though this post is biased towards my hometown, a look into Comcast’s capacity in other markets, to the overwhelming majority of ISPs, presents a similar picture. While Netflix was not blocked by configuration, might it have been blocked by intent?

Conclusion and customer impact
While we have not witnessed a change in peering dynamics as a result of the Netflix/Comcast transaction, a trend we have seen over the past few years is the degrading quality of bandwidth from conventional “tier 1” ISPs, where peering edges have become congested due to the games described above. Network operators commonly discuss on mailing lists how the big four access shops all maintain edges which are boiling hot unless you pay them, or buy from an intermediary paying them. Where it was once possible for an enterprise or content shop to enjoy “good enough” connectivity purchasing from these providers directly, one now must enter the complex game of multi-homing to a half dozen or more providers, or purchase from a route-optimized “tier 2” like an Internap, in order to enjoy a positive and congestion-free user experience.

Contrary to popular opinion, Netflix did not “sell out”, nor were they traitors to the cause of net neutrality; rather, they did what they had to do in order to scale their growing online streaming business. One should not blame Netflix, but rather our marketplace for residential broadband, which allowed such a transaction to occur. I yearn for a day when we see true competition in the last mile — when providers compete on their ability to both innovate and deliver a good service for a fair price. Meanwhile, content and service providers looking to innovate in this space should invest in peer-to-peer delivery as well as on-premise caching. Ordinary users should support a regular framework towards enabling municipal broadband and competitive fiber-to-the-home options.

Internap customers can rest easy knowing that we’ll continue connecting them to Verizon, Comcast, Time Warner Cable, AT&T and a myriad of other access networks with reputations for congestion, without any undue latency or packet loss. Routing customers over constrained connections to prove a point is still not in the cards for us, nor is it something our customers would ever accept, as we sell IP services in a highly competitive marketplace. The Internap team will continue working diligently to circumnavigate the tangled-up tubes; we’ll develop next-generation technologies to keep the bits flowing as efficiently as possible, whatever the regulatory climate.