Evolution of Internet Exchanges

Something we’re fortunate to have at ISC is the founders of some of the most influential Internet Exchanges (IXPs) amongst our senior staff. I haven’t been personally involved in the IXP business for around 7 years now, so it was a pleasure to be invited as guest speaker to the AGM of the TorIX, Canada’s biggest IXP, in Toronto.

It’s great to see TorIX thriving – over 130 peers, 70Gb/s of traffic, and a board demonstrably committed to putting appropriate administration, governance and infrastructure in place. So what has changed about the IXP business since I first started playing in it in 1994, and what are the principles of running an IXP that we’re learned over the years that look like underlying invariants ? One key point seems to be the operational model, and two main approaches have emerged. Either the IXP is operated by the co-location/data-center provider as one more service along with racks and power, or it exists independently, working with a number of data center operators in the same location(s) over time. The former model has taken hold mainly in the US, with co-location providers such as Equinix, Telehouse and CoreSite operating exchanges within their facilities.

In Europe and other parts of the world, many independent IXPs have successfully partnered with various data centers in their metro areas. This has had the advantage of allowing expansion beyond, and avoiding the disadvantage of lock-in, when the first facility inevitably fills up. Neutrality is also a key point (excepting the recent acquisition of a major US regional co-lo operator by a tier1 carrier) in general it seems to be taken for granted these days. The principle that an IXP should not be controlled or exclusively operated by an ISP is well-understood and practiced, and there’s a number of ways of achieving this. I’m pleased to say that the model we developed at LINX, of a mutual non-profit membership association of the participating ISPs, is now by far the dominant one, and this is the path that TorIX has adopted. But other models exist and work well too, for example exchanges operated by national academic networks, or by consortia of ISPs who invest in a for-profit exchanges as shareholders. Each of these models has its particular strengths and weaknesses.

It’s also worth thinking about what “neutrality” means, and in my talk I also outlined a number of possible neutrality principles that have been learned and shown to be successful and/or controversial over the years. One of the biggest changes and challenges for IXP operators today compared to the 1990s is the technology. Back then, we were using LAN technology to give us big metro-area bandwidth wins over the very expensive and limited international bandwidth available – the 10Mb/s Ethernet hub we started the LINX with in 1994 was a big deal compared to the 256kb/s and 2Mb/s circuits we were using to build our long-haul and international backbones from. More importantly, every time the traffic growth needed an upgrade, the next generation of Ethernet technology came along and we could get 10 times the capacity for roughly 2.5 times the cost. But this stopped happening a bit over 5 years ago – since then 10Gb/s Ethernet has been the fastest technology available, but traffic at the major exchanges has grown to many 100s of Gb/s. As outlined at UKNOF19 two weeks ago, 100Gb/s Ethernet and faster is arriving, but it’s not been nearly soon enough, and in the meantime the primary alternative has been extensive use of 10Gb/s multi-link trunking, with all the scaling and stability challenges that go with it.

While the biggest IXPs grapple with the success challenges of traffic growth, the next tier of exchanges deal with the familiar issues of developing their infrastructure and organizations, and in many less-developed parts of the world the challenge is about sustainability or even establishment of a national or regional IXP. The good news is that there are plenty of resources to help with this. In particular, ISC is ready to work with IXPs globally to help provide services that are of benefit to both the IXP operator in attracting members, and the ISP members themselves. This includes instances of our F-Root top-level nameserver and our Security Information Exchange (SIE) projects in particular, but we can also help with various other “peering magnet” projects we are involved in, and are happy to share our decades of IXP operating knowledge – please ask if you think we can help or you would like to support us doing this.

There are other non-profit public benefit resource out there, and I’d particularly like to commend the work of Euro-IX as an association of exchange operators whose mission includes helping new and emerging IXPs everywhere. One final take-home point is that while there’s lots of established knowledge on how to run IXPs, there is no one-size-fits-all principle – ultimately exchanges serve their local communities, and what works for each’s region will depend on the local needs and environment which it is best tailored to in the light of experience. My grateful thanks to William Maton and the Board of TorIX both for inviting me to speak, and for their ongoing support of ISC through our F-Root node there.


Leave a reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Last modified: June 17, 2013 at 5:39 pm