ISC Network Operations Report for 2014

ISC’s Public Benefit network services are: F-Root; SNS-PB, a subsidized anycasted DNS infrastructure for non-profits; Hosted@, subsidized hosting for non-profit projects at our Redwood City location; a municipal network connecting a number of local cities and non-profits to the Internet; and dlv.isc.org, a DNSSEC Look-Aside Validation service.

Network Infrastructure

We maintain approximately 2768 peering sessions across our infrastructure, more if you count all the nodes that peer through route servers. Maintaining these is a significant work item. In 2014 we worked nearly 200 tickets, adding and deleting peers.

A number of service providers donate transit or hosting for F-Root nodes. In addition, we received two significant donations to our network infrastructure in 2014:

• Telia Sonera provided us with free 1G transit links this year both for F-Root and for general ISC network operations (in Atlanta, Chicago, Palo Alto, Frankfurt, Amsterdam).
• NTT has generously upgraded their donation of Internet transit in our main Palo Alto datacenter from 10G to 40G, and also offered us transit at any of our other sites that need it.

F-Root

Completed Projects

1. We audited all of our existing F-Root nodes that don’t have IPv6 connectivity. We contacted both the IX and transit partners and IPv6 enabled the nodes where possible. The nodes that got v6 in 2014 are Phnom Penh, Chennai, Kuala Lumpur, and Torino. At this time 40 of our 57 nodes are IPv6-enabled.
2. We developed a design for a new F-Single node that dramatically reduces the cost for small sites. We solicited new node locations at AfriNIC in Mauritius in November, based on this new single-node design.
3. We added new nodes in 2014 in Kuala Lumpur and Chennai, India. None of our public nodes were decommissioned in 2014. The total number of F-Root nodes went from 55 to 57 in 2014. We signed agreements to add nodes in Beijing, China and Luanda, Angola.

We continue to work on new nodes being added in Uruguay, Angola, and Beijing. It can take several months to half a year to add a node, because for any given node, there are generally two transit providers: a hosting provider and one or more peers that need to be identified, established, configured and tested.

F-Root maintenance

• Hardware refreshes or upgrades in: Amsterdam, Chicago, Frankfurt, and New York. We would like to refresh each F-Root node approximately every 5 years. The frequency of refresh is of course really up to the node sponsor (because they purchase and own the equipment).
• Software rebuilt or upgraded in (at least): Brisbane, Paris, TelAviv, Warsaw.
• Transit provider changed or added more links or backup providers in (at least): Amsterdam, Chennai, Monterrey, Chicago, Franfurt, New York, Atlanta, Palo Alto

F-Root Service Availability

F-Root has not had an actual service outage in over 10 years. Because F-Root is anycasted, an individual node outage does not impair overall service availability, although users who were getting service from that node may see longer response times. Individual node outages can occur when someone makes a (mis) configuration change, or there is a hardware failure. Individual nodes regularly are degraded or taken out of service for a few months at a time due to localized problems (e.g. transit availability; in one case a POP was flooded), moves, or changes in sponsorship. There were 5 F-Root nodes that were degraded in 2014 for a significant period.

We monitored and addressed an attack on the root system in January. Global query loads doubled, but it made no real impact on the overall system.

DITL Data

We participate in the DNS-OARC “A Day in the Life” (DITL) data collection project every year. This year we contributed 58,405,816 Mbytes of data, consisting of 5,586 million queries to F-Root. This was approximately 14% of queries to the root reported in the DITL data this year.

SNS

ISC operated both a commercial secondary name service and a subsidized public-benefit service in 2014. There are several dozen small ccTLDs published through our subsidized SNS service.

As every other network operations team does, we spend considerable resources on DDOS prevention and mitigation. We had to update systems across the board in response to the HeartBleed, NTP, and Poodle issues in 2014.

Early in 2014, our secondary name service was hit by TCP exhaustion attacks. We monitored, increasing the availability of TCP sockets as needed. Later, in July, we were hit with a 200+ gigabit DDOS against both the SNS Public Benefit servers in the San Francisco Bay Area, Chicago, and Amsterdam, as well as AS1280. The target was a ccTLD that we were hosting. We were able to work with both our friends in the community to locate the attack vector (a mixture of NTP and a flood of bad DNS packets from a well-known botnet), and then with our transit providers to filter out the attack traffic.

This attack made isc.org completely unreachable for an hour or so, and degraded our connectivity considerably for most of a day.

After the massive DDOS, we realized that we are do not have the bandwidth or the scale to effectively protect against a modern flooding attack. Making the investment in equipment required to reach that scale is probably not consistent with our non-profit mission. So, we made the decision to recommend that our commercial SNS customers move to another (larger) provider. ISC is no longer accepting commercial SNS customers, although we will continue our subsidized public-benefit SNS service. We have secured additional bandwidth for our subsidized public-benefit SNS to help withstand future DDOS attacks.

Hosted@

We have historically provided free or subsidized hosting for non-profit projects. We did quite a bit of work here in 2014, resolving 234 issues for our hosted partners.

The Kernel.org team refreshed their equipment and donated their old servers to ISC. We are looking into putting them back under maintenance and using them as a VM farm for Hosted@ guests. The goal is to reduce the number of ancient power-hungry guest servers in our Redwood City data center. The electric bill is enormous!

We added NANOG backup servers to Hosted@, and will publish nanog.org via SNS-PB.

Our Hosted@ users include: DNS-OARC, Public Library of Science, the FreeBSD Foundation, Public.Resource.org, the Network Time Foundation, the Measurement Lab (M-Lab), NetBSD, distributed.net, OpenDNS, The Linux Kernel Archives, OpenBSD, The Center for Applied Internet Data Analysis (CAIDA), the Bufferbloat Project, Creative Commons, the Free Software and Open Source Foundation for Africa, HTTP Archive, NANOG, SANOG, and more. In 2014 we removed and returned equipment to OpenLDAP.org, unixheads.com, and Bay Area Children’s Theatre, among others.

NTP Amplification issue

This was a big event in early 2014, and impacted a number of our hosted projects. We helped deflect a NTP amplification attack against one of our hosted customers. We also found that a number of our hosted customers were the source of NTP amplification attacks. We collected incoming abuse reports and contacted administrators for these hosted systems to make sure they locked down their NTP servers.

Municipal network

In 2014 we upgraded our public-benefit municipal network service to 10G. The City of Menlo Park is now getting their Internet entirely through ISC, and we continue to work with the Palo Alto Unified School District to get them a 20x increase in bandwidth. ISC provided free or subsidized connectivity and transit for 11 local towns, schools, and non-profit organizations in 2014.

DLV

ISC operates the DNSSEC Lookaside Validation registry. This was developed as a transition mechanism to help people who wanted their DNSSEC-signed zones to be validated even though their parent had not yet adopted DNSSEC. There is an interface at dlv.isc.org that allows users to register their domain to be validated through the DLV.

In 2014 we modernized the DLV infrastructure, moving it to virtual machines at our Palo Alto datacenter, because it was running on very old hardware and an old OS. We also did an assessment and created a proposed plan for gradually decommissioning the DLV, which will be socialized throughout the DNS community in 2015. We currently have about 2,800 working zones that can be validated through the DLV.

RIPE Atlas Anchor

In 2014 ISC became one of the Anchor Probe sites for the RIPE NCC Atlas project.

AS112 Server

ISC operates one of the AS112 servers. The network of AS112 servers absorbs leaked advertisements for what are supposed to be private RFC1918 addresses, reducing the load on the rest of the DNS infrastructure.

Network Engineering for IETF Meetings

Jim Martin, our Director of Operations, has been the volunteer NOC Team Lead for the IETF for the past 15 years. Jim spends a substantial amount of his time in planning, establishing, and running the conference and hotel IP network for every IETF meeting. He and his team of volunteers create a rapid-deployment event network consisting of an extensive wireless deployment (>100 access points), multiple IPv4 and IPv6 BGP peerings over anything from a DS3 to a 100G experimental link, for about 1500 of the most demanding users on the planet … the people that design the way the Internet works. In 2014 Jim established and worked in the NOC for the 89th IETF in London, the 90th IETF in Toronto, and the 91st IETF in Honolulu, and scouted future possible locations for the IETF.

ISC.org Website

One notable event in 2014 (we can’t call it an accomplishment) was the malware infection on the ISC website. We took our website down between Christmas and New Year’s Day while we scrubbed all the files on our website, updated the software, and added malware detection. We could have restored the website more quickly, but we figured that everyone looking at it was on vacation, so we took our time. We have not had any reports that we actually infected anyone, but we deeply regret the whole situation.