2019 In Review

Highlights

At ISC, 2019 was a solid year of serving our open source users and the Internet community. We continued our engagement with the Internet infrastructure community through work with ICANN, IETF, DNS-OARC, ISOC, UKNOF, NANOG, APNIC and other fora. We joined with our colleagues in other DNS organizations to drive the first “DNS Flag Day” to clean up some lingering technical debt in the DNS system.

We added two new members to the BIND 9 team, Michal Nowak (based in Czechia) and Diego Fronza (in Brazil). The team completed another year of BIND 9 maintenance and development, modernizing BIND’s DNSSEC support, refactoring the network socket code, and establishing both a code of conduct and a new process for removing old features. A major update to Kea added a configuration backend, and we added monthly development releases while supporting a growing Kea support customer base. Our newest Kea team member, Razvan Becheriu (Romania), was an external contributor on the Cassandra backend until we hired him to work on our next big Kea project, multithreading!

We completed the transition to our new Gitlab system with the migration of ISC DHCP, so that we now have integrated issue tracking and development in the open for all our projects. We added many more ISC software packages for our users, in a new ISC Cloudsmith.io repository. We are publishing all of the BIND 9 restricted-access packages (pre-release security fixes and the subscription edition of BIND) and Kea (public plus restricted access) packages there. By making ISC packages we ensure that the latest versions of our applications, and all of their dependencies, are available on the latest stable versions of the major operating systems. We will continue publishing BIND 9 open source packages on the Copr and Launchpad sites, where they are easier for open source BIND 9 users to find.

Our technical support business is stable and even growing modestly. We added 24 new customers in 2019 and only nine left us, for a total of 122 at the end of the year. We needed an additional support engineer and found a very experienced one, Peter Davies, in Copenhagen.

Continuing our tools update from 2018, we migrated from WordPress to a new static web site. The change was motivated by our desire to avoid possible security breaches on the website, and also to improve performance. Our first-ever systems administrator for development systems, Emanuel Petr (Czechia) has been a huge help with this. In the process we also tried to remove all Google tracking tags, a privacy improvement we eventually had to reverse as our website virtually disappeared from the Internet.

We retired ISC’s long-running Domain Survey, which was becoming irrelevant with the adoption of IPv6, and ended the legacy SNS-PB service. SNS-PB was a free network service that ISC provided for over a decade to non-profits and smaller TLDs. Now that secondary name services are now widely available from other providers we would rather focus our network resources on managing F-Root.

Speaking of F-Root, with the retirement of Brian Reid as Director of Network Operations, we reorganized our F-Root and Technical Operations staff. Ray Bellis took over as acting director, and we added a Network Engineer, Elmar Bins, who works from Austria.

We are always looking for opportunities to contribute to open source. We are a long-time user of the open source Etherpad software. Ray Bellis worked on a major refactoring of the code base which formed the bulk of the 1.8.0 release. Ray also published an open source framework for a RADIUS server in Node.JS - https://www.isc.org/blogs/radius-framework/. This implements the RADIUS protocol but does not supply the business logic.

Thank you

Our greatest thanks go to the loyal open source users who have continued to support ISC, many of whom have been support customers of ours for more than a decade. These include some of the most capable independent operators and enterprises in the world.

We also owe a debt to all the wonderful open source users who have submitted feedback and patches, and the open source packagers who maintain distributions for our users. We regard many of these collaborators as our friends.

In 2019 we received generous unrestricted donations from Verisign and Craigslist, and Mozilla agreed to underwrite development of the DNS over HTTPs support in BIND 9. A few individuals have also made small financial contributions to ISC, via Patreon or directly.

We now have a total of 35 staff members, in 11 countries (the US, Brazil, the UK, France, Austria, Denmark, Poland, Romania, Czechia, Australia, and the Netherlands, if you’re keeping score). Including managers, there are eight on the BIND team; seven on the DHCP team; five in Support; six in Sales and Marketing; four in Finance, Facilities, and Exec; and five in F-Root and Technical Operations. We also have two part-time consultants.

Memorial

We sadly note the passing of former ISC employee Kurt Lidl, who worked on BIND from ~2001-2003. Several of us worked with Kurt in other capacities, including at UUNET Technologies, and knew him as a dear friend and co-worker. He was a FreeBSD user and advocate of many years standing, and co-authored RFC2516 which put PPP over ethernet. Kurt was smart, funny, and thoughtful, and we will miss him.

Looking ahead to 2020

Headquarters Move

ISC is moving out of our long-time headquarters, at an old warehouse at 950 Charter Street in Redwood City, CA. We are working to move out by April 30, 2020. Most of our staff already work remotely, but vacating the HQ means relocating our remaining network facilities, records, and business systems elsewhere. We are primarily a virtual company, with contributors in 11 countries and seven US states, but will be headquartered in the New England region of the US, in an old mill building. The physical address will be 53 Main Street, Newmarket, New Hampshire, and our new mailing address will be PO Box 360, Newmarket, NH 03857.

Projects

We started a new open source project, Stork, which will be a graphical dashboard and management application for BIND 9 DNS and Kea DHCP. We expect to do our first public release in the first half of 2020, but anyone who is very eager can follow the development on our GitLab.

Our goal with Stork is to provide a more complete solution for DNS and DHCP, enabling more organizations to use the open source by relieving them of some of the burden of developing or integrating their own management solution. We are leveraging two very popular open source projects, Prometheus and Grafana, for Stork’s time-series data store and data visualization, and using the modern Angular framework for the GUI.

We are excited to be adding HTTPS and TLS transports to BIND in 2020, to enable both leading options for encrypted DNS.

The Kea team, in addition to driving the Stork project, is hard at work on enabling multi-threading to take Kea performance to a new level.

Project Details

BIND and DNS

ISC staff continue to participate vigorously in DNS community events and organizations. In 2019, Ondřej Surý was re-elected to the Board of Directors of DNS-OARC. Cathy Almond continues her role on the Programme Committee for UKNOF, and ISC sent speakers to these and other events in 2019.

ISC staff authored two IETF drafts that were approved as RFCs in 2019:

• RFC8482 Providing Minimal-Sized Responses to DNS Queries That Have QTYPE=ANY, by J. Abley, O. Gudmundsson, M. Majkowski, E. Hunt; and
• RFC8490 DNS Stateful Operations, by R. Bellis, S. Cheshire, J. Dickinson, S. Dickinson, T. Lemon, T. Pusateri.

Mark Andrews has been working for several years to identify and alert operators with DNS systems that do not fully support the EDNS protocol. The open source development community adopted this campaign as the first target for a DNS-wide cleanup event, marketed as “DNS Flag Day” in 2019. This is almost ironic, as EDNS was designed as a feature that should never have required a “Flag Day” because it was backwards-compatible. What had happened was that over a period of years, the DNS had added more features that relied on EDNS, and developers had added a forest of inefficient workarounds to adapt to non-compliant EDNS implementations. With the community support of most of the open source developers and many of the largest hosted-service providers for removing those workarounds, DNS Flag Day 2019 snowballed into a fairly high-profile event by the time the February “deadline” came around.

ISC was taken by surprise by how popular our hosted EDNS compliance checker became. After being available for several years online it suddenly got hammered with traffic, as the news about the Flag Day caught on in the technical press. We had to rate-limit use of the tool because some operators were testing every one of their zones in this very lightly provisioned online tool.

The controversy over the deployment of open DNS resolvers running DNS over HTTP (DoH) accelerated the debate about the impact of DoH on the overall security of the DNS, and about a shift in control and access to end-user data to hosted providers. ISC joined the new Encrypted DNS initiative. This group is beginning a discussion on work that needs to be done to promote and enable deployment of encrypted DNS.

BIND 9

• We published a policy for gradually removing obsolete BIND 9 options, including community consultation for major changes, option deprecation, and then finally removal of the option. Please read our KB article for more details.
• We updated ISC’s security vulnerability handling policy to reduce the number of minor security risks that result in a CVE. Details are in this KB article.
• We established a new Code of Conduct for the BIND 9 project, which applies to speech both on the bind-user mailing list and in our open Gitlab environment.

Releases

Development: 9.13.6, 9.13.7; 9.15.0 - 9.15.7 (10 releases)
Stable: 9.12.4; 9.14.0 - 9.14.9 (10 releases)
ESV: 9.11.6 - 9.11.14 (10 releases)

The major refactoring project for 2019 was replacing the BIND 9 network socket code with the popular open source libuv library. This will simplify code paths, remove one factor limiting performance, and facilitate using other transport layers (including TLS, HTTPS) in the future.

We also implemented a new dnssec-policy key and signing policy (KASP) for zones, enabling automatic key regeneration and rollover.

Feature changes (the complete log of changes is here: https://gitlab.isc.org/isc-projects/bind9/-/blob/master/CHANGES)

• We added support for the new GeoIP2 geolocation API.
• We added improvements for users resigning large zones - spreading out the signing more evenly, adding new statistics on signing timing.
• NSEC aggressive use, formerly enabled by default, is now disabled by default because it was found that it could take longer to assemble an answer from stored NSEC records than it took to re-query in some cases.
• We implemented the new New DNS Cookie algorithm - siphash24 - replacing the old HMAC-SHA DNS Cookie algorithms.
• We added YAML output for dig, mdig, and delv.
• We deprecated Windows 32-bit support and the DNSSEC Look-Aside Validator (DLV).
• We improved DNSSEC trust anchor configuration using the trust-anchors statement, permitting configuration of trust anchors in DS as well as DNSKEY format. (DNSSEC managed-keys was replaced with trust-anchors plus initial-key, and DNSSEC trusted-keys was replaced with trust-anchors plus static-key.)
• We added validate-except Permanent Negative trust anchors.
• DNSSEC validation is enabled by default now.
• We finally deprecated the DLV (DNSSEC Look-Aside Validator) from BIND

CVEs: We published 9 BIND 9 security vulnerability notices in 2019.

Thanks to our code contributors!

• Tony Finch is again our most prolific external contributor, with 5 entries in the changelog for 2019.
• Petr Mensik made two contributions.
• Thomas Jach made 2 commits.
• Timothe Litt made 2 commits.
• Samuel Thibault and Sergei Trofimovich are new contributors for 2019, with one commit each.

DHCP development (Kea and ISC DHCP)

Tomek Mrugalski continues as co-chair of the DHC working group in the IETF, although active work in that group is winding down.

ISC DHCP migrated to GitLab in 2019 (ISC DHCP was the last ISC open source project to migrate). We gained a number of new DHCP support subscribers and had a very busy support year.

Kea releases in 2019: Kea 1.6.0, 1.6.1, 1.7.0, 1.7.1, 1.7.2, 1.7.3

The major development efforts in Kea in 2019 involved providing a configuration database, enabling centralized control and reuse of complex configuration such as subnet definitions, and designing and developing Kea to evolve into a much higher-performance, multi-threaded application. Although DHCP performance is not a significant issue in the enterprise, we are seeing increased adoption of Kea among service providers, many of whom need to provide addresses for millions of devices.

Major features added in Kea include:

• Database - Configuration backend in MySQL
• Client Classification - “drop” class provides a kind of trash can for unwanted queries
• Max/min - lease-lifetime, a popular feature of ISC DHCP, was added to Kea
• Renew/Rebind timers are now calculated automatically

We migrated our Kea Administrative Reference Manual (ARM) to the Sphinx system, so that users could have live on-line documentation for multiple software versions via Read The Docs. See https://kea.readthedocs.io/en/latest/.

We announced and implemented a new Kea release model with Kea 1.6.0, adding development point releases.

We are still experimenting with the best way of making our Kea open source accessible but also financially sustainable. In 2019 ISC added a Kea Basic subscription offering which includes subscriber-only Kea premium software, but does not include technical support. When we migrated to a static website, we also migrated to a different on-line store, and this has enabled us to sell low-cost “subscription” products.

Thanks to our community code contributors

• Franciszek Gorski, our top external contributor with 19 commits, mostly improving Kea statistics.
• Jonatan Raudsepp submitted a fix for Alpine linux in Perfdhcp code.
• Kristoffer Larsen fixed a bug in Makefile.
• Brent Bloxam fixed a bug in kea-admin that assumed the PostgreSQL database was always local.

We added some new KB articles on Kea:

• Kea Performance Optimization
• Kea DHCPv6 Design Considerations
• Using the Kea Configuration Backend
• Understanding Client Configuration

CVEs: We announced 3 Kea security vulnerabilities in 2019.

ISC DHCP releases

We continue to maintain ISC DHCP, while encouraging migration to Kea. We did not do any releases in 2019, but we did commit work in the open repository. As 2019 ends we are in beta for an ISC DHCP maintenance version, with 4.1-ESV-R16b1 and 4.4.2b1 posted before the holidays. We expect to finalize these versions in early 2020.

Thanks to our external reporters and contributors

• Jay Doran from Bluecat Networks suggested we add a ping-timeout knob.
• Jinmei Tatuya at Infoblox reported an issue which caused DDNS updates to be carried out over TCP rather than UDP.
• Fernando Soto from BlueCat Networks reported a dual-stack mixed-mode issue.
• Peter Nagy at Porion-Digital reported an issue that was causing the server, when running in DHPCv4 mode, to segfault when class lease limits are reached. He also submitted a patch.
• Brett Neumeier reported a minor issue with warnings when compiled with GCC 9.
• Tommy Smith updated the URL of IEEE oui.txt in contrib/dhcp-lease-list.pl.
• Joe LeVeque reported a problem with the define flags when using SO_BINDTODEVICE.
• Brad Smith at OpenBSD alerted us to a patch from OpenBSD to always set the scope id of outbound DHPCv6 packets (which only applies when compiling under OpenBSD).
• Jay Doran at BlueCat Networks reported that the “d” domain name option format was incorrectly handled as text instead of RFC 1035 wire format.
• Charles Hedrick pointed out a syntax error in ldap.c which cropped up under Ubuntu 18.04.1/gcc 7.4.0.

CVEs: We did not announce any ISC DHCP security vulnerabilities in 2019.

Technical Support

Customers

Our support business was strong in 2019. We gained 24 new accounts and only 9 left us. All the rest renewed, for a total of 122 at the end of the year. We added a new support engineer, Peter Davies, who has several decades of experience with both BIND 9 and DHCP. Although he lives in Copenhagen now, Peter’s first language is Welsh, bringing ISC’s total number of native languages spoken to eleven (English [American, British, AND Australian], Polish, Czech, French, Romanian, Welsh, German, Dutch, and Portuguese).

Nearly half of our customers opt for the Silver support level, which includes our premium software and 7x24 response to critical issues. Many open source users tell us that they need the option of premium software to justify paying for technical support.

BIND 9 support is our most popular product, but a number of our customers opt for support for multiple open source systems. The strongest trend we saw in 2019 was an uptick in support contracts for the Kea DHCP server.

Most of our customers are Internet service providers, carriers, or large enterprises.

Support Issues and Trends

• We saw a number of significant but intermittent issues with large zone updates (usually for RPZ), impacting BIND 9 performance.
• We received several requests for more metrics and performance improvements for managing ongoing DNSSEC signing and zone propagation, especially for large zones.
• DoH, DoT, and BIND 9 feature availability were asked about frequently.
• There were many requests for EDNS Client Subnet Identifier, often from users interested in using the feature creatively to enable some sort of customized response.
• Questions about Kea high-availability mode, and how to test and monitor that the function is working when there is no failure, became more prevalent. Many new users are looking for advice on Kea operations and asking about best practices for scaling and provisioning.
• As usual a number of queries were from users interested in optimizing their performance, whether for Kea or BIND 9.
• Farsight released their RPZ Services module, which we were able to offer to BIND -S edition customers. This is a plug-in for BIND that works with the BIND RPZ Service interface.
• We have gotten some new customers for ISC DHCP software, and are satisfied to continue maintaining it for the foreseeable future.
• Although ISC and others promoted it widely via conferences and blog posts, the 2019 Flag Day still took some of our users and customers by surprise. We resolved to do a better job of communicating the 2020 Flag Day via direct mail to ISC support customers and other means.
• We restarted our Support customer newsletter, sending a quarterly email to our non-OEM support accounts. We hope this will strike the right balance between too much information and not enough, and will provide us with a vehicle for communicating about longer lead-time events and changes.

Talks

Sam DeFriez from Community Fibre, a Kea support customer, gave a presentation at UKNOF about his experience in deploying Kea DHCPv6.

ISC staff gave 15 DNS technical talks, including five how-to webinars which were recorded and are available for playback. Slides, and recordings if available, are linked on the Presentations page of our website.

Topics included:

• Encrypted DNS: DoH vs. DoT
• BIND Logging: Content Out of Chaos
• UNIX Command-Line Basics
• DNS Flag Day 2020
• Hyper-hyper Local Root Serving
• DNSKey Flood - What Does it Tell Us About Resolvers?
• A Story on Unsupported DNSSEC Algorithms
• DNSSEC: Your Internet Infrastructure Needs Better Protection
• Making a Modern DNS Server
• Stories from BIND 9 Refactoring
• Migration from ISC DHCP to Kea
• Using the Kea Configuration Backend

F-Root & Technical Operations

We continued to upgrade our network of F-Root Anycast instances and also installed new nodes at Harvard in Boston (US), New York (US), La Paz (Bolivia), Hangzhou (China), and Yogyakarta (Indonesia). When combined with the nodes operated in partnership with Cloudflare, our nodes now number in excess of 250. We consistently appear at or very near the top of the performance charts based on query latency, as reported by dnsperf.com.

We started a project to perform a complete refresh and modernization of the F-Root Operations and Management infrastructure.

We announced the shutdown of the SNS-Public Benefit service, to take effect on 31 January 2020. There are now many other organizations offering low-cost or free DNS secondary services to worthy non-profits, as we concentrate our resources on F-Root.

We were previously providing Internet connectivity services to the City of Palo Alto and various other municipal sites in the Bay Area of California. This service has also now been discontinued.

We joined the NLnog Ring which gives us access to 500+ systems from which we can run network tests, in return for us provisioning a virtual machine on our own infrastructure for use by other ring members.

We took advantage of a Fastly offering for non-profits; we now host https://downloads.isc.org and https://www.isc.org on the Fastly network, to offer the best performance possible to our users worldwide.