Kea 2.0 - Performance, Stability and Security
We are very proud to announce that we have just posted a new stable branch of Kea, Kea 2.Read post
At ISC, 2019 was a solid year of serving our open source users and the Internet community. We continued our engagement with the Internet infrastructure community through work with ICANN, IETF, DNS-OARC, ISOC, UKNOF, NANOG, APNIC and other fora. We joined with our colleagues in other DNS organizations to drive the first “DNS Flag Day” to clean up some lingering technical debt in the DNS system.
We added two new members to the BIND 9 team, Michal Nowak (based in Czechia) and Diego Fronza (in Brazil). The team completed another year of BIND 9 maintenance and development, modernizing BIND’s DNSSEC support, refactoring the network socket code, and establishing both a code of conduct and a new process for removing old features. A major update to Kea added a configuration backend, and we added monthly development releases while supporting a growing Kea support customer base. Our newest Kea team member, Razvan Becheriu (Romania), was an external contributor on the Cassandra backend until we hired him to work on our next big Kea project, multithreading!
We completed the transition to our new Gitlab system with the migration of ISC DHCP, so that we now have integrated issue tracking and development in the open for all our projects. We added many more ISC software packages for our users, in a new ISC Cloudsmith.io repository. We are publishing all of the BIND 9 restricted-access packages (pre-release security fixes and the subscription edition of BIND) and Kea (public plus restricted access) packages there. By making ISC packages we ensure that the latest versions of our applications, and all of their dependencies, are available on the latest stable versions of the major operating systems. We will continue publishing BIND 9 open source packages on the Copr and Launchpad sites, where they are easier for open source BIND 9 users to find.
Our technical support business is stable and even growing modestly. We added 24 new customers in 2019 and only nine left us, for a total of 122 at the end of the year. We needed an additional support engineer and found a very experienced one, Peter Davies, in Copenhagen.
Continuing our tools update from 2018, we migrated from WordPress to a new static web site. The change was motivated by our desire to avoid possible security breaches on the website, and also to improve performance. Our first-ever systems administrator for development systems, Emanuel Petr (Czechia) has been a huge help with this. In the process we also tried to remove all Google tracking tags, a privacy improvement we eventually had to reverse as our website virtually disappeared from the Internet.
We retired ISC’s long-running Domain Survey, which was becoming irrelevant with the adoption of IPv6, and ended the legacy SNS-PB service. SNS-PB was a free network service that ISC provided for over a decade to non-profits and smaller TLDs. Now that secondary name services are now widely available from other providers we would rather focus our network resources on managing F-Root.
Speaking of F-Root, with the retirement of Brian Reid as Director of Network Operations, we reorganized our F-Root and Technical Operations staff. Ray Bellis took over as acting director, and we added a Network Engineer, Elmar Bins, who works from Austria.
We are always looking for opportunities to contribute to open source. We are a long-time user of the open source Etherpad software. Ray Bellis worked on a major refactoring of the code base which formed the bulk of the 1.8.0 release. Ray also published an open source framework for a RADIUS server in Node.JS - https://www.isc.org/blogs/radius-framework/. This implements the RADIUS protocol but does not supply the business logic.
Our greatest thanks go to the loyal open source users who have continued to support ISC, many of whom have been support customers of ours for more than a decade. These include some of the most capable independent operators and enterprises in the world.
We also owe a debt to all the wonderful open source users who have submitted feedback and patches, and the open source packagers who maintain distributions for our users. We regard many of these collaborators as our friends.
In 2019 we received generous unrestricted donations from Verisign and Craigslist, and Mozilla agreed to underwrite development of the DNS over HTTPs support in BIND 9. A few individuals have also made small financial contributions to ISC, via Patreon or directly.
We now have a total of 35 staff members, in 11 countries (the US, Brazil, the UK, France, Austria, Denmark, Poland, Romania, Czechia, Australia, and the Netherlands, if you’re keeping score). Including managers, there are eight on the BIND team; seven on the DHCP team; five in Support; six in Sales and Marketing; four in Finance, Facilities, and Exec; and five in F-Root and Technical Operations. We also have two part-time consultants.
We sadly note the passing of former ISC employee Kurt Lidl, who worked on BIND from ~2001-2003. Several of us worked with Kurt in other capacities, including at UUNET Technologies, and knew him as a dear friend and co-worker. He was a FreeBSD user and advocate of many years standing, and co-authored RFC2516 which put PPP over ethernet. Kurt was smart, funny, and thoughtful, and we will miss him.
ISC is moving out of our long-time headquarters, at an old warehouse at 950 Charter Street in Redwood City, CA. We are working to move out by April 30, 2020. Most of our staff already work remotely, but vacating the HQ means relocating our remaining network facilities, records, and business systems elsewhere. We are primarily a virtual company, with contributors in 11 countries and seven US states, but will be headquartered in the New England region of the US, in an old mill building. The physical address will be 53 Main Street, Newmarket, New Hampshire, and our new mailing address will be PO Box 360, Newmarket, NH 03857.
We started a new open source project, Stork, which will be a graphical dashboard and management application for BIND 9 DNS and Kea DHCP. We expect to do our first public release in the first half of 2020, but anyone who is very eager can follow the development on our GitLab.
Our goal with Stork is to provide a more complete solution for DNS and DHCP, enabling more organizations to use the open source by relieving them of some of the burden of developing or integrating their own management solution. We are leveraging two very popular open source projects, Prometheus and Grafana, for Stork’s time-series data store and data visualization, and using the modern Angular framework for the GUI.
We are excited to be adding HTTPS and TLS transports to BIND in 2020, to enable both leading options for encrypted DNS.
The Kea team, in addition to driving the Stork project, is hard at work on enabling multi-threading to take Kea performance to a new level.
ISC staff continue to participate vigorously in DNS community events and organizations. In 2019, Ondřej Surý was re-elected to the Board of Directors of DNS-OARC. Cathy Almond continues her role on the Programme Committee for UKNOF, and ISC sent speakers to these and other events in 2019.
ISC staff authored two IETF drafts that were approved as RFCs in 2019:
Mark Andrews has been working for several years to identify and alert operators with DNS systems that do not fully support the EDNS protocol. The open source development community adopted this campaign as the first target for a DNS-wide cleanup event, marketed as “DNS Flag Day” in 2019. This is almost ironic, as EDNS was designed as a feature that should never have required a “Flag Day” because it was backwards-compatible. What had happened was that over a period of years, the DNS had added more features that relied on EDNS, and developers had added a forest of inefficient workarounds to adapt to non-compliant EDNS implementations. With the community support of most of the open source developers and many of the largest hosted-service providers for removing those workarounds, DNS Flag Day 2019 snowballed into a fairly high-profile event by the time the February “deadline” came around.
ISC was taken by surprise by how popular our hosted EDNS compliance checker became. After being available for several years online it suddenly got hammered with traffic, as the news about the Flag Day caught on in the technical press. We had to rate-limit use of the tool because some operators were testing every one of their zones in this very lightly provisioned online tool.
The controversy over the deployment of open DNS resolvers running DNS over HTTP (DoH) accelerated the debate about the impact of DoH on the overall security of the DNS, and about a shift in control and access to end-user data to hosted providers. ISC joined the new Encrypted DNS initiative. This group is beginning a discussion on work that needs to be done to promote and enable deployment of encrypted DNS.
Development: 9.13.6, 9.13.7; 9.15.0 - 9.15.7 (10 releases)
Stable: 9.12.4; 9.14.0 - 9.14.9 (10 releases)
ESV: 9.11.6 - 9.11.14 (10 releases)
The major refactoring project for 2019 was replacing the BIND 9 network socket code with the popular open source libuv library. This will simplify code paths, remove one factor limiting performance, and facilitate using other transport layers (including TLS, HTTPS) in the future.
We also implemented a new
dnssec-policy key and signing policy (KASP) for zones, enabling automatic key regeneration and rollover.
Feature changes (the complete log of changes is here: https://gitlab.isc.org/isc-projects/bind9/-/blob/master/CHANGES)
trust-anchorsstatement, permitting configuration of trust anchors in DS as well as DNSKEY format. (DNSSEC
managed-keys was replaced with
initial-key, and DNSSEC
trusted-keys was replaced with
validate-exceptPermanent Negative trust anchors.
CVEs: We published 9 BIND 9 security vulnerability notices in 2019.
Tomek Mrugalski continues as co-chair of the DHC working group in the IETF, although active work in that group is winding down.
ISC DHCP migrated to GitLab in 2019 (ISC DHCP was the last ISC open source project to migrate). We gained a number of new DHCP support subscribers and had a very busy support year.
Kea releases in 2019: Kea 1.6.0, 1.6.1, 1.7.0, 1.7.1, 1.7.2, 1.7.3
The major development efforts in Kea in 2019 involved providing a configuration database, enabling centralized control and reuse of complex configuration such as subnet definitions, and designing and developing Kea to evolve into a much higher-performance, multi-threaded application. Although DHCP performance is not a significant issue in the enterprise, we are seeing increased adoption of Kea among service providers, many of whom need to provide addresses for millions of devices.
We migrated our Kea Administrative Reference Manual (ARM) to the Sphinx system, so that users could have live on-line documentation for multiple software versions via Read The Docs. See https://kea.readthedocs.io/en/latest/.
We announced and implemented a new Kea release model with Kea 1.6.0, adding development point releases.
We are still experimenting with the best way of making our Kea open source accessible but also financially sustainable. In 2019 ISC added a Kea Basic subscription offering which includes subscriber-only Kea premium software, but does not include technical support. When we migrated to a static website, we also migrated to a different on-line store, and this has enabled us to sell low-cost “subscription” products.
Thanks to our community code contributors
We added some new KB articles on Kea:
CVEs: We announced 3 Kea security vulnerabilities in 2019.
ISC DHCP releases
We continue to maintain ISC DHCP, while encouraging migration to Kea. We did not do any releases in 2019, but we did commit work in the open repository. As 2019 ends we are in beta for an ISC DHCP maintenance version, with 4.1-ESV-R16b1 and 4.4.2b1 posted before the holidays. We expect to finalize these versions in early 2020.
Thanks to our external reporters and contributors
CVEs: We did not announce any ISC DHCP security vulnerabilities in 2019.
Our support business was strong in 2019. We gained 24 new accounts and only 9 left us. All the rest renewed, for a total of 122 at the end of the year. We added a new support engineer, Peter Davies, who has several decades of experience with both BIND 9 and DHCP. Although he lives in Copenhagen now, Peter’s first language is Welsh, bringing ISC’s total number of native languages spoken to eleven (English [American, British, AND Australian], Polish, Czech, French, Romanian, Welsh, German, Dutch, and Portuguese).
Nearly half of our customers opt for the Silver support level, which includes our premium software and 7x24 response to critical issues. Many open source users tell us that they need the option of premium software to justify paying for technical support.
BIND 9 support is our most popular product, but a number of our customers opt for support for multiple open source systems. The strongest trend we saw in 2019 was an uptick in support contracts for the Kea DHCP server.
Most of our customers are Internet service providers, carriers, or large enterprises.
Support Issues and Trends
Sam DeFriez from Community Fibre, a Kea support customer, gave a presentation at UKNOF about his experience in deploying Kea DHCPv6.
ISC staff gave 15 DNS technical talks, including five how-to webinars which were recorded and are available for playback. Slides, and recordings if available, are linked on the Presentations page of our website.
We continued to upgrade our network of F-Root Anycast instances and also installed new nodes at Harvard in Boston (US), New York (US), La Paz (Bolivia), Hangzhou (China), and Yogyakarta (Indonesia). When combined with the nodes operated in partnership with Cloudflare, our nodes now number in excess of 250. We consistently appear at or very near the top of the performance charts based on query latency, as reported by dnsperf.com.
We started a project to perform a complete refresh and modernization of the F-Root Operations and Management infrastructure.
We announced the shutdown of the SNS-Public Benefit service, to take effect on 31 January 2020. There are now many other organizations offering low-cost or free DNS secondary services to worthy non-profits, as we concentrate our resources on F-Root.
We were previously providing Internet connectivity services to the City of Palo Alto and various other municipal sites in the Bay Area of California. This service has also now been discontinued.
We joined the NLnog Ring which gives us access to 500+ systems from which we can run network tests, in return for us provisioning a virtual machine on our own infrastructure for use by other ring members.
We took advantage of a Fastly offering for non-profits; we now host https://downloads.isc.org and https://www.isc.org on the Fastly network, to offer the best performance possible to our users worldwide.
What's New from ISC