Meet an ISC Engineer - Tony Finch!
Each quarter we like to profile one of our engineers, to acquaint our customers and users with some of the people they may interact with as they use our software.Read post
As a non-profit with few reserves, whether a year was a good one at ISC has a lot to do with our financial status. 2018 was a good year on that score. Most of our funding comes through software support contracts, and in 2018, our support business was strong. We gained 22 new support contracts and lost only 10. As a result, we were able to add quite a lot of staff: three new BIND 9 developers, a senior QA person, a systems and support engineer, and a documentation/communications person. That is a 20% increase in staffing, which is huge. One BIND 9 engineer left and went to one of our OEM partners, and one BIND 9 engineer who had previously left for another open source project returned to ISC.
The biggest event at ISC in 2018 was the migration to GitLab, for both the BIND 9 and Kea DHCP development teams. The ISC-hosted GitLab replaced the separate issue tracker, wikis, and repos for both BIND and Kea with a single development platform. We also are leveraging the continuous integration capabilities of GitLab. We invested quite a bit of effort into getting our automated BIND 9 tests to run in parallel to speed them up. Now nothing is committed to BIND 9 without first passing our continuous integration tests. The Kea team is also making extensive use of the GitLab wiki pages to post design and requirements documents in the open. The adoption of GitLab was fairly disruptive, but overall it has been a positive change.
The shift to GitLab as our primary working repository brought us several unexpected benefits. We opened our issue tracker for BIND so new issues are now open by default; only a few sensitive issues are non-public. (The Kea issue tracker was already open.) We are able to accept contributed patches more gracefully, via merge requests (we used to require email attachments). It seems like we are getting better-quality bug reports, which could be because they are public. We have been pleased to see many people create accounts on https://gitlab.isc.org and start actively opening issues, commenting on existing issues and offering to help (although anyone can browse without an account).
Our support business is a reasonable barometer of how our open source work is being received by users. Many of our new support customers are adopting Kea DHCP, which is gratifying because this project has been under-supported for the past several years. While the IETF work on DHCP is winding down, and the standards-setters are enthusiastic about router-advertised address assignments, DHCP seems to be a continuing core requirement of most networks. We have seen a healthy interest in our “premium” Kea hooks software package, which provides a moderate-cost (US$499) way to support ISC’s open source work.
We are very satisfied with the BIND 9 refactoring we have completed so far. Our initial goal was to reduce software complexity, but we are also adapting to modern cryptography standards and taking more advantage of well-supported LINUX OS features that didn’t exist when BIND 9 was first written. In January we released BIND 9.12, with refactored query logic (query_find and resquery_response). These massive, ridiculously complex functions were split into smaller functions and we created an external library, libns. These changes simplified the query processing code and made it much easier to subsequently implement QNAME minimization. We continued with the refactoring, tackling BIND’s network interface next. The results of that will be seen in 9.14.0, where we expect to see significant improvements in performance on modern hardware.
These are the highlights of 2018 at ISC. We go into more depth, or at least more length, below.
Our busiest quarter for customer support was Q2 (April-June) – and it wasn’t followed by the usual summer holiday slow-down, so our support team has been very busy all year. BIND 9 is a steady workload; ISC DHCP ticket volumes are gradually decreasing, while Kea questions and issues to our support team are steadily increasing. This is to be expected, since we are ending the year with many more Kea support customers than we started it with. We averaged just over six new support issues per week, and the return of Alan Clegg, an experienced, former ISC systems and support engineer, helped us to handle the increased workload.
Screen Shot from kb.isc.org showing the BIND 9 feature matrix
Other support-related activities:
Ondřej Surý, our BIND 9 Development Director, drove a lot of changes in 2018, including ISC’s adoption of GitLab, the addition of three new BIND team members, a new BIND release strategy, and termination of workarounds for legacy systems.
Early in 2018 we implemented the BIND 9 release strategy announced in late 2017; you can read about it at https://www.isc.org/blogs/bind-release-strategy-updated/. The goal was to pare down the number of supported branches to (1) development/unstable, (2) stable, (3) extended support and (4) subscription edition. We accomplished this by declaring End of Life for the 9.9 and 9.10 branches in 2018. It is too early to tell if the new model will result in improving the quality of the early releases on our next stable branch, 9.14, but it has already enabled us to get more users and more feedback on the development branch.
Besides the investment in refactoring described above, 2018 was a year of tough love for BIND users. When BIND 9 was originally written, there was a wider variety of available operating systems, including HP/UX, AIX, BSD, DOS, and Linux, and we proudly supported all of them. Many of these are no longer supported well enough to be viable options for most users, so we decided it was time to cut our support for them and focus on improving BIND for the majority. We didn’t support antiquated software to be stubborn; we did it because we really cared about supporting all our open source users. But we reached a point where that no longer made sense.
Some of the changes:
Windows is still a very popular platform for BIND, but there are no engineers on the BIND team who specialize in it, nor do we have adequate automated test coverage for Windows. We had an external contributor, Rockerinthelocker (later revealed to be Thomas Jach), who contributed several needed Windows fixes and also advised us on some Windows issues.
After removing those legacy system requirements, we were able to modernize our networking stack to take better advantage of threading and reduce context-switching. Witold “hold my beer” Kręcicki spent the last three months of 2018 refactoring the networking code. Through a massive series of commits, he eventually managed to successfully replace our ancient task manager and socket code. The manager uses per-cpu queues for tasks and the network stack runs multiple event loops in CPU-affinitive threads. This greatly improves performance on large systems, especially when using multi-queue NICs. This project is equivalent to removing the bald, flat tires from a car, while it’s in motion, and replacing them with steel-belted radials. We are expecting that these new tires will really pay off when we implement DNS over TLS in 2019.
New features added in 2018 include:
At the end of 2018, we commissioned a BIND 9 project logo from Richard de Ruijter, who also designed the new NLNET Labs project logos. It was nice to be able to afford the small luxury of a professionally-designed logo!
BIND 9 by the Numbers
We issued releases in the following branches of BIND 9 in 2018:
There were six BIND CVEs in 2018.
Here are some approximate numbers of BIND issues created and closed in 2018:
In our pre-GitLab system, there were 96 bug reports closed/rejected and resolved in 2018, and 62 new issues opened.
Our THANKS to the following people for their technical contributions to BIND 9 in 2018
Tony Finch with an impressive 16 commits in GitLab, and five in our pre-GitLab system
Petr Menšík with 12 commits in GitLab, plus one in our pre-GitLab system
We also love to see comments, suggestions, and even complaints from all our users on the bind-users mailing list and in our gitlab.isc.org BIND project.
In 2018 we finally started building a solid base of Kea DHCP support customers, and we began to see more ISC DHCP support customers prepare to migrate to Kea. Our premium hooks packages (sold on our website for US$499) turned out to be a successful experiment that we plan to continue. Our biggest thrill of the year, however, was adding a much-needed new development team member, a senior QA engineer.
In addition, the Kea team spent many hours working on a thorough revision of the DHCPv6 standards, which were standardized in November, 2018 as:
This was a big project: it took 13 revisions to complete this 150-page RFC, which obsoleted seven existing RFCs.
We put out a new branch of ISC DHCP, 4.4, which we intend to be the last branch of ISC DHCP. The key changes included:
Other than that, our work on ISC DHCP was mostly limited to fixing reported bugs and responding to support customer issues. We are minimizing new development on ISC DHCP as it gradually nears its end of active maintenance.
We weren’t able to invest much more work into our ISC DHCP to Kea configuration migration tool, which we are currently testing with some of our ISC DHCP support customers. It is expected that the most difficult part of an ISC DHCP configuration to migrate will be the client classification, because ISC DHCP is so flexible. We have been identifying and tracking all the differences between the two projects that potentially could trip up some users as they migrated in a GitLab milestone (https://gitlab.isc.org/isc-projects/kea/-/milestones/6).
ISC DHCP and Kea by the Numbers
We issued two major feature releases of Kea and published our last planned major branch of ISC DHCP. Kea continues to be a very active project with an emphasis on new feature development. Currently, we produce only new feature versions, and we support only two versions at a time. As our user base grows, we are considering introduction of parallel stable and development branches, with maintenance-only releases on the stable branch.
New features added in 2018 included, by release:
There were two ISC DHCP CVEs and one Kea CVE in 2018.
Because we migrated our issue tracker, and migrated some of the open issues, all of our bug ticket metrics are somewhat messed up, to be honest.
We opened 393 new Kea issues in Gitlab, all of them in 2018. Some of these were migrated over from our older Trac system. We closed 295 issues in our old Trac system in 2018, 184 of them were for Kea 1.4 milestones, which was completed before we migrated to GitLab. We closed 181 tickets in the Kea Gitlab instance to date. So, we closed at least 181 + 184 = 365 issues, and probably more. At any rate, it seems that we are closing issues roughly as fast as we are opening them, which is a good indication that we are keeping up.
Of the 86 new ISC DHCP tickets, 40 (nearly half) were opened as confidential tickets. 21 of the tickets opened in 2018 were resolved or closed in 2018. Overall, we resolved 26 ISC DHCP tickets and closed another four in 2018.
We thank the following people for their technical contributions to ISC DHCP and Kea DHCP in 2018
The following people submitted patches, reported bugs, or suggested valuable features or changes to ISC DHCP in 2018:
Felix Wilhelm of the Google Security Team
Fernando Soto from BlueCat Networks
Indy, of the Fireball ISO open source project
Jiri Popelka of Red Hat
Naiming Shen and Enke Chen of Cisco
Pavel Zhukov at Redhat
Tim DeNike of Lightspeed Communications
For Kea, our top contributor was Razvan Becheriu. Other 2018 contributors included:
Sunil Mayya (our GSOC student)
In 2018, our research team worked on the following projects:
Other Random Statistics
Presentations by ISC: 17 (https://www.isc.org/mission/webinars/)
ISC Staff: 32 in 9 countries and 8 US states
2018 email traffic on user mailing lists:
Emails for help or information to info-at-isc: 883.
What's New from ISC