DLV issue 2020/03/25

[Ray Bellis]

The issue with the dlv.isc.org DNSSEC signatures yesterday (2020/03/25)
was caused by an undetected failure to restore the virtual machine that
runs the hidden master for that zone following a failed upgrade to the
underlying hypervisor.

As a result of this issue the internet facing servers were unable to
fetch the zone from the hidden master and eventually started serving
expired signatures.

The ensuing storm of queries to those servers from resolvers with
outdated configurations and/or software then impeded our ability to
diagnose and correct the issue as quickly as we would have liked.

At some future point ISC would like to completely decommision this zone,
but the number of clients still configured to use it currently makes
that impractical.

Per our announcements and presentations in 2015 through 2017 [1], we
would urge all resolver operators and software packagers to ensure that
DLV is disabled in all configurations.  We have provided some additional
guidance for this on our Knowledge Base.[2]

We apologise for any disruption caused, and will be taking steps to try
to ensure that this does not recur, including improvements to our
monitoring systems.

Ray Bellis
Director of DNS Operations, ISC.

[1] https://www.isc.org/blogs/dlv/
    https://www.isc.org/blogs/dlv-replaced-with-signed-empty-zone/

[2] https://kb.isc.org/docs/disable-dnssec-lookaside-dlv-now-heres-how