Recovering from over enthusiastic key cleanup...

[Warren Kumari]

Hi all,

So, I decided to roll keys on a test zone (af7.org) -- of course, I decided to do this a: late at night and b: while juggling many other things.

So, I generated a new key and submitted my DS to my registrar, and deleted an older one - so far, all good, everything working fine. Problem solved, off to bed...

Oh! Hang on a sec, my keys directory is cluttered with old keys. I've just deleted the DS for one of them from the registrar, so it ain't being used, guess I'll remove the key files. 
rm Kaf7.org.+005+27780.* 
Yay, everything is still working, clean up some unrelated stuff, now really off to bed. Type 'rndc sign af7.org' for giggles, and.... well... 

Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.578 general: info: received control channel command 'sign af7.org'
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.578 general: info: zone af7.org/IN/external: reconfiguring zone keys
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.579 general: warning: dns_dnssec_keylistfromrdataset: error reading private key file af7.org/RSASHA1/27780: file not found
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.580 general: warning: dns_dnssec_findzonekeys2: error reading private key file af7.org/RSASHA1/27780: file not found
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.581 general: notice: zone af7.org/IN/external: setting keywarntime to 01-Feb-2012 22:19:57.578
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.592 general: warning: zone af7.org/IN/external: Key af7.org/RSASHA1/27780 missing or inactive and has no replacement: retaining signatures.
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.603 general: info: zone af7.org/IN/external: next key event: 01-Feb-2012 23:19:57.603
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.604 general: warning: dns_dnssec_findzonekeys2: error reading private key file af7.org/RSASHA1/27780: file not found
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.699 xfer-out: info: client 75.102.1.178#59905: view external: transfer of 'af7.org/IN': IXFR started
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.699 xfer-out: info: client 75.102.1.178#59905: view external: transfer of 'af7.org/IN': IXFR ended
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: rdata.c:393: REQUIRE(((rdata)->data == ((void *)0) && (rdata)->length == 0 && (rdata)->rdclass == 0 && (rdata)->type == 0 &&\
 (rdata)->flags == 0 && !((void *)(((rdata))->link.prev) != (void *)(-1)))) failed, back trace
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #0 0x413f2c in assertion_failed()+0x4c
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #1 0x57a97a in isc_assertion_failed()+0xa
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #2 0x4cc384 in dns_rdata_fromregion()+0x64
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #3 0x4ada8a in rdataset_current()+0x5a
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #4 0x540fb0 in del_sigs()+0x230
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #5 0x5515d7 in zone_sign()+0x7b7
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #6 0x555116 in zone_timer()+0x166
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #7 0x596ef9 in run()+0x1c9
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #8 0x7fac41b94971 in _fini()+0x7fac415e9699
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: #9 0x7fac418f092d in _fini()+0x7fac41345655
Feb  1 22:19:57 vimes named[19281]: 01-Feb-2012 22:19:57.710 general: critical: exiting (due to assertion failure)


Oh. Well, that is sad. Restart BIND. Boom, dies again...
Erm, restore keyfiles from backup (after trying to remember how the restore works, and what passphrase I used for this GPG key)...
Still no love. 

Ended up removing the zone stanza from named.conf so I could start BIND and have a working nameserver, then running ldns-read-zone -s af7.org | grep -v DS | grep -v TYPE65 > af7.org, re-enabling the zone stanza, resigning with new keys, submitting new DS, etc...

So, is there:
A: an easy way to figure out what keyfiles are no longer being used / referenced?
B: a simpler way to recover from this when one *does* make a boo boo?

BIND apparently *tried* to continue running without being able to access the keyfile ("Key af7.org/RSASHA1/27780 missing or inactive and has no replacement: retaining signatures.") but then went "Boom".

W