redundant bump-in-the-wire signers using BIND

[Michael Sinatra]

Hi all:

First, let me explain the trade-off I am trying to manage (as succinctly
as possible):

My current setup has an DNS/IPAM system that backs up to a redundant one
in a different location, a bump-in-the-wire hardware signing appliance
(different from the IPAM), and a bunch of authoritative slaves running
BIND in an anycast cloud.

+---------+     +-----------+
|  IPAM   | --> | HW Signer | --> (Slaves)
+---------+     +-----------+

The HW signer also backs up to a redundant one elsewhere.  Failover for
both can either be manual or a complex combination of heartbeats and
synchronization that yields an active-standby arrangement.  I choose
manual here because our needs don't require immediate, seamless
failover, and I don't want the complexity.

Now, I'd like to replace the HW signers with VMs running BIND.  HSM is
not a requirement for me.  I can run dnssec-keymgr to generate keys on
one of the BIND VMs and then rsync the keys to the other (again,
geographically redundant).  The configurations are similarly synched
between the two.

I am trying to go for reasonable, practical security for the secret
keys, but I also want them backed up to one other location.  So having 2
geographically redundant VMs that can be locked down so that they only
talke to the IPAM and the slaves seems reasonable.  Putting secret keys
on all of the slaves, and having them do inline signing seems a bit more
reckless.  (There are other reasons I'd like to maintain the
bump-in-the-wire method, but I won't go into them now.)

My initial thought is that, as long as I can keep the keys and config in
sync between the two signing VMs, I can keep both VMs running bind *and*
can treat them both as stealth masters and have all of the slaves
configured to use both VMs as masters.  This obviates the need for
manual failover of the *signers*.

This seems to work in practice with a legacy zone that I am using for
testing.  The two signers sometimes sign at slightly different times, so
the signatures are different, but they both produce valid signatures.

My only concern is that serial numbers might get out of sync between the
two signers at some point.  If signer 1 is down for an extended period
of time, signer 2 may go through a few re-sign cycles and have a
consistently higher serial number in the case of a zone that isn't
modified much.

I can see two possible solutions: 1. manually "jiggle the handle" on the
IPAM by bumping the serial number to be greater than that of the larger
signer's SOA serial.  (The IPAM uses date/time serial format, so this
should be easy.) 2. use 'rndc signing -serial' to set the serial number,
possibly in concert with a monitoring script that checks to see if the
serial is out of sync.

Has anyone implemented anything like this?  Any other gotchas I should
be thinking about?  BIND does a good job of doing the right thing with
serial numbers, but I have yet to see (via google-foo or even bing-foo)
any evidence of anyone trying to do an active-active redundant
configuration with BIND inline-signing.

thanks!
michael