Coping with tight address supply when failover member dies

[Nick Urbanik]

Dear Folks,

We have five DHCP server failover pairs, running ISC DHCP version
3.0.4 with some local modifications.

Each member of the failover pair has some address pools that are about
90% or more used.

When a failover member dies (hard disk faile,...), there is a delay
till the box is replaced with a new one, during which time some pools
run out on the partner.  The last time a new partner was brought on
line, it took about one hour for the new primary to transfer the 200MB
of leases from the secondary, and for the new primary to begin
operation.  During that time, perhaps three thousand people were
unable to get new leases.  This is suboptimal.  I need to come up with
a strategy that will cope with such a situation better next time.  I
am able and willing to write code towards that end.

The machine that failed has about 76000 active leases in tight supply,
and the same number again in a loose supply, so about 152,000 active
leases all together.

Note that restarting takes over five minutes, and doing so with no
partner would cause an unacceptable outage.

QUESTIONS:

1. Is there an OMAPI function that I can use to add addresses to the
    backup pool on a lone secondary or to the free pool on a lone primary?
2. Can I use OMAPI to move addresses between the free and backup
    states?
3. I have had problems after putting a DHCP server into the
    partner-down state where both machines would go into a state where
    neither would offer addresses to customers for a good part of an
    hour (somewhat suboptimal :-)  Any suggestions on avoiding that?
4. Would migrating our code to 4.0 (or even 4.1) help in coping with
    the failure of a DHCP failover member?
5. Any other advice anyone may offer?
-- 
Nick Urbanik             http://nicku.org           nicku at nicku.org
GPG: 7FFA CDC7 5A77 0558 DC7A 790A 16DF EC5B BB9D 2C24 ID: BB9D2C24