Good old "peer holds all free leases" appearing in a (formerly) very stable setup

[Nicolas Ecarnot]

Hi,

We have setup a complex DHCP infrastructure 5 years ago, and it is very 
very stable and working very nicely :
- 10 rhel5 servers, serving around 150 shared-networks for 150 physical 
sites
- in every network resides mostly one pool, sometimes 2, and rarely more
- every pool is "failovered" by a couple of two physical servers
- we are using DhcId classes successfully
- our router are using dhcp-helper setup to correctly relay the dhcp traffic

Tests were made, time has passed, all this setup is SVN-versionned, we 
frequently change things/values/subnets, but we never change the basics 
described above, we use it every day and we think we have a clear view 
of it. It is all working FINE.

Today, in a subnet, I just wanted to add a pool which access is 
restricted to a new class. As usual, I wanted this pool to be manageable 
by a server and its failover peer.

What I get is the good old "peer holds all free leases" error, that I 
learned NOT to trust since years, as it appeared even when not related 
at all to leases issues.

The subnet looks like :

shared-network dhcp51-dsi {
   subnet 192.168.51.0 netmask 255.255.255.0 {
     option subnet-mask 255.255.255.0;
     option routers 192.168.51.254;
     option domain-name-servers 192.168.39.215, 192.168.12.215;
   }

   pool {
     allow members of "oldStableClass";
     failover peer "serv-net-adm1_serv-net-adms1";
     deny dynamic bootp clients;
     range 192.168.51.2 192.168.51.229;
   }
   pool {
     allow members of "brandNewClass";
     failover peer "serv-net-adm1_serv-net-adms1";
     deny dynamic bootp clients;
     range 192.168.51.230 192.168.51.241;
   }
}

You can see there are 2 servers implied :
- serv-net-adm1
- serv-net-adms1
Our dhcp test client is a windows XP which NIC got setup with a dhcid 
class like its thousand office neighbors around.

Here are strange things I witnessed :
- When restarting the daemon on adms1, the log file is showing the 
shared-networks it will manage, but sometimes I see "dhcp51-dsi" missing...
- I tried to completely remove the failover setup, and to entrust 
net-adm1 only to manage this subnet. The error message is changing into 
"No free leases". This makes no sense to me as I think/know the pool is 
far from full. I'm using the dhcpd-pools to confirm it and it agrees 
with me.

I had a (read-only) look at the lease file, and it does not show any 
reference to this subnet, though it appears in the log file when 
restarting this server (net-adm1).

I must admit I'd be glad to get some advices on things to check, even 
the obvious ones I may have skipped.

You'll find below the versions used and you'll note these are not recent 
ones, but I think that is not relevant because this whole setup is 
running fine as is since years and no additional weird features is 
required there.

- Redhat RHEL 5.3 64 bits
- isc dhcp 3.0.5-18.el5

Regards,

-- 
Nicolas Ecarnot