dhcpd sporadically ignoring one of two "copies" of a DHCP request

[Phil Mayers]

We run RHEL6 with the stock version of dhcpd from the distro, 
4.1.1-34.P1.el6. I'm aware this is somewhat old, but it's been 
relatively trouble free for us.

Our network is a mix of Cisco/Juniper, and most subnets have resilient 
routing, which means that each DHCP request is seen, and forwarded, by 
two router DHCP relays.

This means we'll often see:

DHCPDISCOVER via router1
DHCPDISCOVER via router2
DHCPOFFER via router1
DHCPOFFER via router2

...or, depending on timing:

DHCPDISCOVER via router1
DHCPOFFER via router1
DHCPDISCOVER via router2
DHCPOFFER via router2

The DISCOVER messages are identical aside from source IP and giaddr. 
They do of course arrive very close together in time - within a couple 
of milliseconds, usually.

The vast majority of the time, this seems to work OK; whichever DISCOVER 
arrives first triggers lease allocation, and the 2nd DISCOVER just hands 
out the same lease.

However, I am seeing odd cases where the 2nd DISCOVER is apparently 
ignored. dhcpd does not log having received it, and does not respond to 
it. The packet is definitely arriving at the machine, as confirmed by 
local tcpdump, but no reply goes out.

For the subnet in question there is no obvious config difference from 
other working subnets, and no wider issue with connectivity to/from them.

Unfortunately this is a real problem, because in many cases, one of the 
two replies will be dropped by uRPF check at the far end. A working 
reply might have:

ip src=dhcpsrv dst=router1
path: routerA -> routerB -> router1

...but a failing reply might have:

ip src=dhcpsrv dst=router2
path: routerA -> routerB -> router1 -> (subnet at layer2) -> router2

i.e. path to "router2" giaddr is via router1 from that position in the 
network. When the packet arrives at router2, it has an invalid source 
and fails the uRPF check.

If the packet dhcpd ignores is the one from giaddr==router1, the client 
can't get a lease.


So: why would dhcpd sporadically ignore a 2nd DISCOVER? How can I debug 
this?