Restarting DNS service causes isc-dhcp-server to stop handing out leases

[Andrew Martin]

Hi Glenn,

----- Original Message -----
> From: "Glenn Satchell" <glenn.satchell at uniq.com.au>
> The dhcpd messages look quite normal. Do you restart the two dhcpd daemons
> simultaneously? This may be introducing a subtle race condition in
> establishing the failover communications.
I restarted one of the servers at 22:54:11 and the other at 22:55:59, so
more than 1 minute apart.

> 
> If so it might be better to stagger the restarts by about 1 minute to
> allow the restarting daemon to establish communication with the peer. It
> doesn't really matter which one is restarted first. You will then see some
> messages about loss of communication with the peer when the other server
> restarts, but this is ok.
The two dhcpd daemons are restarted 2 minutes apart already via cron
whenever changes to dhcpd.conf are pushed, so this is already implemented.

> 
> It might also be more efficient to check the modification time on
> dhcpd.conf, and only restart if this has changed, rather than restarting
> every 15 minutes.
On closer examination it looks like the dhcpd logging every 15 minutes is
when the script checks the new version of dhcpd.conf for syntax. The script
performs the following steps:
 1. grab the latest version of dhcpd.conf, perform syntax validation with
    /usr/sbin/dhcpd -t -cf /path/to/temp/dhcpd.conf
 2. compare modification timestamp of active dhcpd.conf and new dhcpd.conf
 3. if timestamps match, quit; else restart isc-dhcp-server

Calling dhcpd with the -t flag appears to log these messages to the syslog:
dhcpd: Internet Systems Consortium DHCP Server 4.1-ESV-R4
dhcpd: Copyright 2004-2011 Internet Systems Consortium.
dhcpd: All rights reserved.
dhcpd: For info, please visit https://www.isc.org/software/dhcp/
dhcpd: WARNING: Host declarations are global.  They are not limited to the scope you declared them in.

Therefore, the only time isc-dhcp-server was restarted was as follows:
22:54:11 - networking restarted
22:54:11 - isc-dhcp-server restarted
22:54:18 - dnsmasq restarted

Restarting it the following morning via "service isc-dhcp-server restart"
did not resolve the problem the first time:
08:38:44 - service isc-dhcp-server restart
08:41:10 - service dnsmasq restart
08:41:13 - kill isc-dhcp-server ; service isc-dhcp-server start

After the restart at 08:41:13 it was working successfully. Perhaps this is
just a rare bug?

Thanks,

Andrew