DHCP pair messed up, second one only running cant get primary up.

Fri Jan 13 18:31:09 UTC 2017

Two ISC dhcpd servers running as a failover pair running Version 4.3.3-P1
compiled, running on Ubuntu 14.04 64 bit

The Servers are 500 gigs of RAID 1 space with 8 gigs of RAM with quad core
Intel(R) Xeon(R) CPU E31225 @ 3.10GHz

The dhcpd.leases file sits in /ramdisk which is a 4 Gb RAM disk to make sure
that the file can be written to very quickly, this method gives us virtually
a 0.0 WA time when viewing with the top command.

The LEASE_HASH size was increased to 1800017 and enabled debugging with
REPORT_HASH_PERFORMANCE 1 in the dhcpd.h file

We are using 6,657 /24 subnets in our pools file

We give out millions IPs each day

Please see below for config files.

This service has been running fine for the last 346 days, last Saturday, for
an unknown reason dhcp-1 server had issues, so we turned if off, and dhcp-2
server took over dhcp-1’s part just fine.

So using omapi I told dhcp-2 that its partner was down in order to keep
dhp-2 working on its own, so the last log file entry in syslog on dhcp-2
was;  

In recent days dhcp-2 is having some difficulty, after comparing a tcpdump
to dhcp logs we see DISCOVER requests coming in, but no offers going back
out, but this is sporadic, but enough to make users call in.

The last time we saw this issue happen is when the LEASH_HASH size was too
low, we are not sure if this is the same issue , maybe because we are only
on one server? 

Also the dhcpd.leases files grow too big for the /ramdisk, so we are each 10
mins catting /dev/null into /ramdisk/dhcpd.lease! file to save space.

So currently only dhcp-2 is running.

I want to try tonight to simply stop both dhcpd services on both servers,
delete the leases file and “touch” new ones then reboot primary first , then
secondary a few mins later.

What  do you think?

Here are my configs. & Thanks
 :)

DHCP-1 dhcpd.conf file

-----------

authoritative;

log-facility local7;

db-time-format local;

option domain-name "dev"; # TODO

# DNS internal

option domain-name-servers xxx.xx.xx.210, xxx.xx.xx.220;

default-lease-time 1200; # 20 minutes to match the default Tim Hortons'
session duration

max-lease-time 3600; # 1h

# Include EITHER the primary configuration

include "/usr/local/etc/dhcp/dhcpd_primary.conf";

# OR the secondary configuration

#include "/etc/dhcp/dhcpd_secondary.conf";

# No service for the local networks

subnet xxx.xx.0.0 netmask 255.255.255.0 { }

subnet xxx.xx.128.0 netmask 255.255.255.0 { }

subnet xxx.xx.129.0 netmask 255.255.255.0 { }

# All IP ranges for TDL stores

# This file should be automatically generated using the command:

#       ./make_ranges.pl < ranges > dhcpd_pools.conf

include "/usr/local/etc/dhcp/dhcpd_pools.conf";

# Non-standard IP ranges (i.e. big stores)

include "/usr/local/etc/dhcp/dhcpd_special_pools.conf";

pid-file-name "/run/dhcpd.pid";

ddns-update-style none;

omapi-port 7911;

omapi-key omapi_key;

key omapi_key {

     algorithm hmac-md5;

     secret xxxxxxxxxxxxxxxxy==;

}

DHCP-1 dhcpd_primary.conf

## PRIMARY

failover peer "dhcp-failover" {

  primary; # declare this to be the primary server

  address xxx.xx.xx.9;

  port 647;

  peer address xxx.xx.xx.11;

  peer port 647;

  max-response-delay 30;

  max-unacked-updates 10;

  load balance max seconds 3;

  mclt 1800;

  split 128;

}

DHCP-2 dhcp-2.conf

----

authoritative;

log-facility local7;

db-time-format local;

option domain-name "tdl"; # TODO

# DV DNS internal

option domain-name-servers XXX.XX.XX.210, XXX.xx.xx.220;

default-lease-time 1200; # 20 minutes to match the default Tim Hortons'
session duration

max-lease-time 3600; # 1h

### The below commneted as we are to be independant server - Rob Jan 28th
2016

# Include EITHER the primary configuration

#include "/etc/dhcp/dhcpd_primary.conf";

# OR the secondary configuration

include "/usr/local/etc/dhcpd_secondary.conf";

# No service for the local networks

subnet xxx.xx.0.0 netmask 255.255.255.0 { }

subnet xxx.xx.128.0 netmask 255.255.255.0 { }

subnet xxx.xx.129.0 netmask 255.255.255.0 { }

# All IP ranges for TDL stores

# This file should be automatically generated using the command:

#       ./make_ranges.pl < ranges > dhcpd_pools.conf

include "/usr/local/etc/dhcpd_pools.conf";

# Non-standard IP ranges (i.e. big stores)

include "/etc/dhcp/dhcpd_special_pools.conf";

pid-file-name "/run/dhcp-server/dhcpd.pid";

ddns-update-style none;

omapi-port 7911;

omapi-key omapi_key;

key omapi_key {

     algorithm hmac-md5;

     secret xxxxxxxxxxxxxxx==;

}

DHCP-2 dhcpd_secondary.conf

---

## SECONDARY

failover peer "dhcp-failover" {

secondary; 

 address XXX.xx.128.11;

port 647;

peer address xxx.xx.128.9;

peer port 647;

max-response-delay 30;

max-unacked-updates 10;

load balance max seconds 3;

}

Dhcp pools file  snip it
 over 6000 subnets 

--

subnet 10.32.0.0 netmask 255.255.255.0 {

  option routers 10.32.0.1;

  pool {

        failover peer "dhcp-failover";

        range 10.32.0.5 10.32.0.254;

  }

}

  too long to list :)

subnet 10.57.255.0 netmask 255.255.255.0 {

  option routers 10.57.255.1;

  pool {

        failover peer "dhcp-failover";

        range 10.57.255.5 10.57.255.254;

  }

}

Rob Morin

Gestionnaire des systèmes | Senior Systems Administrator

Tel: 514 385-4448 #174                         

DATAVALET.COM

5275, chemin Queen-Mary, Montréal (Québec) H3W 1Y3 Canada

CE COURRIEL AINSI QUE CES DOCUMENTS JOINTS peuvent contenir des
renseignements confidentiels et privilégiés. Si vous n’êtes pas le
destinataire désigné, veuillez nous en informer immédiatement et effacer
toute copie. Merci.

THIS EMAIL AND THE DOCUMENTS ATTACHED may contain privileged or confidential
information. If the reader of this message is not the intended recipient,
please notify the sender immediately and delete the original message. Thank
you.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/dhcp-users/attachments/20170113/78163eab/attachment-0001.html>