DHCP server answers with delay

[Denis Taranushin]

> Has it occurred to you that the two may be related ?
 > Is this a new problem, or has it always been like that ? If it's new, 
has anything changed ?
I don't know. I found this trouble in time migration to other billing.

 > How much memory is the DHCP process taking ? Does the host have 
enough ? Is it swapping ? What is disk I/O like ?
I/O is ok. The DHCP is run on ESX with good a sas storage (RAID50 on 22 
HDDs). And the storage not overloaded.

Load, memory and swap:
top - 11:39:48 up 100 days, 11:18,  1 user,  load average: 0.17, 0.15, 0.15
Tasks:  69 total,   1 running,  68 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.4%us,  3.0%sy,  0.0%ni, 89.9%id,  3.7%wa,  0.0%hi,  1.0%si, 
0.0%st
Mem:   2016588k total,  1342708k used,   673880k free,   171932k buffers
Swap:  2044924k total,        0k used,  2044924k free,  1078632k cached


sudo cat /proc/`pidof dhcpd`/status
Name:   dhcpd
State:  S (sleeping)
Tgid:   1959
Pid:    1959
PPid:   1
TracerPid:      0
Uid:    103     103     103     103
Gid:    107     107     107     107
FDSize: 32
Groups: 107
VmPeak:    62056 kB
VmSize:    15948 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:     54240 kB
VmRSS:      8412 kB
VmData:     5956 kB
VmStk:       136 kB
VmExe:       808 kB
VmLib:      8636 kB
VmPTE:        40 kB
VmSwap:        0 kB
Threads:        1
SigQ:   0/15631
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000
SigCgt: 0000000180000000
CapInh: 0000000000000000
CapPrm: 0000000000002400
CapEff: 0000000000002400
CapBnd: ffffffffffffffff
Cpus_allowed:   1
Cpus_allowed_list:      0
Mems_allowed:   1
Mems_allowed_list:      0
voluntary_ctxt_switches:        789074
nonvoluntary_ctxt_switches:     429717

> How many clients are being served ? Is DNS updating in use ?
about 10k on this server.
No DNS updating.

> Can you provide a shortened form of the config ? Eg, if you have a gazzillion class definitions, just provide a couple and note how many there are. There are some things you can do in the config which are "expensive" in resource usage or time, but without seeing it, no-one here can comment.
(Every ip and name has been changed):

cat dhcpd.conf

failover peer "dhcp" {
   primary;
   address 10.0.19.13;
   port 520;
   peer address 10.0.19.14;
   peer port 520;
   max-response-delay 60;
   max-unacked-updates 10;
   load balance max seconds 5;
   mclt 200;
}

ldap-server "1.1.1.236";
ldap-port 389;
ldap-debug-file "/var/log/dhcp-ldap-startup.log";
ldap-username "cn=dhcp1, dc=t, dc=ru";
ldap-password "1111111";
ldap-base-dn "dc=t, dc=ru";
ldap-method static;




ddns-update-style none;
default-lease-time 600;
max-lease-time 600;
authoritative;
log-facility local7;
option domain-name "t.ru";
option domain-name-servers 10.0.0.19, 10.0.0.252;
option space t;
option t.uid code 101 = text;
option t.packet code 102 = text;
option option-43 code 43 = encapsulate t;
option agent.subscriber-id code 6 = text;
class "RELAY_1.1.1.121" {
match if binary-to-ascii(10, 8, ".", packet(24, 4)) = "1.1.1.121";
}
subnet 10.0.32.0 netmask 255.255.255.0 {
}
shared-network "SharedNetwork1" {
subnet 1.0.0.0 netmask 255.255.224.0 {
}

lots of static pools:
subnet 10.0.17.0 netmask 255.255.255.0 {
option routers 10.0.17.1;
option subnet-mask 255.255.255.0;
option interface-mtu 1500;

class "1000051_001" {
match if option agent.subscriber-id = 31:30:30:30:30:35:31:5f:30:30:31;
}
pool {
range 10.0.17.5 10.0.17.5;
allow members of "1000051_001";
}


lots of dynamic pools:

subnet 10.0.18.128 netmask 255.255.255.128 {
option routers 10.0.18.129;
option subnet-mask 255.255.255.128;
option interface-mtu 1500;
class "RELAY_10.0.18.129" {
match if binary-to-ascii(10, 8, ".", packet(24, 4)) = "10.0.18.129";
}
pool {
range 10.0.18.130 10.0.18.254;
allow members of "RELAY_10.0.18.129";
failover peer "dhcp";
}
}