dhcpd memory leak (was: Re: dhcpd process hitting data size limit)

[sthaug at nethelp.no]

Following up on a thread from early March:

> At my servers the size of the dhcpd process is about 20 MBytes at 
> startup. After about a week the size has increased to about 1 Gbyte. If 
> the dhcpd has not been restarted due to config changes, I usually have 
> to restart once a week due the the memory use. The problem with size 
> growing has been there as far back as I can remember.

Having monitored my DHCP servers rather intensively lately, I'm seeing
a similar behavior - looks like a rather clear memory leak. The first
few hours after startup, the memory usage is constant, e.g.

20080309.0940 93972 93508 /usr/local/sbin/dhcpd
20080309.0945 93972 93508 /usr/local/sbin/dhcpd
20080309.0950 93972 93508 /usr/local/sbin/dhcpd
20080309.0955 93972 93508 /usr/local/sbin/dhcpd
20080309.1000 93972 93508 /usr/local/sbin/dhcpd
20080309.1005 93972 93508 /usr/local/sbin/dhcpd

where the numbers are date/time, process virtual size (VSZ), process
real memory size (RSS). After a few hours, *something* happens which
clearly triggers a rather significant memory leak:

20080309.1735 93972 93508 /usr/local/sbin/dhcpd
20080309.1740 93972 93508 /usr/local/sbin/dhcpd
20080309.1745 93972 93508 /usr/local/sbin/dhcpd
20080309.1750 93972 93508 /usr/local/sbin/dhcpd
20080309.1755 93972 93508 /usr/local/sbin/dhcpd
20080309.1800 94456 93992 /usr/local/sbin/dhcpd
20080309.1805 94936 94472 /usr/local/sbin/dhcpd
20080309.1810 95420 94960 /usr/local/sbin/dhcpd
20080309.1815 95892 95432 /usr/local/sbin/dhcpd
20080309.1820 96368 95908 /usr/local/sbin/dhcpd
20080309.1825 96868 96408 /usr/local/sbin/dhcpd

When the memory leak behavior has started, it lasts until the process
is restarted. I'm seeing a memory leak of around 80 kBytes per minute
with around 100k active leases and 24 hour lease time, in a failover
configuration. Both servers in the configuration show a similar memory
leak - however, the memory leak doesn't start at the same time on both
servers even if the dhcpd processes are restarted at approximately the
same time.

So far I have inspected dhcpd syslog messages for the time when the
memory leak starts, and also looked at tcpdump traces of the actual
traffic to and from the servers. Everything appears normal, as far as
I can see.

Any good suggestions on how to start debugging this? Both servers run
dhcp-3.1.0, OS is FreeBSD 6.3.

Steinar Haug, Nethelp consulting, sthaug at nethelp.no