rsyslog not keeping up with dhcpd

[John Wobus]

On Apr 8, 2016, at 4:33 PM, dave c <dhcp at gvtc.drakkar.org> wrote:
> In addition to the likely rate limiting, I've seen where disk write contention between the leases file and the local syslog can cause issues. Which is why I only send my DHCP logs to an external pair of log servers. This allows me to optimize my log servers for nothing but log appending and have enough storage to keep my logs for 13+ months. Meanwhile, the DHCP servers are purring along and only worrying about minor incidental local logging and the leases file.
> 
> Dave

+1

If syslog is having problems, your dhcp activity may be reaching the point where disk synchs
affect dhcpd performance.  Linux commands that show disk activity can show the issue
("why is the system with very light cpu activity doing huge numbers of disk synchs?”).
DHCP’s retries can hide a growing problem, which might cause a trouble that is non-linearly
related to load: no complaints, then suddenly many.

Logging via udp, or logging without synching eliminates some disk load.  For the lease file
writes, there are smart controllers with NV caching, there are solid state disks, there is a
dhcpd feature to batch disk writes (which may be ready for prime-time), or there is
ramdisk, which unfortunately seriously affects the resiliency of the dhcp service if the OS dies.
The dhcp load tester, dhcperf reveals a lot.

John Wobus
Cornell University IT