Watching performance on a DHCP Server

[David W. Hankins]

On Thu, Feb 07, 2008 at 06:07:51PM -0600, Blake Hudson wrote:
> By default in my distribution the leases file is stored in 
> /var/lib/dhcpd/dhcpd.leases. This happens to be on a RAID1 array with 
> 15k scsi disks and iostat shows the array as being maxed out once it 
> reaches ~ 300 I/O's per second. DHCP logging is done asynchronously to 
> the same array (which normally experiences ~ 50 I/O ops). With CPU and 
> memory barely breaking a sweat, this leads me to believe that the 
> limitation is with the disks (lots of tiny writes).
> 
> I could move the leases file to a different array, or to tmpfs, but 
> before I do I just want to know if these results are typical and that I 
> have interpreted the test data correctly and made the correct 
> determination as to the bottleneck.

those results are typical for that kind of hardware, and you have
interpreted the test data correctly: fsync() is the biggest
bottleneck.

in 4.1.0a1, you will find a feature, however, which was provided to
us in a patch by Christof Chen.  it permits the server to queue
multiple ACKs behind a single fsync(); default 28 (576 byte DHCP
packets filling default socket buffer send sizes).  the burst of acks
are sent presently if the sockets go dry, and shortly will be backed
up with a sub-second timeout.

it has some bugs we're working on, particularly with failover, but
we'll address those in alpha.

you may find that it provides some form of multiplicative benefit to
your performance stats, since fsync() is the bottleneck, and now there
are 28 acks per fsync max.

so if you are only pushing 50 requests/s currently, you may live
comfortably in a 250 request/s buffer for some months until the
4.1.x code is stable?

> Also, I would appreciate any anecdotal evidence with regards to how many 
> requests are typical in a large network under normal (or abnormal) 
> conditions. If 10,000 users all of a sudden came online, how many 
> requests would they really generate per second?

there have been a few folks who suffered mass power outages, i don't
know what search query to use, but you can find them on the old
dhcp-server mailing list.  they did not report problems, rather the
surprise at the lack of problem.

-- 
Ash bugud-gul durbatuluk agh burzum-ishi krimpatul.
Why settle for the lesser evil?	 https://secure.isc.org/store/t-shirt/
-- 
David W. Hankins	"If you don't do it right the first time,
Software Engineer		     you'll just have to do it again."
Internet Systems Consortium, Inc.		-- Jack T. Hankins