Denial of service mitigation techniques? What do you do?

[Paul Keck]

Hello, we're using Internet Systems Consortium DHCP Server V3.0.5-RedHat in
a campus environment.
 
The other day one machine in a random building started pounding the heck out
of the DHCP server, to the extent that our monitoring system was unable to
obtain a lease.  So, even though we could see that many leases were being
fulfilled, I know some must not have been.

Breakdown per minute of log lines:

feta.cc# cat  dhcpd.7 |grep '00:1a:a0:71:2a:72'|cut -c-12|uniq -c
      4 Mar 29 14:46
    171 Mar 29 14:47
   2697 Mar 29 14:48
   2718 Mar 29 14:49
   2520 Mar 29 14:50
   2746 Mar 29 14:51
   2677 Mar 29 14:52
   2712 Mar 29 14:53
   2690 Mar 29 14:54
   2512 Mar 29 14:55
   2684 Mar 29 14:56
   2703 Mar 29 14:57
   2694 Mar 29 14:58
   2681 Mar 29 14:59
   2478 Mar 29 15:00
   2753 Mar 29 15:01
   2717 Mar 29 15:02
   2750 Mar 29 15:03
   2688 Mar 29 15:04
   2572 Mar 29 15:05
   2666 Mar 29 15:06
   2761 Mar 29 15:07
   1402 Mar 29 15:08
      8 Mar 29 15:09
      8 Mar 29 15:18
      2 Mar 29 15:22
      6 Mar 29 15:23
      4 Mar 29 15:28
      4 Mar 29 15:29
      5 Mar 29 15:44
      1 Mar 29 15:46
      2 Mar 29 15:48

Luckily it stopped on its own, but unluckily I was at lunch and didn't catch
it in the act.  It was just one machine that would either take the OFFER and
then DISCOVER again, or not bother taking the OFFER and then DISCOVER again.

This has happened a few times over the years and causes consternation.  What
do you folks do to mitigate this threat?  I trolled the archive and see that
some people get their networking equipment to throttle requests.  I'll
pursue that with our Foundry/Brocade gear but if that doesn't pan out, what
else?  I can see setting up something to look for this situation in the logs
and block any requesting IP with more DISCOVERs than I like, but in this
case it would blackhole an entire building since the router is forwarding
the requests.  Has anyone done that?  Any better ideas?

Also, to the programmers- is DHCPDISCOVER the thing to watch?  What is the
best metric for the "damage" a noisy client is causing?  OFFERs?  ACKs?

Also, my management has taken us over relatively recently and is all about
off-the-shelf solutions and outsourcing.  Do any of the appliances that use
ISC DHCPD (or others for that matter) have a good solution to this problem?

Thanks!

-- 
Paul Keck       pkeck at uga.edu         
University of Georgia                 
EITS Network Operations               mailto:pkeck at ediacara.org
    --Opinions mine.--                Go fighting anomalocaridids!!!