How Somebody Helped Kill dhcpd on Our Network

[Tim Peiffer]

We used to have a problem with rogue servers when the big marketing push 
for NAT routers and wireless complete with DHCP began hitting the 
street.   Since then, we have placed filters on the edge ports to deny 
DHCP server traffic.  Now we only have an issue if a new server pops 
up.  The clients can't get to their server until we remove the filter 
from the server port.

This isn't rate limiting, but it is quite effective at handling the 
original source of the problem


In Cisco parlance:

interface G1/0/1
  ip access-group No_DHCP_SERVER in
!
ip access-list extended Access_IN
  deny udp any eq bootps any log
  permit ip any any

Tim Peiffer

Darren wrote:
> Martin,
>
> I suggest Vlans.  Set up as many Vlans as you can and separate the user 
> traffic.  The rogue DHCP server may not have been malicious in nature.  
> All someone has to do is hook a linksys router up backwards and this 
> will happen.  Its worse when all 10,000 users are in the same broadcast 
> domain.  If it was one vlan of 253 users or something, the damage would 
> have been minimal.  The rogue DHCP server would not have seen the vast 
> majority of user traffic, and therefore would not have been able to 
> respond to it with DHCPNAK.
>
>
> Martin McCormick wrote:
>   
>> 	We recently had our dhcp V3.0.3 system crash or, I should
>> say, crashed by a denial-of-service scheme in which the miscreant
>> ran something on his/her computer that behaved like a dhcp server
>> enough to send DHCPNAK's to every dhcp request it saw on the
>> VLAN.  This made every client send more traffic to the real dhcp
>> server and apparently caused the 1-gig processor with 1 gig of
>> RAM to consume all available memory.  For 3 minutes, it generated
>> about 2,500 "out of memory" messages and then dhcpd finally
>> exited with a 11) SIGSEG and a core dump.  Good old FreeBSD unix
>> is pretty bullet-proof, but this was more than the box could
>> handle.  The platform continued to operate properly afterwards,
>> but dhcpd had to be restarted again.
>>
>> 	We are in the process of instituting dhcp failover with a
>> second server although I suspect that this situation would have
>> added maybe another few minutes to the mayhem before it, to,
>> succumbed to the hammering since failover isn't going to protect
>> against external attacks that hit both servers equally.
>>
>> 	We are also installing switches with port snooping
>> capabilities as funds permit to kill off anybody who aspires to
>> run dhcpd on anything other than the proper dhcp server.  This
>> brings me to my question.
>>
>> 	Is there any particular configuration parameter I can use
>> in dhcpd to make it rate-limit itself since it would have been a
>> much better outcome for it to come up for air, so to speak,
>> rather than crash and have to be restarted.
>>
>> 	This particular event occurred on a Sunday afternoon on
>> the weekend that classes finished for the Summer semester and
>> campus activity was as near to dead as it ever gets around here
>> so we didn't even know anything was wrong for a period of time.
>>
>> 	Our dhcp server gives service to around 10,000 clients of
>> several types and the platform it is on hardly breaks a sweat
>> even on the busiest days, but this was too much to handle.
>>
>> Martin McCormick WB5AGZ  Stillwater, OK 
>> Systems Engineer
>> OSU Information Technology Department Network Operations Group
>>   
>>