Infinite loop in uid_hash_add (possible DoS)

[sthaug at nethelp.no]

Setup: Failover pair running 4.1-ESV-R3 on FreeBSD 7.4-STABLE.

I have a situation here where the servers sometimes get into a loop,
using 100% of the CPU at user level, performing no system calls, and
obviously not handling any new DHCP requests.

I have caught it "in the act" several times now, and the loop is in
server/mdb.c, routine uid_hash_add, starting at line 2035:

		for (scan = head; scan -> n_uid; scan = scan -> n_uid) {
			if (scan -> n_uid == lease) {
				lease_dereference (&scan -> n_uid, MDL);
				if (lease -> n_uid) {
					lease_reference (&scan -> n_uid,
							 lease -> n_uid, MDL);
					lease_dereference (&lease -> n_uid,
							   MDL);
				}
				break;
			}
		}

We have:

(gdb) p scan
$1 = (struct lease *) 0x28d284c8

(gdb) p *scan
$2 = {type = 0x28303280, refcnt = 9, handle = 0, outer = 0x0,
inner = 0x0, next = 0x28cdb284, n_uid = 0x28d284c8, n_hw = 0x0,
...
uid_buf = "\000\000\000\000\000\000"

Note that scan = scan->n_uid = 0x28d284c8, which obviously gives 
the infinite loop.

The call hierarchy:

#0  uid_hash_delete (lease=0x28f929a8) at mdb.c:2036
#1  0x08071522 in dhcp_lease_destroy (h=0x28f929a8, file=0x80ceed4 "dhcp.c", line=2490) at omapi.c:415
#2  0x080b75c2 in omapi_object_dereference (h=0xbfbfd2d8, file=0x80ceed4 "dhcp.c", line=2490) at alloc.c:695
#3  0x08051304 in ack_lease (packet=0x2d9ebc90, lease=0x28d284c8, offer=5, when=0,
    msg=0xbfbfd334 "DHCPREQUEST for 193.90.58.237 from 00:26:61:03:a7:1e via 193.90.48.1", ms_nulltp=0, hp=0x0) at dhcp.c:2490
#4  0x080553ef in dhcprequest (packet=0x2d9ebc90, ms_nulltp=0, ip_lease=0x0) at dhcp.c:660
#5  0x08055f35 in dhcp (packet=0x2d9ebc90) at dhcp.c:225
#6  0x0809ce6f in do_packet (interface=0x28306020, packet=0xbfbfd8d4, len=598, from_port=17152, from=
      {len = 4, iabuf = "ÁK\002.\220ñn-\220ñn-Lë¿¿"}, hfrom=0xbfbfe8ea) at options.c:3753
#7  0x08090457 in got_one (h=0x28306020) at discover.c:1433
#8  0x080bb736 in omapi_one_dispatch (wo=0x0, t=0xbfbfecac) at dispatch.c:520
#9  0x08091b38 in dispatch () at dispatch.c:92
#10 0x0804c2af in main (argc=8, argv=0xbfbfed8c) at dhcpd.c:857

The infinite loop is always triggered by a client with uid (client id)
= 7 zero bytes. Here is tcpdump output of one of the offending requests:

14:00:53.843953 IP (tos 0x0, ttl 61, id 0, offset 0, flags [none], proto UDP (17), length 626)
    193.75.2.46.67 > 193.75.4.2.67: BOOTP/DHCP, Request from 00:26:61:03:a7:1e, length 598, hops 1, xid 0xafdf3d76, secs 3777, Flags [none]
          Client-IP 193.90.58.237
          Gateway-IP 193.90.48.1
          Client-Ethernet-Address 00:26:61:03:a7:1e
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Request
            Lease-Time Option 51, length 4: 7200
            RN Option 58, length 4: 4320
            RB Option 59, length 4: 6480
            Client-ID Option 61, length 7: "^@^@^@^@^@^@"
            Parameter-Request Option 55, length 4:
              Subnet-Mask, Default-Gateway, Domain-Name-Server, Domain-Name
            Agent-Information Option 82, length 48:
              Circuit-ID SubOption 1, length 46: ar1.osls:GigabitEthernet 1/3.31090155:3109-155

As far as I can see, the do_id_hash() routine will return zero given a
uid of 7 zero bytes as input - however, I haven't been able to follow
the various hashing/list manipulation routines to see what happens if 
do_id_hash() returns zero.

At any rate,

- The infinite loop has (so far) always been triggered by one specific
client, with a uid (client id) of 7 zero bytes. I suspect there is a
connection but do not know this for certain.

- There is *some* other condition also needed to trigger the problem,
since the same client has managed to aquire and renew a lease several
times without the infinite loop being triggered.

- The loop happens to both servers in the failover pair. This is of
course bad news, since it means there is a semi-reliable DoS against
4.1-ESV-R3 (and 4.1.1-P1, which I ran before upgrading recently).

If anybody would like to debug this further I have core dumps of the
dhcpd process inside the infinite loop (generated by kill -QUIT on
the running dhcpd process), and I can probably provide ssh access to
the servers.

Steinar Haug, Nethelp consulting, sthaug at nethelp.no