Problem with "corrupt" lease file

[Simon Hobson]

Mike Schmidt wrote:
>True enough, and I should have. But as you know, it's when these
>components break that they come up on the radar. For dhcp I had no
>difficulties upgrading to 3.0.5, but there are a number of other system
>components we use that may have dependencies I can't handle, such as
>kernel changes.  I can't change the kernels we are running in the field
>(about 400 sites geographically dispersed, about 60,000 end users)
>unless it's absolutely necessary, so of course we tend to leave working
>things untouched.

I know that feeling. Fortunately I've got less systems to manage, and 
there are all fairly close (mostly in my office) - however I'm now 
thinking how to upgrade the DNS, Mail, a traffic monitor (which 
carries ALL our traffic), and several others from Debian Sarge to 
Debian Etch. Conveniently we have a fair bit of spare hardware lying 
around so I'm in the lucky position of being able to copy everything 
to a new box, upgrade & test it, then just swap the box out - of 
course being an IT services and hosting company means we don't have a 
quiet time as far as traffic and availability of services is 
concerned !


>Now I'm off to upgrade dhcp at 400 sites without anyone noticing. You
>can imagine I don't want to do that every day.

My experience is that you can take DHCP down for a reasonable time 
before anyone notices. It's not like most other services where they 
are noticed as soon as the customer doesn't see his web page (or 
whatever) - the lack of service only shows for devices 
booting/connecting and those with expiring leases. Unless you go past 
half your lease time then you'll only affect those devices 
booting/connecting which hopefully isn't too many on one box.


I think it was David Hankins came up with an interesting suggestion a 
while back. Run two servers in failover, one 'back in the office', 
the other out at the distribution points in whatever type of network 
you run. The remote units don't need any permanent writable storage 
(ie you could put the OS/software in flash) as you could put the 
leases database in ramdisk. If the device loses power then it will 
simply re-load it's lease database from it's peer which is a 'proper' 
server with hard disk. In normal operation, the remote server will 
handle nearly all the queries because it's much closer in network 
terms to the clients, but you gain the security of having a failover 
peer back in the office where it's easy to manage (and back up).