failover, partner-down state, MCLT and rewind binding

[Gregory Sloop]

So, I'm looking for a little more understanding. I had an outage last week that didn't work out so well.
I've had sort-of-similar problems in the past with this setup, and I *think* I know some of what happened this time, but wanting confirmation.

After the last similar outage, I knew we needed to put the surviving peer in "partner-down" mode, and this, along with the new "rewind state

Here's the config, in general terms.
---
We're doing failover. [ISC dhcpd 2.4.2 on Ubuntu 14.04]

In this specific case, the "free" pool is quite small in comparison to the number of clients. [And this isn't easily fixable - we're addressing, but there are limitations. Lets, for now, ignore that changes might make the situation more tolerable and focus on what's happening.] 

But, since the pool is quite small, in a communication-interrupted state, the surviving peer can easily run out of leases in its part of the split pool. Thus, helping the surviving server do as well as possible in a bad situation is my goal. [Not enough free pool addresses on the surviving peer to handle all the renewals that will come over from the down server.]

Thus, here's what happened:
So the primary and then the secondary went down hard - a few minutes apart.

Fairly quickly, got the secondary back up.
It handled all it's assigned leases fine, as far as I can tell. It then handed out all the remaining pool leases it had. Then it ran out of leases.

I expected it to start recovering the leases from the peer [primary] at this point, and extend [rewind] the leases the primary had already issued.

While I'm not sure how the "rewind" provision worked - we were getting "peer holds all free leases" messages for quite a while.

I didn't think this should be happening - but then I looked at the MCLT time and it was set to 1800 [30m]

One issue to address, I assume:
I assume [now] that I should set MCLT as low as I can, such that, it will also still allow a single server to handle the load. For example; if I set MCLT to 30 seconds; I need to be sure that a single server [or the slowest of the pair] should be able to handle the load of every client renewing every 15s, plus whatever I consider a safety margin. If my servers and network can handle this load, there's no real disadvantage to setting low MCLT times. [I'd perhaps tend to speculate that MCLT times ought to be one-forth as long [or less] as the regular lease time - again if we can handle that load. This would allow the peer, when going into partner down mode to recover leases fairly quickly, in relation to how fast we might expect lease expirations.] Is that reasoning sound?

(The lesson for me here is: I believe I have my MCLT time set too high. At this point, both leases and MCLT are 30 minutes. I can handle a lot more load, so I'm leaning toward a 1-5m MCLT time and perhaps lengthening lease times to help bridge an outage.)

So, my additional questions generally relate to:
1) When can the surviving dhcp fail-over peer recover the unused pool addresses the "down" server had. 

I don't see any specific answer in the list or elsewhere, but it seems logical that this would be the MCLT time. But how does it calculate that time? Again, I assume it would be from the time the second server goes into partner down mode + MCLT. Does this time get "reset" each time the surviving server is restarted? [I'm assuming there's no record of this written to disk. And, the server would have to assume stuff might have happened while it was being restarted - so each time it goes into partner down mode (which it will do when it gets restarted), it will have to wait MCLT time again before starting recovery.]

2) Expired/expiring leases: when can the surviving server recover these, or rewind them?

Recover to be used for another client:
This appears to be whatever the original lease was for [Is that STOS?] + MCLT. Once this time has passed, then the surviving peer who is in "partner-down" mode can take that lease and recover it for use.

Rewind: Does rewind work in "partner-down" mode - or only in communications-interrupted? [Not clear to me from docs.]
I assume that the surviving server would issue "rewind" leases for the MCLT time, as many times as needed, until the peer recovers. But I haven't seen any discussion about how the rewind state actually works. I'd be glad to be pointed at a discussion, if one exists.

However, I'd have expected rewind to work better in our case than it did - or at least better than it appeared. So,

I hope that's not too disjointed. I've also looked at the docs in an attempt to understand things - but what I've written above is the best I have from the docs and list discussion. Thanks for your help, in advance.

-Greg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/dhcp-users/attachments/20151109/39304c06/attachment.html>