[Kea-users] recover from split brain?

[Geist, Daniel (CCI-Atlanta)]

Not sure if KEA has a way to deal with it "gracefully", but in general, once leases are being given out in split-brain, the "train has left the station". A few things that can help:
- ping-verification on lease helps avoid conflicts (although kea doesn't do it https://kb.isc.org/docs/why-doesnt-kea-support-ping-check )
- figure out which of your nodes has the most leases, and use that as your source of truth.
- If they have the same amount of leases, use the one with the NEWEST leases as truth, since clients with older leases will expire and re-discover on their own earlier.
- Once they're back in sync, force reboots if you can (yeah, this is pretty impactful).
- lower your leases while you're trying to get things back in sync. It causes more traffic, but helps things heal on their own.

Dan

>-----Original Message-----
>From: Kea-users <kea-users-bounces at lists.isc.org> On Behalf Of perl-list
>Sent: Wednesday, February 9, 2022 8:00 AM
>To: kea-users <kea-users at lists.isc.org>
>Subject: [EXTERNAL] Re: [Kea-users] recover from split brain?
>
>I think he was asking what happens if the servers can't see each other and
>both start answering clients, possibly creating IP conflicts that must be
>resolved at some point.  Imagine a split fiber ring where the servers can't see
>each other and the clients on each half of the ring can't see each other.  If this
>persists for a while, duplicate IPs could be allocated.  ISC DHCP solved this
>problem by not allocating addresses that were assigned to the other server
>unless you hand set "partner down" mode meaning you knew the other
>server was down and duplicate IPs wouldn't be assigned.
>
>----- Original Message -----
>> From: "Chad Catlett" <chad at catlett.info>
>> To: "David Ramsey" <dmr at ramseyfamily.org>, "kea-users"
>> <kea-users at lists.isc.org>
>> Sent: Tuesday, February 8, 2022 6:21:53 PM
>> Subject: Re: [Kea-users] recover from split brain?
>
>> On 2/8/22 14:12, David Ramsey wrote:
>> > Lab testing I've done shows that the Active/Standby HA configuration
>> > works well if one or the other DHCP servers fails completely.
>> > Automatic takeover and recovery work nicely.
>
>> > If an Active/Standby HA implementation were to go "split brain" with
>> > core network severed between DHCP servers, however, both would go
>> > active and the lease DBs would no longer be in sync.
>
>> > How to recover from that when the network is restored? I know there
>> > are a ton of variables involved - lease times, how long network was
>> > segmented, # subscribers, etc.... but I am wrestling with what would
>> > be the optimal approach/logic to consolidate and recover ?
>
>> > Help?
>
>> > Thanks folks, --David
>
>
>
>
>
>> The admin docs[0] do a fairly decent job of explaining how HA failover
>> situations are recovered from. The amount of time it takes is directly
>> related to the dhcp statemachine timers, which are influenced by your
>> settings.
>
>> 0:
>> https://urldefense.com/v3/__https://kea.readthedocs.io/en/kea-2.1.2/ar
>> m/hooks.html*scope-transition-in-a-partner-down-case__;Iw!!Hit2Ag!i5gg
>> 7MxOAanUKgfx1-ZohtyrYmmud5yJy217j4ao8kUVcctxXvurhzE7lv1FOA$
>
>> Chad
>
>> --
>> Chad Catlett
>> chad at catlett.info
>
>> --
>> ISC funds the development of this software with paid support subscriptions.
>> Contact us at
>https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!i5gg7
>MxOAanUKgfx1-ZohtyrYmmud5yJy217j4ao8kUVcctxXvurhzH2hVg9Ng$  for
>more information.
>
>> To unsubscribe visit
>https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-
>users__;!!Hit2Ag!i5gg7MxOAanUKgfx1-
>ZohtyrYmmud5yJy217j4ao8kUVcctxXvurhzHKeLFyhw$ .
>
>> Kea-users mailing list
>> Kea-users at lists.isc.org
>> https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea
>> -users__;!!Hit2Ag!i5gg7MxOAanUKgfx1-
>ZohtyrYmmud5yJy217j4ao8kUVcctxXvur
>> hzHKeLFyhw$
>--
>ISC funds the development of this software with paid support subscriptions.
>Contact us at
>https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!i5gg7
>MxOAanUKgfx1-ZohtyrYmmud5yJy217j4ao8kUVcctxXvurhzH2hVg9Ng$  for
>more information.
>
>To unsubscribe visit
>https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-
>users__;!!Hit2Ag!i5gg7MxOAanUKgfx1-
>ZohtyrYmmud5yJy217j4ao8kUVcctxXvurhzHKeLFyhw$ .
>
>Kea-users mailing list
>Kea-users at lists.isc.org
>https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-
>users__;!!Hit2Ag!i5gg7MxOAanUKgfx1-
>ZohtyrYmmud5yJy217j4ao8kUVcctxXvurhzHKeLFyhw$