[Kea-users] Planning some experimentation with HA using ECMP

Tue Apr 30 14:11:10 UTC 2024

Brendan, we've nothing special with the use of ECMP as it's just the forwarding decision logic on the spine routers of our datacenter stack. The equal routes are all still inserted there by BGP announcements from the service nodes...I think we have a VERY similar setup. Based on what you need to do to HAVE formal HA mechanisms in place, our thinking is just to avoid it altogether (and let potential conflicts on lease reservations work themselves out with blocking calls to a config database). Hopefully the potential pitfall of having more than one node "bound" to a virtual service IP will be mitigated by nodes not being aware of each others' configurations. 

We'll keep the list updated with progress. 

Dan 

----- On Apr 30, 2024, at 9:41 AM, Brendan Kearney <bpk678 at gmail.com> wrote: 

> On 4/30/24 9:18 AM, Dan Geist wrote:

>> Thanks Vicky.

>> Facebook's dhcplb is an option, but it also solves problems that we don't have
>> (imbalance of DHCP messaging and need for staged deployments). It also requires
>> more infra (either physical or virtual) and a slightly greater network
>> complexity which differs from what we already support in other services. I'm
>> trying to stick with the "just because you have a hammer, you should still try
>> to use a screwdriver on screws" philosophy :)

>> I suppose the WAY in which the traffic is balanced is ultimately a wash, though.
>> Either way, we'd need Kea instances in a horizontal N-number farm with
>> mostly-identical behaviors (regardless of if they listen for the virtual IP or
>> if it's housed one hop upstream). Ideally, having as little state as possible
>> (or as little state that DIFFERS between hosts) is an important aspect.

>> Performance tuning, strategy for maintaining the database backend (monolithic vs
>> multiple replicating instances) and so forth will be important, but is there
>> anything inherent about Kea itself that will break this conceptually (unique
>> metadata payload in messaging that will break on DHCP refresh to a different
>> node or something along those lines)?

>> Thanks
>> Dan

>> ----- On Apr 30, 2024, at 8:39 AM, Victoria Risk [ mailto:vicky at isc.org |
>> <vicky at isc.org> ] wrote:

>>>> On Apr 29, 2024, at 6:16 PM, Dan Geist [ mailto:dan at polter.net |
>>>> <dan at polter.net> ] wrote:

>>>> Hi. I have an environment where many of the network services (DNS, NTP, ToD,
>>>> etc.) provide scaling, fault tolerance, and load sharing via ECMP (in front of
>>>> the service) and BGP. Each (of the 2 or more) service node(s) monitors the
>>>> status of that service and announces/pulls BGP announcements from the upstream
>>>> router pair. This works really well for protocols with simple request/response
>>>> transactions.

>>>> I'd like to try doing this same thing with Kea dhcpv(4|6). In that setup, the
>>>> same "virtual service IP" would be configured on each of several Kea nodes (in
>>>> addition to the real link IPs) and they would announce these to the next hop
>>>> (as above). My thinking is that if there is a common configuration and lease
>>>> backend to these multiple nodes, then this can be a way to provide HA services
>>>> (and scaling) to a very large number of devices. My only concern is how the
>>>> multi-step transaction will be handled.

>>>> Before I spend the time to mock this up, has anyone else tried ECMP load
>>>> distribution with DHCP, specifically on Kea, and are there any "gotchas" to be
>>>> aware of?

>>> You might want to check out the DHCP Load Balancer from Facebook: [
>>> https://github.com/facebookincubator/dhcplb |
>>> https://github.com/facebookincubator/dhcplb ]

>>>> Thanks.
>>>> Dan

>>>> --
>>>> Dan Geist dan(@)polter.net

>>>> --
>>>> ISC funds the development of this software with paid support subscriptions.
>>>> Contact us at [ https://www.isc.org/contact/ | https://www.isc.org/contact/ ]
>>>> for more information.

>>>> To unsubscribe visit [ https://lists.isc.org/mailman/listinfo/kea-users |
>>>> https://lists.isc.org/mailman/listinfo/kea-users ] .

>>>> Kea-users mailing list
>>>> [ mailto:Kea-users at lists.isc.org | Kea-users at lists.isc.org ]
>>>> [ https://lists.isc.org/mailman/listinfo/kea-users |
>>>> https://lists.isc.org/mailman/listinfo/kea-users ]

>>> --
>>> ISC funds the development of this software with paid support subscriptions.
>>> Contact us at [ https://www.isc.org/contact/ | https://www.isc.org/contact/ ]
>>> for more information.

>>> To unsubscribe visit [ https://lists.isc.org/mailman/listinfo/kea-users |
>>> https://lists.isc.org/mailman/listinfo/kea-users ] .

>>> Kea-users mailing list
>>> [ mailto:Kea-users at lists.isc.org | Kea-users at lists.isc.org ]
>>> [ https://lists.isc.org/mailman/listinfo/kea-users |
>>> https://lists.isc.org/mailman/listinfo/kea-users ]

>> --
>> Dan Geist dan(@)polter.net

> i do quite a bit of anycast, using BGP, which may differ from your ECMP desires.
> that said, there is probably some overlap in the what, how and why between the
> setups. i have 3 nodes all participating in BGP, and they inject routes to the
> IPs for several stateless services, like DNS, NTP, Kerberos, etc. in my BGP
> configs, i have set maximum-paths to 4, allowing for multiple routes to the
> same address. then i put the anycast IP on the loopback or some other virtual
> interface, so that the anycast IP is not on the wire. if i understand things
> correctly, this anycast setup is the way the root DNS servers are setup for
> their anycast configurations.

> i have started thinking about setting up Kea to have the "listening" interface
> be a VIP on the loopback, like the rest of my anycast services, but haven't
> gotten to a final design or game plan. i am still muttering through how Kea
> will work when i want to have the "listening" interface receive requests and
> respond to them, while using a different interface to talk to the other HA
> instances or to the database i'm using for configs, etc. the question i have
> not answered yet is, what, if any, stateful requirements are there in Kea, that
> would obviate the use of anycast?

> stateless protocols are fully self contained in request and response, and i dont
> know if dhcp, when served by Kea, is entirely stateless. client to server
> requests, and their responses may be, but what happens when you have a relay in
> between, like i do? can Kea talk to different things from different IPs? like i
> said, can dhcp requests and responses be received and responded to from a VIP
> that is stacked on the loopback interface? can that happen while other
> communications are going on, and using different interfaces/IPs for those other
> communications?

> i could definitely see a great reason for anycast or ECMP, and the scalability,
> reliability and fault tolerance those bring, but its the "how" of it all that i
> have not fully rationalized yet.

> i hope you can accomplish what you are looking for, and would love to hear of
> any progress you make.

> thank you,

> brendan kearney

> --
> ISC funds the development of this software with paid support subscriptions.
> Contact us at https://www.isc.org/contact/ for more information.

> To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.

> Kea-users mailing list
> Kea-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/kea-users

-- 
Dan Geist dan(@)polter.net 
http://www.polter.net 
(404)786-6206 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20240430/7707a2fb/attachment.htm>