[Kea-users] Load-Balancing Network issue between Relay and Kea

Tue Jan 10 02:07:50 UTC 2023

This is my understanding of how the unacked clients functionality works. My explanation is based upon the DHCP4 source code and may differ for DHCP6. I will include references at the bottom of my email which I encourage double-checking for accuracy. I am not a contributor to Kea and have not thoroughly tested the conclusions I draw here.

1. The DHCP packet enters Kea. The HA hook receives the packet in the buffer4Receive[1] function. The packet contents are parsed and dropped if invalid.

2. The packet is checked to be in scope [2][3][4][5][6] (and if it isn't, the packet status is set to NEXT_STEP_DROP [21]). Whether a packet is in scope is decided by the following:
  a. If the packet is not one that can be handled by HA (is one of DHCPDISCOVER, DHCPREQUEST, DHCPDECLINE, DHCPRELEASE, or DHCPINFORM [7]), then the current server will process it [8].
  b. If HA is configured in load balancing mode, the packet is classed according to the aforementioned HBA defined in RFC 3074 section 6 [9][10]. The HBA returns the server that must handle the packet (either primary or secondary). Otherwise (server is in hot standby), the packet is classed as belonging to the primary server in the HA configuration [12]. The class given in either of these conditions is the defined name of the respective server, coming from the HA section of the Kea DHCP4 configuration [13].
  c. The current server will process the packet if it is serving packets with the class determined in (2)(b).

Note: every heartbeat, the servers send each other their scopes [15]. A failed heartbeat sets the HA status to "unavailable" [24], which eventually transitions the server to partner down state.

3. If the server is in a communication interrupted state and the packet is not classed for the current server, then:
  a. Maintain a global counter, incrementing it once per packet (every successful heartbeat counts as a "poke" for the partner [16], which resets this global counter to zero [17]).
  b. Get the "secs" field of the packet. Compare the value to the value configured in the Kea DHCP4 configuration for "max-ack-delay" [18], or 10 seconds by default [19]. If the value of this field is greater than the max-ack-delay, the packet is considered unacked [20]. All packets (unacked or not) are kept track of in a map containing the hardware address, client ID, and last unacked status; if the packet is being received unacked, and it has not been previously recorded as being unacked (that is, the packet secs field just exceeded the max-ack-delay threshold for the first time), the server logs a warning message.

4. A failure is detected if the number of packets in the unacked state is greater than the "max-unacked-clients" setting of the Kea DHCP4 config [22] (or 10 by default [19]). If a failure is detected, the server eventually transitions to partner-down state [23]. More information about when exactly the server transitions to partner-down state is shown by the usages of HAService::shouldPartnerDown() [25] (in other words, I'm not digging into that tonight).

[1]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_impl.cc#L60-L111
[2]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1021
[3]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1029-L1047
[4]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1034
[5]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L376
[6]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L382-L414
[7]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L51-L71
[8]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L395
[9]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L416-L446
[10]: https://www.rfc-editor.org/rfc/rfc3074
[11]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L413
[12]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L398
[13]: https://kea.readthedocs.io/en/kea-2.2.0/arm/hooks.html#load-balancing-configuration
[14]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/communication_state.cc#L617-L625
[15]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1757-L1758
[16]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1793-L1794
[17]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/communication_state.cc#L274
[18]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_config_parser.cc#L180-L181
[19]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_config.cc#L166
[20]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/communication_state.cc#L652
[21]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_impl.cc#L104
[22]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_config_parser.cc#L184-L185
[23]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1097
[24]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1799
[25]: https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1081-L1106

Eric Graham
DevOps Specialist
Direct: 605.990.1859
Eric.Graham at vantagepnt.com<mailto:eric.graham at vantagepnt.com>
[cid:446fdf61-044d-49d2-a820-4da231cbe5e1]
________________________________
From: Kea-users <kea-users-bounces at lists.isc.org> on behalf of Kevin P. Fleming <lists.kea-users at kevin.km6g.us>
Sent: Monday, January 9, 2023 12:38 PM
To: kea-users at lists.isc.org <kea-users at lists.isc.org>
Subject: Re: [Kea-users] Load-Balancing Network issue between Relay and Kea

CAUTION: This email originated outside the organization. Do not click any links or attachments unless you have verified the sender.
On Mon, Jan 9, 2023, at 11:54, Veronique Lefebure wrote:
Very interesting thread.

Mathias, you wrote "Expected behaviour: Kea 2 sees the unacked clients of Kea 1 and sets Kea 1 in partner-down state and handles all requests.", but, If there is no traffic between DHCP clients and Kea1, then the value of max-unacked-clients on server1 cannot increase anyway, right ?  In other words, Kea2 cannot "see" anything ?

It can 'see', because it *also* saw all of the client requests and knows which ones it expected to be handled by Kea1 (as noted earlier in the thread it even emits a log message indicating this).

Forgive my presumption, but I assumed that 'max-unacked-clients' would be a counter of 'unacked clients' which belong to a Kea server *other than this one*. I don't immediately know how counting the number of clients *this server* has not acked would be useful, although I won't be surprised to learn that it is useful to someone.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20230110/8f81f276/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Outlook-fcyi02ly.png
Type: image/png
Size: 16388 bytes
Desc: Outlook-fcyi02ly.png
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20230110/8f81f276/attachment-0001.png>