[Kea-users] Load-Balancing Network issue between Relay and Kea

Tue Jan 10 07:34:57 UTC 2023

Kea HA detects partner failures via the control channel in the first 
place. The servers constantly exchange heartbeats and lease updates. If 
this communication is healthy (i.e., servers receive responses to the 
control commands), the counters of unacked clients are set to 0, and the 
servers do not monitor whether their partners respond to DHCP. In other 
words, if the servers can communicate with each other, the 
"max-unacked-clients" setting has no effect.

Partner failure detection in Kea HA always begins with communication 
failure over the control channel. Usually, it is caused by a partner 
process crash or a network failure. The server tries to send heartbeats 
and lease updates to the partner, for which it gets no responses. 
Suppose the "max-response-delay" setting is 60000 (1 minute) and the 
"heartbeat-delay" is 10000 (10 seconds). In that case, the server sends 
a heartbeat every 10 seconds to the partner. If the partner doesn't 
respond, the server sends the heartbeat again 10 seconds later, and so 
on. It neither counts the unacked clients nor transitions to the 
partner-down state because the communication issue can be temporary. 
Finally, after around six unsuccessful heartbeat attempts (6 * 10 
seconds), the communication interruption becomes longer than 1 minute 
(60000ms). In that case, the server assumes there can be an issue with 
the partner. It is when the "max-unacked-clients" setting finally starts 
to matter.

The server begins to analyze the DHCP messages sent to the partner 
server. The "secs" field in DHCPv4 and "Elapsed Time Option" in DHCPv6 
should be set by a client to indicate how long the client has been 
trying to ask for a new lease or rebind an existing lease. Obviously, 
the server can't see if the partner responds to these queries. It only 
gets copies of the DHCP messages sent by the client. The clients must 
bump these values when they retry to obtain a lease. If these values are 
zero, the server suspecting that the partner is down cannot determine 
whether the partner actually responds to DHCP. If these values are 
greater than 0 (and greater than "max-ack-delay") the server can assume 
that the partner hasn't responded to them because the clients are retrying.

For every client who sends a DHCPDISCOVER or DHCPREBIND to the partner 
server and (finally) sets the "secs" field value greater than 
"max-ack-delay", the other server bumps up its internal counter or 
unacked clients. Again, it only does it when it has been unable to 
communicate with the partner server over the control channel longer than 
the configured "max-response-delay". A single successful heartbeat over 
the control channel will clear the counters of the unacked clients and 
make the server believe that the partner is healthy. It will also stop 
looking at the "secs" and "Elapsed Time" values. The 
"max-unacked-clients" no longer matters until the next communication 
issue over the control channel.

If the "max-unacked-clients" value is exceeded, the server can finally 
transition to the partner-down state and handle both the traffic 
directed to itself and the inoperational partner. Since the state 
transitions are only carried after a heartbeat attempt, there may be a 
slight delay between exceeding the "max-unacked-clients" value and 
actually transitioning to the partner-down state.

I looked into our documentation and realized that although all of these 
pieces are described there, it can be confusing because the ARM lacks a 
sequence diagram or an example of how the failover process can look end 
to end. That's something we should address.

Going over the previous emails, I see that users can see different 
failover strategies, depending on the types of failures they are likely 
to experience in their setup. They are interesting cases, and we will 
discuss them internally. We could consider some alternative failure 
detection strategies, selectable with the HA configuration, but we 
should be aware that there is no one-fits-all solution. There is always 
a possibility that the true failure won't be detected or a false failure 
will be detected, leading to a split-brain situation.

It would be useful if you could please open tickets in Gitlab to 
describe your failover scenarios and the desired behavior. Please 
disregard it if you have already opened them.

Kinds Regards,
Marcin Siodelski

Sr. Software Engineer,
ISC

On 10.01.2023 03:07, Eric Graham wrote:
> This is my understanding of how the unacked clients functionality works. 
> My explanation is based upon the DHCP4 source code and may differ for 
> DHCP6. I will include references at the bottom of my email which I 
> encourage double-checking for accuracy. I am not a contributor to Kea 
> and have not thoroughly tested the conclusions I draw here.
> 
> 1. The DHCP packet enters Kea. The HA hook receives the packet in the 
> buffer4Receive[1] function. The packet contents are parsed and dropped 
> if invalid.
> 
> 2. The packet is checked to be in scope [2][3][4][5][6] (and if it 
> isn't, the packet status is set to NEXT_STEP_DROP [21]). Whether a 
> packet is in scope is decided by the following:
>    a. If the packet is not one that can be handled by HA (is one of 
> DHCPDISCOVER, DHCPREQUEST, DHCPDECLINE, DHCPRELEASE, or DHCPINFORM [7]), 
> then the current server will process it [8].
>    b. If HA is configured in load balancing mode, the packet is classed 
> according to the aforementioned HBA defined in RFC 3074 section 6 
> [9][10]. The HBA returns the server that must handle the packet (either 
> primary or secondary). Otherwise (server is in hot standby), the packet 
> is classed as belonging to the primary server in the HA configuration 
> [12]. The class given in either of these conditions is the defined name 
> of the respective server, coming from the HA section of the Kea DHCP4 
> configuration [13].
>    c. The current server will process the packet if it is serving 
> packets with the class determined in (2)(b).
> 
> Note: every heartbeat, the servers send each other their scopes [15]. A 
> failed heartbeat sets the HA status to "unavailable" [24], which 
> eventually transitions the server to partner down state.
> 
> 3. If the server is in a communication interrupted state and the packet 
> is not classed for the current server, then:
>    a. Maintain a global counter, incrementing it once per packet (every 
> successful heartbeat counts as a "poke" for the partner [16], which 
> resets this global counter to zero [17]).
>    b. Get the "secs" field of the packet. Compare the value to the value 
> configured in the Kea DHCP4 configuration for "max-ack-delay" [18], or 
> 10 seconds by default [19]. If the value of this field is greater than 
> the max-ack-delay, the packet is considered unacked [20]. All packets 
> (unacked or not) are kept track of in a map containing the hardware 
> address, client ID, and last unacked status; if the packet is being 
> received unacked, and it has not been previously recorded as being 
> unacked (that is, the packet secs field just exceeded the max-ack-delay 
> threshold for the first time), the server logs a warning message.
> 
> 4. A failure is detected if the number of packets in the unacked state 
> is greater than the "max-unacked-clients" setting of the Kea DHCP4 
> config [22] (or 10 by default [19]). If a failure is detected, the 
> server eventually transitions to partner-down state [23]. More 
> information about when exactly the server transitions to partner-down 
> state is shown by the usages of HAService::shouldPartnerDown() [25] (in 
> other words, I'm not digging into that tonight).
> 
> [1]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_impl.cc#L60-L111
> [2]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1021
> [3]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1029-L1047
> [4]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1034
> [5]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L376
> [6]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L382-L414
> [7]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L51-L71
> [8]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L395
> [9]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L416-L446
> [10]: https://www.rfc-editor.org/rfc/rfc3074
> [11]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L413
> [12]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/query_filter.cc#L398
> [13]: 
> https://kea.readthedocs.io/en/kea-2.2.0/arm/hooks.html#load-balancing-configuration
> [14]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/communication_state.cc#L617-L625
> [15]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1757-L1758
> [16]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1793-L1794
> [17]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/communication_state.cc#L274
> [18]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_config_parser.cc#L180-L181
> [19]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_config.cc#L166
> [20]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/communication_state.cc#L652
> [21]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_impl.cc#L104
> [22]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_config_parser.cc#L184-L185
> [23]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1097
> [24]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1799
> [25]: 
> https://gitlab.isc.org/isc-projects/kea/-/blob/c4c53a0168ffa385c387ba685ac16e5544feaad4/src/hooks/dhcp/high_availability/ha_service.cc#L1081-L1106
> 
> 
> *Eric Graham*
> /DevOps Specialist/
> Direct: 605.990.1859/
> /
> //Eric.Graham at vantagepnt.com <mailto:eric.graham at vantagepnt.com>/
> /
> /
> /
> ------------------------------------------------------------------------
> *From:* Kea-users <kea-users-bounces at lists.isc.org> on behalf of Kevin 
> P. Fleming <lists.kea-users at kevin.km6g.us>
> *Sent:* Monday, January 9, 2023 12:38 PM
> *To:* kea-users at lists.isc.org <kea-users at lists.isc.org>
> *Subject:* Re: [Kea-users] Load-Balancing Network issue between Relay 
> and Kea
> *CAUTION:* This email originated outside the organization. Do not click 
> any links or attachments unless you have verified the sender.
> On Mon, Jan 9, 2023, at 11:54, Veronique Lefebure wrote:
>> Very interesting thread.
>>
>> Mathias, you wrote "Expected behaviour: Kea 2 sees the unacked clients 
>> of Kea 1 and sets Kea 1 in partner-down state and handles all 
>> requests.", but, If there is no traffic between DHCP clients and Kea1, 
>> then the value of max-unacked-clients on server1 cannot increase 
>> anyway, right ?  In other words, Kea2 cannot "see" anything ?
>>
> 
> It can 'see', because it *also* saw all of the client requests and knows 
> which ones it expected to be handled by Kea1 (as noted earlier in the 
> thread it even emits a log message indicating this).
> 
> Forgive my presumption, but I assumed that 'max-unacked-clients' would 
> be a counter of 'unacked clients' which belong to a Kea server *other 
> than this one*. I don't immediately know how counting the number of 
> clients *this server* has not acked would be useful, although I won't be 
> surprised to learn that it is useful to someone.
> 