[Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Xiao, Yu (CCI-Atlanta)
yu.xiao at cox.com
Tue Mar 26 11:58:05 UTC 2024
Hi Darren,
I am using 2.4.1.
Best Regards,
Yu
From: Kea-users <kea-users-bounces at lists.isc.org> on behalf of Darren Ankney <darren.ankney at gmail.com>
Date: Tuesday, March 26, 2024 at 6:03 AM
To: Kea user's list <kea-users at lists.isc.org>
Subject: [EXTERNAL] Re: [Kea-users] kea-2.4.1 // Strange HA related errors
Hello Yu,
What version of Kea are you running on these servers, out of curiosity?
Thank you,
Darren Ankney
On Mon, Mar 25, 2024 at 10:55 AM Xiao, Yu (CCI-Atlanta) via Kea-users
<kea-users at lists.isc.org> wrote:
>
> Hi experts,
>
>
>
> I have configured two VMs in the same hypervisor as hot-standby mode HA. I believe they are successfully communicating with each other with heart beat packets, as we can see the primary VM kea-1 has successfully received the “ha-heartbeat” from the standby VM kea-2 in green logs. But the red logs indicate that the ha_hooks think the HA heartbeat communications failed due to “no route”. But this is a LAN network, and there’s indeed route installed as we can see below and we can ping the 69 ip.
>
>
>
> [yxiao322 at kea_home1 ~]$ ip route
>
> 192.168.100.0/24 dev ens18 proto kernel scope link src 192.168.100.197 metric 100 <<< This should be the route to 192.168.100.69
>
> 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
>
>
>
> [root at kea_home1 yxiao322]# ping 192.168.100.69
>
> PING 192.168.100.69 (192.168.100.69) 56(84) bytes of data.
>
> 64 bytes from 192.168.100.69: icmp_seq=1 ttl=64 time=0.342 ms
>
> 64 bytes from 192.168.100.69: icmp_seq=2 ttl=64 time=0.302 ms
>
> 64 bytes from 192.168.100.69: icmp_seq=3 ttl=64 time=0.236 ms
>
> ^Z
>
> [1]+ Stopped ping 192.168.100.69
>
>
>
> And if I stop the kea service on primary, then we can see standby server will complain “communication with kea_home1 is interrupted”. And as soon as I start kea service again on primary, then the database began sync again. Thus, I believe there’s indeed communications and syncs between primary and standby VMs. But for some reason, if I shut the kea service on primary, then the standby won’t distribute DHCP leases even after I waited for a long time. Did I miss something here?
>
>
>
>
>
> Primary logs:
>
>
>
> Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.198 INFO [kea-dhcp6.commands/1224.139988007651072] COMMAND_RECEIVED Received command 'ha-heartbeat'
>
> Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN [kea-dhcp6.ha-hooks/1224.139988049614592] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 (https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$ ): No route to host
>
> Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN [kea-dhcp6.ha-hooks/1224.139988049614592] HA_COMMUNICATION_INTERRUPTED communication with kea_home2 is interrupted
>
> Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.199 INFO [kea-dhcp6.commands/1224.139988016043776] COMMAND_RECEIVED Received command 'ha-heartbeat'
>
> Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN [kea-dhcp6.ha-hooks/1224.139988032829184] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 (https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$ ): No route to host
>
> Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN [kea-dhcp6.ha-hooks/1224.139988032829184] HA_COMMUNICATION_INTERRUPTED communication with kea_home2 is interrupted
>
>
>
> Standby logs:
>
>
>
> Mar 25 10:10:24 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:24.129 WARN [kea-dhcp6.ha-hooks/2836.139717198915328] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
>
> Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN [kea-dhcp6.ha-hooks/2836.139717207308032] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$ ): Connection refused
>
> Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN [kea-dhcp6.ha-hooks/2836.139717207308032] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
>
> Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN [kea-dhcp6.ha-hooks/2836.139717190522624] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$ ): Connection refused
>
> Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN [kea-dhcp6.ha-hooks/2836.139717190522624] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.136 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from PARTNER-DOWN to WAITING state, partner state is PARTNER-DOWN
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in WAITING state
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the kea_home2 is in the WAITING state
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from WAITING to SYNCING state, partner state is PARTNER-DOWN
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in SYNCING state
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_SYNC_START starting lease database synchronization with kea_home1
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.140 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 3 leases from kea_home1
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.141 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_SYNC_SUCCESSFUL lease database synchronization with kea_home1 completed successfully in 4.350 ms
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.142 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from SYNCING to READY state, partner state is PARTNER-DOWN
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.142 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in READY state
>
>
>
> Primary Config:
>
>
>
> # HA related hooks configuration
>
> "hooks-libraries": [{
>
> "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so",
>
> "parameters": { }
>
> }, {
>
> "library": "/usr/lib64/kea/hooks/libdhcp_ha.so",
>
> "parameters": {
>
> "high-availability": [{
>
> "this-server-name": "kea_home1",
>
> "mode": "hot-standby",
>
> "heartbeat-delay": 200,
>
> "max-response-delay": 600,
>
> "max-ack-delay": 800,
>
> "max-unacked-clients": 5,
>
> "send-lease-updates": true,
>
> "sync-leases": true,
>
> "sync-page-limit": 200,
>
> "sync-timeout": 600,
>
> "delayed-updates-limit": 0,
>
> "peers": [{
>
> "name": "kea_home1",
>
> "url": https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$> ,
>
> "role": "primary",
>
> "auto-failover": true
>
> }, {
>
> "name": "kea_home2",
>
> "url": https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$<https://urldefense.com/v3/__http:/192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$> ,
>
> "role": "standby",
>
> "auto-failover": true
>
> }]
>
> }]
>
> }
>
> }],
>
>
>
>
>
> "Control-agent": {
>
> "http-host": "192.168.100.197",
>
> // If enabling HA and multi-threading, the 8000 port is used by the HA
>
> // hook library http listener. When using HA hook library with
>
> // multi-threading to function, make sure the port used by dedicated
>
> // listener is different (e.g. 8001) than the one used by CA. Note
>
> // the commands should still be sent via CA. The dedicated listener
>
> // is specifically for HA updates only.
>
> "http-port": 8001,
>
>
>
>
>
> Standby config:
>
>
>
> # HA related hooks configuration
>
> "hooks-libraries": [{
>
> "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so",
>
> "parameters": { }
>
> }, {
>
> "library": "/usr/lib64/kea/hooks/libdhcp_ha.so",
>
> "parameters": {
>
> "high-availability": [{
>
> "this-server-name": "kea_home2",
>
> "mode": "hot-standby",
>
> "heartbeat-delay": 200,
>
> "max-response-delay": 600,
>
> "max-ack-delay": 800,
>
> "max-unacked-clients": 5,
>
> "delayed-updates-limit": 0,
>
> "send-lease-updates": true,
>
> "sync-leases": true,
>
> "sync-page-limit": 200,
>
> "sync-timeout": 600,
>
> "peers": [{
>
> "name": "kea_home1",
>
> "url": https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$> ,
>
> "role": "primary",
>
> "auto-failover": true
>
> }, {
>
> "name": "kea_home2",
>
> "url": https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$<https://urldefense.com/v3/__http:/192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$> ,
>
> "role": "standby",
>
> "auto-failover": true
>
> }]
>
> }]
>
> }
>
> }],
>
>
>
> "Control-agent": {
>
> "http-host": "192.168.100.69",
>
> // If enabling HA and multi-threading, the 8000 port is used by the HA
>
> // hook library http listener. When using HA hook library with
>
> // multi-threading to function, make sure the port used by dedicated
>
> // listener is different (e.g. 8001) than the one used by CA. Note
>
> // the commands should still be sent via CA. The dedicated listener
>
> // is specifically for HA updates only.
>
> "http-port": 8001,
>
>
>
>
>
> Best Regards,
>
> Yu
>
>
>
> --
> ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$> for more information.
>
> To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$> .
>
> Kea-users mailing list
> Kea-users at lists.isc.org
> https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$>
--
ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$> for more information.
To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$> .
Kea-users mailing list
Kea-users at lists.isc.org
https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20240326/eeb63676/attachment-0001.htm>
More information about the Kea-users
mailing list