[Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Xiao, Yu (CCI-Atlanta)
yu.xiao at cox.com
Thu Mar 28 18:03:37 UTC 2024
Hi experts,
I modified the config file on both DHCP VMs, and reversed their roles, now kea2 as primary and kea1 as standby. But I can see still the leases are provided by kea1. I did the packet capture on both hosts.
As we can see, both hosts have received the DHCPv6 Solicit packet from client which has the link local address as fe80::c40b:ebff:fed1:7298, but kea2 immediately did the address resolution process, and then sent the Advertise packet, while even kea2 is the primary host now, it just doesn’t respond to the coming Solicit packet from client.
How should I troubleshoot why kea2 didn’t respond to the Solicit packet from the client? Can I have an example?
Kea1:
[cid:image001.png at 01DA810F.00DD3300]
Kea2:
[cid:image002.png at 01DA810F.24AC5FE0]
Link local Ips:
client
fe80::c40b:ebff:fed1:7298
kea1
fe80::40f1:11ff:fe7a:de7
Kea2
fe80::a8ca:adff:feb9:5db6
Best Regards,
Yu
From: Kea-users <kea-users-bounces at lists.isc.org> on behalf of Xiao, Yu (CCI-Atlanta) via Kea-users <kea-users at lists.isc.org>
Date: Tuesday, March 26, 2024 at 7:58 AM
To: Kea user's list <kea-users at lists.isc.org>
Cc: Xiao, Yu (CCI-Atlanta) <yu.xiao at cox.com>
Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Hi Darren,
I am using 2.4.1.
Best Regards,
Yu
From: Kea-users <kea-users-bounces at lists.isc.org> on behalf of Darren Ankney <darren.ankney at gmail.com>
Date: Tuesday, March 26, 2024 at 6:03 AM
To: Kea user's list <kea-users at lists.isc.org>
Subject: [EXTERNAL] Re: [Kea-users] kea-2.4.1 // Strange HA related errors
Hello Yu,
What version of Kea are you running on these servers, out of curiosity?
Thank you,
Darren Ankney
On Mon, Mar 25, 2024 at 10:55 AM Xiao, Yu (CCI-Atlanta) via Kea-users
<kea-users at lists.isc.org> wrote:
>
> Hi experts,
>
>
>
> I have configured two VMs in the same hypervisor as hot-standby mode HA. I believe they are successfully communicating with each other with heart beat packets, as we can see the primary VM kea-1 has successfully received the “ha-heartbeat” from the standby VM kea-2 in green logs. But the red logs indicate that the ha_hooks think the HA heartbeat communications failed due to “no route”. But this is a LAN network, and there’s indeed route installed as we can see below and we can ping the 69 ip.
>
>
>
> [yxiao322 at kea_home1 ~]$ ip route
>
> 192.168.100.0/24 dev ens18 proto kernel scope link src 192.168.100.197 metric 100 <<< This should be the route to 192.168.100.69
>
> 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
>
>
>
> [root at kea_home1 yxiao322]# ping 192.168.100.69
>
> PING 192.168.100.69 (192.168.100.69) 56(84) bytes of data.
>
> 64 bytes from 192.168.100.69: icmp_seq=1 ttl=64 time=0.342 ms
>
> 64 bytes from 192.168.100.69: icmp_seq=2 ttl=64 time=0.302 ms
>
> 64 bytes from 192.168.100.69: icmp_seq=3 ttl=64 time=0.236 ms
>
> ^Z
>
> [1]+ Stopped ping 192.168.100.69
>
>
>
> And if I stop the kea service on primary, then we can see standby server will complain “communication with kea_home1 is interrupted”. And as soon as I start kea service again on primary, then the database began sync again. Thus, I believe there’s indeed communications and syncs between primary and standby VMs. But for some reason, if I shut the kea service on primary, then the standby won’t distribute DHCP leases even after I waited for a long time. Did I miss something here?
>
>
>
>
>
> Primary logs:
>
>
>
> Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.198 INFO [kea-dhcp6.commands/1224.139988007651072] COMMAND_RECEIVED Received command 'ha-heartbeat'
>
> Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN [kea-dhcp6.ha-hooks/1224.139988049614592] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 (https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$ ): No route to host
>
> Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN [kea-dhcp6.ha-hooks/1224.139988049614592] HA_COMMUNICATION_INTERRUPTED communication with kea_home2 is interrupted
>
> Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.199 INFO [kea-dhcp6.commands/1224.139988016043776] COMMAND_RECEIVED Received command 'ha-heartbeat'
>
> Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN [kea-dhcp6.ha-hooks/1224.139988032829184] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 (https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$ ): No route to host
>
> Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN [kea-dhcp6.ha-hooks/1224.139988032829184] HA_COMMUNICATION_INTERRUPTED communication with kea_home2 is interrupted
>
>
>
> Standby logs:
>
>
>
> Mar 25 10:10:24 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:24.129 WARN [kea-dhcp6.ha-hooks/2836.139717198915328] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
>
> Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN [kea-dhcp6.ha-hooks/2836.139717207308032] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$ ): Connection refused
>
> Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN [kea-dhcp6.ha-hooks/2836.139717207308032] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
>
> Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN [kea-dhcp6.ha-hooks/2836.139717190522624] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$ ): Connection refused
>
> Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN [kea-dhcp6.ha-hooks/2836.139717190522624] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.136 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from PARTNER-DOWN to WAITING state, partner state is PARTNER-DOWN
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in WAITING state
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the kea_home2 is in the WAITING state
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from WAITING to SYNCING state, partner state is PARTNER-DOWN
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in SYNCING state
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_SYNC_START starting lease database synchronization with kea_home1
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.140 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 3 leases from kea_home1
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.141 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_SYNC_SUCCESSFUL lease database synchronization with kea_home1 completed successfully in 4.350 ms
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.142 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from SYNCING to READY state, partner state is PARTNER-DOWN
>
> Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.142 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in READY state
>
>
>
> Primary Config:
>
>
>
> # HA related hooks configuration
>
> "hooks-libraries": [{
>
> "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so",
>
> "parameters": { }
>
> }, {
>
> "library": "/usr/lib64/kea/hooks/libdhcp_ha.so",
>
> "parameters": {
>
> "high-availability": [{
>
> "this-server-name": "kea_home1",
>
> "mode": "hot-standby",
>
> "heartbeat-delay": 200,
>
> "max-response-delay": 600,
>
> "max-ack-delay": 800,
>
> "max-unacked-clients": 5,
>
> "send-lease-updates": true,
>
> "sync-leases": true,
>
> "sync-page-limit": 200,
>
> "sync-timeout": 600,
>
> "delayed-updates-limit": 0,
>
> "peers": [{
>
> "name": "kea_home1",
>
> "url": https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$> ,
>
> "role": "primary",
>
> "auto-failover": true
>
> }, {
>
> "name": "kea_home2",
>
> "url": https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$<https://urldefense.com/v3/__http:/192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$> ,
>
> "role": "standby",
>
> "auto-failover": true
>
> }]
>
> }]
>
> }
>
> }],
>
>
>
>
>
> "Control-agent": {
>
> "http-host": "192.168.100.197",
>
> // If enabling HA and multi-threading, the 8000 port is used by the HA
>
> // hook library http listener. When using HA hook library with
>
> // multi-threading to function, make sure the port used by dedicated
>
> // listener is different (e.g. 8001) than the one used by CA. Note
>
> // the commands should still be sent via CA. The dedicated listener
>
> // is specifically for HA updates only.
>
> "http-port": 8001,
>
>
>
>
>
> Standby config:
>
>
>
> # HA related hooks configuration
>
> "hooks-libraries": [{
>
> "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so",
>
> "parameters": { }
>
> }, {
>
> "library": "/usr/lib64/kea/hooks/libdhcp_ha.so",
>
> "parameters": {
>
> "high-availability": [{
>
> "this-server-name": "kea_home2",
>
> "mode": "hot-standby",
>
> "heartbeat-delay": 200,
>
> "max-response-delay": 600,
>
> "max-ack-delay": 800,
>
> "max-unacked-clients": 5,
>
> "delayed-updates-limit": 0,
>
> "send-lease-updates": true,
>
> "sync-leases": true,
>
> "sync-page-limit": 200,
>
> "sync-timeout": 600,
>
> "peers": [{
>
> "name": "kea_home1",
>
> "url": https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$> ,
>
> "role": "primary",
>
> "auto-failover": true
>
> }, {
>
> "name": "kea_home2",
>
> "url": https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$<https://urldefense.com/v3/__http:/192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$> ,
>
> "role": "standby",
>
> "auto-failover": true
>
> }]
>
> }]
>
> }
>
> }],
>
>
>
> "Control-agent": {
>
> "http-host": "192.168.100.69",
>
> // If enabling HA and multi-threading, the 8000 port is used by the HA
>
> // hook library http listener. When using HA hook library with
>
> // multi-threading to function, make sure the port used by dedicated
>
> // listener is different (e.g. 8001) than the one used by CA. Note
>
> // the commands should still be sent via CA. The dedicated listener
>
> // is specifically for HA updates only.
>
> "http-port": 8001,
>
>
>
>
>
> Best Regards,
>
> Yu
>
>
>
> --
> ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$> for more information.
>
> To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$> .
>
> Kea-users mailing list
> Kea-users at lists.isc.org
> https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$>
--
ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$> for more information.
To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$> .
Kea-users mailing list
Kea-users at lists.isc.org
https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20240328/150c29cb/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 382854 bytes
Desc: image001.png
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20240328/150c29cb/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 155959 bytes
Desc: image002.png
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20240328/150c29cb/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ha_ts_kea2.pcap
Type: application/octet-stream
Size: 4096 bytes
Desc: ha_ts_kea2.pcap
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20240328/150c29cb/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ha_ts_kea1.pcap
Type: application/octet-stream
Size: 4096 bytes
Desc: ha_ts_kea1.pcap
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20240328/150c29cb/attachment-0003.obj>
More information about the Kea-users
mailing list