[Kea-users] kea-2.4.1 // Strange HA related errors
Xiao, Yu (CCI-Atlanta)
yu.xiao at cox.com
Mon Mar 25 14:55:39 UTC 2024
Hi experts,
I have configured two VMs in the same hypervisor as hot-standby mode HA. I believe they are successfully communicating with each other with heart beat packets, as we can see the primary VM kea-1 has successfully received the “ha-heartbeat” from the standby VM kea-2 in green logs. But the red logs indicate that the ha_hooks think the HA heartbeat communications failed due to “no route”. But this is a LAN network, and there’s indeed route installed as we can see below and we can ping the 69 ip.
[yxiao322 at kea_home1 ~]$ ip route
192.168.100.0/24 dev ens18 proto kernel scope link src 192.168.100.197 metric 100 <<< This should be the route to 192.168.100.69
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
[root at kea_home1 yxiao322]# ping 192.168.100.69
PING 192.168.100.69 (192.168.100.69) 56(84) bytes of data.
64 bytes from 192.168.100.69: icmp_seq=1 ttl=64 time=0.342 ms
64 bytes from 192.168.100.69: icmp_seq=2 ttl=64 time=0.302 ms
64 bytes from 192.168.100.69: icmp_seq=3 ttl=64 time=0.236 ms
^Z
[1]+ Stopped ping 192.168.100.69
And if I stop the kea service on primary, then we can see standby server will complain “communication with kea_home1 is interrupted”. And as soon as I start kea service again on primary, then the database began sync again. Thus, I believe there’s indeed communications and syncs between primary and standby VMs. But for some reason, if I shut the kea service on primary, then the standby won’t distribute DHCP leases even after I waited for a long time. Did I miss something here?
Primary logs:
Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.198 INFO [kea-dhcp6.commands/1224.139988007651072] COMMAND_RECEIVED Received command 'ha-heartbeat'
Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN [kea-dhcp6.ha-hooks/1224.139988049614592] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 (http://192.168.100.69:8000/): No route to host
Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN [kea-dhcp6.ha-hooks/1224.139988049614592] HA_COMMUNICATION_INTERRUPTED communication with kea_home2 is interrupted
Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.199 INFO [kea-dhcp6.commands/1224.139988016043776] COMMAND_RECEIVED Received command 'ha-heartbeat'
Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN [kea-dhcp6.ha-hooks/1224.139988032829184] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 (http://192.168.100.69:8000/): No route to host
Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN [kea-dhcp6.ha-hooks/1224.139988032829184] HA_COMMUNICATION_INTERRUPTED communication with kea_home2 is interrupted
Standby logs:
Mar 25 10:10:24 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:24.129 WARN [kea-dhcp6.ha-hooks/2836.139717198915328] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN [kea-dhcp6.ha-hooks/2836.139717207308032] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (http://192.168.100.197:8000/): Connection refused
Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN [kea-dhcp6.ha-hooks/2836.139717207308032] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN [kea-dhcp6.ha-hooks/2836.139717190522624] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (http://192.168.100.197:8000/): Connection refused
Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN [kea-dhcp6.ha-hooks/2836.139717190522624] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.136 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from PARTNER-DOWN to WAITING state, partner state is PARTNER-DOWN
Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in WAITING state
Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the kea_home2 is in the WAITING state
Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from WAITING to SYNCING state, partner state is PARTNER-DOWN
Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in SYNCING state
Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_SYNC_START starting lease database synchronization with kea_home1
Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.140 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 3 leases from kea_home1
Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.141 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_SYNC_SUCCESSFUL lease database synchronization with kea_home1 completed successfully in 4.350 ms
Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.142 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from SYNCING to READY state, partner state is PARTNER-DOWN
Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.142 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in READY state
Primary Config:
# HA related hooks configuration
"hooks-libraries": [{
"library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so",
"parameters": { }
}, {
"library": "/usr/lib64/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [{
"this-server-name": "kea_home1",
"mode": "hot-standby",
"heartbeat-delay": 200,
"max-response-delay": 600,
"max-ack-delay": 800,
"max-unacked-clients": 5,
"send-lease-updates": true,
"sync-leases": true,
"sync-page-limit": 200,
"sync-timeout": 600,
"delayed-updates-limit": 0,
"peers": [{
"name": "kea_home1",
"url": http://192.168.100.197:8000/,
"role": "primary",
"auto-failover": true
}, {
"name": "kea_home2",
"url": http://192.168.100.69:8000/,
"role": "standby",
"auto-failover": true
}]
}]
}
}],
"Control-agent": {
"http-host": "192.168.100.197",
// If enabling HA and multi-threading, the 8000 port is used by the HA
// hook library http listener. When using HA hook library with
// multi-threading to function, make sure the port used by dedicated
// listener is different (e.g. 8001) than the one used by CA. Note
// the commands should still be sent via CA. The dedicated listener
// is specifically for HA updates only.
"http-port": 8001,
Standby config:
# HA related hooks configuration
"hooks-libraries": [{
"library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so",
"parameters": { }
}, {
"library": "/usr/lib64/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [{
"this-server-name": "kea_home2",
"mode": "hot-standby",
"heartbeat-delay": 200,
"max-response-delay": 600,
"max-ack-delay": 800,
"max-unacked-clients": 5,
"delayed-updates-limit": 0,
"send-lease-updates": true,
"sync-leases": true,
"sync-page-limit": 200,
"sync-timeout": 600,
"peers": [{
"name": "kea_home1",
"url": http://192.168.100.197:8000/,
"role": "primary",
"auto-failover": true
}, {
"name": "kea_home2",
"url": http://192.168.100.69:8000/,
"role": "standby",
"auto-failover": true
}]
}]
}
}],
"Control-agent": {
"http-host": "192.168.100.69",
// If enabling HA and multi-threading, the 8000 port is used by the HA
// hook library http listener. When using HA hook library with
// multi-threading to function, make sure the port used by dedicated
// listener is different (e.g. 8001) than the one used by CA. Note
// the commands should still be sent via CA. The dedicated listener
// is specifically for HA updates only.
"http-port": 8001,
Best Regards,
Yu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20240325/e99b90d9/attachment-0001.htm>
More information about the Kea-users
mailing list