[Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Xiao, Yu (CCI-Atlanta)
yu.xiao at cox.com
Thu Apr 4 13:57:38 UTC 2024
Hi Darren,
Thank you for the explanation. I tested again and I do see now the clients could extend their addresses without losing it.
I have another question, I re-enabled the primary kea-01 again, after that, I noticed Kea-01 seems to have retaken the role of the primary again automatically. Is there a way we can keep the standby continue serve as temporary primary instead of revert automatically? Also, is there a convenient way to check both hosts which one is serving the DHCP lease now instead of checking the details of the debug logs? I don’t remember see those details from the 2.4.1 documentation.
Kea-01:
2024-04-04 09:49:40.239 DEBUG [kea-dhcp6.alloc-engine/41273.139801047443200] ALLOC_ENGINE_V6_EXTEND_LEASE duid=[00:04:bf:aa:9f:84:51:61:0b:20:44:a6:6a:ec:1b:99:8b:b3], tid=0xe8593b: extending lifetime of the lease type IA_NA, address fd74:5656:15e2:10::202
2024-04-04 09:49:40.239 DEBUG [kea-dhcp6.alloc-engine/41273.139801047443200] ALLOC_ENGINE_V6_EXTEND_LEASE_DATA duid=[00:04:bf:aa:9f:84:51:61:0b:20:44:a6:6a:ec:1b:99:8b:b3], tid=0xe8593b: detailed information about the lease being extended: Type: IA_NA(0)
Address: fd74:5656:15e2:10::202
Prefix length: 128
IAID: 3394439514
Pref life: 300
Valid life: 400
Cltt: 1712238385
DUID: 00:04:bf:aa:9f:84:51:61:0b:20:44:a6:6a:ec:1b:99:8b:b3
Hardware addr: (none)
Subnet ID: 1
Pool ID: 0
State: default
2024-04-04 09:49:40.239 DEBUG [kea-dhcp6.alloc-engine/41273.139801047443200] ALLOC_ENGINE_V6_EXTEND_NEW_LEASE_DATA duid=[00:04:bf:aa:9f:84:51:61:0b:20:44:a6:6a:ec:1b:99:8b:b3], tid=0xe8593b: new lease information for the lease being extended: Type: IA_NA(0)
Address: fd74:5656:15e2:10::202
Prefix length: 128
IAID: 3394439514
Pref life: 300
Valid life: 400
Cltt: 1712238580
DUID: 00:04:bf:aa:9f:84:51:61:0b:20:44:a6:6a:ec:1b:99:8b:b3
Hardware addr: (none)
Subnet ID: 1
Pool ID: 0
State: default
2024-04-04 09:49:40.239 DEBUG [kea-dhcp6.dhcpsrv/41273.139801047443200] DHCPSRV_MEMFILE_UPDATE_ADDR6 updating IPv6 lease for address fd74:5656:15e2:10::202
2024-04-04 09:49:40.239 INFO [kea-dhcp6.leases/41273.139801047443200] DHCP6_LEASE_RENEW duid=[00:04:bf:aa:9f:84:51:61:0b:20:44:a6:6a:ec:1b:99:8b:b3], tid=0xe8593b: lease for address fd74:5656:15e2:10::202 and iaid=3394439514 has been allocated
2024-04-04 09:49:40.239 DEBUG [kea-dhcp6.callouts/41273.139801047443200] HOOKS_CALLOUTS_BEGIN begin all callouts for hook leases6_committed
2024-04-04 09:49:40.239 DEBUG [kea-dhcp6.http/41273.139801047443200] HTTP_CLIENT_REQUEST_SEND sending HTTP request POST / HTTP/1.1 to http://192.168.100.70:8000/
2024-04-04 09:49:40.239 DEBUG [kea-dhcp6.http/41273.139801047443200] HTTP_CLIENT_REQUEST_SEND_DETAILS detailed information about request sent to http://192.168.100.70:8000/:
POST / HTTP/1.1
Host: 192.168.100.70
Content-Length: 427
Content-Type: application/json
{ "arguments": { "deleted-leases": [ ], "leases": [ { "duid": "00:04:bf:aa:9f:84:51:61:0b:20:44:a6:6a:ec:1b:99:8b:b3", "expire": 1712238980, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "iaid": 3394439514, "ip-address": "fd74:5656:15e2:10::202", "preferred-lft": 300, "state": 0, "subnet-id": 1, "type": "IA_NA", "valid-lft": 400 } ], "origin": "ha-partner" }, "command": "lease6-bulk-apply", "service": [ "dhcp6" ] }
Kea-03:
2024-04-04 09:48:02.103 DEBUG [kea-dhcp6.http/41298.140241352439552] HTTP_SERVER_RESPONSE_SEND_DETAILS detailed information about response sent to 192.168.100.197:
HTTP/1.1 200 OK
Content-Length: 184
Content-Type: application/json
Date: Thu, 04 Apr 2024 13:48:02 GMT
[ { "arguments": { "date-time": "Thu, 04 Apr 2024 13:48:02 GMT", "scopes": [ ], "state": "hot-standby", "unsent-update-count": 4 }, "result": 0, "text": "HA peer status returned." } ]
2024-04-04 09:48:06.081 DEBUG [kea-dhcp6.packets/41298.140241550008320] DHCP6_BUFFER_RECEIVED received buffer from fe80::7cbe:6ff:fe66:e263:546 to ff02::1:2:0 over interface ens18
2024-04-04 09:48:06.081 DEBUG [kea-dhcp6.callouts/41298.140241419581184] HOOKS_CALLOUTS_BEGIN begin all callouts for hook buffer6_receive
2024-04-04 09:48:06.081 DEBUG [kea-dhcp6.ha-hooks/41298.140241419581184] HA_BUFFER6_RECEIVE_NOT_FOR_US duid=[00:04:bf:aa:9f:84:51:61:0b:20:44:a6:6a:ec:1b:99:8b:b3], tid=0x816669: dropping query to be processed by another server
2024-04-04 09:48:06.081 DEBUG [kea-dhcp6.callouts/41298.140241419581184] HOOKS_CALLOUT_CALLED hooks library with index 2 has called a callout on hook buffer6_receive that has address 0x7f8c802af060 (callout duration: 0.121 ms)
2024-04-04 09:48:06.081 DEBUG [kea-dhcp6.callouts/41298.140241419581184] HOOKS_CALLOUTS_COMPLETE completed callouts for hook buffer6_receive (total callouts duration: 0.121 ms)
2024-04-04 09:48:06.081 DEBUG [kea-dhcp6.hooks/41298.140241419581184] DHCP6_HOOK_BUFFER_RCVD_DROP received buffer from fe80::7cbe:6ff:fe66:e263 to ff02::1:2 over interface ens18 was dropped because a callout set the drop flag
2024-04-04 09:48:09.444 DEBUG [kea-dhcp6.packets/41298.140241550008320] DHCP6_BUFFER_RECEIVED received buffer from fe80::c40b:ebff:fed1:7298:546 to ff02::1:2:0 over interface ens18
2024-04-04 09:48:09.445 DEBUG [kea-dhcp6.callouts/41298.140241411188480] HOOKS_CALLOUTS_BEGIN begin all callouts for hook buffer6_receive
2024-04-04 09:48:09.445 DEBUG [kea-dhcp6.ha-hooks/41298.140241411188480] HA_BUFFER6_RECEIVE_NOT_FOR_US duid=[00:04:a3:35:01:e3:85:15:e1:76:3e:47:e3:b0:c5:f8:55:10], tid=0x852612: dropping query to be processed by another server
2024-04-04 09:48:09.445 DEBUG [kea-dhcp6.callouts/41298.140241411188480] HOOKS_CALLOUT_CALLED hooks library with index 2 has called a callout on hook buffer6_receive that has address 0x7f8c802af060 (callout duration: 0.127 ms)
Best Regards,
Yu
From: Kea-users <kea-users-bounces at lists.isc.org> on behalf of Darren Ankney <darren.ankney at gmail.com>
Date: Thursday, April 4, 2024 at 5:48 AM
To: Kea-users at lists.isc.org <kea-users at lists.isc.org>
Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Hi Yu,
Basically you are not waiting for clients to go down. Clients will
attempt to renew at lease half-life. "max-unacked-clients" is a way
of confirming that the other server is actually down. In certain
scenarios it would be possible that the two servers couldn't see each
other but COULD see the clients. If they were both answering the
clients, that could lead to duplicate IP address assignment and the
like.
Thank you,
Darren Ankney
On Wed, Apr 3, 2024 at 5:32 PM Xiao, Yu (CCI-Atlanta) <yu.xiao at cox.com> wrote:
>
> Hi Darren,
>
>
>
> Thank you for your information. I adjusted the parameter and now it is working. But I have a question, why should we need this "max-unacked-clients" parameter? Because imagine we have 1k devices in this network, and they all have their IP assigned by dhcp_1, then suddenly dhcp_1 went down, but at this moment, all devices still have their assigned IPs. Then if we design dhcp_2 to become primary after losing sync time, we may be able to bring dhcp_2 to primary before T2, so there won’t be large number of devices offline. But if we wait until offline devices observed, then it could be too late and many devices already offline.
>
>
>
>
>
>
>
> Best Regards,
>
> Yu
>
>
>
>
>
> From: Kea-users <kea-users-bounces at lists.isc.org> on behalf of Xiao, Yu (CCI-Atlanta) via Kea-users <kea-users at lists.isc.org>
> Date: Wednesday, April 3, 2024 at 9:12 AM
> To: Darren Ankney <darren.ankney at gmail.com>, Kea user's list <kea-users at lists.isc.org>
> Cc: Xiao, Yu (CCI-Atlanta) <yu.xiao at cox.com>
> Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
>
> Hi Darren,
>
>
>
> Thank you for the explanation. Let me investigate further.
>
>
>
>
>
> Best Regards,
>
> Yu
>
>
>
>
>
> From: Darren Ankney <darren.ankney at gmail.com>
> Date: Wednesday, April 3, 2024 at 9:06 AM
> To: Kea user's list <kea-users at lists.isc.org>
> Cc: Xiao, Yu (CCI-Atlanta) <yu.xiao at cox.com>
> Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
>
> Hi Yu,
>
> Look in your logs for messages containing
> HA_COMMUNICATION_INTERRUPTED_CLIENT4_UNACKED. These messages indicate
> how many clients are still needed before the failover will occur.
> Your settings specify:
>
> "max-ack-delay": 300,
> "max-unacked-clients": 5,
>
> max-ack-delay is specified in milliseconds so pretty much every client
> that reaches one in the "elapsed time" option in DHCPv6 will meet the
> criteria of an "unacked" client. Once the sixth client shows up,
> failover should occur. That log message should repeat with each new
> client being tracked giving statistics about how many more are needed.
>
> Thank you,
> Darren Ankney
>
>
> On Tue, Apr 2, 2024 at 12:39 PM Xiao, Yu (CCI-Atlanta) via Kea-users
> <kea-users at lists.isc.org> wrote:
> >
> > Hi experts,
> >
> >
> >
> > I have tested further in my lab and found the answers to my questions. HA works when I don’t enable the control agent. As for the debug log, I think there’s a potential document bug for 2.4.1 document. Cause you can’t assign the log to /var/log folder, you need to assign it to /tmp/ folder.
> >
> >
> >
> > I found a problem on myside on firewall, after I fixed it. Now seems the HA is working. But still having one problem. When I used “systemctl stop kea-dhcp6” on the primary, for example, kea01, then the standby DHCPv6 server kea03 won’t process the DHCP packets from clients. I waited for more time, but the behaviors won’t change. The standby server just drop those DHCP packets from clients due to “dropping query to be processed by another server”.
> >
> >
> >
> > According to Kea documentation, this happens when:
> >
> >
> >
> > “%1: dropping query to be processed by another server
> >
> >
> >
> > This debug message is issued when the received DHCPv6 query is dropped by this server because it should be served by another server. This is the case when the remote server was designated to process the packet as a result of load balancing or because it is a primary server in the hot standby configuration. The argument provides client identification information retrieved from the query.”
> >
> >
> >
> > But now since the primary service is down, so the standby server should become the primary one, right? But seems not the case. Also from the debug logs, I can’t see the current status of the remaining server, technically we should be able to see its status changed from standby to primary, right? But I only see the role information from the logs before I shut down the kea service on primary server.
> >
> >
> >
> > Config:
> >
> > # HA related hooks configuration
> >
> > "hooks-libraries": [{
> >
> > "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so",
> >
> > "parameters": { }
> >
> > }, {
> >
> > "library": "/usr/lib64/kea/hooks/libdhcp_ha.so",
> >
> > "parameters": {
> >
> > "high-availability": [{
> >
> > "this-server-name": "kea_home3",
> >
> > "mode": "hot-standby",
> >
> > "heartbeat-delay": 100,
> >
> > "max-response-delay": 200,
> >
> > "max-ack-delay": 300,
> >
> > "max-unacked-clients": 5,
> >
> > "peers": [{
> >
> > "name": "kea_home1",
> >
> > "url": https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$> ,
> >
> > "role": "primary",
> >
> > "auto-failover": true
> >
> > }, {
> >
> > "name": "kea_home3",
> >
> > "url": https://urldefense.com/v3/__http://192.168.100.70:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvU0f5KAX$<https://urldefense.com/v3/__http:/192.168.100.70:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvU0f5KAX$> ,
> >
> > "role": "standby",
> >
> > "auto-failover": true
> >
> > }]
> >
> > }]
> >
> > }
> >
> > }],
> >
> >
> >
> >
> >
> > Logs:
> >
> > 2024-04-02 12:03:41.696 DEBUG [kea-dhcp6.http/15314.139903918055424] HTTP_CLIENT_REQUEST_SEND_DETAILS detailed information about request sent to https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$> :
> >
> > POST / HTTP/1.1
> >
> > Host: 192.168.100.197
> >
> > Content-Length: 53
> >
> > Content-Type: application/json
> >
> >
> >
> > { "command": "ha-heartbeat", "service": [ "dhcp6" ] }
> >
> > 2024-04-02 12:03:41.696 DEBUG [kea-dhcp6.http/15314.139903754057472] HTTP_BAD_SERVER_RESPONSE_RECEIVED bad response received when communicating with https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$> : Connection refused
> >
> > 2024-04-02 12:03:41.696 WARN [kea-dhcp6.ha-hooks/15314.139903754057472] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$ ): Connection refused
> >
> > 2024-04-02 12:03:41.696 WARN [kea-dhcp6.ha-hooks/15314.139903754057472] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
> >
> > 2024-04-02 12:03:41.773 DEBUG [kea-dhcp6.packets/15314.139903918055424] DHCP6_BUFFER_RECEIVED received buffer from fe80::c40b:ebff:fed1:7298:546 to ff02::1:2:0 over interface ens18
> >
> > 2024-04-02 12:03:41.773 DEBUG [kea-dhcp6.callouts/15314.139903787628288] HOOKS_CALLOUTS_BEGIN begin all callouts for hook buffer6_receive
> >
> > 2024-04-02 12:03:41.773 DEBUG [kea-dhcp6.ha-hooks/15314.139903787628288] HA_BUFFER6_RECEIVE_NOT_FOR_US duid=[00:04:a3:35:01:e3:85:15:e1:76:3e:47:e3:b0:c5:f8:55:10], tid=0x65a570: dropping query to be processed by another server
> >
> > 2024-04-02 12:03:41.773 DEBUG [kea-dhcp6.callouts/15314.139903787628288] HOOKS_CALLOUT_CALLED hooks library with index 2 has called a callout on hook buffer6_receive that has address 0x7f3de3bc3060 (callout duration: 0.161 ms)
> >
> > 2024-04-02 12:03:41.773 DEBUG [kea-dhcp6.callouts/15314.139903787628288] HOOKS_CALLOUTS_COMPLETE completed callouts for hook buffer6_receive (total callouts duration: 0.161 ms)
> >
> > 2024-04-02 12:03:41.773 DEBUG [kea-dhcp6.hooks/15314.139903787628288] DHCP6_HOOK_BUFFER_RCVD_DROP received buffer from fe80::c40b:ebff:fed1:7298 to ff02::1:2 over interface ens18 was dropped because a callout set the drop flag
> >
> > 2024-04-02 12:03:42.774 DEBUG [kea-dhcp6.http/15314.139903918055424] HTTP_CLIENT_REQUEST_SEND sending HTTP request POST / HTTP/1.1 to https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$>
> >
> > 2024-04-02 12:03:42.774 DEBUG [kea-dhcp6.http/15314.139903918055424] HTTP_CLIENT_REQUEST_SEND_DETAILS detailed information about request sent to https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$> :
> >
> > POST / HTTP/1.1
> >
> > Host: 192.168.100.197
> >
> > Content-Length: 53
> >
> > Content-Type: application/json
> >
> >
> >
> > { "command": "ha-heartbeat", "service": [ "dhcp6" ] }
> >
> > 2024-04-02 12:03:42.775 DEBUG [kea-dhcp6.http/15314.139903728879360] HTTP_BAD_SERVER_RESPONSE_RECEIVED bad response received when communicating with https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$> : Connection refused
> >
> > 2024-04-02 12:03:42.775 WARN [kea-dhcp6.ha-hooks/15314.139903728879360] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$ ): Connection refused
> >
> > 2024-04-02 12:03:42.775 WARN [kea-dhcp6.ha-hooks/15314.139903728879360] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
> >
> > 2024-04-02 12:03:43.776 DEBUG [kea-dhcp6.http/15314.139903918055424] HTTP_CLIENT_REQUEST_SEND sending HTTP request POST / HTTP/1.1 to https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$>
> >
> > 2024-04-02 12:03:43.776 DEBUG [kea-dhcp6.http/15314.139903918055424] HTTP_CLIENT_REQUEST_SEND_DETAILS detailed information about request sent to https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$> :
> >
> > POST / HTTP/1.1
> >
> > Host: 192.168.100.197
> >
> > Content-Length: 53
> >
> > Content-Type: application/json
> >
> >
> >
> > Server status from debug logs:
> >
> >
> >
> > 2024-04-02 11:50:32.503 DEBUG [kea-dhcp6.dhcp6/15314.139903918055424] DHCP6_CONFIG_START DHCPv6 server is processing the following configuration: { "control-socket": { "socket-name": "/tmp/kea6-ctrl-socket", "socket-type": "unix" }, "hooks-libraries": [ { "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so", "parameters": { } }, { "library": "/usr/lib64/kea/hooks/libdhcp_ha.so", "parameters": { "high-availability": [ { "heartbeat-delay": 100, "max-ack-delay": 300, "max-response-delay": 200, "max-unacked-clients": 5, "mode": "hot-standby", "peers": [ { "auto-failover": true, "name": "kea_home1", "role": "primary", "url": https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$> }, { "auto-failover": true, "name": "kea_home3", "role": "standby", "url": https://urldefense.com/v3/__http://192.168.100.70:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvU0f5KAX$<https://urldefense.com/v3/__http:/192.168.100.70:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvU0f5KAX$> } ], "this-server-name": "kea_home3" } ] } } ], "interfaces-config": { "interfaces": [ "ens18" ], "service-sockets-require-all": true }, "lease-database": { "name": "/var/lib/kea/dhcp6.leases", "persist": true, "type": "memfile" }, "loggers": [ { "debuglevel": 99, "name": "kea-dhcp6", "output_options": [ { "maxsize": 40480000, "maxver": 8, "output": "/tmp/kea-debug.log" } ], "severity": "DEBUG" } ], "preferred-lifetime": 300, "rebind-timer": 200, "renew-timer": 100, "subnet6": [ { "id": 1, "interface": "ens18", "pools": [ { "pool": "fd74:5656:15e2:10::200-fd74:5656:15e2:10::9999" } ], "subnet": "fd74:5656:15e2:10::/64" } ], "valid-lifetime": 400 }
> >
> >
> >
> >
> >
> > Best Regards,
> >
> > Yu
> >
> >
> >
> >
> >
> > From: Kea-users <kea-users-bounces at lists.isc.org> on behalf of Xiao, Yu (CCI-Atlanta) via Kea-users <kea-users at lists.isc.org>
> > Date: Monday, April 1, 2024 at 5:18 PM
> > To: Kea user's list <kea-users at lists.isc.org>
> > Cc: Xiao, Yu (CCI-Atlanta) <yu.xiao at cox.com>
> > Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
> >
> > Hi Darren/experts,
> >
> >
> >
> > I have changed to 8000 port, but still the HA is not working. I have the following questions:
> >
> >
> >
> > Do I need to enable kea-ctrl-agent to make HA work? Or they are separate?
> > I configured the logger related configuration, but I don’t see the log file has been created, did I miss something? Here’s my config:
> >
> >
> >
> > "loggers": [
> >
> > {
> >
> > "name": "kea-dhcp6",
> >
> > "output_options": [
> >
> > {
> >
> > "output": "/var/log/kea-debug.log",
> >
> > "maxver": 8,
> >
> > "maxsize": 204800,
> >
> > "flush": true,
> >
> > "pattern": "%d{%j %H:%M:%S.%q} %c %m\n"
> >
> > }
> >
> > ],
> >
> > "severity": "DEBUG",
> >
> > "debuglevel": 99
> >
> > }
> >
> >
> >
> > ]
> >
> >
> >
> > [root at kea_home3 kea]# systemctl status kea-dhcp6.service
> >
> > ● kea-dhcp6.service - Kea DHCPv6 Service
> >
> > Loaded: loaded (/usr/lib/systemd/system/kea-dhcp6.service; enabled; vendor preset: disabled)
> >
> > Active: active (running) since Mon 2024-04-01 17:08:48 EDT; 4min 30s ago
> >
> > Docs: man:kea-dhcp6(8)
> >
> > Main PID: 4085 (kea-dhcp6)
> >
> > Tasks: 13 (limit: 10999)
> >
> > Memory: 4.3M
> >
> > CGroup: /system.slice/kea-dhcp6.service
> >
> > └─4085 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf
> >
> >
> >
> > Here’s a kea folder but no kea-debug.log as I defined:
> >
> >
> >
> > [root at kea_home3 kea]# ll /var/log/ |grep kea
> >
> > drwxr-x---. 2 kea kea 6 Nov 23 14:06 kea
> >
> >
> >
> >
> >
> > Best Regards,
> >
> > Yu
> >
> >
> >
> >
> >
> > From: Kea-users <kea-users-bounces at lists.isc.org> on behalf of Xiao, Yu (CCI-Atlanta) via Kea-users <kea-users at lists.isc.org>
> > Date: Friday, March 29, 2024 at 10:47 AM
> > To: Kea user's list <kea-users at lists.isc.org>
> > Cc: Xiao, Yu (CCI-Atlanta) <yu.xiao at cox.com>
> > Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
> >
> > Thank you for the advice, Darren. Let me test on them.
> >
> >
> >
> >
> >
> > Best Regards,
> >
> > Yu
> >
> >
> >
> >
> >
> > From: Kea-users <kea-users-bounces at lists.isc.org> on behalf of Darren Ankney <darren.ankney at gmail.com>
> > Date: Friday, March 29, 2024 at 6:58 AM
> > To: Kea user's list <kea-users at lists.isc.org>
> > Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
> >
> > Hi Yu,
> >
> > You will need to enable multi-threading dedicated listener for HA as
> > described here:
> > https://urldefense.com/v3/__https://kea.readthedocs.io/en/kea-2.4.1/arm/hooks.html*multi-threaded-configuration-ha-mt__;Iw!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1garUIhR_$<https://urldefense.com/v3/__https:/kea.readthedocs.io/en/kea-2.4.1/arm/hooks.html*multi-threaded-configuration-ha-mt__;Iw!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1garUIhR_$>
> > or alter your "url" in each in the peers blocks to port 8001 so that
> > they communicate via the kea-ctrl-agent. I think that should fix it.
> >
> > Thank you,
> > Darren Ankney
> >
> > On Tue, Mar 26, 2024 at 7:58 AM Xiao, Yu (CCI-Atlanta) via Kea-users
> > <kea-users at lists.isc.org> wrote:
> > >
> > > Hi Darren,
> > >
> > >
> > >
> > > I am using 2.4.1.
> > >
> > >
> > >
> > >
> > >
> > > Best Regards,
> > >
> > > Yu
> > >
> > >
> > >
> > >
> > >
> > > From: Kea-users <kea-users-bounces at lists.isc.org> on behalf of Darren Ankney <darren.ankney at gmail.com>
> > > Date: Tuesday, March 26, 2024 at 6:03 AM
> > > To: Kea user's list <kea-users at lists.isc.org>
> > > Subject: [EXTERNAL] Re: [Kea-users] kea-2.4.1 // Strange HA related errors
> > >
> > > Hello Yu,
> > >
> > > What version of Kea are you running on these servers, out of curiosity?
> > >
> > > Thank you,
> > > Darren Ankney
> > >
> > > On Mon, Mar 25, 2024 at 10:55 AM Xiao, Yu (CCI-Atlanta) via Kea-users
> > > <kea-users at lists.isc.org> wrote:
> > > >
> > > > Hi experts,
> > > >
> > > >
> > > >
> > > > I have configured two VMs in the same hypervisor as hot-standby mode HA. I believe they are successfully communicating with each other with heart beat packets, as we can see the primary VM kea-1 has successfully received the “ha-heartbeat” from the standby VM kea-2 in green logs. But the red logs indicate that the ha_hooks think the HA heartbeat communications failed due to “no route”. But this is a LAN network, and there’s indeed route installed as we can see below and we can ping the 69 ip.
> > > >
> > > >
> > > >
> > > > [yxiao322 at kea_home1 ~]$ ip route
> > > >
> > > > 192.168.100.0/24 dev ens18 proto kernel scope link src 192.168.100.197 metric 100 <<< This should be the route to 192.168.100.69
> > > >
> > > > 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
> > > >
> > > >
> > > >
> > > > [root at kea_home1 yxiao322]# ping 192.168.100.69
> > > >
> > > > PING 192.168.100.69 (192.168.100.69) 56(84) bytes of data.
> > > >
> > > > 64 bytes from 192.168.100.69: icmp_seq=1 ttl=64 time=0.342 ms
> > > >
> > > > 64 bytes from 192.168.100.69: icmp_seq=2 ttl=64 time=0.302 ms
> > > >
> > > > 64 bytes from 192.168.100.69: icmp_seq=3 ttl=64 time=0.236 ms
> > > >
> > > > ^Z
> > > >
> > > > [1]+ Stopped ping 192.168.100.69
> > > >
> > > >
> > > >
> > > > And if I stop the kea service on primary, then we can see standby server will complain “communication with kea_home1 is interrupted”. And as soon as I start kea service again on primary, then the database began sync again. Thus, I believe there’s indeed communications and syncs between primary and standby VMs. But for some reason, if I shut the kea service on primary, then the standby won’t distribute DHCP leases even after I waited for a long time. Did I miss something here?
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Primary logs:
> > > >
> > > >
> > > >
> > > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.198 INFO [kea-dhcp6.commands/1224.139988007651072] COMMAND_RECEIVED Received command 'ha-heartbeat'
> > > >
> > > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN [kea-dhcp6.ha-hooks/1224.139988049614592] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 (https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$ ): No route to host
> > > >
> > > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN [kea-dhcp6.ha-hooks/1224.139988049614592] HA_COMMUNICATION_INTERRUPTED communication with kea_home2 is interrupted
> > > >
> > > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.199 INFO [kea-dhcp6.commands/1224.139988016043776] COMMAND_RECEIVED Received command 'ha-heartbeat'
> > > >
> > > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN [kea-dhcp6.ha-hooks/1224.139988032829184] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 (https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$ ): No route to host
> > > >
> > > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN [kea-dhcp6.ha-hooks/1224.139988032829184] HA_COMMUNICATION_INTERRUPTED communication with kea_home2 is interrupted
> > > >
> > > >
> > > >
> > > > Standby logs:
> > > >
> > > >
> > > >
> > > > Mar 25 10:10:24 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:24.129 WARN [kea-dhcp6.ha-hooks/2836.139717198915328] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
> > > >
> > > > Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN [kea-dhcp6.ha-hooks/2836.139717207308032] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$ ): Connection refused
> > > >
> > > > Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN [kea-dhcp6.ha-hooks/2836.139717207308032] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
> > > >
> > > > Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN [kea-dhcp6.ha-hooks/2836.139717190522624] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$ ): Connection refused
> > > >
> > > > Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN [kea-dhcp6.ha-hooks/2836.139717190522624] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
> > > >
> > > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.136 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from PARTNER-DOWN to WAITING state, partner state is PARTNER-DOWN
> > > >
> > > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in WAITING state
> > > >
> > > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the kea_home2 is in the WAITING state
> > > >
> > > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from WAITING to SYNCING state, partner state is PARTNER-DOWN
> > > >
> > > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in SYNCING state
> > > >
> > > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_SYNC_START starting lease database synchronization with kea_home1
> > > >
> > > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.140 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 3 leases from kea_home1
> > > >
> > > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.141 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_SYNC_SUCCESSFUL lease database synchronization with kea_home1 completed successfully in 4.350 ms
> > > >
> > > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.142 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from SYNCING to READY state, partner state is PARTNER-DOWN
> > > >
> > > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.142 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in READY state
> > > >
> > > >
> > > >
> > > > Primary Config:
> > > >
> > > >
> > > >
> > > > # HA related hooks configuration
> > > >
> > > > "hooks-libraries": [{
> > > >
> > > > "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so",
> > > >
> > > > "parameters": { }
> > > >
> > > > }, {
> > > >
> > > > "library": "/usr/lib64/kea/hooks/libdhcp_ha.so",
> > > >
> > > > "parameters": {
> > > >
> > > > "high-availability": [{
> > > >
> > > > "this-server-name": "kea_home1",
> > > >
> > > > "mode": "hot-standby",
> > > >
> > > > "heartbeat-delay": 200,
> > > >
> > > > "max-response-delay": 600,
> > > >
> > > > "max-ack-delay": 800,
> > > >
> > > > "max-unacked-clients": 5,
> > > >
> > > > "send-lease-updates": true,
> > > >
> > > > "sync-leases": true,
> > > >
> > > > "sync-page-limit": 200,
> > > >
> > > > "sync-timeout": 600,
> > > >
> > > > "delayed-updates-limit": 0,
> > > >
> > > > "peers": [{
> > > >
> > > > "name": "kea_home1",
> > > >
> > > > "url": https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$> ,
> > > >
> > > > "role": "primary",
> > > >
> > > > "auto-failover": true
> > > >
> > > > }, {
> > > >
> > > > "name": "kea_home2",
> > > >
> > > > "url": https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$<https://urldefense.com/v3/__http:/192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$> ,
> > > >
> > > > "role": "standby",
> > > >
> > > > "auto-failover": true
> > > >
> > > > }]
> > > >
> > > > }]
> > > >
> > > > }
> > > >
> > > > }],
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > "Control-agent": {
> > > >
> > > > "http-host": "192.168.100.197",
> > > >
> > > > // If enabling HA and multi-threading, the 8000 port is used by the HA
> > > >
> > > > // hook library http listener. When using HA hook library with
> > > >
> > > > // multi-threading to function, make sure the port used by dedicated
> > > >
> > > > // listener is different (e.g. 8001) than the one used by CA. Note
> > > >
> > > > // the commands should still be sent via CA. The dedicated listener
> > > >
> > > > // is specifically for HA updates only.
> > > >
> > > > "http-port": 8001,
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Standby config:
> > > >
> > > >
> > > >
> > > > # HA related hooks configuration
> > > >
> > > > "hooks-libraries": [{
> > > >
> > > > "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so",
> > > >
> > > > "parameters": { }
> > > >
> > > > }, {
> > > >
> > > > "library": "/usr/lib64/kea/hooks/libdhcp_ha.so",
> > > >
> > > > "parameters": {
> > > >
> > > > "high-availability": [{
> > > >
> > > > "this-server-name": "kea_home2",
> > > >
> > > > "mode": "hot-standby",
> > > >
> > > > "heartbeat-delay": 200,
> > > >
> > > > "max-response-delay": 600,
> > > >
> > > > "max-ack-delay": 800,
> > > >
> > > > "max-unacked-clients": 5,
> > > >
> > > > "delayed-updates-limit": 0,
> > > >
> > > > "send-lease-updates": true,
> > > >
> > > > "sync-leases": true,
> > > >
> > > > "sync-page-limit": 200,
> > > >
> > > > "sync-timeout": 600,
> > > >
> > > > "peers": [{
> > > >
> > > > "name": "kea_home1",
> > > >
> > > > "url": https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$> ,
> > > >
> > > > "role": "primary",
> > > >
> > > > "auto-failover": true
> > > >
> > > > }, {
> > > >
> > > > "name": "kea_home2",
> > > >
> > > > "url": https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$<https://urldefense.com/v3/__http:/192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$> ,
> > > >
> > > > "role": "standby",
> > > >
> > > > "auto-failover": true
> > > >
> > > > }]
> > > >
> > > > }]
> > > >
> > > > }
> > > >
> > > > }],
> > > >
> > > >
> > > >
> > > > "Control-agent": {
> > > >
> > > > "http-host": "192.168.100.69",
> > > >
> > > > // If enabling HA and multi-threading, the 8000 port is used by the HA
> > > >
> > > > // hook library http listener. When using HA hook library with
> > > >
> > > > // multi-threading to function, make sure the port used by dedicated
> > > >
> > > > // listener is different (e.g. 8001) than the one used by CA. Note
> > > >
> > > > // the commands should still be sent via CA. The dedicated listener
> > > >
> > > > // is specifically for HA updates only.
> > > >
> > > > "http-port": 8001,
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Best Regards,
> > > >
> > > > Yu
> > > >
> > > >
> > > >
> > > > --
> > > > ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$> for more information.
> > > >
> > > > To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$> .
> > > >
> > > > Kea-users mailing list
> > > > Kea-users at lists.isc.org
> > > > https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$>
> > > --
> > > ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$> for more information.
> > >
> > > To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$> .
> > >
> > > Kea-users mailing list
> > > Kea-users at lists.isc.org
> > > https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$>
> > >
> > > --
> > > ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1gbTSITyu$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1gbTSITyu$> for more information.
> > >
> > > To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1gf4IdHNb$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1gf4IdHNb$> .
> > >
> > > Kea-users mailing list
> > > Kea-users at lists.isc.org
> > > https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1gf4IdHNb$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1gf4IdHNb$>
> > --
> > ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1gbTSITyu$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1gbTSITyu$> for more information.
> >
> > To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1gf4IdHNb$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1gf4IdHNb$> .
> >
> > Kea-users mailing list
> > Kea-users at lists.isc.org
> > https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1gf4IdHNb$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1gf4IdHNb$>
> >
> > --
> > ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvXwmzyK4$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvXwmzyK4$> for more information.
> >
> > To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvXS0EXvM$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvXS0EXvM$> .
> >
> > Kea-users mailing list
> > Kea-users at lists.isc.org
> > https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvXS0EXvM$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvXS0EXvM$>
--
ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!zYf21dDIQNcAF6xDwkGGjmHCI7R_PeM__xtOhpG1x8Uu7R2N95aufWvO8-tvevlBdEJ4PPD7U_xYHuGVs7jO$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!zYf21dDIQNcAF6xDwkGGjmHCI7R_PeM__xtOhpG1x8Uu7R2N95aufWvO8-tvevlBdEJ4PPD7U_xYHuGVs7jO$> for more information.
To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!zYf21dDIQNcAF6xDwkGGjmHCI7R_PeM__xtOhpG1x8Uu7R2N95aufWvO8-tvevlBdEJ4PPD7U_xYHns6h9X9$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!zYf21dDIQNcAF6xDwkGGjmHCI7R_PeM__xtOhpG1x8Uu7R2N95aufWvO8-tvevlBdEJ4PPD7U_xYHns6h9X9$> .
Kea-users mailing list
Kea-users at lists.isc.org
https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!zYf21dDIQNcAF6xDwkGGjmHCI7R_PeM__xtOhpG1x8Uu7R2N95aufWvO8-tvevlBdEJ4PPD7U_xYHns6h9X9$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!zYf21dDIQNcAF6xDwkGGjmHCI7R_PeM__xtOhpG1x8Uu7R2N95aufWvO8-tvevlBdEJ4PPD7U_xYHns6h9X9$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20240404/15af3ebc/attachment-0001.htm>
More information about the Kea-users
mailing list