[Kea-users] HA heartbeat communications failure

CS cs.temp.mail at gmail.com
Wed Dec 27 22:46:44 UTC 2023


telneting around it appears server1 can reach 8001 on itself but not 2.
server 2 can telnet 8001 on server 1 but not itself.
I see nothing logged that indicates a conflict tho and both of these
besides being on different subnets these servers are virtually identical
and firewall free.

CS, cs.Temp.Mail at gMail.com


On Wed, 27 Dec 2023 at 14:38, CS <cs.temp.mail at gmail.com> wrote:

> They made it through. With minor changes (ip addresses, library locations,
> and logging) the files are accepted and daemons are running but... no dice.
>
> server1$ tail -f /var/log/kea/kea-dhcp4.log
> 2023-12-27 22:24:48.486 INFO  [kea-dhcp4.dhcpsrv/1495687]
> DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
> /var/lib/kea/kea-leases4.csv
> 2023-12-27 22:24:48.486 INFO  [kea-dhcp4.dhcpsrv/1495687]
> DHCPSRV_MEMFILE_EXTRACT_EXTENDED_INFO4 extracting extended info saw 0
> leases, extended info sanity checks modified 0 / updated 0 leases and 0
> leases have relay or remote id
> 2023-12-27 22:24:48.486 INFO  [kea-dhcp4.dhcpsrv/1495687]
> DHCPSRV_MEMFILE_LFC_SETUP setting up the Lease File Cleanup interval to
> 3600 sec
> 2023-12-27 22:24:48.486 WARN  [kea-dhcp4.dhcpsrv/1495687]
> DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP traffic
> 2023-12-27 22:24:48.486 INFO  [kea-dhcp4.ha-hooks/1495687]
> HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the server1 is
> in the WAITING state
> 2023-12-27 22:24:48.486 INFO  [kea-dhcp4.ha-hooks/1495687]
> HA_SERVICE_STARTED started high availability service in load-balancing mode
> as primary server
> 2023-12-27 22:24:48.487 WARN  [kea-dhcp4.dhcp4/1495687]
> DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size:
> 64
> 2023-12-27 22:24:48.487 INFO  [kea-dhcp4.dhcp4/1495687] DHCP4_STARTED Kea
> DHCPv4 server version 2.4.0 started
> 2023-12-27 22:24:58.498 WARN  [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> 2023-12-27 22:25:08.510 WARN  [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> 2023-12-27 22:25:18.519 WARN  [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx5:8001): Connection refused
> 2023-12-27 22:25:28.531 WARN  [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> 2023-12-27 22:25:37.660 INFO  [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'ha-heartbeat'
> 2023-12-27 22:25:38.535 WARN  [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> 2023-12-27 22:25:47.674 INFO  [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'ha-heartbeat'
> 2023-12-27 22:25:48.546 WARN  [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> 2023-12-27 22:25:48.546 WARN  [kea-dhcp4.ha-hooks/1495687]
> HA_COMMUNICATION_INTERRUPTED communication with server2 is interrupted
> 2023-12-27 22:25:48.546 INFO  [kea-dhcp4.ha-hooks/1495687]
> HA_STATE_TRANSITION server transitions from WAITING to PARTNER-DOWN state,
> partner state is UNDEFINED
> 2023-12-27 22:25:48.547 INFO  [kea-dhcp4.ha-hooks/1495687]
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner
> while in PARTNER-DOWN state
> 2023-12-27 22:25:48.547 INFO  [kea-dhcp4.ha-hooks/1495687]
> HA_LOCAL_DHCP_ENABLE local DHCP service is enabled while the server1 is in
> the PARTNER-DOWN state
> 2023-12-27 22:25:57.687 INFO  [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'ha-heartbeat'
> 2023-12-27 22:25:57.690 INFO  [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'dhcp-disable'
> 2023-12-27 22:25:57.691 INFO  [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'lease4-get-page'
> 2023-12-27 22:25:57.693 INFO  [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'ha-sync-complete-notify'
> 2023-12-27 22:25:58.557 WARN  [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> 2023-12-27 22:25:58.557 WARN  [kea-dhcp4.ha-hooks/1495687]
> HA_COMMUNICATION_INTERRUPTED communication with server2 is interrupted
>
>
> server2$  tail -f /var/log/kea/kea-dhcp4.log
> 2023-12-27 22:25:26.650 INFO  [kea-dhcp4.dhcp4/1434251]
> DHCP4_CONFIG_COMPLETE DHCPv4 server has completed configuration: no IPv4
> subnets!; DDNS: disabled
> 2023-12-27 22:25:26.650 INFO  [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_MEMFILE_DB opening memory file lease database: type=memfile
> universe=4
> 2023-12-27 22:25:26.650 INFO  [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
> /var/lib/kea/kea-leases4.csv.2
> 2023-12-27 22:25:26.650 INFO  [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
> /var/lib/kea/kea-leases4.csv
> 2023-12-27 22:25:26.650 INFO  [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_MEMFILE_LFC_SETUP setting up the Lease File Cleanup interval to
> 3600 sec
> 2023-12-27 22:25:26.650 WARN  [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP traffic
> 2023-12-27 22:25:26.650 INFO  [kea-dhcp4.ha-hooks/1434251]
> HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the server2 is
> in the WAITING state
> 2023-12-27 22:25:26.650 INFO  [kea-dhcp4.ha-hooks/1434251]
> HA_SERVICE_STARTED started high availability service in load-balancing mode
> as secondary server
> 2023-12-27 22:25:26.650 WARN  [kea-dhcp4.dhcp4/1434251]
> DHCP4_MULTI_THREADING_INFO enabled: no, number of threads: 0, queue size: 0
> 2023-12-27 22:25:26.650 INFO  [kea-dhcp4.dhcp4/1434251] DHCP4_STARTED Kea
> DHCPv4 server version 2.2.0 started
> 2023-12-27 22:25:57.690 INFO  [kea-dhcp4.ha-hooks/1434251]
> HA_STATE_TRANSITION server transitions from WAITING to SYNCING state,
> partner state is PARTNER-DOWN
> 2023-12-27 22:25:57.690 INFO  [kea-dhcp4.ha-hooks/1434251]
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner
> while in SYNCING state
> 2023-12-27 22:25:57.690 INFO  [kea-dhcp4.ha-hooks/1434251] HA_SYNC_START
> starting lease database synchronization with server1
> 2023-12-27 22:25:57.693 INFO  [kea-dhcp4.ha-hooks/1434251]
> HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 0 leases from server1
> 2023-12-27 22:25:57.695 INFO  [kea-dhcp4.ha-hooks/1434251]
> HA_SYNC_SUCCESSFUL lease database synchronization with server1 completed
> successfully in 3.877 ms
> 2023-12-27 22:25:57.695 INFO  [kea-dhcp4.ha-hooks/1434251]
> HA_STATE_TRANSITION server transitions from SYNCING to READY state, partner
> state is PARTNER-DOWN
> 2023-12-27 22:25:57.695 INFO  [kea-dhcp4.ha-hooks/1434251]
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner
> while in READY state
>
> CS, cs.Temp.Mail at gMail.com
>
>
> On Wed, 27 Dec 2023 at 11:22, Darren Ankney <darren.ankney at gmail.com>
> wrote:
>
>> Hi,
>>
>> See attached four files:
>>
>> ca-server1.json (config for kea-ctrl-agent)
>> ca-server2.json  (config for kea-ctrl-agent)
>> dhcp4-server1.json  (config for kea-dhcp4)
>> dhcp4-server2.json  (config for kea-dhcp4)
>>
>> These files use port 8000 for kea-ctrl-agent and 8001 for kea-dhcp4 on
>> version 2.4.0.  They don't do anything other than setup the heartbeats
>> for HA (in fact you don't even need to run the control agent as Kea is
>> communicating directly).  Heartbeats are sent back and forth on port
>> 8001 as expected.  Can you give these a try and see if they work (in
>> testing of course, they won't serve any clients)?  I'm genuinely
>> curious if they work.  Yours should work (unless there is some problem
>> with the certificates or something).  I didn't notice any reason why
>> they wouldn't.
>>
>> Thank you,
>>
>> Darren Ankney
>>
>> PS: I'm not sure if these attachments will make it through to the list.
>>
>> On Wed, Dec 27, 2023 at 7:16 AM CS <cs.temp.mail at gmail.com> wrote:
>> >
>> > Kea 2.4.0
>> >
>> > On Wed, Dec 27, 2023, 03:18 Darren Ankney <darren.ankney at gmail.com>
>> wrote:
>> >>
>> >> Hi,
>> >>
>> >> If I may ask, what version of Kea are you using?  Some defaults have
>> >> changed across versions.
>> >>
>> >> Thank you,
>> >>
>> >> Darren Ankney
>> >>
>> >> On Tue, Dec 26, 2023 at 4:31 PM CS <cs.temp.mail at gmail.com> wrote:
>> >> >
>> >> > >Please describe what you mean by "it doesn't work".
>> >> > I mean I get a pretty useless error: "Unable to connect to Kea
>> Control Agent."
>> >> >
>> >> > > it might be be best to ask Men & Mice about "micetro" and how best
>> to set things
>> >> > I will at some point, when I find a resource with them. But there
>> are two players in this and since kea isn't behaving as expected like you,
>> I and the docs said. I'm starting here.
>> >> >
>> >> > >It actually SHOULDN'T work
>> >> > That's my read on it too. But here's proof. The CA config for one
>> server. It matches for the other server except certs and ip addresses obv.
>> >> >
>> >> >         "Control-agent": {
>> >> >                 "http-host": "xxx.xx1.xxx.xxx",
>> >> >                 "trust-anchor": "Certificate_Autority.pem",
>> >> >                 "cert-file": "ca1_cert.pem",
>> >> >                 "key-file": "ca1_key.pem",
>> >> >                 "cert-required": true,
>> >> >                 "http-port": 8000,
>> >> >                 "authentication": {
>> >> >                         "type": "basic",
>> >> >                         "realm": "kea-control-agent",
>> >> >                         "clients": [{
>> >> >                                 "user": "baduser",
>> >> >                                 "password": "badpassword",
>> >> >                         }]
>> >> >                 },
>> >> >
>> >> > And the dhcp4 config, likewise only the small differences between
>> the two servers
>> >> >
>> >> >  "hooks-libraries": [{
>> >> >                         "library":
>> "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
>> >> >                         "parameters": {}
>> >> >                 },{
>> >> >                         "library" :
>> "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
>> >> >                         "parameters": {
>> >> >                                 "high-availability": [{
>> >> >                                         "this-server-name": "
>> server1.org.org",
>> >> >                                         "mode": "load-balancing",
>> >> >                                         "heartbeat-delay": 10000,
>> >> >                                         "max-response-delay": 60000,
>> >> >                                         "max-ack-delay": 5000,
>> >> >                                         "max-unacked-clients": 0,
>> >> >                                         "require-client-certs": true,
>> >> >                                         "trust-anchor":
>> "Certificate_Autority.pem",
>> >> >                                         "auto-failover": true,
>> >> >
>> >> >                                         "peers": [{
>> >> >                                                 "name": "
>> server1.org.org",
>> >> >                                                 "url": "
>> http://xxx.xx1.xxx.xxx:8000/",
>> >> >                                                 "cert-file":
>> "dhcp1_cert.pem",
>> >> >                                                 "key-file":
>> "dhcp1_key.pem",
>> >> >                                                 "basic-auth-user":
>> "baduser",
>> >> >
>>  "basic-auth-password": "badpassword",
>> >> >                                                 "role": "primary",
>> >> >                                         },{
>> >> >                                                 "name": "
>> server2.org.org",
>> >> >                                                 "url": "
>> http://xxx.xx2.xxx.xxx:8000/",
>> >> >                                                 "cert-file":
>> "dhcp2_cert.pem",
>> >> >                                                 "key-file":
>> "dhcp2_key.pem",
>> >> >                                                 "role": "secondary",
>> >> >                                                 "basic-auth-user":
>> "baduser",
>> >> >
>>  "basic-auth-password": "badpassword",
>> >> >                                         }]
>> >> >                                 }]
>> >> >
>> >> > low and behold it runs. The same nature of daemon status and logs on
>> the other server.
>> >> >
>> >> > $ sudo systemctl restart isc-kea-ctrl-agent.service
>> isc-kea-dhcp4-server.service
>> >> > $ sudo systemctl status isc-kea-ctrl-agent.service
>> isc-kea-dhcp4-server.service
>> >> > ● isc-kea-ctrl-agent.service - Kea Control Agent
>> >> >      Loaded: loaded (/lib/systemd/system/isc-kea-ctrl-agent.service;
>> enabled; vendor preset: enabled)
>> >> >      Active: active (running) since Tue 2023-12-26 20:57:29 UTC; 11s
>> ago
>> >> >        Docs: man:kea-ctrl-agent(8)
>> >> >    Main PID: 1393724 (kea-ctrl-agent)
>> >> >       Tasks: 5 (limit: 19052)
>> >> >      Memory: 2.5M
>> >> >         CPU: 26ms
>> >> >      CGroup: /system.slice/isc-kea-ctrl-agent.service
>> >> >              └─1393724 /usr/sbin/kea-ctrl-agent -c
>> /etc/kea/kea-ctrl-agent.conf
>> >> >
>> >> > Dec 26 20:57:29 kea1 systemd[1]: Started Kea Control Agent.
>> >> >
>> >> > ● isc-kea-dhcp4-server.service - Kea DHCPv4 Service
>> >> >      Loaded: loaded
>> (/lib/systemd/system/isc-kea-dhcp4-server.service; enabled; vendor preset:
>> enabled)
>> >> >      Active: active (running) since Tue 2023-12-26 20:57:29 UTC; 11s
>> ago
>> >> >        Docs: man:kea-dhcp4(8)
>> >> >    Main PID: 1393730 (kea-dhcp4)
>> >> >       Tasks: 9 (limit: 19052)
>> >> >      Memory: 4.5M
>> >> >         CPU: 96ms
>> >> >      CGroup: /system.slice/isc-kea-dhcp4-server.service
>> >> >              └─1393730 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
>> >> >
>> >> > Dec 26 20:57:29 kea1 systemd[1]: isc-kea-dhcp4-server.service:
>> Deactivated successfully.
>> >> > Dec 26 20:57:29 kea1 systemd[1]: Stopped Kea DHCPv4 Service.
>> >> > Dec 26 20:57:29 kea1 systemd[1]: isc-kea-dhcp4-server.service:
>> Consumed 1min 28.504s CPU time.
>> >> > Dec 26 20:57:29 kea1 systemd[1]: Started Kea DHCPv4 Service.
>> >> >
>> >> > $ tail -n10 /var/log/kea/kea-ctrl-agent.log
>> >> > 2023-12-26 20:59:53.827 INFO  [kea-ctrl-agent.ctrl-agent/1393724]
>> CTRL_AGENT_COMMAND_RECEIVED command ha-heartbeat received from remote
>> address xxx.xxx2.xxx.xxx
>> >> > 2023-12-26 20:59:53.828 INFO  [kea-ctrl-agent.ctrl-agent/1393724]
>> CTRL_AGENT_COMMAND_FORWARDED command ha-heartbeat successfully forwarded to
>> the service dhcp4 from remote address xxx.xx2.xxx.xxx
>> >> > 2023-12-26 21:00:03.843 INFO  [kea-ctrl-agent.auth/1393724]
>> HTTP_CLIENT_REQUEST_AUTHORIZED received HTTP request authorized for
>> 'baduser'
>> >> > 2023-12-26 21:00:03.843 INFO  [kea-ctrl-agent.commands/1393724]
>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >> > 2023-12-26 21:00:03.843 INFO  [kea-ctrl-agent.ctrl-agent/1393724]
>> CTRL_AGENT_COMMAND_RECEIVED command ha-heartbeat received from remote
>> address xxx.xxx2.xxx.xxx
>> >> > 2023-12-26 21:00:03.844 INFO  [kea-ctrl-agent.ctrl-agent/1393724]
>> CTRL_AGENT_COMMAND_FORWARDED command ha-heartbeat successfully forwarded to
>> the service dhcp4 from remote address  xxx.xxx2.xxx.xxx
>> >> > 2023-12-26 21:00:13.859 INFO  [kea-ctrl-agent.auth/1393724]
>> HTTP_CLIENT_REQUEST_AUTHORIZED received HTTP request authorized for
>> 'baduser'
>> >> > 2023-12-26 21:00:13.859 INFO  [kea-ctrl-agent.commands/1393724]
>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >> > 2023-12-26 21:00:13.859 INFO  [kea-ctrl-agent.ctrl-agent/1393724]
>> CTRL_AGENT_COMMAND_RECEIVED command ha-heartbeat received from remote
>> address  xxx.xxx2.xxx.xxx
>> >> > 2023-12-26 21:00:13.860 INFO  [kea-ctrl-agent.ctrl-agent/1393724]
>> CTRL_AGENT_COMMAND_FORWARDED command ha-heartbeat successfully forwarded to
>> the service dhcp4 from remote address  xxx.xxx2.xxx.xxx
>> >> > $ tail -n10 /var/log/kea/kea-dhcp4.log
>> >> > 2023-12-26 20:58:53.728 INFO  [kea-dhcp4.commands/1393730]
>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >> > 2023-12-26 20:59:03.745 INFO  [kea-dhcp4.commands/1393730]
>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >> > 2023-12-26 20:59:13.762 INFO  [kea-dhcp4.commands/1393730]
>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >> > 2023-12-26 20:59:23.777 INFO  [kea-dhcp4.commands/1393730]
>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >> > 2023-12-26 20:59:33.793 INFO  [kea-dhcp4.commands/1393730]
>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >> > 2023-12-26 20:59:43.811 INFO  [kea-dhcp4.commands/1393730]
>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >> > 2023-12-26 20:59:53.827 INFO  [kea-dhcp4.commands/1393730]
>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >> > 2023-12-26 21:00:03.844 INFO  [kea-dhcp4.commands/1393730]
>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >> > 2023-12-26 21:00:13.859 INFO  [kea-dhcp4.commands/1393730]
>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >> > 2023-12-26 21:00:23.875 INFO  [kea-dhcp4.commands/1393730]
>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >> >
>> >> > And changing the CA or the server HA paramersts to port 8001 without
>> changing the other (and the other server results in "connection refused"
>> logs. It obv wants the CA port to match the HA parameters port despite what
>> we and the documentation suggests...
>> >> >
>> >> > CS, cs.Temp.Mail at gMail.com
>> >> >
>> >> >
>> >> > On Mon, 25 Dec 2023 at 02:45, Darren Ankney <darren.ankney at gmail.com>
>> wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> It actually SHOULDN'T work to set your control agent and
>> >> >> multi-threaded HA listener to the same port as only one of the
>> >> >> applications should be able to setup a listener on that port.
>> Please
>> >> >> describe what you mean by "it doesn't work".  I'm thinking it might
>> be
>> >> >> be best to ask Men & Mice about "micetro" and how best to set things
>> >> >> up there.
>> >> >>
>> >> >> Thank you,
>> >> >>
>> >> >> Darren Ankney
>> >> >>
>> >> >> On Thu, Dec 21, 2023 at 6:47 PM CS <cs.temp.mail at gmail.com> wrote:
>> >> >> >
>> >> >> > Hi all,
>> >> >> > Moving on from my failure to start and logging issues (thank you
>> for your help btw!) I now don't have my heartbeat/control_agent working
>> correctly.
>> >> >> >
>> >> >> > It works fine so long as I set the ports of my control agents and
>> ha hook parameters to be the same (IE 8000 or 8001)
>> >> >> >
>> >> >> > However I am unable to tie the tiny cluster into micetro,
>> probably because the CA port is occupied with HA heartbeats?
>> >> >> >
>> >> >> > Looking to these examples:
>> >> >> >
>> https://github.com/isc-projects/kea/tree/master/doc/examples/template-ha-mt-tls
>> >> >> >
>> >> >> > Documentation points out
>> >> >> >         //This specifies the port CA will listen on.
>> >> >> >         // If enabling HA and multi-threading, the 8000 port is
>> used by the HA
>> >> >> >         // hook library http listener. When using HA hook library
>> with
>> >> >> >         // multi-threading to function, make sure the port used
>> by dedicated
>> >> >> >         // listener is different (e.g. 8001) than the one used by
>> CA. Note
>> >> >> >         // the commands should still be sent via CA. The
>> dedicated listener
>> >> >> >         // is specifically for HA updates only.
>> >> >> >
>> >> >> > However, how to have a dedicated port for HA and a different one
>> for CA escapes me.
>> >> >> >
>> >> >> > CS, cs.Temp.Mail at gMail.com
>> >> >> > --
>> >> >> > ISC funds the development of this software with paid support
>> subscriptions. Contact us at https://www.isc.org/contact/ for more
>> information.
>> >> >> >
>> >> >> > To unsubscribe visit
>> https://lists.isc.org/mailman/listinfo/kea-users.
>> >> >> >
>> >> >> > Kea-users mailing list
>> >> >> > Kea-users at lists.isc.org
>> >> >> > https://lists.isc.org/mailman/listinfo/kea-users
>> >> >> --
>> >> >> ISC funds the development of this software with paid support
>> subscriptions. Contact us at https://www.isc.org/contact/ for more
>> information.
>> >> >>
>> >> >> To unsubscribe visit
>> https://lists.isc.org/mailman/listinfo/kea-users.
>> >> >>
>> >> >> Kea-users mailing list
>> >> >> Kea-users at lists.isc.org
>> >> >> https://lists.isc.org/mailman/listinfo/kea-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20231227/f47afb5b/attachment-0001.htm>


More information about the Kea-users mailing list