[Kea-users] HA heartbeat communications failure
CS
cs.temp.mail at gmail.com
Thu Dec 28 22:04:26 UTC 2023
Well it worked. wtf happened to server2, I don't know and I can't see. I
built both 1 and 2 at the same time and the same way. Let's just say gamma
rays turned it evil and move on.
I splat back my config file and the control agent is running on 8000 and
the ha heartbeat is beating on 8001, as expected.
I also can parse out the issue with micetro, however I am still not certain
of the solution.
The CA on server1 reports: "HTTP_CONNECTION_HANDSHAKE_FAILED TLS handshake
with x.x.x.x failed with wrong version number"
It is certainly outside the scope of this thread and possibly outside the
scope of this mailing list. I'll follow up a new thread to see what info I
can take forward in my project if I must move-on to micetro support.
Thank you all for your help, especially you Mr Ankney.
CS, cs.Temp.Mail at gMail.com
On Thu, 28 Dec 2023 at 11:45, Darren Ankney <darren.ankney at gmail.com> wrote:
> I'll be excited to here the result!
>
> On Thu, Dec 28, 2023 at 2:43 PM CS <cs.temp.mail at gmail.com> wrote:
> >
> > Me too, especially SELinux. But not in this case. apparmor is disabled
> as far as I know.
> >
> > I'm going to clone server1 into a server3 and see if the new server
> binds to the port then and if one and three will HA effectively. Bit of a
> sledgehammer approach but I'm at a loss too.
> >
> > CS, cs.Temp.Mail at gMail.com
> >
> >
> > On Thu, 28 Dec 2023 at 11:36, Darren Ankney <darren.ankney at gmail.com>
> wrote:
> >>
> >> Hello,
> >>
> >> At some point we got off list. Putting us back on the list. Hope
> >> that is OK. Whatever the ultimate resolution is, the information may
> >> help someone in the future.
> >>
> >> I am at a loss. Have you checked for and/or temporarily disabled
> >> appArmor or SELinux to make sure that they are not the problem?
> >> Either of them might filter one application's access to the network
> >> but not another application even on the same network / port
> >> combination. I have bitten more than once by both of them.
> >>
> >> Thank you,
> >>
> >> Darren Ankney
> >>
> >> On Thu, Dec 28, 2023 at 2:28 PM CS <cs.temp.mail at gmail.com> wrote:
> >> >
> >> > Unfortunately just ip address obscured.
> >> > =/
> >> > Thanks
> >> > CS, cs.Temp.Mail at gMail.com
> >> >
> >> >
> >> > On Thu, 28 Dec 2023 at 11:25, Darren Ankney <darren.ankney at gmail.com>
> wrote:
> >> >>
> >> >> Hello,
> >> >>
> >> >> Is the "url" that is obscured a name or IP address?
> >> >>
> >> >> Thank you,
> >> >>
> >> >> Darren Ankney
> >> >>
> >> >> On Thu, Dec 28, 2023 at 2:07 PM CS <cs.temp.mail at gmail.com> wrote:
> >> >>>
> >> >>> At this moment? You wrote it =)
> >> >>>
> >> >>> {
> >> >>> "Dhcp4": {
> >> >>> "hooks-libraries": [
> >> >>> {
> >> >>> "library": "/xxx/libdhcp_lease_cmds.so"
> >> >>> },
> >> >>> {
> >> >>> "library" : "/xxx/libdhcp_ha.so",
> >> >>> "parameters": {
> >> >>> "high-availability": [
> >> >>> {
> >> >>> "this-server-name": "server2",
> >> >>> "mode": "load-balancing",
> >> >>> "auto-failover": true,
> >> >>> "peers": [
> >> >>> {
> >> >>> "name": "server1",
> >> >>> "url": "http://xxx:8001",
> >> >>> "role": "primary"
> >> >>> },
> >> >>> {
> >> >>> "name": "server2",
> >> >>> "url": "http://xxx:8001",
> >> >>> "role": "secondary"
> >> >>> }
> >> >>> ]
> >> >>> }
> >> >>> ]
> >> >>> }
> >> >>> }
> >> >>> ]
> >> >>> }
> >> >>> }
> >> >>>
> >> >>>
> >> >>> CS, cs.Temp.Mail at gMail.com
> >> >>>
> >> >>>
> >> >>> On Thu, 28 Dec 2023 at 10:34, Darren Ankney <
> darren.ankney at gmail.com> wrote:
> >> >>>>
> >> >>>> Hi,
> >> >>>>
> >> >>>> Could you share your hooks configuration from server2?
> >> >>>>
> >> >>>> Thank you,
> >> >>>>
> >> >>>> Darren Ankney
> >> >>>>
> >> >>>> On Thu, Dec 28, 2023 at 1:29 PM CS <cs.temp.mail at gmail.com> wrote:
> >> >>>>>
> >> >>>>> > there could either be a routing problem or a firewall problem
> >> >>>>> Those things it aint. I setup a tiny webserver with openssl on
> port 8001 and I can telnet back and forth to each server just fine.
> >> >>>>>
> >> >>>>> Kea isn't setting up what it is supposed to and isn't logging a
> failure of any sort:
> >> >>>>> server1$ sudo netstat -lnp | grep 8001
> >> >>>>> tcp 0 0 10.111.45.45:8001 0.0.0.0:*
> LISTEN 1534068/kea-dhcp4
> >> >>>>>
> >> >>>>> server2$ sudo netstat -lnp | grep 8001
> >> >>>>> server2:$ systemctl status isc-kea-dhcp4-server.service
> >> >>>>> ● isc-kea-dhcp4-server.service - Kea IPv4 DHCP daemon
> >> >>>>> Loaded: loaded
> (/lib/systemd/system/isc-kea-dhcp4-server.service; enabled; vendor preset:
> enabled)
> >> >>>>> Active: active (running) since Thu 2023-12-28 18:22:55 UTC;
> 3min 24s ago
> >> >>>>> Docs: man:kea-dhcp4(8)
> >> >>>>> Main PID: 1466785 (kea-dhcp4)
> >> >>>>> Tasks: 5 (limit: 19052)
> >> >>>>> Memory: 2.8M
> >> >>>>> CPU: 48ms
> >> >>>>> CGroup: /system.slice/isc-kea-dhcp4-server.service
> >> >>>>> └─1466785 /usr/sbin/kea-dhcp4 -c
> /etc/kea/kea-dhcp4.conf
> >> >>>>> server2$ tail -n10 /var/log/kea/kea-dhcp4.log
> >> >>>>> 2023-12-28 18:22:55.427 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_SERVICE_STARTED started high availability service in load-balancing mode
> as primary server
> >> >>>>> 2023-12-28 18:22:55.427 WARN [kea-dhcp4.dhcp4/1466785]
> DHCP4_MULTI_THREADING_INFO enabled: no, number of threads: 0, queue size: 0
> >> >>>>> 2023-12-28 18:22:55.427 INFO [kea-dhcp4.dhcp4/1466785]
> DHCP4_STARTED Kea DHCPv4 server version 2.2.0 started
> >> >>>>> 2023-12-28 18:23:06.441 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_STATE_TRANSITION server transitions from WAITING to SYNCING state,
> partner state is WAITING
> >> >>>>> 2023-12-28 18:23:06.442 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner
> while in SYNCING state
> >> >>>>> 2023-12-28 18:23:06.442 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_SYNC_START starting lease database synchronization with server1
> >> >>>>> 2023-12-28 18:23:06.444 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 0 leases from server1
> >> >>>>> 2023-12-28 18:23:06.446 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_SYNC_SUCCESSFUL lease database synchronization with server1 completed
> successfully in 3.534 ms
> >> >>>>> 2023-12-28 18:23:06.446 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_STATE_TRANSITION server transitions from SYNCING to READY state, partner
> state is WAITING
> >> >>>>> 2023-12-28 18:23:06.446 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner
> while in READY state
> >> >>>>>
> >> >>>>>
> >> >>>>> On Thu, Dec 28, 2023, 03:12 Darren Ankney <
> darren.ankney at gmail.com> wrote:
> >> >>>>>>
> >> >>>>>> Hi,
> >> >>>>>>
> >> >>>>>> My advice had nothing to do with micetro as I am not familiar
> with it at all. Micetro shouldn't have anything to do with HA
> functionality, however. If one server can reach the other on port 8001 but
> not the reverse, there could either be a routing problem or a firewall
> problem.
> >> >>>>>>
> >> >>>>>> Thank you,
> >> >>>>>>
> >> >>>>>> Darren Ankney
> >> >>>>>>
> >> >>>>>> On Wed, Dec 27, 2023 at 10:50 PM CS <cs.temp.mail at gmail.com>
> wrote:
> >> >>>>>>>
> >> >>>>>>> Thanks for your input Thijs, tho I find your assessment
> confusing.
> >> >>>>>>> It certainly seems to affirm the observed behavior with my full
> configuration but it flies against Darren's advice and my read of:
> >> >>>>>>>
> >> >>>>>>> // If enabling HA and multi-threading, the 8000 port is
> used by the HA
> >> >>>>>>> // hook library http listener. When using HA hook
> library with
> >> >>>>>>> // multi-threading to function, make sure the port used
> by dedicated
> >> >>>>>>> // listener is different (e.g. 8001) than the one used
> by CA. Note
> >> >>>>>>> // the commands should still be sent via CA. The
> dedicated listener
> >> >>>>>>> // is specifically for HA updates only.
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> >Micetro is not using port 8000 :
> https://docs.menandmice.com/en/10.5/guides/implementation/firewall_ports/
> >> >>>>>>> Doesn't this indicate it is? And as far as I can see in Micetro
> this cannot be changed. So at a minimum the CA must be on port 8000
> >> >>>>>>>
> >> >>>>>>> >This part is also important:
> https://docs.menandmice.com/en/10.5/guides/user-manual/dhcp_kea/#add-kea-hooks
> >> >>>>>>> A good point, however I suspect my deployment is failing before
> any problem with this hook library.
> >> >>>>>>>
> >> >>>>>>> CS, cs.Temp.Mail at gMail.com
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> On Wed, 27 Dec 2023 at 15:51, Thijs Blok <blokthijs at gmail.com>
> wrote:
> >> >>>>>>>>
> >> >>>>>>>> Hi,
> >> >>>>>>>>
> >> >>>>>>>> Your control agent needs to run on the same port as the HA
> configuration parameters.
> >> >>>>>>>> Please make sure you don't listen on the localhost only which
> is the default for the control agent.
> >> >>>>>>>>
> >> >>>>>>>> "Control-agent": {
> >> >>>>>>>>
> >> >>>>>>>> "http-host": "x.x.x.x",
> >> >>>>>>>>
> >> >>>>>>>> "http-port": 8000,
> >> >>>>>>>>
> >> >>>>>>>> ....
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> Micetro is not using port 8000 :
> >> >>>>>>>>
> https://docs.menandmice.com/en/10.5/guides/implementation/firewall_ports/
> >> >>>>>>>>
> >> >>>>>>>> This part is also important:
> >> >>>>>>>>
> https://docs.menandmice.com/en/10.5/guides/user-manual/dhcp_kea/#add-kea-hooks
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> Op wo 27 dec 2023 om 23:47 schreef CS <cs.temp.mail at gmail.com
> >:
> >> >>>>>>>>>
> >> >>>>>>>>> telneting around it appears server1 can reach 8001 on itself
> but not 2. server 2 can telnet 8001 on server 1 but not itself.
> >> >>>>>>>>> I see nothing logged that indicates a conflict tho and both
> of these besides being on different subnets these servers are virtually
> identical and firewall free.
> >> >>>>>>>>>
> >> >>>>>>>>> CS, cs.Temp.Mail at gMail.com
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>> On Wed, 27 Dec 2023 at 14:38, CS <cs.temp.mail at gmail.com>
> wrote:
> >> >>>>>>>>>>
> >> >>>>>>>>>> They made it through. With minor changes (ip addresses,
> library locations, and logging) the files are accepted and daemons are
> running but... no dice.
> >> >>>>>>>>>>
> >> >>>>>>>>>> server1$ tail -f /var/log/kea/kea-dhcp4.log
> >> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.dhcpsrv/1495687]
> DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
> /var/lib/kea/kea-leases4.csv
> >> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.dhcpsrv/1495687]
> DHCPSRV_MEMFILE_EXTRACT_EXTENDED_INFO4 extracting extended info saw 0
> leases, extended info sanity checks modified 0 / updated 0 leases and 0
> leases have relay or remote id
> >> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.dhcpsrv/1495687]
> DHCPSRV_MEMFILE_LFC_SETUP setting up the Lease File Cleanup interval to
> 3600 sec
> >> >>>>>>>>>> 2023-12-27 22:24:48.486 WARN [kea-dhcp4.dhcpsrv/1495687]
> DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP traffic
> >> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.ha-hooks/1495687]
> HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the server1 is
> in the WAITING state
> >> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.ha-hooks/1495687]
> HA_SERVICE_STARTED started high availability service in load-balancing mode
> as primary server
> >> >>>>>>>>>> 2023-12-27 22:24:48.487 WARN [kea-dhcp4.dhcp4/1495687]
> DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size:
> 64
> >> >>>>>>>>>> 2023-12-27 22:24:48.487 INFO [kea-dhcp4.dhcp4/1495687]
> DHCP4_STARTED Kea DHCPv4 server version 2.4.0 started
> >> >>>>>>>>>> 2023-12-27 22:24:58.498 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> >> >>>>>>>>>> 2023-12-27 22:25:08.510 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> >> >>>>>>>>>> 2023-12-27 22:25:18.519 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx5:8001): Connection refused
> >> >>>>>>>>>> 2023-12-27 22:25:28.531 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> >> >>>>>>>>>> 2023-12-27 22:25:37.660 INFO [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'ha-heartbeat'
> >> >>>>>>>>>> 2023-12-27 22:25:38.535 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> >> >>>>>>>>>> 2023-12-27 22:25:47.674 INFO [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'ha-heartbeat'
> >> >>>>>>>>>> 2023-12-27 22:25:48.546 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> >> >>>>>>>>>> 2023-12-27 22:25:48.546 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_COMMUNICATION_INTERRUPTED communication with server2 is interrupted
> >> >>>>>>>>>> 2023-12-27 22:25:48.546 INFO [kea-dhcp4.ha-hooks/1495687]
> HA_STATE_TRANSITION server transitions from WAITING to PARTNER-DOWN state,
> partner state is UNDEFINED
> >> >>>>>>>>>> 2023-12-27 22:25:48.547 INFO [kea-dhcp4.ha-hooks/1495687]
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner
> while in PARTNER-DOWN state
> >> >>>>>>>>>> 2023-12-27 22:25:48.547 INFO [kea-dhcp4.ha-hooks/1495687]
> HA_LOCAL_DHCP_ENABLE local DHCP service is enabled while the server1 is in
> the PARTNER-DOWN state
> >> >>>>>>>>>> 2023-12-27 22:25:57.687 INFO [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'ha-heartbeat'
> >> >>>>>>>>>> 2023-12-27 22:25:57.690 INFO [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'dhcp-disable'
> >> >>>>>>>>>> 2023-12-27 22:25:57.691 INFO [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'lease4-get-page'
> >> >>>>>>>>>> 2023-12-27 22:25:57.693 INFO [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'ha-sync-complete-notify'
> >> >>>>>>>>>> 2023-12-27 22:25:58.557 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> >> >>>>>>>>>> 2023-12-27 22:25:58.557 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_COMMUNICATION_INTERRUPTED communication with server2 is interrupted
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> server2$ tail -f /var/log/kea/kea-dhcp4.log
> >> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcp4/1434251]
> DHCP4_CONFIG_COMPLETE DHCPv4 server has completed configuration: no IPv4
> subnets!; DDNS: disabled
> >> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_MEMFILE_DB opening memory file lease database: type=memfile
> universe=4
> >> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
> /var/lib/kea/kea-leases4.csv.2
> >> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
> /var/lib/kea/kea-leases4.csv
> >> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_MEMFILE_LFC_SETUP setting up the Lease File Cleanup interval to
> 3600 sec
> >> >>>>>>>>>> 2023-12-27 22:25:26.650 WARN [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP traffic
> >> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the server2 is
> in the WAITING state
> >> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_SERVICE_STARTED started high availability service in load-balancing mode
> as secondary server
> >> >>>>>>>>>> 2023-12-27 22:25:26.650 WARN [kea-dhcp4.dhcp4/1434251]
> DHCP4_MULTI_THREADING_INFO enabled: no, number of threads: 0, queue size: 0
> >> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcp4/1434251]
> DHCP4_STARTED Kea DHCPv4 server version 2.2.0 started
> >> >>>>>>>>>> 2023-12-27 22:25:57.690 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_STATE_TRANSITION server transitions from WAITING to SYNCING state,
> partner state is PARTNER-DOWN
> >> >>>>>>>>>> 2023-12-27 22:25:57.690 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner
> while in SYNCING state
> >> >>>>>>>>>> 2023-12-27 22:25:57.690 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_SYNC_START starting lease database synchronization with server1
> >> >>>>>>>>>> 2023-12-27 22:25:57.693 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 0 leases from server1
> >> >>>>>>>>>> 2023-12-27 22:25:57.695 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_SYNC_SUCCESSFUL lease database synchronization with server1 completed
> successfully in 3.877 ms
> >> >>>>>>>>>> 2023-12-27 22:25:57.695 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_STATE_TRANSITION server transitions from SYNCING to READY state, partner
> state is PARTNER-DOWN
> >> >>>>>>>>>> 2023-12-27 22:25:57.695 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner
> while in READY state
> >> >>>>>>>>>>
> >> >>>>>>>>>> CS, cs.Temp.Mail at gMail.com
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> On Wed, 27 Dec 2023 at 11:22, Darren Ankney <
> darren.ankney at gmail.com> wrote:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Hi,
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> See attached four files:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> ca-server1.json (config for kea-ctrl-agent)
> >> >>>>>>>>>>> ca-server2.json (config for kea-ctrl-agent)
> >> >>>>>>>>>>> dhcp4-server1.json (config for kea-dhcp4)
> >> >>>>>>>>>>> dhcp4-server2.json (config for kea-dhcp4)
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> These files use port 8000 for kea-ctrl-agent and 8001 for
> kea-dhcp4 on
> >> >>>>>>>>>>> version 2.4.0. They don't do anything other than setup the
> heartbeats
> >> >>>>>>>>>>> for HA (in fact you don't even need to run the control
> agent as Kea is
> >> >>>>>>>>>>> communicating directly). Heartbeats are sent back and
> forth on port
> >> >>>>>>>>>>> 8001 as expected. Can you give these a try and see if they
> work (in
> >> >>>>>>>>>>> testing of course, they won't serve any clients)? I'm
> genuinely
> >> >>>>>>>>>>> curious if they work. Yours should work (unless there is
> some problem
> >> >>>>>>>>>>> with the certificates or something). I didn't notice any
> reason why
> >> >>>>>>>>>>> they wouldn't.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Thank you,
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Darren Ankney
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> PS: I'm not sure if these attachments will make it through
> to the list.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> On Wed, Dec 27, 2023 at 7:16 AM CS <cs.temp.mail at gmail.com>
> wrote:
> >> >>>>>>>>>>> >
> >> >>>>>>>>>>> > Kea 2.4.0
> >> >>>>>>>>>>> >
> >> >>>>>>>>>>> > On Wed, Dec 27, 2023, 03:18 Darren Ankney <
> darren.ankney at gmail.com> wrote:
> >> >>>>>>>>>>> >>
> >> >>>>>>>>>>> >> Hi,
> >> >>>>>>>>>>> >>
> >> >>>>>>>>>>> >> If I may ask, what version of Kea are you using? Some
> defaults have
> >> >>>>>>>>>>> >> changed across versions.
> >> >>>>>>>>>>> >>
> >> >>>>>>>>>>> >> Thank you,
> >> >>>>>>>>>>> >>
> >> >>>>>>>>>>> >> Darren Ankney
> >> >>>>>>>>>>> >>
> >> >>>>>>>>>>> >> On Tue, Dec 26, 2023 at 4:31 PM CS <
> cs.temp.mail at gmail.com> wrote:
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > >Please describe what you mean by "it doesn't work".
> >> >>>>>>>>>>> >> > I mean I get a pretty useless error: "Unable to
> connect to Kea Control Agent."
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > > it might be be best to ask Men & Mice about
> "micetro" and how best to set things
> >> >>>>>>>>>>> >> > I will at some point, when I find a resource with
> them. But there are two players in this and since kea isn't behaving as
> expected like you, I and the docs said. I'm starting here.
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > >It actually SHOULDN'T work
> >> >>>>>>>>>>> >> > That's my read on it too. But here's proof. The CA
> config for one server. It matches for the other server except certs and ip
> addresses obv.
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > "Control-agent": {
> >> >>>>>>>>>>> >> > "http-host": "xxx.xx1.xxx.xxx",
> >> >>>>>>>>>>> >> > "trust-anchor":
> "Certificate_Autority.pem",
> >> >>>>>>>>>>> >> > "cert-file": "ca1_cert.pem",
> >> >>>>>>>>>>> >> > "key-file": "ca1_key.pem",
> >> >>>>>>>>>>> >> > "cert-required": true,
> >> >>>>>>>>>>> >> > "http-port": 8000,
> >> >>>>>>>>>>> >> > "authentication": {
> >> >>>>>>>>>>> >> > "type": "basic",
> >> >>>>>>>>>>> >> > "realm": "kea-control-agent",
> >> >>>>>>>>>>> >> > "clients": [{
> >> >>>>>>>>>>> >> > "user": "baduser",
> >> >>>>>>>>>>> >> > "password":
> "badpassword",
> >> >>>>>>>>>>> >> > }]
> >> >>>>>>>>>>> >> > },
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > And the dhcp4 config, likewise only the small
> differences between the two servers
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > "hooks-libraries": [{
> >> >>>>>>>>>>> >> > "library":
> "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
> >> >>>>>>>>>>> >> > "parameters": {}
> >> >>>>>>>>>>> >> > },{
> >> >>>>>>>>>>> >> > "library" :
> "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
> >> >>>>>>>>>>> >> > "parameters": {
> >> >>>>>>>>>>> >> > "high-availability": [{
> >> >>>>>>>>>>> >> >
> "this-server-name": "server1.org.org",
> >> >>>>>>>>>>> >> > "mode":
> "load-balancing",
> >> >>>>>>>>>>> >> >
> "heartbeat-delay": 10000,
> >> >>>>>>>>>>> >> >
> "max-response-delay": 60000,
> >> >>>>>>>>>>> >> >
> "max-ack-delay": 5000,
> >> >>>>>>>>>>> >> >
> "max-unacked-clients": 0,
> >> >>>>>>>>>>> >> >
> "require-client-certs": true,
> >> >>>>>>>>>>> >> >
> "trust-anchor": "Certificate_Autority.pem",
> >> >>>>>>>>>>> >> >
> "auto-failover": true,
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > "peers": [{
> >> >>>>>>>>>>> >> >
> "name": "server1.org.org",
> >> >>>>>>>>>>> >> > "url":
> "http://xxx.xx1.xxx.xxx:8000/",
> >> >>>>>>>>>>> >> >
> "cert-file": "dhcp1_cert.pem",
> >> >>>>>>>>>>> >> >
> "key-file": "dhcp1_key.pem",
> >> >>>>>>>>>>> >> >
> "basic-auth-user": "baduser",
> >> >>>>>>>>>>> >> >
> "basic-auth-password": "badpassword",
> >> >>>>>>>>>>> >> >
> "role": "primary",
> >> >>>>>>>>>>> >> > },{
> >> >>>>>>>>>>> >> >
> "name": "server2.org.org",
> >> >>>>>>>>>>> >> > "url":
> "http://xxx.xx2.xxx.xxx:8000/",
> >> >>>>>>>>>>> >> >
> "cert-file": "dhcp2_cert.pem",
> >> >>>>>>>>>>> >> >
> "key-file": "dhcp2_key.pem",
> >> >>>>>>>>>>> >> >
> "role": "secondary",
> >> >>>>>>>>>>> >> >
> "basic-auth-user": "baduser",
> >> >>>>>>>>>>> >> >
> "basic-auth-password": "badpassword",
> >> >>>>>>>>>>> >> > }]
> >> >>>>>>>>>>> >> > }]
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > low and behold it runs. The same nature of daemon
> status and logs on the other server.
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > $ sudo systemctl restart isc-kea-ctrl-agent.service
> isc-kea-dhcp4-server.service
> >> >>>>>>>>>>> >> > $ sudo systemctl status isc-kea-ctrl-agent.service
> isc-kea-dhcp4-server.service
> >> >>>>>>>>>>> >> > ● isc-kea-ctrl-agent.service - Kea Control Agent
> >> >>>>>>>>>>> >> > Loaded: loaded
> (/lib/systemd/system/isc-kea-ctrl-agent.service; enabled; vendor preset:
> enabled)
> >> >>>>>>>>>>> >> > Active: active (running) since Tue 2023-12-26
> 20:57:29 UTC; 11s ago
> >> >>>>>>>>>>> >> > Docs: man:kea-ctrl-agent(8)
> >> >>>>>>>>>>> >> > Main PID: 1393724 (kea-ctrl-agent)
> >> >>>>>>>>>>> >> > Tasks: 5 (limit: 19052)
> >> >>>>>>>>>>> >> > Memory: 2.5M
> >> >>>>>>>>>>> >> > CPU: 26ms
> >> >>>>>>>>>>> >> > CGroup: /system.slice/isc-kea-ctrl-agent.service
> >> >>>>>>>>>>> >> > └─1393724 /usr/sbin/kea-ctrl-agent -c
> /etc/kea/kea-ctrl-agent.conf
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]: Started Kea Control
> Agent.
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > ● isc-kea-dhcp4-server.service - Kea DHCPv4 Service
> >> >>>>>>>>>>> >> > Loaded: loaded
> (/lib/systemd/system/isc-kea-dhcp4-server.service; enabled; vendor preset:
> enabled)
> >> >>>>>>>>>>> >> > Active: active (running) since Tue 2023-12-26
> 20:57:29 UTC; 11s ago
> >> >>>>>>>>>>> >> > Docs: man:kea-dhcp4(8)
> >> >>>>>>>>>>> >> > Main PID: 1393730 (kea-dhcp4)
> >> >>>>>>>>>>> >> > Tasks: 9 (limit: 19052)
> >> >>>>>>>>>>> >> > Memory: 4.5M
> >> >>>>>>>>>>> >> > CPU: 96ms
> >> >>>>>>>>>>> >> > CGroup: /system.slice/isc-kea-dhcp4-server.service
> >> >>>>>>>>>>> >> > └─1393730 /usr/sbin/kea-dhcp4 -c
> /etc/kea/kea-dhcp4.conf
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]:
> isc-kea-dhcp4-server.service: Deactivated successfully.
> >> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]: Stopped Kea DHCPv4
> Service.
> >> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]:
> isc-kea-dhcp4-server.service: Consumed 1min 28.504s CPU time.
> >> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]: Started Kea DHCPv4
> Service.
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > $ tail -n10 /var/log/kea/kea-ctrl-agent.log
> >> >>>>>>>>>>> >> > 2023-12-26 20:59:53.827 INFO
> [kea-ctrl-agent.ctrl-agent/1393724] CTRL_AGENT_COMMAND_RECEIVED command
> ha-heartbeat received from remote address xxx.xxx2.xxx.xxx
> >> >>>>>>>>>>> >> > 2023-12-26 20:59:53.828 INFO
> [kea-ctrl-agent.ctrl-agent/1393724] CTRL_AGENT_COMMAND_FORWARDED command
> ha-heartbeat successfully forwarded to the service dhcp4 from remote
> address xxx.xx2.xxx.xxx
> >> >>>>>>>>>>> >> > 2023-12-26 21:00:03.843 INFO
> [kea-ctrl-agent.auth/1393724] HTTP_CLIENT_REQUEST_AUTHORIZED received HTTP
> request authorized for 'baduser'
> >> >>>>>>>>>>> >> > 2023-12-26 21:00:03.843 INFO
> [kea-ctrl-agent.commands/1393724] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >> >>>>>>>>>>> >> > 2023-12-26 21:00:03.843 INFO
> [kea-ctrl-agent.ctrl-agent/1393724] CTRL_AGENT_COMMAND_RECEIVED command
> ha-heartbeat received from remote address xxx.xxx2.xxx.xxx
> >> >>>>>>>>>>> >> > 2023-12-26 21:00:03.844 INFO
> [kea-ctrl-agent.ctrl-agent/1393724] CTRL_AGENT_COMMAND_FORWARDED command
> ha-heartbeat successfully forwarded to the service dhcp4 from remote
> address xxx.xxx2.xxx.xxx
> >> >>>>>>>>>>> >> > 2023-12-26 21:00:13.859 INFO
> [kea-ctrl-agent.auth/1393724] HTTP_CLIENT_REQUEST_AUTHORIZED received HTTP
> request authorized for 'baduser'
> >> >>>>>>>>>>> >> > 2023-12-26 21:00:13.859 INFO
> [kea-ctrl-agent.commands/1393724] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >> >>>>>>>>>>> >> > 2023-12-26 21:00:13.859 INFO
> [kea-ctrl-agent.ctrl-agent/1393724] CTRL_AGENT_COMMAND_RECEIVED command
> ha-heartbeat received from remote address xxx.xxx2.xxx.xxx
> >> >>>>>>>>>>> >> > 2023-12-26 21:00:13.860 INFO
> [kea-ctrl-agent.ctrl-agent/1393724] CTRL_AGENT_COMMAND_FORWARDED command
> ha-heartbeat successfully forwarded to the service dhcp4 from remote
> address xxx.xxx2.xxx.xxx
> >> >>>>>>>>>>> >> > $ tail -n10 /var/log/kea/kea-dhcp4.log
> >> >>>>>>>>>>> >> > 2023-12-26 20:58:53.728 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >> >>>>>>>>>>> >> > 2023-12-26 20:59:03.745 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >> >>>>>>>>>>> >> > 2023-12-26 20:59:13.762 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >> >>>>>>>>>>> >> > 2023-12-26 20:59:23.777 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >> >>>>>>>>>>> >> > 2023-12-26 20:59:33.793 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >> >>>>>>>>>>> >> > 2023-12-26 20:59:43.811 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >> >>>>>>>>>>> >> > 2023-12-26 20:59:53.827 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >> >>>>>>>>>>> >> > 2023-12-26 21:00:03.844 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >> >>>>>>>>>>> >> > 2023-12-26 21:00:13.859 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >> >>>>>>>>>>> >> > 2023-12-26 21:00:23.875 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > And changing the CA or the server HA paramersts to
> port 8001 without changing the other (and the other server results in
> "connection refused" logs. It obv wants the CA port to match the HA
> parameters port despite what we and the documentation suggests...
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > CS, cs.Temp.Mail at gMail.com
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> >
> >> >>>>>>>>>>> >> > On Mon, 25 Dec 2023 at 02:45, Darren Ankney <
> darren.ankney at gmail.com> wrote:
> >> >>>>>>>>>>> >> >>
> >> >>>>>>>>>>> >> >> Hi,
> >> >>>>>>>>>>> >> >>
> >> >>>>>>>>>>> >> >> It actually SHOULDN'T work to set your control agent
> and
> >> >>>>>>>>>>> >> >> multi-threaded HA listener to the same port as only
> one of the
> >> >>>>>>>>>>> >> >> applications should be able to setup a listener on
> that port. Please
> >> >>>>>>>>>>> >> >> describe what you mean by "it doesn't work". I'm
> thinking it might be
> >> >>>>>>>>>>> >> >> be best to ask Men & Mice about "micetro" and how
> best to set things
> >> >>>>>>>>>>> >> >> up there.
> >> >>>>>>>>>>> >> >>
> >> >>>>>>>>>>> >> >> Thank you,
> >> >>>>>>>>>>> >> >>
> >> >>>>>>>>>>> >> >> Darren Ankney
> >> >>>>>>>>>>> >> >>
> >> >>>>>>>>>>> >> >> On Thu, Dec 21, 2023 at 6:47 PM CS <
> cs.temp.mail at gmail.com> wrote:
> >> >>>>>>>>>>> >> >> >
> >> >>>>>>>>>>> >> >> > Hi all,
> >> >>>>>>>>>>> >> >> > Moving on from my failure to start and logging
> issues (thank you for your help btw!) I now don't have my
> heartbeat/control_agent working correctly.
> >> >>>>>>>>>>> >> >> >
> >> >>>>>>>>>>> >> >> > It works fine so long as I set the ports of my
> control agents and ha hook parameters to be the same (IE 8000 or 8001)
> >> >>>>>>>>>>> >> >> >
> >> >>>>>>>>>>> >> >> > However I am unable to tie the tiny cluster into
> micetro, probably because the CA port is occupied with HA heartbeats?
> >> >>>>>>>>>>> >> >> >
> >> >>>>>>>>>>> >> >> > Looking to these examples:
> >> >>>>>>>>>>> >> >> >
> https://github.com/isc-projects/kea/tree/master/doc/examples/template-ha-mt-tls
> >> >>>>>>>>>>> >> >> >
> >> >>>>>>>>>>> >> >> > Documentation points out
> >> >>>>>>>>>>> >> >> > //This specifies the port CA will listen on.
> >> >>>>>>>>>>> >> >> > // If enabling HA and multi-threading, the
> 8000 port is used by the HA
> >> >>>>>>>>>>> >> >> > // hook library http listener. When using
> HA hook library with
> >> >>>>>>>>>>> >> >> > // multi-threading to function, make sure
> the port used by dedicated
> >> >>>>>>>>>>> >> >> > // listener is different (e.g. 8001) than
> the one used by CA. Note
> >> >>>>>>>>>>> >> >> > // the commands should still be sent via
> CA. The dedicated listener
> >> >>>>>>>>>>> >> >> > // is specifically for HA updates only.
> >> >>>>>>>>>>> >> >> >
> >> >>>>>>>>>>> >> >> > However, how to have a dedicated port for HA and a
> different one for CA escapes me.
> >> >>>>>>>>>>> >> >> >
> >> >>>>>>>>>>> >> >> > CS, cs.Temp.Mail at gMail.com
> >> >>>>>>>>>>> >> >> > --
> >> >>>>>>>>>>> >> >> > ISC funds the development of this software with
> paid support subscriptions. Contact us at https://www.isc.org/contact/
> for more information.
> >> >>>>>>>>>>> >> >> >
> >> >>>>>>>>>>> >> >> > To unsubscribe visit
> https://lists.isc.org/mailman/listinfo/kea-users.
> >> >>>>>>>>>>> >> >> >
> >> >>>>>>>>>>> >> >> > Kea-users mailing list
> >> >>>>>>>>>>> >> >> > Kea-users at lists.isc.org
> >> >>>>>>>>>>> >> >> > https://lists.isc.org/mailman/listinfo/kea-users
> >> >>>>>>>>>>> >> >> --
> >> >>>>>>>>>>> >> >> ISC funds the development of this software with paid
> support subscriptions. Contact us at https://www.isc.org/contact/ for
> more information.
> >> >>>>>>>>>>> >> >>
> >> >>>>>>>>>>> >> >> To unsubscribe visit
> https://lists.isc.org/mailman/listinfo/kea-users.
> >> >>>>>>>>>>> >> >>
> >> >>>>>>>>>>> >> >> Kea-users mailing list
> >> >>>>>>>>>>> >> >> Kea-users at lists.isc.org
> >> >>>>>>>>>>> >> >> https://lists.isc.org/mailman/listinfo/kea-users
> >> >>>>>>>>>
> >> >>>>>>>>> --
> >> >>>>>>>>> ISC funds the development of this software with paid support
> subscriptions. Contact us at https://www.isc.org/contact/ for more
> information.
> >> >>>>>>>>>
> >> >>>>>>>>> To unsubscribe visit
> https://lists.isc.org/mailman/listinfo/kea-users.
> >> >>>>>>>>>
> >> >>>>>>>>> Kea-users mailing list
> >> >>>>>>>>> Kea-users at lists.isc.org
> >> >>>>>>>>> https://lists.isc.org/mailman/listinfo/kea-users
> >> >>>>>>
> >> >>>>>> --
> >> >>>>>> ISC funds the development of this software with paid support
> subscriptions. Contact us at https://www.isc.org/contact/ for more
> information.
> >> >>>>>>
> >> >>>>>> To unsubscribe visit
> https://lists.isc.org/mailman/listinfo/kea-users.
> >> >>>>>>
> >> >>>>>> Kea-users mailing list
> >> >>>>>> Kea-users at lists.isc.org
> >> >>>>>> https://lists.isc.org/mailman/listinfo/kea-users
> --
> ISC funds the development of this software with paid support
> subscriptions. Contact us at https://www.isc.org/contact/ for more
> information.
>
> To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
>
> Kea-users mailing list
> Kea-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/kea-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20231228/c33e09bc/attachment-0001.htm>
More information about the Kea-users
mailing list