[Kea-users] HA heartbeat communications failure
CS
cs.temp.mail at gmail.com
Thu Dec 28 19:45:31 UTC 2023
Me too, especially SELinux. But not in this case. apparmor is disabled as
far as I know.
I'm going to clone server1 into a server3 and see if the new server binds
to the port then and if one and three will HA effectively. Bit of a
sledgehammer approach but I'm at a loss too.
CS, cs.Temp.Mail at gMail.com
On Thu, 28 Dec 2023 at 11:36, Darren Ankney <darren.ankney at gmail.com> wrote:
> Hello,
>
> At some point we got off list. Putting us back on the list. Hope
> that is OK. Whatever the ultimate resolution is, the information may
> help someone in the future.
>
> I am at a loss. Have you checked for and/or temporarily disabled
> appArmor or SELinux to make sure that they are not the problem?
> Either of them might filter one application's access to the network
> but not another application even on the same network / port
> combination. I have bitten more than once by both of them.
>
> Thank you,
>
> Darren Ankney
>
> On Thu, Dec 28, 2023 at 2:28 PM CS <cs.temp.mail at gmail.com> wrote:
> >
> > Unfortunately just ip address obscured.
> > =/
> > Thanks
> > CS, cs.Temp.Mail at gMail.com
> >
> >
> > On Thu, 28 Dec 2023 at 11:25, Darren Ankney <darren.ankney at gmail.com>
> wrote:
> >>
> >> Hello,
> >>
> >> Is the "url" that is obscured a name or IP address?
> >>
> >> Thank you,
> >>
> >> Darren Ankney
> >>
> >> On Thu, Dec 28, 2023 at 2:07 PM CS <cs.temp.mail at gmail.com> wrote:
> >>>
> >>> At this moment? You wrote it =)
> >>>
> >>> {
> >>> "Dhcp4": {
> >>> "hooks-libraries": [
> >>> {
> >>> "library": "/xxx/libdhcp_lease_cmds.so"
> >>> },
> >>> {
> >>> "library" : "/xxx/libdhcp_ha.so",
> >>> "parameters": {
> >>> "high-availability": [
> >>> {
> >>> "this-server-name": "server2",
> >>> "mode": "load-balancing",
> >>> "auto-failover": true,
> >>> "peers": [
> >>> {
> >>> "name": "server1",
> >>> "url": "http://xxx:8001",
> >>> "role": "primary"
> >>> },
> >>> {
> >>> "name": "server2",
> >>> "url": "http://xxx:8001",
> >>> "role": "secondary"
> >>> }
> >>> ]
> >>> }
> >>> ]
> >>> }
> >>> }
> >>> ]
> >>> }
> >>> }
> >>>
> >>>
> >>> CS, cs.Temp.Mail at gMail.com
> >>>
> >>>
> >>> On Thu, 28 Dec 2023 at 10:34, Darren Ankney <darren.ankney at gmail.com>
> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> Could you share your hooks configuration from server2?
> >>>>
> >>>> Thank you,
> >>>>
> >>>> Darren Ankney
> >>>>
> >>>> On Thu, Dec 28, 2023 at 1:29 PM CS <cs.temp.mail at gmail.com> wrote:
> >>>>>
> >>>>> > there could either be a routing problem or a firewall problem
> >>>>> Those things it aint. I setup a tiny webserver with openssl on port
> 8001 and I can telnet back and forth to each server just fine.
> >>>>>
> >>>>> Kea isn't setting up what it is supposed to and isn't logging a
> failure of any sort:
> >>>>> server1$ sudo netstat -lnp | grep 8001
> >>>>> tcp 0 0 10.111.45.45:8001 0.0.0.0:*
> LISTEN 1534068/kea-dhcp4
> >>>>>
> >>>>> server2$ sudo netstat -lnp | grep 8001
> >>>>> server2:$ systemctl status isc-kea-dhcp4-server.service
> >>>>> ● isc-kea-dhcp4-server.service - Kea IPv4 DHCP daemon
> >>>>> Loaded: loaded
> (/lib/systemd/system/isc-kea-dhcp4-server.service; enabled; vendor preset:
> enabled)
> >>>>> Active: active (running) since Thu 2023-12-28 18:22:55 UTC;
> 3min 24s ago
> >>>>> Docs: man:kea-dhcp4(8)
> >>>>> Main PID: 1466785 (kea-dhcp4)
> >>>>> Tasks: 5 (limit: 19052)
> >>>>> Memory: 2.8M
> >>>>> CPU: 48ms
> >>>>> CGroup: /system.slice/isc-kea-dhcp4-server.service
> >>>>> └─1466785 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
> >>>>> server2$ tail -n10 /var/log/kea/kea-dhcp4.log
> >>>>> 2023-12-28 18:22:55.427 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_SERVICE_STARTED started high availability service in load-balancing mode
> as primary server
> >>>>> 2023-12-28 18:22:55.427 WARN [kea-dhcp4.dhcp4/1466785]
> DHCP4_MULTI_THREADING_INFO enabled: no, number of threads: 0, queue size: 0
> >>>>> 2023-12-28 18:22:55.427 INFO [kea-dhcp4.dhcp4/1466785]
> DHCP4_STARTED Kea DHCPv4 server version 2.2.0 started
> >>>>> 2023-12-28 18:23:06.441 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_STATE_TRANSITION server transitions from WAITING to SYNCING state,
> partner state is WAITING
> >>>>> 2023-12-28 18:23:06.442 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner
> while in SYNCING state
> >>>>> 2023-12-28 18:23:06.442 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_SYNC_START starting lease database synchronization with server1
> >>>>> 2023-12-28 18:23:06.444 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 0 leases from server1
> >>>>> 2023-12-28 18:23:06.446 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_SYNC_SUCCESSFUL lease database synchronization with server1 completed
> successfully in 3.534 ms
> >>>>> 2023-12-28 18:23:06.446 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_STATE_TRANSITION server transitions from SYNCING to READY state, partner
> state is WAITING
> >>>>> 2023-12-28 18:23:06.446 INFO [kea-dhcp4.ha-hooks/1466785]
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner
> while in READY state
> >>>>>
> >>>>>
> >>>>> On Thu, Dec 28, 2023, 03:12 Darren Ankney <darren.ankney at gmail.com>
> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> My advice had nothing to do with micetro as I am not familiar with
> it at all. Micetro shouldn't have anything to do with HA functionality,
> however. If one server can reach the other on port 8001 but not the
> reverse, there could either be a routing problem or a firewall problem.
> >>>>>>
> >>>>>> Thank you,
> >>>>>>
> >>>>>> Darren Ankney
> >>>>>>
> >>>>>> On Wed, Dec 27, 2023 at 10:50 PM CS <cs.temp.mail at gmail.com> wrote:
> >>>>>>>
> >>>>>>> Thanks for your input Thijs, tho I find your assessment confusing.
> >>>>>>> It certainly seems to affirm the observed behavior with my full
> configuration but it flies against Darren's advice and my read of:
> >>>>>>>
> >>>>>>> // If enabling HA and multi-threading, the 8000 port is
> used by the HA
> >>>>>>> // hook library http listener. When using HA hook library
> with
> >>>>>>> // multi-threading to function, make sure the port used by
> dedicated
> >>>>>>> // listener is different (e.g. 8001) than the one used by
> CA. Note
> >>>>>>> // the commands should still be sent via CA. The dedicated
> listener
> >>>>>>> // is specifically for HA updates only.
> >>>>>>>
> >>>>>>>
> >>>>>>> >Micetro is not using port 8000 :
> https://docs.menandmice.com/en/10.5/guides/implementation/firewall_ports/
> >>>>>>> Doesn't this indicate it is? And as far as I can see in Micetro
> this cannot be changed. So at a minimum the CA must be on port 8000
> >>>>>>>
> >>>>>>> >This part is also important:
> https://docs.menandmice.com/en/10.5/guides/user-manual/dhcp_kea/#add-kea-hooks
> >>>>>>> A good point, however I suspect my deployment is failing before
> any problem with this hook library.
> >>>>>>>
> >>>>>>> CS, cs.Temp.Mail at gMail.com
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, 27 Dec 2023 at 15:51, Thijs Blok <blokthijs at gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> Your control agent needs to run on the same port as the HA
> configuration parameters.
> >>>>>>>> Please make sure you don't listen on the localhost only which is
> the default for the control agent.
> >>>>>>>>
> >>>>>>>> "Control-agent": {
> >>>>>>>>
> >>>>>>>> "http-host": "x.x.x.x",
> >>>>>>>>
> >>>>>>>> "http-port": 8000,
> >>>>>>>>
> >>>>>>>> ....
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Micetro is not using port 8000 :
> >>>>>>>>
> https://docs.menandmice.com/en/10.5/guides/implementation/firewall_ports/
> >>>>>>>>
> >>>>>>>> This part is also important:
> >>>>>>>>
> https://docs.menandmice.com/en/10.5/guides/user-manual/dhcp_kea/#add-kea-hooks
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Op wo 27 dec 2023 om 23:47 schreef CS <cs.temp.mail at gmail.com>:
> >>>>>>>>>
> >>>>>>>>> telneting around it appears server1 can reach 8001 on itself but
> not 2. server 2 can telnet 8001 on server 1 but not itself.
> >>>>>>>>> I see nothing logged that indicates a conflict tho and both of
> these besides being on different subnets these servers are virtually
> identical and firewall free.
> >>>>>>>>>
> >>>>>>>>> CS, cs.Temp.Mail at gMail.com
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, 27 Dec 2023 at 14:38, CS <cs.temp.mail at gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> They made it through. With minor changes (ip addresses, library
> locations, and logging) the files are accepted and daemons are running
> but... no dice.
> >>>>>>>>>>
> >>>>>>>>>> server1$ tail -f /var/log/kea/kea-dhcp4.log
> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.dhcpsrv/1495687]
> DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
> /var/lib/kea/kea-leases4.csv
> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.dhcpsrv/1495687]
> DHCPSRV_MEMFILE_EXTRACT_EXTENDED_INFO4 extracting extended info saw 0
> leases, extended info sanity checks modified 0 / updated 0 leases and 0
> leases have relay or remote id
> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.dhcpsrv/1495687]
> DHCPSRV_MEMFILE_LFC_SETUP setting up the Lease File Cleanup interval to
> 3600 sec
> >>>>>>>>>> 2023-12-27 22:24:48.486 WARN [kea-dhcp4.dhcpsrv/1495687]
> DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP traffic
> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.ha-hooks/1495687]
> HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the server1 is
> in the WAITING state
> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.ha-hooks/1495687]
> HA_SERVICE_STARTED started high availability service in load-balancing mode
> as primary server
> >>>>>>>>>> 2023-12-27 22:24:48.487 WARN [kea-dhcp4.dhcp4/1495687]
> DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 2, queue size:
> 64
> >>>>>>>>>> 2023-12-27 22:24:48.487 INFO [kea-dhcp4.dhcp4/1495687]
> DHCP4_STARTED Kea DHCPv4 server version 2.4.0 started
> >>>>>>>>>> 2023-12-27 22:24:58.498 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> >>>>>>>>>> 2023-12-27 22:25:08.510 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> >>>>>>>>>> 2023-12-27 22:25:18.519 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx5:8001): Connection refused
> >>>>>>>>>> 2023-12-27 22:25:28.531 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> >>>>>>>>>> 2023-12-27 22:25:37.660 INFO [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'ha-heartbeat'
> >>>>>>>>>> 2023-12-27 22:25:38.535 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> >>>>>>>>>> 2023-12-27 22:25:47.674 INFO [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'ha-heartbeat'
> >>>>>>>>>> 2023-12-27 22:25:48.546 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> >>>>>>>>>> 2023-12-27 22:25:48.546 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_COMMUNICATION_INTERRUPTED communication with server2 is interrupted
> >>>>>>>>>> 2023-12-27 22:25:48.546 INFO [kea-dhcp4.ha-hooks/1495687]
> HA_STATE_TRANSITION server transitions from WAITING to PARTNER-DOWN state,
> partner state is UNDEFINED
> >>>>>>>>>> 2023-12-27 22:25:48.547 INFO [kea-dhcp4.ha-hooks/1495687]
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner
> while in PARTNER-DOWN state
> >>>>>>>>>> 2023-12-27 22:25:48.547 INFO [kea-dhcp4.ha-hooks/1495687]
> HA_LOCAL_DHCP_ENABLE local DHCP service is enabled while the server1 is in
> the PARTNER-DOWN state
> >>>>>>>>>> 2023-12-27 22:25:57.687 INFO [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'ha-heartbeat'
> >>>>>>>>>> 2023-12-27 22:25:57.690 INFO [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'dhcp-disable'
> >>>>>>>>>> 2023-12-27 22:25:57.691 INFO [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'lease4-get-page'
> >>>>>>>>>> 2023-12-27 22:25:57.693 INFO [kea-dhcp4.commands/1495687]
> COMMAND_RECEIVED Received command 'ha-sync-complete-notify'
> >>>>>>>>>> 2023-12-27 22:25:58.557 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to server2 (
> http://xxx:8001): Connection refused
> >>>>>>>>>> 2023-12-27 22:25:58.557 WARN [kea-dhcp4.ha-hooks/1495687]
> HA_COMMUNICATION_INTERRUPTED communication with server2 is interrupted
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> server2$ tail -f /var/log/kea/kea-dhcp4.log
> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcp4/1434251]
> DHCP4_CONFIG_COMPLETE DHCPv4 server has completed configuration: no IPv4
> subnets!; DDNS: disabled
> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_MEMFILE_DB opening memory file lease database: type=memfile
> universe=4
> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
> /var/lib/kea/kea-leases4.csv.2
> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
> /var/lib/kea/kea-leases4.csv
> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_MEMFILE_LFC_SETUP setting up the Lease File Cleanup interval to
> 3600 sec
> >>>>>>>>>> 2023-12-27 22:25:26.650 WARN [kea-dhcp4.dhcpsrv/1434251]
> DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP traffic
> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the server2 is
> in the WAITING state
> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_SERVICE_STARTED started high availability service in load-balancing mode
> as secondary server
> >>>>>>>>>> 2023-12-27 22:25:26.650 WARN [kea-dhcp4.dhcp4/1434251]
> DHCP4_MULTI_THREADING_INFO enabled: no, number of threads: 0, queue size: 0
> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcp4/1434251]
> DHCP4_STARTED Kea DHCPv4 server version 2.2.0 started
> >>>>>>>>>> 2023-12-27 22:25:57.690 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_STATE_TRANSITION server transitions from WAITING to SYNCING state,
> partner state is PARTNER-DOWN
> >>>>>>>>>> 2023-12-27 22:25:57.690 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner
> while in SYNCING state
> >>>>>>>>>> 2023-12-27 22:25:57.690 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_SYNC_START starting lease database synchronization with server1
> >>>>>>>>>> 2023-12-27 22:25:57.693 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 0 leases from server1
> >>>>>>>>>> 2023-12-27 22:25:57.695 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_SYNC_SUCCESSFUL lease database synchronization with server1 completed
> successfully in 3.877 ms
> >>>>>>>>>> 2023-12-27 22:25:57.695 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_STATE_TRANSITION server transitions from SYNCING to READY state, partner
> state is PARTNER-DOWN
> >>>>>>>>>> 2023-12-27 22:25:57.695 INFO [kea-dhcp4.ha-hooks/1434251]
> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner
> while in READY state
> >>>>>>>>>>
> >>>>>>>>>> CS, cs.Temp.Mail at gMail.com
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Wed, 27 Dec 2023 at 11:22, Darren Ankney <
> darren.ankney at gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> See attached four files:
> >>>>>>>>>>>
> >>>>>>>>>>> ca-server1.json (config for kea-ctrl-agent)
> >>>>>>>>>>> ca-server2.json (config for kea-ctrl-agent)
> >>>>>>>>>>> dhcp4-server1.json (config for kea-dhcp4)
> >>>>>>>>>>> dhcp4-server2.json (config for kea-dhcp4)
> >>>>>>>>>>>
> >>>>>>>>>>> These files use port 8000 for kea-ctrl-agent and 8001 for
> kea-dhcp4 on
> >>>>>>>>>>> version 2.4.0. They don't do anything other than setup the
> heartbeats
> >>>>>>>>>>> for HA (in fact you don't even need to run the control agent
> as Kea is
> >>>>>>>>>>> communicating directly). Heartbeats are sent back and forth
> on port
> >>>>>>>>>>> 8001 as expected. Can you give these a try and see if they
> work (in
> >>>>>>>>>>> testing of course, they won't serve any clients)? I'm
> genuinely
> >>>>>>>>>>> curious if they work. Yours should work (unless there is some
> problem
> >>>>>>>>>>> with the certificates or something). I didn't notice any
> reason why
> >>>>>>>>>>> they wouldn't.
> >>>>>>>>>>>
> >>>>>>>>>>> Thank you,
> >>>>>>>>>>>
> >>>>>>>>>>> Darren Ankney
> >>>>>>>>>>>
> >>>>>>>>>>> PS: I'm not sure if these attachments will make it through to
> the list.
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Dec 27, 2023 at 7:16 AM CS <cs.temp.mail at gmail.com>
> wrote:
> >>>>>>>>>>> >
> >>>>>>>>>>> > Kea 2.4.0
> >>>>>>>>>>> >
> >>>>>>>>>>> > On Wed, Dec 27, 2023, 03:18 Darren Ankney <
> darren.ankney at gmail.com> wrote:
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> Hi,
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> If I may ask, what version of Kea are you using? Some
> defaults have
> >>>>>>>>>>> >> changed across versions.
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> Thank you,
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> Darren Ankney
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> On Tue, Dec 26, 2023 at 4:31 PM CS <cs.temp.mail at gmail.com>
> wrote:
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > >Please describe what you mean by "it doesn't work".
> >>>>>>>>>>> >> > I mean I get a pretty useless error: "Unable to connect
> to Kea Control Agent."
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > > it might be be best to ask Men & Mice about "micetro"
> and how best to set things
> >>>>>>>>>>> >> > I will at some point, when I find a resource with them.
> But there are two players in this and since kea isn't behaving as expected
> like you, I and the docs said. I'm starting here.
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > >It actually SHOULDN'T work
> >>>>>>>>>>> >> > That's my read on it too. But here's proof. The CA config
> for one server. It matches for the other server except certs and ip
> addresses obv.
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > "Control-agent": {
> >>>>>>>>>>> >> > "http-host": "xxx.xx1.xxx.xxx",
> >>>>>>>>>>> >> > "trust-anchor":
> "Certificate_Autority.pem",
> >>>>>>>>>>> >> > "cert-file": "ca1_cert.pem",
> >>>>>>>>>>> >> > "key-file": "ca1_key.pem",
> >>>>>>>>>>> >> > "cert-required": true,
> >>>>>>>>>>> >> > "http-port": 8000,
> >>>>>>>>>>> >> > "authentication": {
> >>>>>>>>>>> >> > "type": "basic",
> >>>>>>>>>>> >> > "realm": "kea-control-agent",
> >>>>>>>>>>> >> > "clients": [{
> >>>>>>>>>>> >> > "user": "baduser",
> >>>>>>>>>>> >> > "password": "badpassword",
> >>>>>>>>>>> >> > }]
> >>>>>>>>>>> >> > },
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > And the dhcp4 config, likewise only the small differences
> between the two servers
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > "hooks-libraries": [{
> >>>>>>>>>>> >> > "library":
> "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
> >>>>>>>>>>> >> > "parameters": {}
> >>>>>>>>>>> >> > },{
> >>>>>>>>>>> >> > "library" :
> "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
> >>>>>>>>>>> >> > "parameters": {
> >>>>>>>>>>> >> > "high-availability": [{
> >>>>>>>>>>> >> >
> "this-server-name": "server1.org.org",
> >>>>>>>>>>> >> > "mode":
> "load-balancing",
> >>>>>>>>>>> >> >
> "heartbeat-delay": 10000,
> >>>>>>>>>>> >> >
> "max-response-delay": 60000,
> >>>>>>>>>>> >> > "max-ack-delay":
> 5000,
> >>>>>>>>>>> >> >
> "max-unacked-clients": 0,
> >>>>>>>>>>> >> >
> "require-client-certs": true,
> >>>>>>>>>>> >> > "trust-anchor":
> "Certificate_Autority.pem",
> >>>>>>>>>>> >> > "auto-failover":
> true,
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > "peers": [{
> >>>>>>>>>>> >> > "name": "
> server1.org.org",
> >>>>>>>>>>> >> > "url": "
> http://xxx.xx1.xxx.xxx:8000/",
> >>>>>>>>>>> >> >
> "cert-file": "dhcp1_cert.pem",
> >>>>>>>>>>> >> >
> "key-file": "dhcp1_key.pem",
> >>>>>>>>>>> >> >
> "basic-auth-user": "baduser",
> >>>>>>>>>>> >> >
> "basic-auth-password": "badpassword",
> >>>>>>>>>>> >> > "role":
> "primary",
> >>>>>>>>>>> >> > },{
> >>>>>>>>>>> >> > "name": "
> server2.org.org",
> >>>>>>>>>>> >> > "url": "
> http://xxx.xx2.xxx.xxx:8000/",
> >>>>>>>>>>> >> >
> "cert-file": "dhcp2_cert.pem",
> >>>>>>>>>>> >> >
> "key-file": "dhcp2_key.pem",
> >>>>>>>>>>> >> > "role":
> "secondary",
> >>>>>>>>>>> >> >
> "basic-auth-user": "baduser",
> >>>>>>>>>>> >> >
> "basic-auth-password": "badpassword",
> >>>>>>>>>>> >> > }]
> >>>>>>>>>>> >> > }]
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > low and behold it runs. The same nature of daemon status
> and logs on the other server.
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > $ sudo systemctl restart isc-kea-ctrl-agent.service
> isc-kea-dhcp4-server.service
> >>>>>>>>>>> >> > $ sudo systemctl status isc-kea-ctrl-agent.service
> isc-kea-dhcp4-server.service
> >>>>>>>>>>> >> > ● isc-kea-ctrl-agent.service - Kea Control Agent
> >>>>>>>>>>> >> > Loaded: loaded
> (/lib/systemd/system/isc-kea-ctrl-agent.service; enabled; vendor preset:
> enabled)
> >>>>>>>>>>> >> > Active: active (running) since Tue 2023-12-26
> 20:57:29 UTC; 11s ago
> >>>>>>>>>>> >> > Docs: man:kea-ctrl-agent(8)
> >>>>>>>>>>> >> > Main PID: 1393724 (kea-ctrl-agent)
> >>>>>>>>>>> >> > Tasks: 5 (limit: 19052)
> >>>>>>>>>>> >> > Memory: 2.5M
> >>>>>>>>>>> >> > CPU: 26ms
> >>>>>>>>>>> >> > CGroup: /system.slice/isc-kea-ctrl-agent.service
> >>>>>>>>>>> >> > └─1393724 /usr/sbin/kea-ctrl-agent -c
> /etc/kea/kea-ctrl-agent.conf
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]: Started Kea Control
> Agent.
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > ● isc-kea-dhcp4-server.service - Kea DHCPv4 Service
> >>>>>>>>>>> >> > Loaded: loaded
> (/lib/systemd/system/isc-kea-dhcp4-server.service; enabled; vendor preset:
> enabled)
> >>>>>>>>>>> >> > Active: active (running) since Tue 2023-12-26
> 20:57:29 UTC; 11s ago
> >>>>>>>>>>> >> > Docs: man:kea-dhcp4(8)
> >>>>>>>>>>> >> > Main PID: 1393730 (kea-dhcp4)
> >>>>>>>>>>> >> > Tasks: 9 (limit: 19052)
> >>>>>>>>>>> >> > Memory: 4.5M
> >>>>>>>>>>> >> > CPU: 96ms
> >>>>>>>>>>> >> > CGroup: /system.slice/isc-kea-dhcp4-server.service
> >>>>>>>>>>> >> > └─1393730 /usr/sbin/kea-dhcp4 -c
> /etc/kea/kea-dhcp4.conf
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]:
> isc-kea-dhcp4-server.service: Deactivated successfully.
> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]: Stopped Kea DHCPv4
> Service.
> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]:
> isc-kea-dhcp4-server.service: Consumed 1min 28.504s CPU time.
> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]: Started Kea DHCPv4
> Service.
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > $ tail -n10 /var/log/kea/kea-ctrl-agent.log
> >>>>>>>>>>> >> > 2023-12-26 20:59:53.827 INFO
> [kea-ctrl-agent.ctrl-agent/1393724] CTRL_AGENT_COMMAND_RECEIVED command
> ha-heartbeat received from remote address xxx.xxx2.xxx.xxx
> >>>>>>>>>>> >> > 2023-12-26 20:59:53.828 INFO
> [kea-ctrl-agent.ctrl-agent/1393724] CTRL_AGENT_COMMAND_FORWARDED command
> ha-heartbeat successfully forwarded to the service dhcp4 from remote
> address xxx.xx2.xxx.xxx
> >>>>>>>>>>> >> > 2023-12-26 21:00:03.843 INFO
> [kea-ctrl-agent.auth/1393724] HTTP_CLIENT_REQUEST_AUTHORIZED received HTTP
> request authorized for 'baduser'
> >>>>>>>>>>> >> > 2023-12-26 21:00:03.843 INFO
> [kea-ctrl-agent.commands/1393724] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >>>>>>>>>>> >> > 2023-12-26 21:00:03.843 INFO
> [kea-ctrl-agent.ctrl-agent/1393724] CTRL_AGENT_COMMAND_RECEIVED command
> ha-heartbeat received from remote address xxx.xxx2.xxx.xxx
> >>>>>>>>>>> >> > 2023-12-26 21:00:03.844 INFO
> [kea-ctrl-agent.ctrl-agent/1393724] CTRL_AGENT_COMMAND_FORWARDED command
> ha-heartbeat successfully forwarded to the service dhcp4 from remote
> address xxx.xxx2.xxx.xxx
> >>>>>>>>>>> >> > 2023-12-26 21:00:13.859 INFO
> [kea-ctrl-agent.auth/1393724] HTTP_CLIENT_REQUEST_AUTHORIZED received HTTP
> request authorized for 'baduser'
> >>>>>>>>>>> >> > 2023-12-26 21:00:13.859 INFO
> [kea-ctrl-agent.commands/1393724] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >>>>>>>>>>> >> > 2023-12-26 21:00:13.859 INFO
> [kea-ctrl-agent.ctrl-agent/1393724] CTRL_AGENT_COMMAND_RECEIVED command
> ha-heartbeat received from remote address xxx.xxx2.xxx.xxx
> >>>>>>>>>>> >> > 2023-12-26 21:00:13.860 INFO
> [kea-ctrl-agent.ctrl-agent/1393724] CTRL_AGENT_COMMAND_FORWARDED command
> ha-heartbeat successfully forwarded to the service dhcp4 from remote
> address xxx.xxx2.xxx.xxx
> >>>>>>>>>>> >> > $ tail -n10 /var/log/kea/kea-dhcp4.log
> >>>>>>>>>>> >> > 2023-12-26 20:58:53.728 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >>>>>>>>>>> >> > 2023-12-26 20:59:03.745 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >>>>>>>>>>> >> > 2023-12-26 20:59:13.762 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >>>>>>>>>>> >> > 2023-12-26 20:59:23.777 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >>>>>>>>>>> >> > 2023-12-26 20:59:33.793 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >>>>>>>>>>> >> > 2023-12-26 20:59:43.811 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >>>>>>>>>>> >> > 2023-12-26 20:59:53.827 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >>>>>>>>>>> >> > 2023-12-26 21:00:03.844 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >>>>>>>>>>> >> > 2023-12-26 21:00:13.859 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >>>>>>>>>>> >> > 2023-12-26 21:00:23.875 INFO
> [kea-dhcp4.commands/1393730] COMMAND_RECEIVED Received command
> 'ha-heartbeat'
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > And changing the CA or the server HA paramersts to port
> 8001 without changing the other (and the other server results in
> "connection refused" logs. It obv wants the CA port to match the HA
> parameters port despite what we and the documentation suggests...
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > CS, cs.Temp.Mail at gMail.com
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> >
> >>>>>>>>>>> >> > On Mon, 25 Dec 2023 at 02:45, Darren Ankney <
> darren.ankney at gmail.com> wrote:
> >>>>>>>>>>> >> >>
> >>>>>>>>>>> >> >> Hi,
> >>>>>>>>>>> >> >>
> >>>>>>>>>>> >> >> It actually SHOULDN'T work to set your control agent and
> >>>>>>>>>>> >> >> multi-threaded HA listener to the same port as only one
> of the
> >>>>>>>>>>> >> >> applications should be able to setup a listener on that
> port. Please
> >>>>>>>>>>> >> >> describe what you mean by "it doesn't work". I'm
> thinking it might be
> >>>>>>>>>>> >> >> be best to ask Men & Mice about "micetro" and how best
> to set things
> >>>>>>>>>>> >> >> up there.
> >>>>>>>>>>> >> >>
> >>>>>>>>>>> >> >> Thank you,
> >>>>>>>>>>> >> >>
> >>>>>>>>>>> >> >> Darren Ankney
> >>>>>>>>>>> >> >>
> >>>>>>>>>>> >> >> On Thu, Dec 21, 2023 at 6:47 PM CS <
> cs.temp.mail at gmail.com> wrote:
> >>>>>>>>>>> >> >> >
> >>>>>>>>>>> >> >> > Hi all,
> >>>>>>>>>>> >> >> > Moving on from my failure to start and logging issues
> (thank you for your help btw!) I now don't have my heartbeat/control_agent
> working correctly.
> >>>>>>>>>>> >> >> >
> >>>>>>>>>>> >> >> > It works fine so long as I set the ports of my control
> agents and ha hook parameters to be the same (IE 8000 or 8001)
> >>>>>>>>>>> >> >> >
> >>>>>>>>>>> >> >> > However I am unable to tie the tiny cluster into
> micetro, probably because the CA port is occupied with HA heartbeats?
> >>>>>>>>>>> >> >> >
> >>>>>>>>>>> >> >> > Looking to these examples:
> >>>>>>>>>>> >> >> >
> https://github.com/isc-projects/kea/tree/master/doc/examples/template-ha-mt-tls
> >>>>>>>>>>> >> >> >
> >>>>>>>>>>> >> >> > Documentation points out
> >>>>>>>>>>> >> >> > //This specifies the port CA will listen on.
> >>>>>>>>>>> >> >> > // If enabling HA and multi-threading, the
> 8000 port is used by the HA
> >>>>>>>>>>> >> >> > // hook library http listener. When using HA
> hook library with
> >>>>>>>>>>> >> >> > // multi-threading to function, make sure the
> port used by dedicated
> >>>>>>>>>>> >> >> > // listener is different (e.g. 8001) than the
> one used by CA. Note
> >>>>>>>>>>> >> >> > // the commands should still be sent via CA.
> The dedicated listener
> >>>>>>>>>>> >> >> > // is specifically for HA updates only.
> >>>>>>>>>>> >> >> >
> >>>>>>>>>>> >> >> > However, how to have a dedicated port for HA and a
> different one for CA escapes me.
> >>>>>>>>>>> >> >> >
> >>>>>>>>>>> >> >> > CS, cs.Temp.Mail at gMail.com
> >>>>>>>>>>> >> >> > --
> >>>>>>>>>>> >> >> > ISC funds the development of this software with paid
> support subscriptions. Contact us at https://www.isc.org/contact/ for
> more information.
> >>>>>>>>>>> >> >> >
> >>>>>>>>>>> >> >> > To unsubscribe visit
> https://lists.isc.org/mailman/listinfo/kea-users.
> >>>>>>>>>>> >> >> >
> >>>>>>>>>>> >> >> > Kea-users mailing list
> >>>>>>>>>>> >> >> > Kea-users at lists.isc.org
> >>>>>>>>>>> >> >> > https://lists.isc.org/mailman/listinfo/kea-users
> >>>>>>>>>>> >> >> --
> >>>>>>>>>>> >> >> ISC funds the development of this software with paid
> support subscriptions. Contact us at https://www.isc.org/contact/ for
> more information.
> >>>>>>>>>>> >> >>
> >>>>>>>>>>> >> >> To unsubscribe visit
> https://lists.isc.org/mailman/listinfo/kea-users.
> >>>>>>>>>>> >> >>
> >>>>>>>>>>> >> >> Kea-users mailing list
> >>>>>>>>>>> >> >> Kea-users at lists.isc.org
> >>>>>>>>>>> >> >> https://lists.isc.org/mailman/listinfo/kea-users
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> ISC funds the development of this software with paid support
> subscriptions. Contact us at https://www.isc.org/contact/ for more
> information.
> >>>>>>>>>
> >>>>>>>>> To unsubscribe visit
> https://lists.isc.org/mailman/listinfo/kea-users.
> >>>>>>>>>
> >>>>>>>>> Kea-users mailing list
> >>>>>>>>> Kea-users at lists.isc.org
> >>>>>>>>> https://lists.isc.org/mailman/listinfo/kea-users
> >>>>>>
> >>>>>> --
> >>>>>> ISC funds the development of this software with paid support
> subscriptions. Contact us at https://www.isc.org/contact/ for more
> information.
> >>>>>>
> >>>>>> To unsubscribe visit
> https://lists.isc.org/mailman/listinfo/kea-users.
> >>>>>>
> >>>>>> Kea-users mailing list
> >>>>>> Kea-users at lists.isc.org
> >>>>>> https://lists.isc.org/mailman/listinfo/kea-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20231228/b7cd6f6d/attachment-0001.htm>
More information about the Kea-users
mailing list