[Kea-users] Kea HA issue with terminating connection
Darren Ankney
darren.ankney at gmail.com
Mon Jun 12 18:50:42 UTC 2023
Hi Sebastian,
I was hoping to see something jump out at me here, but it all looks
good to me. Is there some possibility that Kea is running out of
memory or disk space for the memfile? I assume these messages are
appearing on the standby server? It looks like this is using
kea-ctrl-agent as the listener. Is there anything being logged by
kea-ctrl-agent?
I see I never asked you what version of Kea this is, can you include
that as well?
Thank you,
Darren Ankney
On Mon, Jun 12, 2023 at 1:54 PM sebastians <sebastians at selfnet.de> wrote:
>
> Hi Darren,
>
> this is what Kea logs shortly before terminating the HA connection (any
> personal information was removed from the logs):
>
> Mar 12 19:25:06 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670034884352] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678688706, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 2907, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
> Mar 12 19:25:06 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678688706, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 2907, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
> Mar 12 19:27:28 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678688848, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 3812, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
> Mar 12 19:32:05 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670018098944] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678689125, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 274, "valid-lft": 43200 }, reason: ResourceBusy: IP address:100.72.19.33 could not be updated.)
> Mar 12 19:32:34 dhcp-2 kea-dhcp4[203037]: WARN [kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678689154, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 113, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
> Mar 12 19:32:36 dhcp-2 kea-dhcp4[203037]: ERROR [kea-dhcp4.ha-hooks.139670104323840] HA_TERMINATED HA service terminated due to an unrecoverable condition. Check previous error message(s), address the problem and restart!
> Mar 12 22:11:09 dhcp-2 kea-dhcp4[203037]: ERROR [kea-dhcp4.packets.139670138794688] DHCP4_BUFFER_RECEIVE_FAIL error on attempt to receive packet: Truncated DHCPv4 packet (len=0) received, at least 236 is expected.
>
> This is the HA part of the configuration on the standby dhcp server:
>
> "hooks-libraries": [{
> "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
> "parameters": {}
> },
> {
> "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_stat_cmds.so",
> "parameters": {}
> },
> {
> "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
> "parameters": {
> "high-availability": [{
> "this-server-name": "standby-dhcp",
> "mode": "hot-standby",
> "heartbeat-delay": 10000,
> "max-response-delay": 60000,
> "max-ack-delay": 5000,
> "max-unacked-clients": 5,
> "peers": [{
> "name": "primary-dhcp",
> "url": "http://dhcp-1:8001/",
> "role": "primary",
> "auto-failover": true
> }, {
> "name": "standby-dhcp",
> "url": "http://dhcp-2:8001/",
> "role": "standby",
> "auto-failover": true
> }]
> }]
> }
> }]
>
> The configuration on dhcp-1 is similar except for the "this-server-name"
> attribute.
>
> Thanks,
>
> Sebastian
>
> On Sun Jun 11, 2023 at 12:35 PM CEST, Darren Ankney wrote:
> > Hi Sebastian,
> >
> > Please share the HA messages surrounding and including one of these
> > terminations. The termination, or a proceeding message should state
> > the cause. The 'ResourceBusy' messages are not warnings I am familiar
> > with. Could you also send your HA configuration from each server
> > (anonymized of course)?
> >
> > Thank you,
> >
> > Darren Ankney
> >
> > On Sat, Jun 10, 2023 at 11:51 AM sebastians <sebastians at selfnet.de> wrote:
> > >
> > > Hi all,
> > >
> > > we recently migrated our DHCP setup from dhcpd to Kea. It runs on
> > > two servers with hot standby and a memfile backend for the leases. Kea
> > > assigns IP addresses for around 7000 pools.
> > >
> > > Over the past few months the HA connection terminated in random intervals.
> > > From looking at the logs on the passive node I can see a lot of
> > > 'ResourceBusy: IP address ... could not be updated' warnings prior to
> > > the connection terminating. Since multithreading is enabled I suspected
> > > this may be due to the threads encountering a resource lock on the memfile.
> > > I suppose after the lease update fails a few times, the connection is terminated.
> > >
> > > Is the 'ResourceBusy' warning the cause for the terminating HA connection and
> > > is there any way to fix the underlying issue? Any ideas on the issue are greatly
> > > appraciated.
> > >
> > > Thanks,
> > >
> > > Sebastian
> > > --
> > > ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
> > >
> > > To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
> > >
> > > Kea-users mailing list
> > > Kea-users at lists.isc.org
> > > https://lists.isc.org/mailman/listinfo/kea-users
> > --
> > ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
> >
> > To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
> >
> > Kea-users mailing list
> > Kea-users at lists.isc.org
> > https://lists.isc.org/mailman/listinfo/kea-users
>
More information about the Kea-users
mailing list