[Kea-users] Kea HA issue with terminating connection

Mon Jun 12 17:55:38 UTC 2023

Hi Darren,

this is what Kea logs shortly before terminating the HA connection (any
personal information was removed from the logs):

Mar 12 19:25:06 dhcp-2 kea-dhcp4[203037]: WARN  [kea-dhcp4.lease-cmds-hooks.139670034884352] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678688706, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 2907, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:25:06 dhcp-2 kea-dhcp4[203037]: WARN  [kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678688706, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 2907, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:27:28 dhcp-2 kea-dhcp4[203037]: WARN  [kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678688848, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 3812, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:32:05 dhcp-2 kea-dhcp4[203037]: WARN  [kea-dhcp4.lease-cmds-hooks.139670018098944] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678689125, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 274, "valid-lft": 43200 }, reason: ResourceBusy: IP address:100.72.19.33 could not be updated.)
Mar 12 19:32:34 dhcp-2 kea-dhcp4[203037]: WARN  [kea-dhcp4.lease-cmds-hooks.139670009706240] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "client-id": "", "expire": 1678689154, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "", "hw-address": "", "ip-address": "", "state": 0, "subnet-id": 113, "valid-lft": 43200 }, reason: ResourceBusy: IP address:123.123.123.123 could not be updated.)
Mar 12 19:32:36 dhcp-2 kea-dhcp4[203037]: ERROR [kea-dhcp4.ha-hooks.139670104323840] HA_TERMINATED HA service terminated due to an unrecoverable condition. Check previous error message(s), address the problem and restart!
Mar 12 22:11:09 dhcp-2 kea-dhcp4[203037]: ERROR [kea-dhcp4.packets.139670138794688] DHCP4_BUFFER_RECEIVE_FAIL error on attempt to receive packet: Truncated DHCPv4 packet (len=0) received, at least 236 is expected.

This is the HA part of the configuration on the standby dhcp server:

"hooks-libraries": [{
    "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
    "parameters": {}
},
{
    "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_stat_cmds.so",
    "parameters": {}
},
{
    "library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
    "parameters": {
        "high-availability": [{
            "this-server-name": "standby-dhcp",
            "mode": "hot-standby",
            "heartbeat-delay": 10000,
            "max-response-delay": 60000,
            "max-ack-delay": 5000,
            "max-unacked-clients": 5,
            "peers": [{
                "name": "primary-dhcp",
                "url": "http://dhcp-1:8001/",
                "role": "primary",
                "auto-failover": true
            }, {
                "name": "standby-dhcp",
                "url": "http://dhcp-2:8001/",
                "role": "standby",
                "auto-failover": true
            }]
        }]
    }
}]

The configuration on dhcp-1 is similar except for the "this-server-name"
attribute.

Thanks,

Sebastian

On Sun Jun 11, 2023 at 12:35 PM CEST, Darren Ankney wrote:
> Hi Sebastian,
>
> Please share the HA messages surrounding and including one of these
> terminations.  The termination, or a proceeding message should state
> the cause.  The 'ResourceBusy' messages are not warnings I am familiar
> with.  Could you also send your HA configuration from each server
> (anonymized of course)?
>
> Thank you,
>
> Darren Ankney
>
> On Sat, Jun 10, 2023 at 11:51 AM sebastians <sebastians at selfnet.de> wrote:
> >
> > Hi all,
> >
> > we recently migrated our DHCP setup from dhcpd to Kea. It runs on
> > two servers with hot standby and a memfile backend for the leases. Kea
> > assigns IP addresses for around 7000 pools.
> >
> > Over the past few months the HA connection terminated in random intervals.
> > From looking at the logs on the passive node I can see a lot of
> > 'ResourceBusy: IP address ... could not be updated' warnings prior to
> > the connection terminating. Since multithreading is enabled I suspected
> > this may be due to the threads encountering a resource lock on the memfile.
> > I suppose after the lease update fails a few times, the connection is terminated.
> >
> > Is the 'ResourceBusy' warning the cause for the terminating HA connection and
> > is there any way to fix the underlying issue? Any ideas on the issue are greatly
> > appraciated.
> >
> > Thanks,
> >
> > Sebastian
> > --
> > ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
> >
> > To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
> >
> > Kea-users mailing list
> > Kea-users at lists.isc.org
> > https://lists.isc.org/mailman/listinfo/kea-users
> -- 
> ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
>
> To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
>
> Kea-users mailing list
> Kea-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/kea-users