[Kea-users] HA with kea-dhcp4: beginner questions

Wed Nov 23 12:41:04 UTC 2022

It may be that your logs are rotating too quickly after seeing your
logging configuration.  There may be a large amount of logs in a short
period of time during trouble events. depending how many clients you
have.  According to
https://kea.readthedocs.io/en/kea-2.3.2/arm/logging.html maxsize is in
bytes, so you've got it set to 200KB logfile size.  maxver defines how
many rotated files to keep.  So you could keep a maximum of 1.5MB of
logs (unless my math is off).  We get these messages every 10 seconds:

COMMAND_RECEIVED Received command 'ha-heartbeat'

during normal operation.  I think you won't have a very long record of
what is happening with the server with these settings.

06:32:57.393 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
transitions from HOT-STANDBY to PARTNER-DOWN state, partner state is
UNDEFINED

It looks like adc2 lost connection to adc1 (you said adc2 was the
primary).  I guess that may have been the point  that your onsite
administrator unplugged adc1.  Is there anything that is not labeled
kea-dhcp4.ha-hooks in the logfile earlier than this?

On Wed, Nov 23, 2022 at 6:52 AM Stefan G. Weichinger <lists at xunil.at> wrote:
>
> Am 23.11.22 um 12:43 schrieb Darren Ankney:
>
> > If clients were attempting to get addresses at what should have been
> > an elevated pace, there should have been elevated logs in the minutes
> > leading up to the unplug.  There are many nuanced logging options with
> > Kea.  Do you normally see detailed logs of clients attempting to
> > retrieve addresses (successful or unsuccessful)?  If you had no logs
> > at all, then the logging may be broken, not configured correctly in
> > the first place, client messages may not have been getting to the
> > server, or Kea may not have been running.
>
> I am quite sure it was running. Browsed the logs up to that plugging
> event, communication between the nodes was up.
>
> I grepped this out of the logs, maybe something is visible here:
>
> 06:32:57.393 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from HOT-STANDBY to PARTNER-DOWN state, partner state is
> UNDEFINED
> 06:39:50.404 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from HOT-STANDBY to WAITING state, partner state is PARTNER-DOWN
> 06:39:50.404 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from WAITING to SYNCING state, partner state is PARTNER-DOWN
> 06:39:50.441 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from SYNCING to READY state, partner state is PARTNER-DOWN
> 06:39:55.403 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from PARTNER-DOWN to HOT-STANDBY state, partner state is READY
> 06:40:54.795 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from READY to PARTNER-DOWN state, partner state is UNDEFINED
> 06:41:06.394 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from HOT-STANDBY to PARTNER-DOWN state, partner state is
> UNDEFINED
> 07:12:28.853 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from PARTNER-DOWN to WAITING state, partner state is
> PARTNER-DOWN
> 07:12:28.853 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from WAITING to SYNCING state, partner state is PARTNER-DOWN
> 07:12:28.879 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from SYNCING to READY state, partner state is PARTNER-DOWN
> 07:12:35.392 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from PARTNER-DOWN to WAITING state, partner state is READY
> 07:12:35.392 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from WAITING to SYNCING state, partner state is READY
> 07:12:35.420 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from READY to HOT-STANDBY state, partner state is READY
> 07:12:35.420 adc2 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from SYNCING to READY state, partner state is READY
> 07:12:39.060 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from READY to HOT-STANDBY state, partner state is HOT-STANDBY
> 07:23:04.570 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from HOT-STANDBY to PARTNER-DOWN state, partner state is
> UNDEFINED
> 07:24:13.921 adc1 kea-dhcp4.ha-hooks HA_STATE_TRANSITION server
> transitions from WAITING to PARTNER-DOWN state, partner state is UNDEFINED
>
>
> >>
> >> My assumption:
> >>
> >> the samba-installation might have removed or replaced some package kea
> >> depends on. I don't know what ...
> >
> > Most likely, Kea would have failed to run at all if some libraries
> > were unavailable or were incorrect versions (assuming Linux here).
>
> Yes, as mentioned: Debian 11 Bullseye.
>
> Good to know. That was just one thought, as always I try to think "what
> has changed?" to analyse ...
>
> >> so maybe kea on adc2 was already not working OK anymore yesterday
> >> evening and in the night ... but that wasn't noticed because the leases
> >> were still active (?)
> >
> > That is possible.  Depending on your lease length, you wouldn't
> > necessarily notice anything until someone complained about being
> > unable to obtain an IP address.
>
> yes
>
> > Before you rebooted adc2, did you check if Kea was running?
>
> it was running
>
> > Is this
> > DHCPv4, DHCPv6 or both?  What OS is adc2?  Which HA mode did you
> > choose?
>
> Debian 11.5
> isc-kea-2.3.2
> DHCPv4 only
> hot-standby (with adc2 as primary)
>
> > If you would share your logging configuration, that would help
> > too.
>
> Sure:
>
> currently the loglevel is pretty low, I assume. I had "severity" :
> "DEBUG" while the setup phase.
>
> ->
>
> "loggers": [
>          {
>              "name": "kea-dhcp4",
>              "output_options": [
>                  {
>                      "output": "/var/log/kea/kea-debug.log",
>                      "maxver": 8,
>                      "maxsize": 204800,
>                      "flush": true,
>                      "pattern": "%d{%j %H:%M:%S.%q} %c %m\n"
>                  }
>              ],
>              # "severity": "DEBUG",
>              "severity": "INFO",
>              "debuglevel": 99
>          }
>     ]
>
> thanks a lot
>
> --
> ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
>
> To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
>
> Kea-users mailing list
> Kea-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/kea-users