[Kea-users] Weird stats from a shared database

[Jason Guy]

Hi Tomek,

I too was entertaining the idea of deploying 3 Kea servers for HA/Load
Balancing. I had not realized this would not work in 1.3, but your
explanation certainly helps. I suppose there would need to be some type of
sync done between instances (message queue, multicast, etc.).

I wanted to share my design idea as it relates to this discussion, but I
have not done any testing. I was thinking the easiest and most extensible
way to create HA is to use IP "anycast". This requires the hosts are
configured with the same loopback IP address on all servers, and advertise
it out into the network using FRR (routing on the host). This allows the
network to balance the traffic to the servers using ECMP Layer-3 routing. I
would bind Kea to this "anycast" loopback address. Since we use DHCP-Relay
on the ToR/Leaf switches, I would configure them to relay to the Kea
servers' loopback address. All the Kea servers would access the same
backend SQL database cluster. I have not figured out the database side of
things yet either, but assume the backend database is clustered properly
and everything is in sync.

Cheers,
Jason


On Wed, Dec 13, 2017 at 7:12 AM, Tomek Mrugalski <tomasz at isc.org> wrote:

> W dniu 12.12.2017 o 15:48, Munroe Sollog pisze:
> > Let me know if this should be a bug, but I have noticed some weird stats
> > when running two kea-1.3 DHCP servers from the same mysql database
> > (using Galera).
> >
> > I have included a screen grab of the stats.
> >
> > Between noon and 2pm yesterday we when I migrated rogi from the memfile
> > to the mysql database.  I migrated all of the existing leases from the
> > CSV to mysql and started rogi.
> >
> > Igor from around 2:30pm until about 7:45am the next day it steadily
> > declines all the way to -436 leases.  How can it possibly have
> > *negative* leases?
> Rasmus is right. Running more than one Kea server using the same
> database is not officially supported.
>
> Here's what is likely to happen: each Kea instance allocates leases to
> clients. For each allocation, the statistic is increased. The statistic
> is observed on each instance. It is likely to be incorrect as there is
> another instance that also allocates leases.
>
> Now, unless you took extra steps to disable lease expiration on one
> instance and keep it running on another, there are two instances
> periodically looking for leases that are expired. Depending on how many
> leases are expired during exact moment when the expiration triggers, one
> server may get more expired leases to process than the other. Only that
> server will decrease the statistic.
>
> Finally, I don't know how you set this up, but I presume that the server
> that allocated a lease will send its own server-id and thus the release
> messages will be processed only by that server. So this shouldn't
> contribute to the confusion, unless you did some clever things with
> server-id.
>
> You may perceive it as a bug. It's a valid point of view. But I see it
> as Kea being run in a configuration that is not officially supported.
> There's nothing wrong with it. We're happy it provides service and
> generally works. It's just there are quirks like this.
>
> We do have recountLeaseStats4 and recountLeaseStats6 method, but it is
> only used internally. I suppose we could expose it as a command that you
> could call. Kea instance would then consult the database and recalculate
> the values.
>
> As Rasmus mentioned, we do plan to improve the situation significantly
> in 1.4. We want to provide a high availability solution, but also
> improve many aspects of running multiple Kea servers at the same time.
>
> I don't have any specific solution for you right now, just some things
> to consider. Kea doesn't have any notion (at least not yet) of a server
> instance owning a lease. You could try generating the statistic by
> pooling both servers and adding the values together. Consider it an
> experiment. It may or may not work. I'd love to hear about the results.
>
> I'd like to ask you a favour. Can you describe how you did set up Galera
> for MySQL on kea wiki? There are installation instructions here:
> http://kea.isc.org/wiki/Install I was thinking about something similar,
> but with detailed instructions how to set up Galera cluster. This would
> be useful for two reasons. First, other users could set it up in similar
> fashion. Second, one of ISC engineers will get to look at this problem
> one day. It will be very helpful to have an instruction to replicate
> your environment.
>
> Finally, can you submit a bug for this? It would great if this bug
> report had a link to the installation instruction.
>
> Hope that helps,
> Tomek
> _______________________________________________
> Kea-users mailing list
> Kea-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/kea-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20171213/5dc11913/attachment.htm>