[Kea-users] unable to forward command to the dhcp4 service: Connection refused. The server is likely to be offline
Darren Ankney
darren.ankney at gmail.com
Mon Mar 6 13:11:29 UTC 2023
You can certainly see messages in the logs if you setup the ha logs
(see: https://kea.readthedocs.io/en/kea-2.2.0/arm/logging.html and in
particluar this chart
https://kea.readthedocs.io/en/kea-2.2.0/arm/logging.html#id3 which
shows all of the available loggers - you will want to for sure setup
"kea-dhcp4.ha-hooks"). You can also access the status via the API
(see https://kea.readthedocs.io/en/kea-2.2.0/arm/ctrl-channel.html#management-api)
with ha-heartbeat
(https://kea.readthedocs.io/en/kea-2.2.0/api.html#ha-heartbeat).
On Mon, Mar 6, 2023 at 7:51 AM Kraishak Mahtha <kraishak.edu at gmail.com> wrote:
>
> Yes Thanks Darren, found the issue after changing the file name it is working fine and do you have any idea like how can we get the failover status
> In ISC DHCP failover we can see them in the lease file (communication interrupted, recover, partner-down....etc) with the failover name being specified but I don't see any such info in the kea-leas file
>
> Thanks
> Kraishak
>
> On Mon, Mar 6, 2023 at 4:23 PM Darren Ankney <darren.ankney at gmail.com> wrote:
>>
>> Looks like your socket locations don't match between the kea config
>> and the ctrl agent configuration. Those are filenames, so they are
>> going to need to match exactly. Kea is in the driver's seat there as
>> it actually creates the file. the ctrl agent "talks" to kea through
>> such file.
>>
>> example, your primary kea config says: /tmp/kea4-ctrl-socket1 and
>> your ctrl agent config on the primary says: /tmp/control_socket
>>
>> On Mon, Mar 6, 2023 at 4:38 AM Kraishak Mahtha <kraishak.edu at gmail.com> wrote:
>> >
>> > Hi All,
>> >
>> > I am trying to configure the kea-HA setup in my local environment but I am facing an issue the heartbeat is getting failed and the logs are as follows:
>> > Logs on Failover
>> > =============
>> > 2023-03-06 09:04:43.761 DEBUG [kea-dhcp4.http/28550.140575742654656] HTTP_CLIENT_REQUEST_SEND sending HTTP request POST / HTTP/1.1 to http://192.168.0.168:8000/
>> > 2023-03-06 09:04:43.761 DEBUG [kea-dhcp4.http/28550.140575742654656] HTTP_CLIENT_REQUEST_SEND_DETAILS detailed information about request sent to http://192.168.0.168:8000/:
>> > POST / HTTP/1.1
>> > Host: 192.168.0.168
>> > Content-Length: 53
>> > Content-Type: application/json
>> >
>> > { "command": "ha-heartbeat", "service": [ "dhcp4" ] }
>> > 2023-03-06 09:04:43.762 DEBUG [kea-dhcp4.http/28550.140575742654656] HTTP_SERVER_RESPONSE_RECEIVED received HTTP response from http://192.168.0.168:8000/
>> > 2023-03-06 09:04:43.762 DEBUG [kea-dhcp4.http/28550.140575742654656] HTTP_SERVER_RESPONSE_RECEIVED_DETAILS detailed information about well-formed response received from http://192.168.0.168:8000/:
>> > HTTP/1.1 200 OK
>> > Content-Length: 133
>> > Content-Type: application/json
>> > Date: Mon, 06 Mar 2023 09:04:43 GMT
>> >
>> > [ { "result": 1, "text": "unable to forward command to the dhcp4 service: Connection refused. The server is likely to be offline" } ]
>> > 2023-03-06 09:04:43.762 WARN [kea-dhcp4.ha-hooks/28550.140575742654656] HA_HEARTBEAT_FAILED heartbeat to kea-dhcp-168 (http://192.168.0.168:8000/) failed: unable to forward command to the dhcp4 service: Connection refused. The server is likely to be offline, error code 1
>> > 2023-03-06 09:04:43.762 WARN [kea-dhcp4.ha-hooks/28550.140575742654656] HA_COMMUNICATION_INTERRUPTED communication with kea-dhcp-168 is interrupted
>> >
>> >
>> > Logs on Primary
>> > ==============
>> > 2023-03-06 09:01:50.143 DEBUG [kea-dhcp4.dhcpsrv/13620.140081380149440] DHCPSRV_TIMERMGR_START_TIMER starting timer: reclaim-expired-leases
>> > 2023-03-06 09:01:52.145 DEBUG [kea-dhcp4.http/13620.140081380149440] HTTP_CLIENT_REQUEST_SEND sending HTTP request POST / HTTP/1.1 to http://192.168.0.169:8000/
>> > 2023-03-06 09:01:52.145 DEBUG [kea-dhcp4.http/13620.140081380149440] HTTP_CLIENT_REQUEST_SEND_DETAILS detailed information about request sent to http://192.168.0.169:8000/:
>> > POST / HTTP/1.1
>> > Host: 192.168.0.169
>> > Content-Length: 53
>> > Content-Type: application/json
>> >
>> > { "command": "ha-heartbeat", "service": [ "dhcp4" ] }
>> > 2023-03-06 09:01:52.146 DEBUG [kea-dhcp4.http/13620.140081380149440] HTTP_SERVER_RESPONSE_RECEIVED received HTTP response from http://192.168.0.169:8000/
>> > 2023-03-06 09:01:52.146 DEBUG [kea-dhcp4.http/13620.140081380149440] HTTP_SERVER_RESPONSE_RECEIVED_DETAILS detailed information about well-formed response received from http://192.168.0.169:8000/:
>> > HTTP/1.1 200 OK
>> > Content-Length: 133
>> > Content-Type: application/json
>> > Date: Mon, 06 Mar 2023 09:01:52 GMT
>> >
>> > [ { "result": 1, "text": "unable to forward command to the dhcp4 service: Connection refused. The server is likely to be offline" } ]
>> > 2023-03-06 09:01:52.146 WARN [kea-dhcp4.ha-hooks/13620.140081380149440] HA_HEARTBEAT_FAILED heartbeat to appliance-169 (http://192.168.0.169:8000/) failed: unable to forward command to the dhcp4 service: Connection refused. The server is likely to be offline, error code 1
>> > 2023-03-06 09:01:53.558 INFO [kea-ctrl-agent.commands/13581.140159667833024] COMMAND_RECEIVED Received command 'ha-heartbeat'
>> > 2023-03-06 09:01:53.558 INFO [kea-ctrl-agent.ctrl-agent/13581.140159667833024] CTRL_AGENT_COMMAND_RECEIVED command ha-heartbeat received from remote address 192.168.0.169
>> >
>> >
>> > =========
>> > Primary config
>> > ==============
>> > ##DHCPv4 192.168.0.168
>> > {
>> > "Dhcp4": {
>> > "valid-lifetime": 300,
>> > "renew-timer": 150,
>> > "rebind-timer": 200,
>> > "server-tag": "192.168.0.168",
>> > "control-socket": {
>> > "socket-name": "/tmp/kea4-ctrl-socket1",
>> > "socket-type": "unix"
>> > },
>> >
>> > "lease-database": {
>> > "type": "memfile",
>> > "persist": true,
>> > "name": "/var/lib/kea/dhcp4.leases"
>> > },
>> >
>> > "interfaces-config": {
>> > "interfaces": ["eth0"]
>> > },
>> >
>> > "hooks-libraries": [
>> > {
>> > "library": "/usr/local/lib/kea/hooks/libdhcp_lease_cmds.so",
>> > "parameters": { }
>> > },
>> > {
>> > "library": "/usr/local/lib/kea/hooks/libdhcp_bootp.so",
>> > "parameters": { }
>> > },
>> > {
>> > "library": "/usr/local/lib/kea/hooks/libdhcp_ha.so",
>> > "parameters": {
>> > "high-availability": [{
>> > "this-server-name": "kea-dhcp-168",
>> > "mode": "hot-standby",
>> > "heartbeat-delay": 10000,
>> > "max-response-delay": 60000,
>> > "max-ack-delay": 5000,
>> > "max-unacked-clients": 5,
>> > "peers": [{
>> > "name": "kea-dhcp-168",
>> > "url": "http://192.168.0.168:8000/",
>> > "role": "primary",
>> > "auto-failover": true
>> > }, {
>> > "name": "appliance-169",
>> > "url": "http://192.168.0.169:8000/",
>> > "role": "standby",
>> > "auto-failover": true
>> > }]
>> > }]
>> > }
>> > }],
>> > "subnet4": [{
>> > "subnet": "4.0.0.0/24",
>> > "option-data": [
>> > {
>> > "data": "2.2.2.2, 8.8.4.4",
>> > "name": "domain-name-servers"
>> > },
>> > {
>> > "data": "4.0.0.1",
>> > "name": "routers"
>> > }
>> > ],
>> > "pools": [
>> > {
>> > "pool": "4.0.0.25 - 4.0.0.250"
>> > }
>> > ]
>> >
>> > }],
>> > "loggers": [
>> > {
>> > "name": "kea-dhcp4",
>> > "debuglevel": 99,
>> > "output_options": [
>> > {
>> > "output": "/usr/local/var/log/kea-dhcp4.log"
>> > }
>> > ],
>> > "severity": "DEBUG"
>> > }
>> > ]
>> > }
>> > }
>> > kea-ctrl-agent.conf on Primary
>> > =========================
>> >
>> > {
>> > "Control-agent": {
>> > "http-host": "192.168.0.168",
>> > "http-port": 8000,
>> > "control-sockets": {
>> > "dhcp4": {
>> > "socket-type": "unix",
>> > "socket-name": "/tmp/control_socket"
>> > }
>> > },
>> > "cert-required": false
>> > }
>> > }
>> > Netstat output
>> > =============
>> > [root at kea-dhcp-168 ~]# netstat -tulpna | grep 8000
>> > tcp 0 0 192.168.0.168:8000 0.0.0.0:* LISTEN 13581/kea-ctrl-agen
>> > tcp 0 0 192.168.0.168:57308 192.168.0.169:8000 ESTABLISHED 13620/kea-dhcp4
>> > tcp 0 0 192.168.0.168:8000 192.168.0.169:58432 ESTABLISHED 13581/kea-ctrl-agen
>> > [root at kea-dhcp-168 ~]#
>> >
>> > [root at kea-dhcp-168 ~]# netstat -tulpna | grep 67
>> > udp 0 0 192.168.0.168:67 0.0.0.0:* 13620/kea-dhcp4
>> > [root at kea-dhcp-168 ~]#
>> >
>> >
>> > Failover DHCPv4 config
>> > ===================
>> > #DHCPv4 192.168.0.169
>> > {
>> > "Dhcp4": {
>> > "valid-lifetime": 300,
>> > "renew-timer": 150,
>> > "rebind-timer": 200,
>> > "server-tag": "192.168.0.169",
>> > "control-socket": {
>> > "socket-name": "/opt/testUAT/keaFolder/kea4-ctrl-socket",
>> > "socket-type": "unix"
>> > },
>> >
>> > "lease-database": {
>> > "type": "memfile",
>> > "persist": true,
>> > "name": "/opt/testUAT/var/lib/kea/dhcp4.leases"
>> > },
>> >
>> > "interfaces-config": {
>> > "interfaces": ["eth0"]
>> > },
>> >
>> > "hooks-libraries": [
>> > {
>> > "library": "/opt/testUAT/lib/kea/hooks/libdhcp_lease_cmds.so",
>> > "parameters": { }
>> > },
>> > {
>> > "library": "/opt/testUAT/lib/kea/hooks/libdhcp_bootp.so",
>> > "parameters": { }
>> > },
>> > {
>> > "library": "/opt/testUAT/lib/kea/hooks/libdhcp_ha.so",
>> > "parameters": {
>> > "high-availability": [{
>> > "this-server-name": "appliance-169",
>> > "mode": "hot-standby",
>> > "heartbeat-delay": 10000,
>> > "max-response-delay": 60000,
>> > "max-ack-delay": 5000,
>> > "max-unacked-clients": 5,
>> > "peers": [{
>> > "name": "kea-dhcp-168",
>> > "url": "http://192.168.0.168:8000/",
>> > "role": "primary",
>> > "auto-failover": true
>> > }, {
>> > "name": "appliance-169",
>> > "url": "http://192.168.0.169:8000/",
>> > "role": "standby",
>> > "auto-failover": true
>> > }]
>> > }]
>> > }
>> > }],
>> > "subnet4": [{
>> > "subnet": "4.0.0.0/24",
>> > "option-data": [
>> > {
>> > "data": "2.2.2.2, 8.8.4.4",
>> > "name": "domain-name-servers"
>> > },
>> > {
>> > "data": "4.0.0.1",
>> > "name": "routers"
>> > }
>> > ],
>> > "pools": [
>> > {
>> > "pool": "4.0.0.25 - 4.0.0.250"
>> > }
>> > ]
>> >
>> > }],
>> > "loggers": [
>> > {
>> > "name": "kea-dhcp4",
>> > "debuglevel": 99,
>> > "output_options": [
>> > {
>> > "output": "/opt/testUAT/var/log/kea-dhcp4.log"
>> > }
>> > ],
>> > "severity": "DEBUG"
>> > }
>> > ]
>> > }
>> > }
>> >
>> > kea-ctrl-agent.conf on Failover
>> > =======================
>> > {
>> > "Control-agent": {
>> > "http-host": "192.168.0.169",
>> > "http-port": 8000,
>> > "control-sockets": {
>> > "dhcp4": {
>> > "socket-type": "unix",
>> > "socket-name": "/tmp/control_socket"
>> > }
>> > },
>> > "cert-required": false
>> > }
>> > }
>> > Netstat output
>> > =============
>> >
>> > [root at appliance-169 ~]# netstat -tulpna | grep 8000
>> > tcp 0 0 192.168.0.169:8000 0.0.0.0:* LISTEN 28520/kea-ctrl-agen
>> > tcp 0 0 192.168.0.169:8000 192.168.0.168:57308 ESTABLISHED 28520/kea-ctrl-agen
>> > tcp 0 0 192.168.0.169:58432 192.168.0.168:8000 ESTABLISHED 28550/kea-dhcp4
>> > [root at appliance-169 ~]#
>> >
>> > [root at kea-dhcp-168 ~]# netstat -tulpna | grep 67
>> > udp 0 0 192.168.0.169:67 0.0.0.0:* 28550/kea-dhcp4
>> > [root at kea-dhcp-168 ~]#
>> >
>> > I am not sure why it is unable to forward commands to dhcp4 service, Can someone guide or help me if you face the same issue or are aware of the issue?
>> >
>> > Thanks in advance
>> > Kraishak
>> >
>> > --
>> > ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
>> >
>> > To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
>> >
>> > Kea-users mailing list
>> > Kea-users at lists.isc.org
>> > https://lists.isc.org/mailman/listinfo/kea-users
>> --
>> ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information.
>>
>> To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
>>
>> Kea-users mailing list
>> Kea-users at lists.isc.org
>> https://lists.isc.org/mailman/listinfo/kea-users
More information about the Kea-users
mailing list