[Kea-users] Segmentation fault using REST API in DHCPv4 HA setting
Caciano dos Santos Machado
Caciano at cpd.ufrgs.br
Wed Sep 21 20:51:16 UTC 2022
Hi,
I am trying to configure a pair of kea servers running in high
availability mode. However, the servers crash with segfault when I try
to use the HTTP RESP API for some commands. I figured out how to
reproduce the problem in one situation described below.
The server is a Ubuntu 22.04 LTS, running kea 2.2.0 from the repository
https://dl.cloudsmith.io/public/isc/kea-2-2/deb/ubuntu jammy main.
The server is a Xen VM with 2GB of RAM, kernel 5.15.0-43-generic
#46-Ubuntu SMP and glibc 2.35.
This following python script loads a configuration file and calls
config-set using the HTTP API.
------------------------------------------------------------------------------
#!/usr/bin/python3
#coding: utf-8
import json, requests
with open('bug-config.json') as f:
config = json.load(f)
url = 'http://192.168.89.12:8000/'
headers = {'content-type': 'application/json'}
payload = {}
payload['command'] = 'config-set'
payload['service'] = [ 'dhcp4' ]
payload['arguments'] = config
res = requests.post(url, headers=headers, data=json.dumps(payload))
response = res.json()[0]
print('Set config: ' + response['text'])
------------------------------------------------------------------------------
The configuration file sent using the API is this one, without the
shared-networks section. The problem also happens when I send a complete
configuration, but this small one is enough to reproduce the problem.
The configuration file running in the server when I send the config-set
is the same, but has about 19000 host reservations identified by the
hw-address, distributed in 180 networks that are in 90 shared-networks.
------------------------------------------------------------------------------
{
"Dhcp4": {
"authoritative": true,
"client-classes": [
{
"name": "718",
"option-data": [
{
"data": "192.168.1.52,192.168.1.53",
"name": "domain-name-servers"
}
]
},
{
"name": "708",
"option-data": [
{
"data": "192.168.137.7,192.168.1.52,192.168.1.53",
"name": "domain-name-servers"
}
]
},
{
"name": "533",
"option-data": [
{
"data": "192.168.137.7,192.168.1.52,192.168.1.53",
"name": "domain-name-servers"
}
]
},
{
"name": "791",
"option-data": [
{
"data": "192.168.148.30,192.168.1.52,192.168.1.53",
"name": "domain-name-servers"
}
]
},
{
"name": "792",
"option-data": [
{
"data": "192.168.148.30,192.168.1.52,192.168.1.53",
"name": "domain-name-servers"
}
]
},
{
"name": "793",
"option-data": [
{
"data": "192.168.148.30,192.168.1.52,192.168.1.53",
"name": "domain-name-servers"
}
]
},
{
"name": "794",
"option-data": [
{
"data": "192.168.148.30,192.168.1.52,192.168.1.53",
"name": "domain-name-servers"
}
]
},
{
"name": "795",
"option-data": [
{
"data": "192.168.148.30,192.168.1.52,192.168.1.53",
"name": "domain-name-servers"
}
]
},
{
"name": "532",
"option-data": [
{
"data":
"192.168.148.22,192.168.148.30,192.168.1.52,192.168.1.53",
"name": "domain-name-servers"
}
]
},
{
"name": "561",
"option-data": [
{
"data": "192.168.2.165,192.168.2.166",
"name": "domain-name-servers"
}
]
},
{
"name": "568",
"option-data": [
{
"data": "192.168.162.105, 192.168.1.52, 192.168.1.53",
"name": "domain-name-servers"
}
]
},
{
"name": "118",
"option-data": [
{
"data": "ifch.ufrgs.br",
"name": "domain-name"
}
]
},
{
"name": "527",
"option-data": [
{
"data":
"192.168.150.20,192.168.150.23,192.168.1.52,192.168.1.53",
"name": "domain-name-servers"
}
]
}
],
"control-socket": {
"socket-name": "/tmp/kea4-ctrl-socket",
"socket-type": "unix"
},
"dhcp-ddns": {
"enable-updates": false
},
"expired-leases-processing": {
"hold-reclaimed-time": 401
},
"hooks-libraries": [
{
"library":
"/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so"
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_bootp.so"
},
{
"library": "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [
{
"heartbeat-delay": 10000,
"max-unacked-clients": 20,
"min-ack-delay": 5000,
"mode": "hot-standby",
"peers": [
{
"auto-failover": true,
"name": "dhcp",
"role": "primary",
"url": "http://192.168.89.14:8000/"
},
{
"auto-failover": true,
"name": "dhcp-standby",
"role": "standby",
"url": "http://192.168.89.12:8000/"
}
],
"send-lease-updates": false,
"sync-leases": false,
"this-server-name": "dhcp-standby"
}
]
}
}
],
"host-reservation-identifiers": [
"hw-address"
],
"interfaces-config": {
"dhcp-socket-type": "udp",
"interfaces": [
"eth0/192.168.89.12"
]
},
"ip-reservations-unique": true,
"lease-database": {
"lfc-interval": 3600,
"max-row-errors": 100,
"name": "/var/lib/kea/kea-leases4.csv",
"type": "memfile"
},
"loggers": [
{
"debuglevel": 15,
"name": "kea-dhcp4",
"output_options": [
{
"maxver": 32,
"output": "/var/log/kea/kea-dhcp4-server.log"
}
],
"severity": "INFO"
}
],
"match-client-id": false,
"max-valid-lifetime": 1800,
"min-valid-lifetime": 360,
"option-def": [
{
"code": 252,
"name": "wpad",
"type": "string"
},
{
"code": 128,
"name": "option-128",
"type": "string"
},
{
"code": 129,
"name": "etherboot_options",
"type": "string"
}
],
"reservations-global": false,
"reservations-in-subnet": true,
"reservations-lookup-first": true,
"reservations-out-of-pool": true,
"sanity-checks": {
"lease-checks": "fix-del"
},
"store-extended-info": true,
"valid-lifetime": 1800
}
}
------------------------------------------------------------------------------
When the server crashes then "systemctl status" gives the following output:
------------------------------------------------------------------------------
root at keadhcp-dev-02:/etc/kea# systemctl status isc-kea-dhcp4-server.service
× isc-kea-dhcp4-server.service - Kea IPv4 DHCP daemon
Loaded: loaded (/lib/systemd/system/isc-kea-dhcp4-server.service;
enabled; vendor preset: enabled)
Active: failed (Result: core-dump) since Wed 2022-09-21 18:57:31
UTC; 16min ago
Docs: man:kea-dhcp4(8)
Process: 1231471 ExecStart=/usr/sbin/kea-dhcp4 -c
/etc/kea/kea-dhcp4.conf (code=dumped, signal=SEGV)
Main PID: 1231471 (code=dumped, signal=SEGV)
CPU: 7.494s
Sep 21 18:57:19 keadhcp-dev-02 systemd[1]: Started Kea IPv4 DHCP daemon.
Sep 21 18:57:31 keadhcp-dev-02 systemd[1]: isc-kea-dhcp4-server.service:
Main process exited, code=dumped, status=11/SEGV
Sep 21 18:57:31 keadhcp-dev-02 systemd[1]: isc-kea-dhcp4-server.service:
Failed with result 'core-dump'.
Sep 21 18:57:31 keadhcp-dev-02 systemd[1]: isc-kea-dhcp4-server.service:
Consumed 7.494s CPU time.
------------------------------------------------------------------------------
The core dump file has the following data in the first lines:
------------------------------------------------------------------------------
root at keadhcp-dev-02:/etc/kea# head -n 73
/var/crash/_usr_sbin_kea-dhcp4.113.crash
ProblemType: Crash
Architecture: amd64
CrashCounter: 1
Date: Mon Sep 19 18:19:50 2022
DistroRelease: Ubuntu 22.04
ExecutablePath: /usr/sbin/kea-dhcp4
ExecutableTimestamp: 1653069600
ProcCmdline: /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
ProcEnviron: Error: [Errno 13] Permission denied: 'environ'
ProcMaps: Error: [Errno 13] Permission denied: 'maps'
ProcStatus:
Name: kea-dhcp4
Umask: 0022
State: S (sleeping)
Tgid: 1159118
Ngid: 0
Pid: 1159118
PPid: 1
TracerPid: 0
Uid: 113 113 113 113
Gid: 118 118 118 118
FDSize: 64
Groups: 118
NStgid: 1159118
NSpid: 1159118
NSpgid: 1159118
NSsid: 1159118
VmPeak: 213420 kB
VmSize: 184160 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 155992 kB
VmRSS: 124736 kB
RssAnon: 110380 kB
RssFile: 14356 kB
RssShmem: 0 kB
VmData: 149112 kB
VmStk: 132 kB
VmExe: 628 kB
VmLib: 17112 kB
VmPTE: 348 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
CoreDumping: 1
THP_enabled: 1
Threads: 5
SigQ: 0/7428
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000
SigCgt: 0000000100004003
CapInh: 0000000000002400
CapPrm: 0000000000002400
CapEff: 0000000000002400
CapBnd: 000001ffffffffff
CapAmb: 0000000000002400
NoNewPrivs: 0
Seccomp: 0
Seccomp_filters: 0
Speculation_Store_Bypass: vulnerable
SpeculationIndirectBranch: always enabled
Cpus_allowed: 000f
Cpus_allowed_list: 0-3
Mems_allowed:
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 136
nonvoluntary_ctxt_switches: 304
Signal: 11
Uname: Linux 5.15.0-43-generic x86_64
UserGroups: N/A
CoreDump: base64
H4sICAAAAAAC/0NvcmVEdW1wAA==
......
------------------------------------------------------------------------------
The last messages in the log file /var/log/kea/kea-dhcp4-server.log are
these:
------------------------------------------------------------------------------
2022-09-21 19:30:04.973 DEBUG
[kea-dhcp4.commands/1232318.139712599805568]
COMMAND_SOCKET_CONNECTION_OPENED Opened socket 26 for incoming command
connection
2022-09-21 19:30:04.973 DEBUG
[kea-dhcp4.dhcpsrv/1232318.139712599805568]
DHCPSRV_TIMERMGR_RUN_TIMER_OPERATION running operation for timer:
flush-reclaimed-leases
2022-09-21 19:30:04.973 DEBUG
[kea-dhcp4.alloc-engine/1232318.139712599805568]
ALLOC_ENGINE_V4_RECLAIMED_LEASES_DELETE begin deletion of reclaimed
leases expired more than 3600 seconds ago
2022-09-21 19:30:04.973 DEBUG
[kea-dhcp4.dhcpsrv/1232318.139712599805568]
DHCPSRV_MEMFILE_DELETE_EXPIRED_RECLAIMED4 deleting reclaimed IPv4 leases
that expired more than 3600 seconds ago
2022-09-21 19:30:04.973 DEBUG
[kea-dhcp4.alloc-engine/1232318.139712599805568]
ALLOC_ENGINE_V4_RECLAIMED_LEASES_DELETE_COMPLETE successfully deleted 0
expired-reclaimed leases
2022-09-21 19:30:04.973 DEBUG
[kea-dhcp4.dhcpsrv/1232318.139712599805568] DHCPSRV_TIMERMGR_START_TIMER
starting timer: flush-reclaimed-leases
2022-09-21 19:30:04.973 DEBUG
[kea-dhcp4.commands/1232318.139712599805568] COMMAND_SOCKET_READ
Received 3597 bytes over command socket 26
2022-09-21 19:30:04.976 INFO
[kea-dhcp4.commands/1232318.139712599805568] COMMAND_RECEIVED Received
command 'config-set'
2022-09-21 19:30:04.977 INFO [kea-dhcp4.hosts/1232318.139712599805568]
HOSTS_BACKENDS_REGISTERED the following host backend types are
available: mysql postgresql
2022-09-21 19:30:04.977 INFO [kea-dhcp4.dhcpsrv/1232318.139712599805568]
DHCPSRV_CFGMGR_SOCKET_TYPE_SELECT using socket type udp
2022-09-21 19:30:04.977 INFO [kea-dhcp4.dhcpsrv/1232318.139712599805568]
DHCPSRV_CFGMGR_USE_ADDRESS listening on address 192.168.89.12, on
interface eth0
2022-09-21 19:30:04.977 INFO [kea-dhcp4.hooks/1232318.139712599805568]
HOOKS_LIBRARY_CLOSED hooks library
/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so successfully
closed
2022-09-21 19:30:04.978 INFO [kea-dhcp4.hooks/1232318.139712599805568]
HOOKS_LIBRARY_CLOSED hooks library
/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_bootp.so successfully closed
2022-09-21 19:30:04.978 INFO [kea-dhcp4.hooks/1232318.139712599805568]
HOOKS_LIBRARY_CLOSED hooks library
/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so successfully closed
2022-09-21 19:30:04.979 INFO
[kea-dhcp4.ha-hooks/1232318.139712599805568] HA_DEINIT_OK unloading High
Availability hooks library successful
2022-09-21 19:30:04.979 INFO
[kea-dhcp4.bootp-hooks/1232318.139712599805568] BOOTP_UNLOAD Bootp hooks
library has been unloaded
2022-09-21 19:30:04.979 INFO
[kea-dhcp4.lease-cmds-hooks/1232318.139712599805568]
LEASE_CMDS_DEINIT_OK unloading Lease Commands hooks library successful
2022-09-21 19:30:04.980 INFO [kea-dhcp4.hooks/1232318.139712599805568]
HOOKS_LIBRARY_CLOSED hooks library
/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so successfully closed
2022-09-21 19:30:04.980 INFO [kea-dhcp4.hooks/1232318.139712599805568]
HOOKS_LIBRARY_CLOSED hooks library
/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_bootp.so successfully closed
2022-09-21 19:30:04.980 INFO [kea-dhcp4.hooks/1232318.139712599805568]
HOOKS_LIBRARY_CLOSED hooks library
/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so successfully
closed
2022-09-21 19:30:04.982 INFO
[kea-dhcp4.lease-cmds-hooks/1232318.139712599805568] LEASE_CMDS_INIT_OK
loading Lease Commands hooks library successful
2022-09-21 19:30:04.982 INFO [kea-dhcp4.hooks/1232318.139712599805568]
HOOKS_LIBRARY_LOADED hooks library
/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so successfully
loaded
2022-09-21 19:30:04.982 INFO
[kea-dhcp4.bootp-hooks/1232318.139712599805568] BOOTP_LOAD Bootp hooks
library has been loaded
2022-09-21 19:30:04.982 INFO [kea-dhcp4.hooks/1232318.139712599805568]
HOOKS_LIBRARY_LOADED hooks library
/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_bootp.so successfully loaded
2022-09-21 19:30:04.984 INFO
[kea-dhcp4.ha-hooks/1232318.139712599805568] HA_CONFIGURATION_SUCCESSFUL
HA hook library has been successfully configured
2022-09-21 19:30:04.984 WARN
[kea-dhcp4.ha-hooks/1232318.139712599805568]
HA_CONFIG_LEASE_UPDATES_DISABLED lease updates will not be generated
2022-09-21 19:30:04.984 WARN
[kea-dhcp4.ha-hooks/1232318.139712599805568]
HA_CONFIG_LEASE_SYNCING_DISABLED lease database synchronization between
HA servers is disabled
2022-09-21 19:30:04.984 INFO
[kea-dhcp4.ha-hooks/1232318.139712599805568] HA_INIT_OK loading High
Availability hooks library successful
2022-09-21 19:30:04.984 INFO [kea-dhcp4.hooks/1232318.139712599805568]
HOOKS_LIBRARY_LOADED hooks library
/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so successfully loaded
2022-09-21 19:30:04.985 INFO [kea-dhcp4.dhcp4/1232318.139712599805568]
DHCP4_CONFIG_COMPLETE DHCPv4 server has completed configuration: no IPv4
subnets!; DDNS: disabled
2022-09-21 19:30:04.985 INFO [kea-dhcp4.dhcpsrv/1232318.139712599805568]
DHCPSRV_MEMFILE_DB opening memory file lease database: lfc-interval=3600
max-row-errors=100 name=/var/lib/kea/kea-leases4.csv type=memfile universe=4
2022-09-21 19:30:04.985 INFO [kea-dhcp4.dhcpsrv/1232318.139712599805568]
DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
/var/lib/kea/kea-leases4.csv.2
2022-09-21 19:30:04.985 INFO [kea-dhcp4.dhcpsrv/1232318.139712599805568]
DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
/var/lib/kea/kea-leases4.csv
2022-09-21 19:30:04.985 INFO [kea-dhcp4.dhcpsrv/1232318.139712599805568]
DHCPSRV_MEMFILE_LFC_SETUP setting up the Lease File Cleanup interval to
3600 sec
2022-09-21 19:30:04.986 INFO
[kea-dhcp4.ha-hooks/1232318.139712599805568] HA_LOCAL_DHCP_DISABLE local
DHCP service is disabled while the dhcp-standby is in the WAITING state
2022-09-21 19:30:04.986 INFO
[kea-dhcp4.ha-hooks/1232318.139712599805568] HA_SERVICE_STARTED started
high availability service in hot-standby mode as standby server
------------------------------------------------------------------------------
The problem never occurs when I remove the ha hook library section from
the configuration.
Anyone else seen this problem?
Thanks,
Caciano Machado
PhD in Computer Science and Network Engineer
Universidade Federal do Rio Grande do Sul
More information about the Kea-users
mailing list