Log messages disappear
Glenn Satchell
glenn.satchell at uniq.com.au
Fri Jun 15 12:43:31 UTC 2012
Hi Niall
In your dhcpd chroot do you have the log device, /local/var/dhcpd/dev/log?
And is rsyslog configured to listen on that socket as well as /dev/log?
You've jumped a lot of versions of dhcpd, syslog and the kernel, so it's
not clear where the issue might be. Do you use selinux at al?
I wonder, does the paranoia feature change the way logging works? Normally
dhcpd opens the config file, /dev/log, etc, then chroots. Perhaps if any
of those services "go away" dhcpd is stuck in it's chroot and can't open
the files/sockets outside the chroot again? I don't know for sure, but
making sime guesses here to perhaps prompt someone who does know. A bit of
source code perusal may be needed... Maybe try strace on the dhcpd process
after doing a syslog restart?
regards,
-glenn
> During preparation for some server upgrades, I've encountered a problem
> with logging. At a certain point, the log messages from dhcpd no longer
> appear in the log file. This seems to be whenever the syslog daemon is
> reloaded (nightly logrotate, manual -HUP, or whatever). Subsequently
> reloading dhcpd appears to clear the problem, so that messages once again
> appear as expected.
>
> Some relevant details follow below. I'm sure I can forge a solution,
> but I'ld prefer to take advantage of any relevant prior experience to
> save myself some "hammer-and-bellows" work. I'm not looking for
> extended help, but rather for indications such as "Yup, we had that,
> we did X ...", or even (WIBN) "Send in a bug report".
>
> Old server environment -- AFAICS without problem
> hardware platform i386
> RHEL 3.4
> syslogd 1.4.1 (bundled with distro)
> ISC dhcpd 3.1.0 built from source with paranoia feature
> invoked as /usr/sbin/dhcpd -user dhcpd -chroot /local/var/dhcpd/
> default logging configuration (dwz: no logging configured in dhcpd.conf)
>
> New server environment -- with problem
> hardware platform x86_64
> RHEL 6.0
> rsyslogd 4.6.2 (bundled with distro)
> ISC dhcpd 4.2.3-P2 built from source with paranoia feature
> invoked as /usr/local/sbin/dhcpd -user dhcpd -chroot /local/var/dhcpd
> default logging configuration
>
> On the new server, when log messages are no longer appearing, lsof is
> telling me
>
> dhcpd 24138 dhcpd cwd DIR 253,2 4096
> 1048595 /local/var/dhcpd
> dhcpd 24138 dhcpd rtd DIR 253,2 4096
> 1048595 /local/var/dhcpd
> dhcpd 24138 dhcpd txt REG 253,1 6710190
> 295585 /usr/local/sbin/dhcpd
> dhcpd 24138 dhcpd mem REG 253,1 150672
> 295432 /lib64/ld-2.12.so
> dhcpd 24138 dhcpd mem REG 253,1 1838296
> 295433 /lib64/libc-2.12.so
> dhcpd 24138 dhcpd mem REG 253,1 61624
> 264731 /lib64/libnss_files-2.12.so
> dhcpd 24138 dhcpd 0u CHR 1,3 0t0
> 1049796 /local/var/dhcpd/dev/null
> dhcpd 24138 dhcpd 1u CHR 1,3 0t0
> 1049796 /local/var/dhcpd/dev/null
> dhcpd 24138 dhcpd 2u CHR 1,3 0t0
> 1049796 /local/var/dhcpd/dev/null
> dhcpd 24138 dhcpd 4u raw 0t0
> 1937623 00000000:0001->00000000:0000 st=07
> dhcpd 24138 dhcpd 5u pack 1937628 0t0
> ALL type=SOCK_PACKET
> dhcpd 24138 dhcpd 6w REG 253,2 67814
> 1048648 /local/var/dhcpd/db/dhcpd.leases
> dhcpd 24138 dhcpd 7u IPv4 1937630 0t0
> UDP *:bootps
> dhcpd 24138 dhcpd 20u IPv4 1937616 0t0
> UDP *:47430
> dhcpd 24138 dhcpd 21u IPv6 1937617 0t0
> UDP *:57820
>
> Under normal conditions, I see the following
>
> dhcpd 6090 dhcpd cwd DIR 253,2 4096
> 1048595 /local/var/dhcpd
> dhcpd 6090 dhcpd rtd DIR 253,2 4096
> 1048595 /local/var/dhcpd
> dhcpd 6090 dhcpd txt REG 253,1 6710190
> 295585 /usr/local/sbin/dhcpd
> dhcpd 6090 dhcpd mem REG 253,1 150672
> 295432 /lib64/ld-2.12.so
> dhcpd 6090 dhcpd mem REG 253,1 1838296
> 295433 /lib64/libc-2.12.so
> dhcpd 6090 dhcpd mem REG 253,1 61624
> 264731 /lib64/libnss_files-2.12.so
> dhcpd 6090 dhcpd 0u CHR 1,3 0t0
> 1049796 /local/var/dhcpd/dev/null
> dhcpd 6090 dhcpd 1u CHR 1,3 0t0
> 1049796 /local/var/dhcpd/dev/null
> dhcpd 6090 dhcpd 2u CHR 1,3 0t0
> 1049796 /local/var/dhcpd/dev/null
> dhcpd 6090 dhcpd 3u unix 0xffff880294b970c0 0t0
> 2269598 socket
> dhcpd 6090 dhcpd 4u raw 0t0
> 2269603 00000000:0001->00000000:0000 st=07
> dhcpd 6090 dhcpd 5u pack 2269608 0t0
> ALL type=SOCK_PACKET
> dhcpd 6090 dhcpd 6w REG 253,2 26019
> 1048636 /local/var/dhcpd/db/dhcpd.leases
> dhcpd 6090 dhcpd 7u IPv4 2269610 0t0
> UDP *:bootps
> dhcpd 6090 dhcpd 20u IPv4 2269596 0t0
> UDP *:dtpt
> dhcpd 6090 dhcpd 21u IPv6 2269597 0t0
> UDP *:27977
>
> The key difference seems to be that FD 3 is missing under problem
> conditions.
> I expect that this is where log entries should be written.
>
> Ideally, dhcpd would notice this and try to recover. Maybe it does, and
> can't
> succeed on account of the chroot jail. But then we should probably have
> noticed a problem in the old environment.
>
> Otherwise, I expect I'll have to set up triggers for dhcpd in the
> logrotate
> configuration and in the rsyslog init-script.
>
>
> Best regards,
> Niall O'Reilly
>
> _______________________________________________
> dhcp-users mailing list
> dhcp-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/dhcp-users
>
More information about the dhcp-users
mailing list