Question
Leslie Rhorer
lesrhorer at siliconventures.net
Fri Jun 3 09:33:23 UTC 2022
Oi, veh! Something else has died, now, and I don't know what or
how. The only change I made was the one listed below, and now dhcpd
won't run from /etc/init.d/isc-dhcp-server. Rather, it runs, but then
it quits. I can run it manually from the CL with exactly the same
syntax, and it remains up, but then the primary server quits.
On 6/3/2022 4:03 AM, Leslie Rhorer wrote:
> Well, I found one error left over from when this was a /24 network.
> The range definition on the secondary server was from 192.168.1.220 to
> 192.168.1.240, instead of 192.168.0.200 to 192.168.0.240. You can see
> the error in the backup.conf.gz file. I am not sure what issues this
> would cause, other than of course serving addresses in a range I want
> to change.
>
> On 6/3/2022 2:45 AM, Glenn Satchell wrote:
>> Hi Leslie,
>>
>> Ok I can see a packet flow in that pcap file between the two servers.
>> It shows a TCP packet from 192.168.1.50 port 46869 with the SYN [S]
>> flag to 192.168.1.51 port 647 - so that's trying to open the connection.
>> 192.168.1.51 responds with RST [R] flag, so 192.168.50 tries again,
>> and on it goes. So looks like 192.168.51 is not listening on that
>> port. There's no failover connection being established. So we have
>> that to sort out first.
>>
>> $ tcpdump -r secondary.pcap -v
>> reading from file secondary.pcap, link-type EN10MB (Ethernet)
>> 16:23:34.924575 IP (tos 0x0, ttl 64, id 46213, offset 0, flags [DF],
>> proto TCP (6), length 60)
>> 192.168.1.50.46869 > 192.168.1.51.647: Flags [S], cksum 0xdfce
>> (correct), seq 4009562500, win 64240, options [mss 1460,sackOK,TS val
>> 3809692760 ecr 0,nop,wscale 7], length 0
>> 16:23:34.924599 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF],
>> proto TCP (6), length 40)
>> 192.168.1.51.647 > 192.168.1.50.46869: Flags [R.], cksum 0x71fb
>> (correct), seq 0, ack 4009562501, win 0, length 0
>> 16:23:39.925032 IP (tos 0x0, ttl 64, id 20478, offset 0, flags [DF],
>> proto TCP (6), length 60)
>> 192.168.1.50.57529 > 192.168.1.51.647: Flags [S], cksum 0x995f
>> (correct), seq 2790876011, win 64240, options [mss 1460,sackOK,TS val
>> 3809697760 ecr 0,nop,wscale 7], length 0
>> 16:23:39.925054 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF],
>> proto TCP (6), length 40)
>> 192.168.1.51.647 > 192.168.1.50.57529: Flags [R.], cksum 0x3f14
>> (correct), seq 0, ack 2790876012, win 0, length 0
>>
>> When I look at it with wireshark it's the same but perhaps shown a
>> little more clearly
>>
>> 1 0.000000 192.168.1.50 192.168.1.51 TCP 74 46869 →
>> 647 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3809692760
>> TSecr=0 WS=128
>> 2 0.000024 192.168.1.51 192.168.1.50 TCP 54 647 →
>> 46869 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
>> 3 5.000457 192.168.1.50 192.168.1.51 TCP 74 57529 →
>> 647 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3809697760
>> TSecr=0 WS=128
>> 4 5.000479 192.168.1.51 192.168.1.50 TCP 54 647 →
>> 57529 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
>> 5 10.000924 192.168.1.50 192.168.1.51 TCP 74 51935 →
>> 647 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3809702760
>> TSecr=0 WS=128
>> 6 10.000945 192.168.1.51 192.168.1.50 TCP 54 647 →
>> 51935 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
>> 7 15.001390 192.168.1.50 192.168.1.51 TCP 74 57497 →
>> 647 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3809707761
>> TSecr=0 WS=128
>>
>> Can you please post the failover peer definitions for both dhcp
>> servers, I think we need to check that they make sense. Second the
>> interface configs for that interface on each server, output from "ip
>> addr show ethX" or whatever the correct interface name is please. We
>> need to be sure the address, netmask, etc, match up.
>>
>> So that packet capture is very useful. It's pin pointed an issue
>> straight away.
>>
>> regards,
>> Glenn
>>
>> On 2022-06-03 16:37, Leslie Rhorer wrote:
>>> I am seeing a listening connection on the primary server on 647,
>>> but nothing on the secondary. I have included the tcdump from the
>>> secondary on port 647 as a gz file. 'Still waiting on the dumps on
>>> ports 67 and 68 (it's taking a while for 100 packets to pass)
>>>
>>> On 6/3/2022 1:03 AM, Glenn Satchell wrote:
>>>> Hi Leslie,
>>>>
>>>> I know about capturing packets on a 10G interface :) many gigabytes
>>>> in a few seconds...
>>>>
>>>> So you need to use filters when capturing, eg with tcpdump
>>>>
>>>> tcpdump -i eth0 host <other dhcp server IP or name> and tcp port 647
>>>>
>>>> will only capture the failover traffic on eth0 directed to or from
>>>> the other server, and ignore the rest.
>>>>
>>>> tcpdump udp and port 68 or port 67
>>>>
>>>> will capture dhcp packets.
>>>>
>>>> You can add options like "-c 100" to stop after 100 packets are
>>>> captured. "-w filename" will capture to a file and you can copy
>>>> this file to your desktop and use wireshark to read it.
>>>>
>>>> With failover, it's better to restart one dhcp server, wait for it
>>>> to sync, then restart the other one. If you shut down both and then
>>>> start them, then they come up in recover mode.
>>>>
>>>> Also looking at failover connections:
>>>>
>>>> netstat -ant | grep 647
>>>>
>>>> should show an established connection between the two servers.
>>>>
>>>> regards,
>>>> Glenn
>>>>
>>>> On 2022-06-03 15:39, Leslie Rhorer wrote:
>>>>
>>>>> On 6/2/2022 11:30 PM, Gregory Sloop wrote:
>>>>>
>>>>>> Are you seeing balance messages every hour as the two re-balance
>>>>>> the available lease pool?
>>>>> No, I don't think so. It has only been a couple of hours since I
>>>>> have had both online, however.
>>>>>
>>>>>> You say they are both handling leases properly, but how do you
>>>>>> know this? (That a machine gets a lease from somewhere is not
>>>>>> good evidence.)
>>>>>
>>>>> Do you mean because some other machine / device could be issuing
>>>>> leases? No. In that case,
>>>>>
>>>>> 1. Killing both servers would not take down any DHCP clients. If
>>>>> both servers are shut down, DHCP clients start failing in about an
>>>>> hour, until they are all dead.
>>>>>
>>>>> 2. DHCP responses on the LAN stop completely the moment both
>>>>> servers are taken down.
>>>>>
>>>>> 3. No other machine would know anything about the list of
>>>>> dynamically assigned fixed IP addresses in dhcpd.static. None of
>>>>> the addresses of any of the clients ever change.
>>>>>
>>>>> 4. Whenever one server is shut down, the other responds with tons
>>>>> of responses in the log.
>>>>>
>>>>>> A packet capture in front of the secondary might be helpful to
>>>>>> see what traffic is passing - both to the peer and to clients.
>>>>> While not impossible, that is a bit easier said than done. The
>>>>> links between the servers are 10G. I can look into it.
>>>>>
>>>>>> (I hate making captures, at least as much as the next person, but
>>>>>> dang if they don't, nearly always, show something that was
>>>>>> different than I assumed. So, I've just gotten a lot less averse
>>>>>> to getting captures. Yeah, they'll probably take me extra time to
>>>>>> setup and get and paw through, [all when I could be fixin'
>>>>>> stuff!] but they can save hours or days of fruitless searching
>>>>>> for a fix, when I don't even really *know* what's wrong yet.
>>>>>> Don't know about anyone else, but fixing problems gets a whole
>>>>>> lot easier when I actually know what's wrong, or at least have a
>>>>>> good idea what's going on. :)
>>>>>
>>>>> Agreed, although when an interface is chunking away at over 10,000
>>>>> packets per second...
>>>>>
>>>>> If something doesn't break loose, I will see about loading Wireshark.
>
More information about the dhcp-users
mailing list