[Kea-users] perfdhcp error
Vikram Hosakote (vhosakot)
vhosakot at cisco.com
Thu Oct 1 19:45:18 UTC 2015
Hi Tomek, Francis,
This issue is gone after I reloaded the machine! :) May be, a data structure’s
size hit its limit when I ran perfdhcp inside a namespace many times, and this
data structure was not reset/emptied even after uninstalling and re-installing
perfdhcp. Don’t know if this data structure is in perfdhcp's code or in Linux’s
OS code.
When I reloaded the machine, looks like this data structure was finally
reset/emptied and the issue is gone now! :)
Regards,
Vikram Hosakote
OpenStack Software Engineer
vhosakot at cisco.com<mailto:vhosakot at cisco.com>
Cloud and Virtualization Group
Cisco Systems
Boxborough, MA, USA
From: Vikram Hosakote <vhosakot at cisco.com<mailto:vhosakot at cisco.com>>
Date: Monday, September 28, 2015 at 5:36 PM
To: Tomek Mrugalski <tomasz at isc.org<mailto:tomasz at isc.org>>, Marcin Siodelski <marcin at isc.org<mailto:marcin at isc.org>>, "kea-users at lists.isc.org<mailto:kea-users at lists.isc.org>" <kea-users at lists.isc.org<mailto:kea-users at lists.isc.org>>, Francis Dupont <fdupont at isc.org<mailto:fdupont at isc.org>>
Subject: Re: [Kea-users] perfdhcp error
Hi Tomek, Francis,
Hope you guys are doing good!
Just wanted to send an update about this issue (which I think is weird :)) with
no scale. I have just “one" namespace testns-DHCP-network-0. Below is the output
of "sudo ip l” and “ip a” in this namespace.
# sudo ip netns exec testns-DHCP-network-0 sudo ip l
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
227453: tapd2732bf1-97: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether ee:ff:ff:ff:ff:ef brd ff:ff:ff:ff:ff:ff
# sudo ip netns exec testns-DHCP-network-0 sudo ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
227453: tapd2732bf1-97: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether ee:ff:ff:ff:ff:ef brd ff:ff:ff:ff:ff:ff
inet 1.0.0.1/24 brd 1.0.0.255 scope global tapd2732bf1-97
valid_lft forever preferred_lft forever
inet6 fe80::ecff:ffff:feff:ffef/64 scope link
valid_lft forever preferred_lft forever
I see the same error when I run perfdhcp in this namespace. Below is the output
with (with verbose diagnostics enabled). As mentioned, there is only one namespace.
# sudo ip netns exec testns-DHCP-network-0 /usr/local/sbin/perfdhcp -x aeistT -4 -b mac=fa:16:00:00:00:00 -r 10 -R 10 -p 10 -l tapd2732bf1-97
Running: perfdhcp -x aeistT -4 -b mac=fa:16:00:00:00:00 -r 10 -R 10 -p 10 -l tapd2732bf1-97
IPv4
lease-type=address-only (IA_NA option added to the client's request)
rate[1/s]=10
clients=10
base[0]=mac=fa:16:00:00:00:00
test-period=10
drop-time[0]=1
drop-time[1]=1
aggressivity=1
broadcast
elp-offset=-1
sid-offset=-1
rip-offset=-1
diagnostic-selectors=aeistT
interface=tapd2732bf1-97
server=255.255.255.255
Set MAC to fa::16::00::00::00::00
Set DUID to 000100011d9c6933000c01020304
Error running perfdhcp: There is no address for interface: tapd2732bf1-97, port: 67, address family: AF_INET
As we can see in the output of "sudo ip netns exec testns-DHCP-network-0 sudo ip a”
above, the interface tapd2732bf1-97 clearly has an IP address (1.0.0.1/24).
This used to work before. I uninstalled kea and perfdhcp and re-installed them,
and still see the same issue. Have I hit a limit somewhere in code when perfdhcp
is run inside a namespace ? Doesn’t re-installation of perfdhcp fix it ?
Looks like I am hitting line # 743 in the source code kea/src/lib/dhcp/iface_mgr.cc on GitHub.
Am I looking at the right code ?
https://github.com/isc-projects/kea/blob/637a79b28430718cc1fd76f24dfa1951e57b723a/src/lib/dhcp/iface_mgr.cc#L743
In kea/src/lib/dhcp/iface_mgr.cc, should any data structure be reset/emptied (if
full) when the code in lines 727-731 (below) is executed in a namespace ?
if (addr_it->get().getFamily() == family) {
// We have interface and address so let's open socket.
// This may cause isc::Unexpected exception.
return (openSocket(iface->getName(), *addr_it, port, false));
}
Also, in kea/src/lib/dhcp/iface_mgr.cc, at line # 744, the value for the ‘address'
variable is not printed in the exception thrown.
isc_throw(SocketConfigError, "There is no address for interface: "
<< ifname << ", port: " << port << ", address "
" family: " << family_name);
https://github.com/isc-projects/kea/blob/637a79b28430718cc1fd76f24dfa1951e57b723a/src/lib/dhcp/iface_mgr.cc#L744
Also, below is the output of "sudo ip l” in the global/default namespace. The
interfaces in other namespaces are not seen in the output below.
# sudo ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
link/ether c0:67:af:af:72:8f brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000
link/ether c0:67:af:af:72:90 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
link/ether c0:67:af:70:d1:f4 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether c0:67:af:70:d1:f5 brd ff:ff:ff:ff:ff:ff
6: ovs-system: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether a2:c0:30:03:f9:78 brd ff:ff:ff:ff:ff:ff
7: br-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether c0:67:af:af:72:8f brd ff:ff:ff:ff:ff:ff
8: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether c0:67:af:70:d1:f4 brd ff:ff:ff:ff:ff:ff
9: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether 82:9f:84:d9:af:46 brd ff:ff:ff:ff:ff:ff
227046: phy-br-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP mode DEFAULT qlen 1000
link/ether ca:ae:04:18:15:e5 brd ff:ff:ff:ff:ff:ff
227047: int-br-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP mode DEFAULT qlen 1000
link/ether c6:30:51:e4:ba:43 brd ff:ff:ff:ff:ff:ff
[root at controller CPNR_scaling(keystone_admin)]#
Am I hitting a perfdhcp bug (related to running perfdhcp in a namespace), or
I am doing something wrong ?
Regards,
Vikram Hosakote
OpenStack Software Engineer
vhosakot at cisco.com<mailto:vhosakot at cisco.com>
Cloud and Virtualization Group
Cisco Systems
Boxborough, MA, USA
From: Vikram Hosakote <vhosakot at cisco.com<mailto:vhosakot at cisco.com>>
Date: Wednesday, September 2, 2015 at 4:24 PM
To: Tomek Mrugalski <tomasz at isc.org<mailto:tomasz at isc.org>>, Marcin Siodelski <marcin at isc.org<mailto:marcin at isc.org>>, "kea-users at lists.isc.org<mailto:kea-users at lists.isc.org>" <kea-users at lists.isc.org<mailto:kea-users at lists.isc.org>>, Francis Dupont <fdupont at isc.org<mailto:fdupont at isc.org>>
Subject: Re: [Kea-users] perfdhcp error
Hi Tomek, Francis,
Thanks a lot for your replies!
Yes, each namespace in Linux creates an isolated set of networking resources
like interfaces, IPv6/IPv6 protocol stack, routing tables, port numbers (sockets)
etc.
http://man7.org/linux/man-pages/man7/namespaces.7.html
"sudo ip l” in the default/global namespace does not show interfaces in other
namespaces. Below is the output in the default/global namespace.
# sudo ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
link/ether c0:67:af:af:72:8f brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000
link/ether c0:67:af:af:72:90 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP mode DEFAULT qlen 1000
link/ether c0:67:af:70:d1:f4 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
link/ether c0:67:af:70:d1:f5 brd ff:ff:ff:ff:ff:ff
6: ovs-system: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether a2:c0:30:03:f9:78 brd ff:ff:ff:ff:ff:ff
7: br-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether c0:67:af:af:72:8f brd ff:ff:ff:ff:ff:ff
8: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether c0:67:af:70:d1:f4 brd ff:ff:ff:ff:ff:ff
9: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether 82:9f:84:d9:af:46 brd ff:ff:ff:ff:ff:ff
127930: phy-br-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP mode DEFAULT qlen 1000
link/ether ba:92:1d:02:a6:d4 brd ff:ff:ff:ff:ff:ff
127931: int-br-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP mode DEFAULT qlen 1000
link/ether 1e:10:3b:da:a6:86 brd ff:ff:ff:ff:ff:ff
#
"sudo ip l” in a namespace also does not show interfaces in other namespaces.
It shows just the interfaces in a particular namespace. Below is the output from
the first four namespaces (testns-DHCP-network-0….testns-DHCP-network-3)
and the last (2000th) namespace testns-DHCP-network-1999.
# sudo ip netns exec testns-DHCP-network-0 ip l
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
126573: tapa748acb8-a1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether ee:ff:ff:ff:ff:ef brd ff:ff:ff:ff:ff:ff
# sudo ip netns exec testns-DHCP-network-1 ip l
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
127893: tapfa1a4b40-8c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether ee:ff:ff:ff:ff:ef brd ff:ff:ff:ff:ff:ff
# sudo ip netns exec testns-DHCP-network-2 ip l
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
127274: tap109d0256-f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether ee:ff:ff:ff:ff:ef brd ff:ff:ff:ff:ff:ff
# sudo ip netns exec testns-DHCP-network-3 ip l
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
127362: tap4ddbb28d-02: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether ee:ff:ff:ff:ff:ef brd ff:ff:ff:ff:ff:ff
# sudo ip netns exec testns-DHCP-network-1999 ip l
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
126459: tap9283904c-19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether ee:ff:ff:ff:ff:ef brd ff:ff:ff:ff:ff:ff
An interface in one namespace can ping/reach an interface in another
namespace only if there is a layer-2 bridge (like OpenSwitch I’m using) that
bridges both the interfaces so that inter-namespace communication is
possible.
I increased SNDBUF_SIZE and RCVBUF_SIZE in src/lib/dhcp/iface_mgr_linux.cc
to a large value (256K = 256 * 1024 = 262144), rebuilt perfdhcp, and still see the
same error. SNDBUF_SIZE and RCVBUF_SIZE were 32768 before.
const static size_t SNDBUF_SIZE = 262144;
const static size_t RCVBUF_SIZE = 262144;
Also, I see the same error when I run perfdhcp in the first few namespaces (
testns-DHCP-network-0….testns-DHCP-network-3) and the last few namespaces (
testns-DHCP-network-1990….testns-DHCP-network-1999).
Below is the same error when I run perfdhcp in the first namespace
testns-DHCP-network-0 after increasing SNDBUF_SIZE and RCVBUF_SIZE to
262144 and rebuilding perfdhcp.
# sudo ip netns exec testns-DHCP-network-0 /usr/local/sbin/perfdhcp -x aiestT -4 -b mac=fa:16:00:00:00:00 -r 5 -R 5 -p 10 -l tapa748acb8-a1
Running: perfdhcp -x aiestT -4 -b mac=fa:16:00:00:00:00 -r 5 -R 5 -p 10 -l tapa748acb8-a1
IPv4
lease-type=address-only (IA_NA option added to the client's request)
rate[1/s]=5
clients=5
base[0]=mac=fa:16:00:00:00:00
test-period=10
drop-time[0]=1
drop-time[1]=1
aggressivity=1
broadcast
elp-offset=-1
sid-offset=-1
rip-offset=-1
diagnostic-selectors=aiestT
interface=tapa748acb8-a1
server=255.255.255.255
Set MAC to fa::16::00::00::00::00
Set DUID to 000100011d7a145a000c01020304
Error running perfdhcp: There is no address for interface: tapa748acb8-a1, port: 67, address family: AF_INET
#
Regards,
Vikram Hosakote
OpenStack Software Engineer
vhosakot at cisco.com<mailto:vhosakot at cisco.com>
Cloud and Virtualization Group
Cisco Systems
Boxborough, MA, USA
From: Tomek Mrugalski <tomasz at isc.org<mailto:tomasz at isc.org>>
Date: Wednesday, September 2, 2015 at 6:33 AM
To: Vikram Hosakote <vhosakot at cisco.com<mailto:vhosakot at cisco.com>>, Marcin Siodelski <marcin at isc.org<mailto:marcin at isc.org>>, "kea-users at lists.isc.org<mailto:kea-users at lists.isc.org>" <kea-users at lists.isc.org<mailto:kea-users at lists.isc.org>>
Subject: Re: [Kea-users] perfdhcp error
On 02/09/15 07:32, Vikram Hosakote (vhosakot) wrote:
I have been running perfdhcp using an interface inside a Linux namespace
successfully. I have also been able to successfully run multiple
instances of perfdhcp using interfaces in multiple namespaces simultaneously.
Cool. Thanks for sharing this info.
But, when I have 2,000 namespaces and run perfdhcp inside each of these
namespaces, I see that perfdhcp throws the following error even though the
interface used by perfdhcp has an IP address. This error is NOT seen when I
run perfdhcp instances in a few (100-200) namespaces but is seen when I
run them in 2,000 namespaces.
I'm not familiar with the namespaces you mention, but I presume each
namespace creates its own network interface that is visible through ip,
right?
So when creating 2000 namespaces, how many network interfaces are
reported by sudo ip l? I think you may have hit the boundary of our
netlink interface that assumes that the information about interfaces
will not be larger than 32KB. If my theory is right, the info provided
by the kernel is larger than the buffer we provide (32K), so it's
getting truncated and we parse only the first 32KB of data. That means
that if you have a lot (2000 you mentioned definitely falls into that
category)
of interfaces, only the initial couple hundred would be visible by
perfdhcp (or kea for that matter, as they share libdhcp++).
Can you do an experiment? Please configure 2000 namespaces and get a
list interfaces: ip l. Try to run perfdhcp on one of the first 50 or so
interfaces. If my theory is right, it should work. Now try to run it on
one of the last interfaces returned by ip l. It should fail.
If that is so, I have an easy fix for you. Open
src/lib/dhcp/iface_mgr_linux.cc and see lines 123, 125. There are
SNDBUF_SIZE and RCVBUF_SIZE. Please increase RCVBUF_SIZE and see if it
helps.
Tomek
Error running perfdhcp: There is no address for interface:
tapa748acb8-a1, port: 67, address family: AF_INET
Below is the interface tapa748acb8-a1 (inside the namespace
testns-DHCP-network-0) that has the IP address 1.0.0.1. The interface is up.
# sudo ip netns exec testns-DHCP-network-0 ifconfig
tapa748acb8-a1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 1.0.0.1 netmask 255.255.255.0 broadcast 1.0.0.255
ether ee:ff:ff:ff:ff:ef txqueuelen 0 (Ethernet)
RX packets 138 bytes 9280 (9.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 45 bytes 3590 (3.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Below is the perfdhcp output (with verbose diagnostics enabled). Could you
please let me know what the issue is. What am I missing ? Is this a
scale/concurrency issue ? The same exact perfdhcp command below works
perfectly fine when I have a few (100-200) namespaces and run perfdhcp
instances using interfaces in them.
# sudo ip netns exec testns-DHCP-network-0 /usr/local/sbin/perfdhcp -x
aeistT -4 -b mac=fa:16:00:00:00:00 -r 5 -R 5 -p 10 -l tapa748acb8-a1
Running: perfdhcp -x aeistT -4 -b mac=fa:16:00:00:00:00 -r 5 -R 5 -p 10
-l tapa748acb8-a1
IPv4
lease-type=address-only (IA_NA option added to the client's request)
rate[1/s]=5
clients=5
base[0]=mac=fa:16:00:00:00:00
test-period=10
drop-time[0]=1
drop-time[1]=1
aggressivity=1
broadcast
elp-offset=-1
sid-offset=-1
rip-offset=-1
diagnostic-selectors=aeistT
interface=tapa748acb8-a1
server=255.255.255.255
Set MAC to fa::16::00::00::00::00
Set DUID to 000100011d793ce4000c01020304
Error running perfdhcp: There is no address for interface:
tapa748acb8-a1, port: 67, address family: AF_INET
#
Regards,
Vikram Hosakote
OpenStack Software Engineer
vhosakot at cisco.com<mailto:vhosakot at cisco.com> <mailto:vhosakot at cisco.com>
Cloud and Virtualization Group
Cisco Systems
Boxborough, MA, USA
_______________________________________________
Kea-users mailing list
Kea-users at lists.isc.org<mailto:Kea-users at lists.isc.org>
https://lists.isc.org/mailman/listinfo/kea-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/kea-users/attachments/20151001/49675e70/attachment.htm>
More information about the Kea-users
mailing list