Skip to contentSkip to conversation

iproute: Faster ip link add, set and delete

RFC PATCH In review Eric W. Biederman 5 participants 32 messages Mar 22, 2013 – Mar 30, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 iproute: Faster ip link add, set and delete

Because ip link add, set, and delete map the interface name to the interface index by dumping all of the interfaces before performing their respective commands. Operations that should be constant time slow down when lots of network interfaces are in use. Resulting in O(N^2) time to work with O(N) devices.

Make the work that iproute does constant time by passing the interface name to the kernel instead.

In small scale testing on my system this shows dramatic performance increases of ip link add from 120s to just 11s to add 5000 network devices. And from longer than I cared to wait to just 58s to delete all of those interfaces again.

---

I think I am bungling the case where people specify an ifindex as ifNNNN but does anyone care?

 ip/iplink.c |   19 +------------------
 1 files changed, 1 insertions(+), 18 deletions(-)
ip/iplink.c +1 -18
index ad33611..6dffbf0 100644
@@ -533,8 +533,6 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)
 		}
 	}
 
-	ll_init_map(&rth);
-
 	if (!(flags & NLM_F_CREATE)) {
 		if (!dev) {
 			fprintf(stderr, "Not enough information: \"dev\" "
@@ -542,27 +540,12 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)
 			exit(-1);
 		}
 
-		req.i.ifi_index = ll_name_to_index(dev);
-		if (req.i.ifi_index == 0) {
-			fprintf(stderr, "Cannot find device \"%s\"\n", dev);
-			return -1;
-		}
+		name = dev;
 	} else {
 		/* Allow "ip link add dev" and "ip link add name" */
 		if (!name)
 			name = dev;
 
-		if (link) {
-			int ifindex;
-
-			ifindex = ll_name_to_index(link);
-			if (ifindex == 0) {
-				fprintf(stderr, "Cannot find device \"%s\"\n",
-					link);
-				return -1;
-			}
-			addattr_l(&req.n, sizeof(req), IFLA_LINK, &ifindex, 4);
-		}
 	}
 
 	if (name) {
Signature · 1 line
1.7.5.4
#1 lore raw

The whole ifindex map is a design mistake at this point. Better off to do a lazy cache or something like that.

Quoted 71 lines · Eric W. Biederman

Because ip link add, set, and delete map the interface name to the interface index by dumping all of the interfaces before performing their respective commands. Operations that should be constant time slow down when lots of network interfaces are in use. Resulting in O(N^2) time to work with O(N) devices.

Make the work that iproute does constant time by passing the interface name to the kernel instead.

In small scale testing on my system this shows dramatic performance increases of ip link add from 120s to just 11s to add 5000 network devices. And from longer than I cared to wait to just 58s to delete all of those interfaces again.

---

I think I am bungling the case where people specify an ifindex as ifNNNN but does anyone care?

 ip/iplink.c |   19 +------------------
 1 files changed, 1 insertions(+), 18 deletions(-)
ip/iplink.c +1 -19
index ad33611..6dffbf0 100644
@@ -533,8 +533,6 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)
                }
        }
 
-       ll_init_map(&rth);
-
        if (!(flags & NLM_F_CREATE)) {
                if (!dev) {
                        fprintf(stderr, "Not enough information: \"dev\" "
@@ -542,27 +540,12 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)
                        exit(-1);
                }
 
-               req.i.ifi_index = ll_name_to_index(dev);
-               if (req.i.ifi_index == 0) {
-                       fprintf(stderr, "Cannot find device \"%s\"\n", dev);
-                       return -1;
-               }
+               name = dev;
        } else {
                /* Allow "ip link add dev" and "ip link add name" */
                if (!name)
                        name = dev;
 
-               if (link) {
-                       int ifindex;
-
-                       ifindex = ll_name_to_index(link);
-                       if (ifindex == 0) {
-                               fprintf(stderr, "Cannot find device \"%s\"\n",
-                                       link);
-                               return -1;
-                       }
-                       addattr_l(&req.n, sizeof(req), IFLA_LINK, &ifindex, 4);
-               }
        }
 
        if (name) {
--

1.7.5.4

#2 lore raw

Hello,

I re-tested with the patch and got the following results on a 32x 2Ghz core system.

# veth 	add 	delete
1000 	36 	34
3000 	259 	137
4000 	462 	195
5000 	729     N/A

The script to create is the following :
for i in seq 1 5000; do
sudo ip link add type veth
Done

The script to delete:
for d in /sys/class/net/veth*; do

	ip link del `basename $d` 2>/dev/null || true

Done

There is a very good improvement in deletion.

iproute2 does not seems to be well multithread as I get time divided by a factor of 2 with a 8x 3.2 Ghz core system.

I don¹t know if that is the improvement you expected ?

Would the iproute2 redesign you mentioned help improve performance even further ?

As a reference : Iproute2 baseline w/o patch:

# veth 	add 	delete

1000 57 70
2000 193 250
3000 435 510
4000 752 824
5000 1123 1185

Regards

Benoit

Quoted 84 lines · Stephen Hemminger

The whole ifindex map is a design mistake at this point. Better off to do a lazy cache or something like that.

Quoted 77 lines · Eric W. Biederman

Because ip link add, set, and delete map the interface name to the interface index by dumping all of the interfaces before performing their respective commands. Operations that should be constant time slow down when lots of network interfaces are in use. Resulting in O(N^2) time to work with O(N) devices.

Make the work that iproute does constant time by passing the interface name to the kernel instead.

In small scale testing on my system this shows dramatic performance increases of ip link add from 120s to just 11s to add 5000 network devices. And from longer than I cared to wait to just 58s to delete all of those interfaces again.

---

I think I am bungling the case where people specify an ifindex as ifNNNN but does anyone care?

 ip/iplink.c |   19 +------------------
 1 files changed, 1 insertions(+), 18 deletions(-)
ip/iplink.c +0 -2
index ad33611..6dffbf0 100644
@@ -533,8 +533,6 @@ static int iplink_modify(int cmd, unsigned int
 flags, int argc, char **argv)
                }
        }
 
-       ll_init_map(&rth);
-
        if (!(flags & NLM_F_CREATE)) {
                if (!dev) {

fprintf(stderr, "Not enough information:
\"dev\" "

ip/iplink.c +1 -15
@@ -542,27 +540,12 @@ static int iplink_modify(int cmd, unsigned int
 flags, int argc, char **argv)
                        exit(-1);
                }
 
-               req.i.ifi_index = ll_name_to_index(dev);
-               if (req.i.ifi_index == 0) {
-                       fprintf(stderr, "Cannot find device \"%s\"\n",
 dev);
-                       return -1;
-               }
+               name = dev;
        } else {
                /* Allow "ip link add dev" and "ip link add name" */
                if (!name)
                        name = dev;
 
-               if (link) {
-                       int ifindex;
-
-                       ifindex = ll_name_to_index(link);
-                       if (ifindex == 0) {
-                               fprintf(stderr, "Cannot find device
 \"%s\"\n",
-                                       link);
-                               return -1;
-                       }
-                       addattr_l(&req.n, sizeof(req), IFLA_LINK,

&ifindex, 4);
- }

        }

        if (name) {

--
1.7.5.4

#3 lore raw
Quoted 11 lines · Benoit Lourdelet

Hello,

I re-tested with the patch and got the following results on a 32x 2Ghz core system.

# veth 	add 	delete
1000 	36 	34
3000 	259 	137
4000 	462 	195
5000 	729     N/A
Quoted 4 lines

The script to create is the following :
for i in seq 1 5000; do
sudo ip link add type veth
Done

Which performs horribly as I mentioned earlier because you are asking the kernel to create the names. If you want performance you need to specify the names of the network devices you are creating.

aka ip link add a$i type veth name b$i

Quoted 11 lines

The script to delete:
for d in /sys/class/net/veth*; do

	ip link del `basename $d` 2>/dev/null || true

Done

There is a very good improvement in deletion.

iproute2 does not seems to be well multithread as I get time divided by a factor of 2 with a 8x 3.2 Ghz core system.

All netlink traffic and all network stack configuration is serialized by the rtnl_lock in the kernel. This is the slow path in the kernel, not the fast path.

Quoted 4 lines

I don¹t know if that is the improvement you expected ?

Would the iproute2 redesign you mentioned help improve performance even further ?

Specifing the names would dramatically improve your creation performance. It should only take you about 10s for 5000 veth pairs. But you have to specify the names.

Anyway I have exhausted my time, and inclination in this matter. Good luck with whatever your problem is.

Eric

#4 lore raw
Quoted 3 lines · Eric W. Biederman

Specifing the names would dramatically improve your creation performance. It should only take you about 10s for 5000 veth pairs. But you have to specify the names.

Thanks, Eric. I'm going to update lxc to always specify names for the veth pairs, rather than only when they are requested by the user's configuration file.

-serge

#5 lore raw
In reply to Eric W. Biederman

Actually, lxc is using random names now, so it's ok.

Benoit, can you use the patches from Eric with lxc (or use the script you were using before but specify names as he said)?

-serge

#6 lore raw

Hello Serge,

I am indeed using Eric patch with lxc.

It solves the initial problem of slowness to start around 1600 containers.

I am now able to start more than 2000 without having new containers slower and slower to start.

thanks

Benoit

Quoted 7 lines · Serge Hallyn

Actually, lxc is using random names now, so it's ok.

Benoit, can you use the patches from Eric with lxc (or use the script you were using before but specify names as he said)?

-serge

#7 lore raw
Quoted 6 lines · Benoit Lourdelet

Hello Serge,

I am indeed using Eric patch with lxc.

It solves the initial problem of slowness to start around 1600 containers.

Good so now we just need a production ready patch for iproute.

Quoted 2 lines

I am now able to start more than 2000 without having new containers slower and slower to start.

May I ask how large a box you are running and how complex your containers are. I am trying to get a feel for how common it is likely to be to find people running thousands of containers on a single machine.

Eric

#8 lore raw

If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.

I have an updated version of link map hash (index and name). Could you test this patch which applies to latest version in git.

lib/ll_map.c +128 -50
index e9ae129..bf5b0bc 100644
@@ -12,6 +12,7 @@
 
 #include <stdio.h>
 #include <stdlib.h>
+#include <stddef.h>
 #include <unistd.h>
 #include <syslog.h>
 #include <fcntl.h>
@@ -23,9 +24,44 @@
 #include "libnetlink.h"
 #include "ll_map.h"
 
-struct ll_cache
+
+struct hlist_head {
+	struct hlist_node *first;
+};
+
+struct hlist_node {
+	struct hlist_node *next, **pprev;
+};
+
+static inline void hlist_del(struct hlist_node *n)
+{
+	struct hlist_node *next = n->next;
+	struct hlist_node **pprev = n->pprev;
+	*pprev = next;
+	if (next)
+		next->pprev = pprev;
+}
+
+static inline void hlist_add_head(struct hlist_node *n, struct hlist_head *h)
 {
-	struct ll_cache   *idx_next;
+	struct hlist_node *first = h->first;
+	n->next = first;
+	if (first)
+		first->pprev = &n->next;
+	h->first = n;
+	n->pprev = &h->first;
+}
+
+#define hlist_for_each(pos, head) \
+	for (pos = (head)->first; pos ; pos = pos->next)
+
+#define container_of(ptr, type, member) ({			\
+	const typeof( ((type *)0)->member ) *__mptr = (ptr);	\
+	(type *)( (char *)__mptr - offsetof(type,member) );})
+
+struct ll_cache {
+	struct hlist_node idx_hash;
+	struct hlist_node name_hash;
 	unsigned	flags;
 	int		index;
 	unsigned short	type;
@@ -33,49 +69,107 @@ struct ll_cache
 };
 
 #define IDXMAP_SIZE	1024
-static struct ll_cache *idx_head[IDXMAP_SIZE];
+static struct hlist_head idx_head[IDXMAP_SIZE];
+static struct hlist_head name_head[IDXMAP_SIZE];
 
-static inline struct ll_cache *idxhead(int idx)
+static struct ll_cache *ll_get_by_index(unsigned index)
 {
-	return idx_head[idx & (IDXMAP_SIZE - 1)];
+	struct hlist_node *n;
+	unsigned h = index & (IDXMAP_SIZE - 1);
+
+	hlist_for_each(n, &idx_head[h]) {
+		struct ll_cache *im
+			= container_of(n, struct ll_cache, idx_hash);
+		if (im->index == index)
+			return im;
+	}
+
+	return NULL;
+}
+
+static unsigned namehash(const char *str)
+{
+	unsigned hash = 5381;
+
+	while (*str)
+		hash = ((hash << 5) + hash) + *str++; /* hash * 33 + c */
+
+	return hash;
+}
+
+static struct ll_cache *ll_get_by_name(const char *name)
+{
+	struct hlist_node *n;
+	unsigned h = namehash(name) & (IDXMAP_SIZE - 1);
+
+	hlist_for_each(n, &name_head[h]) {
+		struct ll_cache *im
+			= container_of(n, struct ll_cache, name_hash);
+
+		if (strncmp(im->name, name, IFNAMSIZ) == 0)
+			return im;
+	}
+
+	return NULL;
 }
 
 int ll_remember_index(const struct sockaddr_nl *who,
 		      struct nlmsghdr *n, void *arg)
 {
-	int h;
+	unsigned int h;
+	const char *ifname;
 	struct ifinfomsg *ifi = NLMSG_DATA(n);
-	struct ll_cache *im, **imp;
+	struct ll_cache *im;
 	struct rtattr *tb[IFLA_MAX+1];
 
-	if (n->nlmsg_type != RTM_NEWLINK)
+	if (n->nlmsg_type != RTM_NEWLINK && n->nlmsg_type != RTM_DELLINK)
 		return 0;
 
 	if (n->nlmsg_len < NLMSG_LENGTH(sizeof(ifi)))
 		return -1;
 
+	im = ll_get_by_index(ifi->ifi_index);
+	if (n->nlmsg_type == RTM_DELLINK) {
+		if (im) {
+			hlist_del(&im->name_hash);
+			hlist_del(&im->idx_hash);
+			free(im);
+		}
+		return 0;
+	}
+
 	memset(tb, 0, sizeof(tb));
 	parse_rtattr(tb, IFLA_MAX, IFLA_RTA(ifi), IFLA_PAYLOAD(n));
-	if (tb[IFLA_IFNAME] == NULL)
+	ifname = rta_getattr_str(tb[IFLA_IFNAME]);
+	if (ifname == NULL)
 		return 0;
 
-	h = ifi->ifi_index & (IDXMAP_SIZE - 1);
-	for (imp = &idx_head[h]; (im=*imp)!=NULL; imp = &im->idx_next)
-		if (im->index == ifi->ifi_index)
-			break;
-
-	if (im == NULL) {
-		im = malloc(sizeof(*im));
-		if (im == NULL)
-			return 0;
-		im->idx_next = *imp;
-		im->index = ifi->ifi_index;
-		*imp = im;
+	if (im) {
+		/* change to existing entry */
+		if (strcmp(im->name, ifname) != 0) {
+			hlist_del(&im->name_hash);
+			h = namehash(ifname) & (IDXMAP_SIZE - 1);
+			hlist_add_head(&im->name_hash, &name_head[h]);
+		}
+
+		im->flags = ifi->ifi_flags;
+		return 0;
 	}
 
+	im = malloc(sizeof(*im));
+	if (im == NULL)
+		return 0;
+	im->index = ifi->ifi_index;
+	strcpy(im->name, ifname);
 	im->type = ifi->ifi_type;
 	im->flags = ifi->ifi_flags;
-	strcpy(im->name, RTA_DATA(tb[IFLA_IFNAME]));
+
+	h = ifi->ifi_index & (IDXMAP_SIZE - 1);
+	hlist_add_head(&im->idx_hash, &idx_head[h]);
+
+	h = namehash(ifname) & (IDXMAP_SIZE - 1);
+	hlist_add_head(&im->name_hash, &name_head[h]);
+
 	return 0;
 }
 
@@ -86,15 +180,14 @@ const char *ll_idx_n2a(unsigned idx, char *buf)
 	if (idx == 0)
 		return "*";
 
-	for (im = idxhead(idx); im; im = im->idx_next)
-		if (im->index == idx)
-			return im->name;
+	im = ll_get_by_index(idx);
+	if (im)
+		return im->name;
 
 	snprintf(buf, IFNAMSIZ, "if%d", idx);
 	return buf;
 }
 
-
 const char *ll_index_to_name(unsigned idx)
 {
 	static char nbuf[IFNAMSIZ];
@@ -108,10 +201,9 @@ int ll_index_to_type(unsigned idx)
 
 	if (idx == 0)
 		return -1;
-	for (im = idxhead(idx); im; im = im->idx_next)
-		if (im->index == idx)
-			return im->type;
-	return -1;
+
+	im = ll_get_by_index(idx);
+	return im ? im->type : -1;
 }
 
 unsigned ll_index_to_flags(unsigned idx)
@@ -121,35 +213,21 @@ unsigned ll_index_to_flags(unsigned idx)
 	if (idx == 0)
 		return 0;
 
-	for (im = idxhead(idx); im; im = im->idx_next)
-		if (im->index == idx)
-			return im->flags;
-	return 0;
+	im = ll_get_by_index(idx);
+	return im ? im->flags : -1;
 }
 
 unsigned ll_name_to_index(const char *name)
 {
-	static char ncache[IFNAMSIZ];
-	static int icache;
-	struct ll_cache *im;
-	int i;
+	const struct ll_cache *im;
 	unsigned idx;
 
 	if (name == NULL)
 		return 0;
 
-	if (icache && strcmp(name, ncache) == 0)
-		return icache;
-
-	for (i=0; i<IDXMAP_SIZE; i++) {
-		for (im = idx_head[i]; im; im = im->idx_next) {
-			if (strcmp(im->name, name) == 0) {
-				icache = im->index;
-				strcpy(ncache, name);
-				return im->index;
-			}
-		}
-	}
+	im = ll_get_by_name(name);
+	if (im)
+		return im->index;
 
 	idx = if_nametoindex(name);
 	if (idx == 0)
#9 lore raw
Quoted 2 lines · Stephen Hemminger

If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.

The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform the work itself.

Quoted 2 lines

I have an updated version of link map hash (index and name). Could you test this patch which applies to latest version in git.

This still dumps all of the interfaces in ll_init_map causing things to slow down noticably.

# with your patch
# time ~/projects/iproute/iproute2/ip/ip link add a4511 type veth peer name b4511

real 0m0.049s
user 0m0.000s
sys 0m0.048s

# With a hack to make ll_map_init a nop.

# time ~/projects/iproute/iproute2/ip/ip link add a4512 type veth peer name b4512

real 0m0.003s
user 0m0.000s
sys 0m0.000s
eric-ThinkPad-X220 6bed4 #

# Without any patches.

# time ~/projects/iproute/iproute2/ip/ip link add a5002 type veth peer name b5002

real 0m0.052s
user 0m0.004s
sys 0m0.044s

So it looks like dumping all of the interfaces is taking 46 miliseconds, longer than otherwise. Causing ip to take nearly an order of magnitude longer to run when there are a lot of interfaces, and causing ip to slow down with each command.

So the ideal situation is probably just to fill in the ll_map on demand instead of up front.

Eric

#10 lore raw
Quoted 8 lines · Eric W. Biederman
Quoted 2 lines · Stephen Hemminger

If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.

The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform the work itself.

fwiw lxc uses netlink to create new veths, and picks random names with mktemp() ahead of time.

-serge

#11 lore raw
Quoted 6 lines · Serge Hallyn
Quoted 5 lines · Eric W. Biederman
Quoted 2 lines · Stephen Hemminger

If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.

Quoted 6 lines
Quoted 3 lines

The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform the work itself.

fwiw lxc uses netlink to create new veths, and picks random names with mktemp() ahead of time.

I am puzzled where does the slownes in iproute2 come into play?

Eric

#12 lore raw
Quoted 8 lines · Eric W. Biederman
Quoted 6 lines · Serge Hallyn
Quoted 5 lines · Eric W. Biederman
Quoted 2 lines · Stephen Hemminger

If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform the work itself.

fwiw lxc uses netlink to create new veths, and picks random names with mktemp() ahead of time.

I am puzzled where does the slownes in iproute2 come into play?

Benoit originally reported slowness when starting >1500 containers. I asked him to run a few manual tests to figure out what was taking the time. Manually creating a large # of veths was an obvious test, and one which showed poorly scaling performance.

May well be there are other things slowing down lxc of course.

-serge

#13 lore raw
Quoted 16 lines · Serge Hallyn
Quoted 15 lines · Eric W. Biederman
Quoted 12 lines · Serge Hallyn
Quoted 8 lines · Eric W. Biederman
Quoted 2 lines · Stephen Hemminger

If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.

The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform the work itself.

fwiw lxc uses netlink to create new veths, and picks random names with mktemp() ahead of time.

Quoted 6 lines
Quoted 1 line

I am puzzled where does the slownes in iproute2 come into play?

Benoit originally reported slowness when starting >1500 containers. I asked him to run a few manual tests to figure out what was taking the time. Manually creating a large # of veths was an obvious test, and one which showed poorly scaling performance.

Apparently iproute is involved somehwere as when he tested with a patched iproute (as you asked him to) the lxc startup slowdown was gone.

Quoted 1 line

May well be there are other things slowing down lxc of course.

The evidence indicates it was iproute being called somewhere...

Eric

#14 lore raw
Quoted 25 lines · Eric W. Biederman
Quoted 16 lines · Serge Hallyn
Quoted 15 lines · Eric W. Biederman
Quoted 12 lines · Serge Hallyn
Quoted 8 lines · Eric W. Biederman
Quoted 2 lines · Stephen Hemminger

If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.

The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform the work itself.

fwiw lxc uses netlink to create new veths, and picks random names with mktemp() ahead of time.

Quoted 6 lines
Quoted 1 line

I am puzzled where does the slownes in iproute2 come into play?

Benoit originally reported slowness when starting >1500 containers. I asked him to run a few manual tests to figure out what was taking the time. Manually creating a large # of veths was an obvious test, and one which showed poorly scaling performance.

Quoted 7 lines

Apparently iproute is involved somehwere as when he tested with a patched iproute (as you asked him to) the lxc startup slowdown was gone.

Quoted 1 line

May well be there are other things slowing down lxc of course.

The evidence indicates it was iproute being called somewhere...

Benoit can you tell us exactly what test you were running when you saw the slowdown was gone?

-serge

#15 lore raw

Hello,

My test consists in starting small containers (10MB of RAM ) each. Each container has 2x physical VLAN interfaces attached.

lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth6.3
lxc.network.name = eth2
lxc.network.hwaddr = 00:50:56:a8:03:03
lxc.network.ipv4 = 192.168.1.1/24
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth7.3
lxc.network.name = eth1
lxc.network.ipv4 = 2.2.2.2/24
lxc.network.hwaddr = 00:50:57:b8:00:01

With initial iproute2 , when I reach around 1600 containers, container creation almost stops.It takes at least 20s per container to start. With patched iproutes2 , I have started 4000 containers at a rate of 1 per second w/o problem. I have 8000 clan interfaces configured on the host (2x 4000).

Regards

Benoit

Quoted 42 lines · Serge Hallyn
Quoted 28 lines · Eric W. Biederman
Quoted 7 lines · Serge Hallyn
Quoted 6 lines · Eric W. Biederman
Quoted 4 lines · Serge Hallyn
Quoted 3 lines · Eric W. Biederman
Quoted 1 line · Stephen Hemminger

If you need to do lots of operations the --batch mode will be

significantly faster.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

One command start and one link map.

The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

work itself.

fwiw lxc uses netlink to create new veths, and picks random names

with

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

mktemp() ahead of time.

I am puzzled where does the slownes in iproute2 come into play?

Benoit originally reported slowness when starting >1500 containers. I asked him to run a few manual tests to figure out what was taking the time. Manually creating a large # of veths was an obvious test, and one which showed poorly scaling performance.

Quoted 7 lines

Apparently iproute is involved somehwere as when he tested with a patched iproute (as you asked him to) the lxc startup slowdown was gone.

Quoted 1 line

May well be there are other things slowing down lxc of course.

The evidence indicates it was iproute being called somewhere...

Benoit can you tell us exactly what test you were running when you saw the slowdown was gone?

-serge

#16 lore raw
Quoted 4 lines · Benoit Lourdelet

Hello,

My test consists in starting small containers (10MB of RAM ) each. Each container has 2x physical VLAN interfaces attached.

Which commands were you using to create/start them?

Quoted 72 lines

lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth6.3
lxc.network.name = eth2
lxc.network.hwaddr = 00:50:56:a8:03:03
lxc.network.ipv4 = 192.168.1.1/24
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth7.3
lxc.network.name = eth1
lxc.network.ipv4 = 2.2.2.2/24
lxc.network.hwaddr = 00:50:57:b8:00:01

With initial iproute2 , when I reach around 1600 containers, container creation almost stops.It takes at least 20s per container to start. With patched iproutes2 , I have started 4000 containers at a rate of 1 per second w/o problem. I have 8000 clan interfaces configured on the host (2x 4000).

Regards

Benoit

Quoted 42 lines · Serge Hallyn
Quoted 28 lines · Eric W. Biederman
Quoted 7 lines · Serge Hallyn
Quoted 6 lines · Eric W. Biederman
Quoted 4 lines · Serge Hallyn
Quoted 3 lines · Eric W. Biederman
Quoted 1 line · Stephen Hemminger

If you need to do lots of operations the --batch mode will be

significantly faster.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

One command start and one link map.

The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

work itself.

fwiw lxc uses netlink to create new veths, and picks random names

with

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

mktemp() ahead of time.

I am puzzled where does the slownes in iproute2 come into play?

Benoit originally reported slowness when starting >1500 containers. I asked him to run a few manual tests to figure out what was taking the time. Manually creating a large # of veths was an obvious test, and one which showed poorly scaling performance.

Quoted 7 lines

Apparently iproute is involved somehwere as when he tested with a patched iproute (as you asked him to) the lxc startup slowdown was gone.

Quoted 1 line

May well be there are other things slowing down lxc of course.

The evidence indicates it was iproute being called somewhere...

Benoit can you tell us exactly what test you were running when you saw the slowdown was gone?

-serge

#17 lore raw

I use, for each container :

lxc-start -n lwb2001 -f /var/lib/lxc/lwb2001/config -d

I created the containers with lxc-ubuntu -n lwb2001

Benoit

Quoted 89 lines · Serge Hallyn
Quoted 4 lines · Benoit Lourdelet

Hello,

My test consists in starting small containers (10MB of RAM ) each. Each container has 2x physical VLAN interfaces attached.

Which commands were you using to create/start them?

Quoted 80 lines

lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth6.3
lxc.network.name = eth2
lxc.network.hwaddr = 00:50:56:a8:03:03
lxc.network.ipv4 = 192.168.1.1/24
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth7.3
lxc.network.name = eth1
lxc.network.ipv4 = 2.2.2.2/24
lxc.network.hwaddr = 00:50:57:b8:00:01

With initial iproute2 , when I reach around 1600 containers, container creation almost stops.It takes at least 20s per container to start. With patched iproutes2 , I have started 4000 containers at a rate of 1 per
second w/o problem. I have 8000 clan interfaces configured on the host (2x
4000).

Regards

Benoit

Quoted 6 lines · Serge Hallyn
Quoted 5 lines · Eric W. Biederman
Quoted 3 lines · Serge Hallyn
Quoted 2 lines · Eric W. Biederman

Serge Hallyn <[email protected]> writes:

Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines · Eric W. Biederman
Quoted 3 lines
Quoted 1 line · Stephen Hemminger

If you need to do lots of operations the --batch mode will be

significantly faster.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

One command start and one link map.

The problem in this case as I understand it is lots of

independent

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

operations. Now maybe lxc should not shell out to ip and

perform

Quoted 4 lines
Quoted 4 lines

the

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

work itself.

fwiw lxc uses netlink to create new veths, and picks random

names

Quoted 6 lines
Quoted 6 lines

with

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

mktemp() ahead of time.

I am puzzled where does the slownes in iproute2 come into play?

Benoit originally reported slowness when starting >1500

containers. I

Quoted 1 line
Quoted 1 line
Quoted 1 line

asked him to run a few manual tests to figure out what was taking

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

time. Manually creating a large # of veths was an obvious test,

and

Quoted 15 lines
Quoted 9 lines
Quoted 1 line

one which showed poorly scaling performance.

Apparently iproute is involved somehwere as when he tested with a patched iproute (as you asked him to) the lxc startup slowdown was gone.

Quoted 1 line

May well be there are other things slowing down lxc of course.

The evidence indicates it was iproute being called somewhere...

Benoit can you tell us exactly what test you were running when you saw the slowdown was gone?

-serge

#18 lore raw

Try the following two patches. It adds a name hash list, and uses Eric's idea to avoid loading map on add/delete operations.

Quoted 1 line

From 0025e5d63d5d1598ab622867834a3bcb9f518f9f Mon Sep 17 00:00:00 2001

From: Stephen Hemminger <[email protected]>
Date: Thu, 28 Mar 2013 14:57:28 -0700
Subject: [PATCH 1/2] ll_map: add name and index hash

Make ll_ functions faster by having a name hash, and allow for deletion. Also, allow them to work without calling ll_init_map.

---
 include/hlist.h  |   56 ++++++++++++++++++++
 include/ll_map.h |    3 +-
 lib/ll_map.c     |  155 ++++++++++++++++++++++++++++++++++--------------------
 3 files changed, 157 insertions(+), 57 deletions(-)
 create mode 100644 include/hlist.h
include/hlist.h new file +56 -0
new file mode 100644
index 0000000..4e8de9e
@@ -0,0 +1,56 @@
+#ifndef __HLIST_H__
+#define __HLIST_H__ 1
+/* Hash list stuff from kernel */
+
+#include <stddef.h>
+
+#define container_of(ptr, type, member) ({			\
+	const typeof( ((type *)0)->member ) *__mptr = (ptr);	\
+	(type *)( (char *)__mptr - offsetof(type,member) );})
+
+struct hlist_head {
+	struct hlist_node *first;
+};
+
+struct hlist_node {
+	struct hlist_node *next, **pprev;
+};
+
+static inline void hlist_del(struct hlist_node *n)
+{
+	struct hlist_node *next = n->next;
+	struct hlist_node **pprev = n->pprev;
+	*pprev = next;
+	if (next)
+		next->pprev = pprev;
+}
+
+static inline void hlist_add_head(struct hlist_node *n, struct hlist_head *h)
+{
+	struct hlist_node *first = h->first;
+	n->next = first;
+	if (first)
+		first->pprev = &n->next;
+	h->first = n;
+	n->pprev = &h->first;
+}
+
+#define hlist_for_each(pos, head) \
+	for (pos = (head)->first; pos ; pos = pos->next)
+
+
+#define hlist_for_each_safe(pos, n, head) \
+	for (pos = (head)->first; pos && ({ n = pos->next; 1; }); \
+	     pos = n)
+
+#define hlist_entry_safe(ptr, type, member) \
+	({ typeof(ptr) ____ptr = (ptr); \
+	   ____ptr ? hlist_entry(____ptr, type, member) : NULL; \
+	})
+
+#define hlist_for_each_entry(pos, head, member)				\
+	for (pos = hlist_entry_safe((head)->first, typeof(*(pos)), member);\
+	     pos;							\
+	     pos = hlist_entry_safe((pos)->member.next, typeof(*(pos)), member))
+
+#endif /* __HLIST_H__ */
include/ll_map.h +2 -1
index c4d5c6d..f1dda39 100644
@@ -3,7 +3,8 @@
 
 extern int ll_remember_index(const struct sockaddr_nl *who,
 			     struct nlmsghdr *n, void *arg);
-extern int ll_init_map(struct rtnl_handle *rth);
+
+extern void ll_init_map(struct rtnl_handle *rth);
 extern unsigned ll_name_to_index(const char *name);
 extern const char *ll_index_to_name(unsigned idx);
 extern const char *ll_idx_n2a(unsigned idx, char *buf);
lib/ll_map.c +99 -56

236 lines not on this page · raw message

Signature · 1 line
1.7.10.4
Quoted 1 line

From f0124b0f0aa0e5b9288114eb8e6ff9b4f8c33ec8 Mon Sep 17 00:00:00 2001

From: Stephen Hemminger <[email protected]>
Date: Thu, 28 Mar 2013 15:17:47 -0700
Subject: [PATCH 2/2] ip: remove unnecessary ll_init_map

Don't call ll_init_map on modify operations Saves significant overhead with 1000's of devices.

---
 ip/ipaddress.c    |    2 --
 ip/ipaddrlabel.c  |    2 --
 ip/iplink.c       |    2 --
 ip/iproute.c      |    6 ------
 ip/xfrm_monitor.c |    2 --
 5 files changed, 14 deletions(-)
ip/ipaddress.c +0 -2
index 149df69..5b9a438 100644
@@ -1365,8 +1365,6 @@ static int ipaddr_modify(int cmd, int flags, int argc, char **argv)
 	if (!scoped && cmd != RTM_DELADDR)
 		req.ifa.ifa_scope = default_scope(&lcl);
 
-	ll_init_map(&rth);
-
 	if ((req.ifa.ifa_index = ll_name_to_index(d)) == 0) {
 		fprintf(stderr, "Cannot find device \"%s\"\n", d);
 		return -1;
ip/ipaddrlabel.c +0 -2
index eb6a48c..1789d9c 100644
@@ -246,8 +246,6 @@ static int ipaddrlabel_flush(int argc, char **argv)
 
 int do_ipaddrlabel(int argc, char **argv)
 {
-	ll_init_map(&rth);
-
 	if (argc < 1) {
 		return ipaddrlabel_list(0, NULL);
 	} else if (matches(argv[0], "list") == 0 ||
ip/iplink.c +0 -2
index 5c7b43c..dc98019 100644
@@ -533,8 +533,6 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)
 		}
 	}
 
-	ll_init_map(&rth);
-
 	if (!(flags & NLM_F_CREATE)) {
 		if (!dev) {
 			fprintf(stderr, "Not enough information: \"dev\" "
ip/iproute.c +0 -6
index 2c2a331..adef774 100644
@@ -970,8 +970,6 @@ static int iproute_modify(int cmd, unsigned flags, int argc, char **argv)
 	if (d || nhs_ok)  {
 		int idx;
 
-		ll_init_map(&rth);
-
 		if (d) {
 			if ((idx = ll_name_to_index(d)) == 0) {
 				fprintf(stderr, "Cannot find device \"%s\"\n", d);
@@ -1265,8 +1263,6 @@ static int iproute_list_flush_or_save(int argc, char **argv, int action)
 	if (do_ipv6 == AF_UNSPEC && filter.tb)
 		do_ipv6 = AF_INET;
 
-	ll_init_map(&rth);
-
 	if (id || od)  {
 		int idx;
 
@@ -1452,8 +1448,6 @@ static int iproute_get(int argc, char **argv)
 		exit(1);
 	}
 
-	ll_init_map(&rth);
-
 	if (idev || odev)  {
 		int idx;
 
ip/xfrm_monitor.c +0 -2
index bfc48f1..a1f5d53 100644
@@ -408,8 +408,6 @@ int do_xfrm_monitor(int argc, char **argv)
 		return rtnl_from_file(fp, xfrm_accept_msg, (void*)stdout);
 	}
 
-	//ll_init_map(&rth);
-
 	if (rtnl_open_byproto(&rth, groups, NETLINK_XFRM) < 0)
 		exit(1);
 
Signature · 1 line
1.7.10.4
#19 lore raw
Quoted 2 lines · Stephen Hemminger

Try the following two patches. It adds a name hash list, and uses Eric's idea to avoid loading map on add/delete operations.

On my microbenchmark of just creating 5000 veth pairs this takes pairs 16s instead of 13s of my earlier hacks but that is well down in the usable range.

Deleting all of those network interfaces one by one takes me 60s.

So on the microbenchmark side this looks like a good improvement and pretty usable.

I expect Benoit's container startup workload will also reflect this, but it will be interesting to see the actual result.

Eric

#20 lore raw
Quoted 3 lines · Eric W. Biederman

On my microbenchmark of just creating 5000 veth pairs this takes pairs 16s instead of 13s of my earlier hacks but that is well down in the usable range.

I guess most of the time is taken by sysctl_check_table()

#21 lore raw
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Eric W. Biederman

On my microbenchmark of just creating 5000 veth pairs this takes pairs 16s instead of 13s of my earlier hacks but that is well down in the usable range.

I guess most of the time is taken by sysctl_check_table()

All of the significant sysctl slowdowns were fixed in 3.4. If you see something of sysctl show up in a trace I would be happy to talk about it. The kernel side seems to be creating N network devices seems to take NlogN time now. Both sysfs and sysctl store directories as rbtrees removing their previous bottlenecks.

The loop I timed at 16s was just:

time for i in $(seq 1 5000) ; do ip link add a$i type veth peer name b$i; done

There is plenty of room for inefficiencies in 10000 network devices and 5000 forks+execs.

Eric

#22 lore raw
Quoted 16 lines · Eric W. Biederman
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Eric W. Biederman

On my microbenchmark of just creating 5000 veth pairs this takes pairs 16s instead of 13s of my earlier hacks but that is well down in the usable range.

I guess most of the time is taken by sysctl_check_table()

All of the significant sysctl slowdowns were fixed in 3.4. If you see something of sysctl show up in a trace I would be happy to talk about it. The kernel side seems to be creating N network devices seems to take NlogN time now. Both sysfs and sysctl store directories as rbtrees removing their previous bottlenecks.

Quoted 6 lines

The loop I timed at 16s was just:

time for i in $(seq 1 5000) ; do ip link add a$i type veth peer name b$i; done

There is plenty of room for inefficiencies in 10000 network devices and 5000 forks+execs.

Ah right, the sysctl part is fixed ;)

In batch mode, I can create these veth pairs in 4 seconds

for i in $(seq 1 5000) ; do echo link add a$i type veth peer name b$i; done | ip -batch -

#23 lore raw
Quoted 24 lines · Eric Dumazet
Quoted 16 lines · Eric W. Biederman
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Eric W. Biederman

On my microbenchmark of just creating 5000 veth pairs this takes pairs 16s instead of 13s of my earlier hacks but that is well down in the usable range.

I guess most of the time is taken by sysctl_check_table()

All of the significant sysctl slowdowns were fixed in 3.4. If you see something of sysctl show up in a trace I would be happy to talk about it. The kernel side seems to be creating N network devices seems to take NlogN time now. Both sysfs and sysctl store directories as rbtrees removing their previous bottlenecks.

Quoted 6 lines

The loop I timed at 16s was just:

time for i in $(seq 1 5000) ; do ip link add a$i type veth peer name b$i; done

There is plenty of room for inefficiencies in 10000 network devices and 5000 forks+execs.

Quoted 6 lines

Ah right, the sysctl part is fixed ;)

In batch mode, I can create these veth pairs in 4 seconds

for i in $(seq 1 5000) ; do echo link add a$i type veth peer name b$i; done | ip -batch -

Yes. The interesting story here is that the bottleneck before these
patches was the ll_init_map function of iproute2. Which resulted in an
over an order of magnitude slowdown of when starting iproute on a system with lots of network devices.

It is still unclear where iproute comes into the picture in the original problem scenario of creating 2000 containers each with 2 veth pairs. But apparently it was.

As the fundamental use case here was taking 2000 separate independent actions it turns out to be important for things to not slowdown unreasonably outside of batch mode. So I was explicitly testing the non-batch mode performance.

On the flip side it might be interesting to see if we can get batch mode deletes to batch in the kernel, so we don't have to wait for through syncrhonize_rcu_expidited for each of them. Although for the container case I can just drop the last reference to the network namespace and all of the network device removals will batch.

Ultimately shrug. Except in the previous O(N^2) userspace behavior there don't seem to be any practical performance problems with this many network devices. What is interesting is that this many network devices is becoming interesting on inexpensive COTS servers, for cases that are not purely network focused.

Eric

#24 lore raw
In reply to Eric Dumazet
Quoted 4 lines · Eric Dumazet

In batch mode, I can create these veth pairs in 4 seconds

for i in $(seq 1 5000) ; do echo link add a$i type veth peer name b$i; done | ip -batch -

At rmmod time, 30% of cpu is spent in packet_notifier()

Maybe we can do something about this.

30.85% rmmod [kernel.kallsyms] [k]
packet_notifier

                   |
                   --- packet_notifier
                       notifier_call_chain
                       raw_notifier_call_chain
                       call_netdevice_notifiers
                       rollback_registered_many
                       unregister_netdevice_many
                       __rtnl_link_unregister
                       rtnl_link_unregister
                       0xffffffffa0044868
                       sys_delete_module
                       sysenter_dispatch
#25 lore raw
Quoted 11 lines · Eric Dumazet
Quoted 4 lines · Eric Dumazet

In batch mode, I can create these veth pairs in 4 seconds

for i in $(seq 1 5000) ; do echo link add a$i type veth peer name b$i; done | ip -batch -

At rmmod time, 30% of cpu is spent in packet_notifier()

Maybe we can do something about this.

An interesting thought. I had a patch I never got around to pushing a while back that would have had an effect.

It is my observation that the vast majority of packet filters apply not to the entire machine but to an individual interface. In fact you have to work pretty hard to get tools like tcpdump to dump all of the interfaces at once.

So to speed things up for machines that have a lot of these things the idea was to create per device lists for the filters that only needed to be run on a single device. In this case it looks like we could potentially create per device lists for of the listening sockets as well.

In general these lists should be short so the search can also be short.

But I am curious do you actually have a tcpdump or something similar running on your box that is using AF_PACKET sockets? Perhaps a dhcp client?

I am a little surprised that your default case has anything on the lists to trigger any work in the packet_notifier notifier.

Quoted 14 lines

30.85% rmmod [kernel.kallsyms] [k]
packet_notifier

                   |
                   --- packet_notifier
                       notifier_call_chain
                       raw_notifier_call_chain
                       call_netdevice_notifiers
                       rollback_registered_many
                       unregister_netdevice_many
                       __rtnl_link_unregister
                       rtnl_link_unregister
                       0xffffffffa0044868
                       sys_delete_module
                       sysenter_dispatch

Eric

#26 lore raw
Quoted 13 lines · Eric W. Biederman

An interesting thought. I had a patch I never got around to pushing a while back that would have had an effect.

It is my observation that the vast majority of packet filters apply not to the entire machine but to an individual interface. In fact you have to work pretty hard to get tools like tcpdump to dump all of the interfaces at once.

So to speed things up for machines that have a lot of these things the idea was to create per device lists for the filters that only needed to be run on a single device. In this case it looks like we could potentially create per device lists for of the listening sockets as well.

Quoted 8 lines

In general these lists should be short so the search can also be short.

But I am curious do you actually have a tcpdump or something similar running on your box that is using AF_PACKET sockets? Perhaps a dhcp client?

I am a little surprised that your default case has anything on the lists to trigger any work in the packet_notifier notifier.

Hmm, it might be a local daemon on my lab machine which does a PACKET_ADD_MEMBERSHIP for each created interface.

So my machine spend time in packet_dev_mclist(), with a quadratic behavior at rmmod.

#27 lore raw
In reply to Eric W. Biederman

Hello,

Here are my tests of the last patches on 3 different platforms all running 3.8.5 :

Time are in seconds :

8x 3.7Ghz virtual cores

# veth 	create 	delete
1000 	14 	18
2000 	39 	56
5000 	256 	161
10000 	1200 	399

8x 3.2Ghz virtual cores

# veth 	create 	delete

1000 19 40
2000 118 66
5000 305 251

32x 2Ghz virtual cores , 2 sockets

# veth 	create 	delete
1000 	35 	86

2000 120 90

5000 724 245

Compared to initial iproute2 performance on this 32 virtual core system :
5000 1143 1185

"perf record" for creation of 5000 veth on the 32 core system :

# captured on: Fri Mar 29 14:03:35 2013
# hostname : ieng-serv06
# os release : 3.8.5
# perf version : 3.8.5
# arch : x86_64
# nrcpus online : 32
# nrcpus avail : 32
# cpudesc : Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
# cpuid : GenuineIntel,6,45,7
# total memory : 264124548 kB
# cmdline : /usr/src/linux-3.8.5/tools/perf/perf record -a ./test3.script
# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 =
0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_guest = 1,
precise_ip = 0, id = { 36, 37, 38, 39, 40, 41, 42,
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# pmu mappings: cpu = 4, software = 1, uncore_pcu = 15, tracepoint = 2,
uncore_imc_0 = 17, uncore_imc_1 = 18, uncore_imc_2 = 19, uncore_imc_3 =
20, uncore_qpi_0 = 21, uncore_qpi_1 = 22, unco
# ========
#
# Samples: 9M of event 'cycles'
# Event count (approx.): 2894480238483
#
# Overhead          Command                  Shared Object
                          Symbol
# ........  ...............  .............................
...............................................
#
    15.17%             sudo  [kernel.kallsyms]              [k]
snmp_fold_field    
     5.94%             sudo  libc-2.15.so                   [.]
0x00000000000802cd 
     5.64%             sudo  [kernel.kallsyms]              [k]
find_next_bit      
     3.21%             init  libnih.so.1.0.0                [.]
nih_list_add_after 
     2.12%          swapper  [kernel.kallsyms]              [k] intel_idle

1.94% init [kernel.kallsyms] [k] page_fault

1.93% sed libc-2.15.so [.]
0x00000000000a1368
1.93% sudo [kernel.kallsyms] [k]
rtnl_fill_ifinfo
1.92% sudo [veth] [k]
veth_get_stats64
1.78% sudo [kernel.kallsyms] [k] memcpy

1.53% ifquery libc-2.15.so [.]
0x000000000007f52b
1.24% init libc-2.15.so [.]
0x000000000008918f
1.05% sudo [kernel.kallsyms] [k]
inet6_fill_ifla6_attrs
0.98% init [kernel.kallsyms] [k]
copy_pte_range
0.88% irqbalance libc-2.15.so [.]
0x00000000000802cd
0.85% sudo [kernel.kallsyms] [k] memset

0.72% sed ld-2.15.so [.]
0x000000000000a226
0.68% ifquery ld-2.15.so [.]
0x00000000000165a0

     0.64%             init  libnih.so.1.0.0                [.]
nih_tree_next_post_full
     0.61%  bridge-network-  libc-2.15.so                   [.]

0x0000000000131e2a
0.59% init [kernel.kallsyms] [k] do_wp_page

0.59% ifquery [kernel.kallsyms] [k] page_fault

0.54% sed [kernel.kallsyms] [k] page_fault

Regards

Benoit

Quoted 22 lines · Eric W. Biederman
Quoted 3 lines · Stephen Hemminger

Try the following two patches. It adds a name hash list, and uses Eric's idea
to avoid loading map on add/delete operations.

On my microbenchmark of just creating 5000 veth pairs this takes pairs 16s instead of 13s of my earlier hacks but that is well down in the usable range.

Deleting all of those network interfaces one by one takes me 60s.

So on the microbenchmark side this looks like a good improvement and pretty usable.

I expect Benoit's container startup workload will also reflect this, but it will be interesting to see the actual result.

Eric

#28 lore raw
Quoted 121 lines · Benoit Lourdelet

Hello,

Here are my tests of the last patches on 3 different platforms all running 3.8.5 :

Time are in seconds :

8x 3.7Ghz virtual cores

# veth 	create 	delete
1000 	14 	18
2000 	39 	56
5000 	256 	161
10000 	1200 	399

8x 3.2Ghz virtual cores

# veth 	create 	delete

1000 19 40
2000 118 66
5000 305 251

32x 2Ghz virtual cores , 2 sockets

# veth 	create 	delete
1000 	35 	86

2000 120 90

5000 724 245

Compared to initial iproute2 performance on this 32 virtual core system :
5000 1143 1185

"perf record" for creation of 5000 veth on the 32 core system :

# captured on: Fri Mar 29 14:03:35 2013
# hostname : ieng-serv06
# os release : 3.8.5
# perf version : 3.8.5
# arch : x86_64
# nrcpus online : 32
# nrcpus avail : 32
# cpudesc : Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
# cpuid : GenuineIntel,6,45,7
# total memory : 264124548 kB
# cmdline : /usr/src/linux-3.8.5/tools/perf/perf record -a ./test3.script
# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 =
0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_guest = 1,
precise_ip = 0, id = { 36, 37, 38, 39, 40, 41, 42,
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# pmu mappings: cpu = 4, software = 1, uncore_pcu = 15, tracepoint = 2,
uncore_imc_0 = 17, uncore_imc_1 = 18, uncore_imc_2 = 19, uncore_imc_3 =
20, uncore_qpi_0 = 21, uncore_qpi_1 = 22, unco
# ========
#
# Samples: 9M of event 'cycles'
# Event count (approx.): 2894480238483
#
# Overhead          Command                  Shared Object
                          Symbol
# ........  ...............  .............................
...............................................
#
    15.17%             sudo  [kernel.kallsyms]              [k]
snmp_fold_field    
     5.94%             sudo  libc-2.15.so                   [.]
0x00000000000802cd 
     5.64%             sudo  [kernel.kallsyms]              [k]
find_next_bit      
     3.21%             init  libnih.so.1.0.0                [.]
nih_list_add_after 
     2.12%          swapper  [kernel.kallsyms]              [k] intel_idle

1.94% init [kernel.kallsyms] [k] page_fault

1.93% sed libc-2.15.so [.]
0x00000000000a1368
1.93% sudo [kernel.kallsyms] [k]
rtnl_fill_ifinfo
1.92% sudo [veth] [k]
veth_get_stats64
1.78% sudo [kernel.kallsyms] [k] memcpy

1.53% ifquery libc-2.15.so [.]
0x000000000007f52b
1.24% init libc-2.15.so [.]
0x000000000008918f
1.05% sudo [kernel.kallsyms] [k]
inet6_fill_ifla6_attrs
0.98% init [kernel.kallsyms] [k]
copy_pte_range
0.88% irqbalance libc-2.15.so [.]
0x00000000000802cd
0.85% sudo [kernel.kallsyms] [k] memset

0.72% sed ld-2.15.so [.]
0x000000000000a226
0.68% ifquery ld-2.15.so [.]
0x00000000000165a0

     0.64%             init  libnih.so.1.0.0                [.]
nih_tree_next_post_full
     0.61%  bridge-network-  libc-2.15.so                   [.]

0x0000000000131e2a
0.59% init [kernel.kallsyms] [k] do_wp_page

0.59% ifquery [kernel.kallsyms] [k] page_fault

0.54% sed [kernel.kallsyms] [k] page_fault

Quoted 7 lines

Regards

Benoit

This means lxc-start does the same thing than ip :

It fetches the whole device list.

You could strace it to have a confirmation.

#29 lore raw

Sorry Eric,

This is not an lxc-start perf report.This is an "ip" report".

Will run an "lxc-start" perf ASAP.

Regards

Benoit

Quoted 148 lines · Eric Dumazet
Quoted 129 lines · Benoit Lourdelet

Hello,

Here are my tests of the last patches on 3 different platforms all running 3.8.5 :

Time are in seconds :

8x 3.7Ghz virtual cores

# veth 	create 	delete
1000 	14 	18
2000 	39 	56
5000 	256 	161
10000 	1200 	399

8x 3.2Ghz virtual cores

# veth 	create 	delete

1000 19 40
2000 118 66
5000 305 251

32x 2Ghz virtual cores , 2 sockets

# veth 	create 	delete
1000 	35 	86

2000 120 90

5000 724 245

Compared to initial iproute2 performance on this 32 virtual core system :
5000 1143 1185

"perf record" for creation of 5000 veth on the 32 core system :

# captured on: Fri Mar 29 14:03:35 2013
# hostname : ieng-serv06
# os release : 3.8.5
# perf version : 3.8.5
# arch : x86_64
# nrcpus online : 32
# nrcpus avail : 32
# cpudesc : Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
# cpuid : GenuineIntel,6,45,7
# total memory : 264124548 kB
# cmdline : /usr/src/linux-3.8.5/tools/perf/perf record -a
./test3.script
# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2
=
0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_guest = 1,
precise_ip = 0, id = { 36, 37, 38, 39, 40, 41, 42,
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# pmu mappings: cpu = 4, software = 1, uncore_pcu = 15, tracepoint = 2,
uncore_imc_0 = 17, uncore_imc_1 = 18, uncore_imc_2 = 19, uncore_imc_3 =
20, uncore_qpi_0 = 21, uncore_qpi_1 = 22, unco
# ========
#
# Samples: 9M of event 'cycles'
# Event count (approx.): 2894480238483
#
# Overhead          Command                  Shared Object
                          Symbol
# ........  ...............  .............................
...............................................
#
    15.17%             sudo  [kernel.kallsyms]              [k]
snmp_fold_field 
     5.94%             sudo  libc-2.15.so                   [.]
0x00000000000802cd
     5.64%             sudo  [kernel.kallsyms]              [k]
find_next_bit   
     3.21%             init  libnih.so.1.0.0                [.]
nih_list_add_after
     2.12%          swapper  [kernel.kallsyms]              [k]
intel_idle

1.94% init [kernel.kallsyms] [k]
page_fault

1.93% sed libc-2.15.so [.]
0x00000000000a1368
1.93% sudo [kernel.kallsyms] [k]
rtnl_fill_ifinfo
1.92% sudo [veth] [k]
veth_get_stats64
1.78% sudo [kernel.kallsyms] [k] memcpy

1.53% ifquery libc-2.15.so [.]
0x000000000007f52b
1.24% init libc-2.15.so [.]
0x000000000008918f
1.05% sudo [kernel.kallsyms] [k]
inet6_fill_ifla6_attrs
0.98% init [kernel.kallsyms] [k]
copy_pte_range
0.88% irqbalance libc-2.15.so [.]
0x00000000000802cd
0.85% sudo [kernel.kallsyms] [k] memset

0.72% sed ld-2.15.so [.]
0x000000000000a226
0.68% ifquery ld-2.15.so [.]
0x00000000000165a0

     0.64%             init  libnih.so.1.0.0                [.]
nih_tree_next_post_full
     0.61%  bridge-network-  libc-2.15.so                   [.]

0x0000000000131e2a
0.59% init [kernel.kallsyms] [k]
do_wp_page

0.59% ifquery [kernel.kallsyms] [k]
page_fault

0.54% sed [kernel.kallsyms] [k]
page_fault

Quoted 7 lines

Regards

Benoit

This means lxc-start does the same thing than ip :

It fetches the whole device list.

You could strace it to have a confirmation.

#30 lore raw
In reply to Eric W. Biederman

Hello Eric,

I am running simple containers (2 network interfaces, 10MB of RAM, default routing) and want to test scalability.

Out test platform is a 32x 2Ghz cores x86.

Regards

Benoit

Quoted 21 lines · Eric W. Biederman
Quoted 6 lines · Benoit Lourdelet

Hello Serge,

I am indeed using Eric patch with lxc.

It solves the initial problem of slowness to start around 1600 containers.

Good so now we just need a production ready patch for iproute.

Quoted 2 lines

I am now able to start more than 2000 without having new containers slower and slower to start.

May I ask how large a box you are running and how complex your containers are. I am trying to get a feel for how common it is likely to be to find people running thousands of containers on a single machine.

Eric

#31 lore raw
In reply to Benoit Lourdelet
Quoted 6 lines · Benoit Lourdelet

The script to delete:
for d in /sys/class/net/veth*; do

	ip link del `basename $d` 2>/dev/null || true

Done

There is a very good improvement in deletion.

I can do better ;)

If you are really doing this kind of things, you could use :

rmmod veth

Note that "ip" command supports a batch mode

ip -batch filename

In this case, the caching is done only once.

Eric, Stephen, one possibility would be to use the cache only in batch mode.

Anyway caching is wrong because several users can use ip command at the same time.