iproute: Faster ip link add, set and delete
Subsystems (3)
RFC PATCH 1/1 iproute: Faster ip link add, set and delete
Because ip link add, set, and delete map the interface name to the interface index by dumping all of the interfaces before performing their respective commands. Operations that should be constant time slow down when lots of network interfaces are in use. Resulting in O(N^2) time to work with O(N) devices.
Make the work that iproute does constant time by passing the interface name to the kernel instead.
In small scale testing on my system this shows dramatic performance increases of ip link add from 120s to just 11s to add 5000 network devices. And from longer than I cared to wait to just 58s to delete all of those interfaces again.
---
I think I am bungling the case where people specify an ifindex as ifNNNN but does anyone care?
ip/iplink.c | 19 +------------------ 1 files changed, 1 insertions(+), 18 deletions(-)
ip/iplink.c +1 -18
@@ -533,8 +533,6 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv) } } -ll_init_map (&rth); -if (!(flags &NLM_F_CREATE )) {if (!dev) { fprintf(stderr, "Not enough information: \"dev\" "
@@ -542,27 +540,12 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)exit (-1 ); } - req.i.ifi_index =ll_name_to_index (dev); -if (req.i.ifi_index ==0 ) { -fprintf (stderr,"Cannot find device \"%s\"\n" , dev); -return -1 ; - } + name = dev; }else {/* Allow "ip link add dev" and "ip link add name" */ if (!name) name = dev; -if (link) { -int ifindex; - - ifindex =ll_name_to_index (link); -if (ifindex ==0 ) { -fprintf (stderr,"Cannot find device \"%s\"\n" , - link); -return -1 ; - } -addattr_l (&req.n,sizeof (req),IFLA_LINK , &ifindex,4 ); - } }if (name) {
Signature · 1 line
1.7.5.4
The whole ifindex map is a design mistake at this point. Better off to do a lazy cache or something like that.
Quoted 71 lines · Eric W. Biederman
Because ip link add, set, and delete map the interface name to the interface index by dumping all of the interfaces before performing their respective commands. Operations that should be constant time slow down when lots of network interfaces are in use. Resulting in O(N^2) time to work with O(N) devices.
Make the work that iproute does constant time by passing the interface name to the kernel instead.
In small scale testing on my system this shows dramatic performance increases of ip link add from 120s to just 11s to add 5000 network devices. And from longer than I cared to wait to just 58s to delete all of those interfaces again.
Cc Serge Hallyn <[email protected]>Reported-by Benoit Lourdelet <[email protected]>Signed-off-by "Eric W. Biederman" <[email protected]>---
I think I am bungling the case where people specify an ifindex as ifNNNN but does anyone care?
ip/iplink.c | 19 +------------------ 1 files changed, 1 insertions(+), 18 deletions(-)ip/iplink.c +1
-19@@ -533,8 +533,6 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv) } } -ll_init_map (&rth); -if (!(flags &NLM_F_CREATE )) {if (!dev) { fprintf(stderr, "Not enough information: \"dev\" "@@ -542,27 +540,12 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)exit (-1 ); } - req.i.ifi_index =ll_name_to_index (dev); -if (req.i.ifi_index ==0 ) { -fprintf (stderr,"Cannot find device \"%s\"\n" , dev); -return -1 ; - } + name = dev; }else {/* Allow "ip link add dev" and "ip link add name" */ if (!name) name = dev; -if (link) { -int ifindex; - - ifindex =ll_name_to_index (link); -if (ifindex ==0 ) { -fprintf (stderr,"Cannot find device \"%s\"\n" , - link); -return -1 ; - } -addattr_l (&req.n,sizeof (req),IFLA_LINK , &ifindex,4 ); - } }if (name) { --1.7.5.4
Hello,
I re-tested with the patch and got the following results on a 32x 2Ghz core system.
# veth add delete 1000 36 34 3000 259 137 4000 462 195 5000 729 N/A
The script to create is the following :
for i in seq 1 5000; do
sudo ip link add type veth
Done
The script to delete:
for d in /sys/class/net/veth*; do
ip link del `basename $d` 2>/dev/null || true
Done
There is a very good improvement in deletion.
iproute2 does not seems to be well multithread as I get time divided by a factor of 2 with a 8x 3.2 Ghz core system.
I don¹t know if that is the improvement you expected ?
Would the iproute2 redesign you mentioned help improve performance even further ?
As a reference : Iproute2 baseline w/o patch:
# veth add delete
1000 57 70
2000 193 250
3000 435 510
4000 752 824
5000 1123 1185
Regards
Benoit
Quoted 84 lines · Stephen Hemminger
The whole ifindex map is a design mistake at this point. Better off to do a lazy cache or something like that.
Quoted 77 lines · Eric W. Biederman
Because ip link add, set, and delete map the interface name to the interface index by dumping all of the interfaces before performing their respective commands. Operations that should be constant time slow down when lots of network interfaces are in use. Resulting in O(N^2) time to work with O(N) devices.
Make the work that iproute does constant time by passing the interface name to the kernel instead.
In small scale testing on my system this shows dramatic performance increases of ip link add from 120s to just 11s to add 5000 network devices. And from longer than I cared to wait to just 58s to delete all of those interfaces again.
Cc Serge Hallyn <[email protected]>Reported-by Benoit Lourdelet <[email protected]>Signed-off-by "Eric W. Biederman" <[email protected]>---
I think I am bungling the case where people specify an ifindex as ifNNNN but does anyone care?
ip/iplink.c | 19 +------------------ 1 files changed, 1 insertions(+), 18 deletions(-)ip/iplink.c +0
-2@@ -533,8 +533,6 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv) } } -ll_init_map (&rth); -if (!(flags &NLM_F_CREATE )) {if (!dev) {fprintf(stderr, "Not enough information:
\"dev\" "ip/iplink.c +1
-15@@ -542,27 +540,12 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv)exit (-1 ); } - req.i.ifi_index =ll_name_to_index (dev); -if (req.i.ifi_index ==0 ) { -fprintf (stderr,"Cannot find device \"%s\"\n" , dev); -return -1 ; - } + name = dev; }else {/* Allow "ip link add dev" and "ip link add name" */ if (!name) name = dev; -if (link) { -int ifindex; - - ifindex =ll_name_to_index (link); -if (ifindex ==0 ) { -fprintf (stderr,"Cannot find device \"%s\"\n", - link); -return -1 ; - } - addattr_l(&req.n, sizeof(req), IFLA_LINK,&ifindex, 4);
- }} if (name) {--
1.7.5.4
Quoted 11 lines · Benoit Lourdelet
Hello,
I re-tested with the patch and got the following results on a 32x 2Ghz core system.
# veth add delete 1000 36 34 3000 259 137 4000 462 195 5000 729 N/A
Quoted 4 lines
The script to create is the following :
for i inseq 1 5000; do
sudo ip link add type veth
Done
Which performs horribly as I mentioned earlier because you are asking the kernel to create the names. If you want performance you need to specify the names of the network devices you are creating.
aka ip link add a$i type veth name b$i
Quoted 11 lines
The script to delete:
for d in/sys/class/net/veth*; doip link del `basename $d` 2>/dev/null || trueDone
There is a very good improvement in deletion.
iproute2 does not seems to be well multithread as I get time divided by a factor of 2 with a 8x 3.2 Ghz core system.
All netlink traffic and all network stack configuration is serialized by the rtnl_lock in the kernel. This is the slow path in the kernel, not the fast path.
Quoted 4 lines
I don¹t know if that is the improvement you expected ?
Would the iproute2 redesign you mentioned help improve performance even further ?
Specifing the names would dramatically improve your creation performance. It should only take you about 10s for 5000 veth pairs. But you have to specify the names.
Anyway I have exhausted my time, and inclination in this matter. Good luck with whatever your problem is.
Eric
Quoted 3 lines · Eric W. Biederman
Specifing the names would dramatically improve your creation performance. It should only take you about 10s for 5000 veth pairs. But you have to specify the names.
Thanks, Eric. I'm going to update lxc to always specify names for the veth pairs, rather than only when they are requested by the user's configuration file.
-serge
Actually, lxc is using random names now, so it's ok.
Benoit, can you use the patches from Eric with lxc (or use the script you were using before but specify names as he said)?
-serge
Quoted 6 lines · Benoit Lourdelet
The script to delete:
for d in/sys/class/net/veth*; doip link del `basename $d` 2>/dev/null || trueDone
There is a very good improvement in deletion.
I can do better ;)
If you are really doing this kind of things, you could use :
rmmod veth
Note that "ip" command supports a batch mode
ip -batch filename
In this case, the caching is done only once.
Eric, Stephen, one possibility would be to use the cache only in batch mode.
Anyway caching is wrong because several users can use ip command at the same time.
Hello Serge,
I am indeed using Eric patch with lxc.
It solves the initial problem of slowness to start around 1600 containers.
I am now able to start more than 2000 without having new containers slower and slower to start.
thanks
Benoit
Quoted 7 lines · Serge Hallyn
Actually, lxc is using random names now, so it's ok.
Benoit, can you use the patches from Eric with lxc (or use the script you were using before but specify names as he said)?
-serge
Quoted 6 lines · Benoit Lourdelet
Hello Serge,
I am indeed using Eric patch with lxc.
It solves the initial problem of slowness to start around 1600 containers.
Good so now we just need a production ready patch for iproute.
Quoted 2 lines
I am now able to start more than 2000 without having new containers slower and slower to start.
May I ask how large a box you are running and how complex your containers are. I am trying to get a feel for how common it is likely to be to find people running thousands of containers on a single machine.
Eric
If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.
I have an updated version of link map hash (index and name). Could you test this patch which applies to latest version in git.
lib/ll_map.c +128 -50
@@ -12,6 +12,7 @@#include <stdio.h> #include <stdlib.h> +#include <stddef.h> #include <unistd.h> #include <syslog.h> #include <fcntl.h>
@@ -23,9 +24,44 @@#include "libnetlink.h" #include "ll_map.h" -struct ll_cache + +struct hlist_head { +struct hlist_node *first; +}; + +struct hlist_node { +struct hlist_node *next, **pprev; +}; + +static inline void hlist_del (struct hlist_node *n) +{ +struct hlist_node *next = n->next; +struct hlist_node **pprev = n->pprev; + *pprev = next; +if (next) + next->pprev = pprev; +} + +static inline void hlist_add_head (struct hlist_node *n,struct hlist_head *h) { -struct ll_cache *idx_next; +struct hlist_node *first = h->first; + n->next = first; +if (first) + first->pprev = &n->next; + h->first = n; + n->pprev = &h->first; +} + +#define hlist_for_each (pos, head) \ + for (pos = (head)->first; pos ; pos = pos->next) + +#define container_of (ptr, type, member) ({ \ + const typeof( ((type *)0)->member ) *__mptr = (ptr); \ + (type *)( (char *)__mptr - offsetof(type,member) );}) + +struct ll_cache { +struct hlist_node idx_hash; +struct hlist_node name_hash;unsigned flags;int index;unsigned short type;
@@ -33,49 +69,107 @@ struct ll_cache };#define IDXMAP_SIZE 1024 -static struct ll_cache *idx_head[IDXMAP_SIZE ]; +static struct hlist_head idx_head[IDXMAP_SIZE ]; +static struct hlist_head name_head[IDXMAP_SIZE ]; -static inline struct ll_cache *idxhead (int idx) +static struct ll_cache *ll_get_by_index (unsigned index) { -return idx_head[idx & (IDXMAP_SIZE -1 )]; +struct hlist_node *n; +unsigned h = index & (IDXMAP_SIZE -1 ); + +hlist_for_each (n, &idx_head[h]) { +struct ll_cache *im + = container_of(n, struct ll_cache, idx_hash); +if (im->index == index) +return im; + } + +return NULL ; +} + +static unsigned namehash (const char *str) +{ +unsigned hash =5381 ; + +while (*str) + hash = ((hash <<5 ) + hash) + *str++;/* hash * 33 + c */ + +return hash; +} + +static struct ll_cache *ll_get_by_name (const char *name) +{ +struct hlist_node *n; +unsigned h =namehash (name) & (IDXMAP_SIZE -1 ); + +hlist_for_each (n, &name_head[h]) { +struct ll_cache *im + = container_of(n, struct ll_cache, name_hash); + +if (strncmp (im->name, name,IFNAMSIZ ) ==0 ) +return im; + } + +return NULL ; }int ll_remember_index (const struct sockaddr_nl *who,struct nlmsghdr *n,void *arg) { -int h; +unsigned int h; +const char *ifname;struct ifinfomsg *ifi =NLMSG_DATA (n); -struct ll_cache *im, **imp; +struct ll_cache *im;struct rtattr *tb[IFLA_MAX +1 ]; -if (n->nlmsg_type !=RTM_NEWLINK ) +if (n->nlmsg_type !=RTM_NEWLINK && n->nlmsg_type !=RTM_DELLINK )return 0 ;if (n->nlmsg_len <NLMSG_LENGTH (sizeof (ifi)))return -1 ; + im =ll_get_by_index (ifi->ifi_index); +if (n->nlmsg_type ==RTM_DELLINK ) { +if (im) { +hlist_del (&im->name_hash); +hlist_del (&im->idx_hash); +free (im); + } +return 0 ; + } +memset (tb,0 ,sizeof (tb));parse_rtattr (tb,IFLA_MAX ,IFLA_RTA (ifi),IFLA_PAYLOAD (n)); -if (tb[IFLA_IFNAME ] ==NULL ) + ifname =rta_getattr_str (tb[IFLA_IFNAME ]); +if (ifname ==NULL )return 0 ; - h = ifi->ifi_index & (IDXMAP_SIZE -1 ); -for (imp = &idx_head[h]; (im=*imp)!=NULL ; imp = &im->idx_next) -if (im->index == ifi->ifi_index) -break ; - -if (im ==NULL ) { - im =malloc (sizeof (*im)); -if (im ==NULL ) -return 0 ; - im->idx_next = *imp; - im->index = ifi->ifi_index; - *imp = im; +if (im) { +/* change to existing entry */ +if (strcmp (im->name, ifname) !=0 ) { +hlist_del (&im->name_hash); + h =namehash (ifname) & (IDXMAP_SIZE -1 ); +hlist_add_head (&im->name_hash, &name_head[h]); + } + + im->flags = ifi->ifi_flags; +return 0 ; } + im =malloc (sizeof (*im)); +if (im ==NULL ) +return 0 ; + im->index = ifi->ifi_index; +strcpy (im->name, ifname); im->type = ifi->ifi_type; im->flags = ifi->ifi_flags; -strcpy (im->name,RTA_DATA (tb[IFLA_IFNAME ])); + + h = ifi->ifi_index & (IDXMAP_SIZE -1 ); +hlist_add_head (&im->idx_hash, &idx_head[h]); + + h =namehash (ifname) & (IDXMAP_SIZE -1 ); +hlist_add_head (&im->name_hash, &name_head[h]); +return 0 ; }
@@ -86,15 +180,14 @@ const char *ll_idx_n2a(unsigned idx, char *buf)if (idx ==0 )return "*" ; -for (im =idxhead (idx); im; im = im->idx_next) -if (im->index == idx) -return im->name; + im =ll_get_by_index (idx); +if (im) +return im->name;snprintf (buf,IFNAMSIZ ,"if%d" , idx);return buf; } -const char *ll_index_to_name (unsigned idx) {static char nbuf[IFNAMSIZ ];
@@ -108,10 +201,9 @@ int ll_index_to_type(unsigned idx)if (idx ==0 )return -1 ; -for (im =idxhead (idx); im; im = im->idx_next) -if (im->index == idx) -return im->type; -return -1 ; + + im =ll_get_by_index (idx); +return im ? im->type :-1 ; }unsigned ll_index_to_flags (unsigned idx)
@@ -121,35 +213,21 @@ unsigned ll_index_to_flags(unsigned idx)if (idx ==0 )return 0 ; -for (im =idxhead (idx); im; im = im->idx_next) -if (im->index == idx) -return im->flags; -return 0 ; + im =ll_get_by_index (idx); +return im ? im->flags :-1 ; }unsigned ll_name_to_index (const char *name) { -static char ncache[IFNAMSIZ ]; -static int icache; -struct ll_cache *im; -int i; +const struct ll_cache *im;unsigned idx;if (name ==NULL )return 0 ; -if (icache &&strcmp (name, ncache) ==0 ) -return icache; - -for (i=0 ; i<IDXMAP_SIZE ; i++) { -for (im = idx_head[i]; im; im = im->idx_next) { -if (strcmp (im->name, name) ==0 ) { - icache = im->index; -strcpy (ncache, name); -return im->index; - } - } - } + im =ll_get_by_name (name); +if (im) +return im->index; idx =if_nametoindex (name);if (idx ==0 )
Quoted 2 lines · Stephen Hemminger
If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.
The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform the work itself.
Quoted 2 lines
I have an updated version of link map hash (index and name). Could you test this patch which applies to latest version in git.
This still dumps all of the interfaces in ll_init_map causing things to slow down noticably.
# with your patch # time ~/projects/iproute/iproute2/ip/ip link add a4511 type veth peer name b4511
real 0m0.049s
user 0m0.000s
sys 0m0.048s
# With a hack to make ll_map_init a nop.
# time ~/projects/iproute/iproute2/ip/ip link add a4512 type veth peer name b4512
real 0m0.003s
user 0m0.000s
sys 0m0.000s
eric-ThinkPad-X220 6bed4 #
# Without any patches.
# time ~/projects/iproute/iproute2/ip/ip link add a5002 type veth peer name b5002
real 0m0.052s
user 0m0.004s
sys 0m0.044s
So it looks like dumping all of the interfaces is taking 46 miliseconds, longer than otherwise. Causing ip to take nearly an order of magnitude longer to run when there are a lot of interfaces, and causing ip to slow down with each command.
So the ideal situation is probably just to fill in the ll_map on demand instead of up front.
Eric
Quoted 8 lines · Eric W. Biederman
Quoted 2 lines · Stephen Hemminger
If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.
The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform the work itself.
fwiw lxc uses netlink to create new veths, and picks random names with mktemp() ahead of time.
-serge
Quoted 6 lines · Serge Hallyn
Quoted 5 lines · Eric W. Biederman
Quoted 2 lines · Stephen Hemminger
If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.
Quoted 6 lines
Quoted 3 lines
The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform the work itself.
fwiw lxc uses netlink to create new veths, and picks random names with
mktemp()ahead of time.
I am puzzled where does the slownes in iproute2 come into play?
Eric
Quoted 8 lines · Eric W. Biederman
Quoted 6 lines · Serge Hallyn
Quoted 5 lines · Eric W. Biederman
Quoted 2 lines · Stephen Hemminger
If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform the work itself.
fwiw lxc uses netlink to create new veths, and picks random names with
mktemp()ahead of time.I am puzzled where does the slownes in iproute2 come into play?
Benoit originally reported slowness when starting >1500 containers. I asked him to run a few manual tests to figure out what was taking the time. Manually creating a large # of veths was an obvious test, and one which showed poorly scaling performance.
May well be there are other things slowing down lxc of course.
-serge
Quoted 16 lines · Serge Hallyn
Quoted 15 lines · Eric W. Biederman
Quoted 12 lines · Serge Hallyn
Quoted 8 lines · Eric W. Biederman
Quoted 2 lines · Stephen Hemminger
If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.
The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform the work itself.
fwiw lxc uses netlink to create new veths, and picks random names with
mktemp()ahead of time.
Quoted 6 lines
Quoted 1 line
I am puzzled where does the slownes in iproute2 come into play?
Benoit originally reported slowness when starting >1500 containers. I asked him to run a few manual tests to figure out what was taking the time. Manually creating a large # of veths was an obvious test, and one which showed poorly scaling performance.
Apparently iproute is involved somehwere as when he tested with a patched iproute (as you asked him to) the lxc startup slowdown was gone.
Quoted 1 line
May well be there are other things slowing down lxc of course.
The evidence indicates it was iproute being called somewhere...
Eric
Quoted 25 lines · Eric W. Biederman
Quoted 16 lines · Serge Hallyn
Quoted 15 lines · Eric W. Biederman
Quoted 12 lines · Serge Hallyn
Quoted 8 lines · Eric W. Biederman
Quoted 2 lines · Stephen Hemminger
If you need to do lots of operations the --batch mode will be significantly faster. One command start and one link map.
The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform the work itself.
fwiw lxc uses netlink to create new veths, and picks random names with
mktemp()ahead of time.Quoted 6 lines
Quoted 1 line
I am puzzled where does the slownes in iproute2 come into play?
Benoit originally reported slowness when starting >1500 containers. I asked him to run a few manual tests to figure out what was taking the time. Manually creating a large # of veths was an obvious test, and one which showed poorly scaling performance.
Quoted 7 lines
Apparently iproute is involved somehwere as when he tested with a patched iproute (as you asked him to) the lxc startup slowdown was gone.
Quoted 1 line
May well be there are other things slowing down lxc of course.
The evidence indicates it was iproute being called somewhere...
Benoit can you tell us exactly what test you were running when you saw the slowdown was gone?
-serge
Hello,
My test consists in starting small containers (10MB of RAM ) each. Each container has 2x physical VLAN interfaces attached.
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth6.3
lxc.network.name = eth2
lxc.network.hwaddr = 00:50:56:a8:03:03
lxc.network.ipv4 = 192.168.1.1/24
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth7.3
lxc.network.name = eth1
lxc.network.ipv4 = 2.2.2.2/24
lxc.network.hwaddr = 00:50:57:b8:00:01
With initial iproute2 , when I reach around 1600 containers, container creation almost stops.It takes at least 20s per container to start. With patched iproutes2 , I have started 4000 containers at a rate of 1 per second w/o problem. I have 8000 clan interfaces configured on the host (2x 4000).
Regards
Benoit
Quoted 42 lines · Serge Hallyn
Quoted 28 lines · Eric W. Biederman
Quoted 7 lines · Serge Hallyn
Quoted 6 lines · Eric W. Biederman
Quoted 4 lines · Serge Hallyn
Quoted 3 lines · Eric W. Biederman
Quoted 1 line · Stephen Hemminger
If you need to do lots of operations the --batch mode will be
significantly faster.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
One command start and one link map.
The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
work itself.
fwiw lxc uses netlink to create new veths, and picks random names
with
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
mktemp()ahead of time.I am puzzled where does the slownes in iproute2 come into play?
Benoit originally reported slowness when starting >1500 containers. I asked him to run a few manual tests to figure out what was taking the time. Manually creating a large # of veths was an obvious test, and one which showed poorly scaling performance.
Quoted 7 lines
Apparently iproute is involved somehwere as when he tested with a patched iproute (as you asked him to) the lxc startup slowdown was gone.
Quoted 1 line
May well be there are other things slowing down lxc of course.
The evidence indicates it was iproute being called somewhere...
Benoit can you tell us exactly what test you were running when you saw the slowdown was gone?
-serge
Quoted 4 lines · Benoit Lourdelet
Hello,
My test consists in starting small containers (10MB of RAM ) each. Each container has 2x physical VLAN interfaces attached.
Which commands were you using to create/start them?
Quoted 72 lines
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth6.3
lxc.network.name = eth2
lxc.network.hwaddr = 00:50:56:a8:03:03
lxc.network.ipv4 = 192.168.1.1/24
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth7.3
lxc.network.name = eth1
lxc.network.ipv4 = 2.2.2.2/24
lxc.network.hwaddr = 00:50:57:b8:00:01With initial iproute2 , when I reach around 1600 containers, container creation almost stops.It takes at least 20s per container to start. With patched iproutes2 , I have started 4000 containers at a rate of 1 per second w/o problem. I have 8000 clan interfaces configured on the host (2x 4000).
Regards
Benoit
Quoted 42 lines · Serge Hallyn
Quoted 28 lines · Eric W. Biederman
Quoted 7 lines · Serge Hallyn
Quoted 6 lines · Eric W. Biederman
Quoted 4 lines · Serge Hallyn
Quoted 3 lines · Eric W. Biederman
Quoted 1 line · Stephen Hemminger
If you need to do lots of operations the --batch mode will be
significantly faster.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
One command start and one link map.
The problem in this case as I understand it is lots of independent operations. Now maybe lxc should not shell out to ip and perform
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
work itself.
fwiw lxc uses netlink to create new veths, and picks random names
with
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
mktemp()ahead of time.I am puzzled where does the slownes in iproute2 come into play?
Benoit originally reported slowness when starting >1500 containers. I asked him to run a few manual tests to figure out what was taking the time. Manually creating a large # of veths was an obvious test, and one which showed poorly scaling performance.
Quoted 7 lines
Apparently iproute is involved somehwere as when he tested with a patched iproute (as you asked him to) the lxc startup slowdown was gone.
Quoted 1 line
May well be there are other things slowing down lxc of course.
The evidence indicates it was iproute being called somewhere...
Benoit can you tell us exactly what test you were running when you saw the slowdown was gone?
-serge
I use, for each container :
lxc-start -n lwb2001 -f /var/lib/lxc/lwb2001/config -d
I created the containers with lxc-ubuntu -n lwb2001
Benoit
Quoted 89 lines · Serge Hallyn
Quoted 4 lines · Benoit Lourdelet
Hello,
My test consists in starting small containers (10MB of RAM ) each. Each container has 2x physical VLAN interfaces attached.
Which commands were you using to create/start them?
Quoted 80 lines
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth6.3
lxc.network.name = eth2
lxc.network.hwaddr = 00:50:56:a8:03:03
lxc.network.ipv4 = 192.168.1.1/24
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth7.3
lxc.network.name = eth1
lxc.network.ipv4 = 2.2.2.2/24
lxc.network.hwaddr = 00:50:57:b8:00:01With initial iproute2 , when I reach around 1600 containers, container creation almost stops.It takes at least 20s per container to start. With patched iproutes2 , I have started 4000 containers at a rate of 1 per
second w/o problem. I have 8000 clan interfaces configured on the host (2x
4000).Regards
Benoit
Quoted 6 lines · Serge Hallyn
Quoted 5 lines · Eric W. Biederman
Quoted 3 lines · Serge Hallyn
Quoted 2 lines · Eric W. Biederman
Serge Hallyn <[email protected]> writes:
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines · Eric W. Biederman
Quoted 3 lines
Quoted 1 line · Stephen Hemminger
If you need to do lots of operations the --batch mode will be
significantly faster.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
One command start and one link map.
The problem in this case as I understand it is lots of
independent
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
operations. Now maybe lxc should not shell out to ip and
perform
Quoted 4 lines
Quoted 4 lines
the
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
work itself.
fwiw lxc uses netlink to create new veths, and picks random
names
Quoted 6 lines
Quoted 6 lines
with
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
mktemp()ahead of time.I am puzzled where does the slownes in iproute2 come into play?
Benoit originally reported slowness when starting >1500
containers. I
Quoted 1 line
Quoted 1 line
Quoted 1 line
asked him to run a few manual tests to figure out what was taking
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
time. Manually creating a large # of veths was an obvious test,
and
Quoted 15 lines
Quoted 9 lines
Quoted 1 line
one which showed poorly scaling performance.
Apparently iproute is involved somehwere as when he tested with a patched iproute (as you asked him to) the lxc startup slowdown was gone.
Quoted 1 line
May well be there are other things slowing down lxc of course.
The evidence indicates it was iproute being called somewhere...
Benoit can you tell us exactly what test you were running when you saw the slowdown was gone?
-serge
Hello Eric,
I am running simple containers (2 network interfaces, 10MB of RAM, default routing) and want to test scalability.
Out test platform is a 32x 2Ghz cores x86.
Regards
Benoit
Quoted 21 lines · Eric W. Biederman
Quoted 6 lines · Benoit Lourdelet
Hello Serge,
I am indeed using Eric patch with lxc.
It solves the initial problem of slowness to start around 1600 containers.
Good so now we just need a production ready patch for iproute.
Quoted 2 lines
I am now able to start more than 2000 without having new containers slower and slower to start.
May I ask how large a box you are running and how complex your containers are. I am trying to get a feel for how common it is likely to be to find people running thousands of containers on a single machine.
Eric
Try the following two patches. It adds a name hash list, and uses Eric's idea to avoid loading map on add/delete operations.
Quoted 1 line
From
0025e5d63d5d1598ab622867834a3bcb9f518f9fMon Sep 17 00:00:00 2001
From: Stephen Hemminger <[email protected]>
Date: Thu, 28 Mar 2013 14:57:28 -0700
Subject: [PATCH 1/2] ll_map: add name and index hash
Make ll_ functions faster by having a name hash, and allow for deletion. Also, allow them to work without calling ll_init_map.
--- include/hlist.h | 56 ++++++++++++++++++++ include/ll_map.h | 3 +- lib/ll_map.c | 155 ++++++++++++++++++++++++++++++++++-------------------- 3 files changed, 157 insertions(+), 57 deletions(-) create mode 100644 include/hlist.h
include/hlist.h new file +56 -0
@@ -0,0 +1,56 @@ +#ifndef __HLIST_H__ +#define __HLIST_H__ 1 +/* Hash list stuff from kernel */ + +#include <stddef.h> + +#define container_of (ptr, type, member) ({ \ + const typeof( ((type *)0)->member ) *__mptr = (ptr); \ + (type *)( (char *)__mptr - offsetof(type,member) );}) + +struct hlist_head { +struct hlist_node *first; +}; + +struct hlist_node { +struct hlist_node *next, **pprev; +}; + +static inline void hlist_del (struct hlist_node *n) +{ +struct hlist_node *next = n->next; +struct hlist_node **pprev = n->pprev; + *pprev = next; +if (next) + next->pprev = pprev; +} + +static inline void hlist_add_head (struct hlist_node *n,struct hlist_head *h) +{ +struct hlist_node *first = h->first; + n->next = first; +if (first) + first->pprev = &n->next; + h->first = n; + n->pprev = &h->first; +} + +#define hlist_for_each (pos, head) \ + for (pos = (head)->first; pos ; pos = pos->next) + + +#define hlist_for_each_safe (pos, n, head) \ + for (pos = (head)->first; pos && ({ n = pos->next; 1; }); \ + pos = n) + +#define hlist_entry_safe (ptr, type, member) \ + ({ typeof(ptr) ____ptr = (ptr); \ + ____ptr ? hlist_entry(____ptr, type, member) : NULL; \ + }) + +#define hlist_for_each_entry (pos, head, member) \ + for (pos = hlist_entry_safe((head)->first, typeof(*(pos)), member);\ + pos; \ + pos = hlist_entry_safe((pos)->member.next, typeof(*(pos)), member)) + +#endif /* __HLIST_H__ */
include/ll_map.h +2 -1
@@ -3,7 +3,8 @@extern int ll_remember_index (const struct sockaddr_nl *who,struct nlmsghdr *n,void *arg); -extern int ll_init_map (struct rtnl_handle *rth); + +extern void ll_init_map (struct rtnl_handle *rth);extern unsigned ll_name_to_index (const char *name);extern const char *ll_index_to_name (unsigned idx);extern const char *ll_idx_n2a (unsigned idx,char *buf);
lib/ll_map.c +99 -56
236 lines not on this page · raw message
Signature · 1 line
1.7.10.4
Quoted 1 line
From
f0124b0f0aa0e5b9288114eb8e6ff9b4f8c33ec8Mon Sep 17 00:00:00 2001
From: Stephen Hemminger <[email protected]>
Date: Thu, 28 Mar 2013 15:17:47 -0700
Subject: [PATCH 2/2] ip: remove unnecessary ll_init_map
Don't call ll_init_map on modify operations Saves significant overhead with 1000's of devices.
--- ip/ipaddress.c | 2 -- ip/ipaddrlabel.c | 2 -- ip/iplink.c | 2 -- ip/iproute.c | 6 ------ ip/xfrm_monitor.c | 2 -- 5 files changed, 14 deletions(-)
ip/ipaddress.c +0 -2
@@ -1365,8 +1365,6 @@ static int ipaddr_modify(int cmd, int flags, int argc, char **argv)if (!scoped && cmd !=RTM_DELADDR ) req.ifa.ifa_scope =default_scope (&lcl); -ll_init_map (&rth); -if ((req.ifa.ifa_index =ll_name_to_index (d)) ==0 ) {fprintf (stderr,"Cannot find device \"%s\"\n" , d);return -1 ;
ip/ipaddrlabel.c +0 -2
@@ -246,8 +246,6 @@ static int ipaddrlabel_flush(int argc, char **argv)int do_ipaddrlabel (int argc,char **argv) { -ll_init_map (&rth); -if (argc <1 ) {return ipaddrlabel_list (0 ,NULL ); } else if (matches(argv[0], "list") == 0 ||
ip/iplink.c +0 -2
@@ -533,8 +533,6 @@ static int iplink_modify(int cmd, unsigned int flags, int argc, char **argv) } } -ll_init_map (&rth); -if (!(flags &NLM_F_CREATE )) {if (!dev) { fprintf(stderr, "Not enough information: \"dev\" "
ip/iproute.c +0 -6
@@ -970,8 +970,6 @@ static int iproute_modify(int cmd, unsigned flags, int argc, char **argv)if (d || nhs_ok) {int idx; -ll_init_map (&rth); -if (d) {if ((idx =ll_name_to_index (d)) ==0 ) {fprintf (stderr,"Cannot find device \"%s\"\n" , d);
@@ -1265,8 +1263,6 @@ static int iproute_list_flush_or_save(int argc, char **argv, int action)if (do_ipv6 ==AF_UNSPEC && filter.tb) do_ipv6 =AF_INET ; -ll_init_map (&rth); -if (id || od) {int idx;
@@ -1452,8 +1448,6 @@ static int iproute_get(int argc, char **argv)exit (1 ); } -ll_init_map (&rth); -if (idev || odev) {int idx;
ip/xfrm_monitor.c +0 -2
@@ -408,8 +408,6 @@ int do_xfrm_monitor(int argc, char **argv)return rtnl_from_file (fp, xfrm_accept_msg, (void *)stdout); } -//ll_init_map(&rth); -if (rtnl_open_byproto (&rth, groups,NETLINK_XFRM ) <0 )exit (1 );
Signature · 1 line
1.7.10.4
Quoted 2 lines · Stephen Hemminger
Try the following two patches. It adds a name hash list, and uses Eric's idea to avoid loading map on add/delete operations.
On my microbenchmark of just creating 5000 veth pairs this takes pairs 16s instead of 13s of my earlier hacks but that is well down in the usable range.
Deleting all of those network interfaces one by one takes me 60s.
So on the microbenchmark side this looks like a good improvement and pretty usable.
I expect Benoit's container startup workload will also reflect this, but it will be interesting to see the actual result.
Eric
Quoted 3 lines · Eric W. Biederman
On my microbenchmark of just creating 5000 veth pairs this takes pairs 16s instead of 13s of my earlier hacks but that is well down in the usable range.
I guess most of the time is taken by sysctl_check_table()
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Eric W. Biederman
On my microbenchmark of just creating 5000 veth pairs this takes pairs 16s instead of 13s of my earlier hacks but that is well down in the usable range.
I guess most of the time is taken by
sysctl_check_table()
All of the significant sysctl slowdowns were fixed in 3.4. If you see something of sysctl show up in a trace I would be happy to talk about it. The kernel side seems to be creating N network devices seems to take NlogN time now. Both sysfs and sysctl store directories as rbtrees removing their previous bottlenecks.
The loop I timed at 16s was just:
time for i in $(seq 1 5000) ; do ip link add a$i type veth peer name b$i; done
There is plenty of room for inefficiencies in 10000 network devices and 5000 forks+execs.
Eric
Quoted 16 lines · Eric W. Biederman
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Eric W. Biederman
On my microbenchmark of just creating 5000 veth pairs this takes pairs 16s instead of 13s of my earlier hacks but that is well down in the usable range.
I guess most of the time is taken by
sysctl_check_table()All of the significant sysctl slowdowns were fixed in 3.4. If you see something of sysctl show up in a trace I would be happy to talk about it. The kernel side seems to be creating N network devices seems to take NlogN time now. Both sysfs and sysctl store directories as rbtrees removing their previous bottlenecks.
Quoted 6 lines
The loop I timed at 16s was just:
time for i in $(seq 1 5000) ; do ip link add a$i type veth peer name b$i; done
There is plenty of room for inefficiencies in 10000 network devices and 5000 forks+execs.
Ah right, the sysctl part is fixed ;)
In batch mode, I can create these veth pairs in 4 seconds
for i in $(seq 1 5000) ; do echo link add a$i type veth peer name b$i; done | ip -batch -
Quoted 24 lines · Eric Dumazet
Quoted 16 lines · Eric W. Biederman
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Eric W. Biederman
On my microbenchmark of just creating 5000 veth pairs this takes pairs 16s instead of 13s of my earlier hacks but that is well down in the usable range.
I guess most of the time is taken by
sysctl_check_table()All of the significant sysctl slowdowns were fixed in 3.4. If you see something of sysctl show up in a trace I would be happy to talk about it. The kernel side seems to be creating N network devices seems to take NlogN time now. Both sysfs and sysctl store directories as rbtrees removing their previous bottlenecks.
Quoted 6 lines
The loop I timed at 16s was just:
time for i in $(seq 1 5000) ; do ip link add a$i type veth peer name b$i; done
There is plenty of room for inefficiencies in 10000 network devices and 5000 forks+execs.
Quoted 6 lines
Ah right, the sysctl part is fixed ;)
In batch mode, I can create these veth pairs in 4 seconds
for i in $(seq 1 5000) ; do echo link add a$i type veth peer name b$i; done | ip -batch -
Yes. The interesting story here is that the bottleneck before these
patches was the ll_init_map function of iproute2. Which resulted in an
over an order of magnitude slowdown of when starting iproute on a system with lots of network devices.
It is still unclear where iproute comes into the picture in the original problem scenario of creating 2000 containers each with 2 veth pairs. But apparently it was.
As the fundamental use case here was taking 2000 separate independent actions it turns out to be important for things to not slowdown unreasonably outside of batch mode. So I was explicitly testing the non-batch mode performance.
On the flip side it might be interesting to see if we can get batch mode deletes to batch in the kernel, so we don't have to wait for through syncrhonize_rcu_expidited for each of them. Although for the container case I can just drop the last reference to the network namespace and all of the network device removals will batch.
Ultimately shrug. Except in the previous O(N^2) userspace behavior there don't seem to be any practical performance problems with this many network devices. What is interesting is that this many network devices is becoming interesting on inexpensive COTS servers, for cases that are not purely network focused.
Eric
Quoted 4 lines · Eric Dumazet
In batch mode, I can create these veth pairs in 4 seconds
for i in $(seq 1 5000) ; do echo link add a$i type veth peer name b$i; done | ip -batch -
At rmmod time, 30% of cpu is spent in packet_notifier()
Maybe we can do something about this.
30.85% rmmod [kernel.kallsyms] [k]packet_notifier
|
--- packet_notifier
notifier_call_chain
raw_notifier_call_chain
call_netdevice_notifiers
rollback_registered_many
unregister_netdevice_many
__rtnl_link_unregister
rtnl_link_unregister
0xffffffffa0044868
sys_delete_module
sysenter_dispatchQuoted 11 lines · Eric Dumazet
Quoted 4 lines · Eric Dumazet
In batch mode, I can create these veth pairs in 4 seconds
for i in $(seq 1 5000) ; do echo link add a$i type veth peer name b$i; done | ip -batch -
At rmmod time, 30% of cpu is spent in
packet_notifier()Maybe we can do something about this.
An interesting thought. I had a patch I never got around to pushing a while back that would have had an effect.
It is my observation that the vast majority of packet filters apply not to the entire machine but to an individual interface. In fact you have to work pretty hard to get tools like tcpdump to dump all of the interfaces at once.
So to speed things up for machines that have a lot of these things the idea was to create per device lists for the filters that only needed to be run on a single device. In this case it looks like we could potentially create per device lists for of the listening sockets as well.
In general these lists should be short so the search can also be short.
But I am curious do you actually have a tcpdump or something similar running on your box that is using AF_PACKET sockets? Perhaps a dhcp client?
I am a little surprised that your default case has anything on the lists to trigger any work in the packet_notifier notifier.
Quoted 14 lines
30.85% rmmod [kernel.kallsyms] [k]
packet_notifier| --- packet_notifier notifier_call_chain raw_notifier_call_chain call_netdevice_notifiers rollback_registered_many unregister_netdevice_many __rtnl_link_unregister rtnl_link_unregister 0xffffffffa0044868 sys_delete_module sysenter_dispatch
Eric
Quoted 13 lines · Eric W. Biederman
An interesting thought. I had a patch I never got around to pushing a while back that would have had an effect.
It is my observation that the vast majority of packet filters apply not to the entire machine but to an individual interface. In fact you have to work pretty hard to get tools like tcpdump to dump all of the interfaces at once.
So to speed things up for machines that have a lot of these things the idea was to create per device lists for the filters that only needed to be run on a single device. In this case it looks like we could potentially create per device lists for of the listening sockets as well.
Quoted 8 lines
In general these lists should be short so the search can also be short.
But I am curious do you actually have a tcpdump or something similar running on your box that is using
AF_PACKETsockets? Perhaps a dhcp client?I am a little surprised that your default case has anything on the lists to trigger any work in the
packet_notifiernotifier.
Hmm, it might be a local daemon on my lab machine which does a PACKET_ADD_MEMBERSHIP for each created interface.
So my machine spend time in packet_dev_mclist(), with a quadratic behavior at rmmod.
Hello,
Here are my tests of the last patches on 3 different platforms all running 3.8.5 :
Time are in seconds :
8x 3.7Ghz virtual cores
# veth create delete 1000 14 18 2000 39 56 5000 256 161 10000 1200 399
8x 3.2Ghz virtual cores
# veth create delete
1000 19 40
2000 118 66
5000 305 251
32x 2Ghz virtual cores , 2 sockets
# veth create delete 1000 35 86
2000 120 90
5000 724 245
Compared to initial iproute2 performance on this 32 virtual core system :
5000 1143 1185
"perf record" for creation of 5000 veth on the 32 core system :
# captured on: Fri Mar 29 14:03:35 2013
# hostname : ieng-serv06
# os release : 3.8.5
# perf version : 3.8.5
# arch : x86_64
# nrcpus online : 32
# nrcpus avail : 32
# cpudesc : Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
# cpuid : GenuineIntel,6,45,7
# total memory : 264124548 kB
# cmdline : /usr/src/linux-3.8.5/tools/perf/perf record -a ./test3.script
# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 =
0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_guest = 1,
precise_ip = 0, id = { 36, 37, 38, 39, 40, 41, 42,
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# pmu mappings: cpu = 4, software = 1, uncore_pcu = 15, tracepoint = 2,
uncore_imc_0 = 17, uncore_imc_1 = 18, uncore_imc_2 = 19, uncore_imc_3 =
20, uncore_qpi_0 = 21, uncore_qpi_1 = 22, unco
# ========
#
# Samples: 9M of event 'cycles'
# Event count (approx.): 2894480238483
#
# Overhead Command Shared Object
Symbol
# ........ ............... .............................
...............................................
#
15.17% sudo [kernel.kallsyms] [k]
snmp_fold_field
5.94% sudo libc-2.15.so [.]
0x00000000000802cd
5.64% sudo [kernel.kallsyms] [k]
find_next_bit
3.21% init libnih.so.1.0.0 [.]
nih_list_add_after
2.12% swapper [kernel.kallsyms] [k] intel_idle 1.94% init [kernel.kallsyms] [k] page_fault
1.93% sed libc-2.15.so [.]0x00000000000a1368
1.93% sudo [kernel.kallsyms] [k]rtnl_fill_ifinfo
1.92% sudo [veth] [k]veth_get_stats64
1.78% sudo [kernel.kallsyms] [k] memcpy
1.53% ifquery libc-2.15.so [.]0x000000000007f52b
1.24% init libc-2.15.so [.]0x000000000008918f
1.05% sudo [kernel.kallsyms] [k]inet6_fill_ifla6_attrs
0.98% init [kernel.kallsyms] [k]copy_pte_range
0.88% irqbalance libc-2.15.so [.]0x00000000000802cd
0.85% sudo [kernel.kallsyms] [k] memset
0.72% sed ld-2.15.so [.]0x000000000000a226
0.68% ifquery ld-2.15.so [.]0x00000000000165a0
0.64% init libnih.so.1.0.0 [.]
nih_tree_next_post_full
0.61% bridge-network- libc-2.15.so [.]0x0000000000131e2a
0.59% init [kernel.kallsyms] [k] do_wp_page
0.59% ifquery [kernel.kallsyms] [k] page_fault
0.54% sed [kernel.kallsyms] [k] page_fault
Regards
Benoit
Quoted 22 lines · Eric W. Biederman
Quoted 3 lines · Stephen Hemminger
Try the following two patches. It adds a name hash list, and uses Eric's idea
to avoid loading map on add/delete operations.On my microbenchmark of just creating 5000 veth pairs this takes pairs 16s instead of 13s of my earlier hacks but that is well down in the usable range.
Deleting all of those network interfaces one by one takes me 60s.
So on the microbenchmark side this looks like a good improvement and pretty usable.
I expect Benoit's container startup workload will also reflect this, but it will be interesting to see the actual result.
Eric
Quoted 121 lines · Benoit Lourdelet
Hello,
Here are my tests of the last patches on 3 different platforms all running 3.8.5 :
Time are in seconds :
8x 3.7Ghz virtual cores
# veth create delete 1000 14 18 2000 39 56 5000 256 161 10000 1200 3998x 3.2Ghz virtual cores
# veth create delete1000 19 40
2000 118 66
5000 305 25132x 2Ghz virtual cores , 2 sockets
# veth create delete 1000 35 862000 120 90
5000 724 245
Compared to initial iproute2 performance on this 32 virtual core system :
5000 1143 1185"perf record" for creation of 5000 veth on the 32 core system :
# captured on: Fri Mar 29 14:03:35 2013 # hostname : ieng-serv06 # os release : 3.8.5 # perf version : 3.8.5 # arch : x86_64 # nrcpus online : 32 # nrcpus avail : 32 # cpudesc : Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz # cpuid : GenuineIntel,6,45,7 # total memory : 264124548 kB # cmdline : /usr/src/linux-3.8.5/tools/perf/perf record -a ./test3.script # event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_guest = 1, precise_ip = 0, id = { 36, 37, 38, 39, 40, 41, 42, # HEADER_CPU_TOPOLOGY info available, use -I to display # HEADER_NUMA_TOPOLOGY info available, use -I to display # pmu mappings: cpu = 4, software = 1, uncore_pcu = 15, tracepoint = 2, uncore_imc_0 = 17, uncore_imc_1 = 18, uncore_imc_2 = 19, uncore_imc_3 = 20, uncore_qpi_0 = 21, uncore_qpi_1 = 22, unco # ======== # # Samples: 9M of event 'cycles' # Event count (approx.): 2894480238483 # # Overhead Command Shared Object Symbol # ........ ............... ............................. ............................................... # 15.17% sudo [kernel.kallsyms] [k] snmp_fold_field 5.94% sudo libc-2.15.so [.] 0x00000000000802cd 5.64% sudo [kernel.kallsyms] [k] find_next_bit 3.21% init libnih.so.1.0.0 [.] nih_list_add_after 2.12% swapper [kernel.kallsyms] [k] intel_idle1.94% init [kernel.kallsyms] [k]
page_fault1.93% sed libc-2.15.so [.]
0x00000000000a1368
1.93% sudo [kernel.kallsyms] [k]rtnl_fill_ifinfo
1.92% sudo [veth] [k]veth_get_stats64
1.78% sudo [kernel.kallsyms] [k] memcpy1.53% ifquery libc-2.15.so [.]
0x000000000007f52b
1.24% init libc-2.15.so [.]0x000000000008918f
1.05% sudo [kernel.kallsyms] [k]inet6_fill_ifla6_attrs
0.98% init [kernel.kallsyms] [k]copy_pte_range
0.88% irqbalance libc-2.15.so [.]0x00000000000802cd
0.85% sudo [kernel.kallsyms] [k] memset0.72% sed ld-2.15.so [.]
0x000000000000a226
0.68% ifquery ld-2.15.so [.]0x00000000000165a00.64% init libnih.so.1.0.0 [.] nih_tree_next_post_full 0.61% bridge-network- libc-2.15.so [.]
0x0000000000131e2a
0.59% init [kernel.kallsyms] [k]do_wp_page0.59% ifquery [kernel.kallsyms] [k]
page_fault0.54% sed [kernel.kallsyms] [k]
page_fault
Quoted 7 lines
Regards
Benoit
This means lxc-start does the same thing than ip :
It fetches the whole device list.
You could strace it to have a confirmation.
Sorry Eric,
This is not an lxc-start perf report.This is an "ip" report".
Will run an "lxc-start" perf ASAP.
Regards
Benoit
Quoted 148 lines · Eric Dumazet
Quoted 129 lines · Benoit Lourdelet
Hello,
Here are my tests of the last patches on 3 different platforms all running 3.8.5 :
Time are in seconds :
8x 3.7Ghz virtual cores
# veth create delete 1000 14 18 2000 39 56 5000 256 161 10000 1200 3998x 3.2Ghz virtual cores
# veth create delete1000 19 40
2000 118 66
5000 305 25132x 2Ghz virtual cores , 2 sockets
# veth create delete 1000 35 862000 120 90
5000 724 245
Compared to initial iproute2 performance on this 32 virtual core system :
5000 1143 1185"perf record" for creation of 5000 veth on the 32 core system :
# captured on: Fri Mar 29 14:03:35 2013 # hostname : ieng-serv06 # os release : 3.8.5 # perf version : 3.8.5 # arch : x86_64 # nrcpus online : 32 # nrcpus avail : 32 # cpudesc : Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz # cpuid : GenuineIntel,6,45,7 # total memory : 264124548 kB # cmdline : /usr/src/linux-3.8.5/tools/perf/perf record -a ./test3.script # event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_guest = 1, precise_ip = 0, id = { 36, 37, 38, 39, 40, 41, 42, # HEADER_CPU_TOPOLOGY info available, use -I to display # HEADER_NUMA_TOPOLOGY info available, use -I to display # pmu mappings: cpu = 4, software = 1, uncore_pcu = 15, tracepoint = 2, uncore_imc_0 = 17, uncore_imc_1 = 18, uncore_imc_2 = 19, uncore_imc_3 = 20, uncore_qpi_0 = 21, uncore_qpi_1 = 22, unco # ======== # # Samples: 9M of event 'cycles' # Event count (approx.): 2894480238483 # # Overhead Command Shared Object Symbol # ........ ............... ............................. ............................................... # 15.17% sudo [kernel.kallsyms] [k] snmp_fold_field 5.94% sudo libc-2.15.so [.] 0x00000000000802cd 5.64% sudo [kernel.kallsyms] [k] find_next_bit 3.21% init libnih.so.1.0.0 [.] nih_list_add_after 2.12% swapper [kernel.kallsyms] [k] intel_idle1.94% init [kernel.kallsyms] [k]
page_fault1.93% sed libc-2.15.so [.]
0x00000000000a1368
1.93% sudo [kernel.kallsyms] [k]rtnl_fill_ifinfo
1.92% sudo [veth] [k]veth_get_stats64
1.78% sudo [kernel.kallsyms] [k] memcpy1.53% ifquery libc-2.15.so [.]
0x000000000007f52b
1.24% init libc-2.15.so [.]0x000000000008918f
1.05% sudo [kernel.kallsyms] [k]inet6_fill_ifla6_attrs
0.98% init [kernel.kallsyms] [k]copy_pte_range
0.88% irqbalance libc-2.15.so [.]0x00000000000802cd
0.85% sudo [kernel.kallsyms] [k] memset0.72% sed ld-2.15.so [.]
0x000000000000a226
0.68% ifquery ld-2.15.so [.]0x00000000000165a00.64% init libnih.so.1.0.0 [.] nih_tree_next_post_full 0.61% bridge-network- libc-2.15.so [.]
0x0000000000131e2a
0.59% init [kernel.kallsyms] [k]do_wp_page0.59% ifquery [kernel.kallsyms] [k]
page_fault0.54% sed [kernel.kallsyms] [k]
page_faultQuoted 7 lines
Regards
Benoit
This means lxc-start does the same thing than ip :
It fetches the whole device list.
You could strace it to have a confirmation.