Batched netns improvements.
Subsystems (5)
cover Batched netns improvements.
This series of patches takes my simple 4k network namespace exit test from about 44s to 1.6s, with a minuscule increase in code side.
While batching has given me big improvements there are a few changes in there that improve the performance of cleaning up a single network namespace.
Eric
include/net/fib_rules.h | 3 +- include/net/inet_timewait_sock.h | 6 +- include/net/net_namespace.h | 2 + include/net/netns/xfrm.h | 1 + net/core/dev.c | 25 +++++++ net/core/fib_rules.c | 36 +++++++++- net/core/net_namespace.c | 138 ++++++++++++++++---------------------- net/decnet/dn_rules.c | 22 +++--- net/ipv4/fib_rules.c | 12 +--- net/ipv4/inet_timewait_sock.c | 45 ++++++++----- net/ipv4/tcp_ipv4.c | 11 ++- net/ipv6/fib6_rules.c | 22 ++---- net/ipv6/tcp_ipv6.c | 11 ++- net/xfrm/xfrm_user.c | 18 +++-- 14 files changed, 200 insertions(+), 152 deletions(-)
PATCH 1/7 net: Add support for batching network namespace cleanups
From: Eric W. Biederman <[email protected]>
- Add exit_list to struct net to support building lists of network namespaces to cleanup.
- Add exit_batch to pernet_operations to allow running operations only once during a network namespace exit. Instead of once per network namespace.
- Factor opt ops_exit_list and ops_exit_free so the logic with cleanup up a network namespace does not need to be duplicated.
--- include/net/net_namespace.h | 2 + net/core/net_namespace.c | 122 +++++++++++++++++++++--------------------- 2 files changed, 63 insertions(+), 61 deletions(-)
include/net/net_namespace.h +2 -0
@@ -43,6 +43,7 @@ struct net {#endif struct list_head list;/* list of network namespaces */ struct list_head cleanup_list;/* namespaces on death row */ +struct list_head exit_list;/* Use only net_mutex */ struct proc_dir_entry *proc_net;struct proc_dir_entry *proc_net_stat;
@@ -236,6 +237,7 @@ struct pernet_operations {struct list_head list;int (*init)(struct net *net);void (*exit)(struct net *net); +void (*exit_batch)(struct list_head *net_exit_list);int *id;size_t size; };
net/core/net_namespace.c +61 -61
202 lines not on this page · raw message
Signature · 1 line
1.6.5.2.143.g8cc62
PATCH 2/7 net: Move network device exit batching
In reply to Eric W. BiedermanFrom: Eric W. Biederman <[email protected]>
Move network device exit batching from a special case in net_namespace.c to using common mechanisms in dev.c
--- net/core/dev.c | 25 +++++++++++++++++++++++++ net/core/net_namespace.c | 24 ------------------------ 2 files changed, 25 insertions(+), 24 deletions(-)
net/core/dev.c +25 -0
@@ -5766,8 +5766,33 @@ static void __net_exit default_device_exit(struct net *net)rtnl_unlock (); } +static void __net_exit default_device_exit_batch(struct list_head *net_list) +{ +/* At exit all network devices most be removed from a network +* namespace. Do this in the reverse order of registeration. +* Do this across as many network namespaces as possible to +* improve batching efficiency. +*/ +struct net_device *dev; +struct net *net; +LIST_HEAD (dev_kill_list); + +rtnl_lock (); +list_for_each_entry (net, net_list, exit_list) { +for_each_netdev_reverse (net, dev) { +if (dev->rtnl_link_ops) + dev->rtnl_link_ops->dellink (dev, &dev_kill_list); +else +unregister_netdevice_queue (dev, &dev_kill_list); + } + } +unregister_netdevice_many (&dev_kill_list); +rtnl_unlock (); +} +static struct pernet_operations __net_initdata default_device_ops = { .exit = default_device_exit, + .exit_batch = default_device_exit_batch, };/*
net/core/net_namespace.c +0 -24
@@ -8,10 +8,8 @@#include <linux/idr.h> #include <linux/rculist.h> #include <linux/nsproxy.h> -#include <linux/netdevice.h> #include <net/net_namespace.h> #include <net/netns/generic.h> -#include <net/rtnetlink.h> /* * Our network namespace constructor/destructor lists
@@ -29,20 +27,6 @@ EXPORT_SYMBOL(init_net);#define INITIAL_NET_GEN_PTRS 13/* +1 for len +2 for rcu_head */ -static void unregister_netdevices (struct net *net,struct list_head *list) -{ -struct net_device *dev; -/* At exit all network devices most be removed from a network -* namespace. Do this in the reverse order of registeration. -*/ -for_each_netdev_reverse (net, dev) { -if (dev->rtnl_link_ops) - dev->rtnl_link_ops->dellink (dev, list); -else -unregister_netdevice_queue (dev, list); - } -} -static int ops_init (const struct pernet_operations *ops,struct net *net) {int err;
@@ -78,14 +62,6 @@ static void ops_exit_list(const struct pernet_operations *ops,list_for_each_entry (net, net_exit_list, exit_list) ops->exit (net); } -if (&ops->list == first_device) { -LIST_HEAD (dev_kill_list); -rtnl_lock (); -list_for_each_entry (net, net_exit_list, exit_list) -unregister_netdevices (net, &dev_kill_list); -unregister_netdevice_many (&dev_kill_list); -rtnl_unlock (); - }if (ops->exit_batch) ops->exit_batch (net_exit_list); }
Signature · 1 line
1.6.5.2.143.g8cc62
PATCH 3/7 net: Allow xfrm_user_net_exit to batch efficiently.
In reply to Eric W. BiedermanFrom: Eric W. Biederman <[email protected]>
xfrm.nlsk is provided by the xfrm_user module and is access via rcu from other parts of the xfrm code. Add xfrm.nlsk_stash a copy of xfrm.nlsk that will never be set to NULL. This allows the synchronize_net and netlink_kernel_release to be deferred until a whole batch of xfrm.nlsk sockets have been set to NULL.
--- include/net/netns/xfrm.h | 1 + net/xfrm/xfrm_user.c | 18 ++++++++++-------- 2 files changed, 11 insertions(+), 8 deletions(-)
include/net/netns/xfrm.h +1 -0
@@ -43,6 +43,7 @@ struct netns_xfrm {struct work_struct policy_hash_work;struct sock *nlsk; +struct sock *nlsk_stash;u32 sysctl_aevent_etime;u32 sysctl_aevent_rseqth;
net/xfrm/xfrm_user.c +10 -8
@@ -2721,22 +2721,24 @@ static int __net_init xfrm_user_net_init(struct net *net) xfrm_netlink_rcv, NULL, THIS_MODULE);if (nlsk ==NULL )return -ENOMEM ; + net->xfrm.nlsk_stash = nlsk;/* Don't set to NULL */ rcu_assign_pointer (net->xfrm.nlsk, nlsk);return 0 ; } -static void __net_exit xfrm_user_net_exit(struct net *net) +static void __net_exit xfrm_user_net_exit(struct list_head *net_exit_list) { -struct sock *nlsk = net->xfrm.nlsk; - -rcu_assign_pointer (net->xfrm.nlsk,NULL ); -synchronize_rcu (); -netlink_kernel_release (nlsk); +struct net *net; +list_for_each_entry (net, net_exit_list, exit_list) +rcu_assign_pointer (net->xfrm.nlsk,NULL ); +synchronize_net (); +list_for_each_entry (net, net_exit_list, exit_list) +netlink_kernel_release (net->xfrm.nlsk_stash); }static struct pernet_operations xfrm_user_net_ops = { - .init = xfrm_user_net_init, - .exit = xfrm_user_net_exit, + .init = xfrm_user_net_init, + .exit_batch = xfrm_user_net_exit, };static int __init xfrm_user_init (void )
Signature · 1 line
1.6.5.2.143.g8cc62
PATCH 4/7 netns: Add an explicit rcu_barrier to unregister_pernet_{device|subsys}
In reply to Eric W. BiedermanFrom: Eric W. Biederman <[email protected]>
This allows namespace exit methods to batch work that comes requires an rcu barrier using call_rcu without having to treat the unregister_pernet_operations cases specially.
--- net/core/net_namespace.c | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-)
net/core/net_namespace.c +6 -2
@@ -413,8 +413,11 @@ again: } } error =__register_pernet_operations (list, ops); -if (error && ops->id) -ida_remove (&net_generic_ids, *ops->id); +if (error) { +rcu_barrier (); +if (ops->id) +ida_remove (&net_generic_ids, *ops->id); + }return error; }
@@ -423,6 +426,7 @@ static void unregister_pernet_operations(struct pernet_operations *ops) {__unregister_pernet_operations (ops); +rcu_barrier ();if (ops->id)ida_remove (&net_generic_ids, *ops->id); }
Signature · 1 line
1.6.5.2.143.g8cc62
PATCH 5/7 net: Allow fib_rule_unregister to batch
In reply to Eric W. BiedermanFrom: Eric W. Biederman <[email protected]>
Refactor the code so fib_rules_register always takes a template instead of the actual fib_rules_ops structure that will be used. This is required for network namespace support so 2 out of the 3 callers already do this, it allows the error handling to be made common, and it allows fib_rules_unregister to free the template for hte caller.
Modify fib_rules_unregister to use call_rcu instead of syncrhonize_rcu to allw multiple namespaces to be cleaned up in the same rcu grace period.
--- include/net/fib_rules.h | 3 ++- net/core/fib_rules.c | 36 +++++++++++++++++++++++++++++++++--- net/decnet/dn_rules.c | 22 ++++++++++++---------- net/ipv4/fib_rules.c | 12 +++--------- net/ipv6/fib6_rules.c | 22 +++++++--------------- 5 files changed, 57 insertions(+), 38 deletions(-)
include/net/fib_rules.h +2 -1
@@ -64,6 +64,7 @@ struct fib_rules_ops {struct list_head rules_list;struct module *owner;struct net *fro_net; +struct rcu_head rcu; };#define FRA_GENERIC_POLICY \
@@ -99,7 +100,7 @@ static inline u32 frh_get_table(struct fib_rule_hdr *frh, struct nlattr **nla)return frh->table; } -extern int fib_rules_register (struct fib_rules_ops *); +extern struct fib_rules_ops *fib_rules_register (struct fib_rules_ops *,struct net *);extern void fib_rules_unregister (struct fib_rules_ops *);extern void fib_rules_cleanup_ops (struct fib_rules_ops *);
net/core/fib_rules.c +33 -3
@@ -72,7 +72,7 @@ static void flush_route_cache(struct fib_rules_ops *ops) ops->flush_cache (ops); } -int fib_rules_register (struct fib_rules_ops *ops) +static int __fib_rules_register (struct fib_rules_ops *ops) {int err = -EEXIST ;struct fib_rules_ops *o;
@@ -102,6 +102,28 @@ errout:return err; } +struct fib_rules_ops * +fib_rules_register (struct fib_rules_ops *tmpl,struct net *net) +{ +struct fib_rules_ops *ops; +int err; + + ops =kmemdup (tmpl,sizeof (*ops),GFP_KERNEL ); +if (ops ==NULL ) +return ERR_PTR (-ENOMEM ); + +INIT_LIST_HEAD (&ops->rules_list); + ops->fro_net = net; + + err =__fib_rules_register (ops); +if (err) { +kfree (ops); + ops =ERR_PTR (err); + } + +return ops; +} +EXPORT_SYMBOL_GPL (fib_rules_register);void fib_rules_cleanup_ops (struct fib_rules_ops *ops)
@@ -115,6 +137,15 @@ void fib_rules_cleanup_ops(struct fib_rules_ops *ops) }EXPORT_SYMBOL_GPL (fib_rules_cleanup_ops); +static void fib_rules_put_rcu (struct rcu_head *head) +{ + struct fib_rules_ops *ops = container_of(head, struct fib_rules_ops, rcu); +struct net *net = ops->fro_net; + +release_net (net); +kfree (ops); +} +void fib_rules_unregister (struct fib_rules_ops *ops) {struct net *net = ops->fro_net;
@@ -124,8 +155,7 @@ void fib_rules_unregister(struct fib_rules_ops *ops)fib_rules_cleanup_ops (ops);spin_unlock (&net->rules_mod_lock); -synchronize_rcu (); -release_net (net); +call_rcu (&ops->rcu, fib_rules_put_rcu); }EXPORT_SYMBOL_GPL (fib_rules_unregister);
net/decnet/dn_rules.c +12 -10
@@ -33,7 +33,7 @@#include <net/dn_dev.h> #include <net/dn_route.h> -static struct fib_rules_ops dn_fib_rules_ops; +static struct fib_rules_ops *dn_fib_rules_ops;struct dn_fib_rule {
@@ -56,7 +56,7 @@ int dn_fib_lookup(struct flowi *flp, struct dn_fib_res *res) };int err; - err =fib_rules_lookup (&dn_fib_rules_ops, flp,0 , &arg); + err =fib_rules_lookup (dn_fib_rules_ops, flp,0 , &arg); res->r = arg.rule;return err;
@@ -217,9 +217,9 @@ static u32 dn_fib_rule_default_pref(struct fib_rules_ops *ops)struct list_head *pos;struct fib_rule *rule; -if (!list_empty (&dn_fib_rules_ops.rules_list)) { - pos = dn_fib_rules_ops.rules_list.next; -if (pos->next != &dn_fib_rules_ops.rules_list) { +if (!list_empty (&dn_fib_rules_ops->rules_list)) { + pos = dn_fib_rules_ops->rules_list.next; +if (pos->next != &dn_fib_rules_ops->rules_list) { rule = list_entry(pos->next, struct fib_rule, list);if (rule->pref)return rule->pref -1 ;
@@ -234,7 +234,7 @@ static void dn_fib_rule_flush_cache(struct fib_rules_ops *ops)dn_rt_cache_flush (-1 ); } -static struct fib_rules_ops dn_fib_rules_ops = { +static struct fib_rules_ops dn_fib_rules_ops_template = { .family = AF_DECnet, .rule_size =sizeof (struct dn_fib_rule ), .addr_size =sizeof (u16),
@@ -247,21 +247,23 @@ static struct fib_rules_ops dn_fib_rules_ops = { .flush_cache = dn_fib_rule_flush_cache, .nlgroup = RTNLGRP_DECnet_RULE, .policy = dn_fib_rule_policy, - .rules_list = LIST_HEAD_INIT(dn_fib_rules_ops.rules_list), .owner = THIS_MODULE, .fro_net = &init_net, };void __init dn_fib_rules_init (void ) { -BUG_ON (fib_default_rule_add (&dn_fib_rules_ops,0x7fff , + dn_fib_rules_ops = +fib_rules_register (&dn_fib_rules_ops_template, &init_net); +BUG_ON (IS_ERR (dn_fib_rules_ops)); +BUG_ON (fib_default_rule_add (dn_fib_rules_ops,0x7fff ,RT_TABLE_MAIN ,0 )); -fib_rules_register (&dn_fib_rules_ops); } void __exit dn_fib_rules_cleanup(void) { -fib_rules_unregister (&dn_fib_rules_ops); +fib_rules_unregister (dn_fib_rules_ops); +rcu_barrier (); }
net/ipv4/fib_rules.c +3 -9
@@ -301,13 +301,9 @@ int __net_init fib4_rules_init(struct net *net)int err;struct fib_rules_ops *ops; - ops =kmemdup (&fib4_rules_ops_template,sizeof (*ops),GFP_KERNEL ); -if (ops ==NULL ) -return -ENOMEM ; -INIT_LIST_HEAD (&ops->rules_list); - ops->fro_net = net; - -fib_rules_register (ops); + ops =fib_rules_register (&fib4_rules_ops_template, net); +if (IS_ERR (ops)) +return PTR_ERR (ops); err =fib_default_rules_init (ops);if (err <0 )
@@ -318,12 +314,10 @@ int __net_init fib4_rules_init(struct net *net) fail:/* also cleans all rules already added */ fib_rules_unregister (ops); -kfree (ops);return err; } void __net_exit fib4_rules_exit(struct net *net) {fib_rules_unregister (net->ipv4.rules_ops); -kfree (net->ipv4.rules_ops); }
net/ipv6/fib6_rules.c +7 -15
@@ -264,16 +264,14 @@ static struct fib_rules_ops fib6_rules_ops_template = {static int fib6_rules_net_init (struct net *net) { +struct fib_rules_ops *ops;int err = -ENOMEM ; - net->ipv6.fib6_rules_ops =kmemdup (&fib6_rules_ops_template, -sizeof (*net->ipv6.fib6_rules_ops), -GFP_KERNEL ); -if (!net->ipv6.fib6_rules_ops) - goto out; + ops =fib_rules_register (&fib6_rules_ops_template, net); +if (IS_ERR (ops)) +return PTR_ERR (ops); + net->ipv6.fib6_rules_ops = ops; - net->ipv6.fib6_rules_ops->fro_net = net; -INIT_LIST_HEAD (&net->ipv6.fib6_rules_ops->rules_list); err =fib_default_rule_add (net->ipv6.fib6_rules_ops,0 ,RT6_TABLE_LOCAL ,FIB_RULE_PERMANENT );
@@ -283,25 +281,19 @@ static int fib6_rules_net_init(struct net *net) err =fib_default_rule_add (net->ipv6.fib6_rules_ops,0x7FFE ,RT6_TABLE_MAIN ,0 );if (err) - goto out_fib6_default_rule_add; + goto out_fib6_rules_ops; - err =fib_rules_register (net->ipv6.fib6_rules_ops); -if (err) - goto out_fib6_default_rule_add; out:return err; -out_fib6_default_rule_add: -fib_rules_cleanup_ops (net->ipv6.fib6_rules_ops); out_fib6_rules_ops: -kfree (net->ipv6.fib6_rules_ops); +fib_rules_unregister (ops); goto out; }static void fib6_rules_net_exit (struct net *net) {fib_rules_unregister (net->ipv6.fib6_rules_ops); -kfree (net->ipv6.fib6_rules_ops); }static struct pernet_operations fib6_rules_net_ops = {
Signature · 1 line
1.6.5.2.143.g8cc62
PATCH 6/7 net: Use rcu lookups in inet_twsk_purge.
In reply to Eric W. BiedermanFrom: Eric W. Biederman <[email protected]>
While we are looking up entries to free there is no reason to take the lock in inet_twsk_purge. We have to drop locks and restart occassionally anyway so adding a few more in case we get on the wrong list because of a timewait move is no big deal. At the same time not taking the lock for long periods of time is much more polite to the rest of the users of the hash table.
In my test configuration of killing 4k network namespaces this change causes 4k back to back runs of inet_twsk_purge on an empty hash table to go from roughly 20.7s to 3.3s, and the total time to destroy 4k network namespaces goes from roughly 44s to 3.3s.
--- net/ipv4/inet_timewait_sock.c | 39 ++++++++++++++++++++++++--------------- 1 files changed, 24 insertions(+), 15 deletions(-)
net/ipv4/inet_timewait_sock.c +24 -15
@@ -427,31 +427,40 @@ void inet_twsk_purge(struct net *net, struct inet_hashinfo *hashinfo,struct inet_timewait_sock *tw;struct sock *sk;struct hlist_nulls_node *node; -int h; +unsigned int slot; -local_bh_disable (); -for (h =0 ; h <= hashinfo->ehash_mask; h++) { -struct inet_ehash_bucket *head = -inet_ehash_bucket (hashinfo, h); -spinlock_t *lock =inet_ehash_lockp (hashinfo, h); +for (slot =0 ; slot <= hashinfo->ehash_mask; slot++) { +struct inet_ehash_bucket *head = &hashinfo->ehash[slot]; +restart_rcu: +rcu_read_lock (); restart: -spin_lock (lock); -sk_nulls_for_each (sk, node, &head->twchain) { - +sk_nulls_for_each_rcu (sk, node, &head->twchain) { tw =inet_twsk (sk);if (!net_eq (twsk_net (tw), net) || tw->tw_family != family)continue ; -atomic_inc (&tw->tw_refcnt); -spin_unlock (lock); +if (unlikely (!atomic_inc_not_zero (&tw->tw_refcnt))) +continue ; + +if (unlikely (!net_eq (twsk_net (tw), net) || + tw->tw_family != family)) { +inet_twsk_put (tw); + goto restart; + } + +rcu_read_unlock ();inet_twsk_deschedule (tw, twdr);inet_twsk_put (tw); - - goto restart; + goto restart_rcu; } -spin_unlock (lock); +/* If the nulls value we got at the end of this lookup is +* not the expected one, we must restart lookup. +* We probably met an item that was moved to another chain. +*/ +if (get_nulls_value (node) != slot) + goto restart; +rcu_read_unlock (); } -local_bh_enable (); }EXPORT_SYMBOL_GPL (inet_twsk_purge);
Signature · 1 line
1.6.5.2.143.g8cc62
Re: [PATCH 6/7] net: Use rcu lookups in inet_twsk_purge.
Quoted 9 lines · Eric W. Biederman
From: Eric W. Biederman <[email protected]>
While we are looking up entries to free there is no reason to take the lock in
inet_twsk_purge. We have to drop locks and restart occassionally anyway so adding a few more in case we get on the wrong list because of a timewait move is no big deal. At the same time not taking the lock for long periods of time is much more polite to the rest of the users of the hash table.
Quoted 7 lines
In my test configuration of killing 4k network namespaces this change causes 4k back to back runs of
inet_twsk_purgeon an empty hash table to go from roughly 20.7s to 3.3s, and the total time to destroy 4k network namespaces goes from roughly 44s to 3.3s.Signed-off-by Eric W. Biederman <[email protected]>
Very nice patch Eric
Quoted 64 lines
--- net/ipv4/inet_timewait_sock.c | 39 ++++++++++++++++++++++++--------------- 1 files changed, 24 insertions(+), 15 deletions(-)net/ipv4/inet_timewait_sock.c +24
-15@@ -427,31 +427,40 @@ void inet_twsk_purge(struct net *net, struct inet_hashinfo *hashinfo,struct inet_timewait_sock *tw;struct sock *sk;struct hlist_nulls_node *node; -int h; +unsigned int slot; -local_bh_disable (); -for (h =0 ; h <= hashinfo->ehash_mask; h++) { -struct inet_ehash_bucket *head = -inet_ehash_bucket (hashinfo, h); -spinlock_t *lock =inet_ehash_lockp (hashinfo, h); +for (slot =0 ; slot <= hashinfo->ehash_mask; slot++) { +struct inet_ehash_bucket *head = &hashinfo->ehash[slot]; +restart_rcu: +rcu_read_lock (); restart: -spin_lock (lock); -sk_nulls_for_each (sk, node, &head->twchain) { - +sk_nulls_for_each_rcu (sk, node, &head->twchain) { tw =inet_twsk (sk);if (!net_eq (twsk_net (tw), net) || tw->tw_family != family)continue ; -atomic_inc (&tw->tw_refcnt); -spin_unlock (lock); +if (unlikely (!atomic_inc_not_zero (&tw->tw_refcnt))) +continue ; + +if (unlikely (!net_eq (twsk_net (tw), net) || + tw->tw_family != family)) { +inet_twsk_put (tw); + goto restart; + } + +rcu_read_unlock ();inet_twsk_deschedule (tw, twdr);inet_twsk_put (tw); - - goto restart; + goto restart_rcu; } -spin_unlock (lock); +/* If the nulls value we got at the end of this lookup is +* not the expected one, we must restart lookup. +* We probably met an item that was moved to another chain. +*/ +if (get_nulls_value (node) != slot) + goto restart; +rcu_read_unlock (); } -local_bh_enable (); }EXPORT_SYMBOL_GPL (inet_twsk_purge);
PATCH 7/7 net: Batch inet_twsk_purge
In reply to Eric W. BiedermanFrom: Eric W. Biederman <[email protected]>
This function walks the whole hashtable so there is no point in passing it a network namespace. Instead I purge all timewait sockets from dead network namespaces that I find. If the namespace is one of the once I am trying to purge I am guaranteed no new timewait sockets can be formed so this will get them all. If the namespace is one I am not acting for it might form a few more but I will call inet_twsk_purge again and shortly to get rid of them. In any even if the network namespace is dead timewait sockets are useless.
Move the calls of inet_twsk_purge into batch_exit routines so that if I am killing a bunch of namespaces at once I will just call inet_twsk_purge once and save a lot of redundant unnecessary work.
My simple 4k network namespace exit test the cleanup time dropped from roughly 8.2s to 1.6s. While the time spent running inet_twsk_purge fell to about 2ms. 1ms for ipv4 and 1ms for ipv6.
--- include/net/inet_timewait_sock.h | 6 +++--- net/ipv4/inet_timewait_sock.c | 10 +++++----- net/ipv4/tcp_ipv4.c | 11 ++++++++--- net/ipv6/tcp_ipv6.c | 11 ++++++++--- 4 files changed, 24 insertions(+), 14 deletions(-)
include/net/inet_timewait_sock.h +3 -3
@@ -212,14 +212,14 @@ extern void inet_twsk_schedule(struct inet_timewait_sock *tw,extern void inet_twsk_deschedule (struct inet_timewait_sock *tw,struct inet_timewait_death_row *twdr); -extern void inet_twsk_purge (struct net *net,struct inet_hashinfo *hashinfo, +extern void inet_twsk_purge (struct inet_hashinfo *hashinfo,struct inet_timewait_death_row *twdr,int family);static inline struct net *twsk_net (const struct inet_timewait_sock *twsk) {#ifdef CONFIG_NET_NS -return twsk->tw_net; +return rcu_dereference (twsk->tw_net);#else return &init_net;#endif
@@ -229,7 +229,7 @@ static inlinevoid twsk_net_set (struct inet_timewait_sock *twsk,struct net *net) {#ifdef CONFIG_NET_NS - twsk->tw_net = net; +rcu_assign_pointer (twsk->tw_net, net);#endif }#endif /* _INET_TIMEWAIT_SOCK_ */
net/ipv4/inet_timewait_sock.c +5 -5
@@ -421,7 +421,7 @@ out:EXPORT_SYMBOL_GPL (inet_twdr_twcal_tick); -void inet_twsk_purge (struct net *net,struct inet_hashinfo *hashinfo, +void inet_twsk_purge (struct inet_hashinfo *hashinfo,struct inet_timewait_death_row *twdr,int family) {struct inet_timewait_sock *tw;
@@ -436,15 +436,15 @@ restart_rcu: restart:sk_nulls_for_each_rcu (sk, node, &head->twchain) { tw =inet_twsk (sk); -if (!net_eq (twsk_net (tw), net) || - tw->tw_family != family) +if ((tw->tw_family != family) || +atomic_read (&twsk_net (tw)->count))continue ;if (unlikely (!atomic_inc_not_zero (&tw->tw_refcnt)))continue ; -if (unlikely (!net_eq (twsk_net (tw), net) || - tw->tw_family != family)) { +if (unlikely ((tw->tw_family != family) || +atomic_read (&twsk_net (tw)->count))) {inet_twsk_put (tw); goto restart; }
net/ipv4/tcp_ipv4.c +8 -3
@@ -2468,12 +2468,17 @@ static int __net_init tcp_sk_init(struct net *net) static void __net_exit tcp_sk_exit(struct net *net) {inet_ctl_sock_destroy (net->ipv4.tcp_sock); -inet_twsk_purge (net, &tcp_hashinfo, &tcp_death_row,AF_INET ); +} + +static void __net_exit tcp_sk_exit_batch(struct list_head *net_exit_list) +{ +inet_twsk_purge (&tcp_hashinfo, &tcp_death_row,AF_INET ); }static struct pernet_operations __net_initdata tcp_sk_ops = { - .init = tcp_sk_init, - .exit = tcp_sk_exit, + .init = tcp_sk_init, + .exit = tcp_sk_exit, + .exit_batch = tcp_sk_exit_batch, };void __init tcp_v4_init (void )
net/ipv6/tcp_ipv6.c +8 -3
@@ -2126,12 +2126,17 @@ static int tcpv6_net_init(struct net *net)static void tcpv6_net_exit (struct net *net) {inet_ctl_sock_destroy (net->ipv6.tcp_sk); -inet_twsk_purge (net, &tcp_hashinfo, &tcp_death_row,AF_INET6 ); +} + +static void tcpv6_net_exit_batch (struct list_head *net_exit_list) +{ +inet_twsk_purge (&tcp_hashinfo, &tcp_death_row,AF_INET6 ); }static struct pernet_operations tcpv6_net_ops = { - .init = tcpv6_net_init, - .exit = tcpv6_net_exit, + .init = tcpv6_net_init, + .exit = tcpv6_net_exit, + .exit_batch = tcpv6_net_exit_batch, };int __init tcpv6_init (void )
Signature · 1 line
1.6.5.2.143.g8cc62
Re: [PATCH 7/7] net: Batch inet_twsk_purge
Quoted 132 lines · Eric W. Biederman
From: Eric W. Biederman <[email protected]>
This function walks the whole hashtable so there is no point in passing it a network namespace. Instead I purge all timewait sockets from dead network namespaces that I find. If the namespace is one of the once I am trying to purge I am guaranteed no new timewait sockets can be formed so this will get them all. If the namespace is one I am not acting for it might form a few more but I will call
inet_twsk_purgeagain and shortly to get rid of them. In any even if the network namespace is dead timewait sockets are useless.Move the calls of
inet_twsk_purgeintobatch_exitroutines so that if I am killing a bunch of namespaces at once I will just callinet_twsk_purgeonce and save a lot of redundant unnecessary work.My simple 4k network namespace exit test the cleanup time dropped from roughly 8.2s to 1.6s. While the time spent running
inet_twsk_purgefell to about 2ms. 1ms for ipv4 and 1ms for ipv6.Signed-off-by Eric W. Biederman <[email protected]>--- include/net/inet_timewait_sock.h | 6 +++--- net/ipv4/inet_timewait_sock.c | 10 +++++----- net/ipv4/tcp_ipv4.c | 11 ++++++++--- net/ipv6/tcp_ipv6.c | 11 ++++++++--- 4 files changed, 24 insertions(+), 14 deletions(-)include/net/inet_timewait_sock.h +3
-3@@ -212,14 +212,14 @@ extern void inet_twsk_schedule(struct inet_timewait_sock *tw,extern void inet_twsk_deschedule (struct inet_timewait_sock *tw,struct inet_timewait_death_row *twdr); -extern void inet_twsk_purge (struct net *net,struct inet_hashinfo *hashinfo, +extern void inet_twsk_purge (struct inet_hashinfo *hashinfo,struct inet_timewait_death_row *twdr,int family);static inline struct net *twsk_net (const struct inet_timewait_sock *twsk) {#ifdef CONFIG_NET_NS -return twsk->tw_net; +return rcu_dereference (twsk->tw_net);#else return &init_net;#endif @@ -229,7 +229,7 @@ static inlinevoid twsk_net_set (struct inet_timewait_sock *twsk,struct net *net) {#ifdef CONFIG_NET_NS - twsk->tw_net = net; +rcu_assign_pointer (twsk->tw_net, net);#endif }#endif /* _INET_TIMEWAIT_SOCK_ */ net/ipv4/inet_timewait_sock.c +5
-5@@ -421,7 +421,7 @@ out:EXPORT_SYMBOL_GPL (inet_twdr_twcal_tick); -void inet_twsk_purge (struct net *net,struct inet_hashinfo *hashinfo, +void inet_twsk_purge (struct inet_hashinfo *hashinfo,struct inet_timewait_death_row *twdr,int family) {struct inet_timewait_sock *tw;@@ -436,15 +436,15 @@ restart_rcu: restart:sk_nulls_for_each_rcu (sk, node, &head->twchain) { tw =inet_twsk (sk); -if (!net_eq (twsk_net (tw), net) || - tw->tw_family != family) +if ((tw->tw_family != family) || +atomic_read (&twsk_net (tw)->count))continue ;if (unlikely (!atomic_inc_not_zero (&tw->tw_refcnt)))continue ; -if (unlikely (!net_eq (twsk_net (tw), net) || - tw->tw_family != family)) { +if (unlikely ((tw->tw_family != family) || +atomic_read (&twsk_net (tw)->count))) {inet_twsk_put (tw); goto restart; }net/ipv4/tcp_ipv4.c +8
-3@@ -2468,12 +2468,17 @@ static int __net_init tcp_sk_init(struct net *net) static void __net_exit tcp_sk_exit(struct net *net) {inet_ctl_sock_destroy (net->ipv4.tcp_sock); -inet_twsk_purge (net, &tcp_hashinfo, &tcp_death_row,AF_INET ); +} + +static void __net_exit tcp_sk_exit_batch(struct list_head *net_exit_list) +{ +inet_twsk_purge (&tcp_hashinfo, &tcp_death_row,AF_INET ); }static struct pernet_operations __net_initdata tcp_sk_ops = { - .init = tcp_sk_init, - .exit = tcp_sk_exit, + .init = tcp_sk_init, + .exit = tcp_sk_exit, + .exit_batch = tcp_sk_exit_batch, };void __init tcp_v4_init (void )net/ipv6/tcp_ipv6.c +5
-1@@ -2126,12 +2126,17 @@ static int tcpv6_net_init(struct net *net)static void tcpv6_net_exit (struct net *net) {inet_ctl_sock_destroy (net->ipv6.tcp_sk); -inet_twsk_purge (net, &tcp_hashinfo, &tcp_death_row,AF_INET6 ); +} + +static void tcpv6_net_exit_batch (struct list_head *net_exit_list) +{ +inet_twsk_purge (&tcp_hashinfo, &tcp_death_row,AF_INET6 ); }
net/ipv6/tcp_ipv6.c static int tcpv6_net_init(struct net *net) · line 2142
Quoted 10 lines
net/ipv6/tcp_ipv6.c +3
-2static struct pernet_operations tcpv6_net_ops = { - .init = tcpv6_net_init, - .exit = tcpv6_net_exit, + .init = tcpv6_net_init, + .exit = tcpv6_net_exit, + .exit_batch = tcpv6_net_exit_batch, };int __init tcpv6_init (void )
OK, but why calling inet_twsk_purge() twice, one for AF_INET, once for AF_INET6
I believe you could zap family check as well in inet_twsk_purge(), and not need tcpv6_net_ops.exit_batch = tcpv6_net_exit_batch
Re: [PATCH 7/7] net: Batch inet_twsk_purge
Quoted 142 lines · Eric Dumazet
Quoted 141 lines · Eric W. Biederman
From: Eric W. Biederman <[email protected]>
This function walks the whole hashtable so there is no point in passing it a network namespace. Instead I purge all timewait sockets from dead network namespaces that I find. If the namespace is one of the once I am trying to purge I am guaranteed no new timewait sockets can be formed so this will get them all. If the namespace is one I am not acting for it might form a few more but I will call
inet_twsk_purgeagain and shortly to get rid of them. In any even if the network namespace is dead timewait sockets are useless.Move the calls of
inet_twsk_purgeintobatch_exitroutines so that if I am killing a bunch of namespaces at once I will just callinet_twsk_purgeonce and save a lot of redundant unnecessary work.My simple 4k network namespace exit test the cleanup time dropped from roughly 8.2s to 1.6s. While the time spent running
inet_twsk_purgefell to about 2ms. 1ms for ipv4 and 1ms for ipv6.Signed-off-by Eric W. Biederman <[email protected]>--- include/net/inet_timewait_sock.h | 6 +++--- net/ipv4/inet_timewait_sock.c | 10 +++++----- net/ipv4/tcp_ipv4.c | 11 ++++++++--- net/ipv6/tcp_ipv6.c | 11 ++++++++--- 4 files changed, 24 insertions(+), 14 deletions(-)include/net/inet_timewait_sock.h +3
-3@@ -212,14 +212,14 @@ extern void inet_twsk_schedule(struct inet_timewait_sock *tw,extern void inet_twsk_deschedule (struct inet_timewait_sock *tw,struct inet_timewait_death_row *twdr); -extern void inet_twsk_purge (struct net *net,struct inet_hashinfo *hashinfo, +extern void inet_twsk_purge (struct inet_hashinfo *hashinfo,struct inet_timewait_death_row *twdr,int family);static inline struct net *twsk_net (const struct inet_timewait_sock *twsk) {#ifdef CONFIG_NET_NS -return twsk->tw_net; +return rcu_dereference (twsk->tw_net);#else return &init_net;#endif @@ -229,7 +229,7 @@ static inlinevoid twsk_net_set (struct inet_timewait_sock *twsk,struct net *net) {#ifdef CONFIG_NET_NS - twsk->tw_net = net; +rcu_assign_pointer (twsk->tw_net, net);#endif }#endif /* _INET_TIMEWAIT_SOCK_ */ net/ipv4/inet_timewait_sock.c +5
-5@@ -421,7 +421,7 @@ out:EXPORT_SYMBOL_GPL (inet_twdr_twcal_tick); -void inet_twsk_purge (struct net *net,struct inet_hashinfo *hashinfo, +void inet_twsk_purge (struct inet_hashinfo *hashinfo,struct inet_timewait_death_row *twdr,int family) {struct inet_timewait_sock *tw;@@ -436,15 +436,15 @@ restart_rcu: restart:sk_nulls_for_each_rcu (sk, node, &head->twchain) { tw =inet_twsk (sk); -if (!net_eq (twsk_net (tw), net) || - tw->tw_family != family) +if ((tw->tw_family != family) || +atomic_read (&twsk_net (tw)->count))continue ;if (unlikely (!atomic_inc_not_zero (&tw->tw_refcnt)))continue ; -if (unlikely (!net_eq (twsk_net (tw), net) || - tw->tw_family != family)) { +if (unlikely ((tw->tw_family != family) || +atomic_read (&twsk_net (tw)->count))) {inet_twsk_put (tw); goto restart; }net/ipv4/tcp_ipv4.c +8
-3@@ -2468,12 +2468,17 @@ static int __net_init tcp_sk_init(struct net *net) static void __net_exit tcp_sk_exit(struct net *net) {inet_ctl_sock_destroy (net->ipv4.tcp_sock); -inet_twsk_purge (net, &tcp_hashinfo, &tcp_death_row,AF_INET ); +} + +static void __net_exit tcp_sk_exit_batch(struct list_head *net_exit_list) +{ +inet_twsk_purge (&tcp_hashinfo, &tcp_death_row,AF_INET ); }static struct pernet_operations __net_initdata tcp_sk_ops = { - .init = tcp_sk_init, - .exit = tcp_sk_exit, + .init = tcp_sk_init, + .exit = tcp_sk_exit, + .exit_batch = tcp_sk_exit_batch, };void __init tcp_v4_init (void )net/ipv6/tcp_ipv6.c +8
-3@@ -2126,12 +2126,17 @@ static int tcpv6_net_init(struct net *net)static void tcpv6_net_exit (struct net *net) {inet_ctl_sock_destroy (net->ipv6.tcp_sk); -inet_twsk_purge (net, &tcp_hashinfo, &tcp_death_row,AF_INET6 ); +} + +static void tcpv6_net_exit_batch (struct list_head *net_exit_list) +{ +inet_twsk_purge (&tcp_hashinfo, &tcp_death_row,AF_INET6 ); }static struct pernet_operations tcpv6_net_ops = { - .init = tcpv6_net_init, - .exit = tcpv6_net_exit, + .init = tcpv6_net_init, + .exit = tcpv6_net_exit, + .exit_batch = tcpv6_net_exit_batch, };
Quoted 7 lines
net/ipv6/tcp_ipv6.c static int tcpv6_net_init(struct net *net) · line 2142
Quoted 1 line
net/ipv6/tcp_ipv6.c +0
-0int __init tcpv6_init (void )OK, but why calling
inet_twsk_purge()twice, one forAF_INET, once forAF_INET6I believe you could zap family check as well in
inet_twsk_purge(), and not need tcpv6_net_ops.exit_batch =tcpv6_net_exit_batch
Technically it is needed if someone does rmmod ipv6. rmmod ipv6 didn't seem to work for me, but if it ever does... That and the cost is now in the noise in human terms.
I think dccp may also need a inet_twsk_purge as well, but I couldn't figure out what it was doing with timewait sockets.
Eric
David Miller
maintainerRe: [PATCH 7/7] net: Batch inet_twsk_purge
From: [email protected] (Eric W. Biederman)
Date: Thu, 03 Dec 2009 05:36:52 -0800
Quoted 2 lines
I think dccp may also need a
inet_twsk_purgeas well, but I couldn't figure out what it was doing with timewait sockets.
DCCP doesn't have time-wait sockets, it's more like UDP than TCP.
Re: [PATCH 7/7] net: Batch inet_twsk_purge
Quoted 7 lines · David Miller
From: [email protected] (Eric W. Biederman)
Date: Thu, 03 Dec 2009 05:36:52 -0800Quoted 2 lines
I think dccp may also need a
inet_twsk_purgeas well, but I couldn't figure out what it was doing with timewait sockets.DCCP doesn't have time-wait sockets, it's more like UDP than TCP.
There is net/dccp/minisocks.c:dccp_time_wait that calls inet_twsk_alloc().
So dccp is doing something with timewait sockets.
Eric
Quoted 2 lines · Eric W. Biederman
This series of patches takes my simple 4k network namespace exit test from about 44s to 1.6s, with a minuscule increase in code side.
Nice.
I will test these as soon as they make it into net-next (just easier for me given the setup).
I should confirm that:
The previous patches which are already in net-next from yesterday do improve performance to under 1 minute. Can we call that a several-factor improvement?;->
Are you planning to give some tender loving to the virtual devices as well (gre etc) or is that taken care of now you have a pernet batch exit op?
cheers,
jamal
Quoted 12 lines · jamal
Quoted 2 lines · Eric W. Biederman
This series of patches takes my simple 4k network namespace exit test from about 44s to 1.6s, with a minuscule increase in code side.
Nice.
I will test these as soon as they make it into net-next (just easier for me given the setup).
I should confirm that:
The previous patches which are already in net-next from yesterday do improve performance to under 1 minute. Can we call that a several-factor improvement?;->
Quoted 3 lines
Are you planning to give some tender loving to the virtual devices as well (gre etc) or is that taken care of now you have a pernet batch exit op?
The tunnel devices gre, ipip, sit, ip6_tunnel, and the ethernet bridging still need some tlc to give them rtnl_link_ops, and in particular dellink operations so the deletions can be done in one big batch.
The batch operations are mostly a generalization of a bunch of what was already there.
I keep planning on doing something else and then I wind up working on this.
Mostly what I have done turned out to be some very low hanging fruit, and some general cleanups.
Eric
David Miller
maintainerFrom: [email protected] (Eric W. Biederman)
Date: Thu, 03 Dec 2009 04:27:08 -0800
Quoted 7 lines
This series of patches takes my simple 4k network namespace exit test from about 44s to 1.6s, with a minuscule increase in code side.
While batching has given me big improvements there are a few changes in there that improve the performance of cleaning up a single network namespace.
All applied to net-next-2.6, thanks Eric.