rtnetlink: Per-netns RTNL.
Subsystems (2)
v2 cover rtnetlink: Per-netns RTNL.
rtnl_lock() is a "Big Kernel Lock" in the networking slow path and serialised all rtnetlink requests until 4.13.
Since RTNL_FLAG_DOIT_UNLOCKED and RTNL_FLAG_DUMP_UNLOCKED have been introduced in 4.14 and 6.9, respectively, rtnetlink message handlers are ready to be converted to RTNL-less/free.
15 out of 44 dumpit()s have been converted to RCU so far, and the progress is pretty good. We can now dump various major network resources without RTNL.
12 out of 87 doit()s have been converted, but most of the converted doit()s are also on the reader side of RTNL; their message types are RTM_GET*.
So, most of RTM_(NEW|DEL|SET)* operations are still serialised by RTNL.
For example, one of our services creates 2K netns and a small number of network interfaces in each netns that require too many writer-side rtnetlink requests, and setting up a single host takes 10+ minutes.
RTNL is still a huge pain for network configuration paths, and we need more granular locking, given converting all doit()s would be unfeasible.
Actually, most RTNL users do not need to freeze multiple netns, and such users can be protected by per-netns RTNL mutex. The exceptions would be RTM_NEWLINK, RTM_DELLINK, and RTM_SETLINK. (See [0])
This series is the first step of the per-netns RTNL conversion that gradually replaces rtnl_lock() with rtnl_net_lock(net) under CONFIG_DEBUG_NET_SMALL_RTNL.
[0]: https://lpc.events/event/18/contributions/1959/
Changes:
v2:
* Add revert of 464eb03c4a7c
* Fix Kconfig dependency for arch with no lockdep support
v1: https://lore.kernel.org/netdev/[email protected]/
Kuniyuki Iwashima (4):
Revert "rtnetlink: add guard for RTNL"
rtnetlink: Add per-netns RTNL.
rtnetlink: Add assertion helpers for per-netns RTNL.
rtnetlink: Add ASSERT_RTNL_NET() placeholder for netdev notifier.
include/linux/rtnetlink.h | 39 ++++++++++- include/net/net_namespace.h | 4 ++ net/Kconfig.debug | 15 +++++ net/core/Makefile | 1 + net/core/net_namespace.c | 6 ++ net/core/rtnetlink.c | 70 +++++++++++++++++++ net/core/rtnl_net_debug.c | 131 ++++++++++++++++++++++++++++++++++++ 7 files changed, 264 insertions(+), 2 deletions(-) create mode 100644 net/core/rtnl_net_debug.c
Signature · 1 line
2.30.2
PATCH v2 1/4 Revert "rtnetlink: add guard for RTNL"
This reverts commit 464eb03c4a7cfb32cb3324249193cf6bb5b35152.
Once we have a per-netns RTNL, we won't use guard(rtnl).
Also, there's no users for now.
$ grep -rnI "guard(rtnl" || true $
--- include/linux/rtnetlink.h | 3 --- 1 file changed, 3 deletions(-)
include/linux/rtnetlink.h +0 -3
@@ -7,7 +7,6 @@#include <linux/netdevice.h> #include <linux/wait.h> #include <linux/refcount.h> -#include <linux/cleanup.h> #include <uapi/linux/rtnetlink.h> extern int rtnetlink_send (struct sk_buff *skb,struct net *net,u32 pid,u32 group,int echo);
@@ -47,8 +46,6 @@ extern int rtnl_is_locked(void);extern int rtnl_lock_killable (void );extern bool refcount_dec_and_rtnl_lock (refcount_t *r); -DEFINE_LOCK_GUARD_0(rtnl, rtnl_lock(), rtnl_unlock()) -extern wait_queue_head_t netdev_unregistering_wq;extern atomic_t dev_unreg_count;extern struct rw_semaphore pernet_ops_rwsem;
Signature · 1 line
2.30.2
Re: [PATCH v2 net-next 1/4] Revert "rtnetlink: add guard for RTNL"
Quoted 5 lines · Kuniyuki Iwashima
This reverts commit
464eb03c4a7cfb32cb3324249193cf6bb5b35152.Once we have a per-netns RTNL, we won't use guard(rtnl).
Quoted 8 lines
Also, there's no users for now.
$ grep -rnI "guard(rtnl" || true $
PATCH v2 2/4 rtnetlink: Add per-netns RTNL.
In reply to Kuniyuki IwashimaThe goal is to break RTNL down into per-netns mutex.
This patch adds per-netns mutex and its helper functions, rtnl_net_lock() and rtnl_net_unlock().
rtnl_net_lock() acquires the global RTNL and per-netns RTNL mutex, and rtnl_net_unlock() releases them.
We will replace 800+ rtnl_lock() with rtnl_net_lock() and finally removes rtnl_lock() in rtnl_net_lock().
When we need to nest per-netns RTNL mutex, we will use __rtnl_net_lock(), and its locking order is defined by rtnl_net_lock_cmp_fn() as follows:
1. init_net is first
2. netns address ascending order
Note that the conversion will be done under CONFIG_DEBUG_NET_SMALL_RTNL with LOCKDEP so that we can carefully add the extra mutex without slowing down RTNL operations during conversion.
--- include/linux/rtnetlink.h | 13 +++++++++ include/net/net_namespace.h | 4 +++ net/Kconfig.debug | 15 ++++++++++ net/core/net_namespace.c | 6 ++++ net/core/rtnetlink.c | 58 +++++++++++++++++++++++++++++++++++++ 5 files changed, 96 insertions(+)
include/linux/rtnetlink.h +13 -0
@@ -46,6 +46,19 @@ extern int rtnl_is_locked(void);extern int rtnl_lock_killable (void );extern bool refcount_dec_and_rtnl_lock (refcount_t *r); +#ifdef CONFIG_DEBUG_NET_SMALL_RTNL +void __rtnl_net_lock (struct net *net); +void __rtnl_net_unlock (struct net *net); +void rtnl_net_lock (struct net *net); +void rtnl_net_unlock (struct net *net); +int rtnl_net_lock_cmp_fn (const struct lockdep_map *a,const struct lockdep_map *b); +#else +#define __rtnl_net_lock (net) +#define __rtnl_net_unlock (net) +#define rtnl_net_lock (net) rtnl_lock() +#define rtnl_net_unlock (net) rtnl_unlock() +#endif +extern wait_queue_head_t netdev_unregistering_wq;extern atomic_t dev_unreg_count;extern struct rw_semaphore pernet_ops_rwsem;
include/net/net_namespace.h +4 -0
@@ -188,6 +188,10 @@ struct net {#if IS_ENABLED (CONFIG_SMC )struct netns_smc smc;#endif +#ifdef CONFIG_DEBUG_NET_SMALL_RTNL +/* Move to a better place when the config guard is removed. */ +struct mutex rtnl_mutex; +#endif } __randomize_layout;#include <linux/seq_file_net.h>
net/Kconfig.debug +15 -0
@@ -24,3 +24,18 @@ config DEBUG_NET help Enable extra sanity checks in networking. This is mostly used by fuzzers, but is safe to select. + +config DEBUG_NET_SMALL_RTNL + bool "Add extra per-netns mutex inside RTNL" + depends on DEBUG_KERNEL && NET && LOCK_DEBUGGING_SUPPORT + select PROVE_LOCKING + default n + help + rtnl_lock() is being replaced with rtnl_net_lock() that + acquires the global RTNL and a small per-netns RTNL mutex. + + During the conversion, rtnl_net_lock() just adds an extra + mutex in every RTNL scope and slows down the operations. + + Once the conversion completes, rtnl_lock() will be removed + and rtnetlink will gain per-netns scalability.
net/core/net_namespace.c +6 -0
@@ -334,6 +334,12 @@ static __net_init void preinit_net(struct net *net, struct user_namespace *user_idr_init (&net->netns_ids);spin_lock_init (&net->nsid_lock);mutex_init (&net->ipv4.ra_mutex); + +#ifdef CONFIG_DEBUG_NET_SMALL_RTNL +mutex_init (&net->rtnl_mutex); +lock_set_cmp_fn (&net->rtnl_mutex, rtnl_net_lock_cmp_fn,NULL ); +#endif +preinit_net_sysctl (net); }
net/core/rtnetlink.c +58 -0
64 lines not on this page · raw message
Signature · 1 line
2.30.2
Re: [PATCH v2 net-next 2/4] rtnetlink: Add per-netns RTNL.
Quoted 15 lines · Kuniyuki Iwashima
The goal is to break RTNL down into per-netns mutex.
This patch adds per-netns mutex and its helper functions,
rtnl_net_lock()andrtnl_net_unlock().
rtnl_net_lock()acquires the global RTNL and per-netns RTNL mutex, andrtnl_net_unlock()releases them.We will replace 800+
rtnl_lock()withrtnl_net_lock()and finally removesrtnl_lock()inrtnl_net_lock().When we need to nest per-netns RTNL mutex, we will use
__rtnl_net_lock(), and its locking order is defined byrtnl_net_lock_cmp_fn()as follows:
Quoted 8 lines
1.
init_netis first
2. netns address ascending orderNote that the conversion will be done under
CONFIG_DEBUG_NET_SMALL_RTNLwith LOCKDEP so that we can carefully add the extra mutex without slowing down RTNL operations during conversion.Signed-off-by Kuniyuki Iwashima <[email protected]>
Jakub Kicinski
maintainerRe: [PATCH v2 net-next 2/4] rtnetlink: Add per-netns RTNL.
In reply to Kuniyuki Iwashimainclude/linux/rtnetlink.h extern int rtnl_is_locked(void); · line 59
Quoted 11 lines · On Wed, 2 Oct 2024 08:12:38 -0700 Kuniyuki Iwashima wrote:
include/linux/rtnetlink.h +11
-0+#ifdef CONFIG_DEBUG_NET_SMALL_RTNL +void __rtnl_net_lock (struct net *net); +void __rtnl_net_unlock (struct net *net); +void rtnl_net_lock (struct net *net); +void rtnl_net_unlock (struct net *net); +int rtnl_net_lock_cmp_fn (const struct lockdep_map *a,const struct lockdep_map *b); +#else +#define __rtnl_net_lock (net) +#define __rtnl_net_unlock (net) +#define rtnl_net_lock (net) rtnl_lock() +#define rtnl_net_unlock (net) rtnl_unlock()
Let's make sure net is always evaluated?
At the very least make sure the preprocessor doesn't eat it completely otherwise we may end up with config-dependent "unused variable" warnings down the line.
Quoted 1 line
+#endif
Signature · 1 line
pw-bot: cr
Re: [PATCH v2 net-next 2/4] rtnetlink: Add per-netns RTNL.
From: Jakub Kicinski <[email protected]>
Date: Fri, 4 Oct 2024 13:21:45 -0700
Quoted 9 lines · On Wed, 2 Oct 2024 08:12:38 -0700 Kuniyuki Iwashima wrote:
Quoted 8 lines
include/linux/rtnetlink.h +8
-0+#ifdef CONFIG_DEBUG_NET_SMALL_RTNL +void __rtnl_net_lock (struct net *net); +void __rtnl_net_unlock (struct net *net); +void rtnl_net_lock (struct net *net); +void rtnl_net_unlock (struct net *net); +int rtnl_net_lock_cmp_fn (const struct lockdep_map *a,const struct lockdep_map *b); +#else +#define __rtnl_net_lock (net)
Quoted 8 lines
include/linux/rtnetlink.h extern int rtnl_is_locked(void); · line 59
Quoted 3 lines
include/linux/rtnetlink.h +3
-0+#define __rtnl_net_unlock (net) +#define rtnl_net_lock (net) rtnl_lock() +#define rtnl_net_unlock (net) rtnl_unlock()Let's make sure net is always evaluated?
At the very least make sure the preprocessor doesn't eat it completely otherwise we may end up with config-dependent "unused variable" warnings down the line.
Sure, what comes to mind is void casting, which I guess is old-school style ? Do you have any other idea or is this acceptable ?
#define __rtnl_net_lock(net) (void)(net)
#define __rtnl_net_unlock(net) (void)(net)
#define rtnl_net_lock(net) \
do { \
(void)(net); \
rtnl_lock(); \
} while (0)
#define rtnl_net_unlock(net) \
do { \
(void)(net); \
rtnl_unlock(); \
} while (0)Re: [PATCH v2 net-next 2/4] rtnetlink: Add per-netns RTNL.
From: Kuniyuki Iwashima <[email protected]>
Date: Fri, 4 Oct 2024 13:45:26 -0700
Quoted 25 lines
From: Jakub Kicinski <[email protected]>
Date: Fri, 4 Oct 2024 13:21:45 -0700Quoted 9 lines · On Wed, 2 Oct 2024 08:12:38 -0700 Kuniyuki Iwashima wrote:
Quoted 8 lines
include/linux/rtnetlink.h +8
-0+#ifdef CONFIG_DEBUG_NET_SMALL_RTNL +void __rtnl_net_lock (struct net *net); +void __rtnl_net_unlock (struct net *net); +void rtnl_net_lock (struct net *net); +void rtnl_net_unlock (struct net *net); +int rtnl_net_lock_cmp_fn (const struct lockdep_map *a,const struct lockdep_map *b); +#else +#define __rtnl_net_lock (net)Quoted 8 lines
include/linux/rtnetlink.h extern int rtnl_is_locked(void); · line 59
Quoted 3 lines
include/linux/rtnetlink.h +3
-0+#define __rtnl_net_unlock (net) +#define rtnl_net_lock (net) rtnl_lock() +#define rtnl_net_unlock (net) rtnl_unlock()Let's make sure net is always evaluated?
At the very least make sure the preprocessor doesn't eat it completely otherwise we may end up with config-dependent "unused variable" warnings down the line.Sure, what comes to mind is void casting, which I guess is old-school style ? Do you have any other idea or is this acceptable ?
#define
__rtnl_net_lock(net)(void)(net)
#define__rtnl_net_unlock(net)(void)(net)
Quoted 10 lines
#define rtnl_net_lock(net) \
do { \
(void)(net); \
rtnl_lock(); \
} while (0)
#define rtnl_net_unlock(net) \
do { \
(void)(net); \
rtnl_unlock(); \
} while (0)or simply define these as static inline functions and probably this is more preferable ?
static inline void __rtnl_net_lock(struct net *net) {} static inline void __rtnl_net_unlock(struct net *net) {} static inline void rtnl_net_lock(struct net *net)
{
rtnl_lock();
}
static inline void rtnl_net_unlock(struct net *net)
{
rtnl_unlock();
}Re: [PATCH v2 net-next 2/4] rtnetlink: Add per-netns RTNL.
Quoted 45 lines · Kuniyuki Iwashima
From: Kuniyuki Iwashima <[email protected]>
Date: Fri, 4 Oct 2024 13:45:26 -0700Quoted 25 lines
From: Jakub Kicinski <[email protected]>
Date: Fri, 4 Oct 2024 13:21:45 -0700Quoted 9 lines · On Wed, 2 Oct 2024 08:12:38 -0700 Kuniyuki Iwashima wrote:
Quoted 8 lines
include/linux/rtnetlink.h +8
-0+#ifdef CONFIG_DEBUG_NET_SMALL_RTNL +void __rtnl_net_lock (struct net *net); +void __rtnl_net_unlock (struct net *net); +void rtnl_net_lock (struct net *net); +void rtnl_net_unlock (struct net *net); +int rtnl_net_lock_cmp_fn (const struct lockdep_map *a,const struct lockdep_map *b); +#else +#define __rtnl_net_lock (net)Quoted 8 lines
include/linux/rtnetlink.h extern int rtnl_is_locked(void); · line 59
Quoted 3 lines
include/linux/rtnetlink.h +3
-0+#define __rtnl_net_unlock (net) +#define rtnl_net_lock (net) rtnl_lock() +#define rtnl_net_unlock (net) rtnl_unlock()Let's make sure net is always evaluated?
At the very least make sure the preprocessor doesn't eat it completely otherwise we may end up with config-dependent "unused variable" warnings down the line.Sure, what comes to mind is void casting, which I guess is old-school style ? Do you have any other idea or is this acceptable ?
#define
__rtnl_net_lock(net)(void)(net)
#define__rtnl_net_unlock(net)(void)(net)Quoted 10 lines
#define rtnl_net_lock(net) \ do { \ (void)(net); \ rtnl_lock(); \ } while (0) #define rtnl_net_unlock(net) \ do { \ (void)(net); \ rtnl_unlock(); \ } while (0)or simply define these as static inline functions and probably this is more preferable ?
static inline void
__rtnl_net_lock(struct net *net){} static inline void__rtnl_net_unlock(struct net *net){} static inline voidrtnl_net_lock(struct net *net)
Quoted 7 lines
{
rtnl_lock();
}
static inline void rtnl_net_unlock(struct net *net)
{
rtnl_unlock();
}static inline functions seem better to me.
Re: [PATCH v2 net-next 2/4] rtnetlink: Add per-netns RTNL.
From: Eric Dumazet <[email protected]>
Date: Fri, 4 Oct 2024 22:59:32 +0200
Quoted 54 lines · Kuniyuki Iwashima
Quoted 45 lines
From: Kuniyuki Iwashima <[email protected]>
Date: Fri, 4 Oct 2024 13:45:26 -0700Quoted 25 lines
From: Jakub Kicinski <[email protected]>
Date: Fri, 4 Oct 2024 13:21:45 -0700Quoted 9 lines · On Wed, 2 Oct 2024 08:12:38 -0700 Kuniyuki Iwashima wrote:
Quoted 8 lines
include/linux/rtnetlink.h +8
-0+#ifdef CONFIG_DEBUG_NET_SMALL_RTNL +void __rtnl_net_lock (struct net *net); +void __rtnl_net_unlock (struct net *net); +void rtnl_net_lock (struct net *net); +void rtnl_net_unlock (struct net *net); +int rtnl_net_lock_cmp_fn (const struct lockdep_map *a,const struct lockdep_map *b); +#else +#define __rtnl_net_lock (net)Quoted 8 lines
include/linux/rtnetlink.h extern int rtnl_is_locked(void); · line 59
Quoted 3 lines
include/linux/rtnetlink.h +3
-0+#define __rtnl_net_unlock (net) +#define rtnl_net_lock (net) rtnl_lock() +#define rtnl_net_unlock (net) rtnl_unlock()Let's make sure net is always evaluated?
At the very least make sure the preprocessor doesn't eat it completely otherwise we may end up with config-dependent "unused variable" warnings down the line.Sure, what comes to mind is void casting, which I guess is old-school style ? Do you have any other idea or is this acceptable ?
#define
__rtnl_net_lock(net)(void)(net)
#define__rtnl_net_unlock(net)(void)(net)Quoted 10 lines
#define rtnl_net_lock(net) \ do { \ (void)(net); \ rtnl_lock(); \ } while (0) #define rtnl_net_unlock(net) \ do { \ (void)(net); \ rtnl_unlock(); \ } while (0)or simply define these as static inline functions and probably this is more preferable ?
static inline void
__rtnl_net_lock(struct net *net){} static inline void__rtnl_net_unlock(struct net *net){} static inline voidrtnl_net_lock(struct net *net)Quoted 7 lines
{ rtnl_lock(); } static inline void rtnl_net_unlock(struct net *net) { rtnl_unlock(); }
Quoted 1 line
static inline functions seem better to me.
Will use them.
Thanks !
PATCH v2 3/4 rtnetlink: Add assertion helpers for per-netns RTNL.
In reply to Kuniyuki IwashimaOnce an RTNL scope is converted with rtnl_net_lock(), we will replace RTNL helper functions inside the scope with the following per-netns alternatives:
ASSERT_RTNL() -> ASSERT_RTNL_NET(net)
rcu_dereference_rtnl(p) -> rcu_dereference_rtnl_net(net, p)
Note that the per-netns helpers are equivalent to the conventional helpers unless CONFIG_DEBUG_NET_SMALL_RTNL is enabled.
--- include/linux/rtnetlink.h | 25 +++++++++++++++++++++++++ net/core/rtnetlink.c | 12 ++++++++++++ 2 files changed, 37 insertions(+)
include/linux/rtnetlink.h +25 -0
@@ -52,11 +52,36 @@ void __rtnl_net_unlock(struct net *net);void rtnl_net_lock (struct net *net);void rtnl_net_unlock (struct net *net);int rtnl_net_lock_cmp_fn (const struct lockdep_map *a,const struct lockdep_map *b); + +bool rtnl_net_is_locked (struct net *net); + +#define ASSERT_RTNL_NET (net) \ + WARN_ONCE(!rtnl_net_is_locked(net), \ + "RTNL_NET: assertion failed at %s (%d)\n", \ + __FILE__, __LINE__) + +bool lockdep_rtnl_net_is_held (struct net *net); + +#define rcu_dereference_rtnl_net (net, p) \ + rcu_dereference_check(p, lockdep_rtnl_net_is_held(net)) +#define rtnl_net_dereference (net, p) \ + rcu_dereference_protected(p, lockdep_rtnl_net_is_held(net)) +#define rcu_replace_pointer_rtnl_net (net, rp, p) \ + rcu_replace_pointer(rp, p, lockdep_rtnl_net_is_held(net))#else #define __rtnl_net_lock (net)#define __rtnl_net_unlock (net)#define rtnl_net_lock (net) rtnl_lock()#define rtnl_net_unlock (net) rtnl_unlock() + +#define ASSERT_RTNL_NET (net) ASSERT_RTNL() + +#define rcu_dereference_rtnl_net (net, p) \ + rcu_dereference_rtnl(p) +#define rtnl_net_dereference (net, p) \ + rtnl_dereference(p) +#define rcu_replace_pointer_rtnl_net (net, rp, p) \ + rcu_replace_pointer_rtnl(rp, p)#endif extern wait_queue_head_t netdev_unregistering_wq;
net/core/rtnetlink.c +12 -0
@@ -235,6 +235,18 @@ int rtnl_net_lock_cmp_fn(const struct lockdep_map *a, const struct lockdep_map * return rtnl_net_cmp_locks(net_a, net_b); } + +bool rtnl_net_is_locked (struct net *net) +{ +return rtnl_is_locked () &&mutex_is_locked (&net->rtnl_mutex); +} +EXPORT_SYMBOL (rtnl_net_is_locked); + +bool lockdep_rtnl_net_is_held (struct net *net) +{ +return lockdep_rtnl_is_held () &&lockdep_is_held (&net->rtnl_mutex); +} +EXPORT_SYMBOL (lockdep_rtnl_net_is_held);#endif static struct rtnl_link __rcu *__rcu *rtnl_msg_handlers[RTNL_FAMILY_MAX +1 ];
Signature · 1 line
2.30.2
Re: [PATCH v2 net-next 3/4] rtnetlink: Add assertion helpers for per-netns RTNL.
Quoted 5 lines · Kuniyuki Iwashima
Once an RTNL scope is converted with
rtnl_net_lock(), we will replace RTNL helper functions inside the scope with the following per-netns alternatives:
Quoted 7 lines
ASSERT_RTNL()->ASSERT_RTNL_NET(net)
rcu_dereference_rtnl(p)->rcu_dereference_rtnl_net(net, p)Note that the per-netns helpers are equivalent to the conventional helpers unless
CONFIG_DEBUG_NET_SMALL_RTNLis enabled.Signed-off-by Kuniyuki Iwashima <[email protected]>
PATCH v2 4/4 rtnetlink: Add ASSERT_RTNL_NET() placeholder for netdev notifier.
In reply to Kuniyuki IwashimaThe global and per-netns netdev notifier depend on RTNL, and its dependency is not so clear due to nested calls.
Let's add a placeholder to place ASSERT_RTNL_NET() for each event.
--- net/core/Makefile | 1 + net/core/rtnl_net_debug.c | 131 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 132 insertions(+) create mode 100644 net/core/rtnl_net_debug.c
net/core/Makefile +1 -0
@@ -45,3 +45,4 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_sk_storage.o obj-$(CONFIG_OF) += of_net.o obj-$(CONFIG_NET_TEST) += net_test.o obj-$(CONFIG_NET_DEVMEM) += devmem.o +obj-$(CONFIG_DEBUG_NET_SMALL_RTNL) += rtnl_net_debug.o
net/core/rtnl_net_debug.c new file +131 -0
131 lines not on this page · raw message
Signature · 1 line
2.30.2
Re: [PATCH v2 net-next 4/4] rtnetlink: Add ASSERT_RTNL_NET() placeholder for netdev notifier.
Quoted 7 lines · Kuniyuki Iwashima
The global and per-netns netdev notifier depend on RTNL, and its dependency is not so clear due to nested calls.
Let's add a placeholder to place
ASSERT_RTNL_NET()for each event.Signed-off-by Kuniyuki Iwashima <[email protected]>