net: Hold per-netns RTNL during netdev notifier registration.
Subsystems (2)
cover net: Hold per-netns RTNL during netdev notifier registration.
Patch 1 converts the global netdev notifier to blocking_notifier, which will be called under per-netns RTNL without RTNL, then we need to protect the ongoing netdev_chain users from unregistration.
Patch 2 ~ 4 adds per-netns RTNL for registration of the global and per-netns netdev notifiers.
Kuniyuki Iwashima (4):
net: Convert netdev_chain to blocking_notifier.
net: Hold __rtnl_net_lock() in (un)?register_netdevice_notifier().
net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_net().
net: Hold rtnl_net_lock() in
(un)?register_netdevice_notifier_dev_net().
net/core/dev.c | 48 +++++++++++++++++++++++++++++++----------------- 1 file changed, 31 insertions(+), 17 deletions(-)
Signature · 1 line
2.39.5 (Apple Git-154)
PATCH 1/4 net: Convert netdev_chain to blocking_notifier.
Once RTNL is converted to per-netns, nothing protects netdev_chain.
A netdev notifier might disappear while being used by raw_notifier_call_chain().
Let's convert netdev_chain to blocking_notifier to add its dedicated rwsem.
Given netdev_chain is touched under RTNL by only one user at the same time, adding rwsem is unlikely to cause regression.
--- net/core/dev.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
net/core/dev.c +5 -5
@@ -444,7 +444,7 @@ static void unlist_netdevice(struct net_device *dev)* Our notifier list */ -static RAW_NOTIFIER_HEAD (netdev_chain); +static BLOCKING_NOTIFIER_HEAD (netdev_chain);/* * Device drivers call our routines to queue packets here. We empty the
@@ -1770,7 +1770,7 @@ int register_netdevice_notifier(struct notifier_block *nb)/* Close race with setup_net() and cleanup_net() */ down_write (&pernet_ops_rwsem);rtnl_lock (); - err =raw_notifier_chain_register (&netdev_chain, nb); + err =blocking_notifier_chain_register (&netdev_chain, nb);if (err) goto unlock;if (dev_boot_phase)
@@ -1790,7 +1790,7 @@ int register_netdevice_notifier(struct notifier_block *nb)for_each_net_continue_reverse (net)call_netdevice_unregister_net_notifiers (nb, net); -raw_notifier_chain_unregister (&netdev_chain, nb); +blocking_notifier_chain_unregister (&netdev_chain, nb); goto unlock; }EXPORT_SYMBOL (register_netdevice_notifier);
@@ -1817,7 +1817,7 @@ int unregister_netdevice_notifier(struct notifier_block *nb)/* Close race with setup_net() and cleanup_net() */ down_write (&pernet_ops_rwsem);rtnl_lock (); - err =raw_notifier_chain_unregister (&netdev_chain, nb); + err =blocking_notifier_chain_unregister (&netdev_chain, nb);if (err) goto unlock;
@@ -1993,7 +1993,7 @@ int call_netdevice_notifiers_info(unsigned long val, ret =raw_notifier_call_chain (&net->netdev_chain, val, info);if (ret &NOTIFY_STOP_MASK )return ret; -return raw_notifier_call_chain (&netdev_chain, val, info); +return blocking_notifier_call_chain (&netdev_chain, val, info); }/**
Signature · 1 line
2.39.5 (Apple Git-154)
kernel test robot
build botRe: [PATCH v1 net-next 1/4] net: Convert netdev_chain to blocking_notifier.
kernel test robot noticed "WARNING:possible_recursive_locking_detected" on:
Hello,
kernel test robot noticed "WARNING:possible_recursive_locking_detected" on:
commit: a105e36eac9fc55523ee0157ab2aeaf0ebe949ae ("[PATCH v1 net-next 1/4] net: Convert netdev_chain to blocking_notifier.")
url: https://github.com/intel-lab-lkp/linux/commits/Kuniyuki-Iwashima/net-Convert-netdev_chain-to-blocking_notifier/20250104-144302
base: https://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git 385f186aba3d2f7122b71d6d4c7e236b9d4e8003 patch link: https://lore.kernel.org/all/[email protected]/
patch subject: [PATCH v1 net-next 1/4] net: Convert netdev_chain to blocking_notifier.
in testcase: boot
config: i386-randconfig-011-20250106
compiler: clang-19
test machine: qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 4G
(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]
[ 5.611439][ T1] WARNING: possible recursive locking detected
[ 5.612070][ T1] 6.13.0-rc5-00768-ga105e36eac9f #1 Tainted: G T
[ 5.612838][ T1] --------------------------------------------
[ 5.613448][ T1] swapper/1 is trying to acquire lock:
[ 5.613981][ T1] 8bb1f2c0 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain (kernel/notifier.c:?)
[ 5.614973][ T1]
[ 5.614973][ T1] but task is already holding lock:
[ 5.615681][ T1] 8bb1f2c0 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain (kernel/notifier.c:?)
[ 5.616710][ T1]
[ 5.616710][ T1] other info that might help us debug this:
[ 5.617477][ T1] Possible unsafe locking scenario:
[ 5.617477][ T1]
[ 5.618212][ T1] CPU0
[ 5.618531][ T1] ----
[ 5.618867][ T1] lock((netdev_chain).rwsem);
[ 5.619378][ T1] lock((netdev_chain).rwsem);
[ 5.619874][ T1]
[ 5.619874][ T1] *** DEADLOCK ***
[ 5.619874][ T1]
[ 5.620686][ T1] May be due to missing lock nesting notation
[ 5.620686][ T1]
[ 5.621503][ T1] 2 locks held by swapper/1:
[ 5.621956][ T1] #0: 8bb1fd24 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_lock (net/core/rtnetlink.c:80)
[ 5.622737][ T1] #1: 8bb1f2c0 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain (kernel/notifier.c:?)
[ 5.623745][ T1]
[ 5.623745][ T1] stack backtrace:
[ 5.624307][ T1] CPU: 0 UID: 0 PID: 1 Comm: swapper Tainted: G T 6.13.0-rc5-00768-ga105e36eac9f #1
[ 5.625360][ T1] Tainted: [T]=RANDSTRUCT
[ 5.625789][ T1] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
[ 5.626829][ T1] Call Trace:
[ 5.627162][ T1] dump_stack_lvl (lib/dump_stack.c:122)
[ 5.627595][ T1] ? vprintk (kernel/printk/printk_safe.c:?)
[ 5.628004][ T1] dump_stack (lib/dump_stack.c:129)
[ 5.628427][ T1] print_deadlock_bug (kernel/locking/lockdep.c:3039)
[ 5.628931][ T1] __lock_acquire (kernel/locking/lockdep.c:3089 kernel/locking/lockdep.c:3891 kernel/locking/lockdep.c:5226)
[ 5.629437][ T1] lock_acquire (kernel/locking/lockdep.c:5849)
[ 5.629883][ T1] ? blocking_notifier_call_chain (kernel/notifier.c:?)
[ 5.630484][ T1] down_read (kernel/locking/rwsem.c:1524)
[ 5.630907][ T1] ? blocking_notifier_call_chain (kernel/notifier.c:?)
[ 5.631522][ T1] ? notifier_call_chain (kernel/notifier.c:?)
[ 5.632054][ T1] blocking_notifier_call_chain (kernel/notifier.c:?)
[ 5.632631][ T1] call_netdevice_notifiers_info (net/core/dev.c:1997)
[ 5.633224][ T1] register_netdevice (include/linux/notifier.h:207 net/core/dev.c:10597)
[ 5.633717][ T1] bpq_device_event (drivers/net/hamradio/bpqether.c:500 drivers/net/hamradio/bpqether.c:542)
[ 5.634190][ T1] notifier_call_chain (kernel/notifier.c:?)
[ 5.634710][ T1] blocking_notifier_call_chain (kernel/notifier.c:380)
[ 5.635284][ T1] call_netdevice_notifiers_info (net/core/dev.c:1997)
[ 5.635875][ T1] __dev_notify_flags (net/core/dev.c:8995)
[ 5.636372][ T1] dev_change_flags (net/core/dev.c:?)
[ 5.636833][ T1] ic_open_devs (net/ipv4/ipconfig.c:242)
[ 5.637265][ T1] ? wait_for_devices (net/ipv4/ipconfig.c:?)
[ 5.637736][ T1] ip_auto_config (net/ipv4/ipconfig.c:1515)
[ 5.638192][ T1] do_one_initcall (init/main.c:1266)
[ 5.638656][ T1] ? root_nfs_parse_addr (net/ipv4/ipconfig.c:1477)
[ 5.639169][ T1] ? __lock_acquire (kernel/locking/lockdep.c:4670 kernel/locking/lockdep.c:5180)
[ 5.639684][ T1] ? ktime_get (include/linux/seqlock.h:368 (discriminator 3) include/linux/seqlock.h:388 (discriminator 3) kernel/time/timekeeping.c:815 (discriminator 3))
[ 5.640142][ T1] ? irqentry_exit (kernel/entry/common.c:?)
[ 5.640619][ T1] ? check_preemption_disabled (lib/smp_processor_id.c:16 (discriminator 1))
[ 5.641205][ T1] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67)
[ 5.641755][ T1] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4468 (discriminator 5))
[ 5.642293][ T1] ? trace_hardirqs_on (kernel/trace/trace_preemptirq.c:80)
[ 5.642800][ T1] ? irqentry_exit (kernel/entry/common.c:?)
[ 5.643271][ T1] ? vmware_sched_clock (arch/x86/kernel/apic/apic.c:1049)
[ 5.643776][ T1] ? sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1049 (discriminator 6))
[ 5.644336][ T1] ? handle_exception (init_task.c:?)
[ 5.644843][ T1] ? next_arg (lib/cmdline.c:273)
[ 5.645293][ T1] do_initcall_level (init/main.c:1327 (discriminator 6))
[ 5.645781][ T1] do_initcalls (init/main.c:1341 (discriminator 2))
[ 5.646223][ T1] do_basic_setup (init/main.c:1364)
[ 5.646697][ T1] kernel_init_freeable (init/main.c:1579)
[ 5.647216][ T1] ? rest_init (init/main.c:1458)
[ 5.647671][ T1] ? rest_init (init/main.c:1458)
[ 5.648135][ T1] kernel_init (init/main.c:1468)
[ 5.648604][ T1] ret_from_fork (arch/x86/kernel/process.c:153)
[ 5.649060][ T1] ret_from_fork_asm (??:?)
[ 5.649533][ T1] entry_INT80_32 (init_task.c:?)
[ 7.669903][ T35] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 7.682723][ T1] Sending DHCP requests ., OKThe kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250108/[email protected]
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH 2/4 net: Hold __rtnl_net_lock() in (un)?register_netdevice_notifier().
In reply to Kuniyuki Iwashima(un)?register_netdevice_notifier() hold pernet_ops_rwsem and RTNL, iterate all netns, and trigger the notifier for all netdev.
Let's hold __rtnl_net_lock() before triggering the notifier.
--- net/core/dev.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
net/core/dev.c +10 -2
@@ -1776,7 +1776,9 @@ int register_netdevice_notifier(struct notifier_block *nb)if (dev_boot_phase) goto unlock;for_each_net (net) { +__rtnl_net_lock (net); err =call_netdevice_register_net_notifiers (nb, net); +__rtnl_net_unlock (net);if (err) goto rollback; }
@@ -1787,8 +1789,11 @@ int register_netdevice_notifier(struct notifier_block *nb)return err; rollback: -for_each_net_continue_reverse (net) +for_each_net_continue_reverse (net) { +__rtnl_net_lock (net);call_netdevice_unregister_net_notifiers (nb, net); +__rtnl_net_unlock (net); + }blocking_notifier_chain_unregister (&netdev_chain, nb); goto unlock;
@@ -1821,8 +1826,11 @@ int unregister_netdevice_notifier(struct notifier_block *nb)if (err) goto unlock; -for_each_net (net) +for_each_net (net) { +__rtnl_net_lock (net);call_netdevice_unregister_net_notifiers (nb, net); +__rtnl_net_unlock (net); + } unlock:rtnl_unlock ();
Signature · 1 line
2.39.5 (Apple Git-154)
PATCH 3/4 net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_net().
In reply to Kuniyuki Iwashima(un)?register_netdevice_notifier_net() hold RTNL before triggering the notifier for all netdev in the netns.
Let's convert the RTNL to rtnl_net_lock().
Note that the per-netns netdev notifier is protected by per-netns RTNL, so we do not need to convert it to blocking_notifier.
--- net/core/dev.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
net/core/dev.c +6 -4
@@ -1894,9 +1894,10 @@ int register_netdevice_notifier_net(struct net *net, struct notifier_block *nb) {int err; -rtnl_lock (); +rtnl_net_lock (net); err =__register_netdevice_notifier_net (net, nb, false); -rtnl_unlock (); +rtnl_net_unlock (net); +return err; }EXPORT_SYMBOL (register_netdevice_notifier_net);
@@ -1922,9 +1923,10 @@ int unregister_netdevice_notifier_net(struct net *net, {int err; -rtnl_lock (); +rtnl_net_lock (net); err =__unregister_netdevice_notifier_net (net, nb); -rtnl_unlock (); +rtnl_net_unlock (net); +return err; }EXPORT_SYMBOL (unregister_netdevice_notifier_net);
Signature · 1 line
2.39.5 (Apple Git-154)
PATCH 4/4 net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().
In reply to Kuniyuki Iwashima(un)?register_netdevice_notifier_dev_net() hold RTNL before triggering the notifier for all netdev in the netns.
Let's convert the RTNL to rtnl_net_lock().
Note that move_netdevice_notifiers_dev_net() is assumed to be (but not yet) protected by per-netns RTNL of both src and dst netns; we need to convert wireless and hyperv drivers that call dev_change_net_namespace().
--- net/core/dev.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
net/core/dev.c +10 -6
@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev,struct notifier_block *nb, struct netdev_net_notifier *nn) { + struct net *net = dev_net(dev);int err; -rtnl_lock (); - err =__register_netdevice_notifier_net (dev_net (dev), nb, false); +rtnl_net_lock (net); + err =__register_netdevice_notifier_net (net, nb, false);if (!err) { nn->nb = nb;list_add (&nn->list, &dev->net_notifier_list); } -rtnl_unlock (); +rtnl_net_unlock (net); +return err; }EXPORT_SYMBOL (register_netdevice_notifier_dev_net);
@@ -1960,12 +1962,14 @@ int unregister_netdevice_notifier_dev_net(struct net_device *dev,struct notifier_block *nb, struct netdev_net_notifier *nn) { + struct net *net = dev_net(dev);int err; -rtnl_lock (); +rtnl_net_lock (net);list_del (&nn->list); - err =__unregister_netdevice_notifier_net (dev_net (dev), nb); -rtnl_unlock (); + err =__unregister_netdevice_notifier_net (net, nb); +rtnl_net_unlock (net); +return err; }EXPORT_SYMBOL (unregister_netdevice_notifier_dev_net);
Signature · 1 line
2.39.5 (Apple Git-154)
Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().
we observed in our regression tests the following issue:
BUG: KASAN: slab-use-after-free in notifier_call_chain+0x22c/0x280 kasan_report+0xbd/0xf0 RIP: 0033:0x7f70839018b7 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 __kasan_kmalloc+0x83/0x90 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 kasan_save_free_info+0x37/0x50 __kasan_slab_free+0x33/0x40 page dumped because: kasan: bad access detected BUG: KASAN: slab-use-after-free in notifier_call_chain+0x222/0x280 kasan_report+0xbd/0xf0 RIP: 0033:0x7f70839018b7 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 __kasan_kmalloc+0x83/0x90 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 kasan_save_free_info+0x37/0x50 __kasan_slab_free+0x33/0x40 page dumped because: kasan: bad access detected
and there are many more of that kind.
it happens after applying commit 7fb1073300a2 ("net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net()")
test scenario includes configuration and traffic over two namespaces associated with two different VFs.
Quoted 19 lines · Kuniyuki Iwashima
(un)?register_netdevice_notifier_dev_net() hold RTNL before triggering the notifier for all netdev in the netns.
Let's convert the RTNL to
rtnl_net_lock().Note that
move_netdevice_notifiers_dev_net()is assumed to be (but not yet) protected by per-netns RTNL of both src and dst netns; we need to convert wireless and hyperv drivers that calldev_change_net_namespace().Signed-off-by Kuniyuki Iwashima <[email protected]>--- net/core/dev.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)net/core/dev.c +0
-0
net/core/dev.c int register_netdevice_notifier_dev_net(struct net_device *dev, · line 1946
Quoted 4 lines
net/core/dev.c +1
-0@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev,struct notifier_block *nb, struct netdev_net_notifier *nn) { + struct net *net = dev_net(dev);
it seems to happen since the net pointer is acquired here without a lock.
Note that KASAN issue is not triggered when executing with rtnl_lock() taken before this line. and our kernel .config expands rtnl_net_lock(net) to rtnl_lock() (CONFIG_DEBUG_NET_SMALL_RTNL is not set).
Quoted 34 lines
net/core/dev.c +9
-6int err; -rtnl_lock (); - err =__register_netdevice_notifier_net (dev_net (dev), nb, false); +rtnl_net_lock (net); + err =__register_netdevice_notifier_net (net, nb, false);if (!err) { nn->nb = nb;list_add (&nn->list, &dev->net_notifier_list); } -rtnl_unlock (); +rtnl_net_unlock (net); +return err; }EXPORT_SYMBOL (register_netdevice_notifier_dev_net);@@ -1960,12 +1962,14 @@ int unregister_netdevice_notifier_dev_net(struct net_device *dev,struct notifier_block *nb, struct netdev_net_notifier *nn) { + struct net *net = dev_net(dev);int err; -rtnl_lock (); +rtnl_net_lock (net);list_del (&nn->list); - err =__unregister_netdevice_notifier_net (dev_net (dev), nb); -rtnl_unlock (); + err =__unregister_netdevice_notifier_net (net, nb); +rtnl_net_unlock (net); +return err; }EXPORT_SYMBOL (unregister_netdevice_notifier_dev_net);
Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().
Thanks for the report!
From: Yael Chemla <[email protected]>
Date: Thu, 16 Jan 2025 00:16:27 +0200
Quoted 25 lines
we observed in our regression tests the following issue:
BUG: KASAN: slab-use-after-free in notifier_call_chain+0x22c/0x280 kasan_report+0xbd/0xf0 RIP: 0033:0x7f70839018b7 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 __kasan_kmalloc+0x83/0x90 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 kasan_save_free_info+0x37/0x50 __kasan_slab_free+0x33/0x40 page dumped because: kasan: bad access detected BUG: KASAN: slab-use-after-free in notifier_call_chain+0x222/0x280 kasan_report+0xbd/0xf0 RIP: 0033:0x7f70839018b7 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 __kasan_kmalloc+0x83/0x90 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 kasan_save_free_info+0x37/0x50 __kasan_slab_free+0x33/0x40 page dumped because: kasan: bad access detected
Quoted 1 line
and there are many more of that kind.
Do you have any other stack traces with more callers info ? Also can you decode the trace with ./scripts/decode_stacktrace.sh ?
Quoted 6 lines
it happens after applying commit
7fb1073300a2("net: Holdrtnl_net_lock()in (un)?register_netdevice_notifier_dev_net()")test scenario includes configuration and traffic over two namespaces associated with two different VFs.
Could you elaborate more about the test scenario, especially how each device/netns is dismantled after the test case ?
I guess the VF is moved to init_net ?
Quoted 17 lines · Kuniyuki Iwashima
Quoted 14 lines
(un)?register_netdevice_notifier_dev_net() hold RTNL before triggering the notifier for all netdev in the netns.
Let's convert the RTNL to
rtnl_net_lock().Note that
move_netdevice_notifiers_dev_net()is assumed to be (but not yet) protected by per-netns RTNL of both src and dst netns; we need to convert wireless and hyperv drivers that calldev_change_net_namespace().Signed-off-by Kuniyuki Iwashima <[email protected]>--- net/core/dev.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
Quoted 14 lines
net/core/dev.c int register_netdevice_notifier_dev_net(struct net_device *dev, · line 1946
Quoted 9 lines
net/core/dev.c +1
-0@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev,struct notifier_block *nb, struct netdev_net_notifier *nn) { + struct net *net = dev_net(dev);it seems to happen since the net pointer is acquired here without a lock. Note that KASAN issue is not triggered when executing with
rtnl_lock()taken before this line. and our kernel .config expandsrtnl_net_lock(net)tortnl_lock()(CONFIG_DEBUG_NET_SMALL_RTNLis not set).
It sounds like the device was being moved to another netns while unregister_netdevice_notifier_dev_net() was called.
Could you check if dev_net() is changed before/after rtnl_lock() in
* register_netdevice_notifier_dev_net()
* unregister_netdevice_notifier_dev_net()
?
Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().
Quoted 29 lines · Kuniyuki Iwashima
Thanks for the report!
From: Yael Chemla <[email protected]>
Date: Thu, 16 Jan 2025 00:16:27 +0200Quoted 25 lines
we observed in our regression tests the following issue:
BUG: KASAN: slab-use-after-free in notifier_call_chain+0x22c/0x280 kasan_report+0xbd/0xf0 RIP: 0033:0x7f70839018b7 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 __kasan_kmalloc+0x83/0x90 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 kasan_save_free_info+0x37/0x50 __kasan_slab_free+0x33/0x40 page dumped because: kasan: bad access detected BUG: KASAN: slab-use-after-free in notifier_call_chain+0x222/0x280 kasan_report+0xbd/0xf0 RIP: 0033:0x7f70839018b7 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 __kasan_kmalloc+0x83/0x90 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 kasan_save_free_info+0x37/0x50 __kasan_slab_free+0x33/0x40 page dumped because: kasan: bad access detected
Quoted 5 lines
Quoted 1 line
and there are many more of that kind.
Do you have any other stack traces with more callers info ? Also can you decode the trace with
./scripts/decode_stacktrace.sh?
BUG: KASAN: slab-use-after-free in notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2)) Read of size 8 at addr ffff88810cefb4c8 by task test-bridge-lag/21127 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014 Call Trace: <TASK> dump_stack_lvl (/usr/work/linux/lib/dump_stack.c:123) print_report (/usr/work/linux/mm/kasan/report.c:379 /usr/work/linux/mm/kasan/report.c:489) ? __virt_addr_valid (/usr/work/linux/./arch/x86/include/asm/preempt.h:84 (discriminator 13) /usr/work/linux/./include/linux/rcupdate.h:964 (discriminator 13) /usr/work/linux/./include/linux/mmzone.h:2058 (discriminator 13) /usr/work/linux/arch/x86/mm/physaddr.c:65 (discriminator 13)) kasan_report (/usr/work/linux/mm/kasan/report.c:604) ? notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2)) ? notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2)) notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2)) call_netdevice_notifiers_info (/usr/work/linux/net/core/dev.c:2011) unregister_netdevice_many_notify (/usr/work/linux/net/core/dev.c:11551) ? mark_held_locks (/usr/work/linux/kernel/locking/lockdep.c:4321 (discriminator 1)) ? __mutex_lock (/usr/work/linux/kernel/locking/mutex.c:689 (discriminator 2) /usr/work/linux/kernel/locking/mutex.c:735 (discriminator 2)) ? lockdep_hardirqs_on_prepare (/usr/work/linux/kernel/locking/lockdep.c:4347 /usr/work/linux/kernel/locking/lockdep.c:4406) ? dev_ingress_queue_create (/usr/work/linux/net/core/dev.c:11492) ? __mutex_lock (/usr/work/linux/kernel/locking/mutex.c:689 (discriminator 2) /usr/work/linux/kernel/locking/mutex.c:735 (discriminator 2)) ? __mutex_lock (/usr/work/linux/./arch/x86/include/asm/preempt.h:84 (discriminator 13) /usr/work/linux/kernel/locking/mutex.c:715 (discriminator 13) /usr/work/linux/kernel/locking/mutex.c:735 (discriminator 13)) ? unregister_netdev (/usr/work/linux/./include/linux/netdevice.h:3236 /usr/work/linux/net/core/dev.c:11633)
? mutex_lock_io_nested (/usr/work/linux/kernel/locking/mutex.c:734)
? __mutex_unlock_slowpath (/usr/work/linux/./arch/x86/include/asm/atomic64_64.h:101 /usr/work/linux/./include/linux/atomic/atomic-arch-fallback.h:4329 /usr/work/linux/./include/linux/atomic/atomic-long.h:1506 /usr/work/linux/./include/linux/atomic/atomic-instrumented.h:4481 /usr/work/linux/kernel/locking/mutex.c:913) unregister_netdevice_queue (/usr/work/linux/net/core/dev.c:11487)
? unregister_netdevice_many (/usr/work/linux/net/core/dev.c:11476)unregister_netdev (/usr/work/linux/net/core/dev.c:11635)mlx5e_remove (/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/en_main.c:6552 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/en_main.c:6579) mlx5_coreauxiliary_bus_remove (/usr/work/linux/drivers/base/auxiliary.c:230)
device_release_driver_internal (/usr/work/linux/drivers/base/dd.c:1275 /usr/work/linux/drivers/base/dd.c:1296) ? kobject_put (/usr/work/linux/./arch/x86/include/asm/atomic.h:93 (discriminator 4) /usr/work/linux/./include/linux/atomic/atomic-arch-fallback.h:949 (discriminator 4) /usr/work/linux/./include/linux/atomic/atomic-instrumented.h:401 (discriminator 4) /usr/work/linux/./include/linux/refcount.h:264 (discriminator 4) /usr/work/linux/./include/linux/refcount.h:307 (discriminator 4) /usr/work/linux/./include/linux/refcount.h:325 (discriminator 4) /usr/work/linux/./include/linux/kref.h:64 (discriminator 4) /usr/work/linux/lib/kobject.c:737 (discriminator 4)) bus_remove_device (/usr/work/linux/./include/linux/kobject.h:193 /usr/work/linux/drivers/base/base.h:73 /usr/work/linux/drivers/base/bus.c:583) device_del (/usr/work/linux/drivers/base/power/power.h:142 /usr/work/linux/drivers/base/core.c:3855)
? mlx5_core_is_eth_enabled (/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/devlink.h:65) mlx5_core
? __device_link_del (/usr/work/linux/drivers/base/core.c:3809)
? is_ib_enabled (/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/devlink.h:58) mlx5_coremlx5_rescan_drivers_locked (/usr/work/linux/./include/linux/auxiliary_bus.h:241 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/dev.c:333 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/dev.c:535 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/dev.c:549) mlx5_core mlx5_unregister_device (/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/dev.c:468) mlx5_core
mlx5_uninit_one (/usr/work/linux/./include/linux/instrumented.h:68 /usr/work/linux/./include/asm-generic/bitops/instrumented-non-atomic.h:141 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/main.c:1563) mlx5_core remove_one (/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/main.c:965 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/main.c:2019) mlx5_core pci_device_remove (/usr/work/linux/./include/linux/pm_runtime.h:129 /usr/work/linux/drivers/pci/pci-driver.c:475) device_release_driver_internal (/usr/work/linux/drivers/base/dd.c:1275 /usr/work/linux/drivers/base/dd.c:1296) unbind_store (/usr/work/linux/drivers/base/bus.c:245) kernfs_fop_write_iter (/usr/work/linux/fs/kernfs/file.c:338) vfs_write (/usr/work/linux/fs/read_write.c:587 (discriminator 1) /usr/work/linux/fs/read_write.c:679 (discriminator 1)) ? do_user_addr_fault (/usr/work/linux/./include/linux/rcupdate.h:337 /usr/work/linux/./include/linux/rcupdate.h:849 /usr/work/linux/./include/linux/mm.h:740 /usr/work/linux/arch/x86/mm/fault.c:1340)
? kernel_write (/usr/work/linux/fs/read_write.c:660)
? lock_downgrade (/usr/work/linux/kernel/locking/lockdep.c:5857) ksys_write (/usr/work/linux/fs/read_write.c:732)
? __x64_sys_read (/usr/work/linux/fs/read_write.c:721)
? do_user_addr_fault (/usr/work/linux/./arch/x86/include/asm/preempt.h:84 (discriminator 13) /usr/work/linux/./include/linux/rcupdate.h:98 (discriminator 13) /usr/work/linux/./include/linux/rcupdate.h:882 (discriminator 13) /usr/work/linux/./include/linux/mm.h:742 (discriminator 13) /usr/work/linux/arch/x86/mm/fault.c:1340 (discriminator 13)) do_syscall_64 (/usr/work/linux/arch/x86/entry/common.c:52 (discriminator 1) /usr/work/linux/arch/x86/entry/common.c:83 (discriminator 1)) entry_SYSCALL_64_after_hwframe (/usr/work/linux/arch/x86/entry/entry_64.S:130) RIP: 0033:0x7f6a4d5018b7
Quoted 10 lines
Quoted 6 lines
it happens after applying commit
7fb1073300a2("net: Holdrtnl_net_lock()in (un)?register_netdevice_notifier_dev_net()")test scenario includes configuration and traffic over two namespaces associated with two different VFs.
Could you elaborate more about the test scenario, especially how each device/netns is dismantled after the test case ?
we set up a network configuration which includes two VFs isolated using two namespaces (there’s also bridge in this topology), we pass some traffic between VFs. At the end of test (cleanup) we delete network namespaces, wait for 0.5 sec and unbind VFs of NIC.
note that when I extended the timeout after deleting the namespaces the issue doesn’t reproduce.
Quoted 2 lines
I guess the VF is moved to
init_net?
this should be the behavior in case deletion of the namespaces happens before the unbind of VFs.
Quoted 35 lines
Quoted 17 lines · Kuniyuki Iwashima
Quoted 14 lines
(un)?register_netdevice_notifier_dev_net() hold RTNL before triggering the notifier for all netdev in the netns.
Let's convert the RTNL to
rtnl_net_lock().Note that
move_netdevice_notifiers_dev_net()is assumed to be (but not yet) protected by per-netns RTNL of both src and dst netns; we need to convert wireless and hyperv drivers that calldev_change_net_namespace().Signed-off-by Kuniyuki Iwashima <[email protected]>--- net/core/dev.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)Quoted 14 lines
net/core/dev.c int register_netdevice_notifier_dev_net(struct net_device *dev, · line 1946
Quoted 9 lines
net/core/dev.c +1
-0@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev,struct notifier_block *nb, struct netdev_net_notifier *nn) { + struct net *net = dev_net(dev);it seems to happen since the net pointer is acquired here without a lock. Note that KASAN issue is not triggered when executing with
rtnl_lock()taken before this line. and our kernel .config expandsrtnl_net_lock(net)tortnl_lock()(CONFIG_DEBUG_NET_SMALL_RTNLis not set).It sounds like the device was being moved to another netns while
unregister_netdevice_notifier_dev_net()was called.
Quoted 6 lines
Could you check if
dev_net()is changed before/afterrtnl_lock()in*
register_netdevice_notifier_dev_net()
*unregister_netdevice_notifier_dev_net()?
When checking dev_net before and after taking the lock the issue won’t reproduce.
note that when issue reproduce we arrive to unregister_netdevice_notifier_dev_net with an invalid net pointer (verified it with prints of its value, and it's not the same consistent value as is throughout rest of the test). we suspect the issue related to the async ns deletion.
Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().
Hi Kuniyuki,
any update on this?
do you need further info?
Thanks,
Yael
Quoted 160 lines · Yael Chemla
Quoted 29 lines · Kuniyuki Iwashima
Thanks for the report!
From: Yael Chemla <[email protected]>
Date: Thu, 16 Jan 2025 00:16:27 +0200Quoted 25 lines
we observed in our regression tests the following issue:
BUG: KASAN: slab-use-after-free in notifier_call_chain+0x22c/0x280 kasan_report+0xbd/0xf0 RIP: 0033:0x7f70839018b7 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 __kasan_kmalloc+0x83/0x90 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 kasan_save_free_info+0x37/0x50 __kasan_slab_free+0x33/0x40 page dumped because: kasan: bad access detected BUG: KASAN: slab-use-after-free in notifier_call_chain+0x222/0x280 kasan_report+0xbd/0xf0 RIP: 0033:0x7f70839018b7 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 __kasan_kmalloc+0x83/0x90 kasan_save_stack+0x1c/0x40 kasan_save_track+0x10/0x30 kasan_save_free_info+0x37/0x50 __kasan_slab_free+0x33/0x40 page dumped because: kasan: bad access detectedQuoted 5 lines
Quoted 1 line
and there are many more of that kind.
Do you have any other stack traces with more callers info ? Also can you decode the trace with
./scripts/decode_stacktrace.sh?BUG: KASAN: slab-use-after-free in notifier_call_chain (/usr/work/linux/ kernel/notifier.c:75 (discriminator 2)) Read of size 8 at addr ffff88810cefb4c8 by task test-bridge-lag/21127 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0- gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014 Call Trace: <TASK> dump_stack_lvl (/usr/work/linux/lib/dump_stack.c:123) print_report (/usr/work/linux/mm/kasan/report.c:379 /usr/work/linux/mm/ kasan/report.c:489) ? __virt_addr_valid (/usr/work/linux/./arch/x86/include/asm/preempt.h:84 (discriminator 13) /usr/work/linux/./include/linux/rcupdate.h:964 (discriminator 13) /usr/work/linux/./include/linux/mmzone.h:2058 (discriminator 13) /usr/work/linux/arch/x86/mm/physaddr.c:65 (discriminator 13)) kasan_report (/usr/work/linux/mm/kasan/report.c:604) ? notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2)) ? notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2)) notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2)) call_netdevice_notifiers_info (/usr/work/linux/net/core/dev.c:2011) unregister_netdevice_many_notify (/usr/work/linux/net/core/dev.c:11551) ? mark_held_locks (/usr/work/linux/kernel/locking/lockdep.c:4321 (discriminator 1)) ? __mutex_lock (/usr/work/linux/kernel/locking/mutex.c:689 (discriminator 2) /usr/work/linux/kernel/locking/mutex.c:735 (discriminator 2)) ? lockdep_hardirqs_on_prepare (/usr/work/linux/kernel/locking/ lockdep.c:4347 /usr/work/linux/kernel/locking/lockdep.c:4406) ? dev_ingress_queue_create (/usr/work/linux/net/core/dev.c:11492) ? __mutex_lock (/usr/work/linux/kernel/locking/mutex.c:689 (discriminator 2) /usr/work/linux/kernel/locking/mutex.c:735 (discriminator 2)) ? __mutex_lock (/usr/work/linux/./arch/x86/include/asm/preempt.h:84 (discriminator 13) /usr/work/linux/kernel/locking/mutex.c:715 (discriminator 13) /usr/work/linux/kernel/locking/mutex.c:735 (discriminator 13)) ? unregister_netdev (/usr/work/linux/./include/linux/netdevice.h:3236 / usr/work/linux/net/core/dev.c:11633)?
mutex_lock_io_nested(/usr/work/linux/kernel/locking/mutex.c:734)
?__mutex_unlock_slowpath(/usr/work/linux/./arch/x86/include/asm/atomic64_64.h:101/usr/work/linux/./include/linux/atomic/atomic-arch-fallback.h:4329/usr/work/linux/./include/linux/atomic/atomic-long.h:1506/usr/work/linux/./include/linux/atomic/atomic-instrumented.h:4481/usr/work/linux/kernel/locking/mutex.c:913)unregister_netdevice_queue(/usr/work/linux/net/core/dev.c:11487)
?unregister_netdevice_many(/usr/work/linux/net/core/dev.c:11476)unregister_netdev(/usr/work/linux/net/core/dev.c:11635)mlx5e_remove(/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/en_main.c:6552/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/en_main.c:6579)mlx5_coreauxiliary_bus_remove(/usr/work/linux/drivers/base/auxiliary.c:230)device_release_driver_internal (/usr/work/linux/drivers/base/dd.c:1275 / usr/work/linux/drivers/base/dd.c:1296) ? kobject_put (/usr/work/linux/./arch/x86/include/asm/atomic.h:93 (discriminator 4) /usr/work/linux/./include/linux/atomic/atomic-arch- fallback.h:949 (discriminator 4) /usr/work/linux/./include/linux/atomic/ atomic-instrumented.h:401 (discriminator 4) /usr/work/linux/./include/ linux/refcount.h:264 (discriminator 4) /usr/work/linux/./include/linux/ refcount.h:307 (discriminator 4) /usr/work/linux/./include/linux/ refcount.h:325 (discriminator 4) /usr/work/linux/./include/linux/ kref.h:64 (discriminator 4) /usr/work/linux/lib/kobject.c:737 (discriminator 4)) bus_remove_device (/usr/work/linux/./include/linux/kobject.h:193 /usr/ work/linux/drivers/base/base.h:73 /usr/work/linux/drivers/base/bus.c:583) device_del (/usr/work/linux/drivers/base/power/power.h:142 /usr/work/ linux/drivers/base/core.c:3855)?
mlx5_core_is_eth_enabled(/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/devlink.h:65)mlx5_core
?__device_link_del(/usr/work/linux/drivers/base/core.c:3809)
?is_ib_enabled(/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/devlink.h:58)mlx5_coremlx5_rescan_drivers_locked(/usr/work/linux/./include/linux/auxiliary_bus.h:241/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/dev.c:333/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/dev.c:535/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/dev.c:549)mlx5_coremlx5_unregister_device(/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/dev.c:468)mlx5_coremlx5_uninit_one(/usr/work/linux/./include/linux/instrumented.h:68/usr/work/linux/./include/asm-generic/bitops/instrumented-non-atomic.h:141/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/main.c:1563)mlx5_coreremove_one(/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/main.c:965/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/main.c:2019)mlx5_corepci_device_remove (/usr/work/linux/./include/linux/pm_runtime.h:129 / usr/work/linux/drivers/pci/pci-driver.c:475) device_release_driver_internal (/usr/work/linux/drivers/base/dd.c:1275 / usr/work/linux/drivers/base/dd.c:1296) unbind_store (/usr/work/linux/drivers/base/bus.c:245) kernfs_fop_write_iter (/usr/work/linux/fs/kernfs/file.c:338) vfs_write (/usr/work/linux/fs/read_write.c:587 (discriminator 1) /usr/ work/linux/fs/read_write.c:679 (discriminator 1)) ? do_user_addr_fault (/usr/work/linux/./include/linux/rcupdate.h:337 / usr/work/linux/./include/linux/rcupdate.h:849 /usr/work/linux/./include/ linux/mm.h:740 /usr/work/linux/arch/x86/mm/fault.c:1340)?
kernel_write(/usr/work/linux/fs/read_write.c:660)
?lock_downgrade(/usr/work/linux/kernel/locking/lockdep.c:5857)ksys_write(/usr/work/linux/fs/read_write.c:732)
?__x64_sys_read(/usr/work/linux/fs/read_write.c:721)
?do_user_addr_fault(/usr/work/linux/./arch/x86/include/asm/preempt.h:84 (discriminator 13)/usr/work/linux/./include/linux/rcupdate.h:98 (discriminator 13)/usr/work/linux/./include/linux/rcupdate.h:882 (discriminator 13)/usr/work/linux/./include/linux/mm.h:742 (discriminator 13)/usr/work/linux/arch/x86/mm/fault.c:1340(discriminator 13))do_syscall_64(/usr/work/linux/arch/x86/entry/common.c:52(discriminator 1)/usr/work/linux/arch/x86/entry/common.c:83(discriminator 1))entry_SYSCALL_64_after_hwframe(/usr/work/linux/arch/x86/entry/entry_64.S:130)RIP: 0033:0x7f6a4d5018b7Quoted 10 lines
Quoted 6 lines
it happens after applying commit
7fb1073300a2("net: Holdrtnl_net_lock()in (un)?register_netdevice_notifier_dev_net()")test scenario includes configuration and traffic over two namespaces associated with two different VFs.
Could you elaborate more about the test scenario, especially how each device/netns is dismantled after the test case ?
we set up a network configuration which includes two VFs isolated using two namespaces (there’s also bridge in this topology), we pass some traffic between VFs. At the end of test (cleanup) we delete network namespaces, wait for 0.5 sec and unbind VFs of NIC.
note that when I extended the timeout after deleting the namespaces the issue doesn’t reproduce.
Quoted 2 lines
I guess the VF is moved to
init_net?this should be the behavior in case deletion of the namespaces happens before the unbind of VFs.
Quoted 36 lines
Quoted 17 lines · Kuniyuki Iwashima
Quoted 14 lines
(un)?register_netdevice_notifier_dev_net() hold RTNL before triggering the notifier for all netdev in the netns.
Let's convert the RTNL to
rtnl_net_lock().Note that
move_netdevice_notifiers_dev_net()is assumed to be (but not yet) protected by per-netns RTNL of both src and dst netns; we need to convert wireless and hyperv drivers that calldev_change_net_namespace().Signed-off-by Kuniyuki Iwashima <[email protected]>---
net/core/dev.c| 16 ++++++++++------1 file changed, 10 insertions(+), 6 deletions(-)Quoted 15 lines
net/core/dev.c int · line 1947
Quoted 10 lines
net/core/dev.c +1
-0@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev, struct notifier_block *nb, struct netdev_net_notifier *nn) { + struct net *net = dev_net(dev);it seems to happen since the net pointer is acquired here without a lock. Note that KASAN issue is not triggered when executing with
rtnl_lock()taken before this line. and our kernel .config expandsrtnl_net_lock(net)tortnl_lock()(CONFIG_DEBUG_NET_SMALL_RTNLis not set).It sounds like the device was being moved to another netns while
unregister_netdevice_notifier_dev_net()was called.Quoted 6 lines
Could you check if
dev_net()is changed before/afterrtnl_lock()in*
register_netdevice_notifier_dev_net()
*unregister_netdevice_notifier_dev_net()?
When checking
dev_netbefore and after taking the lock the issue won’t reproduce.
note that when issue reproduce we arrive tounregister_netdevice_notifier_dev_netwith an invalid net pointer (verified it with prints of its value, and it's not the same consistent value as is throughout rest of the test). we suspect the issue related to the async ns deletion.
Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().
From: Yael Chemla <[email protected]>
Date: Mon, 27 Jan 2025 19:26:34 +0200
Quoted 4 lines
Hi Kuniyuki,
any update on this?
do you need further info?
Sorry, I was busy as on-call last week.
I'll investigate it this week.
Thanks!
Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().
In reply to Yael ChemlaFrom: Yael Chemla <[email protected]>
Date: Mon, 20 Jan 2025 20:55:07 +0200
Quoted 23 lines
Quoted 23 lines
Quoted 6 lines
Quoted 6 lines
net/core/dev.c +0
-0@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev, struct notifier_block *nb,Quoted 8 lines
net/core/dev.c int register_netdevice_notifier_dev_net(struct net_device *dev, · line 1946
Quoted 3 lines
net/core/dev.c +1
-0struct netdev_net_notifier *nn) { +struct net *net =dev_net (dev);it seems to happen since the net pointer is acquired here without a lock. Note that KASAN issue is not triggered when executing with
rtnl_lock()taken before this line. and our kernel .config expandsrtnl_net_lock(net)tortnl_lock()(CONFIG_DEBUG_NET_SMALL_RTNLis not set).It sounds like the device was being moved to another netns while
unregister_netdevice_notifier_dev_net()was called.Could you check if
dev_net()is changed before/afterrtnl_lock()in*
register_netdevice_notifier_dev_net()
*unregister_netdevice_notifier_dev_net()
Quoted 8 lines
Quoted 1 line
?
When checking
dev_netbefore and after taking the lock the issue won’t reproduce.
note that when issue reproduce we arrive tounregister_netdevice_notifier_dev_netwith an invalid net pointer (verified it with prints of its value, and it's not the same consistent value as is throughout rest of the test).
Does an invalid net pointer means a dead netns pointer ? dev_net() and dev_net_set() use rcu_dereference() and rcu_assign_pointer(), so I guess it should not be an invalid address at least.
Quoted 1 line
we suspect the issue related to the async ns deletion.
I think async netns change would trigger the issue too.
Could you try this patch ?
---8<---
net/core/dev.c +37 -8
@@ -2070,20 +2070,50 @@ static void __move_netdevice_notifier_net(struct net *src_net,__register_netdevice_notifier_net (dst_net, nb, true); } +static void rtnl_net_dev_lock (struct net_device *dev) +{ +struct net *net; + +again: +/* netns might be being dismantled. */ + net =maybe_get_net (dev_net (dev)); +if (!net) { +cond_resched (); + goto again; + } + +rtnl_net_lock (net); + +/* dev might be moved to another netns. */ +if (!net_eq (net,dev_net (dev))) { +rtnl_net_unlock (net); +put_net (net); +cond_resched (); + goto again; + } +} + +static void rtnl_net_dev_unlock (struct net_device *dev) +{ +struct net *net =dev_net (dev); + +rtnl_net_unlock (net); +put_net (net); +} +int register_netdevice_notifier_dev_net (struct net_device *dev,struct notifier_block *nb,struct netdev_net_notifier *nn) { -struct net *net =dev_net (dev);int err; -rtnl_net_lock (net); - err =__register_netdevice_notifier_net (net, nb, false); +rtnl_net_dev_lock (dev); + err =__register_netdevice_notifier_net (dev_net (dev), nb, false);if (!err) { nn->nb = nb;list_add (&nn->list, &dev->net_notifier_list); } -rtnl_net_unlock (net); +rtnl_net_dev_unlock (dev);return err; }
@@ -2093,13 +2123,12 @@ int unregister_netdevice_notifier_dev_net(struct net_device *dev,struct notifier_block *nb, struct netdev_net_notifier *nn) { - struct net *net = dev_net(dev); int err; -rtnl_net_lock (net); +rtnl_net_dev_lock (dev);list_del (&nn->list); - err =__unregister_netdevice_notifier_net (net, nb); -rtnl_net_unlock (net); + err =__unregister_netdevice_notifier_net (dev_net (dev), nb); +rtnl_net_dev_unlock (dev);return err; }
---8<---
Yael Chemla
Tested-byRe: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().
Quoted 34 lines · Kuniyuki Iwashima
From: Yael Chemla <[email protected]>
Date: Mon, 20 Jan 2025 20:55:07 +0200Quoted 23 lines
Quoted 23 lines
Quoted 6 lines
Quoted 6 lines
net/core/dev.c +0
-0@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev, struct notifier_block *nb,Quoted 8 lines
net/core/dev.c static void __move_netdevice_notifier_net(struct net *src_net,
Quoted 3 lines
net/core/dev.c +1
-0struct netdev_net_notifier *nn) { +struct net *net =dev_net (dev);it seems to happen since the net pointer is acquired here without a lock. Note that KASAN issue is not triggered when executing with
rtnl_lock()taken before this line. and our kernel .config expandsrtnl_net_lock(net)tortnl_lock()(CONFIG_DEBUG_NET_SMALL_RTNLis not set).It sounds like the device was being moved to another netns while
unregister_netdevice_notifier_dev_net()was called.Could you check if
dev_net()is changed before/afterrtnl_lock()in*
register_netdevice_notifier_dev_net()
*unregister_netdevice_notifier_dev_net()Quoted 8 lines
Quoted 1 line
?
When checking
dev_netbefore and after taking the lock the issue won’t reproduce.
note that when issue reproduce we arrive tounregister_netdevice_notifier_dev_netwith an invalid net pointer (verified it with prints of its value, and it's not the same consistent value as is throughout rest of the test).
Quoted 4 lines
Does an invalid net pointer means a dead netns pointer ?
dev_net()anddev_net_set()usercu_dereference()andrcu_assign_pointer(), so I guess it should not be an invalid address at least.
I logged several values at the entrance of unregister_netdevice_notifier_dev_net when issue reproduced:
1) fields of net->ns (struct ns_common):
count: the namespace refcount is 0 (i.e. net->ns.count, used refcount_read to read it).
inum: the value doesn't appear to be garbage but differ from its constant value throughout the test.
2) net pointer (struct net): value differ from its constant value observed during the rest of the test.
hope this helps and please let me know if more info is needed.
Quoted 7 lines
Quoted 2 lines
we suspect the issue related to the async ns deletion.
I think async netns change would trigger the issue too.
Could you try this patch ?
I tested your patch and issue won't reproduce with it (CONFIG_DEBUG_NET_SMALL_RTNL is not set in my config).
Thanks a lot!
Quoted 78 lines
---8<---
net/core/dev.c +37
-9@@ -2070,20 +2070,50 @@ static void __move_netdevice_notifier_net(struct net *src_net,__register_netdevice_notifier_net (dst_net, nb, true); } +static void rtnl_net_dev_lock (struct net_device *dev) +{ +struct net *net; + +again: +/* netns might be being dismantled. */ + net =maybe_get_net (dev_net (dev)); +if (!net) { +cond_resched (); + goto again; + } + +rtnl_net_lock (net); + +/* dev might be moved to another netns. */ +if (!net_eq (net,dev_net (dev))) { +rtnl_net_unlock (net); +put_net (net); +cond_resched (); + goto again; + } +} + +static void rtnl_net_dev_unlock (struct net_device *dev) +{ +struct net *net =dev_net (dev); + +rtnl_net_unlock (net); +put_net (net); +} +int register_netdevice_notifier_dev_net (struct net_device *dev,struct notifier_block *nb,struct netdev_net_notifier *nn) { -struct net *net =dev_net (dev);int err; -rtnl_net_lock (net); - err =__register_netdevice_notifier_net (net, nb, false); +rtnl_net_dev_lock (dev); + err =__register_netdevice_notifier_net (dev_net (dev), nb, false);if (!err) { nn->nb = nb;list_add (&nn->list, &dev->net_notifier_list); } -rtnl_net_unlock (net); +rtnl_net_dev_unlock (dev);return err; }@@ -2093,13 +2123,12 @@ int unregister_netdevice_notifier_dev_net(struct net_device *dev,struct notifier_block *nb, struct netdev_net_notifier *nn) { - struct net *net = dev_net(dev); int err; -rtnl_net_lock (net); +rtnl_net_dev_lock (dev);list_del (&nn->list); - err =__unregister_netdevice_notifier_net (net, nb); -rtnl_net_unlock (net); + err =__unregister_netdevice_notifier_net (dev_net (dev), nb); +rtnl_net_dev_unlock (dev);return err; } ---8<---
Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().
From: Yael Chemla <[email protected]>
Date: Wed, 29 Jan 2025 18:21:23 +0200
Quoted 39 lines · Kuniyuki Iwashima
Quoted 34 lines
From: Yael Chemla <[email protected]>
Date: Mon, 20 Jan 2025 20:55:07 +0200Quoted 23 lines
Quoted 23 lines
Quoted 6 lines
Quoted 6 lines
net/core/dev.c +0
-0@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev, struct notifier_block *nb,Quoted 8 lines
net/core/dev.c static void __move_netdevice_notifier_net(struct net *src_net,
Quoted 3 lines
net/core/dev.c +1
-0struct netdev_net_notifier *nn) { +struct net *net =dev_net (dev);it seems to happen since the net pointer is acquired here without a lock. Note that KASAN issue is not triggered when executing with
rtnl_lock()taken before this line. and our kernel .config expandsrtnl_net_lock(net)tortnl_lock()(CONFIG_DEBUG_NET_SMALL_RTNLis not set).It sounds like the device was being moved to another netns while
unregister_netdevice_notifier_dev_net()was called.Could you check if
dev_net()is changed before/afterrtnl_lock()in*
register_netdevice_notifier_dev_net()
*unregister_netdevice_notifier_dev_net()Quoted 8 lines
Quoted 1 line
?
When checking
dev_netbefore and after taking the lock the issue won’t reproduce.
note that when issue reproduce we arrive tounregister_netdevice_notifier_dev_netwith an invalid net pointer (verified it with prints of its value, and it's not the same consistent value as is throughout rest of the test).Quoted 4 lines
Does an invalid net pointer means a dead netns pointer ?
dev_net()anddev_net_set()usercu_dereference()andrcu_assign_pointer(), so I guess it should not be an invalid address at least.
Quoted 5 lines
I logged several values at the entrance of
unregister_netdevice_notifier_dev_netwhen issue reproduced:
1) fields ofnet->ns(structns_common):
count: the namespace refcount is 0 (i.e.net->ns.count, usedrefcount_readto read it).
This indicates here we race with cleanup_net().
Quoted 14 lines
inum: the value doesn't appear to be garbage but differ from its constant value throughout the test.
2) net pointer (struct net): value differ from its constant value observed during the rest of the test.
hope this helps and please let me know if more info is needed.
Quoted 5 lines
Quoted 2 lines
we suspect the issue related to the async ns deletion.
I think async netns change would trigger the issue too.
Quoted 7 lines
Quoted 2 lines
Could you try this patch ?
I tested your patch and issue won't reproduce with it (
CONFIG_DEBUG_NET_SMALL_RTNLis not set in my config).Tested-by Yael Chemla <[email protected]>
Thanks for testing !
Will post the fix officially.
Jakub Kicinski
maintainerQuoted 6 lines · On Sat, 4 Jan 2025 15:37:31 +0900 Kuniyuki Iwashima wrote:
Patch 1 converts the global netdev notifier to
blocking_notifier, which will be called under per-netns RTNL without RTNL, then we need to protect the ongoingnetdev_chainusers from unregistration.Patch 2 ~ 4 adds per-netns RTNL for registration of the global and per-netns netdev notifiers.
Lockdep is not happy:
[ 249.261403][ T11] ============================================
[ 249.261592][ T11] WARNING: possible recursive locking detected
[ 249.261769][ T11] 6.13.0-rc5-virtme #1 Not tainted
[ 249.261920][ T11] --------------------------------------------
[ 249.262094][ T11] kworker/u16:0/11 is trying to acquire lock:
[ 249.262293][ T11] ffffffff8a7f6a70 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90
[ 249.262591][ T11]
[ 249.262591][ T11] but task is already holding lock:
[ 249.262810][ T11] ffffffff8a7f6a70 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90
[ 249.263100][ T11]
[ 249.263100][ T11] other info that might help us debug this:
[ 249.263310][ T11] Possible unsafe locking scenario:
[ 249.263310][ T11]
[ 249.263522][ T11] CPU0
[ 249.263624][ T11] ----
[ 249.263728][ T11] lock((netdev_chain).rwsem);
[ 249.263875][ T11] lock((netdev_chain).rwsem);
[ 249.264020][ T11]
[ 249.264020][ T11] *** DEADLOCK ***
[ 249.264020][ T11]
[ 249.264223][ T11] May be due to missing lock nesting notation
[ 249.264223][ T11]
[ 249.264440][ T11] 5 locks held by kworker/u16:0/11:
[ 249.264582][ T11] #0: ffff8880010b5948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0x7ec/0x16d0
[ 249.264867][ T11] #1: ffffc900000b7da0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xe0b/0x16d0
[ 249.265118][ T11] #2: ffffffff8a7ec4d0 (pernet_ops_rwsem){++++}-{4:4}, at: cleanup_net+0xbc/0xba0
[ 249.265381][ T11] #3: ffffffff8a807e88 (rtnl_mutex){+.+.}-{4:4}, at: default_device_exit_batch+0x81/0x2e0
[ 249.265668][ T11] #4: ffffffff8a7f6a70 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90
[ 249.265954][ T11]
[ 249.265954][ T11] stack backtrace:
[ 249.266126][ T11] CPU: 2 UID: 0 PID: 11 Comm: kworker/u16:0 Not tainted 6.13.0-rc5-virtme #1
[ 249.266389][ T11] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 249.266572][ T11] Workqueue: netns cleanup_net
[ 249.266722][ T11] Call Trace:
[ 249.266826][ T11] <TASK>
[ 249.266907][ T11] dump_stack_lvl+0x82/0xd0
[ 249.267056][ T11] print_deadlock_bug+0x40a/0x650
[ 249.267206][ T11] validate_chain+0x5bf/0xae0
[ 249.267352][ T11] ? __pfx_validate_chain+0x10/0x10
[ 249.267503][ T11] ? hlock_class+0x4e/0x130
[ 249.267642][ T11] ? mark_lock+0x38/0x3e0
[ 249.267751][ T11] __lock_acquire+0xb9a/0x1680
[ 249.267897][ T11] ? spin_bug+0x191/0x1d0
[ 249.268007][ T11] ? debug_object_assert_init+0x2a9/0x370
[ 249.268164][ T11] lock_acquire.part.0+0xeb/0x330
[ 249.268313][ T11] ? blocking_notifier_call_chain+0x50/0x90
[ 249.268497][ T11] ? __pfx_lock_acquire.part.0+0x10/0x10
[ 249.268651][ T11] ? trace_lock_acquire+0x14c/0x1f0
[ 249.268803][ T11] ? lock_acquire+0x32/0xc0
[ 249.268944][ T11] ? blocking_notifier_call_chain+0x50/0x90
[ 249.269132][ T11] down_read+0x9f/0x340
[ 249.269247][ T11] ? blocking_notifier_call_chain+0x50/0x90
[ 249.269436][ T11] ? __pfx_down_read+0x10/0x10
[ 249.269586][ T11] blocking_notifier_call_chain+0x50/0x90
[ 249.269739][ T11] __dev_close_many+0xdf/0x2d0
[ 249.269881][ T11] ? __pfx___dev_close_many+0x10/0x10
[ 249.270031][ T11] dev_close_many+0x202/0x650Signature · 1 line
pw-bot: cr
From: Jakub Kicinski <[email protected]>
Date: Sat, 4 Jan 2025 07:37:40 -0800
Quoted 19 lines · On Sat, 4 Jan 2025 15:37:31 +0900 Kuniyuki Iwashima wrote:
Quoted 6 lines
Patch 1 converts the global netdev notifier to
blocking_notifier, which will be called under per-netns RTNL without RTNL, then we need to protect the ongoingnetdev_chainusers from unregistration.Patch 2 ~ 4 adds per-netns RTNL for registration of the global and per-netns netdev notifiers.
Lockdep is not happy:
[ 249.261403][ T11] ============================================ [ 249.261592][ T11] WARNING: possible recursive locking detected [ 249.261769][ T11] 6.13.0-rc5-virtme #1 Not tainted [ 249.261920][ T11] -------------------------------------------- [ 249.262094][ T11] kworker/u16:0/11 is trying to acquire lock: [ 249.262293][ T11] ffffffff8a7f6a70 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90 [ 249.262591][ T11] [ 249.262591][ T11] but task is already holding lock: [ 249.262810][ T11] ffffffff8a7f6a70 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90
Quoted 8 lines
[ 249.263100][ T11] [ 249.263100][ T11] other info that might help us debug this: [ 249.263310][ T11] Possible unsafe locking scenario: [ 249.263310][ T11] [ 249.263522][ T11] CPU0 [ 249.263624][ T11] ---- [ 249.263728][ T11] lock((netdev_chain).rwsem); [ 249.263875][ T11] lock((netdev_chain).rwsem);
Ah, lockdep annotaion was needed for the nested notifier calls.
But this will not be a meaningful annotation and needs to be changed once rtnl_setlink/dellink supports per-netns RTNL.
I'll drop patch 1 and just leave a comment around RTNL in register_netdevice_notifier() in patch 2.
Another option would be clone each netdev notifier during registration and unshare(2)/clone(2) and force notifiers to be namespacified ?
---8<---
net/core/dev.c +13 -0
@@ -446,6 +446,17 @@ static void unlist_netdevice(struct net_device *dev)static BLOCKING_NOTIFIER_HEAD (netdev_chain); +#ifdef CONFIG_PROVE_LOCKING +static int netdev_chain_lock_cmp_fn (const struct lockdep_map *a, +const struct lockdep_map *b) +{ +if (rtnl_is_locked ()) +return -1 ; + +return 1 ; +} +#endif +/* * Device drivers call our routines to queue packets here. We empty the * queue in the local softnet handler.
@@ -12229,6 +12240,8 @@ static int __init net_dev_init(void)net_dev_struct_check (); +lock_set_cmp_fn (&netdev_chain.rwsem, netdev_chain_lock_cmp_fn,NULL ); +if (dev_proc_init ()) goto out;
---8<---