Skip to contentSkip to conversation

net: Hold per-netns RTNL during netdev notifier registration.

PATCH Superseded Kuniyuki Iwashima 4 participants 16 messages Jan 4, 2025 – Jan 30, 2025

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover net: Hold per-netns RTNL during netdev notifier registration.

Patch 1 converts the global netdev notifier to blocking_notifier, which will be called under per-netns RTNL without RTNL, then we need to protect the ongoing netdev_chain users from unregistration.

Patch 2 ~ 4 adds per-netns RTNL for registration of the global and per-netns netdev notifiers.

Kuniyuki Iwashima (4):
net: Convert netdev_chain to blocking_notifier.
net: Hold __rtnl_net_lock() in (un)?register_netdevice_notifier().
net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_net().
net: Hold rtnl_net_lock() in
(un)?register_netdevice_notifier_dev_net().

 net/core/dev.c | 48 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 31 insertions(+), 17 deletions(-)
Signature · 1 line
2.39.5 (Apple Git-154)
#1 lore raw

PATCH 1/4 net: Convert netdev_chain to blocking_notifier.

Once RTNL is converted to per-netns, nothing protects netdev_chain.

A netdev notifier might disappear while being used by raw_notifier_call_chain().

Let's convert netdev_chain to blocking_notifier to add its dedicated rwsem.

Given netdev_chain is touched under RTNL by only one user at the same time, adding rwsem is unlikely to cause regression.

---
 net/core/dev.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
net/core/dev.c +5 -5
index e7223972b9aa..404f5bda821b 100644
@@ -444,7 +444,7 @@ static void unlist_netdevice(struct net_device *dev)
  *	Our notifier list
  */
 
-static RAW_NOTIFIER_HEAD(netdev_chain);
+static BLOCKING_NOTIFIER_HEAD(netdev_chain);
 
 /*
  *	Device drivers call our routines to queue packets here. We empty the
@@ -1770,7 +1770,7 @@ int register_netdevice_notifier(struct notifier_block *nb)
 	/* Close race with setup_net() and cleanup_net() */
 	down_write(&pernet_ops_rwsem);
 	rtnl_lock();
-	err = raw_notifier_chain_register(&netdev_chain, nb);
+	err = blocking_notifier_chain_register(&netdev_chain, nb);
 	if (err)
 		goto unlock;
 	if (dev_boot_phase)
@@ -1790,7 +1790,7 @@ int register_netdevice_notifier(struct notifier_block *nb)
 	for_each_net_continue_reverse(net)
 		call_netdevice_unregister_net_notifiers(nb, net);
 
-	raw_notifier_chain_unregister(&netdev_chain, nb);
+	blocking_notifier_chain_unregister(&netdev_chain, nb);
 	goto unlock;
 }
 EXPORT_SYMBOL(register_netdevice_notifier);
@@ -1817,7 +1817,7 @@ int unregister_netdevice_notifier(struct notifier_block *nb)
 	/* Close race with setup_net() and cleanup_net() */
 	down_write(&pernet_ops_rwsem);
 	rtnl_lock();
-	err = raw_notifier_chain_unregister(&netdev_chain, nb);
+	err = blocking_notifier_chain_unregister(&netdev_chain, nb);
 	if (err)
 		goto unlock;
 
@@ -1993,7 +1993,7 @@ int call_netdevice_notifiers_info(unsigned long val,
 	ret = raw_notifier_call_chain(&net->netdev_chain, val, info);
 	if (ret & NOTIFY_STOP_MASK)
 		return ret;
-	return raw_notifier_call_chain(&netdev_chain, val, info);
+	return blocking_notifier_call_chain(&netdev_chain, val, info);
 }
 
 /**
Signature · 1 line
2.39.5 (Apple Git-154)
#2 lore raw

Re: [PATCH v1 net-next 1/4] net: Convert netdev_chain to blocking_notifier.

kernel test robot noticed "WARNING:possible_recursive_locking_detected" on:

Hello,

kernel test robot noticed "WARNING:possible_recursive_locking_detected" on:

commit: a105e36eac9fc55523ee0157ab2aeaf0ebe949ae ("[PATCH v1 net-next 1/4] net: Convert netdev_chain to blocking_notifier.")
url: https://github.com/intel-lab-lkp/linux/commits/Kuniyuki-Iwashima/net-Convert-netdev_chain-to-blocking_notifier/20250104-144302
base: https://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git 385f186aba3d2f7122b71d6d4c7e236b9d4e8003 patch link: https://lore.kernel.org/all/[email protected]/
patch subject: [PATCH v1 net-next 1/4] net: Convert netdev_chain to blocking_notifier.

in testcase: boot

config: i386-randconfig-011-20250106
compiler: clang-19
test machine: qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 4G

(please refer to attached dmesg/kmsg for entire log/backtrace)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]

[    5.611439][    T1] WARNING: possible recursive locking detected
[    5.612070][    T1] 6.13.0-rc5-00768-ga105e36eac9f #1 Tainted: G                T
[    5.612838][    T1] --------------------------------------------
[    5.613448][    T1] swapper/1 is trying to acquire lock:
[ 5.613981][ T1] 8bb1f2c0 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain (kernel/notifier.c:?)
[    5.614973][    T1]
[    5.614973][    T1] but task is already holding lock:
[ 5.615681][ T1] 8bb1f2c0 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain (kernel/notifier.c:?)
[    5.616710][    T1]
[    5.616710][    T1] other info that might help us debug this:
[    5.617477][    T1]  Possible unsafe locking scenario:
[    5.617477][    T1]
[    5.618212][    T1]        CPU0
[    5.618531][    T1]        ----
[    5.618867][    T1]   lock((netdev_chain).rwsem);
[    5.619378][    T1]   lock((netdev_chain).rwsem);
[    5.619874][    T1]
[    5.619874][    T1]  *** DEADLOCK ***
[    5.619874][    T1]
[    5.620686][    T1]  May be due to missing lock nesting notation
[    5.620686][    T1]
[    5.621503][    T1] 2 locks held by swapper/1:
[ 5.621956][ T1] #0: 8bb1fd24 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_lock (net/core/rtnetlink.c:80)
[ 5.622737][ T1] #1: 8bb1f2c0 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain (kernel/notifier.c:?)
[    5.623745][    T1]
[    5.623745][    T1] stack backtrace:
[    5.624307][    T1] CPU: 0 UID: 0 PID: 1 Comm: swapper Tainted: G                T  6.13.0-rc5-00768-ga105e36eac9f #1
[    5.625360][    T1] Tainted: [T]=RANDSTRUCT
[    5.625789][    T1] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
[    5.626829][    T1] Call Trace:
[ 5.627162][ T1] dump_stack_lvl (lib/dump_stack.c:122)
[ 5.627595][ T1] ? vprintk (kernel/printk/printk_safe.c:?)
[ 5.628004][ T1] dump_stack (lib/dump_stack.c:129)
[ 5.628427][ T1] print_deadlock_bug (kernel/locking/lockdep.c:3039)
[ 5.628931][ T1] __lock_acquire (kernel/locking/lockdep.c:3089 kernel/locking/lockdep.c:3891 kernel/locking/lockdep.c:5226)
[ 5.629437][ T1] lock_acquire (kernel/locking/lockdep.c:5849)
[ 5.629883][ T1] ? blocking_notifier_call_chain (kernel/notifier.c:?)
[ 5.630484][ T1] down_read (kernel/locking/rwsem.c:1524)
[ 5.630907][ T1] ? blocking_notifier_call_chain (kernel/notifier.c:?)
[ 5.631522][ T1] ? notifier_call_chain (kernel/notifier.c:?)
[ 5.632054][ T1] blocking_notifier_call_chain (kernel/notifier.c:?)
[ 5.632631][ T1] call_netdevice_notifiers_info (net/core/dev.c:1997)
[ 5.633224][ T1] register_netdevice (include/linux/notifier.h:207 net/core/dev.c:10597)
[ 5.633717][ T1] bpq_device_event (drivers/net/hamradio/bpqether.c:500 drivers/net/hamradio/bpqether.c:542)
[ 5.634190][ T1] notifier_call_chain (kernel/notifier.c:?)
[ 5.634710][ T1] blocking_notifier_call_chain (kernel/notifier.c:380)
[ 5.635284][ T1] call_netdevice_notifiers_info (net/core/dev.c:1997)
[ 5.635875][ T1] __dev_notify_flags (net/core/dev.c:8995)
[ 5.636372][ T1] dev_change_flags (net/core/dev.c:?)
[ 5.636833][ T1] ic_open_devs (net/ipv4/ipconfig.c:242)
[ 5.637265][ T1] ? wait_for_devices (net/ipv4/ipconfig.c:?)
[ 5.637736][ T1] ip_auto_config (net/ipv4/ipconfig.c:1515)
[ 5.638192][ T1] do_one_initcall (init/main.c:1266)
[ 5.638656][ T1] ? root_nfs_parse_addr (net/ipv4/ipconfig.c:1477)
[ 5.639169][ T1] ? __lock_acquire (kernel/locking/lockdep.c:4670 kernel/locking/lockdep.c:5180)
[ 5.639684][ T1] ? ktime_get (include/linux/seqlock.h:368 (discriminator 3) include/linux/seqlock.h:388 (discriminator 3) kernel/time/timekeeping.c:815 (discriminator 3))
[ 5.640142][ T1] ? irqentry_exit (kernel/entry/common.c:?)
[ 5.640619][ T1] ? check_preemption_disabled (lib/smp_processor_id.c:16 (discriminator 1))
[ 5.641205][ T1] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67)
[ 5.641755][ T1] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4468 (discriminator 5))
[ 5.642293][ T1] ? trace_hardirqs_on (kernel/trace/trace_preemptirq.c:80)
[ 5.642800][ T1] ? irqentry_exit (kernel/entry/common.c:?)
[ 5.643271][ T1] ? vmware_sched_clock (arch/x86/kernel/apic/apic.c:1049)
[ 5.643776][ T1] ? sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1049 (discriminator 6))
[ 5.644336][ T1] ? handle_exception (init_task.c:?)
[ 5.644843][ T1] ? next_arg (lib/cmdline.c:273)
[ 5.645293][ T1] do_initcall_level (init/main.c:1327 (discriminator 6))
[ 5.645781][ T1] do_initcalls (init/main.c:1341 (discriminator 2))
[ 5.646223][ T1] do_basic_setup (init/main.c:1364)
[ 5.646697][ T1] kernel_init_freeable (init/main.c:1579)
[ 5.647216][ T1] ? rest_init (init/main.c:1458)
[ 5.647671][ T1] ? rest_init (init/main.c:1458)
[ 5.648135][ T1] kernel_init (init/main.c:1468)
[ 5.648604][ T1] ret_from_fork (arch/x86/kernel/process.c:153)
[ 5.649060][ T1] ret_from_fork_asm (??:?)
[ 5.649533][ T1] entry_INT80_32 (init_task.c:?)
[    7.669903][   T35] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[    7.682723][    T1] Sending DHCP requests ., OK

The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250108/[email protected]

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#3 lore raw

PATCH 2/4 net: Hold __rtnl_net_lock() in (un)?register_netdevice_notifier().

In reply to Kuniyuki Iwashima

(un)?register_netdevice_notifier() hold pernet_ops_rwsem and RTNL, iterate all netns, and trigger the notifier for all netdev.

Let's hold __rtnl_net_lock() before triggering the notifier.

---
 net/core/dev.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
net/core/dev.c +10 -2
index 404f5bda821b..7d49b4018ea2 100644
@@ -1776,7 +1776,9 @@ int register_netdevice_notifier(struct notifier_block *nb)
 	if (dev_boot_phase)
 		goto unlock;
 	for_each_net(net) {
+		__rtnl_net_lock(net);
 		err = call_netdevice_register_net_notifiers(nb, net);
+		__rtnl_net_unlock(net);
 		if (err)
 			goto rollback;
 	}
@@ -1787,8 +1789,11 @@ int register_netdevice_notifier(struct notifier_block *nb)
 	return err;
 
 rollback:
-	for_each_net_continue_reverse(net)
+	for_each_net_continue_reverse(net) {
+		__rtnl_net_lock(net);
 		call_netdevice_unregister_net_notifiers(nb, net);
+		__rtnl_net_unlock(net);
+	}
 
 	blocking_notifier_chain_unregister(&netdev_chain, nb);
 	goto unlock;
@@ -1821,8 +1826,11 @@ int unregister_netdevice_notifier(struct notifier_block *nb)
 	if (err)
 		goto unlock;
 
-	for_each_net(net)
+	for_each_net(net) {
+		__rtnl_net_lock(net);
 		call_netdevice_unregister_net_notifiers(nb, net);
+		__rtnl_net_unlock(net);
+	}
 
 unlock:
 	rtnl_unlock();
Signature · 1 line
2.39.5 (Apple Git-154)
#4 lore raw

PATCH 3/4 net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_net().

In reply to Kuniyuki Iwashima

(un)?register_netdevice_notifier_net() hold RTNL before triggering the notifier for all netdev in the netns.

Let's convert the RTNL to rtnl_net_lock().

Note that the per-netns netdev notifier is protected by per-netns RTNL, so we do not need to convert it to blocking_notifier.

---
 net/core/dev.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
net/core/dev.c +6 -4
index 7d49b4018ea2..f6c6559e2548 100644
@@ -1894,9 +1894,10 @@ int register_netdevice_notifier_net(struct net *net, struct notifier_block *nb)
 {
 	int err;
 
-	rtnl_lock();
+	rtnl_net_lock(net);
 	err = __register_netdevice_notifier_net(net, nb, false);
-	rtnl_unlock();
+	rtnl_net_unlock(net);
+
 	return err;
 }
 EXPORT_SYMBOL(register_netdevice_notifier_net);
@@ -1922,9 +1923,10 @@ int unregister_netdevice_notifier_net(struct net *net,
 {
 	int err;
 
-	rtnl_lock();
+	rtnl_net_lock(net);
 	err = __unregister_netdevice_notifier_net(net, nb);
-	rtnl_unlock();
+	rtnl_net_unlock(net);
+
 	return err;
 }
 EXPORT_SYMBOL(unregister_netdevice_notifier_net);
Signature · 1 line
2.39.5 (Apple Git-154)
#5 lore raw

PATCH 4/4 net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().

In reply to Kuniyuki Iwashima

(un)?register_netdevice_notifier_dev_net() hold RTNL before triggering the notifier for all netdev in the netns.

Let's convert the RTNL to rtnl_net_lock().

Note that move_netdevice_notifiers_dev_net() is assumed to be (but not yet) protected by per-netns RTNL of both src and dst netns; we need to convert wireless and hyperv drivers that call dev_change_net_namespace().

---
 net/core/dev.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
net/core/dev.c +10 -6
index f6c6559e2548..a0dd34463901 100644
@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev,
 					struct notifier_block *nb,
 					struct netdev_net_notifier *nn)
 {
+	struct net *net = dev_net(dev);
 	int err;
 
-	rtnl_lock();
-	err = __register_netdevice_notifier_net(dev_net(dev), nb, false);
+	rtnl_net_lock(net);
+	err = __register_netdevice_notifier_net(net, nb, false);
 	if (!err) {
 		nn->nb = nb;
 		list_add(&nn->list, &dev->net_notifier_list);
 	}
-	rtnl_unlock();
+	rtnl_net_unlock(net);
+
 	return err;
 }
 EXPORT_SYMBOL(register_netdevice_notifier_dev_net);
@@ -1960,12 +1962,14 @@ int unregister_netdevice_notifier_dev_net(struct net_device *dev,
 					  struct notifier_block *nb,
 					  struct netdev_net_notifier *nn)
 {
+	struct net *net = dev_net(dev);
 	int err;
 
-	rtnl_lock();
+	rtnl_net_lock(net);
 	list_del(&nn->list);
-	err = __unregister_netdevice_notifier_net(dev_net(dev), nb);
-	rtnl_unlock();
+	err = __unregister_netdevice_notifier_net(net, nb);
+	rtnl_net_unlock(net);
+
 	return err;
 }
 EXPORT_SYMBOL(unregister_netdevice_notifier_dev_net);
Signature · 1 line
2.39.5 (Apple Git-154)
#6 lore raw

Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().

we observed in our regression tests the following issue:

BUG: KASAN: slab-use-after-free in notifier_call_chain+0x22c/0x280
kasan_report+0xbd/0xf0
RIP: 0033:0x7f70839018b7
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
__kasan_kmalloc+0x83/0x90
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
kasan_save_free_info+0x37/0x50
__kasan_slab_free+0x33/0x40
page dumped because: kasan: bad access detected
BUG: KASAN: slab-use-after-free in notifier_call_chain+0x222/0x280
kasan_report+0xbd/0xf0
RIP: 0033:0x7f70839018b7
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
__kasan_kmalloc+0x83/0x90
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
kasan_save_free_info+0x37/0x50
__kasan_slab_free+0x33/0x40
page dumped because: kasan: bad access detected

and there are many more of that kind.

it happens after applying commit 7fb1073300a2 ("net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net()")

test scenario includes configuration and traffic over two namespaces associated with two different VFs.

Quoted 19 lines · Kuniyuki Iwashima

(un)?register_netdevice_notifier_dev_net() hold RTNL before triggering the notifier for all netdev in the netns.

Let's convert the RTNL to rtnl_net_lock().

Note that move_netdevice_notifiers_dev_net() is assumed to be (but not yet) protected by per-netns RTNL of both src and dst netns; we need to convert wireless and hyperv drivers that call dev_change_net_namespace().

---
  net/core/dev.c | 16 ++++++++++------
  1 file changed, 10 insertions(+), 6 deletions(-)
net/core/dev.c +0 -0
index f6c6559e2548..a0dd34463901 100644

net/core/dev.c int register_netdevice_notifier_dev_net(struct net_device *dev, · line 1946

Quoted 4 lines
net/core/dev.c +1 -0
@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev,
  					struct notifier_block *nb,
  					struct netdev_net_notifier *nn)
  {
+	struct net *net = dev_net(dev);

it seems to happen since the net pointer is acquired here without a lock.
Note that KASAN issue is not triggered when executing with rtnl_lock() taken before this line. and our kernel .config expands rtnl_net_lock(net) to rtnl_lock() (CONFIG_DEBUG_NET_SMALL_RTNL is not set).

Quoted 34 lines
net/core/dev.c +9 -6
  	int err;
 
-	rtnl_lock();
-	err = __register_netdevice_notifier_net(dev_net(dev), nb, false);
+	rtnl_net_lock(net);
+	err = __register_netdevice_notifier_net(net, nb, false);
  	if (!err) {
  		nn->nb = nb;
  		list_add(&nn->list, &dev->net_notifier_list);
  	}
-	rtnl_unlock();
+	rtnl_net_unlock(net);
+
  	return err;
  }
  EXPORT_SYMBOL(register_netdevice_notifier_dev_net);
@@ -1960,12 +1962,14 @@ int unregister_netdevice_notifier_dev_net(struct net_device *dev,
  					  struct notifier_block *nb,
  					  struct netdev_net_notifier *nn)
  {
+	struct net *net = dev_net(dev);
  	int err;
 
-	rtnl_lock();
+	rtnl_net_lock(net);
  	list_del(&nn->list);
-	err = __unregister_netdevice_notifier_net(dev_net(dev), nb);
-	rtnl_unlock();
+	err = __unregister_netdevice_notifier_net(net, nb);
+	rtnl_net_unlock(net);
+
  	return err;
  }
  EXPORT_SYMBOL(unregister_netdevice_notifier_dev_net);
#7 lore raw

Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().

Thanks for the report!

From: Yael Chemla <[email protected]>
Date: Thu, 16 Jan 2025 00:16:27 +0200

Quoted 25 lines

we observed in our regression tests the following issue:

BUG: KASAN: slab-use-after-free in notifier_call_chain+0x22c/0x280
kasan_report+0xbd/0xf0
RIP: 0033:0x7f70839018b7
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
__kasan_kmalloc+0x83/0x90
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
kasan_save_free_info+0x37/0x50
__kasan_slab_free+0x33/0x40
page dumped because: kasan: bad access detected
BUG: KASAN: slab-use-after-free in notifier_call_chain+0x222/0x280
kasan_report+0xbd/0xf0
RIP: 0033:0x7f70839018b7
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
__kasan_kmalloc+0x83/0x90
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
kasan_save_free_info+0x37/0x50
__kasan_slab_free+0x33/0x40
page dumped because: kasan: bad access detected
Quoted 1 line

and there are many more of that kind.

Do you have any other stack traces with more callers info ? Also can you decode the trace with ./scripts/decode_stacktrace.sh ?

Quoted 6 lines

it happens after applying commit 7fb1073300a2 ("net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net()")

test scenario includes configuration and traffic over two namespaces associated with two different VFs.

Could you elaborate more about the test scenario, especially how each device/netns is dismantled after the test case ?

I guess the VF is moved to init_net ?

Quoted 17 lines · Kuniyuki Iwashima
Quoted 14 lines

(un)?register_netdevice_notifier_dev_net() hold RTNL before triggering the notifier for all netdev in the netns.

Let's convert the RTNL to rtnl_net_lock().

Note that move_netdevice_notifiers_dev_net() is assumed to be (but not yet) protected by per-netns RTNL of both src and dst netns; we need to convert wireless and hyperv drivers that call dev_change_net_namespace().

---
  net/core/dev.c | 16 ++++++++++------
  1 file changed, 10 insertions(+), 6 deletions(-)
Quoted 14 lines

net/core/dev.c int register_netdevice_notifier_dev_net(struct net_device *dev, · line 1946

Quoted 9 lines
net/core/dev.c +1 -0
index f6c6559e2548..a0dd34463901 100644
@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev,
  					struct notifier_block *nb,
  					struct netdev_net_notifier *nn)
  {
+	struct net *net = dev_net(dev);

it seems to happen since the net pointer is acquired here without a lock. Note that KASAN issue is not triggered when executing with rtnl_lock() taken before this line. and our kernel .config expands rtnl_net_lock(net) to rtnl_lock() (CONFIG_DEBUG_NET_SMALL_RTNL is not set).

It sounds like the device was being moved to another netns while unregister_netdevice_notifier_dev_net() was called.

Could you check if dev_net() is changed before/after rtnl_lock() in

* register_netdevice_notifier_dev_net()
* unregister_netdevice_notifier_dev_net()

?

#8 lore raw

Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().

Quoted 29 lines · Kuniyuki Iwashima

Thanks for the report!

From: Yael Chemla <[email protected]>
Date: Thu, 16 Jan 2025 00:16:27 +0200

Quoted 25 lines

we observed in our regression tests the following issue:

BUG: KASAN: slab-use-after-free in notifier_call_chain+0x22c/0x280
kasan_report+0xbd/0xf0
RIP: 0033:0x7f70839018b7
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
__kasan_kmalloc+0x83/0x90
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
kasan_save_free_info+0x37/0x50
__kasan_slab_free+0x33/0x40
page dumped because: kasan: bad access detected
BUG: KASAN: slab-use-after-free in notifier_call_chain+0x222/0x280
kasan_report+0xbd/0xf0
RIP: 0033:0x7f70839018b7
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
__kasan_kmalloc+0x83/0x90
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
kasan_save_free_info+0x37/0x50
__kasan_slab_free+0x33/0x40
page dumped because: kasan: bad access detected
Quoted 5 lines
Quoted 1 line

and there are many more of that kind.

Do you have any other stack traces with more callers info ? Also can you decode the trace with ./scripts/decode_stacktrace.sh ?

BUG: KASAN: slab-use-after-free in notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2))
Read of size 8 at addr ffff88810cefb4c8 by task test-bridge-lag/21127

Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
Call Trace:
 <TASK>
dump_stack_lvl (/usr/work/linux/lib/dump_stack.c:123)
print_report (/usr/work/linux/mm/kasan/report.c:379 /usr/work/linux/mm/kasan/report.c:489)
? __virt_addr_valid (/usr/work/linux/./arch/x86/include/asm/preempt.h:84 (discriminator 13) /usr/work/linux/./include/linux/rcupdate.h:964 (discriminator 13) /usr/work/linux/./include/linux/mmzone.h:2058 (discriminator 13) /usr/work/linux/arch/x86/mm/physaddr.c:65 (discriminator 13))
kasan_report (/usr/work/linux/mm/kasan/report.c:604)
? notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2))
? notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2))
notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2))
call_netdevice_notifiers_info (/usr/work/linux/net/core/dev.c:2011)
unregister_netdevice_many_notify (/usr/work/linux/net/core/dev.c:11551)
? mark_held_locks (/usr/work/linux/kernel/locking/lockdep.c:4321 (discriminator 1))
? __mutex_lock (/usr/work/linux/kernel/locking/mutex.c:689 (discriminator 2) /usr/work/linux/kernel/locking/mutex.c:735 (discriminator 2))
? lockdep_hardirqs_on_prepare (/usr/work/linux/kernel/locking/lockdep.c:4347 /usr/work/linux/kernel/locking/lockdep.c:4406)
? dev_ingress_queue_create (/usr/work/linux/net/core/dev.c:11492)
? __mutex_lock (/usr/work/linux/kernel/locking/mutex.c:689 (discriminator 2) /usr/work/linux/kernel/locking/mutex.c:735 (discriminator 2))
? __mutex_lock (/usr/work/linux/./arch/x86/include/asm/preempt.h:84 (discriminator 13) /usr/work/linux/kernel/locking/mutex.c:715 (discriminator 13) /usr/work/linux/kernel/locking/mutex.c:735 (discriminator 13))
? unregister_netdev (/usr/work/linux/./include/linux/netdevice.h:3236 /usr/work/linux/net/core/dev.c:11633)

? mutex_lock_io_nested (/usr/work/linux/kernel/locking/mutex.c:734)
? __mutex_unlock_slowpath (/usr/work/linux/./arch/x86/include/asm/atomic64_64.h:101 /usr/work/linux/./include/linux/atomic/atomic-arch-fallback.h:4329 /usr/work/linux/./include/linux/atomic/atomic-long.h:1506 /usr/work/linux/./include/linux/atomic/atomic-instrumented.h:4481 /usr/work/linux/kernel/locking/mutex.c:913) unregister_netdevice_queue (/usr/work/linux/net/core/dev.c:11487)
? unregister_netdevice_many (/usr/work/linux/net/core/dev.c:11476)
unregister_netdev (/usr/work/linux/net/core/dev.c:11635)
mlx5e_remove (/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/en_main.c:6552 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/en_main.c:6579) mlx5_core
auxiliary_bus_remove (/usr/work/linux/drivers/base/auxiliary.c:230)

device_release_driver_internal (/usr/work/linux/drivers/base/dd.c:1275 /usr/work/linux/drivers/base/dd.c:1296)
? kobject_put (/usr/work/linux/./arch/x86/include/asm/atomic.h:93 (discriminator 4) /usr/work/linux/./include/linux/atomic/atomic-arch-fallback.h:949 (discriminator 4) /usr/work/linux/./include/linux/atomic/atomic-instrumented.h:401 (discriminator 4) /usr/work/linux/./include/linux/refcount.h:264 (discriminator 4) /usr/work/linux/./include/linux/refcount.h:307 (discriminator 4) /usr/work/linux/./include/linux/refcount.h:325 (discriminator 4) /usr/work/linux/./include/linux/kref.h:64 (discriminator 4) /usr/work/linux/lib/kobject.c:737 (discriminator 4))
bus_remove_device (/usr/work/linux/./include/linux/kobject.h:193 /usr/work/linux/drivers/base/base.h:73 /usr/work/linux/drivers/base/bus.c:583)
device_del (/usr/work/linux/drivers/base/power/power.h:142 /usr/work/linux/drivers/base/core.c:3855)

? mlx5_core_is_eth_enabled (/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/devlink.h:65) mlx5_core
? __device_link_del (/usr/work/linux/drivers/base/core.c:3809)
? is_ib_enabled (/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/devlink.h:58) mlx5_core
mlx5_rescan_drivers_locked (/usr/work/linux/./include/linux/auxiliary_bus.h:241 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/dev.c:333 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/dev.c:535 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/dev.c:549) mlx5_core mlx5_unregister_device (/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/dev.c:468) mlx5_core

mlx5_uninit_one (/usr/work/linux/./include/linux/instrumented.h:68 /usr/work/linux/./include/asm-generic/bitops/instrumented-non-atomic.h:141 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/main.c:1563) mlx5_core
remove_one (/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/main.c:965 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/main.c:2019) mlx5_core
pci_device_remove (/usr/work/linux/./include/linux/pm_runtime.h:129 /usr/work/linux/drivers/pci/pci-driver.c:475)
device_release_driver_internal (/usr/work/linux/drivers/base/dd.c:1275 /usr/work/linux/drivers/base/dd.c:1296)
unbind_store (/usr/work/linux/drivers/base/bus.c:245)
kernfs_fop_write_iter (/usr/work/linux/fs/kernfs/file.c:338)
vfs_write (/usr/work/linux/fs/read_write.c:587 (discriminator 1) /usr/work/linux/fs/read_write.c:679 (discriminator 1))
? do_user_addr_fault (/usr/work/linux/./include/linux/rcupdate.h:337 /usr/work/linux/./include/linux/rcupdate.h:849 /usr/work/linux/./include/linux/mm.h:740 /usr/work/linux/arch/x86/mm/fault.c:1340)

? kernel_write (/usr/work/linux/fs/read_write.c:660)
? lock_downgrade (/usr/work/linux/kernel/locking/lockdep.c:5857) ksys_write (/usr/work/linux/fs/read_write.c:732)
? __x64_sys_read (/usr/work/linux/fs/read_write.c:721)

? do_user_addr_fault (/usr/work/linux/./arch/x86/include/asm/preempt.h:84 (discriminator 13) /usr/work/linux/./include/linux/rcupdate.h:98 (discriminator 13) /usr/work/linux/./include/linux/rcupdate.h:882 (discriminator 13) /usr/work/linux/./include/linux/mm.h:742 (discriminator 13) /usr/work/linux/arch/x86/mm/fault.c:1340 (discriminator 13))
do_syscall_64 (/usr/work/linux/arch/x86/entry/common.c:52 (discriminator 1) /usr/work/linux/arch/x86/entry/common.c:83 (discriminator 1))
entry_SYSCALL_64_after_hwframe (/usr/work/linux/arch/x86/entry/entry_64.S:130)
RIP: 0033:0x7f6a4d5018b7
Quoted 10 lines
Quoted 6 lines

it happens after applying commit 7fb1073300a2 ("net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net()")

test scenario includes configuration and traffic over two namespaces associated with two different VFs.

Could you elaborate more about the test scenario, especially how each device/netns is dismantled after the test case ?

we set up a network configuration which includes two VFs isolated using two namespaces (there’s also bridge in this topology), we pass some traffic between VFs. At the end of test (cleanup) we delete network namespaces, wait for 0.5 sec and unbind VFs of NIC.

note that when I extended the timeout after deleting the namespaces the issue doesn’t reproduce.

Quoted 2 lines

I guess the VF is moved to init_net ?

this should be the behavior in case deletion of the namespaces happens before the unbind of VFs.

Quoted 35 lines
Quoted 17 lines · Kuniyuki Iwashima
Quoted 14 lines

(un)?register_netdevice_notifier_dev_net() hold RTNL before triggering the notifier for all netdev in the netns.

Let's convert the RTNL to rtnl_net_lock().

Note that move_netdevice_notifiers_dev_net() is assumed to be (but not yet) protected by per-netns RTNL of both src and dst netns; we need to convert wireless and hyperv drivers that call dev_change_net_namespace().

---
   net/core/dev.c | 16 ++++++++++------
   1 file changed, 10 insertions(+), 6 deletions(-)
Quoted 14 lines

net/core/dev.c int register_netdevice_notifier_dev_net(struct net_device *dev, · line 1946

Quoted 9 lines
net/core/dev.c +1 -0
index f6c6559e2548..a0dd34463901 100644
@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev,
   					struct notifier_block *nb,
   					struct netdev_net_notifier *nn)
   {
+	struct net *net = dev_net(dev);

it seems to happen since the net pointer is acquired here without a lock. Note that KASAN issue is not triggered when executing with rtnl_lock() taken before this line. and our kernel .config expands rtnl_net_lock(net) to rtnl_lock() (CONFIG_DEBUG_NET_SMALL_RTNL is not set).

It sounds like the device was being moved to another netns while unregister_netdevice_notifier_dev_net() was called.

Quoted 6 lines

Could you check if dev_net() is changed before/after rtnl_lock() in

* register_netdevice_notifier_dev_net()
* unregister_netdevice_notifier_dev_net()

?

When checking dev_net before and after taking the lock the issue won’t reproduce.
note that when issue reproduce we arrive to unregister_netdevice_notifier_dev_net with an invalid net pointer (verified it with prints of its value, and it's not the same consistent value as is throughout rest of the test). we suspect the issue related to the async ns deletion.

#9 lore raw

Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().

Hi Kuniyuki,

any update on this?
do you need further info?

Thanks,
Yael

Quoted 160 lines · Yael Chemla
Quoted 29 lines · Kuniyuki Iwashima

Thanks for the report!

From: Yael Chemla <[email protected]>
Date: Thu, 16 Jan 2025 00:16:27 +0200

Quoted 25 lines

we observed in our regression tests the following issue:

BUG: KASAN: slab-use-after-free in notifier_call_chain+0x22c/0x280
kasan_report+0xbd/0xf0
RIP: 0033:0x7f70839018b7
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
__kasan_kmalloc+0x83/0x90
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
kasan_save_free_info+0x37/0x50
__kasan_slab_free+0x33/0x40
page dumped because: kasan: bad access detected
BUG: KASAN: slab-use-after-free in notifier_call_chain+0x222/0x280
kasan_report+0xbd/0xf0
RIP: 0033:0x7f70839018b7
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
__kasan_kmalloc+0x83/0x90
kasan_save_stack+0x1c/0x40
kasan_save_track+0x10/0x30
kasan_save_free_info+0x37/0x50
__kasan_slab_free+0x33/0x40
page dumped because: kasan: bad access detected
Quoted 5 lines
Quoted 1 line

and there are many more of that kind.

Do you have any other stack traces with more callers info ? Also can you decode the trace with ./scripts/decode_stacktrace.sh ?

BUG: KASAN: slab-use-after-free in notifier_call_chain (/usr/work/linux/ kernel/notifier.c:75 (discriminator 2))
Read of size 8 at addr ffff88810cefb4c8 by task test-bridge-lag/21127

Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0- gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
Call Trace:
  <TASK>
dump_stack_lvl (/usr/work/linux/lib/dump_stack.c:123)
print_report (/usr/work/linux/mm/kasan/report.c:379 /usr/work/linux/mm/ kasan/report.c:489)
? __virt_addr_valid (/usr/work/linux/./arch/x86/include/asm/preempt.h:84 (discriminator 13) /usr/work/linux/./include/linux/rcupdate.h:964 (discriminator 13) /usr/work/linux/./include/linux/mmzone.h:2058 (discriminator 13) /usr/work/linux/arch/x86/mm/physaddr.c:65 (discriminator 13))
kasan_report (/usr/work/linux/mm/kasan/report.c:604)
? notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2))
? notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2))
notifier_call_chain (/usr/work/linux/kernel/notifier.c:75 (discriminator 2))
call_netdevice_notifiers_info (/usr/work/linux/net/core/dev.c:2011)
unregister_netdevice_many_notify (/usr/work/linux/net/core/dev.c:11551)
? mark_held_locks (/usr/work/linux/kernel/locking/lockdep.c:4321 (discriminator 1))
? __mutex_lock (/usr/work/linux/kernel/locking/mutex.c:689 (discriminator 2) /usr/work/linux/kernel/locking/mutex.c:735 (discriminator 2))
? lockdep_hardirqs_on_prepare (/usr/work/linux/kernel/locking/ lockdep.c:4347 /usr/work/linux/kernel/locking/lockdep.c:4406)
? dev_ingress_queue_create (/usr/work/linux/net/core/dev.c:11492)
? __mutex_lock (/usr/work/linux/kernel/locking/mutex.c:689 (discriminator 2) /usr/work/linux/kernel/locking/mutex.c:735 (discriminator 2))
? __mutex_lock (/usr/work/linux/./arch/x86/include/asm/preempt.h:84 (discriminator 13) /usr/work/linux/kernel/locking/mutex.c:715 (discriminator 13) /usr/work/linux/kernel/locking/mutex.c:735 (discriminator 13))
? unregister_netdev (/usr/work/linux/./include/linux/netdevice.h:3236 / usr/work/linux/net/core/dev.c:11633)

? mutex_lock_io_nested (/usr/work/linux/kernel/locking/mutex.c:734)
? __mutex_unlock_slowpath (/usr/work/linux/./arch/x86/include/asm/ atomic64_64.h:101 /usr/work/linux/./include/linux/atomic/atomic-arch- fallback.h:4329 /usr/work/linux/./include/linux/atomic/atomic- long.h:1506 /usr/work/linux/./include/linux/atomic/atomic- instrumented.h:4481 /usr/work/linux/kernel/locking/mutex.c:913) unregister_netdevice_queue (/usr/work/linux/net/core/dev.c:11487)
? unregister_netdevice_many (/usr/work/linux/net/core/dev.c:11476)
unregister_netdev (/usr/work/linux/net/core/dev.c:11635)
mlx5e_remove (/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/ en_main.c:6552 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/ en_main.c:6579) mlx5_core
auxiliary_bus_remove (/usr/work/linux/drivers/base/auxiliary.c:230)

device_release_driver_internal (/usr/work/linux/drivers/base/dd.c:1275 / usr/work/linux/drivers/base/dd.c:1296)
? kobject_put (/usr/work/linux/./arch/x86/include/asm/atomic.h:93 (discriminator 4) /usr/work/linux/./include/linux/atomic/atomic-arch- fallback.h:949 (discriminator 4) /usr/work/linux/./include/linux/atomic/ atomic-instrumented.h:401 (discriminator 4) /usr/work/linux/./include/ linux/refcount.h:264 (discriminator 4) /usr/work/linux/./include/linux/ refcount.h:307 (discriminator 4) /usr/work/linux/./include/linux/ refcount.h:325 (discriminator 4) /usr/work/linux/./include/linux/ kref.h:64 (discriminator 4) /usr/work/linux/lib/kobject.c:737 (discriminator 4))
bus_remove_device (/usr/work/linux/./include/linux/kobject.h:193 /usr/ work/linux/drivers/base/base.h:73 /usr/work/linux/drivers/base/bus.c:583)
device_del (/usr/work/linux/drivers/base/power/power.h:142 /usr/work/ linux/drivers/base/core.c:3855)

? mlx5_core_is_eth_enabled (/usr/work/linux/drivers/net/ethernet/ mellanox/mlx5/core/devlink.h:65) mlx5_core
? __device_link_del (/usr/work/linux/drivers/base/core.c:3809)
? is_ib_enabled (/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/ core/devlink.h:58) mlx5_core
mlx5_rescan_drivers_locked (/usr/work/linux/./include/linux/ auxiliary_bus.h:241 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/ core/dev.c:333 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/ dev.c:535 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/ dev.c:549) mlx5_core mlx5_unregister_device (/usr/work/linux/drivers/net/ethernet/mellanox/ mlx5/core/dev.c:468) mlx5_core
mlx5_uninit_one (/usr/work/linux/./include/linux/instrumented.h:68 /usr/ work/linux/./include/asm-generic/bitops/instrumented-non-atomic.h:141 / usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/main.c:1563) mlx5_core
remove_one (/usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/ main.c:965 /usr/work/linux/drivers/net/ethernet/mellanox/mlx5/core/ main.c:2019) mlx5_core

pci_device_remove (/usr/work/linux/./include/linux/pm_runtime.h:129 / usr/work/linux/drivers/pci/pci-driver.c:475)
device_release_driver_internal (/usr/work/linux/drivers/base/dd.c:1275 / usr/work/linux/drivers/base/dd.c:1296)
unbind_store (/usr/work/linux/drivers/base/bus.c:245)
kernfs_fop_write_iter (/usr/work/linux/fs/kernfs/file.c:338)
vfs_write (/usr/work/linux/fs/read_write.c:587 (discriminator 1) /usr/ work/linux/fs/read_write.c:679 (discriminator 1))
? do_user_addr_fault (/usr/work/linux/./include/linux/rcupdate.h:337 / usr/work/linux/./include/linux/rcupdate.h:849 /usr/work/linux/./include/ linux/mm.h:740 /usr/work/linux/arch/x86/mm/fault.c:1340)

? kernel_write (/usr/work/linux/fs/read_write.c:660)
? lock_downgrade (/usr/work/linux/kernel/locking/lockdep.c:5857)
ksys_write (/usr/work/linux/fs/read_write.c:732)
? __x64_sys_read (/usr/work/linux/fs/read_write.c:721)
? do_user_addr_fault (/usr/work/linux/./arch/x86/include/asm/ preempt.h:84 (discriminator 13) /usr/work/linux/./include/linux/ rcupdate.h:98 (discriminator 13) /usr/work/linux/./include/linux/ rcupdate.h:882 (discriminator 13) /usr/work/linux/./include/linux/ mm.h:742 (discriminator 13) /usr/work/linux/arch/x86/mm/fault.c:1340 (discriminator 13)) do_syscall_64 (/usr/work/linux/arch/x86/entry/common.c:52 (discriminator 1) /usr/work/linux/arch/x86/entry/common.c:83 (discriminator 1))
entry_SYSCALL_64_after_hwframe (/usr/work/linux/arch/x86/entry/ entry_64.S:130)

RIP: 0033:0x7f6a4d5018b7
Quoted 10 lines
Quoted 6 lines

it happens after applying commit 7fb1073300a2 ("net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net()")

test scenario includes configuration and traffic over two namespaces associated with two different VFs.

Could you elaborate more about the test scenario, especially how each device/netns is dismantled after the test case ?

we set up a network configuration which includes two VFs isolated using two namespaces (there’s also bridge in this topology), we pass some traffic between VFs. At the end of test (cleanup) we delete network namespaces, wait for 0.5 sec and unbind VFs of NIC.

note that when I extended the timeout after deleting the namespaces the issue doesn’t reproduce.

Quoted 2 lines

I guess the VF is moved to init_net ?

this should be the behavior in case deletion of the namespaces happens before the unbind of VFs.

Quoted 36 lines
Quoted 17 lines · Kuniyuki Iwashima
Quoted 14 lines

(un)?register_netdevice_notifier_dev_net() hold RTNL before triggering the notifier for all netdev in the netns.

Let's convert the RTNL to rtnl_net_lock().

Note that move_netdevice_notifiers_dev_net() is assumed to be (but not yet) protected by per-netns RTNL of both src and dst netns; we need to convert wireless and hyperv drivers that call dev_change_net_namespace().

---
   net/core/dev.c | 16 ++++++++++------

   1 file changed, 10 insertions(+), 6 deletions(-)
Quoted 15 lines

net/core/dev.c int · line 1947

Quoted 10 lines
net/core/dev.c +1 -0
index f6c6559e2548..a0dd34463901 100644
@@ -1943,15 +1943,17 @@ int 
 register_netdevice_notifier_dev_net(struct net_device *dev,
                        struct notifier_block *nb,
                        struct netdev_net_notifier *nn)
    {
+    struct net *net = dev_net(dev);

it seems to happen since the net pointer is acquired here without a lock. Note that KASAN issue is not triggered when executing with rtnl_lock() taken before this line. and our kernel .config expands rtnl_net_lock(net) to rtnl_lock() (CONFIG_DEBUG_NET_SMALL_RTNL is not set).

It sounds like the device was being moved to another netns while unregister_netdevice_notifier_dev_net() was called.

Quoted 6 lines

Could you check if dev_net() is changed before/after rtnl_lock() in

   * register_netdevice_notifier_dev_net()
   * unregister_netdevice_notifier_dev_net()

?

When checking dev_net before and after taking the lock the issue won’t reproduce.
note that when issue reproduce we arrive to unregister_netdevice_notifier_dev_net with an invalid net pointer (verified it with prints of its value, and it's not the same consistent value as is throughout rest of the test). we suspect the issue related to the async ns deletion.

#10 lore raw

Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().

From: Yael Chemla <[email protected]>
Date: Mon, 27 Jan 2025 19:26:34 +0200

Quoted 4 lines

Hi Kuniyuki,

any update on this?
do you need further info?

Sorry, I was busy as on-call last week.
I'll investigate it this week.

Thanks!

#11 lore raw

Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().

In reply to Yael Chemla

From: Yael Chemla <[email protected]>
Date: Mon, 20 Jan 2025 20:55:07 +0200

Quoted 23 lines
Quoted 23 lines
Quoted 6 lines
Quoted 6 lines
net/core/dev.c +0 -0
index f6c6559e2548..a0dd34463901 100644
@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev,
   					struct notifier_block *nb,
Quoted 8 lines

net/core/dev.c int register_netdevice_notifier_dev_net(struct net_device *dev, · line 1946

Quoted 3 lines
net/core/dev.c +1 -0
   					struct netdev_net_notifier *nn)
   {
+	struct net *net = dev_net(dev);

it seems to happen since the net pointer is acquired here without a lock. Note that KASAN issue is not triggered when executing with rtnl_lock() taken before this line. and our kernel .config expands rtnl_net_lock(net) to rtnl_lock() (CONFIG_DEBUG_NET_SMALL_RTNL is not set).

It sounds like the device was being moved to another netns while unregister_netdevice_notifier_dev_net() was called.

Could you check if dev_net() is changed before/after rtnl_lock() in

* register_netdevice_notifier_dev_net()
* unregister_netdevice_notifier_dev_net()

Quoted 8 lines
Quoted 1 line

?

When checking dev_net before and after taking the lock the issue won’t reproduce.
note that when issue reproduce we arrive to unregister_netdevice_notifier_dev_net with an invalid net pointer (verified it with prints of its value, and it's not the same consistent value as is throughout rest of the test).

Does an invalid net pointer means a dead netns pointer ? dev_net() and dev_net_set() use rcu_dereference() and rcu_assign_pointer(), so I guess it should not be an invalid address at least.

Quoted 1 line

we suspect the issue related to the async ns deletion.

I think async netns change would trigger the issue too.

Could you try this patch ?

---8<---

net/core/dev.c +37 -8
index afa2282f2604..f4438ec24683 100644
@@ -2070,20 +2070,50 @@ static void __move_netdevice_notifier_net(struct net *src_net,
 	__register_netdevice_notifier_net(dst_net, nb, true);
 }
 
+static void rtnl_net_dev_lock(struct net_device *dev)
+{
+	struct net *net;
+
+again:
+	/* netns might be being dismantled. */
+	net = maybe_get_net(dev_net(dev));
+	if (!net) {
+		cond_resched();
+		goto again;
+	}
+
+	rtnl_net_lock(net);
+
+	/* dev might be moved to another netns. */
+	if (!net_eq(net, dev_net(dev))) {
+		rtnl_net_unlock(net);
+		put_net(net);
+		cond_resched();
+		goto again;
+	}
+}
+
+static void rtnl_net_dev_unlock(struct net_device *dev)
+{
+	struct net *net = dev_net(dev);
+
+	rtnl_net_unlock(net);
+	put_net(net);
+}
+
 int register_netdevice_notifier_dev_net(struct net_device *dev,
 					struct notifier_block *nb,
 					struct netdev_net_notifier *nn)
 {
-	struct net *net = dev_net(dev);
 	int err;
 
-	rtnl_net_lock(net);
-	err = __register_netdevice_notifier_net(net, nb, false);
+	rtnl_net_dev_lock(dev);
+	err = __register_netdevice_notifier_net(dev_net(dev), nb, false);
 	if (!err) {
 		nn->nb = nb;
 		list_add(&nn->list, &dev->net_notifier_list);
 	}
-	rtnl_net_unlock(net);
+	rtnl_net_dev_unlock(dev);
 
 	return err;
 }
@@ -2093,13 +2123,12 @@ int unregister_netdevice_notifier_dev_net(struct net_device *dev,
 					  struct notifier_block *nb,
 					  struct netdev_net_notifier *nn)
 {
-	struct net *net = dev_net(dev);
 	int err;
 
-	rtnl_net_lock(net);
+	rtnl_net_dev_lock(dev);
 	list_del(&nn->list);
-	err = __unregister_netdevice_notifier_net(net, nb);
-	rtnl_net_unlock(net);
+	err = __unregister_netdevice_notifier_net(dev_net(dev), nb);
+	rtnl_net_dev_unlock(dev);
 
 	return err;
 }

---8<---

Tested-by
#12 lore raw

Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().

Quoted 34 lines · Kuniyuki Iwashima

From: Yael Chemla <[email protected]>
Date: Mon, 20 Jan 2025 20:55:07 +0200

Quoted 23 lines
Quoted 23 lines
Quoted 6 lines
Quoted 6 lines
net/core/dev.c +0 -0
index f6c6559e2548..a0dd34463901 100644
@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev,
    					struct notifier_block *nb,
Quoted 8 lines

net/core/dev.c static void __move_netdevice_notifier_net(struct net *src_net,

Quoted 3 lines
net/core/dev.c +1 -0
    					struct netdev_net_notifier *nn)
    {
+	struct net *net = dev_net(dev);

it seems to happen since the net pointer is acquired here without a lock. Note that KASAN issue is not triggered when executing with rtnl_lock() taken before this line. and our kernel .config expands rtnl_net_lock(net) to rtnl_lock() (CONFIG_DEBUG_NET_SMALL_RTNL is not set).

It sounds like the device was being moved to another netns while unregister_netdevice_notifier_dev_net() was called.

Could you check if dev_net() is changed before/after rtnl_lock() in

* register_netdevice_notifier_dev_net()
* unregister_netdevice_notifier_dev_net()

Quoted 8 lines
Quoted 1 line

?

When checking dev_net before and after taking the lock the issue won’t reproduce.
note that when issue reproduce we arrive to unregister_netdevice_notifier_dev_net with an invalid net pointer (verified it with prints of its value, and it's not the same consistent value as is throughout rest of the test).

Quoted 4 lines

Does an invalid net pointer means a dead netns pointer ? dev_net() and dev_net_set() use rcu_dereference() and rcu_assign_pointer(), so I guess it should not be an invalid address at least.

I logged several values at the entrance of unregister_netdevice_notifier_dev_net when issue reproduced:
1) fields of net->ns (struct ns_common):
count: the namespace refcount is 0 (i.e. net->ns.count, used refcount_read to read it).

inum: the value doesn't appear to be garbage but differ from its constant value throughout the test.

2) net pointer (struct net): value differ from its constant value observed during the rest of the test.

hope this helps and please let me know if more info is needed.

Quoted 7 lines
Quoted 2 lines

we suspect the issue related to the async ns deletion.

I think async netns change would trigger the issue too.

Could you try this patch ?

I tested your patch and issue won't reproduce with it (CONFIG_DEBUG_NET_SMALL_RTNL is not set in my config).

Thanks a lot!

Quoted 78 lines

---8<---

net/core/dev.c +37 -9
index afa2282f2604..f4438ec24683 100644
@@ -2070,20 +2070,50 @@ static void __move_netdevice_notifier_net(struct net *src_net,
  	__register_netdevice_notifier_net(dst_net, nb, true);
  }
 
+static void rtnl_net_dev_lock(struct net_device *dev)
+{
+	struct net *net;
+
+again:
+	/* netns might be being dismantled. */
+	net = maybe_get_net(dev_net(dev));
+	if (!net) {
+		cond_resched();
+		goto again;
+	}
+
+	rtnl_net_lock(net);
+
+	/* dev might be moved to another netns. */
+	if (!net_eq(net, dev_net(dev))) {
+		rtnl_net_unlock(net);
+		put_net(net);
+		cond_resched();
+		goto again;
+	}
+}
+
+static void rtnl_net_dev_unlock(struct net_device *dev)
+{
+	struct net *net = dev_net(dev);
+
+	rtnl_net_unlock(net);
+	put_net(net);
+}
+
  int register_netdevice_notifier_dev_net(struct net_device *dev,
  					struct notifier_block *nb,
  					struct netdev_net_notifier *nn)
  {
-	struct net *net = dev_net(dev);
  	int err;
 
-	rtnl_net_lock(net);
-	err = __register_netdevice_notifier_net(net, nb, false);
+	rtnl_net_dev_lock(dev);
+	err = __register_netdevice_notifier_net(dev_net(dev), nb, false);
  	if (!err) {
  		nn->nb = nb;
  		list_add(&nn->list, &dev->net_notifier_list);
  	}
-	rtnl_net_unlock(net);
+	rtnl_net_dev_unlock(dev);
 
  	return err;
  }
@@ -2093,13 +2123,12 @@ int unregister_netdevice_notifier_dev_net(struct net_device *dev,
  					  struct notifier_block *nb,
  					  struct netdev_net_notifier *nn)
  {
-	struct net *net = dev_net(dev);
  	int err;
 
-	rtnl_net_lock(net);
+	rtnl_net_dev_lock(dev);
  	list_del(&nn->list);
-	err = __unregister_netdevice_notifier_net(net, nb);
-	rtnl_net_unlock(net);
+	err = __unregister_netdevice_notifier_net(dev_net(dev), nb);
+	rtnl_net_dev_unlock(dev);
 
  	return err;
  }
---8<---
#13 lore raw

Re: [PATCH v1 net-next 4/4] net: Hold rtnl_net_lock() in (un)?register_netdevice_notifier_dev_net().

From: Yael Chemla <[email protected]>
Date: Wed, 29 Jan 2025 18:21:23 +0200

Quoted 39 lines · Kuniyuki Iwashima
Quoted 34 lines

From: Yael Chemla <[email protected]>
Date: Mon, 20 Jan 2025 20:55:07 +0200

Quoted 23 lines
Quoted 23 lines
Quoted 6 lines
Quoted 6 lines
net/core/dev.c +0 -0
index f6c6559e2548..a0dd34463901 100644
@@ -1943,15 +1943,17 @@ int register_netdevice_notifier_dev_net(struct net_device *dev,
    					struct notifier_block *nb,
Quoted 8 lines

net/core/dev.c static void __move_netdevice_notifier_net(struct net *src_net,

Quoted 3 lines
net/core/dev.c +1 -0
    					struct netdev_net_notifier *nn)
    {
+	struct net *net = dev_net(dev);

it seems to happen since the net pointer is acquired here without a lock. Note that KASAN issue is not triggered when executing with rtnl_lock() taken before this line. and our kernel .config expands rtnl_net_lock(net) to rtnl_lock() (CONFIG_DEBUG_NET_SMALL_RTNL is not set).

It sounds like the device was being moved to another netns while unregister_netdevice_notifier_dev_net() was called.

Could you check if dev_net() is changed before/after rtnl_lock() in

* register_netdevice_notifier_dev_net()
* unregister_netdevice_notifier_dev_net()

Quoted 8 lines
Quoted 1 line

?

When checking dev_net before and after taking the lock the issue won’t reproduce.
note that when issue reproduce we arrive to unregister_netdevice_notifier_dev_net with an invalid net pointer (verified it with prints of its value, and it's not the same consistent value as is throughout rest of the test).

Quoted 4 lines

Does an invalid net pointer means a dead netns pointer ? dev_net() and dev_net_set() use rcu_dereference() and rcu_assign_pointer(), so I guess it should not be an invalid address at least.

Quoted 5 lines

I logged several values at the entrance of unregister_netdevice_notifier_dev_net when issue reproduced:
1) fields of net->ns (struct ns_common):
count: the namespace refcount is 0 (i.e. net->ns.count, used refcount_read to read it).

This indicates here we race with cleanup_net().

Quoted 14 lines

inum: the value doesn't appear to be garbage but differ from its constant value throughout the test.

2) net pointer (struct net): value differ from its constant value observed during the rest of the test.

hope this helps and please let me know if more info is needed.

Quoted 5 lines
Quoted 2 lines

we suspect the issue related to the async ns deletion.

I think async netns change would trigger the issue too.

Quoted 7 lines
Quoted 2 lines

Could you try this patch ?

I tested your patch and issue won't reproduce with it (CONFIG_DEBUG_NET_SMALL_RTNL is not set in my config).

Thanks for testing !

Will post the fix officially.

maintainer
#14 lore raw
In reply to Kuniyuki Iwashima
Quoted 6 lines · On Sat, 4 Jan 2025 15:37:31 +0900 Kuniyuki Iwashima wrote:

Patch 1 converts the global netdev notifier to blocking_notifier, which will be called under per-netns RTNL without RTNL, then we need to protect the ongoing netdev_chain users from unregistration.

Patch 2 ~ 4 adds per-netns RTNL for registration of the global and per-netns netdev notifiers.

Lockdep is not happy:

[  249.261403][   T11] ============================================ 
[  249.261592][   T11] WARNING: possible recursive locking detected
[  249.261769][   T11] 6.13.0-rc5-virtme #1 Not tainted
[  249.261920][   T11] --------------------------------------------
[  249.262094][   T11] kworker/u16:0/11 is trying to acquire lock:
[  249.262293][   T11] ffffffff8a7f6a70 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90
[  249.262591][   T11] 
[  249.262591][   T11] but task is already holding lock:
[  249.262810][   T11] ffffffff8a7f6a70 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90
[  249.263100][   T11] 
[  249.263100][   T11] other info that might help us debug this:
[  249.263310][   T11]  Possible unsafe locking scenario:
[  249.263310][   T11] 
[  249.263522][   T11]        CPU0
[  249.263624][   T11]        ----
[  249.263728][   T11]   lock((netdev_chain).rwsem);
[  249.263875][   T11]   lock((netdev_chain).rwsem);
[  249.264020][   T11] 
[  249.264020][   T11]  *** DEADLOCK ***
[  249.264020][   T11] 
[  249.264223][   T11]  May be due to missing lock nesting notation
[  249.264223][   T11] 
[  249.264440][   T11] 5 locks held by kworker/u16:0/11:
[  249.264582][   T11]  #0: ffff8880010b5948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0x7ec/0x16d0
[  249.264867][   T11]  #1: ffffc900000b7da0 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0xe0b/0x16d0
[  249.265118][   T11]  #2: ffffffff8a7ec4d0 (pernet_ops_rwsem){++++}-{4:4}, at: cleanup_net+0xbc/0xba0
[  249.265381][   T11]  #3: ffffffff8a807e88 (rtnl_mutex){+.+.}-{4:4}, at: default_device_exit_batch+0x81/0x2e0
[  249.265668][   T11]  #4: ffffffff8a7f6a70 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90
[  249.265954][   T11] 
[  249.265954][   T11] stack backtrace:
[  249.266126][   T11] CPU: 2 UID: 0 PID: 11 Comm: kworker/u16:0 Not tainted 6.13.0-rc5-virtme #1
[  249.266389][   T11] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[  249.266572][   T11] Workqueue: netns cleanup_net
[  249.266722][   T11] Call Trace:
[  249.266826][   T11]  <TASK>
[  249.266907][   T11]  dump_stack_lvl+0x82/0xd0
[  249.267056][   T11]  print_deadlock_bug+0x40a/0x650
[  249.267206][   T11]  validate_chain+0x5bf/0xae0
[  249.267352][   T11]  ? __pfx_validate_chain+0x10/0x10
[  249.267503][   T11]  ? hlock_class+0x4e/0x130
[  249.267642][   T11]  ? mark_lock+0x38/0x3e0
[  249.267751][   T11]  __lock_acquire+0xb9a/0x1680
[  249.267897][   T11]  ? spin_bug+0x191/0x1d0
[  249.268007][   T11]  ? debug_object_assert_init+0x2a9/0x370
[  249.268164][   T11]  lock_acquire.part.0+0xeb/0x330
[  249.268313][   T11]  ? blocking_notifier_call_chain+0x50/0x90
[  249.268497][   T11]  ? __pfx_lock_acquire.part.0+0x10/0x10
[  249.268651][   T11]  ? trace_lock_acquire+0x14c/0x1f0
[  249.268803][   T11]  ? lock_acquire+0x32/0xc0
[  249.268944][   T11]  ? blocking_notifier_call_chain+0x50/0x90
[  249.269132][   T11]  down_read+0x9f/0x340
[  249.269247][   T11]  ? blocking_notifier_call_chain+0x50/0x90
[  249.269436][   T11]  ? __pfx_down_read+0x10/0x10
[  249.269586][   T11]  blocking_notifier_call_chain+0x50/0x90
[  249.269739][   T11]  __dev_close_many+0xdf/0x2d0
[  249.269881][   T11]  ? __pfx___dev_close_many+0x10/0x10
[  249.270031][   T11]  dev_close_many+0x202/0x650
Signature · 1 line
pw-bot: cr
#15 lore raw

From: Jakub Kicinski <[email protected]>
Date: Sat, 4 Jan 2025 07:37:40 -0800

Quoted 19 lines · On Sat, 4 Jan 2025 15:37:31 +0900 Kuniyuki Iwashima wrote:
Quoted 6 lines

Patch 1 converts the global netdev notifier to blocking_notifier, which will be called under per-netns RTNL without RTNL, then we need to protect the ongoing netdev_chain users from unregistration.

Patch 2 ~ 4 adds per-netns RTNL for registration of the global and per-netns netdev notifiers.

Lockdep is not happy:

[  249.261403][   T11] ============================================ 
[  249.261592][   T11] WARNING: possible recursive locking detected
[  249.261769][   T11] 6.13.0-rc5-virtme #1 Not tainted
[  249.261920][   T11] --------------------------------------------
[  249.262094][   T11] kworker/u16:0/11 is trying to acquire lock:
[  249.262293][   T11] ffffffff8a7f6a70 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90
[  249.262591][   T11] 
[  249.262591][   T11] but task is already holding lock:
[  249.262810][   T11] ffffffff8a7f6a70 ((netdev_chain).rwsem){++++}-{4:4}, at: blocking_notifier_call_chain+0x50/0x90
Quoted 8 lines
[  249.263100][   T11] 
[  249.263100][   T11] other info that might help us debug this:
[  249.263310][   T11]  Possible unsafe locking scenario:
[  249.263310][   T11] 
[  249.263522][   T11]        CPU0
[  249.263624][   T11]        ----
[  249.263728][   T11]   lock((netdev_chain).rwsem);
[  249.263875][   T11]   lock((netdev_chain).rwsem);

Ah, lockdep annotaion was needed for the nested notifier calls.

But this will not be a meaningful annotation and needs to be changed once rtnl_setlink/dellink supports per-netns RTNL.

I'll drop patch 1 and just leave a comment around RTNL in register_netdevice_notifier() in patch 2.

Another option would be clone each netdev notifier during registration and unshare(2)/clone(2) and force notifiers to be namespacified ?

---8<---

net/core/dev.c +13 -0
index a0dd34463901..8bf8d565f42d 100644
@@ -446,6 +446,17 @@ static void unlist_netdevice(struct net_device *dev)
 
 static BLOCKING_NOTIFIER_HEAD(netdev_chain);
 
+#ifdef CONFIG_PROVE_LOCKING
+static int netdev_chain_lock_cmp_fn(const struct lockdep_map *a,
+				    const struct lockdep_map *b)
+{
+	if (rtnl_is_locked())
+		return -1;
+
+	return 1;
+}
+#endif
+
 /*
  *	Device drivers call our routines to queue packets here. We empty the
  *	queue in the local softnet handler.
@@ -12229,6 +12240,8 @@ static int __init net_dev_init(void)
 
 	net_dev_struct_check();
 
+	lock_set_cmp_fn(&netdev_chain.rwsem, netdev_chain_lock_cmp_fn, NULL);
+
 	if (dev_proc_init())
 		goto out;
 

---8<---