net: watchdog: fix refcount tracking races
Subsystems (3)
PATCH v3 1/1 net: watchdog: fix refcount tracking races
Blamed commit converted the untracked dev_hold()/dev_put() calls in the watchdog code to use the tracked dev_hold_track()/dev_put_track() (which were later renamed/interfaced to netdev_hold() and netdev_put()).
By introducing dev->watchdog_dev_tracker to store the reference tracking information without adding synchronization between netdev_watchdog_up() and dev_watchdog(), it enabled the race condition where this pointer could be overwritten or freed concurrently, leading to the list corruption crash syzbot reported:
list_del corruption, ffff888114a18c00->next is NULL
kernel BUG at lib/list_debug.c:52 !
Oops: invalid opcode: 0000 [#1] SMP KASAN PTI CPU: 1 UID: 0 PID: 91 Comm: kworker/u8:5 Not tainted syzkaller #0 PREEMPT(lazy) Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/09/2026 Workqueue: events_unbound linkwatch_event RIP: 0010:__list_del_entry_valid_or_report.cold+0x22/0x2a lib/list_debug.c:52 Call Trace: <TASK> __list_del_entry_valid include/linux/list.h:132 [inline] __list_del_entry include/linux/list.h:246 [inline] list_move_tail include/linux/list.h:341 [inline] ref_tracker_free+0x1a7/0x6c0 lib/ref_tracker.c:329 netdev_tracker_free include/linux/netdevice.h:4491 [inline] netdev_put include/linux/netdevice.h:4508 [inline] netdev_put include/linux/netdevice.h:4504 [inline] netdev_watchdog_down net/sched/sch_generic.c:600 [inline] dev_deactivate_many+0x28c/0xfe0 net/sched/sch_generic.c:1363 dev_deactivate+0x109/0x1d0 net/sched/sch_generic.c:1397 linkwatch_do_dev net/core/link_watch.c:184 [inline] linkwatch_do_dev+0xd3/0x120 net/core/link_watch.c:166 __linkwatch_run_queue+0x3a5/0x810 net/core/link_watch.c:240 linkwatch_event+0x8f/0xc0 net/core/link_watch.c:314 process_one_work+0xa0e/0x1980 kernel/workqueue.c:3314 process_scheduled_works kernel/workqueue.c:3397 [inline] worker_thread+0x5ef/0xe50 kernel/workqueue.c:3478 kthread+0x370/0x450 kernel/kthread.c:436 ret_from_fork+0x69a/0xc80 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
This patch has three coordinated parts:
1) Add dev->watchdog_lock and dev->watchdog_ref_held to serialize watchdog operations.
2) Remove netdev_watchdog_up() call from netif_carrier_on():
This ensures netdev_watchdog_up() is only called from process/BH context (via linkwatch workqueue dev_activate()), allowing us to use spin_lock_bh() for synchronization.
3) Synchronize watchdog up and watchdog timer:
Protect netdev_watchdog_up() with tx_global_lock and watchdog_lock. Only allocate a new tracker in netdev_watchdog_up() if one is not already present.
In dev_watchdog(), ensure we don't release the tracker if the timer was rescheduled either by dev_watchdog() itself or concurrently by netdev_watchdog_up().
---
v3: added dev->watchdog_lock and dev->watchdog_ref_held instead of messing
with ref tracker infra. (after sashiko feedback on v2)
v2: fix compile error when CONFIG_NET_DEV_REFCNT_TRACKER is not set (Jakub and build bots)
include/linux/netdevice.h | 4 ++++ net/core/dev.c | 3 ++- net/sched/sch_generic.c | 44 +++++++++++++++++++++++++++++---------- 3 files changed, 39 insertions(+), 12 deletions(-)
include/linux/netdevice.h +4 -0
@@ -1980,6 +1980,8 @@ enum netdev_reg_state {* @qdisc_hash: qdisc hash table * @watchdog_timeo: Represents the timeout that is used by * the watchdog (see dev_watchdog()) +* @watchdog_lock: protect watchdog_ref_held +* @watchdog_ref_held: True if the watchdog device ref is taken. * @watchdog_timer: List of timers * * @proto_down_reason: reason a netdev interface is held down
@@ -2392,6 +2394,8 @@ struct net_device {/* These may be needed for future network-power-down code. */ struct timer_list watchdog_timer;int watchdog_timeo; +spinlock_t watchdog_lock; +bool watchdog_ref_held;u32 proto_down_reason;
net/core/dev.c +2 -1
@@ -11217,7 +11217,8 @@ static int netif_alloc_netdev_queues(struct net_device *dev)netdev_for_each_tx_queue (dev, netdev_init_one_queue,NULL );spin_lock_init (&dev->tx_global_lock); - +spin_lock_init (&dev->watchdog_lock); + dev->watchdog_ref_held = false;return 0 ; }
net/sched/sch_generic.c +33 -11
75 lines not on this page · raw message
Signature · 1 line
2.54.0.1099.g489fc7bff1-goog
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:
Quoted 11 lines · On Thu, 11 Jun 2026 15:27:37 +0000 you wrote:
Blamed commit converted the untracked dev_hold()/dev_put() calls in the watchdog code to use the tracked dev_hold_track()/dev_put_track() (which were later renamed/interfaced to
netdev_hold()andnetdev_put()).By introducing
dev->watchdog_dev_trackerto store the reference tracking information without adding synchronization betweennetdev_watchdog_up()anddev_watchdog(), it enabled the race condition where this pointer could be overwritten or freed concurrently, leading to the list corruption crash syzbot reported:[...]
Here is the summary with links:
- [v3,net] net: watchdog: fix refcount tracking races
https://git.kernel.org/netdev/net/c/8eed5519e496
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Dear All,
Quoted 57 lines · Eric Dumazet
Blamed commit converted the untracked dev_hold()/dev_put() calls in the watchdog code to use the tracked dev_hold_track()/dev_put_track() (which were later renamed/interfaced to
netdev_hold()andnetdev_put()).By introducing
dev->watchdog_dev_trackerto store the reference tracking information without adding synchronization betweennetdev_watchdog_up()anddev_watchdog(), it enabled the race condition where this pointer could be overwritten or freed concurrently, leading to the list corruption crash syzbot reported:
list_delcorruption,ffff888114a18c00->nextis NULL
kernel BUG atlib/list_debug.c:52!Oops: invalid opcode: 0000 [#1] SMP KASAN PTI CPU: 1 UID: 0 PID: 91 Comm: kworker/u8:5 Not tainted syzkaller #0 PREEMPT(lazy) Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/09/2026 Workqueue: events_unbound linkwatch_event RIP: 0010:__list_del_entry_valid_or_report.cold+0x22/0x2a lib/list_debug.c:52 Call Trace: <TASK> __list_del_entry_valid include/linux/list.h:132 [inline] __list_del_entry include/linux/list.h:246 [inline] list_move_tail include/linux/list.h:341 [inline] ref_tracker_free+0x1a7/0x6c0 lib/ref_tracker.c:329 netdev_tracker_free include/linux/netdevice.h:4491 [inline] netdev_put include/linux/netdevice.h:4508 [inline] netdev_put include/linux/netdevice.h:4504 [inline] netdev_watchdog_down net/sched/sch_generic.c:600 [inline] dev_deactivate_many+0x28c/0xfe0 net/sched/sch_generic.c:1363 dev_deactivate+0x109/0x1d0 net/sched/sch_generic.c:1397 linkwatch_do_dev net/core/link_watch.c:184 [inline] linkwatch_do_dev+0xd3/0x120 net/core/link_watch.c:166 __linkwatch_run_queue+0x3a5/0x810 net/core/link_watch.c:240 linkwatch_event+0x8f/0xc0 net/core/link_watch.c:314 process_one_work+0xa0e/0x1980 kernel/workqueue.c:3314 process_scheduled_works kernel/workqueue.c:3397 [inline] worker_thread+0x5ef/0xe50 kernel/workqueue.c:3478 kthread+0x370/0x450 kernel/kthread.c:436 ret_from_fork+0x69a/0xc80 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245This patch has three coordinated parts:
1) Add
dev->watchdog_lockanddev->watchdog_ref_heldto serialize watchdog operations.2) Remove
netdev_watchdog_up()call fromnetif_carrier_on():
This ensuresnetdev_watchdog_up()is only called from process/BH context (via linkwatch workqueuedev_activate()), allowing us to usespin_lock_bh()for synchronization.3) Synchronize watchdog up and watchdog timer:
Protectnetdev_watchdog_up()withtx_global_lockandwatchdog_lock. Only allocate a new tracker innetdev_watchdog_up()if one is not already present.
Indev_watchdog(), ensure we don't release the tracker if the timer was rescheduled either bydev_watchdog()itself or concurrently bynetdev_watchdog_up().
Quoted 5 lines
Fixes f12bf6f3f942 ("net: watchdog: add net device refcount tracker")Reported-by [email protected]Tested-by [email protected]Signed-off-by Eric Dumazet <[email protected]>
This patch landed recently in linux-next as commit 8eed5519e496 ("net: watchdog:
fix refcount tracking races"). In my tests I found that it causes the following deadlock during system suspend/resume on QEmu's ARM64bit 'virt' machine:
root@target:~# time rtcwake -s10 -mmem rtcwake: assuming RTC uses UTC ... rtcwake: wakeup from "mem" using /dev/rtc0 at Wed Jun 17 10:46:12 2026 PM: suspend entry (s2idle) Filesystems sync: 0.055 seconds Freezing user space processes Freezing user space processes completed (elapsed 0.006 seconds) OOM killer disabled. Freezing remaining freezable tasks Freezing remaining freezable tasks completed (elapsed 0.003 seconds)
============================================
WARNING: possible recursive locking detected
7.1.0-rc7+ #13003 Not tainted
--------------------------------------------
rtcwake/254 is trying to acquire lock:ffff000006de64e8 (&dev->tx_global_lock){+.-.}-{3:3}, at: netdev_watchdog_up+0x40/0x108
but task is already holding lock:ffff000006de64e8 (&dev->tx_global_lock){+.-.}-{3:3}, at: netif_tx_lock+0x1c/0x34
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&dev->tx_global_lock);
lock(&dev->tx_global_lock);
*** DEADLOCK ***
May be due to missing lock nesting notation
6 locks held by rtcwake/254:
#0: ffff0000071ab3e8 (sb_writers#5){.+.+}-{0:0}, at: vfs_write+0x1ec/0x35c #1: ffff00000d22c480 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0xf0/0x1c4 #2: ffff0000049162c8 (kn->active#61){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x100/0x1c4 #3: ffffaa79533c03b0 (system_transition_mutex){+.+.}-{4:4}, at: pm_suspend+0x98/0x608 #4: ffff000005e3a138 (&dev->mutex){....}-{4:4}, at: device_resume+0xb4/0x254 #5: ffff000006de64e8 (&dev->tx_global_lock){+.-.}-{3:3}, at: netif_tx_lock+0x1c/0x34
stack backtrace: CPU: 1 UID: 0 PID: 254 Comm: rtcwake Not tainted 7.1.0-rc7+ #13003 PREEMPT Hardware name: linux,dummy-virt (DT) Call trace: show_stack+0x18/0x24 (C) dump_stack_lvl+0x90/0xd0 dump_stack+0x18/0x24 print_deadlock_bug+0x260/0x350 __lock_acquire+0x11b8/0x225c lock_acquire+0x1c4/0x3f0 _raw_spin_lock_bh+0x50/0x68 netdev_watchdog_up+0x40/0x108 netif_device_attach+0x9c/0xb0 virtnet_restore+0x100/0x21c virtio_device_restore_priv+0x11c/0x1d0 virtio_device_restore+0x14/0x20 virtio_mmio_restore+0x34/0x40 platform_pm_resume+0x2c/0x68 dpm_run_callback+0xa0/0x240 device_resume+0x120/0x254 dpm_resume+0x1f8/0x2ec dpm_resume_end+0x18/0x34 suspend_devices_and_enter+0x1d0/0x990 pm_suspend+0x1ec/0x608 state_store+0x8c/0x110 kobj_attr_store+0x18/0x2c sysfs_kf_write+0x50/0x7c kernfs_fop_write_iter+0x130/0x1c4 vfs_write+0x2b8/0x35c ksys_write+0x6c/0x104 __arm64_sys_write+0x1c/0x28 invoke_syscall+0x54/0x110 el0_svc_common.constprop.0+0x40/0xe8 do_el0_svc+0x20/0x2c el0_svc+0x54/0x338 el0t_64_sync_handler+0xa0/0xe4 el0t_64_sync+0x198/0x19c
Reverting $subject on top of linux-next fixes this issue.
Quoted 120 lines
---
v3: addeddev->watchdog_lockanddev->watchdog_ref_heldinstead of messing
with ref tracker infra. (after sashiko feedback on v2)
v2: fix compile error whenCONFIG_NET_DEV_REFCNT_TRACKERis not set (Jakub and build bots)include/linux/netdevice.h | 4 ++++ net/core/dev.c | 3 ++- net/sched/sch_generic.c | 44 +++++++++++++++++++++++++++++---------- 3 files changed, 39 insertions(+), 12 deletions(-)include/linux/netdevice.h +4
-0@@ -1980,6 +1980,8 @@ enum netdev_reg_state {* @qdisc_hash: qdisc hash table * @watchdog_timeo: Represents the timeout that is used by * the watchdog (see dev_watchdog()) +* @watchdog_lock: protect watchdog_ref_held +* @watchdog_ref_held: True if the watchdog device ref is taken. * @watchdog_timer: List of timers * * @proto_down_reason: reason a netdev interface is held down @@ -2392,6 +2394,8 @@ struct net_device {/* These may be needed for future network-power-down code. */ struct timer_list watchdog_timer;int watchdog_timeo; +spinlock_t watchdog_lock; +bool watchdog_ref_held;u32 proto_down_reason;net/core/dev.c +2
-1@@ -11217,7 +11217,8 @@ static int netif_alloc_netdev_queues(struct net_device *dev)netdev_for_each_tx_queue (dev, netdev_init_one_queue,NULL );spin_lock_init (&dev->tx_global_lock); - +spin_lock_init (&dev->watchdog_lock); + dev->watchdog_ref_held = false;return 0 ; }net/sched/sch_generic.c +33
-9@@ -568,16 +568,24 @@ static void dev_watchdog(struct timer_list *t) dev->netdev_ops->ndo_tx_timeout (dev, i);netif_unfreeze_queues (dev); } -if (!mod_timer (&dev->watchdog_timer, -round_jiffies (oldest_start + - dev->watchdog_timeo))) - release = false; +spin_lock (&dev->watchdog_lock); +mod_timer (&dev->watchdog_timer, +round_jiffies (oldest_start + + dev->watchdog_timeo)); + release = false; +spin_unlock (&dev->watchdog_lock); } }spin_unlock (&dev->tx_global_lock); -if (release) +spin_lock (&dev->watchdog_lock); +if (timer_pending (&dev->watchdog_timer)) + release = false; +if (release && dev->watchdog_ref_held) {netdev_put (dev, &dev->watchdog_dev_tracker); + dev->watchdog_ref_held = false; + } +spin_unlock (&dev->watchdog_lock); }void netdev_watchdog_up (struct net_device *dev)@@ -586,18 +594,34 @@ void netdev_watchdog_up(struct net_device *dev)return ;if (dev->watchdog_timeo <=0 ) dev->watchdog_timeo =5 *HZ ; +spin_lock_bh (&dev->tx_global_lock); + +spin_lock (&dev->watchdog_lock);if (!mod_timer (&dev->watchdog_timer, -round_jiffies (jiffies + dev->watchdog_timeo))) -netdev_hold (dev, &dev->watchdog_dev_tracker, -GFP_ATOMIC ); +round_jiffies (jiffies + dev->watchdog_timeo))) { +if (!dev->watchdog_ref_held) { +netdev_hold (dev, &dev->watchdog_dev_tracker, +GFP_ATOMIC ); + dev->watchdog_ref_held = true; + } + } +spin_unlock (&dev->watchdog_lock); + +spin_unlock_bh (&dev->tx_global_lock); }EXPORT_SYMBOL_GPL (netdev_watchdog_up);static void netdev_watchdog_down (struct net_device *dev) {netif_tx_lock_bh (dev); -if (timer_delete (&dev->watchdog_timer)) + +spin_lock (&dev->watchdog_lock); +if (timer_delete (&dev->watchdog_timer)) {netdev_put (dev, &dev->watchdog_dev_tracker); + dev->watchdog_ref_held = false; + } +spin_unlock (&dev->watchdog_lock); +netif_tx_unlock_bh (dev); }
net/sched/sch_generic.c void netif_carrier_on(struct net_device *dev) · line 643
Quoted 8 lines
net/sched/sch_generic.c +0
-2@@ -614,8 +638,6 @@ void netif_carrier_on(struct net_device *dev)return ;atomic_inc (&dev->carrier_up_count);linkwatch_fire_event (dev); -if (netif_running (dev)) -netdev_watchdog_up (dev); } }EXPORT_SYMBOL (netif_carrier_on);
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
Quoted 150 lines · Marek Szyprowski
Dear All,
Quoted 57 lines · Eric Dumazet
Blamed commit converted the untracked dev_hold()/dev_put() calls in the watchdog code to use the tracked dev_hold_track()/dev_put_track() (which were later renamed/interfaced to
netdev_hold()andnetdev_put()).By introducing
dev->watchdog_dev_trackerto store the reference tracking information without adding synchronization betweennetdev_watchdog_up()anddev_watchdog(), it enabled the race condition where this pointer could be overwritten or freed concurrently, leading to the list corruption crash syzbot reported:
list_delcorruption,ffff888114a18c00->nextis NULL
kernel BUG atlib/list_debug.c:52!Oops: invalid opcode: 0000 [#1] SMP KASAN PTI CPU: 1 UID: 0 PID: 91 Comm: kworker/u8:5 Not tainted syzkaller #0 PREEMPT(lazy) Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/09/2026 Workqueue: events_unbound linkwatch_event RIP: 0010:__list_del_entry_valid_or_report.cold+0x22/0x2a lib/list_debug.c:52 Call Trace: <TASK> __list_del_entry_valid include/linux/list.h:132 [inline] __list_del_entry include/linux/list.h:246 [inline] list_move_tail include/linux/list.h:341 [inline] ref_tracker_free+0x1a7/0x6c0 lib/ref_tracker.c:329 netdev_tracker_free include/linux/netdevice.h:4491 [inline] netdev_put include/linux/netdevice.h:4508 [inline] netdev_put include/linux/netdevice.h:4504 [inline] netdev_watchdog_down net/sched/sch_generic.c:600 [inline] dev_deactivate_many+0x28c/0xfe0 net/sched/sch_generic.c:1363 dev_deactivate+0x109/0x1d0 net/sched/sch_generic.c:1397 linkwatch_do_dev net/core/link_watch.c:184 [inline] linkwatch_do_dev+0xd3/0x120 net/core/link_watch.c:166 __linkwatch_run_queue+0x3a5/0x810 net/core/link_watch.c:240 linkwatch_event+0x8f/0xc0 net/core/link_watch.c:314 process_one_work+0xa0e/0x1980 kernel/workqueue.c:3314 process_scheduled_works kernel/workqueue.c:3397 [inline] worker_thread+0x5ef/0xe50 kernel/workqueue.c:3478 kthread+0x370/0x450 kernel/kthread.c:436 ret_from_fork+0x69a/0xc80 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245This patch has three coordinated parts:
1) Add
dev->watchdog_lockanddev->watchdog_ref_heldto serialize watchdog operations.2) Remove
netdev_watchdog_up()call fromnetif_carrier_on():
This ensuresnetdev_watchdog_up()is only called from process/BH context (via linkwatch workqueuedev_activate()), allowing us to usespin_lock_bh()for synchronization.3) Synchronize watchdog up and watchdog timer:
Protectnetdev_watchdog_up()withtx_global_lockandwatchdog_lock. Only allocate a new tracker innetdev_watchdog_up()if one is not already present.
Indev_watchdog(), ensure we don't release the tracker if the timer was rescheduled either bydev_watchdog()itself or concurrently bynetdev_watchdog_up().Quoted 5 lines
Fixes f12bf6f3f942 ("net: watchdog: add net device refcount tracker")Reported-by [email protected]Tested-by [email protected]Signed-off-by Eric Dumazet <[email protected]>This patch landed recently in linux-next as commit
8eed5519e496("net: watchdog:
fix refcount tracking races"). In my tests I found that it causes the following deadlock during system suspend/resume on QEmu's ARM64bit 'virt' machine:root@target:~# time rtcwake -s10 -mmem rtcwake: assuming RTC uses UTC ... rtcwake: wakeup from "mem" using /dev/rtc0 at Wed Jun 17 10:46:12 2026 PM: suspend entry (s2idle) Filesystems sync: 0.055 seconds Freezing user space processes Freezing user space processes completed (elapsed 0.006 seconds) OOM killer disabled. Freezing remaining freezable tasks Freezing remaining freezable tasks completed (elapsed 0.003 seconds)============================================
WARNING: possible recursive locking detected
7.1.0-rc7+ #13003 Not tainted
--------------------------------------------
rtcwake/254 is trying to acquire lock:ffff000006de64e8(&dev->tx_global_lock){+.-.}-{3:3}, at:netdev_watchdog_up+0x40/0x108but task is already holding lock:
ffff000006de64e8(&dev->tx_global_lock){+.-.}-{3:3}, at:netif_tx_lock+0x1c/0x34other info that might help us debug this:
Possible unsafe locking scenario:CPU0 ----
lock(&dev->tx_global_lock);
lock(&dev->tx_global_lock);*** DEADLOCK ***
May be due to missing lock nesting notation
6 locks held by rtcwake/254:
#0:ffff0000071ab3e8(sb_writers#5){.+.+}-{0:0}, at:vfs_write+0x1ec/0x35c#1:ffff00000d22c480(&of->mutex#2){+.+.}-{4:4}, at:kernfs_fop_write_iter+0xf0/0x1c4#2:ffff0000049162c8(kn->active#61){.+.+}-{0:0}, at:kernfs_fop_write_iter+0x100/0x1c4#3:ffffaa79533c03b0(system_transition_mutex){+.+.}-{4:4}, at:pm_suspend+0x98/0x608#4:ffff000005e3a138(&dev->mutex){....}-{4:4}, at:device_resume+0xb4/0x254#5:ffff000006de64e8(&dev->tx_global_lock){+.-.}-{3:3}, at:netif_tx_lock+0x1c/0x34stack backtrace: CPU: 1 UID: 0 PID: 254 Comm: rtcwake Not tainted 7.1.0-rc7+ #13003 PREEMPT Hardware name: linux,dummy-virt (DT) Call trace: show_stack+0x18/0x24 (C) dump_stack_lvl+0x90/0xd0 dump_stack+0x18/0x24 print_deadlock_bug+0x260/0x350 __lock_acquire+0x11b8/0x225c lock_acquire+0x1c4/0x3f0 _raw_spin_lock_bh+0x50/0x68 netdev_watchdog_up+0x40/0x108 netif_device_attach+0x9c/0xb0 virtnet_restore+0x100/0x21c virtio_device_restore_priv+0x11c/0x1d0 virtio_device_restore+0x14/0x20 virtio_mmio_restore+0x34/0x40 platform_pm_resume+0x2c/0x68 dpm_run_callback+0xa0/0x240 device_resume+0x120/0x254 dpm_resume+0x1f8/0x2ec dpm_resume_end+0x18/0x34 suspend_devices_and_enter+0x1d0/0x990 pm_suspend+0x1ec/0x608 state_store+0x8c/0x110 kobj_attr_store+0x18/0x2c sysfs_kf_write+0x50/0x7c kernfs_fop_write_iter+0x130/0x1c4 vfs_write+0x2b8/0x35c ksys_write+0x6c/0x104 __arm64_sys_write+0x1c/0x28 invoke_syscall+0x54/0x110 el0_svc_common.constprop.0+0x40/0xe8 do_el0_svc+0x20/0x2c el0_svc+0x54/0x338 el0t_64_sync_handler+0xa0/0xe4 el0t_64_sync+0x198/0x19c
Quoted 1 line
Reverting $subject on top of linux-next fixes this issue.
Thanks for the report Marek!
Acquiring tx_global_lock in netdev_watchdog_up() appears unnecessary anyway because the critical state (timer and refcount tracker) is already protected by dev->watchdog_lock.
Could you try this patch?
net/sched/sch_generic.c +0 -0
100644
net/sched/sch_generic.c +2 -5
@@ -594,9 +594,8 @@ void netdev_watchdog_up(struct net_device *dev)return ;if (dev->watchdog_timeo <=0 ) dev->watchdog_timeo =5 *HZ ; -spin_lock_bh (&dev->tx_global_lock); -spin_lock (&dev->watchdog_lock); +spin_lock_bh (&dev->watchdog_lock);if (!mod_timer (&dev->watchdog_timer,round_jiffies (jiffies + dev->watchdog_timeo))) {if (!dev->watchdog_ref_held) {
@@ -605,9 +604,7 @@ void netdev_watchdog_up(struct net_device *dev) dev->watchdog_ref_held = true; } } -spin_unlock (&dev->watchdog_lock); - -spin_unlock_bh (&dev->tx_global_lock); +spin_unlock_bh (&dev->watchdog_lock); }EXPORT_SYMBOL_GPL (netdev_watchdog_up);
Marek Szyprowski
Tested-byQuoted 149 lines · Eric Dumazet
Quoted 147 lines · Marek Szyprowski
Quoted 57 lines · Eric Dumazet
Blamed commit converted the untracked dev_hold()/dev_put() calls in the watchdog code to use the tracked dev_hold_track()/dev_put_track() (which were later renamed/interfaced to
netdev_hold()andnetdev_put()).By introducing
dev->watchdog_dev_trackerto store the reference tracking information without adding synchronization betweennetdev_watchdog_up()anddev_watchdog(), it enabled the race condition where this pointer could be overwritten or freed concurrently, leading to the list corruption crash syzbot reported:
list_delcorruption,ffff888114a18c00->nextis NULL
kernel BUG atlib/list_debug.c:52!Oops: invalid opcode: 0000 [#1] SMP KASAN PTI CPU: 1 UID: 0 PID: 91 Comm: kworker/u8:5 Not tainted syzkaller #0 PREEMPT(lazy) Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/09/2026 Workqueue: events_unbound linkwatch_event RIP: 0010:__list_del_entry_valid_or_report.cold+0x22/0x2a lib/list_debug.c:52 Call Trace: <TASK> __list_del_entry_valid include/linux/list.h:132 [inline] __list_del_entry include/linux/list.h:246 [inline] list_move_tail include/linux/list.h:341 [inline] ref_tracker_free+0x1a7/0x6c0 lib/ref_tracker.c:329 netdev_tracker_free include/linux/netdevice.h:4491 [inline] netdev_put include/linux/netdevice.h:4508 [inline] netdev_put include/linux/netdevice.h:4504 [inline] netdev_watchdog_down net/sched/sch_generic.c:600 [inline] dev_deactivate_many+0x28c/0xfe0 net/sched/sch_generic.c:1363 dev_deactivate+0x109/0x1d0 net/sched/sch_generic.c:1397 linkwatch_do_dev net/core/link_watch.c:184 [inline] linkwatch_do_dev+0xd3/0x120 net/core/link_watch.c:166 __linkwatch_run_queue+0x3a5/0x810 net/core/link_watch.c:240 linkwatch_event+0x8f/0xc0 net/core/link_watch.c:314 process_one_work+0xa0e/0x1980 kernel/workqueue.c:3314 process_scheduled_works kernel/workqueue.c:3397 [inline] worker_thread+0x5ef/0xe50 kernel/workqueue.c:3478 kthread+0x370/0x450 kernel/kthread.c:436 ret_from_fork+0x69a/0xc80 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245This patch has three coordinated parts:
1) Add
dev->watchdog_lockanddev->watchdog_ref_heldto serialize watchdog operations.2) Remove
netdev_watchdog_up()call fromnetif_carrier_on():
This ensuresnetdev_watchdog_up()is only called from process/BH context (via linkwatch workqueuedev_activate()), allowing us to usespin_lock_bh()for synchronization.3) Synchronize watchdog up and watchdog timer:
Protectnetdev_watchdog_up()withtx_global_lockandwatchdog_lock. Only allocate a new tracker innetdev_watchdog_up()if one is not already present.
Indev_watchdog(), ensure we don't release the tracker if the timer was rescheduled either bydev_watchdog()itself or concurrently bynetdev_watchdog_up().Quoted 5 lines
Fixes f12bf6f3f942 ("net: watchdog: add net device refcount tracker")Reported-by [email protected]Tested-by [email protected]Signed-off-by Eric Dumazet <[email protected]>This patch landed recently in linux-next as commit
8eed5519e496("net: watchdog:
fix refcount tracking races"). In my tests I found that it causes the following deadlock during system suspend/resume on QEmu's ARM64bit 'virt' machine:root@target:~# time rtcwake -s10 -mmem rtcwake: assuming RTC uses UTC ... rtcwake: wakeup from "mem" using /dev/rtc0 at Wed Jun 17 10:46:12 2026 PM: suspend entry (s2idle) Filesystems sync: 0.055 seconds Freezing user space processes Freezing user space processes completed (elapsed 0.006 seconds) OOM killer disabled. Freezing remaining freezable tasks Freezing remaining freezable tasks completed (elapsed 0.003 seconds)============================================
WARNING: possible recursive locking detected
7.1.0-rc7+ #13003 Not tainted
--------------------------------------------
rtcwake/254 is trying to acquire lock:ffff000006de64e8(&dev->tx_global_lock){+.-.}-{3:3}, at:netdev_watchdog_up+0x40/0x108but task is already holding lock:
ffff000006de64e8(&dev->tx_global_lock){+.-.}-{3:3}, at:netif_tx_lock+0x1c/0x34other info that might help us debug this:
Possible unsafe locking scenario:CPU0 ----
lock(&dev->tx_global_lock);
lock(&dev->tx_global_lock);*** DEADLOCK ***
May be due to missing lock nesting notation
6 locks held by rtcwake/254:
#0:ffff0000071ab3e8(sb_writers#5){.+.+}-{0:0}, at:vfs_write+0x1ec/0x35c#1:ffff00000d22c480(&of->mutex#2){+.+.}-{4:4}, at:kernfs_fop_write_iter+0xf0/0x1c4#2:ffff0000049162c8(kn->active#61){.+.+}-{0:0}, at:kernfs_fop_write_iter+0x100/0x1c4#3:ffffaa79533c03b0(system_transition_mutex){+.+.}-{4:4}, at:pm_suspend+0x98/0x608#4:ffff000005e3a138(&dev->mutex){....}-{4:4}, at:device_resume+0xb4/0x254#5:ffff000006de64e8(&dev->tx_global_lock){+.-.}-{3:3}, at:netif_tx_lock+0x1c/0x34stack backtrace: CPU: 1 UID: 0 PID: 254 Comm: rtcwake Not tainted 7.1.0-rc7+ #13003 PREEMPT Hardware name: linux,dummy-virt (DT) Call trace: show_stack+0x18/0x24 (C) dump_stack_lvl+0x90/0xd0 dump_stack+0x18/0x24 print_deadlock_bug+0x260/0x350 __lock_acquire+0x11b8/0x225c lock_acquire+0x1c4/0x3f0 _raw_spin_lock_bh+0x50/0x68 netdev_watchdog_up+0x40/0x108 netif_device_attach+0x9c/0xb0 virtnet_restore+0x100/0x21c virtio_device_restore_priv+0x11c/0x1d0 virtio_device_restore+0x14/0x20 virtio_mmio_restore+0x34/0x40 platform_pm_resume+0x2c/0x68 dpm_run_callback+0xa0/0x240 device_resume+0x120/0x254 dpm_resume+0x1f8/0x2ec dpm_resume_end+0x18/0x34 suspend_devices_and_enter+0x1d0/0x990 pm_suspend+0x1ec/0x608 state_store+0x8c/0x110 kobj_attr_store+0x18/0x2c sysfs_kf_write+0x50/0x7c kernfs_fop_write_iter+0x130/0x1c4 vfs_write+0x2b8/0x35c ksys_write+0x6c/0x104 __arm64_sys_write+0x1c/0x28 invoke_syscall+0x54/0x110 el0_svc_common.constprop.0+0x40/0xe8 do_el0_svc+0x20/0x2c el0_svc+0x54/0x338 el0t_64_sync_handler+0xa0/0xe4 el0t_64_sync+0x198/0x19c
Quoted 8 lines
Quoted 1 line
Reverting $subject on top of linux-next fixes this issue.
Thanks for the report Marek!
Acquiring
tx_global_lockinnetdev_watchdog_up()appears unnecessary anyway because the critical state (timer and refcount tracker) is already protected bydev->watchdog_lock.Could you try this patch?
This fixes the observed issue. Thanks! Feel free to add:
Quoted 17 lines
net/sched/sch_generic.c +0
-0100644
net/sched/sch_generic.c +1
-2@@ -594,9 +594,8 @@ void netdev_watchdog_up(struct net_device *dev)return ;if (dev->watchdog_timeo <=0 ) dev->watchdog_timeo =5 *HZ ; -spin_lock_bh (&dev->tx_global_lock); -spin_lock (&dev->watchdog_lock); +spin_lock_bh (&dev->watchdog_lock);if (!mod_timer (&dev->watchdog_timer,round_jiffies (jiffies + dev->watchdog_timeo))) {if (!dev->watchdog_ref_held) {
net/sched/sch_generic.c void netdev_watchdog_up(struct net_device *dev) · line 610
Quoted 10 lines
net/sched/sch_generic.c +1
-3@@ -605,9 +604,7 @@ void netdev_watchdog_up(struct net_device *dev) dev->watchdog_ref_held = true; } } -spin_unlock (&dev->watchdog_lock); - -spin_unlock_bh (&dev->tx_global_lock); +spin_unlock_bh (&dev->watchdog_lock); }EXPORT_SYMBOL_GPL (netdev_watchdog_up);
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland