"Dead loop on virtual device" error without softirq-BKL on PREEMPT_RT
Subsystems (3)
Starting with linux-6.18, I see the following log messages
2026-02-15T23:50:17.558716+01:00 [ T1559] Dead loop on virtual device wlp4s0, fix it urgently! 2026-02-15T23:50:17.558737+01:00 [ T1559] Dead loop on virtual device wlp4s0, fix it urgently!
[...]
regarding my wireless network device
04:00.0 Network controller [0280]: MEDIATEK Corp. MT7921K (RZ608) Wi-Fi 6E 80MHz [14c3:0608]
I bisected this (from v6.17 to v6.18) and got this as the first bad commit:3253cb49cbad ("softirq: Allow to drop the softirq-BKL lock on PREEMPT_RT")
Using CONFIG_PREEMPT_RT_NEEDS_BH_LOCK=y in v6.18.10 fixes the issue.
Bert Karwatzki
Am Montag, dem 16.02.2026 um 14:43 +0100 schrieb Bert Karwatzki:
Quoted 8 lines
Starting with linux-6.18, I see the following log messages
2026-02-15T23:50:17.558716+01:00 [ T1559] Dead loop on virtual device wlp4s0, fix it urgently! 2026-02-15T23:50:17.558737+01:00 [ T1559] Dead loop on virtual device wlp4s0, fix it urgently![...]
regarding my wireless network device
Quoted 8 lines
04:00.0 Network controller [0280]: MEDIATEK Corp. MT7921K (RZ608) Wi-Fi 6E 80MHz [14c3:0608]
I bisected this (from v6.17 to v6.18) and got this as the first bad commit:
3253cb49cbad("softirq: Allow to drop the softirq-BKL lock onPREEMPT_RT")Using
CONFIG_PREEMPT_RT_NEEDS_BH_LOCK=yin v6.18.10 fixes the issue.Bert Karwatzki
May I presume that to fix this properly at least some of the spin_lock_bh()s in net/mac80211/ need to be converted to local_lock_nested_bh()s?
Bert Karwatzki
Sebastian Andrzej Siewior
maintainerQuoted 14 lines · Bert Karwatzki
Am Montag, dem 16.02.2026 um 14:43 +0100 schrieb Bert Karwatzki:
Quoted 13 lines
Starting with linux-6.18, I see the following log messages
2026-02-15T23:50:17.558716+01:00 [ T1559] Dead loop on virtual device wlp4s0, fix it urgently! 2026-02-15T23:50:17.558737+01:00 [ T1559] Dead loop on virtual device wlp4s0, fix it urgently![...]
regarding my wireless network device
04:00.0 Network controller [0280]: MEDIATEK Corp. MT7921K (RZ608) Wi-Fi 6E 80MHz [14c3:0608]
I bisected this (from v6.17 to v6.18) and got this as the first bad commit:
3253cb49cbad("softirq: Allow to drop the softirq-BKL lock onPREEMPT_RT")
Quoted 6 lines
Quoted 3 lines
Using
CONFIG_PREEMPT_RT_NEEDS_BH_LOCK=yin v6.18.10 fixes the issue.Bert Karwatzki
May I presume that to fix this properly at least some of the spin_lock_bh()s in
net/mac80211/need to be converted to local_lock_nested_bh()s?
I am not sure what issue is so I can't tell. The dev_xmit_recursion*() based counters are per-task so it should be fine. But yet the wifi managed to repeatedly enqueue packets. This might be a real recursion, a stack trace should tell. And then, somewhere synchronisation is missing.
Quoted 1 line
Bert Karwatzki
Sebastian
Am Montag, dem 16.02.2026 um 16:37 +0100 schrieb Sebastian Andrzej Siewior:
Quoted 9 lines
I am not sure what issue is so I can't tell. The dev_xmit_recursion*() based counters are per-task so it should be fine. But yet the wifi managed to repeatedly enqueue packets. This might be a real recursion, a stack trace should tell. And then, somewhere synchronisation is missing.
Quoted 1 line
Bert Karwatzki
Sebastian
The problem seems to be that different preemtible threads try to send skbs.
I used this debug patch for 6.18.10:
net/core/dev.c +10 -1
@@ -4704,6 +4704,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)qdisc_pkt_len_init (skb);tcx_set_ingress (skb, false); +printk (KERN_INFO "%s 0: skb = %px dev = %s\n" , __func__, skb, dev->name);#ifdef CONFIG_NET_EGRESS if (static_branch_unlikely (&egress_needed_key)) {if (nf_hook_egress_active ()) {
@@ -4739,10 +4740,12 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)trace_net_dev_queue (skb);if (q->enqueue) { +printk (KERN_INFO "%s 1: skb = %px dev = %s\n" , __func__, skb, dev->name); rc =__dev_xmit_skb (skb, q, dev, txq); goto out; } +printk (KERN_INFO "%s 2: skb = %px dev = %s txq = %px\n" , __func__, skb, dev->name, txq);/* The device has no queue. Common case for software devices: * loopback, all the sorts of tunnels...
@@ -4761,15 +4764,20 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)/* Other cpus might concurrently change txq->xmit_lock_owner * to -1 or to their cpu id, but not to our id. */ +printk (KERN_INFO "%s: cpu = %d xmit_lock_owner = %d\n" , __func__, cpu,READ_ONCE (txq->xmit_lock_owner));if (READ_ONCE (txq->xmit_lock_owner) != cpu) { -if (dev_xmit_recursion ()) +printk (KERN_INFO "%s 3: skb = %px dev = %s txq = %px\n" , __func__, skb, dev->name, txq); +if (dev_xmit_recursion ()) { +printk (KERN_INFO "%s: recursion alert for device %s!\n" , __func__, dev->name); goto recursion_alert; + } skb =validate_xmit_skb (skb, dev, &again);if (!skb) goto out;HARD_TX_LOCK (dev, txq, cpu); +printk (KERN_INFO "%s 4: skb = %px dev = %s txq = %px\n" , __func__, skb, dev->name, txq);if (!netif_xmit_stopped (txq)) {dev_xmit_recursion_inc ();
@@ -4777,6 +4785,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)dev_xmit_recursion_dec ();if (dev_xmit_complete (rc)) {HARD_TX_UNLOCK (dev, txq); +printk (KERN_INFO "%s 5: skb = %px dev = %s txq = %px\n" , __func__, skb, dev->name, txq); goto out; } }
The normal path of an skb is this:
2026-02-17T00:29:24.124757+01:00 [ T1522] __dev_queue_xmit 0: skb = ffff8c11ed06bd00 dev = wlp4s0 2026-02-17T00:29:24.124845+01:00 [ T1522] __dev_queue_xmit 2: skb = ffff8c11ed06bd00 dev = wlp4s0 txq = ffff8c1145320200 2026-02-17T00:29:24.124851+01:00 [ T1522] __dev_queue_xmit: cpu = 7 xmit_lock_owner = -1 2026-02-17T00:29:24.124853+01:00 [ T1522] __dev_queue_xmit 3: skb = ffff8c11ed06bd00 dev = wlp4s0 txq = ffff8c1145320200 2026-02-17T00:29:24.124855+01:00 [ T1522] __dev_queue_xmit 4: skb = ffff8c11ed06bd00 dev = wlp4s0 txq = ffff8c1145320200 2026-02-17T00:29:24.124857+01:00 [ T1522] __dev_queue_xmit 5: skb = 0000000000000000 dev = wlp4s0 txq = ffff8c1145320200
This is the situation which produces the error messages:
T1522 tries to send an skb on CPU 7:
2026-02-17T00:29:24.212215+01:00 [ T1522] __dev_queue_xmit 0: skb = ffff8c11ed06b100 dev = wlp4s0 2026-02-17T00:29:24.212217+01:00 [ T1522] __dev_queue_xmit 2: skb = ffff8c11ed06b100 dev = wlp4s0 txq = ffff8c1145320200 2026-02-17T00:29:24.212219+01:00 [ T1522] __dev_queue_xmit: cpu = 7 xmit_lock_owner = -1 2026-02-17T00:29:24.212221+01:00 [ T1522] __dev_queue_xmit 3: skb = ffff8c11ed06b100 dev = wlp4s0 txq = ffff8c1145320200 2026-02-17T00:29:24.212223+01:00 [ T1522] __dev_queue_xmit 4: skb = ffff8c11ed06b100 dev = wlp4s0 txq = ffff8c1145320200 Here T1522 gets preempted and T1513 is executed on CPU 7 and also tries to send an skb: 2026-02-17T00:29:24.212225+01:00 [ T1513] __dev_queue_xmit 0: skb = ffff8c11ed06a300 dev = wlp4s0 2026-02-17T00:29:24.212228+01:00 [ T1513] __dev_queue_xmit 2: skb = ffff8c11ed06a300 dev = wlp4s0 txq = ffff8c1145320200 2026-02-17T00:29:24.212230+01:00 [ T1513] __dev_queue_xmit: cpu = 7 xmit_lock_owner = 7 2026-02-17T00:29:24.212231+01:00 [ T1513] Dead loop on virtual device wlp4s0, fix it urgently! 2026-02-17T00:29:24.212234+01:00 [ T1513] __dev_queue_xmit 0: skb = ffff8c11ed06a300 dev = wlp4s0 2026-02-17T00:29:24.212236+01:00 [ T1513] __dev_queue_xmit 2: skb = ffff8c11ed06a300 dev = wlp4s0 txq = ffff8c1145320200 2026-02-17T00:29:24.212238+01:00 [ T1513] __dev_queue_xmit: cpu = 7 xmit_lock_owner = 7 2026-02-17T00:29:24.212240+01:00 [ T1513] Dead loop on virtual device wlp4s0, fix it urgently! 2026-02-17T00:29:24.212242+01:00 [ T1513] __dev_queue_xmit 0: skb = ffff8c11ed06a300 dev = wlp4s0 2026-02-17T00:29:24.212244+01:00 [ T1513] __dev_queue_xmit 2: skb = ffff8c11ed06a300 dev = wlp4s0 txq = ffff8c1145320200 2026-02-17T00:29:24.212246+01:00 [ T1513] __dev_queue_xmit: cpu = 7 xmit_lock_owner = 7 2026-02-17T00:29:24.212247+01:00 [ T1513] Dead loop on virtual device wlp4s0, fix it urgently! T1513 gets preempted and T1522 finishes processing the skb from above: 2026-02-17T00:29:24.212249+01:00 [ T1522] __dev_queue_xmit 5: skb = 0000000000000000 dev = wlp4s0 txq = ffff8c1145320200
Bert Karwatzki
Sebastian Andrzej Siewior
maintainerQuoted 1 line · Bert Karwatzki
The problem seems to be that different preemtible threads try to send skbs.
This does not matter because the counter is per-thread not per-CPU.
Quoted 1 line
2026-02-17T00:29:24.212231+01:00 [ T1513] Dead loop on virtual device wlp4s0, fix it urgently!
Could you please do a backtrace here, for instance via WARN_ON_ONCE()
Quoted 1 line
Bert Karwatzki
Sebastian
Am Dienstag, dem 17.02.2026 um 08:19 +0100 schrieb Sebastian Andrzej Siewior:
Quoted 4 lines · Bert Karwatzki
Quoted 1 line
The problem seems to be that different preemtible threads try to send skbs.
This does not matter because the counter is per-thread not per-CPU.
The "Dead loop on virtual device" messages is not printed because dev_xmit_recursion() returns true, but because READ_ONCE(txq->xmit_lock_owner) == cpu.
Quoted 8 lines
Quoted 2 lines
2026-02-17T00:29:24.212231+01:00 [ T1513] Dead loop on virtual device wlp4s0, fix it urgently!Could you please do a backtrace here, for instance via
WARN_ON_ONCE()Quoted 1 line
Bert Karwatzki
Sebastian
Here's the backtrace:
2026-02-17T09:48:49.553225+01:00 lisa kernel: [ T1521] __dev_queue_xmit 0: skb = ffff9ee68afd3a00 dev = wlp4s0 2026-02-17T09:48:49.553227+01:00 lisa kernel: [ T1521] __dev_queue_xmit 2: skb = ffff9ee68afd3a00 dev = wlp4s0 txq = ffff9ee6820b5a00 2026-02-17T09:48:49.553229+01:00 lisa kernel: [ T1521] __dev_queue_xmit: cpu = 7 xmit_lock_owner = -1 2026-02-17T09:48:49.553232+01:00 lisa kernel: [ T1521] __dev_queue_xmit 3: skb = ffff9ee68afd3a00 dev = wlp4s0 txq = ffff9ee6820b5a00 2026-02-17T09:48:49.553234+01:00 lisa kernel: [ T1521] __dev_queue_xmit 4: skb = ffff9ee68afd3a00 dev = wlp4s0 txq = ffff9ee6820b5a00
2026-02-17T09:48:49.553235+01:00 lisa kernel: [ T1538] __dev_queue_xmit 0: skb = ffff9ee68afd3b00 dev = wlp4s0 2026-02-17T09:48:49.553238+01:00 lisa kernel: [ T1538] __dev_queue_xmit 2: skb = ffff9ee68afd3b00 dev = wlp4s0 txq = ffff9ee6820b5a00 2026-02-17T09:48:49.553241+01:00 lisa kernel: [ T1538] __dev_queue_xmit: cpu = 7 xmit_lock_owner = 7 2026-02-17T09:48:49.553243+01:00 lisa kernel: [ T1538] ------------[ cut here ]------------ 2026-02-17T09:48:49.553245+01:00 lisa kernel: [ T1538] WARNING: CPU: 7 PID: 1538 at net/core/dev.c:4800 __dev_queue_xmit.cold+0x163/0x5bd 2026-02-17T09:48:49.553248+01:00 lisa kernel: [ T1538] Modules linked in: ccm snd_seq_dummy snd_hrtimer snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq
snd_seq_device rfcomm bnep nls_ascii nls_cp437 vfat fat snd_hda_codec_generic snd_hda_codec_atihdmi snd_hda_codec_hdmi snd_hda_intel btusb snd_intel_dspcfg btrtl uvcvideo btintel snd_hda_codec btbcm videobuf2_vmalloc snd_acp3x_pdm_dma snd_soc_dmic snd_acp3x_rn btmtk videobuf2_memops snd_soc_core snd_hda_core uvc videobuf2_v4l2 bluetooth snd_hwdep videodev snd_pcm_oss snd_mixer_oss snd_pcm snd_rn_pci_acp3x snd_acp_config videobuf2_common msi_wmi snd_soc_acpi ecdh_generic ecc mc sparse_keymap wmi_bmof snd_timer snd k10temp ccp soundcore snd_pci_acp3x battery ac button joydev evdev amd_pmc mt7921e mt7921_common mt792x_lib mt76_connac_lib mt76 mac80211 libarc4 cfg80211 rfkill msr fuse nvme_fabrics efi_pstore configfs efivarfs autofs4 ext4 mbcache jbd2 usbhid amdgpu drm_client_lib i2c_algo_bit drm_ttm_helper ttm drm_exec drm_suballoc_helper drm_buddy drm_panel_backlight_quirks gpu_sched xhci_pci amdxcp xhci_hcd
2026-02-17T09:48:49.553251+01:00 lisa kernel: [ T1538] drm_display_helper hid_sensor_hub hid_multitouch mfd_core hid_generic i2c_hid_acpi psmouse
drm_kms_helper usbcore nvme amd_sfh i2c_hid hid cec nvme_core serio_raw i2c_piix4 r8169 crc16 usb_common i2c_smbus i2c_designware_platform i2c_designware_core
2026-02-17T09:48:49.553254+01:00 lisa kernel: [ T1538] CPU: 7 UID: 122 PID: 1538 Comm: isc-loop-0014 Not tainted 6.18.10-deadloop-00005-g56089d3b695a #1153
PREEMPT_{RT,(full)}
2026-02-17T09:48:49.553256+01:00 lisa kernel: [ T1538] Hardware name: Micro-Star International Co., Ltd. Alpha 15 B5EEK/MS-158L, BIOS E158LAMS.10F 11/11/2024
2026-02-17T09:48:49.553258+01:00 lisa kernel: [ T1538] RIP: 0010:__dev_queue_xmit.cold+0x163/0x5bd
2026-02-17T09:48:49.553261+01:00 lisa kernel: [ T1538] Code: 3d 3e 88 5a 01 66 41 83 bf ca 04 00 00 08 0f 86 4d 02 00 00 4c 89 f2 48 c7 c6 80 b3 0c b9 48 c7 c7
b0 b3 2f b9 e8 49 43 fd ff <0f> 0b e8 72 8f 75 00 85 c0 74 0f 4c 89 f6 48 c7 c7 58 b4 2f b9 e8
2026-02-17T09:48:49.553262+01:00 lisa kernel: [ T1538] RSP: 0018:ffffbd6506d53990 EFLAGS: 00010246
2026-02-17T09:48:49.553264+01:00 lisa kernel: [ T1538] RAX: 0000000000000007 RBX: ffff9ee6a089c000 RCX: 0000000000000027
2026-02-17T09:48:49.553265+01:00 lisa kernel: [ T1538] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9ee95e7d6d80
2026-02-17T09:48:49.553266+01:00 lisa kernel: [ T1538] RBP: ffff9ee68afd3b00 R08: 0000000000000000 R09: ffffffffb9685590
2026-02-17T09:48:49.553268+01:00 lisa kernel: [ T1538] R10: ffffffffb96a4210 R11: 0000000000000003 R12: ffff9ee6820b5a00
2026-02-17T09:48:49.553270+01:00 lisa kernel: [ T1538] R13: 0000000000000007 R14: ffff9ee6a089c118 R15: ffff9ee6b455f000
2026-02-17T09:48:49.553273+01:00 lisa kernel: [ T1538] FS: 00007f3ad75ff680(0000) GS:ffff9ee9a4bf3000(0000) knlGS:0000000000000000
2026-02-17T09:48:49.553275+01:00 lisa kernel: [ T1538] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2026-02-17T09:48:49.553277+01:00 lisa kernel: [ T1538] CR2: 00007f3d54a24080 CR3: 0000000102f73000 CR4: 0000000000750ef0
2026-02-17T09:48:49.553279+01:00 lisa kernel: [ T1538] PKRU: 55555554
2026-02-17T09:48:49.553282+01:00 lisa kernel: [ T1538] Call Trace:
2026-02-17T09:48:49.553284+01:00 lisa kernel: [ T1538] <TASK>
2026-02-17T09:48:49.553285+01:00 lisa kernel: [ T1538] ? srso_alias_return_thunk+0x5/0xfbef5
2026-02-17T09:48:49.553287+01:00 lisa kernel: [ T1538] ? __ip_make_skb+0x325/0x560
2026-02-17T09:48:49.553289+01:00 lisa kernel: [ T1538] ip_finish_output2+0x2c8/0x600
2026-02-17T09:48:49.553292+01:00 lisa kernel: [ T1538] ip_local_out+0xc6/0xf0
2026-02-17T09:48:49.553294+01:00 lisa kernel: [ T1538] ip_send_skb+0x14/0x50
2026-02-17T09:48:49.553296+01:00 lisa kernel: [ T1538] udp_send_skb+0x181/0x370
2026-02-17T09:48:49.553299+01:00 lisa kernel: [ T1538] udp_sendmsg+0x8e8/0xbc0
2026-02-17T09:48:49.553301+01:00 lisa kernel: [ T1538] ? ip_frag_init+0x60/0x60
2026-02-17T09:48:49.553302+01:00 lisa kernel: [ T1538] ? srso_alias_return_thunk+0x5/0xfbef5
2026-02-17T09:48:49.553304+01:00 lisa kernel: [ T1538] ? aa_sk_perm+0x8e/0x210
2026-02-17T09:48:49.553306+01:00 lisa kernel: [ T1538] __sock_sendmsg+0x60/0x80
2026-02-17T09:48:49.553308+01:00 lisa kernel: [ T1538] ____sys_sendmsg+0x21d/0x2b0
2026-02-17T09:48:49.553310+01:00 lisa kernel: [ T1538] ? import_iovec+0x1b/0x30
2026-02-17T09:48:49.553312+01:00 lisa kernel: [ T1538] ? srso_alias_return_thunk+0x5/0xfbef5
2026-02-17T09:48:49.553315+01:00 lisa kernel: [ T1538] ? copy_msghdr_from_user+0xe5/0x170
2026-02-17T09:48:49.553316+01:00 lisa kernel: [ T1538] ___sys_sendmsg+0x7e/0xc0
2026-02-17T09:48:49.553318+01:00 lisa kernel: [ T1538] ? srso_alias_return_thunk+0x5/0xfbef5
2026-02-17T09:48:49.553318+01:00 lisa kernel: [ T1538] ? rt_spin_lock+0x38/0x110
2026-02-17T09:48:49.553321+01:00 lisa kernel: [ T1538] ? srso_alias_return_thunk+0x5/0xfbef5
2026-02-17T09:48:49.553323+01:00 lisa kernel: [ T1538] ? rt_spin_lock+0x38/0x110
2026-02-17T09:48:49.553325+01:00 lisa kernel: [ T1538] ? srso_alias_return_thunk+0x5/0xfbef5
2026-02-17T09:48:49.553327+01:00 lisa kernel: [ T1538] ? ipv4_dst_check+0x36/0x60
2026-02-17T09:48:49.553330+01:00 lisa kernel: [ T1538] ? srso_alias_return_thunk+0x5/0xfbef5
2026-02-17T09:48:49.553332+01:00 lisa kernel: [ T1538] ? ip4_datagram_release_cb+0x45/0x1d0
2026-02-17T09:48:49.553334+01:00 lisa kernel: [ T1538] ? rt_spin_lock+0x38/0x110
2026-02-17T09:48:49.553336+01:00 lisa kernel: [ T1538] ? get_random_u16+0xc6/0x1c0
2026-02-17T09:48:49.553338+01:00 lisa kernel: [ T1538] ? srso_alias_return_thunk+0x5/0xfbef5
2026-02-17T09:48:49.553340+01:00 lisa kernel: [ T1538] ? rt_spin_lock+0x38/0x110
2026-02-17T09:48:49.553342+01:00 lisa kernel: [ T1538] ? srso_alias_return_thunk+0x5/0xfbef5
2026-02-17T09:48:49.553344+01:00 lisa kernel: [ T1538] ? rt_spin_unlock+0x5a/0xa0
2026-02-17T09:48:49.553346+01:00 lisa kernel: [ T1538] ? srso_alias_return_thunk+0x5/0xfbef5
2026-02-17T09:48:49.553347+01:00 lisa kernel: [ T1538] ? rt_spin_unlock+0x5a/0xa0
2026-02-17T09:48:49.553349+01:00 lisa kernel: [ T1538] ? srso_alias_return_thunk+0x5/0xfbef5
2026-02-17T09:48:49.553351+01:00 lisa kernel: [ T1538] ? __local_bh_enable_ip+0x73/0xa0
2026-02-17T09:48:49.553353+01:00 lisa kernel: [ T1538] ? srso_alias_return_thunk+0x5/0xfbef5
2026-02-17T09:48:49.553355+01:00 lisa kernel: [ T1538] __sys_sendmsg+0x68/0xc0
2026-02-17T09:48:49.553357+01:00 lisa kernel: [ T1538] do_syscall_64+0x65/0x2f0
2026-02-17T09:48:49.553359+01:00 lisa kernel: [ T1538] entry_SYSCALL_64_after_hwframe+0x55/0x5d
2026-02-17T09:48:49.553361+01:00 lisa kernel: [ T1538] RIP: 0033:0x7f3ae6d239ee
2026-02-17T09:48:49.553363+01:00 lisa kernel: [ T1538] Code: 08 0f 85 f5 4b ff ff 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c
24 10 4c 89 5c 24 08 0f 05 <c3> 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 80 00 00 00 00 48 83 ec 08
2026-02-17T09:48:49.553365+01:00 lisa kernel: [ T1538] RSP: 002b:00007f3ad75fd0a8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
2026-02-17T09:48:49.553367+01:00 lisa kernel: [ T1538] RAX: ffffffffffffffda RBX: 00007f3ad75ff680 RCX: 00007f3ae6d239ee
2026-02-17T09:48:49.553367+01:00 lisa kernel: [ T1538] RDX: 0000000000000000 RSI: 00007f3ad75fd110 RDI: 0000000000000105
2026-02-17T09:48:49.553369+01:00 lisa kernel: [ T1538] RBP: 00007f3ad75fd110 R08: 0000000000000000 R09: 0000000000000000
2026-02-17T09:48:49.553371+01:00 lisa kernel: [ T1538] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
2026-02-17T09:48:49.553373+01:00 lisa kernel: [ T1538] R13: 00007f3ad75fd750 R14: 00007f3ad644c8d0 R15: 00007f3ad75fd750
2026-02-17T09:48:49.553375+01:00 lisa kernel: [ T1538] </TASK>
2026-02-17T09:48:49.553377+01:00 lisa kernel: [ T1538] ---[ end trace 0000000000000000 ]---
2026-02-17T09:48:49.553379+01:00 lisa kernel: [ T1538] Dead loop on virtual device wlp4s0, fix it urgently!
2026-02-17T09:48:49.553380+01:00 lisa kernel: [ T1538] __dev_queue_xmit 0: skb = ffff9ee68afd3b00 dev = wlp4s0
2026-02-17T09:48:49.553381+01:00 lisa kernel: [ T1538] __dev_queue_xmit 2: skb = ffff9ee68afd3b00 dev = wlp4s0 txq = ffff9ee6820b5a00
2026-02-17T09:48:49.553383+01:00 lisa kernel: [ T1538] __dev_queue_xmit: cpu = 7 xmit_lock_owner = 7
2026-02-17T09:48:49.553385+01:00 lisa kernel: [ T1538] Dead loop on virtual device wlp4s0, fix it urgently!2026-02-17T09:48:49.553387+01:00 lisa kernel: [ T1521] __dev_queue_xmit 5: skb = 0000000000000000 dev = wlp4s0 txq = ffff9ee6820b5a00
Bert Karwatzki
Sebastian Andrzej Siewior
maintainerQuoted 8 lines · Bert Karwatzki
Am Dienstag, dem 17.02.2026 um 08:19 +0100 schrieb Sebastian Andrzej Siewior:
Quoted 4 lines · Bert Karwatzki
Quoted 1 line
The problem seems to be that different preemtible threads try to send skbs.
This does not matter because the counter is per-thread not per-CPU.
The "Dead loop on virtual device" messages is not printed because
dev_xmit_recursion()returns true, but becauseREAD_ONCE(txq->xmit_lock_owner)== cpu.
Ach, so it is not the recursion, it is the assigned CPU. This is assigned via __netif_tx_lock(). Here we somehow lack the expected synchronisation. So the queue should be locked but not by the caller.
Quoted 1 line
Here's the backtrace:
thanks.
Quoted 1 line
Bert Karwatzki
Sebastian
Am Dienstag, dem 17.02.2026 um 10:57 +0100 schrieb Sebastian Andrzej Siewior:
Quoted 7 lines · Bert Karwatzki
Quoted 6 lines
Am Dienstag, dem 17.02.2026 um 08:19 +0100 schrieb Sebastian Andrzej Siewior:
Quoted 4 lines · Bert Karwatzki
Quoted 1 line
The problem seems to be that different preemtible threads try to send skbs.
This does not matter because the counter is per-thread not per-CPU.
Quoted 7 lines
Quoted 2 lines
The "Dead loop on virtual device" messages is not printed because
dev_xmit_recursion()returns true, but becauseREAD_ONCE(txq->xmit_lock_owner)== cpu.Ach, so it is not the recursion, it is the assigned CPU. This is assigned via
__netif_tx_lock(). Here we somehow lack the expected synchronisation. So the queue should be locked but not by the caller.
Yes, the queue gets locked by the first thread (via HARD_TX_LOCK), then the thread gets preempted before the processing of the skb is complete, then the next thread on the same CPU calls __dev_queue_xmit() and find that the lockowner has the same CPU id.
I just wondered if we can completely skip the
if (READ_ONCE(txq->xmit_lock_owner) != cpu) {
[...]
} else
{
/* "Recursion" alert */
}check, as the synchronization will we provided by HARD_TX_{LOCK,UNLOCK}.
The comment
/* Other cpus might concurrently change txq->xmit_lock_owner * to -1 or to their cpu id, but not to our id. */
suggests that the case that a thread is preempted while holding the lock was not taken into account here. And in non-RT cases this would be correct as spin_lock() disables preemption in that case.
Bert Karwatzki
Am Dienstag, dem 17.02.2026 um 11:42 +0100 schrieb Bert Karwatzki:
Quoted 12 lines
I just wondered if we can completely skip the
if (READ_ONCE(txq->xmit_lock_owner) != cpu) { [...] } else { /* "Recursion" alert */ }check, as the synchronization will we provided by HARD_TX_{LOCK,UNLOCK}.
I thought about that again, and it seems like a bad idea as (in the non-preempt) case other threads trying to access the queue would wait for the spinlock to be freed, perhaps one can just change the code like this:
commit 05026868843a4eea51d45811d87706f36896e828
Author: Bert Karwatzki <[email protected]> Date: Tue Feb 17 12:08:35 2026 +0100
net: core: dev: don't warn about recursion when on same CPU
This prints a message if we're on the same CPU and the lock is already taken, in a production use we would of course skip this message.
net/core/dev.c +9 -6
@@ -4784,15 +4784,18 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)net_crit_ratelimited ("Virtual device %s asks to queue packet!\n" , dev->name); }else { -/* Recursion is detected! It is possible, -* unfortunately -*/ -recursion_alert: -net_crit_ratelimited ("Dead loop on virtual device %s, fix it urgently!\n" , - dev->name); +net_crit_ratelimited ("Lock taken already on %s!\n" , dev->name); + goto lock_taken; } } +/* Recursion is detected! It is possible, +* unfortunately +*/ +recursion_alert: +net_crit_ratelimited ("Dead loop on virtual device %s, fix it urgently!\n" , + dev->name); +lock_taken: rc = -ENETDOWN ;rcu_read_unlock_bh ();
With this I get these messages (be skipped in production code) instead of the "Dead loop on virtual device":
[ 51.994435] [ T1525] Lock taken already on wlp4s0! [ 51.994546] [ T1525] Lock taken already on wlp4s0! [ 51.994650] [ T1525] Lock taken already on wlp4s0! [ 51.994746] [ T1525] Lock taken already on wlp4s0! [ 51.994845] [ T1525] Lock taken already on wlp4s0! [ 51.994948] [ T1525] Lock taken already on wlp4s0! [ 51.995037] [ T1525] Lock taken already on wlp4s0! [ 51.995128] [ T1525] Lock taken already on wlp4s0! [ 51.995220] [ T1525] Lock taken already on wlp4s0!
Bert Karwatzki
Am Dienstag, dem 17.02.2026 um 12:24 +0100 schrieb Bert Karwatzki:
Quoted 11 lines
Am Dienstag, dem 17.02.2026 um 11:42 +0100 schrieb Bert Karwatzki:
Quoted 10 lines
I just wondered if we can completely skip the
if (READ_ONCE(txq->xmit_lock_owner) != cpu) { [...] } else { /* "Recursion" alert */ }
Quoted 6 lines
Quoted 2 lines
check, as the synchronization will we provided by HARD_TX_{LOCK,UNLOCK}.
I thought about that again, and it seems like a bad idea as (in the non-preempt) case other threads trying to access the queue would wait for the spinlock to be freed, perhaps one can just change the code like this:
My argument above seems wrong: In the non-preempt case we cannot have another thread accessing the txq from the same CPU if the lock is taken (it is not preemptible) and for other threads accessing the txq from different CPUs the check (txq->xmit_lock_owner != cpu) would succeed and they would try the spinlock anyway, So this would not speak against killing the lock owner check. As for the recursion detection, perhaps dev_xmit_recursion() is enough?
Another Idea (more vague ...):
Using the CPU Id as the lock_owner seems to make sense for locks that are not
preemptible (raw_spinlock in the RT case). For preemptible locks a thread ID (which one exactly I'm not sure ...) would perhaps make a better lock owner ...
Bert Karwatzki
Am Dienstag, dem 17.02.2026 um 17:52 +0100 schrieb Bert Karwatzki:
Quoted 19 lines
Am Dienstag, dem 17.02.2026 um 12:24 +0100 schrieb Bert Karwatzki:
Quoted 11 lines
Am Dienstag, dem 17.02.2026 um 11:42 +0100 schrieb Bert Karwatzki:
Quoted 10 lines
I just wondered if we can completely skip the
if (READ_ONCE(txq->xmit_lock_owner) != cpu) { [...] } else { /* "Recursion" alert */ }Quoted 6 lines
Quoted 2 lines
check, as the synchronization will we provided by HARD_TX_{LOCK,UNLOCK}.
I thought about that again, and it seems like a bad idea as (in the non-preempt) case other threads trying to access the queue would wait for the spinlock to be freed, perhaps one can just change the code like this:
Quoted 5 lines
My argument above seems wrong: In the non-preempt case we cannot have another thread accessing the txq from the same CPU if the lock is taken (it is not preemptible) and for other threads accessing the txq from different CPUs the check (
txq->xmit_lock_owner!= cpu) would succeed and they would try the spinlock anyway, So this would not speak against killing the lock owner check. As for the recursion detection, perhapsdev_xmit_recursion()is enough?
I tried to research the original commit which introduced the xmit_lock_owner check, but
it is present since linux 2.3.6 (released 19990610) (when __dev_queue_xmit() was still called dev_queue_xmit()), so I can't tell the original idea behind that check (perhaps recuesion detection ...), so I'm not completely sure if it can be omitted (and just let dev_xmit_recursion() do the recursion checking).
Bert Karwatzki
Sebastian Andrzej Siewior
maintainerQuoted 5 lines · Bert Karwatzki
I tried to research the original commit which introduced the
xmit_lock_ownercheck, but
it is present since linux 2.3.6 (released 19990610) (when__dev_queue_xmit()was still calleddev_queue_xmit()), so I can't tell the original idea behind that check (perhaps recuesion detection ...), so I'm not completely sure if it can be omitted (and just letdev_xmit_recursion()do the recursion checking).
Okay. Thank you. I add it to my list.
Quoted 1 line
Bert Karwatzki
Sebastian
Am Mittwoch, dem 18.02.2026 um 08:30 +0100 schrieb Sebastian Andrzej Siewior:
Quoted 9 lines · Bert Karwatzki
Quoted 5 lines
I tried to research the original commit which introduced the
xmit_lock_ownercheck, but
it is present since linux 2.3.6 (released 19990610) (when__dev_queue_xmit()was still calleddev_queue_xmit()), so I can't tell the original idea behind that check (perhaps recuesion detection ...), so I'm not completely sure if it can be omitted (and just letdev_xmit_recursion()do the recursion checking).Okay. Thank you. I add it to my list.
I've thought about it again and I now think the xmit_lock_owner check IS necessary to avoid deadlocks on recursion in the non-RT case.
My idea to use get_current()->tgid as lock owner also does not work as interrupts are still enabled and __dev_queue_xmit() can be called from interrupt context. So in a situation where an interrupt occurs after the lock has been taken and the interrupt handler calls __dev_queue_xmit() on the same CPU a deadlock would occur.
Bert Karwatzki
Sebastian Andrzej Siewior
maintainerQuoted 10 lines · Bert Karwatzki
Am Mittwoch, dem 18.02.2026 um 08:30 +0100 schrieb Sebastian Andrzej Siewior:
Quoted 9 lines · Bert Karwatzki
Quoted 5 lines
I tried to research the original commit which introduced the
xmit_lock_ownercheck, but
it is present since linux 2.3.6 (released 19990610) (when__dev_queue_xmit()was still calleddev_queue_xmit()), so I can't tell the original idea behind that check (perhaps recuesion detection ...), so I'm not completely sure if it can be omitted (and just letdev_xmit_recursion()do the recursion checking).Okay. Thank you. I add it to my list.
Quoted 7 lines
I've thought about it again and I now think the
xmit_lock_ownercheck IS necessary to avoid deadlocks on recursion in the non-RT case.My idea to use get_current()->tgid as lock owner also does not work as interrupts are still enabled and
__dev_queue_xmit()can be called from interrupt context. So in a situation where an interrupt occurs after the lock has been taken and the interrupt handler calls__dev_queue_xmit()on the same CPU a deadlock would occur.
The warning happens because taskA on cpuX goes through HARD_TX_LOCK(), gets preempted and then taskB on cpuX wants also to send send a packet. The second one throws the warning.
We could ignore this check because a deadlock will throw a warning and "halt" the task that runs into the deadlock.
But then we could be smart about this in the same way !RT is and pro-active check for the simple deadlock. The lock owner of netdev_queue::_xmit_lock is recorded can be checked vs current.
The snippet below should work. I need to see if tomorrow this is still a good idea.
net/core/dev.c +1 -1
@@ -4821,7 +4821,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)/* Other cpus might concurrently change txq->xmit_lock_owner * to -1 or to their cpu id, but not to our id. */ -if (READ_ONCE (txq->xmit_lock_owner) != cpu) { +if (rt_mutex_owner (&txq->_xmit_lock.lock) != current) {if (dev_xmit_recursion ()) goto recursion_alert;
Quoted 1 line
Bert Karwatzki
Sebastian
Quoted 37 lines · Sebastian Andrzej Siewior
Quoted 10 lines · Bert Karwatzki
Am Mittwoch, dem 18.02.2026 um 08:30 +0100 schrieb Sebastian Andrzej Siewior:
Quoted 9 lines · Bert Karwatzki
Quoted 5 lines
I tried to research the original commit which introduced the
xmit_lock_ownercheck, but
it is present since linux 2.3.6 (released 19990610) (when__dev_queue_xmit()was still calleddev_queue_xmit()), so I can't tell the original idea behind that check (perhaps recuesion detection ...), so I'm not completely sure if it can be omitted (and just letdev_xmit_recursion()do the recursion checking).Okay. Thank you. I add it to my list.
Quoted 7 lines
I've thought about it again and I now think the
xmit_lock_ownercheck IS necessary to avoid deadlocks on recursion in the non-RT case.My idea to use get_current()->tgid as lock owner also does not work as interrupts are still enabled and
__dev_queue_xmit()can be called from interrupt context. So in a situation where an interrupt occurs after the lock has been taken and the interrupt handler calls__dev_queue_xmit()on the same CPU a deadlock would occur.The warning happens because taskA on cpuX goes through
HARD_TX_LOCK(), gets preempted and then taskB on cpuX wants also to send send a packet. The second one throws the warning.We could ignore this check because a deadlock will throw a warning and "halt" the task that runs into the deadlock.
But then we could be smart about this in the same way !RT is and pro-active check for the simple deadlock. The lock owner ofnetdev_queue::_xmit_lockis recorded can be checked vs current.The snippet below should work. I need to see if tomorrow this is still a good idea.
net/core/dev.c +0
-0
net/core/dev.c int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) · line 4824
Quoted 5 lines
net/core/dev.c +1
-1@@ -4821,7 +4821,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)/* Other cpus might concurrently change txq->xmit_lock_owner * to -1 or to their cpu id, but not to our id. */ -if (READ_ONCE (txq->xmit_lock_owner) != cpu) { +if (rt_mutex_owner (&txq->_xmit_lock.lock) != current) {
Ain't this changing the behavior for !RT case? Previously, if it was the same thread which has already locked the queue (and hence the same CPU) evaluating this condition, the condition was skipped, which is no longer the case with this change.
How about something like this instead?:
include/linux/netdevice.h +6 -6
@@ -705,7 +705,7 @@ struct netdev_queue {struct dql dql;#endif spinlock_t _xmit_lock____cacheline_aligned_in_smp ; -int xmit_lock_owner; +struct task_struct *xmit_lock_owner;/* * Time (in jiffies) of last Tx */
@@ -4709,7 +4709,7 @@ static inline void __netif_tx_lock(struct netdev_queue *txq, int cpu) {spin_lock (&txq->_xmit_lock);/* Pairs with READ_ONCE() in __dev_queue_xmit() */ -WRITE_ONCE (txq->xmit_lock_owner, cpu); +WRITE_ONCE (txq->xmit_lock_owner, current); }static inline bool __netif_tx_acquire (struct netdev_queue *txq)
@@ -4727,7 +4727,7 @@ static inline void __netif_tx_lock_bh(struct netdev_queue *txq) {spin_lock_bh (&txq->_xmit_lock);/* Pairs with READ_ONCE() in __dev_queue_xmit() */ -WRITE_ONCE (txq->xmit_lock_owner,smp_processor_id ()); +WRITE_ONCE (txq->xmit_lock_owner, current); }static inline bool __netif_tx_trylock (struct netdev_queue *txq)
@@ -4736,7 +4736,7 @@ static inline bool __netif_tx_trylock(struct netdev_queue *txq)if (likely (ok)) {/* Pairs with READ_ONCE() in __dev_queue_xmit() */ -WRITE_ONCE (txq->xmit_lock_owner,smp_processor_id ()); +WRITE_ONCE (txq->xmit_lock_owner, current); }return ok; }
@@ -4744,14 +4744,14 @@ static inline bool __netif_tx_trylock(struct netdev_queue *txq)static inline void __netif_tx_unlock (struct netdev_queue *txq) {/* Pairs with READ_ONCE() in __dev_queue_xmit() */ -WRITE_ONCE (txq->xmit_lock_owner,-1 ); +WRITE_ONCE (txq->xmit_lock_owner,NULL );spin_unlock (&txq->_xmit_lock); }static inline void __netif_tx_unlock_bh (struct netdev_queue *txq) {/* Pairs with READ_ONCE() in __dev_queue_xmit() */ -WRITE_ONCE (txq->xmit_lock_owner,-1 ); +WRITE_ONCE (txq->xmit_lock_owner,NULL );spin_unlock_bh (&txq->_xmit_lock); }
net/core/dev.c +2 -2
@@ -4817,7 +4817,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)/* Other cpus might concurrently change txq->xmit_lock_owner * to -1 or to their cpu id, but not to our id. */ -if (READ_ONCE (txq->xmit_lock_owner) != cpu) { +if (READ_ONCE (txq->xmit_lock_owner) != current) {if (dev_xmit_recursion ()) goto recursion_alert;
@@ -11178,7 +11178,7 @@ static void netdev_init_one_queue(struct net_device *dev,/* Initialize queue lock */ spin_lock_init (&queue->_xmit_lock);netdev_set_xmit_lockdep_class (&queue->_xmit_lock, dev->type); - queue->xmit_lock_owner =-1 ; + queue->xmit_lock_owner =NULL ;netdev_queue_numa_node_write (queue,NUMA_NO_NODE ); queue->dev = dev;#ifdef CONFIG_BQL
Daniel
Quoted 7 lines
net/core/dev.c +0
-0if (dev_xmit_recursion ()) goto recursion_alert;Quoted 2 lines
Bert Karwatzki
Sebastian
Sebastian Andrzej Siewior
maintainerQuoted 4 lines · Daniel Vacek
Quoted 4 lines
net/core/dev.c +0
-0@@ -4821,7 +4821,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)/* Other cpus might concurrently change txq->xmit_lock_owner
Quoted 8 lines
net/core/dev.c int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)
Quoted 4 lines
net/core/dev.c +1
-1* to -1 or to their cpu id, but not to our id. */ -if (READ_ONCE (txq->xmit_lock_owner) != cpu) { +if (rt_mutex_owner (&txq->_xmit_lock.lock) != current) {Ain't this changing the behavior for !RT case? Previously, if it was the same thread which has already locked the queue (and hence the same CPU) evaluating this condition, the condition was skipped, which is no longer the case with this change.
The above was me thinking and does not even compile for !RT. Commit b824c3e16c190 ("net: Provide a PREEMPT_RT specific check for netdev_queue::_xmit_lock") is what was merged in the end.
Sebastian
Quoted 15 lines · Sebastian Andrzej Siewior
Quoted 4 lines · Daniel Vacek
Quoted 4 lines
net/core/dev.c +0
-0@@ -4821,7 +4821,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)/* Other cpus might concurrently change txq->xmit_lock_owner Quoted 8 lines
net/core/dev.c int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)
Quoted 4 lines
net/core/dev.c +1
-1* to -1 or to their cpu id, but not to our id. */ -if (READ_ONCE (txq->xmit_lock_owner) != cpu) { +if (rt_mutex_owner (&txq->_xmit_lock.lock) != current) {Ain't this changing the behavior for !RT case? Previously, if it was the same thread which has already locked the queue (and hence the same CPU) evaluating this condition, the condition was skipped, which is no longer the case with this change.
Quoted 3 lines
The above was me thinking and does not even compile for !RT. Commit
b824c3e16c190("net: Provide aPREEMPT_RTspecific check fornetdev_queue::_xmit_lock") is what was merged in the end.
Hmm, that means txq->xmit_lock_owner is not used at all for PREEMT_RT. It's pointless to even store it. Shall we care?
--nX
Quoted 1 line
Sebastian
Sebastian Andrzej Siewior
maintainerQuoted 17 lines · Daniel Vacek
Quoted 15 lines · Sebastian Andrzej Siewior
Quoted 12 lines · Daniel Vacek
net/core/dev.c int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) · line 4824
Quoted 8 lines
net/core/dev.c +1
-1@@ -4821,7 +4821,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)/* Other cpus might concurrently change txq->xmit_lock_owner * to -1 or to their cpu id, but not to our id. */ -if (READ_ONCE (txq->xmit_lock_owner) != cpu) { +if (rt_mutex_owner (&txq->_xmit_lock.lock) != current) {Ain't this changing the behavior for !RT case? Previously, if it was the same thread which has already locked the queue (and hence the same CPU) evaluating this condition, the condition was skipped, which is no longer the case with this change.
Quoted 6 lines
Quoted 3 lines
The above was me thinking and does not even compile for !RT. Commit
b824c3e16c190("net: Provide aPREEMPT_RTspecific check fornetdev_queue::_xmit_lock") is what was merged in the end.Hmm, that means
txq->xmit_lock_owneris not used at all forPREEMT_RT. It's pointless to even store it. Shall we care?
For PREEMPT_RT the xmit_lock_owner member is only stored and not used otherwise. It could be removed as in ifdef-ed away but I do not care enough to sprinkle it and the gain is little (proof me wrong). So I am happy as-is.
As for the check itself as we have now, we detect the deadlock before it happens and this is nice to have. The alternative (in case of a deadlock) would be the deadlock detection in rtmutex code which would freeze the thread.
Quoted 2 lines
--nX
Sebastian
Quoted 22 lines · Sebastian Andrzej Siewior
Quoted 21 lines · Daniel Vacek
Quoted 18 lines · Sebastian Andrzej Siewior
Quoted 4 lines · Daniel Vacek
Quoted 4 lines
net/core/dev.c +0
-0@@ -4821,7 +4821,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)/* Other cpus might concurrently change txq->xmit_lock_owner Quoted 8 lines
net/core/dev.c int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)
Quoted 4 lines
net/core/dev.c +1
-1* to -1 or to their cpu id, but not to our id. */ -if (READ_ONCE (txq->xmit_lock_owner) != cpu) { +if (rt_mutex_owner (&txq->_xmit_lock.lock) != current) {Ain't this changing the behavior for !RT case? Previously, if it was the same thread which has already locked the queue (and hence the same CPU) evaluating this condition, the condition was skipped, which is no longer the case with this change.
The above was me thinking and does not even compile for !RT. Commit
b824c3e16c190("net: Provide aPREEMPT_RTspecific check fornetdev_queue::_xmit_lock") is what was merged in the end.
Quoted 7 lines
Quoted 2 lines
Hmm, that means
txq->xmit_lock_owneris not used at all forPREEMT_RT. It's pointless to even store it. Shall we care?For
PREEMPT_RTthexmit_lock_ownermember is only stored and not used otherwise. It could be removed as in ifdef-ed away but I do not care enough to sprinkle it and the gain is little (proof me wrong). So I am happy as-is.
Right. Sorry for bothering then. I missed the merged commit.
--nX
Quoted 8 lines
As for the check itself as we have now, we detect the deadlock before it happens and this is nice to have. The alternative (in case of a deadlock) would be the deadlock detection in rtmutex code which would freeze the thread.
Quoted 2 lines
--nX
Sebastian
Quoted 18 lines · Sebastian Andrzej Siewior
Quoted 17 lines · Daniel Vacek
Quoted 15 lines · Sebastian Andrzej Siewior
Quoted 12 lines · Daniel Vacek
net/core/dev.c int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) · line 4824
Quoted 8 lines
net/core/dev.c +1
-1@@ -4821,7 +4821,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)/* Other cpus might concurrently change txq->xmit_lock_owner * to -1 or to their cpu id, but not to our id. */ -if (READ_ONCE (txq->xmit_lock_owner) != cpu) { +if (rt_mutex_owner (&txq->_xmit_lock.lock) != current) {Ain't this changing the behavior for !RT case? Previously, if it was the same thread which has already locked the queue (and hence the same CPU) evaluating this condition, the condition was skipped, which is no longer the case with this change.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The above was me thinking and does not even compile for !RT. Commit
b824c3e16c190("net: Provide aPREEMPT_RTspecific check fornetdev_queue::_xmit_lock") is what was merged in the end.
Thinking about it again, wouldn't it be better to have one generic solution rather then special-casing for PREEMPT_RT vs. !PREEMPT_RT?
--nX
Quoted 17 lines
Quoted 3 lines
Hmm, that means
txq->xmit_lock_owneris not used at all forPREEMT_RT. It's pointless to even store it. Shall we care?For
PREEMPT_RTthexmit_lock_ownermember is only stored and not used otherwise. It could be removed as in ifdef-ed away but I do not care enough to sprinkle it and the gain is little (proof me wrong). So I am happy as-is.As for the check itself as we have now, we detect the deadlock before it happens and this is nice to have. The alternative (in case of a deadlock) would be the deadlock detection in rtmutex code which would freeze the thread.
Quoted 2 lines
--nX
Sebastian
Quoted 39 lines · Sebastian Andrzej Siewior
Quoted 17 lines · Daniel Vacek
Quoted 15 lines · Sebastian Andrzej Siewior
Quoted 12 lines · Daniel Vacek
net/core/dev.c int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) · line 4824
Quoted 8 lines
net/core/dev.c +1
-1@@ -4821,7 +4821,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev)/* Other cpus might concurrently change txq->xmit_lock_owner * to -1 or to their cpu id, but not to our id. */ -if (READ_ONCE (txq->xmit_lock_owner) != cpu) { +if (rt_mutex_owner (&txq->_xmit_lock.lock) != current) {Ain't this changing the behavior for !RT case? Previously, if it was the same thread which has already locked the queue (and hence the same CPU) evaluating this condition, the condition was skipped, which is no longer the case with this change.
Quoted 6 lines
Quoted 3 lines
The above was me thinking and does not even compile for !RT. Commit
b824c3e16c190("net: Provide aPREEMPT_RTspecific check fornetdev_queue::_xmit_lock") is what was merged in the end.Hmm, that means
txq->xmit_lock_owneris not used at all forPREEMT_RT. It's pointless to even store it. Shall we care?For
PREEMPT_RTthexmit_lock_ownermember is only stored and not used otherwise. It could be removed as in ifdef-ed away but I do not care enough to sprinkle it and the gain is little (proof me wrong). So I am happy as-is.As for the check itself as we have now, we detect the deadlock before it happens and this is nice to have. The alternative (in case of a deadlock) would be the deadlock detection in rtmutex code which would freeze the thread.
Quoted 2 lines
--nX
Sebastian
Sebastian Andrzej Siewior
maintainerQuoted 6 lines · Daniel Vacek
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The above was me thinking and does not even compile for !RT. Commit
b824c3e16c190("net: Provide aPREEMPT_RTspecific check fornetdev_queue::_xmit_lock") is what was merged in the end.Thinking about it again, wouldn't it be better to have one generic solution rather then special-casing for
PREEMPT_RTvs. !PREEMPT_RT?
PREEMPT_RT and !PREEMPT_RT is fundamentally different here. The one is not preemptible and records the CPU of the lock owner to detect a recursive deadlock.
The other is preemptible, uses a different locking type/ class which records the lock owner which can be utilised for this purpose.
A generic thing would be to remove this and rely on lockdep. This could work if it is only a devel thing and never "I setup something and make a loop" sort of thing.
Quoted 1 line
--nX
Sebastian
Quoted 5 lines · Sebastian Andrzej Siewior
Quoted 4 lines · Daniel Vacek
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The above was me thinking and does not even compile for !RT. Commit
b824c3e16c190("net: Provide aPREEMPT_RTspecific check fornetdev_queue::_xmit_lock") is what was merged in the end.
Quoted 8 lines
Quoted 2 lines
Thinking about it again, wouldn't it be better to have one generic solution rather then special-casing for
PREEMPT_RTvs. !PREEMPT_RT?
PREEMPT_RTand !PREEMPT_RT is fundamentally different here. The one is not preemptible and records the CPU of the lock owner to detect a recursive deadlock.
The other is preemptible, uses a different locking type/ class which records the lock owner which can be utilised for this purpose.
I understand that (or at least I think I do).
My idea was that the non-preemptible one can record current task instead of the CPU to detect the deadlock. And that would also work for the preemptible case (it would actually match the lock owner approach as you did for the PREEMPT_RT case).
One code for both configurations, no special-casing. I'd argue that's a better result. Am I missing something?
The size of the netdev_queue structure would grow by 8 bytes for !RT case, but that's not a big deal, IMO. For RT case it would just fill the hole.
--nX
Quoted 7 lines
A generic thing would be to remove this and rely on lockdep. This could work if it is only a devel thing and never "I setup something and make a loop" sort of thing.
Quoted 1 line
--nX
Sebastian
Sebastian Andrzej Siewior
maintainerQuoted 10 lines · Daniel Vacek
My idea was that the non-preemptible one can record
currenttask instead of the CPU to detect the deadlock. And that would also work for the preemptible case (it would actually match the lock owner approach as you did for thePREEMPT_RTcase).
One code for both configurations, no special-casing. I'd argue that's a better result. Am I missing something?The size of the
netdev_queuestructure would grow by 8 bytes for !RT case, but that's not a big deal, IMO. For RT case it would just fill the hole.
We have xmit_lock_owner as int. If you replace it with task_struct * then on 64bit the size of the struct netdev_queue will remain unchanged as it fills the hole before the following long.
Then you could record `current' as the lock owner in both cases. This should work.
Quoted 1 line
--nX
Sebastian
Quoted 12 lines · Sebastian Andrzej Siewior
Quoted 10 lines · Daniel Vacek
My idea was that the non-preemptible one can record
currenttask instead of the CPU to detect the deadlock. And that would also work for the preemptible case (it would actually match the lock owner approach as you did for thePREEMPT_RTcase).
One code for both configurations, no special-casing. I'd argue that's a better result. Am I missing something?The size of the
netdev_queuestructure would grow by 8 bytes for !RT case, but that's not a big deal, IMO. For RT case it would just fill the hole.
Quoted 5 lines
We have
xmit_lock_owneras int. If you replace it withtask_struct* then on 64bit the size of the structnetdev_queuewill remain unchanged as it fills the hole before the following long.
Then you could record `current' as the lock owner in both cases. This should work.
Well, that's the patch I originally sent then.
https://lore.kernel.org/linux-rt-devel/[email protected]/
--nX
Quoted 2 lines
Sebastian
Sebastian Andrzej Siewior
maintainerQuoted 4 lines · Daniel Vacek
Well, that's the patch I originally sent then.
That was an alternative after everything was already done.
Quoted 1 line
--nX
Sebastian
Quoted 7 lines · Sebastian Andrzej Siewior
Quoted 4 lines · Daniel Vacek
Well, that's the patch I originally sent then.
That was an alternative after everything was already done.
That is true. I don't mean to complain. But also because I haven't seen the applied version on the mailing list. That's why I originally sent my patch as a reply to this thread.
I'm simply asking whether we should proceed with the discussed approach? Would that be better in the end?
If you don't object, I can send a rebased version along with a follow-up cleanup.
--nX
Quoted 3 lines
Quoted 1 line
--nX
Sebastian