Skip to contentSkip to conversation

xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path

RFC PATCH In review Jason Xing 2 participants 4 messages Dec 9, 2025

Subsystems (4)
lore
Share & tools
lore

Jason Xing

author
#0 lore raw

RFC PATCH v4 1/1 xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path

From: Jason Xing <[email protected]>

We (Paolo and I) noticed that in the sending path touching an extra cacheline due to cq_cached_prod_lock will impact the performance. After moving the lock from struct xsk_buff_pool to struct xsk_queue, the performance is increased by ~5% which can be observed by xdpsock.

An alternative approach [1] can be using atomic_try_cmpxchg() to have the same effect. But unfortunately I don't have evident performance number to prove the atomic approach is better than the current patch. The advantage is to save the contention time among multiple xsks sharing the same pool while the disadvantage is lose good maintenance. The full discussion can be found at the following link.

[1]: https://lore.kernel.org/all/[email protected]/

---
Q: since net-next will be open next year, I wonder if I should post this patch targetting bpf-next?

RFC V4

1. use moving lock method instead (Paolo, Magnus)
2. Add credit to Paolo, thanks!

v3

1. fix one race issue that cannot be resolved by simple seperated atomic operations. So this revision only updates patch [2/3] and tries to use try_cmpxchg method to avoid that problem. (paolo)
2. update commit log accordingly.

V2

1. use separate functions rather than branches within shared routines. (Maciej)
2. make each patch as simple as possible for easier review

---
 include/net/xsk_buff_pool.h | 5 -----
 net/xdp/xsk.c               | 8 ++++----
 net/xdp/xsk_buff_pool.c     | 2 +-
 net/xdp/xsk_queue.h         | 5 +++++
 4 files changed, 10 insertions(+), 10 deletions(-)
include/net/xsk_buff_pool.h +0 -5
index 92a2358c6ce3..0b1abdb99c9e 100644
@@ -90,11 +90,6 @@ struct xsk_buff_pool {
 	 * destructor callback.
 	 */
 	spinlock_t cq_prod_lock;
-	/* Mutual exclusion of the completion ring in the SKB mode.
-	 * Protect: when sockets share a single cq when the same netdev
-	 * and queue id is shared.
-	 */
-	spinlock_t cq_cached_prod_lock;
 	struct xdp_buff_xsk *free_heads[];
 };
 
net/xdp/xsk.c +4 -4
index f093c3453f64..7613887b4122 100644
@@ -543,9 +543,9 @@ static int xsk_cq_reserve_locked(struct xsk_buff_pool *pool)
 {
 	int ret;
 
-	spin_lock(&pool->cq_cached_prod_lock);
+	spin_lock(&pool->cq->cq_cached_prod_lock);
 	ret = xskq_prod_reserve(pool->cq);
-	spin_unlock(&pool->cq_cached_prod_lock);
+	spin_unlock(&pool->cq->cq_cached_prod_lock);
 
 	return ret;
 }
@@ -619,9 +619,9 @@ static void xsk_cq_submit_addr_locked(struct xsk_buff_pool *pool,
 
 static void xsk_cq_cancel_locked(struct xsk_buff_pool *pool, u32 n)
 {
-	spin_lock(&pool->cq_cached_prod_lock);
+	spin_lock(&pool->cq->cq_cached_prod_lock);
 	xskq_prod_cancel_n(pool->cq, n);
-	spin_unlock(&pool->cq_cached_prod_lock);
+	spin_unlock(&pool->cq->cq_cached_prod_lock);
 }
 
 INDIRECT_CALLABLE_SCOPE
net/xdp/xsk_buff_pool.c +1 -1
index 51526034c42a..127c17e02384 100644
@@ -91,7 +91,7 @@ struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,
 	INIT_LIST_HEAD(&pool->xsk_tx_list);
 	spin_lock_init(&pool->xsk_tx_list_lock);
 	spin_lock_init(&pool->cq_prod_lock);
-	spin_lock_init(&pool->cq_cached_prod_lock);
+	spin_lock_init(&xs->cq_tmp->cq_cached_prod_lock);
 	refcount_set(&pool->users, 1);
 
 	pool->fq = xs->fq_tmp;
net/xdp/xsk_queue.h +5 -0
index 1eb8d9f8b104..ec08d9c102b1 100644
@@ -46,6 +46,11 @@ struct xsk_queue {
 	u64 invalid_descs;
 	u64 queue_empty_descs;
 	size_t ring_vmalloc_size;
+	/* Mutual exclusion of the completion ring in the SKB mode.
+	 * Protect: when sockets share a single cq when the same netdev
+	 * and queue id is shared.
+	 */
+	spinlock_t cq_cached_prod_lock;
 };
 
 struct parsed_desc {
Signature · 1 line
2.41.3
syzbot Tested-by [email protected] on v4 1/1
#1 lore raw
syzbot ci has tested the following series

syzbot ci has tested the following series

[v4] xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path
https://lore.kernel.org/all/[email protected]
* [PATCH RFC net-next v4] xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path

and found the following issue:

BUG: unable to handle kernel NULL pointer dereference in xp_create_and_assign_umem

Full report is available here:
https://ci.syzbot.org/series/d7e166a7-a880-4ea1-9707-8889afd4ebe8

***

BUG: unable to handle kernel NULL pointer dereference in xp_create_and_assign_umem
tree:      net-next
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/netdev/net-next.git
base:      0177f0f07886e54e12c6f18fa58f63e63ddd3c58
arch:      amd64
compiler:  Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
config:    https://ci.syzbot.org/builds/d327cc4b-7471-413b-b244-519c6d16d43b/config
C repro:   https://ci.syzbot.org/findings/c8f7aeaf-0e2e-43dd-ae9c-ea2dd8db8d34/c_repro
syz repro: https://ci.syzbot.org/findings/c8f7aeaf-0e2e-43dd-ae9c-ea2dd8db8d34/syz_repro

UDPLite6: UDP-Lite is deprecated and scheduled to be removed in 2025, please contact the netdev mailing list

BUG: kernel NULL pointer dereference, address: 0000000000000058
#PF: supervisor write access in kernel mode
#PF: error_code(0x0002) - not-present page
PGD 80000001b2496067 P4D 80000001b2496067 PUD 0 
Oops: Oops: 0002 [#1] SMP KASAN PTI
CPU: 1 UID: 0 PID: 5973 Comm: syz.0.17 Not tainted syzkaller #0 PREEMPT(full) 
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
RIP: 0010:lockdep_init_map_type+0x1e/0x380 kernel/locking/lockdep.c:4944
Code: 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 55 41 56 53 48 83 ec 10 89 cd 48 89 fb 65 48 8b 05 67 af d1 10 48 89 44 24 08 <48> c7 47 10 00 00 00 00 48 c7 47 08 00 00 00 00 8b 05 8c f2 dc 17
RSP: 0018:ffffc90003c07bb8 EFLAGS: 00010286
RAX: ec490cf5c114aa00 RBX: 0000000000000048 RCX: 0000000000000000
RDX: ffffffff99d16120 RSI: ffffffff8c92a180 RDI: 0000000000000048
RBP: 0000000000000000 R08: 0000000000000003 R09: 0000000000000000
R10: ffffed102e5d7800 R11: fffffbfff1efa3cf R12: dffffc0000000000
R13: ffff8881bdeb3000 R14: ffffffff99d16120 R15: ffffffff8c92a180
FS:  00005555729bd500(0000) GS:ffff8882a9f31000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000058 CR3: 0000000172faa000 CR4: 00000000000006f0
Call Trace:
 <TASK>
 lockdep_init_map_waits include/linux/lockdep.h:135 [inline]
 lockdep_init_map_wait include/linux/lockdep.h:142 [inline]
 __raw_spin_lock_init+0x45/0x100 kernel/locking/spinlock_debug.c:25
 xp_create_and_assign_umem+0x648/0xd40 net/xdp/xsk_buff_pool.c:94
 xsk_bind+0x95a/0xf90 net/xdp/xsk.c:1355
 __sys_bind_socket net/socket.c:1874 [inline]
 __sys_bind+0x2c6/0x3e0 net/socket.c:1905
 __do_sys_bind net/socket.c:1910 [inline]
 __se_sys_bind net/socket.c:1908 [inline]
 __x64_sys_bind+0x7a/0x90 net/socket.c:1908
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0xfa/0xfa0 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f591eb8f7c9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffce6fcef48 EFLAGS: 00000246 ORIG_RAX: 0000000000000031
RAX: ffffffffffffffda RBX: 00007f591ede5fa0 RCX: 00007f591eb8f7c9
RDX: 0000000000000010 RSI: 0000200000000240 RDI: 0000000000000003
RBP: 00007f591ebf297f R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f591ede5fa0 R14: 00007f591ede5fa0 R15: 0000000000000003
 </TASK>
Modules linked in:
CR2: 0000000000000058
---[ end trace 0000000000000000 ]---
RIP: 0010:lockdep_init_map_type+0x1e/0x380 kernel/locking/lockdep.c:4944
Code: 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 55 41 56 53 48 83 ec 10 89 cd 48 89 fb 65 48 8b 05 67 af d1 10 48 89 44 24 08 <48> c7 47 10 00 00 00 00 48 c7 47 08 00 00 00 00 8b 05 8c f2 dc 17
RSP: 0018:ffffc90003c07bb8 EFLAGS: 00010286
RAX: ec490cf5c114aa00 RBX: 0000000000000048 RCX: 0000000000000000
RDX: ffffffff99d16120 RSI: ffffffff8c92a180 RDI: 0000000000000048
RBP: 0000000000000000 R08: 0000000000000003 R09: 0000000000000000
R10: ffffed102e5d7800 R11: fffffbfff1efa3cf R12: dffffc0000000000
R13: ffff8881bdeb3000 R14: ffffffff99d16120 R15: ffffffff8c92a180
FS:  00005555729bd500(0000) GS:ffff8882a9f31000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000058 CR3: 0000000172faa000 CR4: 00000000000006f0
----------------
Code disassembly (best guess):
   0:	90                   	nop
   1:	90                   	nop
   2:	90                   	nop
   3:	90                   	nop
   4:	90                   	nop
   5:	90                   	nop
   6:	90                   	nop
   7:	90                   	nop
   8:	90                   	nop
   9:	90                   	nop
   a:	90                   	nop
   b:	90                   	nop
   c:	f3 0f 1e fa          	endbr64
  10:	55                   	push   %rbp
  11:	41 56                	push   %r14
  13:	53                   	push   %rbx
  14:	48 83 ec 10          	sub    $0x10,%rsp
  18:	89 cd                	mov    %ecx,%ebp
  1a:	48 89 fb             	mov    %rdi,%rbx
  1d:	65 48 8b 05 67 af d1 	mov    %gs:0x10d1af67(%rip),%rax        # 0x10d1af8c
  24:	10
  25:	48 89 44 24 08       	mov    %rax,0x8(%rsp)
* 2a:	48 c7 47 10 00 00 00 	movq   $0x0,0x10(%rdi) <-- trapping instruction
  31:	00
  32:	48 c7 47 08 00 00 00 	movq   $0x0,0x8(%rdi)
  39:	00
  3a:	8b 05 8c f2 dc 17    	mov    0x17dcf28c(%rip),%eax        # 0x17dcf2cc

***

If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:

---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].

author
#2 lore raw
Quoted 17 lines · syzbot ci

syzbot ci has tested the following series

[v4] xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path
https://lore.kernel.org/all/[email protected]
* [PATCH RFC net-next v4] xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path

and found the following issue:

BUG: unable to handle kernel NULL pointer dereference in xp_create_and_assign_umem

Full report is available here:
https://ci.syzbot.org/series/d7e166a7-a880-4ea1-9707-8889afd4ebe8

***

BUG: unable to handle kernel NULL pointer dereference in xp_create_and_assign_umem
Quoted 8 lines
tree:      net-next
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/netdev/net-next.git
base:      0177f0f07886e54e12c6f18fa58f63e63ddd3c58
arch:      amd64
compiler:  Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
config:    https://ci.syzbot.org/builds/d327cc4b-7471-413b-b244-519c6d16d43b/config
C repro:   https://ci.syzbot.org/findings/c8f7aeaf-0e2e-43dd-ae9c-ea2dd8db8d34/c_repro
syz repro: https://ci.syzbot.org/findings/c8f7aeaf-0e2e-43dd-ae9c-ea2dd8db8d34/syz_repro

Interesting. Only rx logic is initialized while tx not. That causes the cq_tmp to be uninitialized.

Quoted 17 lines

UDPLite6: UDP-Lite is deprecated and scheduled to be removed in 2025, please contact the netdev mailing list

BUG: kernel NULL pointer dereference, address: 0000000000000058
#PF: supervisor write access in kernel mode
#PF: error_code(0x0002) - not-present page
PGD 80000001b2496067 P4D 80000001b2496067 PUD 0
Oops: Oops: 0002 [#1] SMP KASAN PTI
CPU: 1 UID: 0 PID: 5973 Comm: syz.0.17 Not tainted syzkaller #0 PREEMPT(full)
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
RIP: 0010:lockdep_init_map_type+0x1e/0x380 kernel/locking/lockdep.c:4944
Code: 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 55 41 56 53 48 83 ec 10 89 cd 48 89 fb 65 48 8b 05 67 af d1 10 48 89 44 24 08 <48> c7 47 10 00 00 00 00 48 c7 47 08 00 00 00 00 8b 05 8c f2 dc 17
RSP: 0018:ffffc90003c07bb8 EFLAGS: 00010286
RAX: ec490cf5c114aa00 RBX: 0000000000000048 RCX: 0000000000000000
RDX: ffffffff99d16120 RSI: ffffffff8c92a180 RDI: 0000000000000048
RBP: 0000000000000000 R08: 0000000000000003 R09: 0000000000000000
R10: ffffed102e5d7800 R11: fffffbfff1efa3cf R12: dffffc0000000000
R13: ffff8881bdeb3000 R14: ffffffff99d16120 R15: ffffffff8c92a180
Quoted 8 lines
FS:  00005555729bd500(0000) GS:ffff8882a9f31000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000058 CR3: 0000000172faa000 CR4: 00000000000006f0
Call Trace:
 <TASK>
 lockdep_init_map_waits include/linux/lockdep.h:135 [inline]
 lockdep_init_map_wait include/linux/lockdep.h:142 [inline]
 __raw_spin_lock_init+0x45/0x100 kernel/locking/spinlock_debug.c:25

Using if-condition to check if cq_tmp is NULL would avoid the corruption.

Will add it in the next version.

Thanks,
Jason

Quoted 74 lines
 xp_create_and_assign_umem+0x648/0xd40 net/xdp/xsk_buff_pool.c:94
 xsk_bind+0x95a/0xf90 net/xdp/xsk.c:1355
 __sys_bind_socket net/socket.c:1874 [inline]
 __sys_bind+0x2c6/0x3e0 net/socket.c:1905
 __do_sys_bind net/socket.c:1910 [inline]
 __se_sys_bind net/socket.c:1908 [inline]
 __x64_sys_bind+0x7a/0x90 net/socket.c:1908
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0xfa/0xfa0 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f591eb8f7c9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffce6fcef48 EFLAGS: 00000246 ORIG_RAX: 0000000000000031
RAX: ffffffffffffffda RBX: 00007f591ede5fa0 RCX: 00007f591eb8f7c9
RDX: 0000000000000010 RSI: 0000200000000240 RDI: 0000000000000003
RBP: 00007f591ebf297f R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f591ede5fa0 R14: 00007f591ede5fa0 R15: 0000000000000003
 </TASK>
Modules linked in:
CR2: 0000000000000058
---[ end trace 0000000000000000 ]---
RIP: 0010:lockdep_init_map_type+0x1e/0x380 kernel/locking/lockdep.c:4944
Code: 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 55 41 56 53 48 83 ec 10 89 cd 48 89 fb 65 48 8b 05 67 af d1 10 48 89 44 24 08 <48> c7 47 10 00 00 00 00 48 c7 47 08 00 00 00 00 8b 05 8c f2 dc 17
RSP: 0018:ffffc90003c07bb8 EFLAGS: 00010286
RAX: ec490cf5c114aa00 RBX: 0000000000000048 RCX: 0000000000000000
RDX: ffffffff99d16120 RSI: ffffffff8c92a180 RDI: 0000000000000048
RBP: 0000000000000000 R08: 0000000000000003 R09: 0000000000000000
R10: ffffed102e5d7800 R11: fffffbfff1efa3cf R12: dffffc0000000000
R13: ffff8881bdeb3000 R14: ffffffff99d16120 R15: ffffffff8c92a180
FS:  00005555729bd500(0000) GS:ffff8882a9f31000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000058 CR3: 0000000172faa000 CR4: 00000000000006f0
----------------
Code disassembly (best guess):
   0:   90                      nop
   1:   90                      nop
   2:   90                      nop
   3:   90                      nop
   4:   90                      nop
   5:   90                      nop
   6:   90                      nop
   7:   90                      nop
   8:   90                      nop
   9:   90                      nop
   a:   90                      nop
   b:   90                      nop
   c:   f3 0f 1e fa             endbr64
  10:   55                      push   %rbp
  11:   41 56                   push   %r14
  13:   53                      push   %rbx
  14:   48 83 ec 10             sub    $0x10,%rsp
  18:   89 cd                   mov    %ecx,%ebp
  1a:   48 89 fb                mov    %rdi,%rbx
  1d:   65 48 8b 05 67 af d1    mov    %gs:0x10d1af67(%rip),%rax        # 0x10d1af8c
  24:   10
  25:   48 89 44 24 08          mov    %rax,0x8(%rsp)
* 2a:   48 c7 47 10 00 00 00    movq   $0x0,0x10(%rdi) <-- trapping instruction
  31:   00
  32:   48 c7 47 08 00 00 00    movq   $0x0,0x8(%rdi)
  39:   00
  3a:   8b 05 8c f2 dc 17       mov    0x17dcf28c(%rip),%eax        # 0x17dcf2cc

***

If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:

---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].

author
#3 lore raw
Quoted 29 lines · Jason Xing
Quoted 17 lines · syzbot ci

syzbot ci has tested the following series

[v4] xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path
https://lore.kernel.org/all/[email protected]
* [PATCH RFC net-next v4] xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path

and found the following issue:

BUG: unable to handle kernel NULL pointer dereference in xp_create_and_assign_umem

Full report is available here:
https://ci.syzbot.org/series/d7e166a7-a880-4ea1-9707-8889afd4ebe8

***

BUG: unable to handle kernel NULL pointer dereference in xp_create_and_assign_umem
Quoted 8 lines
tree:      net-next
URL:       https://kernel.googlesource.com/pub/scm/linux/kernel/git/netdev/net-next.git
base:      0177f0f07886e54e12c6f18fa58f63e63ddd3c58
arch:      amd64
compiler:  Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8
config:    https://ci.syzbot.org/builds/d327cc4b-7471-413b-b244-519c6d16d43b/config
C repro:   https://ci.syzbot.org/findings/c8f7aeaf-0e2e-43dd-ae9c-ea2dd8db8d34/c_repro
syz repro: https://ci.syzbot.org/findings/c8f7aeaf-0e2e-43dd-ae9c-ea2dd8db8d34/syz_repro
Quoted 1 line

Interesting. Only rx logic is initialized while tx not. That causes

Actually, it should not happen or we can say the usage is against what we expect because at the first time cq and fq need to be both initialized and then the shared umem function can be used[1]. But the repro tries to directly use the shared umem at the first time.

[1]:
net/xdp/xsk.c +1401
Corresponding patches can be seen here:
commit 7361f9c3d71 ("xsk: Move fill and completion rings to buffer pool") commit fe2308328cd2f ("xsk: add umem completion queue support and mmap")

The above two patches make sure that in xdp_umem_validate_queues() (which was renamed to xsk_validate_queues()) doesn't allow _only one_ of them to be initialized at the first time. To put it in a simpler way, cq and fq are required to be initialized. So now what I have in my mind is can we have the same test when using XDP_SHARED_UMEM?

I think this repro found an existing bug before this patch, IIUC.

Thanks,
Jason