xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path
Subsystems (4)
RFC PATCH v4 1/1 xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path
From: Jason Xing <[email protected]>
We (Paolo and I) noticed that in the sending path touching an extra cacheline due to cq_cached_prod_lock will impact the performance. After moving the lock from struct xsk_buff_pool to struct xsk_queue, the performance is increased by ~5% which can be observed by xdpsock.
An alternative approach [1] can be using atomic_try_cmpxchg() to have the same effect. But unfortunately I don't have evident performance number to prove the atomic approach is better than the current patch. The advantage is to save the contention time among multiple xsks sharing the same pool while the disadvantage is lose good maintenance. The full discussion can be found at the following link.
[1]: https://lore.kernel.org/all/[email protected]/
---
Q: since net-next will be open next year, I wonder if I should post this patch targetting bpf-next?
RFC V4
1. use moving lock method instead (Paolo, Magnus)
2. Add credit to Paolo, thanks!
v3
1. fix one race issue that cannot be resolved by simple seperated atomic operations. So this revision only updates patch [2/3] and tries to use try_cmpxchg method to avoid that problem. (paolo)
2. update commit log accordingly.
V2
1. use separate functions rather than branches within shared routines. (Maciej)
2. make each patch as simple as possible for easier review
--- include/net/xsk_buff_pool.h | 5 ----- net/xdp/xsk.c | 8 ++++---- net/xdp/xsk_buff_pool.c | 2 +- net/xdp/xsk_queue.h | 5 +++++ 4 files changed, 10 insertions(+), 10 deletions(-)
include/net/xsk_buff_pool.h +0 -5
@@ -90,11 +90,6 @@ struct xsk_buff_pool {* destructor callback. */ spinlock_t cq_prod_lock; -/* Mutual exclusion of the completion ring in the SKB mode. -* Protect: when sockets share a single cq when the same netdev -* and queue id is shared. -*/ -spinlock_t cq_cached_prod_lock;struct xdp_buff_xsk *free_heads[]; };
net/xdp/xsk.c +4 -4
@@ -543,9 +543,9 @@ static int xsk_cq_reserve_locked(struct xsk_buff_pool *pool) {int ret; -spin_lock (&pool->cq_cached_prod_lock); +spin_lock (&pool->cq->cq_cached_prod_lock); ret =xskq_prod_reserve (pool->cq); -spin_unlock (&pool->cq_cached_prod_lock); +spin_unlock (&pool->cq->cq_cached_prod_lock);return ret; }
@@ -619,9 +619,9 @@ static void xsk_cq_submit_addr_locked(struct xsk_buff_pool *pool,static void xsk_cq_cancel_locked (struct xsk_buff_pool *pool,u32 n) { -spin_lock (&pool->cq_cached_prod_lock); +spin_lock (&pool->cq->cq_cached_prod_lock);xskq_prod_cancel_n (pool->cq, n); -spin_unlock (&pool->cq_cached_prod_lock); +spin_unlock (&pool->cq->cq_cached_prod_lock); }INDIRECT_CALLABLE_SCOPE
net/xdp/xsk_buff_pool.c +1 -1
@@ -91,7 +91,7 @@ struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,INIT_LIST_HEAD (&pool->xsk_tx_list);spin_lock_init (&pool->xsk_tx_list_lock);spin_lock_init (&pool->cq_prod_lock); -spin_lock_init (&pool->cq_cached_prod_lock); +spin_lock_init (&xs->cq_tmp->cq_cached_prod_lock);refcount_set (&pool->users,1 ); pool->fq = xs->fq_tmp;
net/xdp/xsk_queue.h +5 -0
@@ -46,6 +46,11 @@ struct xsk_queue {u64 invalid_descs;u64 queue_empty_descs;size_t ring_vmalloc_size; +/* Mutual exclusion of the completion ring in the SKB mode. +* Protect: when sockets share a single cq when the same netdev +* and queue id is shared. +*/ +spinlock_t cq_cached_prod_lock; };struct parsed_desc {
Signature · 1 line
2.41.3
syzbot ci has tested the following series
syzbot ci has tested the following series
[v4] xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path
https://lore.kernel.org/all/[email protected]
* [PATCH RFC net-next v4] xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path
and found the following issue:
BUG: unable to handle kernel NULL pointer dereference in xp_create_and_assign_umem
Full report is available here:
https://ci.syzbot.org/series/d7e166a7-a880-4ea1-9707-8889afd4ebe8
***
BUG: unable to handle kernel NULL pointer dereference in xp_create_and_assign_umem
tree: net-next URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/netdev/net-next.git base: 0177f0f07886e54e12c6f18fa58f63e63ddd3c58 arch: amd64 compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 config: https://ci.syzbot.org/builds/d327cc4b-7471-413b-b244-519c6d16d43b/config C repro: https://ci.syzbot.org/findings/c8f7aeaf-0e2e-43dd-ae9c-ea2dd8db8d34/c_repro syz repro: https://ci.syzbot.org/findings/c8f7aeaf-0e2e-43dd-ae9c-ea2dd8db8d34/syz_repro
UDPLite6: UDP-Lite is deprecated and scheduled to be removed in 2025, please contact the netdev mailing list
BUG: kernel NULL pointer dereference, address: 0000000000000058 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page PGD 80000001b2496067 P4D 80000001b2496067 PUD 0 Oops: Oops: 0002 [#1] SMP KASAN PTI CPU: 1 UID: 0 PID: 5973 Comm: syz.0.17 Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:lockdep_init_map_type+0x1e/0x380 kernel/locking/lockdep.c:4944 Code: 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 55 41 56 53 48 83 ec 10 89 cd 48 89 fb 65 48 8b 05 67 af d1 10 48 89 44 24 08 <48> c7 47 10 00 00 00 00 48 c7 47 08 00 00 00 00 8b 05 8c f2 dc 17 RSP: 0018:ffffc90003c07bb8 EFLAGS: 00010286 RAX: ec490cf5c114aa00 RBX: 0000000000000048 RCX: 0000000000000000 RDX: ffffffff99d16120 RSI: ffffffff8c92a180 RDI: 0000000000000048 RBP: 0000000000000000 R08: 0000000000000003 R09: 0000000000000000 R10: ffffed102e5d7800 R11: fffffbfff1efa3cf R12: dffffc0000000000 R13: ffff8881bdeb3000 R14: ffffffff99d16120 R15: ffffffff8c92a180 FS: 00005555729bd500(0000) GS:ffff8882a9f31000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000058 CR3: 0000000172faa000 CR4: 00000000000006f0 Call Trace: <TASK> lockdep_init_map_waits include/linux/lockdep.h:135 [inline] lockdep_init_map_wait include/linux/lockdep.h:142 [inline] __raw_spin_lock_init+0x45/0x100 kernel/locking/spinlock_debug.c:25 xp_create_and_assign_umem+0x648/0xd40 net/xdp/xsk_buff_pool.c:94 xsk_bind+0x95a/0xf90 net/xdp/xsk.c:1355 __sys_bind_socket net/socket.c:1874 [inline] __sys_bind+0x2c6/0x3e0 net/socket.c:1905 __do_sys_bind net/socket.c:1910 [inline] __se_sys_bind net/socket.c:1908 [inline] __x64_sys_bind+0x7a/0x90 net/socket.c:1908 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0xfa/0xfa0 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f591eb8f7c9 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ffce6fcef48 EFLAGS: 00000246 ORIG_RAX: 0000000000000031 RAX: ffffffffffffffda RBX: 00007f591ede5fa0 RCX: 00007f591eb8f7c9 RDX: 0000000000000010 RSI: 0000200000000240 RDI: 0000000000000003 RBP: 00007f591ebf297f R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f591ede5fa0 R14: 00007f591ede5fa0 R15: 0000000000000003 </TASK> Modules linked in: CR2: 0000000000000058 ---[ end trace 0000000000000000 ]--- RIP: 0010:lockdep_init_map_type+0x1e/0x380 kernel/locking/lockdep.c:4944 Code: 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 55 41 56 53 48 83 ec 10 89 cd 48 89 fb 65 48 8b 05 67 af d1 10 48 89 44 24 08 <48> c7 47 10 00 00 00 00 48 c7 47 08 00 00 00 00 8b 05 8c f2 dc 17 RSP: 0018:ffffc90003c07bb8 EFLAGS: 00010286 RAX: ec490cf5c114aa00 RBX: 0000000000000048 RCX: 0000000000000000 RDX: ffffffff99d16120 RSI: ffffffff8c92a180 RDI: 0000000000000048 RBP: 0000000000000000 R08: 0000000000000003 R09: 0000000000000000 R10: ffffed102e5d7800 R11: fffffbfff1efa3cf R12: dffffc0000000000 R13: ffff8881bdeb3000 R14: ffffffff99d16120 R15: ffffffff8c92a180 FS: 00005555729bd500(0000) GS:ffff8882a9f31000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000058 CR3: 0000000172faa000 CR4: 00000000000006f0 ---------------- Code disassembly (best guess): 0: 90 nop 1: 90 nop 2: 90 nop 3: 90 nop 4: 90 nop 5: 90 nop 6: 90 nop 7: 90 nop 8: 90 nop 9: 90 nop a: 90 nop b: 90 nop c: f3 0f 1e fa endbr64 10: 55 push %rbp 11: 41 56 push %r14 13: 53 push %rbx 14: 48 83 ec 10 sub $0x10,%rsp 18: 89 cd mov %ecx,%ebp 1a: 48 89 fb mov %rdi,%rbx 1d: 65 48 8b 05 67 af d1 mov %gs:0x10d1af67(%rip),%rax # 0x10d1af8c 24: 10 25: 48 89 44 24 08 mov %rax,0x8(%rsp) * 2a: 48 c7 47 10 00 00 00 movq $0x0,0x10(%rdi) <-- trapping instruction 31: 00 32: 48 c7 47 08 00 00 00 movq $0x0,0x8(%rdi) 39: 00 3a: 8b 05 8c f2 dc 17 mov 0x17dcf28c(%rip),%eax # 0x17dcf2cc
***
If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:
---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].
Quoted 17 lines · syzbot ci
syzbot ci has tested the following series
[v4] xsk: move
cq_cached_prod_lockto avoid touching a cacheline in sending path
https://lore.kernel.org/all/[email protected]
* [PATCH RFC net-next v4] xsk: movecq_cached_prod_lockto avoid touching a cacheline in sending pathand found the following issue:
BUG: unable to handle kernel NULL pointer dereference in xp_create_and_assign_umemFull report is available here:
https://ci.syzbot.org/series/d7e166a7-a880-4ea1-9707-8889afd4ebe8***
BUG: unable to handle kernel NULL pointer dereference in xp_create_and_assign_umem
Quoted 8 lines
tree: net-next URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/netdev/net-next.git base: 0177f0f07886e54e12c6f18fa58f63e63ddd3c58 arch: amd64 compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 config: https://ci.syzbot.org/builds/d327cc4b-7471-413b-b244-519c6d16d43b/config C repro: https://ci.syzbot.org/findings/c8f7aeaf-0e2e-43dd-ae9c-ea2dd8db8d34/c_repro syz repro: https://ci.syzbot.org/findings/c8f7aeaf-0e2e-43dd-ae9c-ea2dd8db8d34/syz_repro
Interesting. Only rx logic is initialized while tx not. That causes the cq_tmp to be uninitialized.
Quoted 17 lines
UDPLite6: UDP-Lite is deprecated and scheduled to be removed in 2025, please contact the netdev mailing list
BUG: kernel NULL pointer dereference, address: 0000000000000058 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page PGD 80000001b2496067 P4D 80000001b2496067 PUD 0 Oops: Oops: 0002 [#1] SMP KASAN PTI CPU: 1 UID: 0 PID: 5973 Comm: syz.0.17 Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:lockdep_init_map_type+0x1e/0x380 kernel/locking/lockdep.c:4944 Code: 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 55 41 56 53 48 83 ec 10 89 cd 48 89 fb 65 48 8b 05 67 af d1 10 48 89 44 24 08 <48> c7 47 10 00 00 00 00 48 c7 47 08 00 00 00 00 8b 05 8c f2 dc 17 RSP: 0018:ffffc90003c07bb8 EFLAGS: 00010286 RAX: ec490cf5c114aa00 RBX: 0000000000000048 RCX: 0000000000000000 RDX: ffffffff99d16120 RSI: ffffffff8c92a180 RDI: 0000000000000048 RBP: 0000000000000000 R08: 0000000000000003 R09: 0000000000000000 R10: ffffed102e5d7800 R11: fffffbfff1efa3cf R12: dffffc0000000000 R13: ffff8881bdeb3000 R14: ffffffff99d16120 R15: ffffffff8c92a180
Quoted 8 lines
FS: 00005555729bd500(0000) GS:ffff8882a9f31000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000058 CR3: 0000000172faa000 CR4: 00000000000006f0 Call Trace: <TASK> lockdep_init_map_waits include/linux/lockdep.h:135 [inline] lockdep_init_map_wait include/linux/lockdep.h:142 [inline] __raw_spin_lock_init+0x45/0x100 kernel/locking/spinlock_debug.c:25
Using if-condition to check if cq_tmp is NULL would avoid the corruption.
Will add it in the next version.
Thanks,
Jason
Quoted 74 lines
xp_create_and_assign_umem+0x648/0xd40 net/xdp/xsk_buff_pool.c:94 xsk_bind+0x95a/0xf90 net/xdp/xsk.c:1355 __sys_bind_socket net/socket.c:1874 [inline] __sys_bind+0x2c6/0x3e0 net/socket.c:1905 __do_sys_bind net/socket.c:1910 [inline] __se_sys_bind net/socket.c:1908 [inline] __x64_sys_bind+0x7a/0x90 net/socket.c:1908 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0xfa/0xfa0 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f591eb8f7c9 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ffce6fcef48 EFLAGS: 00000246 ORIG_RAX: 0000000000000031 RAX: ffffffffffffffda RBX: 00007f591ede5fa0 RCX: 00007f591eb8f7c9 RDX: 0000000000000010 RSI: 0000200000000240 RDI: 0000000000000003 RBP: 00007f591ebf297f R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f591ede5fa0 R14: 00007f591ede5fa0 R15: 0000000000000003 </TASK> Modules linked in: CR2: 0000000000000058 ---[ end trace 0000000000000000 ]--- RIP: 0010:lockdep_init_map_type+0x1e/0x380 kernel/locking/lockdep.c:4944 Code: 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 55 41 56 53 48 83 ec 10 89 cd 48 89 fb 65 48 8b 05 67 af d1 10 48 89 44 24 08 <48> c7 47 10 00 00 00 00 48 c7 47 08 00 00 00 00 8b 05 8c f2 dc 17 RSP: 0018:ffffc90003c07bb8 EFLAGS: 00010286 RAX: ec490cf5c114aa00 RBX: 0000000000000048 RCX: 0000000000000000 RDX: ffffffff99d16120 RSI: ffffffff8c92a180 RDI: 0000000000000048 RBP: 0000000000000000 R08: 0000000000000003 R09: 0000000000000000 R10: ffffed102e5d7800 R11: fffffbfff1efa3cf R12: dffffc0000000000 R13: ffff8881bdeb3000 R14: ffffffff99d16120 R15: ffffffff8c92a180 FS: 00005555729bd500(0000) GS:ffff8882a9f31000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000058 CR3: 0000000172faa000 CR4: 00000000000006f0 ---------------- Code disassembly (best guess): 0: 90 nop 1: 90 nop 2: 90 nop 3: 90 nop 4: 90 nop 5: 90 nop 6: 90 nop 7: 90 nop 8: 90 nop 9: 90 nop a: 90 nop b: 90 nop c: f3 0f 1e fa endbr64 10: 55 push %rbp 11: 41 56 push %r14 13: 53 push %rbx 14: 48 83 ec 10 sub $0x10,%rsp 18: 89 cd mov %ecx,%ebp 1a: 48 89 fb mov %rdi,%rbx 1d: 65 48 8b 05 67 af d1 mov %gs:0x10d1af67(%rip),%rax # 0x10d1af8c 24: 10 25: 48 89 44 24 08 mov %rax,0x8(%rsp) * 2a: 48 c7 47 10 00 00 00 movq $0x0,0x10(%rdi) <-- trapping instruction 31: 00 32: 48 c7 47 08 00 00 00 movq $0x0,0x8(%rdi) 39: 00 3a: 8b 05 8c f2 dc 17 mov 0x17dcf28c(%rip),%eax # 0x17dcf2cc***
If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:
Tested-by [email protected]---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].
Quoted 29 lines · Jason Xing
Quoted 17 lines · syzbot ci
syzbot ci has tested the following series
[v4] xsk: move
cq_cached_prod_lockto avoid touching a cacheline in sending path
https://lore.kernel.org/all/[email protected]
* [PATCH RFC net-next v4] xsk: movecq_cached_prod_lockto avoid touching a cacheline in sending pathand found the following issue:
BUG: unable to handle kernel NULL pointer dereference in xp_create_and_assign_umemFull report is available here:
https://ci.syzbot.org/series/d7e166a7-a880-4ea1-9707-8889afd4ebe8***
BUG: unable to handle kernel NULL pointer dereference in xp_create_and_assign_umemQuoted 8 lines
tree: net-next URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/netdev/net-next.git base: 0177f0f07886e54e12c6f18fa58f63e63ddd3c58 arch: amd64 compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 config: https://ci.syzbot.org/builds/d327cc4b-7471-413b-b244-519c6d16d43b/config C repro: https://ci.syzbot.org/findings/c8f7aeaf-0e2e-43dd-ae9c-ea2dd8db8d34/c_repro syz repro: https://ci.syzbot.org/findings/c8f7aeaf-0e2e-43dd-ae9c-ea2dd8db8d34/syz_repro
Quoted 1 line
Interesting. Only rx logic is initialized while tx not. That causes
Actually, it should not happen or we can say the usage is against what we expect because at the first time cq and fq need to be both initialized and then the shared umem function can be used[1]. But the repro tries to directly use the shared umem at the first time.
[1]:net/xdp/xsk.c +1401
Corresponding patches can be seen here:
commit 7361f9c3d71 ("xsk: Move fill and completion rings to buffer pool") commit fe2308328cd2f ("xsk: add umem completion queue support and mmap")
The above two patches make sure that in xdp_umem_validate_queues() (which was renamed to xsk_validate_queues()) doesn't allow _only one_ of them to be initialized at the first time. To put it in a simpler way, cq and fq are required to be initialized. So now what I have in my mind is can we have the same test when using XDP_SHARED_UMEM?
I think this repro found an existing bug before this patch, IIUC.
Thanks,
Jason