xsk: move cq_cached_prod_lock
Subsystems (4)
v6 cover xsk: move cq_cached_prod_lock
From: Jason Xing <[email protected]>
Move cq_cached_prod_lock to avoid touching new cacheline.
---
V6
1. only rebase
RFC V5
1. From what I lately know from the repro at the above link, application can use the shared umem mode directly but the kernel will eventually return error that is reflected in the xp_assign_dev_shared(). Advancing the check can avoid the crash in patch [1/2] and be good to avoid unnecessary memory allocation.
RFC V4
1. use moving lock method instead (Paolo, Magnus)
2. Add credit to Paolo, thanks!
v3
1. fix one race issue that cannot be resolved by simple seperated atomic operations. So this revision only updates patch [2/3] and tries to use try_cmpxchg method to avoid that problem. (paolo)
2. update commit log accordingly.
V2
1. use separate functions rather than branches within shared routines. (Maciej)
2. make each patch as simple as possible for easier review
Jason Xing (2):
xsk: advance cq/fq check when shared umem is used
xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending
path
include/net/xsk_buff_pool.h | 5 ----- net/xdp/xsk.c | 15 +++++++++++---- net/xdp/xsk_buff_pool.c | 6 +----- net/xdp/xsk_queue.h | 5 +++++ 4 files changed, 17 insertions(+), 14 deletions(-)
Signature · 1 line
2.41.3
PATCH v6 1/2 xsk: advance cq/fq check when shared umem is used
From: Jason Xing <[email protected]>
In the shared umem mode with different queues or devices, either uninitialized cq or fq is not allowed which was previously done in xp_assign_dev_shared(). The patch advances the check at the beginning so that 1) we can avoid a few memory allocation and stuff if cq or fq is NULL, 2) it can be regarded as preparation for the next patch in the series.
--- net/xdp/xsk.c | 7 +++++++ net/xdp/xsk_buff_pool.c | 4 ---- 2 files changed, 7 insertions(+), 4 deletions(-)
net/xdp/xsk.c +7 -0
@@ -1349,6 +1349,13 @@ static int xsk_bind(struct socket *sock, struct sockaddr_unsized *addr, int addr }if (umem_xs->queue_id != qid || umem_xs->dev != dev) { +/* One fill and completion ring required for each queue id. */ +if (!xsk_validate_queues (xs)) { + err = -EINVAL ; +sockfd_put (sock); + goto out_unlock; + } +/* Share the umem with another socket on another qid * and/or device. */
net/xdp/xsk_buff_pool.c +0 -4
@@ -247,10 +247,6 @@ int xp_assign_dev_shared(struct xsk_buff_pool *pool, struct xdp_sock *umem_xs,u16 flags;struct xdp_umem *umem = umem_xs->umem; -/* One fill and completion ring required for each queue id. */ -if (!pool->fq || !pool->cq) -return -EINVAL ; - flags = umem->zc ?XDP_ZEROCOPY :XDP_COPY ;if (umem_xs->pool->uses_need_wakeup) flags |=XDP_USE_NEED_WAKEUP ;
Signature · 1 line
2.41.3
PATCH v6 2/2 xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path
In reply to Jason XingFrom: Jason Xing <[email protected]>
We (Paolo and I) noticed that in the sending path touching an extra cacheline due to cq_cached_prod_lock will impact the performance. After moving the lock from struct xsk_buff_pool to struct xsk_queue, the performance is increased by ~5% which can be observed by xdpsock.
An alternative approach [1] can be using atomic_try_cmpxchg() to have the same effect. But unfortunately I don't have evident performance numbers to prove the atomic approach is better than the current patch. The advantage is to save the contention time among multiple xsks sharing the same pool while the disadvantage is losing good maintenance. The full discussion can be found at the following link.
[1]: https://lore.kernel.org/all/[email protected]/
--- include/net/xsk_buff_pool.h | 5 ----- net/xdp/xsk.c | 8 ++++---- net/xdp/xsk_buff_pool.c | 2 +- net/xdp/xsk_queue.h | 5 +++++ 4 files changed, 10 insertions(+), 10 deletions(-)
include/net/xsk_buff_pool.h +0 -5
@@ -90,11 +90,6 @@ struct xsk_buff_pool {* destructor callback. */ spinlock_t cq_prod_lock; -/* Mutual exclusion of the completion ring in the SKB mode. -* Protect: when sockets share a single cq when the same netdev -* and queue id is shared. -*/ -spinlock_t cq_cached_prod_lock;struct xdp_buff_xsk *free_heads[]; };
net/xdp/xsk.c +4 -4
@@ -543,9 +543,9 @@ static int xsk_cq_reserve_locked(struct xsk_buff_pool *pool) {int ret; -spin_lock (&pool->cq_cached_prod_lock); +spin_lock (&pool->cq->cq_cached_prod_lock); ret =xskq_prod_reserve (pool->cq); -spin_unlock (&pool->cq_cached_prod_lock); +spin_unlock (&pool->cq->cq_cached_prod_lock);return ret; }
@@ -619,9 +619,9 @@ static void xsk_cq_submit_addr_locked(struct xsk_buff_pool *pool,static void xsk_cq_cancel_locked (struct xsk_buff_pool *pool,u32 n) { -spin_lock (&pool->cq_cached_prod_lock); +spin_lock (&pool->cq->cq_cached_prod_lock);xskq_prod_cancel_n (pool->cq, n); -spin_unlock (&pool->cq_cached_prod_lock); +spin_unlock (&pool->cq->cq_cached_prod_lock); }INDIRECT_CALLABLE_SCOPE
net/xdp/xsk_buff_pool.c +1 -1
@@ -91,7 +91,7 @@ struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,INIT_LIST_HEAD (&pool->xsk_tx_list);spin_lock_init (&pool->xsk_tx_list_lock);spin_lock_init (&pool->cq_prod_lock); -spin_lock_init (&pool->cq_cached_prod_lock); +spin_lock_init (&xs->cq_tmp->cq_cached_prod_lock);refcount_set (&pool->users,1 ); pool->fq = xs->fq_tmp;
net/xdp/xsk_queue.h +5 -0
@@ -46,6 +46,11 @@ struct xsk_queue {u64 invalid_descs;u64 queue_empty_descs;size_t ring_vmalloc_size; +/* Mutual exclusion of the completion ring in the SKB mode. +* Protect: when sockets share a single cq when the same netdev +* and queue id is shared. +*/ +spinlock_t cq_cached_prod_lock; };struct parsed_desc {
Signature · 1 line
2.41.3
Paolo Abeni
maintainerRe: [PATCH net-next v6 2/2] xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path
net/xdp/xsk_buff_pool.c struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs, · line 95
Quoted 11 lines · Jason Xing
net/xdp/xsk_buff_pool.c +1
-1@@ -91,7 +91,7 @@ struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,INIT_LIST_HEAD (&pool->xsk_tx_list);spin_lock_init (&pool->xsk_tx_list_lock);spin_lock_init (&pool->cq_prod_lock); -spin_lock_init (&pool->cq_cached_prod_lock); +spin_lock_init (&xs->cq_tmp->cq_cached_prod_lock);refcount_set (&pool->users,1 );
Very minor nit: moving the init later, after:
pool->cq = xs->cq_tmp;
would avoid touching a '_tmp' field, which looks strange.
Please do not resubmit just for this. Waiting a little longer for Magnus, Maciej or Stanislav ack.
/P
Re: [PATCH net-next v6 2/2] xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path
Quoted 16 lines · Paolo Abeni
net/xdp/xsk_buff_pool.c struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs, · line 95
Quoted 11 lines · Jason Xing
net/xdp/xsk_buff_pool.c +1
-1@@ -91,7 +91,7 @@ struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,INIT_LIST_HEAD (&pool->xsk_tx_list);spin_lock_init (&pool->xsk_tx_list_lock);spin_lock_init (&pool->cq_prod_lock); -spin_lock_init (&pool->cq_cached_prod_lock); +spin_lock_init (&xs->cq_tmp->cq_cached_prod_lock);refcount_set (&pool->users,1 );Very minor nit: moving the init later, after:
Quoted 6 lines
pool->cq = xs->cq_tmp;would avoid touching a '
_tmp' field, which looks strange.Please do not resubmit just for this. Waiting a little longer for Magnus, Maciej or Stanislav ack.
Got it. Thanks for the review.
Thanks,
Jason
Quoted 3 lines
/P
Re: [PATCH net-next v6 2/2] xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path
In reply to Jason XingQuoted 15 lines · Jason Xing
From: Jason Xing <[email protected]>
We (Paolo and I) noticed that in the sending path touching an extra cacheline due to
cq_cached_prod_lockwill impact the performance. After moving the lock from structxsk_buff_poolto structxsk_queue, the performance is increased by ~5% which can be observed by xdpsock.An alternative approach [1] can be using
atomic_try_cmpxchg()to have the same effect. But unfortunately I don't have evident performance numbers to prove the atomic approach is better than the current patch. The advantage is to save the contention time among multiple xsks sharing the same pool while the disadvantage is losing good maintenance. The full discussion can be found at the following link.
Quoted 4 lines
[1]: https://lore.kernel.org/all/[email protected]/
Suggested-by Paolo Abeni <[email protected]>Signed-off-by Jason Xing <[email protected]>
Hi Magnus, Maciej and Stanislav,
Any feedback on the whole series?
Thanks,
Jason
Re: [PATCH net-next v6 2/2] xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path
Quoted 16 lines · Jason Xing
Quoted 15 lines · Jason Xing
From: Jason Xing <[email protected]>
We (Paolo and I) noticed that in the sending path touching an extra cacheline due to
cq_cached_prod_lockwill impact the performance. After moving the lock from structxsk_buff_poolto structxsk_queue, the performance is increased by ~5% which can be observed by xdpsock.An alternative approach [1] can be using
atomic_try_cmpxchg()to have the same effect. But unfortunately I don't have evident performance numbers to prove the atomic approach is better than the current patch. The advantage is to save the contention time among multiple xsks sharing the same pool while the disadvantage is losing good maintenance. The full discussion can be found at the following link.
Quoted 8 lines
Quoted 4 lines
[1]: https://lore.kernel.org/all/[email protected]/
Suggested-by Paolo Abeni <[email protected]>Signed-off-by Jason Xing <[email protected]>Hi Magnus, Maciej and Stanislav,
Any feedback on the whole series?
LGTM, thanks! (I'm gonna be a bit slow on the mailing list in Jan/Feb)
Quoted 8 lines · Jason Xing
From: Jason Xing <[email protected]>
Move
cq_cached_prod_lockto avoid touching new cacheline.---
V61. only rebase
This series was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Quoted 10 lines · On Sun, 4 Jan 2026 09:21:23 +0800 you wrote:
From: Jason Xing <[email protected]>
Move
cq_cached_prod_lockto avoid touching new cacheline.---
V61. only rebase
[...]
Here is the summary with links:
- [net-next,v6,1/2] xsk: advance cq/fq check when shared umem is used
https://git.kernel.org/netdev/net-next/c/cee715d907d0 - [net-next,v6,2/2] xsk: move cq_cached_prod_lock to avoid touching a cacheline in sending path https://git.kernel.org/netdev/net-next/c/a2cb2e23b2bc
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html