Skip to contentSkip to conversation

xsk: Fix race condition in AF_XDP generic RX path

PATCH Applied e.kubanski 3 participants 3 messages Apr 16, 2025 – Apr 25, 2025 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): CGME20250416101926eucas1p193c52e72b20321605905f1c465c8ac06@eucas1p1.samsung.com

e.kubanski

author
#1 lore raw

PATCH v2 1/1 xsk: Fix race condition in AF_XDP generic RX path

Move rx_lock from xsk_socket to xsk_buff_pool.
Fix synchronization for shared umem mode in
generic RX path where multiple sockets share
single xsk_buff_pool.

RX queue is exclusive to xsk_socket, while FILL
queue can be shared between multiple sockets.
This could result in race condition where two
CPU cores access RX path of two different sockets
sharing the same umem.

Protect both queues by acquiring spinlock in shared xsk_buff_pool.

Lock contention may be minimized in the future by some per-thread FQ buffering.

It's safe and necessary to move spin_lock_bh(rx_lock) after xsk_rcv_check():
* xs->pool and spinlock_init is synchronized by xsk_bind() -> xsk_is_bound() memory barriers.
* xsk_rcv_check() may return true at the moment of xsk_release() or xsk_unbind_dev(), however this will not cause any data races or race conditions. xsk_unbind_dev() removes xdp socket from all maps and waits for completion of all outstanding rx operations. Packets in RX path will either complete safely or drop.

Fixes bf0bdd1343efb ("xdp: fix race on generic receive path")
---
 include/net/xdp_sock.h      | 3 ---
 include/net/xsk_buff_pool.h | 2 ++
 net/xdp/xsk.c               | 6 +++---
 net/xdp/xsk_buff_pool.c     | 1 +
 4 files changed, 6 insertions(+), 6 deletions(-)
include/net/xdp_sock.h +0 -3
index bfe625b55d55..df3f5f07bc7c 100644
@@ -71,9 +71,6 @@ struct xdp_sock {
 	 */
 	u32 tx_budget_spent;
 
-	/* Protects generic receive. */
-	spinlock_t rx_lock;
-
 	/* Statistics */
 	u64 rx_dropped;
 	u64 rx_queue_full;
include/net/xsk_buff_pool.h +2 -0
index 50779406bc2d..7f0a75d6563d 100644
@@ -53,6 +53,8 @@ struct xsk_buff_pool {
 	refcount_t users;
 	struct xdp_umem *umem;
 	struct work_struct work;
+	/* Protects generic receive in shared and non-shared umem mode. */
+	spinlock_t rx_lock;
 	struct list_head free_list;
 	struct list_head xskb_list;
 	u32 heads_cnt;
net/xdp/xsk.c +3 -3
index 89d2bef96469..e2a75f3be237 100644
@@ -337,13 +337,14 @@ int xsk_generic_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
 	u32 len = xdp_get_buff_len(xdp);
 	int err;
 
-	spin_lock_bh(&xs->rx_lock);
 	err = xsk_rcv_check(xs, xdp, len);
 	if (!err) {
+		spin_lock_bh(&xs->pool->rx_lock);
 		err = __xsk_rcv(xs, xdp, len);
 		xsk_flush(xs);
+		spin_unlock_bh(&xs->pool->rx_lock);
 	}
-	spin_unlock_bh(&xs->rx_lock);
+
 	return err;
 }
 
@@ -1724,7 +1725,6 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol,
 	xs = xdp_sk(sk);
 	xs->state = XSK_READY;
 	mutex_init(&xs->mutex);
-	spin_lock_init(&xs->rx_lock);
 
 	INIT_LIST_HEAD(&xs->map_list);
 	spin_lock_init(&xs->map_list_lock);
net/xdp/xsk_buff_pool.c +1 -0
index 1f7975b49657..3a5f16f53178 100644
@@ -87,6 +87,7 @@ struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,
 	pool->addrs = umem->addrs;
 	pool->tx_metadata_len = umem->tx_metadata_len;
 	pool->tx_sw_csum = umem->flags & XDP_UMEM_TX_SW_CSUM;
+	spin_lock_init(&pool->rx_lock);
 	INIT_LIST_HEAD(&pool->free_list);
 	INIT_LIST_HEAD(&pool->xskb_list);
 	INIT_LIST_HEAD(&pool->xsk_tx_list);
Signature · 1 line
2.34.1
Acked-by on v2 1/1
#2 lore raw
Quoted 18 lines · e.kubanski

Move rx_lock from xsk_socket to xsk_buff_pool.
Fix synchronization for shared umem mode in
generic RX path where multiple sockets share
single xsk_buff_pool.

RX queue is exclusive to xsk_socket, while FILL
queue can be shared between multiple sockets.
This could result in race condition where two
CPU cores access RX path of two different sockets
sharing the same umem.

Protect both queues by acquiring spinlock in shared xsk_buff_pool.

Lock contention may be minimized in the future by some per-thread FQ buffering.

Quoted 11 lines

It's safe and necessary to move spin_lock_bh(rx_lock) after xsk_rcv_check():
* xs->pool and spinlock_init is synchronized by xsk_bind() -> xsk_is_bound() memory barriers.
* xsk_rcv_check() may return true at the moment of xsk_release() or xsk_unbind_dev(), however this will not cause any data races or race conditions. xsk_unbind_dev() removes xdp socket from all maps and waits for completion of all outstanding rx operations. Packets in RX path will either complete safely or drop.

Thanks Eryk.

Quoted 81 lines
Fixes bf0bdd1343efb ("xdp: fix race on generic receive path")
---
 include/net/xdp_sock.h      | 3 ---
 include/net/xsk_buff_pool.h | 2 ++
 net/xdp/xsk.c               | 6 +++---
 net/xdp/xsk_buff_pool.c     | 1 +
 4 files changed, 6 insertions(+), 6 deletions(-)
include/net/xdp_sock.h +0 -3
index bfe625b55d55..df3f5f07bc7c 100644
@@ -71,9 +71,6 @@ struct xdp_sock {
         */
        u32 tx_budget_spent;
 
-       /* Protects generic receive. */
-       spinlock_t rx_lock;
-
        /* Statistics */
        u64 rx_dropped;
        u64 rx_queue_full;
include/net/xsk_buff_pool.h +2 -0
index 50779406bc2d..7f0a75d6563d 100644
@@ -53,6 +53,8 @@ struct xsk_buff_pool {
        refcount_t users;
        struct xdp_umem *umem;
        struct work_struct work;
+       /* Protects generic receive in shared and non-shared umem mode. */
+       spinlock_t rx_lock;
        struct list_head free_list;
        struct list_head xskb_list;
        u32 heads_cnt;
net/xdp/xsk.c +3 -3
index 89d2bef96469..e2a75f3be237 100644
@@ -337,13 +337,14 @@ int xsk_generic_rcv(struct xdp_sock *xs, struct xdp_buff *xdp)
        u32 len = xdp_get_buff_len(xdp);
        int err;
 
-       spin_lock_bh(&xs->rx_lock);
        err = xsk_rcv_check(xs, xdp, len);
        if (!err) {
+               spin_lock_bh(&xs->pool->rx_lock);
                err = __xsk_rcv(xs, xdp, len);
                xsk_flush(xs);
+               spin_unlock_bh(&xs->pool->rx_lock);
        }
-       spin_unlock_bh(&xs->rx_lock);
+
        return err;
 }
 
@@ -1724,7 +1725,6 @@ static int xsk_create(struct net *net, struct socket *sock, int protocol,
        xs = xdp_sk(sk);
        xs->state = XSK_READY;
        mutex_init(&xs->mutex);
-       spin_lock_init(&xs->rx_lock);
 
        INIT_LIST_HEAD(&xs->map_list);
        spin_lock_init(&xs->map_list_lock);
net/xdp/xsk_buff_pool.c +1 -1
index 1f7975b49657..3a5f16f53178 100644
@@ -87,6 +87,7 @@ struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs,
        pool->addrs = umem->addrs;
        pool->tx_metadata_len = umem->tx_metadata_len;
        pool->tx_sw_csum = umem->flags & XDP_UMEM_TX_SW_CSUM;
+       spin_lock_init(&pool->rx_lock);
        INIT_LIST_HEAD(&pool->free_list);
        INIT_LIST_HEAD(&pool->xskb_list);
        INIT_LIST_HEAD(&pool->xsk_tx_list);
--

2.34.1

#3 lore raw
In reply to e.kubanski
This patch was applied to netdev/net.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <[email protected]>:

Quoted 5 lines · On Wed, 16 Apr 2025 12:19:08 +0200 you wrote:

Move rx_lock from xsk_socket to xsk_buff_pool. Fix synchronization for shared umem mode in generic RX path where multiple sockets share single xsk_buff_pool.

Quoted 7 lines

RX queue is exclusive to xsk_socket, while FILL queue can be shared between multiple sockets. This could result in race condition where two CPU cores access RX path of two different sockets sharing the same umem.

[...]

Here is the summary with links:
- [v2,bpf] xsk: Fix race condition in AF_XDP generic RX path

    https://git.kernel.org/netdev/net/c/a1356ac7749c

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html