Skip to contentSkip to conversation

net: use READ_ONCE/WRITE_ONCE for ring index accesses

PATCH Applied Jakub Kicinski 6 participants 10 messages Apr 12, 2023 – Apr 13, 2023

Subsystems (8)
lore
Share & tools
lore
author maintainer
#1 lore raw

PATCH v2 1/3 net: docs: update the sample code in driver.rst

In reply to Jakub Kicinski

The sample code talks about single-queue devices and uses locks. Update it to something resembling more modern code. Make sure we mention use of READ_ONCE() / WRITE_ONCE().

Change the comment which talked about consumer on the xmit side. AFAIU xmit is the producer and completions are a consumer.

---
CC: [email protected]
CC: [email protected]

---
 Documentation/networking/driver.rst | 61 +++++++++++++----------------
 1 file changed, 27 insertions(+), 34 deletions(-)
Documentation/networking/driver.rst +27 -34
index 4071f2c00f8b..4f5dfa9c022e 100644
@@ -47,30 +47,43 @@ Instead it must maintain the queue properly.  For example,
 
 .. code-block:: c
 
+	static u32 drv_tx_avail(struct drv_ring *dr)
+	{
+		u32 used = READ_ONCE(dr->prod) - READ_ONCE(dr->cons);
+
+		return dr->tx_ring_size - (used & bp->tx_ring_mask);
+	}
+
 	static netdev_tx_t drv_hard_start_xmit(struct sk_buff *skb,
 					       struct net_device *dev)
 	{
 		struct drv *dp = netdev_priv(dev);
+		struct netdev_queue *txq;
+		struct drv_ring *dr;
+		int idx;
+
+		idx = skb_get_queue_mapping(skb);
+		dr = dp->tx_rings[idx];
+		txq = netdev_get_tx_queue(dev, idx);
 
-		lock_tx(dp);
 		//...
-		/* This is a hard error log it. */
-		if (TX_BUFFS_AVAIL(dp) <= (skb_shinfo(skb)->nr_frags + 1)) {
+		/* This should be a very rare race - log it. */
+		if (drv_tx_avail(dr) <= skb_shinfo(skb)->nr_frags + 1) {
 			netif_stop_queue(dev);
-			unlock_tx(dp);
-			printk(KERN_ERR PFX "%s: BUG! Tx Ring full when queue awake!\n",
-			       dev->name);
+			netdev_warn(dev, "Tx Ring full when queue awake!\n");
 			return NETDEV_TX_BUSY;
 		}
 
 		//... queue packet to card ...
-		//... update tx consumer index ...
 
-		if (TX_BUFFS_AVAIL(dp) <= (MAX_SKB_FRAGS + 1))
-			netif_stop_queue(dev);
+		netdev_tx_sent_queue(txq, skb->len);
+
+		//... update tx producer index using WRITE_ONCE() ...
+
+		if (!netif_txq_maybe_stop(txq, drv_tx_avail(dr),
+					  MAX_SKB_FRAGS + 1, 2 * MAX_SKB_FRAGS))
+			dr->stats.stopped++;
 
-		//...
-		unlock_tx(dp);
 		//...
 		return NETDEV_TX_OK;
 	}
@@ -79,30 +92,10 @@ Instead it must maintain the queue properly.  For example,
 
 .. code-block:: c
 
-	if (netif_queue_stopped(dp->dev) &&
-	    TX_BUFFS_AVAIL(dp) > (MAX_SKB_FRAGS + 1))
-		netif_wake_queue(dp->dev);
-
-For a non-scatter-gather supporting card, the three tests simply become:
-
-.. code-block:: c
-
-		/* This is a hard error log it. */
-		if (TX_BUFFS_AVAIL(dp) <= 0)
-
-and:
-
-.. code-block:: c
-
-		if (TX_BUFFS_AVAIL(dp) == 0)
-
-and:
-
-.. code-block:: c
+	//... update tx consumer index using WRITE_ONCE() ...
 
-	if (netif_queue_stopped(dp->dev) &&
-	    TX_BUFFS_AVAIL(dp) > 0)
-		netif_wake_queue(dp->dev);
+	netif_txq_completed_wake(txq, cmpl_pkts, cmpl_bytes,
+				 drv_tx_avail(dr), 2 * MAX_SKB_FRAGS);
 
 Lockless queue stop / wake helper macros
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Signature · 1 line
2.39.2

Jakub Kicinski

author maintainer
#0 lore raw

v2 cover net: use READ_ONCE/WRITE_ONCE for ring index accesses

Small follow up to the lockless ring stop/start macros. Update the doc and the drivers suggested by Eric:
https://lore.kernel.org/all/CANn89iJrBGSybMX1FqrhCEMWT3Nnz2=2+aStsbbwpWzKHjk51g@mail.gmail.com/

v2:
- convert xdp paths in bnxt as well
v1: https://lore.kernel.org/all/[email protected]/

Jakub Kicinski (3):
net: docs: update the sample code in driver.rst
bnxt: use READ_ONCE/WRITE_ONCE for ring indexes
mlx4: use READ_ONCE/WRITE_ONCE for ring indexes

 Documentation/networking/driver.rst           | 61 ++++++++-----------
 drivers/net/ethernet/broadcom/bnxt/bnxt.c     |  6 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |  9 ++-
 drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c |  6 +-
 drivers/net/ethernet/mellanox/mlx4/en_tx.c    |  8 ++-
 5 files changed, 42 insertions(+), 48 deletions(-)
Signature · 1 line
2.39.2
author maintainer
#7 lore raw

PATCH v2 3/3 mlx4: use READ_ONCE/WRITE_ONCE for ring indexes

Eric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.

---
CC: [email protected]
CC: [email protected]

---
 drivers/net/ethernet/mellanox/mlx4/en_tx.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx4/en_tx.c +5 -3
index 2f79378fbf6e..65cb63f6c465 100644
@@ -228,7 +228,9 @@ void mlx4_en_deactivate_tx_ring(struct mlx4_en_priv *priv,
 
 static inline bool mlx4_en_is_tx_ring_full(struct mlx4_en_tx_ring *ring)
 {
-	return ring->prod - ring->cons > ring->full_size;
+	u32 used = READ_ONCE(ring->prod) - READ_ONCE(ring->cons);
+
+	return used > ring->full_size;
 }
 
 static void mlx4_en_stamp_wqe(struct mlx4_en_priv *priv,
@@ -1083,7 +1085,7 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
 			op_own |= cpu_to_be32(MLX4_WQE_CTRL_IIP);
 	}
 
-	ring->prod += nr_txbb;
+	WRITE_ONCE(ring->prod, ring->prod + nr_txbb);
 
 	/* If we used a bounce buffer then copy descriptor back into place */
 	if (unlikely(bounce))
@@ -1214,7 +1216,7 @@ netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring,
 
 	rx_ring->xdp_tx++;
 
-	ring->prod += MLX4_EN_XDP_TX_NRTXBB;
+	WRITE_ONCE(ring->prod, ring->prod + MLX4_EN_XDP_TX_NRTXBB);
 
 	/* Ensure new descriptor hits memory
 	 * before setting ownership of this descriptor to HW
Signature · 1 line
2.39.2
author maintainer
#2 lore raw

PATCH v2 2/3 bnxt: use READ_ONCE/WRITE_ONCE for ring indexes

In reply to Jakub Kicinski

Eric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.

---
v2:
- cover writes in bnxt_xdp.c
v1: https://lore.kernel.org/all/[email protected]/
---
CC: [email protected]

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c     | 6 +++---
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     | 9 ++++-----
 drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c | 6 +++---
 3 files changed, 10 insertions(+), 11 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +3 -3
index f7602d8d79e3..92289ab2f34a 100644
@@ -472,7 +472,7 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev)
 		prod = NEXT_TX(prod);
 		tx_push->doorbell =
 			cpu_to_le32(DB_KEY_TX_PUSH | DB_LONG_TX_PUSH | prod);
-		txr->tx_prod = prod;
+		WRITE_ONCE(txr->tx_prod, prod);
 
 		tx_buf->is_push = 1;
 		netdev_tx_sent_queue(txq, skb->len);
@@ -583,7 +583,7 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev)
 	wmb();
 
 	prod = NEXT_TX(prod);
-	txr->tx_prod = prod;
+	WRITE_ONCE(txr->tx_prod, prod);
 
 	if (!netdev_xmit_more() || netif_xmit_stopped(txq))
 		bnxt_txr_db_kick(bp, txr, prod);
@@ -688,7 +688,7 @@ static void bnxt_tx_int(struct bnxt *bp, struct bnxt_napi *bnapi, int nr_pkts)
 		dev_kfree_skb_any(skb);
 	}
 
-	txr->tx_cons = cons;
+	WRITE_ONCE(txr->tx_cons, cons);
 
 	__netif_txq_completed_wake(txq, nr_pkts, tx_bytes,
 				   bnxt_tx_avail(bp, txr), bp->tx_wake_thresh,
drivers/net/ethernet/broadcom/bnxt/bnxt.h +4 -5
index 18cac98ba58e..080e73496066 100644
@@ -2231,13 +2231,12 @@ struct bnxt {
 #define SFF_MODULE_ID_QSFP28			0x11
 #define BNXT_MAX_PHY_I2C_RESP_SIZE		64
 
-static inline u32 bnxt_tx_avail(struct bnxt *bp, struct bnxt_tx_ring_info *txr)
+static inline u32 bnxt_tx_avail(struct bnxt *bp,
+				const struct bnxt_tx_ring_info *txr)
 {
-	/* Tell compiler to fetch tx indices from memory. */
-	barrier();
+	u32 used = READ_ONCE(txr->tx_prod) - READ_ONCE(txr->tx_cons);
 
-	return bp->tx_ring_size -
-		((txr->tx_prod - txr->tx_cons) & bp->tx_ring_mask);
+	return bp->tx_ring_size - (used & bp->tx_ring_mask);
 }
 
 static inline void bnxt_writeq(struct bnxt *bp, u64 val,
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +3 -3
index 5843c93b1711..4efa5fe6972b 100644
@@ -64,7 +64,7 @@ struct bnxt_sw_tx_bd *bnxt_xmit_bd(struct bnxt *bp,
 		int frag_len;
 
 		prod = NEXT_TX(prod);
-		txr->tx_prod = prod;
+		WRITE_ONCE(txr->tx_prod, prod);
 
 		/* first fill up the first buffer */
 		frag_tx_buf = &txr->tx_buf_ring[prod];
@@ -94,7 +94,7 @@ struct bnxt_sw_tx_bd *bnxt_xmit_bd(struct bnxt *bp,
 	/* Sync TX BD */
 	wmb();
 	prod = NEXT_TX(prod);
-	txr->tx_prod = prod;
+	WRITE_ONCE(txr->tx_prod, prod);
 
 	return tx_buf;
 }
@@ -161,7 +161,7 @@ void bnxt_tx_int_xdp(struct bnxt *bp, struct bnxt_napi *bnapi, int nr_pkts)
 		}
 		tx_cons = NEXT_TX(tx_cons);
 	}
-	txr->tx_cons = tx_cons;
+	WRITE_ONCE(txr->tx_cons, tx_cons);
 	if (rx_doorbell_needed) {
 		tx_buf = &txr->tx_buf_ring[last_tx_cons];
 		bnxt_db_write(bp, &rxr->rx_db, tx_buf->rx_prod);
Signature · 1 line
2.39.2
maintainer Reviewed-by on v2 2/3
#3 lore raw

Re: [PATCH net-next v2 2/3] bnxt: use READ_ONCE/WRITE_ONCE for ring indexes

Quoted 4 lines · Jakub Kicinski

Eric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.

Quoted 8 lines

---
v2:
- cover writes in bnxt_xdp.c
v1: https://lore.kernel.org/all/[email protected]/

Thanks.

Reviewed-by on v2 3/3
#8 lore raw

Re: [PATCH net-next v2 3/3] mlx4: use READ_ONCE/WRITE_ONCE for ring indexes

In reply to Jakub Kicinski
Quoted 7 lines · Jakub Kicinski

Eric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.

Quoted 7 lines

---
CC: [email protected]
CC: [email protected]

---
  drivers/net/ethernet/mellanox/mlx4/en_tx.c | 8 +++++---
  1 file changed, 5 insertions(+), 3 deletions(-)

Thanks!

#4 lore raw

RE: [PATCH net-next v2 2/3] bnxt: use READ_ONCE/WRITE_ONCE for ring indexes

In reply to Jakub Kicinski

From: Jakub Kicinski

Quoted 5 lines

Sent: 12 April 2023 02:51

Eric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.

...

drivers/net/ethernet/broadcom/bnxt/bnxt.h struct bnxt { · line 2240

Quoted 12 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.h +4 -5
-static inline u32 bnxt_tx_avail(struct bnxt *bp, struct bnxt_tx_ring_info *txr)
+static inline u32 bnxt_tx_avail(struct bnxt *bp,
+				const struct bnxt_tx_ring_info *txr)
 {
-	/* Tell compiler to fetch tx indices from memory. */
-	barrier();
+	u32 used = READ_ONCE(txr->tx_prod) - READ_ONCE(txr->tx_cons);
 
-	return bp->tx_ring_size -
-		((txr->tx_prod - txr->tx_cons) & bp->tx_ring_mask);
+	return bp->tx_ring_size - (used & bp->tx_ring_mask);
 }

Doesn't that function only make sense if only one of the ring index can be changing?
In this case I think this is being used in the transmit path so that 'tx_prod' is constant and is either already read or need not be read again.

Having written a lot of 'ring access' functions over the years if the ring size is a power of 2 I'd mask the 'tx_prod' value when it is being used rather than on the increment. (So the value just wraps modulo 2**32.)
This tends to make the code safer - especially since the 'ring full' and 'ring empty' conditions are different.

Also that code is masking with bp->tx_ring_mask, but the increments (in hunks I've chopped) use NEXT_TX(prod). If that is masking with bp->tx_ring_mask then 'bp' should be a parameter.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

maintainer
#5 lore raw

Re: [PATCH net-next v2 2/3] bnxt: use READ_ONCE/WRITE_ONCE for ring indexes

Quoted 33 lines · David Laight

From: Jakub Kicinski

Quoted 5 lines

Sent: 12 April 2023 02:51

Eric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.

...

drivers/net/ethernet/broadcom/bnxt/bnxt.h struct bnxt { · line 2240

Quoted 12 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.h +4 -5
-static inline u32 bnxt_tx_avail(struct bnxt *bp, struct bnxt_tx_ring_info *txr)
+static inline u32 bnxt_tx_avail(struct bnxt *bp,
+				const struct bnxt_tx_ring_info *txr)
 {
-	/* Tell compiler to fetch tx indices from memory. */
-	barrier();
+	u32 used = READ_ONCE(txr->tx_prod) - READ_ONCE(txr->tx_cons);
 
-	return bp->tx_ring_size -
-		((txr->tx_prod - txr->tx_cons) & bp->tx_ring_mask);
+	return bp->tx_ring_size - (used & bp->tx_ring_mask);
 }

Doesn't that function only make sense if only one of the ring index can be changing?
In this case I think this is being used in the transmit path so that 'tx_prod' is constant and is either already read or need not be read again.

Having written a lot of 'ring access' functions over the years if the ring size is a power of 2 I'd mask the 'tx_prod' value when it is being used rather than on the increment. (So the value just wraps modulo 2**32.)
This tends to make the code safer - especially since the 'ring full' and 'ring empty' conditions are different.

Quoted 4 lines

Also that code is masking with bp->tx_ring_mask, but the increments (in hunks I've chopped) use NEXT_TX(prod). If that is masking with bp->tx_ring_mask then 'bp' should be a parameter.

AFAICS bnxt_tx_avail() is also used in TX interrupt, outside tx path/tx lock.

I think all the above consideration are more suited for a driver refactor, while the current patch specifically address potential data race issues.

Cheers,

Paolo

#9 lore raw
In reply to Jakub Kicinski
This series was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:

Quoted 9 lines · On Tue, 11 Apr 2023 18:50:35 -0700 you wrote:

Small follow up to the lockless ring stop/start macros. Update the doc and the drivers suggested by Eric:
https://lore.kernel.org/all/CANn89iJrBGSybMX1FqrhCEMWT3Nnz2=2+aStsbbwpWzKHjk51g@mail.gmail.com/

v2:
- convert xdp paths in bnxt as well
v1: https://lore.kernel.org/all/[email protected]/

[...]

Here is the summary with links:
- [net-next,v2,1/3] net: docs: update the sample code in driver.rst

    https://git.kernel.org/netdev/net-next/c/50762d9af307
  - [net-next,v2,2/3] bnxt: use READ_ONCE/WRITE_ONCE for ring indexes
    https://git.kernel.org/netdev/net-next/c/36647b206c01
  - [net-next,v2,3/3] mlx4: use READ_ONCE/WRITE_ONCE for ring indexes
    https://git.kernel.org/netdev/net-next/c/9a714997386b

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
#6 lore raw

RE: [PATCH net-next v2 2/3] bnxt: use READ_ONCE/WRITE_ONCE for ring indexes

In reply to Paolo Abeni

From: Paolo Abeni

Quoted 29 lines

Sent: 13 April 2023 12:38

Quoted 26 lines · David Laight

From: Jakub Kicinski

Quoted 5 lines

Sent: 12 April 2023 02:51

Eric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.

...

drivers/net/ethernet/broadcom/bnxt/bnxt.h struct bnxt { · line 2240

Quoted 12 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.h +4 -5
-static inline u32 bnxt_tx_avail(struct bnxt *bp, struct bnxt_tx_ring_info *txr)
+static inline u32 bnxt_tx_avail(struct bnxt *bp,
+				const struct bnxt_tx_ring_info *txr)
 {
-	/* Tell compiler to fetch tx indices from memory. */
-	barrier();
+	u32 used = READ_ONCE(txr->tx_prod) - READ_ONCE(txr->tx_cons);
 
-	return bp->tx_ring_size -
-		((txr->tx_prod - txr->tx_cons) & bp->tx_ring_mask);
+	return bp->tx_ring_size - (used & bp->tx_ring_mask);
 }

Doesn't that function only make sense if only one of the ring index can be changing?
In this case I think this is being used in the transmit path so that 'tx_prod' is constant and is either already read or need not be read again.

...

Quoted 3 lines

AFAICS bnxt_tx_avail() is also used in TX interrupt, outside tx path/tx lock.

In which case both tx_prod and tx_cons are subject to possible updates. It is even possible that the two values have absolutely no relation to each other, it requires some unusual circumstances, but isn't impossible.
- A high priority interrupt (eg x86 SMM mode) could separate the READ_ONCE().
- Transmit setup will increase tx_prod.
- End of transmit 'reap' often done by other code paths (like rx processing or tx setup) can change tx_cons.
So not only is the value immediately stale, it can be just plain wrong.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)