net: use READ_ONCE/WRITE_ONCE for ring index accesses
Subsystems (8)
Jakub Kicinski
maintainerv2 cover net: use READ_ONCE/WRITE_ONCE for ring index accesses
Small follow up to the lockless ring stop/start macros. Update the doc and the drivers suggested by Eric:
https://lore.kernel.org/all/CANn89iJrBGSybMX1FqrhCEMWT3Nnz2=2+aStsbbwpWzKHjk51g@mail.gmail.com/
v2:
- convert xdp paths in bnxt as well
v1: https://lore.kernel.org/all/[email protected]/
Jakub Kicinski (3):
net: docs: update the sample code in driver.rst
bnxt: use READ_ONCE/WRITE_ONCE for ring indexes
mlx4: use READ_ONCE/WRITE_ONCE for ring indexes
Documentation/networking/driver.rst | 61 ++++++++----------- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 6 +- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 9 ++- drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c | 6 +- drivers/net/ethernet/mellanox/mlx4/en_tx.c | 8 ++- 5 files changed, 42 insertions(+), 48 deletions(-)
Signature · 1 line
2.39.2
Jakub Kicinski
maintainerPATCH v2 1/3 net: docs: update the sample code in driver.rst
The sample code talks about single-queue devices and uses locks. Update it to something resembling more modern code. Make sure we mention use of READ_ONCE() / WRITE_ONCE().
Change the comment which talked about consumer on the xmit side. AFAIU xmit is the producer and completions are a consumer.
---
CC: [email protected]
CC: [email protected]
--- Documentation/networking/driver.rst | 61 +++++++++++++---------------- 1 file changed, 27 insertions(+), 34 deletions(-)
Documentation/networking/driver.rst +27 -34
@@ -47,30 +47,43 @@ Instead it must maintain the queue properly. For example, .. code-block:: c + static u32 drv_tx_avail(struct drv_ring *dr) + { + u32 used = READ_ONCE(dr->prod) - READ_ONCE(dr->cons); + + return dr->tx_ring_size - (used & bp->tx_ring_mask); + } + static netdev_tx_t drv_hard_start_xmit(struct sk_buff *skb, struct net_device *dev) { struct drv *dp = netdev_priv(dev); + struct netdev_queue *txq; + struct drv_ring *dr; + int idx; + + idx = skb_get_queue_mapping(skb); + dr = dp->tx_rings[idx]; + txq = netdev_get_tx_queue(dev, idx); - lock_tx(dp); //... - /* This is a hard error log it. */ - if (TX_BUFFS_AVAIL(dp) <= (skb_shinfo(skb)->nr_frags + 1)) { + /* This should be a very rare race - log it. */ + if (drv_tx_avail(dr) <= skb_shinfo(skb)->nr_frags + 1) { netif_stop_queue(dev); - unlock_tx(dp); - printk(KERN_ERR PFX "%s: BUG! Tx Ring full when queue awake!\n", - dev->name); + netdev_warn(dev, "Tx Ring full when queue awake!\n"); return NETDEV_TX_BUSY; } //... queue packet to card ... - //... update tx consumer index ... - if (TX_BUFFS_AVAIL(dp) <= (MAX_SKB_FRAGS + 1)) - netif_stop_queue(dev); + netdev_tx_sent_queue(txq, skb->len); + + //... update tx producer index using WRITE_ONCE() ... + + if (!netif_txq_maybe_stop(txq, drv_tx_avail(dr), + MAX_SKB_FRAGS + 1, 2 * MAX_SKB_FRAGS)) + dr->stats.stopped++; - //... - unlock_tx(dp); //... return NETDEV_TX_OK; }
@@ -79,30 +92,10 @@ Instead it must maintain the queue properly. For example, .. code-block:: c - if (netif_queue_stopped(dp->dev) && - TX_BUFFS_AVAIL(dp) > (MAX_SKB_FRAGS + 1)) - netif_wake_queue(dp->dev); - -For a non-scatter-gather supporting card, the three tests simply become: - -.. code-block:: c - - /* This is a hard error log it. */ - if (TX_BUFFS_AVAIL(dp) <= 0) - -and: - -.. code-block:: c - - if (TX_BUFFS_AVAIL(dp) == 0) - -and: - -.. code-block:: c + //... update tx consumer index using WRITE_ONCE() ... - if (netif_queue_stopped(dp->dev) && - TX_BUFFS_AVAIL(dp) > 0) - netif_wake_queue(dp->dev); + netif_txq_completed_wake(txq, cmpl_pkts, cmpl_bytes, + drv_tx_avail(dr), 2 * MAX_SKB_FRAGS); Lockless queue stop / wake helper macros ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Signature · 1 line
2.39.2
Jakub Kicinski
maintainerPATCH v2 2/3 bnxt: use READ_ONCE/WRITE_ONCE for ring indexes
In reply to Jakub KicinskiEric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.
---
v2:
- cover writes in bnxt_xdp.c
v1: https://lore.kernel.org/all/[email protected]/
---
CC: [email protected]
--- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 6 +++--- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 9 ++++----- drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c | 6 +++--- 3 files changed, 10 insertions(+), 11 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +3 -3
@@ -472,7 +472,7 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev) prod =NEXT_TX (prod); tx_push->doorbell =cpu_to_le32 (DB_KEY_TX_PUSH |DB_LONG_TX_PUSH | prod); - txr->tx_prod = prod; +WRITE_ONCE (txr->tx_prod, prod); tx_buf->is_push =1 ;netdev_tx_sent_queue (txq, skb->len);
@@ -583,7 +583,7 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev)wmb (); prod =NEXT_TX (prod); - txr->tx_prod = prod; +WRITE_ONCE (txr->tx_prod, prod);if (!netdev_xmit_more () ||netif_xmit_stopped (txq))bnxt_txr_db_kick (bp, txr, prod);
@@ -688,7 +688,7 @@ static void bnxt_tx_int(struct bnxt *bp, struct bnxt_napi *bnapi, int nr_pkts)dev_kfree_skb_any (skb); } - txr->tx_cons = cons; +WRITE_ONCE (txr->tx_cons, cons); __netif_txq_completed_wake(txq, nr_pkts, tx_bytes, bnxt_tx_avail(bp, txr), bp->tx_wake_thresh,
drivers/net/ethernet/broadcom/bnxt/bnxt.h +4 -5
@@ -2231,13 +2231,12 @@ struct bnxt {#define SFF_MODULE_ID_QSFP28 0x11#define BNXT_MAX_PHY_I2C_RESP_SIZE 64 -static inline u32 bnxt_tx_avail (struct bnxt *bp,struct bnxt_tx_ring_info *txr) +static inline u32 bnxt_tx_avail (struct bnxt *bp, +const struct bnxt_tx_ring_info *txr) { -/* Tell compiler to fetch tx indices from memory. */ -barrier (); +u32 used =READ_ONCE (txr->tx_prod) -READ_ONCE (txr->tx_cons); -return bp->tx_ring_size - - ((txr->tx_prod - txr->tx_cons) & bp->tx_ring_mask); +return bp->tx_ring_size - (used & bp->tx_ring_mask); } static inline void bnxt_writeq(struct bnxt *bp, u64 val,
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +3 -3
@@ -64,7 +64,7 @@ struct bnxt_sw_tx_bd *bnxt_xmit_bd(struct bnxt *bp,int frag_len; prod =NEXT_TX (prod); - txr->tx_prod = prod; +WRITE_ONCE (txr->tx_prod, prod);/* first fill up the first buffer */ frag_tx_buf = &txr->tx_buf_ring[prod];
@@ -94,7 +94,7 @@ struct bnxt_sw_tx_bd *bnxt_xmit_bd(struct bnxt *bp,/* Sync TX BD */ wmb (); prod =NEXT_TX (prod); - txr->tx_prod = prod; +WRITE_ONCE (txr->tx_prod, prod);return tx_buf; }
@@ -161,7 +161,7 @@ void bnxt_tx_int_xdp(struct bnxt *bp, struct bnxt_napi *bnapi, int nr_pkts) } tx_cons =NEXT_TX (tx_cons); } - txr->tx_cons = tx_cons; +WRITE_ONCE (txr->tx_cons, tx_cons);if (rx_doorbell_needed) { tx_buf = &txr->tx_buf_ring[last_tx_cons];bnxt_db_write (bp, &rxr->rx_db, tx_buf->rx_prod);
Signature · 1 line
2.39.2
Re: [PATCH net-next v2 2/3] bnxt: use READ_ONCE/WRITE_ONCE for ring indexes
Quoted 4 lines · Jakub Kicinski
Eric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.
Quoted 8 lines
Suggested-by Eric Dumazet <[email protected]>Reviewed-by Eric Dumazet <[email protected]>Reviewed-by Jesse Brandeburg <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>---
v2:
- cover writes inbnxt_xdp.c
v1: https://lore.kernel.org/all/[email protected]/
Thanks.
RE: [PATCH net-next v2 2/3] bnxt: use READ_ONCE/WRITE_ONCE for ring indexes
In reply to Jakub KicinskiFrom: Jakub Kicinski
Quoted 5 lines
Sent: 12 April 2023 02:51
Eric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.
...
drivers/net/ethernet/broadcom/bnxt/bnxt.h struct bnxt { · line 2240
Quoted 12 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.h +4
-5-static inline u32 bnxt_tx_avail (struct bnxt *bp,struct bnxt_tx_ring_info *txr) +static inline u32 bnxt_tx_avail (struct bnxt *bp, +const struct bnxt_tx_ring_info *txr) { -/* Tell compiler to fetch tx indices from memory. */ -barrier (); +u32 used =READ_ONCE (txr->tx_prod) -READ_ONCE (txr->tx_cons); -return bp->tx_ring_size - - ((txr->tx_prod - txr->tx_cons) & bp->tx_ring_mask); +return bp->tx_ring_size - (used & bp->tx_ring_mask); }
Doesn't that function only make sense if only one of the ring index can be changing?
In this case I think this is being used in the transmit path so that 'tx_prod' is constant and is either already read or need not be read again.
Having written a lot of 'ring access' functions over the years if the ring size is a power of 2 I'd mask the 'tx_prod' value when it is being used rather than on the increment. (So the value just wraps modulo 2**32.)
This tends to make the code safer - especially since the 'ring full' and 'ring empty' conditions are different.
Also that code is masking with bp->tx_ring_mask, but the increments (in hunks I've chopped) use NEXT_TX(prod). If that is masking with bp->tx_ring_mask then 'bp' should be a parameter.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Paolo Abeni
maintainerRe: [PATCH net-next v2 2/3] bnxt: use READ_ONCE/WRITE_ONCE for ring indexes
Quoted 33 lines · David Laight
From: Jakub Kicinski
Quoted 5 lines
Sent: 12 April 2023 02:51
Eric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.
...
drivers/net/ethernet/broadcom/bnxt/bnxt.h struct bnxt { · line 2240
Quoted 12 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.h +4
-5-static inline u32 bnxt_tx_avail (struct bnxt *bp,struct bnxt_tx_ring_info *txr) +static inline u32 bnxt_tx_avail (struct bnxt *bp, +const struct bnxt_tx_ring_info *txr) { -/* Tell compiler to fetch tx indices from memory. */ -barrier (); +u32 used =READ_ONCE (txr->tx_prod) -READ_ONCE (txr->tx_cons); -return bp->tx_ring_size - - ((txr->tx_prod - txr->tx_cons) & bp->tx_ring_mask); +return bp->tx_ring_size - (used & bp->tx_ring_mask); }Doesn't that function only make sense if only one of the ring index can be changing?
In this case I think this is being used in the transmit path so that 'tx_prod' is constant and is either already read or need not be read again.Having written a lot of 'ring access' functions over the years if the ring size is a power of 2 I'd mask the '
tx_prod' value when it is being used rather than on the increment. (So the value just wraps modulo 2**32.)
This tends to make the code safer - especially since the 'ring full' and 'ring empty' conditions are different.
Quoted 4 lines
Also that code is masking with
bp->tx_ring_mask, but the increments (in hunks I've chopped) useNEXT_TX(prod). If that is masking withbp->tx_ring_maskthen 'bp' should be a parameter.
AFAICS bnxt_tx_avail() is also used in TX interrupt, outside tx path/tx lock.
I think all the above consideration are more suited for a driver refactor, while the current patch specifically address potential data race issues.
Cheers,
Paolo
RE: [PATCH net-next v2 2/3] bnxt: use READ_ONCE/WRITE_ONCE for ring indexes
From: Paolo Abeni
Quoted 29 lines
Sent: 13 April 2023 12:38
Quoted 26 lines · David Laight
From: Jakub Kicinski
Quoted 5 lines
Sent: 12 April 2023 02:51
Eric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.
...
drivers/net/ethernet/broadcom/bnxt/bnxt.h struct bnxt { · line 2240
Quoted 12 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.h +4
-5-static inline u32 bnxt_tx_avail (struct bnxt *bp,struct bnxt_tx_ring_info *txr) +static inline u32 bnxt_tx_avail (struct bnxt *bp, +const struct bnxt_tx_ring_info *txr) { -/* Tell compiler to fetch tx indices from memory. */ -barrier (); +u32 used =READ_ONCE (txr->tx_prod) -READ_ONCE (txr->tx_cons); -return bp->tx_ring_size - - ((txr->tx_prod - txr->tx_cons) & bp->tx_ring_mask); +return bp->tx_ring_size - (used & bp->tx_ring_mask); }Doesn't that function only make sense if only one of the ring index can be changing?
In this case I think this is being used in the transmit path so that 'tx_prod' is constant and is either already read or need not be read again.
...
Quoted 3 lines
AFAICS
bnxt_tx_avail()is also used in TX interrupt, outside tx path/tx lock.
In which case both tx_prod and tx_cons are subject to possible updates. It is even possible that the two values have absolutely no relation to each other, it requires some unusual circumstances, but isn't impossible.
- A high priority interrupt (eg x86 SMM mode) could separate the READ_ONCE().
- Transmit setup will increase tx_prod.
- End of transmit 'reap' often done by other code paths (like rx processing or tx setup) can change tx_cons.
So not only is the value immediately stale, it can be just plain wrong.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Jakub Kicinski
maintainerPATCH v2 3/3 mlx4: use READ_ONCE/WRITE_ONCE for ring indexes
In reply to Jakub KicinskiEric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.
---
CC: [email protected]
CC: [email protected]
--- drivers/net/ethernet/mellanox/mlx4/en_tx.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
drivers/net/ethernet/mellanox/mlx4/en_tx.c +5 -3
@@ -228,7 +228,9 @@ void mlx4_en_deactivate_tx_ring(struct mlx4_en_priv *priv,static inline bool mlx4_en_is_tx_ring_full (struct mlx4_en_tx_ring *ring) { -return ring->prod - ring->cons > ring->full_size; +u32 used =READ_ONCE (ring->prod) -READ_ONCE (ring->cons); + +return used > ring->full_size; } static void mlx4_en_stamp_wqe(struct mlx4_en_priv *priv,
@@ -1083,7 +1085,7 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev) op_own |=cpu_to_be32 (MLX4_WQE_CTRL_IIP ); } - ring->prod += nr_txbb; +WRITE_ONCE (ring->prod, ring->prod + nr_txbb);/* If we used a bounce buffer then copy descriptor back into place */ if (unlikely (bounce))
@@ -1214,7 +1216,7 @@ netdev_tx_t mlx4_en_xmit_frame(struct mlx4_en_rx_ring *rx_ring, rx_ring->xdp_tx++; - ring->prod +=MLX4_EN_XDP_TX_NRTXBB ; +WRITE_ONCE (ring->prod, ring->prod +MLX4_EN_XDP_TX_NRTXBB );/* Ensure new descriptor hits memory * before setting ownership of this descriptor to HW
Signature · 1 line
2.39.2
Re: [PATCH net-next v2 3/3] mlx4: use READ_ONCE/WRITE_ONCE for ring indexes
Quoted 7 lines · Jakub Kicinski
Eric points out that we should make sure that ring index updates are wrapped in the appropriate READ_ONCE/WRITE_ONCE macros.
Suggested-by Eric Dumazet <[email protected]>Reviewed-by Eric Dumazet <[email protected]>Reviewed-by Jesse Brandeburg <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>
Quoted 7 lines
---
CC: [email protected]
CC: [email protected]--- drivers/net/ethernet/mellanox/mlx4/en_tx.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
Thanks!
This series was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Quoted 9 lines · On Tue, 11 Apr 2023 18:50:35 -0700 you wrote:
Small follow up to the lockless ring stop/start macros. Update the doc and the drivers suggested by Eric:
https://lore.kernel.org/all/CANn89iJrBGSybMX1FqrhCEMWT3Nnz2=2+aStsbbwpWzKHjk51g@mail.gmail.com/v2:
- convert xdp paths in bnxt as well
v1: https://lore.kernel.org/all/[email protected]/[...]
Here is the summary with links:
- [net-next,v2,1/3] net: docs: update the sample code in driver.rst
https://git.kernel.org/netdev/net-next/c/50762d9af307 - [net-next,v2,2/3] bnxt: use READ_ONCE/WRITE_ONCE for ring indexes https://git.kernel.org/netdev/net-next/c/36647b206c01 - [net-next,v2,3/3] mlx4: use READ_ONCE/WRITE_ONCE for ring indexes https://git.kernel.org/netdev/net-next/c/9a714997386b
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html