avoid compiler and IQ/OQ reordering
Subsystems (4)
v2 cover avoid compiler and IQ/OQ reordering
Utilize READ_ONCE and WRITE_ONCE APIs to prevent compiler optimization and reordering. Ensure IO queue OUT/IN_CNT registers are flushed. Relocate IQ/OQ IN/OUT_CNTS updates to occur before NAPI completion, and replace napi_complete with napi_complete_done.
Vimlesh Kumar (4):
octeon_ep: Relocate counter updates before NAPI
octeon_ep: avoid compiler and IQ/OQ reordering
octeon_ep_vf: Relocate counter updates before NAPI
octeon_ep_vf: avoid compiler and IQ/OQ reordering
V1: https://lore.kernel.org/all/[email protected]/
.../ethernet/marvell/octeon_ep/octep_main.c | 40 +++++++++++++------ .../net/ethernet/marvell/octeon_ep/octep_rx.c | 27 +++++++++---- .../marvell/octeon_ep_vf/octep_vf_main.c | 38 +++++++++++++----- .../marvell/octeon_ep_vf/octep_vf_rx.c | 28 +++++++++---- 4 files changed, 95 insertions(+), 38 deletions(-)
Signature · 1 line
2.47.3
PATCH v2 1/4 octeon_ep: Relocate counter updates before NAPI
Relocate IQ/OQ IN/OUT_CNTS updates to occur before NAPI completion, and replace napi_complete with napi_complete_done.
Moving the IQ/OQ counter updates before napi_complete_done ensures
1. Counter registers are updated before re-enabling interrupts.
2. Prevents a race where new packets arrive but counters aren't properly synchronized.napi_complete_done (vs napi_complete) allows for better interrupt coalescing.
--- .../ethernet/marvell/octeon_ep/octep_main.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)
drivers/net/ethernet/marvell/octeon_ep/octep_main.c +15 -4
@@ -555,12 +555,12 @@ static void octep_clean_irqs(struct octep_device *oct) }/** -* octep_enable_ioq_irq() - Enable MSI-x interrupt of a Tx/Rx queue. +* octep_update_pkt() - Update IQ/OQ IN/OUT_CNT registers. * * @iq: Octeon Tx queue data structure. * @oq: Octeon Rx queue data structure. */ -static void octep_enable_ioq_irq (struct octep_iq *iq,struct octep_oq *oq) +static void octep_update_pkt (struct octep_iq *iq,struct octep_oq *oq) {u32 pkts_pend = oq->pkts_pending;
@@ -576,7 +576,17 @@ static void octep_enable_ioq_irq(struct octep_iq *iq, struct octep_oq *oq) }/* Flush the previous wrties before writing to RESEND bit */ -wmb (); +smp_wmb (); +} + +/** +* octep_enable_ioq_irq() - Enable MSI-x interrupt of a Tx/Rx queue. +* +* @iq: Octeon Tx queue data structure. +* @oq: Octeon Rx queue data structure. +*/ +static void octep_enable_ioq_irq (struct octep_iq *iq,struct octep_oq *oq) +{writeq (1UL <<OCTEP_OQ_INTR_RESEND_BIT , oq->pkts_sent_reg);writeq (1UL <<OCTEP_IQ_INTR_RESEND_BIT , iq->inst_cnt_reg); }
@@ -602,7 +612,8 @@ static int octep_napi_poll(struct napi_struct *napi, int budget)if (tx_pending || rx_done >= budget)return budget; -napi_complete (napi); +octep_update_pkt (ioq_vector->iq, ioq_vector->oq); +napi_complete_done (napi, rx_done);octep_enable_ioq_irq (ioq_vector->iq, ioq_vector->oq);return rx_done; }
Signature · 1 line
2.47.3
PATCH v2 2/4 octeon_ep: avoid compiler and IQ/OQ reordering
In reply to Vimlesh KumarUtilize READ_ONCE and WRITE_ONCE APIs for IO queue Tx/Rx variable access to prevent compiler optimization and reordering. Additionally, ensure IO queue OUT/IN_CNT registers are flushed by performing a read-back after writing.
The compiler could reorder reads/writes to pkts_pending, last_pkt_count, etc., causing stale values to be used when calculating packets to process or register updates to send to hardware. The Octeon hardware requires a read-back after writing to OUT_CNT/IN_CNT registers to ensure the write has been flushed through any posted write buffers before the interrupt resend bit is set. Without this, we have observed cases where the hardware didn't properly update its internal state.
wmb/rmb only provides ordering guarantees but doesn't prevent the compiler from performing optimizations like caching in registers, load tearing etc.
---
V1: https://lore.kernel.org/all/[email protected]/
.../ethernet/marvell/octeon_ep/octep_main.c | 21 +++++++++------ .../net/ethernet/marvell/octeon_ep/octep_rx.c | 27 +++++++++++++------ 2 files changed, 32 insertions(+), 16 deletions(-)
drivers/net/ethernet/marvell/octeon_ep/octep_main.c +13 -8
@@ -562,17 +562,22 @@ static void octep_clean_irqs(struct octep_device *oct)*/ static void octep_update_pkt (struct octep_iq *iq,struct octep_oq *oq) { -u32 pkts_pend = oq->pkts_pending; +u32 pkts_pend =READ_ONCE (oq->pkts_pending); +u32 last_pkt_count =READ_ONCE (oq->last_pkt_count); +u32 pkts_processed =READ_ONCE (iq->pkts_processed); +u32 pkt_in_done =READ_ONCE (iq->pkt_in_done);netdev_dbg (iq->netdev,"enabling intr for Q-%u\n" , iq->q_no); -if (iq->pkts_processed) { -writel (iq->pkts_processed, iq->inst_cnt_reg); - iq->pkt_in_done -= iq->pkts_processed; - iq->pkts_processed =0 ; +if (pkts_processed) { +writel (pkts_processed, iq->inst_cnt_reg); +readl (iq->inst_cnt_reg); +WRITE_ONCE (iq->pkt_in_done, (pkt_in_done - pkts_processed)); +WRITE_ONCE (iq->pkts_processed,0 ); } -if (oq->last_pkt_count - pkts_pend) { -writel (oq->last_pkt_count - pkts_pend, oq->pkts_sent_reg); - oq->last_pkt_count = pkts_pend; +if (last_pkt_count - pkts_pend) { +writel (last_pkt_count - pkts_pend, oq->pkts_sent_reg); +readl (oq->pkts_sent_reg); +WRITE_ONCE (oq->last_pkt_count, pkts_pend); }/* Flush the previous wrties before writing to RESEND bit */
drivers/net/ethernet/marvell/octeon_ep/octep_rx.c +19 -8
@@ -318,10 +318,16 @@ static int octep_oq_check_hw_for_pkts(struct octep_device *oct, struct octep_oq *oq) {u32 pkt_count, new_pkts; +u32 last_pkt_count, pkts_pending; pkt_count =readl (oq->pkts_sent_reg); - new_pkts = pkt_count - oq->last_pkt_count; + last_pkt_count =READ_ONCE (oq->last_pkt_count); + new_pkts = pkt_count - last_pkt_count; +if (pkt_count < last_pkt_count) { +dev_err (oq->dev,"OQ-%u pkt_count(%u) < oq->last_pkt_count(%u)\n" , + oq->q_no, pkt_count, last_pkt_count); + }/* Clear the hardware packets counter register if the rx queue is * being processed continuously with-in a single interrupt and * reached half its max value.
@@ -332,8 +338,9 @@ static int octep_oq_check_hw_for_pkts(struct octep_device *oct, pkt_count =readl (oq->pkts_sent_reg); new_pkts += pkt_count; } - oq->last_pkt_count = pkt_count; - oq->pkts_pending += new_pkts; +WRITE_ONCE (oq->last_pkt_count, pkt_count); + pkts_pending =READ_ONCE (oq->pkts_pending); +WRITE_ONCE (oq->pkts_pending, (pkts_pending + new_pkts));return new_pkts; }
@@ -408,7 +415,7 @@ static int __octep_oq_process_rx(struct octep_device *oct,u16 rx_ol_flags;u32 read_idx; - read_idx = oq->host_read_idx; + read_idx =READ_ONCE (oq->host_read_idx); rx_bytes =0 ; desc_used =0 ;for (pkt =0 ; pkt < pkts_to_process; pkt++) {
@@ -493,7 +500,7 @@ static int __octep_oq_process_rx(struct octep_device *oct,napi_gro_receive (oq->napi, skb); } - oq->host_read_idx = read_idx; +WRITE_ONCE (oq->host_read_idx, read_idx); oq->refill_count += desc_used; oq->stats->packets += pkt; oq->stats->bytes += rx_bytes;
@@ -516,22 +523,26 @@ int octep_oq_process_rx(struct octep_oq *oq, int budget) {u32 pkts_available, pkts_processed, total_pkts_processed;struct octep_device *oct = oq->octep_dev; +u32 pkts_pending; pkts_available =0 ; pkts_processed =0 ; total_pkts_processed =0 ;while (total_pkts_processed < budget) {/* update pending count only when current one exhausted */ -if (oq->pkts_pending ==0 ) + pkts_pending =READ_ONCE (oq->pkts_pending); +if (pkts_pending ==0 )octep_oq_check_hw_for_pkts (oct, oq); + pkts_pending =READ_ONCE (oq->pkts_pending); pkts_available =min (budget - total_pkts_processed, - oq->pkts_pending); + pkts_pending);if (!pkts_available)break ; pkts_processed =__octep_oq_process_rx (oct, oq, pkts_available); - oq->pkts_pending -= pkts_processed; + pkts_pending =READ_ONCE (oq->pkts_pending); +WRITE_ONCE (oq->pkts_pending, (pkts_pending - pkts_processed)); total_pkts_processed += pkts_processed; }
Signature · 1 line
2.47.3
PATCH v2 3/4 octeon_ep_vf: Relocate counter updates before NAPI
In reply to Vimlesh KumarRelocate IQ/OQ IN/OUT_CNTS updates to occur before NAPI completion. Moving the IQ/OQ counter updates before napi_complete_done ensures
1. Counter registers are updated before re-enabling interrupts.
2. Prevents a race where new packets arrive but counters aren't properly synchronized.
--- .../marvell/octeon_ep_vf/octep_vf_main.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-)
drivers/net/ethernet/marvell/octeon_ep_vf/octep_vf_main.c +15 -2
@@ -286,12 +286,13 @@ static void octep_vf_clean_irqs(struct octep_vf_device *oct) }/** -* octep_vf_enable_ioq_irq() - Enable MSI-x interrupt of a Tx/Rx queue. +* octep_vf_update_pkt() - Update IQ/OQ IN/OUT_CNT registers. * * @iq: Octeon Tx queue data structure. * @oq: Octeon Rx queue data structure. */ -static void octep_vf_enable_ioq_irq (struct octep_vf_iq *iq,struct octep_vf_oq *oq) + +static void octep_vf_update_pkt (struct octep_vf_iq *iq,struct octep_vf_oq *oq) {u32 pkts_pend = oq->pkts_pending;
@@ -308,6 +309,17 @@ static void octep_vf_enable_ioq_irq(struct octep_vf_iq *iq, struct octep_vf_oq */* Flush the previous wrties before writing to RESEND bit */ smp_wmb (); +} + +/** +* octep_vf_enable_ioq_irq() - Enable MSI-x interrupt of a Tx/Rx queue. +* +* @iq: Octeon Tx queue data structure. +* @oq: Octeon Rx queue data structure. +*/ +static void octep_vf_enable_ioq_irq (struct octep_vf_iq *iq, +struct octep_vf_oq *oq) +{writeq (1UL <<OCTEP_VF_OQ_INTR_RESEND_BIT , oq->pkts_sent_reg);writeq (1UL <<OCTEP_VF_IQ_INTR_RESEND_BIT , iq->inst_cnt_reg); }
@@ -333,6 +345,7 @@ static int octep_vf_napi_poll(struct napi_struct *napi, int budget)if (tx_pending || rx_done >= budget)return budget; +octep_vf_update_pkt (ioq_vector->iq, ioq_vector->oq);if (likely (napi_complete_done (napi, rx_done)))octep_vf_enable_ioq_irq (ioq_vector->iq, ioq_vector->oq);
Signature · 1 line
2.47.3
PATCH v2 4/4 octeon_ep_vf: avoid compiler and IQ/OQ reordering
In reply to Vimlesh KumarUtilize READ_ONCE and WRITE_ONCE APIs for IO queue Tx/Rx variable access to prevent compiler optimization and reordering. Additionally, ensure IO queue OUT/IN_CNT registers are flushed by performing a read-back after writing.
The compiler could reorder reads/writes to pkts_pending, last_pkt_count, etc., causing stale values to be used when calculating packets to process or register updates to send to hardware. The Octeon hardware requires a read-back after writing to OUT_CNT/IN_CNT registers to ensure the write has been flushed through any posted write buffers before the interrupt resend bit is set. Without this, we have observed cases where the hardware didn't properly update its internal state.
wmb/rmb only provides ordering guarantees but doesn't prevent the compiler from performing optimizations like caching in registers, load tearing etc.
---
V1: https://lore.kernel.org/all/[email protected]/
.../marvell/octeon_ep_vf/octep_vf_main.c | 21 ++++++++------ .../marvell/octeon_ep_vf/octep_vf_rx.c | 28 +++++++++++++------ 2 files changed, 33 insertions(+), 16 deletions(-)
drivers/net/ethernet/marvell/octeon_ep_vf/octep_vf_main.c +13 -8
@@ -294,17 +294,22 @@ static void octep_vf_clean_irqs(struct octep_vf_device *oct)static void octep_vf_update_pkt (struct octep_vf_iq *iq,struct octep_vf_oq *oq) { -u32 pkts_pend = oq->pkts_pending; +u32 pkts_pend =READ_ONCE (oq->pkts_pending); +u32 last_pkt_count =READ_ONCE (oq->last_pkt_count); +u32 pkts_processed =READ_ONCE (iq->pkts_processed); +u32 pkt_in_done =READ_ONCE (iq->pkt_in_done);netdev_dbg (iq->netdev,"enabling intr for Q-%u\n" , iq->q_no); -if (iq->pkts_processed) { -writel (iq->pkts_processed, iq->inst_cnt_reg); - iq->pkt_in_done -= iq->pkts_processed; - iq->pkts_processed =0 ; +if (pkts_processed) { +writel (pkts_processed, iq->inst_cnt_reg); +readl (iq->inst_cnt_reg); +WRITE_ONCE (iq->pkt_in_done, (pkt_in_done - pkts_processed)); +WRITE_ONCE (iq->pkts_processed,0 ); } -if (oq->last_pkt_count - pkts_pend) { -writel (oq->last_pkt_count - pkts_pend, oq->pkts_sent_reg); - oq->last_pkt_count = pkts_pend; +if (last_pkt_count - pkts_pend) { +writel (last_pkt_count - pkts_pend, oq->pkts_sent_reg); +readl (oq->pkts_sent_reg); +WRITE_ONCE (oq->last_pkt_count, pkts_pend); }/* Flush the previous wrties before writing to RESEND bit */
drivers/net/ethernet/marvell/octeon_ep_vf/octep_vf_rx.c +20 -8
@@ -319,9 +319,16 @@ static int octep_vf_oq_check_hw_for_pkts(struct octep_vf_device *oct, struct octep_vf_oq *oq) {u32 pkt_count, new_pkts; +u32 last_pkt_count, pkts_pending; pkt_count =readl (oq->pkts_sent_reg); - new_pkts = pkt_count - oq->last_pkt_count; + last_pkt_count =READ_ONCE (oq->last_pkt_count); + new_pkts = pkt_count - last_pkt_count; + +if (pkt_count < last_pkt_count) { +dev_err (oq->dev,"OQ-%u pkt_count(%u) < oq->last_pkt_count(%u)\n" , + oq->q_no, pkt_count, last_pkt_count); + }/* Clear the hardware packets counter register if the rx queue is * being processed continuously with-in a single interrupt and
@@ -333,8 +340,9 @@ static int octep_vf_oq_check_hw_for_pkts(struct octep_vf_device *oct, pkt_count =readl (oq->pkts_sent_reg); new_pkts += pkt_count; } - oq->last_pkt_count = pkt_count; - oq->pkts_pending += new_pkts; +WRITE_ONCE (oq->last_pkt_count, pkt_count); + pkts_pending =READ_ONCE (oq->pkts_pending); +WRITE_ONCE (oq->pkts_pending, (pkts_pending + new_pkts));return new_pkts; }
@@ -363,7 +371,7 @@ static int __octep_vf_oq_process_rx(struct octep_vf_device *oct,struct sk_buff *skb;u32 read_idx; - read_idx = oq->host_read_idx; + read_idx =READ_ONCE (oq->host_read_idx); rx_bytes =0 ; desc_used =0 ;for (pkt =0 ; pkt < pkts_to_process; pkt++) {
@@ -457,7 +465,7 @@ static int __octep_vf_oq_process_rx(struct octep_vf_device *oct,napi_gro_receive (oq->napi, skb); } - oq->host_read_idx = read_idx; +WRITE_ONCE (oq->host_read_idx, read_idx); oq->refill_count += desc_used; oq->stats->packets += pkt; oq->stats->bytes += rx_bytes;
@@ -480,22 +488,26 @@ int octep_vf_oq_process_rx(struct octep_vf_oq *oq, int budget) {u32 pkts_available, pkts_processed, total_pkts_processed;struct octep_vf_device *oct = oq->octep_vf_dev; +u32 pkts_pending; pkts_available =0 ; pkts_processed =0 ; total_pkts_processed =0 ;while (total_pkts_processed < budget) {/* update pending count only when current one exhausted */ -if (oq->pkts_pending ==0 ) + pkts_pending =READ_ONCE (oq->pkts_pending); +if (pkts_pending ==0 )octep_vf_oq_check_hw_for_pkts (oct, oq); + pkts_pending =READ_ONCE (oq->pkts_pending); pkts_available =min (budget - total_pkts_processed, - oq->pkts_pending); + pkts_pending);if (!pkts_available)break ; pkts_processed =__octep_vf_oq_process_rx (oct, oq, pkts_available); - oq->pkts_pending -= pkts_processed; + pkts_pending =READ_ONCE (oq->pkts_pending); +WRITE_ONCE (oq->pkts_pending, (pkts_pending - pkts_processed)); total_pkts_processed += pkts_processed; }
Signature · 1 line
2.47.3
This series was applied to netdev/net.git (main) by Paolo Abeni <[email protected]>:
Hello:
This series was applied to netdev/net.git (main)
by Paolo Abeni <[email protected]>:
Quoted 6 lines · On Fri, 27 Feb 2026 09:13:56 +0000 you wrote:
Utilize
READ_ONCEandWRITE_ONCEAPIs to prevent compiler optimization and reordering. Ensure IO queue OUT/IN_CNT registers are flushed. Relocate IQ/OQ IN/OUT_CNTS updates to occur before NAPI completion, and replacenapi_completewithnapi_complete_done.
Quoted 7 lines
Vimlesh Kumar (4):
octeon_ep: Relocate counter updates before NAPI
octeon_ep: avoid compiler and IQ/OQ reordering
octeon_ep_vf: Relocate counter updates before NAPI
octeon_ep_vf: avoid compiler and IQ/OQ reordering[...]
Here is the summary with links:
- [net,v2,1/4] octeon_ep: Relocate counter updates before NAPI
https://git.kernel.org/netdev/net/c/18c04a808c43 - [net,v2,2/4] octeon_ep: avoid compiler and IQ/OQ reordering https://git.kernel.org/netdev/net/c/43b3160cb639 - [net,v2,3/4] octeon_ep_vf: Relocate counter updates before NAPI https://git.kernel.org/netdev/net/c/2ae7d20fb24f - [net,v2,4/4] octeon_ep_vf: avoid compiler and IQ/OQ reordering https://git.kernel.org/netdev/net/c/6c73126ecd10
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html