net: axienet: Be more careful about updating tx_bd_tail
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH v7 1/2 net: axienet: Be more careful about updating tx_bd_tail
The axienet_start_xmit function was updating the tx_bd_tail variable multiple times, with potential rollbacks on error or invalid intermediate positions, even though this variable is also used in the TX completion path. Use READ_ONCE where this variable is read and WRITE_ONCE where it is written to make this update more atomic, and move the write before the MMIO write to start the transfer, so it is protected by that implicit write barrier.
--- .../net/ethernet/xilinx/xilinx_axienet_main.c | 26 +++++++++++-------- 1 file changed, 15 insertions(+), 11 deletions(-)
drivers/net/ethernet/xilinx/xilinx_axienet_main.c +15 -11
@@ -746,7 +746,8 @@ static inline int axienet_check_tx_bd_space(struct axienet_local *lp,/* Ensure we see all descriptor updates from device or TX IRQ path */ rmb (); - cur_p = &lp->tx_bd_v[(lp->tx_bd_tail + num_frag) % lp->tx_bd_num]; + cur_p = &lp->tx_bd_v[(READ_ONCE (lp->tx_bd_tail) + num_frag) % + lp->tx_bd_num];if (cur_p->cntrl)return NETDEV_TX_BUSY ;return 0 ;
@@ -807,12 +808,15 @@ axienet_start_xmit(struct sk_buff *skb, struct net_device *ndev)u32 csum_index_off;skb_frag_t *frag;dma_addr_t tail_p, phys; +u32 orig_tail_ptr, new_tail_ptr;struct axienet_local *lp =netdev_priv (ndev);struct axidma_bd *cur_p; -u32 orig_tail_ptr = lp->tx_bd_tail; + + orig_tail_ptr = lp->tx_bd_tail; + new_tail_ptr = orig_tail_ptr; num_frag =skb_shinfo (skb)->nr_frags; - cur_p = &lp->tx_bd_v[lp->tx_bd_tail]; + cur_p = &lp->tx_bd_v[orig_tail_ptr];if (axienet_check_tx_bd_space (lp, num_frag +1 )) {/* Should not happen as last start_xmit call should have
@@ -852,9 +856,9 @@ axienet_start_xmit(struct sk_buff *skb, struct net_device *ndev) cur_p->cntrl =skb_headlen (skb) |XAXIDMA_BD_CTRL_TXSOF_MASK ;for (ii =0 ; ii < num_frag; ii++) { -if (++lp->tx_bd_tail >= lp->tx_bd_num) - lp->tx_bd_tail =0 ; - cur_p = &lp->tx_bd_v[lp->tx_bd_tail]; +if (++new_tail_ptr >= lp->tx_bd_num) + new_tail_ptr =0 ; + cur_p = &lp->tx_bd_v[new_tail_ptr]; frag = &skb_shinfo (skb)->frags[ii]; phys = dma_map_single(lp->dev, skb_frag_address(frag),
@@ -866,8 +870,6 @@ axienet_start_xmit(struct sk_buff *skb, struct net_device *ndev) ndev->stats.tx_dropped++;axienet_free_tx_chain (ndev, orig_tail_ptr, ii +1 ,NULL ); - lp->tx_bd_tail = orig_tail_ptr; -return NETDEV_TX_OK ; }desc_set_phys_addr (lp, phys, cur_p);
@@ -877,11 +879,13 @@ axienet_start_xmit(struct sk_buff *skb, struct net_device *ndev) cur_p->cntrl |=XAXIDMA_BD_CTRL_TXEOF_MASK ; cur_p->skb = skb; - tail_p = lp->tx_bd_p +sizeof (*lp->tx_bd_v) * lp->tx_bd_tail; + tail_p = lp->tx_bd_p +sizeof (*lp->tx_bd_v) * new_tail_ptr; +if (++new_tail_ptr >= lp->tx_bd_num) + new_tail_ptr =0 ; +WRITE_ONCE (lp->tx_bd_tail, new_tail_ptr); +/* Start the transfer */ axienet_dma_out_addr (lp,XAXIDMA_TX_TDESC_OFFSET , tail_p); -if (++lp->tx_bd_tail >= lp->tx_bd_num) - lp->tx_bd_tail =0 ;/* Stop queue if next transmit may not have space */ if (axienet_check_tx_bd_space (lp,MAX_SKB_FRAGS +1 )) {
Signature · 1 line
2.31.1
This series was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (master) by David S. Miller <[email protected]>:
Quoted 9 lines · On Thu, 12 May 2022 11:18:52 -0600 you wrote:
The
axienet_start_xmitfunction was updating thetx_bd_tailvariable multiple times, with potential rollbacks on error or invalid intermediate positions, even though this variable is also used in the TX completion path. UseREAD_ONCEwhere this variable is read andWRITE_ONCEwhere it is written to make this update more atomic, and move the write before the MMIO write to start the transfer, so it is protected by that implicit write barrier.[...]
Here is the summary with links:
- [net-next,v7,1/2] net: axienet: Be more careful about updating tx_bd_tail
https://git.kernel.org/netdev/net-next/c/f0cf4000f586 - [net-next,v7,2/2] net: axienet: Use NAPI for TX completion path https://git.kernel.org/netdev/net-next/c/9e2bc267e780
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
PATCH v7 2/2 net: axienet: Use NAPI for TX completion path
In reply to a message not in the archiveThis driver was using the TX IRQ handler to perform all TX completion tasks. Under heavy TX network load, this can cause significant irqs-off latencies (found to be in the hundreds of microseconds using ftrace). This can cause other issues, such as overrunning serial UART FIFOs when using high baud rates with limited UART FIFO sizes.
Switch to using a NAPI poll handler to perform the TX completion work to get this out of hard IRQ context and avoid the IRQ latency impact. A separate poll handler is used for TX and RX since they have separate IRQs on this controller, so that the completion work for each of them stays on the same CPU as the interrupt.
Testing on a Xilinx MPSoC ZU9EG platform using iperf3 from a Linux PC through a switch at 1G link speed showed no significant change in TX or RX throughput, with approximately 941 Mbps before and after. Hard IRQ time in the TX throughput test was significantly reduced from 12% to below 1% on the CPU handling TX interrupts, with total hard+soft IRQ CPU usage dropping from about 56% down to 48%.
--- drivers/net/ethernet/xilinx/xilinx_axienet.h | 54 +++---- .../net/ethernet/xilinx/xilinx_axienet_main.c | 142 ++++++++++-------- 2 files changed, 111 insertions(+), 85 deletions(-)
drivers/net/ethernet/xilinx/xilinx_axienet.h +28 -26
@@ -385,7 +385,6 @@ struct axidma_bd {* @phy_node: Pointer to device node structure * @phylink: Pointer to phylink instance * @phylink_config: phylink configuration settings -* @napi: NAPI control structure * @pcs_phy: Reference to PCS/PMA PHY if used * @pcs: phylink pcs structure for PCS PHY * @switch_x_sgmii: Whether switchable 1000BaseX/SGMII mode is enabled in the core
@@ -396,7 +395,22 @@ struct axidma_bd {* @regs_start: Resource start for axienet device addresses * @regs: Base address for the axienet_local device address space * @dma_regs: Base address for the axidma device address space +* @napi_rx: NAPI RX control structure * @rx_dma_cr: Nominal content of RX DMA control register +* @rx_bd_v: Virtual address of the RX buffer descriptor ring +* @rx_bd_p: Physical address(start address) of the RX buffer descr. ring +* @rx_bd_num: Size of RX buffer descriptor ring +* @rx_bd_ci: Stores the index of the Rx buffer descriptor in the ring being +* accessed currently. +* @napi_tx: NAPI TX control structure +* @tx_dma_cr: Nominal content of TX DMA control register +* @tx_bd_v: Virtual address of the TX buffer descriptor ring +* @tx_bd_p: Physical address(start address) of the TX buffer descr. ring +* @tx_bd_num: Size of TX buffer descriptor ring +* @tx_bd_ci: Stores the next Tx buffer descriptor in the ring that may be +* complete. Only updated at runtime by TX NAPI poll. +* @tx_bd_tail: Stores the index of the next Tx buffer descriptor in the ring +* to be populated. * @dma_err_task: Work structure to process Axi DMA errors * @tx_irq: Axidma TX IRQ number * @rx_irq: Axidma RX IRQ number
@@ -404,19 +418,6 @@ struct axidma_bd {* @phy_mode: Phy type to identify between MII/GMII/RGMII/SGMII/1000 Base-X * @options: AxiEthernet option word * @features: Stores the extended features supported by the axienet hw -* @tx_bd_v: Virtual address of the TX buffer descriptor ring -* @tx_bd_p: Physical address(start address) of the TX buffer descr. ring -* @tx_bd_num: Size of TX buffer descriptor ring -* @rx_bd_v: Virtual address of the RX buffer descriptor ring -* @rx_bd_p: Physical address(start address) of the RX buffer descr. ring -* @rx_bd_num: Size of RX buffer descriptor ring -* @tx_bd_ci: Stores the index of the Tx buffer descriptor in the ring being -* accessed currently. Used while alloc. BDs before a TX starts -* @tx_bd_tail: Stores the index of the Tx buffer descriptor in the ring being -* accessed currently. Used while processing BDs after the TX -* completed. -* @rx_bd_ci: Stores the index of the Rx buffer descriptor in the ring being -* accessed currently. * @max_frm_size: Stores the maximum size of the frame that can be that * Txed/Rxed in the existing hardware. If jumbo option is * supported, the maximum frame size would be 9k. Else it is
@@ -436,8 +437,6 @@ struct axienet_local {struct phylink *phylink;struct phylink_config phylink_config; -struct napi_struct napi; -struct mdio_device *pcs_phy;struct phylink_pcs pcs;
@@ -453,7 +452,20 @@ struct axienet_local {void __iomem *regs;void __iomem *dma_regs; +struct napi_struct napi_rx;u32 rx_dma_cr; +struct axidma_bd *rx_bd_v; +dma_addr_t rx_bd_p; +u32 rx_bd_num; +u32 rx_bd_ci; + +struct napi_struct napi_tx; +u32 tx_dma_cr; +struct axidma_bd *tx_bd_v; +dma_addr_t tx_bd_p; +u32 tx_bd_num; +u32 tx_bd_ci; +u32 tx_bd_tail;struct work_struct dma_err_task;
@@ -465,16 +477,6 @@ struct axienet_local {u32 options;u32 features; -struct axidma_bd *tx_bd_v; -dma_addr_t tx_bd_p; -u32 tx_bd_num; -struct axidma_bd *rx_bd_v; -dma_addr_t rx_bd_p; -u32 rx_bd_num; -u32 tx_bd_ci; -u32 tx_bd_tail; -u32 rx_bd_ci; -u32 max_frm_size;u32 rxmem;
drivers/net/ethernet/xilinx/xilinx_axienet_main.c +83 -59
308 lines not on this page · raw message
Signature · 1 line
2.31.1
Re: [PATCH net-next v7 0/2] axienet NAPI improvements
In reply to a message not in the archiveQuoted 25 lines · Robert Hancock
Changes to support TX NAPI in the axienet driver, as well as fixing a potential concurrency issue in the TX ring pointer handling.
Supersedes v5 of the individual patch
"net: axienet: Use NAPI for TX completion path".Changed since v6: Use
READ_ONCEwhere the value actually does need to be read once, inaxienet_check_tx_bd_space, not inaxienet_start_xmitwhere it doesn't.Changed since v5: Replaced spinlock with fixes to the way the TX ring tail pointer is updated, and broke those changes into a separate patch.
Changed since v4: Added locking to protect TX ring tail pointer against concurrent access by TX transmit and TX poll paths.
Changed since v3: Fixed references to renamed function in comments
Changed since v2: Use separate TX and RX NAPI poll handlers to keep completion handling on same CPU as TX/RX IRQ. Added hard/soft IRQ benchmark information to commit message.
Changed since v1: Added benchmark information to commit message, no code changes.
Quoted 8 lines
Robert Hancock (2):
net: axienet: Be more careful about updatingtx_bd_tail
net: axienet: Use NAPI for TX completion pathdrivers/net/ethernet/xilinx/xilinx_axienet.h | 54 +++--- .../net/ethernet/xilinx/xilinx_axienet_main.c | 168 ++++++++++-------- 2 files changed, 126 insertions(+), 96 deletions(-)
Looking at Patchwork and Lore, seems like this cover letter may not have made it to the list, though the two patches did? Can resend if it doesn't eventually show up..
Signature · 3 lines
Robert Hancock Senior Hardware Designer, Calian Advanced Technologies www.calian.com
Jakub Kicinski
maintainerRe: [PATCH net-next v7 0/2] axienet NAPI improvements
Quoted 3 lines · On Thu, 12 May 2022 23:00:02 +0000 Robert Hancock wrote:
Looking at Patchwork and Lore, seems like this cover letter may not have made it to the list, though the two patches did? Can resend if it doesn't eventually show up..
I may be wrong but I think it's fine, the cover letter doesn't have much extra info, lore will catch you quoting it here and thread that with the patches.