net: fec: add XDP_TX feature support
Subsystems (5)
Wei Fang
maintainerPATCH v3 1/1 net: fec: add XDP_TX feature support
The XDP_TX feature is not supported before, and all the frames which are deemed to do XDP_TX action actually do the XDP_DROP action. So this patch adds the XDP_TX support to FEC driver.
I tested the performance of XDP_TX feature in XDP_DRV and XDP_SKB modes on i.MX8MM-EVK and i.MX8MP-EVK platforms respectively, and the test steps and results are as follows.
Step 1: Board A connects to the FEC port of the DUT and runs the pktgen_sample03_burst_single_flow.sh script to generate and send burst traffic to DUT. Note that the length of packet was set to 64 bytes and the procotol of packet was UDP in my test scenario.
Step 2: The DUT runs the xdp2 program to transmit received UDP packets back out on the same port where they were received.
root@imx8mmevk:~# ./xdp2 eth0 proto 17: 150326 pkt/s proto 17: 141920 pkt/s proto 17: 147338 pkt/s proto 17: 140783 pkt/s proto 17: 150400 pkt/s proto 17: 134651 pkt/s proto 17: 134676 pkt/s proto 17: 134959 pkt/s proto 17: 148152 pkt/s proto 17: 149885 pkt/s
root@imx8mmevk:~# ./xdp2 -S eth0 proto 17: 131094 pkt/s proto 17: 134691 pkt/s proto 17: 138930 pkt/s proto 17: 129347 pkt/s proto 17: 133050 pkt/s proto 17: 132932 pkt/s proto 17: 136628 pkt/s proto 17: 132964 pkt/s proto 17: 131265 pkt/s proto 17: 135794 pkt/s
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 135817 pkt/s proto 17: 142776 pkt/s proto 17: 142237 pkt/s proto 17: 135673 pkt/s proto 17: 139508 pkt/s proto 17: 147340 pkt/s proto 17: 133329 pkt/s proto 17: 141171 pkt/s proto 17: 146917 pkt/s proto 17: 135488 pkt/s
root@imx8mpevk:~# ./xdp2 -S eth0 proto 17: 133150 pkt/s proto 17: 133127 pkt/s proto 17: 133538 pkt/s proto 17: 133094 pkt/s proto 17: 133690 pkt/s proto 17: 133199 pkt/s proto 17: 133905 pkt/s proto 17: 132908 pkt/s proto 17: 133292 pkt/s proto 17: 133511 pkt/s
---
V2 changes:
According to Jakub's comments, the V2 patch adds two changes.
1. Call txq_trans_cond_update() in fec_enet_xdp_tx_xmit() to avoid tx timeout as XDP shares the queues with kernel stack.
2. Tx processing shouldn't call any XDP (or page pool) APIs if the "budget" is 0.
V3 changes:
1. Remove the second change in V2, because this change has been separated into another patch and it has been submmitted to the upstream [1].
[1] https://lore.kernel.org/r/[email protected]
--- drivers/net/ethernet/freescale/fec.h | 1 + drivers/net/ethernet/freescale/fec_main.c | 80 ++++++++++++++++++----- 2 files changed, 65 insertions(+), 16 deletions(-)
drivers/net/ethernet/freescale/fec.h +1 -0
@@ -547,6 +547,7 @@ enum {enum fec_txbuf_type {FEC_TXBUF_T_SKB ,FEC_TXBUF_T_XDP_NDO , +FEC_TXBUF_T_XDP_TX , };struct fec_tx_buffer {
drivers/net/ethernet/freescale/fec_main.c +64 -16
141 lines not on this page · raw message
Signature · 1 line
2.25.1
Quoted 64 lines · Wei Fang
The
XDP_TXfeature is not supported before, and all the frames which are deemed to doXDP_TXaction actually do theXDP_DROPaction. So this patch adds theXDP_TXsupport to FEC driver.I tested the performance of
XDP_TXfeature inXDP_DRVandXDP_SKBmodes on i.MX8MM-EVK and i.MX8MP-EVK platforms respectively, and the test steps and results are as follows.Step 1: Board A connects to the FEC port of the DUT and runs the
pktgen_sample03_burst_single_flow.shscript to generate and send burst traffic to DUT. Note that the length of packet was set to 64 bytes and the procotol of packet was UDP in my test scenario.Step 2: The DUT runs the xdp2 program to transmit received UDP packets back out on the same port where they were received.
root@imx8mmevk:~# ./xdp2 eth0 proto 17: 150326 pkt/s proto 17: 141920 pkt/s proto 17: 147338 pkt/s proto 17: 140783 pkt/s proto 17: 150400 pkt/s proto 17: 134651 pkt/s proto 17: 134676 pkt/s proto 17: 134959 pkt/s proto 17: 148152 pkt/s proto 17: 149885 pkt/sroot@imx8mmevk:~# ./xdp2 -S eth0 proto 17: 131094 pkt/s proto 17: 134691 pkt/s proto 17: 138930 pkt/s proto 17: 129347 pkt/s proto 17: 133050 pkt/s proto 17: 132932 pkt/s proto 17: 136628 pkt/s proto 17: 132964 pkt/s proto 17: 131265 pkt/s proto 17: 135794 pkt/sroot@imx8mpevk:~# ./xdp2 eth0 proto 17: 135817 pkt/s proto 17: 142776 pkt/s proto 17: 142237 pkt/s proto 17: 135673 pkt/s proto 17: 139508 pkt/s proto 17: 147340 pkt/s proto 17: 133329 pkt/s proto 17: 141171 pkt/s proto 17: 146917 pkt/s proto 17: 135488 pkt/sroot@imx8mpevk:~# ./xdp2 -S eth0 proto 17: 133150 pkt/s proto 17: 133127 pkt/s proto 17: 133538 pkt/s proto 17: 133094 pkt/s proto 17: 133690 pkt/s proto 17: 133199 pkt/s proto 17: 133905 pkt/s proto 17: 132908 pkt/s proto 17: 133292 pkt/s proto 17: 133511 pkt/s
Quoted 1 line
Signed-off-by Wei Fang <[email protected]>
But I thought XDP-related code should go to bpf-next :/ Could somebody clarify this?
Quoted 189 lines
---
V2 changes:
According to Jakub's comments, the V2 patch adds two changes.
1. Calltxq_trans_cond_update()infec_enet_xdp_tx_xmit()to avoid tx timeout as XDP shares the queues with kernel stack.
2. Tx processing shouldn't call any XDP (or page pool) APIs if the "budget" is 0.V3 changes:
1. Remove the second change in V2, because this change has been separated into another patch and it has been submmitted to the upstream [1].
[1] https://lore.kernel.org/r/[email protected]--- drivers/net/ethernet/freescale/fec.h | 1 + drivers/net/ethernet/freescale/fec_main.c | 80 ++++++++++++++++++----- 2 files changed, 65 insertions(+), 16 deletions(-)drivers/net/ethernet/freescale/fec.h +1
-0@@ -547,6 +547,7 @@ enum {enum fec_txbuf_type {FEC_TXBUF_T_SKB ,FEC_TXBUF_T_XDP_NDO , +FEC_TXBUF_T_XDP_TX , };struct fec_tx_buffer {drivers/net/ethernet/freescale/fec_main.c +64
-16@@ -75,6 +75,8 @@static void set_multicast_list (struct net_device *ndev);static void fec_enet_itr_coal_set (struct net_device *ndev); +static int fec_enet_xdp_tx_xmit (struct net_device *ndev, +struct xdp_buff *xdp);#define DRIVER_NAME "fec"@@ -960,7 +962,8 @@ static void fec_enet_bd_init(struct net_device *dev) txq->tx_buf[i].skb =NULL ; } }else { -if (bdp->cbd_bufaddr) +if (bdp->cbd_bufaddr && + txq->tx_buf[i].type ==FEC_TXBUF_T_XDP_NDO ) dma_unmap_single(&fep->pdev->dev, fec32_to_cpu(bdp->cbd_bufaddr), fec16_to_cpu(bdp->cbd_datlen),@@ -1423,7 +1426,8 @@ fec_enet_tx_queue(struct net_device *ndev, u16 queue_id, int budget)break ; xdpf = txq->tx_buf[index].xdp; -if (bdp->cbd_bufaddr) +if (bdp->cbd_bufaddr && + txq->tx_buf[index].type ==FEC_TXBUF_T_XDP_NDO ) dma_unmap_single(&fep->pdev->dev, fec32_to_cpu(bdp->cbd_bufaddr), fec16_to_cpu(bdp->cbd_datlen),@@ -1482,7 +1486,7 @@ fec_enet_tx_queue(struct net_device *ndev, u16 queue_id, int budget)/* Free the sk buffer associated with this last transmit */ dev_kfree_skb_any (skb); }else { -xdp_return_frame (xdpf); +xdp_return_frame_rx_napi (xdpf); txq->tx_buf[index].xdp =NULL ;/* restore default tx buffer type: FEC_TXBUF_T_SKB */ @@ -1573,11 +1577,18 @@ fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, }break ; -default : -bpf_warn_invalid_xdp_action (fep->netdev, prog, act); - fallthrough; -case XDP_TX : + err =fec_enet_xdp_tx_xmit (fep->netdev, xdp); +if (err) { + ret =FEC_ENET_XDP_CONSUMED ; + page =virt_to_head_page (xdp->data); +page_pool_put_page (rxq->page_pool, page, sync, true); + }else { + ret =FEC_ENET_XDP_TX ; + } +break ; + +default :bpf_warn_invalid_xdp_action (fep->netdev, prog, act); fallthrough;@@ -3793,7 +3804,8 @@ fec_enet_xdp_get_tx_queue(struct fec_enet_private *fep, int index)static int fec_enet_txq_xmit_frame (struct fec_enet_private *fep,struct fec_enet_priv_tx_q *txq, -struct xdp_frame *frame) +struct xdp_frame *frame, +bool ndo_xmit) {unsigned int index, status, estatus;struct bufdesc *bdp;@@ -3813,10 +3825,24 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, index =fec_enet_get_bd_index (bdp, &txq->bd); - dma_addr =dma_map_single (&fep->pdev->dev, frame->data, - frame->len,DMA_TO_DEVICE ); -if (dma_mapping_error (&fep->pdev->dev, dma_addr)) -return -ENOMEM ; +if (ndo_xmit) { + dma_addr =dma_map_single (&fep->pdev->dev, frame->data, + frame->len,DMA_TO_DEVICE ); +if (dma_mapping_error (&fep->pdev->dev, dma_addr)) +return -ENOMEM ; + + txq->tx_buf[index].type =FEC_TXBUF_T_XDP_NDO ; + }else { +struct page *page =virt_to_page (frame->data); + + dma_addr =page_pool_get_dma_addr (page) +sizeof (*frame) + + frame->headroom; +dma_sync_single_for_device (&fep->pdev->dev, dma_addr, + frame->len,DMA_BIDIRECTIONAL ); + txq->tx_buf[index].type =FEC_TXBUF_T_XDP_TX ; + } + + txq->tx_buf[index].xdp = frame; status |= (BD_ENET_TX_INTR |BD_ENET_TX_LAST );if (fep->bufdesc_ex)@@ -3835,9 +3861,6 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, ebdp->cbd_esc =cpu_to_fec32 (estatus); } - txq->tx_buf[index].type =FEC_TXBUF_T_XDP_NDO ; - txq->tx_buf[index].xdp = frame; -/* Make sure the updates to rest of the descriptor are performed before * transferring ownership. */ @@ -3863,6 +3886,31 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep,return 0 ; } +static int fec_enet_xdp_tx_xmit (struct net_device *ndev, +struct xdp_buff *xdp) +{ +struct xdp_frame *xdpf =xdp_convert_buff_to_frame (xdp); +struct fec_enet_private *fep =netdev_priv (ndev); +struct fec_enet_priv_tx_q *txq; +int cpu =smp_processor_id (); +struct netdev_queue *nq; +int queue, ret; + + queue =fec_enet_xdp_get_tx_queue (fep, cpu); + txq = fep->tx_queue[queue]; + nq =netdev_get_tx_queue (fep->netdev, queue); + +__netif_tx_lock (nq, cpu); + +/* Avoid tx timeout as XDP shares the queue with kernel stack */ +txq_trans_cond_update (nq); + ret =fec_enet_txq_xmit_frame (fep, txq, xdpf, false); + +__netif_tx_unlock (nq); + +return ret; +} + static int fec_enet_xdp_xmit(struct net_device *dev, int num_frames, struct xdp_frame **frames,@@ -3885,7 +3933,7 @@ static int fec_enet_xdp_xmit(struct net_device *dev,/* Avoid tx timeout as XDP shares the queue with kernel stack */ txq_trans_cond_update (nq);for (i =0 ; i < num_frames; i++) { -if (fec_enet_txq_xmit_frame (fep, txq, frames[i]) <0 ) +if (fec_enet_txq_xmit_frame (fep, txq, frames[i], true) <0 )break ; sent_frames++; }Signature · 1 line
2.25.1
Jakub Kicinski
maintainerQuoted 2 lines · On Tue, 1 Aug 2023 17:34:00 +0200 Larysa Zaremba wrote:
But I thought XDP-related code should go to bpf-next :/ Could somebody clarify this?
AFAIU pure driver XDP implementations end up flowing directly into net-next most of the time. If there are core changes or new API additions in the series - that's when bpf-next becomes important.
Quoted 8 lines · Jakub Kicinski
Quoted 2 lines · On Tue, 1 Aug 2023 17:34:00 +0200 Larysa Zaremba wrote:
But I thought XDP-related code should go to bpf-next :/ Could somebody clarify this?
AFAIU pure driver XDP implementations end up flowing directly into net-next most of the time. If there are core changes or new API additions in the series - that's when bpf-next becomes important.
Thanks for the explanation!
Jakub Kicinski
maintainerdrivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, · line 1585
Quoted 6 lines · On Mon, 31 Jul 2023 14:00:25 +0800 Wei Fang wrote:
drivers/net/ethernet/freescale/fec_main.c +5
-0case XDP_TX : + err =fec_enet_xdp_tx_xmit (fep->netdev, xdp); +if (err) { + ret =FEC_ENET_XDP_CONSUMED ; + page =virt_to_head_page (xdp->data); +page_pool_put_page (rxq->page_pool, page, sync, true);
The error path should call trace_xdp_exception().
Signature · 1 line
pw-bot: cr
Wei Fang
maintainerQuoted 4 lines
-----Original Message-----
Subject: Re: [PATCH V3 net-next] net: fec: addXDP_TXfeature supportOn Mon, 31 Jul 2023 14:00:25 +0800 Wei Fang wrote:
Quoted 8 lines
drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, · line 1585
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c +5
-0case XDP_TX : + err =fec_enet_xdp_tx_xmit (fep->netdev, xdp); +if (err) { + ret =FEC_ENET_XDP_CONSUMED ; + page =virt_to_head_page (xdp->data); +page_pool_put_page (rxq->page_pool, page, sync, true);The error path should call
trace_xdp_exception().
Thanks for your reminder, it made me realize that the error processing of other XDP actions also needs to add exception tracing. I'll add the exception tracing for XDP_TX in V4 patch, and add the tracing for other XDP actions in a separate patch.
Quoted 8 lines · Wei Fang
The
XDP_TXfeature is not supported before, and all the frames which are deemed to doXDP_TXaction actually do theXDP_DROPaction. So this patch adds theXDP_TXsupport to FEC driver.I tested the performance of
XDP_TXfeature inXDP_DRVandXDP_SKBmodes on i.MX8MM-EVK and i.MX8MP-EVK platforms respectively, and the test steps and results are as follows.
Quoted 8 lines
Step 1: Board A connects to the FEC port of the DUT and runs the
pktgen_sample03_burst_single_flow.shscript to generate and send burst traffic to DUT. Note that the length of packet was set to 64 bytes and the procotol of packet was UDP in my test scenario.Step 2: The DUT runs the xdp2 program to transmit received UDP packets back out on the same port where they were received.
Below test result runs should have some more explaination, please. (more inline code comments below)
Quoted 36 lines
root@imx8mmevk:~# ./xdp2 eth0 proto 17: 150326 pkt/s proto 17: 141920 pkt/s proto 17: 147338 pkt/s proto 17: 140783 pkt/s proto 17: 150400 pkt/s proto 17: 134651 pkt/s proto 17: 134676 pkt/s proto 17: 134959 pkt/s proto 17: 148152 pkt/s proto 17: 149885 pkt/sroot@imx8mmevk:~# ./xdp2 -S eth0 proto 17: 131094 pkt/s proto 17: 134691 pkt/s proto 17: 138930 pkt/s proto 17: 129347 pkt/s proto 17: 133050 pkt/s proto 17: 132932 pkt/s proto 17: 136628 pkt/s proto 17: 132964 pkt/s proto 17: 131265 pkt/s proto 17: 135794 pkt/sroot@imx8mpevk:~# ./xdp2 eth0 proto 17: 135817 pkt/s proto 17: 142776 pkt/s proto 17: 142237 pkt/s proto 17: 135673 pkt/s proto 17: 139508 pkt/s proto 17: 147340 pkt/s proto 17: 133329 pkt/s proto 17: 141171 pkt/s proto 17: 146917 pkt/s proto 17: 135488 pkt/s
Quoted 12 lines
root@imx8mpevk:~# ./xdp2 -S eth0 proto 17: 133150 pkt/s proto 17: 133127 pkt/s proto 17: 133538 pkt/s proto 17: 133094 pkt/s proto 17: 133690 pkt/s proto 17: 133199 pkt/s proto 17: 133905 pkt/s proto 17: 132908 pkt/s proto 17: 133292 pkt/s proto 17: 133511 pkt/s
For this driver, I would like to see a benchmark comparison between XDP_TX and XDP_REDIRECT.
As below code does could create a situation where XDP_REDIRECT is just as fast as XDP_TX. (Note, that I expect XDP_TX to be faster than XDP_REDIRECT.)
Quoted 74 lines
Signed-off-by Wei Fang <[email protected]>---
V2 changes:
According to Jakub's comments, the V2 patch adds two changes.
1. Calltxq_trans_cond_update()infec_enet_xdp_tx_xmit()to avoid tx timeout as XDP shares the queues with kernel stack.
2. Tx processing shouldn't call any XDP (or page pool) APIs if the "budget" is 0.V3 changes:
1. Remove the second change in V2, because this change has been separated into another patch and it has been submmitted to the upstream [1].
[1] https://lore.kernel.org/r/[email protected]--- drivers/net/ethernet/freescale/fec.h | 1 + drivers/net/ethernet/freescale/fec_main.c | 80 ++++++++++++++++++----- 2 files changed, 65 insertions(+), 16 deletions(-)drivers/net/ethernet/freescale/fec.h +1
-0@@ -547,6 +547,7 @@ enum {enum fec_txbuf_type {FEC_TXBUF_T_SKB ,FEC_TXBUF_T_XDP_NDO , +FEC_TXBUF_T_XDP_TX , };struct fec_tx_buffer {drivers/net/ethernet/freescale/fec_main.c +7
-3@@ -75,6 +75,8 @@static void set_multicast_list (struct net_device *ndev);static void fec_enet_itr_coal_set (struct net_device *ndev); +static int fec_enet_xdp_tx_xmit (struct net_device *ndev, +struct xdp_buff *xdp);#define DRIVER_NAME "fec"@@ -960,7 +962,8 @@ static void fec_enet_bd_init(struct net_device *dev) txq->tx_buf[i].skb =NULL ; } }else { -if (bdp->cbd_bufaddr) +if (bdp->cbd_bufaddr && + txq->tx_buf[i].type ==FEC_TXBUF_T_XDP_NDO ) dma_unmap_single(&fep->pdev->dev, fec32_to_cpu(bdp->cbd_bufaddr), fec16_to_cpu(bdp->cbd_datlen),@@ -1423,7 +1426,8 @@ fec_enet_tx_queue(struct net_device *ndev, u16 queue_id, int budget)break ; xdpf = txq->tx_buf[index].xdp; -if (bdp->cbd_bufaddr) +if (bdp->cbd_bufaddr && + txq->tx_buf[index].type ==FEC_TXBUF_T_XDP_NDO ) dma_unmap_single(&fep->pdev->dev, fec32_to_cpu(bdp->cbd_bufaddr), fec16_to_cpu(bdp->cbd_datlen),@@ -1482,7 +1486,7 @@ fec_enet_tx_queue(struct net_device *ndev, u16 queue_id, int budget)/* Free the sk buffer associated with this last transmit */ dev_kfree_skb_any (skb); }else { -xdp_return_frame (xdpf); +xdp_return_frame_rx_napi (xdpf); txq->tx_buf[index].xdp =NULL ;/* restore default tx buffer type: FEC_TXBUF_T_SKB */
drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, · line 1581
Quoted 9 lines
drivers/net/ethernet/freescale/fec_main.c +1
-4@@ -1573,11 +1577,18 @@ fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, }break ; -default : -bpf_warn_invalid_xdp_action (fep->netdev, prog, act); - fallthrough; -case XDP_TX : + err =fec_enet_xdp_tx_xmit (fep->netdev, xdp);
You should pass along the "sync" length value to fec_enet_xdp_tx_xmit(). Because we know DMA comes from same device (it is already DMA mapped to), then we can do a DMA sync "to_device" with only the sync length.
drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, · line 1582
Quoted 1 line
drivers/net/ethernet/freescale/fec_main.c +1
-0+if (err) {
Add an unlikely(err) or do like above case XDP_REDIRECT, where it takes the likely case "if (!err)" first.
Quoted 13 lines
drivers/net/ethernet/freescale/fec_main.c +9
-0+ ret =FEC_ENET_XDP_CONSUMED ; + page =virt_to_head_page (xdp->data); +page_pool_put_page (rxq->page_pool, page, sync, true); + }else { + ret =FEC_ENET_XDP_TX ; + } +break ; + +default :bpf_warn_invalid_xdp_action (fep->netdev, prog, act); fallthrough;
drivers/net/ethernet/freescale/fec_main.c fec_enet_xdp_get_tx_queue(struct fec_enet_private *fep, int index) · line 3808
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c +2
-1@@ -3793,7 +3804,8 @@ fec_enet_xdp_get_tx_queue(struct fec_enet_private *fep, int index)static int fec_enet_txq_xmit_frame (struct fec_enet_private *fep,struct fec_enet_priv_tx_q *txq, -struct xdp_frame *frame) +struct xdp_frame *frame, +bool ndo_xmit)
E.g add parameter dma_sync_len.
Quoted 15 lines
drivers/net/ethernet/freescale/fec_main.c +4
-4{unsigned int index, status, estatus;struct bufdesc *bdp;@@ -3813,10 +3825,24 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, index =fec_enet_get_bd_index (bdp, &txq->bd); - dma_addr =dma_map_single (&fep->pdev->dev, frame->data, - frame->len,DMA_TO_DEVICE ); -if (dma_mapping_error (&fep->pdev->dev, dma_addr)) -return -ENOMEM ; +if (ndo_xmit) { + dma_addr =dma_map_single (&fep->pdev->dev, frame->data, + frame->len,DMA_TO_DEVICE ); +if (dma_mapping_error (&fep->pdev->dev, dma_addr))
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3841
Quoted 10 lines
drivers/net/ethernet/freescale/fec_main.c +10
-0+return -ENOMEM ; + + txq->tx_buf[index].type =FEC_TXBUF_T_XDP_NDO ; + }else { +struct page *page =virt_to_page (frame->data); + + dma_addr =page_pool_get_dma_addr (page) +sizeof (*frame) + + frame->headroom; +dma_sync_single_for_device (&fep->pdev->dev, dma_addr, + frame->len,DMA_BIDIRECTIONAL );
Optimization: use dma_sync_len here instead of frame->len.
Quoted 18 lines
drivers/net/ethernet/freescale/fec_main.c +4
-3+ txq->tx_buf[index].type =FEC_TXBUF_T_XDP_TX ; + } + + txq->tx_buf[index].xdp = frame; status |= (BD_ENET_TX_INTR |BD_ENET_TX_LAST );if (fep->bufdesc_ex)@@ -3835,9 +3861,6 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, ebdp->cbd_esc =cpu_to_fec32 (estatus); } - txq->tx_buf[index].type =FEC_TXBUF_T_XDP_NDO ; - txq->tx_buf[index].xdp = frame; -/* Make sure the updates to rest of the descriptor are performed before * transferring ownership. */
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3891
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c +3
-0@@ -3863,6 +3886,31 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep,return 0 ; } +static int fec_enet_xdp_tx_xmit (struct net_device *ndev, +struct xdp_buff *xdp) +{
E.g add parameter dma_sync_len.
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3892
Quoted 1 line
drivers/net/ethernet/freescale/fec_main.c +1
-0+struct xdp_frame *xdpf =xdp_convert_buff_to_frame (xdp);
XDP_TX can avoid this conversion to xdp_frame.
It would requires some refactor of fec_enet_txq_xmit_frame().
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3903
Quoted 11 lines
drivers/net/ethernet/freescale/fec_main.c +11
-0+struct fec_enet_private *fep =netdev_priv (ndev); +struct fec_enet_priv_tx_q *txq; +int cpu =smp_processor_id (); +struct netdev_queue *nq; +int queue, ret; + + queue =fec_enet_xdp_get_tx_queue (fep, cpu); + txq = fep->tx_queue[queue]; + nq =netdev_get_tx_queue (fep->netdev, queue); + +__netif_tx_lock (nq, cpu);
It is sad that XDP_TX takes a lock for each frame.
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3907
Quoted 4 lines
drivers/net/ethernet/freescale/fec_main.c +4
-0+ +/* Avoid tx timeout as XDP shares the queue with kernel stack */ +txq_trans_cond_update (nq); + ret =fec_enet_txq_xmit_frame (fep, txq, xdpf, false);
Add/pass parameter dma_sync_len to fec_enet_txq_xmit_frame().
Quoted 18 lines
drivers/net/ethernet/freescale/fec_main.c +7
-1+ +__netif_tx_unlock (nq); + +return ret; +} + static int fec_enet_xdp_xmit(struct net_device *dev, int num_frames, struct xdp_frame **frames,@@ -3885,7 +3933,7 @@ static int fec_enet_xdp_xmit(struct net_device *dev,/* Avoid tx timeout as XDP shares the queue with kernel stack */ txq_trans_cond_update (nq);for (i =0 ; i < num_frames; i++) { -if (fec_enet_txq_xmit_frame (fep, txq, frames[i]) <0 ) +if (fec_enet_txq_xmit_frame (fep, txq, frames[i], true) <0 )break ; sent_frames++; }
Wei Fang
maintainerQuoted 71 lines · Wei Fang
Quoted 8 lines
The
XDP_TXfeature is not supported before, and all the frames which are deemed to doXDP_TXaction actually do theXDP_DROPaction. So this patch adds theXDP_TXsupport to FEC driver.I tested the performance of
XDP_TXfeature inXDP_DRVandXDP_SKBmodes on i.MX8MM-EVK and i.MX8MP-EVK platforms respectively, and the test steps and results are as follows.Quoted 8 lines
Step 1: Board A connects to the FEC port of the DUT and runs the
pktgen_sample03_burst_single_flow.shscript to generate and send burst traffic to DUT. Note that the length of packet was set to 64 bytes and the procotol of packet was UDP in my test scenario.Step 2: The DUT runs the xdp2 program to transmit received UDP packets back out on the same port where they were received.
Below test result runs should have some more explaination, please. (more inline code comments below)
Quoted 36 lines
root@imx8mmevk:~# ./xdp2 eth0 proto 17: 150326 pkt/s proto 17: 141920 pkt/s proto 17: 147338 pkt/s proto 17: 140783 pkt/s proto 17: 150400 pkt/s proto 17: 134651 pkt/s proto 17: 134676 pkt/s proto 17: 134959 pkt/s proto 17: 148152 pkt/s proto 17: 149885 pkt/sroot@imx8mmevk:~# ./xdp2 -S eth0 proto 17: 131094 pkt/s proto 17: 134691 pkt/s proto 17: 138930 pkt/s proto 17: 129347 pkt/s proto 17: 133050 pkt/s proto 17: 132932 pkt/s proto 17: 136628 pkt/s proto 17: 132964 pkt/s proto 17: 131265 pkt/s proto 17: 135794 pkt/sroot@imx8mpevk:~# ./xdp2 eth0 proto 17: 135817 pkt/s proto 17: 142776 pkt/s proto 17: 142237 pkt/s proto 17: 135673 pkt/s proto 17: 139508 pkt/s proto 17: 147340 pkt/s proto 17: 133329 pkt/s proto 17: 141171 pkt/s proto 17: 146917 pkt/s proto 17: 135488 pkt/sQuoted 12 lines
root@imx8mpevk:~# ./xdp2 -S eth0 proto 17: 133150 pkt/s proto 17: 133127 pkt/s proto 17: 133538 pkt/s proto 17: 133094 pkt/s proto 17: 133690 pkt/s proto 17: 133199 pkt/s proto 17: 133905 pkt/s proto 17: 132908 pkt/s proto 17: 133292 pkt/s proto 17: 133511 pkt/s
Quoted 3 lines
For this driver, I would like to see a benchmark comparison between
XDP_TXandXDP_REDIRECT.
Okay, I'll do a comparison test.
Quoted 4 lines
As below code does could create a situation where
XDP_REDIRECTis just as fast asXDP_TX. (Note, that I expectXDP_TXto be faster thanXDP_REDIRECT.)
Could you explain why you expect XDP_TX should be faster than XDP_REDIRECT? What's the problem if XDP_TX is as fast ad XDP_REDIRECT?
Quoted 235 lines
Quoted 9 lines
Signed-off-by Wei Fang <[email protected]>---
V2 changes:
According to Jakub's comments, the V2 patch adds two changes.
1. Calltxq_trans_cond_update()infec_enet_xdp_tx_xmit()to avoid tx timeout as XDP shares the queues with kernel stack.
2. Tx processing shouldn't call any XDP (or page pool) APIs if the "budget" is 0.Quoted 5 lines
V3 changes:
1. Remove the second change in V2, because this change has been separated into another patch and it has been submmitted to the upstream [1].
[1]https://lore.k/
ernel.org%2Fr%2F20230725074148.2936402-1-wei.fang%40nxp.com&data= 05%7C01%7Cwei.fang%40nxp.com%7C9a2fc5bab84947e4bea608db933aa5 e9%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638265652320 018962%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wc xe8nBeLS9uQrbphuNI18owgDNHJq9478V53KybWB8%3D&reserved=0Quoted 3 lines
--- drivers/net/ethernet/freescale/fec.h | 1 + drivers/net/ethernet/freescale/fec_main.c | 80++++++++++++++++++-----
Review
Quoted 3 lines
2 files changed, 65 insertions(+), 16 deletions(-)(quoted diff) +0
-0
b/drivers/net/ethernet/freescale/fec.hReview
Quoted 12 lines
index
8f1edcca96c4..f35445bddc7a100644drivers/net/ethernet/freescale/fec.h +1
-0@@ -547,6 +547,7 @@ enum {enum fec_txbuf_type {FEC_TXBUF_T_SKB ,FEC_TXBUF_T_XDP_NDO , +FEC_TXBUF_T_XDP_TX , };struct fec_tx_buffer {(quoted diff) +0
-0
b/drivers/net/ethernet/freescale/fec_main.cdrivers/net/ethernet/freescale/fec_main.c static void fec_enet_bd_init(struct net_device
Quoted 13 lines
index
14d0dc7ba3c9..2068fe95504e100644drivers/net/ethernet/freescale/fec_main.c +2
-0@@ -75,6 +75,8 @@static void set_multicast_list (struct net_device *ndev);static void fec_enet_itr_coal_set (struct net_device *ndev); +static int fec_enet_xdp_tx_xmit (struct net_device *ndev, +struct xdp_buff *xdp);#define DRIVER_NAME "fec"@@ -960,7 +962,8 @@ static void fec_enet_bd_init(struct net_device*dev)
drivers/net/ethernet/freescale/fec_main.c fec_enet_tx_queue(struct net_device *ndev, u16
Quoted 10 lines
drivers/net/ethernet/freescale/fec_main.c +2
-1txq->tx_buf[i].skb =NULL ; } }else { -if (bdp->cbd_bufaddr) +if (bdp->cbd_bufaddr && + txq->tx_buf[i].type ==FEC_TXBUF_T_XDP_NDO ) dma_unmap_single(&fep->pdev->dev, fec32_to_cpu(bdp->cbd_bufaddr), fec16_to_cpu(bdp->cbd_datlen),@@ -1423,7 +1426,8 @@ fec_enet_tx_queue(struct net_device *ndev, u16
queue_id, int budget)drivers/net/ethernet/freescale/fec_main.c fec_enet_tx_queue(struct net_device *ndev, u16
Quoted 10 lines
drivers/net/ethernet/freescale/fec_main.c +2
-1break ; xdpf = txq->tx_buf[index].xdp; -if (bdp->cbd_bufaddr) +if (bdp->cbd_bufaddr && + txq->tx_buf[index].type ==FEC_TXBUF_T_XDP_NDO ) dma_unmap_single(&fep->pdev->dev, fec32_to_cpu(bdp->cbd_bufaddr), fec16_to_cpu(bdp->cbd_datlen),@@ -1482,7 +1486,7 @@ fec_enet_tx_queue(struct net_device *ndev, u16
queue_id, int budget)drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private
Quoted 9 lines
drivers/net/ethernet/freescale/fec_main.c +1
-1/* Free the sk buffer associated with this last transmit */ dev_kfree_skb_any (skb); }else { -xdp_return_frame (xdpf); +xdp_return_frame_rx_napi (xdpf); txq->tx_buf[index].xdp =NULL ;/* restore default tx buffer type: FEC_TXBUF_T_SKB */ @@ -1573,11 +1577,18 @@ fec_enet_run_xdp(struct fec_enet_private*fep, struct
bpf_prog*prog,drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, · line 1581
Quoted 9 lines
drivers/net/ethernet/freescale/fec_main.c +1
-4}break ; -default : -bpf_warn_invalid_xdp_action (fep->netdev, prog, act); - fallthrough; -case XDP_TX : + err =fec_enet_xdp_tx_xmit (fep->netdev, xdp);You should pass along the "sync" length value to
fec_enet_xdp_tx_xmit(). Because we know DMA comes from same device (it is already DMA mapped to), then we can do a DMA sync "to_device" with only the sync length.drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, · line 1582
Quoted 1 line
drivers/net/ethernet/freescale/fec_main.c +1
-0+if (err) {Add an unlikely(err) or do like above case
XDP_REDIRECT, where it takes the likely case "if (!err)" first.drivers/net/ethernet/freescale/fec_main.c fec_enet_xdp_get_tx_queue(struct
Quoted 13 lines
drivers/net/ethernet/freescale/fec_main.c +9
-0+ ret =FEC_ENET_XDP_CONSUMED ; + page =virt_to_head_page (xdp->data); +page_pool_put_page (rxq->page_pool, page, sync, true); + }else { + ret =FEC_ENET_XDP_TX ; + } +break ; + +default :bpf_warn_invalid_xdp_action (fep->netdev, prog, act); fallthrough;@@ -3793,7 +3804,8 @@ fec_enet_xdp_get_tx_queue(struct
fec_enet_private*fep, int index)drivers/net/ethernet/freescale/fec_main.c fec_enet_xdp_get_tx_queue(struct fec_enet_private *fep, int index) · line 3808
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c +2
-1static int fec_enet_txq_xmit_frame (struct fec_enet_private *fep,struct fec_enet_priv_tx_q *txq, -struct xdp_frame *frame) +struct xdp_frame *frame, +bool ndo_xmit)E.g add parameter
dma_sync_len.drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct
Quoted 4 lines
drivers/net/ethernet/freescale/fec_main.c +0
-0{unsigned int index, status, estatus;struct bufdesc *bdp;@@ -3813,10 +3825,24 @@ static int fec_enet_txq_xmit_frame(struct
fec_enet_private*fep,Quoted 11 lines
drivers/net/ethernet/freescale/fec_main.c +4
-4index =fec_enet_get_bd_index (bdp, &txq->bd); - dma_addr =dma_map_single (&fep->pdev->dev, frame->data, - frame->len,DMA_TO_DEVICE ); -if (dma_mapping_error (&fep->pdev->dev, dma_addr)) -return -ENOMEM ; +if (ndo_xmit) { + dma_addr =dma_map_single (&fep->pdev->dev, frame->data, + frame->len,DMA_TO_DEVICE ); +if (dma_mapping_error (&fep->pdev->dev, dma_addr))drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3841
Quoted 10 lines
drivers/net/ethernet/freescale/fec_main.c +10
-0+return -ENOMEM ; + + txq->tx_buf[index].type =FEC_TXBUF_T_XDP_NDO ; + }else { +struct page *page =virt_to_page (frame->data); + + dma_addr =page_pool_get_dma_addr (page) +sizeof (*frame) + + frame->headroom; +dma_sync_single_for_device (&fep->pdev->dev, dma_addr, + frame->len,DMA_BIDIRECTIONAL );Optimization: use
dma_sync_lenhere instead offrame->len.drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct
Quoted 8 lines
drivers/net/ethernet/freescale/fec_main.c +4
-0+ txq->tx_buf[index].type =FEC_TXBUF_T_XDP_TX ; + } + + txq->tx_buf[index].xdp = frame; status |= (BD_ENET_TX_INTR |BD_ENET_TX_LAST );if (fep->bufdesc_ex)@@ -3835,9 +3861,6 @@ static int fec_enet_txq_xmit_frame(struct
fec_enet_private*fep,drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep,
Quoted 7 lines
drivers/net/ethernet/freescale/fec_main.c +0
-3ebdp->cbd_esc =cpu_to_fec32 (estatus); } - txq->tx_buf[index].type =FEC_TXBUF_T_XDP_NDO ; - txq->tx_buf[index].xdp = frame; -/* Make sure the updates to rest of the descriptor are performed before
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct
Quoted 3 lines
drivers/net/ethernet/freescale/fec_main.c +0
-0* transferring ownership. */ @@ -3863,6 +3886,31 @@ static int fec_enet_txq_xmit_frame(struct
fec_enet_private*fep,drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3891
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c +3
-0return 0 ; } +static int fec_enet_xdp_tx_xmit (struct net_device *ndev, +struct xdp_buff *xdp) +{E.g add parameter
dma_sync_len.drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3892
Quoted 1 line
drivers/net/ethernet/freescale/fec_main.c +1
-0+struct xdp_frame *xdpf =xdp_convert_buff_to_frame (xdp);
XDP_TXcan avoid this conversion toxdp_frame.
It would requires some refactor offec_enet_txq_xmit_frame().drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3903
Quoted 11 lines
drivers/net/ethernet/freescale/fec_main.c +11
-0+struct fec_enet_private *fep =netdev_priv (ndev); +struct fec_enet_priv_tx_q *txq; +int cpu =smp_processor_id (); +struct netdev_queue *nq; +int queue, ret; + + queue =fec_enet_xdp_get_tx_queue (fep, cpu); + txq = fep->tx_queue[queue]; + nq =netdev_get_tx_queue (fep->netdev, queue); + +__netif_tx_lock (nq, cpu);It is sad that
XDP_TXtakes a lock for each frame.drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3907
Quoted 4 lines
drivers/net/ethernet/freescale/fec_main.c +4
-0+ +/* Avoid tx timeout as XDP shares the queue with kernel stack */ +txq_trans_cond_update (nq); + ret =fec_enet_txq_xmit_frame (fep, txq, xdpf, false);Add/pass parameter
dma_sync_lentofec_enet_txq_xmit_frame().drivers/net/ethernet/freescale/fec_main.c static int fec_enet_xdp_xmit(struct net_device
Quoted 10 lines
drivers/net/ethernet/freescale/fec_main.c +6
-0+ +__netif_tx_unlock (nq); + +return ret; +} + static int fec_enet_xdp_xmit(struct net_device *dev, int num_frames, struct xdp_frame **frames,@@ -3885,7 +3933,7 @@ static int fec_enet_xdp_xmit(struct net_device*dev,
Quoted 8 lines
drivers/net/ethernet/freescale/fec_main.c +1
-1/* Avoid tx timeout as XDP shares the queue with kernel stack */ txq_trans_cond_update (nq);for (i =0 ; i < num_frames; i++) { -if (fec_enet_txq_xmit_frame (fep, txq, frames[i]) <0 ) +if (fec_enet_txq_xmit_frame (fep, txq, frames[i], true) <0 )break ; sent_frames++; }
Quoted 71 lines · Wei Fang
Quoted 71 lines · Wei Fang
Quoted 8 lines
The
XDP_TXfeature is not supported before, and all the frames which are deemed to doXDP_TXaction actually do theXDP_DROPaction. So this patch adds theXDP_TXsupport to FEC driver.I tested the performance of
XDP_TXfeature inXDP_DRVandXDP_SKBmodes on i.MX8MM-EVK and i.MX8MP-EVK platforms respectively, and the test steps and results are as follows.Quoted 8 lines
Step 1: Board A connects to the FEC port of the DUT and runs the
pktgen_sample03_burst_single_flow.shscript to generate and send burst traffic to DUT. Note that the length of packet was set to 64 bytes and the procotol of packet was UDP in my test scenario.Step 2: The DUT runs the xdp2 program to transmit received UDP packets back out on the same port where they were received.
Below test result runs should have some more explaination, please. (more inline code comments below)
Quoted 48 lines
root@imx8mmevk:~# ./xdp2 eth0 proto 17: 150326 pkt/s proto 17: 141920 pkt/s proto 17: 147338 pkt/s proto 17: 140783 pkt/s proto 17: 150400 pkt/s proto 17: 134651 pkt/s proto 17: 134676 pkt/s proto 17: 134959 pkt/s proto 17: 148152 pkt/s proto 17: 149885 pkt/sroot@imx8mmevk:~# ./xdp2 -S eth0 proto 17: 131094 pkt/s proto 17: 134691 pkt/s proto 17: 138930 pkt/s proto 17: 129347 pkt/s proto 17: 133050 pkt/s proto 17: 132932 pkt/s proto 17: 136628 pkt/s proto 17: 132964 pkt/s proto 17: 131265 pkt/s proto 17: 135794 pkt/sroot@imx8mpevk:~# ./xdp2 eth0 proto 17: 135817 pkt/s proto 17: 142776 pkt/s proto 17: 142237 pkt/s proto 17: 135673 pkt/s proto 17: 139508 pkt/s proto 17: 147340 pkt/s proto 17: 133329 pkt/s proto 17: 141171 pkt/s proto 17: 146917 pkt/s proto 17: 135488 pkt/sroot@imx8mpevk:~# ./xdp2 -S eth0 proto 17: 133150 pkt/s proto 17: 133127 pkt/s proto 17: 133538 pkt/s proto 17: 133094 pkt/s proto 17: 133690 pkt/s proto 17: 133199 pkt/s proto 17: 133905 pkt/s proto 17: 132908 pkt/s proto 17: 133292 pkt/s proto 17: 133511 pkt/s
Quoted 4 lines
Quoted 3 lines
For this driver, I would like to see a benchmark comparison between
XDP_TXandXDP_REDIRECT.Okay, I'll do a comparison test.
Thanks.
Quoted 6 lines
Quoted 5 lines
As below code does could create a situation where
XDP_REDIRECTis just as fast asXDP_TX. (Note, that I expectXDP_TXto be faster thanXDP_REDIRECT.)Could you explain why you expect
XDP_TXshould be faster thanXDP_REDIRECT?
First of all: I explained the changes needed to improve XDP_TX, below as comments on the code. Please read and address.
XDP_TX should always be faster than XDP_REDIRECT, because it happens locally in the driver and doesn't have to go through any generic xdp_do_redirect code.
Like benchmarks shown in our XDP-paper[1] on mlx5 the graph[2] shows XDP_TX vs XDP_REDIRECT vs. DPDK.
[1] https://dl.acm.org/doi/10.1145/3281411.3281443
[2]
https://github.com/xdp-project/xdp-paper/blob/master/figures/redirect-test.pdf
Quoted 238 lines
What's the problem if
XDP_TXis as fast adXDP_REDIRECT?Quoted 235 lines
Quoted 9 lines
Signed-off-by Wei Fang <[email protected]>---
V2 changes:
According to Jakub's comments, the V2 patch adds two changes.
1. Calltxq_trans_cond_update()infec_enet_xdp_tx_xmit()to avoid tx timeout as XDP shares the queues with kernel stack.
2. Tx processing shouldn't call any XDP (or page pool) APIs if the "budget" is 0.Quoted 5 lines
V3 changes:
1. Remove the second change in V2, because this change has been separated into another patch and it has been submmitted to the upstream [1].
[1]https://lore.k/
ernel.org%2Fr%2F20230725074148.2936402-1-wei.fang%40nxp.com&data= 05%7C01%7Cwei.fang%40nxp.com%7C9a2fc5bab84947e4bea608db933aa5 e9%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638265652320 018962%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wc xe8nBeLS9uQrbphuNI18owgDNHJq9478V53KybWB8%3D&reserved=0Quoted 3 lines
--- drivers/net/ethernet/freescale/fec.h | 1 + drivers/net/ethernet/freescale/fec_main.c | 80++++++++++++++++++-----
Review
Quoted 3 lines
2 files changed, 65 insertions(+), 16 deletions(-)(quoted diff) +0
-0
b/drivers/net/ethernet/freescale/fec.hReview
Quoted 12 lines
index
8f1edcca96c4..f35445bddc7a100644drivers/net/ethernet/freescale/fec.h +1
-0@@ -547,6 +547,7 @@ enum {enum fec_txbuf_type {FEC_TXBUF_T_SKB ,FEC_TXBUF_T_XDP_NDO , +FEC_TXBUF_T_XDP_TX , };struct fec_tx_buffer {(quoted diff) +0
-0
b/drivers/net/ethernet/freescale/fec_main.cdrivers/net/ethernet/freescale/fec_main.c static void fec_enet_bd_init(struct net_device
Quoted 13 lines
index
14d0dc7ba3c9..2068fe95504e100644drivers/net/ethernet/freescale/fec_main.c +2
-0@@ -75,6 +75,8 @@static void set_multicast_list (struct net_device *ndev);static void fec_enet_itr_coal_set (struct net_device *ndev); +static int fec_enet_xdp_tx_xmit (struct net_device *ndev, +struct xdp_buff *xdp);#define DRIVER_NAME "fec"@@ -960,7 +962,8 @@ static void fec_enet_bd_init(struct net_device*dev)
drivers/net/ethernet/freescale/fec_main.c fec_enet_tx_queue(struct net_device *ndev, u16
Quoted 10 lines
drivers/net/ethernet/freescale/fec_main.c +2
-1txq->tx_buf[i].skb =NULL ; } }else { -if (bdp->cbd_bufaddr) +if (bdp->cbd_bufaddr && + txq->tx_buf[i].type ==FEC_TXBUF_T_XDP_NDO ) dma_unmap_single(&fep->pdev->dev, fec32_to_cpu(bdp->cbd_bufaddr), fec16_to_cpu(bdp->cbd_datlen),@@ -1423,7 +1426,8 @@ fec_enet_tx_queue(struct net_device *ndev, u16
queue_id, int budget)drivers/net/ethernet/freescale/fec_main.c fec_enet_tx_queue(struct net_device *ndev, u16
Quoted 10 lines
drivers/net/ethernet/freescale/fec_main.c +2
-1break ; xdpf = txq->tx_buf[index].xdp; -if (bdp->cbd_bufaddr) +if (bdp->cbd_bufaddr && + txq->tx_buf[index].type ==FEC_TXBUF_T_XDP_NDO ) dma_unmap_single(&fep->pdev->dev, fec32_to_cpu(bdp->cbd_bufaddr), fec16_to_cpu(bdp->cbd_datlen),@@ -1482,7 +1486,7 @@ fec_enet_tx_queue(struct net_device *ndev, u16
queue_id, int budget)drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private
Quoted 9 lines
drivers/net/ethernet/freescale/fec_main.c +1
-1/* Free the sk buffer associated with this last transmit */ dev_kfree_skb_any (skb); }else { -xdp_return_frame (xdpf); +xdp_return_frame_rx_napi (xdpf); txq->tx_buf[index].xdp =NULL ;/* restore default tx buffer type: FEC_TXBUF_T_SKB */ @@ -1573,11 +1577,18 @@ fec_enet_run_xdp(struct fec_enet_private*fep, struct
bpf_prog*prog,drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, · line 1581
Quoted 9 lines
drivers/net/ethernet/freescale/fec_main.c +1
-4}break ; -default : -bpf_warn_invalid_xdp_action (fep->netdev, prog, act); - fallthrough; -case XDP_TX : + err =fec_enet_xdp_tx_xmit (fep->netdev, xdp);You should pass along the "sync" length value to
fec_enet_xdp_tx_xmit(). Because we know DMA comes from same device (it is already DMA mapped to), then we can do a DMA sync "to_device" with only the sync length.drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, · line 1582
Quoted 1 line
drivers/net/ethernet/freescale/fec_main.c +1
-0+if (err) {Add an unlikely(err) or do like above case
XDP_REDIRECT, where it takes the likely case "if (!err)" first.drivers/net/ethernet/freescale/fec_main.c fec_enet_xdp_get_tx_queue(struct
Quoted 13 lines
drivers/net/ethernet/freescale/fec_main.c +9
-0+ ret =FEC_ENET_XDP_CONSUMED ; + page =virt_to_head_page (xdp->data); +page_pool_put_page (rxq->page_pool, page, sync, true); + }else { + ret =FEC_ENET_XDP_TX ; + } +break ; + +default :bpf_warn_invalid_xdp_action (fep->netdev, prog, act); fallthrough;@@ -3793,7 +3804,8 @@ fec_enet_xdp_get_tx_queue(struct
fec_enet_private*fep, int index)drivers/net/ethernet/freescale/fec_main.c fec_enet_xdp_get_tx_queue(struct fec_enet_private *fep, int index) · line 3808
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c +2
-1static int fec_enet_txq_xmit_frame (struct fec_enet_private *fep,struct fec_enet_priv_tx_q *txq, -struct xdp_frame *frame) +struct xdp_frame *frame, +bool ndo_xmit)E.g add parameter
dma_sync_len.drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct
Quoted 4 lines
drivers/net/ethernet/freescale/fec_main.c +0
-0{unsigned int index, status, estatus;struct bufdesc *bdp;@@ -3813,10 +3825,24 @@ static int fec_enet_txq_xmit_frame(struct
fec_enet_private*fep,Quoted 11 lines
drivers/net/ethernet/freescale/fec_main.c +4
-4index =fec_enet_get_bd_index (bdp, &txq->bd); - dma_addr =dma_map_single (&fep->pdev->dev, frame->data, - frame->len,DMA_TO_DEVICE ); -if (dma_mapping_error (&fep->pdev->dev, dma_addr)) -return -ENOMEM ; +if (ndo_xmit) { + dma_addr =dma_map_single (&fep->pdev->dev, frame->data, + frame->len,DMA_TO_DEVICE ); +if (dma_mapping_error (&fep->pdev->dev, dma_addr))drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3841
Quoted 10 lines
drivers/net/ethernet/freescale/fec_main.c +10
-0+return -ENOMEM ; + + txq->tx_buf[index].type =FEC_TXBUF_T_XDP_NDO ; + }else { +struct page *page =virt_to_page (frame->data); + + dma_addr =page_pool_get_dma_addr (page) +sizeof (*frame) + + frame->headroom; +dma_sync_single_for_device (&fep->pdev->dev, dma_addr, + frame->len,DMA_BIDIRECTIONAL );Optimization: use
dma_sync_lenhere instead offrame->len.drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct
Quoted 8 lines
drivers/net/ethernet/freescale/fec_main.c +4
-0+ txq->tx_buf[index].type =FEC_TXBUF_T_XDP_TX ; + } + + txq->tx_buf[index].xdp = frame; status |= (BD_ENET_TX_INTR |BD_ENET_TX_LAST );if (fep->bufdesc_ex)@@ -3835,9 +3861,6 @@ static int fec_enet_txq_xmit_frame(struct
fec_enet_private*fep,drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep,
Quoted 7 lines
drivers/net/ethernet/freescale/fec_main.c +0
-3ebdp->cbd_esc =cpu_to_fec32 (estatus); } - txq->tx_buf[index].type =FEC_TXBUF_T_XDP_NDO ; - txq->tx_buf[index].xdp = frame; -/* Make sure the updates to rest of the descriptor are performed before
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct
Quoted 3 lines
drivers/net/ethernet/freescale/fec_main.c +0
-0* transferring ownership. */ @@ -3863,6 +3886,31 @@ static int fec_enet_txq_xmit_frame(struct
fec_enet_private*fep,drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3891
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c +3
-0return 0 ; } +static int fec_enet_xdp_tx_xmit (struct net_device *ndev, +struct xdp_buff *xdp) +{E.g add parameter
dma_sync_len.drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3892
Quoted 1 line
drivers/net/ethernet/freescale/fec_main.c +1
-0+struct xdp_frame *xdpf =xdp_convert_buff_to_frame (xdp);
XDP_TXcan avoid this conversion toxdp_frame.
It would requires some refactor offec_enet_txq_xmit_frame().drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3903
Quoted 11 lines
drivers/net/ethernet/freescale/fec_main.c +11
-0+struct fec_enet_private *fep =netdev_priv (ndev); +struct fec_enet_priv_tx_q *txq; +int cpu =smp_processor_id (); +struct netdev_queue *nq; +int queue, ret; + + queue =fec_enet_xdp_get_tx_queue (fep, cpu); + txq = fep->tx_queue[queue]; + nq =netdev_get_tx_queue (fep->netdev, queue); + +__netif_tx_lock (nq, cpu);It is sad that
XDP_TXtakes a lock for each frame.drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3907
Quoted 4 lines
drivers/net/ethernet/freescale/fec_main.c +4
-0+ +/* Avoid tx timeout as XDP shares the queue with kernel stack */ +txq_trans_cond_update (nq); + ret =fec_enet_txq_xmit_frame (fep, txq, xdpf, false);Add/pass parameter
dma_sync_lentofec_enet_txq_xmit_frame().drivers/net/ethernet/freescale/fec_main.c static int fec_enet_xdp_xmit(struct net_device
Quoted 10 lines
drivers/net/ethernet/freescale/fec_main.c +6
-0+ +__netif_tx_unlock (nq); + +return ret; +} + static int fec_enet_xdp_xmit(struct net_device *dev, int num_frames, struct xdp_frame **frames,@@ -3885,7 +3933,7 @@ static int fec_enet_xdp_xmit(struct net_device*dev,
Quoted 8 lines
drivers/net/ethernet/freescale/fec_main.c +1
-1/* Avoid tx timeout as XDP shares the queue with kernel stack */ txq_trans_cond_update (nq);for (i =0 ; i < num_frames; i++) { -if (fec_enet_txq_xmit_frame (fep, txq, frames[i]) <0 ) +if (fec_enet_txq_xmit_frame (fep, txq, frames[i], true) <0 )break ; sent_frames++; }
Wei Fang
maintainerSorry, I missed some comments before.
Quoted 15 lines
Review fec_enet_tx_queue(struct net_device *ndev, u16
Quoted 1 line
(quoted diff) +0
-0@@ -1482,7 +1486,7 @@ fec_enet_tx_queue(struct net_device *ndev, u16
queue_id, int budget)drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private
Quoted 9 lines
drivers/net/ethernet/freescale/fec_main.c +1
-1/* Free the sk buffer associated with this last transmit */ dev_kfree_skb_any (skb); }else { -xdp_return_frame (xdpf); +xdp_return_frame_rx_napi (xdpf); txq->tx_buf[index].xdp =NULL ;/* restore default tx buffer type: FEC_TXBUF_T_SKB */ @@ -1573,11 +1577,18 @@ fec_enet_run_xdp(struct fec_enet_private*fep, struct
bpf_prog*prog,Quoted 3 lines
(quoted diff) +0
-0} break;
Quoted 11 lines
drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, · line 1581
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c +1
-4-default : -bpf_warn_invalid_xdp_action (fep->netdev, prog, act); - fallthrough; -case XDP_TX : + err =fec_enet_xdp_tx_xmit (fep->netdev, xdp);You should pass along the "sync" length value to
fec_enet_xdp_tx_xmit(). Because we know DMA comes from same device (it is already DMA mapped to), then we can do a DMA sync "to_device" with only the sync length.
I think it's okay if the frame length does not change, but if we increase the length of the received frame, such as add a VLAN header into the frame, I think the "sync" length value is not correct.
Quoted 4 lines
drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, · line 1582
Quoted 1 line
drivers/net/ethernet/freescale/fec_main.c +1
-0+if (err) {Add an unlikely(err) or do like above case
XDP_REDIRECT, where it takes the likely case "if (!err)" first.
Yes, you are right, I will improve it.
Quoted 83 lines
drivers/net/ethernet/freescale/fec_main.c fec_enet_xdp_get_tx_queue(struct
Quoted 14 lines
drivers/net/ethernet/freescale/fec_main.c +9
-0+ ret =FEC_ENET_XDP_CONSUMED ; + page =virt_to_head_page (xdp->data); +page_pool_put_page (rxq->page_pool, page, sync, true); + }else { + ret =FEC_ENET_XDP_TX ; + } +break ; + +default :bpf_warn_invalid_xdp_action (fep->netdev, prog, act); fallthrough;@@ -3793,7 +3804,8 @@ fec_enet_xdp_get_tx_queue(struct
fec_enet_private*fep, int index)drivers/net/ethernet/freescale/fec_main.c fec_enet_xdp_get_tx_queue(struct fec_enet_private *fep, int index) · line 3808
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c +2
-1static int fec_enet_txq_xmit_frame (struct fec_enet_private *fep,struct fec_enet_priv_tx_q *txq, -struct xdp_frame *frame) +struct xdp_frame *frame, +bool ndo_xmit)E.g add parameter
dma_sync_len.drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct
Quoted 4 lines
drivers/net/ethernet/freescale/fec_main.c +0
-0{unsigned int index, status, estatus;struct bufdesc *bdp;@@ -3813,10 +3825,24 @@ static int fec_enet_txq_xmit_frame(struct
fec_enet_private*fep,Quoted 11 lines
drivers/net/ethernet/freescale/fec_main.c +4
-4index =fec_enet_get_bd_index (bdp, &txq->bd); - dma_addr =dma_map_single (&fep->pdev->dev, frame->data, - frame->len,DMA_TO_DEVICE ); -if (dma_mapping_error (&fep->pdev->dev, dma_addr)) -return -ENOMEM ; +if (ndo_xmit) { + dma_addr =dma_map_single (&fep->pdev->dev, frame->data, + frame->len,DMA_TO_DEVICE ); +if (dma_mapping_error (&fep->pdev->dev, dma_addr))drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3841
Quoted 10 lines
drivers/net/ethernet/freescale/fec_main.c +10
-0+return -ENOMEM ; + + txq->tx_buf[index].type =FEC_TXBUF_T_XDP_NDO ; + }else { +struct page *page =virt_to_page (frame->data); + + dma_addr =page_pool_get_dma_addr (page) +sizeof (*frame) + + frame->headroom; +dma_sync_single_for_device (&fep->pdev->dev, dma_addr, + frame->len,DMA_BIDIRECTIONAL );Optimization: use
dma_sync_lenhere instead offrame->len.drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct
Quoted 8 lines
drivers/net/ethernet/freescale/fec_main.c +4
-0+ txq->tx_buf[index].type =FEC_TXBUF_T_XDP_TX ; + } + + txq->tx_buf[index].xdp = frame; status |= (BD_ENET_TX_INTR |BD_ENET_TX_LAST );if (fep->bufdesc_ex)@@ -3835,9 +3861,6 @@ static int fec_enet_txq_xmit_frame(struct
fec_enet_private*fep,drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep,
Quoted 7 lines
drivers/net/ethernet/freescale/fec_main.c +0
-3ebdp->cbd_esc =cpu_to_fec32 (estatus); } - txq->tx_buf[index].type =FEC_TXBUF_T_XDP_NDO ; - txq->tx_buf[index].xdp = frame; -/* Make sure the updates to rest of the descriptor are performed before
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct
Quoted 3 lines
drivers/net/ethernet/freescale/fec_main.c +0
-0* transferring ownership. */ @@ -3863,6 +3886,31 @@ static int fec_enet_txq_xmit_frame(struct
fec_enet_private*fep,drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3891
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c +3
-0return 0 ; } +static int fec_enet_xdp_tx_xmit (struct net_device *ndev, +struct xdp_buff *xdp) +{E.g add parameter
dma_sync_len.
Quoted 5 lines
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3892
Quoted 1 line
drivers/net/ethernet/freescale/fec_main.c +1
-0+struct xdp_frame *xdpf =xdp_convert_buff_to_frame (xdp);
XDP_TXcan avoid this conversion toxdp_frame.
It would requires some refactor offec_enet_txq_xmit_frame().
Yes, but I'm not intend to change it, using the existing interface is enough.
Quoted 5 lines
Quoted 5 lines
drivers/net/ethernet/freescale/fec_main.c +5
-0+struct fec_enet_private *fep =netdev_priv (ndev); +struct fec_enet_priv_tx_q *txq; +int cpu =smp_processor_id (); +struct netdev_queue *nq; +int queue, ret;
Quoted 9 lines
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3903
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c +6
-0+ + queue =fec_enet_xdp_get_tx_queue (fep, cpu); + txq = fep->tx_queue[queue]; + nq =netdev_get_tx_queue (fep->netdev, queue); + +__netif_tx_lock (nq, cpu);It is sad that
XDP_TXtakes a lock for each frame.
Yes, but the XDP path share the queue with the kernel network stack, so we need a lock here, unless there is a dedicated queue for XDP path. Do you have a better solution?
Quoted 27 lines
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3907
Quoted 4 lines
drivers/net/ethernet/freescale/fec_main.c +4
-0+ +/* Avoid tx timeout as XDP shares the queue with kernel stack */ +txq_trans_cond_update (nq); + ret =fec_enet_txq_xmit_frame (fep, txq, xdpf, false);Add/pass parameter
dma_sync_lentofec_enet_txq_xmit_frame().drivers/net/ethernet/freescale/fec_main.c static int fec_enet_xdp_xmit(struct net_device
Quoted 10 lines
drivers/net/ethernet/freescale/fec_main.c +6
-0+ +__netif_tx_unlock (nq); + +return ret; +} + static int fec_enet_xdp_xmit(struct net_device *dev, int num_frames, struct xdp_frame **frames,@@ -3885,7 +3933,7 @@ static int fec_enet_xdp_xmit(struct net_device*dev,
Quoted 8 lines
drivers/net/ethernet/freescale/fec_main.c +1
-1/* Avoid tx timeout as XDP shares the queue with kernel stack */ txq_trans_cond_update (nq);for (i =0 ; i < num_frames; i++) { -if (fec_enet_txq_xmit_frame (fep, txq, frames[i]) <0 ) +if (fec_enet_txq_xmit_frame (fep, txq, frames[i], true) <0 )break ; sent_frames++; }
Jesper Dangaard Brouer
maintainerQuoted 6 lines · Wei Fang
Quoted 4 lines
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3892
Quoted 1 line
drivers/net/ethernet/freescale/fec_main.c +1
-0+struct xdp_frame *xdpf =xdp_convert_buff_to_frame (xdp);
XDP_TXcan avoid this conversion toxdp_frame.
It would requires some refactor offec_enet_txq_xmit_frame().Yes, but I'm not intend to change it, using the existing interface is enough.
Quoted 8 lines
Quoted 8 lines
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3900
Quoted 8 lines
drivers/net/ethernet/freescale/fec_main.c +8
-0+struct fec_enet_private *fep =netdev_priv (ndev); +struct fec_enet_priv_tx_q *txq; +int cpu =smp_processor_id (); +struct netdev_queue *nq; +int queue, ret; + + queue =fec_enet_xdp_get_tx_queue (fep, cpu); + txq = fep->tx_queue[queue];
Notice how TXQ gets selected based on CPU.
Thus it will be the same for all the frames.
Quoted 10 lines
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3903
Quoted 3 lines
drivers/net/ethernet/freescale/fec_main.c +3
-0+ nq =netdev_get_tx_queue (fep->netdev, queue); + +__netif_tx_lock (nq, cpu);It is sad that
XDP_TXtakes a lock for each frame.Yes, but the XDP path share the queue with the kernel network stack, so we need a lock here, unless there is a dedicated queue for XDP path. Do you have a better solution?
Yes, the solution would be to keep a stack local (or per-CPU) queue for all the XDP_TX frames, and send them at the xdp_do_flush_map() call site. This is basically what happens with xdp_do_redirect() in cpumap.c and devmap.c code, that have a per-CPU bulk queue and sends a bulk of packets into fec_enet_xdp_xmit / ndo_xdp_xmit.
I understand if you don't want to add the complexity to the driver. And I guess, it should be a followup patch to make sure this actually improves performance.
--Jesper
Wei Fang
maintainerQuoted 9 lines · Wei Fang
Quoted 6 lines
Quoted 4 lines
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3892
Quoted 1 line
drivers/net/ethernet/freescale/fec_main.c +1
-0+struct xdp_frame *xdpf =xdp_convert_buff_to_frame (xdp);
XDP_TXcan avoid this conversion toxdp_frame.
It would requires some refactor offec_enet_txq_xmit_frame().Yes, but I'm not intend to change it, using the existing interface is enough.
Quoted 12 lines
Quoted 8 lines
Quoted 8 lines
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3900
Quoted 8 lines
drivers/net/ethernet/freescale/fec_main.c +8
-0+struct fec_enet_private *fep =netdev_priv (ndev); +struct fec_enet_priv_tx_q *txq; +int cpu =smp_processor_id (); +struct netdev_queue *nq; +int queue, ret; + + queue =fec_enet_xdp_get_tx_queue (fep, cpu); + txq = fep->tx_queue[queue];Notice how TXQ gets selected based on CPU.
Thus it will be the same for all the frames.
Yes, I'll optimize it, thanks!
Quoted 17 lines
Quoted 10 lines
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, · line 3903
Quoted 3 lines
drivers/net/ethernet/freescale/fec_main.c +3
-0+ nq =netdev_get_tx_queue (fep->netdev, queue); + +__netif_tx_lock (nq, cpu);It is sad that
XDP_TXtakes a lock for each frame.Yes, but the XDP path share the queue with the kernel network stack, so we need a lock here, unless there is a dedicated queue for XDP path. Do you have a better solution?
Yes, the solution would be to keep a stack local (or per-CPU) queue for all the
XDP_TXframes, and send them at thexdp_do_flush_map()call site. This is basically what happens withxdp_do_redirect()incpumap.canddevmap.ccode, that have a per-CPU bulk queue and sends a bulk of packets intofec_enet_xdp_xmit/ndo_xdp_xmit.
Quoted 4 lines
I understand if you don't want to add the complexity to the driver. And I guess, it should be a followup patch to make sure this actually improves performance.
Thanks, I got it. I'll optimize in a followup patch if it really improves the performance.
Jakub Kicinski
maintainerdrivers/net/ethernet/freescale/fec_main.c fec_enet_tx_queue(struct net_device *ndev, u16 queue_id, int budget) · line 1489
Quoted 3 lines · On Mon, 31 Jul 2023 14:00:25 +0800 Wei Fang wrote:
drivers/net/ethernet/freescale/fec_main.c +1
-1}else { -xdp_return_frame (xdpf); +xdp_return_frame_rx_napi (xdpf);
If you implement Jesper's syncing suggestions, I think you can use
page_pool_put_page(pool, page, 0, true);
for XDP_TX here to avoid the DMA sync on page recycle.
Wei Fang
maintainerQuoted 9 lines
Review
Quoted 3 lines
(quoted diff) +1
-1} else { - xdp_return_frame(xdpf); + xdp_return_frame_rx_napi(xdpf);If you implement Jesper's syncing suggestions, I think you can use
page_pool_put_page(pool, page, 0, true);for
XDP_TXhere to avoid the DMA sync on page recycle.
I tried Jasper's syncing suggestion and used page_pool_put_page() to recycle pages, but the results does not seem to improve the performance of XDP_TX, it even degrades the speed.
The result of the current modification.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 260180 pkt/s proto 17: 260373 pkt/s proto 17: 260363 pkt/s proto 17: 259036 pkt/s proto 17: 260180 pkt/s proto 17: 260048 pkt/s proto 17: 260029 pkt/s proto 17: 260133 pkt/s proto 17: 260021 pkt/s proto 17: 260203 pkt/s proto 17: 260293 pkt/s proto 17: 259418 pkt/s
After using the sync suggestion, the result shows as follow.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 255956 pkt/s proto 17: 255841 pkt/s proto 17: 255835 pkt/s proto 17: 255381 pkt/s proto 17: 255736 pkt/s proto 17: 255779 pkt/s proto 17: 254135 pkt/s proto 17: 255584 pkt/s proto 17: 255855 pkt/s proto 17: 255664 pkt/s
Below are my changes, I don't know what cause it. Based on the results, it's better to keep the current modification.
drivers/net/ethernet/freescale/fec_main.c +18 -8
@@ -77,7 +77,8 @@static void set_multicast_list (struct net_device *ndev);static void fec_enet_itr_coal_set (struct net_device *ndev);static int fec_enet_xdp_tx_xmit (struct net_device *ndev, -struct xdp_buff *xdp); +struct xdp_buff *xdp, +u32 dma_sync_len);#define DRIVER_NAME "fec"
@@ -1487,7 +1488,14 @@ fec_enet_tx_queue(struct net_device *ndev, u16 queue_id, int budget)/* Free the sk buffer associated with this last transmit */ dev_kfree_skb_any (skb); }else { -xdp_return_frame_rx_napi (xdpf); +if (txq->tx_buf[index].type ==FEC_TXBUF_T_XDP_NDO ) +xdp_return_frame_rx_napi (xdpf); +else { +struct page *page; + + page =virt_to_head_page (xdpf->data); +page_pool_put_page (page->pp, page,0 , true); + } txq->tx_buf[index].xdp =NULL ;/* restore default tx buffer type: FEC_TXBUF_T_SKB */
@@ -1557,7 +1565,8 @@ fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, act =bpf_prog_run_xdp (prog, xdp);/* Due xdp_adjust_tail: DMA sync for_device cover max len CPU touch */ - sync = xdp->data_end - xdp->data_hard_start -FEC_ENET_XDP_HEADROOM ; + sync = xdp->data_end - xdp->data; sync =max (sync, len);switch (act) {
@@ -1579,7 +1588,7 @@ fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog,break ;case XDP_TX : - err =fec_enet_xdp_tx_xmit (fep->netdev, xdp); + err =fec_enet_xdp_tx_xmit (fep->netdev, xdp, sync);if (unlikely (err)) { ret =FEC_ENET_XDP_CONSUMED ; page =virt_to_head_page (xdp->data);
@@ -3807,6 +3816,7 @@ fec_enet_xdp_get_tx_queue(struct fec_enet_private *fep, int index)static int fec_enet_txq_xmit_frame (struct fec_enet_private *fep,struct fec_enet_priv_tx_q *txq,struct xdp_frame *frame, +u32 dma_sync_len,bool ndo_xmit) {unsigned int index, status, estatus;
@@ -3840,7 +3850,7 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, dma_addr =page_pool_get_dma_addr (page) +sizeof (*frame) + frame->headroom;dma_sync_single_for_device (&fep->pdev->dev, dma_addr, - frame->len,DMA_BIDIRECTIONAL ); + dma_sync_len,DMA_BIDIRECTIONAL ); txq->tx_buf[index].type =FEC_TXBUF_T_XDP_TX ; }
@@ -3889,7 +3899,8 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, }static int fec_enet_xdp_tx_xmit (struct net_device *ndev, -struct xdp_buff *xdp) +struct xdp_buff *xdp, +u32 dma_sync_len) {struct xdp_frame *xdpf =xdp_convert_buff_to_frame (xdp);struct fec_enet_private *fep =netdev_priv (ndev);
@@ -3909,7 +3920,7 @@ static int fec_enet_xdp_tx_xmit(struct net_device *ndev,/* Avoid tx timeout as XDP shares the queue with kernel stack */ txq_trans_cond_update (nq); - ret =fec_enet_txq_xmit_frame (fep, txq, xdpf, false); + ret =fec_enet_txq_xmit_frame (fep, txq, xdpf, dma_sync_len, false);__netif_tx_unlock (nq);
@@ -3938,7 +3949,7 @@ static int fec_enet_xdp_xmit(struct net_device *dev,/* Avoid tx timeout as XDP shares the queue with kernel stack */ txq_trans_cond_update (nq);for (i =0 ; i < num_frames; i++) { -if (fec_enet_txq_xmit_frame (fep, txq, frames[i], true) <0 ) +if (fec_enet_txq_xmit_frame (fep, txq, frames[i],0 , true) <0 )break ; sent_frames++; }
Quoted 7 lines · Wei Fang
Quoted 7 lines
Review
Quoted 3 lines
} else {(quoted diff) +1
-1- xdp_return_frame(xdpf); + xdp_return_frame_rx_napi(xdpf);If you implement Jesper's syncing suggestions, I think you can use
page_pool_put_page(pool, page, 0, true);
To Jakub, using 0 here you are trying to bypass the DMA-sync (which is valid as driver knows XDP_TX have already done the sync). The code will still call into DMA-sync calls with zero as size, so wonder if we should detect size zero and skip that call?
(I mean is this something page_pool should support.)
net/core/page_pool.c +2 -1
@@ -323,7 +323,8 @@ static void page_pool_dma_sync_for_device(struct page_pool *pool,dma_addr_t dma_addr =page_pool_get_dma_addr (page); dma_sync_size =min (dma_sync_size, pool->p.max_len); -dma_sync_single_range_for_device (pool->p.dev, dma_addr, +if (dma_sync_size) +dma_sync_single_range_for_device (pool->p.dev, dma_addr, pool->p.offset, dma_sync_size, pool->p.dma_dir);
Quoted 5 lines
Quoted 2 lines
for
XDP_TXhere to avoid the DMA sync on page recycle.I tried Jasper's syncing suggestion and used
page_pool_put_page()to recycle pages, but the results does not seem to improve the performance ofXDP_TX,
The optimization will only have effect on those devices which have dev->dma_coherent=false else DMA function [1] (e.g. dma_direct_sync_single_for_device) will skip the sync calls.
[1]
https://elixir.bootlin.com/linux/v6.5-rc4/source/kernel/dma/direct.h#L63
(Cc. Andrew Lunn)
Does any of the imx generations have dma-noncoherent memory?
And does any of these use the fec NIC driver?
Quoted 1 line
it even degrades the speed.
Could be low runs simply be a variation between your test runs?
The specific device (imx8mpevk) this was tested on, clearly have dma_coherent=true, or else we would have seen a difference. But the code change should not have any overhead for the dma_coherent=true case, the only extra overhead is the extra empty DMA sync call with size zero (as discussed in top).
Quoted 4 lines
The result of the current modification.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 260180 pkt/s
These results are *significantly* better than reported in patch-1. What happened?!?
e.g.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 135817 pkt/s proto 17: 142776 pkt/s
Quoted 122 lines
proto 17: 260373 pkt/s proto 17: 260363 pkt/s proto 17: 259036 pkt/s proto 17: 260180 pkt/s proto 17: 260048 pkt/s proto 17: 260029 pkt/s proto 17: 260133 pkt/s proto 17: 260021 pkt/s proto 17: 260203 pkt/s proto 17: 260293 pkt/s proto 17: 259418 pkt/sAfter using the sync suggestion, the result shows as follow.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 255956 pkt/s proto 17: 255841 pkt/s proto 17: 255835 pkt/s proto 17: 255381 pkt/s proto 17: 255736 pkt/s proto 17: 255779 pkt/s proto 17: 254135 pkt/s proto 17: 255584 pkt/s proto 17: 255855 pkt/s proto 17: 255664 pkt/sBelow are my changes, I don't know what cause it. Based on the results, it's better to keep the current modification.
drivers/net/ethernet/freescale/fec_main.c +18
-8@@ -77,7 +77,8 @@static void set_multicast_list (struct net_device *ndev);static void fec_enet_itr_coal_set (struct net_device *ndev);static int fec_enet_xdp_tx_xmit (struct net_device *ndev, -struct xdp_buff *xdp); +struct xdp_buff *xdp, +u32 dma_sync_len);#define DRIVER_NAME "fec"@@ -1487,7 +1488,14 @@ fec_enet_tx_queue(struct net_device *ndev, u16 queue_id, int budget)/* Free the sk buffer associated with this last transmit */ dev_kfree_skb_any (skb); }else { -xdp_return_frame_rx_napi (xdpf); +if (txq->tx_buf[index].type ==FEC_TXBUF_T_XDP_NDO ) +xdp_return_frame_rx_napi (xdpf); +else { +struct page *page; + + page =virt_to_head_page (xdpf->data); +page_pool_put_page (page->pp, page,0 , true); + } txq->tx_buf[index].xdp =NULL ;/* restore default tx buffer type: FEC_TXBUF_T_SKB */ @@ -1557,7 +1565,8 @@ fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog, act =bpf_prog_run_xdp (prog, xdp);/* Due xdp_adjust_tail: DMA sync for_device cover max len CPU touch */ - sync = xdp->data_end - xdp->data_hard_start -FEC_ENET_XDP_HEADROOM ; + sync = xdp->data_end - xdp->data; sync =max (sync, len);switch (act) {@@ -1579,7 +1588,7 @@ fec_enet_run_xdp(struct fec_enet_private *fep, struct bpf_prog *prog,break ;case XDP_TX : - err =fec_enet_xdp_tx_xmit (fep->netdev, xdp); + err =fec_enet_xdp_tx_xmit (fep->netdev, xdp, sync);if (unlikely (err)) { ret =FEC_ENET_XDP_CONSUMED ; page =virt_to_head_page (xdp->data);@@ -3807,6 +3816,7 @@ fec_enet_xdp_get_tx_queue(struct fec_enet_private *fep, int index)static int fec_enet_txq_xmit_frame (struct fec_enet_private *fep,struct fec_enet_priv_tx_q *txq,struct xdp_frame *frame, +u32 dma_sync_len,bool ndo_xmit) {unsigned int index, status, estatus;@@ -3840,7 +3850,7 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, dma_addr =page_pool_get_dma_addr (page) +sizeof (*frame) + frame->headroom;dma_sync_single_for_device (&fep->pdev->dev, dma_addr, - frame->len,DMA_BIDIRECTIONAL ); + dma_sync_len,DMA_BIDIRECTIONAL ); txq->tx_buf[index].type =FEC_TXBUF_T_XDP_TX ; }@@ -3889,7 +3899,8 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, }static int fec_enet_xdp_tx_xmit (struct net_device *ndev, -struct xdp_buff *xdp) +struct xdp_buff *xdp, +u32 dma_sync_len) {struct xdp_frame *xdpf =xdp_convert_buff_to_frame (xdp);struct fec_enet_private *fep =netdev_priv (ndev);@@ -3909,7 +3920,7 @@ static int fec_enet_xdp_tx_xmit(struct net_device *ndev,/* Avoid tx timeout as XDP shares the queue with kernel stack */ txq_trans_cond_update (nq); - ret =fec_enet_txq_xmit_frame (fep, txq, xdpf, false); + ret =fec_enet_txq_xmit_frame (fep, txq, xdpf, dma_sync_len, false);__netif_tx_unlock (nq);@@ -3938,7 +3949,7 @@ static int fec_enet_xdp_xmit(struct net_device *dev,/* Avoid tx timeout as XDP shares the queue with kernel stack */ txq_trans_cond_update (nq);for (i =0 ; i < num_frames; i++) { -if (fec_enet_txq_xmit_frame (fep, txq, frames[i], true) <0 ) +if (fec_enet_txq_xmit_frame (fep, txq, frames[i],0 , true) <0 )break ; sent_frames++; }
Wei Fang
maintainerQuoted 56 lines · Wei Fang
Quoted 7 lines
Quoted 7 lines
Review
Quoted 3 lines
} else {(quoted diff) +1
-1- xdp_return_frame(xdpf); + xdp_return_frame_rx_napi(xdpf);If you implement Jesper's syncing suggestions, I think you can use
page_pool_put_page(pool, page, 0, true);To Jakub, using 0 here you are trying to bypass the DMA-sync (which is valid as driver knows
XDP_TXhave already done the sync).
The code will still call into DMA-sync calls with zero as size, so wonder if we should detect size zero and skip that call?
(I mean is this somethingpage_poolshould support.)net/core/page_pool.c index renamed from net/core/page_pool.c +0
-0
7ca456bfab71..778d061e4f2c100644net/core/page_pool.c +2
-1@@ -323,7 +323,8 @@ static void page_pool_dma_sync_for_device(structpage_pool *pool, dma_addr_t dma_addr = page_pool_get_dma_addr(page); dma_sync_size =min (dma_sync_size, pool->p.max_len); - dma_sync_single_range_for_device(pool->p.dev, dma_addr, +if (dma_sync_size) + dma_sync_single_range_for_device(pool->p.dev, dma_addr, pool->p.offset,
dma_sync_size,pool->p.dma_dir);Quoted 6 lines
Quoted 2 lines
for
XDP_TXhere to avoid the DMA sync on page recycle.I tried Jasper's syncing suggestion and used
page_pool_put_page()to recycle pages, but the results does not seem to improve the performance ofXDP_TX,The optimization will only have effect on those devices which have dev->dma_coherent=false else DMA function [1] (e.g.
dma_direct_sync_single_for_device) will skip the sync calls.[1]
https://elixir.b/
ootlin.com%2Flinux%2Fv6.5-rc4%2Fsource%2Fkernel%2Fdma%2Fdirect.h%2 3L63&data=05%7C01%7Cwei.fang%40nxp.com%7Cb81436deb63d41dd41a1 08db93f454cb%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63 8266449821982804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwM DAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C% 7C&sdata=CSi%2Fzld%2FucIYo6VDb9YdO91D8HKV1tbzPq8fB5X%2Bs3s%3D& reserved=0
Quoted 9 lines
(Cc. Andrew Lunn)
Does any of the imx generations have dma-noncoherent memory?And does any of these use the fec NIC driver?
Quoted 1 line
it even degrades the speed.
Could be low runs simply be a variation between your test runs?
Maybe, I just tested once before. So I test several times again, the results of the two methods do not seem to be much different so far, both about 255000 pkt/s.
Quoted 6 lines
The specific device (imx8mpevk) this was tested on, clearly have dma_coherent=true, or else we would have seen a difference. But the code change should not have any overhead for the dma_coherent=true case, the only extra overhead is the extra empty DMA sync call with size zero (as discussed in top).
The FEC of i.MX8MP-EVK has dma_coherent=false, and as I mentioned above, I did not see an obvious difference in the performance. :(
Quoted 8 lines
Quoted 4 lines
The result of the current modification.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 260180 pkt/sThese results are *significantly* better than reported in patch-1. What happened?!?
The test environment is slightly different, in patch-1, the FEC port was directly connected to the port of another board. But in the latest test, the ports of the two boards were connected to a switch, so the ports of the two boards are not directly connected.
Quoted 148 lines
e.g.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 135817 pkt/s proto 17: 142776 pkt/sdrivers/net/ethernet/freescale/fec_main.c fec_enet_tx_queue(struct net_device *ndev,
Quoted 44 lines
proto 17: 260373 pkt/s proto 17: 260363 pkt/s proto 17: 259036 pkt/s proto 17: 260180 pkt/s proto 17: 260048 pkt/s proto 17: 260029 pkt/s proto 17: 260133 pkt/s proto 17: 260021 pkt/s proto 17: 260203 pkt/s proto 17: 260293 pkt/s proto 17: 259418 pkt/sAfter using the sync suggestion, the result shows as follow.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 255956 pkt/s proto 17: 255841 pkt/s proto 17: 255835 pkt/s proto 17: 255381 pkt/s proto 17: 255736 pkt/s proto 17: 255779 pkt/s proto 17: 254135 pkt/s proto 17: 255584 pkt/s proto 17: 255855 pkt/s proto 17: 255664 pkt/sBelow are my changes, I don't know what cause it. Based on the results, it's better to keep the current modification.
drivers/net/ethernet/freescale/fec_main.c +2
-1@@ -77,7 +77,8 @@static void set_multicast_list (struct net_device *ndev);static void fec_enet_itr_coal_set (struct net_device *ndev);static int fec_enet_xdp_tx_xmit (struct net_device *ndev, -struct xdp_buff *xdp); +struct xdp_buff *xdp, +u32 dma_sync_len);#define DRIVER_NAME "fec"@@ -1487,7 +1488,14 @@ fec_enet_tx_queue(struct net_device *ndev,u16
queue_id, int budget)drivers/net/ethernet/freescale/fec_main.c fec_enet_tx_queue(struct net_device *ndev, · line 1488
Quoted 1 line
drivers/net/ethernet/freescale/fec_main.c +0
-0/* Free the sk buffer associated with this last transmit */
drivers/net/ethernet/freescale/fec_main.c fec_enet_tx_queue(struct net_device *ndev, · line 1491
Quoted 4 lines
drivers/net/ethernet/freescale/fec_main.c +1
-1dev_kfree_skb_any (skb); }else { -xdp_return_frame_rx_napi (xdpf); + if (txq->tx_buf[index].type ==
FEC_TXBUF_T_XDP_NDO)drivers/net/ethernet/freescale/fec_main.c fec_enet_tx_queue(struct net_device *ndev, · line 1496
Quoted 5 lines
drivers/net/ethernet/freescale/fec_main.c +5
-0+xdp_return_frame_rx_napi (xdpf); + else { +struct page *page; + + page =
virt_to_head_page(xdpf->data);drivers/net/ethernet/freescale/fec_main.c fec_enet_tx_queue(struct net_device *ndev, · line 1497
Quoted 1 line
drivers/net/ethernet/freescale/fec_main.c +1
-0+ page_pool_put_page(page->pp, page,0, true);
Quoted 5 lines
drivers/net/ethernet/freescale/fec_main.c +1
-0+ } txq->tx_buf[index].xdp =NULL ;/* restore default tx buffer type:
FEC_TXBUF_T_SKB*/ @@ -1557,7 +1565,8 @@ fec_enet_run_xdp(struct
fec_enet_private*fep, structbpf_prog*prog,Quoted 3 lines
act = bpf_prog_run_xdp(prog, xdp); /* Due xdp_adjust_tail: DMA sync for_device cover max len CPUtouch */
Quoted 1 line
- sync =
xdp->data_end-xdp->data_hard_start-
FEC_ENET_XDP_HEADROOM;drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private *fep,
Quoted 5 lines
drivers/net/ethernet/freescale/fec_main.c +1
-0+ sync = xdp->data_end - xdp->data; sync =max (sync, len);switch (act) {@@ -1579,7 +1588,7 @@ fec_enet_run_xdp(struct fec_enet_private *fep,struct
bpf_prog*prog,drivers/net/ethernet/freescale/fec_main.c fec_enet_run_xdp(struct fec_enet_private *fep,
Quoted 9 lines
drivers/net/ethernet/freescale/fec_main.c +1
-2break ;case XDP_TX : - err =fec_enet_xdp_tx_xmit (fep->netdev, xdp); + err =fec_enet_xdp_tx_xmit (fep->netdev, xdp, sync);if (unlikely (err)) { ret =FEC_ENET_XDP_CONSUMED ; page = virt_to_head_page(xdp->data); @@ -3807,6 +3816,7 @@ fec_enet_xdp_get_tx_queue(struct fec_enet_private*fep, int index)
Quoted 9 lines
static int fec_enet_txq_xmit_frame(struct
fec_enet_private*fep,
structfec_enet_priv_tx_q*txq, structxdp_frame*frame,
+ u32dma_sync_len,
boolndo_xmit){ unsigned int index, status, estatus; @@ -3840,7 +3850,7 @@static int fec_enet_txq_xmit_frame(struct
fec_enet_private*fep,
dma_addr=page_pool_get_dma_addr(page)+
sizeof(*frame)+Quoted 2 lines
frame->headroom; dma_sync_single_for_device(&fep->pdev->dev,
dma_addr,Quoted 1 line
-
frame->len,
DMA_BIDIRECTIONAL);drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct
Quoted 6 lines
drivers/net/ethernet/freescale/fec_main.c +2
-0+ dma_sync_len, + DMA_BIDIRECTIONAL); txq->tx_buf[index].type =FEC_TXBUF_T_XDP_TX ; }@@ -3889,7 +3899,8 @@ static int fec_enet_txq_xmit_frame(struct
fec_enet_private*fep,drivers/net/ethernet/freescale/fec_main.c static int fec_enet_txq_xmit_frame(struct
Quoted 12 lines
drivers/net/ethernet/freescale/fec_main.c +3
-1}static int fec_enet_xdp_tx_xmit (struct net_device *ndev, -struct xdp_buff *xdp) +struct xdp_buff *xdp, +u32 dma_sync_len) {struct xdp_frame *xdpf =xdp_convert_buff_to_frame (xdp); struct fec_enet_private *fep = netdev_priv(ndev); @@ -3909,7 +3920,7 @@ static int fec_enet_xdp_tx_xmit(struct net_device *ndev,/* Avoid tx timeout as XDP shares the queue with kernel stack */
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_xdp_xmit(struct net_device
Quoted 8 lines
txq_trans_cond_update(nq);drivers/net/ethernet/freescale/fec_main.c +2
-1- ret =fec_enet_txq_xmit_frame (fep, txq, xdpf, false); + ret =fec_enet_txq_xmit_frame (fep, txq, xdpf, dma_sync_len, + false);__netif_tx_unlock (nq);@@ -3938,7 +3949,7 @@ static int fec_enet_xdp_xmit(struct net_device*dev,
drivers/net/ethernet/freescale/fec_main.c static int fec_enet_xdp_xmit(struct net_device · line 3949
Quoted 1 line
drivers/net/ethernet/freescale/fec_main.c +0
-0/* Avoid tx timeout as XDP shares the queue with kernel stack */
Quoted 9 lines
drivers/net/ethernet/freescale/fec_main.c +2
-1txq_trans_cond_update (nq);for (i =0 ; i < num_frames; i++) { -if (fec_enet_txq_xmit_frame (fep, txq, frames[i], true) <0 ) +if (fec_enet_txq_xmit_frame (fep, txq, frames[i],0 , + true) <0 )break ; sent_frames++; }
Jesper Dangaard Brouer
maintainerQuoted 13 lines · Wei Fang
Quoted 13 lines · Wei Fang
Quoted 6 lines
Quoted 6 lines
Review
Quoted 3 lines
} else {(quoted diff) +1
-1- xdp_return_frame(xdpf); + xdp_return_frame_rx_napi(xdpf);If you implement Jesper's syncing suggestions, I think you can use
page_pool_put_page(pool, page, 0, true);To Jakub, using 0 here you are trying to bypass the DMA-sync (which is valid as driver knows
XDP_TXhave already done the sync).
The code will still call into DMA-sync calls with zero as size, so wonder if we should detect size zero and skip that call?
(I mean is this somethingpage_poolshould support.)
[...]
Quoted 25 lines
Quoted 12 lines
Quoted 6 lines
Quoted 3 lines
for
XDP_TXhere to avoid the DMA sync on page recycle.I tried Jasper's syncing suggestion and used
page_pool_put_page()to recycle pages, but the results does not seem to improve the performance ofXDP_TX,The optimization will only have effect on those devices which have dev->dma_coherent=false else DMA function [1] (e.g.
dma_direct_sync_single_for_device) will skip the sync calls.[1] https://elixir.bootlin.com/linux/v6.5-rc4/source/kernel/dma/direct.h#L63
Quoted 9 lines
(Cc. Andrew Lunn)
Does any of the imx generations have dma-noncoherent memory?And does any of these use the fec NIC driver?
Quoted 1 line
it even degrades the speed.
Could be low runs simply be a variation between your test runs?
Maybe, I just tested once before. So I test several times again, the results of the two methods do not seem to be much different so far, both about 255000 pkt/s.
Quoted 8 lines
Quoted 6 lines
The specific device (imx8mpevk) this was tested on, clearly have dma_coherent=true, or else we would have seen a difference. But the code change should not have any overhead for the dma_coherent=true case, the only extra overhead is the extra empty DMA sync call with size zero (as discussed in top).
The FEC of i.MX8MP-EVK has dma_coherent=false, and as I mentioned above, I did not see an obvious difference in the performance. :(
That is surprising - given the results.
(see below, lack of perf/diff might be caused by Ethernet flow-control).
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
The result of the current modification.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 260180 pkt/s
Quoted 8 lines
Quoted 3 lines
These results are*significantly* better than reported in patch-1. What happened?!?
The test environment is slightly different, in patch-1, the FEC port was directly connected to the port of another board. But in the latest test, the ports of the two boards were connected to a switch, so the ports of the two boards are not directly connected.
Hmm, I've seen this kind of perf behavior of direct-connected or via switch before. The mistake I made was, that I had not disabled Ethernet flow-control. The xdp2 XDP_TX program will swap the mac addresses, and send the packet back to the packet generator (running pktgen), which will get overloaded itself and starts sending Ethernet flow-control pause frames.
Command line to disable:
# ethtool -A eth0 rx off tx off
Can I ask/get you to make sure that Ethernet flow-control is disabled (on both generator and DUT (to be on safe-side)) and run the test again?
--Jesper
Quoted 8 lines
Quoted 8 lines
e.g.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 135817 pkt/s proto 17: 142776 pkt/sQuoted 3 lines
proto 17: 260373 pkt/s proto 17: 260363 pkt/s proto 17: 259036 pkt/s
[...]
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
After using the sync suggestion, the result shows as follow.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 255956 pkt/s proto 17: 255841 pkt/s proto 17: 255835 pkt/s
Wei Fang
maintainerQuoted 27 lines
Quoted 2 lines
The FEC of i.MX8MP-EVK has dma_coherent=false, and as I mentioned above, I did not see an obvious difference in the performance. :(
That is surprising - given the results.
(see below, lack of perf/diff might be caused by Ethernet flow-control).
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
The result of the current modification.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 260180 pkt/sQuoted 8 lines
Quoted 3 lines
These results are*significantly* better than reported in patch-1. What happened?!?
The test environment is slightly different, in patch-1, the FEC port was directly connected to the port of another board. But in the latest test, the ports of the two boards were connected to a switch, so the ports of the two boards are not directly connected.
Hmm, I've seen this kind of perf behavior of direct-connected or via switch before. The mistake I made was, that I had not disabled Ethernet flow-control. The xdp2
XDP_TXprogram will swap the mac addresses, and send the packet back to the packet generator (running pktgen), which will get overloaded itself and starts sending Ethernet flow-control pause frames.
Quoted 6 lines
Command line to disable:
# ethtool -A eth0 rx off tx offCan I ask/get you to make sure that Ethernet flow-control is disabled (on both generator and DUT (to be on safe-side)) and run the test again?
The flow-control was not disabled before, so according to your suggestion, I disable the flow-control on the both boards and run the test again, the performance is slightly improved, but still can not see a clear difference between the two methods. Below are the results.
Result: use "sync_dma_len" method
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 258886 pkt/s proto 17: 258879 pkt/s proto 17: 258872 pkt/s proto 17: 258312 pkt/s proto 17: 258926 pkt/s proto 17: 259057 pkt/s proto 17: 258437 pkt/s proto 17: 259242 pkt/s proto 17: 258665 pkt/s proto 17: 258779 pkt/s proto 17: 259209 pkt/s
Result: use the current method
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 258752 pkt/s proto 17: 258672 pkt/s proto 17: 258317 pkt/s proto 17: 258787 pkt/s proto 17: 258757 pkt/s proto 17: 258542 pkt/s proto 17: 258829 pkt/s proto 17: 258480 pkt/s proto 17: 258859 pkt/s proto 17: 258918 pkt/s proto 17: 258782 pkt/s proto 17: 259086 pkt/s proto 17: 258337 pkt/s
Jesper Dangaard Brouer
maintainerQuoted 30 lines · Wei Fang
Quoted 30 lines
Quoted 2 lines
The FEC of i.MX8MP-EVK has dma_coherent=false, and as I mentioned above, I did not see an obvious difference in the performance. :(
That is surprising - given the results.
(see below, lack of perf/diff might be caused by Ethernet flow-control).
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
The result of the current modification.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 260180 pkt/sQuoted 8 lines
Quoted 3 lines
These results are*significantly* better than reported in patch-1. What happened?!?
The test environment is slightly different, in patch-1, the FEC port was directly connected to the port of another board. But in the latest test, the ports of the two boards were connected to a switch, so the ports of the two boards are not directly connected.
Hmm, I've seen this kind of perf behavior of direct-connected or via switch before. The mistake I made was, that I had not disabled Ethernet flow-control. The xdp2
XDP_TXprogram will swap the mac addresses, and send the packet back to the packet generator (running pktgen), which will get overloaded itself and starts sending Ethernet flow-control pause frames.Command line to disable:
# ethtool -A eth0 rx off tx off
Quoted 7 lines
Quoted 3 lines
Can I ask/get you to make sure that Ethernet flow-control is disabled (on both generator and DUT (to be on safe-side)) and run the test again?
The flow-control was not disabled before, so according to your suggestion, I disable the flow-control on the both boards and run the test again, the performance is slightly improved, but still can not see a clear difference between the two methods. Below are the results.
Something else must be stalling the CPU.
When looking at fec_main.c code, I noticed that fec_enet_txq_xmit_frame() will do a MMIO write for every xdp_frame (to trigger transmit start), which I believe will stall the CPU. The ndo_xdp_xmit/fec_enet_xdp_xmit does bulking, and should be the function that does the MMIO write to trigger transmit start.
$ git diff
drivers/net/ethernet/freescale/fec_main.c +3 -3
@@ -3849,9 +3849,6 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, txq->bd.cur = bdp; -/* Trigger transmission start */ -writel (0 , txq->bd.reg_desc_active); -return 0 ; }
@@ -3880,6 +3877,9 @@ static int fec_enet_xdp_xmit(struct net_device *dev, sent_frames++; } +/* Trigger transmission start */ +writel (0 , txq->bd.reg_desc_active); +__netif_tx_unlock (nq);return sent_frames;
Quoted 2 lines
Result: use "
sync_dma_len" methodroot@imx8mpevk:~# ./xdp2 eth0
The xdp2 (and xdp1) program(s) have a performance issue (due to using
Can I ask you to test using xdp_rxq_info, like:
sudo ./xdp_rxq_info --dev mlx5p1 --action XDP_TX
Quoted 2 lines
proto 17: 258886 pkt/s proto 17: 258879 pkt/s
If you provide numbers for xdp_redirect, then we could better evaluate if changing the lock per xdp_frame, for XDP_TX also, is worth it.
And also find out of moving the MMIO write have any effect.
I also noticed driver does a MMIO write (on rxq) for every RX-packet in fec_enet_rx_queue() napi-poll loop. This also looks like a potential performance stall.
--Jesper
Wei Fang
maintainerQuoted 5 lines
Quoted 4 lines
The flow-control was not disabled before, so according to your suggestion, I disable the flow-control on the both boards and run the test again, the performance is slightly improved, but still can not see a clear difference between the two methods. Below are the results.
Quoted 7 lines
Something else must be stalling the CPU.
When looking atfec_main.ccode, I noticed thatfec_enet_txq_xmit_frame()will do a MMIO write for everyxdp_frame(to trigger transmit start), which I believe will stall the CPU. The ndo_xdp_xmit/fec_enet_xdp_xmit does bulking, and should be the function that does the MMIO write to trigger transmit start.
We'd better keep a MMIO write for every xdp_frame on txq, as you know, the txq will be inactive when no additional ready descriptors remain in the tx-BDR. So it may increase the delay of the packets if we do a MMIO write for multiple packets.
Quoted 30 lines
$ git diffdrivers/net/ethernet/freescale/fec_main.c +3
-3@@ -3849,9 +3849,6 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, txq->bd.cur = bdp; -/* Trigger transmission start */ -writel (0 , txq->bd.reg_desc_active); -return 0 ; }@@ -3880,6 +3877,9 @@ static int fec_enet_xdp_xmit(struct net_device *dev, sent_frames++; } +/* Trigger transmission start */ +writel (0 , txq->bd.reg_desc_active); +__netif_tx_unlock (nq);return sent_frames;
Quoted 9 lines
Quoted 2 lines
Result: use "
sync_dma_len" methodroot@imx8mpevk:~# ./xdp2 eth0The xdp2 (and xdp1) program(s) have a performance issue (due to using
Can I ask you to test using
xdp_rxq_info, like:sudo ./xdp_rxq_info --dev mlx5p1 --action XDP_TX
Yes, below are the results, the results are also basically the same. Result 1: current method
./xdp_rxq_info --dev eth0 --action XDP_TX
Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmac
XDP stats CPU pps issue-pps XDP-RX CPU 0 259,102 0 XDP-RX CPU total 259,102 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 0:0 259,102 0 rx_queue_index 0:sum 259,102 Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmac XDP stats CPU pps issue-pps XDP-RX CPU 0 259,498 0 XDP-RX CPU total 259,498 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 0:0 259,496 0 rx_queue_index 0:sum 259,496 Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmac XDP stats CPU pps issue-pps XDP-RX CPU 0 259,408 0 XDP-RX CPU total 259,408
Result 2: dma_sync_len method
Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmac
XDP stats CPU pps issue-pps XDP-RX CPU 0 258,254 0 XDP-RX CPU total 258,254 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 0:0 258,254 0 rx_queue_index 0:sum 258,254 Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmac XDP stats CPU pps issue-pps XDP-RX CPU 0 259,316 0 XDP-RX CPU total 259,316 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 0:0 259,318 0 rx_queue_index 0:sum 259,318 Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmac XDP stats CPU pps issue-pps XDP-RX CPU 0 259,554 0 XDP-RX CPU total 259,554 RXQ stats RXQ:CPU pps issue-pps
rx_queue_index 0:0 259,553 0rx_queue_index 0:sum 259,553
Quoted 7 lines
Quoted 3 lines
proto 17: 258886 pkt/s proto 17: 258879 pkt/sIf you provide numbers for
xdp_redirect, then we could better evaluate if changing the lock perxdp_frame, forXDP_TXalso, is worth it.
For XDP_REDIRECT, the performance show as follow.
root@imx8mpevk:~# ./xdp_redirect eth1 eth0 Redirecting from eth1 (ifindex 3; driver st_gmac) to eth0 (ifindex 2; driver fec) eth1->eth0 221,642 rx/s 0 err,drop/s 221,643 xmit/s eth1->eth0 221,761 rx/s 0 err,drop/s 221,760 xmit/s eth1->eth0 221,793 rx/s 0 err,drop/s 221,794 xmit/s eth1->eth0 221,825 rx/s 0 err,drop/s 221,825 xmit/s eth1->eth0 221,823 rx/s 0 err,drop/s 221,821 xmit/s eth1->eth0 221,815 rx/s 0 err,drop/s 221,816 xmit/s eth1->eth0 222,016 rx/s 0 err,drop/s 222,016 xmit/s eth1->eth0 222,059 rx/s 0 err,drop/s 222,059 xmit/s eth1->eth0 222,085 rx/s 0 err,drop/s 222,089 xmit/s eth1->eth0 221,956 rx/s 0 err,drop/s 221,952 xmit/s eth1->eth0 222,070 rx/s 0 err,drop/s 222,071 xmit/s eth1->eth0 222,017 rx/s 0 err,drop/s 222,017 xmit/s eth1->eth0 222,069 rx/s 0 err,drop/s 222,067 xmit/s eth1->eth0 221,986 rx/s 0 err,drop/s 221,987 xmit/s eth1->eth0 221,932 rx/s 0 err,drop/s 221,936 xmit/s eth1->eth0 222,045 rx/s 0 err,drop/s 222,041 xmit/s eth1->eth0 222,014 rx/s 0 err,drop/s 222,014 xmit/s Packets received : 3,772,908 Average packets/s : 221,936 Packets transmitted : 3,772,908 Average transmit/s : 221,936
Quoted 2 lines
And also find out of moving the MMIO write have any effect.
I move the MMIO write to fec_enet_xdp_xmit(), the result shows as follow, the performance is slightly improved.
root@imx8mpevk:~# ./xdp_redirect eth1 eth0 Redirecting from eth1 (ifindex 3; driver st_gmac) to eth0 (ifindex 2; driver fec) eth1->eth0 222,666 rx/s 0 err,drop/s 222,668 xmit/s eth1->eth0 221,663 rx/s 0 err,drop/s 221,664 xmit/s eth1->eth0 222,743 rx/s 0 err,drop/s 222,741 xmit/s eth1->eth0 222,917 rx/s 0 err,drop/s 222,923 xmit/s eth1->eth0 221,810 rx/s 0 err,drop/s 221,808 xmit/s eth1->eth0 222,891 rx/s 0 err,drop/s 222,888 xmit/s eth1->eth0 222,983 rx/s 0 err,drop/s 222,984 xmit/s eth1->eth0 221,655 rx/s 0 err,drop/s 221,653 xmit/s eth1->eth0 222,827 rx/s 0 err,drop/s 222,827 xmit/s eth1->eth0 221,728 rx/s 0 err,drop/s 221,728 xmit/s eth1->eth0 222,790 rx/s 0 err,drop/s 222,789 xmit/s eth1->eth0 222,874 rx/s 0 err,drop/s 222,874 xmit/s eth1->eth0 221,888 rx/s 0 err,drop/s 221,887 xmit/s eth1->eth0 223,057 rx/s 0 err,drop/s 223,056 xmit/s eth1->eth0 222,219 rx/s 0 err,drop/s 222,220 xmit/s Packets received : 3,336,711 Average packets/s : 222,447 Packets transmitted : 3,336,710 Average transmit/s : 222,447
Quoted 4 lines
I also noticed driver does a MMIO write (on rxq) for every RX-packet in
fec_enet_rx_queue()napi-poll loop. This also looks like a potential performance stall.
The same as txq, the rxq will be inactive if the rx-BDR has no free BDs, so we'd better do a MMIO write when we recycle a BD, so that the hardware can timely attach the received pakcets on the rx-BDR.
In addition, I also tried to avoid using xdp_convert_buff_to_frame(), but the performance of XDP_TX is still not improved. :(
After these days of testing, I think it's best to keep the solution in V3, and then make some optimizations on the V3 patch.
Jesper Dangaard Brouer
maintainerQuoted 12 lines · Wei Fang
Quoted 5 lines
Quoted 4 lines
The flow-control was not disabled before, so according to your suggestion, I disable the flow-control on the both boards and run the test again, the performance is slightly improved, but still can not see a clear difference between the two methods. Below are the results.
Quoted 7 lines
Something else must be stalling the CPU.
When looking atfec_main.ccode, I noticed thatfec_enet_txq_xmit_frame()will do a MMIO write for everyxdp_frame(to trigger transmit start), which I believe will stall the CPU. The ndo_xdp_xmit/fec_enet_xdp_xmit does bulking, and should be the function that does the MMIO write to trigger transmit start.
Quoted 5 lines
We'd better keep a MMIO write for every
xdp_frameon txq, as you know, the txq will be inactive when no additional ready descriptors remain in the tx-BDR. So it may increase the delay of the packets if we do a MMIO write for multiple packets.
You know this hardware better than me, so I will leave to you.
Quoted 81 lines
Quoted 30 lines
$ git diffdrivers/net/ethernet/freescale/fec_main.c +3
-3@@ -3849,9 +3849,6 @@ static int fec_enet_txq_xmit_frame(struct fec_enet_private *fep, txq->bd.cur = bdp; -/* Trigger transmission start */ -writel (0 , txq->bd.reg_desc_active); -return 0 ; }@@ -3880,6 +3877,9 @@ static int fec_enet_xdp_xmit(struct net_device *dev, sent_frames++; } +/* Trigger transmission start */ +writel (0 , txq->bd.reg_desc_active); +__netif_tx_unlock (nq);return sent_frames;Quoted 9 lines
Quoted 2 lines
Result: use "
sync_dma_len" methodroot@imx8mpevk:~# ./xdp2 eth0The xdp2 (and xdp1) program(s) have a performance issue (due to using
Can I ask you to test using
xdp_rxq_info, like:sudo ./xdp_rxq_info --dev mlx5p1 --action XDP_TXYes, below are the results, the results are also basically the same. Result 1: current method
./xdp_rxq_info --dev eth0 --actionXDP_TX
Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmacXDP stats CPU pps issue-pps XDP-RX CPU 0 259,102 0 XDP-RX CPU total 259,102 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 0:0 259,102 0 rx_queue_index 0:sum 259,102 Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmac XDP stats CPU pps issue-pps XDP-RX CPU 0 259,498 0 XDP-RX CPU total 259,498 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 0:0 259,496 0 rx_queue_index 0:sum 259,496 Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmac XDP stats CPU pps issue-pps XDP-RX CPU 0 259,408 0 XDP-RX CPU total 259,408Result 2:
dma_sync_lenmethod
Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmacXDP stats CPU pps issue-pps XDP-RX CPU 0 258,254 0 XDP-RX CPU total 258,254 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 0:0 258,254 0 rx_queue_index 0:sum 258,254 Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmac XDP stats CPU pps issue-pps XDP-RX CPU 0 259,316 0 XDP-RX CPU total 259,316 RXQ stats RXQ:CPU pps issue-pps rx_queue_index 0:0 259,318 0 rx_queue_index 0:sum 259,318 Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmac XDP stats CPU pps issue-pps XDP-RX CPU 0 259,554 0 XDP-RX CPU total 259,554 RXQ stats RXQ:CPU pps issue-pps
Quoted 3 lines
rx_queue_index0:0 259,553 0rx_queue_index0:sum 259,553
Thanks for running this.
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
proto 17: 258886 pkt/s proto 17: 258879 pkt/sIf you provide numbers for
xdp_redirect, then we could better evaluate if changing the lock perxdp_frame, forXDP_TXalso, is worth it.For
XDP_REDIRECT, the performance show as follow.root@imx8mpevk:~# ./xdp_redirect eth1 eth0 Redirecting from eth1 (ifindex 3; driver st_gmac) to eth0 (ifindex 2; driver fec)
This is not exactly the same as XDP_TX setup as here you choose to redirect between eth1 (driver st_gmac) and to eth0 (driver fec).
I would like to see eth0 to eth0 XDP_REDIRECT, so we can compare to XDP_TX performance. Sorry for all the requests, but can you provide those numbers?
Quoted 1 line
eth1->eth0221,642 rx/s 0 err,drop/s 221,643 xmit/s
So, XDP_REDIRECT is approx (1-(221825/259554))*100 = 14.53% slower. But as this is 'eth1->eth0' this isn't true comparison to XDP_TX.
Quoted 16 lines
eth1->eth0 221,761 rx/s 0 err,drop/s 221,760 xmit/s eth1->eth0 221,793 rx/s 0 err,drop/s 221,794 xmit/s eth1->eth0 221,825 rx/s 0 err,drop/s 221,825 xmit/s eth1->eth0 221,823 rx/s 0 err,drop/s 221,821 xmit/s eth1->eth0 221,815 rx/s 0 err,drop/s 221,816 xmit/s eth1->eth0 222,016 rx/s 0 err,drop/s 222,016 xmit/s eth1->eth0 222,059 rx/s 0 err,drop/s 222,059 xmit/s eth1->eth0 222,085 rx/s 0 err,drop/s 222,089 xmit/s eth1->eth0 221,956 rx/s 0 err,drop/s 221,952 xmit/s eth1->eth0 222,070 rx/s 0 err,drop/s 222,071 xmit/s eth1->eth0 222,017 rx/s 0 err,drop/s 222,017 xmit/s eth1->eth0 222,069 rx/s 0 err,drop/s 222,067 xmit/s eth1->eth0 221,986 rx/s 0 err,drop/s 221,987 xmit/s eth1->eth0 221,932 rx/s 0 err,drop/s 221,936 xmit/s eth1->eth0 222,045 rx/s 0 err,drop/s 222,041 xmit/s eth1->eth0 222,014 rx/s 0 err,drop/s 222,014 xmit/s
Quoted 9 lines
Packets received : 3,772,908
Average packets/s : 221,936
Packets transmitted : 3,772,908
Average transmit/s : 221,936Quoted 2 lines
And also find out of moving the MMIO write have any effect.
I move the MMIO write to
fec_enet_xdp_xmit(), the result shows as follow, the performance is slightly improved.
I'm puzzled that moving the MMIO write isn't change performance.
Can you please verify that the packet generator machine is sending more frame than the system can handle?
(meaning the pktgen_sample03_burst_single_flow.sh script fast enough?)
Quoted 26 lines
root@imx8mpevk:~# ./xdp_redirect eth1 eth0 Redirecting from eth1 (ifindex 3; driver st_gmac) to eth0 (ifindex 2; driver fec) eth1->eth0 222,666 rx/s 0 err,drop/s 222,668 xmit/s eth1->eth0 221,663 rx/s 0 err,drop/s 221,664 xmit/s eth1->eth0 222,743 rx/s 0 err,drop/s 222,741 xmit/s eth1->eth0 222,917 rx/s 0 err,drop/s 222,923 xmit/s eth1->eth0 221,810 rx/s 0 err,drop/s 221,808 xmit/s eth1->eth0 222,891 rx/s 0 err,drop/s 222,888 xmit/s eth1->eth0 222,983 rx/s 0 err,drop/s 222,984 xmit/s eth1->eth0 221,655 rx/s 0 err,drop/s 221,653 xmit/s eth1->eth0 222,827 rx/s 0 err,drop/s 222,827 xmit/s eth1->eth0 221,728 rx/s 0 err,drop/s 221,728 xmit/s eth1->eth0 222,790 rx/s 0 err,drop/s 222,789 xmit/s eth1->eth0 222,874 rx/s 0 err,drop/s 222,874 xmit/s eth1->eth0 221,888 rx/s 0 err,drop/s 221,887 xmit/s eth1->eth0 223,057 rx/s 0 err,drop/s 223,056 xmit/s eth1->eth0 222,219 rx/s 0 err,drop/s 222,220 xmit/s Packets received : 3,336,711 Average packets/s : 222,447 Packets transmitted : 3,336,710 Average transmit/s : 222,447Quoted 4 lines
I also noticed driver does a MMIO write (on rxq) for every RX-packet in
fec_enet_rx_queue()napi-poll loop. This also looks like a potential performance stall.
Quoted 7 lines
The same as txq, the rxq will be inactive if the rx-BDR has no free BDs, so we'd better do a MMIO write when we recycle a BD, so that the hardware can timely attach the received pakcets on the rx-BDR.
In addition, I also tried to avoid using
xdp_convert_buff_to_frame(), but the performance ofXDP_TXis still not improved. :(
I would not expect much performance improvement from this anyhow.
Quoted 2 lines
After these days of testing, I think it's best to keep the solution in V3, and then make some optimizations on the V3 patch.
I agree.
I think you need to send a V5, and then I can ACK that.
Thanks for all this testing,
--Jesper
Wei Fang
maintainerQuoted 11 lines
Quoted 3 lines
For
XDP_REDIRECT, the performance show as follow.root@imx8mpevk:~# ./xdp_redirect eth1 eth0 Redirecting from eth1 (ifindex 3; driver st_gmac) to eth0 (ifindex 2; driver fec)This is not exactly the same as
XDP_TXsetup as here you choose to redirect between eth1 (driverst_gmac) and to eth0 (driver fec).I would like to see eth0 to eth0
XDP_REDIRECT, so we can compare toXDP_TXperformance.
Sorry for all the requests, but can you provide those numbers?
Oh, sorry, I thought what you wanted were XDP_REDIRECT results for different NICs. Below is the result of XDP_REDIRECT on the same NIC.
root@imx8mpevk:~# ./xdp_redirect eth0 eth0 Redirecting from eth0 (ifindex 2; driver fec) to eth0 (ifindex 2; driver fec) Summary 232,302 rx/s 0 err,drop/s 232,344 xmit/s Summary 234,579 rx/s 0 err,drop/s 234,577 xmit/s Summary 235,548 rx/s 0 err,drop/s 235,549 xmit/s Summary 234,704 rx/s 0 err,drop/s 234,703 xmit/s Summary 235,504 rx/s 0 err,drop/s 235,504 xmit/s Summary 235,223 rx/s 0 err,drop/s 235,224 xmit/s Summary 234,509 rx/s 0 err,drop/s 234,507 xmit/s Summary 235,481 rx/s 0 err,drop/s 235,482 xmit/s Summary 234,684 rx/s 0 err,drop/s 234,683 xmit/s Summary 235,520 rx/s 0 err,drop/s 235,520 xmit/s Summary 235,461 rx/s 0 err,drop/s 235,461 xmit/s Summary 234,627 rx/s 0 err,drop/s 234,627 xmit/s Summary 235,611 rx/s 0 err,drop/s 235,611 xmit/s Packets received : 3,053,753 Average packets/s : 234,904 Packets transmitted : 3,053,792 Average transmit/s : 234,907
Quoted 8 lines
I'm puzzled that moving the MMIO write isn't change performance.
Can you please verify that the packet generator machine is sending more frame than the system can handle?
(meaning the
pktgen_sample03_burst_single_flow.shscript fast enough?)
Thanks very much!
You remind me, I always started the pktgen script first and then ran the xdp2 program in the previous tests. So I saw the transmit speed of the generator was always greater than the speed of XDP_TX when I stopped the script. But actually, the real-time transmit speed of the generator was degraded to as equal to the speed of XDP_TX.
So I turned off the rx function of the generator in case of increasing the CPU loading of the generator due to the returned traffic from xdp2. And I tested the performance again. Below are the results.
Result 1: current method
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 326539 pkt/s proto 17: 326464 pkt/s proto 17: 326528 pkt/s proto 17: 326465 pkt/s proto 17: 326550 pkt/s
Result 2: sync_dma_len method
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 353918 pkt/s proto 17: 352923 pkt/s proto 17: 353900 pkt/s proto 17: 352672 pkt/s proto 17: 353912 pkt/s
Note: the speed of the generator is about 935397pps.
Compared result 1 with result 2. The "sync_dma_len" method actually improves the performance of XDP_TX, so the conclusion from the previous tests is *incorrect*. I'm so sorry for that. :(
In addition, I also tried the "dma_sync_len" + not use xdp_convert_buff_to_frame() method, the performance has been further improved. Below is the result.
Result 3: sync_dma_len + not use xdp_convert_buff_to_frame() method
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 369261 pkt/s proto 17: 369267 pkt/s proto 17: 369206 pkt/s proto 17: 369214 pkt/s proto 17: 369126 pkt/s
Therefore, I'm intend to use the "dma_sync_len"+ not use xdp_convert_buff_to_frame() method in the V5 patch. Thank you again, Jesper and Jakub. You really helped me a lot. :)
Jesper Dangaard Brouer
maintainerQuoted 42 lines · Wei Fang
Quoted 11 lines
Quoted 3 lines
For
XDP_REDIRECT, the performance show as follow.root@imx8mpevk:~# ./xdp_redirect eth1 eth0 Redirecting from eth1 (ifindex 3; driver st_gmac) to eth0 (ifindex 2; driver fec)This is not exactly the same as
XDP_TXsetup as here you choose to redirect between eth1 (driverst_gmac) and to eth0 (driver fec).I would like to see eth0 to eth0
XDP_REDIRECT, so we can compare toXDP_TXperformance.
Sorry for all the requests, but can you provide those numbers?Oh, sorry, I thought what you wanted were
XDP_REDIRECTresults for different NICs. Below is the result ofXDP_REDIRECTon the same NIC.root@imx8mpevk:~# ./xdp_redirect eth0 eth0 Redirecting from eth0 (ifindex 2; driver fec) to eth0 (ifindex 2; driver fec) Summary 232,302 rx/s 0 err,drop/s 232,344 xmit/s Summary 234,579 rx/s 0 err,drop/s 234,577 xmit/s Summary 235,548 rx/s 0 err,drop/s 235,549 xmit/s Summary 234,704 rx/s 0 err,drop/s 234,703 xmit/s Summary 235,504 rx/s 0 err,drop/s 235,504 xmit/s Summary 235,223 rx/s 0 err,drop/s 235,224 xmit/s Summary 234,509 rx/s 0 err,drop/s 234,507 xmit/s Summary 235,481 rx/s 0 err,drop/s 235,482 xmit/s Summary 234,684 rx/s 0 err,drop/s 234,683 xmit/s Summary 235,520 rx/s 0 err,drop/s 235,520 xmit/s Summary 235,461 rx/s 0 err,drop/s 235,461 xmit/s Summary 234,627 rx/s 0 err,drop/s 234,627 xmit/s Summary 235,611 rx/s 0 err,drop/s 235,611 xmit/s Packets received : 3,053,753 Average packets/s : 234,904 Packets transmitted : 3,053,792 Average transmit/s : 234,907Quoted 8 lines
I'm puzzled that moving the MMIO write isn't change performance.
Can you please verify that the packet generator machine is sending more frame than the system can handle?
(meaning the
pktgen_sample03_burst_single_flow.shscript fast enough?)
Quoted 7 lines
Thanks very much!
You remind me, I always started the pktgen script first and then ran the xdp2 program in the previous tests. So I saw the transmit speed of the generator was always greater than the speed ofXDP_TXwhen I stopped the script. But actually, the real-time transmit speed of the generator was degraded to as equal to the speed ofXDP_TX.
Good that we finally found the root-cause, that explains why it seems our code changes didn't have any effect. The generator gets affected and slowed down due to the traffic that is bounced back to it. (I tried to hint this earlier with the Ethernet Flow-Control settings).
Quoted 2 lines
So I turned off the rx function of the generator in case of increasing the CPU loading of the generator due to the returned traffic from xdp2.
How did you turned off the rx function of the generator? (I a couple of tricks I use)
Quoted 11 lines
And I tested
the performance again. Below are the results.Result 1: current method
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 326539 pkt/s proto 17: 326464 pkt/s proto 17: 326528 pkt/s proto 17: 326465 pkt/s proto 17: 326550 pkt/s
Quoted 8 lines
Result 2:
sync_dma_lenmethodroot@imx8mpevk:~# ./xdp2 eth0 proto 17: 353918 pkt/s proto 17: 352923 pkt/s proto 17: 353900 pkt/s proto 17: 352672 pkt/s proto 17: 353912 pkt/s
This looks more promising:
((353912/326550)-1)*100 = 8.37% faster.
Or gaining/saving approx 236 nanosec per packet ((1/326550-1/353912)*10^9).
Quoted 6 lines
Note: the speed of the generator is about 935397pps.
Compared result 1 with result 2. The "
sync_dma_len" method actually improves the performance ofXDP_TX, so the conclusion from the previous tests is *incorrect*. I'm so sorry for that. :(
I'm happy that we finally found the root-cause. Thanks for doing all the requested tests I asked for.
Quoted 11 lines
In addition, I also tried the "
dma_sync_len" + not usexdp_convert_buff_to_frame()method, the performance has been further improved. Below is the result.Result 3:
sync_dma_len+ not usexdp_convert_buff_to_frame()methodroot@imx8mpevk:~# ./xdp2 eth0 proto 17: 369261 pkt/s proto 17: 369267 pkt/s proto 17: 369206 pkt/s proto 17: 369214 pkt/s proto 17: 369126 pkt/s
Quoted 3 lines
Therefore, I'm intend to use the "dma_sync_len"+ not use
xdp_convert_buff_to_frame()method in the V5 patch. Thank you again, Jesper and Jakub. You really helped me a lot. :)
I suggest, that V5 patch still use xdp_convert_buff_to_frame(), and then you send followup patch (or as 2/2 patch) that remove the use of xdp_convert_buff_to_frame() for XDP_TX. This way it is easier to keep track of the changes and improvements.
I would be very interested in knowing if the MMIO test change after this correction to the testlab/generator.
--Jesper
Wei Fang
maintainerQuoted 13 lines
Quoted 7 lines
Thanks very much!
You remind me, I always started the pktgen script first and then ran the xdp2 program in the previous tests. So I saw the transmit speed of the generator was always greater than the speed ofXDP_TXwhen I stopped the script. But actually, the real-time transmit speed of the generator was degraded to as equal to the speed ofXDP_TX.Good that we finally found the root-cause, that explains why it seems our code changes didn't have any effect. The generator gets affected and slowed down due to the traffic that is bounced back to it. (I tried to hint this earlier with the Ethernet Flow-Control settings).
Quoted 6 lines
Quoted 2 lines
So I turned off the rx function of the generator in case of increasing the CPU loading of the generator due to the returned traffic from xdp2.
How did you turned off the rx function of the generator? (I a couple of tricks I use)
Actually, I didn't really disable the rx function of the generator, I just made the generator hardware automatically discard the returned traffic from xdp2. So I utilized the MAC filter feature of the hardware and did some modification to the pktgen script to make the SMAC of the packet is different from the MAC address of the generator.
Quoted 54 lines
Quoted 11 lines
And I tested
the performance again. Below are the results.Result 1: current method
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 326539 pkt/s proto 17: 326464 pkt/s proto 17: 326528 pkt/s proto 17: 326465 pkt/s proto 17: 326550 pkt/sQuoted 8 lines
Result 2:
sync_dma_lenmethodroot@imx8mpevk:~# ./xdp2 eth0 proto 17: 353918 pkt/s proto 17: 352923 pkt/s proto 17: 353900 pkt/s proto 17: 352672 pkt/s proto 17: 353912 pkt/sThis looks more promising:
((353912/326550)-1)*100 = 8.37% faster.Or gaining/saving approx 236 nanosec per packet
((1/326550-1/353912)*10^9).Quoted 4 lines
Note: the speed of the generator is about 935397pps.
Compared result 1 with result 2. The "
sync_dma_len" method actually improves the performance ofXDP_TX, so the conclusion from the previoustests is *incorrect*.
Quoted 2 lines
I'm so sorry for that. :(
I'm happy that we finally found the root-cause. Thanks for doing all the requested tests I asked for.
Quoted 2 lines
In addition, I also tried the "
dma_sync_len" + not usexdp_convert_buff_to_frame()method, the performance has been furtherimproved. Below is the result.
Quoted 9 lines
Result 3:
sync_dma_len+ not usexdp_convert_buff_to_frame()methodroot@imx8mpevk:~# ./xdp2 eth0 proto 17: 369261 pkt/s proto 17: 369267 pkt/s proto 17: 369206 pkt/s proto 17: 369214 pkt/s proto 17: 369126 pkt/sQuoted 4 lines
Therefore, I'm intend to use the "dma_sync_len"+ not use
xdp_convert_buff_to_frame()method in the V5 patch. Thank you again, Jesper and Jakub. You really helped me a lot. :)
Quoted 5 lines
I suggest, that V5 patch still use
xdp_convert_buff_to_frame(), and then you send followup patch (or as 2/2 patch) that remove the use ofxdp_convert_buff_to_frame()forXDP_TX. This way it is easier to keep track of the changes and improvements.
Okay, I will do it.
Quoted 3 lines
I would be very interested in knowing if the MMIO test change after this correction to the testlab/generator.
The performance is significantly improved as you expected, but as I explained before, I'm not sure whether there are the potential risks other than increase latency. So I'm not going to modify it at the moment.
Below is the result that I changed the logic to do a MMIO-write on rx-BDR and tx-BDR respectively in the end of the NPI callback.
root@imx8mpevk:~# ./xdp2 eth0 proto 17: 436020 pkt/s proto 17: 436167 pkt/s proto 17: 434205 pkt/s proto 17: 436140 pkt/s proto 17: 436115 pkt/s
Jakub Kicinski
maintainerQuoted 11 lines · On Mon, 7 Aug 2023 10:30:34 +0000 Wei Fang wrote:
./xdp_rxq_info --dev eth0 --action
XDP_TX
Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmacXDP stats CPU pps issue-pps XDP-RX CPU 0 259,102 0 XDP-RX CPU total 259,102Result 2:
dma_sync_lenmethod
Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmacXDP stats CPU pps issue-pps XDP-RX CPU 0 258,254 0 XDP-RX CPU total 258,254
Just to be clear are these number with xdp_return_frame() replaced with page_pool_put_page(pool, page, 0, true); ?
Wei Fang
maintainerQuoted 7 lines · On Mon, 7 Aug 2023 10:30:34 +0000 Wei Fang wrote:
Quoted 5 lines
./xdp_rxq_info --dev eth0 --action
XDP_TXRunning XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmacXDP stats CPU pps issue-pps XDP-RX CPU 0 259,102 0 XDP-RX CPU total 259,102
Quoted 8 lines
Quoted 5 lines
Result 2:
dma_sync_lenmethod
Running XDP on dev:eth0 (ifindex:2) action:XDP_TX options:swapmacXDP stats CPU pps issue-pps XDP-RX CPU 0 258,254 0 XDP-RX CPU total 258,254Just to be clear are these number with
xdp_return_frame()replaced withpage_pool_put_page(pool, page, 0, true); ?
Yes, I used the page_pool_put_page() to instead of xdp_return_frame() when I tried the "dma_sync_len" method.