net: dpaa_eth: convert to napi_gro_receive
Subsystems (3)
PATCH 1/1 net: dpaa_eth: convert to napi_gro_receive
Replace netif_receive_skb() with napi_gro_receive() to improve receive performance for this driver. It has rx checksum support so routing speed shouldn't suffer.
Tested on a WatchGuard Firebox M300.
iperf3 bidir speed test:
[ ID][Role] Interval Transfer Bitrate Retr [ 5][TX-C] 0.00-60.01 sec 5.35 GBytes 766 Mbits/sec 184 sender [ 5][TX-C] 0.00-60.02 sec 5.35 GBytes 766 Mbits/sec receiver [ 7][RX-C] 0.00-60.01 sec 5.50 GBytes 787 Mbits/sec 124 sender [ 7][RX-C] 0.00-60.02 sec 5.49 GBytes 786 Mbits/sec receiver
After:
[ ID][Role] Interval Transfer Bitrate Retr [ 5][TX-C] 0.00-60.01 sec 6.49 GBytes 929 Mbits/sec 0 sender [ 5][TX-C] 0.00-60.02 sec 6.49 GBytes 928 Mbits/sec receiver [ 7][RX-C] 0.00-60.01 sec 6.55 GBytes 938 Mbits/sec 0 sender [ 7][RX-C] 0.00-60.02 sec 6.55 GBytes 937 Mbits/sec receiver
--- drivers/net/ethernet/freescale/dpaa/dpaa_eth.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/net/ethernet/freescale/dpaa/dpaa_eth.c +1 -4
@@ -2824,10 +2824,7 @@ static enum qman_cb_dqrr_result rx_default_dqrr(struct qman_portal *portal, skb_len = skb->len; -if (unlikely (netif_receive_skb (skb) ==NET_RX_DROP )) { - percpu_stats->rx_dropped++; -return qman_cb_dqrr_consume; - } +napi_gro_receive (&np->napi, skb); percpu_stats->rx_packets++; percpu_stats->rx_bytes += skb_len;
Signature · 1 line
2.55.0
Hi Rosen,
Quoted 48 lines · Rosen Penev
Replace
netif_receive_skb()withnapi_gro_receive()to improve receive performance for this driver. It has rx checksum support so routing speed shouldn't suffer.Tested on a WatchGuard Firebox M300.
iperf3 bidir speed test:
[ ID][Role] Interval Transfer Bitrate Retr [ 5][TX-C] 0.00-60.01 sec 5.35 GBytes 766 Mbits/sec 184 sender [ 5][TX-C] 0.00-60.02 sec 5.35 GBytes 766 Mbits/sec receiver [ 7][RX-C] 0.00-60.01 sec 5.50 GBytes 787 Mbits/sec 124 sender [ 7][RX-C] 0.00-60.02 sec 5.49 GBytes 786 Mbits/sec receiverAfter:
[ ID][Role] Interval Transfer Bitrate Retr [ 5][TX-C] 0.00-60.01 sec 6.49 GBytes 929 Mbits/sec 0 sender [ 5][TX-C] 0.00-60.02 sec 6.49 GBytes 928 Mbits/sec receiver [ 7][RX-C] 0.00-60.01 sec 6.55 GBytes 938 Mbits/sec 0 sender [ 7][RX-C] 0.00-60.02 sec 6.55 GBytes 937 Mbits/sec receiverAssisted-by Opencode:big-pickleSigned-off-by Rosen Penev <[email protected]>--- drivers/net/ethernet/freescale/dpaa/dpaa_eth.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)drivers/net/ethernet/freescale/dpaa/dpaa_eth.c +1
-4@@ -2824,10 +2824,7 @@ static enum qman_cb_dqrr_result rx_default_dqrr(struct qman_portal *portal, skb_len = skb->len; -if (unlikely (netif_receive_skb (skb) ==NET_RX_DROP )) { - percpu_stats->rx_dropped++; -return qman_cb_dqrr_consume; - } +napi_gro_receive (&np->napi, skb); percpu_stats->rx_packets++; percpu_stats->rx_bytes += skb_len;Signature · 1 line
2.55.0
(thanks to Madalin for pointing out this change to me)
I do not have time this week to test this patch, but in premise, you are creating exactly the conditions for this bug to occur:
https://github.com/nxp-qoriq/linux/commit/d0ebec2092d6c5fe327513cb66e02d8c2c1a8f87
In sdk_dpaa we already do GRO for TCP flows and it has the problems pointed out in the above commit. With your change to use GRO for all capable flows, I currently have no reason to believe that without similar countermeasures as those taken by sdk_dpaa (napi_gro_receive() global to the entire QMan portal) the outcome would be different.
Have you tested traffic in mixed scenarios, where flows from multiple interfaces land on the same CPU?
Until further evidence comes in:
pw-bot: cr
Quoted 59 lines · Vladimir Oltean
Hi Rosen,
Quoted 45 lines · Rosen Penev
Replace
netif_receive_skb()withnapi_gro_receive()to improve receive performance for this driver. It has rx checksum support so routing speed shouldn't suffer.Tested on a WatchGuard Firebox M300.
iperf3 bidir speed test:
[ ID][Role] Interval Transfer Bitrate Retr [ 5][TX-C] 0.00-60.01 sec 5.35 GBytes 766 Mbits/sec 184 sender [ 5][TX-C] 0.00-60.02 sec 5.35 GBytes 766 Mbits/sec receiver [ 7][RX-C] 0.00-60.01 sec 5.50 GBytes 787 Mbits/sec 124 sender [ 7][RX-C] 0.00-60.02 sec 5.49 GBytes 786 Mbits/sec receiverAfter:
[ ID][Role] Interval Transfer Bitrate Retr [ 5][TX-C] 0.00-60.01 sec 6.49 GBytes 929 Mbits/sec 0 sender [ 5][TX-C] 0.00-60.02 sec 6.49 GBytes 928 Mbits/sec receiver [ 7][RX-C] 0.00-60.01 sec 6.55 GBytes 938 Mbits/sec 0 sender [ 7][RX-C] 0.00-60.02 sec 6.55 GBytes 937 Mbits/sec receiverAssisted-by Opencode:big-pickleSigned-off-by Rosen Penev <[email protected]>--- drivers/net/ethernet/freescale/dpaa/dpaa_eth.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)drivers/net/ethernet/freescale/dpaa/dpaa_eth.c +1
-5@@ -2824,10 +2824,7 @@ static enum qman_cb_dqrr_result rx_default_dqrr(struct qman_portal *portal, skb_len = skb->len; -if (unlikely (netif_receive_skb (skb) ==NET_RX_DROP )) { - percpu_stats->rx_dropped++; -return qman_cb_dqrr_consume; - } +napi_gro_receive (&np->napi, skb); percpu_stats->rx_packets++; percpu_stats->rx_bytes += skb_len; --Quoted 3 lines
2.55.0
(thanks to Madalin for pointing out this change to me)
I do not have time this week to test this patch, but in premise, you are creating exactly the conditions for this bug to occur:
https://github.com/nxp-qoriq/linux/commit/d0ebec2092d6c5fe327513cb66e02d8c2c1a8f87
Quoted 8 lines
In
sdk_dpaawe already do GRO for TCP flows and it has the problems pointed out in the above commit. With your change to use GRO for all capable flows, I currently have no reason to believe that without similar countermeasures as those taken bysdk_dpaa(napi_gro_receive()global to the entire QMan portal) the outcome would be different.Have you tested traffic in mixed scenarios, where flows from multiple interfaces land on the same CPU?
Just iperf3 --bidir.
Quoted 2 lines
Until further evidence comes in:
Maybe netif_receive_skb_list is a better fit here.
Quoted 2 lines
pw-bot: cr
Quoted 4 lines · Rosen Penev
Quoted 2 lines · Vladimir Oltean
Have you tested traffic in mixed scenarios, where flows from multiple interfaces land on the same CPU?
Just iperf3 --bidir.
Just this would not have reproduced the problem I pointed out.
Quoted 2 lines
Quoted 1 line
Until further evidence comes in:
Maybe
netif_receive_skb_listis a better fit here.
Maybe or maybe not. I don't know how netif_receive_skb_list() is implemented and whether it shares the same pain points as napi_gro_receive() on DPAA1, or what performance it has to offer. Testing in a wide variety of circumstances should show that.
I currently have very limited access to email for the rest of this week. I did not mean to completely dismiss napi_gro_receive(), just point out what kind of complexity is required for proper operation when the QMan portal mixes frames from different interfaces in the same NAPI poll cycle.