Skip to contentSkip to conversation

net: dpaa_eth: convert to napi_gro_receive

PATCH In review Rosen Penev 2 participants 4 messages Jul 6 – Jul 7

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net: dpaa_eth: convert to napi_gro_receive

Replace netif_receive_skb() with napi_gro_receive() to improve receive performance for this driver. It has rx checksum support so routing speed shouldn't suffer.

Tested on a WatchGuard Firebox M300.

iperf3 bidir speed test:

[ ID][Role]   Interval         Transfer      Bitrate        Retr
[  5][TX-C]   0.00-60.01  sec  5.35 GBytes   766 Mbits/sec  184    sender
[  5][TX-C]   0.00-60.02  sec  5.35 GBytes   766 Mbits/sec         receiver
[  7][RX-C]   0.00-60.01  sec  5.50 GBytes   787 Mbits/sec  124    sender
[  7][RX-C]   0.00-60.02  sec  5.49 GBytes   786 Mbits/sec         receiver

After:

[ ID][Role]   Interval         Transfer      Bitrate        Retr
[  5][TX-C]   0.00-60.01  sec  6.49 GBytes   929 Mbits/sec    0    sender
[  5][TX-C]   0.00-60.02  sec  6.49 GBytes   928 Mbits/sec         receiver
[  7][RX-C]   0.00-60.01  sec  6.55 GBytes   938 Mbits/sec    0    sender
[  7][RX-C]   0.00-60.02  sec  6.55 GBytes   937 Mbits/sec         receiver
Assisted-by Opencode:big-pickle
---
 drivers/net/ethernet/freescale/dpaa/dpaa_eth.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/net/ethernet/freescale/dpaa/dpaa_eth.c +1 -4
index ad2d8256eb8d..83191f636ec7 100644
@@ -2824,10 +2824,7 @@ static enum qman_cb_dqrr_result rx_default_dqrr(struct qman_portal *portal,
 
 	skb_len = skb->len;
 
-	if (unlikely(netif_receive_skb(skb) == NET_RX_DROP)) {
-		percpu_stats->rx_dropped++;
-		return qman_cb_dqrr_consume;
-	}
+	napi_gro_receive(&np->napi, skb);
 
 	percpu_stats->rx_packets++;
 	percpu_stats->rx_bytes += skb_len;
Signature · 1 line
2.55.0
#1 lore raw

Hi Rosen,

Quoted 48 lines · Rosen Penev

Replace netif_receive_skb() with napi_gro_receive() to improve receive performance for this driver. It has rx checksum support so routing speed shouldn't suffer.

Tested on a WatchGuard Firebox M300.

iperf3 bidir speed test:

[ ID][Role]   Interval         Transfer      Bitrate        Retr
[  5][TX-C]   0.00-60.01  sec  5.35 GBytes   766 Mbits/sec  184    sender
[  5][TX-C]   0.00-60.02  sec  5.35 GBytes   766 Mbits/sec         receiver
[  7][RX-C]   0.00-60.01  sec  5.50 GBytes   787 Mbits/sec  124    sender
[  7][RX-C]   0.00-60.02  sec  5.49 GBytes   786 Mbits/sec         receiver

After:

[ ID][Role]   Interval         Transfer      Bitrate        Retr
[  5][TX-C]   0.00-60.01  sec  6.49 GBytes   929 Mbits/sec    0    sender
[  5][TX-C]   0.00-60.02  sec  6.49 GBytes   928 Mbits/sec         receiver
[  7][RX-C]   0.00-60.01  sec  6.55 GBytes   938 Mbits/sec    0    sender
[  7][RX-C]   0.00-60.02  sec  6.55 GBytes   937 Mbits/sec         receiver
Assisted-by Opencode:big-pickle
---
 drivers/net/ethernet/freescale/dpaa/dpaa_eth.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/net/ethernet/freescale/dpaa/dpaa_eth.c +1 -4
index ad2d8256eb8d..83191f636ec7 100644
@@ -2824,10 +2824,7 @@ static enum qman_cb_dqrr_result rx_default_dqrr(struct qman_portal *portal,
 
 	skb_len = skb->len;
 
-	if (unlikely(netif_receive_skb(skb) == NET_RX_DROP)) {
-		percpu_stats->rx_dropped++;
-		return qman_cb_dqrr_consume;
-	}
+	napi_gro_receive(&np->napi, skb);
 
 	percpu_stats->rx_packets++;
 	percpu_stats->rx_bytes += skb_len;
Signature · 1 line
2.55.0

(thanks to Madalin for pointing out this change to me)

I do not have time this week to test this patch, but in premise, you are creating exactly the conditions for this bug to occur:
https://github.com/nxp-qoriq/linux/commit/d0ebec2092d6c5fe327513cb66e02d8c2c1a8f87

In sdk_dpaa we already do GRO for TCP flows and it has the problems pointed out in the above commit. With your change to use GRO for all capable flows, I currently have no reason to believe that without similar countermeasures as those taken by sdk_dpaa (napi_gro_receive() global to the entire QMan portal) the outcome would be different.

Have you tested traffic in mixed scenarios, where flows from multiple interfaces land on the same CPU?

Until further evidence comes in:

pw-bot: cr

#2 lore raw
Quoted 59 lines · Vladimir Oltean

Hi Rosen,

Quoted 45 lines · Rosen Penev

Replace netif_receive_skb() with napi_gro_receive() to improve receive performance for this driver. It has rx checksum support so routing speed shouldn't suffer.

Tested on a WatchGuard Firebox M300.

iperf3 bidir speed test:

[ ID][Role]   Interval         Transfer      Bitrate        Retr
[  5][TX-C]   0.00-60.01  sec  5.35 GBytes   766 Mbits/sec  184    sender
[  5][TX-C]   0.00-60.02  sec  5.35 GBytes   766 Mbits/sec         receiver
[  7][RX-C]   0.00-60.01  sec  5.50 GBytes   787 Mbits/sec  124    sender
[  7][RX-C]   0.00-60.02  sec  5.49 GBytes   786 Mbits/sec         receiver

After:

[ ID][Role]   Interval         Transfer      Bitrate        Retr
[  5][TX-C]   0.00-60.01  sec  6.49 GBytes   929 Mbits/sec    0    sender
[  5][TX-C]   0.00-60.02  sec  6.49 GBytes   928 Mbits/sec         receiver
[  7][RX-C]   0.00-60.01  sec  6.55 GBytes   938 Mbits/sec    0    sender
[  7][RX-C]   0.00-60.02  sec  6.55 GBytes   937 Mbits/sec         receiver
Assisted-by Opencode:big-pickle
---
 drivers/net/ethernet/freescale/dpaa/dpaa_eth.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/net/ethernet/freescale/dpaa/dpaa_eth.c +1 -5
index ad2d8256eb8d..83191f636ec7 100644
@@ -2824,10 +2824,7 @@ static enum qman_cb_dqrr_result rx_default_dqrr(struct qman_portal *portal,
 
      skb_len = skb->len;
 
-     if (unlikely(netif_receive_skb(skb) == NET_RX_DROP)) {
-             percpu_stats->rx_dropped++;
-             return qman_cb_dqrr_consume;
-     }
+     napi_gro_receive(&np->napi, skb);
 
      percpu_stats->rx_packets++;
      percpu_stats->rx_bytes += skb_len;
--
Quoted 3 lines

2.55.0

(thanks to Madalin for pointing out this change to me)

I do not have time this week to test this patch, but in premise, you are creating exactly the conditions for this bug to occur:
https://github.com/nxp-qoriq/linux/commit/d0ebec2092d6c5fe327513cb66e02d8c2c1a8f87

Quoted 8 lines

In sdk_dpaa we already do GRO for TCP flows and it has the problems pointed out in the above commit. With your change to use GRO for all capable flows, I currently have no reason to believe that without similar countermeasures as those taken by sdk_dpaa (napi_gro_receive() global to the entire QMan portal) the outcome would be different.

Have you tested traffic in mixed scenarios, where flows from multiple interfaces land on the same CPU?

Just iperf3 --bidir.

Quoted 2 lines

Until further evidence comes in:

Maybe netif_receive_skb_list is a better fit here.

Quoted 2 lines

pw-bot: cr

#3 lore raw
Quoted 4 lines · Rosen Penev
Quoted 2 lines · Vladimir Oltean

Have you tested traffic in mixed scenarios, where flows from multiple interfaces land on the same CPU?

Just iperf3 --bidir.

Just this would not have reproduced the problem I pointed out.

Quoted 2 lines
Quoted 1 line

Until further evidence comes in:

Maybe netif_receive_skb_list is a better fit here.

Maybe or maybe not. I don't know how netif_receive_skb_list() is implemented and whether it shares the same pain points as napi_gro_receive() on DPAA1, or what performance it has to offer. Testing in a wide variety of circumstances should show that.

I currently have very limited access to email for the rest of this week. I did not mean to completely dismiss napi_gro_receive(), just point out what kind of complexity is required for proper operation when the QMan portal mixes frames from different interfaces in the same NAPI poll cycle.