net: gro: do not keep too many GRO packets in napi->rx_list
Subsystems (2)
PATCH 1/1 net: gro: do not keep too many GRO packets in napi->rx_list
From: Eric Dumazet <[email protected]>
Commit c80794323e82 ("net: Fix packet reordering caused by GRO and listified RX cooperation") had the unfortunate effect of adding latencies in common workloads.
Before the patch, GRO packets were immediately passed to upper stacks.
After the patch, we can accumulate quite a lot of GRO packets (depdending on NAPI budget).
My fix is counting in napi->rx_count number of segments instead of number of logical packets.
--- net/core/dev.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
net/core/dev.c +6 -5
@@ -5735,10 +5735,11 @@ static void gro_normal_list(struct napi_struct *napi)/* Queue one GRO_NORMAL SKB up for list processing. If batch size exceeded, * pass the whole batch up to the stack. */ -static void gro_normal_one (struct napi_struct *napi,struct sk_buff *skb) +static void gro_normal_one (struct napi_struct *napi,struct sk_buff *skb,int segs) {list_add_tail (&skb->list, &napi->rx_list); -if (++napi->rx_count >= gro_normal_batch) + napi->rx_count += segs; +if (napi->rx_count >= gro_normal_batch)gro_normal_list (napi); }
@@ -5777,7 +5778,7 @@ static int napi_gro_complete(struct napi_struct *napi, struct sk_buff *skb) } out: -gro_normal_one (napi, skb); +gro_normal_one (napi, skb,NAPI_GRO_CB (skb)->count);return NET_RX_SUCCESS ; }
@@ -6067,7 +6068,7 @@ static gro_result_t napi_skb_finish(struct napi_struct *napi, {switch (ret) {case GRO_NORMAL : -gro_normal_one (napi, skb); +gro_normal_one (napi, skb,1 );break ;case GRO_DROP :
@@ -6155,7 +6156,7 @@ static gro_result_t napi_frags_finish(struct napi_struct *napi,__skb_push (skb,ETH_HLEN ); skb->protocol =eth_type_trans (skb, skb->dev);if (ret ==GRO_NORMAL ) -gro_normal_one (napi, skb); +gro_normal_one (napi, skb,1 );break ;case GRO_DROP :
Signature · 1 line
2.30.0.365.g02bc693789-goog
Quoted 6 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Commit
c80794323e82("net: Fix packet reordering caused by GRO and listified RX cooperation") had the unfortunate effect of adding latencies in common workloads.
Quoted 6 lines
Before the patch, GRO packets were immediately passed to upper stacks.
After the patch, we can accumulate quite a lot of GRO packets (depdending on NAPI budget).
Why napi budget ? looking at the code it seems to be more related to MAX_GRO_SKBS * gro_normal_batch, since we are counting GRO SKBs as 1
but maybe i am missing some information about the actual issue you are hitting.
Quoted 43 lines
My fix is counting in
napi->rx_countnumber of segments instead of number of logical packets.Fixes c80794323e82 ("net: Fix packet reordering caused by GRO andlistified RX cooperation")
Signed-off-by Eric Dumazet <[email protected]>Bisected-by John Sperbeck <[email protected]>Tested-by Jian Yang <[email protected]>Cc Maxim Mikityanskiy <[email protected]>Cc Alexander Lobakin <[email protected]>Cc Saeed Mahameed <[email protected]>Cc Edward Cree <[email protected]>---
net/core/dev.c | 11 ++++++-----1 file changed, 6 insertions(+), 5 deletions(-)net/core/dev.c +0
-0index
a979b86dbacda9dfe31dd8b269024f7f0f5a8ef1..449b45b843d40ece7dd1e2ed6a5996ee1db9f591100644net/core/dev.c +3
-2@@ -5735,10 +5735,11 @@ static void gro_normal_list(struct napi_struct *napi) /* Queue one GRO_NORMAL SKB up for list processing. If batch sizeexceeded, * pass the whole batch up to the stack. */ -static void gro_normal_one (struct napi_struct *napi,struct sk_buff *skb) +static void gro_normal_one(struct napi_struct *napi, struct sk_buff *skb,int segs) { list_add_tail(&skb->list, &napi->rx_list); - if (++napi->rx_count >= gro_normal_batch) + napi->rx_count += segs; + if (napi->rx_count >= gro_normal_batch)gro_normal_list(napi);
}net/core/dev.c +0
-0@@ -5777,7 +5778,7 @@ static int napi_gro_complete(struct napi_struct *napi, struct sk_buff *skb) }
Quoted 4 lines
out:
- gro_normal_one(napi, skb);
+ gro_normal_one(napi, skb, NAPI_GRO_CB(skb)->count);
Seems correct to me,
Quoted 11 lines · Saeed Mahameed
Quoted 9 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Commit
c80794323e82("net: Fix packet reordering caused by GRO and listified RX cooperation") had the unfortunate effect of adding latencies in common workloads.Before the patch, GRO packets were immediately passed to upper stacks.
Quoted 6 lines
Quoted 3 lines
After the patch, we can accumulate quite a lot of GRO packets (depdending on NAPI budget).
Why napi budget ? looking at the code it seems to be more related to
MAX_GRO_SKBS*gro_normal_batch, since we are counting GRO SKBs as 1
Simply because we call gro_normal_list() from napi_poll(),
So we flush the napi rx_list every 64 packets under stress.(assuming NIC driver uses NAPI_POLL_WEIGHT),
or more often if napi_complete_done() is called if the budget was not exhausted.
GRO always has been able to keep MAX_GRO_SKBS in its layer, but no recent patch has changed this part.
Quoted 4 lines
but maybe i am missing some information about the actual issue you are hitting.
Well, the issue is precisely described in the changelog.
Quoted 55 lines
Quoted 43 lines
My fix is counting in
napi->rx_countnumber of segments instead of number of logical packets.Fixes c80794323e82 ("net: Fix packet reordering caused by GRO andlistified RX cooperation")
Signed-off-by Eric Dumazet <[email protected]>Bisected-by John Sperbeck <[email protected]>Tested-by Jian Yang <[email protected]>Cc Maxim Mikityanskiy <[email protected]>Cc Alexander Lobakin <[email protected]>Cc Saeed Mahameed <[email protected]>Cc Edward Cree <[email protected]>--- net/core/dev.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)net/core/dev.c +0
-0index
a979b86dbacda9dfe31dd8b269024f7f0f5a8ef1..449b45b843d40ece7dd1e2ed6a5996ee1db9f591100644net/core/dev.c +3
-2@@ -5735,10 +5735,11 @@ static void gro_normal_list(struct napi_struct *napi)/* Queue one GRO_NORMAL SKB up for list processing. If batch size exceeded, * pass the whole batch up to the stack. */ -static void gro_normal_one (struct napi_struct *napi,struct sk_buff *skb) +static void gro_normal_one(struct napi_struct *napi, struct sk_buff *skb,int segs) {list_add_tail (&skb->list, &napi->rx_list); -if (++napi->rx_count >= gro_normal_batch) + napi->rx_count += segs; +if (napi->rx_count >= gro_normal_batch)gro_normal_list (napi); }net/core/dev.c static int napi_gro_complete(struct napi_struct · line 5782
Quoted 6 lines
net/core/dev.c +1
-1@@ -5777,7 +5778,7 @@ static int napi_gro_complete(struct napi_struct *napi, struct sk_buff *skb) } out: -gro_normal_one (napi, skb); +gro_normal_one (napi, skb,NAPI_GRO_CB (skb)->count);Seems correct to me,
Reviewed-by Saeed Mahameed <[email protected]>
Alexander Lobakin
Reviewed-byFrom: Eric Dumazet <[email protected]>
Date: Thu, 4 Feb 2021 23:44:17 +0100
Quoted 20 lines · Saeed Mahameed
Quoted 11 lines · Eric Dumazet
Quoted 9 lines
From: Eric Dumazet <[email protected]>
Commit
c80794323e82("net: Fix packet reordering caused by GRO and listified RX cooperation") had the unfortunate effect of adding latencies in common workloads.Before the patch, GRO packets were immediately passed to upper stacks.
Quoted 6 lines
Quoted 3 lines
After the patch, we can accumulate quite a lot of GRO packets (depdending on NAPI budget).
Why napi budget ? looking at the code it seems to be more related to
MAX_GRO_SKBS*gro_normal_batch, since we are counting GRO SKBs as 1
Quoted 5 lines
Simply because we call
gro_normal_list()fromnapi_poll(),So we flush the napi
rx_listevery 64 packets under stress.(assuming NIC driver usesNAPI_POLL_WEIGHT),
or more often ifnapi_complete_done()is called if the budget was not exhausted.
Saeed,
Eric means that if we have e.g. 8 GRO packets with 8 segs each, then rx_list will be flushed only after processing of 64 ingress frames.
Quoted 6 lines
GRO always has been able to keep
MAX_GRO_SKBSin its layer, but no recent patch has changed this part.Quoted 2 lines
Quoted 18 lines
Quoted 2 lines
but maybe i am missing some information about the actual issue you are hitting.
Well, the issue is precisely described in the changelog.
Quoted 12 lines
Quoted 12 lines
My fix is counting in
napi->rx_countnumber of segments instead of number of logical packets.Fixes c80794323e82 ("net: Fix packet reordering caused by GRO andlistified RX cooperation")
Signed-off-by Eric Dumazet <[email protected]>Bisected-by John Sperbeck <[email protected]>Tested-by Jian Yang <[email protected]>Cc Maxim Mikityanskiy <[email protected]>Cc Alexander Lobakin <[email protected]>
It's strange why mailmap didn't pick up my active email at pm.me.
Anyways, this fix is correct for me. It restores the original Edward's logics, but without spurious out-of-order deliveries. Moreover, the pre-patch behaviour can easily be achieved by increasing net.core.gro_normal_batch if needed.
Thanks!
Quoted 34 lines
Quoted 34 lines
Quoted 34 lines
Cc Saeed Mahameed <[email protected]>Cc Edward Cree <[email protected]>--- net/core/dev.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)net/core/dev.c +0
-0index
a979b86dbacda9dfe31dd8b269024f7f0f5a8ef1..449b45b843d40ece7dd1e2ed6a5996ee1db9f591100644net/core/dev.c +3
-2@@ -5735,10 +5735,11 @@ static void gro_normal_list(struct napi_struct *napi)/* Queue one GRO_NORMAL SKB up for list processing. If batch size exceeded, * pass the whole batch up to the stack. */ -static void gro_normal_one (struct napi_struct *napi,struct sk_buff *skb) +static void gro_normal_one(struct napi_struct *napi, struct sk_buff *skb,int segs) {list_add_tail (&skb->list, &napi->rx_list); -if (++napi->rx_count >= gro_normal_batch) + napi->rx_count += segs; +if (napi->rx_count >= gro_normal_batch)gro_normal_list (napi); }@@ -5777,7 +5778,7 @@ static int napi_gro_complete(struct napi_struct *napi, struct sk_buff *skb) }
Quoted 7 lines
Quoted 7 lines
net/core/dev.c static int napi_gro_complete(struct napi_struct *napi, struct sk_buff *skb) · line 5781
Quoted 3 lines
net/core/dev.c +1
-1out: -gro_normal_one (napi, skb); +gro_normal_one (napi, skb,NAPI_GRO_CB (skb)->count);Seems correct to me,
Reviewed-by Saeed Mahameed <[email protected]>
Al
Quoted 4 lines · Alexander Lobakin
It's strange why mailmap didn't pick up my active email at pm.me.
I took the signatures from c80794323e82, I CCed all people involved in this recent patch.
It is very rare I use scripts/get_maintainer.pl since it tends to be noisy.
Quoted 10 lines
Anyways, this fix is correct for me. It restores the original Edward's logics, but without spurious out-of-order deliveries. Moreover, the pre-patch behaviour can easily be achieved by increasing net.core.gro_normal_batch if needed.
Thanks!
Reviewed-by Alexander Lobakin <[email protected]>
Thanks.
Quoted 15 lines · Eric Dumazet
From: Eric Dumazet <[email protected]>
Commit
c80794323e82("net: Fix packet reordering caused by GRO and listified RX cooperation") had the unfortunate effect of adding latencies in common workloads.Before the patch, GRO packets were immediately passed to upper stacks.
After the patch, we can accumulate quite a lot of GRO packets (depdending on NAPI budget).
My fix is counting in
napi->rx_countnumber of segments instead of number of logical packets.
Quoted 8 lines
Fixes c80794323e82 ("net: Fix packet reordering caused by GRO and listified RX cooperation")Signed-off-by Eric Dumazet <[email protected]>Bisected-by John Sperbeck <[email protected]>Tested-by Jian Yang <[email protected]>Cc Maxim Mikityanskiy <[email protected]>Cc Alexander Lobakin <[email protected]>Cc Saeed Mahameed <[email protected]>Cc Edward Cree <[email protected]>
This patch was applied to netdev/net.git (refs/heads/master):
Hello:
This patch was applied to netdev/net.git (refs/heads/master):
Quoted 10 lines · On Thu, 4 Feb 2021 13:31:46 -0800 you wrote:
From: Eric Dumazet <[email protected]>
Commit
c80794323e82("net: Fix packet reordering caused by GRO and listified RX cooperation") had the unfortunate effect of adding latencies in common workloads.Before the patch, GRO packets were immediately passed to upper stacks.
[...]
Here is the summary with links:
- [net] net: gro: do not keep too many GRO packets in napi->rx_list
https://git.kernel.org/netdev/net/c/8dc1c444df19
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html