net: optimize TX throughput and efficiency
Subsystems (4)
v2 cover net: optimize TX throughput and efficiency
In this series, I replace the busylock spinlock we have in __dev_queue_xmit() and use lockless list (llist) to reduce spinlock contention to the minimum.
Idea is that only one cpu might spin on the qdisc spinlock, while others simply add their skb in the llist.
After this series, we get a 300 % (4x) improvement on heavy TX workloads, sending twice the number of packets per second, for half the cpu cycles.
v2: deflake tcp_user_timeout_user-timeout-probe.pkt.
Ability to return a different code than NET_XMIT_SUCCESS when __dev_xmit_skb() has a single skb to send.
Eric Dumazet (6):
selftests/net: packetdrill: unflake
tcp_user_timeout_user-timeout-probe.pkt
net: add add indirect call wrapper in skb_release_head_state() net/sched: act_mirred: add loop detection
Revert "net/sched: Fix mirred deadlock on device recursion"
net: sched: claim one cache line in Qdisc
net: dev_queue_xmit() llist adoption
include/linux/netdevice_xmit.h | 9 +- include/net/sch_generic.h | 23 ++--- net/core/dev.c | 97 +++++++++++-------- net/core/skbuff.c | 11 ++- net/sched/act_mirred.c | 62 +++++------- net/sched/sch_generic.c | 7 -- .../tcp_user_timeout_user-timeout-probe.pkt | 6 +- 7 files changed, 111 insertions(+), 104 deletions(-)
Signature · 1 line
2.51.0.788.g6d19910ace-goog
PATCH v2 1/6 selftests/net: packetdrill: unflake tcp_user_timeout_user-timeout-probe.pkt
This test fails the first time I am running it after a fresh virtme-ng boot.
tcp_user_timeout_user-timeout-probe.pkt:33: runtime error in write call: Expected result -1 but got 24 with errno 2 (No such file or directory)
Tweaks the timings a bit, to reduce flakiness.
--- .../net/packetdrill/tcp_user_timeout_user-timeout-probe.pkt | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
tools/testing/selftests/net/packetdrill/tcp_user_timeout_user-timeout-probe.pkt +4 -2
@@ -23,14 +23,16 @@ // install a qdisc dropping all packets +0 `tc qdisc delete dev tun0 root 2>/dev/null ; tc qdisc add dev tun0 root pfifo limit 0` + +0 write(4, ..., 24) = 24 // When qdisc is congested we retry every 500ms // (TCP_RESOURCE_PROBE_INTERVAL) and therefore // we retry 6 times before hitting 3s timeout. // First verify that the connection is alive: -+3.250 write(4, ..., 24) = 24 ++3 write(4, ..., 24) = 24 + // Now verify that shortly after that the socket is dead: - +.100 write(4, ..., 24) = -1 ETIMEDOUT (Connection timed out) ++1 write(4, ..., 24) = -1 ETIMEDOUT (Connection timed out) +0 %{ assert tcpi_probes == 6, tcpi_probes; \ assert tcpi_backoff == 0, tcpi_backoff }%
Signature · 1 line
2.51.0.788.g6d19910ace-goog
Re: [PATCH v2 net-next 1/6] selftests/net: packetdrill: unflake tcp_user_timeout_user-timeout-probe.pkt
Quoted 8 lines · Eric Dumazet
This test fails the first time I am running it after a fresh virtme-ng boot.
tcp_user_timeout_user-timeout-probe.pkt:33: runtime error in write call: Expected result -1 but got 24 with errno 2 (No such file or directory)
Tweaks the timings a bit, to reduce flakiness.
Signed-off-by Eric Dumazet <[email protected]>
PATCH v2 2/6 net: add add indirect call wrapper in skb_release_head_state()
In reply to Eric DumazetWhile stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in sock_wfree() due to CONFIG_MITIGATION_RETPOLINE=y.
Take care of TCP and UDP TX destructors and use INDIRECT_CALL_3() macro.
--- net/core/skbuff.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
net/core/skbuff.c +10 -1
@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)skb_dst_drop (skb);if (skb->destructor) {DEBUG_NET_WARN_ON_ONCE (in_hardirq ()); - skb->destructor (skb); +#ifdef CONFIG_INET +INDIRECT_CALL_3 (skb->destructor, + tcp_wfree, __sock_wfree, sock_wfree, + skb); +#else +INDIRECT_CALL_1 (skb->destructor, + sock_wfree, + skb); + +#endif }#if IS_ENABLED (CONFIG_NF_CONNTRACK )nf_conntrack_put (skb_nfct (skb));
Signature · 1 line
2.51.0.788.g6d19910ace-goog
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
Quoted 9 lines · Eric Dumazet
While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in
sock_wfree()due toCONFIG_MITIGATION_RETPOLINE=y.Take care of TCP and UDP TX destructors and use
INDIRECT_CALL_3()macro.Signed-off-by Eric Dumazet <[email protected]>
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
In reply to Eric DumazetQuoted 8 lines · Eric Dumazet
While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in
sock_wfree()due toCONFIG_MITIGATION_RETPOLINE=y.Take care of TCP and UDP TX destructors and use
INDIRECT_CALL_3()macro.Signed-off-by Eric Dumazet <[email protected]>
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
In reply to Eric DumazetFrom: Eric Dumazet <[email protected]>
Date: Tue, 14 Oct 2025 17:19:03 +0000
Quoted 21 lines
While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in
sock_wfree()due toCONFIG_MITIGATION_RETPOLINE=y.Take care of TCP and UDP TX destructors and use
INDIRECT_CALL_3()macro.Signed-off-by Eric Dumazet <[email protected]>--- net/core/skbuff.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)net/core/skbuff.c +0
-1@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)skb_dst_drop (skb);if (skb->destructor) {DEBUG_NET_WARN_ON_ONCE (in_hardirq ()); - skb->destructor (skb);
net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1148
Quoted 10 lines
net/core/skbuff.c +10
-0+#ifdef CONFIG_INET +INDIRECT_CALL_3 (skb->destructor, + tcp_wfree, __sock_wfree, sock_wfree, + skb); +#else +INDIRECT_CALL_1 (skb->destructor, + sock_wfree, + skb); + +#endif
Is it just me or seems like you ignored the suggestion/discussion under v1 of this patch...
net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1151
Quoted 3 lines
net/core/skbuff.c +0
-0}#if IS_ENABLED (CONFIG_NF_CONNTRACK )nf_conntrack_put (skb_nfct (skb));
Thanks,
Olek
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
Quoted 30 lines · Alexander Lobakin
From: Eric Dumazet <[email protected]>
Date: Tue, 14 Oct 2025 17:19:03 +0000Quoted 26 lines
While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in
sock_wfree()due toCONFIG_MITIGATION_RETPOLINE=y.Take care of TCP and UDP TX destructors and use
INDIRECT_CALL_3()macro.Signed-off-by Eric Dumazet <[email protected]>--- net/core/skbuff.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)net/core/skbuff.c +5
-1@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)skb_dst_drop (skb);if (skb->destructor) {DEBUG_NET_WARN_ON_ONCE (in_hardirq ()); - skb->destructor (skb); +#ifdef CONFIG_INET +INDIRECT_CALL_3 (skb->destructor, + tcp_wfree, __sock_wfree, sock_wfree, + skb); +#else
Quoted 9 lines
net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1148
Quoted 5 lines
net/core/skbuff.c +5
-0+INDIRECT_CALL_1 (skb->destructor, + sock_wfree, + skb); + +#endif Is it just me or seems like you ignored the suggestion/discussion under v1 of this patch...
I did not. Please send a patch when you can demonstrate the difference.
We are not going to add all the possible destructors unless there is evidence.
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
From: Eric Dumazet <[email protected]>
Date: Wed, 15 Oct 2025 05:16:05 -0700
Quoted 42 lines
Quoted 30 lines · Alexander Lobakin
From: Eric Dumazet <[email protected]>
Date: Tue, 14 Oct 2025 17:19:03 +0000Quoted 26 lines
While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in
sock_wfree()due toCONFIG_MITIGATION_RETPOLINE=y.Take care of TCP and UDP TX destructors and use
INDIRECT_CALL_3()macro.Signed-off-by Eric Dumazet <[email protected]>--- net/core/skbuff.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)net/core/skbuff.c +5
-1@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)skb_dst_drop (skb);if (skb->destructor) {DEBUG_NET_WARN_ON_ONCE (in_hardirq ()); - skb->destructor (skb); +#ifdef CONFIG_INET +INDIRECT_CALL_3 (skb->destructor, + tcp_wfree, __sock_wfree, sock_wfree, + skb); +#else Quoted 9 lines
net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1148
Quoted 5 lines
net/core/skbuff.c +5
-0+INDIRECT_CALL_1 (skb->destructor, + sock_wfree, + skb); + +#endif Is it just me or seems like you ignored the suggestion/discussion under v1 of this patch...
Quoted 1 line
I did not. Please send a patch when you can demonstrate the difference.
You "did not", but you didn't reply there, only sent v2 w/o any mention.
Quoted 2 lines
We are not going to add all the possible destructors unless there is evidence.
There are numbers in the original discussion, you'd have noticed if you did read.
We only ask to add one more destructor which will help certain perf-critical workloads. Add it to the end of the list, so that it won't hurt your optimization.
"Send a patch" means you're now changing these lines now and then they would be changed once again, why...
Thanks,
Olek
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
Quoted 56 lines · Alexander Lobakin
From: Eric Dumazet <[email protected]>
Date: Wed, 15 Oct 2025 05:16:05 -0700Quoted 42 lines
Quoted 30 lines · Alexander Lobakin
From: Eric Dumazet <[email protected]>
Date: Tue, 14 Oct 2025 17:19:03 +0000Quoted 26 lines
While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in
sock_wfree()due toCONFIG_MITIGATION_RETPOLINE=y.Take care of TCP and UDP TX destructors and use
INDIRECT_CALL_3()macro.Signed-off-by Eric Dumazet <[email protected]>--- net/core/skbuff.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)net/core/skbuff.c +5
-1@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)skb_dst_drop (skb);if (skb->destructor) {DEBUG_NET_WARN_ON_ONCE (in_hardirq ()); - skb->destructor (skb); +#ifdef CONFIG_INET +INDIRECT_CALL_3 (skb->destructor, + tcp_wfree, __sock_wfree, sock_wfree, + skb); +#else Quoted 9 lines
net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1148
Quoted 5 lines
net/core/skbuff.c +5
-0+INDIRECT_CALL_1 (skb->destructor, + sock_wfree, + skb); + +#endif Is it just me or seems like you ignored the suggestion/discussion under v1 of this patch...
Quoted 1 line
I did not. Please send a patch when you can demonstrate the difference.
You "did not", but you didn't reply there, only sent v2 w/o any mention.
Quoted 2 lines
We are not going to add all the possible destructors unless there is evidence.
There are numbers in the original discussion, you'd have noticed if you did read.
Quoted 6 lines
We only ask to add one more destructor which will help certain perf-critical workloads. Add it to the end of the list, so that it won't hurt your optimization.
"Send a patch" means you're now changing these lines now and then they would be changed once again, why...
I can not test what you propose.
I can drop this patch instead, and keep it in Google kernels, (we had TCP support for years)
Or... you can send a patch on top of it later.
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
Quoted 8 lines · Eric Dumazet
I can not test what you propose.
I can drop this patch instead, and keep it in Google kernels, (we had TCP support for years)
Or... you can send a patch on top of it later.
To be very clear : my Signed-off-by: means that I have strong confidence on what I tested.
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
In reply to Eric DumazetFrom: Eric Dumazet <[email protected]>
Date: Wed, 15 Oct 2025 05:46:27 -0700
Quoted 58 lines
Quoted 56 lines · Alexander Lobakin
From: Eric Dumazet <[email protected]>
Date: Wed, 15 Oct 2025 05:16:05 -0700Quoted 42 lines
Quoted 30 lines · Alexander Lobakin
From: Eric Dumazet <[email protected]>
Date: Tue, 14 Oct 2025 17:19:03 +0000Quoted 26 lines
While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in
sock_wfree()due toCONFIG_MITIGATION_RETPOLINE=y.Take care of TCP and UDP TX destructors and use
INDIRECT_CALL_3()macro.Signed-off-by Eric Dumazet <[email protected]>--- net/core/skbuff.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)net/core/skbuff.c +5
-1@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)skb_dst_drop (skb);if (skb->destructor) {DEBUG_NET_WARN_ON_ONCE (in_hardirq ()); - skb->destructor (skb); +#ifdef CONFIG_INET +INDIRECT_CALL_3 (skb->destructor, + tcp_wfree, __sock_wfree, sock_wfree, + skb); +#else Quoted 9 lines
net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1148
Quoted 5 lines
net/core/skbuff.c +5
-0+INDIRECT_CALL_1 (skb->destructor, + sock_wfree, + skb); + +#endif Is it just me or seems like you ignored the suggestion/discussion under v1 of this patch...
Quoted 1 line
I did not. Please send a patch when you can demonstrate the difference.
You "did not", but you didn't reply there, only sent v2 w/o any mention.
Quoted 2 lines
We are not going to add all the possible destructors unless there is evidence.
There are numbers in the original discussion, you'd have noticed if you did read.
Quoted 8 lines
Quoted 6 lines
We only ask to add one more destructor which will help certain perf-critical workloads. Add it to the end of the list, so that it won't hurt your optimization.
"Send a patch" means you're now changing these lines now and then they would be changed once again, why...
I can not test what you propose.
You asked *me* to show the difference, in the orig discussion there's a patch, there are tests and there is difference... :D
Quoted 3 lines
I can drop this patch instead, and keep it in Google kernels, (we had TCP support for years)
Ok, enough, leave this one as it is, we'll send the XSk bit ourselves.
Quoted 2 lines
Or... you can send a patch on top of it later.
Re "my Signed-off-by means I have strong confidence" -- sometimes we also have Tested-by from other folks and it's never been a problem, hey we're the community.
Thanks,
Olek
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
Quoted 2 lines · Alexander Lobakin
You asked *me* to show the difference, in the orig discussion there's a patch, there are tests and there is difference... :D
I am afraid I have not seen this.
The only thing I found was :
<quote>
Not sure, but maybe we could add generic XSk skb destructor here as well? Or it's not that important as generic XSk is not the best way to use XDP sockets?
Maciej, what do you think?
</quote>
No numbers.
Quoted 16 lines
Quoted 4 lines
I can drop this patch instead, and keep it in Google kernels, (we had TCP support for years)
Ok, enough, leave this one as it is, we'll send the XSk bit ourselves.
Quoted 2 lines
Or... you can send a patch on top of it later.
Re "my Signed-off-by means I have strong confidence" -- sometimes we also have Tested-by from other folks and it's never been a problem, hey we're the community.
Thanks,
Olek
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
From: Eric Dumazet <[email protected]>
Date: Wed, 15 Oct 2025 06:01:40 -0700
Quoted 15 lines
Quoted 2 lines · Alexander Lobakin
You asked *me* to show the difference, in the orig discussion there's a patch, there are tests and there is difference... :D
I am afraid I have not seen this.
The only thing I found was :
<quote>
Not sure, but maybe we could add generic XSk skb destructor here as well? Or it's not that important as generic XSk is not the best way to use XDP sockets?
Quoted 4 lines
Maciej, what do you think?
</quote>No numbers.
From [0]:
From: Jason Xing <[email protected]>
Date: Thu, 9 Oct 2025 16:37:56 +0800
Quoted 3 lines
Quoted 1 line · Maciej Fijalkowski
[...]
Quoted 22 lines
Quoted 2 lines
Quoted 2 lines
Not sure, but maybe we could add generic XSk skb destructor here as well?
I added the following snippet[1] and only saw a stable ~1% improvement when sending 64 size packets with xdpsock.
I'm not so sure it deserves a follow-up patch to Eric's series. Better than nothing? Any ideas on this one?
[1]
INDIRECT_CALL_4(skb->destructor, tcp_wfree, __sock_wfree, sock_wfree, xsk_destruct_skb, skb);Quoted 2 lines
Quoted 2 lines
Or it's not that important as generic XSk is not the best way to
use XDP sockets?Yes, it surely matters. At least,
virtio_netand veth need this copy mode. And I've been working on batch xmit to ramp up the generic path.Quoted 3 lines
Quoted 2 lines
Maciej, what do you think?
Quoted 7 lines
Quoted 2 lines
I would appreciate it as there has been various attempts to optmize xsk generic xmit path.
So do I!
Thanks,
Jason
[0]
https://lore.kernel.org/netdev/CAL+tcoBN9puWX-sTGvTiBN0Hg5oXKR3mjv783YXeR4Bsovuxkw@mail.gmail.com
Thanks,
Olek
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
Quoted 4 lines · Alexander Lobakin
Quoted 4 lines
I added the following snippet[1] and only saw a stable ~1% improvement when sending 64 size packets with xdpsock.
1% is noise level, I am definitely not convinced.
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
In reply to Eric DumazetQuoted 66 lines · Eric Dumazet
Quoted 56 lines · Alexander Lobakin
From: Eric Dumazet <[email protected]>
Date: Wed, 15 Oct 2025 05:16:05 -0700Quoted 42 lines
Quoted 30 lines · Alexander Lobakin
From: Eric Dumazet <[email protected]>
Date: Tue, 14 Oct 2025 17:19:03 +0000Quoted 26 lines
While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in
sock_wfree()due toCONFIG_MITIGATION_RETPOLINE=y.Take care of TCP and UDP TX destructors and use
INDIRECT_CALL_3()macro.Signed-off-by Eric Dumazet <[email protected]>--- net/core/skbuff.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)net/core/skbuff.c +5
-1@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)skb_dst_drop (skb);if (skb->destructor) {DEBUG_NET_WARN_ON_ONCE (in_hardirq ()); - skb->destructor (skb); +#ifdef CONFIG_INET +INDIRECT_CALL_3 (skb->destructor, + tcp_wfree, __sock_wfree, sock_wfree, + skb); +#else Quoted 9 lines
net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1148
Quoted 5 lines
net/core/skbuff.c +5
-0+INDIRECT_CALL_1 (skb->destructor, + sock_wfree, + skb); + +#endif Is it just me or seems like you ignored the suggestion/discussion under v1 of this patch...
Quoted 1 line
I did not. Please send a patch when you can demonstrate the difference.
You "did not", but you didn't reply there, only sent v2 w/o any mention.
Quoted 2 lines
We are not going to add all the possible destructors unless there is evidence.
There are numbers in the original discussion, you'd have noticed if you did read.
Quoted 6 lines
We only ask to add one more destructor which will help certain perf-critical workloads. Add it to the end of the list, so that it won't hurt your optimization.
"Send a patch" means you're now changing these lines now and then they would be changed once again, why...
Quoted 7 lines
I can not test what you propose.
I can drop this patch instead, and keep it in Google kernels, (we had TCP support for years)
Or... you can send a patch on top of it later.
Sorry, I've been away from the keyboard for a few days. I think it's fair to let us (who are currently working on the xsk improvement) post a simple patch based on the series.
Regarding what you mentioned that 1% is a noisy number, I disagree. The overall numbers are improved, rather than only one or small part of them. I've done a few tests under different servers, so I believe what I've seen. BTW, xdpsock is the test tool that gives a stable number especially when running on the physical machine.
@ Alexander I think I can post that patch with more test numbers and your 'suggested-by' tag included if you have no objection:) Or if you wish you could do it on your own, please feel free to send one then :)
Thanks,
Jason
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
Quoted 13 lines · Jason Xing
Sorry, I've been away from the keyboard for a few days. I think it's fair to let us (who are currently working on the xsk improvement) post a simple patch based on the series.
Regarding what you mentioned that 1% is a noisy number, I disagree. The overall numbers are improved, rather than only one or small part of them. I've done a few tests under different servers, so I believe what I've seen. BTW, xdpsock is the test tool that gives a stable number especially when running on the physical machine.
Quoted 3 lines
@ Alexander I think I can post that patch with more test numbers and your 'suggested-by' tag included if you have no objection:) Or if you wish you could do it on your own, please feel free to send one then :)
The series focus was on something bringing 100% improvement. The 1% figure _was_ noise.
I think you are mistaken on what a "SIgned-off-by: Eric Dumazet <[email protected]>" means.
I am not opposed to a patch that you will support by yourself. I am opposed to you trying to let me take responsibility for something I have no time/desire to support.
I added the indirect call wrapper mostly at the last moment, so that anyone wanting to test my series
like I described (UDP workload) would not have to mention the sock_wfree cost.
Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()
Quoted 25 lines · Eric Dumazet
Quoted 13 lines · Jason Xing
Sorry, I've been away from the keyboard for a few days. I think it's fair to let us (who are currently working on the xsk improvement) post a simple patch based on the series.
Regarding what you mentioned that 1% is a noisy number, I disagree. The overall numbers are improved, rather than only one or small part of them. I've done a few tests under different servers, so I believe what I've seen. BTW, xdpsock is the test tool that gives a stable number especially when running on the physical machine.
Quoted 3 lines
@ Alexander I think I can post that patch with more test numbers and your 'suggested-by' tag included if you have no objection:) Or if you wish you could do it on your own, please feel free to send one then :)
The series focus was on something bringing 100% improvement. The 1% figure _was_ noise.
I think you are mistaken on what a "SIgned-off-by: Eric Dumazet <[email protected]>" means.
Quoted 7 lines
I am not opposed to a patch that you will support by yourself. I am opposed to you trying to let me take responsibility for something I have no time/desire to support.
I added the indirect call wrapper mostly at the last moment, so that anyone wanting to test my series
like I described (UDP workload) would not have to mention thesock_wfreecost.
Eric, I totally understand what you meant and thanks for bringing up this idea :) Agree that one patch should do one thing at a time. It's clean. If people smell something wrong in the future, they can easily bisect and revert. So what I replied was that I decided to add a follow-up patch to only support the xsk scenario.
Thanks,
Jason
PATCH v2 3/6 net/sched: act_mirred: add loop detection
In reply to Eric DumazetCommit 0f022d32c3ec ("net/sched: Fix mirred deadlock on device recursion") added code in the fast path, even when act_mirred is not used.
Prepare its revert by implementing loop detection in act_mirred.
Adds an array of device pointers in struct netdev_xmit.
tcf_mirred_is_act_redirect() can detect if the array already contains the target device.
--- include/linux/netdevice_xmit.h | 9 ++++- net/sched/act_mirred.c | 62 +++++++++++++--------------------- 2 files changed, 31 insertions(+), 40 deletions(-)
include/linux/netdevice_xmit.h +8 -1
@@ -2,6 +2,12 @@ #ifndef _LINUX_NETDEVICE_XMIT_H #define _LINUX_NETDEVICE_XMIT_H +#if IS_ENABLED(CONFIG_NET_ACT_MIRRED) +#define MIRRED_NEST_LIMIT 4 +#endif + +struct net_device; + struct netdev_xmit { u16 recursion; u8 more;
@@ -9,7 +15,8 @@ struct netdev_xmit {u8 skip_txqueue;#endif #if IS_ENABLED (CONFIG_NET_ACT_MIRRED ) -u8 sched_mirred_nest; +u8 sched_mirred_nest; +struct net_device *sched_mirred_dev[MIRRED_NEST_LIMIT ];#endif #if IS_ENABLED (CONFIG_NF_DUP_NETDEV )u8 nf_dup_skb_recursion;
net/sched/act_mirred.c +23 -39
98 lines not on this page · raw message
Signature · 1 line
2.51.0.788.g6d19910ace-goog
Re: [PATCH v2 net-next 3/6] net/sched: act_mirred: add loop detection
Quoted 4 lines · Eric Dumazet
Commit
0f022d32c3ec("net/sched: Fix mirred deadlock on device recursion") added code in the fast path, even whenact_mirredis not used.
Quoted 8 lines
Prepare its revert by implementing loop detection in
act_mirred.Adds an array of device pointers in struct
netdev_xmit.
tcf_mirred_is_act_redirect()can detect if the array already contains the target device.Signed-off-by Eric Dumazet <[email protected]>
Re: [PATCH v2 net-next 3/6] net/sched: act_mirred: add loop detection
In reply to Eric DumazetQuoted 11 lines · Eric Dumazet
Commit
0f022d32c3ec("net/sched: Fix mirred deadlock on device recursion") added code in the fast path, even whenact_mirredis not used.Prepare its revert by implementing loop detection in
act_mirred.Adds an array of device pointers in struct
netdev_xmit.
tcf_mirred_is_act_redirect()can detect if the array already contains the target device.Signed-off-by Eric Dumazet <[email protected]>
PATCH v2 4/6 Revert "net/sched: Fix mirred deadlock on device recursion"
In reply to Eric DumazetThis reverts commits 0f022d32c3eca477fbf79a205243a6123ed0fe11 and 44180feaccf266d9b0b28cc4ceaac019817deb5c.
Prior patch in this series implemented loop detection in act_mirred, we can remove q->owner to save some cycles in the fast path.
--- include/net/sch_generic.h | 1 - net/core/dev.c | 6 ------ net/sched/sch_generic.c | 2 -- 3 files changed, 9 deletions(-)
include/net/sch_generic.h +0 -1
@@ -117,7 +117,6 @@ struct Qdisc {struct qdisc_skb_head q;struct gnet_stats_basic_sync bstats;struct gnet_stats_queue qstats; -int owner;unsigned long state;unsigned long state2;/* must be written under qdisc spinlock */ struct Qdisc *next_sched;
net/core/dev.c +0 -6
@@ -4167,10 +4167,6 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,return rc; } -if (unlikely (READ_ONCE (q->owner) ==smp_processor_id ())) { -kfree_skb_reason (skb,SKB_DROP_REASON_TC_RECLASSIFY_LOOP ); -return NET_XMIT_DROP ; - }/* * Heuristic to force contended enqueues to serialize on a * separate lock before trying to get qdisc main lock.
@@ -4210,9 +4206,7 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,qdisc_run_end (q); rc =NET_XMIT_SUCCESS ; }else { -WRITE_ONCE (q->owner,smp_processor_id ()); rc =dev_qdisc_enqueue (skb, q, &to_free, txq); -WRITE_ONCE (q->owner,-1 );if (qdisc_run_begin (q)) {if (unlikely (contended)) {spin_unlock (&q->busylock);
net/sched/sch_generic.c +0 -2
@@ -679,7 +679,6 @@ struct Qdisc noop_qdisc = { .qlen = 0, .lock = __SPIN_LOCK_UNLOCKED(noop_qdisc.skb_bad_txq.lock), }, - .owner = -1, };EXPORT_SYMBOL (noop_qdisc);
@@ -985,7 +984,6 @@ struct Qdisc *qdisc_alloc(struct netdev_queue *dev_queue, sch->enqueue = ops->enqueue; sch->dequeue = ops->dequeue; sch->dev_queue = dev_queue; - sch->owner =-1 ;netdev_hold (dev, &sch->dev_tracker,GFP_KERNEL );refcount_set (&sch->refcnt,1 );
Signature · 1 line
2.51.0.788.g6d19910ace-goog
Re: [PATCH v2 net-next 4/6] Revert "net/sched: Fix mirred deadlock on device recursion"
Quoted 9 lines · Eric Dumazet
This reverts commits
0f022d32c3eca477fbf79a205243a6123ed0fe11and44180feaccf266d9b0b28cc4ceaac019817deb5c.Prior patch in this series implemented loop detection in
act_mirred, we can removeq->ownerto save some cycles in the fast path.Signed-off-by Eric Dumazet <[email protected]>
Re: [PATCH v2 net-next 4/6] Revert "net/sched: Fix mirred deadlock on device recursion"
In reply to Eric DumazetQuoted 8 lines · Eric Dumazet
This reverts commits
0f022d32c3eca477fbf79a205243a6123ed0fe11and44180feaccf266d9b0b28cc4ceaac019817deb5c.Prior patch in this series implemented loop detection in
act_mirred, we can removeq->ownerto save some cycles in the fast path.Signed-off-by Eric Dumazet <[email protected]>
Re: [PATCH v2 net-next 4/6] Revert "net/sched: Fix mirred deadlock on device recursion"
In reply to Eric DumazetQuoted 8 lines · Eric Dumazet
This reverts commits
0f022d32c3eca477fbf79a205243a6123ed0fe11and44180feaccf266d9b0b28cc4ceaac019817deb5c.Prior patch in this series implemented loop detection in
act_mirred, we can removeq->ownerto save some cycles in the fast path.Signed-off-by Eric Dumazet <[email protected]>
PATCH v2 5/6 net: sched: claim one cache line in Qdisc
In reply to Eric DumazetReplace state2 field with a boolean.
Move it to a hole between qstats and state so that we shrink Qdisc by a full cache line.
--- include/net/sch_generic.h | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-)
include/net/sch_generic.h +7 -11
@@ -41,13 +41,6 @@ enum qdisc_state_t { __QDISC_STATE_DRAINING, }; -enum qdisc_state2_t { -/* Only for !TCQ_F_NOLOCK qdisc. Never access it directly. -* Use qdisc_run_begin/end() or qdisc_is_running() instead. -*/ - __QDISC_STATE2_RUNNING, -}; -#define QDISC_STATE_MISSED BIT(__QDISC_STATE_MISSED)#define QDISC_STATE_DRAINING BIT(__QDISC_STATE_DRAINING)
@@ -117,8 +110,8 @@ struct Qdisc {struct qdisc_skb_head q;struct gnet_stats_basic_sync bstats;struct gnet_stats_queue qstats; +bool running;/* must be written under qdisc spinlock */ unsigned long state; -unsigned long state2;/* must be written under qdisc spinlock */ struct Qdisc *next_sched;struct sk_buff_head skb_bad_txq;
@@ -167,7 +160,7 @@ static inline bool qdisc_is_running(struct Qdisc *qdisc) {if (qdisc->flags &TCQ_F_NOLOCK )return spin_is_locked (&qdisc->seqlock); -return test_bit (__QDISC_STATE2_RUNNING, &qdisc->state2); +return READ_ONCE (qdisc->running); }static inline bool nolock_qdisc_is_empty (const struct Qdisc *qdisc)
@@ -210,7 +203,10 @@ static inline bool qdisc_run_begin(struct Qdisc *qdisc)*/ return spin_trylock (&qdisc->seqlock); } -return !__test_and_set_bit (__QDISC_STATE2_RUNNING, &qdisc->state2); +if (READ_ONCE (qdisc->running)) +return false; +WRITE_ONCE (qdisc->running, true); +return true; }static inline void qdisc_run_end (struct Qdisc *qdisc)
@@ -228,7 +224,7 @@ static inline void qdisc_run_end(struct Qdisc *qdisc) &qdisc->state)))__netif_schedule (qdisc); }else { -__clear_bit (__QDISC_STATE2_RUNNING, &qdisc->state2); +WRITE_ONCE (qdisc->running, false); } }
Signature · 1 line
2.51.0.788.g6d19910ace-goog
Re: [PATCH v2 net-next 5/6] net: sched: claim one cache line in Qdisc
Quoted 7 lines · Eric Dumazet
Replace state2 field with a boolean.
Move it to a hole between qstats and state so that we shrink Qdisc by a full cache line.
Signed-off-by Eric Dumazet <[email protected]>
Re: [PATCH v2 net-next 5/6] net: sched: claim one cache line in Qdisc
In reply to Eric DumazetQuoted 6 lines · Eric Dumazet
Replace state2 field with a boolean.
Move it to a hole between qstats and state so that we shrink Qdisc by a full cache line.
Signed-off-by Eric Dumazet <[email protected]>
PATCH v2 6/6 net: dev_queue_xmit() llist adoption
In reply to Eric DumazetRemove busylock spinlock and use a lockless list (llist) to reduce spinlock contention to the minimum.
Idea is that only one cpu might spin on the qdisc spinlock, while others simply add their skb in the llist.
After this patch, we get a 300 % improvement on heavy TX workloads.
- Sending twice the number of packets per second.
- While consuming 50 % less cycles.
Note that this also allows in the future to submit batches to various qdisc->enqueue() methods.
Tested:
- Dual Intel(R) Xeon(R) 6985P-C (480 hyper threads).
- 100Gbit NIC, 30 TX queues with FQ packet scheduler.
- echo 64 >/sys/kernel/slab/skbuff_small_head/cpu_partial (avoid contention in mm)
- 240 concurrent "netperf -t UDP_STREAM -- -m 120 -n"
Before:
16 Mpps (41 Mpps if each thread is pinned to a different cpu)
vmstat 2 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
243 0 0 2368988672 51036 1100852 0 0 146 1 242 60 0 9 91 0 0 244 0 0 2368988672 51036 1100852 0 0 536 10 487745 14718 0 52 48 0 0 244 0 0 2368988672 51036 1100852 0 0 512 0 503067 46033 0 52 48 0 0 244 0 0 2368988672 51036 1100852 0 0 512 0 494807 12107 0 52 48 0 0 244 0 0 2368988672 51036 1100852 0 0 702 26 492845 10110 0 52 48 0 0
Lock contention (1 second sample taken on 8 cores) perf lock record -C0-7 sleep 1; perf lock contention
contended total wait max wait avg wait type caller
442111 6.79 s 162.47 ms 15.35 us spinlock dev_hard_start_xmit+0xcd
5961 9.57 ms 8.12 us 1.60 us spinlock __dev_queue_xmit+0x3a0
244 560.63 us 7.63 us 2.30 us spinlock do_softirq+0x5b
13 25.09 us 3.21 us 1.93 us spinlock net_tx_action+0xf8If netperf threads are pinned, spinlock stress is very high. perf lock record -C0-7 sleep 1; perf lock contention
contended total wait max wait avg wait type caller
964508 7.10 s 147.25 ms 7.36 us spinlock dev_hard_start_xmit+0xcd
201 268.05 us 4.65 us 1.33 us spinlock __dev_queue_xmit+0x3a0
12 26.05 us 3.84 us 2.17 us spinlock do_softirq+0x5b@__dev_queue_xmit_ns:
[256, 512) 21 | | [512, 1K) 631 | | [1K, 2K) 27328 |@ | [2K, 4K) 265392 |@@@@@@@@@@@@@@@@ | [4K, 8K) 417543 |@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8K, 16K) 826292 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16K, 32K) 733822 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [32K, 64K) 19055 |@ | [64K, 128K) 17240 |@ | [128K, 256K) 25633 |@ | [256K, 512K) 4 | |
After:
29 Mpps (57 Mpps if each thread is pinned to a different cpu)
vmstat 2 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st 78 0 0 2369573632 32896 1350988 0 0 22 0 331 254 0 8 92 0 0 75 0 0 2369573632 32896 1350988 0 0 22 50 425713 280199 0 23 76 0 0 104 0 0 2369573632 32896 1350988 0 0 290 0 430238 298247 0 23 76 0 0 86 0 0 2369573632 32896 1350988 0 0 132 0 428019 291865 0 24 76 0 0 90 0 0 2369573632 32896 1350988 0 0 502 0 422498 278672 0 23 76 0 0
perf lock record -C0-7 sleep 1; perf lock contention
contended total wait max wait avg wait type caller
2524 116.15 ms 486.61 us 46.02 us spinlock __dev_queue_xmit+0x55b
5821 107.18 ms 371.67 us 18.41 us spinlock dev_hard_start_xmit+0xcd
2377 9.73 ms 35.86 us 4.09 us spinlock ___slab_alloc+0x4e0
923 5.74 ms 20.91 us 6.22 us spinlock ___slab_alloc+0x5c9
121 3.42 ms 193.05 us 28.24 us spinlock net_tx_action+0xf8
6 564.33 us 167.60 us 94.05 us spinlock do_softirq+0x5bIf netperf threads are pinned (~54 Mpps) perf lock record -C0-7 sleep 1; perf lock contention
32907 316.98 ms 195.98 us 9.63 us spinlock dev_hard_start_xmit+0xcd
4507 61.83 ms 212.73 us 13.72 us spinlock __dev_queue_xmit+0x554
2781 23.53 ms 40.03 us 8.46 us spinlock ___slab_alloc+0x5c9
3554 18.94 ms 34.69 us 5.33 us spinlock ___slab_alloc+0x4e0
233 9.09 ms 215.70 us 38.99 us spinlock do_softirq+0x5b
153 930.66 us 48.67 us 6.08 us spinlock net_tx_action+0xfd
84 331.10 us 14.22 us 3.94 us spinlock ___slab_alloc+0x5c9
140 323.71 us 9.94 us 2.31 us spinlock ___slab_alloc+0x4e0@__dev_queue_xmit_ns:
[128, 256) 1539830 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [256, 512) 2299558 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [512, 1K) 483936 |@@@@@@@@@@ | [1K, 2K) 265345 |@@@@@@ | [2K, 4K) 145463 |@@@ | [4K, 8K) 54571 |@ | [8K, 16K) 10270 | | [16K, 32K) 9385 | | [32K, 64K) 7749 | | [64K, 128K) 26799 | | [128K, 256K) 2665 | | [256K, 512K) 665 | |
--- include/net/sch_generic.h | 4 +- net/core/dev.c | 91 ++++++++++++++++++++++++--------------- net/sched/sch_generic.c | 5 --- 3 files changed, 59 insertions(+), 41 deletions(-)
include/net/sch_generic.h +3 -1
@@ -115,7 +115,9 @@ struct Qdisc {struct Qdisc *next_sched;struct sk_buff_head skb_bad_txq; -spinlock_t busylock____cacheline_aligned_in_smp ; +atomic_long_t defer_count____cacheline_aligned_in_smp ; +struct llist_head defer_list; +spinlock_t seqlock;struct rcu_head rcu;
net/core/dev.c +56 -35
126 lines not on this page · raw message
net/sched/sch_generic.c +0 -5
@@ -666,7 +666,6 @@ struct Qdisc noop_qdisc = { .ops = &noop_qdisc_ops, .q.lock = __SPIN_LOCK_UNLOCKED(noop_qdisc.q.lock), .dev_queue = &noop_netdev_queue, - .busylock = __SPIN_LOCK_UNLOCKED(noop_qdisc.busylock), .gso_skb = { .next = (struct sk_buff *)&noop_qdisc.gso_skb, .prev = (struct sk_buff *)&noop_qdisc.gso_skb,
@@ -970,10 +969,6 @@ struct Qdisc *qdisc_alloc(struct netdev_queue *dev_queue, } } -spin_lock_init (&sch->busylock); -lockdep_set_class (&sch->busylock, - dev->qdisc_tx_busylock ?: &qdisc_tx_busylock); -/* seqlock has the same scope of busylock, for NOLOCK qdisc */ spin_lock_init (&sch->seqlock); lockdep_set_class(&sch->seqlock,
Signature · 1 line
2.51.0.788.g6d19910ace-goog
Re: [PATCH v2 net-next 6/6] net: dev_queue_xmit() llist adoption
Quoted 112 lines · Eric Dumazet
Remove busylock spinlock and use a lockless list (llist) to reduce spinlock contention to the minimum.
Idea is that only one cpu might spin on the qdisc spinlock, while others simply add their skb in the llist.
After this patch, we get a 300 % improvement on heavy TX workloads.
- Sending twice the number of packets per second.
- While consuming 50 % less cycles.Note that this also allows in the future to submit batches to various
qdisc->enqueue()methods.Tested:
- Dual Intel(R) Xeon(R) 6985P-C (480 hyper threads).
- 100Gbit NIC, 30 TX queues with FQ packet scheduler.
- echo 64 >/sys/kernel/slab/skbuff_small_head/cpu_partial (avoid contention in mm)
- 240 concurrent "netperf -tUDP_STREAM-- -m 120 -n"Before:
16 Mpps (41 Mpps if each thread is pinned to a different cpu)
vmstat 2 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st243 0 0 2368988672 51036 1100852 0 0 146 1 242 60 0 9 91 0 0 244 0 0 2368988672 51036 1100852 0 0 536 10 487745 14718 0 52 48 0 0 244 0 0 2368988672 51036 1100852 0 0 512 0 503067 46033 0 52 48 0 0 244 0 0 2368988672 51036 1100852 0 0 512 0 494807 12107 0 52 48 0 0 244 0 0 2368988672 51036 1100852 0 0 702 26 492845 10110 0 52 48 0 0Lock contention (1 second sample taken on 8 cores) perf lock record -C0-7 sleep 1; perf lock contention
contended total wait max wait avg wait type caller442111 6.79 s 162.47 ms 15.35 us spinlock dev_hard_start_xmit+0xcd 5961 9.57 ms 8.12 us 1.60 us spinlock __dev_queue_xmit+0x3a0 244 560.63 us 7.63 us 2.30 us spinlock do_softirq+0x5b 13 25.09 us 3.21 us 1.93 us spinlock net_tx_action+0xf8If netperf threads are pinned, spinlock stress is very high. perf lock record -C0-7 sleep 1; perf lock contention
contended total wait max wait avg wait type caller964508 7.10 s 147.25 ms 7.36 us spinlock dev_hard_start_xmit+0xcd 201 268.05 us 4.65 us 1.33 us spinlock __dev_queue_xmit+0x3a0 12 26.05 us 3.84 us 2.17 us spinlock do_softirq+0x5b@__dev_queue_xmit_ns:
[256, 512) 21 | | [512, 1K) 631 | | [1K, 2K) 27328 |@ | [2K, 4K) 265392 |@@@@@@@@@@@@@@@@ | [4K, 8K) 417543 |@@@@@@@@@@@@@@@@@@@@@@@@@@ | [8K, 16K) 826292 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [16K, 32K) 733822 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [32K, 64K) 19055 |@ | [64K, 128K) 17240 |@ | [128K, 256K) 25633 |@ | [256K, 512K) 4 | |After:
29 Mpps (57 Mpps if each thread is pinned to a different cpu)
vmstat 2 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----r b swpd free buff cache si so bi bo in cs us sy id wa st 78 0 0 2369573632 32896 1350988 0 0 22 0 331 254 0 8 92 0 0 75 0 0 2369573632 32896 1350988 0 0 22 50 425713 280199 0 23 76 0 0 104 0 0 2369573632 32896 1350988 0 0 290 0 430238 298247 0 23 76 0 0 86 0 0 2369573632 32896 1350988 0 0 132 0 428019 291865 0 24 76 0 0 90 0 0 2369573632 32896 1350988 0 0 502 0 422498 278672 0 23 76 0 0perf lock record -C0-7 sleep 1; perf lock contention
contended total wait max wait avg wait type caller2524 116.15 ms 486.61 us 46.02 us spinlock __dev_queue_xmit+0x55b 5821 107.18 ms 371.67 us 18.41 us spinlock dev_hard_start_xmit+0xcd 2377 9.73 ms 35.86 us 4.09 us spinlock ___slab_alloc+0x4e0 923 5.74 ms 20.91 us 6.22 us spinlock ___slab_alloc+0x5c9 121 3.42 ms 193.05 us 28.24 us spinlock net_tx_action+0xf8 6 564.33 us 167.60 us 94.05 us spinlock do_softirq+0x5bIf netperf threads are pinned (~54 Mpps) perf lock record -C0-7 sleep 1; perf lock contention
32907 316.98 ms 195.98 us 9.63 us spinlock dev_hard_start_xmit+0xcd 4507 61.83 ms 212.73 us 13.72 us spinlock __dev_queue_xmit+0x554 2781 23.53 ms 40.03 us 8.46 us spinlock ___slab_alloc+0x5c9 3554 18.94 ms 34.69 us 5.33 us spinlock ___slab_alloc+0x4e0 233 9.09 ms 215.70 us 38.99 us spinlock do_softirq+0x5b 153 930.66 us 48.67 us 6.08 us spinlock net_tx_action+0xfd 84 331.10 us 14.22 us 3.94 us spinlock ___slab_alloc+0x5c9 140 323.71 us 9.94 us 2.31 us spinlock ___slab_alloc+0x4e0@__dev_queue_xmit_ns:
[128, 256) 1539830 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | [256, 512) 2299558 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [512, 1K) 483936 |@@@@@@@@@@ | [1K, 2K) 265345 |@@@@@@ | [2K, 4K) 145463 |@@@ | [4K, 8K) 54571 |@ | [8K, 16K) 10270 | | [16K, 32K) 9385 | | [32K, 64K) 7749 | | [64K, 128K) 26799 | | [128K, 256K) 2665 | | [256K, 512K) 665 | |
Quoted 2 lines
Signed-off-by Eric Dumazet <[email protected]>Reviewed-by Toke Høiland-Jørgensen <[email protected]>
Thanks for big improvement!
Quoted 11 lines · Eric Dumazet
In this series, I replace the busylock spinlock we have in
__dev_queue_xmit()and use lockless list (llist) to reduce spinlock contention to the minimum.Idea is that only one cpu might spin on the qdisc spinlock, while others simply add their skb in the llist.
After this series, we get a 300 % (4x) improvement on heavy TX workloads, sending twice the number of packets per second, for half the cpu cycles.
Not important but i am curious: you didnt mention what NIC this was in the commit messages ;->
For the patchset, I have done testing with existing tdc tests and no regression..
It does inspire new things when time becomes available.... so will be doing more testing and likely small extensions etc.
So:
(For the tc bits, since the majority of the code touches tc related stuff)
cheers,
jamal
Quoted 25 lines
v2: deflake tcp_user_timeout_user-timeout-probe.pkt.
Ability to return a different code thanNET_XMIT_SUCCESSwhen__dev_xmit_skb()has a single skb to send.Eric Dumazet (6):
selftests/net: packetdrill: unflake
tcp_user_timeout_user-timeout-probe.pkt
net: add add indirect call wrapper inskb_release_head_state()net/sched:act_mirred: add loop detection
Revert "net/sched: Fix mirred deadlock on device recursion"
net: sched: claim one cache line in Qdisc
net:dev_queue_xmit()llist adoptioninclude/linux/netdevice_xmit.h | 9 +- include/net/sch_generic.h | 23 ++--- net/core/dev.c | 97 +++++++++++-------- net/core/skbuff.c | 11 ++- net/sched/act_mirred.c | 62 +++++------- net/sched/sch_generic.c | 7 -- .../tcp_user_timeout_user-timeout-probe.pkt | 6 +- 7 files changed, 111 insertions(+), 104 deletions(-)--
2.51.0.788.g6d19910ace-goog
Quoted 10 lines · Jamal Hadi Salim
Quoted 8 lines · Eric Dumazet
In this series, I replace the busylock spinlock we have in
__dev_queue_xmit()and use lockless list (llist) to reduce spinlock contention to the minimum.Idea is that only one cpu might spin on the qdisc spinlock, while others simply add their skb in the llist.
Quoted 6 lines
Quoted 3 lines
After this series, we get a 300 % (4x) improvement on heavy TX workloads, sending twice the number of packets per second, for half the cpu cycles.
Not important but i am curious: you didnt mention what NIC this was in the commit messages ;->
I have used two NIC : IDPF (200Gbit), and GQ (100Gbit Google NIC) (Usually with 32 TX queues)
And a variety of platforms, up to 512 cores sharing these 32 TX queues.
Quoted 8 lines
For the patchset, I have done testing with existing tdc tests and no regression..
It does inspire new things when time becomes available.... so will be doing more testing and likely small extensions etc.
So:Tested-by Jamal Hadi Salim <[email protected]>Acked-by Jamal Hadi Salim <[email protected]>
Thanks Jamal !
I also have this idea:
cpus serving NIC interrupts and specifically TX completions are often trapped in also restarting a busy qdisc (because it was stopped by BQL or the driver's own flow control).
We can do better:
1) In the TX completion loop, collect the skbs and do not free them immediately.
2) Store them in a private list and sum their skb->len while doing so.
3) Then call netdev_tx_completed_queue() call and netif_tx_wake_queue().
If the queue was stopped, this might add the qdisc in our per-cpu private list (sd->output_queue), raising NET_TX_SOFTIRQ (no immediate action because napi poll runs while BH are blocked)
4) Then, take care of all dev_consume_skb_any().
Quite often freeing these skbs can take a lot of time, because of mm contention, and other false sharing, or expensive skb->destructor like TCP.
5) By the time net_tx_action() finally runs, perhaps another cpu saw the queue being in XON state and was able to push more packets to the queue.
This means net_tx_action() might have nothing to do, saving precious cycles.
We should add extra logic in net_tx_action() to not even grab qdisc_lock() spinlock at all.
My thinking is to add back a sequence (to replace q->running boolean), and store a snapshot of this sequence every time we restart a queue. -> net_tx_action can compare the sequence against the last snapshot.
Quoted 31 lines
(For the tc bits, since the majority of the code touches tc related stuff)
cheers,
jamalQuoted 25 lines
v2: deflake tcp_user_timeout_user-timeout-probe.pkt.
Ability to return a different code thanNET_XMIT_SUCCESSwhen__dev_xmit_skb()has a single skb to send.Eric Dumazet (6):
selftests/net: packetdrill: unflake
tcp_user_timeout_user-timeout-probe.pkt
net: add add indirect call wrapper inskb_release_head_state()net/sched:act_mirred: add loop detection
Revert "net/sched: Fix mirred deadlock on device recursion"
net: sched: claim one cache line in Qdisc
net:dev_queue_xmit()llist adoptioninclude/linux/netdevice_xmit.h | 9 +- include/net/sch_generic.h | 23 ++--- net/core/dev.c | 97 +++++++++++-------- net/core/skbuff.c | 11 ++- net/sched/act_mirred.c | 62 +++++------- net/sched/sch_generic.c | 7 -- .../tcp_user_timeout_user-timeout-probe.pkt | 6 +- 7 files changed, 111 insertions(+), 104 deletions(-)--
2.51.0.788.g6d19910ace-goog
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 8 lines · On Tue, 14 Oct 2025 17:19:01 +0000 you wrote:
In this series, I replace the busylock spinlock we have in
__dev_queue_xmit()and use lockless list (llist) to reduce spinlock contention to the minimum.Idea is that only one cpu might spin on the qdisc spinlock, while others simply add their skb in the llist.
[...]
Here is the summary with links:
- [v2,net-next,1/6] selftests/net: packetdrill: unflake tcp_user_timeout_user-timeout-probe.pkt
https://git.kernel.org/netdev/net-next/c/56cef47c28dc - [v2,net-next,2/6] net: add add indirect call wrapper in skb_release_head_state() https://git.kernel.org/netdev/net-next/c/5b2b7dec05f3 - [v2,net-next,3/6] net/sched: act_mirred: add loop detection https://git.kernel.org/netdev/net-next/c/fe946a751d9b - [v2,net-next,4/6] Revert "net/sched: Fix mirred deadlock on device recursion" https://git.kernel.org/netdev/net-next/c/178ca30889a1 - [v2,net-next,5/6] net: sched: claim one cache line in Qdisc https://git.kernel.org/netdev/net-next/c/526f5fb112f7 - [v2,net-next,6/6] net: dev_queue_xmit() llist adoption https://git.kernel.org/netdev/net-next/c/100dfa74cad9
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html