Skip to contentSkip to conversation

net: optimize TX throughput and efficiency

PATCH Applied Eric Dumazet 8 participants 33 messages Oct 14, 2025 – Oct 20, 2025

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v2 cover net: optimize TX throughput and efficiency

In this series, I replace the busylock spinlock we have in __dev_queue_xmit() and use lockless list (llist) to reduce spinlock contention to the minimum.

Idea is that only one cpu might spin on the qdisc spinlock, while others simply add their skb in the llist.

After this series, we get a 300 % (4x) improvement on heavy TX workloads, sending twice the number of packets per second, for half the cpu cycles.

v2: deflake tcp_user_timeout_user-timeout-probe.pkt.
Ability to return a different code than NET_XMIT_SUCCESS when __dev_xmit_skb() has a single skb to send.

Eric Dumazet (6):
selftests/net: packetdrill: unflake
tcp_user_timeout_user-timeout-probe.pkt
net: add add indirect call wrapper in skb_release_head_state() net/sched: act_mirred: add loop detection
Revert "net/sched: Fix mirred deadlock on device recursion"
net: sched: claim one cache line in Qdisc
net: dev_queue_xmit() llist adoption

 include/linux/netdevice_xmit.h                |  9 +-
 include/net/sch_generic.h                     | 23 ++---
 net/core/dev.c                                | 97 +++++++++++--------
 net/core/skbuff.c                             | 11 ++-
 net/sched/act_mirred.c                        | 62 +++++-------
 net/sched/sch_generic.c                       |  7 --
 .../tcp_user_timeout_user-timeout-probe.pkt   |  6 +-
 7 files changed, 111 insertions(+), 104 deletions(-)
Signature · 1 line
2.51.0.788.g6d19910ace-goog
#1 lore raw

PATCH v2 1/6 selftests/net: packetdrill: unflake tcp_user_timeout_user-timeout-probe.pkt

This test fails the first time I am running it after a fresh virtme-ng boot.

tcp_user_timeout_user-timeout-probe.pkt:33: runtime error in write call: Expected result -1 but got 24 with errno 2 (No such file or directory)

Tweaks the timings a bit, to reduce flakiness.

Cc Soham Chakradeo <[email protected]>
Cc Willem de Bruijn <[email protected]>
---
 .../net/packetdrill/tcp_user_timeout_user-timeout-probe.pkt | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/testing/selftests/net/packetdrill/tcp_user_timeout_user-timeout-probe.pkt +4 -2
index 183051ba0cae..6882b8240a8a 100644
@@ -23,14 +23,16 @@
 
 // install a qdisc dropping all packets
    +0 `tc qdisc delete dev tun0 root 2>/dev/null ; tc qdisc add dev tun0 root pfifo limit 0`
+
    +0 write(4, ..., 24) = 24
    // When qdisc is congested we retry every 500ms
    // (TCP_RESOURCE_PROBE_INTERVAL) and therefore
    // we retry 6 times before hitting 3s timeout.
    // First verify that the connection is alive:
-+3.250 write(4, ..., 24) = 24
++3 write(4, ..., 24) = 24
+
    // Now verify that shortly after that the socket is dead:
- +.100 write(4, ..., 24) = -1 ETIMEDOUT (Connection timed out)
++1 write(4, ..., 24) = -1 ETIMEDOUT (Connection timed out)
 
    +0 %{ assert tcpi_probes == 6, tcpi_probes; \
          assert tcpi_backoff == 0, tcpi_backoff }%
Signature · 1 line
2.51.0.788.g6d19910ace-goog
Reviewed-by on v2 1/6
#2 lore raw

Re: [PATCH v2 net-next 1/6] selftests/net: packetdrill: unflake tcp_user_timeout_user-timeout-probe.pkt

Quoted 8 lines · Eric Dumazet

This test fails the first time I am running it after a fresh virtme-ng boot.

tcp_user_timeout_user-timeout-probe.pkt:33: runtime error in write call: Expected result -1 but got 24 with errno 2 (No such file or directory)

Tweaks the timings a bit, to reduce flakiness.

#3 lore raw

PATCH v2 2/6 net: add add indirect call wrapper in skb_release_head_state()

In reply to Eric Dumazet

While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in sock_wfree() due to CONFIG_MITIGATION_RETPOLINE=y.

Take care of TCP and UDP TX destructors and use INDIRECT_CALL_3() macro.

---
 net/core/skbuff.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
net/core/skbuff.c +10 -1
index bc12790017b0..692e3a70e75e 100644
@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)
 	skb_dst_drop(skb);
 	if (skb->destructor) {
 		DEBUG_NET_WARN_ON_ONCE(in_hardirq());
-		skb->destructor(skb);
+#ifdef CONFIG_INET
+		INDIRECT_CALL_3(skb->destructor,
+				tcp_wfree, __sock_wfree, sock_wfree,
+				skb);
+#else
+		INDIRECT_CALL_1(skb->destructor,
+				sock_wfree,
+				skb);
+
+#endif
 	}
 #if IS_ENABLED(CONFIG_NF_CONNTRACK)
 	nf_conntrack_put(skb_nfct(skb));
Signature · 1 line
2.51.0.788.g6d19910ace-goog
Reviewed-by on v2 2/6
#4 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

Quoted 9 lines · Eric Dumazet

While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in sock_wfree() due to CONFIG_MITIGATION_RETPOLINE=y.

Take care of TCP and UDP TX destructors and use INDIRECT_CALL_3() macro.

Reviewed-by on v2 2/6
#5 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

In reply to Eric Dumazet
Quoted 8 lines · Eric Dumazet

While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in sock_wfree() due to CONFIG_MITIGATION_RETPOLINE=y.

Take care of TCP and UDP TX destructors and use INDIRECT_CALL_3() macro.

#6 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Tue, 14 Oct 2025 17:19:03 +0000

Quoted 21 lines

While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in sock_wfree() due to CONFIG_MITIGATION_RETPOLINE=y.

Take care of TCP and UDP TX destructors and use INDIRECT_CALL_3() macro.

---
 net/core/skbuff.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
net/core/skbuff.c +0 -1
index bc12790017b0..692e3a70e75e 100644
@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)
 	skb_dst_drop(skb);
 	if (skb->destructor) {
 		DEBUG_NET_WARN_ON_ONCE(in_hardirq());
-		skb->destructor(skb);

net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1148

Quoted 10 lines
net/core/skbuff.c +10 -0
+#ifdef CONFIG_INET
+		INDIRECT_CALL_3(skb->destructor,
+				tcp_wfree, __sock_wfree, sock_wfree,
+				skb);
+#else
+		INDIRECT_CALL_1(skb->destructor,
+				sock_wfree,
+				skb);
+
+#endif

Is it just me or seems like you ignored the suggestion/discussion under v1 of this patch...

net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1151

Quoted 3 lines
net/core/skbuff.c +0 -0
 	}
 #if IS_ENABLED(CONFIG_NF_CONNTRACK)
 	nf_conntrack_put(skb_nfct(skb));

Thanks,
Olek

#7 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

Quoted 30 lines · Alexander Lobakin

From: Eric Dumazet <[email protected]>
Date: Tue, 14 Oct 2025 17:19:03 +0000

Quoted 26 lines

While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in sock_wfree() due to CONFIG_MITIGATION_RETPOLINE=y.

Take care of TCP and UDP TX destructors and use INDIRECT_CALL_3() macro.

---
 net/core/skbuff.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
net/core/skbuff.c +5 -1
index bc12790017b0..692e3a70e75e 100644
@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)
      skb_dst_drop(skb);
      if (skb->destructor) {
              DEBUG_NET_WARN_ON_ONCE(in_hardirq());
-             skb->destructor(skb);
+#ifdef CONFIG_INET
+             INDIRECT_CALL_3(skb->destructor,
+                             tcp_wfree, __sock_wfree, sock_wfree,
+                             skb);
+#else
Quoted 9 lines

net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1148

Quoted 5 lines
net/core/skbuff.c +5 -0
+             INDIRECT_CALL_1(skb->destructor,
+                             sock_wfree,
+                             skb);
+
+#endif

Is it just me or seems like you ignored the suggestion/discussion under v1 of this patch...

I did not. Please send a patch when you can demonstrate the difference.

We are not going to add all the possible destructors unless there is evidence.

#8 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

From: Eric Dumazet <[email protected]>
Date: Wed, 15 Oct 2025 05:16:05 -0700

Quoted 42 lines
Quoted 30 lines · Alexander Lobakin

From: Eric Dumazet <[email protected]>
Date: Tue, 14 Oct 2025 17:19:03 +0000

Quoted 26 lines

While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in sock_wfree() due to CONFIG_MITIGATION_RETPOLINE=y.

Take care of TCP and UDP TX destructors and use INDIRECT_CALL_3() macro.

---
 net/core/skbuff.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
net/core/skbuff.c +5 -1
index bc12790017b0..692e3a70e75e 100644
@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)
      skb_dst_drop(skb);
      if (skb->destructor) {
              DEBUG_NET_WARN_ON_ONCE(in_hardirq());
-             skb->destructor(skb);
+#ifdef CONFIG_INET
+             INDIRECT_CALL_3(skb->destructor,
+                             tcp_wfree, __sock_wfree, sock_wfree,
+                             skb);
+#else
Quoted 9 lines

net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1148

Quoted 5 lines
net/core/skbuff.c +5 -0
+             INDIRECT_CALL_1(skb->destructor,
+                             sock_wfree,
+                             skb);
+
+#endif

Is it just me or seems like you ignored the suggestion/discussion under v1 of this patch...

Quoted 1 line

I did not. Please send a patch when you can demonstrate the difference.

You "did not", but you didn't reply there, only sent v2 w/o any mention.

Quoted 2 lines

We are not going to add all the possible destructors unless there is evidence.

There are numbers in the original discussion, you'd have noticed if you did read.

We only ask to add one more destructor which will help certain perf-critical workloads. Add it to the end of the list, so that it won't hurt your optimization.

"Send a patch" means you're now changing these lines now and then they would be changed once again, why...

Thanks,
Olek

#9 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

Quoted 56 lines · Alexander Lobakin

From: Eric Dumazet <[email protected]>
Date: Wed, 15 Oct 2025 05:16:05 -0700

Quoted 42 lines
Quoted 30 lines · Alexander Lobakin

From: Eric Dumazet <[email protected]>
Date: Tue, 14 Oct 2025 17:19:03 +0000

Quoted 26 lines

While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in sock_wfree() due to CONFIG_MITIGATION_RETPOLINE=y.

Take care of TCP and UDP TX destructors and use INDIRECT_CALL_3() macro.

---
 net/core/skbuff.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
net/core/skbuff.c +5 -1
index bc12790017b0..692e3a70e75e 100644
@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)
      skb_dst_drop(skb);
      if (skb->destructor) {
              DEBUG_NET_WARN_ON_ONCE(in_hardirq());
-             skb->destructor(skb);
+#ifdef CONFIG_INET
+             INDIRECT_CALL_3(skb->destructor,
+                             tcp_wfree, __sock_wfree, sock_wfree,
+                             skb);
+#else
Quoted 9 lines

net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1148

Quoted 5 lines
net/core/skbuff.c +5 -0
+             INDIRECT_CALL_1(skb->destructor,
+                             sock_wfree,
+                             skb);
+
+#endif

Is it just me or seems like you ignored the suggestion/discussion under v1 of this patch...

Quoted 1 line

I did not. Please send a patch when you can demonstrate the difference.

You "did not", but you didn't reply there, only sent v2 w/o any mention.

Quoted 2 lines

We are not going to add all the possible destructors unless there is evidence.

There are numbers in the original discussion, you'd have noticed if you did read.

Quoted 6 lines

We only ask to add one more destructor which will help certain perf-critical workloads. Add it to the end of the list, so that it won't hurt your optimization.

"Send a patch" means you're now changing these lines now and then they would be changed once again, why...

I can not test what you propose.

I can drop this patch instead, and keep it in Google kernels, (we had TCP support for years)

Or... you can send a patch on top of it later.

#10 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

Quoted 8 lines · Eric Dumazet

I can not test what you propose.

I can drop this patch instead, and keep it in Google kernels, (we had TCP support for years)

Or... you can send a patch on top of it later.

To be very clear : my Signed-off-by: means that I have strong confidence on what I tested.

#11 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Wed, 15 Oct 2025 05:46:27 -0700

Quoted 58 lines
Quoted 56 lines · Alexander Lobakin

From: Eric Dumazet <[email protected]>
Date: Wed, 15 Oct 2025 05:16:05 -0700

Quoted 42 lines
Quoted 30 lines · Alexander Lobakin

From: Eric Dumazet <[email protected]>
Date: Tue, 14 Oct 2025 17:19:03 +0000

Quoted 26 lines

While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in sock_wfree() due to CONFIG_MITIGATION_RETPOLINE=y.

Take care of TCP and UDP TX destructors and use INDIRECT_CALL_3() macro.

---
 net/core/skbuff.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
net/core/skbuff.c +5 -1
index bc12790017b0..692e3a70e75e 100644
@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)
      skb_dst_drop(skb);
      if (skb->destructor) {
              DEBUG_NET_WARN_ON_ONCE(in_hardirq());
-             skb->destructor(skb);
+#ifdef CONFIG_INET
+             INDIRECT_CALL_3(skb->destructor,
+                             tcp_wfree, __sock_wfree, sock_wfree,
+                             skb);
+#else
Quoted 9 lines

net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1148

Quoted 5 lines
net/core/skbuff.c +5 -0
+             INDIRECT_CALL_1(skb->destructor,
+                             sock_wfree,
+                             skb);
+
+#endif

Is it just me or seems like you ignored the suggestion/discussion under v1 of this patch...

Quoted 1 line

I did not. Please send a patch when you can demonstrate the difference.

You "did not", but you didn't reply there, only sent v2 w/o any mention.

Quoted 2 lines

We are not going to add all the possible destructors unless there is evidence.

There are numbers in the original discussion, you'd have noticed if you did read.

Quoted 8 lines
Quoted 6 lines

We only ask to add one more destructor which will help certain perf-critical workloads. Add it to the end of the list, so that it won't hurt your optimization.

"Send a patch" means you're now changing these lines now and then they would be changed once again, why...

I can not test what you propose.

You asked *me* to show the difference, in the orig discussion there's a patch, there are tests and there is difference... :D

Quoted 3 lines

I can drop this patch instead, and keep it in Google kernels, (we had TCP support for years)

Ok, enough, leave this one as it is, we'll send the XSk bit ourselves.

Quoted 2 lines

Or... you can send a patch on top of it later.

Re "my Signed-off-by means I have strong confidence" -- sometimes we also have Tested-by from other folks and it's never been a problem, hey we're the community.

Thanks,
Olek

#12 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

Quoted 2 lines · Alexander Lobakin

You asked *me* to show the difference, in the orig discussion there's a patch, there are tests and there is difference... :D

I am afraid I have not seen this.

The only thing I found was :

<quote>
Not sure, but maybe we could add generic XSk skb destructor here as well? Or it's not that important as generic XSk is not the best way to use XDP sockets?

Maciej, what do you think?
</quote>

No numbers.

Quoted 16 lines
Quoted 4 lines

I can drop this patch instead, and keep it in Google kernels, (we had TCP support for years)

Ok, enough, leave this one as it is, we'll send the XSk bit ourselves.

Quoted 2 lines

Or... you can send a patch on top of it later.

Re "my Signed-off-by means I have strong confidence" -- sometimes we also have Tested-by from other folks and it's never been a problem, hey we're the community.

Thanks,
Olek

#13 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

From: Eric Dumazet <[email protected]>
Date: Wed, 15 Oct 2025 06:01:40 -0700

Quoted 15 lines
Quoted 2 lines · Alexander Lobakin

You asked *me* to show the difference, in the orig discussion there's a patch, there are tests and there is difference... :D

I am afraid I have not seen this.

The only thing I found was :

<quote>
Not sure, but maybe we could add generic XSk skb destructor here as well? Or it's not that important as generic XSk is not the best way to use XDP sockets?

Quoted 4 lines

Maciej, what do you think?
</quote>

No numbers.

From [0]:

From: Jason Xing <[email protected]>
Date: Thu, 9 Oct 2025 16:37:56 +0800

Quoted 3 lines
Quoted 1 line · Maciej Fijalkowski

[...]

Quoted 22 lines
Quoted 2 lines
Quoted 2 lines

Not sure, but maybe we could add generic XSk skb destructor here as well?

I added the following snippet[1] and only saw a stable ~1% improvement when sending 64 size packets with xdpsock.

I'm not so sure it deserves a follow-up patch to Eric's series. Better than nothing? Any ideas on this one?

[1]
INDIRECT_CALL_4(skb->destructor, tcp_wfree, __sock_wfree, sock_wfree, xsk_destruct_skb, skb);

Quoted 2 lines
Quoted 2 lines

Or it's not that important as generic XSk is not the best way to
use XDP sockets?

Yes, it surely matters. At least, virtio_net and veth need this copy mode. And I've been working on batch xmit to ramp up the generic path.

Quoted 3 lines
Quoted 2 lines

Maciej, what do you think?

Quoted 7 lines
Quoted 2 lines

I would appreciate it as there has been various attempts to optmize xsk generic xmit path.

So do I!

Thanks,
Jason

[0]
https://lore.kernel.org/netdev/CAL+tcoBN9puWX-sTGvTiBN0Hg5oXKR3mjv783YXeR4Bsovuxkw@mail.gmail.com

Thanks,
Olek

#14 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

Quoted 4 lines · Alexander Lobakin
Quoted 4 lines

I added the following snippet[1] and only saw a stable ~1% improvement when sending 64 size packets with xdpsock.

1% is noise level, I am definitely not convinced.

#15 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

In reply to Eric Dumazet
Quoted 66 lines · Eric Dumazet
Quoted 56 lines · Alexander Lobakin

From: Eric Dumazet <[email protected]>
Date: Wed, 15 Oct 2025 05:16:05 -0700

Quoted 42 lines
Quoted 30 lines · Alexander Lobakin

From: Eric Dumazet <[email protected]>
Date: Tue, 14 Oct 2025 17:19:03 +0000

Quoted 26 lines

While stress testing UDP senders on a host with expensive indirect calls, I found cpus processing TX completions where showing a very high cost (20%) in sock_wfree() due to CONFIG_MITIGATION_RETPOLINE=y.

Take care of TCP and UDP TX destructors and use INDIRECT_CALL_3() macro.

---
 net/core/skbuff.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
net/core/skbuff.c +5 -1
index bc12790017b0..692e3a70e75e 100644
@@ -1136,7 +1136,16 @@ void skb_release_head_state(struct sk_buff *skb)
      skb_dst_drop(skb);
      if (skb->destructor) {
              DEBUG_NET_WARN_ON_ONCE(in_hardirq());
-             skb->destructor(skb);
+#ifdef CONFIG_INET
+             INDIRECT_CALL_3(skb->destructor,
+                             tcp_wfree, __sock_wfree, sock_wfree,
+                             skb);
+#else
Quoted 9 lines

net/core/skbuff.c void skb_release_head_state(struct sk_buff *skb) · line 1148

Quoted 5 lines
net/core/skbuff.c +5 -0
+             INDIRECT_CALL_1(skb->destructor,
+                             sock_wfree,
+                             skb);
+
+#endif

Is it just me or seems like you ignored the suggestion/discussion under v1 of this patch...

Quoted 1 line

I did not. Please send a patch when you can demonstrate the difference.

You "did not", but you didn't reply there, only sent v2 w/o any mention.

Quoted 2 lines

We are not going to add all the possible destructors unless there is evidence.

There are numbers in the original discussion, you'd have noticed if you did read.

Quoted 6 lines

We only ask to add one more destructor which will help certain perf-critical workloads. Add it to the end of the list, so that it won't hurt your optimization.

"Send a patch" means you're now changing these lines now and then they would be changed once again, why...

Quoted 7 lines

I can not test what you propose.

I can drop this patch instead, and keep it in Google kernels, (we had TCP support for years)

Or... you can send a patch on top of it later.

Sorry, I've been away from the keyboard for a few days. I think it's fair to let us (who are currently working on the xsk improvement) post a simple patch based on the series.

Regarding what you mentioned that 1% is a noisy number, I disagree. The overall numbers are improved, rather than only one or small part of them. I've done a few tests under different servers, so I believe what I've seen. BTW, xdpsock is the test tool that gives a stable number especially when running on the physical machine.

@ Alexander I think I can post that patch with more test numbers and your 'suggested-by' tag included if you have no objection:) Or if you wish you could do it on your own, please feel free to send one then :)

Thanks,
Jason

#16 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

Quoted 13 lines · Jason Xing

Sorry, I've been away from the keyboard for a few days. I think it's fair to let us (who are currently working on the xsk improvement) post a simple patch based on the series.

Regarding what you mentioned that 1% is a noisy number, I disagree. The overall numbers are improved, rather than only one or small part of them. I've done a few tests under different servers, so I believe what I've seen. BTW, xdpsock is the test tool that gives a stable number especially when running on the physical machine.

Quoted 3 lines

@ Alexander I think I can post that patch with more test numbers and your 'suggested-by' tag included if you have no objection:) Or if you wish you could do it on your own, please feel free to send one then :)

The series focus was on something bringing 100% improvement. The 1% figure _was_ noise.

I think you are mistaken on what a "SIgned-off-by: Eric Dumazet <[email protected]>" means.

I am not opposed to a patch that you will support by yourself. I am opposed to you trying to let me take responsibility for something I have no time/desire to support.

I added the indirect call wrapper mostly at the last moment, so that anyone wanting to test my series
like I described (UDP workload) would not have to mention the sock_wfree cost.

#17 lore raw

Re: [PATCH v2 net-next 2/6] net: add add indirect call wrapper in skb_release_head_state()

Quoted 25 lines · Eric Dumazet
Quoted 13 lines · Jason Xing

Sorry, I've been away from the keyboard for a few days. I think it's fair to let us (who are currently working on the xsk improvement) post a simple patch based on the series.

Regarding what you mentioned that 1% is a noisy number, I disagree. The overall numbers are improved, rather than only one or small part of them. I've done a few tests under different servers, so I believe what I've seen. BTW, xdpsock is the test tool that gives a stable number especially when running on the physical machine.

Quoted 3 lines

@ Alexander I think I can post that patch with more test numbers and your 'suggested-by' tag included if you have no objection:) Or if you wish you could do it on your own, please feel free to send one then :)

The series focus was on something bringing 100% improvement. The 1% figure _was_ noise.

I think you are mistaken on what a "SIgned-off-by: Eric Dumazet <[email protected]>" means.

Quoted 7 lines

I am not opposed to a patch that you will support by yourself. I am opposed to you trying to let me take responsibility for something I have no time/desire to support.

I added the indirect call wrapper mostly at the last moment, so that anyone wanting to test my series
like I described (UDP workload) would not have to mention the sock_wfree cost.

Eric, I totally understand what you meant and thanks for bringing up this idea :) Agree that one patch should do one thing at a time. It's clean. If people smell something wrong in the future, they can easily bisect and revert. So what I replied was that I decided to add a follow-up patch to only support the xsk scenario.

Thanks,
Jason

#18 lore raw

PATCH v2 3/6 net/sched: act_mirred: add loop detection

In reply to Eric Dumazet

Commit 0f022d32c3ec ("net/sched: Fix mirred deadlock on device recursion") added code in the fast path, even when act_mirred is not used.

Prepare its revert by implementing loop detection in act_mirred.

Adds an array of device pointers in struct netdev_xmit.

tcf_mirred_is_act_redirect() can detect if the array already contains the target device.

---
 include/linux/netdevice_xmit.h |  9 ++++-
 net/sched/act_mirred.c         | 62 +++++++++++++---------------------
 2 files changed, 31 insertions(+), 40 deletions(-)
include/linux/netdevice_xmit.h +8 -1
index 813a19122ebb..cc232508e695 100644
@@ -2,6 +2,12 @@
 #ifndef _LINUX_NETDEVICE_XMIT_H
 #define _LINUX_NETDEVICE_XMIT_H
 
+#if IS_ENABLED(CONFIG_NET_ACT_MIRRED)
+#define MIRRED_NEST_LIMIT	4
+#endif
+
+struct net_device;
+
 struct netdev_xmit {
 	u16 recursion;
 	u8  more;
@@ -9,7 +15,8 @@ struct netdev_xmit {
 	u8  skip_txqueue;
 #endif
 #if IS_ENABLED(CONFIG_NET_ACT_MIRRED)
-	u8 sched_mirred_nest;
+	u8			sched_mirred_nest;
+	struct net_device	*sched_mirred_dev[MIRRED_NEST_LIMIT];
 #endif
 #if IS_ENABLED(CONFIG_NF_DUP_NETDEV)
 	u8 nf_dup_skb_recursion;
net/sched/act_mirred.c +23 -39

98 lines not on this page · raw message

Signature · 1 line
2.51.0.788.g6d19910ace-goog
Reviewed-by on v2 3/6
#19 lore raw

Re: [PATCH v2 net-next 3/6] net/sched: act_mirred: add loop detection

Quoted 4 lines · Eric Dumazet

Commit 0f022d32c3ec ("net/sched: Fix mirred deadlock on device recursion") added code in the fast path, even when act_mirred is not used.

Quoted 8 lines

Prepare its revert by implementing loop detection in act_mirred.

Adds an array of device pointers in struct netdev_xmit.

tcf_mirred_is_act_redirect() can detect if the array already contains the target device.

Reviewed-by on v2 3/6
#20 lore raw

Re: [PATCH v2 net-next 3/6] net/sched: act_mirred: add loop detection

In reply to Eric Dumazet
Quoted 11 lines · Eric Dumazet

Commit 0f022d32c3ec ("net/sched: Fix mirred deadlock on device recursion") added code in the fast path, even when act_mirred is not used.

Prepare its revert by implementing loop detection in act_mirred.

Adds an array of device pointers in struct netdev_xmit.

tcf_mirred_is_act_redirect() can detect if the array already contains the target device.

#21 lore raw

PATCH v2 4/6 Revert "net/sched: Fix mirred deadlock on device recursion"

In reply to Eric Dumazet

This reverts commits 0f022d32c3eca477fbf79a205243a6123ed0fe11 and 44180feaccf266d9b0b28cc4ceaac019817deb5c.

Prior patch in this series implemented loop detection in act_mirred, we can remove q->owner to save some cycles in the fast path.

---
 include/net/sch_generic.h | 1 -
 net/core/dev.c            | 6 ------
 net/sched/sch_generic.c   | 2 --
 3 files changed, 9 deletions(-)
include/net/sch_generic.h +0 -1
index 738cd5b13c62..32e9961570b4 100644
@@ -117,7 +117,6 @@ struct Qdisc {
 	struct qdisc_skb_head	q;
 	struct gnet_stats_basic_sync bstats;
 	struct gnet_stats_queue	qstats;
-	int                     owner;
 	unsigned long		state;
 	unsigned long		state2; /* must be written under qdisc spinlock */
 	struct Qdisc            *next_sched;
net/core/dev.c +0 -6
index a64cef2c537e..0ff178399b2d 100644
@@ -4167,10 +4167,6 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,
 		return rc;
 	}
 
-	if (unlikely(READ_ONCE(q->owner) == smp_processor_id())) {
-		kfree_skb_reason(skb, SKB_DROP_REASON_TC_RECLASSIFY_LOOP);
-		return NET_XMIT_DROP;
-	}
 	/*
 	 * Heuristic to force contended enqueues to serialize on a
 	 * separate lock before trying to get qdisc main lock.
@@ -4210,9 +4206,7 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,
 		qdisc_run_end(q);
 		rc = NET_XMIT_SUCCESS;
 	} else {
-		WRITE_ONCE(q->owner, smp_processor_id());
 		rc = dev_qdisc_enqueue(skb, q, &to_free, txq);
-		WRITE_ONCE(q->owner, -1);
 		if (qdisc_run_begin(q)) {
 			if (unlikely(contended)) {
 				spin_unlock(&q->busylock);
net/sched/sch_generic.c +0 -2
index 1e008a228ebd..dfa8e8e667d2 100644
@@ -679,7 +679,6 @@ struct Qdisc noop_qdisc = {
 		.qlen = 0,
 		.lock = __SPIN_LOCK_UNLOCKED(noop_qdisc.skb_bad_txq.lock),
 	},
-	.owner = -1,
 };
 EXPORT_SYMBOL(noop_qdisc);
 
@@ -985,7 +984,6 @@ struct Qdisc *qdisc_alloc(struct netdev_queue *dev_queue,
 	sch->enqueue = ops->enqueue;
 	sch->dequeue = ops->dequeue;
 	sch->dev_queue = dev_queue;
-	sch->owner = -1;
 	netdev_hold(dev, &sch->dev_tracker, GFP_KERNEL);
 	refcount_set(&sch->refcnt, 1);
 
Signature · 1 line
2.51.0.788.g6d19910ace-goog
Reviewed-by on v2 4/6
#22 lore raw

Re: [PATCH v2 net-next 4/6] Revert "net/sched: Fix mirred deadlock on device recursion"

Quoted 9 lines · Eric Dumazet

This reverts commits 0f022d32c3eca477fbf79a205243a6123ed0fe11 and 44180feaccf266d9b0b28cc4ceaac019817deb5c.

Prior patch in this series implemented loop detection in act_mirred, we can remove q->owner to save some cycles in the fast path.

Reviewed-by on v2 4/6
#23 lore raw

Re: [PATCH v2 net-next 4/6] Revert "net/sched: Fix mirred deadlock on device recursion"

In reply to Eric Dumazet
Quoted 8 lines · Eric Dumazet

This reverts commits 0f022d32c3eca477fbf79a205243a6123ed0fe11 and 44180feaccf266d9b0b28cc4ceaac019817deb5c.

Prior patch in this series implemented loop detection in act_mirred, we can remove q->owner to save some cycles in the fast path.

Reviewed-by on v2 4/6
#24 lore raw

Re: [PATCH v2 net-next 4/6] Revert "net/sched: Fix mirred deadlock on device recursion"

In reply to Eric Dumazet
Quoted 8 lines · Eric Dumazet

This reverts commits 0f022d32c3eca477fbf79a205243a6123ed0fe11 and 44180feaccf266d9b0b28cc4ceaac019817deb5c.

Prior patch in this series implemented loop detection in act_mirred, we can remove q->owner to save some cycles in the fast path.

#25 lore raw

PATCH v2 5/6 net: sched: claim one cache line in Qdisc

In reply to Eric Dumazet

Replace state2 field with a boolean.

Move it to a hole between qstats and state so that we shrink Qdisc by a full cache line.

---
 include/net/sch_generic.h | 18 +++++++-----------
 1 file changed, 7 insertions(+), 11 deletions(-)
include/net/sch_generic.h +7 -11
index 32e9961570b4..31561291bc92 100644
@@ -41,13 +41,6 @@ enum qdisc_state_t {
 	__QDISC_STATE_DRAINING,
 };
 
-enum qdisc_state2_t {
-	/* Only for !TCQ_F_NOLOCK qdisc. Never access it directly.
-	 * Use qdisc_run_begin/end() or qdisc_is_running() instead.
-	 */
-	__QDISC_STATE2_RUNNING,
-};
-
 #define QDISC_STATE_MISSED	BIT(__QDISC_STATE_MISSED)
 #define QDISC_STATE_DRAINING	BIT(__QDISC_STATE_DRAINING)
 
@@ -117,8 +110,8 @@ struct Qdisc {
 	struct qdisc_skb_head	q;
 	struct gnet_stats_basic_sync bstats;
 	struct gnet_stats_queue	qstats;
+	bool			running; /* must be written under qdisc spinlock */
 	unsigned long		state;
-	unsigned long		state2; /* must be written under qdisc spinlock */
 	struct Qdisc            *next_sched;
 	struct sk_buff_head	skb_bad_txq;
 
@@ -167,7 +160,7 @@ static inline bool qdisc_is_running(struct Qdisc *qdisc)
 {
 	if (qdisc->flags & TCQ_F_NOLOCK)
 		return spin_is_locked(&qdisc->seqlock);
-	return test_bit(__QDISC_STATE2_RUNNING, &qdisc->state2);
+	return READ_ONCE(qdisc->running);
 }
 
 static inline bool nolock_qdisc_is_empty(const struct Qdisc *qdisc)
@@ -210,7 +203,10 @@ static inline bool qdisc_run_begin(struct Qdisc *qdisc)
 		 */
 		return spin_trylock(&qdisc->seqlock);
 	}
-	return !__test_and_set_bit(__QDISC_STATE2_RUNNING, &qdisc->state2);
+	if (READ_ONCE(qdisc->running))
+		return false;
+	WRITE_ONCE(qdisc->running, true);
+	return true;
 }
 
 static inline void qdisc_run_end(struct Qdisc *qdisc)
@@ -228,7 +224,7 @@ static inline void qdisc_run_end(struct Qdisc *qdisc)
 				      &qdisc->state)))
 			__netif_schedule(qdisc);
 	} else {
-		__clear_bit(__QDISC_STATE2_RUNNING, &qdisc->state2);
+		WRITE_ONCE(qdisc->running, false);
 	}
 }
 
Signature · 1 line
2.51.0.788.g6d19910ace-goog
#28 lore raw

PATCH v2 6/6 net: dev_queue_xmit() llist adoption

In reply to Eric Dumazet

Remove busylock spinlock and use a lockless list (llist) to reduce spinlock contention to the minimum.

Idea is that only one cpu might spin on the qdisc spinlock, while others simply add their skb in the llist.

After this patch, we get a 300 % improvement on heavy TX workloads.
- Sending twice the number of packets per second.
- While consuming 50 % less cycles.

Note that this also allows in the future to submit batches to various qdisc->enqueue() methods.

Tested:

- Dual Intel(R) Xeon(R) 6985P-C (480 hyper threads).
- 100Gbit NIC, 30 TX queues with FQ packet scheduler.
- echo 64 >/sys/kernel/slab/skbuff_small_head/cpu_partial (avoid contention in mm)
- 240 concurrent "netperf -t UDP_STREAM -- -m 120 -n"

Before:

16 Mpps (41 Mpps if each thread is pinned to a different cpu)

vmstat 2 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st

243  0      0 2368988672  51036 1100852    0    0   146     1  242   60  0  9 91  0  0
244  0      0 2368988672  51036 1100852    0    0   536    10 487745 14718  0 52 48  0  0
244  0      0 2368988672  51036 1100852    0    0   512     0 503067 46033  0 52 48  0  0
244  0      0 2368988672  51036 1100852    0    0   512     0 494807 12107  0 52 48  0  0
244  0      0 2368988672  51036 1100852    0    0   702    26 492845 10110  0 52 48  0  0

Lock contention (1 second sample taken on 8 cores) perf lock record -C0-7 sleep 1; perf lock contention
contended total wait max wait avg wait type caller

    442111      6.79 s     162.47 ms     15.35 us     spinlock   dev_hard_start_xmit+0xcd
      5961      9.57 ms      8.12 us      1.60 us     spinlock   __dev_queue_xmit+0x3a0
       244    560.63 us      7.63 us      2.30 us     spinlock   do_softirq+0x5b
        13     25.09 us      3.21 us      1.93 us     spinlock   net_tx_action+0xf8

If netperf threads are pinned, spinlock stress is very high. perf lock record -C0-7 sleep 1; perf lock contention
contended total wait max wait avg wait type caller

    964508      7.10 s     147.25 ms      7.36 us     spinlock   dev_hard_start_xmit+0xcd
       201    268.05 us      4.65 us      1.33 us     spinlock   __dev_queue_xmit+0x3a0
        12     26.05 us      3.84 us      2.17 us     spinlock   do_softirq+0x5b

@__dev_queue_xmit_ns:

[256, 512)            21 |                                                    |
[512, 1K)            631 |                                                    |
[1K, 2K)           27328 |@                                                   |
[2K, 4K)          265392 |@@@@@@@@@@@@@@@@                                    |
[4K, 8K)          417543 |@@@@@@@@@@@@@@@@@@@@@@@@@@                          |
[8K, 16K)         826292 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16K, 32K)        733822 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@      |
[32K, 64K)         19055 |@                                                   |
[64K, 128K)        17240 |@                                                   |
[128K, 256K)       25633 |@                                                   |
[256K, 512K)           4 |                                                    |

After:

29 Mpps (57 Mpps if each thread is pinned to a different cpu)

vmstat 2 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
78  0      0 2369573632  32896 1350988    0    0    22     0  331  254  0  8 92  0  0
75  0      0 2369573632  32896 1350988    0    0    22    50 425713 280199  0 23 76  0  0
104  0      0 2369573632  32896 1350988    0    0   290     0 430238 298247  0 23 76  0  0
86  0      0 2369573632  32896 1350988    0    0   132     0 428019 291865  0 24 76  0  0
90  0      0 2369573632  32896 1350988    0    0   502     0 422498 278672  0 23 76  0  0

perf lock record -C0-7 sleep 1; perf lock contention
contended total wait max wait avg wait type caller

      2524    116.15 ms    486.61 us     46.02 us     spinlock   __dev_queue_xmit+0x55b
      5821    107.18 ms    371.67 us     18.41 us     spinlock   dev_hard_start_xmit+0xcd
      2377      9.73 ms     35.86 us      4.09 us     spinlock   ___slab_alloc+0x4e0
       923      5.74 ms     20.91 us      6.22 us     spinlock   ___slab_alloc+0x5c9
       121      3.42 ms    193.05 us     28.24 us     spinlock   net_tx_action+0xf8
         6    564.33 us    167.60 us     94.05 us     spinlock   do_softirq+0x5b

If netperf threads are pinned (~54 Mpps) perf lock record -C0-7 sleep 1; perf lock contention

     32907    316.98 ms    195.98 us      9.63 us     spinlock   dev_hard_start_xmit+0xcd
      4507     61.83 ms    212.73 us     13.72 us     spinlock   __dev_queue_xmit+0x554
      2781     23.53 ms     40.03 us      8.46 us     spinlock   ___slab_alloc+0x5c9
      3554     18.94 ms     34.69 us      5.33 us     spinlock   ___slab_alloc+0x4e0
       233      9.09 ms    215.70 us     38.99 us     spinlock   do_softirq+0x5b
       153    930.66 us     48.67 us      6.08 us     spinlock   net_tx_action+0xfd
        84    331.10 us     14.22 us      3.94 us     spinlock   ___slab_alloc+0x5c9
       140    323.71 us      9.94 us      2.31 us     spinlock   ___slab_alloc+0x4e0

@__dev_queue_xmit_ns:

[128, 256)       1539830 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[256, 512)       2299558 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)         483936 |@@@@@@@@@@                                          |
[1K, 2K)          265345 |@@@@@@                                              |
[2K, 4K)          145463 |@@@                                                 |
[4K, 8K)           54571 |@                                                   |
[8K, 16K)          10270 |                                                    |
[16K, 32K)          9385 |                                                    |
[32K, 64K)          7749 |                                                    |
[64K, 128K)        26799 |                                                    |
[128K, 256K)        2665 |                                                    |
[256K, 512K)         665 |                                                    |
---
 include/net/sch_generic.h |  4 +-
 net/core/dev.c            | 91 ++++++++++++++++++++++++---------------
 net/sched/sch_generic.c   |  5 ---
 3 files changed, 59 insertions(+), 41 deletions(-)
include/net/sch_generic.h +3 -1
index 31561291bc92..94966692ccdf 100644
@@ -115,7 +115,9 @@ struct Qdisc {
 	struct Qdisc            *next_sched;
 	struct sk_buff_head	skb_bad_txq;
 
-	spinlock_t		busylock ____cacheline_aligned_in_smp;
+	atomic_long_t		defer_count ____cacheline_aligned_in_smp;
+	struct llist_head	defer_list;
+
 	spinlock_t		seqlock;
 
 	struct rcu_head		rcu;
net/core/dev.c +56 -35

126 lines not on this page · raw message

net/sched/sch_generic.c +0 -5
index dfa8e8e667d2..d9a98d02a55f 100644
@@ -666,7 +666,6 @@ struct Qdisc noop_qdisc = {
 	.ops		=	&noop_qdisc_ops,
 	.q.lock		=	__SPIN_LOCK_UNLOCKED(noop_qdisc.q.lock),
 	.dev_queue	=	&noop_netdev_queue,
-	.busylock	=	__SPIN_LOCK_UNLOCKED(noop_qdisc.busylock),
 	.gso_skb = {
 		.next = (struct sk_buff *)&noop_qdisc.gso_skb,
 		.prev = (struct sk_buff *)&noop_qdisc.gso_skb,
@@ -970,10 +969,6 @@ struct Qdisc *qdisc_alloc(struct netdev_queue *dev_queue,
 		}
 	}
 
-	spin_lock_init(&sch->busylock);
-	lockdep_set_class(&sch->busylock,
-			  dev->qdisc_tx_busylock ?: &qdisc_tx_busylock);
-
 	/* seqlock has the same scope of busylock, for NOLOCK qdisc */
 	spin_lock_init(&sch->seqlock);
 	lockdep_set_class(&sch->seqlock,
Signature · 1 line
2.51.0.788.g6d19910ace-goog
Reviewed-by on v2 6/6
#29 lore raw

Re: [PATCH v2 net-next 6/6] net: dev_queue_xmit() llist adoption

Quoted 112 lines · Eric Dumazet

Remove busylock spinlock and use a lockless list (llist) to reduce spinlock contention to the minimum.

Idea is that only one cpu might spin on the qdisc spinlock, while others simply add their skb in the llist.

After this patch, we get a 300 % improvement on heavy TX workloads.
- Sending twice the number of packets per second.
- While consuming 50 % less cycles.

Note that this also allows in the future to submit batches to various qdisc->enqueue() methods.

Tested:

- Dual Intel(R) Xeon(R) 6985P-C (480 hyper threads).
- 100Gbit NIC, 30 TX queues with FQ packet scheduler.
- echo 64 >/sys/kernel/slab/skbuff_small_head/cpu_partial (avoid contention in mm)
- 240 concurrent "netperf -t UDP_STREAM -- -m 120 -n"

Before:

16 Mpps (41 Mpps if each thread is pinned to a different cpu)

vmstat 2 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st

243  0      0 2368988672  51036 1100852    0    0   146     1  242   60  0  9 91  0  0
244  0      0 2368988672  51036 1100852    0    0   536    10 487745 14718  0 52 48  0  0
244  0      0 2368988672  51036 1100852    0    0   512     0 503067 46033  0 52 48  0  0
244  0      0 2368988672  51036 1100852    0    0   512     0 494807 12107  0 52 48  0  0
244  0      0 2368988672  51036 1100852    0    0   702    26 492845 10110  0 52 48  0  0

Lock contention (1 second sample taken on 8 cores) perf lock record -C0-7 sleep 1; perf lock contention
contended total wait max wait avg wait type caller

    442111      6.79 s     162.47 ms     15.35 us     spinlock   dev_hard_start_xmit+0xcd
      5961      9.57 ms      8.12 us      1.60 us     spinlock   __dev_queue_xmit+0x3a0
       244    560.63 us      7.63 us      2.30 us     spinlock   do_softirq+0x5b
        13     25.09 us      3.21 us      1.93 us     spinlock   net_tx_action+0xf8

If netperf threads are pinned, spinlock stress is very high. perf lock record -C0-7 sleep 1; perf lock contention
contended total wait max wait avg wait type caller

    964508      7.10 s     147.25 ms      7.36 us     spinlock   dev_hard_start_xmit+0xcd
       201    268.05 us      4.65 us      1.33 us     spinlock   __dev_queue_xmit+0x3a0
        12     26.05 us      3.84 us      2.17 us     spinlock   do_softirq+0x5b

@__dev_queue_xmit_ns:

[256, 512)            21 |                                                    |
[512, 1K)            631 |                                                    |
[1K, 2K)           27328 |@                                                   |
[2K, 4K)          265392 |@@@@@@@@@@@@@@@@                                    |
[4K, 8K)          417543 |@@@@@@@@@@@@@@@@@@@@@@@@@@                          |
[8K, 16K)         826292 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16K, 32K)        733822 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@      |
[32K, 64K)         19055 |@                                                   |
[64K, 128K)        17240 |@                                                   |
[128K, 256K)       25633 |@                                                   |
[256K, 512K)           4 |                                                    |

After:

29 Mpps (57 Mpps if each thread is pinned to a different cpu)

vmstat 2 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
78  0      0 2369573632  32896 1350988    0    0    22     0  331  254  0  8 92  0  0
75  0      0 2369573632  32896 1350988    0    0    22    50 425713 280199  0 23 76  0  0
104  0      0 2369573632  32896 1350988    0    0   290     0 430238 298247  0 23 76  0  0
86  0      0 2369573632  32896 1350988    0    0   132     0 428019 291865  0 24 76  0  0
90  0      0 2369573632  32896 1350988    0    0   502     0 422498 278672  0 23 76  0  0

perf lock record -C0-7 sleep 1; perf lock contention
contended total wait max wait avg wait type caller

      2524    116.15 ms    486.61 us     46.02 us     spinlock   __dev_queue_xmit+0x55b
      5821    107.18 ms    371.67 us     18.41 us     spinlock   dev_hard_start_xmit+0xcd
      2377      9.73 ms     35.86 us      4.09 us     spinlock   ___slab_alloc+0x4e0
       923      5.74 ms     20.91 us      6.22 us     spinlock   ___slab_alloc+0x5c9
       121      3.42 ms    193.05 us     28.24 us     spinlock   net_tx_action+0xf8
         6    564.33 us    167.60 us     94.05 us     spinlock   do_softirq+0x5b

If netperf threads are pinned (~54 Mpps) perf lock record -C0-7 sleep 1; perf lock contention

     32907    316.98 ms    195.98 us      9.63 us     spinlock   dev_hard_start_xmit+0xcd
      4507     61.83 ms    212.73 us     13.72 us     spinlock   __dev_queue_xmit+0x554
      2781     23.53 ms     40.03 us      8.46 us     spinlock   ___slab_alloc+0x5c9
      3554     18.94 ms     34.69 us      5.33 us     spinlock   ___slab_alloc+0x4e0
       233      9.09 ms    215.70 us     38.99 us     spinlock   do_softirq+0x5b
       153    930.66 us     48.67 us      6.08 us     spinlock   net_tx_action+0xfd
        84    331.10 us     14.22 us      3.94 us     spinlock   ___slab_alloc+0x5c9
       140    323.71 us      9.94 us      2.31 us     spinlock   ___slab_alloc+0x4e0

@__dev_queue_xmit_ns:

[128, 256)       1539830 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                  |
[256, 512)       2299558 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K)         483936 |@@@@@@@@@@                                          |
[1K, 2K)          265345 |@@@@@@                                              |
[2K, 4K)          145463 |@@@                                                 |
[4K, 8K)           54571 |@                                                   |
[8K, 16K)          10270 |                                                    |
[16K, 32K)          9385 |                                                    |
[32K, 64K)          7749 |                                                    |
[64K, 128K)        26799 |                                                    |
[128K, 256K)        2665 |                                                    |
[256K, 512K)         665 |                                                    |
Quoted 2 lines

Thanks for big improvement!

maintainer Tested-by Acked-by on the v2 series
#30 lore raw
In reply to Eric Dumazet
Quoted 11 lines · Eric Dumazet

In this series, I replace the busylock spinlock we have in __dev_queue_xmit() and use lockless list (llist) to reduce spinlock contention to the minimum.

Idea is that only one cpu might spin on the qdisc spinlock, while others simply add their skb in the llist.

After this series, we get a 300 % (4x) improvement on heavy TX workloads, sending twice the number of packets per second, for half the cpu cycles.

Not important but i am curious: you didnt mention what NIC this was in the commit messages ;->

For the patchset, I have done testing with existing tdc tests and no regression..
It does inspire new things when time becomes available.... so will be doing more testing and likely small extensions etc.
So:

(For the tc bits, since the majority of the code touches tc related stuff)

cheers,
jamal

Quoted 25 lines

v2: deflake tcp_user_timeout_user-timeout-probe.pkt.
Ability to return a different code than NET_XMIT_SUCCESS when __dev_xmit_skb() has a single skb to send.

Eric Dumazet (6):
selftests/net: packetdrill: unflake
tcp_user_timeout_user-timeout-probe.pkt
net: add add indirect call wrapper in skb_release_head_state() net/sched: act_mirred: add loop detection
Revert "net/sched: Fix mirred deadlock on device recursion"
net: sched: claim one cache line in Qdisc
net: dev_queue_xmit() llist adoption

 include/linux/netdevice_xmit.h                |  9 +-
 include/net/sch_generic.h                     | 23 ++---
 net/core/dev.c                                | 97 +++++++++++--------
 net/core/skbuff.c                             | 11 ++-
 net/sched/act_mirred.c                        | 62 +++++-------
 net/sched/sch_generic.c                       |  7 --
 .../tcp_user_timeout_user-timeout-probe.pkt   |  6 +-
 7 files changed, 111 insertions(+), 104 deletions(-)

--
2.51.0.788.g6d19910ace-goog

#31 lore raw
Quoted 10 lines · Jamal Hadi Salim
Quoted 8 lines · Eric Dumazet

In this series, I replace the busylock spinlock we have in __dev_queue_xmit() and use lockless list (llist) to reduce spinlock contention to the minimum.

Idea is that only one cpu might spin on the qdisc spinlock, while others simply add their skb in the llist.

Quoted 6 lines
Quoted 3 lines

After this series, we get a 300 % (4x) improvement on heavy TX workloads, sending twice the number of packets per second, for half the cpu cycles.

Not important but i am curious: you didnt mention what NIC this was in the commit messages ;->

I have used two NIC : IDPF (200Gbit), and GQ (100Gbit Google NIC) (Usually with 32 TX queues)

And a variety of platforms, up to 512 cores sharing these 32 TX queues.

Quoted 8 lines

For the patchset, I have done testing with existing tdc tests and no regression..
It does inspire new things when time becomes available.... so will be doing more testing and likely small extensions etc.
So:

Thanks Jamal !

I also have this idea:

cpus serving NIC interrupts and specifically TX completions are often trapped in also restarting a busy qdisc (because it was stopped by BQL or the driver's own flow control).

We can do better:

1) In the TX completion loop, collect the skbs and do not free them immediately.

2) Store them in a private list and sum their skb->len while doing so.

3) Then call netdev_tx_completed_queue() call and netif_tx_wake_queue().

If the queue was stopped, this might add the qdisc in our per-cpu private list (sd->output_queue), raising NET_TX_SOFTIRQ (no immediate action because napi poll runs while BH are blocked)

4) Then, take care of all dev_consume_skb_any().

Quite often freeing these skbs can take a lot of time, because of mm contention, and other false sharing, or expensive skb->destructor like TCP.

5) By the time net_tx_action() finally runs, perhaps another cpu saw the queue being in XON state and was able to push more packets to the queue.

This means net_tx_action() might have nothing to do, saving precious cycles.

We should add extra logic in net_tx_action() to not even grab qdisc_lock() spinlock at all.

My thinking is to add back a sequence (to replace q->running boolean), and store a snapshot of this sequence every time we restart a queue. -> net_tx_action can compare the sequence against the last snapshot.

Quoted 31 lines

(For the tc bits, since the majority of the code touches tc related stuff)

cheers,
jamal

Quoted 25 lines

v2: deflake tcp_user_timeout_user-timeout-probe.pkt.
Ability to return a different code than NET_XMIT_SUCCESS when __dev_xmit_skb() has a single skb to send.

Eric Dumazet (6):
selftests/net: packetdrill: unflake
tcp_user_timeout_user-timeout-probe.pkt
net: add add indirect call wrapper in skb_release_head_state() net/sched: act_mirred: add loop detection
Revert "net/sched: Fix mirred deadlock on device recursion"
net: sched: claim one cache line in Qdisc
net: dev_queue_xmit() llist adoption

 include/linux/netdevice_xmit.h                |  9 +-
 include/net/sch_generic.h                     | 23 ++---
 net/core/dev.c                                | 97 +++++++++++--------
 net/core/skbuff.c                             | 11 ++-
 net/sched/act_mirred.c                        | 62 +++++-------
 net/sched/sch_generic.c                       |  7 --
 .../tcp_user_timeout_user-timeout-probe.pkt   |  6 +-
 7 files changed, 111 insertions(+), 104 deletions(-)

--
2.51.0.788.g6d19910ace-goog

#32 lore raw
In reply to Eric Dumazet
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 8 lines · On Tue, 14 Oct 2025 17:19:01 +0000 you wrote:

In this series, I replace the busylock spinlock we have in __dev_queue_xmit() and use lockless list (llist) to reduce spinlock contention to the minimum.

Idea is that only one cpu might spin on the qdisc spinlock, while others simply add their skb in the llist.

[...]

Here is the summary with links:
- [v2,net-next,1/6] selftests/net: packetdrill: unflake tcp_user_timeout_user-timeout-probe.pkt

    https://git.kernel.org/netdev/net-next/c/56cef47c28dc
  - [v2,net-next,2/6] net: add add indirect call wrapper in skb_release_head_state()
    https://git.kernel.org/netdev/net-next/c/5b2b7dec05f3
  - [v2,net-next,3/6] net/sched: act_mirred: add loop detection
    https://git.kernel.org/netdev/net-next/c/fe946a751d9b
  - [v2,net-next,4/6] Revert "net/sched: Fix mirred deadlock on device recursion"
    https://git.kernel.org/netdev/net-next/c/178ca30889a1
  - [v2,net-next,5/6] net: sched: claim one cache line in Qdisc
    https://git.kernel.org/netdev/net-next/c/526f5fb112f7
  - [v2,net-next,6/6] net: dev_queue_xmit() llist adoption
    https://git.kernel.org/netdev/net-next/c/100dfa74cad9

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html