Skip to contentSkip to conversation

tcp: try to defer / return acked skbs to originating CPU

PATCH In review Jakub Kicinski 2 participants 15 messages Jan 17 – Feb 17

Subsystems (4)
lore
Share & tools
lore

Jakub Kicinski

author maintainer
#0 lore raw

PATCH 1/1 tcp: try to defer / return acked skbs to originating CPU

Running a memcache-like workload under production(ish) load on a 300 thread AMD machine we see ~3% of CPU time spent in kmem_cache_free() via tcp_ack(), freeing skbs from rtx queue. This workloads pins workers away from softirq CPU so the Tx skbs are pretty much always allocated on a different CPU than where the ACKs arrive. Try to use the defer skb free queue to return the skbs back to where they came from. This results in a ~4% performance improvement for the workload.

---
 include/net/tcp.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/net/tcp.h +1 -1
index ef0fee58fde8..e290651da508 100644
@@ -332,7 +332,7 @@ static inline void tcp_wmem_free_skb(struct sock *sk, struct sk_buff *skb)
 		sk_mem_uncharge(sk, skb->truesize);
 	else
 		sk_mem_uncharge(sk, SKB_TRUESIZE(skb_end_offset(skb)));
-	__kfree_skb(skb);
+	skb_attempt_defer_free(skb);
 }
 
 void sk_forced_mem_schedule(struct sock *sk, int size);
Signature · 1 line
2.52.0
author maintainer
#1 lore raw
Quoted 8 lines · On Sat, 17 Jan 2026 08:42:55 -0800 Jakub Kicinski wrote:

Running a memcache-like workload under production(ish) load on a 300 thread AMD machine we see ~3% of CPU time spent in kmem_cache_free() via tcp_ack(), freeing skbs from rtx queue. This workloads pins workers away from softirq CPU so the Tx skbs are pretty much always allocated on a different CPU than where the ACKs arrive. Try to use the defer skb free queue to return the skbs back to where they came from. This results in a ~4% performance improvement for the workload.

In the interest of full transparency the performance testing was done on a 6.13-ish kernel. But I don't see anything that'd make the situation better upstream..

#2 lore raw
In reply to Jakub Kicinski
Quoted 10 lines · Jakub Kicinski

Running a memcache-like workload under production(ish) load on a 300 thread AMD machine we see ~3% of CPU time spent in kmem_cache_free() via tcp_ack(), freeing skbs from rtx queue. This workloads pins workers away from softirq CPU so the Tx skbs are pretty much always allocated on a different CPU than where the ACKs arrive. Try to use the defer skb free queue to return the skbs back to where they came from. This results in a ~4% performance improvement for the workload.

This probably makes sense when RFS is not used.
Here, RFS gives us ~40% performance improvement for typical RPC workloads, so I never took a look at this side :)

Have you tested what happens for bulk sends ?
sendmsg() allocates skbs and push them to transmit queue, but ACK can decide to split TSO packets, and the new allocation is done on the softirq CPU (assuming RFS is not used)

Perhaps tso_fragment()/tcp_fragment() could copy the source skb->alloc_cpu to (new)buff->alloc_cpu.

Also, if workers are away from softirq, they will only process the defer queue in large patches, after receiving an trigger_rx_softirq() IPI.
Any idea of skb_defer_free_flush() latency when dealing with batches of ~64 big TSO packets ?

Quoted 21 lines
---
 include/net/tcp.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/net/tcp.h +1 -2
index ef0fee58fde8..e290651da508 100644
@@ -332,7 +332,7 @@ static inline void tcp_wmem_free_skb(struct sock *sk, struct sk_buff *skb)
                sk_mem_uncharge(sk, skb->truesize);
        else
                sk_mem_uncharge(sk, SKB_TRUESIZE(skb_end_offset(skb)));
-       __kfree_skb(skb);
+       skb_attempt_defer_free(skb);
 }
 
 void sk_forced_mem_schedule(struct sock *sk, int size);
--

2.52.0

author maintainer
#3 lore raw
Quoted 10 lines · On Sat, 17 Jan 2026 19:16:57 +0100 Eric Dumazet wrote:
Quoted 8 lines · Jakub Kicinski

Running a memcache-like workload under production(ish) load on a 300 thread AMD machine we see ~3% of CPU time spent in kmem_cache_free() via tcp_ack(), freeing skbs from rtx queue. This workloads pins workers away from softirq CPU so the Tx skbs are pretty much always allocated on a different CPU than where the ACKs arrive. Try to use the defer skb free queue to return the skbs back to where they came from. This results in a ~4% performance improvement for the workload.

Quoted 3 lines

This probably makes sense when RFS is not used.
Here, RFS gives us ~40% performance improvement for typical RPC workloads, so I never took a look at this side :)

This workload doesn't like RFS. Maybe because it has 1M sockets.. I'll need to look closer, the patchwork queue first tho.. :)

Quoted 7 lines

Have you tested what happens for bulk sends ?
sendmsg() allocates skbs and push them to transmit queue, but ACK can decide to split TSO packets, and the new allocation is done on the softirq CPU (assuming RFS is not used)

Perhaps tso_fragment()/tcp_fragment() could copy the source skb->alloc_cpu to (new)buff->alloc_cpu.

I'll do some synthetic testing and get back.

Quoted 5 lines

Also, if workers are away from softirq, they will only process the defer queue in large patches, after receiving an trigger_rx_softirq() IPI.
Any idea of skb_defer_free_flush() latency when dealing with batches of ~64 big TSO packets ?

Not sure if there's much we can do about that.. Perhaps we should have a shrinker that flushes the defer queues? I chatted with Shakeel briefly and it sounded fairly straightforward.

Reviewed-by
#4 lore raw
Quoted 35 lines · Jakub Kicinski
Quoted 10 lines · On Sat, 17 Jan 2026 19:16:57 +0100 Eric Dumazet wrote:
Quoted 8 lines · Jakub Kicinski

Running a memcache-like workload under production(ish) load on a 300 thread AMD machine we see ~3% of CPU time spent in kmem_cache_free() via tcp_ack(), freeing skbs from rtx queue. This workloads pins workers away from softirq CPU so the Tx skbs are pretty much always allocated on a different CPU than where the ACKs arrive. Try to use the defer skb free queue to return the skbs back to where they came from. This results in a ~4% performance improvement for the workload.

Quoted 3 lines

This probably makes sense when RFS is not used.
Here, RFS gives us ~40% performance improvement for typical RPC workloads, so I never took a look at this side :)

This workload doesn't like RFS. Maybe because it has 1M sockets.. I'll need to look closer, the patchwork queue first tho.. :)

Quoted 7 lines

Have you tested what happens for bulk sends ?
sendmsg() allocates skbs and push them to transmit queue, but ACK can decide to split TSO packets, and the new allocation is done on the softirq CPU (assuming RFS is not used)

Perhaps tso_fragment()/tcp_fragment() could copy the source skb->alloc_cpu to (new)buff->alloc_cpu.

I'll do some synthetic testing and get back.

Quoted 5 lines

Also, if workers are away from softirq, they will only process the defer queue in large patches, after receiving an trigger_rx_softirq() IPI.
Any idea of skb_defer_free_flush() latency when dealing with batches of ~64 big TSO packets ?

Quoted 3 lines

Not sure if there's much we can do about that.. Perhaps we should have a shrinker that flushes the defer queues? I chatted with Shakeel briefly and it sounded fairly straightforward.

I was mostly concerned about latency spikes, I did some tests here and this seems fine.
(I assume you asked Shakeel about the extra memory being held in the per-cpu queue, and pcp implications ?)

author maintainer
#5 lore raw
Quoted 6 lines · On Sun, 18 Jan 2026 13:15:00 +0100 Eric Dumazet wrote:
Quoted 6 lines
Quoted 5 lines

Also, if workers are away from softirq, they will only process the defer queue in large patches, after receiving an trigger_rx_softirq() IPI.
Any idea of skb_defer_free_flush() latency when dealing with batches of ~64 big TSO packets ?

Quoted 6 lines
Quoted 3 lines

Not sure if there's much we can do about that.. Perhaps we should have a shrinker that flushes the defer queues? I chatted with Shakeel briefly and it sounded fairly straightforward.

I was mostly concerned about latency spikes, I did some tests here and this seems fine.

Looks like selftests run into the zerocopy Tx latency issue. I'll drop this version from patchwork..

Quoted 2 lines

(I assume you asked Shakeel about the extra memory being held in the per-cpu queue, and pcp implications ?)

Under real load it helps quite a bit but real load flushes the queues frequently. I'll talk to him.

author maintainer
#6 lore raw
Quoted 9 lines · On Mon, 19 Jan 2026 09:04:35 -0800 Jakub Kicinski wrote:
Quoted 6 lines
Quoted 3 lines

Not sure if there's much we can do about that.. Perhaps we should have a shrinker that flushes the defer queues? I chatted with Shakeel briefly and it sounded fairly straightforward.

I was mostly concerned about latency spikes, I did some tests here and this seems fine.

Looks like selftests run into the zerocopy Tx latency issue. I'll drop this version from patchwork..

Delaying zero copy forever is a bit of an annoyance. I believe the same thing can happen in net-next with UDP but I haven't tested to confirm.

I assume the attached patch is out of question since it came up before?

From ec1dcd1d542880fdaa126a5d4d57147ce69c80d3 Mon Sep 17 00:00:00 2001
From: Jakub Kicinski <[email protected]>
Date: Wed, 28 Jan 2026 15:01:27 -0800
Subject: [--tree name--] net: periodically flush the defer queues

Zero-copy skbs may sit in the defer skb queue forever. AFAIU we don't see this for UDP in SW tests because over veth we end up hitting skb_orphan_frags_rx(). But it should happen when we zero-copy Tx on a real interface, which frees skbs via napi_consume_skb(). Since commit 6471658dc66c ("udp: use skb_attempt_defer_free()") we will queue the skb with a ubuf to a remote core where it may never be freed.

If we make TCP defer freeing Tx skbs this will be even more obvious and trigger over veth. In TCP the stack is traversed by a skb clone, and freeing happens when the ACK comes in, so no lucky skb_orphan_frags_rx().

This patch attempts a periodic scrub of the defer queues. Hopefully perf overhead is negligible under normal load, even on huge machines, as we'll just check that timer is already pending vast majority of the time.

---
 Documentation/admin-guide/sysctl/net.rst |  13 ++
 include/net/hotdata.h                    |   1 +
 net/core/dev.h                           |   4 +
 net/core/dev.c                           |   1 +
 net/core/skbuff.c                        | 152 +++++++++++++++++++++++
 net/core/sysctl_net_core.c               |   8 ++
 6 files changed, 179 insertions(+)
Documentation/admin-guide/sysctl/net.rst +13 -0
index 19408da2390b..dbfae381818a 100644
@@ -368,6 +368,19 @@ by the cpu which allocated them.
 
 Default: 128
 
+skb_defer_timeout_us
+--------------------
+
+Timeout in microseconds for the safety timer that flushes deferred skb
+queues. When skbs are deferred to a CPU that never runs network softirq
+(e.g., application threads that allocated TX skbs for zero-copy sends),
+they may remain queued indefinitely. This timer periodically triggers
+an IPI to the affected CPUs to drain their defer queues.
+
+Setting this to 0 disables the safety timer.
+
+Default: 20000 (20 milliseconds)
+
 optmem_max
 ----------
 
include/net/hotdata.h +1 -0
index 6632b1aa7584..fb738da24fe6 100644
@@ -10,6 +10,7 @@
 struct skb_defer_node {
 	struct llist_head	defer_list;
 	atomic_long_t		defer_count;
+	u8			needs_flush;
 } ____cacheline_aligned_in_smp;
 
 /* Read mostly data used in network fast paths. */
net/core/dev.h +4 -0
index 98793a738f43..e481b5e67363 100644
@@ -366,6 +366,10 @@ static inline void napi_assert_will_not_race(const struct napi_struct *napi)
 
 void kick_defer_list_purge(unsigned int cpu);
 
+extern unsigned int sysctl_skb_defer_timeout_us;
+int sysctl_skb_defer_timeout(const struct ctl_table *table, int write,
+			     void *buffer, size_t *lenp, loff_t *ppos);
+
 #define XMIT_RECURSION_LIMIT	8
 
 #ifndef CONFIG_PREEMPT_RT
net/core/dev.c +1 -0
index 43de5af0d6ec..e6e004f56c58 100644
@@ -6801,6 +6801,7 @@ static void skb_defer_free_flush(void)
 		if (llist_empty(&sdn->defer_list))
 			continue;
 		atomic_long_set(&sdn->defer_count, 0);
+		WRITE_ONCE(sdn->needs_flush, 0);
 		free_list = llist_del_all(&sdn->defer_list);
 
 		llist_for_each_entry_safe(skb, next, free_list, ll_node) {
net/core/skbuff.c +152 -0

174 lines not on this page · raw message

net/core/sysctl_net_core.c +8 -0
index 03aea10073f0..af8528a2337c 100644
@@ -645,6 +645,14 @@ static struct ctl_table net_core_table[] = {
 		.proc_handler	= proc_dointvec_minmax,
 		.extra1		= SYSCTL_ZERO,
 	},
+	{
+		.procname	= "skb_defer_timeout_us",
+		.data		= &sysctl_skb_defer_timeout_us,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= sysctl_skb_defer_timeout,
+		.extra1		= SYSCTL_ZERO,
+	},
 };
 
 static struct ctl_table netns_core_table[] = {
Signature · 1 line
2.52.0
author maintainer
#7 lore raw
Quoted 6 lines · On Thu, 29 Jan 2026 15:04:59 -0800 Jakub Kicinski wrote:
Quoted 5 lines
Quoted 2 lines

I was mostly concerned about latency spikes, I did some tests here and this seems fine.

Looks like selftests run into the zerocopy Tx latency issue. I'll drop this version from patchwork..

Quoted 7 lines

Delaying zero copy forever is a bit of an annoyance. I believe the same thing can happen in net-next with UDP but I haven't tested to confirm.

I assume the attached patch is out of question since it came up before?

[0001-net-periodically-flush-the-defer-queues.patch text/x-patch (9441 bytes)]

Ugh, too many patches in my /tmp. I attached an old buggy version, but you get the point.

#8 lore raw
In reply to Jakub Kicinski
Quoted 9 lines · Jakub Kicinski
Quoted 7 lines · On Mon, 19 Jan 2026 09:04:35 -0800 Jakub Kicinski wrote:
Quoted 6 lines
Quoted 3 lines

Not sure if there's much we can do about that.. Perhaps we should have a shrinker that flushes the defer queues? I chatted with Shakeel briefly and it sounded fairly straightforward.

I was mostly concerned about latency spikes, I did some tests here and this seems fine.

Quoted 8 lines
Quoted 2 lines

Looks like selftests run into the zerocopy Tx latency issue. I'll drop this version from patchwork..

Delaying zero copy forever is a bit of an annoyance. I believe the same thing can happen in net-next with UDP but I haven't tested to confirm.

I assume the attached patch is out of question since it came up before?

I think I totally missed your email :/

What about not attempting defer for zero copy skbs ?

It turns out existing patch e20dfbad8aab ("net: fix napi_consume_skb() with alien skbs")
is already a problem for zcopy.

net/core/skbuff.c +0 -0
index 699c401a5eae9c497a42b6bdd8593af7890529f4..dc47d3efc72ed86dce5e382d505eda7bc863669a

100644

net/core/skbuff.c +6 -1
@@ -7266,10 +7266,15 @@ void skb_attempt_defer_free(struct sk_buff *skb)
 {
        struct skb_defer_node *sdn;
        unsigned long defer_count;
-       int cpu = skb->alloc_cpu;
        unsigned int defer_max;
        bool kick;
+       int cpu;
 
+       /* zero copy notifications should not be delayed. */
+       if (skb_zcopy(skb))
+               goto nodefer;
+
+       cpu = skb->alloc_cpu;
        if (cpu == raw_smp_processor_id() ||
            WARN_ON_ONCE(cpu >= nr_cpu_ids) ||
            !cpu_online(cpu)) {
author maintainer
#9 lore raw
Quoted 7 lines · On Mon, 16 Feb 2026 17:06:58 +0100 Eric Dumazet wrote:
Quoted 7 lines
Quoted 2 lines

Looks like selftests run into the zerocopy Tx latency issue. I'll drop this version from patchwork..

Delaying zero copy forever is a bit of an annoyance. I believe the same thing can happen in net-next with UDP but I haven't tested to confirm.

Quoted 8 lines
Quoted 1 line

I assume the attached patch is out of question since it came up before?

I think I totally missed your email :/

What about not attempting defer for zero copy skbs ?

It turns out existing patch e20dfbad8aab ("net: fix napi_consume_skb() with alien skbs") is already a problem for zcopy.

We definitely need either this or the timer patch I attached, for UDP.

I put the TCP write side on a back burner because slab sheaves got merged, I think skb defer free will still make a difference but IDK how much. Maybe juice will no longer be worth the squeeze?

#10 lore raw
Quoted 15 lines · Jakub Kicinski
Quoted 13 lines · On Mon, 16 Feb 2026 17:06:58 +0100 Eric Dumazet wrote:
Quoted 8 lines
Quoted 2 lines

Looks like selftests run into the zerocopy Tx latency issue. I'll drop this version from patchwork..

Delaying zero copy forever is a bit of an annoyance. I believe the same thing can happen in net-next with UDP but I haven't tested to confirm.

I assume the attached patch is out of question since it came up before?

I think I totally missed your email :/

What about not attempting defer for zero copy skbs ?

Quoted 8 lines
Quoted 2 lines

It turns out existing patch e20dfbad8aab ("net: fix napi_consume_skb() with alien skbs") is already a problem for zcopy.

We definitely need either this or the timer patch I attached, for UDP.

I put the TCP write side on a back burner because slab sheaves got merged, I think skb defer free will still make a difference but IDK how much. Maybe juice will no longer be worth the squeeze?

It depends how many frags are attached to each skb.

page frags are not yet handled by SLUB sheaves :)

author maintainer
#11 lore raw
Quoted 9 lines · On Mon, 16 Feb 2026 18:58:08 +0100 Eric Dumazet wrote:
Quoted 9 lines
Quoted 6 lines

I think I totally missed your email :/

What about not attempting defer for zero copy skbs ?

It turns out existing patch e20dfbad8aab ("net: fix napi_consume_skb() with alien skbs") is already a problem for zcopy.

We definitely need either this or the timer patch I attached, for UDP.

Quoted 7 lines
Quoted 3 lines

I put the TCP write side on a back burner because slab sheaves got merged, I think skb defer free will still make a difference but IDK how much. Maybe juice will no longer be worth the squeeze?

It depends how many frags are attached to each skb.

page frags are not yet handled by SLUB sheaves :)

Ack, my recollection is that more of the cycles are spent in slab than pcp handling, tho. I could be misremembering. But my thinking was that if we both:
- lose zc sends due to the unbounded completion time
- lose the slab benefit due to sheaves
we are only left with fairly narrow case of page handling of small skbs. My gut feeling was that however unclean the timer fix would be best since it gives us zc back. But I don't have a way to experiment with sheaves yet to get data. All of this is pure speculation.

Regardless the patch you shared earlier is probably best as a fix for zero-copy UDP for now.

#12 lore raw
Quoted 29 lines · Jakub Kicinski
Quoted 9 lines · On Mon, 16 Feb 2026 18:58:08 +0100 Eric Dumazet wrote:
Quoted 9 lines
Quoted 6 lines

I think I totally missed your email :/

What about not attempting defer for zero copy skbs ?

It turns out existing patch e20dfbad8aab ("net: fix napi_consume_skb() with alien skbs") is already a problem for zcopy.

We definitely need either this or the timer patch I attached, for UDP.

Quoted 7 lines
Quoted 3 lines

I put the TCP write side on a back burner because slab sheaves got merged, I think skb defer free will still make a difference but IDK how much. Maybe juice will no longer be worth the squeeze?

It depends how many frags are attached to each skb.

page frags are not yet handled by SLUB sheaves :)

Ack, my recollection is that more of the cycles are spent in slab than pcp handling, tho. I could be misremembering. But my thinking was that if we both:
- lose zc sends due to the unbounded completion time
- lose the slab benefit due to sheaves
we are only left with fairly narrow case of page handling of small skbs. My gut feeling was that however unclean the timer fix would be best since it gives us zc back. But I don't have a way to experiment with sheaves yet to get data. All of this is pure speculation.

Quoted 2 lines

Regardless the patch you shared earlier is probably best as a fix for zero-copy UDP for now.

I am cooking a formal patch, but I do not see why UDP is a problem today ?

Definitely I am seeing problems with e20dfbad8aab ("net: fix napi_consume_skb() with alien skbs"), so was tempted to use it for the FIxes: tag.

This is the first time a TX skb would be potentially delayed.

UDP use of skb_attempt_defer_free() is with RX skbs, and normally their zcopy status has been cleared ?

Thanks !

author maintainer
#13 lore raw
Quoted 2 lines · On Mon, 16 Feb 2026 19:16:46 +0100 Eric Dumazet wrote:

UDP use of skb_attempt_defer_free() is with RX skbs, and normally their zcopy status has been cleared ?

I meant coming in via napi_consume_skb() -> skb_attempt_defer_free(). There we are freeing Tx skbs, which for UDP these may legitimately have zc state. No?

#14 lore raw
Quoted 8 lines · Jakub Kicinski
Quoted 2 lines · On Mon, 16 Feb 2026 19:16:46 +0100 Eric Dumazet wrote:

UDP use of skb_attempt_defer_free() is with RX skbs, and normally their zcopy status has been cleared ?

I meant coming in via napi_consume_skb() -> skb_attempt_defer_free(). There we are freeing Tx skbs, which for UDP these may legitimately have zc state. No?

Ah, I guess I was referring to a kernel before e20dfbad8aab ("net:
fix napi_consume_skb()
with alien skbs")

I guess we agree then ;)