tcp: try to defer / return acked skbs to originating CPU
Subsystems (4)
Jakub Kicinski
maintainerPATCH 1/1 tcp: try to defer / return acked skbs to originating CPU
Running a memcache-like workload under production(ish) load on a 300 thread AMD machine we see ~3% of CPU time spent in kmem_cache_free() via tcp_ack(), freeing skbs from rtx queue. This workloads pins workers away from softirq CPU so the Tx skbs are pretty much always allocated on a different CPU than where the ACKs arrive. Try to use the defer skb free queue to return the skbs back to where they came from. This results in a ~4% performance improvement for the workload.
--- include/net/tcp.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/net/tcp.h +1 -1
@@ -332,7 +332,7 @@ static inline void tcp_wmem_free_skb(struct sock *sk, struct sk_buff *skb)sk_mem_uncharge (sk, skb->truesize); elsesk_mem_uncharge (sk ,SKB_TRUESIZE (skb_end_offset (skb ))); -__kfree_skb (skb); +skb_attempt_defer_free (skb); }void sk_forced_mem_schedule (struct sock *sk,int size);
Signature · 1 line
2.52.0
Jakub Kicinski
maintainerQuoted 8 lines · On Sat, 17 Jan 2026 08:42:55 -0800 Jakub Kicinski wrote:
Running a memcache-like workload under production(ish) load on a 300 thread AMD machine we see ~3% of CPU time spent in
kmem_cache_free()viatcp_ack(), freeing skbs from rtx queue. This workloads pins workers away from softirq CPU so the Tx skbs are pretty much always allocated on a different CPU than where the ACKs arrive. Try to use the defer skb free queue to return the skbs back to where they came from. This results in a ~4% performance improvement for the workload.
In the interest of full transparency the performance testing was done on a 6.13-ish kernel. But I don't see anything that'd make the situation better upstream..
Quoted 10 lines · Jakub Kicinski
Running a memcache-like workload under production(ish) load on a 300 thread AMD machine we see ~3% of CPU time spent in
kmem_cache_free()viatcp_ack(), freeing skbs from rtx queue. This workloads pins workers away from softirq CPU so the Tx skbs are pretty much always allocated on a different CPU than where the ACKs arrive. Try to use the defer skb free queue to return the skbs back to where they came from. This results in a ~4% performance improvement for the workload.
This probably makes sense when RFS is not used.
Here, RFS gives us ~40% performance improvement for typical RPC workloads, so I never took a look at this side :)
Have you tested what happens for bulk sends ?sendmsg() allocates skbs and push them to transmit queue, but ACK can decide to split TSO packets, and the new allocation is done on the softirq CPU (assuming RFS is not used)
Perhaps tso_fragment()/tcp_fragment() could copy the source skb->alloc_cpu to (new)buff->alloc_cpu.
Also, if workers are away from softirq, they will only process the defer queue in large patches, after receiving an trigger_rx_softirq() IPI.
Any idea of skb_defer_free_flush() latency when dealing with batches of ~64 big TSO packets ?
Quoted 21 lines
Signed-off-by Jakub Kicinski <[email protected]>--- include/net/tcp.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)include/net/tcp.h +1
-2@@ -332,7 +332,7 @@ static inline void tcp_wmem_free_skb(struct sock *sk, struct sk_buff *skb)sk_mem_uncharge (sk, skb->truesize); elsesk_mem_uncharge (sk ,SKB_TRUESIZE (skb_end_offset (skb ))); -__kfree_skb (skb); +skb_attempt_defer_free (skb); }void sk_forced_mem_schedule (struct sock *sk,int size); --2.52.0
Jakub Kicinski
maintainerQuoted 10 lines · On Sat, 17 Jan 2026 19:16:57 +0100 Eric Dumazet wrote:
Quoted 8 lines · Jakub Kicinski
Running a memcache-like workload under production(ish) load on a 300 thread AMD machine we see ~3% of CPU time spent in
kmem_cache_free()viatcp_ack(), freeing skbs from rtx queue. This workloads pins workers away from softirq CPU so the Tx skbs are pretty much always allocated on a different CPU than where the ACKs arrive. Try to use the defer skb free queue to return the skbs back to where they came from. This results in a ~4% performance improvement for the workload.
Quoted 3 lines
This probably makes sense when RFS is not used.
Here, RFS gives us ~40% performance improvement for typical RPC workloads, so I never took a look at this side :)
This workload doesn't like RFS. Maybe because it has 1M sockets.. I'll need to look closer, the patchwork queue first tho.. :)
Quoted 7 lines
Have you tested what happens for bulk sends ?
sendmsg()allocates skbs and push them to transmit queue, but ACK can decide to split TSO packets, and the new allocation is done on the softirq CPU (assuming RFS is not used)Perhaps tso_fragment()/tcp_fragment() could copy the source
skb->alloc_cputo (new)buff->alloc_cpu.
I'll do some synthetic testing and get back.
Quoted 5 lines
Also, if workers are away from softirq, they will only process the defer queue in large patches, after receiving an
trigger_rx_softirq()IPI.
Any idea ofskb_defer_free_flush()latency when dealing with batches of ~64 big TSO packets ?
Not sure if there's much we can do about that.. Perhaps we should have a shrinker that flushes the defer queues? I chatted with Shakeel briefly and it sounded fairly straightforward.
Eric Dumazet
Reviewed-byQuoted 35 lines · Jakub Kicinski
Quoted 10 lines · On Sat, 17 Jan 2026 19:16:57 +0100 Eric Dumazet wrote:
Quoted 8 lines · Jakub Kicinski
Running a memcache-like workload under production(ish) load on a 300 thread AMD machine we see ~3% of CPU time spent in
kmem_cache_free()viatcp_ack(), freeing skbs from rtx queue. This workloads pins workers away from softirq CPU so the Tx skbs are pretty much always allocated on a different CPU than where the ACKs arrive. Try to use the defer skb free queue to return the skbs back to where they came from. This results in a ~4% performance improvement for the workload.Quoted 3 lines
This probably makes sense when RFS is not used.
Here, RFS gives us ~40% performance improvement for typical RPC workloads, so I never took a look at this side :)This workload doesn't like RFS. Maybe because it has 1M sockets.. I'll need to look closer, the patchwork queue first tho.. :)
Quoted 7 lines
Have you tested what happens for bulk sends ?
sendmsg()allocates skbs and push them to transmit queue, but ACK can decide to split TSO packets, and the new allocation is done on the softirq CPU (assuming RFS is not used)Perhaps tso_fragment()/tcp_fragment() could copy the source
skb->alloc_cputo (new)buff->alloc_cpu.I'll do some synthetic testing and get back.
Quoted 5 lines
Also, if workers are away from softirq, they will only process the defer queue in large patches, after receiving an
trigger_rx_softirq()IPI.
Any idea ofskb_defer_free_flush()latency when dealing with batches of ~64 big TSO packets ?
Quoted 3 lines
Not sure if there's much we can do about that.. Perhaps we should have a shrinker that flushes the defer queues? I chatted with Shakeel briefly and it sounded fairly straightforward.
I was mostly concerned about latency spikes, I did some tests here and this seems fine.
(I assume you asked Shakeel about the extra memory being held in the per-cpu queue, and pcp implications ?)
Jakub Kicinski
maintainerQuoted 6 lines · On Sun, 18 Jan 2026 13:15:00 +0100 Eric Dumazet wrote:
Quoted 6 lines
Quoted 5 lines
Also, if workers are away from softirq, they will only process the defer queue in large patches, after receiving an
trigger_rx_softirq()IPI.
Any idea ofskb_defer_free_flush()latency when dealing with batches of ~64 big TSO packets ?
Quoted 6 lines
Quoted 3 lines
Not sure if there's much we can do about that.. Perhaps we should have a shrinker that flushes the defer queues? I chatted with Shakeel briefly and it sounded fairly straightforward.
I was mostly concerned about latency spikes, I did some tests here and this seems fine.
Looks like selftests run into the zerocopy Tx latency issue. I'll drop this version from patchwork..
Quoted 2 lines
(I assume you asked Shakeel about the extra memory being held in the per-cpu queue, and pcp implications ?)
Under real load it helps quite a bit but real load flushes the queues frequently. I'll talk to him.
Jakub Kicinski
maintainerQuoted 9 lines · On Mon, 19 Jan 2026 09:04:35 -0800 Jakub Kicinski wrote:
Quoted 6 lines
Quoted 3 lines
Not sure if there's much we can do about that.. Perhaps we should have a shrinker that flushes the defer queues? I chatted with Shakeel briefly and it sounded fairly straightforward.
I was mostly concerned about latency spikes, I did some tests here and this seems fine.
Looks like selftests run into the zerocopy Tx latency issue. I'll drop this version from patchwork..
Delaying zero copy forever is a bit of an annoyance. I believe the same thing can happen in net-next with UDP but I haven't tested to confirm.
I assume the attached patch is out of question since it came up before?
From ec1dcd1d542880fdaa126a5d4d57147ce69c80d3 Mon Sep 17 00:00:00 2001
From: Jakub Kicinski <[email protected]>
Date: Wed, 28 Jan 2026 15:01:27 -0800
Subject: [--tree name--] net: periodically flush the defer queues
Zero-copy skbs may sit in the defer skb queue forever. AFAIU we don't see this for UDP in SW tests because over veth we end up hitting skb_orphan_frags_rx(). But it should happen when we zero-copy Tx on a real interface, which frees skbs via napi_consume_skb(). Since commit 6471658dc66c ("udp: use skb_attempt_defer_free()") we will queue the skb with a ubuf to a remote core where it may never be freed.
If we make TCP defer freeing Tx skbs this will be even more obvious and trigger over veth. In TCP the stack is traversed by a skb clone, and freeing happens when the ACK comes in, so no lucky skb_orphan_frags_rx().
This patch attempts a periodic scrub of the defer queues. Hopefully perf overhead is negligible under normal load, even on huge machines, as we'll just check that timer is already pending vast majority of the time.
--- Documentation/admin-guide/sysctl/net.rst | 13 ++ include/net/hotdata.h | 1 + net/core/dev.h | 4 + net/core/dev.c | 1 + net/core/skbuff.c | 152 +++++++++++++++++++++++ net/core/sysctl_net_core.c | 8 ++ 6 files changed, 179 insertions(+)
Documentation/admin-guide/sysctl/net.rst +13 -0
@@ -368,6 +368,19 @@ by the cpu which allocated them. Default: 128 +skb_defer_timeout_us +-------------------- + +Timeout in microseconds for the safety timer that flushes deferred skb +queues. When skbs are deferred to a CPU that never runs network softirq +(e.g., application threads that allocated TX skbs for zero-copy sends), +they may remain queued indefinitely. This timer periodically triggers +an IPI to the affected CPUs to drain their defer queues. + +Setting this to 0 disables the safety timer. + +Default: 20000 (20 milliseconds) + optmem_max ----------
include/net/hotdata.h +1 -0
@@ -10,6 +10,7 @@struct skb_defer_node {struct llist_head defer_list;atomic_long_t defer_count; +u8 needs_flush; }____cacheline_aligned_in_smp ;/* Read mostly data used in network fast paths. */
net/core/dev.h +4 -0
@@ -366,6 +366,10 @@ static inline void napi_assert_will_not_race(const struct napi_struct *napi)void kick_defer_list_purge (unsigned int cpu); +extern unsigned int sysctl_skb_defer_timeout_us; +int sysctl_skb_defer_timeout (const struct ctl_table *table,int write, +void *buffer,size_t *lenp,loff_t *ppos); +#define XMIT_RECURSION_LIMIT 8#ifndef CONFIG_PREEMPT_RT
net/core/dev.c +1 -0
@@ -6801,6 +6801,7 @@ static void skb_defer_free_flush(void)if (llist_empty (&sdn->defer_list))continue ;atomic_long_set (&sdn->defer_count,0 ); +WRITE_ONCE (sdn->needs_flush,0 ); free_list =llist_del_all (&sdn->defer_list);llist_for_each_entry_safe (skb, next, free_list, ll_node) {
net/core/skbuff.c +152 -0
174 lines not on this page · raw message
net/core/sysctl_net_core.c +8 -0
@@ -645,6 +645,14 @@ static struct ctl_table net_core_table[] = { .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO, }, + { + .procname = "skb_defer_timeout_us", + .data = &sysctl_skb_defer_timeout_us, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = sysctl_skb_defer_timeout, + .extra1 = SYSCTL_ZERO, + }, };static struct ctl_table netns_core_table[] = {
Signature · 1 line
2.52.0
Jakub Kicinski
maintainerQuoted 6 lines · On Thu, 29 Jan 2026 15:04:59 -0800 Jakub Kicinski wrote:
Quoted 5 lines
Quoted 2 lines
I was mostly concerned about latency spikes, I did some tests here and this seems fine.
Looks like selftests run into the zerocopy Tx latency issue. I'll drop this version from patchwork..
Quoted 7 lines
Delaying zero copy forever is a bit of an annoyance. I believe the same thing can happen in net-next with UDP but I haven't tested to confirm.
I assume the attached patch is out of question since it came up before?
[0001-net-periodically-flush-the-defer-queues.patch text/x-patch (9441 bytes)]
Ugh, too many patches in my /tmp. I attached an old buggy version, but you get the point.
Quoted 9 lines · Jakub Kicinski
Quoted 7 lines · On Mon, 19 Jan 2026 09:04:35 -0800 Jakub Kicinski wrote:
Quoted 6 lines
Quoted 3 lines
Not sure if there's much we can do about that.. Perhaps we should have a shrinker that flushes the defer queues? I chatted with Shakeel briefly and it sounded fairly straightforward.
I was mostly concerned about latency spikes, I did some tests here and this seems fine.
Quoted 8 lines
Quoted 2 lines
Looks like selftests run into the zerocopy Tx latency issue. I'll drop this version from patchwork..
Delaying zero copy forever is a bit of an annoyance. I believe the same thing can happen in net-next with UDP but I haven't tested to confirm.
I assume the attached patch is out of question since it came up before?
I think I totally missed your email :/
What about not attempting defer for zero copy skbs ?
It turns out existing patch e20dfbad8aab ("net: fix napi_consume_skb() with alien skbs")
is already a problem for zcopy.
net/core/skbuff.c +0 -0
100644
net/core/skbuff.c +6 -1
@@ -7266,10 +7266,15 @@ void skb_attempt_defer_free(struct sk_buff *skb) {struct skb_defer_node *sdn;unsigned long defer_count; -int cpu = skb->alloc_cpu;unsigned int defer_max;bool kick; +int cpu; +/* zero copy notifications should not be delayed. */ +if (skb_zcopy (skb)) + goto nodefer; + + cpu = skb->alloc_cpu;if (cpu ==raw_smp_processor_id () ||WARN_ON_ONCE (cpu >= nr_cpu_ids) || !cpu_online (cpu)) {
Jakub Kicinski
maintainerQuoted 7 lines · On Mon, 16 Feb 2026 17:06:58 +0100 Eric Dumazet wrote:
Quoted 7 lines
Quoted 2 lines
Looks like selftests run into the zerocopy Tx latency issue. I'll drop this version from patchwork..
Delaying zero copy forever is a bit of an annoyance. I believe the same thing can happen in net-next with UDP but I haven't tested to confirm.
Quoted 8 lines
Quoted 1 line
I assume the attached patch is out of question since it came up before?
I think I totally missed your email :/
What about not attempting defer for zero copy skbs ?
It turns out existing patch
e20dfbad8aab("net: fixnapi_consume_skb()with alien skbs") is already a problem for zcopy.
We definitely need either this or the timer patch I attached, for UDP.
I put the TCP write side on a back burner because slab sheaves got merged, I think skb defer free will still make a difference but IDK how much. Maybe juice will no longer be worth the squeeze?
Quoted 15 lines · Jakub Kicinski
Quoted 13 lines · On Mon, 16 Feb 2026 17:06:58 +0100 Eric Dumazet wrote:
Quoted 8 lines
Quoted 2 lines
Looks like selftests run into the zerocopy Tx latency issue. I'll drop this version from patchwork..
Delaying zero copy forever is a bit of an annoyance. I believe the same thing can happen in net-next with UDP but I haven't tested to confirm.
I assume the attached patch is out of question since it came up before?
I think I totally missed your email :/
What about not attempting defer for zero copy skbs ?
Quoted 8 lines
Quoted 2 lines
It turns out existing patch
e20dfbad8aab("net: fixnapi_consume_skb()with alien skbs") is already a problem for zcopy.We definitely need either this or the timer patch I attached, for UDP.
I put the TCP write side on a back burner because slab sheaves got merged, I think skb defer free will still make a difference but IDK how much. Maybe juice will no longer be worth the squeeze?
It depends how many frags are attached to each skb.
page frags are not yet handled by SLUB sheaves :)
Jakub Kicinski
maintainerQuoted 9 lines · On Mon, 16 Feb 2026 18:58:08 +0100 Eric Dumazet wrote:
Quoted 9 lines
Quoted 6 lines
I think I totally missed your email :/
What about not attempting defer for zero copy skbs ?
It turns out existing patch
e20dfbad8aab("net: fixnapi_consume_skb()with alien skbs") is already a problem for zcopy.We definitely need either this or the timer patch I attached, for UDP.
Quoted 7 lines
Quoted 3 lines
I put the TCP write side on a back burner because slab sheaves got merged, I think skb defer free will still make a difference but IDK how much. Maybe juice will no longer be worth the squeeze?
It depends how many frags are attached to each skb.
page frags are not yet handled by SLUB sheaves :)
Ack, my recollection is that more of the cycles are spent in slab than pcp handling, tho. I could be misremembering. But my thinking was that if we both:
- lose zc sends due to the unbounded completion time
- lose the slab benefit due to sheaves
we are only left with fairly narrow case of page handling of small skbs. My gut feeling was that however unclean the timer fix would be best since it gives us zc back. But I don't have a way to experiment with sheaves yet to get data. All of this is pure speculation.
Regardless the patch you shared earlier is probably best as a fix for zero-copy UDP for now.
Quoted 29 lines · Jakub Kicinski
Quoted 9 lines · On Mon, 16 Feb 2026 18:58:08 +0100 Eric Dumazet wrote:
Quoted 9 lines
Quoted 6 lines
I think I totally missed your email :/
What about not attempting defer for zero copy skbs ?
It turns out existing patch
e20dfbad8aab("net: fixnapi_consume_skb()with alien skbs") is already a problem for zcopy.We definitely need either this or the timer patch I attached, for UDP.
Quoted 7 lines
Quoted 3 lines
I put the TCP write side on a back burner because slab sheaves got merged, I think skb defer free will still make a difference but IDK how much. Maybe juice will no longer be worth the squeeze?
It depends how many frags are attached to each skb.
page frags are not yet handled by SLUB sheaves :)
Ack, my recollection is that more of the cycles are spent in slab than pcp handling, tho. I could be misremembering. But my thinking was that if we both:
- lose zc sends due to the unbounded completion time
- lose the slab benefit due to sheaves
we are only left with fairly narrow case of page handling of small skbs. My gut feeling was that however unclean the timer fix would be best since it gives us zc back. But I don't have a way to experiment with sheaves yet to get data. All of this is pure speculation.
Quoted 2 lines
Regardless the patch you shared earlier is probably best as a fix for zero-copy UDP for now.
I am cooking a formal patch, but I do not see why UDP is a problem today ?
Definitely I am seeing problems with e20dfbad8aab ("net: fix napi_consume_skb() with alien skbs"), so was tempted to use it for the FIxes: tag.
This is the first time a TX skb would be potentially delayed.
UDP use of skb_attempt_defer_free() is with RX skbs, and normally their zcopy status has been cleared ?
Thanks !
Jakub Kicinski
maintainerQuoted 2 lines · On Mon, 16 Feb 2026 19:16:46 +0100 Eric Dumazet wrote:
UDP use of
skb_attempt_defer_free()is with RX skbs, and normally their zcopy status has been cleared ?
I meant coming in via napi_consume_skb() -> skb_attempt_defer_free(). There we are freeing Tx skbs, which for UDP these may legitimately have zc state. No?
Quoted 8 lines · Jakub Kicinski
Quoted 2 lines · On Mon, 16 Feb 2026 19:16:46 +0100 Eric Dumazet wrote:
UDP use of
skb_attempt_defer_free()is with RX skbs, and normally their zcopy status has been cleared ?I meant coming in via
napi_consume_skb()->skb_attempt_defer_free(). There we are freeing Tx skbs, which for UDP these may legitimately have zc state. No?
Ah, I guess I was referring to a kernel before e20dfbad8aab ("net:
fix napi_consume_skb()
with alien skbs")
I guess we agree then ;)