tcp: move sk_forced_mem_schedule() to tcp.c
Subsystems (3)
PATCH 1/1 tcp: move sk_forced_mem_schedule() to tcp.c
TCP fast path can (auto)inline this helper, instead of (auto)inling it from tcp_send_fin().
No change of overall code size, but tcp_sendmsg() is faster.
$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1) Function old new delta tcp_stream_alloc_skb 216 357 +141 tcp_send_fin 688 548 -140 Total: Before=22236729, After=22236730, chg +0.00%
BTW, we might change tcp_send_fin() to use tcp_stream_alloc_skb().
--- net/ipv4/tcp.c | 27 +++++++++++++++++++++++++++ net/ipv4/tcp_output.c | 27 --------------------------- 2 files changed, 27 insertions(+), 27 deletions(-)
net/ipv4/tcp.c +27 -0
@@ -902,6 +902,33 @@ ssize_t tcp_splice_read(struct socket *sock, loff_t *ppos, }EXPORT_IPV6_MOD (tcp_splice_read); +/* We allow to exceed memory limits for FIN packets to expedite +* connection tear down and (memory) recovery. +* Otherwise tcp_send_fin() could be tempted to either delay FIN +* or even be forced to close flow without any FIN. +* In general, we want to allow one skb per socket to avoid hangs +* with edge trigger epoll() +*/ +void sk_forced_mem_schedule (struct sock *sk,int size) +{ +int delta, amt; + + delta = size - sk->sk_forward_alloc; +if (delta <=0 ) +return ; + + amt =sk_mem_pages (delta); +sk_forward_alloc_add (sk, amt <<PAGE_SHIFT ); + +if (mem_cgroup_sk_enabled (sk)) +mem_cgroup_sk_charge (sk, amt,gfp_memcg_charge () | __GFP_NOFAIL); + +if (sk->sk_bypass_prot_mem) +return ; + +sk_memory_allocated_add (sk, amt); +} +struct sk_buff *tcp_stream_alloc_skb (struct sock *sk,gfp_t gfp,bool force_schedule) {
net/ipv4/tcp_output.c +0 -27
@@ -3767,33 +3767,6 @@ void tcp_xmit_retransmit_queue(struct sock *sk) inet_csk(sk)->icsk_rto, true); } -/* We allow to exceed memory limits for FIN packets to expedite -* connection tear down and (memory) recovery. -* Otherwise tcp_send_fin() could be tempted to either delay FIN -* or even be forced to close flow without any FIN. -* In general, we want to allow one skb per socket to avoid hangs -* with edge trigger epoll() -*/ -void sk_forced_mem_schedule (struct sock *sk,int size) -{ -int delta, amt; - - delta = size - sk->sk_forward_alloc; -if (delta <=0 ) -return ; - - amt =sk_mem_pages (delta); -sk_forward_alloc_add (sk, amt <<PAGE_SHIFT ); - -if (mem_cgroup_sk_enabled (sk)) -mem_cgroup_sk_charge (sk, amt,gfp_memcg_charge () | __GFP_NOFAIL); - -if (sk->sk_bypass_prot_mem) -return ; - -sk_memory_allocated_add (sk, amt); -} -/* Send a FIN. The caller locks the socket for us. * We should try to send a FIN packet really hard, but eventually give up. */
Signature · 1 line
2.52.0.457.g6b5491de43-goog
Quoted 13 lines · Eric Dumazet
TCP fast path can (auto)inline this helper, instead of (auto)inling it from
tcp_send_fin().No change of overall code size, but
tcp_sendmsg()is faster.$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1) Function old new delta tcp_stream_alloc_skb 216 357 +141 tcp_send_fin 688 548 -140 Total: Before=22236729, After=22236730, chg +0.00%
Quoted 4 lines
BTW, we might change
tcp_send_fin()to usetcp_stream_alloc_skb().Signed-off-by Eric Dumazet <[email protected]>---
Thanks, Eric!
neal
Paolo Abeni
maintainerHi Eric,
Quoted 1 line · Eric Dumazet
TCP fast path can (auto)inline this helper, instead
of (auto)inling it from tcp_send_fin().
Quoted 10 lines
Quoted 2 lines
No change of overall code size, but
tcp_sendmsg()is faster.$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1) Function old new delta tcp_stream_alloc_skb 216 357 +141 tcp_send_fin 688 548 -140 Total: Before=22236729, After=22236730, chg +0.00%
Quoted 3 lines
BTW, we might change
tcp_send_fin()to usetcp_stream_alloc_skb().Signed-off-by Eric Dumazet <[email protected]>
I've a question out of sheer curiosity: are you using some specific tool to look for inline opportunity, or "just" careful code and/or objdump analysis?
Thanks,
Paolo
Quoted 16 lines · Paolo Abeni
Hi Eric,
Quoted 1 line · Eric Dumazet
TCP fast path can (auto)inline this helper, instead
of (auto)inling it from
tcp_send_fin().Quoted 10 lines
Quoted 2 lines
No change of overall code size, but
tcp_sendmsg()is faster.$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1) Function old new delta tcp_stream_alloc_skb 216 357 +141 tcp_send_fin 688 548 -140 Total: Before=22236729, After=22236730, chg +0.00%
Quoted 7 lines
Quoted 3 lines
BTW, we might change
tcp_send_fin()to usetcp_stream_alloc_skb().Signed-off-by Eric Dumazet <[email protected]>I've a question out of sheer curiosity: are you using some specific tool to look for inline opportunity, or "just" careful code and/or objdump analysis?
I am studying performance profiles on a stress test using Google production kernels,
on platforms that are a big chunk of the fleet.
These kernels are very close to upstream (at least for core and TCP networking stacks), and use clang and FDO.
I am currently focusing on some functions that even FDO does not inline, for some reasons.
Thanks !
Hi Eric,
Quoted 18 lines · Eric Dumazet
Quoted 16 lines · Paolo Abeni
Hi Eric,
Quoted 1 line · Eric Dumazet
TCP fast path can (auto)inline this helper, instead
of (auto)inling it from
tcp_send_fin().Quoted 10 lines
Quoted 2 lines
No change of overall code size, but
tcp_sendmsg()is faster.$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1) Function old new delta tcp_stream_alloc_skb 216 357 +141 tcp_send_fin 688 548 -140 Total: Before=22236729, After=22236730, chg +0.00%
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
BTW, we might change
tcp_send_fin()to usetcp_stream_alloc_skb().Signed-off-by Eric Dumazet <[email protected]>I've a question out of sheer curiosity: are you using some specific tool to look for inline opportunity, or "just" careful code and/or objdump analysis?
Paolo, right, you beat me to it! I'm also curious :)
Quoted 9 lines
I am studying performance profiles on a stress test using Google production kernels,
on platforms that are a big chunk of the fleet.
These kernels are very close to upstream (at least for core and TCP networking stacks), and use clang and FDO.I am currently focusing on some functions that even FDO does not inline, for some reasons.
Eric, could you share your more valuable experience and methodology behind this?
Prior to the recent work you've done, I completely missed that we could even do something with the inline functions. Normally using perf doesn't give us a clear micro observation of those functions especially about some hints on whether we should inline some functions. If there is any tool, it would be super awesome!
Let me guess, are you trying to avoid function call action as much as you can in the hot path based on your deep understanding of ingress/egress performance?
Thanks,
Jason
Quoted 42 lines · Jason Xing
Hi Eric,
Quoted 18 lines · Eric Dumazet
Quoted 16 lines · Paolo Abeni
Hi Eric,
Quoted 1 line · Eric Dumazet
TCP fast path can (auto)inline this helper, instead
of (auto)inling it from
tcp_send_fin().Quoted 10 lines
Quoted 2 lines
No change of overall code size, but
tcp_sendmsg()is faster.$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1) Function old new delta tcp_stream_alloc_skb 216 357 +141 tcp_send_fin 688 548 -140 Total: Before=22236729, After=22236730, chg +0.00%Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
BTW, we might change
tcp_send_fin()to usetcp_stream_alloc_skb().Signed-off-by Eric Dumazet <[email protected]>I've a question out of sheer curiosity: are you using some specific tool to look for inline opportunity, or "just" careful code and/or objdump analysis?
Paolo, right, you beat me to it! I'm also curious :)
Quoted 9 lines
I am studying performance profiles on a stress test using Google production kernels,
on platforms that are a big chunk of the fleet.
These kernels are very close to upstream (at least for core and TCP networking stacks), and use clang and FDO.I am currently focusing on some functions that even FDO does not inline, for some reasons.
Quoted 8 lines
Eric, could you share your more valuable experience and methodology behind this?
Prior to the recent work you've done, I completely missed that we could even do something with the inline functions. Normally using perf doesn't give us a clear micro observation of those functions especially about some hints on whether we should inline some functions. If there is any tool, it would be super awesome!
perf is your friend, and a deep knowledge of your cpu-du-jour behavior (assembly)
perf record -C xxx -g sleep 10
perf report --no-children
<Use the perf UI appropriately>
Quoted 3 lines
Let me guess, are you trying to avoid function call action as much as you can in the hot path based on your deep understanding of ingress/egress performance?
I think you could answer the question by parsing the changelogs of my recent commits ;)
Quoted 59 lines · Eric Dumazet
Quoted 42 lines · Jason Xing
Hi Eric,
Quoted 18 lines · Eric Dumazet
Quoted 16 lines · Paolo Abeni
Hi Eric,
Quoted 1 line · Eric Dumazet
TCP fast path can (auto)inline this helper, instead
of (auto)inling it from
tcp_send_fin().Quoted 10 lines
Quoted 2 lines
No change of overall code size, but
tcp_sendmsg()is faster.$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1) Function old new delta tcp_stream_alloc_skb 216 357 +141 tcp_send_fin 688 548 -140 Total: Before=22236729, After=22236730, chg +0.00%Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
BTW, we might change
tcp_send_fin()to usetcp_stream_alloc_skb().Signed-off-by Eric Dumazet <[email protected]>I've a question out of sheer curiosity: are you using some specific tool to look for inline opportunity, or "just" careful code and/or objdump analysis?
Paolo, right, you beat me to it! I'm also curious :)
Quoted 9 lines
I am studying performance profiles on a stress test using Google production kernels,
on platforms that are a big chunk of the fleet.
These kernels are very close to upstream (at least for core and TCP networking stacks), and use clang and FDO.I am currently focusing on some functions that even FDO does not inline, for some reasons.
Quoted 8 lines
Eric, could you share your more valuable experience and methodology behind this?
Prior to the recent work you've done, I completely missed that we could even do something with the inline functions. Normally using perf doesn't give us a clear micro observation of those functions especially about some hints on whether we should inline some functions. If there is any tool, it would be super awesome!
perf is your friend, and a deep knowledge of your cpu-du-jour behavior (assembly)
perf record -C xxx -g sleep 10
perf report --no-children
Quoted 8 lines
<Use the perf UI appropriately>
Quoted 3 lines
Let me guess, are you trying to avoid function call action as much as you can in the hot path based on your deep understanding of ingress/egress performance?
I think you could answer the question by parsing the changelogs of my recent commits ;)
Right, I've already catched some insightful knowledge from your commits :) Thanks for the interesting work!
Thanks,
Jason
This patch was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Hello:
This patch was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:
Quoted 5 lines · On Fri, 23 Jan 2026 11:16:05 +0000 you wrote:
TCP fast path can (auto)inline this helper, instead of (auto)inling it from
tcp_send_fin().No change of overall code size, but
tcp_sendmsg()is faster.
Quoted 8 lines
$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1) Function old new delta tcp_stream_alloc_skb 216 357 +141 tcp_send_fin 688 548 -140 Total: Before=22236729, After=22236730, chg +0.00%[...]
Here is the summary with links:
- [net-next] tcp: move sk_forced_mem_schedule() to tcp.c
https://git.kernel.org/netdev/net-next/c/a18056a6c11c
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html