Skip to contentSkip to conversation

tcp: move sk_forced_mem_schedule() to tcp.c

PATCH Applied Eric Dumazet 5 participants 8 messages Jan 23 – Jan 27

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tcp: move sk_forced_mem_schedule() to tcp.c

TCP fast path can (auto)inline this helper, instead of (auto)inling it from tcp_send_fin().

No change of overall code size, but tcp_sendmsg() is faster.

$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new
add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1)
Function                                     old     new   delta
tcp_stream_alloc_skb                         216     357    +141
tcp_send_fin                                 688     548    -140
Total: Before=22236729, After=22236730, chg +0.00%

BTW, we might change tcp_send_fin() to use tcp_stream_alloc_skb().

---
 net/ipv4/tcp.c        | 27 +++++++++++++++++++++++++++
 net/ipv4/tcp_output.c | 27 ---------------------------
 2 files changed, 27 insertions(+), 27 deletions(-)
net/ipv4/tcp.c +27 -0
index 148cdf3cd6233add37ea52e273cb4fb3e75fcbcb..87acd01274601eca799e42b8f895fff4c35cf25a 100644
@@ -902,6 +902,33 @@ ssize_t tcp_splice_read(struct socket *sock, loff_t *ppos,
 }
 EXPORT_IPV6_MOD(tcp_splice_read);
 
+/* We allow to exceed memory limits for FIN packets to expedite
+ * connection tear down and (memory) recovery.
+ * Otherwise tcp_send_fin() could be tempted to either delay FIN
+ * or even be forced to close flow without any FIN.
+ * In general, we want to allow one skb per socket to avoid hangs
+ * with edge trigger epoll()
+ */
+void sk_forced_mem_schedule(struct sock *sk, int size)
+{
+	int delta, amt;
+
+	delta = size - sk->sk_forward_alloc;
+	if (delta <= 0)
+		return;
+
+	amt = sk_mem_pages(delta);
+	sk_forward_alloc_add(sk, amt << PAGE_SHIFT);
+
+	if (mem_cgroup_sk_enabled(sk))
+		mem_cgroup_sk_charge(sk, amt, gfp_memcg_charge() | __GFP_NOFAIL);
+
+	if (sk->sk_bypass_prot_mem)
+		return;
+
+	sk_memory_allocated_add(sk, amt);
+}
+
 struct sk_buff *tcp_stream_alloc_skb(struct sock *sk, gfp_t gfp,
 				     bool force_schedule)
 {
net/ipv4/tcp_output.c +0 -27
index 256b669e8d3b4a4d191e61e79784e412aaef8965..597e888af36d8697cd696964077b423e79fdb83e 100644
@@ -3767,33 +3767,6 @@ void tcp_xmit_retransmit_queue(struct sock *sk)
 				     inet_csk(sk)->icsk_rto, true);
 }
 
-/* We allow to exceed memory limits for FIN packets to expedite
- * connection tear down and (memory) recovery.
- * Otherwise tcp_send_fin() could be tempted to either delay FIN
- * or even be forced to close flow without any FIN.
- * In general, we want to allow one skb per socket to avoid hangs
- * with edge trigger epoll()
- */
-void sk_forced_mem_schedule(struct sock *sk, int size)
-{
-	int delta, amt;
-
-	delta = size - sk->sk_forward_alloc;
-	if (delta <= 0)
-		return;
-
-	amt = sk_mem_pages(delta);
-	sk_forward_alloc_add(sk, amt << PAGE_SHIFT);
-
-	if (mem_cgroup_sk_enabled(sk))
-		mem_cgroup_sk_charge(sk, amt, gfp_memcg_charge() | __GFP_NOFAIL);
-
-	if (sk->sk_bypass_prot_mem)
-		return;
-
-	sk_memory_allocated_add(sk, amt);
-}
-
 /* Send a FIN. The caller locks the socket for us.
  * We should try to send a FIN packet really hard, but eventually give up.
  */
Signature · 1 line
2.52.0.457.g6b5491de43-goog
maintainer Reviewed-by on v1 1/1
#1 lore raw
Quoted 13 lines · Eric Dumazet

TCP fast path can (auto)inline this helper, instead of (auto)inling it from tcp_send_fin().

No change of overall code size, but tcp_sendmsg() is faster.

$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new
add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1)
Function                                     old     new   delta
tcp_stream_alloc_skb                         216     357    +141
tcp_send_fin                                 688     548    -140
Total: Before=22236729, After=22236730, chg +0.00%
Quoted 4 lines

BTW, we might change tcp_send_fin() to use tcp_stream_alloc_skb().

---

Thanks, Eric!

neal

maintainer
#2 lore raw

Hi Eric,

Quoted 1 line · Eric Dumazet

TCP fast path can (auto)inline this helper, instead

of (auto)inling it from tcp_send_fin().

Quoted 10 lines
Quoted 2 lines

No change of overall code size, but tcp_sendmsg() is faster.

$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new
add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1)
Function                                     old     new   delta
tcp_stream_alloc_skb                         216     357    +141
tcp_send_fin                                 688     548    -140
Total: Before=22236729, After=22236730, chg +0.00%
Quoted 3 lines

BTW, we might change tcp_send_fin() to use tcp_stream_alloc_skb().

I've a question out of sheer curiosity: are you using some specific tool to look for inline opportunity, or "just" careful code and/or objdump analysis?

Thanks,

Paolo

#3 lore raw
Quoted 16 lines · Paolo Abeni

Hi Eric,

Quoted 1 line · Eric Dumazet

TCP fast path can (auto)inline this helper, instead

of (auto)inling it from tcp_send_fin().

Quoted 10 lines
Quoted 2 lines

No change of overall code size, but tcp_sendmsg() is faster.

$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new
add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1)
Function                                     old     new   delta
tcp_stream_alloc_skb                         216     357    +141
tcp_send_fin                                 688     548    -140
Total: Before=22236729, After=22236730, chg +0.00%
Quoted 7 lines
Quoted 3 lines

BTW, we might change tcp_send_fin() to use tcp_stream_alloc_skb().

I've a question out of sheer curiosity: are you using some specific tool to look for inline opportunity, or "just" careful code and/or objdump analysis?

I am studying performance profiles on a stress test using Google production kernels,
on platforms that are a big chunk of the fleet.
These kernels are very close to upstream (at least for core and TCP networking stacks), and use clang and FDO.

I am currently focusing on some functions that even FDO does not inline, for some reasons.

Thanks !

#4 lore raw

Hi Eric,

Quoted 18 lines · Eric Dumazet
Quoted 16 lines · Paolo Abeni

Hi Eric,

Quoted 1 line · Eric Dumazet

TCP fast path can (auto)inline this helper, instead

of (auto)inling it from tcp_send_fin().

Quoted 10 lines
Quoted 2 lines

No change of overall code size, but tcp_sendmsg() is faster.

$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new
add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1)
Function                                     old     new   delta
tcp_stream_alloc_skb                         216     357    +141
tcp_send_fin                                 688     548    -140
Total: Before=22236729, After=22236730, chg +0.00%
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

BTW, we might change tcp_send_fin() to use tcp_stream_alloc_skb().

I've a question out of sheer curiosity: are you using some specific tool to look for inline opportunity, or "just" careful code and/or objdump analysis?

Paolo, right, you beat me to it! I'm also curious :)

Quoted 9 lines

I am studying performance profiles on a stress test using Google production kernels,
on platforms that are a big chunk of the fleet.
These kernels are very close to upstream (at least for core and TCP networking stacks), and use clang and FDO.

I am currently focusing on some functions that even FDO does not inline, for some reasons.

Eric, could you share your more valuable experience and methodology behind this?

Prior to the recent work you've done, I completely missed that we could even do something with the inline functions. Normally using perf doesn't give us a clear micro observation of those functions especially about some hints on whether we should inline some functions. If there is any tool, it would be super awesome!

Let me guess, are you trying to avoid function call action as much as you can in the hot path based on your deep understanding of ingress/egress performance?

Thanks,
Jason

#5 lore raw
Quoted 42 lines · Jason Xing

Hi Eric,

Quoted 18 lines · Eric Dumazet
Quoted 16 lines · Paolo Abeni

Hi Eric,

Quoted 1 line · Eric Dumazet

TCP fast path can (auto)inline this helper, instead

of (auto)inling it from tcp_send_fin().

Quoted 10 lines
Quoted 2 lines

No change of overall code size, but tcp_sendmsg() is faster.

$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new
add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1)
Function                                     old     new   delta
tcp_stream_alloc_skb                         216     357    +141
tcp_send_fin                                 688     548    -140
Total: Before=22236729, After=22236730, chg +0.00%
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

BTW, we might change tcp_send_fin() to use tcp_stream_alloc_skb().

I've a question out of sheer curiosity: are you using some specific tool to look for inline opportunity, or "just" careful code and/or objdump analysis?

Paolo, right, you beat me to it! I'm also curious :)

Quoted 9 lines

I am studying performance profiles on a stress test using Google production kernels,
on platforms that are a big chunk of the fleet.
These kernels are very close to upstream (at least for core and TCP networking stacks), and use clang and FDO.

I am currently focusing on some functions that even FDO does not inline, for some reasons.

Quoted 8 lines

Eric, could you share your more valuable experience and methodology behind this?

Prior to the recent work you've done, I completely missed that we could even do something with the inline functions. Normally using perf doesn't give us a clear micro observation of those functions especially about some hints on whether we should inline some functions. If there is any tool, it would be super awesome!

perf is your friend, and a deep knowledge of your cpu-du-jour behavior (assembly)

perf record -C xxx -g sleep 10
perf report --no-children

<Use the perf UI appropriately>

Quoted 3 lines

Let me guess, are you trying to avoid function call action as much as you can in the hot path based on your deep understanding of ingress/egress performance?

I think you could answer the question by parsing the changelogs of my recent commits ;)

#6 lore raw
Quoted 59 lines · Eric Dumazet
Quoted 42 lines · Jason Xing

Hi Eric,

Quoted 18 lines · Eric Dumazet
Quoted 16 lines · Paolo Abeni

Hi Eric,

Quoted 1 line · Eric Dumazet

TCP fast path can (auto)inline this helper, instead

of (auto)inling it from tcp_send_fin().

Quoted 10 lines
Quoted 2 lines

No change of overall code size, but tcp_sendmsg() is faster.

$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new
add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1)
Function                                     old     new   delta
tcp_stream_alloc_skb                         216     357    +141
tcp_send_fin                                 688     548    -140
Total: Before=22236729, After=22236730, chg +0.00%
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

BTW, we might change tcp_send_fin() to use tcp_stream_alloc_skb().

I've a question out of sheer curiosity: are you using some specific tool to look for inline opportunity, or "just" careful code and/or objdump analysis?

Paolo, right, you beat me to it! I'm also curious :)

Quoted 9 lines

I am studying performance profiles on a stress test using Google production kernels,
on platforms that are a big chunk of the fleet.
These kernels are very close to upstream (at least for core and TCP networking stacks), and use clang and FDO.

I am currently focusing on some functions that even FDO does not inline, for some reasons.

Quoted 8 lines

Eric, could you share your more valuable experience and methodology behind this?

Prior to the recent work you've done, I completely missed that we could even do something with the inline functions. Normally using perf doesn't give us a clear micro observation of those functions especially about some hints on whether we should inline some functions. If there is any tool, it would be super awesome!

perf is your friend, and a deep knowledge of your cpu-du-jour behavior (assembly)

perf record -C xxx -g sleep 10
perf report --no-children

Quoted 8 lines

<Use the perf UI appropriately>

Quoted 3 lines

Let me guess, are you trying to avoid function call action as much as you can in the hot path based on your deep understanding of ingress/egress performance?

I think you could answer the question by parsing the changelogs of my recent commits ;)

Right, I've already catched some insightful knowledge from your commits :) Thanks for the interesting work!

Thanks,
Jason

#7 lore raw
In reply to Eric Dumazet
This patch was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:

Hello:

This patch was applied to netdev/net-next.git (main) by Paolo Abeni <[email protected]>:

Quoted 5 lines · On Fri, 23 Jan 2026 11:16:05 +0000 you wrote:

TCP fast path can (auto)inline this helper, instead of (auto)inling it from tcp_send_fin().

No change of overall code size, but tcp_sendmsg() is faster.

Quoted 8 lines
$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new
add/remove: 0/0 grow/shrink: 1/1 up/down: 141/-140 (1)
Function                                     old     new   delta
tcp_stream_alloc_skb                         216     357    +141
tcp_send_fin                                 688     548    -140
Total: Before=22236729, After=22236730, chg +0.00%

[...]

Here is the summary with links:
- [net-next] tcp: move sk_forced_mem_schedule() to tcp.c

    https://git.kernel.org/netdev/net-next/c/a18056a6c11c

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html