tcp: make tcp_ack() faster
Subsystems (3)
cover tcp: make tcp_ack() faster
Move tcp_rack_update_reo_wnd() and tcp_rack_advance() to tcp_input.c to allow their (auto)inlining.
No functional change in this series.
Eric Dumazet (3):
tcp: mark tcp_process_tlp_ack() as unlikely
tcp: move tcp_rack_update_reo_wnd() to tcp_input.c
tcp: move tcp_rack_advance() to tcp_input.c
include/net/tcp.h | 3 -- net/ipv4/tcp_input.c | 79 +++++++++++++++++++++++++++++++++++++++-- net/ipv4/tcp_recovery.c | 75 -------------------------------------- 3 files changed, 77 insertions(+), 80 deletions(-)
Signature · 1 line
2.53.0.rc1.217.geba53bf80e-goog
PATCH 1/3 tcp: mark tcp_process_tlp_ack() as unlikely
It is unlikely we have to call tcp_process_tlp_ack().
--- net/ipv4/tcp_input.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
net/ipv4/tcp_input.c +2 -2
@@ -4283,7 +4283,7 @@ static int tcp_ack(struct sock *sk, const struct sk_buff *skb, int flag)tcp_in_ack_event (sk, flag); -if (tp->tlp_high_seq) +if (unlikely (tp->tlp_high_seq))tcp_process_tlp_ack (sk, ack, flag);if (tcp_ack_is_dubious (sk, flag)) {
@@ -4333,7 +4333,7 @@ static int tcp_ack(struct sock *sk, const struct sk_buff *skb, int flag)*/ tcp_ack_probe (sk); -if (tp->tlp_high_seq) +if (unlikely (tp->tlp_high_seq))tcp_process_tlp_ack (sk, ack, flag);return 1 ;
Signature · 1 line
2.53.0.rc1.217.geba53bf80e-goog
Re: [PATCH net-next 1/3] tcp: mark tcp_process_tlp_ack() as unlikely
Quoted 4 lines · Eric Dumazet
It is unlikely we have to call
tcp_process_tlp_ack().Signed-off-by Eric Dumazet <[email protected]>
PATCH 2/3 tcp: move tcp_rack_update_reo_wnd() to tcp_input.c
In reply to Eric Dumazettcp_rack_update_reo_wnd() is called only once from tcp_ack()
Move it to tcp_input.c so that it can be inlined by the compiler to save space and cpu cycles.
$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new add/remove: 0/2 grow/shrink: 1/0 up/down: 110/-153 (-43) Function old new delta tcp_ack 5631 5741 +110 __pfx_tcp_rack_update_reo_wnd 16 - -16 tcp_rack_update_reo_wnd 137 - -137 Total: Before=22572723, After=22572680, chg -0.00%
--- include/net/tcp.h | 1 - net/ipv4/tcp_input.c | 43 +++++++++++++++++++++++++++++++++++++++++ net/ipv4/tcp_recovery.c | 43 ----------------------------------------- 3 files changed, 43 insertions(+), 44 deletions(-)
include/net/tcp.h +0 -1
@@ -2517,7 +2517,6 @@ extern bool tcp_rack_mark_lost(struct sock *sk);extern void tcp_rack_advance (struct tcp_sock *tp,u8 sacked,u32 end_seq,u64 xmit_time);extern void tcp_rack_reo_timeout (struct sock *sk); -extern void tcp_rack_update_reo_wnd (struct sock *sk,struct rate_sample *rs);/* tcp_plb.c */
net/ipv4/tcp_input.c +43 -0
49 lines not on this page · raw message
net/ipv4/tcp_recovery.c +0 -43
49 lines not on this page · raw message
Signature · 1 line
2.53.0.rc1.217.geba53bf80e-goog
Re: [PATCH net-next 2/3] tcp: move tcp_rack_update_reo_wnd() to tcp_input.c
Quoted 14 lines · Eric Dumazet
tcp_rack_update_reo_wnd()is called only once fromtcp_ack()Move it to
tcp_input.cso that it can be inlined by the compiler to save space and cpu cycles.$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new add/remove: 0/2 grow/shrink: 1/0 up/down: 110/-153 (-43) Function old new delta tcp_ack 5631 5741 +110 __pfx_tcp_rack_update_reo_wnd 16 - -16 tcp_rack_update_reo_wnd 137 - -137 Total: Before=22572723, After=22572680, chg -0.00%
Quoted 1 line
Signed-off-by Eric Dumazet <[email protected]>
PATCH 3/3 tcp: move tcp_rack_advance() to tcp_input.c
In reply to Eric Dumazettcp_rack_advance() is called from tcp_ack() and tcp_sacktag_one().
Moving it to tcp_input.c allows the compiler to inline it and save both space and cpu cycles in TCP fast path.
$ scripts/bloat-o-meter -t vmlinux.1 vmlinux.2 add/remove: 0/2 grow/shrink: 1/1 up/down: 98/-132 (-34) Function old new delta tcp_ack 5741 5839 +98 tcp_sacktag_one 407 395 -12 __pfx_tcp_rack_advance 16 - -16 tcp_rack_advance 104 - -104 Total: Before=22572680, After=22572646, chg -0.00%
--- include/net/tcp.h | 2 -- net/ipv4/tcp_input.c | 32 ++++++++++++++++++++++++++++++++ net/ipv4/tcp_recovery.c | 32 -------------------------------- 3 files changed, 32 insertions(+), 34 deletions(-)
include/net/tcp.h +0 -2
@@ -2514,8 +2514,6 @@ void tcp_newreno_mark_lost(struct sock *sk, bool snd_una_advanced);extern s32 tcp_rack_skb_timeout (struct tcp_sock *tp,struct sk_buff *skb,u32 reo_wnd);extern bool tcp_rack_mark_lost (struct sock *sk); -extern void tcp_rack_advance (struct tcp_sock *tp,u8 sacked,u32 end_seq, -u64 xmit_time);extern void tcp_rack_reo_timeout (struct sock *sk);/* tcp_plb.c */
net/ipv4/tcp_input.c +32 -0
@@ -1558,6 +1558,38 @@ static int tcp_match_skb_to_sack(struct sock *sk, struct sk_buff *skb,return in_sack; } +/* Record the most recently (re)sent time among the (s)acked packets +* This is "Step 3: Advance RACK.xmit_time and update RACK.RTT" from +* draft-cheng-tcpm-rack-00.txt +*/ +static void tcp_rack_advance (struct tcp_sock *tp,u8 sacked, +u32 end_seq,u64 xmit_time) +{ +u32 rtt_us; + + rtt_us =tcp_stamp_us_delta (tp->tcp_mstamp, xmit_time); +if (rtt_us <tcp_min_rtt (tp) && (sacked &TCPCB_RETRANS )) { +/* If the sacked packet was retransmitted, it's ambiguous +* whether the retransmission or the original (or the prior +* retransmission) was sacked. +* +* If the original is lost, there is no ambiguity. Otherwise +* we assume the original can be delayed up to aRTT + min_rtt. +* the aRTT term is bounded by the fast recovery or timeout, +* so it's at least one RTT (i.e., retransmission is at least +* an RTT later). +*/ +return ; + } + tp->rack.advanced =1 ; + tp->rack.rtt_us = rtt_us; +if (tcp_skb_sent_after (xmit_time, tp->rack.mstamp, + end_seq, tp->rack.end_seq)) { + tp->rack.mstamp = xmit_time; + tp->rack.end_seq = end_seq; + } +} +/* Mark the given newly-SACKed range as such, adjusting counters and hints. */ static u8 tcp_sacktag_one(struct sock *sk, struct tcp_sacktag_state *state, u8 sacked,
net/ipv4/tcp_recovery.c +0 -32
@@ -111,38 +111,6 @@ bool tcp_rack_mark_lost(struct sock *sk)return !!timeout; } -/* Record the most recently (re)sent time among the (s)acked packets -* This is "Step 3: Advance RACK.xmit_time and update RACK.RTT" from -* draft-cheng-tcpm-rack-00.txt -*/ -void tcp_rack_advance (struct tcp_sock *tp,u8 sacked,u32 end_seq, -u64 xmit_time) -{ -u32 rtt_us; - - rtt_us =tcp_stamp_us_delta (tp->tcp_mstamp, xmit_time); -if (rtt_us <tcp_min_rtt (tp) && (sacked &TCPCB_RETRANS )) { -/* If the sacked packet was retransmitted, it's ambiguous -* whether the retransmission or the original (or the prior -* retransmission) was sacked. -* -* If the original is lost, there is no ambiguity. Otherwise -* we assume the original can be delayed up to aRTT + min_rtt. -* the aRTT term is bounded by the fast recovery or timeout, -* so it's at least one RTT (i.e., retransmission is at least -* an RTT later). -*/ -return ; - } - tp->rack.advanced =1 ; - tp->rack.rtt_us = rtt_us; -if (tcp_skb_sent_after (xmit_time, tp->rack.mstamp, - end_seq, tp->rack.end_seq)) { - tp->rack.mstamp = xmit_time; - tp->rack.end_seq = end_seq; - } -} -/* We have waited long enough to accommodate reordering. Mark the expired * packets lost and retransmit them. */
Signature · 1 line
2.53.0.rc1.217.geba53bf80e-goog
Re: [PATCH net-next 3/3] tcp: move tcp_rack_advance() to tcp_input.c
Quoted 15 lines · Eric Dumazet
tcp_rack_advance()is called fromtcp_ack()andtcp_sacktag_one().Moving it to
tcp_input.callows the compiler to inline it and save both space and cpu cycles in TCP fast path.$ scripts/bloat-o-meter -t vmlinux.1 vmlinux.2 add/remove: 0/2 grow/shrink: 1/1 up/down: 98/-132 (-34) Function old new delta tcp_ack 5741 5839 +98 tcp_sacktag_one 407 395 -12 __pfx_tcp_rack_advance 16 - -16 tcp_rack_advance 104 - -104 Total: Before=22572680, After=22572646, chg -0.00%
Quoted 1 line
Signed-off-by Eric Dumazet <[email protected]>
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 11 lines · On Tue, 27 Jan 2026 03:21:44 +0000 you wrote:
Move
tcp_rack_update_reo_wnd()andtcp_rack_advance()totcp_input.cto allow their (auto)inlining.No functional change in this series.
Eric Dumazet (3):
tcp: marktcp_process_tlp_ack()as unlikely
tcp: movetcp_rack_update_reo_wnd()totcp_input.c
tcp: movetcp_rack_advance()totcp_input.c[...]
Here is the summary with links:
- [net-next,1/3] tcp: mark tcp_process_tlp_ack() as unlikely
https://git.kernel.org/netdev/net-next/c/773a70021313 - [net-next,2/3] tcp: move tcp_rack_update_reo_wnd() to tcp_input.c https://git.kernel.org/netdev/net-next/c/629a68865abb - [net-next,3/3] tcp: move tcp_rack_advance() to tcp_input.c https://git.kernel.org/netdev/net-next/c/d5fb143dbe8d
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html