Skip to contentSkip to conversation

tcp: make tcp_ack() faster

PATCH Applied Eric Dumazet 3 participants 8 messages Jan 27 – Jan 29

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover tcp: make tcp_ack() faster

Move tcp_rack_update_reo_wnd() and tcp_rack_advance() to tcp_input.c to allow their (auto)inlining.

No functional change in this series.

Eric Dumazet (3):
tcp: mark tcp_process_tlp_ack() as unlikely
tcp: move tcp_rack_update_reo_wnd() to tcp_input.c
tcp: move tcp_rack_advance() to tcp_input.c

 include/net/tcp.h       |  3 --
 net/ipv4/tcp_input.c    | 79 +++++++++++++++++++++++++++++++++++++++--
 net/ipv4/tcp_recovery.c | 75 --------------------------------------
 3 files changed, 77 insertions(+), 80 deletions(-)
Signature · 1 line
2.53.0.rc1.217.geba53bf80e-goog
#1 lore raw

PATCH 1/3 tcp: mark tcp_process_tlp_ack() as unlikely

It is unlikely we have to call tcp_process_tlp_ack().

---
 net/ipv4/tcp_input.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
net/ipv4/tcp_input.c +2 -2
index 9e91ddbc6253..f2fafba95705 100644
@@ -4283,7 +4283,7 @@ static int tcp_ack(struct sock *sk, const struct sk_buff *skb, int flag)
 
 	tcp_in_ack_event(sk, flag);
 
-	if (tp->tlp_high_seq)
+	if (unlikely(tp->tlp_high_seq))
 		tcp_process_tlp_ack(sk, ack, flag);
 
 	if (tcp_ack_is_dubious(sk, flag)) {
@@ -4333,7 +4333,7 @@ static int tcp_ack(struct sock *sk, const struct sk_buff *skb, int flag)
 	 */
 	tcp_ack_probe(sk);
 
-	if (tp->tlp_high_seq)
+	if (unlikely(tp->tlp_high_seq))
 		tcp_process_tlp_ack(sk, ack, flag);
 	return 1;
 
Signature · 1 line
2.53.0.rc1.217.geba53bf80e-goog
#3 lore raw

PATCH 2/3 tcp: move tcp_rack_update_reo_wnd() to tcp_input.c

In reply to Eric Dumazet

tcp_rack_update_reo_wnd() is called only once from tcp_ack()

Move it to tcp_input.c so that it can be inlined by the compiler to save space and cpu cycles.

$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new
add/remove: 0/2 grow/shrink: 1/0 up/down: 110/-153 (-43)
Function                                     old     new   delta
tcp_ack                                     5631    5741    +110
__pfx_tcp_rack_update_reo_wnd                 16       -     -16
tcp_rack_update_reo_wnd                      137       -    -137
Total: Before=22572723, After=22572680, chg -0.00%
---
 include/net/tcp.h       |  1 -
 net/ipv4/tcp_input.c    | 43 +++++++++++++++++++++++++++++++++++++++++
 net/ipv4/tcp_recovery.c | 43 -----------------------------------------
 3 files changed, 43 insertions(+), 44 deletions(-)
include/net/tcp.h +0 -1
index efff433de9a4..a4d9f263ea68 100644
@@ -2517,7 +2517,6 @@ extern bool tcp_rack_mark_lost(struct sock *sk);
 extern void tcp_rack_advance(struct tcp_sock *tp, u8 sacked, u32 end_seq,
 			     u64 xmit_time);
 extern void tcp_rack_reo_timeout(struct sock *sk);
-extern void tcp_rack_update_reo_wnd(struct sock *sk, struct rate_sample *rs);
 
 /* tcp_plb.c */
 
net/ipv4/tcp_input.c +43 -0

49 lines not on this page · raw message

net/ipv4/tcp_recovery.c +0 -43

49 lines not on this page · raw message

Signature · 1 line
2.53.0.rc1.217.geba53bf80e-goog
reviewer Reviewed-by on v1 2/3
#4 lore raw

Re: [PATCH net-next 2/3] tcp: move tcp_rack_update_reo_wnd() to tcp_input.c

Quoted 14 lines · Eric Dumazet

tcp_rack_update_reo_wnd() is called only once from tcp_ack()

Move it to tcp_input.c so that it can be inlined by the compiler to save space and cpu cycles.

$ scripts/bloat-o-meter -t vmlinux.old vmlinux.new
add/remove: 0/2 grow/shrink: 1/0 up/down: 110/-153 (-43)
Function                                     old     new   delta
tcp_ack                                     5631    5741    +110
__pfx_tcp_rack_update_reo_wnd                 16       -     -16
tcp_rack_update_reo_wnd                      137       -    -137
Total: Before=22572723, After=22572680, chg -0.00%
Quoted 1 line
#5 lore raw

PATCH 3/3 tcp: move tcp_rack_advance() to tcp_input.c

In reply to Eric Dumazet

tcp_rack_advance() is called from tcp_ack() and tcp_sacktag_one().

Moving it to tcp_input.c allows the compiler to inline it and save both space and cpu cycles in TCP fast path.

$ scripts/bloat-o-meter -t vmlinux.1 vmlinux.2
add/remove: 0/2 grow/shrink: 1/1 up/down: 98/-132 (-34)
Function                                     old     new   delta
tcp_ack                                     5741    5839     +98
tcp_sacktag_one                              407     395     -12
__pfx_tcp_rack_advance                        16       -     -16
tcp_rack_advance                             104       -    -104
Total: Before=22572680, After=22572646, chg -0.00%
---
 include/net/tcp.h       |  2 --
 net/ipv4/tcp_input.c    | 32 ++++++++++++++++++++++++++++++++
 net/ipv4/tcp_recovery.c | 32 --------------------------------
 3 files changed, 32 insertions(+), 34 deletions(-)
include/net/tcp.h +0 -2
index a4d9f263ea68..f1cf9e6730c8 100644
@@ -2514,8 +2514,6 @@ void tcp_newreno_mark_lost(struct sock *sk, bool snd_una_advanced);
 extern s32 tcp_rack_skb_timeout(struct tcp_sock *tp, struct sk_buff *skb,
 				u32 reo_wnd);
 extern bool tcp_rack_mark_lost(struct sock *sk);
-extern void tcp_rack_advance(struct tcp_sock *tp, u8 sacked, u32 end_seq,
-			     u64 xmit_time);
 extern void tcp_rack_reo_timeout(struct sock *sk);
 
 /* tcp_plb.c */
net/ipv4/tcp_input.c +32 -0
index d504a9a9b6ec..a2a872382fc0 100644
@@ -1558,6 +1558,38 @@ static int tcp_match_skb_to_sack(struct sock *sk, struct sk_buff *skb,
 	return in_sack;
 }
 
+/* Record the most recently (re)sent time among the (s)acked packets
+ * This is "Step 3: Advance RACK.xmit_time and update RACK.RTT" from
+ * draft-cheng-tcpm-rack-00.txt
+ */
+static void tcp_rack_advance(struct tcp_sock *tp, u8 sacked,
+			     u32 end_seq, u64 xmit_time)
+{
+	u32 rtt_us;
+
+	rtt_us = tcp_stamp_us_delta(tp->tcp_mstamp, xmit_time);
+	if (rtt_us < tcp_min_rtt(tp) && (sacked & TCPCB_RETRANS)) {
+		/* If the sacked packet was retransmitted, it's ambiguous
+		 * whether the retransmission or the original (or the prior
+		 * retransmission) was sacked.
+		 *
+		 * If the original is lost, there is no ambiguity. Otherwise
+		 * we assume the original can be delayed up to aRTT + min_rtt.
+		 * the aRTT term is bounded by the fast recovery or timeout,
+		 * so it's at least one RTT (i.e., retransmission is at least
+		 * an RTT later).
+		 */
+		return;
+	}
+	tp->rack.advanced = 1;
+	tp->rack.rtt_us = rtt_us;
+	if (tcp_skb_sent_after(xmit_time, tp->rack.mstamp,
+			       end_seq, tp->rack.end_seq)) {
+		tp->rack.mstamp = xmit_time;
+		tp->rack.end_seq = end_seq;
+	}
+}
+
 /* Mark the given newly-SACKed range as such, adjusting counters and hints. */
 static u8 tcp_sacktag_one(struct sock *sk,
 			  struct tcp_sacktag_state *state, u8 sacked,
net/ipv4/tcp_recovery.c +0 -32
index 40732b84771e..139646751073 100644
@@ -111,38 +111,6 @@ bool tcp_rack_mark_lost(struct sock *sk)
 	return !!timeout;
 }
 
-/* Record the most recently (re)sent time among the (s)acked packets
- * This is "Step 3: Advance RACK.xmit_time and update RACK.RTT" from
- * draft-cheng-tcpm-rack-00.txt
- */
-void tcp_rack_advance(struct tcp_sock *tp, u8 sacked, u32 end_seq,
-		      u64 xmit_time)
-{
-	u32 rtt_us;
-
-	rtt_us = tcp_stamp_us_delta(tp->tcp_mstamp, xmit_time);
-	if (rtt_us < tcp_min_rtt(tp) && (sacked & TCPCB_RETRANS)) {
-		/* If the sacked packet was retransmitted, it's ambiguous
-		 * whether the retransmission or the original (or the prior
-		 * retransmission) was sacked.
-		 *
-		 * If the original is lost, there is no ambiguity. Otherwise
-		 * we assume the original can be delayed up to aRTT + min_rtt.
-		 * the aRTT term is bounded by the fast recovery or timeout,
-		 * so it's at least one RTT (i.e., retransmission is at least
-		 * an RTT later).
-		 */
-		return;
-	}
-	tp->rack.advanced = 1;
-	tp->rack.rtt_us = rtt_us;
-	if (tcp_skb_sent_after(xmit_time, tp->rack.mstamp,
-			       end_seq, tp->rack.end_seq)) {
-		tp->rack.mstamp = xmit_time;
-		tp->rack.end_seq = end_seq;
-	}
-}
-
 /* We have waited long enough to accommodate reordering. Mark the expired
  * packets lost and retransmit them.
  */
Signature · 1 line
2.53.0.rc1.217.geba53bf80e-goog
reviewer Reviewed-by on v1 3/3
#6 lore raw

Re: [PATCH net-next 3/3] tcp: move tcp_rack_advance() to tcp_input.c

Quoted 15 lines · Eric Dumazet

tcp_rack_advance() is called from tcp_ack() and tcp_sacktag_one().

Moving it to tcp_input.c allows the compiler to inline it and save both space and cpu cycles in TCP fast path.

$ scripts/bloat-o-meter -t vmlinux.1 vmlinux.2
add/remove: 0/2 grow/shrink: 1/1 up/down: 98/-132 (-34)
Function                                     old     new   delta
tcp_ack                                     5741    5839     +98
tcp_sacktag_one                              407     395     -12
__pfx_tcp_rack_advance                        16       -     -16
tcp_rack_advance                             104       -    -104
Total: Before=22572680, After=22572646, chg -0.00%
Quoted 1 line
#7 lore raw
In reply to Eric Dumazet
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Hello:

This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:

Quoted 11 lines · On Tue, 27 Jan 2026 03:21:44 +0000 you wrote:

Move tcp_rack_update_reo_wnd() and tcp_rack_advance() to tcp_input.c to allow their (auto)inlining.

No functional change in this series.

Eric Dumazet (3):
tcp: mark tcp_process_tlp_ack() as unlikely
tcp: move tcp_rack_update_reo_wnd() to tcp_input.c
tcp: move tcp_rack_advance() to tcp_input.c

[...]

Here is the summary with links:
- [net-next,1/3] tcp: mark tcp_process_tlp_ack() as unlikely

    https://git.kernel.org/netdev/net-next/c/773a70021313
  - [net-next,2/3] tcp: move tcp_rack_update_reo_wnd() to tcp_input.c
    https://git.kernel.org/netdev/net-next/c/629a68865abb
  - [net-next,3/3] tcp: move tcp_rack_advance() to tcp_input.c
    https://git.kernel.org/netdev/net-next/c/d5fb143dbe8d

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html