Skip to contentSkip to conversation

tcp: make cwnd-limited not affected by tcp internal pacing

RFC PATCH In review Pengcheng Yang 2 participants 2 messages Mar 19, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 tcp: make cwnd-limited not affected by tcp internal pacing

The current cwnd-limited is set when cwnd is fully used (inflight >= cwnd), which allows the congestion algorithm to accurately determine whether cwnd needs to be added.

However, there may be a problem when using tcp internal pacing:
In congestion avoidance phase, when a burst of packets are acked by a stretched ACK or a burst of ACKs, this makes a large reduction in inflight in a short time. At this time, the sender sends data according to the pacing rate cannot fill CWND and cwnd-limited is not set. The worst case is that cwnd-limited is set only after the last packet in a window is sent. This causes the congestion algorithm to be too conservative to increase CWND.

The idea is that once cwnd-limited is set, it maintains a window period. In this period, it is considered that the CWND is limited. This makes the congestion algorithm unaffected by tcp internal pacing.

---
 include/linux/tcp.h   |  2 +-
 net/ipv4/tcp_output.c | 14 ++++++++------
 2 files changed, 9 insertions(+), 7 deletions(-)
include/linux/tcp.h +1 -1
index 3dc9640..3b3329f 100644
@@ -286,7 +286,7 @@ struct tcp_sock {
 	u32	packets_out;	/* Packets which are "in flight"	*/
 	u32	retrans_out;	/* Retransmitted packets out		*/
 	u32	max_packets_out;  /* max packets_out in last window */
-	u32	max_packets_seq;  /* right edge of max_packets_out flight */
+	u32	cwnd_limited_seq; /* snd_nxt at cwnd limited */
 
 	u16	urg_data;	/* Saved octet of OOB data and control flags */
 	u8	ecn_flags;	/* ECN status bits.			*/
net/ipv4/tcp_output.c +8 -6
index 306e25d..31dd6dc 100644
@@ -1705,14 +1705,16 @@ static void tcp_cwnd_validate(struct sock *sk, bool is_cwnd_limited)
 	const struct tcp_congestion_ops *ca_ops = inet_csk(sk)->icsk_ca_ops;
 	struct tcp_sock *tp = tcp_sk(sk);
 
-	/* Track the maximum number of outstanding packets in each
-	 * window, and remember whether we were cwnd-limited then.
+	/* Remember whether we were cwnd-limited in last window,
+	 * and track the maximum number of outstanding packets in each window.
 	 */
-	if (!before(tp->snd_una, tp->max_packets_seq) ||
-	    tp->packets_out > tp->max_packets_out) {
-		tp->max_packets_out = tp->packets_out;
-		tp->max_packets_seq = tp->snd_nxt;
+	if (is_cwnd_limited ||
+	    !before(tp->snd_una, tp->cwnd_limited_seq)) {
 		tp->is_cwnd_limited = is_cwnd_limited;
+		tp->cwnd_limited_seq = tp->snd_nxt;
+		tp->max_packets_out = tp->packets_out;
+	} else if (tp->packets_out > tp->max_packets_out) {
+		tp->max_packets_out = tp->packets_out;
 	}
 
 	if (tcp_is_cwnd_limited(sk)) {
Signature · 1 line
1.8.3.1
maintainer
#1 lore raw
Quoted 14 lines · Pengcheng Yang

The current cwnd-limited is set when cwnd is fully used (inflight >= cwnd), which allows the congestion algorithm to accurately determine whether cwnd needs to be added.

However, there may be a problem when using tcp internal pacing:
In congestion avoidance phase, when a burst of packets are acked by a stretched ACK or a burst of ACKs, this makes a large reduction in inflight in a short time. At this time, the sender sends data according to the pacing rate cannot fill CWND and cwnd-limited is not set. The worst case is that cwnd-limited is set only after the last packet in a window is sent. This causes the congestion algorithm to be too conservative to increase CWND.

Quoted 6 lines

The idea is that once cwnd-limited is set, it maintains a window period. In this period, it is considered that the CWND is limited. This makes the congestion algorithm unaffected by tcp internal pacing.

---

Thanks for sending this patch! We ran into this bug in our team recently as well, and have been working on iterating on patches to fix it.

I think this particular proposal in this patch does not properly persist the max_packets_out until all the ACKs have been received for a flight of data. The consequence of this would be that the cwnd does not grow properly in slow-start for cases where the max_packets_out is high enough to merit growing cwnd, but the connection is not strictly cwnd-limited.

I'm a bit busy this week but I will try to put together and send out a proposed patch ASAP.

best,
neal