tcp_cubic: fix to achieve at least the same throughput as Reno
Subsystems (3)
v2 cover tcp_cubic: fix to achieve at least the same throughput as Reno
This series patches fixes some CUBIC bugs so that "CUBIC achieves at least the same throughput as Reno in small-BDP networks"
[RFC 9438: https://www.rfc-editor.org/rfc/rfc9438.html]
It consists of three bug fixes, all changing function bictcp_update() of tcp_cubic.c, which controls how fast CUBIC increases its congestion window size snd_cwnd.
(1) tcp_cubic: fix to run bictcp_update() at least once per RTT
(2) tcp_cubic: fix to match Reno additive increment
(3) tcp_cubic: fix to use emulated Reno cwnd one RTT in the future
Experiments:
Below are Mininet experiments to demonstrate the performance difference between the original CUBIC and patched CUBIC.
Network: link capacity = 100Mbps, RTT = 4ms
TCP flows: one RENO and one CUBIC. initial cwnd = 10 packets. The first data packet of each flow is lost
snd_cwnd of RENO and original CUBIC flows
https://github.com/zmrui/tcp_cubic_fix/blob/main/renocubic_fixb0.jpg
snd_cwnd of RENO and patched CUBIC (with bug fixes 1, 2, and 3) flows.
https://github.com/zmrui/tcp_cubic_fix/blob/main/renocubic_fixb1b2b3.jpg
The result of patched CUBIC with different combinations of bug fixes 1, 2, and 3 can be found at the following link, where you can also find more experiment results.
https://github.com/zmrui/tcp_cubic_fix
Changes:
v1->v2:
- Separate patches;
- Add new cwnd_prior field for cwnd before a loss event
- https://lore.kernel.org/netdev/[email protected]/T/#t
Thanks
Mingrui, and Lisong
Mingrui Zhang (3):
tcp_cubic: fix to run bictcp_update() at least once per RTT
tcp_cubic: fix to match Reno additive increment
tcp_cubic: fix to use emulated Reno cwnd one RTT in the future
net/ipv4/tcp_cubic.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
Signature · 1 line
2.34.1
PATCH v2 1/3 tcp_cubic: fix to run bictcp_update() at least once per RTT
The original code bypasses bictcp_update() under certain conditions to reduce the CPU overhead. Intuitively, when last_cwnd==cwnd, bictcp_update() is executed 32 times per second. As a result, it is possible that bictcp_update() is not executed for several RTTs when RTT is short (specifically < 1/32 second = 31 ms and last_cwnd==cwnd which may happen in small-BDP networks), thus leading to low throughput in these RTTs.
The patched code executes bictcp_update() 32 times per second if RTT > 31 ms or every RTT if RTT < 31 ms, when last_cwnd==cwnd.
Thanks
Mingrui, and Lisong
--- net/ipv4/tcp_cubic.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
net/ipv4/tcp_cubic.c +5 -1
@@ -218,8 +218,12 @@ static inline void bictcp_update(struct bictcp *ca, u32 cwnd, u32 acked) ca->ack_cnt += acked;/* count the number of ACKed packets */ +/* Update 32 times per second if RTT > 1/32 second, +* every RTT if RTT < 1/32 second +* even when last_cwnd == cwnd +*/ if (ca->last_cwnd == cwnd && - (s32)(tcp_jiffies32 - ca->last_time) <=HZ /32 ) + (s32)(tcp_jiffies32 - ca->last_time) <=min (HZ /32 ,usecs_to_jiffies (ca->delay_min)))return ;/* The CUBIC function can update ca->cnt at most once per jiffy.
Signature · 1 line
2.34.1
Neal Cardwell
maintainerRe: [PATCH net v2 1/3] tcp_cubic: fix to run bictcp_update() at least once per RTT
Quoted 9 lines · Mingrui Zhang
The original code bypasses
bictcp_update()under certain conditions to reduce the CPU overhead. Intuitively, when last_cwnd==cwnd,bictcp_update()is executed 32 times per second. As a result, it is possible thatbictcp_update()is not executed for several RTTs when RTT is short (specifically < 1/32 second = 31 ms and last_cwnd==cwnd which may happen in small-BDP networks), thus leading to low throughput in these RTTs.
Quoted 7 lines
The patched code executes
bictcp_update()32 times per second if RTT > 31 ms or every RTT if RTT < 31 ms, when last_cwnd==cwnd.Thanks
Mingrui, and LisongFixes 91a4599c2ad8 ("tcp_cubic: fix to run bictcp_update() at least once per RTT")
The Fixes: footer is not for restating the commit title in your new patch. Instead, it should list the SHA1 and first commit message line from the old commit (potentially years ago) that has buggy behavior that you are fixing. That way the Linux maintainers know which Linux releases have the bug, and they know how far back in release history the fix should be applied, when backporting fixes to stable releases.
More information:
https://www.kernel.org/doc/html/v6.10/process/submitting-patches.html#using-reported-by-tested-by-reviewed-by-suggested-by-and-fixes
Please update the Fixes footers in the three commits and repost. :-)
Thanks!
neal
Quoted 28 lines
Signed-off-by Mingrui Zhang <[email protected]>Signed-off-by Lisong Xu <[email protected]>--- net/ipv4/tcp_cubic.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)net/ipv4/tcp_cubic.c +5
-2@@ -218,8 +218,12 @@ static inline void bictcp_update(struct bictcp *ca, u32 cwnd, u32 acked) ca->ack_cnt += acked;/* count the number of ACKed packets */ +/* Update 32 times per second if RTT > 1/32 second, +* every RTT if RTT < 1/32 second +* even when last_cwnd == cwnd +*/ if (ca->last_cwnd == cwnd && - (s32)(tcp_jiffies32 - ca->last_time) <=HZ /32 ) + (s32)(tcp_jiffies32 - ca->last_time) <=min (HZ /32 ,usecs_to_jiffies (ca->delay_min)))return ;/* The CUBIC function can update ca->cnt at most once per jiffy. -- 2.34.1
Re: [PATCH net v2 1/3] tcp_cubic: fix to run bictcp_update() at least once per RTT
Quoted 23 lines · Neal Cardwell
Quoted 8 lines · Mingrui Zhang
The original code bypasses
bictcp_update()under certain conditions to reduce the CPU overhead. Intuitively, when last_cwnd==cwnd,bictcp_update()is executed 32 times per second. As a result, it is possible thatbictcp_update()is not executed for several RTTs when RTT is short (specifically < 1/32 second = 31 ms and last_cwnd==cwnd which may happen in small-BDP networks), thus leading to low throughput in these RTTs.Quoted 7 lines
The patched code executes
bictcp_update()32 times per second if RTT > 31 ms or every RTT if RTT < 31 ms, when last_cwnd==cwnd.Thanks
Mingrui, and LisongFixes 91a4599c2ad8 ("tcp_cubic: fix to run bictcp_update() at least once per RTT")The Fixes: footer is not for restating the commit title in your new patch. Instead, it should list the SHA1 and first commit message line from the old commit (potentially years ago) that has buggy behavior that you are fixing. That way the Linux maintainers know which Linux releases have the bug, and they know how far back in release history the fix should be applied, when backporting fixes to stable releases.
Quoted 8 lines
More information:
https://www.kernel.org/doc/html/v6.10/process/submitting-patches.html#using-reported-by-tested-by-reviewed-by-suggested-by-and-fixesPlease update the Fixes footers in the three commits and repost. :-)
Thanks!
neal
Thank you again, Neal.
I appreciate your detailed instructions on Fixes footer. I misunderstood its meaning.
Thanks
Mingrui
Quoted 28 lines
Quoted 28 lines
Signed-off-by Mingrui Zhang <[email protected]>Signed-off-by Lisong Xu <[email protected]>--- net/ipv4/tcp_cubic.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)net/ipv4/tcp_cubic.c +5
-2@@ -218,8 +218,12 @@ static inline void bictcp_update(struct bictcp *ca, u32 cwnd, u32 acked) ca->ack_cnt += acked;/* count the number of ACKed packets */ +/* Update 32 times per second if RTT > 1/32 second, +* every RTT if RTT < 1/32 second +* even when last_cwnd == cwnd +*/ if (ca->last_cwnd == cwnd && - (s32)(tcp_jiffies32 - ca->last_time) <=HZ /32 ) + (s32)(tcp_jiffies32 - ca->last_time) <=min (HZ /32 ,usecs_to_jiffies (ca->delay_min)))return ;/* The CUBIC function can update ca->cnt at most once per jiffy. -- 2.34.1
Neal Cardwell
maintainerRe: [PATCH net v2 1/3] tcp_cubic: fix to run bictcp_update() at least once per RTT
In reply to Mingrui ZhangQuoted 9 lines · Mingrui Zhang
The original code bypasses
bictcp_update()under certain conditions to reduce the CPU overhead. Intuitively, when last_cwnd==cwnd,bictcp_update()is executed 32 times per second. As a result, it is possible thatbictcp_update()is not executed for several RTTs when RTT is short (specifically < 1/32 second = 31 ms and last_cwnd==cwnd which may happen in small-BDP networks), thus leading to low throughput in these RTTs.
Quoted 5 lines
The patched code executes
bictcp_update()32 times per second if RTT > 31 ms or every RTT if RTT < 31 ms, when last_cwnd==cwnd.Thanks
Mingrui, and Lisong
Another note: please remove these commit message lines like this with "Thanks" and your names, from each of the 3 commits.
Please run "git log" in your git Linux working directory to get a sense of the conventions for Linux commit messages. :-)
Thanks!
neal
Quoted 29 lines
Fixes 91a4599c2ad8 ("tcp_cubic: fix to run bictcp_update() at least once per RTT")Signed-off-by Mingrui Zhang <[email protected]>Signed-off-by Lisong Xu <[email protected]>--- net/ipv4/tcp_cubic.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)net/ipv4/tcp_cubic.c +5
-2@@ -218,8 +218,12 @@ static inline void bictcp_update(struct bictcp *ca, u32 cwnd, u32 acked) ca->ack_cnt += acked;/* count the number of ACKed packets */ +/* Update 32 times per second if RTT > 1/32 second, +* every RTT if RTT < 1/32 second +* even when last_cwnd == cwnd +*/ if (ca->last_cwnd == cwnd && - (s32)(tcp_jiffies32 - ca->last_time) <=HZ /32 ) + (s32)(tcp_jiffies32 - ca->last_time) <=min (HZ /32 ,usecs_to_jiffies (ca->delay_min)))return ;/* The CUBIC function can update ca->cnt at most once per jiffy. -- 2.34.1
PATCH v2 2/3 tcp_cubic: fix to match Reno additive increment
In reply to Mingrui ZhangAdd new field 'cwnd_prior' in bictcp for cwnd before a loss event Suggested by Neal Cardwell
https://lore.kernel.org/netdev/[email protected]/T/#t
The original code follows RFC 8312 (obsoleted CUBIC RFC).
The patched code follows RFC 9438 (new CUBIC RFC).
"Once _W_est_ has grown to reach the _cwnd_ at the time of most recently setting _ssthresh_ -- that is, _W_est_ >= _cwnd_prior_ -- the sender SHOULD set α__cubic_ to 1 to ensure that it can achieve the same congestion window increment rate as Reno, which uses AIMD (1,0.5)."
Thanks
Mingrui, and Lisong
--- net/ipv4/tcp_cubic.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
net/ipv4/tcp_cubic.c +6 -1
@@ -102,6 +102,7 @@ struct bictcp {u32 end_seq;/* end_seq of the round */ u32 last_ack;/* last time when the ACK spacing is close */ u32 curr_rtt;/* the minimum rtt of current round */ +u32 cwnd_prior;/* cwnd before a loss event */ };static inline void bictcp_reset (struct bictcp *ca)
@@ -305,7 +306,10 @@ static inline void bictcp_update(struct bictcp *ca, u32 cwnd, u32 acked)if (tcp_friendliness) {u32 scale = beta_scale; - delta = (cwnd * scale) >>3 ; +if (cwnd < ca->cwnd_prior) + delta = (cwnd * scale) >>3 ;/* CUBIC additive increment */ +else + delta = cwnd;/* Reno additive increment */ while (ca->ack_cnt > delta) {/* update tcp cwnd */ ca->ack_cnt -= delta; ca->tcp_cwnd++;
@@ -355,6 +359,7 @@ __bpf_kfunc static u32 cubictcp_recalc_ssthresh(struct sock *sk) / (2 * BICTCP_BETA_SCALE); else ca->last_max_cwnd = tcp_snd_cwnd(tp); + ca->cwnd_prior =tcp_snd_cwnd (tp);return max ((tcp_snd_cwnd (tp) * beta) /BICTCP_BETA_SCALE ,2U ); }
Signature · 1 line
2.34.1
PATCH v2 3/3 tcp_cubic: fix to use emulated Reno cwnd one RTT in the future
In reply to Mingrui ZhangThe original code estimates RENO snd_cwnd using the estimated RENO snd_cwnd at the current time (i.e., tcp_cwnd).
The patched code estimates RENO snd_cwnd using the estimated RENO snd_cwnd after one RTT (i.e., tcp_cwnd_next_rtt), because ca->cnt is used to increase snd_cwnd for the next RTT.
Thanks
Mingrui, and Lisong
--- net/ipv4/tcp_cubic.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
net/ipv4/tcp_cubic.c +5 -2
@@ -315,8 +315,11 @@ static inline void bictcp_update(struct bictcp *ca, u32 cwnd, u32 acked) ca->tcp_cwnd++; } -if (ca->tcp_cwnd > cwnd) {/* if bic is slower than tcp */ - delta = ca->tcp_cwnd - cwnd; +/* Reno cwnd one RTT in the future */ +u32 tcp_cwnd_next_rtt = ca->tcp_cwnd + (ca->ack_cnt + cwnd) / delta; + +if (tcp_cwnd_next_rtt > cwnd) {/* if bic is slower than Reno */ + delta = tcp_cwnd_next_rtt - cwnd; max_cnt = cwnd / delta;if (ca->cnt > max_cnt) ca->cnt = max_cnt;
Signature · 1 line
2.34.1