tcp: Fix orphaned socket stalling indefinitely in FIN-WAIT-1
Subsystems (3)
RFC 1/1 tcp: Fix orphaned socket stalling indefinitely in FIN-WAIT-1
From: MingMing Wang <[email protected]>
An orphaned TCP socket can stall indefinitely in FIN-WAIT-1 if the following conditions are met:
1. net.ipv4.tcp_retries2 is set to a value ≤ 8;
2. The peer advertises a zero window, and the window never reopens.
Steps to reproduce:
1. Set up two instances with nmap installed: one will act as the server the other as the client
2. Execute on the server:
a. lower rmem : sysctl -w net.ipv4.tcp_rmem="16 32 32" b. start a listener: nc -l -p 1234
3. Execute on the client:
a. lower tcp_retries2: sysctl -w net.ipv4.tcp_retries2=8 b. send pakcets: cat /dev/zero | nc <server-ip> 1234
c. after five seconds, stop the process: killall nc
4. Execute on the server: killall -STOP nc
5. Expected abnormal result: using ss command, we'll notice that the client connection remains stuck in the FIN_WAIT1 state, and the backoff counter always be 8 and no longer increased, as shown below:
```
FIN-WAIT-1 0 1389 172.16.0.2:50316 172.16.0.1:1234
cubic wscale:2,7 rto:201 backoff:8 rtt:0.078/0.007 mss:36
... other fields omitted ...
```6. If we set tcp_retries2 to 15 and repeat the steps above, the FIN_WAIT1 state will be forcefully reclaimed after about 5 minutes.
During the zero-window probe retry process, it will check whether the current connection is alive or not. If the connection is not alive and the counter of retries exceeds the maximum allowed max_probes, retry process will be terminated.
In our case, when we set net.ipv4.tcp_retries2 to 8 or a less value, according to the current implementation, the icsk->icsk_backoff counter will be capped at net.ipv4.tcp_retries2. The value calculated by inet_csk_rto_backoff will always be too small, which means the computed backoff duration will always be less than rto_max. As a result, the alive check will always return true. The condition before the goto abort statement is an logical AND condition, the abort branch can never be reached.
So, the TCP retransmission backoff mechanism has two issues:
1. icsk->icsk_backoff should monotonically increase during probe transmission and, upon reaching the maximum backoff limit, the connection should be terminated. However, the backoff value itself must not be capped prematurely — it should only control when to abort.
2. The condition for orphaned connection abort was incorrectly based on connection liveness and probe count. It should instead consider whether the number of orphaned probes exceeds the intended limit.
To fix this, introduce a local variable orphan_probes to track orphan probe attempts separately from max_probes, which is used for RTO retransmissions. This decouples the two counters and prevents accidental overwrites, ensuring correct timeout behavior for orphaned connections.
---
We couldn't determine the rationale behind the following check in tcp_send_probe0():
```
if (icsk->icsk_backoff < READ_ONCE(net->ipv4.sysctl_tcp_retries2))
icsk->icsk_backoff++;
```This condition appears to be the root cause of the observed stall. However, it has existed in the kernel for over 20 years — which suggests there might be a historical or subtle reason for its presence.
We would greatly appreciate it if anyone could shed
--- net/ipv4/tcp_output.c | 4 +--- net/ipv4/tcp_timer.c | 4 ++-- 2 files changed, 3 insertions(+), 5 deletions(-)
net/ipv4/tcp_output.c +1 -3
@@ -4385,7 +4385,6 @@ void tcp_send_probe0(struct sock *sk) {struct inet_connection_sock *icsk =inet_csk (sk);struct tcp_sock *tp =tcp_sk (sk); -struct net *net =sock_net (sk);unsigned long timeout;int err;
@@ -4401,8 +4400,7 @@ void tcp_send_probe0(struct sock *sk) icsk->icsk_probes_out++;if (err <=0 ) { -if (icsk->icsk_backoff <READ_ONCE (net->ipv4.sysctl_tcp_retries2)) - icsk->icsk_backoff++; + icsk->icsk_backoff++; timeout =tcp_probe0_when (sk,tcp_rto_max (sk)); }else {/* If packet was not sent due to local congestion,
net/ipv4/tcp_timer.c +2 -2
@@ -419,9 +419,9 @@ static void tcp_probe_timer(struct sock *sk)if (sock_flag (sk,SOCK_DEAD )) {unsigned int rto_max =tcp_rto_max (sk);const bool alive =inet_csk_rto_backoff (icsk, rto_max) < rto_max; +int orphan_probes =tcp_orphan_retries (sk, alive); - max_probes =tcp_orphan_retries (sk, alive); -if (!alive && icsk->icsk_backoff >= max_probes) +if (!alive || icsk->icsk_backoff >= orphan_probes) goto abort;if (tcp_out_of_resources (sk, true))return ;
Signature · 1 line
2.46.0
Quoted 22 lines · MingMing Wang
From: MingMing Wang <[email protected]>
An orphaned TCP socket can stall indefinitely in FIN-WAIT-1 if the following conditions are met:
1. net.ipv4.tcp_retries2 is set to a value ≤ 8;
2. The peer advertises a zero window, and the window never reopens.Steps to reproduce:
1. Set up two instances with nmap installed: one will act as the server the other as the client
2. Execute on the server:a. lower rmem : `sysctl -w net.ipv4.tcp_rmem="16 32 32"` b. start a listener: `nc -l -p 1234`3. Execute on the client:
a. lowertcp_retries2:sysctl -w net.ipv4.tcp_retries2=8b. send pakcets:cat /dev/zero | nc <server-ip> 1234
c. after five seconds, stop the process:killall nc
4. Execute on the server:killall -STOP nc
5. Expected abnormal result: usingsscommand, we'll notice that the client connection remains stuck in theFIN_WAIT1state, and the backoff counter always be 8 and no longer increased, as shown below:
Quoted 1 line
```
Hi MingMing
Please prepare and share with us a packetdrill test, instead of this 'repro', which is the old way of describing things :/
- This will be easier for us to understand the issue.
- It will be added to existing tests in tools/testing/selftests/net/packetdrill if your patch is accepted, so that we can make sure future changes are not breaking this again.
Ideally, you should attach this packetdrill test in a second patch (thus sending a series of two patches)
Thank you.
Quoted 73 lines
FIN-WAIT-1 0 1389 172.16.0.2:50316 172.16.0.1:1234 cubic wscale:2,7 rto:201 backoff:8 rtt:0.078/0.007 mss:36 ... other fields omitted ... ```6. If we set
tcp_retries2to 15 and repeat the steps above, theFIN_WAIT1state will be forcefully reclaimed after about 5 minutes.During the zero-window probe retry process, it will check whether the current connection is alive or not. If the connection is not alive and the counter of retries exceeds the maximum allowed
max_probes, retry process will be terminated.In our case, when we set
net.ipv4.tcp_retries2to 8 or a less value, according to the current implementation, theicsk->icsk_backoffcounter will be capped atnet.ipv4.tcp_retries2. The value calculated byinet_csk_rto_backoffwill always be too small, which means the computed backoff duration will always be less thanrto_max. As a result, the alive check will always return true. The condition before thegoto abortstatement is an logical AND condition, the abort branch can never be reached.So, the TCP retransmission backoff mechanism has two issues:
1.
icsk->icsk_backoffshould monotonically increase during probe transmission and, upon reaching the maximum backoff limit, the connection should be terminated. However, the backoff value itself must not be capped prematurely — it should only control when to abort.2. The condition for orphaned connection abort was incorrectly based on connection liveness and probe count. It should instead consider whether the number of orphaned probes exceeds the intended limit.
To fix this, introduce a local variable
orphan_probesto track orphan probe attempts separately frommax_probes, which is used for RTO retransmissions. This decouples the two counters and prevents accidental overwrites, ensuring correct timeout behavior for orphaned connections.Fixes b248230c34970 ("tcp: abort orphan sockets stalling on zero window probes")Co-developed-by Dust Li <[email protected]>Signed-off-by Dust Li <[email protected]>Co-developed-by MingMing Wang <[email protected]>Signed-off-by MingMing Wang <[email protected]>---
We couldn't determine the rationale behind the following check intcp_send_probe0():``` if (icsk->icsk_backoff < READ_ONCE(net->ipv4.sysctl_tcp_retries2)) icsk->icsk_backoff++; ```This condition appears to be the root cause of the observed stall. However, it has existed in the kernel for over 20 years — which suggests there might be a historical or subtle reason for its presence.
We would greatly appreciate it if anyone could shed
--- net/ipv4/tcp_output.c | 4 +--- net/ipv4/tcp_timer.c | 4 ++-- 2 files changed, 3 insertions(+), 5 deletions(-)net/ipv4/tcp_output.c +0
-1@@ -4385,7 +4385,6 @@ void tcp_send_probe0(struct sock *sk) {struct inet_connection_sock *icsk =inet_csk (sk);struct tcp_sock *tp =tcp_sk (sk); -struct net *net =sock_net (sk);unsigned long timeout;int err;
net/ipv4/tcp_output.c void tcp_send_probe0(struct sock *sk) · line 4403
Quoted 6 lines
net/ipv4/tcp_output.c +1
-2@@ -4401,8 +4400,7 @@ void tcp_send_probe0(struct sock *sk) icsk->icsk_probes_out++;if (err <=0 ) { -if (icsk->icsk_backoff <READ_ONCE (net->ipv4.sysctl_tcp_retries2)) - icsk->icsk_backoff++; + icsk->icsk_backoff++;
I think we need to have a cap, otherwise we risk overflows in inet_csk_rto_backoff()
Quoted 22 lines
net/ipv4/tcp_output.c +0
-0timeout =tcp_probe0_when (sk,tcp_rto_max (sk)); }else {/* If packet was not sent due to local congestion, net/ipv4/tcp_timer.c +2
-3@@ -419,9 +419,9 @@ static void tcp_probe_timer(struct sock *sk)if (sock_flag (sk,SOCK_DEAD )) {unsigned int rto_max =tcp_rto_max (sk);const bool alive =inet_csk_rto_backoff (icsk, rto_max) < rto_max; +int orphan_probes =tcp_orphan_retries (sk, alive); - max_probes =tcp_orphan_retries (sk, alive); -if (!alive && icsk->icsk_backoff >= max_probes) +if (!alive || icsk->icsk_backoff >= orphan_probes) goto abort;if (tcp_out_of_resources (sk, true))return ; --2.46.0
Quoted 33 lines · Eric Dumazet
Quoted 22 lines · MingMing Wang
From: MingMing Wang <[email protected]>
An orphaned TCP socket can stall indefinitely in FIN-WAIT-1 if the following conditions are met:
1. net.ipv4.tcp_retries2 is set to a value ≤ 8;
2. The peer advertises a zero window, and the window never reopens.Steps to reproduce:
1. Set up two instances with nmap installed: one will act as the server the other as the client
2. Execute on the server:a. lower rmem : `sysctl -w net.ipv4.tcp_rmem="16 32 32"` b. start a listener: `nc -l -p 1234`3. Execute on the client:
a. lowertcp_retries2:sysctl -w net.ipv4.tcp_retries2=8b. send pakcets:cat /dev/zero | nc <server-ip> 1234
c. after five seconds, stop the process:killall nc
4. Execute on the server:killall -STOP nc
5. Expected abnormal result: usingsscommand, we'll notice that the client connection remains stuck in theFIN_WAIT1state, and the backoff counter always be 8 and no longer increased, as shown below:Quoted 1 line
```Hi MingMing
Please prepare and share with us a packetdrill test, instead of this 'repro', which is the old way of describing things :/
- This will be easier for us to understand the issue.
Quoted 47 lines
- It will be added to existing tests in
tools/testing/selftests/net/packetdrillif your patch is accepted, so that we can make sure future changes are not breaking this again.Ideally, you should attach this packetdrill test in a second patch (thus sending a series of two patches)
Thank you.
Quoted 38 lines
FIN-WAIT-1 0 1389 172.16.0.2:50316 172.16.0.1:1234 cubic wscale:2,7 rto:201 backoff:8 rtt:0.078/0.007 mss:36 ... other fields omitted ... ```6. If we set
tcp_retries2to 15 and repeat the steps above, theFIN_WAIT1state will be forcefully reclaimed after about 5 minutes.During the zero-window probe retry process, it will check whether the current connection is alive or not. If the connection is not alive and the counter of retries exceeds the maximum allowed
max_probes, retry process will be terminated.In our case, when we set
net.ipv4.tcp_retries2to 8 or a less value, according to the current implementation, theicsk->icsk_backoffcounter will be capped atnet.ipv4.tcp_retries2. The value calculated byinet_csk_rto_backoffwill always be too small, which means the computed backoff duration will always be less thanrto_max. As a result, the alive check will always return true. The condition before thegoto abortstatement is an logical AND condition, the abort branch can never be reached.So, the TCP retransmission backoff mechanism has two issues:
1.
icsk->icsk_backoffshould monotonically increase during probe transmission and, upon reaching the maximum backoff limit, the connection should be terminated. However, the backoff value itself must not be capped prematurely — it should only control when to abort.2. The condition for orphaned connection abort was incorrectly based on connection liveness and probe count. It should instead consider whether the number of orphaned probes exceeds the intended limit.
To fix this, introduce a local variable
orphan_probesto track orphan probe attempts separately frommax_probes, which is used for RTO retransmissions. This decouples the two counters and prevents accidental overwrites, ensuring correct timeout behavior for orphaned connections.Fixes b248230c34970 ("tcp: abort orphan sockets stalling on zero window probes")
Thanks for catching this corner case. Feel free to add a Acked-by:
<[email protected]> after the packetdrill test in your respin
Quoted 68 lines
Quoted 35 lines
Co-developed-by Dust Li <[email protected]>Signed-off-by Dust Li <[email protected]>Co-developed-by MingMing Wang <[email protected]>Signed-off-by MingMing Wang <[email protected]>---
We couldn't determine the rationale behind the following check intcp_send_probe0():``` if (icsk->icsk_backoff < READ_ONCE(net->ipv4.sysctl_tcp_retries2)) icsk->icsk_backoff++; ```This condition appears to be the root cause of the observed stall. However, it has existed in the kernel for over 20 years — which suggests there might be a historical or subtle reason for its presence.
We would greatly appreciate it if anyone could shed
--- net/ipv4/tcp_output.c | 4 +--- net/ipv4/tcp_timer.c | 4 ++-- 2 files changed, 3 insertions(+), 5 deletions(-)net/ipv4/tcp_output.c +0
-1@@ -4385,7 +4385,6 @@ void tcp_send_probe0(struct sock *sk) {struct inet_connection_sock *icsk =inet_csk (sk);struct tcp_sock *tp =tcp_sk (sk); -struct net *net =sock_net (sk);unsigned long timeout;int err;net/ipv4/tcp_output.c void tcp_send_probe0(struct sock *sk) · line 4403
Quoted 6 lines
net/ipv4/tcp_output.c +1
-2@@ -4401,8 +4400,7 @@ void tcp_send_probe0(struct sock *sk) icsk->icsk_probes_out++;if (err <=0 ) { -if (icsk->icsk_backoff <READ_ONCE (net->ipv4.sysctl_tcp_retries2)) - icsk->icsk_backoff++; + icsk->icsk_backoff++;I think we need to have a cap, otherwise we risk overflows in
inet_csk_rto_backoff()Quoted 22 lines
net/ipv4/tcp_output.c +0
-0timeout =tcp_probe0_when (sk,tcp_rto_max (sk)); }else {/* If packet was not sent due to local congestion, net/ipv4/tcp_timer.c +2
-3@@ -419,9 +419,9 @@ static void tcp_probe_timer(struct sock *sk)if (sock_flag (sk,SOCK_DEAD )) {unsigned int rto_max =tcp_rto_max (sk);const bool alive =inet_csk_rto_backoff (icsk, rto_max) < rto_max; +int orphan_probes =tcp_orphan_retries (sk, alive); - max_probes =tcp_orphan_retries (sk, alive); -if (!alive && icsk->icsk_backoff >= max_probes) +if (!alive || icsk->icsk_backoff >= orphan_probes) goto abort;if (tcp_out_of_resources (sk, true))return ; --2.46.0
Quoted 9 lines · Eric Dumazet
Quoted 8 lines · MingMing Wang
From: MingMing Wang <[email protected]>
An orphaned TCP socket can stall indefinitely in FIN-WAIT-1 if the following conditions are met:
1. net.ipv4.tcp_retries2 is set to a value ≤ 8;
2. The peer advertises a zero window, and the window never reopens.
Quoted 15 lines
Quoted 6 lines
Steps to reproduce:
1. Set up two instances with nmap installed: one will act as the server the other as the client
2. Execute on the server:a. lower rmem : `sysctl -w net.ipv4.tcp_rmem="16 32 32"` b. start a listener: `nc -l -p 1234`Quoted 9 lines
3. Execute on the client:
a. lowertcp_retries2:sysctl -w net.ipv4.tcp_retries2=8b. send pakcets:cat /dev/zero | nc <server-ip> 1234c. after five seconds, stop the process:killall nc
4. Execute on the server:killall -STOP nc
5. Expected abnormal result: usingsscommand, we'll notice that the client connection remains stuck in theFIN_WAIT1state, and the backoff counter always be 8 and no longer increased, as shown below:
```
Thanks for your suggestions, Eric. We will prepare the packetdrill test and resend a series of two patches.
Quoted 89 lines
Hi MingMing
Please prepare and share with us a packetdrill test, instead of this 'repro', which is the old way of describing things :/
- This will be easier for us to understand the issue.
- It will be added to existing tests in
tools/testing/selftests/net/packetdrillif your patch is accepted, so that we can make sure future changes are not breaking this again.Ideally, you should attach this packetdrill test in a second patch (thus sending a series of two patches)
Thank you.
Quoted 72 lines
FIN-WAIT-1 0 1389 172.16.0.2:50316 172.16.0.1:1234 cubic wscale:2,7 rto:201 backoff:8 rtt:0.078/0.007 mss:36 ... other fields omitted ... ```6. If we set
tcp_retries2to 15 and repeat the steps above, theFIN_WAIT1state will be forcefully reclaimed after about 5 minutes.During the zero-window probe retry process, it will check whether the current connection is alive or not. If the connection is not alive and the counter of retries exceeds the maximum allowed
max_probes, retry process will be terminated.In our case, when we set
net.ipv4.tcp_retries2to 8 or a less value, according to the current implementation, theicsk->icsk_backoffcounter will be capped atnet.ipv4.tcp_retries2. The value calculated byinet_csk_rto_backoffwill always be too small, which means the computed backoff duration will always be less thanrto_max. As a result, the alive check will always return true. The condition before thegoto abortstatement is an logical AND condition, the abort branch can never be reached.So, the TCP retransmission backoff mechanism has two issues:
1.
icsk->icsk_backoffshould monotonically increase during probe transmission and, upon reaching the maximum backoff limit, the connection should be terminated. However, the backoff value itself must not be capped prematurely — it should only control when to abort.2. The condition for orphaned connection abort was incorrectly based on connection liveness and probe count. It should instead consider whether the number of orphaned probes exceeds the intended limit.
To fix this, introduce a local variable
orphan_probesto track orphan probe attempts separately frommax_probes, which is used for RTO retransmissions. This decouples the two counters and prevents accidental overwrites, ensuring correct timeout behavior for orphaned connections.Fixes b248230c34970 ("tcp: abort orphan sockets stalling on zero window probes")Co-developed-by Dust Li <[email protected]>Signed-off-by Dust Li <[email protected]>Co-developed-by MingMing Wang <[email protected]>Signed-off-by MingMing Wang <[email protected]>---
We couldn't determine the rationale behind the following check intcp_send_probe0():``` if (icsk->icsk_backoff < READ_ONCE(net->ipv4.sysctl_tcp_retries2)) icsk->icsk_backoff++; ```This condition appears to be the root cause of the observed stall. However, it has existed in the kernel for over 20 years — which suggests there might be a historical or subtle reason for its presence.
We would greatly appreciate it if anyone could shed
--- net/ipv4/tcp_output.c | 4 +--- net/ipv4/tcp_timer.c | 4 ++-- 2 files changed, 3 insertions(+), 5 deletions(-)net/ipv4/tcp_output.c +0
-1@@ -4385,7 +4385,6 @@ void tcp_send_probe0(struct sock *sk) {struct inet_connection_sock *icsk =inet_csk (sk);struct tcp_sock *tp =tcp_sk (sk); -struct net *net =sock_net (sk);unsigned long timeout;int err;
Quoted 7 lines
net/ipv4/tcp_output.c void tcp_send_probe0(struct sock *sk) · line 4403
Quoted 7 lines
net/ipv4/tcp_output.c +1
-2@@ -4401,8 +4400,7 @@ void tcp_send_probe0(struct sock *sk) icsk->icsk_probes_out++;if (err <=0 ) { -if (icsk->icsk_backoff <READ_ONCE (net->ipv4.sysctl_tcp_retries2)) - icsk->icsk_backoff++; + icsk->icsk_backoff++;
We agree with your perspective. Futhermore, as mentioned in the raw text, we would appreciate it if you could clarify whether this cap is just to prevent overflow by using a huge value, or if it should be set to a specific meaningful value.
Quoted 27 lines
I think we need to have a cap, otherwise we risk overflows in
inet_csk_rto_backoff()Quoted 22 lines
net/ipv4/tcp_output.c +0
-0timeout =tcp_probe0_when (sk,tcp_rto_max (sk)); }else {/* If packet was not sent due to local congestion, net/ipv4/tcp_timer.c +2
-3@@ -419,9 +419,9 @@ static void tcp_probe_timer(struct sock *sk)if (sock_flag (sk,SOCK_DEAD )) {unsigned int rto_max =tcp_rto_max (sk);const bool alive =inet_csk_rto_backoff (icsk, rto_max) < rto_max; +int orphan_probes =tcp_orphan_retries (sk, alive); - max_probes =tcp_orphan_retries (sk, alive); -if (!alive && icsk->icsk_backoff >= max_probes) +if (!alive || icsk->icsk_backoff >= orphan_probes) goto abort;if (tcp_out_of_resources (sk, true))return ; --2.46.0