Fix reconnection latency caused by FIN/ACK handling race
Subsystems (4)
cover Fix reconnection latency caused by FIN/ACK handling race
From: SeongJae Park <[email protected]>
When closing a connection, the two acks that required to change closing socket's status to FIN_WAIT_2 and then TIME_WAIT could be processed in reverse order. This is possible in RSS disabled environments such as a connection inside a host.
For example, expected state transitions and required packets for the disconnection will be similar to below flow.
00 (Process A) (Process B) 01 ESTABLISHED ESTABLISHED 02 close() 03 FIN_WAIT_1 04 ---FIN--> 05 CLOSE_WAIT 06 <--ACK--- 07 FIN_WAIT_2 08 <--FIN/ACK--- 09 TIME_WAIT 10 ---ACK--> 11 LAST_ACK 12 CLOSED CLOSED
The acks in lines 6 and 8 are the acks. If the line 8 packet is processed before the line 6 packet, it will be just ignored as it is not a expected packet, and the later process of the line 6 packet will change the status of Process A to FIN_WAIT_2, but as it has already handled line 8 packet, it will not go to TIME_WAIT and thus will not send the line 10 packet to Process B. Thus, Process B will left in CLOSE_WAIT status, as below.
00 (Process A) (Process B) 01 ESTABLISHED ESTABLISHED 02 close() 03 FIN_WAIT_1 04 ---FIN--> 05 CLOSE_WAIT 06 (<--ACK---) 07 (<--FIN/ACK---) 08 (fired in right order) 09 <--FIN/ACK--- 10 <--ACK--- 11 (processed in reverse order) 12 FIN_WAIT_2
Later, if the Process B sends SYN to Process A for reconnection using the same port, Process A will responds with an ACK for the last flow, which has no increased sequence number. Thus, Process A will send RST, wait for TIMEOUT_INIT (one second in default), and then try reconnection. If reconnections are frequent, the one second latency spikes can be a big problem. Below is a tcpdump results of the problem:
14.436259 IP 127.0.0.1.45150 > 127.0.0.1.4242: Flags [S], seq 2560603644
14.436266 IP 127.0.0.1.4242 > 127.0.0.1.45150: Flags [.], ack 5, win 512
14.436271 IP 127.0.0.1.45150 > 127.0.0.1.4242: Flags [R], seq 2541101298
/* ONE SECOND DELAY */
15.464613 IP 127.0.0.1.45150 > 127.0.0.1.4242: Flags [S], seq 2560603644Patchset Organization
---------------------
The first patch fix a trivial nit. The second one fix the problem by adjusting the resend delay of the SYN in the case. Finally, the third patch adds a user space test to reproduce this problem.
The patches are based on the v5.5. You can also clone the complete git tree:
$ git clone git://github.com/sjp38/linux -b patches/finack_lat/v1
The web is also available:
https://github.com/sjp38/linux/tree/patches/finack_lat/v1
SeongJae Park (3):
net/ipv4/inet_timewait_sock: Fix inconsistent comments
tcp: Reduce SYN resend delay if a suspicous ACK is received
selftests: net: Add FIN_ACK processing order related latency spike
test
net/ipv4/inet_timewait_sock.c | 1 + net/ipv4/tcp_input.c | 6 +- tools/testing/selftests/net/.gitignore | 2 + tools/testing/selftests/net/Makefile | 2 + tools/testing/selftests/net/fin_ack_lat.sh | 42 ++++++++++ .../selftests/net/fin_ack_lat_accept.c | 49 +++++++++++ .../selftests/net/fin_ack_lat_connect.c | 81 +++++++++++++++++++ 7 files changed, 182 insertions(+), 1 deletion(-) create mode 100755 tools/testing/selftests/net/fin_ack_lat.sh create mode 100644 tools/testing/selftests/net/fin_ack_lat_accept.c create mode 100644 tools/testing/selftests/net/fin_ack_lat_connect.c
Signature · 1 line
2.17.1
PATCH 1/3 net/ipv4/inet_timewait_sock: Fix inconsistent comments
From: SeongJae Park <[email protected]>
Commit ec94c2696f0b ("tcp/dccp: avoid one atomic operation for timewait hashdance") mistakenly erased a comment for the second step of inet_twsk_hashdance(). This commit restores it for better readability.
--- net/ipv4/inet_timewait_sock.c | 1 + 1 file changed, 1 insertion(+)
net/ipv4/inet_timewait_sock.c +1 -0
@@ -120,6 +120,7 @@ void inet_twsk_hashdance(struct inet_timewait_sock *tw, struct sock *sk,spin_lock (lock); +/* Step 2: Hash TW into tcp ehash chain. */ inet_twsk_add_node_rcu (tw, &ehead->chain);/* Step 3: Remove SK from hash chain */
Signature · 1 line
2.17.1
Re: [PATCH 1/3] net/ipv4/inet_timewait_sock: Fix inconsistent comments
Quoted 18 lines · On Fri, Jan 31, 2020 at 4:24 AM <[email protected]> wrote:
From: SeongJae Park <[email protected]>
Commit
ec94c2696f0b("tcp/dccp: avoid one atomic operation for timewait hashdance") mistakenly erased a comment for the second step ofinet_twsk_hashdance(). This commit restores it for better readability.Signed-off-by SeongJae Park <[email protected]>--- net/ipv4/inet_timewait_sock.c | 1 + 1 file changed, 1 insertion(+)net/ipv4/inet_timewait_sock.c +0
-0
net/ipv4/inet_timewait_sock.c void inet_twsk_hashdance(struct inet_timewait_sock *tw, struct sock *sk, · line 123
Quoted 4 lines
net/ipv4/inet_timewait_sock.c +1
-0@@ -120,6 +120,7 @@ void inet_twsk_hashdance(struct inet_timewait_sock *tw, struct sock *sk,spin_lock (lock); +/* Step 2: Hash TW into tcp ehash chain. */
This comment adds no value, please do not bring it back.
net-next is closed, now is not the time for cosmetic changes.
Also take a look at Documentation/networking/netdev-FAQ.rst
Re: Re: [PATCH 1/3] net/ipv4/inet_timewait_sock: Fix inconsistent comments
Quoted 22 lines · On Fri, 31 Jan 2020 06:54:53 -0800 Eric Dumazet <[email protected]> wrote:
Quoted 21 lines · On Fri, Jan 31, 2020 at 4:24 AM <[email protected]> wrote:
From: SeongJae Park <[email protected]>
Commit
ec94c2696f0b("tcp/dccp: avoid one atomic operation for timewait hashdance") mistakenly erased a comment for the second step ofinet_twsk_hashdance(). This commit restores it for better readability.Signed-off-by SeongJae Park <[email protected]>--- net/ipv4/inet_timewait_sock.c | 1 + 1 file changed, 1 insertion(+)net/ipv4/inet_timewait_sock.c +0
-0@@ -120,6 +120,7 @@ void inet_twsk_hashdance(struct inet_timewait_sock *tw, struct sock *sk,spin_lock (lock);
Quoted 7 lines
net/ipv4/inet_timewait_sock.c void inet_twsk_hashdance(struct inet_timewait_sock *tw, struct sock *sk, · line 123
Quoted 1 line
net/ipv4/inet_timewait_sock.c +1
-0+/* Step 2: Hash TW into tcp ehash chain. */ This comment adds no value, please do not bring it back.
net-next is closed, now is not the time for cosmetic changes.
Also take a look at
Documentation/networking/netdev-FAQ.rst
Thank you for this kind reference. Will drop this in next spin.
Thanks,
SeongJae Park
Quoted 1 line
PATCH 2/3 tcp: Reduce SYN resend delay if a suspicous ACK is received
In reply to [email protected]From: SeongJae Park <[email protected]>
When closing a connection, the two acks that required to change closing socket's status to FIN_WAIT_2 and then TIME_WAIT could be processed in reverse order. This is possible in RSS disabled environments such as a connection inside a host.
For example, expected state transitions and required packets for the disconnection will be similar to below flow.
00 (Process A) (Process B) 01 ESTABLISHED ESTABLISHED 02 close() 03 FIN_WAIT_1 04 ---FIN--> 05 CLOSE_WAIT 06 <--ACK--- 07 FIN_WAIT_2 08 <--FIN/ACK--- 09 TIME_WAIT 10 ---ACK--> 11 LAST_ACK 12 CLOSED CLOSED
The acks in lines 6 and 8 are the acks. If the line 8 packet is processed before the line 6 packet, it will be just ignored as it is not a expected packet, and the later process of the line 6 packet will change the status of Process A to FIN_WAIT_2, but as it has already handled line 8 packet, it will not go to TIME_WAIT and thus will not send the line 10 packet to Process B. Thus, Process B will left in CLOSE_WAIT status, as below.
00 (Process A) (Process B) 01 ESTABLISHED ESTABLISHED 02 close() 03 FIN_WAIT_1 04 ---FIN--> 05 CLOSE_WAIT 06 (<--ACK---) 07 (<--FIN/ACK---) 08 (fired in right order) 09 <--FIN/ACK--- 10 <--ACK--- 11 (processed in reverse order) 12 FIN_WAIT_2
Later, if the Process B sends SYN to Process A for reconnection using the same port, Process A will responds with an ACK for the last flow, which has no increased sequence number. Thus, Process A will send RST, wait for TIMEOUT_INIT (one second in default), and then try reconnection. If reconnections are frequent, the one second latency spikes can be a big problem. Below is a tcpdump results of the problem:
14.436259 IP 127.0.0.1.45150 > 127.0.0.1.4242: Flags [S], seq 2560603644
14.436266 IP 127.0.0.1.4242 > 127.0.0.1.45150: Flags [.], ack 5, win 512
14.436271 IP 127.0.0.1.45150 > 127.0.0.1.4242: Flags [R], seq 2541101298
/* ONE SECOND DELAY */
15.464613 IP 127.0.0.1.45150 > 127.0.0.1.4242: Flags [S], seq 2560603644This commit mitigates the problem by reducing the delay for the next SYN if the suspicous ACK is received while in SYN_SENT state.
Following commit will add a selftest, which can be also helpful for understanding of this issue.
--- net/ipv4/tcp_input.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
net/ipv4/tcp_input.c +5 -1
@@ -5893,8 +5893,12 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,* the segment and return)" */ if (!after (TCP_SKB_CB (skb)->ack_seq, tp->snd_una) || -after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) +after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) { +/* Previous FIN/ACK or RST/ACK might be ignore. */ +inet_csk_reset_xmit_timer (sk,ICSK_TIME_RETRANS , +TCP_ATO_MIN ,TCP_RTO_MAX ); goto reset_and_undo; + }if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr && !between(tp->rx_opt.rcv_tsecr, tp->retrans_stamp,
Signature · 1 line
2.17.1
Re: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
Quoted 10 lines · On Fri, Jan 31, 2020 at 4:25 AM <[email protected]> wrote:
Signed-off-by SeongJae Park <[email protected]>--- net/ipv4/tcp_input.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)net/ipv4/tcp_input.c +0
-0
net/ipv4/tcp_input.c static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb, · line 5899
Quoted 8 lines
net/ipv4/tcp_input.c +4
-1@@ -5893,8 +5893,12 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,* the segment and return)" */ if (!after (TCP_SKB_CB (skb)->ack_seq, tp->snd_una) || -after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) +after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) { +/* Previous FIN/ACK or RST/ACK might be ignore. */ +inet_csk_reset_xmit_timer (sk,ICSK_TIME_RETRANS , +TCP_ATO_MIN ,TCP_RTO_MAX );
This is not what I suggested.
I suggested implementing a strategy where only the _first_ retransmit would be done earlier.
So you need to look at the current counter of retransmit attempts, then reset the timer if this SYN_SENT
socket never resent a SYN.
We do not want to trigger packet storms, if for some reason the remote peer constantly sends
us the same packet.
Thanks.
Quoted 8 lines
net/ipv4/tcp_input.c +1
-1goto reset_and_undo; + }if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr && !between(tp->rx_opt.rcv_tsecr, tp->retrans_stamp, --2.17.1
Re: Re: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
Quoted 26 lines · On Fri, 31 Jan 2020 07:01:21 -0800 Eric Dumazet <[email protected]> wrote:
Quoted 10 lines · On Fri, Jan 31, 2020 at 4:25 AM <[email protected]> wrote:
Signed-off-by SeongJae Park <[email protected]>--- net/ipv4/tcp_input.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)net/ipv4/tcp_input.c +0
-0net/ipv4/tcp_input.c static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb, · line 5899
Quoted 8 lines
net/ipv4/tcp_input.c +4
-1@@ -5893,8 +5893,12 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,* the segment and return)" */ if (!after (TCP_SKB_CB (skb)->ack_seq, tp->snd_una) || -after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) +after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) { +/* Previous FIN/ACK or RST/ACK might be ignore. */ +inet_csk_reset_xmit_timer (sk,ICSK_TIME_RETRANS , +TCP_ATO_MIN ,TCP_RTO_MAX );This is not what I suggested.
I suggested implementing a strategy where only the _first_ retransmit would be done earlier.
Quoted 7 lines
So you need to look at the current counter of retransmit attempts, then reset the timer if this
SYN_SENT
socket never resent a SYN.We do not want to trigger packet storms, if for some reason the remote peer constantly sends
us the same packet.
You're right, I missed the important point, thank you for pointing it. Among retransmission related fields of 'tcp_sock', I think '->total_retrans' would fit for this check. How about below change?
```
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 2a976f57f7e7..29fc0e4da931 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -5893,8 +5893,14 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
* the segment and return)"
*/
if (!after(TCP_SKB_CB(skb)->ack_seq, tp->snd_una) ||
- after(TCP_SKB_CB(skb)->ack_seq, tp->snd_nxt))
+ after(TCP_SKB_CB(skb)->ack_seq, tp->snd_nxt)) {
+ /* Previous FIN/ACK or RST/ACK might be ignored. */
+ if (tp->total_retrans == 0)
+ inet_csk_reset_xmit_timer(sk,
+ ICSK_TIME_RETRANS, TCP_ATO_MIN,
+ TCP_RTO_MAX);
goto reset_and_undo;
+ }
if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr &&
!between(tp->rx_opt.rcv_tsecr, tp->retrans_stamp,
```Thanks,
SeongJae Park
Quoted 12 lines
Thanks.
Quoted 8 lines
net/ipv4/tcp_input.c +1
-1goto reset_and_undo; + }if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr && !between(tp->rx_opt.rcv_tsecr, tp->retrans_stamp, --2.17.1
Re: Re: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
Quoted 47 lines · On Fri, Jan 31, 2020 at 8:12 AM <[email protected]> wrote:
Quoted 26 lines · On Fri, 31 Jan 2020 07:01:21 -0800 Eric Dumazet <[email protected]> wrote:
Quoted 10 lines · On Fri, Jan 31, 2020 at 4:25 AM <[email protected]> wrote:
Signed-off-by SeongJae Park <[email protected]>--- net/ipv4/tcp_input.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)net/ipv4/tcp_input.c +0
-0net/ipv4/tcp_input.c static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb, · line 5899
Quoted 8 lines
net/ipv4/tcp_input.c +4
-1@@ -5893,8 +5893,12 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,* the segment and return)" */ if (!after (TCP_SKB_CB (skb)->ack_seq, tp->snd_una) || -after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) +after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) { +/* Previous FIN/ACK or RST/ACK might be ignore. */ +inet_csk_reset_xmit_timer (sk,ICSK_TIME_RETRANS , +TCP_ATO_MIN ,TCP_RTO_MAX );This is not what I suggested.
I suggested implementing a strategy where only the _first_ retransmit would be done earlier.
Quoted 7 lines
So you need to look at the current counter of retransmit attempts, then reset the timer if this
SYN_SENT
socket never resent a SYN.We do not want to trigger packet storms, if for some reason the remote peer constantly sends
us the same packet.You're right, I missed the important point, thank you for pointing it. Among retransmission related fields of '
tcp_sock', I think '->total_retrans' would fit for this check. How about below change?```
net/ipv4/tcp_input.c +0
-0
net/ipv4/tcp_input.c static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb, · line 5898
Quoted 7 lines
net/ipv4/tcp_input.c +3
-1@@ -5893,8 +5893,14 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,* the segment and return)" */ if (!after (TCP_SKB_CB (skb)->ack_seq, tp->snd_una) || -after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) +after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) { +/* Previous FIN/ACK or RST/ACK might be ignored. */ +if (tp->total_retrans ==0 )
canonical fied would be icsk->icsk_retransmits (look in net/ipv4/tcp_timer.c )
AFAIK, it seems we forget to clear tp->total_retrans in tcp_disconnect() I will send a patch for this tp->total_retrans thing.
Quoted 25 lines
net/ipv4/tcp_input.c +4
-0+inet_csk_reset_xmit_timer (sk, +ICSK_TIME_RETRANS ,TCP_ATO_MIN , +TCP_RTO_MAX ); goto reset_and_undo; + }if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr && !between(tp->rx_opt.rcv_tsecr, tp->retrans_stamp,```
Thanks,
SeongJae ParkQuoted 12 lines
Thanks.
Quoted 8 lines
net/ipv4/tcp_input.c +1
-1goto reset_and_undo; + }if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr && !between(tp->rx_opt.rcv_tsecr, tp->retrans_stamp, --2.17.1
Re: Re: Re: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
Quoted 58 lines · On Fri, 31 Jan 2020 08:55:08 -0800 Eric Dumazet <[email protected]> wrote:
Quoted 47 lines · On Fri, Jan 31, 2020 at 8:12 AM <[email protected]> wrote:
Quoted 26 lines · On Fri, 31 Jan 2020 07:01:21 -0800 Eric Dumazet <[email protected]> wrote:
Quoted 10 lines · On Fri, Jan 31, 2020 at 4:25 AM <[email protected]> wrote:
Signed-off-by SeongJae Park <[email protected]>--- net/ipv4/tcp_input.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)net/ipv4/tcp_input.c +0
-0net/ipv4/tcp_input.c static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb, · line 5899
Quoted 8 lines
net/ipv4/tcp_input.c +4
-1@@ -5893,8 +5893,12 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,* the segment and return)" */ if (!after (TCP_SKB_CB (skb)->ack_seq, tp->snd_una) || -after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) +after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) { +/* Previous FIN/ACK or RST/ACK might be ignore. */ +inet_csk_reset_xmit_timer (sk,ICSK_TIME_RETRANS , +TCP_ATO_MIN ,TCP_RTO_MAX );This is not what I suggested.
I suggested implementing a strategy where only the _first_ retransmit would be done earlier.
Quoted 7 lines
So you need to look at the current counter of retransmit attempts, then reset the timer if this
SYN_SENT
socket never resent a SYN.We do not want to trigger packet storms, if for some reason the remote peer constantly sends
us the same packet.You're right, I missed the important point, thank you for pointing it. Among retransmission related fields of '
tcp_sock', I think '->total_retrans' would fit for this check. How about below change?```
net/ipv4/tcp_input.c +0
-0net/ipv4/tcp_input.c static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb, · line 5898
Quoted 7 lines
net/ipv4/tcp_input.c +3
-1@@ -5893,8 +5893,14 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,* the segment and return)" */ if (!after (TCP_SKB_CB (skb)->ack_seq, tp->snd_una) || -after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) +after (TCP_SKB_CB (skb)->ack_seq, tp->snd_nxt)) { +/* Previous FIN/ACK or RST/ACK might be ignored. */ +if (tp->total_retrans ==0 )canonical fied would be
icsk->icsk_retransmits(look innet/ipv4/tcp_timer.c)
Quoted 2 lines
AFAIK, it seems we forget to clear
tp->total_retransintcp_disconnect()I will send a patch for thistp->total_retransthing.
Oh, then I will use 'tcsk->icsk_retransmits' instead of 'tp->total_retrans', in next spin. May I also ask you to Cc me for your 'tp->total_retrans' fix patch?
Thanks,
SeongJae Park
Quoted 27 lines
Quoted 26 lines
net/ipv4/tcp_input.c +4
-0+inet_csk_reset_xmit_timer (sk, +ICSK_TIME_RETRANS ,TCP_ATO_MIN , +TCP_RTO_MAX ); goto reset_and_undo; + }if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr && !between(tp->rx_opt.rcv_tsecr, tp->retrans_stamp,```
Thanks,
SeongJae ParkQuoted 12 lines
Thanks.
Quoted 8 lines
net/ipv4/tcp_input.c +1
-1goto reset_and_undo; + }if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr && !between(tp->rx_opt.rcv_tsecr, tp->retrans_stamp, --2.17.1
Re: Re: Re: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
Quoted 3 lines · On Fri, Jan 31, 2020 at 9:05 AM <[email protected]> wrote:
Oh, then I will use '
tcsk->icsk_retransmits' instead of 'tp->total_retrans', in next spin. May I also ask you to Cc me for your 'tp->total_retrans' fix patch?
Sure, but I usually send my patches to netdev@
Please subscribe to the list if you want to get a copy of all TCP patches in the future.
Neal Cardwell
maintainerRe: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
In reply to [email protected]Quoted 11 lines · On Fri, Jan 31, 2020 at 7:25 AM <[email protected]> wrote:
From: SeongJae Park <[email protected]>
When closing a connection, the two acks that required to change closing socket's status to
FIN_WAIT_2and thenTIME_WAITcould be processed in reverse order. This is possible in RSS disabled environments such as a connection inside a host.For example, expected state transitions and required packets for the disconnection will be similar to below flow.
Quoted 13 lines
00 (Process A) (Process B)
01 ESTABLISHED ESTABLISHED
02 close()
03 FIN_WAIT_1
04 ---FIN-->
05 CLOSE_WAIT
06 <--ACK---
07 FIN_WAIT_2
08 <--FIN/ACK---
09 TIME_WAIT
10 ---ACK-->
11 LAST_ACK
12 CLOSED CLOSEDAFAICT this sequence is not quite what would happen, and that it would be different starting in line 8, and would unfold as follows:
08 close()
09 LAST_ACK
10 <--FIN/ACK---
11 TIME_WAIT
12 ---ACK-->
13 CLOSED CLOSEDQuoted 3 lines
The acks in lines 6 and 8 are the acks. If the line 8 packet is processed before the line 6 packet, it will be just ignored as it is not a expected packet,
AFAICT that is where the bug starts.
AFAICT, from first principles, when process A receives the FIN/ACK it should move to TIME_WAIT even if it has not received a preceding ACK. That's because ACKs are cumulative. So receiving a later cumulative ACK conveys all the information in the previous ACKs.
Also, consider the de facto standard state transition diagram from "TCP/IP Illustrated, Volume 2: The Implementation", by Wright and Stevens, e.g.:
https://courses.cs.washington.edu/courses/cse461/19sp/lectures/TCPIP_State_Transition_Diagram.pdf
This first-principles analysis agrees with the Wright/Stevens diagram, which says that a connection in FIN_WAIT_1 that receives a FIN/ACK should move to TIME_WAIT.
This seems like a faster and more robust solution than installing special timers.
Thoughts?
neal
Re: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
Quoted 54 lines · Neal Cardwell
Quoted 11 lines · On Fri, Jan 31, 2020 at 7:25 AM <[email protected]> wrote:
From: SeongJae Park <[email protected]>
When closing a connection, the two acks that required to change closing socket's status to
FIN_WAIT_2and thenTIME_WAITcould be processed in reverse order. This is possible in RSS disabled environments such as a connection inside a host.For example, expected state transitions and required packets for the disconnection will be similar to below flow.
Quoted 13 lines
00 (Process A) (Process B) 01 ESTABLISHED ESTABLISHED 02 close() 03 FIN_WAIT_1 04 ---FIN--> 05 CLOSE_WAIT 06 <--ACK--- 07 FIN_WAIT_2 08 <--FIN/ACK--- 09 TIME_WAIT 10 ---ACK--> 11 LAST_ACK 12 CLOSED CLOSEDAFAICT this sequence is not quite what would happen, and that it would be different starting in line 8, and would unfold as follows:
08 close() 09 LAST_ACK 10 <--FIN/ACK--- 11 TIME_WAIT 12 ---ACK--> 13 CLOSED CLOSEDQuoted 3 lines
The acks in lines 6 and 8 are the acks. If the line 8 packet is processed before the line 6 packet, it will be just ignored as it is not a expected packet,
AFAICT that is where the bug starts.
AFAICT, from first principles, when process A receives the FIN/ACK it should move to
TIME_WAITeven if it has not received a preceding ACK. That's because ACKs are cumulative. So receiving a later cumulative ACK conveys all the information in the previous ACKs.Also, consider the de facto standard state transition diagram from "TCP/IP Illustrated, Volume 2: The Implementation", by Wright and Stevens, e.g.:
https://courses.cs.washington.edu/courses/cse461/19sp/lectures/TCPIP_State_Transition_Diagram.pdf
Quoted 8 lines
This first-principles analysis agrees with the Wright/Stevens diagram, which says that a connection in
FIN_WAIT_1that receives a FIN/ACK should move toTIME_WAIT.This seems like a faster and more robust solution than installing special timers.
Thoughts?
This is orthogonal I think.
No matter how hard we fix the other side, we should improve the active side.
Since we send a RST, sending the SYN a few ms after the RST seems way better than waiting 1 second as if we received no packet at all.
Receiving this ACK tells us something about networking health, no need to be very cautious about the next attempt.
Of course, if you have a fix for the passive side, that would be nice to review !
Neal Cardwell
maintainerRe: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
Quoted 70 lines · Eric Dumazet
Quoted 54 lines · Neal Cardwell
Quoted 11 lines · On Fri, Jan 31, 2020 at 7:25 AM <[email protected]> wrote:
From: SeongJae Park <[email protected]>
When closing a connection, the two acks that required to change closing socket's status to
FIN_WAIT_2and thenTIME_WAITcould be processed in reverse order. This is possible in RSS disabled environments such as a connection inside a host.For example, expected state transitions and required packets for the disconnection will be similar to below flow.
Quoted 13 lines
00 (Process A) (Process B) 01 ESTABLISHED ESTABLISHED 02 close() 03 FIN_WAIT_1 04 ---FIN--> 05 CLOSE_WAIT 06 <--ACK--- 07 FIN_WAIT_2 08 <--FIN/ACK--- 09 TIME_WAIT 10 ---ACK--> 11 LAST_ACK 12 CLOSED CLOSEDAFAICT this sequence is not quite what would happen, and that it would be different starting in line 8, and would unfold as follows:
08 close() 09 LAST_ACK 10 <--FIN/ACK--- 11 TIME_WAIT 12 ---ACK--> 13 CLOSED CLOSEDQuoted 3 lines
The acks in lines 6 and 8 are the acks. If the line 8 packet is processed before the line 6 packet, it will be just ignored as it is not a expected packet,
AFAICT that is where the bug starts.
AFAICT, from first principles, when process A receives the FIN/ACK it should move to
TIME_WAITeven if it has not received a preceding ACK. That's because ACKs are cumulative. So receiving a later cumulative ACK conveys all the information in the previous ACKs.Also, consider the de facto standard state transition diagram from "TCP/IP Illustrated, Volume 2: The Implementation", by Wright and Stevens, e.g.:
https://courses.cs.washington.edu/courses/cse461/19sp/lectures/TCPIP_State_Transition_Diagram.pdf
Quoted 8 lines
This first-principles analysis agrees with the Wright/Stevens diagram, which says that a connection in
FIN_WAIT_1that receives a FIN/ACK should move toTIME_WAIT.This seems like a faster and more robust solution than installing special timers.
Thoughts?
This is orthogonal I think.
Quoted 7 lines
No matter how hard we fix the other side, we should improve the active side.
Since we send a RST, sending the SYN a few ms after the RST seems way better than waiting 1 second as if we received no packet at all.
Receiving this ACK tells us something about networking health, no need to be very cautious about the next attempt.
Yes, all good points. Thanks!
Quoted 1 line
Of course, if you have a fix for the passive side, that would be nice to review !
I looked into fixing this, but my quick reading of the Linux tcp_rcv_state_process() code is that it should behave correctly and that a connection in FIN_WAIT_1 that receives a FIN/ACK should move to TIME_WAIT.
SeongJae, do you happen to have a tcpdump trace of the problematic sequence where the "process A" ends up in FIN_WAIT_2 when it should be in TIME_WAIT?
If I have time I will try to construct a packetdrill case to verify the behavior in this case.
thanks,
neal
Quoted 3 lines
Re: Re: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
Quoted 13 lines · On Fri, 31 Jan 2020 17:11:35 -0500 Neal Cardwell <[email protected]> wrote:
Quoted 12 lines · Eric Dumazet
Quoted 8 lines · Neal Cardwell
Quoted 7 lines · On Fri, Jan 31, 2020 at 7:25 AM <[email protected]> wrote:
From: SeongJae Park <[email protected]>
When closing a connection, the two acks that required to change closing socket's status to
FIN_WAIT_2and thenTIME_WAITcould be processed in reverse order. This is possible in RSS disabled environments such as a connection inside a host.
[...]
Quoted 9 lines
I looked into fixing this, but my quick reading of the Linux
tcp_rcv_state_process()code is that it should behave correctly and that a connection inFIN_WAIT_1that receives a FIN/ACK should move toTIME_WAIT.SeongJae, do you happen to have a tcpdump trace of the problematic sequence where the "process A" ends up in
FIN_WAIT_2when it should be inTIME_WAIT?
Hi Neal,
Yes, I have. You can get it from the previous discussion for this patchset
(https://lore.kernel.org/bpf/[email protected]/). As it also has a reproducer program and how I got the tcpdump trace, I believe you could get your own trace, too. If you have any question or need help, feel free to let me know. :)
Thanks,
SeongJae Park
Quoted 8 lines
If I have time I will try to construct a packetdrill case to verify the behavior in this case.
thanks,
nealQuoted 1 line
Neal Cardwell
maintainerRe: Re: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
Quoted 27 lines · SeongJae Park
Quoted 13 lines · On Fri, 31 Jan 2020 17:11:35 -0500 Neal Cardwell <[email protected]> wrote:
Quoted 12 lines · Eric Dumazet
Quoted 8 lines · Neal Cardwell
Quoted 7 lines · On Fri, Jan 31, 2020 at 7:25 AM <[email protected]> wrote:
From: SeongJae Park <[email protected]>
When closing a connection, the two acks that required to change closing socket's status to
FIN_WAIT_2and thenTIME_WAITcould be processed in reverse order. This is possible in RSS disabled environments such as a connection inside a host.[...]
Quoted 9 lines
I looked into fixing this, but my quick reading of the Linux
tcp_rcv_state_process()code is that it should behave correctly and that a connection inFIN_WAIT_1that receives a FIN/ACK should move toTIME_WAIT.SeongJae, do you happen to have a tcpdump trace of the problematic sequence where the "process A" ends up in
FIN_WAIT_2when it should be inTIME_WAIT?
Quoted 8 lines
Hi Neal,
Yes, I have. You can get it from the previous discussion for this patchset
(https://lore.kernel.org/bpf/[email protected]/). As it also has a reproducer program and how I got the tcpdump trace, I believe you could get your own trace, too. If you have any question or need help, feel free to let me know. :)
Great. Thank you for the pointer.
I had one quick question: in the message:
https://lore.kernel.org/bpf/[email protected]/
... it showed a trace with the client sending a RST/ACK, but this email thread shows a FIN/ACK. I am curious about the motivation for the difference?
Anyway, thanks for the report, and thanks to Eric for further clarifying!
neal
Re: Re: Re: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
Quoted 37 lines · On Fri, 31 Jan 2020 22:55:34 -0500 Neal Cardwell <[email protected]> wrote:
Quoted 27 lines · SeongJae Park
Quoted 13 lines · On Fri, 31 Jan 2020 17:11:35 -0500 Neal Cardwell <[email protected]> wrote:
Quoted 12 lines · Eric Dumazet
Quoted 8 lines · Neal Cardwell
Quoted 7 lines · On Fri, Jan 31, 2020 at 7:25 AM <[email protected]> wrote:
From: SeongJae Park <[email protected]>
When closing a connection, the two acks that required to change closing socket's status to
FIN_WAIT_2and thenTIME_WAITcould be processed in reverse order. This is possible in RSS disabled environments such as a connection inside a host.[...]
Quoted 9 lines
I looked into fixing this, but my quick reading of the Linux
tcp_rcv_state_process()code is that it should behave correctly and that a connection inFIN_WAIT_1that receives a FIN/ACK should move toTIME_WAIT.SeongJae, do you happen to have a tcpdump trace of the problematic sequence where the "process A" ends up in
FIN_WAIT_2when it should be inTIME_WAIT?Quoted 8 lines
Hi Neal,
Yes, I have. You can get it from the previous discussion for this patchset
(https://lore.kernel.org/bpf/[email protected]/). As it also has a reproducer program and how I got the tcpdump trace, I believe you could get your own trace, too. If you have any question or need help, feel free to let me know. :)
Quoted 7 lines
Great. Thank you for the pointer.
I had one quick question: in the message:
https://lore.kernel.org/bpf/[email protected]/
... it showed a trace with the client sending a RST/ACK, but this email thread shows a FIN/ACK. I am curious about the motivation for the difference?
RST/ACK is traced if LINGER socket option is applied in the reproduce program, and FIN/ACK is traced if it is not applied. LINGER applied version shows the spikes more frequently, but the main problem logic has no difference. I confirmed this by testing both of the two versions.
In the previous discussion, I showed the LINGER applied trace. However, as many other documents are using FIN/ACK, I changed the trace to FIN/ACK version in this patchset for better understanding. I will comment that it doesn't matter whether it is FIN/ACK or RST/ACK in the next spin.
Thanks,
SeongJae Park
Quoted 5 lines
Anyway, thanks for the report, and thanks to Eric for further clarifying!
neal
Neal Cardwell
maintainerRe: Re: Re: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
Quoted 9 lines · SeongJae Park
RST/ACK is traced if LINGER socket option is applied in the reproduce program, and FIN/ACK is traced if it is not applied. LINGER applied version shows the spikes more frequently, but the main problem logic has no difference. I confirmed this by testing both of the two versions.
In the previous discussion, I showed the LINGER applied trace. However, as many other documents are using FIN/ACK, I changed the trace to FIN/ACK version in this patchset for better understanding. I will comment that it doesn't matter whether it is FIN/ACK or RST/ACK in the next spin.
Great. Thanks for the details!
neal
Re: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
In reply to Neal CardwellQuoted 11 lines · Neal Cardwell
I looked into fixing this, but my quick reading of the Linux
tcp_rcv_state_process()code is that it should behave correctly and that a connection inFIN_WAIT_1that receives a FIN/ACK should move toTIME_WAIT.SeongJae, do you happen to have a tcpdump trace of the problematic sequence where the "process A" ends up in
FIN_WAIT_2when it should be inTIME_WAIT?If I have time I will try to construct a packetdrill case to verify the behavior in this case.
Unfortunately you wont be able to reproduce the issue with packetdrill, since it involved packets being processed at the same time (race window)
RE: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
From: Eric Dumazet
Quoted 12 lines
Sent: 31 January 2020 22:54
Quoted 9 lines · Neal Cardwell
I looked into fixing this, but my quick reading of the Linux
tcp_rcv_state_process()code is that it should behave correctly and that a connection inFIN_WAIT_1that receives a FIN/ACK should move toTIME_WAIT.SeongJae, do you happen to have a tcpdump trace of the problematic sequence where the "process A" ends up in
FIN_WAIT_2when it should be inTIME_WAIT?
Quoted 5 lines
Quoted 2 lines
If I have time I will try to construct a packetdrill case to verify the behavior in this case.
Unfortunately you wont be able to reproduce the issue with packetdrill, since it involved packets being processed at the same time (race window)
You might be able to force the timing race by adding a sleep in one of the code paths.
No good for a regression test, but ok for code testing.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: [PATCH 2/3] tcp: Reduce SYN resend delay if a suspicous ACK is received
Quoted 17 lines · David Laight
From: Eric Dumazet
Quoted 15 lines
Sent: 31 January 2020 22:54
Quoted 11 lines · Neal Cardwell
I looked into fixing this, but my quick reading of the Linux
tcp_rcv_state_process()code is that it should behave correctly and that a connection inFIN_WAIT_1that receives a FIN/ACK should move toTIME_WAIT.SeongJae, do you happen to have a tcpdump trace of the problematic sequence where the "process A" ends up in
FIN_WAIT_2when it should be inTIME_WAIT?If I have time I will try to construct a packetdrill case to verify the behavior in this case.
Quoted 7 lines
Quoted 2 lines
Unfortunately you wont be able to reproduce the issue with packetdrill, since it involved packets being processed at the same time (race window)
You might be able to force the timing race by adding a sleep in one of the code paths.
No good for a regression test, but ok for code testing.
Please take a look at packetdrill, there is no possibility for it to send more than one packet at a time.
Even if we modify packetdrill adding the possibility of feeding packets to its tun device from multiple threads,
the race is tiny and you would have to run the packetdrill thousands of times to eventually trigger the race once.
While the test SeongJae provided is using two threads and regular TCP stack over loopback interface,
it triggers the race more reliably.
PATCH 3/3 selftests: net: Add FIN_ACK processing order related latency spike test
In reply to [email protected]From: SeongJae Park <[email protected]>
This commit adds a test for FIN_ACK process races related reconnection latency spike issues. The issue has described and solved by the previous commit ("tcp: Reduce SYN resend delay if a suspicous ACK is received").
--- tools/testing/selftests/net/.gitignore | 2 + tools/testing/selftests/net/Makefile | 2 + tools/testing/selftests/net/fin_ack_lat.sh | 42 ++++++++++ .../selftests/net/fin_ack_lat_accept.c | 49 +++++++++++ .../selftests/net/fin_ack_lat_connect.c | 81 +++++++++++++++++++ 5 files changed, 176 insertions(+) create mode 100755 tools/testing/selftests/net/fin_ack_lat.sh create mode 100644 tools/testing/selftests/net/fin_ack_lat_accept.c create mode 100644 tools/testing/selftests/net/fin_ack_lat_connect.c
tools/testing/selftests/net/.gitignore +2 -0
@@ -22,3 +22,5 @@ ipv6_flowlabel_mgr so_txtime tcp_fastopen_backup_key nettest +fin_ack_lat_accept +fin_ack_lat_connect
tools/testing/selftests/net/Makefile +2 -0
@@ -11,6 +11,7 @@ TEST_PROGS += udpgso_bench.sh fib_rule_tests.sh msg_zerocopy.sh psock_snd.sh TEST_PROGS += udpgro_bench.sh udpgro.sh test_vxlan_under_vrf.sh reuseport_addr_any.sh TEST_PROGS += test_vxlan_fdb_changelink.sh so_txtime.sh ipv6_flowlabel.sh TEST_PROGS += tcp_fastopen_backup_key.sh fcnal-test.sh l2tp.sh traceroute.sh +TEST_PROGS += fin_ack_lat.sh TEST_PROGS_EXTENDED := in_netns.sh TEST_GEN_FILES = socket nettest TEST_GEN_FILES += psock_fanout psock_tpacket msg_zerocopy reuseport_addr_any
@@ -18,6 +19,7 @@ TEST_GEN_FILES += tcp_mmap tcp_inq psock_snd txring_overwrite TEST_GEN_FILES += udpgso udpgso_bench_tx udpgso_bench_rx ip_defrag TEST_GEN_FILES += so_txtime ipv6_flowlabel ipv6_flowlabel_mgr TEST_GEN_FILES += tcp_fastopen_backup_key +TEST_GEN_FILES += fin_ack_lat_accept fin_ack_lat_connect TEST_GEN_PROGS = reuseport_bpf reuseport_bpf_cpu reuseport_bpf_numa TEST_GEN_PROGS += reuseport_dualstack reuseaddr_conflict tls
tools/testing/selftests/net/fin_ack_lat.sh new file +42 -0
42 lines not on this page · raw message
tools/testing/selftests/net/fin_ack_lat_accept.c new file +49 -0
49 lines not on this page · raw message
tools/testing/selftests/net/fin_ack_lat_connect.c new file +81 -0
81 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH 3/3] selftests: net: Add FIN_ACK processing order related latency spike test
Quoted 8 lines · On Fri, Jan 31, 2020 at 4:25 AM <[email protected]> wrote:
From: SeongJae Park <[email protected]>
This commit adds a test for
FIN_ACKprocess races related reconnection latency spike issues. The issue has described and solved by the previous commit ("tcp: Reduce SYN resend delay if a suspicous ACK is received").
I do not know for other tests, but using a hard coded port (4242) is going to be flakky, since the port might be already used.
Please make sure to run tests on a separate namespace.
Re: Re: [PATCH 3/3] selftests: net: Add FIN_ACK processing order related latency spike test
Quoted 10 lines · On Fri, 31 Jan 2020 06:56:13 -0800 Eric Dumazet <[email protected]> wrote:
Quoted 8 lines · On Fri, Jan 31, 2020 at 4:25 AM <[email protected]> wrote:
From: SeongJae Park <[email protected]>
This commit adds a test for
FIN_ACKprocess races related reconnection latency spike issues. The issue has described and solved by the previous commit ("tcp: Reduce SYN resend delay if a suspicous ACK is received").
Quoted 4 lines
I do not know for other tests, but using a hard coded port (4242) is going to be flakky, since the port might be already used.
Please make sure to run tests on a separate namespace.
Agreed, will do so in next spin.
Thanks,
SeongJae Park
From: [email protected]
Quoted 1 line
Sent: 31 January 2020 12:24
...
Quoted 8 lines
The acks in lines 6 and 8 are the acks. If the line 8 packet is processed before the line 6 packet, it will be just ignored as it is not a expected packet, and the later process of the line 6 packet will change the status of Process A to
FIN_WAIT_2, but as it has already handled line 8 packet, it will not go toTIME_WAITand thus will not send the line 10 packet to Process B. Thus, Process B will left inCLOSE_WAITstatus, as below.
Quoted 13 lines
00 (Process A) (Process B) 01 ESTABLISHED ESTABLISHED 02 close() 03 FIN_WAIT_1 04 ---FIN--> 05 CLOSE_WAIT 06 (<--ACK---) 07 (<--FIN/ACK---) 08 (fired in right order) 09 <--FIN/ACK--- 10 <--ACK--- 11 (processed in reverse order) 12 FIN_WAIT_2
Why doesn't A treat the FIN/ACK (09) as valid (as if the ACK had got lost) and then ignore the ACK (10) because it refers to a closed socket?
I presume that B sends two ACKs (06 and 07) because it can sit in an intermediate state and the first ACK stops the FIN being resent?
I've implemented lots of protocols in my time, but not TCP.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 25 lines · On Fri, 31 Jan 2020 14:00:27 +0000 David Laight <[email protected]> wrote:
From: [email protected]
Quoted 1 line
Sent: 31 January 2020 12:24
...
Quoted 8 lines
The acks in lines 6 and 8 are the acks. If the line 8 packet is processed before the line 6 packet, it will be just ignored as it is not a expected packet, and the later process of the line 6 packet will change the status of Process A to
FIN_WAIT_2, but as it has already handled line 8 packet, it will not go toTIME_WAITand thus will not send the line 10 packet to Process B. Thus, Process B will left inCLOSE_WAITstatus, as below.Quoted 13 lines
00 (Process A) (Process B) 01 ESTABLISHED ESTABLISHED 02 close() 03 FIN_WAIT_1 04 ---FIN--> 05 CLOSE_WAIT 06 (<--ACK---) 07 (<--FIN/ACK---) 08 (fired in right order) 09 <--FIN/ACK--- 10 <--ACK--- 11 (processed in reverse order) 12 FIN_WAIT_2
Quoted 3 lines
Why doesn't A treat the FIN/ACK (09) as valid (as if the ACK had got lost) and then ignore the ACK (10) because it refers to a closed socket?
Because the TCP protocol (RFC 793) doesn't have such speculation. TCP is stateful protocol. Thus, packets arrived in unexpected state are not required to be respected, AFAIU.
Quoted 4 lines
I presume that B sends two ACKs (06 and 07) because it can sit in an intermediate state and the first ACK stops the FIN being resent?
I think there is no such presume in the protocol, either.
Quoted 2 lines
I've implemented lots of protocols in my time, but not TCP.
If you find anything I'm misunderstanding, please don't hesitate to yell at me. Hope the previous discussion[1] regarding this issue to be helpful.
Thanks,
SeongJae Park
[1] https://lore.kernel.org/bpf/[email protected]/
Quoted 6 lines
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)