net: tcp: move to timewait when receiving data post active-close
Subsystems (3)
PATCH 1/1 net: tcp: move to timewait when receiving data post active-close
RFC 1122, 4.2.2.13:
[..] if new data is received after CLOSE is called, its TCP SHOULD send a RST to show that data was lost.
When a connection is closed actively, it MUST linger in TIME-WAIT state [..].
We reset a connection, but destroy state immediately.
After discussing this with Hannes, we decided it was preferable to also move to TW state to avoid immediate port reuse.
packetdrill testcase:
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < S 0:0(0) win 29200 <mss 1460>
0.100 > S. 0:0(0) ack 1 <mss 1460>
0.200 < . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
// close our side.
0.210 close(4) = 0
// we should expect to see FIN now, sk moves to FIN_WAIT_1 0.210 > F. 1:1(0) ack 1 win 29200
// receive data, but sk already closed -> Reset 0.300 < P. 1:1001(1000) ack 1 win 46
0.300 > R 1:1(0) win 0
---
We got complaint from customer that CLOSED state transition is RFC violation.
The advantage of current behaviour is that further packets will also result in RST.
I'm interested if others think its worth changing this now.
Thanks.
net/ipv4/tcp_input.c +15 -4
@@ -5904,11 +5904,16 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb)break ; } -if (tp->linger2 <0 || - (TCP_SKB_CB (skb)->end_seq !=TCP_SKB_CB (skb)->seq && -after (TCP_SKB_CB (skb)->end_seq - th->fin, tp->rcv_nxt))) { +if (tp->linger2 <0 ) { +NET_INC_STATS_BH (sock_net (sk),LINUX_MIB_TCPABORTONDATA );tcp_done (sk); +return 1 ; + } + +if (TCP_SKB_CB (skb)->end_seq !=TCP_SKB_CB (skb)->seq && +after (TCP_SKB_CB (skb)->end_seq - th->fin, tp->rcv_nxt)) {NET_INC_STATS_BH (sock_net (sk),LINUX_MIB_TCPABORTONDATA ); +tcp_time_wait (sk,TCP_TIME_WAIT ,0 );return 1 ; }
@@ -5966,7 +5971,13 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb)if (TCP_SKB_CB (skb)->end_seq !=TCP_SKB_CB (skb)->seq &&after (TCP_SKB_CB (skb)->end_seq - th->fin, tp->rcv_nxt)) {NET_INC_STATS_BH (sock_net (sk),LINUX_MIB_TCPABORTONDATA ); -tcp_reset (sk); + +if (sk->sk_state ==TCP_CLOSE_WAIT || + sk->sk_state ==TCP_LAST_ACK ) +tcp_reset (sk); +else +tcp_time_wait (sk,TCP_TIME_WAIT ,0 ); +return 1 ; } }
Signature · 1 line
2.4.10
Quoted 14 lines · Florian Westphal
RFC 1122, 4.2.2.13:
[..] if new data is received after CLOSE is called, its TCP SHOULD send a RST to show that data was lost.When a connection is closed actively, it MUST linger in TIME-WAIT state [..].
We reset a connection, but destroy state immediately.
After discussing this with Hannes, we decided it was preferable to also move to TW state to avoid immediate port reuse.
packetdrill testcase:
Quoted 15 lines
0.000
socket(..., SOCK_STREAM, IPPROTO_TCP)= 3 0.000setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4)= 0 0.000bind(3, ..., ...)= 0
0.000listen(3, 1)= 0
0.100 < S 0:0(0) win 29200 <mss 1460>
0.100 > S. 0:0(0) ack 1 <mss 1460>
0.200 < . 1:1(0) ack 1 win 257
0.200accept(3, ..., ...)= 4
// close our side.
0.210 close(4) = 0
// we should expect to see FIN now, sk moves toFIN_WAIT_10.210 > F. 1:1(0) ack 1 win 29200
// receive data, but sk already closed -> Reset 0.300 < P. 1:1001(1000) ack 1 win 46
0.300 > R 1:1(0) win 0
Thanks for using packetdrill ;)
This packetdrill test shows nothing special regarding your patch, it should work right now with current kernels ???
lpk51:~# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes lpk51:~# ./packetdrill Florian.pkt 07:26:49.656548 IP 192.0.2.1.59883 > 192.168.155.177.8080: Flags [S], seq 0, win 29200, options [mss 1460], length 0 07:26:49.656611 IP 192.168.155.177.8080 > 192.0.2.1.59883: Flags [S.], seq 1433468786, ack 1, win 29200, options [mss 1460], length 0 07:26:49.756530 IP 192.0.2.1.59883 > 192.168.155.177.8080: Flags [.], ack 1, win 257, length 0 07:26:49.766529 IP 192.168.155.177.8080 > 192.0.2.1.59883: Flags [F.], seq 1, ack 1, win 29200, length 0 07:26:49.856529 IP 192.0.2.1.59883 > 192.168.155.177.8080: Flags [P.], seq 1:1001, ack 1, win 46, length 1000: HTTP 07:26:49.856560 IP 192.168.155.177.8080 > 192.0.2.1.59883: Flags [R], seq 1433468787, win 0, length 0 07:26:49.856806 IP 192.0.2.1.59883 > 192.168.155.177.8080: Flags [R.], seq 1001, ack 1, win 46, length 0
Quoted 32 lines · Eric Dumazet
Quoted 14 lines · Florian Westphal
RFC 1122, 4.2.2.13:
[..] if new data is received after CLOSE is called, its TCP SHOULD send a RST to show that data was lost.When a connection is closed actively, it MUST linger in TIME-WAIT state [..].
We reset a connection, but destroy state immediately.
After discussing this with Hannes, we decided it was preferable to also move to TW state to avoid immediate port reuse.
packetdrill testcase:
Quoted 15 lines
0.000
socket(..., SOCK_STREAM, IPPROTO_TCP)= 3 0.000setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4)= 0 0.000bind(3, ..., ...)= 0
0.000listen(3, 1)= 0
0.100 < S 0:0(0) win 29200 <mss 1460>
0.100 > S. 0:0(0) ack 1 <mss 1460>
0.200 < . 1:1(0) ack 1 win 257
0.200accept(3, ..., ...)= 4
// close our side.
0.210 close(4) = 0
// we should expect to see FIN now, sk moves toFIN_WAIT_10.210 > F. 1:1(0) ack 1 win 29200
// receive data, but sk already closed -> Reset 0.300 < P. 1:1001(1000) ack 1 win 46
0.300 > R 1:1(0) win 0
Quoted 2 lines
This packetdrill test shows nothing special regarding your patch, it should work right now with current kernels ???
Yes, but we kill the socket.
I should have added
0.400 ss -nito state time-wait
as last line...
Before patch: no output
after patch: tw socket shown.
The on-wire behavior doesn't change unless further packets arrive. Old behaviour: more RST
New behaviour: acks+tw timer restart
Sorry for the confusion.
Quoted 6 lines · Florian Westphal
Yes, but we kill the socket.
I should have added
0.400
ss -nito state time-wait
Quoted 8 lines
as last line...
Before patch: no output
after patch: tw socket shown.The on-wire behavior doesn't change unless further packets arrive. Old behaviour: more RST
New behaviour: acks+tw timer restart
Just add few more incoming packets to the packetdrill test then ?
Also, is your customer really _not_ using TCP timestamps ?
This is kind of a requirement for port reuse anyway.
Anyway, having a TIMEWAIT setup after sending a RST makes little sense to me.
When a RST packet is sent, the remote peer will forget everything about this previous connection, and another connect() might reuse the tuple and I do not think we should forbid this. Normal PAWS checks were invented for a good reason.
RFC 1122, 4.2.2.13 can be interpreted in very different ways.
Please show us real issue your customer has.
Quoted 13 lines · Eric Dumazet
Quoted 11 lines · Florian Westphal
Yes, but we kill the socket.
I should have added
0.400
ss -nito state time-waitas last line...
Before patch: no output
after patch: tw socket shown.
Quoted 7 lines
Quoted 3 lines
The on-wire behavior doesn't change unless further packets arrive. Old behaviour: more RST
New behaviour: acks+tw timer restartJust add few more incoming packets to the packetdrill test then ?
Also, is your customer really _not_ using TCP timestamps ?
Windows mostly does not use TCP timestamps. Also we have cases were security folks tell customers to turn off timestamps as they enable attackers to guess uptime. :(
Quoted 9 lines
This is kind of a requirement for port reuse anyway.
Anyway, having a TIMEWAIT setup after sending a RST makes little sense to me.
When a RST packet is sent, the remote peer will forget everything about this previous connection, and another
connect()might reuse the tuple and I do not think we should forbid this. Normal PAWS checks were invented for a good reason.
Still, the RST packet can be dropped along the way. So the teardown of the socket on the other side might not happen.
Bye,
Hannes
Quoted 2 lines · Hannes Frederic Sowa
Still, the RST packet can be dropped along the way. So the teardown of the socket on the other side might not happen.
This is why it is better to send RST for every incoming in-excess packet
Try following packetdrill test, before and after your patch :
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < S 0:0(0) win 29200 <mss 1460>
0.100 > S. 0:0(0) ack 1 <mss 1460>
0.200 < . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
// close our side.
0.210 close(4) = 0
// we should expect to see FIN now, sk moves to FIN_WAIT_1 0.210 > F. 1:1(0) ack 1 win 29200
// receive data, but sk already closed -> Reset
+.010 < P. 1:1001(1000) ack 1 win 46
+0 > R 1:1(0) win 0
// Are we properly sending a RST like prior packet did ? +.010 < P. 1001:2001(1000) ack 1 win 46
+0 > R 1:1(0) win 0
Quoted 6 lines · Eric Dumazet
Quoted 2 lines · Hannes Frederic Sowa
Still, the RST packet can be dropped along the way. So the teardown of the socket on the other side might not happen.
This is why it is better to send RST for every incoming in-excess packet
I'm convinced and marked patch as rejected in patchwork.
Thanks Eric!
Quoted 6 lines · Eric Dumazet
Quoted 2 lines · Hannes Frederic Sowa
Still, the RST packet can be dropped along the way. So the teardown of the socket on the other side might not happen.
This is why it is better to send RST for every incoming in-excess packet
I agree it would be better to send a RST than ACK incoming data. Just thinking out loud, would it make sense to add a sub-state to timewait so we RST all other packets for TIMEWAIT_LEN duration?
Quoted 15 lines
Try following packetdrill test, before and after your patch :
0.000
socket(..., SOCK_STREAM, IPPROTO_TCP)= 3 0.000setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4)= 0 0.000bind(3, ..., ...)= 0
0.000listen(3, 1)= 0
0.100 < S 0:0(0) win 29200 <mss 1460>
0.100 > S. 0:0(0) ack 1 <mss 1460>
0.200 < . 1:1(0) ack 1 win 257
0.200accept(3, ..., ...)= 4
// close our side.
0.210 close(4) = 0
// we should expect to see FIN now, sk moves toFIN_WAIT_10.210 > F. 1:1(0) ack 1 win 29200
Quoted 7 lines
// receive data, but sk already closed -> Reset
+.010 < P. 1:1001(1000) ack 1 win 46
+0 > R 1:1(0) win 0// Are we properly sending a RST like prior packet did ? +.010 < P. 1001:2001(1000) ack 1 win 46
+0 > R 1:1(0) win 0
Thanks,
Hannes
Quoted 11 lines · Hannes Frederic Sowa
Quoted 6 lines · Eric Dumazet
Quoted 2 lines · Hannes Frederic Sowa
Still, the RST packet can be dropped along the way. So the teardown of the socket on the other side might not happen.
This is why it is better to send RST for every incoming in-excess packet
I agree it would be better to send a RST than ACK incoming data. Just thinking out loud, would it make sense to add a sub-state to timewait so we RST all other packets for
TIMEWAIT_LENduration?
Intuitively this timewait creation looks dangerous to me, maybe from an attack surface perspective.
An out of sequence packet sends an advisory RST packet, which is already a hint for building some kinds of attacks.
Now if we also create a timewait socket, and its associated timer, we also add quite a risk of memory and cpu consumption.
So I am fine you create a timewait, only if you can prove there is no change in security.
Thanks !
David Miller
maintainerFrom: Eric Dumazet <[email protected]>
Date: Wed, 18 Nov 2015 07:46:31 -0800
Quoted 9 lines
Anyway, having a TIMEWAIT setup after sending a RST makes little sense to me.
When a RST packet is sent, the remote peer will forget everything about this previous connection, and another
connect()might reuse the tuple and I do not think we should forbid this. Normal PAWS checks were invented for a good reason.RFC 1122, 4.2.2.13 can be interpreted in very different ways.
I think it is clear that once RST is emitted, that connection ID no longer exists, on both ends.
Sender of RST must not have that matching state, and receiver of RST must tear things down.