Skip to contentSkip to conversation

net: tcp: move to timewait when receiving data post active-close

PATCH In review Florian Westphal 4 participants 10 messages Nov 18, 2015 – Nov 19, 2015

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 net: tcp: move to timewait when receiving data post active-close

RFC 1122, 4.2.2.13:
[..] if new data is received after CLOSE is called, its TCP SHOULD send a RST to show that data was lost.

When a connection is closed actively, it MUST linger in TIME-WAIT state [..].

We reset a connection, but destroy state immediately.

After discussing this with Hannes, we decided it was preferable to also move to TW state to avoid immediate port reuse.

packetdrill testcase:

0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < S 0:0(0) win 29200 <mss 1460>
0.100 > S. 0:0(0) ack 1 <mss 1460>
0.200 < . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
// close our side.
0.210 close(4) = 0
// we should expect to see FIN now, sk moves to FIN_WAIT_1 0.210 > F. 1:1(0) ack 1 win 29200
// receive data, but sk already closed -> Reset 0.300 < P. 1:1001(1000) ack 1 win 46
0.300 > R 1:1(0) win 0

---
We got complaint from customer that CLOSED state transition is RFC violation.

The advantage of current behaviour is that further packets will also result in RST.

I'm interested if others think its worth changing this now.

Thanks.

net/ipv4/tcp_input.c +15 -4
index fdd88c3..3bcdad1 100644
@@ -5904,11 +5904,16 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb)
 			break;
 		}
 
-		if (tp->linger2 < 0 ||
-		    (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
-		     after(TCP_SKB_CB(skb)->end_seq - th->fin, tp->rcv_nxt))) {
+		if (tp->linger2 < 0) {
+			NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
 			tcp_done(sk);
+			return 1;
+		}
+
+		if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
+		    after(TCP_SKB_CB(skb)->end_seq - th->fin, tp->rcv_nxt)) {
 			NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
+			tcp_time_wait(sk, TCP_TIME_WAIT, 0);
 			return 1;
 		}
 
@@ -5966,7 +5971,13 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb)
 			if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
 			    after(TCP_SKB_CB(skb)->end_seq - th->fin, tp->rcv_nxt)) {
 				NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
-				tcp_reset(sk);
+
+				if (sk->sk_state == TCP_CLOSE_WAIT ||
+				    sk->sk_state == TCP_LAST_ACK)
+					tcp_reset(sk);
+				else
+					tcp_time_wait(sk, TCP_TIME_WAIT, 0);
+
 				return 1;
 			}
 		}
Signature · 1 line
2.4.10
#1 lore raw
Quoted 14 lines · Florian Westphal

RFC 1122, 4.2.2.13:
[..] if new data is received after CLOSE is called, its TCP SHOULD send a RST to show that data was lost.

When a connection is closed actively, it MUST linger in TIME-WAIT state [..].

We reset a connection, but destroy state immediately.

After discussing this with Hannes, we decided it was preferable to also move to TW state to avoid immediate port reuse.

packetdrill testcase:

Quoted 15 lines

0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < S 0:0(0) win 29200 <mss 1460>
0.100 > S. 0:0(0) ack 1 <mss 1460>
0.200 < . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
// close our side.
0.210 close(4) = 0
// we should expect to see FIN now, sk moves to FIN_WAIT_1 0.210 > F. 1:1(0) ack 1 win 29200
// receive data, but sk already closed -> Reset 0.300 < P. 1:1001(1000) ack 1 win 46
0.300 > R 1:1(0) win 0

Thanks for using packetdrill ;)

This packetdrill test shows nothing special regarding your patch, it should work right now with current kernels ???

lpk51:~# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
lpk51:~# ./packetdrill Florian.pkt 
07:26:49.656548 IP 192.0.2.1.59883 > 192.168.155.177.8080: Flags [S], seq 0, win 29200, options [mss 1460], length 0
07:26:49.656611 IP 192.168.155.177.8080 > 192.0.2.1.59883: Flags [S.], seq 1433468786, ack 1, win 29200, options [mss 1460], length 0
07:26:49.756530 IP 192.0.2.1.59883 > 192.168.155.177.8080: Flags [.], ack 1, win 257, length 0
07:26:49.766529 IP 192.168.155.177.8080 > 192.0.2.1.59883: Flags [F.], seq 1, ack 1, win 29200, length 0
07:26:49.856529 IP 192.0.2.1.59883 > 192.168.155.177.8080: Flags [P.], seq 1:1001, ack 1, win 46, length 1000: HTTP
07:26:49.856560 IP 192.168.155.177.8080 > 192.0.2.1.59883: Flags [R], seq 1433468787, win 0, length 0
07:26:49.856806 IP 192.0.2.1.59883 > 192.168.155.177.8080: Flags [R.], seq 1001, ack 1, win 46, length 0
#2 lore raw
Quoted 32 lines · Eric Dumazet
Quoted 14 lines · Florian Westphal

RFC 1122, 4.2.2.13:
[..] if new data is received after CLOSE is called, its TCP SHOULD send a RST to show that data was lost.

When a connection is closed actively, it MUST linger in TIME-WAIT state [..].

We reset a connection, but destroy state immediately.

After discussing this with Hannes, we decided it was preferable to also move to TW state to avoid immediate port reuse.

packetdrill testcase:

Quoted 15 lines

0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < S 0:0(0) win 29200 <mss 1460>
0.100 > S. 0:0(0) ack 1 <mss 1460>
0.200 < . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
// close our side.
0.210 close(4) = 0
// we should expect to see FIN now, sk moves to FIN_WAIT_1 0.210 > F. 1:1(0) ack 1 win 29200
// receive data, but sk already closed -> Reset 0.300 < P. 1:1001(1000) ack 1 win 46
0.300 > R 1:1(0) win 0

Quoted 2 lines

This packetdrill test shows nothing special regarding your patch, it should work right now with current kernels ???

Yes, but we kill the socket.

I should have added

0.400 ss -nito state time-wait

as last line...

Before patch: no output
after patch: tw socket shown.

The on-wire behavior doesn't change unless further packets arrive. Old behaviour: more RST
New behaviour: acks+tw timer restart

Sorry for the confusion.

#3 lore raw
Quoted 6 lines · Florian Westphal

Yes, but we kill the socket.

I should have added

0.400 ss -nito state time-wait

Quoted 8 lines

as last line...

Before patch: no output
after patch: tw socket shown.

The on-wire behavior doesn't change unless further packets arrive. Old behaviour: more RST
New behaviour: acks+tw timer restart

Just add few more incoming packets to the packetdrill test then ?

Also, is your customer really _not_ using TCP timestamps ?

This is kind of a requirement for port reuse anyway.

Anyway, having a TIMEWAIT setup after sending a RST makes little sense to me.

When a RST packet is sent, the remote peer will forget everything about this previous connection, and another connect() might reuse the tuple and I do not think we should forbid this. Normal PAWS checks were invented for a good reason.

RFC 1122, 4.2.2.13 can be interpreted in very different ways.

Please show us real issue your customer has.

#4 lore raw
Quoted 13 lines · Eric Dumazet
Quoted 11 lines · Florian Westphal

Yes, but we kill the socket.

I should have added

0.400 ss -nito state time-wait

as last line...

Before patch: no output
after patch: tw socket shown.

Quoted 7 lines
Quoted 3 lines

The on-wire behavior doesn't change unless further packets arrive. Old behaviour: more RST
New behaviour: acks+tw timer restart

Just add few more incoming packets to the packetdrill test then ?

Also, is your customer really _not_ using TCP timestamps ?

Windows mostly does not use TCP timestamps. Also we have cases were security folks tell customers to turn off timestamps as they enable attackers to guess uptime. :(

Quoted 9 lines

This is kind of a requirement for port reuse anyway.

Anyway, having a TIMEWAIT setup after sending a RST makes little sense to me.

When a RST packet is sent, the remote peer will forget everything about this previous connection, and another connect() might reuse the tuple and I do not think we should forbid this. Normal PAWS checks were invented for a good reason.

Still, the RST packet can be dropped along the way. So the teardown of the socket on the other side might not happen.

Bye,
Hannes

#5 lore raw
Quoted 2 lines · Hannes Frederic Sowa

Still, the RST packet can be dropped along the way. So the teardown of the socket on the other side might not happen.

This is why it is better to send RST for every incoming in-excess packet

Try following packetdrill test, before and after your patch :

0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < S 0:0(0) win 29200 <mss 1460>
0.100 > S. 0:0(0) ack 1 <mss 1460>
0.200 < . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
// close our side.
0.210 close(4) = 0
// we should expect to see FIN now, sk moves to FIN_WAIT_1 0.210 > F. 1:1(0) ack 1 win 29200

// receive data, but sk already closed -> Reset
+.010 < P. 1:1001(1000) ack 1 win 46
+0 > R 1:1(0) win 0

// Are we properly sending a RST like prior packet did ? +.010 < P. 1001:2001(1000) ack 1 win 46
+0 > R 1:1(0) win 0

#6 lore raw
Quoted 6 lines · Eric Dumazet
Quoted 2 lines · Hannes Frederic Sowa

Still, the RST packet can be dropped along the way. So the teardown of the socket on the other side might not happen.

This is why it is better to send RST for every incoming in-excess packet

I'm convinced and marked patch as rejected in patchwork.

Thanks Eric!

#7 lore raw
In reply to Eric Dumazet
Quoted 6 lines · Eric Dumazet
Quoted 2 lines · Hannes Frederic Sowa

Still, the RST packet can be dropped along the way. So the teardown of the socket on the other side might not happen.

This is why it is better to send RST for every incoming in-excess packet

I agree it would be better to send a RST than ACK incoming data. Just thinking out loud, would it make sense to add a sub-state to timewait so we RST all other packets for TIMEWAIT_LEN duration?

Quoted 15 lines

Try following packetdrill test, before and after your patch :

0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3 0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 0.000 bind(3, ..., ...) = 0
0.000 listen(3, 1) = 0
0.100 < S 0:0(0) win 29200 <mss 1460>
0.100 > S. 0:0(0) ack 1 <mss 1460>
0.200 < . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
// close our side.
0.210 close(4) = 0
// we should expect to see FIN now, sk moves to FIN_WAIT_1 0.210 > F. 1:1(0) ack 1 win 29200

Quoted 7 lines

// receive data, but sk already closed -> Reset
+.010 < P. 1:1001(1000) ack 1 win 46
+0 > R 1:1(0) win 0

// Are we properly sending a RST like prior packet did ? +.010 < P. 1001:2001(1000) ack 1 win 46
+0 > R 1:1(0) win 0

Thanks,
Hannes

#8 lore raw
Quoted 11 lines · Hannes Frederic Sowa
Quoted 6 lines · Eric Dumazet
Quoted 2 lines · Hannes Frederic Sowa

Still, the RST packet can be dropped along the way. So the teardown of the socket on the other side might not happen.

This is why it is better to send RST for every incoming in-excess packet

I agree it would be better to send a RST than ACK incoming data. Just thinking out loud, would it make sense to add a sub-state to timewait so we RST all other packets for TIMEWAIT_LEN duration?

Intuitively this timewait creation looks dangerous to me, maybe from an attack surface perspective.

An out of sequence packet sends an advisory RST packet, which is already a hint for building some kinds of attacks.

Now if we also create a timewait socket, and its associated timer, we also add quite a risk of memory and cpu consumption.

So I am fine you create a timewait, only if you can prove there is no change in security.

Thanks !

maintainer
#9 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Wed, 18 Nov 2015 07:46:31 -0800

Quoted 9 lines

Anyway, having a TIMEWAIT setup after sending a RST makes little sense to me.

When a RST packet is sent, the remote peer will forget everything about this previous connection, and another connect() might reuse the tuple and I do not think we should forbid this. Normal PAWS checks were invented for a good reason.

RFC 1122, 4.2.2.13 can be interpreted in very different ways.

I think it is clear that once RST is emitted, that connection ID no longer exists, on both ends.

Sender of RST must not have that matching state, and receiver of RST must tear things down.