Linux 2.2.16 through 2.2.18preX TCP hang bug triggered by rsync
I read through the tcpdump, and it seems that Linux completely ignores packets with out-of-window sequence numbers:
* the solaris computers (dynamic...) sends further data although the Linux box (static) says 'win 0'.
See lines 2067, 2069, 2076, ...
2066 16:31:43.108759 eth0 > static.8664 > dynamic.ih.lucent.com.39406:
. 1583720:1583720(0) ack 69041 win 0 (DF)
2067 16:31:43.110623 eth0 < dynamic.ih.lucent.com.39406 > static.8664:
P 69041:69628(587) ack 1583721 win 0 (DF)
2078 16:31:43.896657 eth0 < dynamic.ih.lucent.com.39406 > static.8664:
. 69041:69041(0) ack 1583721 win 8760 (DF)
* this is the last ack with an in-window sequence number.
.
.
.
2136 16:35:08.488774 eth0 > static.8664 > dynamic.ih.lucent.com.39406:
. 1583721:1585181(1460) ack 69041 win 0 (DF)
* the linux computer sends data
2137 16:35:08.492158 eth0 < dynamic.ih.lucent.com.39406 > static.8664:
. 70501:70501(0) ack 1592481 win 8760 (DF)
* but ignores the ack, probably because the sequence number is out of window
Perhaps someone who understand TCP could check the code and compare it with the RFC's?
--
Manfred
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
I checked RFC793, and AFAICS Solaris is the culprit:
it sends out invalid packets, Linux ignores them and thus Linux doesn't receive acks.
Which Solaris version do you use?
* The last valid ack from the Solaris computer is for byte 1583721, win 8760 (line 2078)
* No packet after line 2078 from the Solaris computer passed the acceptability test from RFC793, page 69. Thus Linux ignores these packets completely.
* Linux sends out packets up to 1591021:1592481(1460) without receiving _valid_ acks, then it begins to retry 1583721:1585181(1460) every 2 seconds until the end of the tcpdump.
--
Manfred
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Hello!
Quoted 2 lines
I read through the tcpdump, and it seems that Linux completely ignores packets with out-of-window sequence numbers:
Yes, Linux is __very__ not right doing this. RFC requires to accept ACK, URG and RST on any segment adjacent to window, even if window is zero.
Solaris also does thing, formally wrong, but it would work if linux would be formally correct.
O-ho-ho... It is difficult bug.
Alexey
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 4 lines
Yes, Linux is
__very__not right doing this. RFC requires to accept ACK, URG and RST on any segment adjacent to window, even if window is zero.
Interesting: I checked the RFC 793 and came to the conclusion that Linux is correct. ("special allowance should be made to accept valid ACKs" not must be). Is there another RFC?
Note that either segment length is > 0 or SEG.SEQ != RCV.NXT
But I spotted a second problem, might be minor:
Could you check what happened in line 2066 of this tcpdump?
in line 2064 static (2.2.18-pre17) ack'ed 1583721.
in line 2066 dynamic (Solaris 7.0) sends data although the window is 0 in line 2067 static complains, but now ack is 1583720
2060 16:31:42.879337 eth0 < dynamic.ih.lucent.com.39406 > static.8664:
. 67580:67580(0) ack 1582261 win 1460 (DF)
2061 16:31:42.907940 eth0 > static.8664 > dynamic.ih.lucent.com.39406:
. 1583721:1583721(0) ack 67580 win 1460 (DF)
2062 16:31:42.908620 eth0 < dynamic.ih.lucent.com.39406 > static.8664:
. 67580:67581(1) ack 1583721 win 0 (DF)
2063 16:31:43.098761 eth0 > static.8664 > dynamic.ih.lucent.com.39406:
. 1583721:1583721(0) ack 67581 win 1460 (DF)
2064 16:31:43.100993 eth0 < dynamic.ih.lucent.com.39406 > static.8664:
P 67581:68456(875) ack 1583721 win 0 (DF)
2065 16:31:43.101524 eth0 < dynamic.ih.lucent.com.39406 > static.8664:
P 68456:69041(585) ack 1583721 win 0 (DF)
2066 16:31:43.108759 eth0 > static.8664 > dynamic.ih.lucent.com.39406:
. 1583720:1583720(0) ack 69041 win 0 (DF)
2067 16:31:43.110623 eth0 < dynamic.ih.lucent.com.39406 > static.8664:
P 69041:69628(587) ack 1583721 win 0 (DF)
2068 16:31:43.110679 eth0 > static.8664 > dynamic.ih.lucent.com.39406:
. 1583721:1583721(0) ack 69041 win 0 (DF)
An off-by-one error somewhere in tcp_send_ack()?
A second packet with the off-by-one ack number is send a few packets later, all other packets are correct.
--
Manfred
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Hello!
Quoted 1 line
must be). Is there another RFC?
It is exactly this place.
As soon as BSD uses this feature, it is must for us.
Quoted 1 line
Could you check what happened in line 2066 of this tcpdump?
....
Quoted 2 lines
2066 16:31:43.108759 eth0 > static.8664 > dynamic.ih.lucent.com.39406:
. 1583720:1583720(0) ack 69041 win 0 (DF)
This is usual RFC-style zero-window probe: null segment out of window.
Alexey
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Hello!
I take my words back. Manfred is right, this requirement is not a MUST.
Real problem is much worse, and it is wholly on the shame of solaris. Tcpdump shows at least two different bugs there.
2060 16:31:42.879337 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 675
80:67580(0) ack 1582261 win 1460 (DF)
2061 16:31:42.907940 eth0 > static.8664 > dynamic.ih.lucent.com.39406: . 158
3721:1583721(0) ack 67580 win 1460 (DF)
All is OK until now. Solaris's state should be:
SND.NXT=SND.UNA=67580
SND.WND=1460
RCV.NXT=1582261
2062 16:31:42.908620 eth0 < dynamic.ih.lucent.com.39406 > static.8664: . 675
80:67581(1) ack 1583721 win 0 (DF)
Solaris sends one byte.
SND.NXT++
RCV.NXT=1583721
2063 16:31:43.098761 eth0 > static.8664 > dynamic.ih.lucent.com.39406: . 158
3721:1583721(0) ack 67581 win 1460 (DF)
We ACK it.
2064 16:31:43.100993 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P 675
81:68456(875) ack 1583721 win 0 (DF)
2065 16:31:43.101524 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P 684
56:69041(585) ack 1583721 win 0 (DF)
Solaris sends two segments, filling all the window.
SND.NXT=69041
2066 16:31:43.108759 eth0 > static.8664 > dynamic.ih.lucent.com.39406: . 158
3720:1583720(0) ack 69041 win 0 (DF)
We send zero window probe. SEG.SEQ=1583720.
Solaris accepts ACK from it!!! (bug #1) But does not accept window.
So, now it thinks that SND.UNA=SND.NXT=69041
SND.WND=1460
State is corrupted.
This is hard bug. But it is still not fatal. Actually, such corruptions (but by different reasons) are common with stacks, which borrowed code from BSD. Look into tcp-impl, Subj: "Send window update algorithm ..." They are recoverable, provided stack is sane.
2067 16:31:43.110623 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P 690
41:69628(587) ack 1583721 win 0 (DF)
Solaris send some crap out of window, because of corrupted state. No problems.
2068 16:31:43.110679 eth0 > static.8664 > dynamic.ih.lucent.com.39406: . 158
3721:1583721(0) ack 69041 win 0 (DF)
We tell "No pasaran", of course.
According to rules, Solaris must shrink window now. This is the only way to recover corrupted state.
2069 16:31:43.111641 eth0 < dynamic.ih.lucent.com.39406 > static.8664: P 696
28:70501(873) ack 1583721 win 0 (DF)
It does not. And this is point after which recovery is impossible. Fatal bug#2.
To resume: it is impossible to help to this from Linux side. We may accept ACK&WIN from out-of-window segments, and this will help in this case _occasionally_. But Solaris is still deemed to lockup randomly with such sawdust in the head.
Alexey
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 8 lines · On Wed, Jan 24, 2001 at 11:03:34PM +0300, [email protected] wrote:
Hello!
Quoted 2 lines
I read through the tcpdump, and it seems that Linux completely ignores packets with out-of-window sequence numbers:
Yes, Linux is
__very__not right doing this. RFC requires to accept ACK, URG and RST on any segment adjacent to window, even if window is zero.
It's mostly for security to make it more difficult to nuke connections without knowing the sequence number.
Remember RFC is from a very different internet with much less DoS attacks.
-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 4 lines · Andi Kleen
It's mostly for security to make it more difficult to nuke connections without knowing the sequence number.
Remember RFC is from a very different internet with much less DoS attacks.
Andi, one of the worst DoSs in the world is not being able to communicate with half of the systems out there.
BSD and Solaris both make these kinds of packets, therefore it is must to handle them properly. So we will fix Linux, there is no argument.
Later,
David S. Miller
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines · David S. Miller
Quoted 3 lines · Andi Kleen
It's mostly for security to make it more difficult to nuke connections without knowing the sequence number.
Quoted 7 lines
Quoted 1 line
Remember RFC is from a very different internet with much less DoS attacks.
Andi, one of the worst DoSs in the world is not being able to communicate with half of the systems out there.
BSD and Solaris both make these kinds of packets, therefore it is must to handle them properly. So we will fix Linux, there is no argument.
Hang on... From what was quoted of the RFC, this behaviour (accepting these packets) isn't required of hosts? In which case, if BSD or Solaris depend on it, THEY are violating the protocol, not Linux??
James.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 4 lines · Andi Kleen
It's mostly for security to make it more difficult to nuke connections without knowing the sequence number.
Remember RFC is from a very different internet with much less DoS attacks.
If you're deliberately breaking compatibility by violating the specs, you're making your own DoS if your machines can't chat to each other. If you insist on breaking the RFC, make a sysctl for this behaviour that defaults to "off".
Signature · 5 lines
Matthias Andree - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 11 lines · Matthias Andree
Quoted 4 lines · Andi Kleen
It's mostly for security to make it more difficult to nuke connections without knowing the sequence number.
Remember RFC is from a very different internet with much less DoS attacks.
If you're deliberately breaking compatibility by violating the specs, you're making your own DoS if your machines can't chat to each other. If you insist on breaking the RFC, make a sysctl for this behaviour that defaults to "off".
This isn't a violation - the section quoted does not REQUIRE the behaviour, it only RECOMMENDS it as being a good idea. Since implementing it apparently makes DoS attacks easier, NOT implementing it is now a better idea...
James.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 4 lines · James Sutherland
This isn't a violation - the section quoted does not REQUIRE the behaviour, it only RECOMMENDS it as being a good idea. Since implementing it apparently makes DoS attacks easier, NOT implementing it is now a better idea...
Ok, now for the interoperability. Is there a problem with this when the recommendation is not followed?
Signature · 5 lines
Matthias Andree - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
Quoted 9 lines · David S. Miller
Quoted 4 lines · Andi Kleen
It's mostly for security to make it more difficult to nuke connections without knowing the sequence number.
Remember RFC is from a very different internet with much less DoS attacks.
Andi, one of the worst DoSs in the world is not being able to communicate with half of the systems out there.
If it was that serious then there would be surely more reports ;)
Quoted 3 lines
BSD and Solaris both make these kinds of packets, therefore it is must to handle them properly. So we will fix Linux, there is no argument.
How do you propose to handle them? Queue the data anyways or just process the ACK?
-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines · Andi Kleen
Quoted 2 lines
BSD and Solaris both make these kinds of packets, therefore it is must to handle them properly. So we will fix Linux, there is no argument.
How do you propose to handle them? Queue the data anyways or just process the ACK?
tcp_sequence returns two flag bits instead of it's current binary state. One bit says "accept data", the other says "accept control bits" (such as RST, ACK, etc.)
tcp_sequence also will truncate the data len of the SKB area if necessary, BSD really puts total crap in the probe byte.
Callers of tcp_sequence check the return value bits accordingly. This is all slow path code, so there are no performance issues.
Later,
David S. Miller
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 11 lines · David S. Miller
Quoted 5 lines · Andi Kleen
Quoted 2 lines
BSD and Solaris both make these kinds of packets, therefore it is must to handle them properly. So we will fix Linux, there is no argument.
How do you propose to handle them? Queue the data anyways or just process the ACK?
tcp_sequencereturns two flag bits instead of it's current binary state. One bit says "accept data", the other says "accept control bits" (such as RST, ACK, etc.)
Sounds ugly @) tcp_sequence is already far too complicated.
Quoted 6 lines
tcp_sequencealso will truncate the data len of the SKB area if necessary, BSD really puts total crap in the probe byte.Callers of
tcp_sequencecheck the return value bits accordingly. This is all slow path code, so there are no performance issues.
How about simply queueing the data in that case if it already fits into the receive buffer? The alternative would be to skb_trim() it to 0 in the slow path of tcp_sequence and queue, but that looks wasteful. Basically it would accept the acks with the data in most cases except when the application has totally stopped reading and in that case it doesn't harm to ignore the acks.
I have been played with a different embedded stack and it uses that approach and it seems to work and makes much simpler code.
-Andi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/