connect() stalls after 4.9 -> 4.10 upgrade
Hi all,
the following regression a colleage and me experienced on two different machines after upgrading from linux-4.9 to linux-4.10 is so obvious and so easy to reproduce that I am surprised we could not find any reports of it on the Internet:
After upgrading a machine running the latest CentOS from using mainline kernel linux-4.9 to linux-4.10, attempts to connect() via IPv4 to localhost (and seemingly also other hosts) fail in about half of the cases, leaving the process trying to connect() stalled until timeout.
Reproduction:
Quoted 2 lines
ncat -k -l 19999 &
C=1 ; while true ; do echo -n "$C " ; echo ping | ncat localhost 19999 ; C=expr $C + 1; sleep 1 ; done
Using linux-4.10, the output looks like this:
Quoted 21 lines
1 ping
2 Ncat: Connection timed out.
3 ping
4 Ncat: Connection timed out.
5 ping
6 ping
7 ping
8 Ncat: Connection timed out.
9 ping
10 ping
11 Ncat: Connection timed out.
12 ping
13 Ncat: Connection timed out.
14 ping
15 Ncat: Connection timed out.
16 ping
17 Ncat: Connection timed out.
18 ping
19 ping
20 Ncat: Connection timed out.
21 ping
Quoted 8 lines
22 ping
23 Ncat: Connection timed out.
24 ping
25 Ncat: Connection timed out.
26 Ncat: Connection timed out.
27 ping
28 Ncat: Connection timed out.
29 ping
Using linux-4.9, the output looks like this:
Quoted 11 lines
1 ping
2 ping
3 ping
4 ping
5 ping
6 ping
7 ping
8 ping
9 ping
10 ping
11 ping
Quoted 8 lines
12 ping
13 ping
14 ping
15 ping
16 ping
17 ping
18 ping
19 ping
The two machines we observed this symptom on were running different distributions (CentOS 7.3 and Ubuntu) and were of completely different hardware.
Didn't anybody else experience the same symptom?
(I also created https://bugzilla.kernel.org/show_bug.cgi?id=194723 on this topic.)
Regards,
Lutz Vieweg
change "tcp: randomize tcp timestamp offsets for each connection" broke networking
Dear Linux Developers,
change set "tcp: randomize tcp timestamp offsets for each connection"
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=95a22caee396cef0bb2ca8fafdd82966a49367bb
broke networking for several users starting from linux-4.10 - connects (even to localhost) would just time out in many cases.
See
https://bugzilla.kernel.org/show_bug.cgi?id=194723
for a detailed description and the "git bisect" log.
Thanks for considering any fix in advance, best regards,
Lutz Vieweg
Re: change "tcp: randomize tcp timestamp offsets for each connection" broke networking
Quoted 11 lines · Lutz Vieweg
Dear Linux Developers,
change set "tcp: randomize tcp timestamp offsets for each connection"
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/co
mmit/?id=95a22caee396cef0bb2ca8fafdd82966a49367bb
broke networking for several users starting from linux-4.10 - connects (even to localhost) would just time out in many cases.See
https://bugzilla.kernel.org/show_bug.cgi?id=194723
for a detailed description and the "git bisect" log.
The netdev thread therein indicates that the fix is in the works...
-Mike