Skip to contentSkip to conversation

connect() stalls after 4.9 -> 4.10 upgrade

DISCUSSION Lutz Vieweg 2 participants 3 messages Feb 27, 2017 – Mar 16, 2017

lore
Share & tools
lore
#0 lore raw

Hi all,

the following regression a colleage and me experienced on two different machines after upgrading from linux-4.9 to linux-4.10 is so obvious and so easy to reproduce that I am surprised we could not find any reports of it on the Internet:

After upgrading a machine running the latest CentOS from using mainline kernel linux-4.9 to linux-4.10, attempts to connect() via IPv4 to localhost (and seemingly also other hosts) fail in about half of the cases, leaving the process trying to connect() stalled until timeout.

Reproduction:

Quoted 2 lines

ncat -k -l 19999 &
C=1 ; while true ; do echo -n "$C " ; echo ping | ncat localhost 19999 ; C=expr $C + 1 ; sleep 1 ; done

Using linux-4.10, the output looks like this:

Quoted 21 lines

1 ping
2 Ncat: Connection timed out.
3 ping
4 Ncat: Connection timed out.
5 ping
6 ping
7 ping
8 Ncat: Connection timed out.
9 ping
10 ping
11 Ncat: Connection timed out.
12 ping
13 Ncat: Connection timed out.
14 ping
15 Ncat: Connection timed out.
16 ping
17 Ncat: Connection timed out.
18 ping
19 ping
20 Ncat: Connection timed out.
21 ping

Quoted 8 lines

22 ping
23 Ncat: Connection timed out.
24 ping
25 Ncat: Connection timed out.
26 Ncat: Connection timed out.
27 ping
28 Ncat: Connection timed out.
29 ping

Using linux-4.9, the output looks like this:

Quoted 11 lines

1 ping
2 ping
3 ping
4 ping
5 ping
6 ping
7 ping
8 ping
9 ping
10 ping
11 ping

Quoted 8 lines

12 ping
13 ping
14 ping
15 ping
16 ping
17 ping
18 ping
19 ping

The two machines we observed this symptom on were running different distributions (CentOS 7.3 and Ubuntu) and were of completely different hardware.

Didn't anybody else experience the same symptom?

(I also created https://bugzilla.kernel.org/show_bug.cgi?id=194723 on this topic.)

Regards,

Lutz Vieweg

#1 lore raw

change "tcp: randomize tcp timestamp offsets for each connection" broke networking

Dear Linux Developers,

change set "tcp: randomize tcp timestamp offsets for each connection"
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=95a22caee396cef0bb2ca8fafdd82966a49367bb
broke networking for several users starting from linux-4.10 - connects (even to localhost) would just time out in many cases.

See
https://bugzilla.kernel.org/show_bug.cgi?id=194723
for a detailed description and the "git bisect" log.

Thanks for considering any fix in advance, best regards,

Lutz Vieweg

#2 lore raw

Re: change "tcp: randomize tcp timestamp offsets for each connection" broke networking

Quoted 11 lines · Lutz Vieweg

Dear Linux Developers,

change set "tcp: randomize tcp timestamp offsets for each connection"
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/co
mmit/?id=95a22caee396cef0bb2ca8fafdd82966a49367bb
broke networking for several users starting from linux-4.10 - connects (even to localhost) would just time out in many cases.

See
https://bugzilla.kernel.org/show_bug.cgi?id=194723
for a detailed description and the "git bisect" log.

The netdev thread therein indicates that the fix is in the works...

-Mike