Skip to contentSkip to conversation

New: Regression and incompatibility with Windows SP2-SP3-Vista TCP stack causing lost connections

Bugme-new Andrew Morton 3 participants 4 messages Aug 12, 2010 1 message not in the archive

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

Andrew Morton

author
#1 lore raw
In reply to a message not in the archive

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 141 lines · On Thu, 12 Aug 2010 08:20:01 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=16568

Summary: Regression and incompatibility with Windows
SP2-SP3-Vista TCP stack causing lost connections
Product: Networking
Version: 2.5
Kernel Version: 2.6.30+
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: IPV4
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No

Hi.
I administer about 50 highly-loaded web servers (free CMS hosting) under linux. Having on most of them kernel versions between 2.6.24 and 2.6.29 at the beginnig of the year, I made TCP sysctls tunings for increasing DDOS and different flooding protection (our servers have attacks rather often). tcp_tw_recyle=1 was among of them, as many manuals in the net recommend to do this and linux documentation does not say anything bad. Having periodic kernel panics connected with bugs in ethernet card drivers and ext3 and after founding that 2.6.31+ kernels work faster with ext3, I upgraded almost all kernels to 2.6.32.8, which was already being tested on several servers for several months. Somewhen after that we began to receive complaints from our users (site owners) that they (and their visitors) see very unstable work of their sites. It looked like HTTP-connections were just lost in a random way. Not everybody had the problem, just a small percent. We tried to find problem with internet providers or buggy firewalls, but finally came to conclusion that problem is connected with our servers. Analizing situations with lost connections using tcpdump i found that client host send packets, BUT LINUX JUST IGNORES THEM, there was SYN-packet repeated 3 times with interval of 3 secs, but NO SYN-ACK reply. Most problems had users with Windows SP3 (i.e. almost all users with SP3 had the problem). I booted one server with old 2.6.24 kernel and found that problem dissappeared. Then began look for exact kernel version, that introduced incompatibility. Using binary search I compiled several kernels between 2.6.24 and 2.6.32.8 and found that 2.6.29.6 DO NO have the problem, but 2.6.30 DOES. Studing commits made to tcp_input.c and tcp_ipv4.c (which i supposed were involved) between that releases I found this one.
author Eric Dumazet <[email protected]>

    Wed, 11 Mar 2009 16:23:57 +0000 (09:23 -0700)
  committer    David S. Miller <[email protected]>    
    Wed, 11 Mar 2009 16:23:57 +0000 (09:23 -0700)

commit fc1ad92dfc4e363a055053746552cdb445ba5c57

tcp: allow timestamps even if SYN packet has tsval=0

Some systems send SYN packets with apparently wrong RFC1323 timestamp option values [timestamp tsval=0 tsecr=0].
It might be for security reasons (http://www.secuobs.com/plugs/25220.shtml ) Linux TCP stack ignores this option and sends back a SYN+ACK packet without timestamp option, thus many TCP flows cannot use timestamps and lose some benefit of RFC1323.
Other operating systems seem to not care about initial tsval value, and let tcp flows to negotiate timestamp option.

net/ipv4/tcp_ipv4.c diff :

net/ipv4/tcp_ipv4.c +0 -9
@@ -1226,15 +1226,6 @@ int tcp_v4_conn_request(struct sock *sk, struct sk_buff
 *skb)
        if (want_cookie && !tmp_opt.saw_tstamp)
                tcp_clear_options(&tmp_opt);
 
-       if (tmp_opt.saw_tstamp && !tmp_opt.rcv_tsval) {
-               /* Some OSes (unknown ones, but I see them on web server, which
-                * contains information interesting only for windows'
-                * users) do not send their stamp in SYN. It is easy case.
-                * We simply do not advertise TS support.
-                */
-               tmp_opt.saw_tstamp = 0;
-               tmp_opt.tstamp_ok  = 0;
-       }
        tmp_opt.tstamp_ok = tmp_opt.saw_tstamp;
 

        tcp_openreq_init(req, &tmp_opt, skb);

Removing that was not very good. Having analized lost connections from SP3 I know that they have timestamps turned on and timestamp value is 0. Here is it:
13:39:10.430498 IP 192.168.99.130.3493 > 192.168.99.100.80: S 2507911465:2507911465(0) win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 0 0,nop,nop,sackOK>

        0x0000:  4500 0040 2bda 4000 8006 86a6 c0a8 6382  E..@[email protected].
        0x0010:  c0a8 6364 0da5 0050 957b b129 0000 0000  ..cd...P.{.)....
        0x0020:  b002 ffff 992c 0000 0204 05b4 0103 0303  .....,..........
        0x0030:  0101 080a 0000 0000 0000 0000 0101 0402  ................

Having above code fragment removed we got tmp_opt.tstamp_ok=1, as i understand. But a little later in source code of tcp_ipv4.c read:

        /* VJ's idea. We save last timestamp seen
         * from the destination in peer table, when entering
         * state TIME-WAIT, and check against it before
         * accepting new connection request.
         *
         * If "isn" is not zero, this request hit alive
         * timewait bucket, so that all the necessary checks
         * are made in the function processing timewait state.
         */
        if (tmp_opt.saw_tstamp &&
            tcp_death_row.sysctl_tw_recycle &&
            (dst = inet_csk_route_req(sk, req)) != NULL &&
            (peer = rt_get_peer((struct rtable *)dst)) != NULL &&
            peer->v4daddr == saddr) {
            if ((u32)get_seconds() - peer->tcp_ts_stamp < TCP_PAWS_MSL &&
                (s32)(peer->tcp_ts - req->ts_recent) >
                            TCP_PAWS_WINDOW) {
                NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_PAWSPASSIVEREJECTED);
                goto drop_and_release;
            }
        }

which in some way (tmp_opt.saw_tstamp && tcp_death_row.sysctl_tw_recycle are true), random way, having not closed time-wait sockets from the pear, leads to packet ignorence.

As for me, i understand, that i should not enable tw_recycle, BUT DOCUMENTATION DOES NOT STATE, that enabling it i'll got random and rather often lost of connections from some types of popular clients (like Windows). Concerning above stated commit, it should include something to prevent above condition to become true if tmp_opt.rcv_tsval==0. I'm not sure, but something like

        if (tmp_opt.saw_tstamp &&
+           tmp_opt.rcv_tsval &&
            tcp_death_row.sysctl_tw_recycle &&
            (dst = inet_csk_route_req(sk, req)) != NULL &&
            (peer = rt_get_peer((struct rtable *)dst)) != NULL &&

just to not provide regression and strong TCP-stack incompatibility in case tw_recycle is enabled.
Also documentation does not state, that tw_recyle should not be used at all for internet servers, because web-clients, which are behind NAT, will have problems connected with the same above condition because successive connections from different clients (which have common IP) could have incompatible timestamps.

Sorry if i detracted somebody busy from his work with my unimportant problem.

#2 lore raw
Quoted 145 lines · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 139 lines · On Thu, 12 Aug 2010 08:20:01 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=16568

Summary: Regression and incompatibility with Windows
SP2-SP3-Vista TCP stack causing lost connections
Product: Networking
Version: 2.5
Kernel Version: 2.6.30+
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: IPV4
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No

Hi.
I administer about 50 highly-loaded web servers (free CMS hosting) under linux. Having on most of them kernel versions between 2.6.24 and 2.6.29 at the beginnig of the year, I made TCP sysctls tunings for increasing DDOS and different flooding protection (our servers have attacks rather often). tcp_tw_recyle=1 was among of them, as many manuals in the net recommend to do this and linux documentation does not say anything bad. Having periodic kernel panics connected with bugs in ethernet card drivers and ext3 and after founding that 2.6.31+ kernels work faster with ext3, I upgraded almost all kernels to 2.6.32.8, which was already being tested on several servers for several months. Somewhen after that we began to receive complaints from our users (site owners) that they (and their visitors) see very unstable work of their sites. It looked like HTTP-connections were just lost in a random way. Not everybody had the problem, just a small percent. We tried to find problem with internet providers or buggy firewalls, but finally came to conclusion that problem is connected with our servers. Analizing situations with lost connections using tcpdump i found that client host send packets, BUT LINUX JUST IGNORES THEM, there was SYN-packet repeated 3 times with interval of 3 secs, but NO SYN-ACK reply. Most problems had users with Windows SP3 (i.e. almost all users with SP3 had the problem). I booted one server with old 2.6.24 kernel and found that problem dissappeared. Then began look for exact kernel version, that introduced incompatibility. Using binary search I compiled several kernels between 2.6.24 and 2.6.32.8 and found that 2.6.29.6 DO NO have the problem, but 2.6.30 DOES. Studing commits made to tcp_input.c and tcp_ipv4.c (which i supposed were involved) between that releases I found this one.
author Eric Dumazet <[email protected]>

    Wed, 11 Mar 2009 16:23:57 +0000 (09:23 -0700)
  committer    David S. Miller <[email protected]>    
    Wed, 11 Mar 2009 16:23:57 +0000 (09:23 -0700)

commit fc1ad92dfc4e363a055053746552cdb445ba5c57

tcp: allow timestamps even if SYN packet has tsval=0

Some systems send SYN packets with apparently wrong RFC1323 timestamp option values [timestamp tsval=0 tsecr=0].
It might be for security reasons (http://www.secuobs.com/plugs/25220.shtml ) Linux TCP stack ignores this option and sends back a SYN+ACK packet without timestamp option, thus many TCP flows cannot use timestamps and lose some benefit of RFC1323.
Other operating systems seem to not care about initial tsval value, and let tcp flows to negotiate timestamp option.

net/ipv4/tcp_ipv4.c diff :

net/ipv4/tcp_ipv4.c +0 -9
@@ -1226,15 +1226,6 @@ int tcp_v4_conn_request(struct sock *sk, struct sk_buff
 *skb)
        if (want_cookie && !tmp_opt.saw_tstamp)
                tcp_clear_options(&tmp_opt);
 
-       if (tmp_opt.saw_tstamp && !tmp_opt.rcv_tsval) {
-               /* Some OSes (unknown ones, but I see them on web server, which
-                * contains information interesting only for windows'
-                * users) do not send their stamp in SYN. It is easy case.
-                * We simply do not advertise TS support.
-                */
-               tmp_opt.saw_tstamp = 0;
-               tmp_opt.tstamp_ok  = 0;
-       }
        tmp_opt.tstamp_ok = tmp_opt.saw_tstamp;
 

        tcp_openreq_init(req, &tmp_opt, skb);

Removing that was not very good. Having analized lost connections from SP3 I know that they have timestamps turned on and timestamp value is 0. Here is it:
13:39:10.430498 IP 192.168.99.130.3493 > 192.168.99.100.80: S 2507911465:2507911465(0) win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 0 0,nop,nop,sackOK>

        0x0000:  4500 0040 2bda 4000 8006 86a6 c0a8 6382  E..@[email protected].
        0x0010:  c0a8 6364 0da5 0050 957b b129 0000 0000  ..cd...P.{.)....
        0x0020:  b002 ffff 992c 0000 0204 05b4 0103 0303  .....,..........
        0x0030:  0101 080a 0000 0000 0000 0000 0101 0402  ................

Having above code fragment removed we got tmp_opt.tstamp_ok=1, as i understand. But a little later in source code of tcp_ipv4.c read:

        /* VJ's idea. We save last timestamp seen
         * from the destination in peer table, when entering
         * state TIME-WAIT, and check against it before
         * accepting new connection request.
         *
         * If "isn" is not zero, this request hit alive
         * timewait bucket, so that all the necessary checks
         * are made in the function processing timewait state.
         */
        if (tmp_opt.saw_tstamp &&
            tcp_death_row.sysctl_tw_recycle &&
            (dst = inet_csk_route_req(sk, req)) != NULL &&
            (peer = rt_get_peer((struct rtable *)dst)) != NULL &&
            peer->v4daddr == saddr) {
            if ((u32)get_seconds() - peer->tcp_ts_stamp < TCP_PAWS_MSL &&
                (s32)(peer->tcp_ts - req->ts_recent) >
                            TCP_PAWS_WINDOW) {
                NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_PAWSPASSIVEREJECTED);
                goto drop_and_release;
            }
        }

which in some way (tmp_opt.saw_tstamp && tcp_death_row.sysctl_tw_recycle are true), random way, having not closed time-wait sockets from the pear, leads to packet ignorence.

As for me, i understand, that i should not enable tw_recycle, BUT DOCUMENTATION DOES NOT STATE, that enabling it i'll got random and rather often lost of connections from some types of popular clients (like Windows). Concerning above stated commit, it should include something to prevent above condition to become true if tmp_opt.rcv_tsval==0. I'm not sure, but something like

        if (tmp_opt.saw_tstamp &&
+           tmp_opt.rcv_tsval &&
            tcp_death_row.sysctl_tw_recycle &&
            (dst = inet_csk_route_req(sk, req)) != NULL &&
            (peer = rt_get_peer((struct rtable *)dst)) != NULL &&

just to not provide regression and strong TCP-stack incompatibility in case tw_recycle is enabled.
Also documentation does not state, that tw_recyle should not be used at all for internet servers, because web-clients, which are behind NAT, will have problems connected with the same above condition because successive connections from different clients (which have common IP) could have incompatible timestamps.

Quoted 4 lines
Quoted 2 lines

Sorry if i detracted somebody busy from his work with my unimportant problem.

--

Hi Yuriy

Interesting analysis but wrong conclusions :)

Clients using RFC1323 (timestamps) and behind a NAT device will barf on your setup. No matter they use Windows SP3 or other operating system.

Only because RFC1323 is more often enabled at client level (a registry change on Windows XP, Vista or Seven I dont know), you start noticing your server drops more connections than before.

Point is :

Dont mess with tcp_tw_recycle=1, tcp_timestamps=1 on public machines

Its a non working setup, for clients behind NAT devices (since their TSVAL will probably lead to incorrect behavior on server, with infamous LINUX_MIB_PAWSPASSIVEREJECTED status seen on netstat -s, as you discovered.

And your patch solves nothing for this very common case, unless the NAT device is able to overwrite TSVAL values with its own values (very unlikely !!!)

A working setup is (and is the default) :

tcp_tw_recycle=0
tcp_timestamps=1

Documentation might be improved, but I feel whole "tcp_tw_recycle" affair is really too tricky to be ever documented (not mentioning using it ;) )

#4 lore raw
In reply to Yuriy
Quoted 4 lines · Yuriy

Thanks for reply.

Main idea that i wanted to say is just to document this feature appropriately as internet is full of recommendations to enable it. Just few words like "do not used it on public servers" would be much better than now.

Sure, but we dont maintain nor correct the recommandations found on various Internet pages ;)

BTW, a google search on "tcp_tw_recycle" gives many results on problems with this setting, not improvements.

Also, many "recommandations" found on Internet suggest to disable tcp_timestamps, only because it adds 12 bytes to TCP header.

Apparently you chose to follow the tcp_tw_recycle=1 recommandation, not the tcp_timestamps=0 one ;)

#3 lore raw
In reply to Eric Dumazet

Hi, Eric.

You wrote 12.08.2010, 18:09:33:

Quoted 145 lines · Andrew Morton
Quoted 145 lines

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 139 lines · On Thu, 12 Aug 2010 08:20:01 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=16568

Summary: Regression and incompatibility with Windows
SP2-SP3-Vista TCP stack causing lost connections
Product: Networking
Version: 2.5
Kernel Version: 2.6.30+
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: IPV4
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: No

Hi.
I administer about 50 highly-loaded web servers (free CMS hosting) under linux. Having on most of them kernel versions between 2.6.24 and 2.6.29 at the beginnig of the year, I made TCP sysctls tunings for increasing DDOS and different flooding protection (our servers have attacks rather often). tcp_tw_recyle=1 was among of them, as many manuals in the net recommend to do this and linux documentation does not say anything bad. Having periodic kernel panics connected with bugs in ethernet card drivers and ext3 and after founding that 2.6.31+ kernels work faster with ext3, I upgraded almost all kernels to 2.6.32.8, which was already being tested on several servers for several months. Somewhen after that we began to receive complaints from our users (site owners) that they (and their visitors) see very unstable work of their sites. It looked like HTTP-connections were just lost in a random way. Not everybody had the problem, just a small percent. We tried to find problem with internet providers or buggy firewalls, but finally came to conclusion that problem is connected with our servers. Analizing situations with lost connections using tcpdump i found that client host send packets, BUT LINUX JUST IGNORES THEM, there was SYN-packet repeated 3 times with interval of 3 secs, but NO SYN-ACK reply. Most problems had users with Windows SP3 (i.e. almost all users with SP3 had the problem). I booted one server with old 2.6.24 kernel and found that problem dissappeared. Then began look for exact kernel version, that introduced incompatibility. Using binary search I compiled several kernels between 2.6.24 and 2.6.32.8 and found that 2.6.29.6 DO NO have the problem, but 2.6.30 DOES. Studing commits made to tcp_input.c and tcp_ipv4.c (which i supposed were involved) between that releases I found this one.
author Eric Dumazet <[email protected]>

    Wed, 11 Mar 2009 16:23:57 +0000 (09:23 -0700)
  committer    David S. Miller <[email protected]>    
    Wed, 11 Mar 2009 16:23:57 +0000 (09:23 -0700)

commit fc1ad92dfc4e363a055053746552cdb445ba5c57

tcp: allow timestamps even if SYN packet has tsval=0

Some systems send SYN packets with apparently wrong RFC1323 timestamp option values [timestamp tsval=0 tsecr=0].
It might be for security reasons (http://www.secuobs.com/plugs/25220.shtml ) Linux TCP stack ignores this option and sends back a SYN+ACK packet without timestamp option, thus many TCP flows cannot use timestamps and lose some benefit of RFC1323.
Other operating systems seem to not care about initial tsval value, and let tcp flows to negotiate timestamp option.

net/ipv4/tcp_ipv4.c diff :

net/ipv4/tcp_ipv4.c +0 -9
@@ -1226,15 +1226,6 @@ int tcp_v4_conn_request(struct sock *sk, struct sk_buff
 *skb)
        if (want_cookie && !tmp_opt.saw_tstamp)
                tcp_clear_options(&tmp_opt);
 
-       if (tmp_opt.saw_tstamp && !tmp_opt.rcv_tsval) {
-               /* Some OSes (unknown ones, but I see them on web server, which
-                * contains information interesting only for windows'
-                * users) do not send their stamp in SYN. It is easy case.
-                * We simply do not advertise TS support.
-                */
-               tmp_opt.saw_tstamp = 0;
-               tmp_opt.tstamp_ok  = 0;
-       }
        tmp_opt.tstamp_ok = tmp_opt.saw_tstamp;
 

        tcp_openreq_init(req, &tmp_opt, skb);

Removing that was not very good. Having analized lost connections from SP3 I know that they have timestamps turned on and timestamp value is 0. Here is it:
13:39:10.430498 IP 192.168.99.130.3493 > 192.168.99.100.80: S 2507911465:2507911465(0) win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 0 0,nop,nop,sackOK>

        0x0000:  4500 0040 2bda 4000 8006 86a6 c0a8 6382  E..@[email protected].
        0x0010:  c0a8 6364 0da5 0050 957b b129 0000 0000  ..cd...P.{.)....
        0x0020:  b002 ffff 992c 0000 0204 05b4 0103 0303  .....,..........
        0x0030:  0101 080a 0000 0000 0000 0000 0101 0402  ................

Having above code fragment removed we got tmp_opt.tstamp_ok=1, as i understand. But a little later in source code of tcp_ipv4.c read:

        /* VJ's idea. We save last timestamp seen
         * from the destination in peer table, when entering
         * state TIME-WAIT, and check against it before
         * accepting new connection request.
         *
         * If "isn" is not zero, this request hit alive
         * timewait bucket, so that all the necessary checks
         * are made in the function processing timewait state.
         */
        if (tmp_opt.saw_tstamp &&
            tcp_death_row.sysctl_tw_recycle &&
            (dst = inet_csk_route_req(sk, req)) != NULL &&
            (peer = rt_get_peer((struct rtable *)dst)) != NULL &&
            peer->v4daddr == saddr) {
            if ((u32)get_seconds() - peer->tcp_ts_stamp < TCP_PAWS_MSL &&
                (s32)(peer->tcp_ts - req->ts_recent) >
                            TCP_PAWS_WINDOW) {
                NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_PAWSPASSIVEREJECTED);
                goto drop_and_release;
            }
        }

which in some way (tmp_opt.saw_tstamp && tcp_death_row.sysctl_tw_recycle are true), random way, having not closed time-wait sockets from the pear, leads to packet ignorence.

As for me, i understand, that i should not enable tw_recycle, BUT DOCUMENTATION DOES NOT STATE, that enabling it i'll got random and rather often lost of connections from some types of popular clients (like Windows). Concerning above stated commit, it should include something to prevent above condition to become true if tmp_opt.rcv_tsval==0. I'm not sure, but something like

        if (tmp_opt.saw_tstamp &&
+           tmp_opt.rcv_tsval &&
            tcp_death_row.sysctl_tw_recycle &&
            (dst = inet_csk_route_req(sk, req)) != NULL &&
            (peer = rt_get_peer((struct rtable *)dst)) != NULL &&

just to not provide regression and strong TCP-stack incompatibility in case tw_recycle is enabled.
Also documentation does not state, that tw_recyle should not be used at all for internet servers, because web-clients, which are behind NAT, will have problems connected with the same above condition because successive connections from different clients (which have common IP) could have incompatible timestamps.

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Sorry if i detracted somebody busy from his work with my unimportant problem.

--

ED> Hi Yuriy

ED> Interesting analysis but wrong conclusions :)

ED> Clients using RFC1323 (timestamps) and behind a NAT device will barf on ED> your setup. No matter they use Windows SP3 or other operating system.

ED> Only because RFC1323 is more often enabled at client level (a registry ED> change on Windows XP, Vista or Seven I dont know), you start noticing ED> your server drops more connections than before.

ED> Point is :

ED> Dont mess with tcp_tw_recycle=1, tcp_timestamps=1 on public machines

ED> Its a non working setup, for clients behind NAT devices (since their ED> TSVAL will probably lead to incorrect behavior on server, with infamous ED> LINUX_MIB_PAWSPASSIVEREJECTED status seen on netstat -s, as you ED> discovered.

ED> And your patch solves nothing for this very common case, unless the NAT ED> device is able to overwrite TSVAL values with its own values (very ED> unlikely !!!)

ED> A working setup is (and is the default) :

ED> tcp_tw_recycle=0
ED> tcp_timestamps=1

ED> Documentation might be improved, but I feel whole "tcp_tw_recycle" ED> affair is really too tricky to be ever documented (not mentioning using ED> it ;) )

Thanks for reply.

Main idea that i wanted to say is just to document this feature appropriately as internet is full of recommendations to enable it. Just few words like "do not used it on public servers" would be much better than now.

Signature · 2 lines
Regards,
 Yuriy                          mailto:[email protected]