Skip to contentSkip to conversation

Out of window filter catches too much

DISCUSSION Pierre Ossman 2 participants 7 messages Feb 26, 2005 – Mar 3, 2005 1 message not in the archive

lore
Share & tools
lore

Pierre Ossman

author
#0 lore raw

I'm having problem with the out of window filter throwing a way packets in otherwise perfectly good connections. The problem appears when doing a rather large rsync between two linux machines on the network here.

The rsync server is running 2.6.9, the client 2.6.10 and the router 2.6.10.

Since there is only linux machines involved here this must be a kernel bug. Either in the TCP layer or in netfilters detection. Here is a dump from the router when it starts throwing away packets:

ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=10234 DF PROTO=TCP SPT=3851 DPT=873 SEQ=2763580423 ACK=299956256 WINDOW=95 RES=0x00 ACK URGP=0 OPT (0101080AC4C2FDE77E1D58C1)
ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=10236 DF PROTO=TCP SPT=3851 DPT=873 SEQ=2763581871 ACK=299956256 WINDOW=95 RES=0x00 ACK URGP=0 OPT (0101080AC4C2FDE77E1D58C1)
ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=10238 DF PROTO=TCP SPT=3851 DPT=873 SEQ=2763583319 ACK=299956256 WINDOW=95 RES=0x00 ACK URGP=0 OPT (0101080AC4C2FDE77E1D58C1)
ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=10240 DF PROTO=TCP SPT=3851 DPT=873 SEQ=2763584767 ACK=299956256 WINDOW=95 RES=0x00 ACK URGP=0 OPT (0101080AC4C2FDE77E1D58C1)
ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=10242 DF PROTO=TCP SPT=3851 DPT=873 SEQ=2763586215 ACK=299956256 WINDOW=95 RES=0x00 ACK URGP=0 OPT (0101080AC4C2FDE77E1D58C1)
ip_ct_tcp: ACK is over the upper bound (ACKed data has never seen yet) IN= OUT= SRC=10.8.5.10 DST=10.8.0.24 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=23961 DF PROTO=TCP SPT=873 DPT=3851 SEQ=299956256 ACK=2763580423 WINDOW=0 RES=0x00 ACK URGP=0 OPT (0101080A7E1D58E9C4C2FDE7)
ip_ct_tcp: ACK is over the upper bound (ACKed data has never seen yet) IN= OUT= SRC=10.8.5.10 DST=10.8.0.24 LEN=432 TOS=0x00 PREC=0x00 TTL=64 ID=23963 DF PROTO=TCP SPT=873 DPT=3851 SEQ=299956256 ACK=2763580423 WINDOW=0 RES=0x00 ACK PSH URGP=0 OPT (0101080A7E1D5927C4C2FDE7)
ip_ct_tcp: ACK is over the upper bound (ACKed data has never seen yet) IN= OUT= SRC=10.8.5.10 DST=10.8.0.24 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=23965 DF PROTO=TCP SPT=873 DPT=3851 SEQ=299956636 ACK=2763580423 WINDOW=1718 RES=0x00 ACK URGP=0 OPT (0101080A7E1D5952C4C2FDE7)
ip_ct_tcp: ACK is over the upper bound (ACKed data has never seen yet) IN= OUT= SRC=10.8.5.10 DST=10.8.0.24 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=23967 DF PROTO=TCP SPT=873 DPT=3851 SEQ=299956636 ACK=2763580423 WINDOW=3788 RES=0x00 ACK URGP=0 OPT (0101080A7E1D599DC4C2FDE7)
ip_ct_tcp: ACK is over the upper bound (ACKed data has never seen yet) IN= OUT= SRC=10.8.5.10 DST=10.8.0.24 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=23969 DF PROTO=TCP SPT=873 DPT=3851 SEQ=299956636 ACK=2763580423 WINDOW=3788 RES=0x00 ACK URGP=0 OPT (0101080A7E1D59B1C4C2FED70101050AA4B8DE5FA4B8E407)
printk: 7 messages suppressed.
ip_ct_tcp: ACK is over the upper bound (ACKed data has never seen yet) IN= OUT= SRC=10.8.5.10 DST=10.8.0.24 LEN=432 TOS=0x00 PREC=0x00 TTL=64 ID=23985 DF PROTO=TCP SPT=873 DPT=3851 SEQ=299956256 ACK=2763580423 WINDOW=5792 RES=0x00 ACK PSH URGP=0 OPT (0101080A7E1D73CBC4C30BF7)
printk: 1 messages suppressed.
ip_ct_tcp: ACK is over the upper bound (ACKed data has never seen yet) IN= OUT= SRC=10.8.5.10 DST=10.8.0.24 LEN=432 TOS=0x00 PREC=0x00 TTL=64 ID=23989 DF PROTO=TCP SPT=873 DPT=3851 SEQ=299956256 ACK=2763580423 WINDOW=5792 RES=0x00 ACK PSH URGP=0 OPT (0101080A7E1D8F4BC4C31AF7)
ip_ct_tcp: ACK is over the upper bound (ACKed data has never seen yet) IN= OUT= SRC=10.8.5.10 DST=10.8.0.24 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=23991 DF PROTO=TCP SPT=873 DPT=3851 SEQ=299956636 ACK=2763580423 WINDOW=5792 RES=0x00 ACK URGP=0 OPT (0101080A7E1D93D0C4C338F70101050AA4B8DE5FA4B8E407)
ip_ct_tcp: ACK is over the upper bound (ACKed data has never seen yet) IN= OUT= SRC=10.8.5.10 DST=10.8.0.24 LEN=432 TOS=0x00 PREC=0x00 TTL=64 ID=23993 DF PROTO=TCP SPT=873 DPT=3851 SEQ=299956256 ACK=2763580423 WINDOW=5792 RES=0x00 ACK PSH URGP=0 OPT (0101080A7E1DC64BC4C338F7)
ip_ct_tcp: ACK is over the upper bound (ACKed data has never seen yet) IN= OUT= SRC=10.8.5.10 DST=10.8.0.24 LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=23995 DF PROTO=TCP SPT=873 DPT=3851 SEQ=299956636 ACK=2763580423 WINDOW=5792 RES=0x00 ACK URGP=0 OPT (0101080A7E1DCFCFC4C374F70101050AA4B8DE5FA4B8E407)
ip_ct_tcp: ACK is over the upper bound (ACKed data has never seen yet) IN= OUT= SRC=10.8.5.10 DST=10.8.0.24 LEN=432 TOS=0x00 PREC=0x00 TTL=64 ID=23997 DF PROTO=TCP SPT=873 DPT=3851 SEQ=299956256 ACK=2763580423 WINDOW=5792 RES=0x00 ACK PSH URGP=0 OPT (0101080A7E1E344BC4C374F7)
ip_ct_tcp: invalid RST (ignored) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=10258 DF PROTO=TCP SPT=3851 DPT=873 SEQ=2763587663 ACK=299956256 WINDOW=724 RES=0x00 ACK RST URGP=0 OPT (0101080AC4C3E8BE7E1D58C1)

The connection recovered from the first couple of these, but the later ones causes the connection to die.

This is very annoying so I hope someone has the time to help me fix this
as soon as possible.

Rgds
Pierre

#1 lore raw

Hi,

Quoted 9 lines · Pierre Ossman

Since there is only linux machines involved here this must be a kernel bug. Either in the TCP layer or in netfilters detection. Here is a dump from the router when it starts throwing away packets:

ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=10234 DF PROTO=TCP SPT=3851 DPT=873 SEQ=2763580423 ACK=299956256 WINDOW=95 RES=0x00 ACK URGP=0 OPT (0101080AC4C2FDE77E1D58C1)
ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver)

On Mon, 21 Feb 2005 I posted a patch to netfilter-devel which addresses this and other issues in TCP window tracking. Please try the patch.

Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary
author
#2 lore raw
Quoted 18 lines · Jozsef Kadlecsik

Hi,

Quoted 11 lines · Pierre Ossman

Since there is only linux machines involved here this must be a kernel bug. Either in the TCP layer or in netfilters detection. Here is a dump from the router when it starts throwing away packets:

ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=10234 DF PROTO=TCP SPT=3851 DPT=873 SEQ=2763580423 ACK=299956256 WINDOW=95 RES=0x00 ACK URGP=0 OPT (0101080AC4C2FDE77E1D58C1)
ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver)

Quoted 4 lines

On Mon, 21 Feb 2005 I posted a patch to netfilter-devel which addresses this and other issues in TCP window tracking. Please try the patch.

I assume you meant:
https://lists.netfilter.org/pipermail/netfilter-devel/2005-February/018598.html

I've tried the patch and it seems to keep it from dropping the ACKs which is enough to keep the connection going. I still get some errors the other way though:

Mar  2 01:36:22 prometheus kernel: ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=52959 DF PROTO=TCP SPT=1053 DPT=873 SEQ=3991302411 ACK=1391445765 WINDOW=115 RES=0x00 ACK URGP=0 OPT (0101080AD974090C92CE1415)
Mar  2 01:36:24 prometheus kernel: ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=53577 DF PROTO=TCP SPT=1053 DPT=873 SEQ=3991735363 ACK=1391446225 WINDOW=0 RES=0x00 ACK URGP=0 OPT (0101080AD974111492CE1C1D)
Mar  2 01:37:55 prometheus kernel: ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=5615 DF PROTO=TCP SPT=1053 DPT=873 SEQ=4004321678 ACK=1391476149 WINDOW=74 RES=0x00 ACK URGP=0 OPT (0101080AD97576E992CF81EC)
Mar  2 01:37:55 prometheus kernel: ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=5617 DF PROTO=TCP SPT=1053 DPT=873 SEQ=4004323126 ACK=1391476149 WINDOW=74 RES=0x00 ACK URGP=0 OPT (0101080AD97576E992CF81EC)
Mar  2 01:37:55 prometheus kernel: ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=5619 DF PROTO=TCP SPT=1053 DPT=873 SEQ=4004324574 ACK=1391476149 WINDOW=74 RES=0x00 ACK URGP=0 OPT (0101080AD97576E992CF81EC)
Mar  2 01:37:55 prometheus kernel: ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=5621 DF PROTO=TCP SPT=1053 DPT=873 SEQ=4004326022 ACK=1391476149 WINDOW=74 RES=0x00 ACK URGP=0 OPT (0101080AD97576E992CF81EC)
Mar  2 01:37:55 prometheus kernel: ip_ct_tcp: SEQ is over the upper bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=5623 DF PROTO=TCP SPT=1053 DPT=873 SEQ=4004327470 ACK=1391476149 WINDOW=74 RES=0x00 ACK URGP=0 OPT (0101080AD97576E992CF81EC)

Rgds
Pierre

#3 lore raw
Quoted 10 lines · Pierre Ossman
Quoted 2 lines

On Mon, 21 Feb 2005 I posted a patch to netfilter-devel which addresses this and other issues in TCP window tracking. Please try the patch.

I assume you meant:
https://lists.netfilter.org/pipermail/netfilter-devel/2005-February/018598.html

I've tried the patch and it seems to keep it from dropping the ACKs which is enough to keep the connection going. I still get some errors the other way though:

Quoted 5 lines
Mar  2 01:36:22 prometheus kernel: ip_ct_tcp: SEQ is over the upper

bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=52959 DF PROTO=TCP SPT=1053 DPT=873 SEQ=3991302411 ACK=1391445765 WINDOW=115 RES=0x00 ACK URGP=0 OPT (0101080AD974090C92CE1415)

If it is reproducible then could you capture the traffic with tcpdump and send me the results together with the corresponding log lines? Please dump on both sides of the firewall.

Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary
author
#4 lore raw
Quoted 23 lines · Jozsef Kadlecsik
Quoted 11 lines · Pierre Ossman
Quoted 4 lines

On Mon, 21 Feb 2005 I posted a patch to netfilter-devel which addresses this and other issues in TCP window tracking. Please try the patch.

I assume you meant:
https://lists.netfilter.org/pipermail/netfilter-devel/2005-February/018598.html

I've tried the patch and it seems to keep it from dropping the ACKs which is enough to keep the connection going. I still get some errors the other way though:

Quoted 7 lines
Mar  2 01:36:22 prometheus kernel: ip_ct_tcp: SEQ is over the upper

bound (over the window of the receiver) IN= OUT= SRC=10.8.0.24 DST=10.8.5.10 LEN=1500 TOS=0x00 PREC=0x00 TTL=64 ID=52959 DF PROTO=TCP SPT=1053 DPT=873 SEQ=3991302411 ACK=1391445765 WINDOW=115 RES=0x00 ACK URGP=0 OPT (0101080AD974090C92CE1415)

Quoted 6 lines

If it is reproducible then could you capture the traffic with tcpdump and send me the results together with the corresponding log lines? Please dump on both sides of the firewall.

It's a lot of traffic so that will be difficult. The problems appear after at least 100 MB has been transfered. Is there some way I can reduce this to just the parts that are of relevance to you? If you have a decent connection (or a lot of time ;)) I suppose I could put up the entire thing on the local webserver for you to download at your own leisure.

Rgds
Pierrre

#5 lore raw
Quoted 10 lines · Pierre Ossman
Quoted 4 lines

If it is reproducible then could you capture the traffic with tcpdump and send me the results together with the corresponding log lines? Please dump on both sides of the firewall.

It's a lot of traffic so that will be difficult. The problems appear after at least 100 MB has been transfered. Is there some way I can reduce this to just the parts that are of relevance to you? If you have a decent connection (or a lot of time ;)) I suppose I could put up the entire thing on the local webserver for you to download at your own leisure.

There is no need to capture the data, the protocol headers are enough (so default snaplen of tcpdump is just fine). Write me the url when I can download the files and I fire up the debugging environment.

Best regards,
Jozsef
-
E-mail : [email protected], [email protected] PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics

          H-1525 Budapest 114, POB. 49, Hungary

A message that is not in the archive (sent off-list or not yet received): [email protected]