Skip to contentSkip to conversation

New: Network connections hang going through an OpenBSD firewall]

Fwd: Nivedita Singhvi 5 participants 8 messages Sep 14, 2004

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Can you reproduce on the latest kernel, please?
Is the OpenBSD mangling the packet in any way?

Can anyone tell me if this smells like something
recently fixed (MTU issues)? Doesn't sound like
the windowscaling problem but could be related.

thanks,
Nivedita

-------- Original Message --------
Subject: [Bug 3397] New: Network connections hang going through an OpenBSD firewall
Date: Tue, 14 Sep 2004 09:38:24 -0700
From: [email protected]
To: [email protected]

http://bugme.osdl.org/show_bug.cgi?id=3397

Summary: Network connections hang going through an OpenBSD
firewall
Kernel Version: 2.6.6+
Status: NEW
Severity: blocking
Owner: [email protected]
Submitter: [email protected]

Distribution: Fedora Core 2, Gentoo
Hardware Environment: All
Software Environment: All
Problem Description:
We have seen a number of issues with people accessing our website
http://www.cs.unm.edu/ using kernels 2.6.6+. Doing some network sniffing we can
see that data request is received by the web server and the web server responds
however after certain amount of bytes it simply stops.

I have played with MTU sizes and if Ethereal is to be believed the transfer stops after MTU + 77 bytes.

This has been reported to us by a number of different people running different
distributions ie. Fedora Core 2, Gentoo. For example 2.6.5 kernel that comes with FC2 works. Secure IMAP and Secure POP don't seem to work either using 2.6.6+.

What is even stranger is that SSH connections don't exhibit this kind of a problem ie. you can SSH withouth a hitch.

A data point is that we are using transparent (in-line) OpenBSD firewall with
Packetfilter.

Steps to reproduce:

Boot into 2.6.6+ kernel try pulling up http://www.cs.unm.edu/ through a web browser. It won't show up. Boot into a 2.6.5 and below and the page will show up.

Any clues will be appreciated.

------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.

#1 lore raw
Quoted 6 lines · On Tue, 14 Sep 2004 14:00:26 -0700 Nivedita Singhvi <[email protected]> wrote:

Can you reproduce on the latest kernel, please?
Is the OpenBSD mangling the packet in any way?

Can anyone tell me if this smells like something
recently fixed (MTU issues)? Doesn't sound like
the windowscaling problem but could be related.

OpenBSD pf is easy to configure to break window scaling. The developer claims its not a bug. Basically stateless TCP connection tracking will never work, it's a bad idea.

Quoted 10 lines · Daniel Hartmeier

The problem arises when the user creates a complicated ruleset that passes the first SYN of a connection without creating a state entry. You might argue that this shouldn't be possible, but some forms of stateless filtering are being used. The man page warns against this, too.

In this particular case, the ruleset tells pf to pass the initial SYN without creating state (and therefore without any place to note the window option and first scale factor). When, later, the SYN+ACK creates state, the state just doesn't contain the information to follow the scaled windows.

#2 lore raw
In reply to Nivedita Singhvi
Quoted 1 line · Nivedita Singhvi

Can you reproduce on the latest kernel, please?

I will try it with the latest kernel. I have tried this with kernel 2.6.8-1.521 from

http://www.atrpms.net/dist/

It doesn't work with it. Should I download 2.6.8-1 from kernel.org and compile it by hand ?

Quoted 1 line

Is the OpenBSD mangling the packet in any way?

It is certainly possible but everything works fine with all other OSes and kernels 2.6.5 and below which leads me to believe something was changed in 2.6.6 that broke it.

Are you able to reproduce it ?

Thanks,

Vladimir Vuksan

Quoted 7 lines

Can anyone tell me if this smells like something
recently fixed (MTU issues)? Doesn't sound like
the windowscaling problem but could be related.

http://bugme.osdl.org/show_bug.cgi?id=3397

maintainer
#3 lore raw
Quoted 7 lines · On Tue, 14 Sep 2004 15:09:50 -0600 Vladimir <[email protected]> wrote:
Quoted 1 line

Is the OpenBSD mangling the packet in any way?

It is certainly possible but everything works fine with all other OSes and kernels 2.6.5 and below which leads me to believe something was changed in 2.6.6 that broke it.

Are you able to reproduce it ?

OpenBSD packet filter is busted, and the maintainer of it claims this is not a bug.

That changes in 2.6.6 didn't "break" things, it enabled a feature in TCP that OpenBSD stateless TCP connection tracking cannot handle, and old TCP feature in fact, window scaling.

See here for more info:

http://lwn.net/Articles/92727/

#5 lore raw
In reply to Vladimir
Quoted 1 line · Vladimir

It doesn't work with it. Should I download 2.6.8-1 from kernel.org and compile it by hand ?

If you can, download the latest (linux-2.6.9-rc2). You can get the patch against linux-2.6.8.1 vanilla from kernel.org.

Quoted 3 lines

It is certainly possible but everything works fine with all other OSes and kernels 2.6.5 and below which leads me to believe something was changed in 2.6.6 that broke it.

Are you able to reproduce it ?

Unfortunately, I have zero cycles right now (and in the near future) to address any issues, I'm afraid. Someone on netdev will likely be able to help you or have run into it already. See Stephen's post from a few minutes ago..

thanks,
Nivedita

#4 lore raw
In reply to David S. Miller
Quoted 11 lines · David S. Miller

OpenBSD packet filter is busted, and the maintainer of it claims this is not a bug.

That changes in 2.6.6 didn't "break" things, it enabled a feature in TCP that OpenBSD stateless TCP connection tracking cannot handle, and old TCP feature in fact, window scaling.

See here for more info:

http://lwn.net/Articles/92727/

Thanks. We were able to fix our firewall so things work properly now. The problem is that this is "insidious" since it is not immediately apparent what the problem is especially since it tends to work with all other OSes except Linux with 2.6.6+.

I will note this on the bug I submitted and close it.

Thanks a lot,

Vladimir Vuksan

#6 lore raw
In reply to Nivedita Singhvi
Quoted 3 lines · Nivedita Singhvi

If you can, download the latest (linux-2.6.9-rc2). You can get the patch against linux-2.6.8.1 vanilla from kernel.org.

Patch against 2.6.8. It won't apply clean on 2.6.8.1.

Signature · 2 lines
Tomasz Torcz                 Morality must always be based on practicality.
[email protected]                -- Baron Vladimir Harkonnen
#7 lore raw
In reply to Nivedita Singhvi
Quoted 13 lines · On Tue, 14 Sep 2004 14:21:41 -0700 Nivedita Singhvi <[email protected]> wrote:
Quoted 2 lines · Vladimir

It doesn't work with it. Should I download 2.6.8-1 from kernel.org and compile it by hand ?

If you can, download the latest (linux-2.6.9-rc2). You can get the patch against linux-2.6.8.1 vanilla from kernel.org.

Quoted 4 lines

It is certainly possible but everything works fine with all other OSes and kernels 2.6.5 and below which leads me to believe something was changed in 2.6.6 that broke it.

Quoted 6 lines
Quoted 1 line

Are you able to reproduce it ?

Unfortunately, I have zero cycles right now (and in the near future) to address any issues, I'm afraid. Someone on netdev will likely be able to help you or have run into it already. See Stephen's post from a few minutes ago..

Since the latest kernel chooses window scaling factor automatically, the value chosen by default is small enough that OpenBSD
busted pf will just make performance suck. If you want test it, you have to increase tcp_rmem[2] or tcp_rmem_max to force a window scale > 2.