tg3 in 2.6.12-rc6 and Cisco PIX SMTP fixup
Hi
I am testing kernel 2.6.12-rc6 on a 6 IBM 335 servers. The NICs are gigabit broadcom. If I use the tg3 driver then each of the servers are unable to communicate with a Cisco PIX using SMTP fixup, the connection simply get cut:
-------------
telnet xx.x.xx.xx 25
Trying xx.x.xxx.xx...
Connected to xx.x.xxx.xx.
Escape character is '^]'.
mail to: <[email protected]>
Connection closed by foreign host.
-------------
Using tcpdump does not give me any clue as to what goes wrong, the connection is simply lost so I am suspecting some kind of TX/RX mess up. If I instead use the tg3 driver in kernel 2.6.8.1 (or the official broadcom bcm5700 driver (version 8.1.55) with kernel 2.6.12-rc6) then I get:
-------------
telnet xx.x.xxx.xx 25
Trying xx.x.xxx.xx...
Connected to xx.x.xxx.xx.
Escape character is '^]'.
220 ***************
mail to: <[email protected]>
250 ok
quit
221 test.com
Connection closed by foreign host.
-------------
So are there any differences in the tg3 driver between 2.6.8.1 and 2.6.12-rc6 that would cause this kind of behaviour ?.
I know that SMTP fixup is mostly a poorly implemented Sendmail security fix left over from the pre ESMTP era that cripples SMTP connectivity without offering any real
security advantages. So the best thing would be to turn it off, but given that I do not control the firewall and the admin refuses to change it because he believes it to be a security risk then I am looking for another solution (still hoping that it is not shifting NICs in all my servers).
Regards.
Lars Roland
Quoted 2 lines · Lars Roland
So are there any differences in the tg3 driver between 2.6.8.1 and 2.6.12-rc6 that would cause this kind of behaviour ?.
i'd say: "certainly", but best you find out by diff'ing the versions and/or eventually put 2.6.8.1's tg3 driver in a 2.6.12-rc6 tree, compile, hope it builds, then try again to connect.
Quoted 1 line
I know that SMTP fixup is mostly a poorly implemented Sendmail
i don't know what a "smtp fixup" would be, but does the disconnect happen to other applications too?
if it really turns out to be a tg3 problem, maybe [email protected] should be Cc'ed.
Signature · 3 lines
BOFH excuse #67: descramble code needed from software company
From: Christian Kujau <[email protected]>
Date: Thu, 16 Jun 2005 23:57:31 +0200
Quoted 2 lines
if it really turns out to be a tg3 problem, maybe [email protected] should be Cc'ed.
Make that [email protected]
Quoted 7 lines · David S. Miller
From: Christian Kujau <[email protected]>
Date: Thu, 16 Jun 2005 23:57:31 +0200Quoted 2 lines
if it really turns out to be a tg3 problem, maybe [email protected] should be Cc'ed.
Make that [email protected]
Is that a permanent change of address? And is the old address dead? If so, then the patch beneath should probably be applied.
Update the netdev mailing list address in MAINTAINERS.
---
MAINTAINERS | 48 ++++++++++++++++++++++++------------------------ 1 files changed, 24 insertions(+), 24 deletions(-)
MAINTAINERS +24 -24
@@ -73,7 +73,7 @@ 3C359 NETWORK DRIVER P: Mike Phillips M: [email protected] -L: [email protected] +L: [email protected] L: [email protected] W: http://www.linuxtr.net S: Maintained
@@ -81,13 +81,13 @@ 3C505 NETWORK DRIVER P: Philip Blundell M: [email protected] -L: [email protected] +L: [email protected] S: Maintained 3CR990 NETWORK DRIVER P: David Dillow M: [email protected] -L: [email protected] +L: [email protected] S: Maintained 3W-XXXX ATA-RAID CONTROLLER DRIVER
@@ -130,7 +130,7 @@ 8169 10/100/1000 GIGABIT ETHERNET DRIVER P: Francois Romieu M: [email protected] -L: [email protected] +L: [email protected] S: Maintained 8250/16?50 (AND CLONE UARTS) SERIAL DRIVER
@@ -143,7 +143,7 @@ 8390 NETWORK DRIVERS [WD80x3/SMC-ELITE, SMC-ULTRA, NE2000, 3C503, etc.] P: Paul Gortmaker M: [email protected] -L: [email protected] +L: [email protected] S: Maintained A2232 SERIAL BOARD DRIVER
@@ -337,7 +337,7 @@ ARPD SUPPORT P: Jonathan Layes -L: [email protected] +L: [email protected] S: Maintained ASUS ACPI EXTRAS DRIVER
@@ -710,7 +710,7 @@ DIGI RIGHTSWITCH NETWORK DRIVER P: Rick Richardson -L: [email protected] +L: [email protected] W: http://www.digi.com S: Orphaned
@@ -821,7 +821,7 @@ ETHEREXPRESS-16 NETWORK DRIVER P: Philip Blundell M: [email protected] -L: [email protected] +L: [email protected] S: Maintained ETHERNET BRIDGE
@@ -884,7 +884,7 @@ FRAME RELAY DLCI/FRAD (Sangoma drivers too) P: Mike McLagan M: [email protected] -L: [email protected] +L: [email protected] S: Maintained FREEVXFS FILESYSTEM
@@ -1238,7 +1238,7 @@ IPX NETWORK LAYER P: Arnaldo Carvalho de Melo M: [email protected] -L: [email protected] +L: [email protected] S: Maintained IRDA SUBSYSTEM
@@ -1521,7 +1521,7 @@ P: Manish Lachwani M: [email protected] L: [email protected] -L: [email protected] +L: [email protected] S: Supported MATROX FRAMEBUFFER DRIVER
@@ -1631,13 +1631,13 @@ M: [email protected] P: Jeff Garzik M: [email protected] -L: [email protected] +L: [email protected] S: Maintained NETWORKING [GENERAL] P: Networking Team -M: [email protected] -L: [email protected] +M: [email protected] +L: [email protected] S: Maintained NETWORKING [IPv4/IPv6]
@@ -1653,7 +1653,7 @@ M: [email protected] P: Patrick McHardy M: [email protected] -L: [email protected] +L: [email protected] S: Maintained IPVS
@@ -1673,7 +1673,7 @@ P: Jan-Pascal van Best and Andreas Mohr M: Jan-Pascal van Best <[email protected]> M: Andreas Mohr <[email protected]> -L: [email protected] +L: [email protected] S: Maintained NINJA SCSI-3 / NINJA SCSI-32Bi (16bit/CardBus) PCMCIA SCSI HOST ADAPTER DRIVER
@@ -1715,7 +1715,7 @@ M: [email protected] P: Mike Phillips M: [email protected] -L: [email protected] +L: [email protected] L: [email protected] W: http://www.linuxtr.net S: Maintained
@@ -1822,7 +1822,7 @@ PCNET32 NETWORK DRIVER P: Thomas Bogendörfer M: [email protected] -L: [email protected] +L: [email protected] S: Maintained PHRAM MTD DRIVER
@@ -1834,7 +1834,7 @@ POSIX CLOCKS and TIMERS P: George Anzinger M: [email protected] -L: [email protected] +L: [email protected] S: Supported PERFORMANCE-MONITORING COUNTERS DRIVER
@@ -1875,7 +1875,7 @@ PRISM54 WIRELESS DRIVER P: Prism54 Development Team M: [email protected] -L: [email protected] +L: [email protected] W: http://prism54.org S: Maintained
@@ -2092,7 +2092,7 @@ P: Daniele Venzano M: [email protected] W: http://www.brownhat.org/sis900.html -L: [email protected] +L: [email protected] S: Maintained SIS FRAMEBUFFER DRIVER
@@ -2149,7 +2149,7 @@ SONIC NETWORK DRIVER P: Thomas Bogendoerfer M: [email protected] -L: [email protected] +L: [email protected] S: Maintained SONY VAIO CONTROL DEVICE DRIVER
@@ -2211,7 +2211,7 @@ SPX NETWORK LAYER P: Jay Schulist M: [email protected] -L: [email protected] +L: [email protected] S: Supported SRM (Alpha) environment access
@@ -2290,7 +2290,7 @@ TOKEN-RING NETWORK DRIVER P: Mike Phillips M: [email protected] -L: [email protected] +L: [email protected] L: [email protected] W: http://www.linuxtr.net S: Maintained
From: Jesper Juhl <[email protected]>
Date: Fri, 17 Jun 2005 00:41:28 +0200 (CEST)
Quoted 2 lines
Is that a permanent change of address? And is the old address dead? If so, then the patch beneath should probably be applied.
This has been fixed in Linus's tree already for a few days now.
Quoted 7 lines · Christian Kujau
Quoted 2 lines · Lars Roland
So are there any differences in the tg3 driver between 2.6.8.1 and 2.6.12-rc6 that would cause this kind of behaviour ?.
i'd say: "certainly", but best you find out by diff'ing the versions and/or eventually put 2.6.8.1's tg3 driver in a 2.6.12-rc6 tree, compile, hope it builds, then try again to connect.
It does not seams to be limited to braodcom cards. 3com and Intel e100 cards does the exact same stunt on kernels never than 2.6.8.1. Intel e1000 and realtek 8139 cards do however work.
Quoted 5 lines
Quoted 2 lines
I know that SMTP fixup is mostly a poorly implemented Sendmail
i don't know what a "smtp fixup" would be, but does the disconnect happen to other applications too?
SMTP fixup is a dirty hack where the firewall limits the amount of SMTP commands that can be used in the session to only the core rfc 821 commands. Most people do however use it just to hide the actual receiving mail server from the sender - my guess is that the Cisco PIX changes the frames/packages and then the connection gets dropped. I got the admin to turn it off temporally and then the tg3 works just fine.
Regards.
Lars Roland
Quoted 3 lines · Lars Roland
It does not seams to be limited to braodcom cards. 3com and Intel e100 cards does the exact same stunt on kernels never than 2.6.8.1. Intel e1000 and realtek 8139 cards do however work.
hm - tricky, i think. because no kernel oopses, nothing to look at in the syslog (yes?), various nic drivers affected, others not...in cases like these only Documentation/BUG-HUNTING comes to my mind: if 2.6.8.1 works, and 2.6.12-rc6 does not, we'll need to find out the kernelversion which introduced this behaviour.
sorry,
Christian.
Signature · 3 lines
BOFH excuse #318: Your EMAIL is now being delivered by the USPS.
Quoted 7 lines · Christian Kujau
Quoted 3 lines · Lars Roland
It does not seams to be limited to braodcom cards. 3com and Intel e100 cards does the exact same stunt on kernels never than 2.6.8.1. Intel e1000 and realtek 8139 cards do however work.
hm - tricky, i think. because no kernel oopses, nothing to look at in the syslog (yes?),
Nothing anywhere, even tcpdump just seams to get cut off - I have not been debugging ethernet drivers for years, getting a little rusty at that, so nothing there yet.
Quoted 4 lines
various nic drivers affected, others not...in cases like these only
Documentation/BUG-HUNTINGcomes to my mind: if 2.6.8.1 works, and 2.6.12-rc6 does not, we'll need to find out the kernelversion which introduced this behaviour.
That I can give you, kernel 2.6.8.1 works but 2.6.9 does not (at least not with tg3 and tulip cards).
Regards.
Lars Roland
Quoted 29 lines · Lars Roland
Quoted 12 lines · Christian Kujau
Quoted 5 lines · Lars Roland
It does not seams to be limited to braodcom cards. 3com and Intel e100 cards does the exact same stunt on kernels never than 2.6.8.1. Intel e1000 and realtek 8139 cards do however work.
hm - tricky, i think. because no kernel oopses, nothing to look at in the syslog (yes?),
Nothing anywhere, even tcpdump just seams to get cut off - I have not been debugging ethernet drivers for years, getting a little rusty at that, so nothing there yet.
Quoted 6 lines
various nic drivers affected, others not...in cases like these only
Documentation/BUG-HUNTINGcomes to my mind: if 2.6.8.1 works, and 2.6.12-rc6 does not, we'll need to find out the kernelversion which introduced this behaviour.
Quoted 10 lines
That I can give you, kernel 2.6.8.1 works but 2.6.9 does not (at least not with tg3 and tulip cards).
Regards.
Lars Roland
one question,
Can I know what is the problem? I have 2 tg3 adapters, lots of
e100's and some Cisco PIX and devices.
I can try to reproduce it and see if anyone has something to say about it.
Let me know,
.Alejandro
Quoted 6 lines · Alejandro Bonilla
one question,
Can I know what is the problem?
:I have 2 tg3 adapters, lots e100's and some Cisco PIX and devices.I can try to reproduce it and see if anyone has something to say about it.
Yes please. As I see it. Enable smtp fixup protocol on your cisco pix (you will need to have a smtp server to point it to), then on some linux system running with a kernel greater than 2.6.8.1 do a telnet to the smtp server that is firewalled and try to issue a smtp command.
Note that cisco has a bug report on smtp fixup banner hiding issues in cisco os 6.3.4 but it should not result in the connection getting dropped, it also does not explain why this problem does not seam to exists on kernels prior to 2.6.9.
Regards.
Lars Roland
Quoted 23 lines · Lars Roland
Quoted 8 lines · Alejandro Bonilla
one question,
Can I know what is the problem?
:I have 2 tg3 adapters, lots e100's and some Cisco PIX and devices.I can try to reproduce it and see if anyone has something to say about it.
Yes please. As I see it. Enable smtp fixup protocol on your cisco pix (you will need to have a smtp server to point it to), then on some linux system running with a kernel greater than 2.6.8.1 do a telnet to the smtp server that is firewalled and try to issue a smtp command.
Note that cisco has a bug report on smtp fixup banner hiding issues in cisco os 6.3.4 but it should not result in the connection getting dropped, it also does not explain why this problem does not seam to exists on kernels prior to 2.6.9.
Quoted 5 lines
Regards.
Lars Roland
Lars,
I might be able to try this tomorrow. Just need to setup the PIX.
If you have that bug ID, let me know. ;-)
.Alejandro
Quoted 15 lines · Lars Roland
Quoted 6 lines · Alejandro Bonilla
one question,
Can I know what is the problem?
:I have 2 tg3 adapters, lots e100's and some Cisco PIX and devices.I can try to reproduce it and see if anyone has something to say about it.
Yes please. As I see it. Enable smtp fixup protocol on your cisco pix (you will need to have a smtp server to point it to), then on some linux system running with a kernel greater than 2.6.8.1 do a telnet to the smtp server that is firewalled and try to issue a smtp command.
Quoted 4 lines
Note that cisco has a bug report on smtp fixup banner hiding issues in cisco os 6.3.4 but it should not result in the connection getting dropped, it also does not explain why this problem does not seam to exists on kernels prior to 2.6.9.
2.6.9? This rings a bell.. ;)
Does disabling TCP window scaling fix it?
echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
A number of firewalls just stomp on the scaling bits - the end result is that the perceived window size is too small to make any progress and the connection wedges up.
Quoted 21 lines · On 6/17/05, [email protected] <[email protected]> wrote:
Quoted 15 lines · Lars Roland
Quoted 6 lines · Alejandro Bonilla
one question,
Can I know what is the problem?
:I have 2 tg3 adapters, lots e100's and some Cisco PIX and devices.I can try to reproduce it and see if anyone has something to say about it.
Yes please. As I see it. Enable smtp fixup protocol on your cisco pix (you will need to have a smtp server to point it to), then on some linux system running with a kernel greater than 2.6.8.1 do a telnet to the smtp server that is firewalled and try to issue a smtp command.
Quoted 4 lines
Note that cisco has a bug report on smtp fixup banner hiding issues in cisco os 6.3.4 but it should not result in the connection getting dropped, it also does not explain why this problem does not seam to exists on kernels prior to 2.6.9.
Quoted 5 lines
2.6.9? This rings a bell.. ;)
Does disabling TCP window scaling fix it?
echo 0 >
/proc/sys/net/ipv4/tcp_window_scaling
Yes it does solve it.
Thanks so much - this will be much easier than getting the largest ISP in Denmark to update there Cisco to a new version.
Regards.
Lars Roland
Quoted 35 lines
Quoted 7 lines
Quoted 7 lines · Alejandro Bonilla
Quoted 4 lines
one question,
Can I know what is the problem?
:I have 2 tg3 adapters, lots e100's and some Cisco PIXand devices.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I can try to reproduce it and see if anyone has
something to say about it.
Quoted 2 lines
Quoted 2 lines
Yes please. As I see it. Enable smtp fixup protocol on
your cisco pix
Quoted 2 lines
Quoted 2 lines
(you will need to have a smtp server to point it to), then on some linux system running with a kernel greater than 2.6.8.1
do a telnet to
Quoted 1 line
Quoted 1 line
the smtp server that is firewalled and try to issue a
smtp command.
Quoted 2 lines
Quoted 2 lines
Note that cisco has a bug report on smtp fixup banner
hiding issues in
Quoted 2 lines
Quoted 2 lines
cisco os 6.3.4 but it should not result in the connection getting dropped, it also does not explain why this problem does
not seam to
Quoted 7 lines
Quoted 1 line
exists on kernels prior to 2.6.9.
2.6.9? This rings a bell.. ;)
Does disabling TCP window scaling fix it?
echo 0 >
/proc/sys/net/ipv4/tcp_window_scalingYes it does solve it.
Quoted 7 lines
Thanks so much - this will be much easier than getting the largest ISP in Denmark to update there Cisco to a new version.
Regards.
Lars Roland
Lars, Valdis,
So what do we really have here? Problem with Cisco or a problem in the
driver? Both?
.Alejandro
Quoted 2 lines · Alejandro Bonilla
So what do we really have here? Problem with Cisco or a problem in the
driver? Both?
My bet is that this is a Cisco bug. The only fix for this that I have found on cisco is turning smtp fixup off, even upgrading to the latest cisco does not fix it completely.
Regards.
Lars Roland
Quoted 12 lines · Alejandro Bonilla
Quoted 1 line
So what do we really have here? Problem with Cisco
or a problem in the
Quoted 1 line
driver? Both?
My bet is that this is a Cisco bug. The only fix for this that I have found on cisco is turning smtp fixup off, even upgrading to the latest cisco does not fix it completely.
Quoted 4 lines
Regards.
Lars Roland
Lars,
Issue is supoused to be fixed with 6.3.4.115 or later.
Please let me know if that is not the case with you, or if you would like to get the Image.
.Alejandro
Quoted 2 lines · Alejandro Bonilla
So what do we really have here? Problem with Cisco or a problem in the
driver? Both?
The Cisco PIX is gratuitously clearing the TCP window scaling bits. So if you have tcp_adv_win_scale set to (for example) 6, you'll send a window advertisement of (say) 4096, represented as 64 and a "shift left 6 bits". The PIX whacks the "6 bits" part, and the other end thinks the window is 64 bytes and wedges when a response is over 64 bytes long.
There was quite a discussion of this on lkml back last July.
Quoted 9 lines · [email protected] wrote:
Quoted 4 lines · Alejandro Bonilla
So what do we really have here? Problem with Cisco or a problem in the
driver? Both?
Quoted 8 lines
The Cisco PIX is gratuitously clearing the TCP window scaling bits. So if you have
tcp_adv_win_scaleset to (for example) 6, you'll send a window advertisement of (say) 4096, represented as 64 and a "shift left 6 bits". The PIX whacks the "6 bits" part, and the other end thinks the window is 64 bytes and wedges when a response is over 64 bytes long.
there _was_ a bug in the Cisco PIX whereby it cleared TCP window-scaling bits. this can be tracked through cisco bug-id CSCdy29514.
this was fixed back in August 2002 with the fix incorporated into PIX software releases 6.1.5 and 6.2.3 and later. any 'recent' (i.e. last 2.5 years) releases don't have this problem. (or, at least, we don't think so..).
cheers,
lincoln.
Quoted 8 lines · Lincoln Dale
there _was_ a bug in the Cisco PIX whereby it cleared TCP window-scaling bits.
this can be tracked through cisco bug-id CSCdy29514.this was fixed back in August 2002 with the fix incorporated into PIX software releases 6.1.5 and 6.2.3 and later.
any 'recent' (i.e. last 2.5 years) releases don't have this problem. (or, at least, we don't think so..).
I have identified two firewalls with this problem and both of then are running PIX software version 6.3.4 - I have not yet managed to persuade there respective admins to update to 7.0.1 (or 6.3.4.115) - so until then I am just turning window-scaling off.
Regards.
Lars Roland
Quoted 14 lines · Lars Roland
Quoted 10 lines · Lincoln Dale
there _was_ a bug in the Cisco PIX whereby it cleared TCP window-scaling bits.
this can be tracked through cisco bug-id CSCdy29514.this was fixed back in August 2002 with the fix incorporated into PIX software releases 6.1.5 and 6.2.3 and later.
any 'recent' (i.e. last 2.5 years) releases don't have this problem. (or, at least, we don't think so..).
Quoted 6 lines
I have identified two firewalls with this problem and both of then are running PIX software version 6.3.4 - I have not yet managed to persuade there respective admins to update to 7.0.1 (or 6.3.4.115) - so until then I am just turning window-scaling off.
If you have a Cisco contract, you need to make sure that your account manager will allow the upgrades.
But as I said before, this was fixed long time ago.
I hope you can upgrade soon.
.Alejandro
Quoted 7 lines
Regards.
Lars Roland
Quoted 4 lines · Alejandro Bonilla
If you have a Cisco contract, you need to make sure that your account manager will allow the upgrades.
But as I said before, this was fixed long time ago.
It was not fixed completly in PIX 6.1.5, at this point I now have 3 pixes that all have this problem and all of them are running 6.3.4 I have asked them to upgrade to 6.3.4.115 and will report back as soon I have any results.
Quoted 2 lines · Alejandro Bonilla
So what do we really have here? Problem with Cisco or a problem in the
driver? Both?
Oh - the TCP scaling bits are sent in a TCP Option header - which is what the PIX is gratuitously throwing out (presumably because they're "optional", given the sorts of dain bramage we've seen from PIX boxen before. For the longest time, their 'SMTP Fixup' was anything but....)
Quoted 13 lines · Lars Roland
Quoted 7 lines · Christian Kujau
Quoted 3 lines · Lars Roland
It does not seams to be limited to braodcom cards. 3com and Intel e100 cards does the exact same stunt on kernels never than 2.6.8.1. Intel e1000 and realtek 8139 cards do however work.
hm - tricky, i think. because no kernel oopses, nothing to look at in the syslog (yes?),
Nothing anywhere, even tcpdump just seams to get cut off - I have not been debugging ethernet drivers for years, getting a little rusty at that, so nothing there yet.
Quoted 7 lines
Quoted 4 lines
various nic drivers affected, others not...in cases like these only
Documentation/BUG-HUNTINGcomes to my mind: if 2.6.8.1 works, and 2.6.12-rc6 does not, we'll need to find out the kernelversion which introduced this behaviour.That I can give you, kernel 2.6.8.1 works but 2.6.9 does not (at least not with tg3 and tulip cards).
Maybe some checksumming code has changed, and some of the packets which are checksummed by the hardware get wrong on the wire ?
Willy
Quoted 2 lines · Willy Tarreau
Maybe some checksumming code has changed, and some of the packets which are checksummed by the hardware get wrong on the wire ?
Yes my exact thought, it is fine by me if it is a cisco problem that needs to be fixed in the firewall but it would be nice knowing what exactly changed from 2.6.8.1 -> 2.6.9 so it stopped working.
Regards.
Lars Roland