New: Machine hangs with EIP at skb_copy_and_csum_dev

Bugme-new In review Andrew Morton 5 participants 39 messages Aug 19, 2010 – Aug 29, 2010 1 message not in the archive

Subsystems (3)

Networking Network drivers IPv4/IPv6

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

Andrew Morton

author

2010-08-19 22:22 UTC #1 lore raw

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 16 lines · On Thu, 19 Aug 2010 09:57:25 GMT [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=16626

           Summary: Machine hangs with EIP at skb_copy_and_csum_dev
           Product: Drivers
           Version: 2.5
    Kernel Version: 2.6.36-rc1-00127-g763008c
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: blocking
          Priority: P1
         Component: PCI
        AssignedTo: [email protected]
        ReportedBy: [email protected]
        Regression: Yes

A post-2.6.35 regression.

Quoted 11 lines

After upgrade from 2.6.33.7 to 2.6.35.2 a server hanged twice, so continued on 2.6.33.7.
Today decided to try lates Linus' tree with no luck.
The first time I started on 2.6.36-rc1-00127-g763008c it ran for a few minutes, then whent dead with this on the screen:
[picture 1]
http://picpaste.com/9cfb03116d41f27568e1bb2a67b7f4dc.jpg

Quoted 8 lines

[picture 2]
Then I power-cycled the machine, only two get this:
http://picpaste.com/6d70f453e462d1aed038781ad4bdb741.jpg
And because [picture 2] seemed too bad on the lower half of the screen, here is
[picture 3]
http://picpaste.com/0a51ae079ace2e4abd9e9d29226069f7.jpg

Might have triggered the BUG_ON() in skb_copy_and_csum_dev(). Might be a tg3 thing. Hard to tell.

It'd be really nice to get that first screenful. Sigh. How long have we had this oops-scrolls-off problem?? Perhaps you could set /proc/sys/kernel/printk_delay to 100 (it's in milliseconds) so that the oops scrolls past nice and slowly?

Plamen Petrov

2010-08-20 05:09 UTC #2 lore raw

(responding via emailed reply-to-all)

На 20.8.2010 г. 01:21, Andrew Morton написа:

Quoted 46 lines

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 16 lines · On Thu, 19 Aug 2010 09:57:25 GMT [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=16626
            Summary: Machine hangs with EIP at skb_copy_and_csum_dev
            Product: Drivers
            Version: 2.5
     Kernel Version: 2.6.36-rc1-00127-g763008c
           Platform: All
         OS/Version: Linux
               Tree: Mainline
             Status: NEW
           Severity: blocking
           Priority: P1
          Component: PCI
         AssignedTo: [email protected]
         ReportedBy: [email protected]
         Regression: Yes
A post-2.6.35 regression.
Quoted 11 lines
After upgrade from 2.6.33.7 to 2.6.35.2 a server hanged twice, so continued on 2.6.33.7.
Today decided to try lates Linus' tree with no luck.
The first time I started on 2.6.36-rc1-00127-g763008c it ran for a few minutes, then whent dead with this on the screen:
[picture 1]
http://picpaste.com/9cfb03116d41f27568e1bb2a67b7f4dc.jpg
Quoted 8 lines
[picture 2]
Then I power-cycled the machine, only two get this:
http://picpaste.com/6d70f453e462d1aed038781ad4bdb741.jpg
And because [picture 2] seemed too bad on the lower half of the screen, here is
[picture 3]
http://picpaste.com/0a51ae079ace2e4abd9e9d29226069f7.jpg

Quoted 8 lines

Might have triggered the BUG_ON() in skb_copy_and_csum_dev(). Might be a tg3 thing. Hard to tell.
It'd be really nice to get that first screenful. Sigh. How long have we had this oops-scrolls-off problem?? Perhaps you could set /proc/sys/kernel/printk_delay to 100 (it's in milliseconds) so that the oops scrolls past nice and slowly?

So you need the begining of the oops screen - I will try to get that with the proposed pirntk_delay setting.
But wich kernel should I use? Linus' latest tree or 2.6.35.2 ? They both fail the same way here, as far as I can say.

Andrew Morton

author

2010-08-20 05:09 UTC #3 lore raw

Quoted 52 lines · On Fri, 20 Aug 2010 08:03:21 +0300 Plamen Petrov <[email protected]>…

(responding via emailed reply-to-all)
____ 20.8.2010 __. 01:21, Andrew Morton ____________:
Quoted 49 lines
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 16 lines · On Thu, 19 Aug 2010 09:57:25 GMT [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=16626
            Summary: Machine hangs with EIP at skb_copy_and_csum_dev
            Product: Drivers
            Version: 2.5
     Kernel Version: 2.6.36-rc1-00127-g763008c
           Platform: All
         OS/Version: Linux
               Tree: Mainline
             Status: NEW
           Severity: blocking
           Priority: P1
          Component: PCI
         AssignedTo: [email protected]
         ReportedBy: [email protected]
         Regression: Yes
A post-2.6.35 regression.
Quoted 11 lines
After upgrade from 2.6.33.7 to 2.6.35.2 a server hanged twice, so continued on 2.6.33.7.
Today decided to try lates Linus' tree with no luck.
The first time I started on 2.6.36-rc1-00127-g763008c it ran for a few minutes, then whent dead with this on the screen:
[picture 1]
http://picpaste.com/9cfb03116d41f27568e1bb2a67b7f4dc.jpg
Quoted 8 lines
[picture 2]
Then I power-cycled the machine, only two get this:
http://picpaste.com/6d70f453e462d1aed038781ad4bdb741.jpg
And because [picture 2] seemed too bad on the lower half of the screen, here is
[picture 3]
http://picpaste.com/0a51ae079ace2e4abd9e9d29226069f7.jpg
Might have triggered the BUG_ON() in skb_copy_and_csum_dev(). Might be a tg3 thing. Hard to tell.

Quoted 7 lines

Quoted 5 lines
It'd be really nice to get that first screenful. Sigh. How long have we had this oops-scrolls-off problem?? Perhaps you could set /proc/sys/kernel/printk_delay to 100 (it's in milliseconds) so that the oops scrolls past nice and slowly?
So you need the begining of the oops screen - I will try to get that with the proposed pirntk_delay setting.

Thanks.

Quoted 2 lines

But wich kernel should I use? Linus' latest tree or 2.6.35.2 ? They both fail the same way here, as far as I can say.

Current mainline would be best, because we'd fix the bug there first then backport the fix into -stable. But it doesn't matter a lot in this case - whatever's most convenient for you, I'd say.

Plamen Petrov

2010-08-20 06:12 UTC #4 lore raw

На 20.8.2010 г. 08:11, Andrew Morton написа:

Quoted 62 lines · On Fri, 20 Aug 2010 08:03:21 +0300 Plamen Petrov<[email protected]>…

Quoted 53 lines
(responding via emailed reply-to-all)
____ 20.8.2010 __. 01:21, Andrew Morton ____________:
Quoted 49 lines
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 16 lines · On Thu, 19 Aug 2010 09:57:25 GMT [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=16626
             Summary: Machine hangs with EIP at skb_copy_and_csum_dev
             Product: Drivers
             Version: 2.5
      Kernel Version: 2.6.36-rc1-00127-g763008c
            Platform: All
          OS/Version: Linux
                Tree: Mainline
              Status: NEW
            Severity: blocking
            Priority: P1
           Component: PCI
          AssignedTo: [email protected]
          ReportedBy: [email protected]
          Regression: Yes
A post-2.6.35 regression.
Quoted 11 lines
After upgrade from 2.6.33.7 to 2.6.35.2 a server hanged twice, so continued on 2.6.33.7.
Today decided to try lates Linus' tree with no luck.
The first time I started on 2.6.36-rc1-00127-g763008c it ran for a few minutes, then whent dead with this on the screen:
[picture 1]
http://picpaste.com/9cfb03116d41f27568e1bb2a67b7f4dc.jpg
Quoted 8 lines
[picture 2]
Then I power-cycled the machine, only two get this:
http://picpaste.com/6d70f453e462d1aed038781ad4bdb741.jpg
And because [picture 2] seemed too bad on the lower half of the screen, here is
[picture 3]
http://picpaste.com/0a51ae079ace2e4abd9e9d29226069f7.jpg
Might have triggered the BUG_ON() in skb_copy_and_csum_dev(). Might be a tg3 thing. Hard to tell.
Quoted 7 lines
Quoted 5 lines
It'd be really nice to get that first screenful. Sigh. How long have we had this oops-scrolls-off problem?? Perhaps you could set /proc/sys/kernel/printk_delay to 100 (it's in milliseconds) so that the oops scrolls past nice and slowly?
So you need the begining of the oops screen - I will try to get that with the proposed pirntk_delay setting.

Quoted 9 lines

Thanks.
Quoted 2 lines
But wich kernel should I use? Linus' latest tree or 2.6.35.2 ? They both fail the same way here, as far as I can say.
Current mainline would be best, because we'd fix the bug there first then backport the fix into -stable. But it doesn't matter a lot in this case - whatever's most convenient for you, I'd say.

With the "echo 100 > /proc/sys/kernel/printk_delay" command run by /etc/rc.d/rc.local, while still on 2.6.36-rc1-00127-g763008c, I got these:

[picture 4]
http://picpaste.com/aa3e373e894179e8ba19587ed63d8104.jpg

[picture 5]
http://picpaste.com/9bc4bdc04f5a84fdaf49d6e1db23ede8.jpg

[picture 6]
http://picpaste.com/da3ccd69a0a1221bb55f48b39c4ad950.jpg

Hope the above help.

And by the way, I think you are correct that this is a post-2.6.35 thing, because 2.6.35.2 was the first to give me this kind of problems, and I can confirm that 2.6.34 does not have it, because the system was on 2.6.34.4 for the last 12 hours without problems, then just a moment ago crashed on 2.6.36-rc1-00127-g763008c, and now back on 2.6.34.4

P.S. Shouldn't "echo 100 > /proc/sys/kernel/printk_delay" be somewhere on the "How to debug a crashing kernel guide" somewhere?

Thanks!

Andrew Morton

author

2010-08-20 06:18 UTC #5 lore raw

Quoted 6 lines · On Fri, 20 Aug 2010 09:12:10 +0300 Plamen Petrov <[email protected]>…

With the "echo 100 > /proc/sys/kernel/printk_delay" command run by /etc/rc.d/rc.local, while still on 2.6.36-rc1-00127-g763008c, I got these:
[picture 4]
http://picpaste.com/aa3e373e894179e8ba19587ed63d8104.jpg

bewdy, thanks.

        BUG_ON(csstart > skb_headlen(skb));

Hopefully that's enough for the net guys to work with.

Quoted 8 lines

[picture 5]
http://picpaste.com/9bc4bdc04f5a84fdaf49d6e1db23ede8.jpg
[picture 6]
http://picpaste.com/da3ccd69a0a1221bb55f48b39c4ad950.jpg
Hope the above help.

Quoted 7 lines

And by the way, I think you are correct that this is a post-2.6.35 thing, because 2.6.35.2 was the first to give me this kind of problems, and I can confirm that 2.6.34 does not have it, because the system was on 2.6.34.4 for the last 12 hours without problems, then just a moment ago crashed on 2.6.36-rc1-00127-g763008c, and now back on 2.6.34.4

OK.

Quoted 3 lines

P.S. Shouldn't "echo 100 > /proc/sys/kernel/printk_delay" be somewhere on the "How to debug a crashing kernel guide" somewhere?

I only just thought of it ;)

I'm thinking perhaps that we should have a lines_after_oops boot parameter which makes the console shut up N lines after someone called oops_enter().

Plamen Petrov

2010-08-20 06:26 UTC #6 lore raw

Posting this only in the hope that it will be helpfull to the "net guys"...

root@fs:~# uname -a; lspci -vvv; ethtool -i eth1

Quoted 306 lines

Linux fs 2.6.34.4-FS #1 SMP Thu Aug 19 18:09:58 UTC 2010 i686 Intel(R) Pentium(R) D CPU 3.00GHz GenuineIntel GNU/Linux 00:00.0 Host bridge: Intel Corporation E7230/3000/3010 Memory Controller Hub
Subsystem: Dell Unknown device 01df
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
Latency: 0
Capabilities: [e0] Vendor Specific Information
00:01.0 PCI bridge: Intel Corporation E7230/3000/3010 PCI Express Root Port (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: efe00000-efefffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
Capabilities: [88] Subsystem: Intel Corporation Unknown device 0000
Capabilities: [80] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [90] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+
Address: fee0300c Data: 4149
Capabilities: [a0] Express Root Port (Slot+) IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 2
Link: Latency L0s <256ns, L1 <4us
Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
Link: Speed 2.5Gb/s, Width x0
Slot: AtnBtn- PwrCtrl- MRL- AtnInd- PwrInd- HotPlug- Surpise-
Slot: Number 1, PowerLimit 25.000000
Slot: Enabled AtnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq-
Slot: AttnInd Off, PwrInd On, Power-
Root: Correctable- Non-Fatal- Fatal- PME-
Capabilities: [100] Virtual Channel
Capabilities: [140] Unknown (5)
00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1 (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 00001000-00001fff
Memory behind bridge: efd00000-efdfffff
Prefetchable memory behind bridge: 0000000020000000-00000000201fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Express Root Port (Slot+) IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s unlimited, L1 unlimited
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 1
Link: Latency L0s <256ns, L1 <4us
Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
Link: Speed 2.5Gb/s, Width x0
Slot: AtnBtn- PwrCtrl- MRL- AtnInd- PwrInd- HotPlug+ Surpise+
Slot: Number 4, PowerLimit 10.000000
Slot: Enabled AtnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq-
Slot: AttnInd Unknown, PwrInd Unknown, Power-
Root: Correctable- Non-Fatal- Fatal- PME-
Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+
Address: fee0300c Data: 4151
Capabilities: [90] Subsystem: Gammagraphx, Inc. Unknown device 0000
Capabilities: [a0] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100] Virtual Channel
Capabilities: [180] Unknown (5)
00:1c.4 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 5 (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
I/O behind bridge: 00002000-00002fff
Memory behind bridge: 20200000-203fffff
Prefetchable memory behind bridge: 0000000020400000-00000000205fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Express Root Port (Slot+) IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s unlimited, L1 unlimited
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 5
Link: Latency L0s <256ns, L1 <4us
Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
Link: Speed 2.5Gb/s, Width x0
Slot: AtnBtn- PwrCtrl- MRL- AtnInd- PwrInd- HotPlug+ Surpise+
Slot: Number 1, PowerLimit 10.000000
Slot: Enabled AtnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq-
Slot: AttnInd Unknown, PwrInd Unknown, Power-
Root: Correctable- Non-Fatal- Fatal- PME-
Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+
Address: fee0300c Data: 4159
Capabilities: [90] Subsystem: Gammagraphx, Inc. Unknown device 0000
Capabilities: [a0] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100] Virtual Channel
Capabilities: [180] Unknown (5)
00:1c.5 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 6 (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: 00003000-00003fff
Memory behind bridge: efc00000-efcfffff
Prefetchable memory behind bridge: 0000000020600000-00000000207fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Express Root Port (Slot+) IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s unlimited, L1 unlimited
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 6
Link: Latency L0s <256ns, L1 <4us
Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
Link: Speed 2.5Gb/s, Width x1
Slot: AtnBtn- PwrCtrl- MRL- AtnInd- PwrInd- HotPlug+ Surpise+
Slot: Number 0, PowerLimit 10.000000
Slot: Enabled AtnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq-
Slot: AttnInd Unknown, PwrInd Unknown, Power-
Root: Correctable- Non-Fatal- Fatal- PME-
Capabilities: [80] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+
Address: fee0300c Data: 4161
Capabilities: [90] Subsystem: Gammagraphx, Inc. Unknown device 0000
Capabilities: [a0] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100] Virtual Channel
Capabilities: [180] Unknown (5)
00:1d.0 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 (rev 01) (prog-if 00 [UHCI])
Subsystem: Dell Unknown device 01df
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Interrupt: pin A routed to IRQ 21
Region 4: I/O ports at ff80 [size=32]
00:1d.1 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #2 (rev 01) (prog-if 00 [UHCI])
Subsystem: Dell Unknown device 01df
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Interrupt: pin B routed to IRQ 22
Region 4: I/O ports at ff60 [size=32]
00:1d.2 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #3 (rev 01) (prog-if 00 [UHCI])
Subsystem: Dell Unknown device 01df
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Interrupt: pin C routed to IRQ 18
Region 4: I/O ports at ff40 [size=32]
00:1d.7 USB Controller: Intel Corporation 82801G (ICH7 Family) USB2 EHCI Controller (rev 01) (prog-if 20 [EHCI])
Subsystem: Dell Unknown device 01df
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Interrupt: pin A routed to IRQ 21
Region 0: Memory at ff980800 (32-bit, non-prefetchable) [size=1K]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Debug port
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev e1) (prog-if 01 [Subtractive decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Bus: primary=00, secondary=05, subordinate=05, sec-latency=32
I/O behind bridge: 0000d000-0000dfff
Memory behind bridge: efa00000-efbfffff
Prefetchable memory behind bridge: 00000000e0000000-00000000e7ffffff
Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
Capabilities: [50] Subsystem: Gammagraphx, Inc. Unknown device 0000
00:1f.0 ISA bridge: Intel Corporation 82801GB/GR (ICH7 Family) LPC Interface Bridge (rev 01)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Capabilities: [e0] Vendor Specific Information
00:1f.1 IDE interface: Intel Corporation 82801G (ICH7 Family) IDE Controller (rev 01) (prog-if 8a [Master SecP PriP])
Subsystem: Dell Unknown device 01df
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Interrupt: pin A routed to IRQ 16
Region 0: I/O ports at 01f0 [size=8]
Region 1: I/O ports at 03f4 [size=1]
Region 2: I/O ports at 0170 [size=8]
Region 3: I/O ports at 0374 [size=1]
Region 4: I/O ports at ffa0 [size=16]
00:1f.2 IDE interface: Intel Corporation 82801GB/GR/GH (ICH7 Family) SATA IDE Controller (rev 01) (prog-if 8f [Master SecP SecO PriP PriO])
Subsystem: Dell Unknown device 01df
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Interrupt: pin C routed to IRQ 20
Region 0: I/O ports at fe00 [size=8]
Region 1: I/O ports at fe10 [size=4]
Region 2: I/O ports at fe20 [size=8]
Region 3: I/O ports at fe30 [size=4]
Region 4: I/O ports at fec0 [size=16]
Region 5: Memory at effffc00 (32-bit, non-prefetchable) [size=1K]
Capabilities: [70] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:1f.3 SMBus: Intel Corporation 82801G (ICH7 Family) SMBus Controller (rev 01)
Subsystem: Dell Unknown device 01df
Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin B routed to IRQ 17
Region 4: I/O ports at ece0 [size=32]
04:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5754 Gigabit Ethernet PCI Express (rev 02)
Subsystem: Dell Unknown device 01df
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 28
Region 0: Memory at efcf0000 (64-bit, non-prefetchable) [size=64K]
Expansion ROM at <ignored> [disabled]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Capabilities: [58] Vendor Specific Information
Capabilities: [e8] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable+
Address: 00000000fee0300c Data: 4189
Capabilities: [d0] Express Endpoint IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag+
Device: Latency L0s <4us, L1 unlimited
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
Device: RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
Link: Latency L0s <4us, L1 <64us
Link: ASPM Disabled RCB 64 bytes CommClk+ ExtSynch-
Link: Speed 2.5Gb/s, Width x1
Capabilities: [100] Advanced Error Reporting
Capabilities: [13c] Virtual Channel
Capabilities: [160] Device Serial Number 1b-8a-38-fe-ff-a0-1a-00
Capabilities: [16c] Power Budgeting
05:02.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RT8139
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (8000ns min, 16000ns max)
Interrupt: pin A routed to IRQ 18
Region 0: I/O ports at d400 [size=256]
Region 1: Memory at efaefe00 (32-bit, non-prefetchable) [size=256]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
05:04.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RT8139
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (8000ns min, 16000ns max)
Interrupt: pin A routed to IRQ 16
Region 0: I/O ports at d800 [size=256]
Region 1: Memory at efaeff00 (32-bit, non-prefetchable) [size=256]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
05:07.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02) (prog-if 00 [VGA])
Subsystem: Dell Unknown device 01df
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR+ FastB2B-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin A routed to IRQ 5
Region 0: Memory at e0000000 (32-bit, prefetchable) [size=128M]
Region 1: I/O ports at dc00 [size=256]
Region 2: Memory at efaf0000 (32-bit, non-prefetchable) [size=64K]
Expansion ROM at efb00000 [disabled] [size=128K]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
driver: tg3
version: 3.108
firmware-version: 5754-v3.12
bus-info: 0000:04:00.0
root@fs:~#

Eric Dumazet

2010-08-20 06:35 UTC #7 lore raw

Quoted 2 lines · Plamen Petrov

Posting this only in the hope that it will be helpfull to the "net guys"...

Its a forwarding setup.

Please post

ifconfig -a
iptables -nvL
iptables -t nat -nvL
iptables -t mangle -nvL
ip route
ethtool -k eth0 (& eth1 ...)

Try to disable gro ?
ethtool -K eth0 gro off
ethtool -K eth1 gro off

Plamen Petrov

2010-08-20 06:57 UTC #8 lore raw

На 20.8.2010 г. 09:34, Eric Dumazet написа:

Quoted 15 lines · Plamen Petrov

Quoted 2 lines
Posting this only in the hope that it will be helpfull to the "net guys"...
Its a forwarding setup.
Please post
ifconfig -a
iptables -nvL
iptables -t nat -nvL
iptables -t mangle -nvL
ip route
ethtool -k eth0 (& eth1 ...)

Quoted 5 lines

Try to disable gro ?
ethtool -K eth0 gro off
ethtool -K eth1 gro off

Here goes the output of the above:

root@fs:~# ifconfig -a

Quoted 98 lines

eth0 Link encap:Ethernet HWaddr 00:0E:2E:5C:27:EF
          inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20e:2eff:fe5c:27ef/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:40986 errors:0 dropped:0 overruns:0 frame:0
          TX packets:32825 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:36081064 (34.4 MiB)  TX bytes:6457150 (6.1 MiB)
          Interrupt:18 Base address:0xe00
eth1 Link encap:Ethernet HWaddr 00:1A:A0:38:8A:1B
          inet addr:192.168.10.1  Bcast:192.168.10.255  Mask:255.255.255.0
          inet6 addr: fe80::21a:a0ff:fe38:8a1b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1220443 errors:0 dropped:0 overruns:0 frame:0
          TX packets:568452 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1477959893 (1.3 GiB)  TX bytes:144840881 (138.1 MiB)
          Interrupt:17
eth2 Link encap:Ethernet HWaddr 00:0E:2E:5C:27:E6
          inet addr:192.168.199.1  Bcast:192.168.199.255  Mask:255.255.255.0
          inet6 addr: fe80::20e:2eff:fe5c:27e6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1119 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1149 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:210401 (205.4 KiB)  TX bytes:505063 (493.2 KiB)
          Interrupt:16 Base address:0x4f00
gre0 Link encap:UNSPEC HWaddr 00-00-00-00-FF-00-73-69-00-00-00-00-00-00-00-00
NOARP MTU:1476 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
ifb0 Link encap:Ethernet HWaddr 96:9F:C4:43:78:01
BROADCAST NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:32
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
ifb1 Link encap:Ethernet HWaddr EA:B9:BD:02:C1:77
BROADCAST NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:32
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
ip6tnl0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
NOARP MTU:1460 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
lo Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:13546 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13546 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1051287 (1.0 MiB)  TX bytes:1051287 (1.0 MiB)
sit0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-73-69-00-00-00-00-00-00-00-00
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
sixxs_t Link encap:UNSPEC HWaddr C0-A8-01-02-00-00-73-69-00-00-00-00-00-00-00-00
inet6 addr: 2001:15c0:65ff:64::2/64 Scope:Global inet6 addr: fe80::c0a8:102/128 Scope:Link
UP POINTOPOINT RUNNING NOARP MTU:1280 Metric:1 RX packets:145 errors:0 dropped:0 overruns:0 frame:0 TX packets:145 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:15080 (14.7 KiB) TX bytes:15080 (14.7 KiB)
teql0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:192.168.11.1 P-t-P:192.168.11.2 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

Quoted 7 lines

tunl0 Link encap:IPIP Tunnel HWaddr
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

root@fs:~# iptables -nvL

Quoted 32 lines

Chain INPUT (policy ACCEPT 1207K packets, 1455M bytes)

 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:53
    0     0 DROP       udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           udp dpt:53
    0     0 DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:135
    0     0 DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:137
    0     0 DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:138
    0     0 DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:139
    0     0 DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:445
    0     0 DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:993
    0     0 DROP       udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           udp dpt:135
    0     0 DROP       udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           udp dpt:137
    0     0 DROP       udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           udp dpt:138
    0     0 DROP       udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           udp dpt:139
    0     0 DROP       udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           udp dpt:445
    0     0 DROP       udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           udp dpt:993

Chain FORWARD (policy DROP 0 packets, 0 bytes)

 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  tun0   *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      tun0    0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  eth2   eth1    192.168.199.15       192.168.10.0/24
    0     0 REJECT     all  --  eth2   *       0.0.0.0/0            192.168.1.1         reject-with icmp-net-unreachable
    0     0 REJECT     all  --  eth1   *       0.0.0.0/0            192.168.1.1         reject-with icmp-net-unreachable
29650 5450K ACCEPT     all  --  eth1   eth0    192.168.10.0/24     !192.168.1.1
  821  165K ACCEPT     all  --  eth2   eth0    192.168.199.0/24    !192.168.1.1
    0     0 ACCEPT     all  --  eth2   eth1    192.168.199.0/24     192.168.10.5
  624  438K ACCEPT     tcp  --  *      eth2    0.0.0.0/0            192.168.199.0/24    tcp flags:!0x17/0x02
33228   34M ACCEPT     tcp  --  *      eth1    0.0.0.0/0            192.168.10.0/24     tcp flags:!0x17/0x02
  199 14395 ACCEPT    !tcp  --  *      eth2    0.0.0.0/0            192.168.199.0/24
 2891  597K ACCEPT    !tcp  --  *      eth1    0.0.0.0/0            192.168.10.0/24

Quoted 2 lines

Chain OUTPUT (policy ACCEPT 546K packets, 100M bytes)
pkts bytes target prot opt in out source destination

root@fs:~# iptables -t nat -nvL

Quoted 41 lines

Chain PREROUTING (policy DROP 701 packets, 94310 bytes)

 pkts bytes target     prot opt in     out     source               destination
    0     0 LOG        tcp  --  eth1   *       192.168.10.0/24     !192.168.10.1        tcp dpt:25 LOG flags 0 level 4
    0     0 DROP       tcp  --  eth1   *       192.168.10.0/24     !192.168.10.1        tcp dpt:25
    0     0 LOG        tcp  --  eth2   *       192.168.199.0/24    !192.168.199.1       tcp dpt:25 LOG flags 0 level 4
    0     0 DROP       tcp  --  eth2   *       192.168.199.0/24    !192.168.199.1       tcp dpt:25
    0     0 LOG        tcp  --  tun0   *       192.168.11.0/24     !192.168.11.1        tcp dpt:25 LOG flags 0 level 4
    0     0 DROP       tcp  --  tun0   *       192.168.11.0/24     !192.168.11.1        tcp dpt:25
    0     0 ACCEPT     all  --  tun0   *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            192.168.11.0/24
    0     0 ACCEPT     all  --  *      *       192.168.11.0/24      0.0.0.0/0
    0     0 DROP       tcp  --  *      *       87.249.45.135        0.0.0.0/0           tcp dpts:20:22
    0     0 ACCEPT     all  --  eth0   *       212.18.63.73         192.168.1.2
    0     0 ACCEPT     all  --  eth2   *       192.168.199.15       192.168.10.0/24
    0     0 DROP       all  --  eth2   *       192.168.199.0/24     192.168.10.0/24
    0     0 DROP       all  --  eth1   *       192.168.10.0/24      192.168.199.0/24
 5629  597K ACCEPT     all  --  eth1   *       192.168.10.0/24      0.0.0.0/0
  262 26231 ACCEPT     all  --  eth2   *       192.168.199.0/24     0.0.0.0/0
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            192.168.1.2         tcp dpt:21 state NEW,ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            192.168.1.2         tcp spts:1024:65535 dpts:1024:65535 state RELATED,ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            192.168.1.2         tcp dpts:20:21 flags:0x17/0x02
    0     0 ACCEPT     tcp  --  *      *       77.0.0.0/8           192.168.1.2         tcp dpt:22 flags:0x17/0x02
    0     0 ACCEPT     tcp  --  *      *       87.0.0.0/8           192.168.1.2         tcp dpt:22 flags:0x17/0x02
    0     0 ACCEPT     tcp  --  *      *       90.0.0.0/8           192.168.1.2         tcp dpt:22 flags:0x17/0x02
    0     0 ACCEPT     tcp  --  *      *       95.0.0.0/8           192.168.1.2         tcp dpt:22 flags:0x17/0x02
    0     0 ACCEPT     tcp  --  *      *       212.45.77.11         192.168.1.2         tcp dpt:22 flags:0x17/0x02
    0     0 LOG        tcp  --  *      *       0.0.0.0/0            192.168.1.2         tcp dpt:22 flags:0x17/0x02 LOG flags 0 level 4
    0     0 DROP       tcp  --  *      *       0.0.0.0/0            192.168.1.2         tcp dpt:22 flags:0x17/0x02
   11   628 ACCEPT     tcp  --  *      *       0.0.0.0/0            192.168.1.2         tcp dpt:25 flags:0x17/0x02
    2    96 ACCEPT     tcp  --  *      *       0.0.0.0/0            192.168.1.2         tcp dpt:80 flags:0x17/0x02
    2   100 ACCEPT     tcp  --  *      *       0.0.0.0/0            192.168.1.2         tcp dpt:443 flags:0x17/0x02
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            192.168.1.2         tcp dpt:41414 flags:0x17/0x02
 1197 58544 DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp flags:0x17/0x02
    0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            192.168.1.2         tcp dpts:1023:65500
    6  2006 ACCEPT     udp  --  eth1   *       0.0.0.0              255.255.255.255     udp spts:67:68 dpts:67:68
    1   328 ACCEPT     udp  --  eth2   *       0.0.0.0              255.255.255.255     udp spts:67:68 dpts:67:68

Chain OUTPUT (policy ACCEPT 2685 packets, 196K bytes)
pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 2656 packets, 186K bytes)

Quoted 16 lines

 pkts bytes target     prot opt in     out     source               destination
    1   124 ACCEPT     all  --  *      eth0    192.168.1.2          212.18.63.73
    0     0 ACCEPT     tcp  --  *      eth0    192.168.1.2          0.0.0.0/0           tcp spts:1024:65535 dpts:1024:65535 state RELATED,ESTABLISHED
    0     0 ACCEPT     tcp  --  *      eth0    192.168.1.2          0.0.0.0/0           tcp spt:20 state RELATED,ESTABLISHED
    0     0 SNAT       all  --  *      eth0    192.168.10.16/29     0.0.0.0/0           to:192.168.1.2
    4   192 SNAT       all  --  *      eth0    192.168.10.24/29     0.0.0.0/0           to:192.168.1.2
    0     0 SNAT       all  --  *      eth0    192.168.10.32/29     0.0.0.0/0           to:192.168.1.2
    0     0 SNAT       all  --  *      eth0    192.168.10.40/29     0.0.0.0/0           to:192.168.1.2
    0     0 SNAT       all  --  *      eth0    192.168.10.48/29     0.0.0.0/0           to:192.168.1.2
    0     0 SNAT       all  --  *      eth0    192.168.10.56/29     0.0.0.0/0           to:192.168.1.2
    0     0 SNAT       all  --  *      eth0    192.168.10.64/28     0.0.0.0/0           to:192.168.1.2
 4893  516K SNAT       all  --  *      eth0    192.168.10.0/24      0.0.0.0/0           to:192.168.1.2
  211 16721 SNAT       all  --  *      eth0    192.168.199.0/24     0.0.0.0/0           to:192.168.1.2
   24  8328 ACCEPT     udp  --  *      eth1    192.168.10.1         255.255.255.255     udp spts:67:68 dpts:67:68
    4  1388 ACCEPT     udp  --  *      eth2    192.168.199.1        255.255.255.255     udp spts:67:68 dpts:67:68
    0     0 DROP       all  --  *      *       0.0.0.0              0.0.0.0/0

root@fs:~# iptables -t mangle -nvL

Quoted 6 lines

Chain PREROUTING (policy ACCEPT 1287K packets, 1508M bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 1212K packets, 1461M bytes)
pkts bytes target prot opt in out source destination

Quoted 8 lines

Chain FORWARD (policy ACCEPT 73581 packets, 47M bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 548K packets, 100M bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 622K packets, 147M bytes)
pkts bytes target prot opt in out source destination

root@fs:~# ip route

Quoted 7 lines

192.168.11.2 dev tun0 proto kernel scope link src 192.168.11.1 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.2 192.168.199.0/24 dev eth2 proto kernel scope link src 192.168.199.1 192.168.11.0/24 via 192.168.11.2 dev tun0
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.1 127.0.0.0/8 dev lo scope link
default via 192.168.1.1 dev eth0 metric 1

root@fs:~# ethtool -k eth0

Quoted 7 lines

Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: on

root@fs:~# ethtool -k eth1

Quoted 7 lines

Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on

root@fs:~# ethtool -k eth2

Quoted 7 lines

Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: on

root@fs:~# uname -a

Quoted 1 line

Linux fs 2.6.34.4-FS #1 SMP Thu Aug 19 18:09:58 UTC 2010 i686 Intel(R) Pentium(R) D CPU 3.00GHz GenuineIntel GNU/Linux

Plamen Petrov

2010-08-20 07:38 UTC #9 lore raw

In reply to Eric Dumazet

На 20.8.2010 г. 09:34, Eric Dumazet написа:

Quoted 15 lines · Plamen Petrov

Quoted 2 lines
Posting this only in the hope that it will be helpfull to the "net guys"...
Its a forwarding setup.
Please post
ifconfig -a
iptables -nvL
iptables -t nat -nvL
iptables -t mangle -nvL
ip route
ethtool -k eth0 (& eth1 ...)

Quoted 5 lines

Try to disable gro ?
ethtool -K eth0 gro off
ethtool -K eth1 gro off

Sorry about this, but when I dig a bit into this, I found that I had a pretty old ethtool:

Quoted 5 lines

root@fs:~# ethtool --help
ethtool version 5
Usage:
ethtool DEVNAME Display standard information about device
...

So my ethtool did not know anything about gro, because by looking at
http://git.kernel.org/?p=network/ethtool/ethtool.git;a=shortlog
I saw I had this one:
2006-09-01 Jeff Garzik Release version 5. origin v5
and gro reading/setting was added "a bit later"...
2009-03-06 Jeff Garzik Get/set GRO settings.

So, I went on and compiled myself a new ethtool:

Quoted 5 lines

root@fs:~# ethtool -h
ethtool version 2.6.35
Usage:
ethtool DEVNAME Display standard information about device
...

And run it on my 2.6.34.4 kernel, which produced these:

Quoted 1 line

root@fs:~# ethtool -k eth0

Quoted 11 lines

Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off
ntuple-filters: off
receive-hashing: off

root@fs:~# ethtool -k eth1

Quoted 11 lines

Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off
ntuple-filters: off
receive-hashing: off

root@fs:~# ethtool -k eth2

Quoted 11 lines

Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off
ntuple-filters: off
receive-hashing: off

root@fs:~#

Quoted 1 line

But again, this is on 2.6.34.4. Could it be that "generic-receive-offload" is ON by default on my only gigabit capable tg3 card on 2.6.35.x and later?

I will try to determine this myself later, but the machine is a gateway on the local area network, and is needed from the user here...

Thanks!

Plamen Petrov

2010-08-20 08:31 UTC #10 lore raw

In reply to Eric Dumazet

На 20.8.2010 г. 10:38, Plamen Petrov написа:

Quoted 5 lines

На 20.8.2010 г. 09:34, Eric Dumazet написа:
Quoted 4 lines
Quoted 4 lines · Plamen Petrov
Quoted 2 lines
Quoted 2 lines
Posting this only in the hope that it will be helpfull to the "net guys"...

...

Quoted 5 lines

Quoted 1 line
Quoted 1 line
But again, this is on 2.6.34.4. Could it be that "generic-receive-offload" is ON by default on my only gigabit capable tg3 card on 2.6.35.x and later?

Well, as it turns out - the commit enabling GRO for tg3 cards was from the 2.6.35 merge window:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=cb903bf4ee2d6e53210e2174d363e10698112042

Quoted 9 lines

tg3: Enable GRO by default.author David S. Miller <[email protected]>
	Wed, 21 Apr 2010 01:49:45 +0000 (18:49 -0700)
committer	David S. Miller <[email protected]>	
	Wed, 21 Apr 2010 01:49:45 +0000 (18:49 -0700)
This was merely an oversight when I added the *_gro_receive() calls.
Signed-off-by David S. Miller <[email protected]>

The next thing I will be trying is disabling
GRO for the tg3 network card in my network
startup scripts via my shiny new ethtool... ;)

Plamen Petrov

2010-08-20 09:19 UTC #11 lore raw

На 20.8.2010 г. 11:31, Plamen Petrov написа:
...

Quoted 3 lines

The next thing I will be trying is disabling
GRO for the tg3 network card in my network
startup scripts via my shiny new ethtool... ;)

So, for nearly half an hour now running 2.6.36-rc1-FS-00127-g763008c:

Quoted 1 line

root@fs:~# w; uname -a

Quoted 4 lines

12:09:48 up 26 min, 1 user, load average: 2.43, 1.93, 1.26
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    192.168.10.159   12:09    0.00s  0.02s  0.00s w
Linux fs 2.6.36-rc1-FS-00127-g763008c #1 SMP Thu Aug 19 07:10:57 UTC 2010 i686 Intel(R) Pentium(R) D CPU 3.00GHz GenuineIntel GNU/Linux

And that's with the usual load, which can be found here:
http://pvp.now.im/s/drraw.cgi?Mode=view;Dashboard=1173713287.13177

The only addition I made was running
ethtool -K ethX gro off
for each network card.

Which means that I have a Broadcom Tygon 3 (tg3) which doesn't like "generic-receive-offload" at all!

With "generic-receive-offload" off, I will stay on 2.6.36-rc1-FS-00127-g763008c for as long as I can, just to confirm that in fact the "default on" setting for gro in the tg3 driver is the culprit for my troubles...

Where to next?

Eric Dumazet

2010-08-20 10:26 UTC #12 lore raw

Quoted 25 lines · Plamen Petrov

На 20.8.2010 г. 11:31, Plamen Petrov написа:
...
Quoted 3 lines
The next thing I will be trying is disabling
GRO for the tg3 network card in my network
startup scripts via my shiny new ethtool... ;)
So, for nearly half an hour now running 2.6.36-rc1-FS-00127-g763008c:
Quoted 1 line
root@fs:~# w; uname -a
Quoted 4 lines
12:09:48 up 26 min, 1 user, load average: 2.43, 1.93, 1.26
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    192.168.10.159   12:09    0.00s  0.02s  0.00s w
Linux fs 2.6.36-rc1-FS-00127-g763008c #1 SMP Thu Aug 19 07:10:57 UTC 2010 i686 Intel(R) Pentium(R) D CPU 3.00GHz GenuineIntel GNU/Linux
And that's with the usual load, which can be found here:
http://pvp.now.im/s/drraw.cgi?Mode=view;Dashboard=1173713287.13177
The only addition I made was running
ethtool -K ethX gro off
for each network card.
Which means that I have a Broadcom Tygon 3 (tg3) which doesn't like "generic-receive-offload" at all!

Quoted 6 lines

With "generic-receive-offload" off, I will stay on 2.6.36-rc1-FS-00127-g763008c for as long as I can, just to confirm that in fact the "default on" setting for gro in the tg3 driver is the culprit for my troubles...
Where to next?

First, thanks a lot for finding this so fast.

Hmmm, please check that on your net/core/dev.c file, line 3146 you have the fix added in

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=e5093aec2e6b60c3df2420057ffab9ed4a6d2792

commit e5093aec2e6b60c3df2420057ffab9ed4a6d2792

Author: Jarek Poplawski <[email protected]>
Date:   Wed Aug 11 02:02:10 2010 +0000

net: Fix a memmove bug in dev_gro_receive()

--skb_shinfo(skb)->nr_frags * sizeof(skb_frag_t));

If yes, I think David & Herbert will probably have to dig again into gro code :(

Plamen Petrov

2010-08-20 10:53 UTC #13 lore raw

На 20.8.2010 г. 13:26, Eric Dumazet написа:

Quoted 3 lines · Plamen Petrov

Quoted 2 lines
На 20.8.2010 г. 11:31, Plamen Petrov написа:
...

...

Quoted 5 lines

Quoted 2 lines
Where to next?
First, thanks a lot for finding this so fast.

For nothing.

Quoted 7 lines

Hmmm, please check that on your net/core/dev.c file, line 3146 you have the fix added in
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=e5093aec2e6b60c3df2420057ffab9ed4a6d2792
commit e5093aec2e6b60c3df2420057ffab9ed4a6d2792

Quoted 9 lines

Author: Jarek Poplawski<[email protected]>
Date:   Wed Aug 11 02:02:10 2010 +0000
net: Fix a memmove bug in dev_gro_receive()
--skb_shinfo(skb)->nr_frags * sizeof(skb_frag_t));

Yes, I confirm I have the above code in place.

Quoted 4 lines

If yes, I think David& Herbert will probably have to dig again into gro code :(

So, I guess its David and Herbert's turn?...

Thanks!
Plamen

Jarek Poplawski

2010-08-20 19:38 UTC #14 lore raw

Plamen Petrov wrote, On 20.08.2010 12:53:

Quoted 1 line

So, I guess its David and Herbert's turn?...

If you're bored in the meantime I'd suggest to do check the realtek driver eg:
- for locking with the patch below,
- to turn off with ethtool its tx-checksumming and/or scatter-gather, or if possible try to reproduce this with other NIC.

Thanks,
Jarek P.
---

drivers/net/8139too.c +4 -2

index f5166dc..aaaccc5 100644
@@ -1692,6 +1692,8 @@ static netdev_tx_t rtl8139_start_xmit (struct sk_buff *skb,
 	unsigned int len = skb->len;
 	unsigned long flags;
 
+	spin_lock_irqsave(&tp->lock, flags);
+
 	/* Calculate the next Tx descriptor entry. */
 	entry = tp->cur_tx % NUM_TX_DESC;

@@ -1700,14 +1702,14 @@ static netdev_tx_t rtl8139_start_xmit (struct sk_buff *skb,
 		if (len < ETH_ZLEN)
 			memset(tp->tx_buf[entry], 0, ETH_ZLEN);
 		skb_copy_and_csum_dev(skb, tp->tx_buf[entry]);
-		dev_kfree_skb(skb);
+		dev_kfree_skb_irq(skb);
 	} else {
+		spin_unlock_irqrestore(&tp->lock, flags);
 		dev_kfree_skb(skb);
 		dev->stats.tx_dropped++;
 		return NETDEV_TX_OK;
 	}
 
-	spin_lock_irqsave(&tp->lock, flags);
 	/*
 	 * Writing to TxStatus triggers a DMA transfer of the data
 	 * copied to tp->tx_buf[entry] above. Use a memory barrier

Jarek Poplawski

2010-08-21 07:47 UTC #15 lore raw

Quoted 7 lines · Jarek Poplawski

Plamen Petrov wrote, On 20.08.2010 12:53:
Quoted 1 line
So, I guess its David and Herbert's turn?...
If you're bored in the meantime I'd suggest to do check the realtek driver eg:
- for locking with the patch below,
- to turn off with ethtool its tx-checksumming and/or scatter-gather,

After rethinking, it's almost impossible this patch could change anything here, so don't bother, but consider mainly the second proposal.

Jarek P.

Eric Dumazet

2010-08-21 07:51 UTC #16 lore raw

Quoted 9 lines · Jarek Poplawski

Quoted 7 lines · Jarek Poplawski
Plamen Petrov wrote, On 20.08.2010 12:53:
Quoted 1 line
So, I guess its David and Herbert's turn?...
If you're bored in the meantime I'd suggest to do check the realtek driver eg:
- for locking with the patch below,
- to turn off with ethtool its tx-checksumming and/or scatter-gather,

Quoted 5 lines

After rethinking, it's almost impossible this patch could change anything here, so don't bother, but consider mainly the second proposal.
Jarek P.

Indeed ;)

Its true that not many nics use the skb_copy_and_csum_dev() helper, maybe this one must be updated somehow ?

Jarek Poplawski

2010-08-21 08:07 UTC #17 lore raw

Quoted 14 lines · Eric Dumazet

Quoted 13 lines · Jarek Poplawski
Quoted 7 lines · Jarek Poplawski
Plamen Petrov wrote, On 20.08.2010 12:53:
Quoted 1 line
So, I guess its David and Herbert's turn?...
If you're bored in the meantime I'd suggest to do check the realtek driver eg:
- for locking with the patch below,
- to turn off with ethtool its tx-checksumming and/or scatter-gather,
After rethinking, it's almost impossible this patch could change anything here, so don't bother, but consider mainly the second proposal.

Quoted 7 lines

Quoted 1 line
Jarek P.
Indeed ;)
Its true that not many nics use the skb_copy_and_csum_dev() helper, maybe this one must be updated somehow ?

Yes, it seems it should be possible at least to handle the bug with a warning and error return, considering Plamen's problems with getting the trace.

Jarek P.

Plamen Petrov

2010-08-23 11:47 UTC #18 lore raw

На 21.8.2010 г. 11:07, Jarek Poplawski написа:

Quoted 19 lines · Eric Dumazet

Quoted 18 lines · Jarek Poplawski
Quoted 9 lines · Jarek Poplawski
Quoted 7 lines
Plamen Petrov wrote, On 20.08.2010 12:53:
Quoted 1 line
So, I guess its David and Herbert's turn?...
If you're bored in the meantime I'd suggest to do check the realtek driver eg:
- for locking with the patch below,
- to turn off with ethtool its tx-checksumming and/or scatter-gather,
Quoted 5 lines
After rethinking, it's almost impossible this patch could change anything here, so don't bother, but consider mainly the second proposal.
Jarek P.
Indeed ;)

Quoted 8 lines

Quoted 3 lines
Its true that not many nics use the skb_copy_and_csum_dev() helper, maybe this one must be updated somehow ?
Yes, it seems it should be possible at least to handle the bug with a warning and error return, considering Plamen's problems with getting the trace.
Jarek P.

Well, here is the current status:

Last I promised I will stay on 2.6.36-rc1-git for as long as possible, so here is what I achieved:

Quoted 1 line

root@fs:/boot# w; uname -a

Quoted 4 lines

12:08:18 up 3 days, 24 min, 1 user, load average: 1.21, 1.29, 1.17
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    192.168.10.159   12:04    0.00s  0.02s  0.00s w
Linux fs 2.6.36-rc1-FS-00127-g763008c #1 SMP Thu Aug 19 07:10:57 UTC 2010 i686 Intel(R) Pentium(R) D CPU 3.00GHz GenuineIntel GNU/Linux

Yeah, 3 days and counting, right until I decided to try the freshly announced 2.6.36-rc2.

So I upgraded the kernel, but left the scripts that turn GRO off for the tg3 card still run at system startup. This way the system ran for 2 and a half hours, when I decided its time to try turning GRO on.

I first tried to turn GRO on for the tg3 nic, and the system oopsed immediately (if the panic screen is necessary - please, ask for it).

After the system came back, I tried turning GRO on for the 2 RealTek 8139 nics, too, but ethtool only accepted turning GRO off.

And unfortunately, I can't test if other nics will fail the same way as the motherboard integrated tg3 I have does, so for now, this is only a tg3 + GRO on problem; I don't have any other hardware to test with available.

Thanks,
Plamen

Eric Dumazet

2010-08-23 12:35 UTC #19 lore raw

Quoted 22 lines · Plamen Petrov

Well, here is the current status:
Last I promised I will stay on 2.6.36-rc1-git for as long as possible, so here is what I achieved:
Quoted 1 line
root@fs:/boot# w; uname -a
Quoted 4 lines
12:08:18 up 3 days, 24 min, 1 user, load average: 1.21, 1.29, 1.17
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    192.168.10.159   12:04    0.00s  0.02s  0.00s w
Linux fs 2.6.36-rc1-FS-00127-g763008c #1 SMP Thu Aug 19 07:10:57 UTC 2010 i686 Intel(R) Pentium(R) D CPU 3.00GHz GenuineIntel GNU/Linux
Yeah, 3 days and counting, right until I decided to try the freshly announced 2.6.36-rc2.
So I upgraded the kernel, but left the scripts that turn GRO off for the tg3 card still run at system startup. This way the system ran for 2 and a half hours, when I decided its time to try turning GRO on.
I first tried to turn GRO on for the tg3 nic, and the system oopsed immediately (if the panic screen is necessary - please, ask for it).

Quoted 7 lines

After the system came back, I tried turning GRO on for the 2 RealTek 8139 nics, too, but ethtool only accepted turning GRO off.
And unfortunately, I can't test if other nics will fail the same way as the motherboard integrated tg3 I have does, so for now, this is only a tg3 + GRO on problem; I don't have any other hardware to test with available.

There was no change in latest kernel in this area.

Should you have only tg3 cards, I guess there would be no bug.

Bug is probably a combination of :

1) tg3 + GRO , or any card enabling GRO
2) Some network code (netfilter ?)
3) a 8139too, or any card calling skb_copy_and_csum_dev()

Jarek Poplawski

2010-08-23 12:47 UTC #20 lore raw

In reply to Plamen Petrov

Quoted 11 lines · Plamen Petrov

???? 21.8.2010 ??. 11:07, Jarek Poplawski ????????????:
Quoted 10 lines · Eric Dumazet
Quoted 9 lines · Jarek Poplawski
Quoted 8 lines · Jarek Poplawski
Quoted 7 lines
Plamen Petrov wrote, On 20.08.2010 12:53:
Quoted 1 line
So, I guess its David and Herbert's turn?...
If you're bored in the meantime I'd suggest to do check the realtek driver eg:
- for locking with the patch below,
- to turn off with ethtool its tx-checksumming and/or scatter-gather,

...

Quoted 10 lines

Yeah, 3 days and counting, right until I decided to try the freshly announced 2.6.36-rc2.
So I upgraded the kernel, but left the scripts that turn GRO off for the tg3 card still run at system startup. This way the system ran for 2 and a half hours, when I decided its time to try turning GRO on.
I first tried to turn GRO on for the tg3 nic, and the system oopsed immediately (if the panic screen is necessary - please, ask for it).

Quoted 7 lines

After the system came back, I tried turning GRO on for the 2 RealTek 8139 nics, too, but ethtool only accepted turning GRO off.
And unfortunately, I can't test if other nics will fail the same way as the motherboard integrated tg3 I have does, so for now, this is only a tg3 + GRO on problem; I don't have any other hardware to test with available.

A little misunderstanding: I was intersted with turning off some features on realteks to change the packet path from tg3 with gro to realtek without gro and without tx-checksumming etc.

But maybe you could try the patch below instead (so the patched kernel, tg3 with gro on, and realteks without any change).

Thanks,
Jarek P.

--- (for debugging only)

net/core/dev.c +25 -1

index 3721fbb..51823cd 100644
@@ -1935,6 +1935,23 @@ static inline int skb_needs_linearize(struct sk_buff *skb,
 					      illegal_highdma(dev, skb))));
 }
 
+static int skb_csum_start_bug(struct sk_buff *skb)
+{
+
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		long csstart;
+
+		csstart = skb->csum_start - skb_headroom(skb);
+		if (WARN_ON(csstart > skb_headlen(skb))) {
+			pr_warning("csum_start %d, headroom %d, headlen %d\n",
+				   skb->csum_start, skb_headroom(skb),
+				   skb_headlen(skb));
+			return 1;
+		}
+	}
+	return 0;
+}
+
 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 			struct netdev_queue *txq)
 {

@@ -1955,11 +1972,13 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 		skb_orphan_try(skb);
 
 		if (netif_needs_gso(dev, skb)) {
+			skb_csum_start_bug(skb);
 			if (unlikely(dev_gso_segment(skb)))
 				goto out_kfree_skb;
 			if (skb->next)
 				goto gso;
 		} else {
+			skb_csum_start_bug(skb);
 			if (skb_needs_linearize(skb, dev) &&
 			    __skb_linearize(skb))
 				goto out_kfree_skb;

@@ -1997,7 +2016,12 @@ gso:
 		if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
 			skb_dst_drop(nskb);
 
-		rc = ops->ndo_start_xmit(nskb, dev);
+		if (skb_csum_start_bug(skb)) {
+			kfree_skb(skb);
+			rc = NETDEV_TX_OK;
+		} else
+			rc = ops->ndo_start_xmit(nskb, dev);
+
 		if (unlikely(rc != NETDEV_TX_OK)) {
 			if (rc & ~NETDEV_TX_MASK)
 				goto out_kfree_gso_skb;

Eric Dumazet

2010-08-23 13:00 UTC #21 lore raw

Quoted 52 lines · Jarek Poplawski

Quoted 11 lines · Plamen Petrov
???? 21.8.2010 ??. 11:07, Jarek Poplawski ????????????:
Quoted 10 lines · Eric Dumazet
Quoted 9 lines · Jarek Poplawski
Quoted 8 lines · Jarek Poplawski
Quoted 7 lines
Plamen Petrov wrote, On 20.08.2010 12:53:
Quoted 1 line
So, I guess its David and Herbert's turn?...
If you're bored in the meantime I'd suggest to do check the realtek driver eg:
- for locking with the patch below,
- to turn off with ethtool its tx-checksumming and/or scatter-gather,
...
Quoted 10 lines
Yeah, 3 days and counting, right until I decided to try the freshly announced 2.6.36-rc2.
So I upgraded the kernel, but left the scripts that turn GRO off for the tg3 card still run at system startup. This way the system ran for 2 and a half hours, when I decided its time to try turning GRO on.
I first tried to turn GRO on for the tg3 nic, and the system oopsed immediately (if the panic screen is necessary - please, ask for it).
Quoted 7 lines
After the system came back, I tried turning GRO on for the 2 RealTek 8139 nics, too, but ethtool only accepted turning GRO off.
And unfortunately, I can't test if other nics will fail the same way as the motherboard integrated tg3 I have does, so for now, this is only a tg3 + GRO on problem; I don't have any other hardware to test with available.
A little misunderstanding: I was intersted with turning off some features on realteks to change the packet path from tg3 with gro to realtek without gro and without tx-checksumming etc.
But maybe you could try the patch below instead (so the patched kernel, tg3 with gro on, and realteks without any change).
Thanks,
Jarek P.
--- (for debugging only)
net/core/dev.c +1 -0
index 3721fbb..51823cd 100644
@@ -1935,6 +1935,23 @@ static inline int skb_needs_linearize(struct sk_buff *skb,
 					      illegal_highdma(dev, skb))));
 }
 
+static int skb_csum_start_bug(struct sk_buff *skb)

net/core/dev.c static inline int skb_needs_linearize(struct sk_buff *skb, · line 1948

Quoted 10 lines

net/core/dev.c +10 -0

+{
+
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		long csstart;
+
+		csstart = skb->csum_start - skb_headroom(skb);
+		if (WARN_ON(csstart > skb_headlen(skb))) {
+			pr_warning("csum_start %d, headroom %d, headlen %d\n",
+				   skb->csum_start, skb_headroom(skb),
+				   skb_headlen(skb));

I was about to suggest a similar patch ;)

Also prints skb->csum_offset and skb->len if possible

pr_err("csum_start %u, offset %u, headroom %d, headlen %d, len %d\n",

        skb->csum_start,
	skb->csum_offset,
	skb_headroom(skb),
        skb_headlen(skb),
	skb->len);

Quoted 37 lines

net/core/dev.c +14 -1

+			return 1;
+		}
+	}
+	return 0;
+}
+
 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 			struct netdev_queue *txq)
 {

@@ -1955,11 +1972,13 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 		skb_orphan_try(skb);
 
 		if (netif_needs_gso(dev, skb)) {
+			skb_csum_start_bug(skb);
 			if (unlikely(dev_gso_segment(skb)))
 				goto out_kfree_skb;
 			if (skb->next)
 				goto gso;
 		} else {
+			skb_csum_start_bug(skb);
 			if (skb_needs_linearize(skb, dev) &&
 			    __skb_linearize(skb))
 				goto out_kfree_skb;

@@ -1997,7 +2016,12 @@ gso:
 		if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
 			skb_dst_drop(nskb);
 
-		rc = ops->ndo_start_xmit(nskb, dev);
+		if (skb_csum_start_bug(skb)) {
+			kfree_skb(skb);
+			rc = NETDEV_TX_OK;
+		} else
+			rc = ops->ndo_start_xmit(nskb, dev);
+
 		if (unlikely(rc != NETDEV_TX_OK)) {
 			if (rc & ~NETDEV_TX_MASK)
 				goto out_kfree_gso_skb;

Jarek Poplawski

2010-08-23 13:11 UTC #22 lore raw

On Mon, Aug 23, 2010 at 03:00:43PM +0200, Eric Dumazet wrote:
...

Quoted 3 lines

I was about to suggest a similar patch ;)
Also prints skb->csum_offset and skb->len if possible

Feel free to send it: I'm a bit in hurry now...

Jarek P.

Plamen Petrov

2010-08-23 13:43 UTC #23 lore raw

На 23.8.2010 г. 16:10, Jarek Poplawski написа:

Quoted 9 lines · Eric Dumazet

...
Quoted 3 lines
I was about to suggest a similar patch ;)
Also prints skb->csum_offset and skb->len if possible
Feel free to send it: I'm a bit in hurry now...
Jarek P.

Currently compiling the kernel with the attached patch.

And, by the way, if there are any patches to follow, would you please, send them as attachments? I think my mail client is line-wrapping the patches...

The results will be sent soon, too.

Plamen

net/core/dev.c +28 -1

index 3721fbb..24a53d0 100644
@@ -1935,6 +1935,26 @@ static inline int skb_needs_linearize(struct sk_buff *skb,
 					      illegal_highdma(dev, skb))));
 }
 
+static int skb_csum_start_bug(struct sk_buff *skb)
+{
+
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		long csstart;
+		
+		csstart = skb->csum_start - skb_headroom(skb);
+		if (WARN_ON(csstart > skb_headlen(skb))) {
+		    pr_warning("csum_start %u, offset %u, headroom %d, headlen %d, len %d\n",
+			skb->csum_start,
+			skb->csum_offset,
+			skb_headroom(skb),
+			skb_headlen(skb),
+			skb->len);
+		    return 1;
+		}
+	}
+	return 0;
+}
+
 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 			struct netdev_queue *txq)
 {

@@ -1955,11 +1975,13 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 		skb_orphan_try(skb);
 
 		if (netif_needs_gso(dev, skb)) {
+			skb_csum_start_bug(skb);
 			if (unlikely(dev_gso_segment(skb)))
 				goto out_kfree_skb;
 			if (skb->next)
 				goto gso;
 		} else {
+			skb_csum_start_bug(skb);
 			if (skb_needs_linearize(skb, dev) &&
 			    __skb_linearize(skb))
 				goto out_kfree_skb;

@@ -1997,7 +2019,12 @@ gso:
 		if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
 			skb_dst_drop(nskb);
 
-		rc = ops->ndo_start_xmit(nskb, dev);
+		if (skb_csum_start_bug(skb)) {
+		    kfree_skb(skb);
+		    rc = NETDEV_TX_OK;
+		} else
+		    rc = ops->ndo_start_xmit(nskb, dev);
+		    
 		if (unlikely(rc != NETDEV_TX_OK)) {
 			if (rc & ~NETDEV_TX_MASK)
 				goto out_kfree_gso_skb;

Plamen Petrov

2010-08-23 14:05 UTC #24 lore raw

На 23.8.2010 г. 16:43, Plamen Petrov написа:

Quoted 11 lines

На 23.8.2010 г. 16:10, Jarek Poplawski написа:
Quoted 9 lines · Eric Dumazet
...
Quoted 3 lines
I was about to suggest a similar patch ;)
Also prints skb->csum_offset and skb->len if possible
Feel free to send it: I'm a bit in hurry now...
Jarek P.

Quoted 7 lines

Currently compiling the kernel with the attached patch.
And, by the way, if there are any patches to follow, would you please, send them as attachments? I think my mail client is line-wrapping the patches...
The results will be sent soon, too.

Well, so far - so good. No oopses.

I think I'm hitting some compiler related issue here...

The kernel with the patch I sent applied is working pretty fine even with "generic-receive-offload" ON for my tg3 nic...

What is the big difference there?

Or do we need to dig into the object files now? Because that's what's too scary for me...

Plamen

P.S. The compiler I am using is :

root@crux:~# gcc -v

Quoted 5 lines

Using built-in specs.
Target: i686-pc-linux-gnu
Configured with: ../gcc-4.4.4/configure --prefix=/usr --libexecdir=/usr/lib --enable-languages=c,c++,objc --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-shared --disable-nls --with-x=no Thread model: posix
gcc version 4.4.4 (CRUX) (GCC)

Anything obviously special about it?

Jarek Poplawski

2010-08-23 14:14 UTC #25 lore raw

Quoted 8 lines · Plamen Petrov

Well, so far - so good. No oopses.
I think I'm hitting some compiler related issue here...
The kernel with the patch I sent applied is working pretty fine even with "generic-receive-offload" ON for my tg3 nic...
What is the big difference there?

Oopses should be replaced with warnings, so please check the syslog etc from time to time.

Jarek P.

Plamen Petrov

2010-08-24 04:51 UTC #26 lore raw

На 23.8.2010 г. 17:14, Jarek Poplawski написа:

Quoted 8 lines · Plamen Petrov

Quoted 7 lines
Well, so far - so good. No oopses.
I think I'm hitting some compiler related issue here...
The kernel with the patch I sent applied is working pretty fine even with "generic-receive-offload" ON for my tg3 nic...

Quoted 6 lines

Quoted 1 line
What is the big difference there?
Oopses should be replaced with warnings, so please check the syslog etc from time to time.
Jarek P.

So far - not so good. :(

After I left the machine, i rebooted once. Then I logged in via ssh, and turned gro back ON for the tg3 nic - and in the next 30 minutes - another reboot.

Today I will try to capture a picture of the oops/warning with my phone camera, because contrary to what you suggest, Jarek, there are no messages in the system logs.

Plamen

Eric Dumazet

2010-08-24 05:01 UTC #27 lore raw

Quoted 9 lines · Plamen Petrov

So far - not so good. :(
After I left the machine, i rebooted once. Then I logged in via ssh, and turned gro back ON for the tg3 nic - and in the next 30 minutes - another reboot.
Today I will try to capture a picture of the oops/warning with my phone camera, because contrary to what you suggest, Jarek, there are no messages in the system logs.

Hmm... I was thinking adding a call __skb_linearize(), just in case...

drivers/net/8139too.c +1 -1

index f5166dc..10928a2 100644
@@ -1696,7 +1696,7 @@ static netdev_tx_t rtl8139_start_xmit (struct sk_buff *skb,
 	entry = tp->cur_tx % NUM_TX_DESC;
 
 	/* Note: the chip doesn't have auto-pad! */
-	if (likely(len < TX_BUF_SIZE)) {
+	if (likely(len < TX_BUF_SIZE && !__skb_linearize(skb))) {
 		if (len < ETH_ZLEN)
 			memset(tp->tx_buf[entry], 0, ETH_ZLEN);
 		skb_copy_and_csum_dev(skb, tp->tx_buf[entry]);

Plamen Petrov

2010-08-24 08:43 UTC #28 lore raw

На 24.8.2010 г. 08:01, Eric Dumazet написа:

Quoted 2 lines · Plamen Petrov

Quoted 1 line
So far - not so good. :(

...

Quoted 10 lines

Hmm... I was thinking adding a call __skb_linearize(), just in case...
drivers/net/8139too.c +0 -0
index f5166dc..10928a2 100644

drivers/net/8139too.c static netdev_tx_t rtl8139_start_xmit (struct sk_buff *skb, · line 1702

Quoted 8 lines

drivers/net/8139too.c +1 -1

@@ -1696,7 +1696,7 @@ static netdev_tx_t rtl8139_start_xmit (struct sk_buff *skb,
  	entry = tp->cur_tx % NUM_TX_DESC;
 
  	/* Note: the chip doesn't have auto-pad! */
-	if (likely(len<  TX_BUF_SIZE)) {
+	if (likely(len<  TX_BUF_SIZE&&  !__skb_linearize(skb))) {
  		if (len<  ETH_ZLEN)
  			memset(tp->tx_buf[entry], 0, ETH_ZLEN);
  		skb_copy_and_csum_dev(skb, tp->tx_buf[entry]);

Here is what I've got while running the kernel with the patch I attached earlier plus the above one, after turning GRO on for my tg3 nic:

[picture 10]
http://picpaste.com/37e37f9ff9504e3a003f49092f9b1be6.jpg

[picture 11]
http://picpaste.com/5663ca7c5041c0ed7a1f3e6a9aa17a9e.jpg

[picture 12]
http://picpaste.com/17f2ecaa409a1ebbf835a2e0519d3099.jpg

Now moving on to the second proposed patch from Jarek.

Thanks,
Plamen

Plamen Petrov

2010-08-24 13:28 UTC #29 lore raw

На 24.8.2010 г. 11:43, Plamen Petrov написа:

Quoted 3 lines

На 24.8.2010 г. 08:01, Eric Dumazet написа:
Quoted 2 lines · Plamen Petrov
Quoted 1 line
So far - not so good. :(

...

Quoted 11 lines

Quoted 1 line
Here is what I've got while running the kernel with the patch I attached earlier plus the above one, after turning GRO on for my tg3 nic:
[picture 10]
http://picpaste.com/37e37f9ff9504e3a003f49092f9b1be6.jpg
[picture 11]
http://picpaste.com/5663ca7c5041c0ed7a1f3e6a9aa17a9e.jpg

Quoted 7 lines

[picture 12]
http://picpaste.com/17f2ecaa409a1ebbf835a2e0519d3099.jpg
Now moving on to the second proposed patch from Jarek.
Thanks,
Plamen

The current status: if I enable GRO on the tg3 - the kernel oopses. It just takes a different amount of time to trigger: somewhere from 30 seconds to 30 minutes.

The oopses looks the same, and here are the latest:

[picture 13]
http://picpaste.com/c8dbda8f5c15d9ce3e050dd7f245f5d0.jpg

[picture 14]
http://picpaste.com/646cca586b704c5b72d3cf9fa54c7344.jpg

I was wondering which debug options could help us track this down?

Thanks,
Plamen

Eric Dumazet

2010-08-24 15:08 UTC #30 lore raw

Quoted 6 lines · Plamen Petrov

The current status: if I enable GRO on the tg3 - the kernel oopses. It just takes a different amount of time to trigger: somewhere from 30 seconds to 30 minutes.
The oopses looks the same, and here are the latest:

Quoted 8 lines

[picture 13]
http://picpaste.com/c8dbda8f5c15d9ce3e050dd7f245f5d0.jpg
[picture 14]
http://picpaste.com/646cca586b704c5b72d3cf9fa54c7344.jpg
I was wondering which debug options could help us track this down?

Thanks, here is an updated patch (against linux-2.6)

net/core/dev.c +43 -2

index 3721fbb..77c8eb7 100644
@@ -1935,6 +1935,32 @@ static inline int skb_needs_linearize(struct sk_buff *skb,
 					      illegal_highdma(dev, skb))));
 }
 
+int skb_csum_start_bug(const struct sk_buff *skb, int pos)
+{
+
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		long csstart;
+
+		csstart = skb->csum_start - skb_headroom(skb);
+		if (WARN_ON(csstart > skb_headlen(skb))) {
+			int i;
+
+			pr_err("%d: csum_start %u, offset %u, headroom %d, headlen %d, len %d\n",
+				   pos, skb->csum_start, skb->csum_offset, skb_headroom(skb),
+				   skb_headlen(skb), skb->len);
+			pr_err("nr_frags=%u gso_size=%u ",
+					skb_shinfo(skb)->nr_frags,
+					skb_shinfo(skb)->gso_size);
+			for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
+				pr_err("frag_size=%u ", skb_shinfo(skb)->frags[i].size);
+			}
+			pr_err("\n");
+			return 1;
+		}
+	}
+	return 0;
+}
+
 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 			struct netdev_queue *txq)
 {

@@ -1959,11 +1985,15 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 				goto out_kfree_skb;
 			if (skb->next)
 				goto gso;
+			if (skb_csum_start_bug(skb, 10))
+				goto out_kfree_skb;
 		} else {
 			if (skb_needs_linearize(skb, dev) &&
 			    __skb_linearize(skb))
 				goto out_kfree_skb;
 
+			if (skb_csum_start_bug(skb, 20))
+				goto out_kfree_skb;
 			/* If packet is not checksummed and device does not
 			 * support checksumming for this protocol, complete
 			 * checksumming here.

@@ -1974,10 +2004,16 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 				if (!dev_can_checksum(dev, skb) &&
 				     skb_checksum_help(skb))
 					goto out_kfree_skb;
+				if (skb_csum_start_bug(skb, 30))
+					goto out_kfree_skb;
 			}
 		}
 
-		rc = ops->ndo_start_xmit(skb, dev);
+		if (skb_csum_start_bug(skb, 40)) {
+			kfree_skb(skb);
+			rc = NETDEV_TX_OK;
+		} else
+			rc = ops->ndo_start_xmit(skb, dev);
 		if (rc == NETDEV_TX_OK)
 			txq_trans_update(txq);
 		return rc;

@@ -1997,7 +2033,12 @@ gso:
 		if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
 			skb_dst_drop(nskb);
 
-		rc = ops->ndo_start_xmit(nskb, dev);
+		if (skb_csum_start_bug(skb, 50)) {
+			kfree_skb(skb);
+			rc = NETDEV_TX_OK;
+		} else
+			rc = ops->ndo_start_xmit(nskb, dev);
+
 		if (unlikely(rc != NETDEV_TX_OK)) {
 			if (rc & ~NETDEV_TX_MASK)
 				goto out_kfree_gso_skb;

net/core/skbuff.c +3 -1

index 3a2513f..3d54a1b 100644
@@ -1824,13 +1824,15 @@ void skb_copy_and_csum_dev(const struct sk_buff *skb, u8 *to)
 {
 	__wsum csum;
 	long csstart;
+	extern int skb_csum_start_bug(const struct sk_buff *skb, int pos);
 
 	if (skb->ip_summed == CHECKSUM_PARTIAL)
 		csstart = skb->csum_start - skb_headroom(skb);
 	else
 		csstart = skb_headlen(skb);
 
-	BUG_ON(csstart > skb_headlen(skb));
+	if (skb_csum_start_bug(skb, 100))
+		return;
 
 	skb_copy_from_linear_data(skb, to, csstart);

Plamen Petrov

2010-08-24 17:30 UTC #31 lore raw

Eric Dumazet написа:

Quoted 117 lines · Plamen Petrov

Quoted 7 lines

The current status: if I enable GRO on the tg3 - the kernel oopses. It just takes a different amount of time to trigger: somewhere from 30 seconds to 30 minutes.
The oopses looks the same, and here are the latest:

Quoted 8 lines

[picture 13]
http://picpaste.com/c8dbda8f5c15d9ce3e050dd7f245f5d0.jpg
[picture 14]
http://picpaste.com/646cca586b704c5b72d3cf9fa54c7344.jpg
I was wondering which debug options could help us track this down?

Thanks, here is an updated patch (against linux-2.6)

net/core/dev.c +43 -2

index 3721fbb..77c8eb7 100644
@@ -1935,6 +1935,32 @@ static inline int skb_needs_linearize(struct sk_buff *skb,
 					      illegal_highdma(dev, skb))));
 }
 
+int skb_csum_start_bug(const struct sk_buff *skb, int pos)
+{
+
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		long csstart;
+
+		csstart = skb->csum_start - skb_headroom(skb);
+		if (WARN_ON(csstart > skb_headlen(skb))) {
+			int i;
+
+			pr_err("%d: csum_start %u, offset %u, headroom %d, headlen %d, len %d\n",
+				   pos, skb->csum_start, skb->csum_offset, skb_headroom(skb),
+				   skb_headlen(skb), skb->len);
+			pr_err("nr_frags=%u gso_size=%u ",
+					skb_shinfo(skb)->nr_frags,
+					skb_shinfo(skb)->gso_size);
+			for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
+				pr_err("frag_size=%u ", skb_shinfo(skb)->frags[i].size);
+			}
+			pr_err("\n");
+			return 1;
+		}
+	}
+	return 0;
+}
+
 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 			struct netdev_queue *txq)
 {

@@ -1959,11 +1985,15 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 				goto out_kfree_skb;
 			if (skb->next)
 				goto gso;
+			if (skb_csum_start_bug(skb, 10))
+				goto out_kfree_skb;
 		} else {
 			if (skb_needs_linearize(skb, dev) &&
 			    __skb_linearize(skb))
 				goto out_kfree_skb;
 
+			if (skb_csum_start_bug(skb, 20))
+				goto out_kfree_skb;
 			/* If packet is not checksummed and device does not
 			 * support checksumming for this protocol, complete
 			 * checksumming here.

@@ -1974,10 +2004,16 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 				if (!dev_can_checksum(dev, skb) &&
 				     skb_checksum_help(skb))
 					goto out_kfree_skb;
+				if (skb_csum_start_bug(skb, 30))
+					goto out_kfree_skb;
 			}
 		}
 
-		rc = ops->ndo_start_xmit(skb, dev);
+		if (skb_csum_start_bug(skb, 40)) {
+			kfree_skb(skb);
+			rc = NETDEV_TX_OK;
+		} else
+			rc = ops->ndo_start_xmit(skb, dev);
 		if (rc == NETDEV_TX_OK)
 			txq_trans_update(txq);
 		return rc;

@@ -1997,7 +2033,12 @@ gso:
 		if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
 			skb_dst_drop(nskb);
 
-		rc = ops->ndo_start_xmit(nskb, dev);
+		if (skb_csum_start_bug(skb, 50)) {
+			kfree_skb(skb);
+			rc = NETDEV_TX_OK;
+		} else
+			rc = ops->ndo_start_xmit(nskb, dev);
+
 		if (unlikely(rc != NETDEV_TX_OK)) {
 			if (rc & ~NETDEV_TX_MASK)
 				goto out_kfree_gso_skb;

net/core/skbuff.c +1 -0

index 3a2513f..3d54a1b 100644
@@ -1824,13 +1824,15 @@ void skb_copy_and_csum_dev(const struct sk_buff *skb, u8 *to)
 {
 	__wsum csum;
 	long csstart;
+	extern int skb_csum_start_bug(const struct sk_buff *skb, int pos);
 
 	if (skb->ip_summed == CHECKSUM_PARTIAL)

net/core/skbuff.c void skb_copy_and_csum_dev(const struct sk_buff *skb, u8 *to) · line 1838

Quoted 10 lines

net/core/skbuff.c +2 -1

 		csstart = skb->csum_start - skb_headroom(skb);
 	else
 		csstart = skb_headlen(skb);
 
-	BUG_ON(csstart > skb_headlen(skb));
+	if (skb_csum_start_bug(skb, 100))
+		return;
 
 	skb_copy_from_linear_data(skb, to, csstart);

Above patch applied, and happy to report the machine now spits data in the logs instead of oopsing. Here is what we have now:

[   10.721802] Ending clean XFS mount for filesystem: md12
[   11.669013] IPv4 FIB: Using LC-trie version 0.409
[   11.669101] eth2: link up, 100Mbps, full-duplex, lpa 0x45E1
[   11.746792] eth0: link up, 100Mbps, full-duplex, lpa 0x41E1
[   11.757230] tg3 0000:04:00.0: irq 44 for MSI/MSI-X
[   11.810133] ADDRCONF(NETDEV_UP): eth1: link is not ready
[   11.957523] sixxs_t: Disabled Privacy Extensions
[   14.843711] tg3 0000:04:00.0: eth1: Link is up at 1000 Mbps, full duplex
[   14.843717] tg3 0000:04:00.0: eth1: Flow control is on for TX and on for RX
[   14.843753] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   15.854861] tun0: Disabled Privacy Extensions
[  699.375620] ------------[ cut here ]------------
[  699.475648] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0xf2()
[  699.575667] Hardware name: PowerEdge SC440
[  699.675688] Pid: 2963, comm: FahCore_78.exe Not tainted 2.6.36-rc2-FS-00103-g2d6fa25 #1
[  699.775706] Call Trace:
[  699.975744]  [<c102d86c>] ? warn_slowpath_common+0x67/0x8c
[  700.175779]  [<c12abc76>] ? skb_csum_start_bug+0x46/0xf2
[  700.375813]  [<c12abc76>] ? skb_csum_start_bug+0x46/0xf2
[  700.575848]  [<c102d8ac>] ? warn_slowpath_null+0x1b/0x1f
[  700.775882]  [<c12abc76>] ? skb_csum_start_bug+0x46/0xf2
[  700.975918]  [<c1024569>] ? __wake_up_sync_key+0x3c/0x52
[  701.175953]  [<c12a7bab>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  701.375989]  [<c122483b>] ? rtl8139_start_xmit+0x4a/0x13a
[  701.576026]  [<c12ae29e>] ? dev_hard_start_xmit+0x220/0x4cc
[  701.776062]  [<c12bfbed>] ? sch_direct_xmit+0xac/0x174
[  701.976096]  [<c12c3f69>] ? nf_iterate+0x69/0x7c
[  702.176131]  [<c12e8976>] ? ip_finish_output+0x0/0x2b6
[  702.376165]  [<c12b00eb>] ? dev_queue_xmit+0xc7/0x355
[  702.576198]  [<c12e8976>] ? ip_finish_output+0x0/0x2b6
[  702.776232]  [<c12e8a92>] ? ip_finish_output+0x11c/0x2b6
[  702.976266]  [<c12e8f11>] ? ip_output+0xa4/0xc3
[  703.176299]  [<c12e8976>] ? ip_finish_output+0x0/0x2b6
[  703.376332]  [<c12e4ff9>] ? ip_forward_finish+0x39/0x44
[  703.576365]  [<c12e3a38>] ? ip_rcv_finish+0xe8/0x39f
[  703.776398]  [<c12ad0fd>] ? __netif_receive_skb+0x237/0x2b3
[  703.976431]  [<c12ad70b>] ? netif_receive_skb+0x5f/0x64
[  704.176464]  [<c12ad75e>] ? napi_gro_complete+0x4e/0x94
[  704.376497]  [<c12ada9a>] ? dev_gro_receive+0x158/0x1f5
[  704.576530]  [<c12adc84>] ? napi_gro_receive+0x16/0x1f
[  704.776563]  [<c1217efb>] ? tg3_poll_work+0x5bc/0xbfb
[  704.976597]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  705.176631]  [<c121ce68>] ? tg3_poll+0x43/0x194
[  705.376665]  [<c12ad8b3>] ? net_rx_action+0xcc/0x15b
[  705.576699]  [<c1031cad>] ? __do_softirq+0x7f/0xfa
[  705.776733]  [<c1053dc9>] ? handle_IRQ_event+0x48/0xa6
[  705.976767]  [<c105689b>] ? move_native_irq+0x9/0x3e
[  706.176799]  [<c1031d4f>] ? do_softirq+0x27/0x2a
[  706.376832]  [<c1031e9d>] ? irq_exit+0x63/0x68
[  706.576864]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  706.776897]  [<c1031e6b>] ? irq_exit+0x31/0x68
[  706.976930]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  707.176963]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  707.276981] ---[ end trace 75e4f8534893c910 ]---
[  707.376998] 100: csum_start 306, offset 16, headroom 390, headlen 70, len 70
[  707.477015] nr_frags=0 gso_size=0
[  707.577031]
[ 1012.931455] ------------[ cut here ]------------
[ 1013.031482] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0xf2()
[ 1013.131501] Hardware name: PowerEdge SC440
[ 1013.231521] Pid: 2963, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00103-g2d6fa25 #1
[ 1013.331538] Call Trace:
[ 1013.531575]  [<c102d86c>] ? warn_slowpath_common+0x67/0x8c
[ 1013.731608]  [<c12abc76>] ? skb_csum_start_bug+0x46/0xf2
[ 1013.931641]  [<c12abc76>] ? skb_csum_start_bug+0x46/0xf2
[ 1014.131675]  [<c102d8ac>] ? warn_slowpath_null+0x1b/0x1f
[ 1014.331708]  [<c12abc76>] ? skb_csum_start_bug+0x46/0xf2
[ 1014.531742]  [<c1024569>] ? __wake_up_sync_key+0x3c/0x52
[ 1014.731775]  [<c12a7bab>] ? skb_copy_and_csum_dev+0x2a/0xaf
[ 1014.931809]  [<c122483b>] ? rtl8139_start_xmit+0x4a/0x13a
[ 1015.131841]  [<c12ae29e>] ? dev_hard_start_xmit+0x220/0x4cc
[ 1015.331875]  [<c12bfbed>] ? sch_direct_xmit+0xac/0x174
[ 1015.531908]  [<c12c3f69>] ? nf_iterate+0x69/0x7c
[ 1015.731941]  [<c12e8976>] ? ip_finish_output+0x0/0x2b6
[ 1015.931973]  [<c12b00eb>] ? dev_queue_xmit+0xc7/0x355
[ 1016.132007]  [<c12e8976>] ? ip_finish_output+0x0/0x2b6
[ 1016.332039]  [<c12e8a92>] ? ip_finish_output+0x11c/0x2b6
[ 1016.532071]  [<c12e8f11>] ? ip_output+0xa4/0xc3
[ 1016.732103]  [<c12e8976>] ? ip_finish_output+0x0/0x2b6
[ 1016.932135]  [<c12e4ff9>] ? ip_forward_finish+0x39/0x44
[ 1017.132166]  [<c12e3a38>] ? ip_rcv_finish+0xe8/0x39f
[ 1017.332198]  [<c12ad0fd>] ? __netif_receive_skb+0x237/0x2b3
[ 1017.532230]  [<c12ad70b>] ? netif_receive_skb+0x5f/0x64
[ 1017.732262]  [<c12ad75e>] ? napi_gro_complete+0x4e/0x94
[ 1017.932294]  [<c12ada9a>] ? dev_gro_receive+0x158/0x1f5
[ 1018.132326]  [<c12adc84>] ? napi_gro_receive+0x16/0x1f
[ 1018.332358]  [<c1217efb>] ? tg3_poll_work+0x5bc/0xbfb
[ 1018.532392]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[ 1018.732424]  [<c121ce68>] ? tg3_poll+0x43/0x194
[ 1018.932456]  [<c12ad8b3>] ? net_rx_action+0xcc/0x15b
[ 1019.132489]  [<c1031cad>] ? __do_softirq+0x7f/0xfa
[ 1019.332522]  [<c1053dc9>] ? handle_IRQ_event+0x48/0xa6
[ 1019.532554]  [<c105689b>] ? move_native_irq+0x9/0x3e
[ 1019.732586]  [<c1031d4f>] ? do_softirq+0x27/0x2a
[ 1019.932617]  [<c1031e9d>] ? irq_exit+0x63/0x68
[ 1020.132648]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[ 1020.332680]  [<c1031e6b>] ? irq_exit+0x31/0x68
[ 1020.532713]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[ 1020.732745]  [<c1002d29>] ? common_interrupt+0x29/0x30
[ 1020.932777]  [<c1390000>] ? quirk_io_region+0x1c/0x91
[ 1021.032794] ---[ end trace 75e4f8534893c911 ]---
[ 1021.132812] 100: csum_start 306, offset 16, headroom 390, headlen 153, len 153
[ 1021.232828] nr_frags=0 gso_size=0
[ 1021.332844]

Now what?

Thanks a lot, Eric and Jarek!

Plamen

_
___
_____
------------------------------------------
This message was sent by the mail server
at fs.ru.acad.bg using the web interface:
https://fs.ru.acad.bg/s/m/webmail
E-mail [email protected] with anything,
regarding the server itself

Jarek Poplawski

2010-08-24 18:22 UTC #32 lore raw

Quoted 1 line · Plamen Petrov

Eric Dumazet ????????????:

...

Quoted 2 lines

Now what?

Good question. IMHO it looks like skbs are overwritten, so better turn off gro until some next patch.

Quoted 2 lines

Thanks a lot, Eric and Jarek!

Thanks a lot, Plamen and Eric, too!

Jarek P.

Eric Dumazet

2010-08-24 19:20 UTC #33 lore raw

In reply to Plamen Petrov

Quoted 13 lines · Plamen Petrov

Above patch applied, and happy to report the machine now spits data in the logs instead of oopsing. Here is what we have now:

[  707.276981] ---[ end trace 75e4f8534893c910 ]---
[  707.376998] 100: csum_start 306, offset 16, headroom 390, headlen 70, 
len 70
[  707.477015] nr_frags=0 gso_size=0
[  707.577031]
[ 1021.032794] ---[ end trace 75e4f8534893c911 ]---
[ 1021.132812] 100: csum_start 306, offset 16, headroom 390, headlen 153, 
len 153
[ 1021.232828] nr_frags=0 gso_size=0
[ 1021.332844]

Thanks !

csum_offset = 16.

so its offsetof(struct tcphdr, check)

maybe a bug in net/ipv4/netfilter/nf_nat_helper.c ?

We should trace all spots where we set csum_start/csum_offset

Or/And trace the skb content.

Please add a :

print_hex_dump(KERN_ERR, "skb data:", DUMP_PREFIX_OFFSET,

               16, 1, skb->head, skb_end_pointer(skb)-skb->head,true);

call in skb_csum_start_bug(), right after the pr_err("\n") and before the "return 1;"

int skb_csum_start_bug(const struct sk_buff *skb, int pos) {

        if (skb->ip_summed == CHECKSUM_PARTIAL) {
                long csstart;

                csstart = skb->csum_start - skb_headroom(skb);
                if (WARN_ON(csstart > skb_headlen(skb))) {
                        int i;

                        pr_err("%d: csum_start %u, offset %u, headroom %d, headlen %d, len %d\n",
                                   pos, skb->csum_start, skb->csum_offset, skb_headroom(skb),
                                   skb_headlen(skb), skb->len);
                        pr_err("nr_frags=%u gso_size=%u ",
                                        skb_shinfo(skb)->nr_frags,
                                        skb_shinfo(skb)->gso_size);
                        for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
                                pr_err("frag_size=%u ", skb_shinfo(skb)->frags[i].size);
                        }
                        pr_err("\n");
                        print_hex_dump(KERN_ERR, "skb data:", DUMP_PREFIX_OFFSET,
                                16, 1, skb->head, skb_end_pointer(skb) - skb->head, true);
                        return 1;
                }
        }
        return 0;
}

Plamen Petrov

2010-08-25 07:05 UTC #34 lore raw

На 24.8.2010 г. 22:19, Eric Dumazet написа:

Quoted 23 lines · Plamen Petrov

Quoted 13 lines
Above patch applied, and happy to report the machine now spits data in the logs instead of oopsing. Here is what we have now:
[  707.276981] ---[ end trace 75e4f8534893c910 ]---
[  707.376998] 100: csum_start 306, offset 16, headroom 390, headlen 70,
len 70
[  707.477015] nr_frags=0 gso_size=0
[  707.577031]
[ 1021.032794] ---[ end trace 75e4f8534893c911 ]---
[ 1021.132812] 100: csum_start 306, offset 16, headroom 390, headlen 153,
len 153
[ 1021.232828] nr_frags=0 gso_size=0
[ 1021.332844]
Thanks !
csum_offset = 16.
so its offsetof(struct tcphdr, check)
maybe a bug in net/ipv4/netfilter/nf_nat_helper.c ?

Quoted 9 lines

We should trace all spots where we set csum_start/csum_offset
Or/And trace the skb content.
Please add a :
print_hex_dump(KERN_ERR, "skb data:", DUMP_PREFIX_OFFSET,
                16, 1, skb->head, skb_end_pointer(skb)-skb->head,true);

Done! See the results below.

Quoted 24 lines

call in skb_csum_start_bug(), right after the pr_err("\n") and before the "return 1;"

int skb_csum_start_bug(const struct sk_buff *skb, int pos) {

         if (skb->ip_summed == CHECKSUM_PARTIAL) {
                 long csstart;

                 csstart = skb->csum_start - skb_headroom(skb);
                 if (WARN_ON(csstart>  skb_headlen(skb))) {
                         int i;

                         pr_err("%d: csum_start %u, offset %u, headroom %d, headlen %d, len %d\n",
                                    pos, skb->csum_start, skb->csum_offset, skb_headroom(skb),
                                    skb_headlen(skb), skb->len);
                         pr_err("nr_frags=%u gso_size=%u ",
                                         skb_shinfo(skb)->nr_frags,
                                         skb_shinfo(skb)->gso_size);
                         for (i = 0; i<  skb_shinfo(skb)->nr_frags; i++) {
                                 pr_err("frag_size=%u ", skb_shinfo(skb)->frags[i].size);
                         }

Quoted 10 lines

                         pr_err("\n");
                         print_hex_dump(KERN_ERR, "skb data:", DUMP_PREFIX_OFFSET,
                                 16, 1, skb->head, skb_end_pointer(skb) - skb->head, true);
                         return 1;
                 }
         }
         return 0;
}

I see you liked the previous one, here's some more. ;)

This one is based on Linus' latest tree,
hence the ID "2.6.36-rc2-FS-00210-geedff42".

Quoted 90 lines

[   10.510191] XFS mounting filesystem md12
[   10.693540] Ending clean XFS mount for filesystem: md12
[   11.592737] IPv4 FIB: Using LC-trie version 0.409
[   11.592827] eth2: link up, 100Mbps, full-duplex, lpa 0x45E1
[   11.677311] eth0: link up, 100Mbps, full-duplex, lpa 0x41E1
[   11.687604] tg3 0000:04:00.0: irq 44 for MSI/MSI-X
[   11.719166] ADDRCONF(NETDEV_UP): eth1: link is not ready
[   11.845858] sixxs_t: Disabled Privacy Extensions
[   14.815688] tg3 0000:04:00.0: eth1: Link is up at 1000 Mbps, full duplex
[   14.815693] tg3 0000:04:00.0: eth1: Flow control is on for TX and on for RX
[   14.815740] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   15.470040] tun0: Disabled Privacy Extensions
[  310.470021] ------------[ cut here ]------------
[  310.570041] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  310.670050] Hardware name: PowerEdge SC440
[  310.770060] Pid: 2960, comm: FahCore_78.exe Not tainted 2.6.36-rc2-FS-00210-geedff42 #1
[  310.870069] Call Trace:
[  311.070087]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  311.270103]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  311.470126]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  311.670144]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  311.870167]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  312.070191]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  312.270207]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  312.470224]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  312.670238]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  312.870253]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  313.070268]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  313.270283]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  313.470297]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  313.670312]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  313.870326]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  314.070341]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  314.270355]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  314.470370]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  314.670385]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  314.870399]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  315.070413]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  315.270427]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  315.470440]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  315.670454]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  315.870468]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  316.070483]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  316.270498]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  316.470512]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  316.670526]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  316.870541]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  317.070555]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  317.270569]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  317.470582]  [<c1031ead>] ? irq_exit+0x63/0x68
[  317.670596]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  317.870610]  [<c10035c3>] ? do_device_not_available+0x0/0x49
[  318.070624]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  318.270639]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  318.370647] ---[ end trace df8deff2ad2a9760 ]---
[  318.470656] 100: csum_start 306, offset 16, headroom 390, headlen 151, len 151
[  318.570664] nr_frags=0 gso_size=0
[  318.670671]
[  318.770680] skb data:00000000: 00 a4 27 cc 17 5e ef ec 00 1a a0 38 8a 1b 08 00  ..'..^.....8....
[  318.870688] skb data:00000010: 45 00 00 b7 00 00 40 00 40 11 a4 62 c0 a8 0a 01  E.....@[email protected]....
[  318.970697] skb data:00000020: c0 a8 0a 82 00 35 f2 69 00 a3 96 88 d3 4e 81 80  .....5.i.....N..
[  319.070706] skb data:00000030: 00 01 00 04 00 00 00 00 06 61 6b 61 6d 61 69 0d  .........akamai.
[  319.170714] skb data:00000040: 73 6d 61 72 74 61 64 73 65 72 76 65 72 03 63 6f  smartadserver.co
[  319.270723] skb data:00000050: 6d 00 00 01 00 01 c0 0c 00 05 00 01 00 00 81 0a  m...............
[  319.370731] skb data:00000060: 00 28 06 61 6b 61 6d 61 69 0d 73 6d 61 72 74 61  .(.akamai.smarta
[  319.470739] skb data:00000070: 64 73 65 72 76 65 72 03 63 6f 6d 09 65 64 67 65  dserver.com.edge
[  319.570747] skb data:00000080: 73 75 69 74 65 03 6e 65 74 00 c0 36 00 00 00 00  suite.net..6....
[  319.670756] skb data:00000090: 00 00 00 00 00 00 00 00 00 00 08 00 45 00 00 34  ............E..4
[  319.770764] skb data:000000a0: 0d 39 40 00 40 06 2f 89 7f 00 00 01 7f 00 00 01  .9@.@./.........
[  319.870772] skb data:000000b0: be 75 19 4e 14 43 0f 38 14 5d 49 65 00 00 00 00  .u.N.C.8.]Ie....
[  319.970780] skb data:000000c0: 00 00 00 00 00 00 00 00 00 00 08 00 45 00 00 57  ............E..W
[  320.070789] skb data:000000d0: 4b 40 40 00 40 06 f1 5e 7f 00 00 01 7f 00 00 01  K@@.@..^........
[  320.170797] skb data:000000e0: 19 4e be 75 14 5d 49 65 14 43 0f 38 80 18 04 00  .N.u.]Ie.C.8....
[  320.270805] skb data:000000f0: fe 4b 00 00 01 01 08 0a 00 00 02 03 00 00 02 03  .K..............
[  320.370813] skb data:00000100: 00 a8 27 cc 00 00 00 00 00 00 00 00 00 00 00 00  ..'.............
[  320.470821] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.570829] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.670837] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.770845] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.870853] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.970862] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  321.070870] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  321.170878] skb data:00000180: 00 0a e6 ac 07 db 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  321.270887] skb data:00000190: 27 ef 08 00 45 00 00 89 09 27 40 00 7f 06 46 74  '...E....'@...Ft
[  321.370895] skb data:000001a0: c0 a8 01 02 5b 67 8e c2 04 8f 00 50 98 49 d7 bc  ....[g.....P.I..
[  321.470903] skb data:000001b0: ff 40 e9 4a 50 18 ff ff ac 4f 00 00 33 42 25 32  [email protected]%2
[  321.570911] skb data:000001c0: 34 73 68 25 33 44 33 25 33 42 25 32 34 73 77 25  4sh%3D3%3B%24sw%
[  321.670919] skb data:000001d0: 33 44 33 3b 20 70 69 64 3d 35 30 32 31 37 34 33  3D3; pid=5021743
[  321.770927] skb data:000001e0: 34 32 30 33 31 30 32 39 39 37 38 33 0d 0a 43 6f  420310299783..Co
[  321.870936] skb data:000001f0: 6f 6b 69 65 32 3a 20 24 56 65 72 73 69 6f 6e 3d  okie2: $Version=

Quoted 8 lines

[  321.970944] skb data:00000200: 31 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 4b  1..Connection: K
[  322.070952] skb data:00000210: 65 65 70 2d 41 6c 69 76 65 0d 0a 0d 0a 00 00 00  eep-Alive.......
[  322.170960] skb data:00000220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.270968] skb data:00000230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.370977] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.470985] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.570993] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.671004] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

The rest is in the attached file, in case you need to run it trough some debugging app...

Thanks,
Plamen

[   10.510191] XFS mounting filesystem md12
[   10.693540] Ending clean XFS mount for filesystem: md12
[   11.592737] IPv4 FIB: Using LC-trie version 0.409
[   11.592827] eth2: link up, 100Mbps, full-duplex, lpa 0x45E1
[   11.677311] eth0: link up, 100Mbps, full-duplex, lpa 0x41E1
[   11.687604] tg3 0000:04:00.0: irq 44 for MSI/MSI-X
[   11.719166] ADDRCONF(NETDEV_UP): eth1: link is not ready
[   11.845858] sixxs_t: Disabled Privacy Extensions
[   14.815688] tg3 0000:04:00.0: eth1: Link is up at 1000 Mbps, full duplex
[   14.815693] tg3 0000:04:00.0: eth1: Flow control is on for TX and on for RX
[   14.815740] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   15.470040] tun0: Disabled Privacy Extensions
[  310.470021] ------------[ cut here ]------------
[  310.570041] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  310.670050] Hardware name: PowerEdge SC440
[  310.770060] Pid: 2960, comm: FahCore_78.exe Not tainted 2.6.36-rc2-FS-00210-geedff42 #1
[  310.870069] Call Trace:
[  311.070087]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  311.270103]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  311.470126]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  311.670144]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  311.870167]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  312.070191]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  312.270207]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  312.470224]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  312.670238]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  312.870253]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  313.070268]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  313.270283]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  313.470297]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  313.670312]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  313.870326]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  314.070341]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  314.270355]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  314.470370]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  314.670385]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  314.870399]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  315.070413]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  315.270427]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  315.470440]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  315.670454]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  315.870468]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  316.070483]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  316.270498]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  316.470512]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  316.670526]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  316.870541]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  317.070555]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  317.270569]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  317.470582]  [<c1031ead>] ? irq_exit+0x63/0x68
[  317.670596]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  317.870610]  [<c10035c3>] ? do_device_not_available+0x0/0x49
[  318.070624]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  318.270639]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  318.370647] ---[ end trace df8deff2ad2a9760 ]---
[  318.470656] 100: csum_start 306, offset 16, headroom 390, headlen 151, len 151
[  318.570664] nr_frags=0 gso_size=0
[  318.670671]
[  318.770680] skb data:00000000: 00 a4 27 cc 17 5e ef ec 00 1a a0 38 8a 1b 08 00  ..'..^.....8....
[  318.870688] skb data:00000010: 45 00 00 b7 00 00 40 00 40 11 a4 62 c0 a8 0a 01  E.....@[email protected]....
[  318.970697] skb data:00000020: c0 a8 0a 82 00 35 f2 69 00 a3 96 88 d3 4e 81 80  .....5.i.....N..
[  319.070706] skb data:00000030: 00 01 00 04 00 00 00 00 06 61 6b 61 6d 61 69 0d  .........akamai.
[  319.170714] skb data:00000040: 73 6d 61 72 74 61 64 73 65 72 76 65 72 03 63 6f  smartadserver.co
[  319.270723] skb data:00000050: 6d 00 00 01 00 01 c0 0c 00 05 00 01 00 00 81 0a  m...............
[  319.370731] skb data:00000060: 00 28 06 61 6b 61 6d 61 69 0d 73 6d 61 72 74 61  .(.akamai.smarta
[  319.470739] skb data:00000070: 64 73 65 72 76 65 72 03 63 6f 6d 09 65 64 67 65  dserver.com.edge
[  319.570747] skb data:00000080: 73 75 69 74 65 03 6e 65 74 00 c0 36 00 00 00 00  suite.net..6....
[  319.670756] skb data:00000090: 00 00 00 00 00 00 00 00 00 00 08 00 45 00 00 34  ............E..4
[  319.770764] skb data:000000a0: 0d 39 40 00 40 06 2f 89 7f 00 00 01 7f 00 00 01  .9@.@./.........
[  319.870772] skb data:000000b0: be 75 19 4e 14 43 0f 38 14 5d 49 65 00 00 00 00  .u.N.C.8.]Ie....
[  319.970780] skb data:000000c0: 00 00 00 00 00 00 00 00 00 00 08 00 45 00 00 57  ............E..W
[  320.070789] skb data:000000d0: 4b 40 40 00 40 06 f1 5e 7f 00 00 01 7f 00 00 01  K@@.@..^........
[  320.170797] skb data:000000e0: 19 4e be 75 14 5d 49 65 14 43 0f 38 80 18 04 00  .N.u.]Ie.C.8....
[  320.270805] skb data:000000f0: fe 4b 00 00 01 01 08 0a 00 00 02 03 00 00 02 03  .K..............
[  320.370813] skb data:00000100: 00 a8 27 cc 00 00 00 00 00 00 00 00 00 00 00 00  ..'.............
[  320.470821] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.570829] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.670837] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.770845] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.870853] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.970862] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  321.070870] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  321.170878] skb data:00000180: 00 0a e6 ac 07 db 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  321.270887] skb data:00000190: 27 ef 08 00 45 00 00 89 09 27 40 00 7f 06 46 74  '...E....'@...Ft
[  321.370895] skb data:000001a0: c0 a8 01 02 5b 67 8e c2 04 8f 00 50 98 49 d7 bc  ....[g.....P.I..
[  321.470903] skb data:000001b0: ff 40 e9 4a 50 18 ff ff ac 4f 00 00 33 42 25 32  [email protected]%2
[  321.570911] skb data:000001c0: 34 73 68 25 33 44 33 25 33 42 25 32 34 73 77 25  4sh%3D3%3B%24sw%
[  321.670919] skb data:000001d0: 33 44 33 3b 20 70 69 64 3d 35 30 32 31 37 34 33  3D3; pid=5021743
[  321.770927] skb data:000001e0: 34 32 30 33 31 30 32 39 39 37 38 33 0d 0a 43 6f  420310299783..Co
[  321.870936] skb data:000001f0: 6f 6b 69 65 32 3a 20 24 56 65 72 73 69 6f 6e 3d  okie2: $Version=
[  321.970944] skb data:00000200: 31 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 4b  1..Connection: K
[  322.070952] skb data:00000210: 65 65 70 2d 41 6c 69 76 65 0d 0a 0d 0a 00 00 00  eep-Alive.......
[  322.170960] skb data:00000220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.270968] skb data:00000230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.370977] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.470985] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.570993] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.671004] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  324.960350] ------------[ cut here ]------------
[  325.060369] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  325.160378] Hardware name: PowerEdge SC440
[  325.260392] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  325.360405] Call Trace:
[  325.560423]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  325.760438]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  325.960453]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  326.160469]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  326.360485]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  326.560502]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  326.760518]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  326.960535]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  327.160550]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  327.360566]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  327.560581]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  327.760598]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  327.960614]  [<c12b5601>] ? neigh_resolve_output+0xd0/0x280
[  328.160630]  [<c12be0a5>] ? eth_header+0x0/0x9b
[  328.360645]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  328.560660]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  328.760675]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  328.960691]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  329.160706]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  329.360721]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  329.560735]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  329.760749]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  329.960763]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  330.160777]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  330.360791]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  330.560806]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  330.760820]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  330.960834]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  331.160849]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  331.360864]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  331.560878]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  331.760893]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  331.960907]  [<c1031ead>] ? irq_exit+0x63/0x68
[  332.160921]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  332.360934]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  332.560949]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  332.760964]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  332.960979]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  333.060987] ---[ end trace df8deff2ad2a9761 ]---
[  333.160995] 100: csum_start 306, offset 16, headroom 390, headlen 158, len 158
[  333.261006] nr_frags=0 gso_size=0
[  333.361013]
[  333.461022] skb data:00000000: 00 9c 27 cc 00 00 00 00 00 00 00 00 00 00 00 00  ..'.............
[  333.561030] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  333.661038] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  333.761046] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  333.861055] skb data:00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  333.961063] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  334.061071] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  334.161079] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  334.261087] skb data:00000080: 00 04 61 98 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ..a......\'....\
[  334.361095] skb data:00000090: 27 ef 08 00 45 00 00 28 15 28 40 00 7f 06 7d a2  '...E..(.(@...}.
[  334.461103] skb data:000000a0: c0 a8 01 02 4f 63 57 f8 04 2e 00 50 94 c2 2d a9  ....OcW....P..-.
[  334.561111] skb data:000000b0: d6 2f bb 73 50 10 ff ff ee 41 00 00 00 00 00 00  ./.sP....A......
[  334.661120] skb data:000000c0: 00 00 20 4f 4b 0d 0a 53 65 72 76 65 72 3a 20 6e  .. OK..Server: n
[  334.761128] skb data:000000d0: 67 69 6e 78 2f 30 2e 36 2e 33 39 0d 0a 44 61 74  ginx/0.6.39..Dat
[  334.861136] skb data:000000e0: 65 3a 20 57 65 64 2c 20 32 35 20 41 75 67 20 32  e: Wed, 25 Aug 2
[  334.961144] skb data:000000f0: 30 31 30 20 30 36 3a 35 32 3a 34 35 20 47 4d 54  010 06:52:45 GMT
[  335.061152] skb data:00000100: 00 b0 27 cc 00 00 00 00 00 00 00 00 00 00 00 00  ..'.............
[  335.161160] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.261168] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.361176] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.461184] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.561192] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.661200] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.761208] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.861216] skb data:00000180: 00 04 61 98 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ..a......\'....\
[  335.961224] skb data:00000190: 27 ef 08 00 45 00 00 90 1a fa 40 00 7f 06 34 db  '[email protected].
[  336.061233] skb data:000001a0: c0 a8 01 02 5b 67 8e 81 04 f7 00 50 2e 1a b7 d1  ....[g.....P....
[  336.161241] skb data:000001b0: 09 f2 34 df 50 18 ff ff ac 15 00 00 25 33 44 39  ..4.P.......%3D9
[  336.261249] skb data:000001c0: 25 35 46 30 25 35 46 30 74 3b 20 41 53 50 53 45  %5F0%5F0t; ASPSE
[  336.361257] skb data:000001d0: 53 53 49 4f 4e 49 44 43 41 44 41 42 54 43 52 3d  SSIONIDCADABTCR=
[  336.461265] skb data:000001e0: 41 4e 4b 50 4b 41 42 43 49 4d 46 4e 42 42 41 45  ANKPKABCIMFNBBAE
[  336.561273] skb data:000001f0: 4e 42 49 45 49 43 4d 50 3b 20 42 49 47 69 70 53  NBIEICMP; BIGipS
[  336.661281] skb data:00000200: 65 72 76 65 72 53 4d 41 52 54 32 3d 38 38 39 36  erverSMART2=8896
[  336.761289] skb data:00000210: 35 31 39 37 38 2e 32 33 30 34 30 2e 30 30 30 30  51978.23040.0000
[  336.861297] skb data:00000220: 0d 0a 0d 0a 00 00 00 00 10 0c c6 a0 15 ed 28 8d  ..............(.
[  336.961306] skb data:00000230: 9c 30 02 94 60 09 ef 98 89 9b 28 8d c9 a0 0c ff  .0..`.....(.....
[  337.061314] skb data:00000240: f8 90 00 f9 8f c7 a0 0c 0e 19 09 92 30 08 84 60  ............0..`
[  337.161322] skb data:00000250: 08 78 d2 7c 8c 60 91 1e f9 91 20 19 92 22 19 09  .x.|.`.... .."..
[  337.261330] skb data:00000260: 81 00 00 3b 00 00 00 00 00 00 00 00 00 00 00 00  ...;............
[  337.361338] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  352.042649] ------------[ cut here ]------------
[  352.142668] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  352.242677] Hardware name: PowerEdge SC440
[  352.342688] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  352.442697] Call Trace:
[  352.642715]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  352.842731]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  353.042746]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  353.242761]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  353.442776]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  353.642792]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  353.842809]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  354.042825]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  354.242840]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  354.442856]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  354.642871]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  354.842887]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  355.042902]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  355.242917]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  355.442932]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  355.642957]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  355.842980]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  356.043009]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  356.243024]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  356.443039]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  356.643053]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  356.843067]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  357.043081]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  357.243095]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  357.443110]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  357.643126]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  357.843139]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  358.043153]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  358.243167]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  358.443183]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  358.643197]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  358.843210]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  359.043224]  [<c1031ead>] ? irq_exit+0x63/0x68
[  359.243238]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  359.443251]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  359.643265]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  359.843279]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  360.043294]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  360.143302] ---[ end trace df8deff2ad2a9762 ]---
[  360.243310] 100: csum_start 306, offset 16, headroom 390, headlen 143, len 143
[  360.343318] nr_frags=0 gso_size=0
[  360.443324]
[  360.543334] skb data:00000000: 00 bc 2e cc 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  360.643343] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  360.743351] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  360.843360] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  360.943368] skb data:00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  361.043377] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  361.143385] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  361.243393] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  361.343402] skb data:00000080: 00 19 66 2f 00 00 00 1a a0 38 8a 1b 00 19 66 2f  ..f/.....8....f/
[  361.443410] skb data:00000090: f9 3e 08 00 45 00 00 dc ac 84 40 00 80 06 a3 5d  .>..E.....@....]
[  361.543419] skb data:000000a0: c0 a8 0a 9f 5e f6 7f fc 07 15 01 bb 0a 27 94 09  ....^........'..
[  361.643427] skb data:000000b0: 86 26 d0 7a 50 18 ff ff 01 08 00 00 16 03 01 00  .&.zP...........
[  361.743436] skb data:000000c0: af 01 00 00 ab 03 01 4c 74 bd d6 04 bf 0c 49 6f  .......Lt.....Io
[  361.843444] skb data:000000d0: 2f 6e b3 4c a6 56 dc 7b 56 76 ba 7c 2d 5e b8 a0  /n.L.V.{Vv.|-^..
[  361.943453] skb data:000000e0: d8 c5 13 f3 47 ba d5 20 43 87 a4 82 fa cb 00 e1  ....G.. C.......
[  362.043461] skb data:000000f0: a0 fa aa ff 3d 0b 1d bc 13 e2 75 b8 a7 81 d0 bc  ....=.....u.....
[  362.143470] skb data:00000100: 00 b4 2e cc 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.243478] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.343486] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.443495] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.543503] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.643512] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.743520] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.843528] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.943537] skb data:00000180: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...\'....\'....\
[  363.043545] skb data:00000190: 27 ef 08 00 45 00 00 81 21 9a 40 00 7f 06 2e 4a  '[email protected]
[  363.143554] skb data:000001a0: c0 a8 01 02 5b 67 8e 81 04 f7 00 50 2e 1a ba 51  ....[g.....P...Q
[  363.243562] skb data:000001b0: 09 f2 3c 0c 50 18 fe 86 ac 06 00 00 41 53 50 53  ..<.P.......ASPS
[  363.343571] skb data:000001c0: 45 53 53 49 4f 4e 49 44 43 41 44 41 42 54 43 52  ESSIONIDCADABTCR
[  363.443580] skb data:000001d0: 3d 41 4e 4b 50 4b 41 42 43 49 4d 46 4e 42 42 41  =ANKPKABCIMFNBBA
[  363.543588] skb data:000001e0: 45 4e 42 49 45 49 43 4d 50 3b 20 42 49 47 69 70  ENBIEICMP; BIGip
[  363.643596] skb data:000001f0: 53 65 72 76 65 72 53 4d 41 52 54 32 3d 38 38 39  ServerSMART2=889
[  363.743605] skb data:00000200: 36 35 31 39 37 38 2e 32 33 30 34 30 2e 30 30 30  651978.23040.000
[  363.843613] skb data:00000210: 30 0d 0a 0d 0a 00 00 00 00 00 00 00 00 00 00 00  0...............
[  363.943622] skb data:00000220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  364.043630] skb data:00000230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  364.143638] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  364.243647] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  364.343655] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  364.443663] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  364.680629] ------------[ cut here ]------------
[  364.780652] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  364.880664] Hardware name: PowerEdge SC440
[  364.980679] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  365.080691] Call Trace:
[  365.280717]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  365.480741]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  365.680756]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  365.880772]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  366.080788]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  366.280804]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  366.480820]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  366.680836]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  366.880851]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  367.080868]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  367.280883]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  367.480899]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  367.680914]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  367.880928]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  368.080943]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  368.280958]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  368.480972]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  368.680987]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  368.881002]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  369.081017]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  369.281031]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  369.481046]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  369.681061]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  369.881076]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  370.081092]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  370.281108]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  370.481123]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  370.681138]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  370.881153]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  371.081168]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  371.281183]  [<c1031cd4>] ? __do_softirq+0x96/0xfa
[  371.481196]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  371.681210]  [<c1031ead>] ? irq_exit+0x63/0x68
[  371.881224]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  372.081236]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  372.281250]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  372.481265]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  372.681280]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  372.781288] ---[ end trace df8deff2ad2a9763 ]---
[  372.881297] 100: csum_start 306, offset 16, headroom 390, headlen 135, len 135
[  372.981304] nr_frags=0 gso_size=0
[  373.081311]
[  373.181321] skb data:00000000: 00 38 35 cc 02 00 08 00 41 00 31 32 34 38 33 34  .85.....A.124834
[  373.281330] skb data:00000010: 31 35 35 31 2e 4d 33 35 35 34 32 50 31 37 31 31  1551.M35542P1711
[  373.381339] skb data:00000020: 32 56 30 30 30 30 30 30 30 30 30 30 30 30 30 39  2V00000000000009
[  373.481347] skb data:00000030: 30 36 49 30 32 38 30 36 32 32 42 5f 31 2e 66 73  06I0280622B_1.fs
[  373.581355] skb data:00000040: 2c 53 3d 32 39 38 32 3a 32 2c 53 00 00 00 00 00  ,S=2982:2,S.....
[  373.681364] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  373.781372] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  373.881380] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  373.981389] skb data:00000080: 00 0e 2e 5c 27 e6 00 30 05 07 f3 1a 08 00 45 00  ...\'..0......E.
[  374.081397] skb data:00000090: 01 48 4c bc 00 00 80 11 dd 86 c0 a8 c7 0f c0 a8  .HL.............
[  374.181406] skb data:000000a0: c7 01 00 44 00 43 01 34 e5 4a 01 01 06 00 8e 0e  ...D.C.4.J......
[  374.281415] skb data:000000b0: c5 2f 36 6d 00 00 c0 a8 c7 0f 00 00 00 00 00 00  ./6m............
[  374.381423] skb data:000000c0: 00 00 00 00 00 00 00 30 05 07 f3 1a 00 00 00 00  .......0........
[  374.481432] skb data:000000d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  374.581440] skb data:000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  374.681448] skb data:000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  374.781457] skb data:00000100: 00 30 35 cc 80 35 54 c1 00 00 00 00 00 00 00 00  .05..5T.........
[  374.881465] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 c0 31 7a 03 c1  ............1z..
[  374.981474] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  375.081482] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 24 54 81 b7  ............$T..
[  375.181491] skb data:00000140: 00 00 00 00 00 00 00 00 9d 6e ac 57 2c 53 00 00  .........n.W,S..
[  375.281499] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  375.381508] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  375.481516] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  375.581524] skb data:00000180: 00 16 17 5e ef ec 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...^.....\'....\
[  375.681534] skb data:00000190: 27 ef 08 00 45 00 00 79 22 02 40 00 7f 06 2d ea  '...E..y"[email protected].
[  375.781543] skb data:000001a0: c0 a8 01 02 5b 67 8e 81 04 f7 00 50 2e 1a bc c2  ....[g.....P....
[  375.881552] skb data:000001b0: 09 f2 3e f6 50 18 ff ff ab fe 00 00 4f 4e 49 44  ..>.P.......ONID
[  375.981561] skb data:000001c0: 43 41 44 41 42 54 43 52 3d 41 4e 4b 50 4b 41 42  CADABTCR=ANKPKAB
[  376.081569] skb data:000001d0: 43 49 4d 46 4e 42 42 41 45 4e 42 49 45 49 43 4d  CIMFNBBAENBIEICM
[  376.181578] skb data:000001e0: 50 3b 20 42 49 47 69 70 53 65 72 76 65 72 53 4d  P; BIGipServerSM
[  376.281586] skb data:000001f0: 41 52 54 32 3d 38 38 39 36 35 31 39 37 38 2e 32  ART2=889651978.2
[  376.381594] skb data:00000200: 33 30 34 30 2e 30 30 30 30 0d 0a 0d 0a 00 00 00  3040.0000.......
[  376.481603] skb data:00000210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  376.581611] skb data:00000220: 00 00 00 00 00 00 00 00 30 30 30 30 30 30 30 39  ........00000009
[  376.681620] skb data:00000230: 30 36 49 30 32 38 30 36 45 36 35 5f 31 2e 66 73  06I02806E65_1.fs
[  376.781628] skb data:00000240: 2c 53 3d 31 35 37 38 3a 32 2c 53 00 00 00 00 00  ,S=1578:2,S.....
[  376.881636] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  376.981645] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  377.081653] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  454.636420] ------------[ cut here ]------------
[  454.736438] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  454.836447] Hardware name: PowerEdge SC440
[  454.936458] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  455.036466] Call Trace:
[  455.236484]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  455.436499]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  455.636513]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  455.836528]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  456.036543]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  456.236559]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  456.436574]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  456.636589]  [<c1224852>] ? rtl8139_start_xmit+0x51/0x13a
[  456.836603]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  457.036617]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  457.236632]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  457.436650]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  457.636673]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  457.836696]  [<c12b5601>] ? neigh_resolve_output+0xd0/0x280
[  458.036718]  [<c12be0a5>] ? eth_header+0x0/0x9b
[  458.236740]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  458.436762]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  458.636783]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  458.836805]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  459.036827]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  459.236848]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  459.436870]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  459.636891]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  459.836912]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  460.036934]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  460.236956]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  460.436979]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  460.637004]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  460.837026]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  461.037048]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  461.237071]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  461.437092]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  461.637114]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  461.837129]  [<c1031ead>] ? irq_exit+0x63/0x68
[  462.037143]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  462.237156]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  462.437171]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  462.637184]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  462.837199]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  462.937207] ---[ end trace df8deff2ad2a9764 ]---
[  463.037215] 100: csum_start 306, offset 16, headroom 390, headlen 151, len 151
[  463.137223] nr_frags=0 gso_size=0
[  463.237230]
[  463.337239] skb data:00000000: 00 00 00 00 39 c6 77 33 00 1a a0 38 8a 1b 08 00  ....9.w3...8....
[  463.437247] skb data:00000010: 45 00 00 68 00 00 40 00 ff 01 e6 3b c0 a8 0a 01  E..h..@....;....
[  463.537256] skb data:00000020: c0 a8 0a 07 08 00 ba 22 96 80 00 25 00 00 00 00  ......."...%....
[  463.637264] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  463.737272] skb data:00000040: 46 6c 6f 72 69 61 6e 20 46 6f 72 73 74 65 72 20  Florian Forster
[  463.837280] skb data:00000050: 3c 6f 63 74 6f 40 76 65 72 70 6c 61 6e 74 2e 6f  <[email protected]
[  463.937289] skb data:00000060: 72 67 3e 20 68 74 74 70 3a 2f 2f 76 65 72 70 6c  rg> http://verpl
[  464.037297] skb data:00000070: 61 6e 74 2e 6f 72 67 2f 00 00 00 00 00 00 00 00  ant.org/........
[  464.137305] skb data:00000080: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 08 00 45 00  ...\'....\'...E.
[  464.237313] skb data:00000090: 00 28 e7 f2 40 00 f5 06 1d 3a 5b cb 63 2d c0 a8  .(..@....:[.c-..
[  464.337321] skb data:000000a0: 01 02 00 50 04 4c 41 a1 8a 47 70 4f ca fc 50 10  ...P.LA..GpO..P.
[  464.437330] skb data:000000b0: 13 2e 10 33 00 00 00 00 00 00 00 00 54 54 50 2f  ...3........TTP/
[  464.537338] skb data:000000c0: 31 2e c7 4e 9d d1 da 14 6d 47 02 98 f4 c1 58 a3  1..N....mG....X.
[  464.637346] skb data:000000d0: 44 0b 73 14 a1 67 43 77 d3 95 06 31 bb 39 6c 78  D.s..gCw...1.9lx
[  464.737354] skb data:000000e0: 24 a8 c8 3c 8c d9 89 ba 5c e9 a3 1b 7f f8 e8 0c  $..<....\.......
[  464.837363] skb data:000000f0: f7 68 57 e4 1b 4d e8 3b 19 8b 88 8e ef 20 f9 47  .hW..M.;..... .G
[  464.937371] skb data:00000100: 00 34 16 cc 39 c6 77 33 00 1a a0 38 8a 1b 08 00  .4..9.w3...8....
[  465.037379] skb data:00000110: 45 00 00 68 00 00 40 00 ff 01 e6 3b c0 a8 0a 01  E..h..@....;....
[  465.137388] skb data:00000120: c0 a8 0a 07 08 00 ba 22 96 80 00 25 00 00 00 00  ......."...%....
[  465.237396] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  465.337404] skb data:00000140: 46 6c 6f 72 69 61 6e 20 46 6f 72 73 74 65 72 20  Florian Forster
[  465.437412] skb data:00000150: 3c 6f 63 74 6f 40 76 65 72 70 6c 61 6e 74 2e 6f  <[email protected]
[  465.537420] skb data:00000160: 72 67 3e 20 68 74 74 70 3a 2f 2f 76 65 72 70 6c  rg> http://verpl
[  465.637428] skb data:00000170: 61 6e 74 2e 6f 72 67 2f 00 00 00 00 00 00 00 00  ant.org/........
[  465.737436] skb data:00000180: 00 0a e6 ac 07 db 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  465.837444] skb data:00000190: 27 ef 08 00 45 00 00 89 04 28 40 00 7f 06 4b b4  '...E....(@...K.
[  465.937452] skb data:000001a0: c0 a8 01 02 5b 67 8e 81 04 4f 00 50 f6 d5 f5 07  ....[g...O.P....
[  466.037461] skb data:000001b0: 5b 82 93 48 50 18 ff ff ac 0e 00 00 33 42 25 32  [..HP.......3B%2
[  466.137469] skb data:000001c0: 34 73 68 25 33 44 33 25 33 42 25 32 34 73 77 25  4sh%3D3%3B%24sw%
[  466.237477] skb data:000001d0: 33 44 33 3b 20 70 69 64 3d 35 30 32 31 37 34 33  3D3; pid=5021743
[  466.337485] skb data:000001e0: 34 32 30 33 31 30 32 39 39 37 38 33 0d 0a 43 6f  420310299783..Co
[  466.437493] skb data:000001f0: 6f 6b 69 65 32 3a 20 24 56 65 72 73 69 6f 6e 3d  okie2: $Version=
[  466.537501] skb data:00000200: 31 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 4b  1..Connection: K
[  466.637509] skb data:00000210: 65 65 70 2d 41 6c 69 76 65 0d 0a 0d 0a ec 00 00  eep-Alive.......
[  466.737517] skb data:00000220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  466.837526] skb data:00000230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  466.937534] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  467.037542] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  467.137550] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  467.237558] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  470.600713] ------------[ cut here ]------------
[  470.700732] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  470.800741] Hardware name: PowerEdge SC440
[  470.900752] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  471.000761] Call Trace:
[  471.200779]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  471.400796]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  471.600810]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  471.800826]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  472.000841]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  472.200858]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  472.400874]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  472.600890]  [<c1224852>] ? rtl8139_start_xmit+0x51/0x13a
[  472.800905]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  473.000920]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  473.200936]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  473.400952]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  473.600969]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  473.800985]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  474.001000]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  474.201018]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  474.401032]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  474.601046]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  474.801060]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  475.001074]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  475.201088]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  475.401102]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  475.601115]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  475.801129]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  476.001142]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  476.201156]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  476.401171]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  476.601185]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  476.801199]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  477.001213]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  477.201229]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  477.401243]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  477.601257]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  477.801271]  [<c1031ead>] ? irq_exit+0x63/0x68
[  478.001284]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  478.201299]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  478.401314]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  478.601329]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  478.801344]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  478.901352] ---[ end trace df8deff2ad2a9765 ]---
[  479.001361] 100: csum_start 306, offset 16, headroom 390, headlen 168, len 168
[  479.101368] nr_frags=0 gso_size=0
[  479.201375]
[  479.301384] skb data:00000000: 00 7c de c8 e6 ac 07 db 00 1a a0 38 8a 1b 08 00  .|.........8....
[  479.401394] skb data:00000010: 45 00 00 6c 00 00 40 00 40 11 a4 9b c0 a8 0a 01  E..l..@.@.......
[  479.501403] skb data:00000020: c0 a8 0a 94 00 35 04 05 00 58 96 4f 6e d3 81 80  .....5...X.On...
[  479.601411] skb data:00000030: 00 01 00 02 00 00 00 00 04 6f 63 73 70 06 74 68  .........ocsp.th
[  479.701420] skb data:00000040: 61 77 74 65 03 63 6f 6d 00 00 01 00 01 c0 0c 00  awte.com........
[  479.801428] skb data:00000050: 05 00 01 00 00 03 67 00 13 04 6f 63 73 70 08 76  ......g...ocsp.v
[  479.901437] skb data:00000060: 65 72 69 73 69 67 6e 03 6e 65 74 00 c0 2d 00 01  erisign.net..-..
[  480.001445] skb data:00000070: 00 01 00 00 00 00 00 04 c7 07 36 48 61 30 00 00  ..........6Ha0..
[  480.101453] skb data:00000080: 00 0a e6 ac 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  480.201462] skb data:00000090: 27 ef 08 00 45 00 00 28 04 d0 00 00 7f 06 8b 6d  '...E..(.......m
[  480.301470] skb data:000000a0: c0 a8 01 02 5b 67 8e 81 04 4f 00 50 f6 d5 f5 69  ....[g...O.P...i
[  480.401479] skb data:000000b0: f6 d5 f5 69 50 04 00 00 27 2f 00 00 54 54 50 2f  ...iP...'/..TTP/
[  480.501487] skb data:000000c0: 31 2e 28 3d 33 d1 bc 47 66 61 86 93 94 7a 11 b5  1.(=3..Gfa...z..
[  480.601495] skb data:000000d0: 6b d8 32 97 e7 ce 6f 04 bb b9 d9 69 4a 5c 4b 4f  k.2...o....iJ\KO
[  480.701504] skb data:000000e0: a0 17 fa 64 c8 95 51 95 45 c8 dc 1d ef b0 4e 76  ...d..Q.E.....Nv
[  480.801512] skb data:000000f0: 22 6e 78 55 96 9d d4 78 73 2c 45 a3 00 d1 6e 22  "nxU...xs,E...n"
[  480.901521] skb data:00000100: 00 68 de c8 e6 ac 07 db 00 1a a0 38 8a 1b 08 00  .h.........8....
[  481.001529] skb data:00000110: 45 00 00 6c 00 00 40 00 40 11 a4 9b c0 a8 0a 01  E..l..@.@.......
[  481.101537] skb data:00000120: c0 a8 0a 94 00 35 04 05 00 58 96 4f 6e d3 81 80  .....5...X.On...
[  481.201546] skb data:00000130: 00 01 00 02 00 00 00 00 04 6f 63 73 70 06 74 68  .........ocsp.th
[  481.301554] skb data:00000140: 61 77 74 65 03 63 6f 6d 00 00 01 00 01 c0 0c 00  awte.com........
[  481.401562] skb data:00000150: 05 00 01 00 00 03 67 00 13 04 6f 63 73 70 08 76  ......g...ocsp.v
[  481.501571] skb data:00000160: 65 72 69 73 69 67 6e 03 6e 65 74 00 c0 2d 00 01  erisign.net..-..
[  481.601579] skb data:00000170: 00 01 00 00 00 00 00 04 c7 07 36 48 61 30 00 00  ..........6Ha0..
[  481.701588] skb data:00000180: 00 16 17 5e 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...^.....\'....\
[  481.801597] skb data:00000190: 27 ef 08 00 45 00 00 9a 04 dc 40 00 7f 06 4a ef  '[email protected].
[  481.901605] skb data:000001a0: c0 a8 01 02 5b 67 8e 81 04 5f 00 50 65 db a5 dc  ....[g..._.Pe...
[  482.001614] skb data:000001b0: 1c 11 e6 62 50 18 ff ff ac 1f 00 00 39 25 33 42  ...bP.......9%3B
[  482.101622] skb data:000001c0: 25 32 34 6f 25 33 44 31 31 30 35 31 25 33 42 25  %24o%3D11051%3B%
[  482.201630] skb data:000001d0: 32 34 73 68 25 33 44 33 25 33 42 25 32 34 73 77  24sh%3D3%3B%24sw
[  482.301639] skb data:000001e0: 25 33 44 33 3b 20 70 69 64 3d 35 30 32 31 37 34  %3D3; pid=502174
[  482.401647] skb data:000001f0: 33 34 32 30 33 31 30 32 39 39 37 38 33 0d 0a 43  3420310299783..C
[  482.501656] skb data:00000200: 6f 6f 6b 69 65 32 3a 20 24 56 65 72 73 69 6f 6e  ookie2: $Version
[  482.601665] skb data:00000210: 3d 31 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20  =1..Connection:
[  482.701673] skb data:00000220: 4b 65 65 70 2d 41 6c 69 76 65 0d 0a 0d 0a 00 00  Keep-Alive......
[  482.801681] skb data:00000230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  482.901689] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  483.001698] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  483.101706] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  483.201715] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  595.934671] ------------[ cut here ]------------
[  596.034690] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  596.134699] Hardware name: PowerEdge SC440
[  596.234711] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  596.334723] Call Trace:
[  596.534750]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  596.734773]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  596.934795]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  597.134811]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  597.334826]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  597.534842]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  597.734858]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  597.934873]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  598.134888]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  598.334904]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  598.534919]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  598.734934]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  598.934948]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  599.134962]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  599.334976]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  599.534990]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  599.735008]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  599.935022]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  600.135036]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  600.335049]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  600.535063]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  600.735076]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  600.935089]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  601.135103]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  601.335116]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  601.535131]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  601.735145]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  601.935159]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  602.135173]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  602.335188]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  602.535202]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  602.735215]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  602.935228]  [<c1031ead>] ? irq_exit+0x63/0x68
[  603.135242]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  603.335256]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  603.535272]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  603.735285]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  603.935300]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  604.035308] ---[ end trace df8deff2ad2a9766 ]---
[  604.135316] 100: csum_start 306, offset 16, headroom 390, headlen 98, len 98
[  604.235324] nr_frags=0 gso_size=0
[  604.335330]
[  604.435339] skb data:00000000: 00 28 37 cc 00 00 00 00 00 00 00 00 00 00 00 00  .(7.............
[  604.535348] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  604.635356] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  604.735364] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  604.835372] skb data:00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  604.935381] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  605.035389] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  605.135397] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  605.235406] skb data:00000080: 00 15 58 8f 00 00 00 1a a0 38 8a 1b 00 50 56 00  ..X......8...PV.
[  605.335414] skb data:00000090: 00 01 08 00 45 00 00 68 b2 9c 00 00 40 01 32 a4  [email protected].
[  605.435422] skb data:000000a0: c0 a8 0a 03 c0 a8 0a 01 00 00 50 56 08 3d 00 35  ..........PV.=.5
[  605.535430] skb data:000000b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  605.635439] skb data:000000c0: 00 00 00 00 46 6c 6f 72 69 61 6e 20 46 6f 72 73  ....Florian Fors
[  605.735447] skb data:000000d0: 74 65 72 20 3c 6f 63 74 6f 40 76 65 72 70 6c 61  ter <octo@verpla
[  605.835455] skb data:000000e0: 6e 74 2e 6f 72 67 3e 20 68 74 74 70 3a 2f 2f 76  nt.org> http://v
[  605.935463] skb data:000000f0: 65 72 70 6c 61 6e 74 2e 6f 72 67 2f df 0b 91 35  erplant.org/...5
[  606.035472] skb data:00000100: 00 e8 92 d2 61 98 72 a5 00 1a a0 38 8a 1b 08 00  ....a.r....8....
[  606.135480] skb data:00000110: 45 00 00 a0 00 00 40 00 40 11 a4 64 c0 a8 0a 01  E.....@[email protected]....
[  606.235489] skb data:00000120: c0 a8 0a 97 00 35 04 6e 00 8c 96 86 9e 46 81 80  .....5.n.....F..
[  606.335497] skb data:00000130: 00 01 00 03 00 00 00 00 01 6c 0a 61 64 64 74 68  .........l.addth
[  606.435505] skb data:00000140: 69 73 63 64 6e 03 63 6f 6d 00 00 01 00 01 c0 0c  iscdn.com.......
[  606.535514] skb data:00000150: 00 05 00 01 00 00 00 be 00 25 08 77 69 6c 64 63  .........%.wildc
[  606.635522] skb data:00000160: 61 72 64 0a 61 64 64 74 68 69 73 63 64 6e 03 63  ard.addthiscdn.c
[  606.735531] skb data:00000170: 6f 6d 07 65 64 67 65 6b 65 79 03 6e 65 74 00 00  om.edgekey.net..
[  606.835539] skb data:00000180: 00 0a e6 ac 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  606.935547] skb data:00000190: 27 ef 08 00 45 00 00 54 0d da 40 00 7f 06 d3 63  '[email protected]
[  607.035556] skb data:000001a0: c0 a8 01 02 d1 55 87 66 05 5d 00 50 f0 d2 2a b2  .....U.f.].P..*.
[  607.135564] skb data:000001b0: b6 81 e7 7d 50 18 ff ff 1a ad 00 00 66 6c 61 74  ...}P.......flat
[  607.235573] skb data:000001c0: 65 2c 20 67 7a 69 70 2c 20 63 68 75 6e 6b 65 64  e, gzip, chunked
[  607.335581] skb data:000001d0: 2c 20 69 64 65 6e 74 69 74 79 2c 20 74 72 61 69  , identity, trai
[  607.435589] skb data:000001e0: 6c 65 72 73 0d 0a 0d 0a 00 00 00 00 00 00 00 00  lers............
[  607.535597] skb data:000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  609.648527] ------------[ cut here ]------------
[  609.748545] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  609.848554] Hardware name: PowerEdge SC440
[  609.948564] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  610.048573] Call Trace:
[  610.248591]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  610.448607]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  610.648622]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  610.848638]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  611.048654]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  611.248672]  [<c13311fe>] ? xfrm_lookup+0x1c/0x3a
[  611.448688]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  611.648704]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  611.848719]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  612.048735]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  612.248750]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  612.448766]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  612.648781]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  612.848797]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  613.048813]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  613.248828]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  613.448843]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  613.648858]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  613.848873]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  614.048886]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  614.248901]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  614.448915]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  614.648930]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  614.848943]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  615.048957]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  615.248973]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  615.448987]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  615.649004]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  615.849018]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  616.049033]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  616.249048]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  616.449062]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  616.649075]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  616.849089]  [<c1031ead>] ? irq_exit+0x63/0x68
[  617.049102]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  617.249116]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  617.449130]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  617.649144]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  617.849160]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  617.949169] ---[ end trace df8deff2ad2a9767 ]---
[  618.049177] 100: csum_start 306, offset 16, headroom 390, headlen 91, len 91
[  618.149185] nr_frags=0 gso_size=0
[  618.249191]
[  618.349200] skb data:00000000: 00 00 00 00 41 75 67 20 32 35 20 30 39 3a 35 37  ....Aug 25 09:57
[  618.449209] skb data:00000010: 3a 31 32 20 61 75 74 68 64 61 65 6d 6f 6e 64 3a  :12 authdaemond:
[  618.549217] skb data:00000020: 20 75 73 65 72 64 62 3a 20 68 6f 6d 65 3d 2f 68   userdb: home=/h
[  618.649225] skb data:00000030: 6f 6d 65 2f 76 6d 61 69 6c 2f 63 61 72 70 6f 2c  ome/vmail/carpo,
[  618.749233] skb data:00000040: 20 75 69 64 3d 31 30 30 36 2c 20 67 69 64 3d 31   uid=1006, gid=1
[  618.849241] skb data:00000050: 30 30 2c 20 73 68 65 6c 6c 3d 3c 75 6e 73 65 74  00, shell=<unset
[  618.949249] skb data:00000060: 3e 2c 20 6d 61 69 6c 3d 2f 68 6f 6d 65 2f 76 6d  >, mail=/home/vm
[  619.049258] skb data:00000070: 61 69 6c 2f 63 61 72 70 6f 2f 4d 61 69 6c 00 00  ail/carpo/Mail..
[  619.149266] skb data:00000080: 00 0e 2e 5c 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...\.....\'....\
[  619.249274] skb data:00000090: 27 ef 08 00 45 00 00 34 29 df 40 00 3f 06 29 b8  '...E..4).@.?.).
[  619.349283] skb data:000000a0: c0 a8 01 02 c0 57 66 2b b4 46 fc 20 4a 18 7e 15  .....Wf+.F. J.~.
[  619.449291] skb data:000000b0: ca 51 03 5b 80 10 27 06 2b a0 00 00 01 01 08 0a  .Q.[..'.+.......
[  619.549299] skb data:000000c0: 00 ea 5e 99 eb 9c aa 87 6f 6e 6e 65 63 74 69 6f  ..^.....onnectio
[  619.649307] skb data:000000d0: 6e 3a 20 63 6c 6f 73 65 0d 0a 43 6f 6e 74 65 6e  n: close..Conten
[  619.749316] skb data:000000e0: 74 2d 54 79 70 65 3a 20 69 6d 61 67 65 2f 67 69  t-Type: image/gi
[  619.849324] skb data:000000f0: 66 0d 0a 43 61 63 68 65 2d 43 6f 6e 74 72 6f 6c  f..Cache-Control
[  619.949332] skb data:00000100: 00 6c de c8 00 00 00 00 00 00 00 00 00 00 00 00  .l..............
[  620.049340] skb data:00000110: 45 00 00 68 00 00 40 00 ff 01 e5 5d c0 a8 0a 01  E..h..@....]....
[  620.149348] skb data:00000120: c0 a8 0a e5 08 00 09 1f 47 73 00 36 00 00 00 00  ........Gs.6....
[  620.249357] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  620.349365] skb data:00000140: 46 6c 6f 72 69 61 6e 20 46 6f 72 73 74 65 72 20  Florian Forster
[  620.449373] skb data:00000150: 3c 6f 63 74 6f 40 76 65 72 70 6c 61 6e 74 2e 6f  <[email protected]
[  620.549381] skb data:00000160: 72 67 3e 20 68 74 74 70 3a 2f 2f 76 65 72 70 6c  rg> http://verpl
[  620.649389] skb data:00000170: 61 6e 74 2e 6f 72 67 2f 00 00 00 00 00 00 00 00  ant.org/........
[  620.749397] skb data:00000180: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...\'....\'....\
[  620.849406] skb data:00000190: 27 ef 08 00 45 00 00 4d 0e 33 40 00 7f 06 d3 11  '...E..M.3@.....
[  620.949414] skb data:000001a0: c0 a8 01 02 d1 55 87 66 05 67 00 50 b0 ed 26 e3  .....U.f.g.P..&.
[  621.049423] skb data:000001b0: 21 67 b2 5b 50 18 ff ff 1a a6 00 00 67 7a 69 70  !g.[P.......gzip
[  621.149431] skb data:000001c0: 2c 20 63 68 75 6e 6b 65 64 2c 20 69 64 65 6e 74  , chunked, ident
[  621.249439] skb data:000001d0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  621.349448] skb data:000001e0: 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  621.449456] skb data:000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  621.551553] ------------[ cut here ]------------
[  621.651565] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  621.751576] Hardware name: PowerEdge SC440
[  621.851591] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  621.951600] Call Trace:
[  622.151616]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  622.351631]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  622.551646]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  622.751661]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  622.951676]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  623.151692]  [<c1083e35>] ? __slab_alloc+0x285/0x4ef
[  623.351707]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  623.551723]  [<c1224852>] ? rtl8139_start_xmit+0x51/0x13a
[  623.751737]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  623.951752]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  624.151767]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  624.351782]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  624.551798]  [<c12bfd67>] ? __qdisc_run+0x62/0xdf
[  624.751813]  [<c12b01ca>] ? dev_queue_xmit+0x155/0x354
[  624.951828]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  625.151843]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  625.351858]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  625.551872]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  625.751887]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  625.951903]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  626.151917]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  626.351932]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  626.551947]  [<c12adb80>] ? napi_skb_finish+0x2b/0x39
[  626.751961]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  626.951976]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  627.151992]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  627.352009]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  627.552024]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  627.752039]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  627.952054]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  628.152068]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  628.352083]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  628.552096]  [<c1031ead>] ? irq_exit+0x63/0x68
[  628.752110]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  628.952123]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  629.152137]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  629.352151]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  629.552165]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  629.652173] ---[ end trace df8deff2ad2a9768 ]---
[  629.752181] 100: csum_start 306, offset 16, headroom 390, headlen 91, len 91
[  629.852188] nr_frags=0 gso_size=0
[  629.952195]
[  630.052203] skb data:00000000: 00 4c f9 c8 4a 00 08 00 42 00 31 32 38 32 35 33  .L..J...B.128253
[  630.152212] skb data:00000010: 39 33 36 39 2e 4d 34 37 30 34 33 38 50 32 31 39  9369.M470438P219
[  630.252220] skb data:00000020: 32 56 30 30 30 30 30 30 30 30 30 30 30 30 30 39  2V00000000000009
[  630.352228] skb data:00000030: 30 36 49 30 31 32 39 36 31 46 42 5f 31 33 2e 66  06I012961FB_13.f
[  630.452237] skb data:00000040: 73 2c 53 3d 38 34 32 39 3a 32 2c 53 2c 53 53 30  s,S=8429:2,S,SS0
[  630.552245] skb data:00000050: 29 0d 0a 35 32 20 4f 4b 20 53 54 41 54 55 53 20  )..52 OK STATUS
[  630.652253] skb data:00000060: 43 6f 6d 70 6c 65 74 65 64 2e 0d 0a 00 00 00 00  Completed.......
[  630.752261] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  630.852270] skb data:00000080: 00 00 00 00 00 00 00 1a a0 38 8a 1b 00 04 61 98  .........8....a.
[  630.952279] skb data:00000090: 72 a5 08 00 45 00 00 30 30 83 40 00 80 06 2e 17  r...E..00.@.....
[  631.052287] skb data:000000a0: c0 a8 0a 97 ae 89 22 65 05 27 00 50 95 60 06 10  ......"e.'.P.`..
[  631.152296] skb data:000000b0: 00 00 00 00 70 02 ff ff 46 0a 00 00 02 04 05 b4  ....p...F.......
[  631.252304] skb data:000000c0: 01 01 04 02 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  631.352312] skb data:000000d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  631.452321] skb data:000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  631.552329] skb data:000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  631.652338] skb data:00000100: 00 ec fa c8 4a 00 08 00 42 00 31 32 38 32 37 31  ....J...B.128271
[  631.752346] skb data:00000110: 35 30 31 36 2e 4d 38 33 37 37 32 35 50 32 31 39  5016.M837725P219
[  631.852354] skb data:00000120: 32 37 56 30 30 30 30 30 30 30 30 30 30 30 30 30  27V0000000000000
[  631.952362] skb data:00000130: 39 30 36 49 30 30 45 41 30 37 45 45 5f 36 2e 66  906I00EA07EE_6.f
[  632.052371] skb data:00000140: 73 2c 53 3d 36 30 30 31 3a 32 2c 53 53 32 2c 53  s,S=6001:2,SS2,S
[  632.152379] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  632.252388] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  632.352396] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  632.452404] skb data:00000180: 00 00 00 00 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  632.552413] skb data:00000190: 27 ef 08 00 45 00 00 4d 0e 3d 40 00 7f 06 d3 07  '...E..M.=@.....
[  632.652422] skb data:000001a0: c0 a8 01 02 d1 55 87 66 05 67 00 50 b0 ed 26 e3  .....U.f.g.P..&.
[  632.752430] skb data:000001b0: 21 67 b2 5b 50 18 ff ff 1a a6 00 00 67 7a 69 70  !g.[P.......gzip
[  632.852438] skb data:000001c0: 2c 20 63 68 75 6e 6b 65 64 2c 20 69 64 65 6e 74  , chunked, ident
[  632.952447] skb data:000001d0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  633.052455] skb data:000001e0: 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  633.152464] skb data:000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  639.625714] ------------[ cut here ]------------
[  639.725733] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  639.825742] Hardware name: PowerEdge SC440
[  639.925752] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  640.025761] Call Trace:
[  640.225779]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  640.425795]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  640.625810]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  640.825825]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  641.025844]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  641.225869]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  641.425893]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  641.625917]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  641.825932]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  642.025948]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  642.225963]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  642.425980]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  642.625995]  [<c12b5601>] ? neigh_resolve_output+0xd0/0x280
[  642.826016]  [<c12be0a5>] ? eth_header+0x0/0x9b
[  643.026032]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  643.226048]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  643.426063]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  643.626078]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  643.826092]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  644.026107]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  644.226122]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  644.426136]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  644.626152]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  644.826167]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  645.026181]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  645.226197]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  645.426212]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  645.626226]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  645.826240]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  646.026255]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  646.226269]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  646.426283]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  646.626297]  [<c1031ead>] ? irq_exit+0x63/0x68
[  646.826310]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  647.026323]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  647.226338]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  647.426351]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  647.626366]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  647.726374] ---[ end trace df8deff2ad2a9769 ]---
[  647.826383] 100: csum_start 306, offset 16, headroom 390, headlen 96, len 96
[  647.926390] nr_frags=0 gso_size=0
[  648.026397]
[  648.126406] skb data:00000000: 00 c0 1c cc 05 07 f3 1a 00 0e 2e 5c 27 e6 08 00  ...........\'...
[  648.226416] skb data:00000010: 45 00 00 c2 00 00 40 00 40 11 2a c9 c0 a8 c7 01  E.....@.@.*.....
[  648.326424] skb data:00000020: c0 a8 c7 0f 00 35 05 16 00 ae 10 22 01 4d 81 80  .....5.....".M..
[  648.426433] skb data:00000030: 00 01 00 07 00 00 00 00 08 63 6c 69 65 6e 74 73  .........clients
[  648.526441] skb data:00000040: 31 06 67 6f 6f 67 6c 65 02 62 67 00 00 01 00 01  1.google.bg.....
[  648.626450] skb data:00000050: c0 0c 00 05 00 01 00 00 13 fa 00 16 07 63 6c 69  .............cli
[  648.726459] skb data:00000060: 65 6e 74 73 01 6c 06 67 6f 6f 67 6c 65 03 63 6f  ents.l.google.co
[  648.826467] skb data:00000070: 6d 00 c0 30 00 01 00 01 00 00 00 55 00 04 00 00  m..0.......U....
[  648.926475] skb data:00000080: 00 0a e6 ac 07 db 00 1a a0 38 8a 1b 08 00 45 00  .........8....E.
[  649.026484] skb data:00000090: 00 28 00 00 40 00 38 06 a1 fd d4 32 01 64 c0 a8  .([email protected]..
[  649.126492] skb data:000000a0: 0a 94 00 50 05 64 27 80 ba dc 66 e9 47 b0 50 10  ...P.d'...f.G.P.
[  649.226501] skb data:000000b0: 19 f0 5e 67 00 00 ff ff 4b df 00 00 01 01 08 0a  ..^g....K.......
[  649.326510] skb data:000000c0: 00 ea 5e 7e eb 9c 7b 39 01 01 05 0a ca 4e e7 5b  ..^~..{9.....N.[
[  649.426518] skb data:000000d0: ca 4e ec fb 46 2a 33 d2 18 82 46 b7 eb 8e 44 77  .N..F*3...F...Dw
[  649.526526] skb data:000000e0: bc 35 37 3a 31 30 20 47 4d 54 0d 0a 53 65 72 76  .57:10 GMT..Serv
[  649.626535] skb data:000000f0: 65 72 3a 20 41 70 61 63 68 65 0d 0a 4c 61 73 74  er: Apache..Last
[  649.726543] skb data:00000100: 00 fc 1c cc 00 00 00 00 00 00 00 00 00 00 08 00  ................
[  649.826551] skb data:00000110: 45 c0 00 84 09 52 00 00 40 01 db 14 c0 a8 0a 01  E....R..@.......
[  649.926560] skb data:00000120: c0 a8 0a 01 03 01 fc fe 00 00 00 00 45 00 00 68  ............E..h
[  650.026568] skb data:00000130: 00 00 40 00 ff 01 e5 5d c0 a8 0a 01 c0 a8 0a e5  ..@....]........
[  650.126576] skb data:00000140: 08 00 09 1d 47 73 00 38 00 00 00 00 00 00 00 00  ....Gs.8........
[  650.226585] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 46 6c 6f 72  ............Flor
[  650.326593] skb data:00000160: 69 61 6e 20 46 6f 72 73 74 65 72 20 3c 6f 63 74  ian Forster <oct
[  650.426601] skb data:00000170: 6f 40 76 65 72 70 6c 61 6e 74 2e 6f 72 67 00 00  [email protected]..
[  650.526609] skb data:00000180: 00 15 58 8f 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ..X......\'....\
[  650.626618] skb data:00000190: 27 ef 08 00 45 00 00 52 af d0 40 00 7f 06 18 13  '...E..R..@.....
[  650.726626] skb data:000001a0: c0 a8 01 02 4a 7d 27 9b 05 dd 00 50 21 f4 27 de  ....J}'....P!.'.
[  650.826635] skb data:000001b0: 5d d2 44 c1 50 18 ff ff 34 07 00 00 61 74 65 2c  ].D.P...4...ate,
[  650.926643] skb data:000001c0: 20 67 7a 69 70 2c 20 63 68 75 6e 6b 65 64 2c 20   gzip, chunked,
[  651.026651] skb data:000001d0: 69 64 65 6e 74 69 74 79 2c 20 74 72 61 69 6c 65  identity, traile
[  651.126660] skb data:000001e0: 72 73 0d 0a 0d 0a 20 6e 6f 2d 63 61 63 68 65 0d  rs.... no-cache.
[  651.226668] skb data:000001f0: 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a  .Content-Length:
[  651.778140] ------------[ cut here ]------------
[  651.878158] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  651.978167] Hardware name: PowerEdge SC440
[  652.078177] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  652.178185] Call Trace:
[  652.378204]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  652.578220]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  652.778235]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  652.978252]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  653.178267]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  653.378284]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  653.578300]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  653.778316]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  653.978331]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  654.178348]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  654.378364]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  654.578380]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  654.778395]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  654.978409]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  655.178424]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  655.378438]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  655.578453]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  655.778468]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  655.978482]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  656.178497]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  656.378512]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  656.578526]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  656.778539]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  656.978552]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  657.178565]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  657.378579]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  657.578592]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  657.778605]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  657.978619]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  658.178632]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  658.378644]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  658.578656]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  658.778669]  [<c1031ead>] ? irq_exit+0x63/0x68
[  658.978683]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  659.178696]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  659.378710]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  659.578723]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  659.778739]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  659.878748] ---[ end trace df8deff2ad2a976a ]---
[  659.978756] 100: csum_start 306, offset 16, headroom 390, headlen 95, len 95
[  660.078764] nr_frags=0 gso_size=0
[  660.178770]
[  660.278779] skb data:00000000: 00 9c 27 cc 2e 5c 27 b2 00 0e 2e 5c 27 ef 08 00  ..'..\'....\'...
[  660.378788] skb data:00000010: 45 c0 00 83 c3 a0 00 00 40 01 45 4d c0 a8 01 02  [email protected]....
[  660.478797] skb data:00000020: d0 43 de de 03 03 6e 2e 00 00 00 00 45 00 00 67  .C....n.....E..g
[  660.578806] skb data:00000030: 00 00 40 00 39 11 d0 b9 d0 43 de de c0 a8 01 02  [email protected]......
[  660.678814] skb data:00000040: 00 35 3a 76 00 53 db d6 33 72 81 80 00 01 00 01  .5:v.S..3r......
[  660.778822] skb data:00000050: 00 00 00 00 03 31 31 34 03 32 31 37 02 31 32 03  .....114.217.12.
[  660.878831] skb data:00000060: 32 31 36 07 69 6e 2d 61 64 64 72 04 61 72 70 61  216.in-addr.arpa
[  660.978840] skb data:00000070: 00 00 0c 00 01 c0 0c 00 0c 00 01 00 00 13 df 00  ................
[  661.078848] skb data:00000080: 00 0e 2e 5c 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...\.....\'....\
[  661.178857] skb data:00000090: 27 ef 08 00 45 00 00 28 b0 5c 40 00 7f 06 2f c7  '...E..(.\@.../.
[  661.278865] skb data:000000a0: c0 a8 01 02 55 0e 04 f4 05 df 00 50 bb 6b 1e cb  ....U......P.k..
[  661.378874] skb data:000000b0: b8 42 fe b0 50 10 ff ff fc ce 00 00 c6 4c a3 d6  .B..P........L..
[  661.478882] skb data:000000c0: 50 82 60 6f d3 94 63 5e 4f e6 84 d9 c7 26 05 b3  P.`o..c^O....&..
[  661.578891] skb data:000000d0: 74 4c 39 c0 8d b5 fd c0 0c 00 0c 00 01 00 00 19  tL9.............
[  661.678899] skb data:000000e0: 29 00 12 06 61 39 30 35 73 6d 05 61 76 61 73 74  )...a905sm.avast
[  661.778907] skb data:000000f0: 03 63 6f 6d 00 00 00 00 00 00 00 00 00 00 00 00  .com............
[  661.878916] skb data:00000100: 00 ac 27 cc 05 07 f3 1a 00 0e 2e 5c 27 e6 08 00  ..'........\'...
[  661.978924] skb data:00000110: 45 00 00 ac 00 00 40 00 40 11 2a df c0 a8 c7 01  E.....@.@.*.....
[  662.078932] skb data:00000120: c0 a8 c7 0f 00 35 04 04 00 98 10 0c 01 55 81 80  .....5.......U..
[  662.178941] skb data:00000130: 00 01 00 05 00 00 00 00 07 70 61 67 65 61 64 32  .........pagead2
[  662.278949] skb data:00000140: 11 67 6f 6f 67 6c 65 73 79 6e 64 69 63 61 74 69  .googlesyndicati
[  662.378958] skb data:00000150: 6f 6e 03 63 6f 6d 00 00 01 00 01 c0 0c 00 05 00  on.com..........
[  662.478966] skb data:00000160: 01 00 01 46 5c 00 15 06 70 61 67 65 61 64 01 6c  ...F\...pagead.l
[  662.578975] skb data:00000170: 06 67 6f 6f 67 6c 65 03 63 6f 6d 00 c0 3b 00 00  .google.com..;..
[  662.678983] skb data:00000180: 00 26 9e 0f 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  .&.......\'....\
[  662.778991] skb data:00000190: 27 ef 08 00 45 00 00 51 b0 d2 40 00 7f 06 30 70  '[email protected]
[  662.879003] skb data:000001a0: c0 a8 01 02 d1 55 87 64 05 8c 00 50 fc 1e ec 32  .....U.d...P...2
[  662.979012] skb data:000001b0: ce 21 af 11 50 18 fc 27 1a a8 00 00 74 65 2c 20  .!..P..'....te,
[  663.079020] skb data:000001c0: 67 7a 69 70 2c 20 63 68 75 6e 6b 65 64 2c 20 69  gzip, chunked, i
[  663.179029] skb data:000001d0: 64 65 6e 74 69 74 79 2c 20 74 72 61 69 6c 65 72  dentity, trailer
[  663.279037] skb data:000001e0: 73 0d 0a 0d 0a 22 3a 22 52 55 42 49 43 4f 4e 5f  s....":"RUBICON_
[  663.379045] skb data:000001f0: 50 52 4f 5f 42 55 5f 52 4f 4e 5f 37 32 38 58 39  PRO_BU_RON_728X9
[  690.540721] ------------[ cut here ]------------
[  690.640741] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  690.740750] Hardware name: PowerEdge SC440
[  690.840761] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  690.940770] Call Trace:
[  691.140789]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  691.340804]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  691.540820]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  691.740836]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  691.940852]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  692.140868]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  692.340884]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  692.540901]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  692.740916]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  692.940932]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  693.140948]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  693.340965]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  693.540981]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  693.740997]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  693.941017]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  694.141032]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  694.341046]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  694.541061]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  694.741075]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  694.941089]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  695.141103]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  695.341116]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  695.541131]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  695.741145]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  695.941160]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  696.141175]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  696.341189]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  696.541203]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  696.741218]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  696.941233]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  697.141247]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  697.341261]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  697.541275]  [<c1031ead>] ? irq_exit+0x63/0x68
[  697.741289]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  697.941303]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  698.141318]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  698.341332]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  698.541348]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  698.641356] ---[ end trace df8deff2ad2a976b ]---
[  698.741364] 100: csum_start 306, offset 16, headroom 390, headlen 75, len 75
[  698.841372] nr_frags=0 gso_size=0
[  698.941378]
[  699.041387] skb data:00000000: 00 c4 28 db 3b 07 9b 3f 00 1a a0 38 8a 1b 08 00  ..(.;..?...8....
[  699.141397] skb data:00000010: 45 00 00 68 00 00 40 00 ff 01 e6 3e c0 a8 0a 01  E..h..@....>....
[  699.241406] skb data:00000020: c0 a8 0a 04 08 00 99 ce b6 bb 00 3e 00 00 00 00  ...........>....
[  699.341414] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  699.441422] skb data:00000040: 46 6c 6f 72 69 61 6e 20 46 6f 72 73 74 65 72 20  Florian Forster
[  699.541431] skb data:00000050: 3c 6f 63 74 6f 40 76 65 72 70 6c 61 6e 74 2e 6f  <[email protected]
[  699.641439] skb data:00000060: 72 67 3e 20 68 74 74 70 3a 2f 2f 76 65 72 70 6c  rg> http://verpl
[  699.741447] skb data:00000070: 61 6e 74 2e 6f 72 67 2f 00 00 00 00 00 00 00 00  ant.org/........
[  699.841456] skb data:00000080: 00 00 00 00 00 00 00 1a a0 38 8a 1b 00 19 66 2f  .........8....f/
[  699.941464] skb data:00000090: f9 3e 08 00 45 00 00 28 da 32 40 00 80 06 8a ac  .>..E..(.2@.....
[  700.041473] skb data:000000a0: c0 a8 0a 9f c0 a8 0a 01 07 2f 00 8f 75 27 b1 29  ........./..u'.)
[  700.141481] skb data:000000b0: 7a 97 06 bb 50 10 b0 27 ba 5a 00 00 00 00 00 00  z...P..'.Z......
[  700.241490] skb data:000000c0: 00 00 4d 61 69 6c 64 69 00 00 00 19 66 2f f9 3e  ..Maildi....f/.>
[  700.341498] skb data:000000d0: 00 1a a0 38 8a 1b 08 00 45 00 05 dc f1 ed 40 00  ...8....E.....@.
[  700.441506] skb data:000000e0: 40 06 00 00 c0 a8 0a 01 c0 a8 0a 9f 00 8f 07 2f  @............../
[  700.541514] skb data:000000f0: 7a 97 b1 d3 75 27 b1 29 50 10 1d 8e 00 00 00 00  z...u'.)P.......
[  700.641523] skb data:00000100: 00 f0 28 db 39 c6 77 33 00 1a a0 38 8a 1b 08 00  ..(.9.w3...8....
[  700.741531] skb data:00000110: 45 00 00 68 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  E..h.....\'....\
[  700.841539] skb data:00000120: 27 ef 08 00 45 00 00 7c 00 00 40 00 40 29 65 53  '...E..|..@.@)eS
[  700.941548] skb data:00000130: c0 a8 01 02 d4 12 3f 49 60 00 00 00 00 40 3a 40  ......?I`....@:@
[  701.041556] skb data:00000140: 20 01 15 c0 65 ff 00 64 00 00 00 00 00 00 00 02   ...e..d........
[  701.141565] skb data:00000150: 20 01 15 c0 65 ff 00 64 00 00 00 00 00 00 00 01   ...e..d........
[  701.241573] skb data:00000160: 80 00 c2 0f 69 0b 00 17 24 bf 74 4c 8a f8 0d 00  ....i...$.tL....
[  701.341581] skb data:00000170: 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 00 00  ................
[  701.441589] skb data:00000180: 00 15 58 8f a0 21 00 0e 2e 5c 27 b2 00 0e 2e 5c  ..X..!...\'....\
[  701.541598] skb data:00000190: 27 ef 08 00 45 00 00 3d b6 47 40 00 7f 06 a3 44  '[email protected]
[  701.641606] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 0d 00 50 d0 e5 75 18  .....2.R...P..u.
[  701.741615] skb data:000001b0: 30 a2 40 58 50 18 ff ff a2 5e 00 00 64 65 6e 74  0.@XP....^..dent
[  701.841623] skb data:000001c0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  701.941632] skb data:000001d0: 0a 74 65 73 0d 0a 0d 0a 76 01 15 41 79 79 e9 99  .tes....v..Ayy..
[  702.041639] skb data:000001e0: 79 15 fa 06 7a 66 7a c6 96 c8 4a f4 41 96 80 0c  y...zfz...J.A...
[  702.141647] skb data:000001f0: d5 87 3a 10 00 77 17 1a 5e a9 00 00 00 0d 0a 30  ..:..w..^......0
[  702.324707] ------------[ cut here ]------------
[  702.424725] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  702.524734] Hardware name: PowerEdge SC440
[  702.624744] Pid: 3324, comm: couriertls Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  702.724752] Call Trace:
[  702.924771]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  703.124786]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  703.324801]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  703.524816]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  703.724831]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  703.924847]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  704.124862]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  704.324879]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  704.524894]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  704.724910]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  704.924925]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  705.124941]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  705.324956]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  705.524971]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  705.724986]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  705.925005]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  706.125020]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  706.325036]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  706.525051]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  706.725066]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  706.925081]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  707.125095]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  707.325111]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  707.525126]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  707.725141]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  707.925157]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  708.125172]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  708.325187]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  708.525201]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  708.725216]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  708.925231]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  709.125246]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  709.325260]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  709.525273]  [<c1031ead>] ? irq_exit+0x63/0x68
[  709.725287]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  709.925300]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  710.125314]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  710.325329]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  710.425338] ---[ end trace df8deff2ad2a976c ]---
[  710.525346] 100: csum_start 306, offset 16, headroom 390, headlen 75, len 75
[  710.625353] nr_frags=0 gso_size=0
[  710.725360]
[  710.825369] skb data:00000000: 00 48 4f c7 51 c3 f2 5d 04 1c 78 62 65 c1 6b f9  .HO.Q..]..xbe.k.
[  710.925378] skb data:00000010: ab 7a d9 42 35 93 16 47 f4 a4 84 16 d5 4c ee 15  .z.B5..G.....L..
[  711.025386] skb data:00000020: 4f 1c f1 93 b6 b6 a8 29 0b 45 9c 6e da a6 79 47  O......).E.n..yG
[  711.125395] skb data:00000030: 62 61 04 40 f8 6f 0f 09 ab 27 07 05 48 73 bb 37  [email protected]...'..Hs.7
[  711.225403] skb data:00000040: 59 b9 1e 1e eb 42 89 3d 05 92 ba 1b 13 11 d8 9f  Y....B.=........
[  711.325412] skb data:00000050: b1 fc 54 19 2d 1f 30 f2 11 99 cb b4 51 5c df 52  ..T.-.0.....Q\.R
[  711.425420] skb data:00000060: 84 c8 00 06 a2 73 f6 39 a5 c5 3f b8 31 04 2f c9  .....s.9..?.1./.
[  711.525429] skb data:00000070: 44 2d 0e ca 95 99 aa a9 b1 52 aa e9 eb 3f a5 2e  D-.......R...?..
[  711.625437] skb data:00000080: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 08 00 45 c0  ...\'....\'...E.
[  711.725446] skb data:00000090: 00 5f 4f da 00 00 40 01 a6 b0 c0 a8 01 01 c0 a8  ._O...@.........
[  711.825454] skb data:000000a0: 01 02 03 03 80 91 00 00 00 00 45 00 00 43 00 00  ..........E..C..
[  711.925462] skb data:000000b0: 40 00 40 11 b7 56 c0 a8 01 02 c0 a8 01 01 b0 b4  @[email protected]..........
[  712.025471] skb data:000000c0: 00 35 00 2f 8e 51 ea df 01 00 00 01 00 00 00 00  .5./.Q..........
[  712.125479] skb data:000000d0: 00 00 0b 64 6f 77 6e 6c 6f 61 64 39 30 36 05 61  ...download906.a
[  712.225488] skb data:000000e0: 76 61 73 74 03 63 6f 6d 00 00 01 00 01 c0 da 16  vast.com........
[  712.325496] skb data:000000f0: ca 17 2a 8d 37 40 df 9d 82 d4 85 9d 28 da 7c 82  ..*.7@......(.|.
[  712.425505] skb data:00000100: 00 7c 4f c7 7a b7 cc c1 ec 9e 02 66 9e 02 c8 b0  .|O.z......f....
[  712.525513] skb data:00000110: b4 5c 6a 24 8a 04 1c c3 4d c0 c8 fa 0f 2d a6 cc  .\j$....M....-..
[  712.625521] skb data:00000120: a3 5a c9 25 25 c5 10 34 e9 ac c1 2f 62 28 dd bf  .Z.%%..4.../b(..
[  712.725530] skb data:00000130: 19 74 43 9b 1a e0 46 5a 44 c9 0d 03 7f 21 61 b2  .tC...FZD....!a.
[  712.825538] skb data:00000140: 5e a6 09 8b e1 47 99 f5 00 f0 d4 ec 33 79 bb 08  ^....G......3y..
[  712.925547] skb data:00000150: ab 16 6a 17 27 5f 1f 6a 6f fd ae c3 e6 ab 35 24  ..j.'_.jo.....5$
[  713.025555] skb data:00000160: 6d 8b c0 33 1f c8 ff 06 c0 3e eb 8e 93 af 5e 0c  m..3.....>....^.
[  713.125563] skb data:00000170: 47 1b 41 40 28 57 48 be 6f 79 e2 dc c3 ed 00 00  G.A@(WH.oy......
[  713.225572] skb data:00000180: 00 15 58 8f 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ..X......\'....\
[  713.325580] skb data:00000190: 27 ef 08 00 45 00 00 3d b6 9a 40 00 7f 06 a2 f1  '...E..=..@.....
[  713.425589] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 15 00 50 0a db 39 df  .....2.R...P..9.
[  713.525597] skb data:000001b0: 3c 2f 3e 11 50 18 ff ff a2 5e 00 00 64 65 6e 74  </>.P....^..dent
[  713.625605] skb data:000001c0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  713.725613] skb data:000001d0: 0a 98 0d 99 47 2b ed 9f 6a 89 9f 84 62 18 59 ce  ....G+..j...b.Y.
[  713.825621] skb data:000001e0: f9 c7 c6 08 33 7c db 30 53 9e 58 3e 33 35 5a 6a  ....3|.0S.X>35Zj
[  713.925630] skb data:000001f0: a8 99 4b 93 e0 bd 9b b5 8c 93 52 a4 eb 33 9d f4  ..K.......R..3..
[  714.086695] ------------[ cut here ]------------
[  714.186713] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  714.286722] Hardware name: PowerEdge SC440
[  714.386733] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  714.486742] Call Trace:
[  714.686760]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  714.886776]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  715.086791]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  715.286806]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  715.486822]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  715.686839]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  715.886854]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  716.086870]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  716.286885]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  716.486901]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  716.686917]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  716.886932]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  717.086947]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  717.286960]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  717.486974]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  717.686987]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  717.887005]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  718.087019]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  718.287034]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  718.487049]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  718.687064]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  718.887077]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  719.087092]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  719.287105]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  719.487119]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  719.687133]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  719.887147]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  720.087160]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  720.287173]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  720.487188]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  720.687203]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  720.887217]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  721.087231]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  721.287244]  [<c1031ead>] ? irq_exit+0x63/0x68
[  721.487257]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  721.687270]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  721.887284]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  722.087297]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  722.287312]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  722.387323] ---[ end trace df8deff2ad2a976d ]---
[  722.487334] 100: csum_start 306, offset 16, headroom 390, headlen 87, len 87
[  722.587345] nr_frags=0 gso_size=0
[  722.687355]
[  722.787367] skb data:00000000: 00 78 1d c7 00 00 00 00 00 00 00 00 00 00 00 00  .x..............
[  722.887379] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  722.987390] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  723.087399] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  723.187407] skb data:00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  723.287415] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  723.387423] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  723.487431] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  723.587439] skb data:00000080: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 08 00 45 00  ...\'....\'...E.
[  723.687448] skb data:00000090: 00 30 00 00 40 00 3a 06 9e 95 d4 32 0c 56 c0 a8  .0..@.:....2.V..
[  723.787456] skb data:000000a0: 01 02 00 50 06 17 46 e7 e5 65 ca 4a f8 64 70 12  ...P..F..e.J.dp.
[  723.887464] skb data:000000b0: 16 d0 d4 b0 00 00 02 04 05 ac 01 01 04 02 da 76  ...............v
[  723.987473] skb data:000000c0: 00 35 00 2c df 47 69 dd 01 00 00 01 00 00 00 00  .5.,.Gi.........
[  724.087481] skb data:000000d0: 00 00 07 74 6f 6f 6c 62 61 72 06 67 6f 6f 67 6c  ...toolbar.googl
[  724.187490] skb data:000000e0: 65 03 63 6f 6d 00 00 01 00 01 00 00 00 00 00 00  e.com...........
[  724.287498] skb data:000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  724.387506] skb data:00000100: 00 54 1d c7 00 00 00 00 00 00 00 00 00 00 00 00  .T..............
[  724.487515] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  724.587523] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  724.687531] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  724.787540] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  724.887548] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  724.987556] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  725.087565] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  725.187573] skb data:00000180: 00 0a e6 ac 07 db 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  725.287581] skb data:00000190: 27 ef 08 00 45 00 00 49 b6 f5 40 00 7f 06 a2 8a  '...E..I..@.....
[  725.387590] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 19 00 50 cc 53 da 48  .....2.R...P.S.H
[  725.487598] skb data:000001b0: 47 54 a4 18 50 18 ff ff a2 6a 00 00 2c 20 63 68  GT..P....j.., ch
[  725.587606] skb data:000001c0: 75 6e 6b 65 64 2c 20 69 64 65 6e 74 69 74 79 2c  unked, identity,
[  725.687614] skb data:000001d0: 20 74 72 61 69 6c 65 72 73 0d 0a 0d 0a 00 00 00   trailers.......
[  725.787623] skb data:000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  725.887631] skb data:000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  725.989879] ------------[ cut here ]------------
[  726.089990] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  726.190007] Hardware name: PowerEdge SC440
[  726.290017] Pid: 2956, comm: fah6 Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  726.390025] Call Trace:
[  726.590041]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  726.790056]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  726.990071]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  727.190086]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  727.390101]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  727.590117]  [<c1083e35>] ? __slab_alloc+0x285/0x4ef
[  727.790132]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  727.990148]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  728.190162]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  728.390178]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  728.590193]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  728.790208]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  728.990224]  [<c12b5601>] ? neigh_resolve_output+0xd0/0x280
[  729.190239]  [<c12be0a5>] ? eth_header+0x0/0x9b
[  729.390255]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  729.590270]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  729.790285]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  729.990300]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  730.190315]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  730.390330]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  730.590345]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  730.790359]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  730.990374]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  731.190389]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  731.390404]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  731.590420]  [<c1129273>] ? xfs_file_aio_read+0x181/0x27a
[  731.790435]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  731.990450]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  732.190465]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  732.390479]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  732.590494]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  732.790508]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  732.990522]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  733.190536]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  733.390549]  [<c1031ead>] ? irq_exit+0x63/0x68
[  733.590563]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  733.790577]  [<c1088c0f>] ? sys_read+0x68/0x6a
[  733.990591]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  734.190605]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  734.290613] ---[ end trace df8deff2ad2a976e ]---
[  734.390622] 100: csum_start 306, offset 16, headroom 390, headlen 146, len 146
[  734.490630] nr_frags=0 gso_size=0
[  734.590636]
[  734.690644] skb data:00000000: 00 84 5b c7 80 35 54 c1 00 00 00 00 00 00 00 00  ..[..5T.........
[  734.790653] skb data:00000010: 01 00 00 00 00 00 00 00 00 00 00 c0 31 7a 03 c1  ............1z..
[  734.890662] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  734.990670] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 24 f4 7c b7  ............$.|.
[  735.090678] skb data:00000040: 00 00 00 00 00 00 00 00 9d 6e ac 57 32 2c 53 a7  .........n.W2,S.
[  735.190687] skb data:00000050: fa 82 6a 39 12 28 ba 9d 4b 9c bf 30 64 a5 71 30  ..j9.(..K..0d.q0
[  735.290695] skb data:00000060: 85 2f e2 93 e5 72 f6 56 d8 df 5e a9 88 bb 07 63  ./...r.V..^....c
[  735.390704] skb data:00000070: c6 e8 91 b1 b7 40 77 8f 54 9f df 81 5c 16 56 64  [email protected]...\.Vd
[  735.490712] skb data:00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  735.590720] skb data:00000090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  735.690729] skb data:000000a0: 00 00 00 00 01 00 00 00 67 d5 da 34 c6 d0 e2 4b  ........g..4...K
[  735.790737] skb data:000000b0: b2 85 0e 8f c3 67 ac 47 20 85 3c 7f 11 86 a2 4d  .....g.G .<....M
[  735.890746] skb data:000000c0: e1 a3 b0 c1 54 86 8d 7b e9 f1 0c dc 03 b0 6f cd  ....T..{......o.
[  735.990754] skb data:000000d0: c8 94 7d 6e 78 fb 1b 8a b6 da 78 3a d0 c7 17 7a  ..}nx.....x:...z
[  736.090762] skb data:000000e0: 10 1b 6f 83 97 ba c0 bb ff ef 3f 12 3f 12 89 14  ..o.......?.?...
[  736.190771] skb data:000000f0: 3d 01 66 bf 5c f2 59 24 b7 60 5e 76 f7 e5 5f 00  =.f.\.Y$.`^v.._.
[  736.290779] skb data:00000100: 00 f8 2c c7 00 00 00 00 00 00 00 00 00 00 00 00  ..,.............
[  736.390787] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  736.490795] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  736.590804] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  736.690812] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  736.790820] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  736.890829] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  736.990837] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  737.090845] skb data:00000180: 00 00 00 00 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  737.190853] skb data:00000190: 27 ef 08 00 45 00 00 84 b7 05 40 00 7f 06 10 ac  '...E.....@.....
[  737.290862] skb data:000001a0: c0 a8 01 02 4a 7d 27 9b 06 1b 00 50 55 fa 51 79  ....J}'....PU.Qy
[  737.390870] skb data:000001b0: f0 df ec 83 50 18 ff ff 34 39 00 00 3a 20 24 56  ....P...49..: $V
[  737.490878] skb data:000001c0: 65 72 73 69 6f 6e 3d 31 0d 0a 43 6f 6e 6e 65 63  ersion=1..Connec
[  737.590887] skb data:000001d0: 74 69 6f 6e 3a 20 4b 65 65 70 2d 41 6c 69 76 65  tion: Keep-Alive
[  737.690895] skb data:000001e0: 2c 20 54 45 0d 0a 54 45 3a 20 64 65 66 6c 61 74  , TE..TE: deflat
[  737.790903] skb data:000001f0: 65 2c 20 67 7a 69 70 2c 20 63 68 75 6e 6b 65 64  e, gzip, chunked
[  737.890911] skb data:00000200: 2c 20 69 64 65 6e 74 69 74 79 2c 20 74 72 61 69  , identity, trai
[  737.990920] skb data:00000210: 6c 65 72 73 0d 0a 0d 0a 34 32 36 36 50 31 34 35  lers....4266P145
[  738.090928] skb data:00000220: 31 31 56 30 30 30 30 30 30 30 30 30 30 30 30 30  11V0000000000000
[  738.190936] skb data:00000230: 39 30 36 49 30 30 45 37 37 42 45 35 5f 35 37 35  906I00E77BE5_575
[  738.290944] skb data:00000240: 2e 66 73 2c 53 3d 31 32 34 32 39 3a 32 2c 53 9a  .fs,S=12429:2,S.
[  738.390953] skb data:00000250: 73 e3 6a fd 88 f1 5f 91 84 4a df b2 dc e5 04 0b  s.j..._..J......
[  738.490961] skb data:00000260: 83 fd 06 69 f9 fb 5b c2 55 fe c5 a6 df 11 b6 1c  ...i..[.U.......
[  738.590969] skb data:00000270: b3 b3 a8 e5 37 56 12 23 a3 05 d9 34 15 06 71 ab  ....7V.#...4..q.
[  739.154435] ------------[ cut here ]------------
[  739.254453] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  739.354462] Hardware name: PowerEdge SC440
[  739.454472] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  739.554481] Call Trace:
[  739.754499]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  739.954515]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  740.154530]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  740.354546]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  740.554562]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  740.754578]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  740.954594]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  741.154611]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  741.354626]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  741.554643]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  741.754658]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  741.954674]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  742.154690]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  742.354705]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  742.554721]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  742.754736]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  742.954751]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  743.154765]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  743.354780]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  743.554795]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  743.754810]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  743.954825]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  744.154840]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  744.354855]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  744.554871]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  744.754887]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  744.954902]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  745.154916]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  745.354932]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  745.554948]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  745.754964]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  745.954979]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  746.154992]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  746.355009]  [<c1031ead>] ? irq_exit+0x63/0x68
[  746.555023]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  746.755036]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  746.955050]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  747.155063]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  747.355078]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  747.455086] ---[ end trace df8deff2ad2a976f ]---
[  747.555094] 100: csum_start 306, offset 16, headroom 390, headlen 173, len 173
[  747.655102] nr_frags=0 gso_size=0
[  747.755108]
[  747.855117] skb data:00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  747.955125] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.055134] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.155142] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.255150] skb data:00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.355159] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.455167] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.555175] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.655184] skb data:00000080: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 08 00 45 20  ...\'....\'...E
[  748.755192] skb data:00000090: 00 30 74 a0 00 00 30 11 c7 88 18 38 74 92 c0 a8  .0t...0....8t...
[  748.855200] skb data:000000a0: 01 02 6c 91 3d 63 00 1c 66 5f 27 c6 02 94 fb da  ..l.=c..f_'.....
[  748.955209] skb data:000000b0: f9 96 7a cb d2 6c 26 66 b7 0e 71 37 e5 3c d7 8d  ..z..l&f..q7.<..
[  749.055217] skb data:000000c0: 24 5f 58 61 a6 ad 55 0a 9d 1f ce 5f fa 92 d1 67  $_Xa..U...._...g
[  749.155226] skb data:000000d0: fa e2 9a 9d 2a 68 6e 06 fd 0d 79 b8 cb 02 9a 3e  ....*hn...y....>
[  749.255234] skb data:000000e0: 9f 84 be 76 65 73 47 64 7f 07 88 fa cf 85 03 8a  ...vesGd........
[  749.355243] skb data:000000f0: 72 37 d0 0f bc 43 55 2d df 44 6f e0 dd 9b b6 1a  r7...CU-.Do.....
[  749.455251] skb data:00000100: 00 98 29 c7 00 00 00 00 00 00 00 00 00 00 00 00  ..).............
[  749.555259] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  749.655268] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  749.755277] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  749.855285] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  749.955294] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  750.055302] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  750.155310] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  750.255319] skb data:00000180: 00 16 17 a4 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  750.355327] skb data:00000190: 27 ef 08 00 45 00 00 9f b7 a2 40 00 7f 06 0f f4  '...E.....@.....
[  750.455335] skb data:000001a0: c0 a8 01 02 4a 7d 27 9b 06 25 00 50 58 65 0f b7  ....J}'..%.PXe..
[  750.555344] skb data:000001b0: 58 0c fe 7d 50 18 ff ff 34 54 00 00 65 74 3d 37  X..}P...4T..et=7
[  750.655352] skb data:000001c0: 33 30 7c 63 73 3d 61 33 79 30 67 39 72 35 0d 0a  30|cs=a3y0g9r5..
[  750.755360] skb data:000001d0: 43 6f 6f 6b 69 65 32 3a 20 24 56 65 72 73 69 6f  Cookie2: $Versio
[  750.855369] skb data:000001e0: 6e 3d 31 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a  n=1..Connection:
[  750.955377] skb data:000001f0: 20 4b 65 65 70 2d 41 6c 69 76 65 2c 20 54 45 0d   Keep-Alive, TE.
[  751.055385] skb data:00000200: 0a 54 45 3a 20 64 65 66 6c 61 74 65 2c 20 67 7a  .TE: deflate, gz
[  751.155393] skb data:00000210: 69 70 2c 20 63 68 75 6e 6b 65 64 2c 20 69 64 65  ip, chunked, ide
[  751.255402] skb data:00000220: 6e 74 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d  ntity, trailers.
[  751.355410] skb data:00000230: 0a 0d 0a 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  751.455418] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  751.555426] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  751.655434] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  751.755443] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  752.055018] ------------[ cut here ]------------
[  752.155056] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  752.255104] Hardware name: PowerEdge SC440
[  752.355152] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  752.455162] Call Trace:
[  752.655194]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  752.855215]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  753.055245]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  753.255262]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  753.455277]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  753.655294]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  753.855310]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  754.055327]  [<c1224852>] ? rtl8139_start_xmit+0x51/0x13a
[  754.255341]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  754.455357]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  754.655373]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  754.855389]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  755.055405]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  755.255420]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  755.455435]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  755.655450]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  755.855465]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  756.055479]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  756.255495]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  756.455510]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  756.655525]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  756.855541]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  757.055555]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  757.255571]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  757.455586]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  757.655601]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  757.855617]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  758.055632]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  758.255647]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  758.455662]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  758.655677]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  758.855692]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  759.055707]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  759.255720]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  759.455734]  [<c1031ead>] ? irq_exit+0x63/0x68
[  759.655747]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  759.855761]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  760.055775]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  760.255787]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  760.455801]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  760.555810] ---[ end trace df8deff2ad2a9770 ]---
[  760.655819] 100: csum_start 306, offset 16, headroom 390, headlen 88, len 88
[  760.755826] nr_frags=0 gso_size=0
[  760.855833]
[  760.955842] skb data:00000000: 00 90 92 d2 00 00 00 00 00 00 00 00 00 00 08 00  ................
[  761.055851] skb data:00000010: 45 c0 00 84 09 5b 00 00 40 01 db 0b c0 a8 0a 01  E....[..@.......
[  761.155859] skb data:00000020: c0 a8 0a 01 03 01 fc fe 00 00 00 00 45 00 00 68  ............E..h
[  761.255867] skb data:00000030: 00 00 40 00 ff 01 e5 5d c0 a8 0a 01 c0 a8 0a e5  ..@....]........
[  761.355875] skb data:00000040: 08 00 09 14 47 73 00 41 00 00 00 00 00 00 00 00  ....Gs.A........
[  761.455884] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 46 6c 6f 72  ............Flor
[  761.555892] skb data:00000060: 69 61 6e 20 46 6f 72 73 74 65 72 20 3c 6f 63 74  ian Forster <oct
[  761.655900] skb data:00000070: 6f 40 76 65 72 70 6c 61 6e 74 2e 6f 72 67 00 00  [email protected]..
[  761.755909] skb data:00000080: 00 0e 2e 5c 27 e6 00 30 05 07 f3 1a 08 00 45 00  ...\'..0......E.
[  761.855917] skb data:00000090: 00 28 51 b6 40 00 80 06 c7 5f c0 a8 c7 0f 55 0e  .(Q.@...._....U.
[  761.955926] skb data:000000a0: 04 f4 0c 41 00 50 0c c5 91 a2 c0 76 ad e2 50 10  ...A.P.....v..P.
[  762.055934] skb data:000000b0: ff 3c b5 8b 00 00 00 00 00 00 00 00 bd 6b fb 67  .<...........k.g
[  762.155942] skb data:000000c0: f4 05 fe 14 bb f0 73 7f a4 27 3f 47 39 b1 9e 78  ......s..'?G9..x
[  762.255951] skb data:000000d0: 17 94 e8 43 3f 1f dd e4 a3 db 7c a4 d7 f0 15 6b  ...C?.....|....k
[  762.355959] skb data:000000e0: 9d 16 1b 6a f0 73 a1 e5 8b 17 ad aa 1e 4e 2e 64  ...j.s.......N.d
[  762.455967] skb data:000000f0: 2e 8e ea e1 c3 41 b1 98 0b 77 aa 6b 7c d4 ab db  .....A...w.k|...
[  762.555976] skb data:00000100: 00 bc 92 d2 00 00 00 00 00 00 00 00 00 00 08 00  ................
[  762.655984] skb data:00000110: 45 c0 00 84 09 5b 00 00 40 01 db 0b c0 a8 0a 01  E....[..@.......
[  762.755992] skb data:00000120: c0 a8 0a 01 03 01 fc fe 00 00 00 00 45 00 00 68  ............E..h
[  762.856004] skb data:00000130: 00 00 40 00 ff 01 e5 5d c0 a8 0a 01 c0 a8 0a e5  ..@....]........
[  762.956012] skb data:00000140: 08 00 09 14 47 73 00 41 00 00 00 00 00 00 00 00  ....Gs.A........
[  763.056020] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 46 6c 6f 72  ............Flor
[  763.156028] skb data:00000160: 69 61 6e 20 46 6f 72 73 74 65 72 20 3c 6f 63 74  ian Forster <oct
[  763.256037] skb data:00000170: 6f 40 76 65 72 70 6c 61 6e 74 2e 6f 72 67 00 00  [email protected]..
[  763.356045] skb data:00000180: 00 15 58 8f 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ..X......\'....\
[  763.456053] skb data:00000190: 27 ef 08 00 45 00 00 4a b8 16 40 00 7f 06 a1 68  '[email protected]
[  763.556061] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 2b 00 50 ff e6 5c a2  .....2.R.+.P..\.
[  763.656070] skb data:000001b0: 6a a6 77 69 50 18 fa f0 a2 6b 00 00 70 2c 20 63  j.wiP....k..p, c
[  763.756078] skb data:000001c0: 68 75 6e 6b 65 64 2c 20 69 64 65 6e 74 69 74 79  hunked, identity
[  763.856087] skb data:000001d0: 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d 0a 00 00  , trailers......
[  763.956096] skb data:000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  764.056104] skb data:000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  764.454219] ------------[ cut here ]------------
[  764.554236] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  764.654245] Hardware name: PowerEdge SC440
[  764.754255] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  764.854263] Call Trace:
[  765.054281]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  765.254297]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  765.454313]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  765.654328]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  765.854343]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  766.054359]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  766.254374]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  766.454392]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  766.654407]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  766.854423]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  767.054438]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  767.254454]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  767.454470]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  767.654485]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  767.854500]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  768.054514]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  768.254529]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  768.454544]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  768.654560]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  768.854575]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  769.054590]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  769.254605]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  769.454620]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  769.654635]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  769.854650]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  770.054666]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  770.254681]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  770.454695]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  770.654709]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  770.854724]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  771.054738]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  771.254751]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  771.454765]  [<c1031ead>] ? irq_exit+0x63/0x68
[  771.654778]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  771.854792]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  772.054806]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  772.254820]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  772.454834]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  772.554843] ---[ end trace df8deff2ad2a9771 ]---
[  772.654851] 100: csum_start 306, offset 16, headroom 390, headlen 75, len 75
[  772.754859] nr_frags=0 gso_size=0
[  772.854866]
[  772.954875] skb data:00000000: 00 b0 27 cc 93 4a d3 70 a0 6a d9 ec d7 87 92 21  ..'..J.p.j.....!
[  773.054883] skb data:00000010: 30 ee 16 e5 ea 75 ad 30 f3 93 99 1e d1 76 0b c2  0....u.0.....v..
[  773.154892] skb data:00000020: 60 32 0c 69 62 56 99 d8 53 7d 0f d0 ab f1 a5 66  `2.ibV..S}.....f
[  773.254900] skb data:00000030: 7b a1 ae e7 18 ab 62 fa fc 87 20 f6 46 50 7d 76  {.....b... .FP}v
[  773.354908] skb data:00000040: af b9 53 21 17 56 24 b0 a1 d1 bb fc 57 0d d7 ab  ..S!.V$.....W...
[  773.454917] skb data:00000050: 97 78 b8 47 32 60 46 5e 37 af b1 95 d4 2b 25 f1  .x.G2`F^7....+%.
[  773.554925] skb data:00000060: 27 a6 25 c6 dc 7b 5b 62 89 5c f9 a2 d7 ba af da  '.%..{[b.\......
[  773.654933] skb data:00000070: a6 93 be 49 35 81 3d 6a 54 e7 8c 9a cb 3d 00 00  ...I5.=jT....=..
[  773.754942] skb data:00000080: 00 21 97 d3 fe cb 00 1a a0 38 8a 1b 08 00 45 00  .!.......8....E.
[  773.854950] skb data:00000090: 00 2f a0 85 00 00 71 11 20 3c 63 ee 59 c9 c0 a8  ./....q. <c.Y...
[  773.954958] skb data:000000a0: 0a 9d 9d 41 14 b9 00 1b 85 1a bc e9 02 53 5e 2f  ...A.........S^/
[  774.054967] skb data:000000b0: f7 80 13 fe f6 e0 7f 34 51 c2 e5 e2 69 00 00 00  .......4Q...i...
[  774.154975] skb data:000000c0: 00 00 00 32 3f 5e bc e5 01 00 00 01 00 00 00 00  ...2?^..........
[  774.254983] skb data:000000d0: 00 00 05 69 6d 67 30 33 02 77 61 02 75 73 08 6d  ...img03.wa.us.m
[  774.354992] skb data:000000e0: 61 6e 67 61 66 6f 78 03 63 6f 6d 00 00 01 00 01  angafox.com.....
[  774.455004] skb data:000000f0: 01 00 01 c0 0c 00 01 00 01 00 00 00 43 00 04 5e  ............C..^
[  774.555012] skb data:00000100: 00 bc 27 cc 00 10 00 00 00 00 00 00 d6 8f 0e c1  ..'.............
[  774.655021] skb data:00000110: c3 fb 7a 03 f2 41 b6 93 9b 0d 78 73 3c c3 07 11  ..z..A....xs<...
[  774.755029] skb data:00000120: 3f dc db 4e b8 03 5c 7a 57 c0 47 fc 2a 86 b4 08  ?..N..\zW.G.*...
[  774.855037] skb data:00000130: 7e 8b 66 bc 2d 54 c5 c9 84 ee dd 66 84 93 1b 49  ~.f.-T.....f...I
[  774.955046] skb data:00000140: c1 27 b1 75 fe 8d c3 e1 32 92 dd 99 85 41 a0 7f  .'.u....2....A..
[  775.055054] skb data:00000150: 96 24 95 6f f7 a3 2b 3d e7 e6 67 12 e9 8a 08 2d  .$.o..+=..g....-
[  775.155062] skb data:00000160: 0c f4 cf f2 c1 4e 91 a0 30 b9 24 05 ec 00 39 14  .....N..0.$...9.
[  775.255071] skb data:00000170: 7c f7 21 e2 c8 93 19 2b cf 0f eb 7b f4 98 00 00  |.!....+...{....
[  775.355079] skb data:00000180: 00 21 97 4b 5a 01 00 0e 2e 5c 27 b2 00 0e 2e 5c  .!.KZ....\'....\
[  775.455087] skb data:00000190: 27 ef 08 00 45 00 00 3d b8 67 40 00 7f 06 a1 24  '...E..=.g@....$
[  775.555096] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 1f 00 50 50 9b 3d 36  .....2.R...PP.=6
[  775.655104] skb data:000001b0: 5e 28 1c 8a 50 18 fa 8f a2 5e 00 00 64 65 6e 74  ^(..P....^..dent
[  775.755112] skb data:000001c0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  775.855120] skb data:000001d0: 0a 45 45 0d 1f 69 15 f3 b3 31 c6 9c df 9e b9 06  .EE..i...1......
[  775.955129] skb data:000001e0: 34 17 58 c8 be af 97 02 5d 82 36 73 34 dd 97 96  4.X.....].6s4...
[  776.055137] skb data:000001f0: 1e ac 90 ae 21 d0 09 eb f0 b5 53 10 ed 6a eb b7  ....!.....S..j..
[  776.186389] ------------[ cut here ]------------
[  776.286408] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  776.386417] Hardware name: PowerEdge SC440
[  776.486428] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  776.586437] Call Trace:
[  776.786454]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  776.986470]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  777.186485]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  777.386500]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  777.586514]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  777.786530]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  777.986546]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  778.186561]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  778.386576]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  778.586592]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  778.786607]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  778.986623]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  779.186638]  [<c12b5601>] ? neigh_resolve_output+0xd0/0x280
[  779.386653]  [<c12be0a5>] ? eth_header+0x0/0x9b
[  779.586668]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  779.786682]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  779.986697]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  780.186712]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  780.386735]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  780.586750]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  780.786764]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  780.986779]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  781.186793]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  781.386808]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  781.586823]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  781.786839]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  781.986854]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  782.186869]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  782.386883]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  782.586897]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  782.786911]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  782.986924]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  783.186937]  [<c1031ead>] ? irq_exit+0x63/0x68
[  783.386951]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  783.586964]  [<c10035c3>] ? do_device_not_available+0x0/0x49
[  783.786977]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  783.986992]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  784.087003] ---[ end trace df8deff2ad2a9772 ]---
[  784.187012] 100: csum_start 306, offset 16, headroom 390, headlen 75, len 75
[  784.287019] nr_frags=0 gso_size=0
[  784.387025]
[  784.487034] skb data:00000000: 00 9c 92 d2 2e 5c 27 b2 00 0e 2e 5c 27 ef 08 00  .....\'....\'...
[  784.587043] skb data:00000010: 45 00 00 68 00 00 40 00 ff 01 f8 40 c0 a8 01 02  E..h..@....@....
[  784.687051] skb data:00000020: c0 a8 01 01 08 00 98 9a b7 ec 00 41 00 00 00 00  ...........A....
[  784.787059] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  784.887067] skb data:00000040: 46 6c 6f 72 69 61 6e 20 46 6f 72 73 74 65 72 20  Florian Forster
[  784.987076] skb data:00000050: 3c 6f 63 74 6f 40 76 65 72 70 6c 61 6e 74 2e 6f  <[email protected]
[  785.087084] skb data:00000060: 72 67 3e 20 68 74 74 70 3a 2f 2f 76 65 72 70 6c  rg> http://verpl
[  785.187092] skb data:00000070: 61 6e 74 2e 6f 72 67 2f 00 00 00 00 00 00 00 00  ant.org/........
[  785.287100] skb data:00000080: 00 0e 2e 5c 27 e6 00 30 05 07 f3 1a 08 00 45 00  ...\'..0......E.
[  785.387109] skb data:00000090: 00 28 51 f5 40 00 80 06 c7 20 c0 a8 c7 0f 55 0e  .(Q.@.... ....U.
[  785.487117] skb data:000000a0: 04 f4 0c 46 00 50 0d 1e ce f0 05 98 13 27 50 10  ...F.P.......'P.
[  785.587125] skb data:000000b0: ff 3c cd 79 00 00 00 00 00 00 00 00 82 42 b1 5c  .<.y.........B.\
[  785.687133] skb data:000000c0: 34 46 b7 4f e5 16 cf 1e 4e f9 d0 af 6b 6b 62 89  4F.O....N...kkb.
[  785.787141] skb data:000000d0: 4b 20 74 72 61 69 6c 65 72 73 0d 0a 0d 0a 00 01  K trailers......
[  785.887150] skb data:000000e0: 19 4e e9 b4 b4 41 ce 1b b4 a8 fd b5 80 18 04 00  .N...A..........
[  785.987158] skb data:000000f0: fe 29 00 00 01 01 08 0a 00 00 b0 77 00 00 b0 77  .).........w...w
[  786.087166] skb data:00000100: 00 8c 92 d2 2e 5c 27 b2 00 0e 2e 5c 27 ef 08 00  .....\'....\'...
[  786.187175] skb data:00000110: 45 00 00 68 00 00 40 00 ff 01 f8 40 c0 a8 01 02  E..h..@....@....
[  786.287183] skb data:00000120: c0 a8 01 01 08 00 98 9a b7 ec 00 41 00 00 00 00  ...........A....
[  786.387191] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  786.487199] skb data:00000140: 46 6c 6f 72 69 61 6e 20 46 6f 72 73 74 65 72 20  Florian Forster
[  786.587207] skb data:00000150: 3c 6f 63 74 6f 40 76 65 72 70 6c 61 6e 74 2e 6f  <[email protected]
[  786.687215] skb data:00000160: 72 67 3e 20 68 74 74 70 3a 2f 2f 76 65 72 70 6c  rg> http://verpl
[  786.787223] skb data:00000170: 61 6e 74 2e 6f 72 67 2f 00 00 00 00 00 00 00 00  ant.org/........
[  786.887231] skb data:00000180: 00 0e 2e 5c 27 b2 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...\'....\'....\
[  786.987240] skb data:00000190: 27 ef 08 00 45 00 00 3d b8 85 40 00 7f 06 a1 06  '...E..=..@.....
[  787.087248] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 2f 00 50 f1 3f dd 4e  .....2.R./.P.?.N
[  787.187256] skb data:000001b0: 80 91 32 21 50 18 ff ff a2 5e 00 00 64 65 6e 74  ..2!P....^..dent
[  787.287264] skb data:000001c0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  787.387273] skb data:000001d0: 0a 6f 73 74 2d 63 68 65 63 6b 3d 30 2c 20 70 72  .ost-check=0, pr
[  787.487281] skb data:000001e0: 65 2d 63 68 65 63 6b 3d 30 0d 0a 50 72 61 67 6d  e-check=0..Pragm
[  787.587289] skb data:000001f0: 61 3a 20 6e 6f 2d 63 61 63 68 65 0d 0a 4c 6f 63  a: no-cache..Loc
[  788.016486] ------------[ cut here ]------------
[  788.116504] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  788.216513] Hardware name: PowerEdge SC440
[  788.316524] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  788.416532] Call Trace:
[  788.616550]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  788.816566]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  789.016581]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  789.216596]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  789.416612]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  789.616630]  [<c13311fe>] ? xfrm_lookup+0x1c/0x3a
[  789.816646]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  790.016662]  [<c1224852>] ? rtl8139_start_xmit+0x51/0x13a
[  790.216677]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  790.416692]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  790.616708]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  790.816723]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  791.016740]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  791.216756]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  791.416771]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  791.616786]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  791.816801]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  792.016816]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  792.216830]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  792.416845]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  792.616860]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  792.816876]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  793.016890]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  793.216906]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  793.416921]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  793.616936]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  793.816953]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  794.016968]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  794.216982]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  794.416997]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  794.617013]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  794.817027]  [<c1031cd4>] ? __do_softirq+0x96/0xfa
[  795.017042]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  795.217056]  [<c1031ead>] ? irq_exit+0x63/0x68
[  795.417070]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  795.617084]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  795.817099]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  796.017114]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  796.217129]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  796.317137] ---[ end trace df8deff2ad2a9773 ]---
[  796.417145] 100: csum_start 306, offset 16, headroom 390, headlen 75, len 75
[  796.517152] nr_frags=0 gso_size=0
[  796.617159]
[  796.717168] skb data:00000000: 00 b8 05 c7 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  796.817177] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  796.917186] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  797.017194] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  797.117202] skb data:00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  797.217211] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  797.317219] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  797.417227] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  797.517236] skb data:00000080: 00 0e 2e 5c 27 e6 00 30 05 07 f3 1a 08 00 45 00  ...\'..0......E.
[  797.617244] skb data:00000090: 00 28 52 0c 40 00 80 06 60 17 c0 a8 c7 0f bc 8a  .(R.@...`.......
[  797.717253] skb data:000000a0: 04 6a 0c 45 00 50 0d 05 55 80 1b ff cc ee 50 10  .j.E.P..U.....P.
[  797.817261] skb data:000000b0: fa 40 14 df 00 00 00 00 00 00 00 00 f6 da 3f 4e  .@............?N
[  797.917270] skb data:000000c0: f6 db 20 4f 4b 0d 0a 54 72 61 6e 73 66 65 72 2d  .. OK..Transfer-
[  798.017278] skb data:000000d0: 45 6e 63 6f 64 69 6e 67 3a 20 63 68 75 6e 6b 65  Encoding: chunke
[  798.117286] skb data:000000e0: 64 0d 0a 58 2d 50 6f 77 65 72 65 64 2d 42 79 3a  d..X-Powered-By:
[  798.217294] skb data:000000f0: 20 50 48 50 2f 34 2e 34 2e 34 2d 38 2b 65 74 63   PHP/4.4.4-8+etc
[  798.317303] skb data:00000100: 00 b0 05 c7 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  798.417311] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  798.517319] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  798.617327] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  798.717335] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  798.817344] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  798.917352] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  799.017361] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  799.117369] skb data:00000180: 00 16 17 5e ef ec 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...^.....\'....\
[  799.217378] skb data:00000190: 27 ef 08 00 45 00 00 3d b8 f5 40 00 7f 06 a0 96  '...E..=..@.....
[  799.317386] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 31 00 50 28 3b 73 4a  .....2.R.1.P(;sJ
[  799.417394] skb data:000001b0: 8b 40 32 30 50 18 ff ff a2 5e 00 00 64 65 6e 74  .@20P....^..dent
[  799.517403] skb data:000001c0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  799.617411] skb data:000001d0: 0a 92 5f 9e 97 93 9f 98 92 9a 02 14 b2 30 b7 04  .._..........0..
[  799.717420] skb data:000001e0: 89 65 e6 21 94 19 99 a6 02 01 00 00 00 ff ff 03  .e.!............
[  799.817428] skb data:000001f0: 00 cc a7 a3 30 5a 00 00 00 0d 0a 30 0d 0a 0d 0a  ....0Z.....0....
[  800.314252] ------------[ cut here ]------------
[  800.414269] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  800.514278] Hardware name: PowerEdge SC440
[  800.614288] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  800.714297] Call Trace:
[  800.914315]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  801.114330]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  801.314345]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  801.514360]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  801.714376]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  801.914393]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  802.114408]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  802.314424]  [<c1224852>] ? rtl8139_start_xmit+0x51/0x13a
[  802.514439]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  802.714454]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  802.914469]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  803.114483]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  803.314498]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  803.514513]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  803.714528]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  803.914543]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  804.114557]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  804.314570]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  804.514583]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  804.714596]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  804.914609]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  805.114625]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  805.314640]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  805.514654]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  805.714668]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  805.914683]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  806.114698]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  806.314713]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  806.514726]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  806.714740]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  806.914755]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  807.114769]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  807.314783]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  807.514796]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  807.714810]  [<c1031ead>] ? irq_exit+0x63/0x68
[  807.914823]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  808.114836]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  808.314849]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  808.514863]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  808.714877]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  808.814885] ---[ end trace df8deff2ad2a9774 ]---
[  808.914893] 100: csum_start 306, offset 16, headroom 390, headlen 179, len 179
[  809.014900] nr_frags=0 gso_size=0
[  809.114906]
[  809.214915] skb data:00000000: 00 a8 05 c7 00 10 00 00 00 00 00 00 00 00 00 00  ................
[  809.314923] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  809.414931] skb data:00000020: 00 00 00 00 00 00 00 00 30 30 30 30 30 30 30 39  ........00000009
[  809.514940] skb data:00000030: 30 36 49 30 31 42 32 45 41 32 45 5f 31 2e 66 73  06I01B2EA2E_1.fs
[  809.614947] skb data:00000040: 2c 53 3d 35 31 37 31 3a 32 2c 53 53 53 2c 53 01  ,S=5171:2,SSS,S.
[  809.714956] skb data:00000050: 00 00 00 00 01 69 03 64 69 72 02 62 67 00 00 01  .....i.dir.bg...
[  809.814964] skb data:00000060: 00 01 c0 0c 00 01 00 01 00 00 07 c9 00 04 c2 91  ................
[  809.914972] skb data:00000070: 3f 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ?...............
[  810.014980] skb data:00000080: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 08 00 45 00  ...\'....\'...E.
[  810.114989] skb data:00000090: 00 2e c2 18 00 00 72 11 58 b7 57 06 15 3f c0 a8  ......r.X.W..?..
[  810.214997] skb data:000000a0: 01 02 85 42 13 02 00 1a e2 be cc 12 02 3e b5 ed  ...B.........>..
[  810.315007] skb data:000000b0: 0b c6 2d 3e a7 ad 70 7c 6c af 14 ab 8e 6e 3d 63  ..->..p|l....n=c
[  810.415015] skb data:000000c0: 12 85 00 68 d3 e5 0a 44 61 74 65 3a 20 57 65 64  ...h...Date: Wed
[  810.515023] skb data:000000d0: 2c 20 32 35 20 41 75 67 20 32 30 31 30 20 30 37  , 25 Aug 2010 07
[  810.615031] skb data:000000e0: 3a 30 30 3a 34 31 20 47 4d 54 0d 0a 53 65 72 76  :00:41 GMT..Serv
[  810.715040] skb data:000000f0: 65 72 3a 20 41 70 61 63 68 65 0d 0a 56 61 72 79  er: Apache..Vary
[  810.815048] skb data:00000100: 00 b8 05 c7 00 10 00 00 00 00 00 00 00 00 00 00  ................
[  810.915056] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  811.015064] skb data:00000120: 00 00 00 00 00 00 00 00 30 30 30 30 30 30 30 39  ........00000009
[  811.115072] skb data:00000130: 30 36 49 30 31 42 32 45 41 32 45 5f 31 2e 66 73  06I01B2EA2E_1.fs
[  811.215081] skb data:00000140: 2c 53 3d 35 31 37 31 3a 32 2c 53 53 53 2c 53 01  ,S=5171:2,SSS,S.
[  811.315089] skb data:00000150: 00 00 00 00 01 69 03 64 69 72 02 62 67 00 00 01  .....i.dir.bg...
[  811.415097] skb data:00000160: 00 01 c0 0c 00 01 00 01 00 00 07 c9 00 04 c2 91  ................
[  811.515105] skb data:00000170: 3f 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ?...............
[  811.615113] skb data:00000180: 00 26 9e 0f b0 bf 00 0e 2e 5c 27 b2 00 0e 2e 5c  .&.......\'....\
[  811.715122] skb data:00000190: 27 ef 08 00 45 00 00 a5 04 e8 40 00 7f 06 4a 97  '[email protected].
[  811.815130] skb data:000001a0: c0 a8 01 02 5b 67 8e c2 04 3e 00 50 1b 94 70 c6  ....[g...>.P..p.
[  811.915138] skb data:000001b0: fd ca fd 54 50 18 ff ff ac 6b 00 00 44 25 33 42  ...TP....k..D%3B
[  812.015146] skb data:000001c0: 25 32 34 71 70 70 25 33 44 30 25 33 42 25 32 34  %24qpp%3D0%3B%24
[  812.115154] skb data:000001d0: 71 74 25 33 44 39 25 35 46 30 25 35 46 30 74 3b  qt%3D9%5F0%5F0t;
[  812.215162] skb data:000001e0: 20 41 53 50 53 45 53 53 49 4f 4e 49 44 53 43 43   ASPSESSIONIDSCC
[  812.315170] skb data:000001f0: 44 51 41 42 54 3d 41 45 48 48 44 4e 43 43 50 4c  DQABT=AEHHDNCCPL
[  812.415178] skb data:00000200: 44 42 4a 42 49 4b 46 46 41 42 4f 47 4c 42 3b 20  DBJBIKFFABOGLB;
[  812.515186] skb data:00000210: 42 49 47 69 70 53 65 72 76 65 72 53 4d 41 52 54  BIGipServerSMART
[  812.615194] skb data:00000220: 3d 36 38 38 33 32 35 33 38 36 2e 32 33 30 34 30  =688325386.23040
[  812.715203] skb data:00000230: 2e 30 30 30 30 0d 0a 0d 0a 00 00 00 00 00 00 00  .0000...........
[  812.815211] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  812.915219] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  813.015228] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  813.115237] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

Eric Dumazet

2010-08-29 07:48 UTC #35 lore raw

PATCH 1/1 net: reset ip_summed in skb_tunnel_rx()

It seems we forget to reset skb->ip_summed to CHECKSUM_NONE in various tunnels (gre, ipip, sit, ip6_tunnel), before re-entering stack.

Add ip_summed initialization in skb_tunnel_rx(), and remove it from ipmr / ip6mr

Signed-off-by Eric Dumazet <[email protected]>

---

include/net/dst.h |    1 +
net/ipv4/ipmr.c   |    1 -
net/ipv6/ip6mr.c  |    1 -
3 files changed, 1 insertion(+), 2 deletions(-)

include/net/dst.h +1 -0

index 81d1413..2ef5580 100644
@@ -237,6 +237,7 @@ static inline void skb_dst_force(struct sk_buff *skb)
  */
 static inline void skb_tunnel_rx(struct sk_buff *skb, struct net_device *dev)
 {
+	skb->ip_summed = CHECKSUM_NONE;
 	skb->dev = dev;
 	/* TODO : stats should be SMP safe */
 	dev->stats.rx_packets++;

net/ipv4/ipmr.c +0 -1

index 179fcab..70fa43b 100644
@@ -1837,7 +1837,6 @@ static int __pim_rcv(struct mr_table *mrt, struct sk_buff *skb,
 	skb_pull(skb, (u8*)encap - skb->data);
 	skb_reset_network_header(skb);
 	skb->protocol = htons(ETH_P_IP);
-	skb->ip_summed = 0;
 	skb->pkt_type = PACKET_HOST;
 
 	skb_tunnel_rx(skb, reg_dev);

net/ipv6/ip6mr.c +0 -1

index 66078da..c4d0146 100644
@@ -661,7 +661,6 @@ static int pim6_rcv(struct sk_buff *skb)
 	skb_pull(skb, (u8 *)encap - skb->data);
 	skb_reset_network_header(skb);
 	skb->protocol = htons(ETH_P_IPV6);
-	skb->ip_summed = 0;
 	skb->pkt_type = PACKET_HOST;
 
 	skb_tunnel_rx(skb, reg_dev);

Jesse Gross

2010-08-29 15:35 UTC #36 lore raw

Re: [PATCH] net: reset ip_summed in skb_tunnel_rx()

Quoted 7 lines · Eric Dumazet

It seems we forget to reset skb->ip_summed to CHECKSUM_NONE in various tunnels (gre, ipip, sit, ip6_tunnel), before re-entering stack.
Add ip_summed initialization in skb_tunnel_rx(), and remove it from ipmr / ip6mr
Signed-off-by Eric Dumazet <[email protected]>

This is intentional behavior. For CHECKSUM_COMPLETE packets we subtract off the checksum of the encapsulating headers that we remove so we can still get the benefit of checksum hardware for the inner packet as well. GRE does this using skb_postpull_rcsum(), for IPIP it is implicit because the outer IP header was already removed, etc.

Plamen Petrov

2010-08-27 08:44 UTC #37 lore raw

In reply to Eric Dumazet

Current status refresh:

Just tried 2.6.35.4, but without luck.

See the oops at http://picpaste.com/8240d73e92bd2ba25a9b4019010fcabf.jpg

Eric Dumazet and I are continuing to try and find a solution for my problem via private email.

Thanks,
Plamen

Plamen Petrov

2010-08-24 05:19 UTC #38 lore raw

In reply to Plamen Petrov

На 24.8.2010 г. 07:51, Plamen Petrov написа:

Quoted 4 lines

На 23.8.2010 г. 17:14, Jarek Poplawski написа:
Quoted 3 lines · Plamen Petrov
Quoted 2 lines
Well, so far - so good. No oopses.

...

Quoted 5 lines

Quoted 4 lines
Oopses should be replaced with warnings, so please check the syslog etc from time to time.
Jarek P.

Quoted 8 lines

So far - not so good. :(
After I left the machine, i rebooted once. Then I logged in via ssh, and turned gro back ON for the tg3 nic - and in the next 30 minutes - another reboot.
Today I will try to capture a picture of the oops/warning with my phone camera,

And here is what I've got:

[picture 7]
http://picpaste.com/31d6a54fec9e87de0d1550ee02d9c336.jpg

[picture 8]
http://picpaste.com/02db6ad8abec6281065328fb52d328cf.jpg

[picture 9]
http://picpaste.com/9fbaaa14c679f57c82e96884d3274090.jpg

Sorry for the really bad quality, even for a phone, but the problem is that I don't know when its going to happen, so... you see the results.

because contrary to what you suggest,

Quoted 3 lines

Jarek, there are no messages in the system logs.
Plamen

Ideas?

Thanks,
Plamen

Jarek Poplawski

2010-08-24 06:18 UTC #39 lore raw

On Tue, Aug 24, 2010 at 08:19:37AM +0300, Plamen Petrov wrote:
...

Quoted 8 lines

And here is what I've got:
[picture 7]
http://picpaste.com/31d6a54fec9e87de0d1550ee02d9c336.jpg
[picture 8]
http://picpaste.com/02db6ad8abec6281065328fb52d328cf.jpg

Quoted 9 lines

[picture 9]
http://picpaste.com/9fbaaa14c679f57c82e96884d3274090.jpg
Sorry for the really bad quality, even for a phone, but the problem is that I don't know when its going to happen, so... you see the results.
because contrary to what you suggest,
Quoted 1 line
Jarek, there are no messages in the system logs.

I'm extremely sorry: I missed 1 place. Anyway, it's very helpful too.

Quoted 4 lines

Quoted 2 lines
Plamen
Ideas?

Try Eric's patch, and later maybe this one.

Thanks,
Jarek P.

net/core/dev.c +34 -2

index 3721fbb..01241fc 100644
@@ -1935,6 +1935,23 @@ static inline int skb_needs_linearize(struct sk_buff *skb,
 					      illegal_highdma(dev, skb))));
 }
 
+static int skb_csum_start_bug(struct sk_buff *skb, int pos)
+{
+
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		long csstart;
+
+		csstart = skb->csum_start - skb_headroom(skb);
+		if (WARN_ON(csstart > skb_headlen(skb))) {
+			pr_warning("%d: csum_start %d, headroom %d, headlen %d\n",
+				   pos, skb->csum_start, skb_headroom(skb),
+				   skb_headlen(skb));
+			return 1;
+		}
+	}
+	return 0;
+}
+
 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 			struct netdev_queue *txq)
 {

@@ -1959,11 +1976,15 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 				goto out_kfree_skb;
 			if (skb->next)
 				goto gso;
+			if (skb_csum_start_bug(skb, 10))
+				goto out_kfree_skb;
 		} else {
 			if (skb_needs_linearize(skb, dev) &&
 			    __skb_linearize(skb))
 				goto out_kfree_skb;
 
+			if (skb_csum_start_bug(skb, 20))
+				goto out_kfree_skb;
 			/* If packet is not checksummed and device does not
 			 * support checksumming for this protocol, complete
 			 * checksumming here.

@@ -1974,10 +1995,16 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 				if (!dev_can_checksum(dev, skb) &&
 				     skb_checksum_help(skb))
 					goto out_kfree_skb;
+				if (skb_csum_start_bug(skb, 30))
+					goto out_kfree_skb;
 			}
 		}
 
-		rc = ops->ndo_start_xmit(skb, dev);
+		if (skb_csum_start_bug(skb, 40)) {
+			kfree_skb(skb);
+			rc = NETDEV_TX_OK;
+		} else
+			rc = ops->ndo_start_xmit(skb, dev);
 		if (rc == NETDEV_TX_OK)
 			txq_trans_update(txq);
 		return rc;

@@ -1997,7 +2024,12 @@ gso:
 		if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
 			skb_dst_drop(nskb);
 
-		rc = ops->ndo_start_xmit(nskb, dev);
+		if (skb_csum_start_bug(skb, 50)) {
+			kfree_skb(skb);
+			rc = NETDEV_TX_OK;
+		} else
+			rc = ops->ndo_start_xmit(nskb, dev);
+
 		if (unlikely(rc != NETDEV_TX_OK)) {
 			if (rc & ~NETDEV_TX_MASK)
 				goto out_kfree_gso_skb;