New: Machine hangs with EIP at skb_copy_and_csum_dev

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

A post-2.6.35 regression.

Might have triggered the BUG_ON() in skb_copy_and_csum_dev(). Might be a tg3 thing. Hard to tell.

It'd be really nice to get that first screenful. Sigh. How long have we had this oops-scrolls-off problem?? Perhaps you could set /proc/sys/kernel/printk_delay to 100 (it's in milliseconds) so that the oops scrolls past nice and slowly?

(responding via emailed reply-to-all)

На 20.8.2010 г. 01:21, Andrew Morton написа:

So you need the begining of the oops screen - I will try to get that with the proposed pirntk_delay setting.
But wich kernel should I use? Linus' latest tree or 2.6.35.2 ? They both fail the same way here, as far as I can say.

Thanks.

Current mainline would be best, because we'd fix the bug there first then backport the fix into -stable. But it doesn't matter a lot in this case - whatever's most convenient for you, I'd say.

На 20.8.2010 г. 08:11, Andrew Morton написа:

With the "echo 100 > /proc/sys/kernel/printk_delay" command run by /etc/rc.d/rc.local, while still on 2.6.36-rc1-00127-g763008c, I got these:

[picture 4]
http://picpaste.com/aa3e373e894179e8ba19587ed63d8104.jpg

[picture 5]
http://picpaste.com/9bc4bdc04f5a84fdaf49d6e1db23ede8.jpg

[picture 6]
http://picpaste.com/da3ccd69a0a1221bb55f48b39c4ad950.jpg

Hope the above help.

And by the way, I think you are correct that this is a post-2.6.35 thing, because 2.6.35.2 was the first to give me this kind of problems, and I can confirm that 2.6.34 does not have it, because the system was on 2.6.34.4 for the last 12 hours without problems, then just a moment ago crashed on 2.6.36-rc1-00127-g763008c, and now back on 2.6.34.4

P.S. Shouldn't "echo 100 > /proc/sys/kernel/printk_delay" be somewhere on the "How to debug a crashing kernel guide" somewhere?

Thanks!

bewdy, thanks.

        BUG_ON(csstart > skb_headlen(skb));

Hopefully that's enough for the net guys to work with.

OK.

I only just thought of it ;)

I'm thinking perhaps that we should have a lines_after_oops boot parameter which makes the console shut up N lines after someone called oops_enter().

Posting this only in the hope that it will be helpfull to the "net guys"...

root@fs:~# uname -a; lspci -vvv; ethtool -i eth1

Its a forwarding setup.

Please post

ifconfig -a
iptables -nvL
iptables -t nat -nvL
iptables -t mangle -nvL
ip route
ethtool -k eth0 (& eth1 ...)

Try to disable gro ?
ethtool -K eth0 gro off
ethtool -K eth1 gro off

На 20.8.2010 г. 09:34, Eric Dumazet написа:

Here goes the output of the above:

root@fs:~# ifconfig -a

root@fs:~# iptables -nvL

root@fs:~# iptables -t nat -nvL

root@fs:~# iptables -t mangle -nvL

root@fs:~# ip route

root@fs:~# ethtool -k eth0

root@fs:~# ethtool -k eth1

root@fs:~# ethtool -k eth2

root@fs:~# uname -a

На 20.8.2010 г. 09:34, Eric Dumazet написа:

Sorry about this, but when I dig a bit into this, I found that I had a pretty old ethtool:

So my ethtool did not know anything about gro, because by looking at
http://git.kernel.org/?p=network/ethtool/ethtool.git;a=shortlog
I saw I had this one:
2006-09-01 Jeff Garzik Release version 5. origin v5
and gro reading/setting was added "a bit later"...
2009-03-06 Jeff Garzik Get/set GRO settings.

So, I went on and compiled myself a new ethtool:

And run it on my 2.6.34.4 kernel, which produced these:

root@fs:~# ethtool -k eth0

root@fs:~# ethtool -k eth1

root@fs:~# ethtool -k eth2

root@fs:~#

But again, this is on 2.6.34.4. Could it be that "generic-receive-offload" is ON by default on my only gigabit capable tg3 card on 2.6.35.x and later?

I will try to determine this myself later, but the machine is a gateway on the local area network, and is needed from the user here...

Thanks!

На 20.8.2010 г. 10:38, Plamen Petrov написа:

...

Well, as it turns out - the commit enabling GRO for tg3 cards was from the 2.6.35 merge window:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=cb903bf4ee2d6e53210e2174d363e10698112042

The next thing I will be trying is disabling
GRO for the tg3 network card in my network
startup scripts via my shiny new ethtool... ;)

На 20.8.2010 г. 11:31, Plamen Petrov написа:
...

So, for nearly half an hour now running 2.6.36-rc1-FS-00127-g763008c:

root@fs:~# w; uname -a

And that's with the usual load, which can be found here:
http://pvp.now.im/s/drraw.cgi?Mode=view;Dashboard=1173713287.13177

The only addition I made was running
ethtool -K ethX gro off
for each network card.

Which means that I have a Broadcom Tygon 3 (tg3) which doesn't like "generic-receive-offload" at all!

With "generic-receive-offload" off, I will stay on 2.6.36-rc1-FS-00127-g763008c for as long as I can, just to confirm that in fact the "default on" setting for gro in the tg3 driver is the culprit for my troubles...

Where to next?

First, thanks a lot for finding this so fast.

Hmmm, please check that on your net/core/dev.c file, line 3146 you have the fix added in

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=e5093aec2e6b60c3df2420057ffab9ed4a6d2792

commit e5093aec2e6b60c3df2420057ffab9ed4a6d2792

Author: Jarek Poplawski <[email protected]>
Date:   Wed Aug 11 02:02:10 2010 +0000

net: Fix a memmove bug in dev_gro_receive()

--skb_shinfo(skb)->nr_frags * sizeof(skb_frag_t));

If yes, I think David & Herbert will probably have to dig again into gro code :(

На 20.8.2010 г. 13:26, Eric Dumazet написа:

...

For nothing.

Yes, I confirm I have the above code in place.

So, I guess its David and Herbert's turn?...

Thanks!
Plamen

Plamen Petrov wrote, On 20.08.2010 12:53:

If you're bored in the meantime I'd suggest to do check the realtek driver eg:
- for locking with the patch below,
- to turn off with ethtool its tx-checksumming and/or scatter-gather, or if possible try to reproduce this with other NIC.

Thanks,
Jarek P.
---

index f5166dc..aaaccc5 100644
@@ -1692,6 +1692,8 @@ static netdev_tx_t rtl8139_start_xmit (struct sk_buff *skb,
 	unsigned int len = skb->len;
 	unsigned long flags;
 
+	spin_lock_irqsave(&tp->lock, flags);
+
 	/* Calculate the next Tx descriptor entry. */
 	entry = tp->cur_tx % NUM_TX_DESC;
 

@@ -1700,14 +1702,14 @@ static netdev_tx_t rtl8139_start_xmit (struct sk_buff *skb,
 		if (len < ETH_ZLEN)
 			memset(tp->tx_buf[entry], 0, ETH_ZLEN);
 		skb_copy_and_csum_dev(skb, tp->tx_buf[entry]);
-		dev_kfree_skb(skb);
+		dev_kfree_skb_irq(skb);
 	} else {
+		spin_unlock_irqrestore(&tp->lock, flags);
 		dev_kfree_skb(skb);
 		dev->stats.tx_dropped++;
 		return NETDEV_TX_OK;
 	}
 
-	spin_lock_irqsave(&tp->lock, flags);
 	/*
 	 * Writing to TxStatus triggers a DMA transfer of the data
 	 * copied to tp->tx_buf[entry] above. Use a memory barrier

After rethinking, it's almost impossible this patch could change anything here, so don't bother, but consider mainly the second proposal.

Jarek P.

Indeed ;)

Its true that not many nics use the skb_copy_and_csum_dev() helper, maybe this one must be updated somehow ?

Yes, it seems it should be possible at least to handle the bug with a warning and error return, considering Plamen's problems with getting the trace.

Jarek P.

На 21.8.2010 г. 11:07, Jarek Poplawski написа:

Well, here is the current status:

Last I promised I will stay on 2.6.36-rc1-git for as long as possible, so here is what I achieved:

root@fs:/boot# w; uname -a

Yeah, 3 days and counting, right until I decided to try the freshly announced 2.6.36-rc2.

So I upgraded the kernel, but left the scripts that turn GRO off for the tg3 card still run at system startup. This way the system ran for 2 and a half hours, when I decided its time to try turning GRO on.

I first tried to turn GRO on for the tg3 nic, and the system oopsed immediately (if the panic screen is necessary - please, ask for it).

After the system came back, I tried turning GRO on for the 2 RealTek 8139 nics, too, but ethtool only accepted turning GRO off.

And unfortunately, I can't test if other nics will fail the same way as the motherboard integrated tg3 I have does, so for now, this is only a tg3 + GRO on problem; I don't have any other hardware to test with available.

Thanks,
Plamen

There was no change in latest kernel in this area.

Should you have only tg3 cards, I guess there would be no bug.

Bug is probably a combination of :

1) tg3 + GRO , or any card enabling GRO
2) Some network code (netfilter ?)
3) a 8139too, or any card calling skb_copy_and_csum_dev()

...

A little misunderstanding: I was intersted with turning off some features on realteks to change the packet path from tg3 with gro to realtek without gro and without tx-checksumming etc.

But maybe you could try the patch below instead (so the patched kernel, tg3 with gro on, and realteks without any change).

Thanks,
Jarek P.

--- (for debugging only)

index 3721fbb..51823cd 100644
@@ -1935,6 +1935,23 @@ static inline int skb_needs_linearize(struct sk_buff *skb,
 					      illegal_highdma(dev, skb))));
 }
 
+static int skb_csum_start_bug(struct sk_buff *skb)
+{
+
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		long csstart;
+
+		csstart = skb->csum_start - skb_headroom(skb);
+		if (WARN_ON(csstart > skb_headlen(skb))) {
+			pr_warning("csum_start %d, headroom %d, headlen %d\n",
+				   skb->csum_start, skb_headroom(skb),
+				   skb_headlen(skb));
+			return 1;
+		}
+	}
+	return 0;
+}
+
 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 			struct netdev_queue *txq)
 {

@@ -1955,11 +1972,13 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 		skb_orphan_try(skb);
 
 		if (netif_needs_gso(dev, skb)) {
+			skb_csum_start_bug(skb);
 			if (unlikely(dev_gso_segment(skb)))
 				goto out_kfree_skb;
 			if (skb->next)
 				goto gso;
 		} else {
+			skb_csum_start_bug(skb);
 			if (skb_needs_linearize(skb, dev) &&
 			    __skb_linearize(skb))
 				goto out_kfree_skb;

@@ -1997,7 +2016,12 @@ gso:
 		if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
 			skb_dst_drop(nskb);
 
-		rc = ops->ndo_start_xmit(nskb, dev);
+		if (skb_csum_start_bug(skb)) {
+			kfree_skb(skb);
+			rc = NETDEV_TX_OK;
+		} else
+			rc = ops->ndo_start_xmit(nskb, dev);
+
 		if (unlikely(rc != NETDEV_TX_OK)) {
 			if (rc & ~NETDEV_TX_MASK)
 				goto out_kfree_gso_skb;

On Mon, Aug 23, 2010 at 03:00:43PM +0200, Eric Dumazet wrote:
...

Feel free to send it: I'm a bit in hurry now...

Jarek P.

На 23.8.2010 г. 16:10, Jarek Poplawski написа:

Currently compiling the kernel with the attached patch.

And, by the way, if there are any patches to follow, would you please, send them as attachments? I think my mail client is line-wrapping the patches...

The results will be sent soon, too.

Plamen

index 3721fbb..24a53d0 100644
@@ -1935,6 +1935,26 @@ static inline int skb_needs_linearize(struct sk_buff *skb,
 					      illegal_highdma(dev, skb))));
 }
 
+static int skb_csum_start_bug(struct sk_buff *skb)
+{
+
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		long csstart;
+		
+		csstart = skb->csum_start - skb_headroom(skb);
+		if (WARN_ON(csstart > skb_headlen(skb))) {
+		    pr_warning("csum_start %u, offset %u, headroom %d, headlen %d, len %d\n",
+			skb->csum_start,
+			skb->csum_offset,
+			skb_headroom(skb),
+			skb_headlen(skb),
+			skb->len);
+		    return 1;
+		}
+	}
+	return 0;
+}
+
 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 			struct netdev_queue *txq)
 {

@@ -1955,11 +1975,13 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 		skb_orphan_try(skb);
 
 		if (netif_needs_gso(dev, skb)) {
+			skb_csum_start_bug(skb);
 			if (unlikely(dev_gso_segment(skb)))
 				goto out_kfree_skb;
 			if (skb->next)
 				goto gso;
 		} else {
+			skb_csum_start_bug(skb);
 			if (skb_needs_linearize(skb, dev) &&
 			    __skb_linearize(skb))
 				goto out_kfree_skb;

@@ -1997,7 +2019,12 @@ gso:
 		if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
 			skb_dst_drop(nskb);
 
-		rc = ops->ndo_start_xmit(nskb, dev);
+		if (skb_csum_start_bug(skb)) {
+		    kfree_skb(skb);
+		    rc = NETDEV_TX_OK;
+		} else
+		    rc = ops->ndo_start_xmit(nskb, dev);
+		    
 		if (unlikely(rc != NETDEV_TX_OK)) {
 			if (rc & ~NETDEV_TX_MASK)
 				goto out_kfree_gso_skb;

На 23.8.2010 г. 16:43, Plamen Petrov написа:

Well, so far - so good. No oopses.

I think I'm hitting some compiler related issue here...

The kernel with the patch I sent applied is working pretty fine even with "generic-receive-offload" ON for my tg3 nic...

What is the big difference there?

Or do we need to dig into the object files now? Because that's what's too scary for me...

Plamen

P.S. The compiler I am using is :

root@crux:~# gcc -v

Anything obviously special about it?

Oopses should be replaced with warnings, so please check the syslog etc from time to time.

Jarek P.

На 23.8.2010 г. 17:14, Jarek Poplawski написа:

So far - not so good. :(

After I left the machine, i rebooted once. Then I logged in via ssh, and turned gro back ON for the tg3 nic - and in the next 30 minutes - another reboot.

Today I will try to capture a picture of the oops/warning with my phone camera, because contrary to what you suggest, Jarek, there are no messages in the system logs.

Plamen

Hmm... I was thinking adding a call __skb_linearize(), just in case...

index f5166dc..10928a2 100644
@@ -1696,7 +1696,7 @@ static netdev_tx_t rtl8139_start_xmit (struct sk_buff *skb,
 	entry = tp->cur_tx % NUM_TX_DESC;
 
 	/* Note: the chip doesn't have auto-pad! */
-	if (likely(len < TX_BUF_SIZE)) {
+	if (likely(len < TX_BUF_SIZE && !__skb_linearize(skb))) {
 		if (len < ETH_ZLEN)
 			memset(tp->tx_buf[entry], 0, ETH_ZLEN);
 		skb_copy_and_csum_dev(skb, tp->tx_buf[entry]);

На 24.8.2010 г. 07:51, Plamen Petrov написа:

...

And here is what I've got:

[picture 7]
http://picpaste.com/31d6a54fec9e87de0d1550ee02d9c336.jpg

[picture 8]
http://picpaste.com/02db6ad8abec6281065328fb52d328cf.jpg

[picture 9]
http://picpaste.com/9fbaaa14c679f57c82e96884d3274090.jpg

Sorry for the really bad quality, even for a phone, but the problem is that I don't know when its going to happen, so... you see the results.

because contrary to what you suggest,

Ideas?

Thanks,
Plamen

On Tue, Aug 24, 2010 at 08:19:37AM +0300, Plamen Petrov wrote:
...

I'm extremely sorry: I missed 1 place. Anyway, it's very helpful too.

Try Eric's patch, and later maybe this one.

Thanks,
Jarek P.

index 3721fbb..01241fc 100644
@@ -1935,6 +1935,23 @@ static inline int skb_needs_linearize(struct sk_buff *skb,
 					      illegal_highdma(dev, skb))));
 }
 
+static int skb_csum_start_bug(struct sk_buff *skb, int pos)
+{
+
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		long csstart;
+
+		csstart = skb->csum_start - skb_headroom(skb);
+		if (WARN_ON(csstart > skb_headlen(skb))) {
+			pr_warning("%d: csum_start %d, headroom %d, headlen %d\n",
+				   pos, skb->csum_start, skb_headroom(skb),
+				   skb_headlen(skb));
+			return 1;
+		}
+	}
+	return 0;
+}
+
 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 			struct netdev_queue *txq)
 {

@@ -1959,11 +1976,15 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 				goto out_kfree_skb;
 			if (skb->next)
 				goto gso;
+			if (skb_csum_start_bug(skb, 10))
+				goto out_kfree_skb;
 		} else {
 			if (skb_needs_linearize(skb, dev) &&
 			    __skb_linearize(skb))
 				goto out_kfree_skb;
 
+			if (skb_csum_start_bug(skb, 20))
+				goto out_kfree_skb;
 			/* If packet is not checksummed and device does not
 			 * support checksumming for this protocol, complete
 			 * checksumming here.

@@ -1974,10 +1995,16 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 				if (!dev_can_checksum(dev, skb) &&
 				     skb_checksum_help(skb))
 					goto out_kfree_skb;
+				if (skb_csum_start_bug(skb, 30))
+					goto out_kfree_skb;
 			}
 		}
 
-		rc = ops->ndo_start_xmit(skb, dev);
+		if (skb_csum_start_bug(skb, 40)) {
+			kfree_skb(skb);
+			rc = NETDEV_TX_OK;
+		} else
+			rc = ops->ndo_start_xmit(skb, dev);
 		if (rc == NETDEV_TX_OK)
 			txq_trans_update(txq);
 		return rc;

@@ -1997,7 +2024,12 @@ gso:
 		if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
 			skb_dst_drop(nskb);
 
-		rc = ops->ndo_start_xmit(nskb, dev);
+		if (skb_csum_start_bug(skb, 50)) {
+			kfree_skb(skb);
+			rc = NETDEV_TX_OK;
+		} else
+			rc = ops->ndo_start_xmit(nskb, dev);
+
 		if (unlikely(rc != NETDEV_TX_OK)) {
 			if (rc & ~NETDEV_TX_MASK)
 				goto out_kfree_gso_skb;

На 24.8.2010 г. 08:01, Eric Dumazet написа:

...

На 24.8.2010 г. 11:43, Plamen Petrov написа:

...

The current status: if I enable GRO on the tg3 - the kernel oopses. It just takes a different amount of time to trigger: somewhere from 30 seconds to 30 minutes.

The oopses looks the same, and here are the latest:

[picture 13]
http://picpaste.com/c8dbda8f5c15d9ce3e050dd7f245f5d0.jpg

[picture 14]
http://picpaste.com/646cca586b704c5b72d3cf9fa54c7344.jpg

I was wondering which debug options could help us track this down?

Thanks,
Plamen

Thanks, here is an updated patch (against linux-2.6)

index 3721fbb..77c8eb7 100644
@@ -1935,6 +1935,32 @@ static inline int skb_needs_linearize(struct sk_buff *skb,
 					      illegal_highdma(dev, skb))));
 }
 
+int skb_csum_start_bug(const struct sk_buff *skb, int pos)
+{
+
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		long csstart;
+
+		csstart = skb->csum_start - skb_headroom(skb);
+		if (WARN_ON(csstart > skb_headlen(skb))) {
+			int i;
+
+			pr_err("%d: csum_start %u, offset %u, headroom %d, headlen %d, len %d\n",
+				   pos, skb->csum_start, skb->csum_offset, skb_headroom(skb),
+				   skb_headlen(skb), skb->len);
+			pr_err("nr_frags=%u gso_size=%u ",
+					skb_shinfo(skb)->nr_frags,
+					skb_shinfo(skb)->gso_size);
+			for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
+				pr_err("frag_size=%u ", skb_shinfo(skb)->frags[i].size);
+			}
+			pr_err("\n");
+			return 1;
+		}
+	}
+	return 0;
+}
+
 int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 			struct netdev_queue *txq)
 {

@@ -1959,11 +1985,15 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 				goto out_kfree_skb;
 			if (skb->next)
 				goto gso;
+			if (skb_csum_start_bug(skb, 10))
+				goto out_kfree_skb;
 		} else {
 			if (skb_needs_linearize(skb, dev) &&
 			    __skb_linearize(skb))
 				goto out_kfree_skb;
 
+			if (skb_csum_start_bug(skb, 20))
+				goto out_kfree_skb;
 			/* If packet is not checksummed and device does not
 			 * support checksumming for this protocol, complete
 			 * checksumming here.

@@ -1974,10 +2004,16 @@ int dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
 				if (!dev_can_checksum(dev, skb) &&
 				     skb_checksum_help(skb))
 					goto out_kfree_skb;
+				if (skb_csum_start_bug(skb, 30))
+					goto out_kfree_skb;
 			}
 		}
 
-		rc = ops->ndo_start_xmit(skb, dev);
+		if (skb_csum_start_bug(skb, 40)) {
+			kfree_skb(skb);
+			rc = NETDEV_TX_OK;
+		} else
+			rc = ops->ndo_start_xmit(skb, dev);
 		if (rc == NETDEV_TX_OK)
 			txq_trans_update(txq);
 		return rc;

@@ -1997,7 +2033,12 @@ gso:
 		if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
 			skb_dst_drop(nskb);
 
-		rc = ops->ndo_start_xmit(nskb, dev);
+		if (skb_csum_start_bug(skb, 50)) {
+			kfree_skb(skb);
+			rc = NETDEV_TX_OK;
+		} else
+			rc = ops->ndo_start_xmit(nskb, dev);
+
 		if (unlikely(rc != NETDEV_TX_OK)) {
 			if (rc & ~NETDEV_TX_MASK)
 				goto out_kfree_gso_skb;

index 3a2513f..3d54a1b 100644
@@ -1824,13 +1824,15 @@ void skb_copy_and_csum_dev(const struct sk_buff *skb, u8 *to)
 {
 	__wsum csum;
 	long csstart;
+	extern int skb_csum_start_bug(const struct sk_buff *skb, int pos);
 
 	if (skb->ip_summed == CHECKSUM_PARTIAL)
 		csstart = skb->csum_start - skb_headroom(skb);
 	else
 		csstart = skb_headlen(skb);
 
-	BUG_ON(csstart > skb_headlen(skb));
+	if (skb_csum_start_bug(skb, 100))
+		return;
 
 	skb_copy_from_linear_data(skb, to, csstart);
 

Eric Dumazet написа:

[   10.721802] Ending clean XFS mount for filesystem: md12
[   11.669013] IPv4 FIB: Using LC-trie version 0.409
[   11.669101] eth2: link up, 100Mbps, full-duplex, lpa 0x45E1
[   11.746792] eth0: link up, 100Mbps, full-duplex, lpa 0x41E1
[   11.757230] tg3 0000:04:00.0: irq 44 for MSI/MSI-X
[   11.810133] ADDRCONF(NETDEV_UP): eth1: link is not ready
[   11.957523] sixxs_t: Disabled Privacy Extensions
[   14.843711] tg3 0000:04:00.0: eth1: Link is up at 1000 Mbps, full duplex
[   14.843717] tg3 0000:04:00.0: eth1: Flow control is on for TX and on for RX
[   14.843753] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   15.854861] tun0: Disabled Privacy Extensions
[  699.375620] ------------[ cut here ]------------
[  699.475648] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0xf2()
[  699.575667] Hardware name: PowerEdge SC440
[  699.675688] Pid: 2963, comm: FahCore_78.exe Not tainted 2.6.36-rc2-FS-00103-g2d6fa25 #1
[  699.775706] Call Trace:
[  699.975744]  [<c102d86c>] ? warn_slowpath_common+0x67/0x8c
[  700.175779]  [<c12abc76>] ? skb_csum_start_bug+0x46/0xf2
[  700.375813]  [<c12abc76>] ? skb_csum_start_bug+0x46/0xf2
[  700.575848]  [<c102d8ac>] ? warn_slowpath_null+0x1b/0x1f
[  700.775882]  [<c12abc76>] ? skb_csum_start_bug+0x46/0xf2
[  700.975918]  [<c1024569>] ? __wake_up_sync_key+0x3c/0x52
[  701.175953]  [<c12a7bab>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  701.375989]  [<c122483b>] ? rtl8139_start_xmit+0x4a/0x13a
[  701.576026]  [<c12ae29e>] ? dev_hard_start_xmit+0x220/0x4cc
[  701.776062]  [<c12bfbed>] ? sch_direct_xmit+0xac/0x174
[  701.976096]  [<c12c3f69>] ? nf_iterate+0x69/0x7c
[  702.176131]  [<c12e8976>] ? ip_finish_output+0x0/0x2b6
[  702.376165]  [<c12b00eb>] ? dev_queue_xmit+0xc7/0x355
[  702.576198]  [<c12e8976>] ? ip_finish_output+0x0/0x2b6
[  702.776232]  [<c12e8a92>] ? ip_finish_output+0x11c/0x2b6
[  702.976266]  [<c12e8f11>] ? ip_output+0xa4/0xc3
[  703.176299]  [<c12e8976>] ? ip_finish_output+0x0/0x2b6
[  703.376332]  [<c12e4ff9>] ? ip_forward_finish+0x39/0x44
[  703.576365]  [<c12e3a38>] ? ip_rcv_finish+0xe8/0x39f
[  703.776398]  [<c12ad0fd>] ? __netif_receive_skb+0x237/0x2b3
[  703.976431]  [<c12ad70b>] ? netif_receive_skb+0x5f/0x64
[  704.176464]  [<c12ad75e>] ? napi_gro_complete+0x4e/0x94
[  704.376497]  [<c12ada9a>] ? dev_gro_receive+0x158/0x1f5
[  704.576530]  [<c12adc84>] ? napi_gro_receive+0x16/0x1f
[  704.776563]  [<c1217efb>] ? tg3_poll_work+0x5bc/0xbfb
[  704.976597]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  705.176631]  [<c121ce68>] ? tg3_poll+0x43/0x194
[  705.376665]  [<c12ad8b3>] ? net_rx_action+0xcc/0x15b
[  705.576699]  [<c1031cad>] ? __do_softirq+0x7f/0xfa
[  705.776733]  [<c1053dc9>] ? handle_IRQ_event+0x48/0xa6
[  705.976767]  [<c105689b>] ? move_native_irq+0x9/0x3e
[  706.176799]  [<c1031d4f>] ? do_softirq+0x27/0x2a
[  706.376832]  [<c1031e9d>] ? irq_exit+0x63/0x68
[  706.576864]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  706.776897]  [<c1031e6b>] ? irq_exit+0x31/0x68
[  706.976930]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  707.176963]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  707.276981] ---[ end trace 75e4f8534893c910 ]---
[  707.376998] 100: csum_start 306, offset 16, headroom 390, headlen 70, len 70
[  707.477015] nr_frags=0 gso_size=0
[  707.577031]
[ 1012.931455] ------------[ cut here ]------------
[ 1013.031482] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0xf2()
[ 1013.131501] Hardware name: PowerEdge SC440
[ 1013.231521] Pid: 2963, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00103-g2d6fa25 #1
[ 1013.331538] Call Trace:
[ 1013.531575]  [<c102d86c>] ? warn_slowpath_common+0x67/0x8c
[ 1013.731608]  [<c12abc76>] ? skb_csum_start_bug+0x46/0xf2
[ 1013.931641]  [<c12abc76>] ? skb_csum_start_bug+0x46/0xf2
[ 1014.131675]  [<c102d8ac>] ? warn_slowpath_null+0x1b/0x1f
[ 1014.331708]  [<c12abc76>] ? skb_csum_start_bug+0x46/0xf2
[ 1014.531742]  [<c1024569>] ? __wake_up_sync_key+0x3c/0x52
[ 1014.731775]  [<c12a7bab>] ? skb_copy_and_csum_dev+0x2a/0xaf
[ 1014.931809]  [<c122483b>] ? rtl8139_start_xmit+0x4a/0x13a
[ 1015.131841]  [<c12ae29e>] ? dev_hard_start_xmit+0x220/0x4cc
[ 1015.331875]  [<c12bfbed>] ? sch_direct_xmit+0xac/0x174
[ 1015.531908]  [<c12c3f69>] ? nf_iterate+0x69/0x7c
[ 1015.731941]  [<c12e8976>] ? ip_finish_output+0x0/0x2b6
[ 1015.931973]  [<c12b00eb>] ? dev_queue_xmit+0xc7/0x355
[ 1016.132007]  [<c12e8976>] ? ip_finish_output+0x0/0x2b6
[ 1016.332039]  [<c12e8a92>] ? ip_finish_output+0x11c/0x2b6
[ 1016.532071]  [<c12e8f11>] ? ip_output+0xa4/0xc3
[ 1016.732103]  [<c12e8976>] ? ip_finish_output+0x0/0x2b6
[ 1016.932135]  [<c12e4ff9>] ? ip_forward_finish+0x39/0x44
[ 1017.132166]  [<c12e3a38>] ? ip_rcv_finish+0xe8/0x39f
[ 1017.332198]  [<c12ad0fd>] ? __netif_receive_skb+0x237/0x2b3
[ 1017.532230]  [<c12ad70b>] ? netif_receive_skb+0x5f/0x64
[ 1017.732262]  [<c12ad75e>] ? napi_gro_complete+0x4e/0x94
[ 1017.932294]  [<c12ada9a>] ? dev_gro_receive+0x158/0x1f5
[ 1018.132326]  [<c12adc84>] ? napi_gro_receive+0x16/0x1f
[ 1018.332358]  [<c1217efb>] ? tg3_poll_work+0x5bc/0xbfb
[ 1018.532392]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[ 1018.732424]  [<c121ce68>] ? tg3_poll+0x43/0x194
[ 1018.932456]  [<c12ad8b3>] ? net_rx_action+0xcc/0x15b
[ 1019.132489]  [<c1031cad>] ? __do_softirq+0x7f/0xfa
[ 1019.332522]  [<c1053dc9>] ? handle_IRQ_event+0x48/0xa6
[ 1019.532554]  [<c105689b>] ? move_native_irq+0x9/0x3e
[ 1019.732586]  [<c1031d4f>] ? do_softirq+0x27/0x2a
[ 1019.932617]  [<c1031e9d>] ? irq_exit+0x63/0x68
[ 1020.132648]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[ 1020.332680]  [<c1031e6b>] ? irq_exit+0x31/0x68
[ 1020.532713]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[ 1020.732745]  [<c1002d29>] ? common_interrupt+0x29/0x30
[ 1020.932777]  [<c1390000>] ? quirk_io_region+0x1c/0x91
[ 1021.032794] ---[ end trace 75e4f8534893c911 ]---
[ 1021.132812] 100: csum_start 306, offset 16, headroom 390, headlen 153, len 153
[ 1021.232828] nr_frags=0 gso_size=0
[ 1021.332844] 

...

Good question. IMHO it looks like skbs are overwritten, so better turn off gro until some next patch.

Thanks a lot, Plamen and Eric, too!

Jarek P.

Thanks !

csum_offset = 16.

so its offsetof(struct tcphdr, check)

maybe a bug in net/ipv4/netfilter/nf_nat_helper.c ?

We should trace all spots where we set csum_start/csum_offset

Or/And trace the skb content.

Please add a :

print_hex_dump(KERN_ERR, "skb data:", DUMP_PREFIX_OFFSET,

               16, 1, skb->head, skb_end_pointer(skb)-skb->head,true);

call in skb_csum_start_bug(), right after the pr_err("\n") and before the "return 1;"

int skb_csum_start_bug(const struct sk_buff *skb, int pos) {

        if (skb->ip_summed == CHECKSUM_PARTIAL) {
                long csstart;

                csstart = skb->csum_start - skb_headroom(skb);
                if (WARN_ON(csstart > skb_headlen(skb))) {
                        int i;

                        pr_err("%d: csum_start %u, offset %u, headroom %d, headlen %d, len %d\n",
                                   pos, skb->csum_start, skb->csum_offset, skb_headroom(skb),
                                   skb_headlen(skb), skb->len);
                        pr_err("nr_frags=%u gso_size=%u ",
                                        skb_shinfo(skb)->nr_frags,
                                        skb_shinfo(skb)->gso_size);
                        for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) {
                                pr_err("frag_size=%u ", skb_shinfo(skb)->frags[i].size);
                        }
                        pr_err("\n");
                        print_hex_dump(KERN_ERR, "skb data:", DUMP_PREFIX_OFFSET,
                                16, 1, skb->head, skb_end_pointer(skb) - skb->head, true);
                        return 1;
                }
        }
        return 0;
}

На 24.8.2010 г. 22:19, Eric Dumazet написа:

Done! See the results below.

I see you liked the previous one, here's some more. ;)

This one is based on Linus' latest tree,
hence the ID "2.6.36-rc2-FS-00210-geedff42".

The rest is in the attached file, in case you need to run it trough some debugging app...

Thanks,
Plamen

[   10.510191] XFS mounting filesystem md12
[   10.693540] Ending clean XFS mount for filesystem: md12
[   11.592737] IPv4 FIB: Using LC-trie version 0.409
[   11.592827] eth2: link up, 100Mbps, full-duplex, lpa 0x45E1
[   11.677311] eth0: link up, 100Mbps, full-duplex, lpa 0x41E1
[   11.687604] tg3 0000:04:00.0: irq 44 for MSI/MSI-X
[   11.719166] ADDRCONF(NETDEV_UP): eth1: link is not ready
[   11.845858] sixxs_t: Disabled Privacy Extensions
[   14.815688] tg3 0000:04:00.0: eth1: Link is up at 1000 Mbps, full duplex
[   14.815693] tg3 0000:04:00.0: eth1: Flow control is on for TX and on for RX
[   14.815740] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   15.470040] tun0: Disabled Privacy Extensions
[  310.470021] ------------[ cut here ]------------
[  310.570041] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  310.670050] Hardware name: PowerEdge SC440
[  310.770060] Pid: 2960, comm: FahCore_78.exe Not tainted 2.6.36-rc2-FS-00210-geedff42 #1
[  310.870069] Call Trace:
[  311.070087]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  311.270103]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  311.470126]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  311.670144]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  311.870167]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  312.070191]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  312.270207]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  312.470224]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  312.670238]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  312.870253]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  313.070268]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  313.270283]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  313.470297]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  313.670312]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  313.870326]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  314.070341]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  314.270355]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  314.470370]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  314.670385]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  314.870399]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  315.070413]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  315.270427]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  315.470440]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  315.670454]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  315.870468]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  316.070483]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  316.270498]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  316.470512]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  316.670526]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  316.870541]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  317.070555]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  317.270569]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  317.470582]  [<c1031ead>] ? irq_exit+0x63/0x68
[  317.670596]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  317.870610]  [<c10035c3>] ? do_device_not_available+0x0/0x49
[  318.070624]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  318.270639]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  318.370647] ---[ end trace df8deff2ad2a9760 ]---
[  318.470656] 100: csum_start 306, offset 16, headroom 390, headlen 151, len 151
[  318.570664] nr_frags=0 gso_size=0
[  318.670671]
[  318.770680] skb data:00000000: 00 a4 27 cc 17 5e ef ec 00 1a a0 38 8a 1b 08 00  ..'..^.....8....
[  318.870688] skb data:00000010: 45 00 00 b7 00 00 40 00 40 11 a4 62 c0 a8 0a 01  E.....@[email protected]....
[  318.970697] skb data:00000020: c0 a8 0a 82 00 35 f2 69 00 a3 96 88 d3 4e 81 80  .....5.i.....N..
[  319.070706] skb data:00000030: 00 01 00 04 00 00 00 00 06 61 6b 61 6d 61 69 0d  .........akamai.
[  319.170714] skb data:00000040: 73 6d 61 72 74 61 64 73 65 72 76 65 72 03 63 6f  smartadserver.co
[  319.270723] skb data:00000050: 6d 00 00 01 00 01 c0 0c 00 05 00 01 00 00 81 0a  m...............
[  319.370731] skb data:00000060: 00 28 06 61 6b 61 6d 61 69 0d 73 6d 61 72 74 61  .(.akamai.smarta
[  319.470739] skb data:00000070: 64 73 65 72 76 65 72 03 63 6f 6d 09 65 64 67 65  dserver.com.edge
[  319.570747] skb data:00000080: 73 75 69 74 65 03 6e 65 74 00 c0 36 00 00 00 00  suite.net..6....
[  319.670756] skb data:00000090: 00 00 00 00 00 00 00 00 00 00 08 00 45 00 00 34  ............E..4
[  319.770764] skb data:000000a0: 0d 39 40 00 40 06 2f 89 7f 00 00 01 7f 00 00 01  .9@.@./.........
[  319.870772] skb data:000000b0: be 75 19 4e 14 43 0f 38 14 5d 49 65 00 00 00 00  .u.N.C.8.]Ie....
[  319.970780] skb data:000000c0: 00 00 00 00 00 00 00 00 00 00 08 00 45 00 00 57  ............E..W
[  320.070789] skb data:000000d0: 4b 40 40 00 40 06 f1 5e 7f 00 00 01 7f 00 00 01  K@@.@..^........
[  320.170797] skb data:000000e0: 19 4e be 75 14 5d 49 65 14 43 0f 38 80 18 04 00  .N.u.]Ie.C.8....
[  320.270805] skb data:000000f0: fe 4b 00 00 01 01 08 0a 00 00 02 03 00 00 02 03  .K..............
[  320.370813] skb data:00000100: 00 a8 27 cc 00 00 00 00 00 00 00 00 00 00 00 00  ..'.............
[  320.470821] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.570829] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.670837] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.770845] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.870853] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  320.970862] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  321.070870] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  321.170878] skb data:00000180: 00 0a e6 ac 07 db 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  321.270887] skb data:00000190: 27 ef 08 00 45 00 00 89 09 27 40 00 7f 06 46 74  '...E....'@...Ft
[  321.370895] skb data:000001a0: c0 a8 01 02 5b 67 8e c2 04 8f 00 50 98 49 d7 bc  ....[g.....P.I..
[  321.470903] skb data:000001b0: ff 40 e9 4a 50 18 ff ff ac 4f 00 00 33 42 25 32  [email protected]%2
[  321.570911] skb data:000001c0: 34 73 68 25 33 44 33 25 33 42 25 32 34 73 77 25  4sh%3D3%3B%24sw%
[  321.670919] skb data:000001d0: 33 44 33 3b 20 70 69 64 3d 35 30 32 31 37 34 33  3D3; pid=5021743
[  321.770927] skb data:000001e0: 34 32 30 33 31 30 32 39 39 37 38 33 0d 0a 43 6f  420310299783..Co
[  321.870936] skb data:000001f0: 6f 6b 69 65 32 3a 20 24 56 65 72 73 69 6f 6e 3d  okie2: $Version=
[  321.970944] skb data:00000200: 31 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 4b  1..Connection: K
[  322.070952] skb data:00000210: 65 65 70 2d 41 6c 69 76 65 0d 0a 0d 0a 00 00 00  eep-Alive.......
[  322.170960] skb data:00000220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.270968] skb data:00000230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.370977] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.470985] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.570993] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  322.671004] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  324.960350] ------------[ cut here ]------------
[  325.060369] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  325.160378] Hardware name: PowerEdge SC440
[  325.260392] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  325.360405] Call Trace:
[  325.560423]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  325.760438]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  325.960453]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  326.160469]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  326.360485]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  326.560502]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  326.760518]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  326.960535]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  327.160550]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  327.360566]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  327.560581]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  327.760598]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  327.960614]  [<c12b5601>] ? neigh_resolve_output+0xd0/0x280
[  328.160630]  [<c12be0a5>] ? eth_header+0x0/0x9b
[  328.360645]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  328.560660]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  328.760675]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  328.960691]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  329.160706]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  329.360721]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  329.560735]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  329.760749]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  329.960763]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  330.160777]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  330.360791]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  330.560806]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  330.760820]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  330.960834]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  331.160849]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  331.360864]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  331.560878]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  331.760893]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  331.960907]  [<c1031ead>] ? irq_exit+0x63/0x68
[  332.160921]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  332.360934]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  332.560949]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  332.760964]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  332.960979]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  333.060987] ---[ end trace df8deff2ad2a9761 ]---
[  333.160995] 100: csum_start 306, offset 16, headroom 390, headlen 158, len 158
[  333.261006] nr_frags=0 gso_size=0
[  333.361013]
[  333.461022] skb data:00000000: 00 9c 27 cc 00 00 00 00 00 00 00 00 00 00 00 00  ..'.............
[  333.561030] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  333.661038] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  333.761046] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  333.861055] skb data:00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  333.961063] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  334.061071] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  334.161079] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  334.261087] skb data:00000080: 00 04 61 98 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ..a......\'....\
[  334.361095] skb data:00000090: 27 ef 08 00 45 00 00 28 15 28 40 00 7f 06 7d a2  '...E..(.(@...}.
[  334.461103] skb data:000000a0: c0 a8 01 02 4f 63 57 f8 04 2e 00 50 94 c2 2d a9  ....OcW....P..-.
[  334.561111] skb data:000000b0: d6 2f bb 73 50 10 ff ff ee 41 00 00 00 00 00 00  ./.sP....A......
[  334.661120] skb data:000000c0: 00 00 20 4f 4b 0d 0a 53 65 72 76 65 72 3a 20 6e  .. OK..Server: n
[  334.761128] skb data:000000d0: 67 69 6e 78 2f 30 2e 36 2e 33 39 0d 0a 44 61 74  ginx/0.6.39..Dat
[  334.861136] skb data:000000e0: 65 3a 20 57 65 64 2c 20 32 35 20 41 75 67 20 32  e: Wed, 25 Aug 2
[  334.961144] skb data:000000f0: 30 31 30 20 30 36 3a 35 32 3a 34 35 20 47 4d 54  010 06:52:45 GMT
[  335.061152] skb data:00000100: 00 b0 27 cc 00 00 00 00 00 00 00 00 00 00 00 00  ..'.............
[  335.161160] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.261168] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.361176] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.461184] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.561192] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.661200] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.761208] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  335.861216] skb data:00000180: 00 04 61 98 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ..a......\'....\
[  335.961224] skb data:00000190: 27 ef 08 00 45 00 00 90 1a fa 40 00 7f 06 34 db  '[email protected].
[  336.061233] skb data:000001a0: c0 a8 01 02 5b 67 8e 81 04 f7 00 50 2e 1a b7 d1  ....[g.....P....
[  336.161241] skb data:000001b0: 09 f2 34 df 50 18 ff ff ac 15 00 00 25 33 44 39  ..4.P.......%3D9
[  336.261249] skb data:000001c0: 25 35 46 30 25 35 46 30 74 3b 20 41 53 50 53 45  %5F0%5F0t; ASPSE
[  336.361257] skb data:000001d0: 53 53 49 4f 4e 49 44 43 41 44 41 42 54 43 52 3d  SSIONIDCADABTCR=
[  336.461265] skb data:000001e0: 41 4e 4b 50 4b 41 42 43 49 4d 46 4e 42 42 41 45  ANKPKABCIMFNBBAE
[  336.561273] skb data:000001f0: 4e 42 49 45 49 43 4d 50 3b 20 42 49 47 69 70 53  NBIEICMP; BIGipS
[  336.661281] skb data:00000200: 65 72 76 65 72 53 4d 41 52 54 32 3d 38 38 39 36  erverSMART2=8896
[  336.761289] skb data:00000210: 35 31 39 37 38 2e 32 33 30 34 30 2e 30 30 30 30  51978.23040.0000
[  336.861297] skb data:00000220: 0d 0a 0d 0a 00 00 00 00 10 0c c6 a0 15 ed 28 8d  ..............(.
[  336.961306] skb data:00000230: 9c 30 02 94 60 09 ef 98 89 9b 28 8d c9 a0 0c ff  .0..`.....(.....
[  337.061314] skb data:00000240: f8 90 00 f9 8f c7 a0 0c 0e 19 09 92 30 08 84 60  ............0..`
[  337.161322] skb data:00000250: 08 78 d2 7c 8c 60 91 1e f9 91 20 19 92 22 19 09  .x.|.`.... .."..
[  337.261330] skb data:00000260: 81 00 00 3b 00 00 00 00 00 00 00 00 00 00 00 00  ...;............
[  337.361338] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  352.042649] ------------[ cut here ]------------
[  352.142668] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  352.242677] Hardware name: PowerEdge SC440
[  352.342688] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  352.442697] Call Trace:
[  352.642715]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  352.842731]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  353.042746]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  353.242761]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  353.442776]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  353.642792]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  353.842809]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  354.042825]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  354.242840]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  354.442856]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  354.642871]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  354.842887]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  355.042902]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  355.242917]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  355.442932]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  355.642957]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  355.842980]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  356.043009]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  356.243024]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  356.443039]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  356.643053]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  356.843067]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  357.043081]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  357.243095]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  357.443110]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  357.643126]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  357.843139]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  358.043153]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  358.243167]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  358.443183]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  358.643197]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  358.843210]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  359.043224]  [<c1031ead>] ? irq_exit+0x63/0x68
[  359.243238]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  359.443251]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  359.643265]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  359.843279]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  360.043294]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  360.143302] ---[ end trace df8deff2ad2a9762 ]---
[  360.243310] 100: csum_start 306, offset 16, headroom 390, headlen 143, len 143
[  360.343318] nr_frags=0 gso_size=0
[  360.443324]
[  360.543334] skb data:00000000: 00 bc 2e cc 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  360.643343] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  360.743351] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  360.843360] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  360.943368] skb data:00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  361.043377] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  361.143385] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  361.243393] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  361.343402] skb data:00000080: 00 19 66 2f 00 00 00 1a a0 38 8a 1b 00 19 66 2f  ..f/.....8....f/
[  361.443410] skb data:00000090: f9 3e 08 00 45 00 00 dc ac 84 40 00 80 06 a3 5d  .>..E.....@....]
[  361.543419] skb data:000000a0: c0 a8 0a 9f 5e f6 7f fc 07 15 01 bb 0a 27 94 09  ....^........'..
[  361.643427] skb data:000000b0: 86 26 d0 7a 50 18 ff ff 01 08 00 00 16 03 01 00  .&.zP...........
[  361.743436] skb data:000000c0: af 01 00 00 ab 03 01 4c 74 bd d6 04 bf 0c 49 6f  .......Lt.....Io
[  361.843444] skb data:000000d0: 2f 6e b3 4c a6 56 dc 7b 56 76 ba 7c 2d 5e b8 a0  /n.L.V.{Vv.|-^..
[  361.943453] skb data:000000e0: d8 c5 13 f3 47 ba d5 20 43 87 a4 82 fa cb 00 e1  ....G.. C.......
[  362.043461] skb data:000000f0: a0 fa aa ff 3d 0b 1d bc 13 e2 75 b8 a7 81 d0 bc  ....=.....u.....
[  362.143470] skb data:00000100: 00 b4 2e cc 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.243478] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.343486] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.443495] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.543503] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.643512] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.743520] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.843528] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  362.943537] skb data:00000180: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...\'....\'....\
[  363.043545] skb data:00000190: 27 ef 08 00 45 00 00 81 21 9a 40 00 7f 06 2e 4a  '[email protected]
[  363.143554] skb data:000001a0: c0 a8 01 02 5b 67 8e 81 04 f7 00 50 2e 1a ba 51  ....[g.....P...Q
[  363.243562] skb data:000001b0: 09 f2 3c 0c 50 18 fe 86 ac 06 00 00 41 53 50 53  ..<.P.......ASPS
[  363.343571] skb data:000001c0: 45 53 53 49 4f 4e 49 44 43 41 44 41 42 54 43 52  ESSIONIDCADABTCR
[  363.443580] skb data:000001d0: 3d 41 4e 4b 50 4b 41 42 43 49 4d 46 4e 42 42 41  =ANKPKABCIMFNBBA
[  363.543588] skb data:000001e0: 45 4e 42 49 45 49 43 4d 50 3b 20 42 49 47 69 70  ENBIEICMP; BIGip
[  363.643596] skb data:000001f0: 53 65 72 76 65 72 53 4d 41 52 54 32 3d 38 38 39  ServerSMART2=889
[  363.743605] skb data:00000200: 36 35 31 39 37 38 2e 32 33 30 34 30 2e 30 30 30  651978.23040.000
[  363.843613] skb data:00000210: 30 0d 0a 0d 0a 00 00 00 00 00 00 00 00 00 00 00  0...............
[  363.943622] skb data:00000220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  364.043630] skb data:00000230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  364.143638] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  364.243647] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  364.343655] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  364.443663] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  364.680629] ------------[ cut here ]------------
[  364.780652] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  364.880664] Hardware name: PowerEdge SC440
[  364.980679] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  365.080691] Call Trace:
[  365.280717]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  365.480741]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  365.680756]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  365.880772]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  366.080788]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  366.280804]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  366.480820]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  366.680836]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  366.880851]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  367.080868]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  367.280883]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  367.480899]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  367.680914]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  367.880928]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  368.080943]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  368.280958]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  368.480972]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  368.680987]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  368.881002]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  369.081017]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  369.281031]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  369.481046]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  369.681061]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  369.881076]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  370.081092]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  370.281108]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  370.481123]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  370.681138]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  370.881153]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  371.081168]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  371.281183]  [<c1031cd4>] ? __do_softirq+0x96/0xfa
[  371.481196]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  371.681210]  [<c1031ead>] ? irq_exit+0x63/0x68
[  371.881224]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  372.081236]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  372.281250]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  372.481265]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  372.681280]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  372.781288] ---[ end trace df8deff2ad2a9763 ]---
[  372.881297] 100: csum_start 306, offset 16, headroom 390, headlen 135, len 135
[  372.981304] nr_frags=0 gso_size=0
[  373.081311]
[  373.181321] skb data:00000000: 00 38 35 cc 02 00 08 00 41 00 31 32 34 38 33 34  .85.....A.124834
[  373.281330] skb data:00000010: 31 35 35 31 2e 4d 33 35 35 34 32 50 31 37 31 31  1551.M35542P1711
[  373.381339] skb data:00000020: 32 56 30 30 30 30 30 30 30 30 30 30 30 30 30 39  2V00000000000009
[  373.481347] skb data:00000030: 30 36 49 30 32 38 30 36 32 32 42 5f 31 2e 66 73  06I0280622B_1.fs
[  373.581355] skb data:00000040: 2c 53 3d 32 39 38 32 3a 32 2c 53 00 00 00 00 00  ,S=2982:2,S.....
[  373.681364] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  373.781372] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  373.881380] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  373.981389] skb data:00000080: 00 0e 2e 5c 27 e6 00 30 05 07 f3 1a 08 00 45 00  ...\'..0......E.
[  374.081397] skb data:00000090: 01 48 4c bc 00 00 80 11 dd 86 c0 a8 c7 0f c0 a8  .HL.............
[  374.181406] skb data:000000a0: c7 01 00 44 00 43 01 34 e5 4a 01 01 06 00 8e 0e  ...D.C.4.J......
[  374.281415] skb data:000000b0: c5 2f 36 6d 00 00 c0 a8 c7 0f 00 00 00 00 00 00  ./6m............
[  374.381423] skb data:000000c0: 00 00 00 00 00 00 00 30 05 07 f3 1a 00 00 00 00  .......0........
[  374.481432] skb data:000000d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  374.581440] skb data:000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  374.681448] skb data:000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  374.781457] skb data:00000100: 00 30 35 cc 80 35 54 c1 00 00 00 00 00 00 00 00  .05..5T.........
[  374.881465] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 c0 31 7a 03 c1  ............1z..
[  374.981474] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  375.081482] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 24 54 81 b7  ............$T..
[  375.181491] skb data:00000140: 00 00 00 00 00 00 00 00 9d 6e ac 57 2c 53 00 00  .........n.W,S..
[  375.281499] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  375.381508] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  375.481516] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  375.581524] skb data:00000180: 00 16 17 5e ef ec 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...^.....\'....\
[  375.681534] skb data:00000190: 27 ef 08 00 45 00 00 79 22 02 40 00 7f 06 2d ea  '...E..y"[email protected].
[  375.781543] skb data:000001a0: c0 a8 01 02 5b 67 8e 81 04 f7 00 50 2e 1a bc c2  ....[g.....P....
[  375.881552] skb data:000001b0: 09 f2 3e f6 50 18 ff ff ab fe 00 00 4f 4e 49 44  ..>.P.......ONID
[  375.981561] skb data:000001c0: 43 41 44 41 42 54 43 52 3d 41 4e 4b 50 4b 41 42  CADABTCR=ANKPKAB
[  376.081569] skb data:000001d0: 43 49 4d 46 4e 42 42 41 45 4e 42 49 45 49 43 4d  CIMFNBBAENBIEICM
[  376.181578] skb data:000001e0: 50 3b 20 42 49 47 69 70 53 65 72 76 65 72 53 4d  P; BIGipServerSM
[  376.281586] skb data:000001f0: 41 52 54 32 3d 38 38 39 36 35 31 39 37 38 2e 32  ART2=889651978.2
[  376.381594] skb data:00000200: 33 30 34 30 2e 30 30 30 30 0d 0a 0d 0a 00 00 00  3040.0000.......
[  376.481603] skb data:00000210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  376.581611] skb data:00000220: 00 00 00 00 00 00 00 00 30 30 30 30 30 30 30 39  ........00000009
[  376.681620] skb data:00000230: 30 36 49 30 32 38 30 36 45 36 35 5f 31 2e 66 73  06I02806E65_1.fs
[  376.781628] skb data:00000240: 2c 53 3d 31 35 37 38 3a 32 2c 53 00 00 00 00 00  ,S=1578:2,S.....
[  376.881636] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  376.981645] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  377.081653] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  454.636420] ------------[ cut here ]------------
[  454.736438] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  454.836447] Hardware name: PowerEdge SC440
[  454.936458] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  455.036466] Call Trace:
[  455.236484]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  455.436499]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  455.636513]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  455.836528]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  456.036543]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  456.236559]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  456.436574]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  456.636589]  [<c1224852>] ? rtl8139_start_xmit+0x51/0x13a
[  456.836603]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  457.036617]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  457.236632]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  457.436650]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  457.636673]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  457.836696]  [<c12b5601>] ? neigh_resolve_output+0xd0/0x280
[  458.036718]  [<c12be0a5>] ? eth_header+0x0/0x9b
[  458.236740]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  458.436762]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  458.636783]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  458.836805]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  459.036827]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  459.236848]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  459.436870]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  459.636891]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  459.836912]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  460.036934]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  460.236956]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  460.436979]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  460.637004]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  460.837026]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  461.037048]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  461.237071]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  461.437092]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  461.637114]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  461.837129]  [<c1031ead>] ? irq_exit+0x63/0x68
[  462.037143]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  462.237156]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  462.437171]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  462.637184]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  462.837199]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  462.937207] ---[ end trace df8deff2ad2a9764 ]---
[  463.037215] 100: csum_start 306, offset 16, headroom 390, headlen 151, len 151
[  463.137223] nr_frags=0 gso_size=0
[  463.237230]
[  463.337239] skb data:00000000: 00 00 00 00 39 c6 77 33 00 1a a0 38 8a 1b 08 00  ....9.w3...8....
[  463.437247] skb data:00000010: 45 00 00 68 00 00 40 00 ff 01 e6 3b c0 a8 0a 01  E..h..@....;....
[  463.537256] skb data:00000020: c0 a8 0a 07 08 00 ba 22 96 80 00 25 00 00 00 00  ......."...%....
[  463.637264] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  463.737272] skb data:00000040: 46 6c 6f 72 69 61 6e 20 46 6f 72 73 74 65 72 20  Florian Forster
[  463.837280] skb data:00000050: 3c 6f 63 74 6f 40 76 65 72 70 6c 61 6e 74 2e 6f  <[email protected]
[  463.937289] skb data:00000060: 72 67 3e 20 68 74 74 70 3a 2f 2f 76 65 72 70 6c  rg> http://verpl
[  464.037297] skb data:00000070: 61 6e 74 2e 6f 72 67 2f 00 00 00 00 00 00 00 00  ant.org/........
[  464.137305] skb data:00000080: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 08 00 45 00  ...\'....\'...E.
[  464.237313] skb data:00000090: 00 28 e7 f2 40 00 f5 06 1d 3a 5b cb 63 2d c0 a8  .(..@....:[.c-..
[  464.337321] skb data:000000a0: 01 02 00 50 04 4c 41 a1 8a 47 70 4f ca fc 50 10  ...P.LA..GpO..P.
[  464.437330] skb data:000000b0: 13 2e 10 33 00 00 00 00 00 00 00 00 54 54 50 2f  ...3........TTP/
[  464.537338] skb data:000000c0: 31 2e c7 4e 9d d1 da 14 6d 47 02 98 f4 c1 58 a3  1..N....mG....X.
[  464.637346] skb data:000000d0: 44 0b 73 14 a1 67 43 77 d3 95 06 31 bb 39 6c 78  D.s..gCw...1.9lx
[  464.737354] skb data:000000e0: 24 a8 c8 3c 8c d9 89 ba 5c e9 a3 1b 7f f8 e8 0c  $..<....\.......
[  464.837363] skb data:000000f0: f7 68 57 e4 1b 4d e8 3b 19 8b 88 8e ef 20 f9 47  .hW..M.;..... .G
[  464.937371] skb data:00000100: 00 34 16 cc 39 c6 77 33 00 1a a0 38 8a 1b 08 00  .4..9.w3...8....
[  465.037379] skb data:00000110: 45 00 00 68 00 00 40 00 ff 01 e6 3b c0 a8 0a 01  E..h..@....;....
[  465.137388] skb data:00000120: c0 a8 0a 07 08 00 ba 22 96 80 00 25 00 00 00 00  ......."...%....
[  465.237396] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  465.337404] skb data:00000140: 46 6c 6f 72 69 61 6e 20 46 6f 72 73 74 65 72 20  Florian Forster
[  465.437412] skb data:00000150: 3c 6f 63 74 6f 40 76 65 72 70 6c 61 6e 74 2e 6f  <[email protected]
[  465.537420] skb data:00000160: 72 67 3e 20 68 74 74 70 3a 2f 2f 76 65 72 70 6c  rg> http://verpl
[  465.637428] skb data:00000170: 61 6e 74 2e 6f 72 67 2f 00 00 00 00 00 00 00 00  ant.org/........
[  465.737436] skb data:00000180: 00 0a e6 ac 07 db 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  465.837444] skb data:00000190: 27 ef 08 00 45 00 00 89 04 28 40 00 7f 06 4b b4  '...E....(@...K.
[  465.937452] skb data:000001a0: c0 a8 01 02 5b 67 8e 81 04 4f 00 50 f6 d5 f5 07  ....[g...O.P....
[  466.037461] skb data:000001b0: 5b 82 93 48 50 18 ff ff ac 0e 00 00 33 42 25 32  [..HP.......3B%2
[  466.137469] skb data:000001c0: 34 73 68 25 33 44 33 25 33 42 25 32 34 73 77 25  4sh%3D3%3B%24sw%
[  466.237477] skb data:000001d0: 33 44 33 3b 20 70 69 64 3d 35 30 32 31 37 34 33  3D3; pid=5021743
[  466.337485] skb data:000001e0: 34 32 30 33 31 30 32 39 39 37 38 33 0d 0a 43 6f  420310299783..Co
[  466.437493] skb data:000001f0: 6f 6b 69 65 32 3a 20 24 56 65 72 73 69 6f 6e 3d  okie2: $Version=
[  466.537501] skb data:00000200: 31 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 4b  1..Connection: K
[  466.637509] skb data:00000210: 65 65 70 2d 41 6c 69 76 65 0d 0a 0d 0a ec 00 00  eep-Alive.......
[  466.737517] skb data:00000220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  466.837526] skb data:00000230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  466.937534] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  467.037542] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  467.137550] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  467.237558] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  470.600713] ------------[ cut here ]------------
[  470.700732] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  470.800741] Hardware name: PowerEdge SC440
[  470.900752] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  471.000761] Call Trace:
[  471.200779]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  471.400796]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  471.600810]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  471.800826]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  472.000841]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  472.200858]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  472.400874]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  472.600890]  [<c1224852>] ? rtl8139_start_xmit+0x51/0x13a
[  472.800905]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  473.000920]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  473.200936]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  473.400952]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  473.600969]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  473.800985]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  474.001000]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  474.201018]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  474.401032]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  474.601046]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  474.801060]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  475.001074]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  475.201088]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  475.401102]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  475.601115]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  475.801129]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  476.001142]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  476.201156]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  476.401171]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  476.601185]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  476.801199]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  477.001213]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  477.201229]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  477.401243]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  477.601257]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  477.801271]  [<c1031ead>] ? irq_exit+0x63/0x68
[  478.001284]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  478.201299]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  478.401314]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  478.601329]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  478.801344]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  478.901352] ---[ end trace df8deff2ad2a9765 ]---
[  479.001361] 100: csum_start 306, offset 16, headroom 390, headlen 168, len 168
[  479.101368] nr_frags=0 gso_size=0
[  479.201375]
[  479.301384] skb data:00000000: 00 7c de c8 e6 ac 07 db 00 1a a0 38 8a 1b 08 00  .|.........8....
[  479.401394] skb data:00000010: 45 00 00 6c 00 00 40 00 40 11 a4 9b c0 a8 0a 01  E..l..@.@.......
[  479.501403] skb data:00000020: c0 a8 0a 94 00 35 04 05 00 58 96 4f 6e d3 81 80  .....5...X.On...
[  479.601411] skb data:00000030: 00 01 00 02 00 00 00 00 04 6f 63 73 70 06 74 68  .........ocsp.th
[  479.701420] skb data:00000040: 61 77 74 65 03 63 6f 6d 00 00 01 00 01 c0 0c 00  awte.com........
[  479.801428] skb data:00000050: 05 00 01 00 00 03 67 00 13 04 6f 63 73 70 08 76  ......g...ocsp.v
[  479.901437] skb data:00000060: 65 72 69 73 69 67 6e 03 6e 65 74 00 c0 2d 00 01  erisign.net..-..
[  480.001445] skb data:00000070: 00 01 00 00 00 00 00 04 c7 07 36 48 61 30 00 00  ..........6Ha0..
[  480.101453] skb data:00000080: 00 0a e6 ac 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  480.201462] skb data:00000090: 27 ef 08 00 45 00 00 28 04 d0 00 00 7f 06 8b 6d  '...E..(.......m
[  480.301470] skb data:000000a0: c0 a8 01 02 5b 67 8e 81 04 4f 00 50 f6 d5 f5 69  ....[g...O.P...i
[  480.401479] skb data:000000b0: f6 d5 f5 69 50 04 00 00 27 2f 00 00 54 54 50 2f  ...iP...'/..TTP/
[  480.501487] skb data:000000c0: 31 2e 28 3d 33 d1 bc 47 66 61 86 93 94 7a 11 b5  1.(=3..Gfa...z..
[  480.601495] skb data:000000d0: 6b d8 32 97 e7 ce 6f 04 bb b9 d9 69 4a 5c 4b 4f  k.2...o....iJ\KO
[  480.701504] skb data:000000e0: a0 17 fa 64 c8 95 51 95 45 c8 dc 1d ef b0 4e 76  ...d..Q.E.....Nv
[  480.801512] skb data:000000f0: 22 6e 78 55 96 9d d4 78 73 2c 45 a3 00 d1 6e 22  "nxU...xs,E...n"
[  480.901521] skb data:00000100: 00 68 de c8 e6 ac 07 db 00 1a a0 38 8a 1b 08 00  .h.........8....
[  481.001529] skb data:00000110: 45 00 00 6c 00 00 40 00 40 11 a4 9b c0 a8 0a 01  E..l..@.@.......
[  481.101537] skb data:00000120: c0 a8 0a 94 00 35 04 05 00 58 96 4f 6e d3 81 80  .....5...X.On...
[  481.201546] skb data:00000130: 00 01 00 02 00 00 00 00 04 6f 63 73 70 06 74 68  .........ocsp.th
[  481.301554] skb data:00000140: 61 77 74 65 03 63 6f 6d 00 00 01 00 01 c0 0c 00  awte.com........
[  481.401562] skb data:00000150: 05 00 01 00 00 03 67 00 13 04 6f 63 73 70 08 76  ......g...ocsp.v
[  481.501571] skb data:00000160: 65 72 69 73 69 67 6e 03 6e 65 74 00 c0 2d 00 01  erisign.net..-..
[  481.601579] skb data:00000170: 00 01 00 00 00 00 00 04 c7 07 36 48 61 30 00 00  ..........6Ha0..
[  481.701588] skb data:00000180: 00 16 17 5e 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...^.....\'....\
[  481.801597] skb data:00000190: 27 ef 08 00 45 00 00 9a 04 dc 40 00 7f 06 4a ef  '[email protected].
[  481.901605] skb data:000001a0: c0 a8 01 02 5b 67 8e 81 04 5f 00 50 65 db a5 dc  ....[g..._.Pe...
[  482.001614] skb data:000001b0: 1c 11 e6 62 50 18 ff ff ac 1f 00 00 39 25 33 42  ...bP.......9%3B
[  482.101622] skb data:000001c0: 25 32 34 6f 25 33 44 31 31 30 35 31 25 33 42 25  %24o%3D11051%3B%
[  482.201630] skb data:000001d0: 32 34 73 68 25 33 44 33 25 33 42 25 32 34 73 77  24sh%3D3%3B%24sw
[  482.301639] skb data:000001e0: 25 33 44 33 3b 20 70 69 64 3d 35 30 32 31 37 34  %3D3; pid=502174
[  482.401647] skb data:000001f0: 33 34 32 30 33 31 30 32 39 39 37 38 33 0d 0a 43  3420310299783..C
[  482.501656] skb data:00000200: 6f 6f 6b 69 65 32 3a 20 24 56 65 72 73 69 6f 6e  ookie2: $Version
[  482.601665] skb data:00000210: 3d 31 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20  =1..Connection:
[  482.701673] skb data:00000220: 4b 65 65 70 2d 41 6c 69 76 65 0d 0a 0d 0a 00 00  Keep-Alive......
[  482.801681] skb data:00000230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  482.901689] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  483.001698] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  483.101706] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  483.201715] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  595.934671] ------------[ cut here ]------------
[  596.034690] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  596.134699] Hardware name: PowerEdge SC440
[  596.234711] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  596.334723] Call Trace:
[  596.534750]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  596.734773]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  596.934795]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  597.134811]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  597.334826]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  597.534842]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  597.734858]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  597.934873]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  598.134888]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  598.334904]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  598.534919]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  598.734934]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  598.934948]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  599.134962]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  599.334976]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  599.534990]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  599.735008]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  599.935022]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  600.135036]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  600.335049]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  600.535063]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  600.735076]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  600.935089]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  601.135103]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  601.335116]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  601.535131]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  601.735145]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  601.935159]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  602.135173]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  602.335188]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  602.535202]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  602.735215]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  602.935228]  [<c1031ead>] ? irq_exit+0x63/0x68
[  603.135242]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  603.335256]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  603.535272]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  603.735285]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  603.935300]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  604.035308] ---[ end trace df8deff2ad2a9766 ]---
[  604.135316] 100: csum_start 306, offset 16, headroom 390, headlen 98, len 98
[  604.235324] nr_frags=0 gso_size=0
[  604.335330]
[  604.435339] skb data:00000000: 00 28 37 cc 00 00 00 00 00 00 00 00 00 00 00 00  .(7.............
[  604.535348] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  604.635356] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  604.735364] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  604.835372] skb data:00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  604.935381] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  605.035389] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  605.135397] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  605.235406] skb data:00000080: 00 15 58 8f 00 00 00 1a a0 38 8a 1b 00 50 56 00  ..X......8...PV.
[  605.335414] skb data:00000090: 00 01 08 00 45 00 00 68 b2 9c 00 00 40 01 32 a4  [email protected].
[  605.435422] skb data:000000a0: c0 a8 0a 03 c0 a8 0a 01 00 00 50 56 08 3d 00 35  ..........PV.=.5
[  605.535430] skb data:000000b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  605.635439] skb data:000000c0: 00 00 00 00 46 6c 6f 72 69 61 6e 20 46 6f 72 73  ....Florian Fors
[  605.735447] skb data:000000d0: 74 65 72 20 3c 6f 63 74 6f 40 76 65 72 70 6c 61  ter <octo@verpla
[  605.835455] skb data:000000e0: 6e 74 2e 6f 72 67 3e 20 68 74 74 70 3a 2f 2f 76  nt.org> http://v
[  605.935463] skb data:000000f0: 65 72 70 6c 61 6e 74 2e 6f 72 67 2f df 0b 91 35  erplant.org/...5
[  606.035472] skb data:00000100: 00 e8 92 d2 61 98 72 a5 00 1a a0 38 8a 1b 08 00  ....a.r....8....
[  606.135480] skb data:00000110: 45 00 00 a0 00 00 40 00 40 11 a4 64 c0 a8 0a 01  E.....@[email protected]....
[  606.235489] skb data:00000120: c0 a8 0a 97 00 35 04 6e 00 8c 96 86 9e 46 81 80  .....5.n.....F..
[  606.335497] skb data:00000130: 00 01 00 03 00 00 00 00 01 6c 0a 61 64 64 74 68  .........l.addth
[  606.435505] skb data:00000140: 69 73 63 64 6e 03 63 6f 6d 00 00 01 00 01 c0 0c  iscdn.com.......
[  606.535514] skb data:00000150: 00 05 00 01 00 00 00 be 00 25 08 77 69 6c 64 63  .........%.wildc
[  606.635522] skb data:00000160: 61 72 64 0a 61 64 64 74 68 69 73 63 64 6e 03 63  ard.addthiscdn.c
[  606.735531] skb data:00000170: 6f 6d 07 65 64 67 65 6b 65 79 03 6e 65 74 00 00  om.edgekey.net..
[  606.835539] skb data:00000180: 00 0a e6 ac 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  606.935547] skb data:00000190: 27 ef 08 00 45 00 00 54 0d da 40 00 7f 06 d3 63  '[email protected]
[  607.035556] skb data:000001a0: c0 a8 01 02 d1 55 87 66 05 5d 00 50 f0 d2 2a b2  .....U.f.].P..*.
[  607.135564] skb data:000001b0: b6 81 e7 7d 50 18 ff ff 1a ad 00 00 66 6c 61 74  ...}P.......flat
[  607.235573] skb data:000001c0: 65 2c 20 67 7a 69 70 2c 20 63 68 75 6e 6b 65 64  e, gzip, chunked
[  607.335581] skb data:000001d0: 2c 20 69 64 65 6e 74 69 74 79 2c 20 74 72 61 69  , identity, trai
[  607.435589] skb data:000001e0: 6c 65 72 73 0d 0a 0d 0a 00 00 00 00 00 00 00 00  lers............
[  607.535597] skb data:000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  609.648527] ------------[ cut here ]------------
[  609.748545] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  609.848554] Hardware name: PowerEdge SC440
[  609.948564] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  610.048573] Call Trace:
[  610.248591]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  610.448607]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  610.648622]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  610.848638]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  611.048654]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  611.248672]  [<c13311fe>] ? xfrm_lookup+0x1c/0x3a
[  611.448688]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  611.648704]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  611.848719]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  612.048735]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  612.248750]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  612.448766]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  612.648781]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  612.848797]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  613.048813]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  613.248828]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  613.448843]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  613.648858]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  613.848873]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  614.048886]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  614.248901]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  614.448915]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  614.648930]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  614.848943]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  615.048957]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  615.248973]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  615.448987]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  615.649004]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  615.849018]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  616.049033]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  616.249048]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  616.449062]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  616.649075]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  616.849089]  [<c1031ead>] ? irq_exit+0x63/0x68
[  617.049102]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  617.249116]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  617.449130]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  617.649144]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  617.849160]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  617.949169] ---[ end trace df8deff2ad2a9767 ]---
[  618.049177] 100: csum_start 306, offset 16, headroom 390, headlen 91, len 91
[  618.149185] nr_frags=0 gso_size=0
[  618.249191]
[  618.349200] skb data:00000000: 00 00 00 00 41 75 67 20 32 35 20 30 39 3a 35 37  ....Aug 25 09:57
[  618.449209] skb data:00000010: 3a 31 32 20 61 75 74 68 64 61 65 6d 6f 6e 64 3a  :12 authdaemond:
[  618.549217] skb data:00000020: 20 75 73 65 72 64 62 3a 20 68 6f 6d 65 3d 2f 68   userdb: home=/h
[  618.649225] skb data:00000030: 6f 6d 65 2f 76 6d 61 69 6c 2f 63 61 72 70 6f 2c  ome/vmail/carpo,
[  618.749233] skb data:00000040: 20 75 69 64 3d 31 30 30 36 2c 20 67 69 64 3d 31   uid=1006, gid=1
[  618.849241] skb data:00000050: 30 30 2c 20 73 68 65 6c 6c 3d 3c 75 6e 73 65 74  00, shell=<unset
[  618.949249] skb data:00000060: 3e 2c 20 6d 61 69 6c 3d 2f 68 6f 6d 65 2f 76 6d  >, mail=/home/vm
[  619.049258] skb data:00000070: 61 69 6c 2f 63 61 72 70 6f 2f 4d 61 69 6c 00 00  ail/carpo/Mail..
[  619.149266] skb data:00000080: 00 0e 2e 5c 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...\.....\'....\
[  619.249274] skb data:00000090: 27 ef 08 00 45 00 00 34 29 df 40 00 3f 06 29 b8  '...E..4).@.?.).
[  619.349283] skb data:000000a0: c0 a8 01 02 c0 57 66 2b b4 46 fc 20 4a 18 7e 15  .....Wf+.F. J.~.
[  619.449291] skb data:000000b0: ca 51 03 5b 80 10 27 06 2b a0 00 00 01 01 08 0a  .Q.[..'.+.......
[  619.549299] skb data:000000c0: 00 ea 5e 99 eb 9c aa 87 6f 6e 6e 65 63 74 69 6f  ..^.....onnectio
[  619.649307] skb data:000000d0: 6e 3a 20 63 6c 6f 73 65 0d 0a 43 6f 6e 74 65 6e  n: close..Conten
[  619.749316] skb data:000000e0: 74 2d 54 79 70 65 3a 20 69 6d 61 67 65 2f 67 69  t-Type: image/gi
[  619.849324] skb data:000000f0: 66 0d 0a 43 61 63 68 65 2d 43 6f 6e 74 72 6f 6c  f..Cache-Control
[  619.949332] skb data:00000100: 00 6c de c8 00 00 00 00 00 00 00 00 00 00 00 00  .l..............
[  620.049340] skb data:00000110: 45 00 00 68 00 00 40 00 ff 01 e5 5d c0 a8 0a 01  E..h..@....]....
[  620.149348] skb data:00000120: c0 a8 0a e5 08 00 09 1f 47 73 00 36 00 00 00 00  ........Gs.6....
[  620.249357] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  620.349365] skb data:00000140: 46 6c 6f 72 69 61 6e 20 46 6f 72 73 74 65 72 20  Florian Forster
[  620.449373] skb data:00000150: 3c 6f 63 74 6f 40 76 65 72 70 6c 61 6e 74 2e 6f  <[email protected]
[  620.549381] skb data:00000160: 72 67 3e 20 68 74 74 70 3a 2f 2f 76 65 72 70 6c  rg> http://verpl
[  620.649389] skb data:00000170: 61 6e 74 2e 6f 72 67 2f 00 00 00 00 00 00 00 00  ant.org/........
[  620.749397] skb data:00000180: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...\'....\'....\
[  620.849406] skb data:00000190: 27 ef 08 00 45 00 00 4d 0e 33 40 00 7f 06 d3 11  '...E..M.3@.....
[  620.949414] skb data:000001a0: c0 a8 01 02 d1 55 87 66 05 67 00 50 b0 ed 26 e3  .....U.f.g.P..&.
[  621.049423] skb data:000001b0: 21 67 b2 5b 50 18 ff ff 1a a6 00 00 67 7a 69 70  !g.[P.......gzip
[  621.149431] skb data:000001c0: 2c 20 63 68 75 6e 6b 65 64 2c 20 69 64 65 6e 74  , chunked, ident
[  621.249439] skb data:000001d0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  621.349448] skb data:000001e0: 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  621.449456] skb data:000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  621.551553] ------------[ cut here ]------------
[  621.651565] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  621.751576] Hardware name: PowerEdge SC440
[  621.851591] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  621.951600] Call Trace:
[  622.151616]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  622.351631]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  622.551646]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  622.751661]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  622.951676]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  623.151692]  [<c1083e35>] ? __slab_alloc+0x285/0x4ef
[  623.351707]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  623.551723]  [<c1224852>] ? rtl8139_start_xmit+0x51/0x13a
[  623.751737]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  623.951752]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  624.151767]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  624.351782]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  624.551798]  [<c12bfd67>] ? __qdisc_run+0x62/0xdf
[  624.751813]  [<c12b01ca>] ? dev_queue_xmit+0x155/0x354
[  624.951828]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  625.151843]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  625.351858]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  625.551872]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  625.751887]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  625.951903]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  626.151917]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  626.351932]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  626.551947]  [<c12adb80>] ? napi_skb_finish+0x2b/0x39
[  626.751961]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  626.951976]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  627.151992]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  627.352009]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  627.552024]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  627.752039]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  627.952054]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  628.152068]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  628.352083]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  628.552096]  [<c1031ead>] ? irq_exit+0x63/0x68
[  628.752110]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  628.952123]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  629.152137]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  629.352151]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  629.552165]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  629.652173] ---[ end trace df8deff2ad2a9768 ]---
[  629.752181] 100: csum_start 306, offset 16, headroom 390, headlen 91, len 91
[  629.852188] nr_frags=0 gso_size=0
[  629.952195]
[  630.052203] skb data:00000000: 00 4c f9 c8 4a 00 08 00 42 00 31 32 38 32 35 33  .L..J...B.128253
[  630.152212] skb data:00000010: 39 33 36 39 2e 4d 34 37 30 34 33 38 50 32 31 39  9369.M470438P219
[  630.252220] skb data:00000020: 32 56 30 30 30 30 30 30 30 30 30 30 30 30 30 39  2V00000000000009
[  630.352228] skb data:00000030: 30 36 49 30 31 32 39 36 31 46 42 5f 31 33 2e 66  06I012961FB_13.f
[  630.452237] skb data:00000040: 73 2c 53 3d 38 34 32 39 3a 32 2c 53 2c 53 53 30  s,S=8429:2,S,SS0
[  630.552245] skb data:00000050: 29 0d 0a 35 32 20 4f 4b 20 53 54 41 54 55 53 20  )..52 OK STATUS
[  630.652253] skb data:00000060: 43 6f 6d 70 6c 65 74 65 64 2e 0d 0a 00 00 00 00  Completed.......
[  630.752261] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  630.852270] skb data:00000080: 00 00 00 00 00 00 00 1a a0 38 8a 1b 00 04 61 98  .........8....a.
[  630.952279] skb data:00000090: 72 a5 08 00 45 00 00 30 30 83 40 00 80 06 2e 17  r...E..00.@.....
[  631.052287] skb data:000000a0: c0 a8 0a 97 ae 89 22 65 05 27 00 50 95 60 06 10  ......"e.'.P.`..
[  631.152296] skb data:000000b0: 00 00 00 00 70 02 ff ff 46 0a 00 00 02 04 05 b4  ....p...F.......
[  631.252304] skb data:000000c0: 01 01 04 02 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  631.352312] skb data:000000d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  631.452321] skb data:000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  631.552329] skb data:000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  631.652338] skb data:00000100: 00 ec fa c8 4a 00 08 00 42 00 31 32 38 32 37 31  ....J...B.128271
[  631.752346] skb data:00000110: 35 30 31 36 2e 4d 38 33 37 37 32 35 50 32 31 39  5016.M837725P219
[  631.852354] skb data:00000120: 32 37 56 30 30 30 30 30 30 30 30 30 30 30 30 30  27V0000000000000
[  631.952362] skb data:00000130: 39 30 36 49 30 30 45 41 30 37 45 45 5f 36 2e 66  906I00EA07EE_6.f
[  632.052371] skb data:00000140: 73 2c 53 3d 36 30 30 31 3a 32 2c 53 53 32 2c 53  s,S=6001:2,SS2,S
[  632.152379] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  632.252388] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  632.352396] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  632.452404] skb data:00000180: 00 00 00 00 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  632.552413] skb data:00000190: 27 ef 08 00 45 00 00 4d 0e 3d 40 00 7f 06 d3 07  '...E..M.=@.....
[  632.652422] skb data:000001a0: c0 a8 01 02 d1 55 87 66 05 67 00 50 b0 ed 26 e3  .....U.f.g.P..&.
[  632.752430] skb data:000001b0: 21 67 b2 5b 50 18 ff ff 1a a6 00 00 67 7a 69 70  !g.[P.......gzip
[  632.852438] skb data:000001c0: 2c 20 63 68 75 6e 6b 65 64 2c 20 69 64 65 6e 74  , chunked, ident
[  632.952447] skb data:000001d0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  633.052455] skb data:000001e0: 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  633.152464] skb data:000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  639.625714] ------------[ cut here ]------------
[  639.725733] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  639.825742] Hardware name: PowerEdge SC440
[  639.925752] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  640.025761] Call Trace:
[  640.225779]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  640.425795]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  640.625810]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  640.825825]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  641.025844]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  641.225869]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  641.425893]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  641.625917]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  641.825932]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  642.025948]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  642.225963]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  642.425980]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  642.625995]  [<c12b5601>] ? neigh_resolve_output+0xd0/0x280
[  642.826016]  [<c12be0a5>] ? eth_header+0x0/0x9b
[  643.026032]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  643.226048]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  643.426063]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  643.626078]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  643.826092]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  644.026107]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  644.226122]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  644.426136]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  644.626152]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  644.826167]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  645.026181]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  645.226197]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  645.426212]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  645.626226]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  645.826240]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  646.026255]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  646.226269]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  646.426283]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  646.626297]  [<c1031ead>] ? irq_exit+0x63/0x68
[  646.826310]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  647.026323]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  647.226338]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  647.426351]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  647.626366]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  647.726374] ---[ end trace df8deff2ad2a9769 ]---
[  647.826383] 100: csum_start 306, offset 16, headroom 390, headlen 96, len 96
[  647.926390] nr_frags=0 gso_size=0
[  648.026397]
[  648.126406] skb data:00000000: 00 c0 1c cc 05 07 f3 1a 00 0e 2e 5c 27 e6 08 00  ...........\'...
[  648.226416] skb data:00000010: 45 00 00 c2 00 00 40 00 40 11 2a c9 c0 a8 c7 01  E.....@.@.*.....
[  648.326424] skb data:00000020: c0 a8 c7 0f 00 35 05 16 00 ae 10 22 01 4d 81 80  .....5.....".M..
[  648.426433] skb data:00000030: 00 01 00 07 00 00 00 00 08 63 6c 69 65 6e 74 73  .........clients
[  648.526441] skb data:00000040: 31 06 67 6f 6f 67 6c 65 02 62 67 00 00 01 00 01  1.google.bg.....
[  648.626450] skb data:00000050: c0 0c 00 05 00 01 00 00 13 fa 00 16 07 63 6c 69  .............cli
[  648.726459] skb data:00000060: 65 6e 74 73 01 6c 06 67 6f 6f 67 6c 65 03 63 6f  ents.l.google.co
[  648.826467] skb data:00000070: 6d 00 c0 30 00 01 00 01 00 00 00 55 00 04 00 00  m..0.......U....
[  648.926475] skb data:00000080: 00 0a e6 ac 07 db 00 1a a0 38 8a 1b 08 00 45 00  .........8....E.
[  649.026484] skb data:00000090: 00 28 00 00 40 00 38 06 a1 fd d4 32 01 64 c0 a8  .([email protected]..
[  649.126492] skb data:000000a0: 0a 94 00 50 05 64 27 80 ba dc 66 e9 47 b0 50 10  ...P.d'...f.G.P.
[  649.226501] skb data:000000b0: 19 f0 5e 67 00 00 ff ff 4b df 00 00 01 01 08 0a  ..^g....K.......
[  649.326510] skb data:000000c0: 00 ea 5e 7e eb 9c 7b 39 01 01 05 0a ca 4e e7 5b  ..^~..{9.....N.[
[  649.426518] skb data:000000d0: ca 4e ec fb 46 2a 33 d2 18 82 46 b7 eb 8e 44 77  .N..F*3...F...Dw
[  649.526526] skb data:000000e0: bc 35 37 3a 31 30 20 47 4d 54 0d 0a 53 65 72 76  .57:10 GMT..Serv
[  649.626535] skb data:000000f0: 65 72 3a 20 41 70 61 63 68 65 0d 0a 4c 61 73 74  er: Apache..Last
[  649.726543] skb data:00000100: 00 fc 1c cc 00 00 00 00 00 00 00 00 00 00 08 00  ................
[  649.826551] skb data:00000110: 45 c0 00 84 09 52 00 00 40 01 db 14 c0 a8 0a 01  E....R..@.......
[  649.926560] skb data:00000120: c0 a8 0a 01 03 01 fc fe 00 00 00 00 45 00 00 68  ............E..h
[  650.026568] skb data:00000130: 00 00 40 00 ff 01 e5 5d c0 a8 0a 01 c0 a8 0a e5  ..@....]........
[  650.126576] skb data:00000140: 08 00 09 1d 47 73 00 38 00 00 00 00 00 00 00 00  ....Gs.8........
[  650.226585] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 46 6c 6f 72  ............Flor
[  650.326593] skb data:00000160: 69 61 6e 20 46 6f 72 73 74 65 72 20 3c 6f 63 74  ian Forster <oct
[  650.426601] skb data:00000170: 6f 40 76 65 72 70 6c 61 6e 74 2e 6f 72 67 00 00  [email protected]..
[  650.526609] skb data:00000180: 00 15 58 8f 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ..X......\'....\
[  650.626618] skb data:00000190: 27 ef 08 00 45 00 00 52 af d0 40 00 7f 06 18 13  '...E..R..@.....
[  650.726626] skb data:000001a0: c0 a8 01 02 4a 7d 27 9b 05 dd 00 50 21 f4 27 de  ....J}'....P!.'.
[  650.826635] skb data:000001b0: 5d d2 44 c1 50 18 ff ff 34 07 00 00 61 74 65 2c  ].D.P...4...ate,
[  650.926643] skb data:000001c0: 20 67 7a 69 70 2c 20 63 68 75 6e 6b 65 64 2c 20   gzip, chunked,
[  651.026651] skb data:000001d0: 69 64 65 6e 74 69 74 79 2c 20 74 72 61 69 6c 65  identity, traile
[  651.126660] skb data:000001e0: 72 73 0d 0a 0d 0a 20 6e 6f 2d 63 61 63 68 65 0d  rs.... no-cache.
[  651.226668] skb data:000001f0: 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a  .Content-Length:
[  651.778140] ------------[ cut here ]------------
[  651.878158] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  651.978167] Hardware name: PowerEdge SC440
[  652.078177] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  652.178185] Call Trace:
[  652.378204]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  652.578220]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  652.778235]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  652.978252]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  653.178267]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  653.378284]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  653.578300]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  653.778316]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  653.978331]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  654.178348]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  654.378364]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  654.578380]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  654.778395]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  654.978409]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  655.178424]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  655.378438]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  655.578453]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  655.778468]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  655.978482]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  656.178497]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  656.378512]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  656.578526]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  656.778539]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  656.978552]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  657.178565]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  657.378579]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  657.578592]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  657.778605]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  657.978619]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  658.178632]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  658.378644]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  658.578656]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  658.778669]  [<c1031ead>] ? irq_exit+0x63/0x68
[  658.978683]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  659.178696]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  659.378710]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  659.578723]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  659.778739]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  659.878748] ---[ end trace df8deff2ad2a976a ]---
[  659.978756] 100: csum_start 306, offset 16, headroom 390, headlen 95, len 95
[  660.078764] nr_frags=0 gso_size=0
[  660.178770]
[  660.278779] skb data:00000000: 00 9c 27 cc 2e 5c 27 b2 00 0e 2e 5c 27 ef 08 00  ..'..\'....\'...
[  660.378788] skb data:00000010: 45 c0 00 83 c3 a0 00 00 40 01 45 4d c0 a8 01 02  [email protected]....
[  660.478797] skb data:00000020: d0 43 de de 03 03 6e 2e 00 00 00 00 45 00 00 67  .C....n.....E..g
[  660.578806] skb data:00000030: 00 00 40 00 39 11 d0 b9 d0 43 de de c0 a8 01 02  [email protected]......
[  660.678814] skb data:00000040: 00 35 3a 76 00 53 db d6 33 72 81 80 00 01 00 01  .5:v.S..3r......
[  660.778822] skb data:00000050: 00 00 00 00 03 31 31 34 03 32 31 37 02 31 32 03  .....114.217.12.
[  660.878831] skb data:00000060: 32 31 36 07 69 6e 2d 61 64 64 72 04 61 72 70 61  216.in-addr.arpa
[  660.978840] skb data:00000070: 00 00 0c 00 01 c0 0c 00 0c 00 01 00 00 13 df 00  ................
[  661.078848] skb data:00000080: 00 0e 2e 5c 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...\.....\'....\
[  661.178857] skb data:00000090: 27 ef 08 00 45 00 00 28 b0 5c 40 00 7f 06 2f c7  '...E..(.\@.../.
[  661.278865] skb data:000000a0: c0 a8 01 02 55 0e 04 f4 05 df 00 50 bb 6b 1e cb  ....U......P.k..
[  661.378874] skb data:000000b0: b8 42 fe b0 50 10 ff ff fc ce 00 00 c6 4c a3 d6  .B..P........L..
[  661.478882] skb data:000000c0: 50 82 60 6f d3 94 63 5e 4f e6 84 d9 c7 26 05 b3  P.`o..c^O....&..
[  661.578891] skb data:000000d0: 74 4c 39 c0 8d b5 fd c0 0c 00 0c 00 01 00 00 19  tL9.............
[  661.678899] skb data:000000e0: 29 00 12 06 61 39 30 35 73 6d 05 61 76 61 73 74  )...a905sm.avast
[  661.778907] skb data:000000f0: 03 63 6f 6d 00 00 00 00 00 00 00 00 00 00 00 00  .com............
[  661.878916] skb data:00000100: 00 ac 27 cc 05 07 f3 1a 00 0e 2e 5c 27 e6 08 00  ..'........\'...
[  661.978924] skb data:00000110: 45 00 00 ac 00 00 40 00 40 11 2a df c0 a8 c7 01  E.....@.@.*.....
[  662.078932] skb data:00000120: c0 a8 c7 0f 00 35 04 04 00 98 10 0c 01 55 81 80  .....5.......U..
[  662.178941] skb data:00000130: 00 01 00 05 00 00 00 00 07 70 61 67 65 61 64 32  .........pagead2
[  662.278949] skb data:00000140: 11 67 6f 6f 67 6c 65 73 79 6e 64 69 63 61 74 69  .googlesyndicati
[  662.378958] skb data:00000150: 6f 6e 03 63 6f 6d 00 00 01 00 01 c0 0c 00 05 00  on.com..........
[  662.478966] skb data:00000160: 01 00 01 46 5c 00 15 06 70 61 67 65 61 64 01 6c  ...F\...pagead.l
[  662.578975] skb data:00000170: 06 67 6f 6f 67 6c 65 03 63 6f 6d 00 c0 3b 00 00  .google.com..;..
[  662.678983] skb data:00000180: 00 26 9e 0f 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  .&.......\'....\
[  662.778991] skb data:00000190: 27 ef 08 00 45 00 00 51 b0 d2 40 00 7f 06 30 70  '[email protected]
[  662.879003] skb data:000001a0: c0 a8 01 02 d1 55 87 64 05 8c 00 50 fc 1e ec 32  .....U.d...P...2
[  662.979012] skb data:000001b0: ce 21 af 11 50 18 fc 27 1a a8 00 00 74 65 2c 20  .!..P..'....te,
[  663.079020] skb data:000001c0: 67 7a 69 70 2c 20 63 68 75 6e 6b 65 64 2c 20 69  gzip, chunked, i
[  663.179029] skb data:000001d0: 64 65 6e 74 69 74 79 2c 20 74 72 61 69 6c 65 72  dentity, trailer
[  663.279037] skb data:000001e0: 73 0d 0a 0d 0a 22 3a 22 52 55 42 49 43 4f 4e 5f  s....":"RUBICON_
[  663.379045] skb data:000001f0: 50 52 4f 5f 42 55 5f 52 4f 4e 5f 37 32 38 58 39  PRO_BU_RON_728X9
[  690.540721] ------------[ cut here ]------------
[  690.640741] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  690.740750] Hardware name: PowerEdge SC440
[  690.840761] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  690.940770] Call Trace:
[  691.140789]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  691.340804]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  691.540820]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  691.740836]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  691.940852]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  692.140868]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  692.340884]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  692.540901]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  692.740916]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  692.940932]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  693.140948]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  693.340965]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  693.540981]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  693.740997]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  693.941017]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  694.141032]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  694.341046]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  694.541061]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  694.741075]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  694.941089]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  695.141103]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  695.341116]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  695.541131]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  695.741145]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  695.941160]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  696.141175]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  696.341189]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  696.541203]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  696.741218]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  696.941233]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  697.141247]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  697.341261]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  697.541275]  [<c1031ead>] ? irq_exit+0x63/0x68
[  697.741289]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  697.941303]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  698.141318]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  698.341332]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  698.541348]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  698.641356] ---[ end trace df8deff2ad2a976b ]---
[  698.741364] 100: csum_start 306, offset 16, headroom 390, headlen 75, len 75
[  698.841372] nr_frags=0 gso_size=0
[  698.941378]
[  699.041387] skb data:00000000: 00 c4 28 db 3b 07 9b 3f 00 1a a0 38 8a 1b 08 00  ..(.;..?...8....
[  699.141397] skb data:00000010: 45 00 00 68 00 00 40 00 ff 01 e6 3e c0 a8 0a 01  E..h..@....>....
[  699.241406] skb data:00000020: c0 a8 0a 04 08 00 99 ce b6 bb 00 3e 00 00 00 00  ...........>....
[  699.341414] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  699.441422] skb data:00000040: 46 6c 6f 72 69 61 6e 20 46 6f 72 73 74 65 72 20  Florian Forster
[  699.541431] skb data:00000050: 3c 6f 63 74 6f 40 76 65 72 70 6c 61 6e 74 2e 6f  <[email protected]
[  699.641439] skb data:00000060: 72 67 3e 20 68 74 74 70 3a 2f 2f 76 65 72 70 6c  rg> http://verpl
[  699.741447] skb data:00000070: 61 6e 74 2e 6f 72 67 2f 00 00 00 00 00 00 00 00  ant.org/........
[  699.841456] skb data:00000080: 00 00 00 00 00 00 00 1a a0 38 8a 1b 00 19 66 2f  .........8....f/
[  699.941464] skb data:00000090: f9 3e 08 00 45 00 00 28 da 32 40 00 80 06 8a ac  .>..E..(.2@.....
[  700.041473] skb data:000000a0: c0 a8 0a 9f c0 a8 0a 01 07 2f 00 8f 75 27 b1 29  ........./..u'.)
[  700.141481] skb data:000000b0: 7a 97 06 bb 50 10 b0 27 ba 5a 00 00 00 00 00 00  z...P..'.Z......
[  700.241490] skb data:000000c0: 00 00 4d 61 69 6c 64 69 00 00 00 19 66 2f f9 3e  ..Maildi....f/.>
[  700.341498] skb data:000000d0: 00 1a a0 38 8a 1b 08 00 45 00 05 dc f1 ed 40 00  ...8....E.....@.
[  700.441506] skb data:000000e0: 40 06 00 00 c0 a8 0a 01 c0 a8 0a 9f 00 8f 07 2f  @............../
[  700.541514] skb data:000000f0: 7a 97 b1 d3 75 27 b1 29 50 10 1d 8e 00 00 00 00  z...u'.)P.......
[  700.641523] skb data:00000100: 00 f0 28 db 39 c6 77 33 00 1a a0 38 8a 1b 08 00  ..(.9.w3...8....
[  700.741531] skb data:00000110: 45 00 00 68 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  E..h.....\'....\
[  700.841539] skb data:00000120: 27 ef 08 00 45 00 00 7c 00 00 40 00 40 29 65 53  '...E..|..@.@)eS
[  700.941548] skb data:00000130: c0 a8 01 02 d4 12 3f 49 60 00 00 00 00 40 3a 40  ......?I`....@:@
[  701.041556] skb data:00000140: 20 01 15 c0 65 ff 00 64 00 00 00 00 00 00 00 02   ...e..d........
[  701.141565] skb data:00000150: 20 01 15 c0 65 ff 00 64 00 00 00 00 00 00 00 01   ...e..d........
[  701.241573] skb data:00000160: 80 00 c2 0f 69 0b 00 17 24 bf 74 4c 8a f8 0d 00  ....i...$.tL....
[  701.341581] skb data:00000170: 08 09 0a 0b 0c 0d 0e 0f 10 11 12 13 14 15 00 00  ................
[  701.441589] skb data:00000180: 00 15 58 8f a0 21 00 0e 2e 5c 27 b2 00 0e 2e 5c  ..X..!...\'....\
[  701.541598] skb data:00000190: 27 ef 08 00 45 00 00 3d b6 47 40 00 7f 06 a3 44  '[email protected]
[  701.641606] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 0d 00 50 d0 e5 75 18  .....2.R...P..u.
[  701.741615] skb data:000001b0: 30 a2 40 58 50 18 ff ff a2 5e 00 00 64 65 6e 74  0.@XP....^..dent
[  701.841623] skb data:000001c0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  701.941632] skb data:000001d0: 0a 74 65 73 0d 0a 0d 0a 76 01 15 41 79 79 e9 99  .tes....v..Ayy..
[  702.041639] skb data:000001e0: 79 15 fa 06 7a 66 7a c6 96 c8 4a f4 41 96 80 0c  y...zfz...J.A...
[  702.141647] skb data:000001f0: d5 87 3a 10 00 77 17 1a 5e a9 00 00 00 0d 0a 30  ..:..w..^......0
[  702.324707] ------------[ cut here ]------------
[  702.424725] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  702.524734] Hardware name: PowerEdge SC440
[  702.624744] Pid: 3324, comm: couriertls Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  702.724752] Call Trace:
[  702.924771]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  703.124786]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  703.324801]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  703.524816]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  703.724831]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  703.924847]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  704.124862]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  704.324879]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  704.524894]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  704.724910]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  704.924925]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  705.124941]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  705.324956]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  705.524971]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  705.724986]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  705.925005]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  706.125020]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  706.325036]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  706.525051]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  706.725066]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  706.925081]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  707.125095]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  707.325111]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  707.525126]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  707.725141]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  707.925157]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  708.125172]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  708.325187]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  708.525201]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  708.725216]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  708.925231]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  709.125246]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  709.325260]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  709.525273]  [<c1031ead>] ? irq_exit+0x63/0x68
[  709.725287]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  709.925300]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  710.125314]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  710.325329]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  710.425338] ---[ end trace df8deff2ad2a976c ]---
[  710.525346] 100: csum_start 306, offset 16, headroom 390, headlen 75, len 75
[  710.625353] nr_frags=0 gso_size=0
[  710.725360]
[  710.825369] skb data:00000000: 00 48 4f c7 51 c3 f2 5d 04 1c 78 62 65 c1 6b f9  .HO.Q..]..xbe.k.
[  710.925378] skb data:00000010: ab 7a d9 42 35 93 16 47 f4 a4 84 16 d5 4c ee 15  .z.B5..G.....L..
[  711.025386] skb data:00000020: 4f 1c f1 93 b6 b6 a8 29 0b 45 9c 6e da a6 79 47  O......).E.n..yG
[  711.125395] skb data:00000030: 62 61 04 40 f8 6f 0f 09 ab 27 07 05 48 73 bb 37  [email protected]...'..Hs.7
[  711.225403] skb data:00000040: 59 b9 1e 1e eb 42 89 3d 05 92 ba 1b 13 11 d8 9f  Y....B.=........
[  711.325412] skb data:00000050: b1 fc 54 19 2d 1f 30 f2 11 99 cb b4 51 5c df 52  ..T.-.0.....Q\.R
[  711.425420] skb data:00000060: 84 c8 00 06 a2 73 f6 39 a5 c5 3f b8 31 04 2f c9  .....s.9..?.1./.
[  711.525429] skb data:00000070: 44 2d 0e ca 95 99 aa a9 b1 52 aa e9 eb 3f a5 2e  D-.......R...?..
[  711.625437] skb data:00000080: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 08 00 45 c0  ...\'....\'...E.
[  711.725446] skb data:00000090: 00 5f 4f da 00 00 40 01 a6 b0 c0 a8 01 01 c0 a8  ._O...@.........
[  711.825454] skb data:000000a0: 01 02 03 03 80 91 00 00 00 00 45 00 00 43 00 00  ..........E..C..
[  711.925462] skb data:000000b0: 40 00 40 11 b7 56 c0 a8 01 02 c0 a8 01 01 b0 b4  @[email protected]..........
[  712.025471] skb data:000000c0: 00 35 00 2f 8e 51 ea df 01 00 00 01 00 00 00 00  .5./.Q..........
[  712.125479] skb data:000000d0: 00 00 0b 64 6f 77 6e 6c 6f 61 64 39 30 36 05 61  ...download906.a
[  712.225488] skb data:000000e0: 76 61 73 74 03 63 6f 6d 00 00 01 00 01 c0 da 16  vast.com........
[  712.325496] skb data:000000f0: ca 17 2a 8d 37 40 df 9d 82 d4 85 9d 28 da 7c 82  ..*.7@......(.|.
[  712.425505] skb data:00000100: 00 7c 4f c7 7a b7 cc c1 ec 9e 02 66 9e 02 c8 b0  .|O.z......f....
[  712.525513] skb data:00000110: b4 5c 6a 24 8a 04 1c c3 4d c0 c8 fa 0f 2d a6 cc  .\j$....M....-..
[  712.625521] skb data:00000120: a3 5a c9 25 25 c5 10 34 e9 ac c1 2f 62 28 dd bf  .Z.%%..4.../b(..
[  712.725530] skb data:00000130: 19 74 43 9b 1a e0 46 5a 44 c9 0d 03 7f 21 61 b2  .tC...FZD....!a.
[  712.825538] skb data:00000140: 5e a6 09 8b e1 47 99 f5 00 f0 d4 ec 33 79 bb 08  ^....G......3y..
[  712.925547] skb data:00000150: ab 16 6a 17 27 5f 1f 6a 6f fd ae c3 e6 ab 35 24  ..j.'_.jo.....5$
[  713.025555] skb data:00000160: 6d 8b c0 33 1f c8 ff 06 c0 3e eb 8e 93 af 5e 0c  m..3.....>....^.
[  713.125563] skb data:00000170: 47 1b 41 40 28 57 48 be 6f 79 e2 dc c3 ed 00 00  G.A@(WH.oy......
[  713.225572] skb data:00000180: 00 15 58 8f 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ..X......\'....\
[  713.325580] skb data:00000190: 27 ef 08 00 45 00 00 3d b6 9a 40 00 7f 06 a2 f1  '...E..=..@.....
[  713.425589] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 15 00 50 0a db 39 df  .....2.R...P..9.
[  713.525597] skb data:000001b0: 3c 2f 3e 11 50 18 ff ff a2 5e 00 00 64 65 6e 74  </>.P....^..dent
[  713.625605] skb data:000001c0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  713.725613] skb data:000001d0: 0a 98 0d 99 47 2b ed 9f 6a 89 9f 84 62 18 59 ce  ....G+..j...b.Y.
[  713.825621] skb data:000001e0: f9 c7 c6 08 33 7c db 30 53 9e 58 3e 33 35 5a 6a  ....3|.0S.X>35Zj
[  713.925630] skb data:000001f0: a8 99 4b 93 e0 bd 9b b5 8c 93 52 a4 eb 33 9d f4  ..K.......R..3..
[  714.086695] ------------[ cut here ]------------
[  714.186713] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  714.286722] Hardware name: PowerEdge SC440
[  714.386733] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  714.486742] Call Trace:
[  714.686760]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  714.886776]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  715.086791]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  715.286806]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  715.486822]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  715.686839]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  715.886854]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  716.086870]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  716.286885]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  716.486901]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  716.686917]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  716.886932]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  717.086947]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  717.286960]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  717.486974]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  717.686987]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  717.887005]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  718.087019]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  718.287034]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  718.487049]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  718.687064]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  718.887077]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  719.087092]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  719.287105]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  719.487119]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  719.687133]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  719.887147]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  720.087160]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  720.287173]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  720.487188]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  720.687203]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  720.887217]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  721.087231]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  721.287244]  [<c1031ead>] ? irq_exit+0x63/0x68
[  721.487257]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  721.687270]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  721.887284]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  722.087297]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  722.287312]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  722.387323] ---[ end trace df8deff2ad2a976d ]---
[  722.487334] 100: csum_start 306, offset 16, headroom 390, headlen 87, len 87
[  722.587345] nr_frags=0 gso_size=0
[  722.687355]
[  722.787367] skb data:00000000: 00 78 1d c7 00 00 00 00 00 00 00 00 00 00 00 00  .x..............
[  722.887379] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  722.987390] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  723.087399] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  723.187407] skb data:00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  723.287415] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  723.387423] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  723.487431] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  723.587439] skb data:00000080: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 08 00 45 00  ...\'....\'...E.
[  723.687448] skb data:00000090: 00 30 00 00 40 00 3a 06 9e 95 d4 32 0c 56 c0 a8  .0..@.:....2.V..
[  723.787456] skb data:000000a0: 01 02 00 50 06 17 46 e7 e5 65 ca 4a f8 64 70 12  ...P..F..e.J.dp.
[  723.887464] skb data:000000b0: 16 d0 d4 b0 00 00 02 04 05 ac 01 01 04 02 da 76  ...............v
[  723.987473] skb data:000000c0: 00 35 00 2c df 47 69 dd 01 00 00 01 00 00 00 00  .5.,.Gi.........
[  724.087481] skb data:000000d0: 00 00 07 74 6f 6f 6c 62 61 72 06 67 6f 6f 67 6c  ...toolbar.googl
[  724.187490] skb data:000000e0: 65 03 63 6f 6d 00 00 01 00 01 00 00 00 00 00 00  e.com...........
[  724.287498] skb data:000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  724.387506] skb data:00000100: 00 54 1d c7 00 00 00 00 00 00 00 00 00 00 00 00  .T..............
[  724.487515] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  724.587523] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  724.687531] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  724.787540] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  724.887548] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  724.987556] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  725.087565] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  725.187573] skb data:00000180: 00 0a e6 ac 07 db 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  725.287581] skb data:00000190: 27 ef 08 00 45 00 00 49 b6 f5 40 00 7f 06 a2 8a  '...E..I..@.....
[  725.387590] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 19 00 50 cc 53 da 48  .....2.R...P.S.H
[  725.487598] skb data:000001b0: 47 54 a4 18 50 18 ff ff a2 6a 00 00 2c 20 63 68  GT..P....j.., ch
[  725.587606] skb data:000001c0: 75 6e 6b 65 64 2c 20 69 64 65 6e 74 69 74 79 2c  unked, identity,
[  725.687614] skb data:000001d0: 20 74 72 61 69 6c 65 72 73 0d 0a 0d 0a 00 00 00   trailers.......
[  725.787623] skb data:000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  725.887631] skb data:000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  725.989879] ------------[ cut here ]------------
[  726.089990] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  726.190007] Hardware name: PowerEdge SC440
[  726.290017] Pid: 2956, comm: fah6 Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  726.390025] Call Trace:
[  726.590041]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  726.790056]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  726.990071]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  727.190086]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  727.390101]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  727.590117]  [<c1083e35>] ? __slab_alloc+0x285/0x4ef
[  727.790132]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  727.990148]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  728.190162]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  728.390178]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  728.590193]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  728.790208]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  728.990224]  [<c12b5601>] ? neigh_resolve_output+0xd0/0x280
[  729.190239]  [<c12be0a5>] ? eth_header+0x0/0x9b
[  729.390255]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  729.590270]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  729.790285]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  729.990300]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  730.190315]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  730.390330]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  730.590345]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  730.790359]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  730.990374]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  731.190389]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  731.390404]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  731.590420]  [<c1129273>] ? xfs_file_aio_read+0x181/0x27a
[  731.790435]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  731.990450]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  732.190465]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  732.390479]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  732.590494]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  732.790508]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  732.990522]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  733.190536]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  733.390549]  [<c1031ead>] ? irq_exit+0x63/0x68
[  733.590563]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  733.790577]  [<c1088c0f>] ? sys_read+0x68/0x6a
[  733.990591]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  734.190605]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  734.290613] ---[ end trace df8deff2ad2a976e ]---
[  734.390622] 100: csum_start 306, offset 16, headroom 390, headlen 146, len 146
[  734.490630] nr_frags=0 gso_size=0
[  734.590636]
[  734.690644] skb data:00000000: 00 84 5b c7 80 35 54 c1 00 00 00 00 00 00 00 00  ..[..5T.........
[  734.790653] skb data:00000010: 01 00 00 00 00 00 00 00 00 00 00 c0 31 7a 03 c1  ............1z..
[  734.890662] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  734.990670] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 24 f4 7c b7  ............$.|.
[  735.090678] skb data:00000040: 00 00 00 00 00 00 00 00 9d 6e ac 57 32 2c 53 a7  .........n.W2,S.
[  735.190687] skb data:00000050: fa 82 6a 39 12 28 ba 9d 4b 9c bf 30 64 a5 71 30  ..j9.(..K..0d.q0
[  735.290695] skb data:00000060: 85 2f e2 93 e5 72 f6 56 d8 df 5e a9 88 bb 07 63  ./...r.V..^....c
[  735.390704] skb data:00000070: c6 e8 91 b1 b7 40 77 8f 54 9f df 81 5c 16 56 64  [email protected]...\.Vd
[  735.490712] skb data:00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  735.590720] skb data:00000090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  735.690729] skb data:000000a0: 00 00 00 00 01 00 00 00 67 d5 da 34 c6 d0 e2 4b  ........g..4...K
[  735.790737] skb data:000000b0: b2 85 0e 8f c3 67 ac 47 20 85 3c 7f 11 86 a2 4d  .....g.G .<....M
[  735.890746] skb data:000000c0: e1 a3 b0 c1 54 86 8d 7b e9 f1 0c dc 03 b0 6f cd  ....T..{......o.
[  735.990754] skb data:000000d0: c8 94 7d 6e 78 fb 1b 8a b6 da 78 3a d0 c7 17 7a  ..}nx.....x:...z
[  736.090762] skb data:000000e0: 10 1b 6f 83 97 ba c0 bb ff ef 3f 12 3f 12 89 14  ..o.......?.?...
[  736.190771] skb data:000000f0: 3d 01 66 bf 5c f2 59 24 b7 60 5e 76 f7 e5 5f 00  =.f.\.Y$.`^v.._.
[  736.290779] skb data:00000100: 00 f8 2c c7 00 00 00 00 00 00 00 00 00 00 00 00  ..,.............
[  736.390787] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  736.490795] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  736.590804] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  736.690812] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  736.790820] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  736.890829] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  736.990837] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  737.090845] skb data:00000180: 00 00 00 00 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  737.190853] skb data:00000190: 27 ef 08 00 45 00 00 84 b7 05 40 00 7f 06 10 ac  '...E.....@.....
[  737.290862] skb data:000001a0: c0 a8 01 02 4a 7d 27 9b 06 1b 00 50 55 fa 51 79  ....J}'....PU.Qy
[  737.390870] skb data:000001b0: f0 df ec 83 50 18 ff ff 34 39 00 00 3a 20 24 56  ....P...49..: $V
[  737.490878] skb data:000001c0: 65 72 73 69 6f 6e 3d 31 0d 0a 43 6f 6e 6e 65 63  ersion=1..Connec
[  737.590887] skb data:000001d0: 74 69 6f 6e 3a 20 4b 65 65 70 2d 41 6c 69 76 65  tion: Keep-Alive
[  737.690895] skb data:000001e0: 2c 20 54 45 0d 0a 54 45 3a 20 64 65 66 6c 61 74  , TE..TE: deflat
[  737.790903] skb data:000001f0: 65 2c 20 67 7a 69 70 2c 20 63 68 75 6e 6b 65 64  e, gzip, chunked
[  737.890911] skb data:00000200: 2c 20 69 64 65 6e 74 69 74 79 2c 20 74 72 61 69  , identity, trai
[  737.990920] skb data:00000210: 6c 65 72 73 0d 0a 0d 0a 34 32 36 36 50 31 34 35  lers....4266P145
[  738.090928] skb data:00000220: 31 31 56 30 30 30 30 30 30 30 30 30 30 30 30 30  11V0000000000000
[  738.190936] skb data:00000230: 39 30 36 49 30 30 45 37 37 42 45 35 5f 35 37 35  906I00E77BE5_575
[  738.290944] skb data:00000240: 2e 66 73 2c 53 3d 31 32 34 32 39 3a 32 2c 53 9a  .fs,S=12429:2,S.
[  738.390953] skb data:00000250: 73 e3 6a fd 88 f1 5f 91 84 4a df b2 dc e5 04 0b  s.j..._..J......
[  738.490961] skb data:00000260: 83 fd 06 69 f9 fb 5b c2 55 fe c5 a6 df 11 b6 1c  ...i..[.U.......
[  738.590969] skb data:00000270: b3 b3 a8 e5 37 56 12 23 a3 05 d9 34 15 06 71 ab  ....7V.#...4..q.
[  739.154435] ------------[ cut here ]------------
[  739.254453] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  739.354462] Hardware name: PowerEdge SC440
[  739.454472] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  739.554481] Call Trace:
[  739.754499]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  739.954515]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  740.154530]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  740.354546]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  740.554562]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  740.754578]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  740.954594]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  741.154611]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  741.354626]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  741.554643]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  741.754658]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  741.954674]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  742.154690]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  742.354705]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  742.554721]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  742.754736]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  742.954751]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  743.154765]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  743.354780]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  743.554795]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  743.754810]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  743.954825]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  744.154840]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  744.354855]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  744.554871]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  744.754887]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  744.954902]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  745.154916]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  745.354932]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  745.554948]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  745.754964]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  745.954979]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  746.154992]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  746.355009]  [<c1031ead>] ? irq_exit+0x63/0x68
[  746.555023]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  746.755036]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  746.955050]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  747.155063]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  747.355078]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  747.455086] ---[ end trace df8deff2ad2a976f ]---
[  747.555094] 100: csum_start 306, offset 16, headroom 390, headlen 173, len 173
[  747.655102] nr_frags=0 gso_size=0
[  747.755108]
[  747.855117] skb data:00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  747.955125] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.055134] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.155142] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.255150] skb data:00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.355159] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.455167] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.555175] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  748.655184] skb data:00000080: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 08 00 45 20  ...\'....\'...E
[  748.755192] skb data:00000090: 00 30 74 a0 00 00 30 11 c7 88 18 38 74 92 c0 a8  .0t...0....8t...
[  748.855200] skb data:000000a0: 01 02 6c 91 3d 63 00 1c 66 5f 27 c6 02 94 fb da  ..l.=c..f_'.....
[  748.955209] skb data:000000b0: f9 96 7a cb d2 6c 26 66 b7 0e 71 37 e5 3c d7 8d  ..z..l&f..q7.<..
[  749.055217] skb data:000000c0: 24 5f 58 61 a6 ad 55 0a 9d 1f ce 5f fa 92 d1 67  $_Xa..U...._...g
[  749.155226] skb data:000000d0: fa e2 9a 9d 2a 68 6e 06 fd 0d 79 b8 cb 02 9a 3e  ....*hn...y....>
[  749.255234] skb data:000000e0: 9f 84 be 76 65 73 47 64 7f 07 88 fa cf 85 03 8a  ...vesGd........
[  749.355243] skb data:000000f0: 72 37 d0 0f bc 43 55 2d df 44 6f e0 dd 9b b6 1a  r7...CU-.Do.....
[  749.455251] skb data:00000100: 00 98 29 c7 00 00 00 00 00 00 00 00 00 00 00 00  ..).............
[  749.555259] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  749.655268] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  749.755277] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  749.855285] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  749.955294] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  750.055302] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  750.155310] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  750.255319] skb data:00000180: 00 16 17 a4 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  .........\'....\
[  750.355327] skb data:00000190: 27 ef 08 00 45 00 00 9f b7 a2 40 00 7f 06 0f f4  '...E.....@.....
[  750.455335] skb data:000001a0: c0 a8 01 02 4a 7d 27 9b 06 25 00 50 58 65 0f b7  ....J}'..%.PXe..
[  750.555344] skb data:000001b0: 58 0c fe 7d 50 18 ff ff 34 54 00 00 65 74 3d 37  X..}P...4T..et=7
[  750.655352] skb data:000001c0: 33 30 7c 63 73 3d 61 33 79 30 67 39 72 35 0d 0a  30|cs=a3y0g9r5..
[  750.755360] skb data:000001d0: 43 6f 6f 6b 69 65 32 3a 20 24 56 65 72 73 69 6f  Cookie2: $Versio
[  750.855369] skb data:000001e0: 6e 3d 31 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a  n=1..Connection:
[  750.955377] skb data:000001f0: 20 4b 65 65 70 2d 41 6c 69 76 65 2c 20 54 45 0d   Keep-Alive, TE.
[  751.055385] skb data:00000200: 0a 54 45 3a 20 64 65 66 6c 61 74 65 2c 20 67 7a  .TE: deflate, gz
[  751.155393] skb data:00000210: 69 70 2c 20 63 68 75 6e 6b 65 64 2c 20 69 64 65  ip, chunked, ide
[  751.255402] skb data:00000220: 6e 74 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d  ntity, trailers.
[  751.355410] skb data:00000230: 0a 0d 0a 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  751.455418] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  751.555426] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  751.655434] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  751.755443] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  752.055018] ------------[ cut here ]------------
[  752.155056] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  752.255104] Hardware name: PowerEdge SC440
[  752.355152] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  752.455162] Call Trace:
[  752.655194]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  752.855215]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  753.055245]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  753.255262]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  753.455277]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  753.655294]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  753.855310]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  754.055327]  [<c1224852>] ? rtl8139_start_xmit+0x51/0x13a
[  754.255341]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  754.455357]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  754.655373]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  754.855389]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  755.055405]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  755.255420]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  755.455435]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  755.655450]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  755.855465]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  756.055479]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  756.255495]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  756.455510]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  756.655525]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  756.855541]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  757.055555]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  757.255571]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  757.455586]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  757.655601]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  757.855617]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  758.055632]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  758.255647]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  758.455662]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  758.655677]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  758.855692]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  759.055707]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  759.255720]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  759.455734]  [<c1031ead>] ? irq_exit+0x63/0x68
[  759.655747]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  759.855761]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  760.055775]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  760.255787]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  760.455801]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  760.555810] ---[ end trace df8deff2ad2a9770 ]---
[  760.655819] 100: csum_start 306, offset 16, headroom 390, headlen 88, len 88
[  760.755826] nr_frags=0 gso_size=0
[  760.855833]
[  760.955842] skb data:00000000: 00 90 92 d2 00 00 00 00 00 00 00 00 00 00 08 00  ................
[  761.055851] skb data:00000010: 45 c0 00 84 09 5b 00 00 40 01 db 0b c0 a8 0a 01  E....[..@.......
[  761.155859] skb data:00000020: c0 a8 0a 01 03 01 fc fe 00 00 00 00 45 00 00 68  ............E..h
[  761.255867] skb data:00000030: 00 00 40 00 ff 01 e5 5d c0 a8 0a 01 c0 a8 0a e5  ..@....]........
[  761.355875] skb data:00000040: 08 00 09 14 47 73 00 41 00 00 00 00 00 00 00 00  ....Gs.A........
[  761.455884] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 46 6c 6f 72  ............Flor
[  761.555892] skb data:00000060: 69 61 6e 20 46 6f 72 73 74 65 72 20 3c 6f 63 74  ian Forster <oct
[  761.655900] skb data:00000070: 6f 40 76 65 72 70 6c 61 6e 74 2e 6f 72 67 00 00  [email protected]..
[  761.755909] skb data:00000080: 00 0e 2e 5c 27 e6 00 30 05 07 f3 1a 08 00 45 00  ...\'..0......E.
[  761.855917] skb data:00000090: 00 28 51 b6 40 00 80 06 c7 5f c0 a8 c7 0f 55 0e  .(Q.@...._....U.
[  761.955926] skb data:000000a0: 04 f4 0c 41 00 50 0c c5 91 a2 c0 76 ad e2 50 10  ...A.P.....v..P.
[  762.055934] skb data:000000b0: ff 3c b5 8b 00 00 00 00 00 00 00 00 bd 6b fb 67  .<...........k.g
[  762.155942] skb data:000000c0: f4 05 fe 14 bb f0 73 7f a4 27 3f 47 39 b1 9e 78  ......s..'?G9..x
[  762.255951] skb data:000000d0: 17 94 e8 43 3f 1f dd e4 a3 db 7c a4 d7 f0 15 6b  ...C?.....|....k
[  762.355959] skb data:000000e0: 9d 16 1b 6a f0 73 a1 e5 8b 17 ad aa 1e 4e 2e 64  ...j.s.......N.d
[  762.455967] skb data:000000f0: 2e 8e ea e1 c3 41 b1 98 0b 77 aa 6b 7c d4 ab db  .....A...w.k|...
[  762.555976] skb data:00000100: 00 bc 92 d2 00 00 00 00 00 00 00 00 00 00 08 00  ................
[  762.655984] skb data:00000110: 45 c0 00 84 09 5b 00 00 40 01 db 0b c0 a8 0a 01  E....[..@.......
[  762.755992] skb data:00000120: c0 a8 0a 01 03 01 fc fe 00 00 00 00 45 00 00 68  ............E..h
[  762.856004] skb data:00000130: 00 00 40 00 ff 01 e5 5d c0 a8 0a 01 c0 a8 0a e5  ..@....]........
[  762.956012] skb data:00000140: 08 00 09 14 47 73 00 41 00 00 00 00 00 00 00 00  ....Gs.A........
[  763.056020] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 46 6c 6f 72  ............Flor
[  763.156028] skb data:00000160: 69 61 6e 20 46 6f 72 73 74 65 72 20 3c 6f 63 74  ian Forster <oct
[  763.256037] skb data:00000170: 6f 40 76 65 72 70 6c 61 6e 74 2e 6f 72 67 00 00  [email protected]..
[  763.356045] skb data:00000180: 00 15 58 8f 00 00 00 0e 2e 5c 27 b2 00 0e 2e 5c  ..X......\'....\
[  763.456053] skb data:00000190: 27 ef 08 00 45 00 00 4a b8 16 40 00 7f 06 a1 68  '[email protected]
[  763.556061] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 2b 00 50 ff e6 5c a2  .....2.R.+.P..\.
[  763.656070] skb data:000001b0: 6a a6 77 69 50 18 fa f0 a2 6b 00 00 70 2c 20 63  j.wiP....k..p, c
[  763.756078] skb data:000001c0: 68 75 6e 6b 65 64 2c 20 69 64 65 6e 74 69 74 79  hunked, identity
[  763.856087] skb data:000001d0: 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d 0a 00 00  , trailers......
[  763.956096] skb data:000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  764.056104] skb data:000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  764.454219] ------------[ cut here ]------------
[  764.554236] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  764.654245] Hardware name: PowerEdge SC440
[  764.754255] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  764.854263] Call Trace:
[  765.054281]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  765.254297]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  765.454313]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  765.654328]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  765.854343]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  766.054359]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  766.254374]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  766.454392]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  766.654407]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  766.854423]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  767.054438]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  767.254454]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  767.454470]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  767.654485]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  767.854500]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  768.054514]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  768.254529]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  768.454544]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  768.654560]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  768.854575]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  769.054590]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  769.254605]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  769.454620]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  769.654635]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  769.854650]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  770.054666]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  770.254681]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  770.454695]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  770.654709]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  770.854724]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  771.054738]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  771.254751]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  771.454765]  [<c1031ead>] ? irq_exit+0x63/0x68
[  771.654778]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  771.854792]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  772.054806]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  772.254820]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  772.454834]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  772.554843] ---[ end trace df8deff2ad2a9771 ]---
[  772.654851] 100: csum_start 306, offset 16, headroom 390, headlen 75, len 75
[  772.754859] nr_frags=0 gso_size=0
[  772.854866]
[  772.954875] skb data:00000000: 00 b0 27 cc 93 4a d3 70 a0 6a d9 ec d7 87 92 21  ..'..J.p.j.....!
[  773.054883] skb data:00000010: 30 ee 16 e5 ea 75 ad 30 f3 93 99 1e d1 76 0b c2  0....u.0.....v..
[  773.154892] skb data:00000020: 60 32 0c 69 62 56 99 d8 53 7d 0f d0 ab f1 a5 66  `2.ibV..S}.....f
[  773.254900] skb data:00000030: 7b a1 ae e7 18 ab 62 fa fc 87 20 f6 46 50 7d 76  {.....b... .FP}v
[  773.354908] skb data:00000040: af b9 53 21 17 56 24 b0 a1 d1 bb fc 57 0d d7 ab  ..S!.V$.....W...
[  773.454917] skb data:00000050: 97 78 b8 47 32 60 46 5e 37 af b1 95 d4 2b 25 f1  .x.G2`F^7....+%.
[  773.554925] skb data:00000060: 27 a6 25 c6 dc 7b 5b 62 89 5c f9 a2 d7 ba af da  '.%..{[b.\......
[  773.654933] skb data:00000070: a6 93 be 49 35 81 3d 6a 54 e7 8c 9a cb 3d 00 00  ...I5.=jT....=..
[  773.754942] skb data:00000080: 00 21 97 d3 fe cb 00 1a a0 38 8a 1b 08 00 45 00  .!.......8....E.
[  773.854950] skb data:00000090: 00 2f a0 85 00 00 71 11 20 3c 63 ee 59 c9 c0 a8  ./....q. <c.Y...
[  773.954958] skb data:000000a0: 0a 9d 9d 41 14 b9 00 1b 85 1a bc e9 02 53 5e 2f  ...A.........S^/
[  774.054967] skb data:000000b0: f7 80 13 fe f6 e0 7f 34 51 c2 e5 e2 69 00 00 00  .......4Q...i...
[  774.154975] skb data:000000c0: 00 00 00 32 3f 5e bc e5 01 00 00 01 00 00 00 00  ...2?^..........
[  774.254983] skb data:000000d0: 00 00 05 69 6d 67 30 33 02 77 61 02 75 73 08 6d  ...img03.wa.us.m
[  774.354992] skb data:000000e0: 61 6e 67 61 66 6f 78 03 63 6f 6d 00 00 01 00 01  angafox.com.....
[  774.455004] skb data:000000f0: 01 00 01 c0 0c 00 01 00 01 00 00 00 43 00 04 5e  ............C..^
[  774.555012] skb data:00000100: 00 bc 27 cc 00 10 00 00 00 00 00 00 d6 8f 0e c1  ..'.............
[  774.655021] skb data:00000110: c3 fb 7a 03 f2 41 b6 93 9b 0d 78 73 3c c3 07 11  ..z..A....xs<...
[  774.755029] skb data:00000120: 3f dc db 4e b8 03 5c 7a 57 c0 47 fc 2a 86 b4 08  ?..N..\zW.G.*...
[  774.855037] skb data:00000130: 7e 8b 66 bc 2d 54 c5 c9 84 ee dd 66 84 93 1b 49  ~.f.-T.....f...I
[  774.955046] skb data:00000140: c1 27 b1 75 fe 8d c3 e1 32 92 dd 99 85 41 a0 7f  .'.u....2....A..
[  775.055054] skb data:00000150: 96 24 95 6f f7 a3 2b 3d e7 e6 67 12 e9 8a 08 2d  .$.o..+=..g....-
[  775.155062] skb data:00000160: 0c f4 cf f2 c1 4e 91 a0 30 b9 24 05 ec 00 39 14  .....N..0.$...9.
[  775.255071] skb data:00000170: 7c f7 21 e2 c8 93 19 2b cf 0f eb 7b f4 98 00 00  |.!....+...{....
[  775.355079] skb data:00000180: 00 21 97 4b 5a 01 00 0e 2e 5c 27 b2 00 0e 2e 5c  .!.KZ....\'....\
[  775.455087] skb data:00000190: 27 ef 08 00 45 00 00 3d b8 67 40 00 7f 06 a1 24  '...E..=.g@....$
[  775.555096] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 1f 00 50 50 9b 3d 36  .....2.R...PP.=6
[  775.655104] skb data:000001b0: 5e 28 1c 8a 50 18 fa 8f a2 5e 00 00 64 65 6e 74  ^(..P....^..dent
[  775.755112] skb data:000001c0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  775.855120] skb data:000001d0: 0a 45 45 0d 1f 69 15 f3 b3 31 c6 9c df 9e b9 06  .EE..i...1......
[  775.955129] skb data:000001e0: 34 17 58 c8 be af 97 02 5d 82 36 73 34 dd 97 96  4.X.....].6s4...
[  776.055137] skb data:000001f0: 1e ac 90 ae 21 d0 09 eb f0 b5 53 10 ed 6a eb b7  ....!.....S..j..
[  776.186389] ------------[ cut here ]------------
[  776.286408] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  776.386417] Hardware name: PowerEdge SC440
[  776.486428] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  776.586437] Call Trace:
[  776.786454]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  776.986470]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  777.186485]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  777.386500]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  777.586514]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  777.786530]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  777.986546]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  778.186561]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  778.386576]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  778.586592]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  778.786607]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  778.986623]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  779.186638]  [<c12b5601>] ? neigh_resolve_output+0xd0/0x280
[  779.386653]  [<c12be0a5>] ? eth_header+0x0/0x9b
[  779.586668]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  779.786682]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  779.986697]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  780.186712]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  780.386735]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  780.586750]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  780.786764]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  780.986779]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  781.186793]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  781.386808]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  781.586823]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  781.786839]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  781.986854]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  782.186869]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  782.386883]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  782.586897]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  782.786911]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  782.986924]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  783.186937]  [<c1031ead>] ? irq_exit+0x63/0x68
[  783.386951]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  783.586964]  [<c10035c3>] ? do_device_not_available+0x0/0x49
[  783.786977]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  783.986992]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  784.087003] ---[ end trace df8deff2ad2a9772 ]---
[  784.187012] 100: csum_start 306, offset 16, headroom 390, headlen 75, len 75
[  784.287019] nr_frags=0 gso_size=0
[  784.387025]
[  784.487034] skb data:00000000: 00 9c 92 d2 2e 5c 27 b2 00 0e 2e 5c 27 ef 08 00  .....\'....\'...
[  784.587043] skb data:00000010: 45 00 00 68 00 00 40 00 ff 01 f8 40 c0 a8 01 02  E..h..@....@....
[  784.687051] skb data:00000020: c0 a8 01 01 08 00 98 9a b7 ec 00 41 00 00 00 00  ...........A....
[  784.787059] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  784.887067] skb data:00000040: 46 6c 6f 72 69 61 6e 20 46 6f 72 73 74 65 72 20  Florian Forster
[  784.987076] skb data:00000050: 3c 6f 63 74 6f 40 76 65 72 70 6c 61 6e 74 2e 6f  <[email protected]
[  785.087084] skb data:00000060: 72 67 3e 20 68 74 74 70 3a 2f 2f 76 65 72 70 6c  rg> http://verpl
[  785.187092] skb data:00000070: 61 6e 74 2e 6f 72 67 2f 00 00 00 00 00 00 00 00  ant.org/........
[  785.287100] skb data:00000080: 00 0e 2e 5c 27 e6 00 30 05 07 f3 1a 08 00 45 00  ...\'..0......E.
[  785.387109] skb data:00000090: 00 28 51 f5 40 00 80 06 c7 20 c0 a8 c7 0f 55 0e  .(Q.@.... ....U.
[  785.487117] skb data:000000a0: 04 f4 0c 46 00 50 0d 1e ce f0 05 98 13 27 50 10  ...F.P.......'P.
[  785.587125] skb data:000000b0: ff 3c cd 79 00 00 00 00 00 00 00 00 82 42 b1 5c  .<.y.........B.\
[  785.687133] skb data:000000c0: 34 46 b7 4f e5 16 cf 1e 4e f9 d0 af 6b 6b 62 89  4F.O....N...kkb.
[  785.787141] skb data:000000d0: 4b 20 74 72 61 69 6c 65 72 73 0d 0a 0d 0a 00 01  K trailers......
[  785.887150] skb data:000000e0: 19 4e e9 b4 b4 41 ce 1b b4 a8 fd b5 80 18 04 00  .N...A..........
[  785.987158] skb data:000000f0: fe 29 00 00 01 01 08 0a 00 00 b0 77 00 00 b0 77  .).........w...w
[  786.087166] skb data:00000100: 00 8c 92 d2 2e 5c 27 b2 00 0e 2e 5c 27 ef 08 00  .....\'....\'...
[  786.187175] skb data:00000110: 45 00 00 68 00 00 40 00 ff 01 f8 40 c0 a8 01 02  E..h..@....@....
[  786.287183] skb data:00000120: c0 a8 01 01 08 00 98 9a b7 ec 00 41 00 00 00 00  ...........A....
[  786.387191] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  786.487199] skb data:00000140: 46 6c 6f 72 69 61 6e 20 46 6f 72 73 74 65 72 20  Florian Forster
[  786.587207] skb data:00000150: 3c 6f 63 74 6f 40 76 65 72 70 6c 61 6e 74 2e 6f  <[email protected]
[  786.687215] skb data:00000160: 72 67 3e 20 68 74 74 70 3a 2f 2f 76 65 72 70 6c  rg> http://verpl
[  786.787223] skb data:00000170: 61 6e 74 2e 6f 72 67 2f 00 00 00 00 00 00 00 00  ant.org/........
[  786.887231] skb data:00000180: 00 0e 2e 5c 27 b2 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...\'....\'....\
[  786.987240] skb data:00000190: 27 ef 08 00 45 00 00 3d b8 85 40 00 7f 06 a1 06  '...E..=..@.....
[  787.087248] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 2f 00 50 f1 3f dd 4e  .....2.R./.P.?.N
[  787.187256] skb data:000001b0: 80 91 32 21 50 18 ff ff a2 5e 00 00 64 65 6e 74  ..2!P....^..dent
[  787.287264] skb data:000001c0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  787.387273] skb data:000001d0: 0a 6f 73 74 2d 63 68 65 63 6b 3d 30 2c 20 70 72  .ost-check=0, pr
[  787.487281] skb data:000001e0: 65 2d 63 68 65 63 6b 3d 30 0d 0a 50 72 61 67 6d  e-check=0..Pragm
[  787.587289] skb data:000001f0: 61 3a 20 6e 6f 2d 63 61 63 68 65 0d 0a 4c 6f 63  a: no-cache..Loc
[  788.016486] ------------[ cut here ]------------
[  788.116504] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  788.216513] Hardware name: PowerEdge SC440
[  788.316524] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  788.416532] Call Trace:
[  788.616550]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  788.816566]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  789.016581]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  789.216596]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  789.416612]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  789.616630]  [<c13311fe>] ? xfrm_lookup+0x1c/0x3a
[  789.816646]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  790.016662]  [<c1224852>] ? rtl8139_start_xmit+0x51/0x13a
[  790.216677]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  790.416692]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  790.616708]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  790.816723]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  791.016740]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  791.216756]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  791.416771]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  791.616786]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  791.816801]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  792.016816]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  792.216830]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  792.416845]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  792.616860]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  792.816876]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  793.016890]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  793.216906]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  793.416921]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  793.616936]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  793.816953]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  794.016968]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  794.216982]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  794.416997]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  794.617013]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  794.817027]  [<c1031cd4>] ? __do_softirq+0x96/0xfa
[  795.017042]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  795.217056]  [<c1031ead>] ? irq_exit+0x63/0x68
[  795.417070]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  795.617084]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  795.817099]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  796.017114]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  796.217129]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  796.317137] ---[ end trace df8deff2ad2a9773 ]---
[  796.417145] 100: csum_start 306, offset 16, headroom 390, headlen 75, len 75
[  796.517152] nr_frags=0 gso_size=0
[  796.617159]
[  796.717168] skb data:00000000: 00 b8 05 c7 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  796.817177] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  796.917186] skb data:00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  797.017194] skb data:00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  797.117202] skb data:00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  797.217211] skb data:00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  797.317219] skb data:00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  797.417227] skb data:00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  797.517236] skb data:00000080: 00 0e 2e 5c 27 e6 00 30 05 07 f3 1a 08 00 45 00  ...\'..0......E.
[  797.617244] skb data:00000090: 00 28 52 0c 40 00 80 06 60 17 c0 a8 c7 0f bc 8a  .(R.@...`.......
[  797.717253] skb data:000000a0: 04 6a 0c 45 00 50 0d 05 55 80 1b ff cc ee 50 10  .j.E.P..U.....P.
[  797.817261] skb data:000000b0: fa 40 14 df 00 00 00 00 00 00 00 00 f6 da 3f 4e  .@............?N
[  797.917270] skb data:000000c0: f6 db 20 4f 4b 0d 0a 54 72 61 6e 73 66 65 72 2d  .. OK..Transfer-
[  798.017278] skb data:000000d0: 45 6e 63 6f 64 69 6e 67 3a 20 63 68 75 6e 6b 65  Encoding: chunke
[  798.117286] skb data:000000e0: 64 0d 0a 58 2d 50 6f 77 65 72 65 64 2d 42 79 3a  d..X-Powered-By:
[  798.217294] skb data:000000f0: 20 50 48 50 2f 34 2e 34 2e 34 2d 38 2b 65 74 63   PHP/4.4.4-8+etc
[  798.317303] skb data:00000100: 00 b0 05 c7 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  798.417311] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  798.517319] skb data:00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  798.617327] skb data:00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  798.717335] skb data:00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  798.817344] skb data:00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  798.917352] skb data:00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  799.017361] skb data:00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  799.117369] skb data:00000180: 00 16 17 5e ef ec 00 0e 2e 5c 27 b2 00 0e 2e 5c  ...^.....\'....\
[  799.217378] skb data:00000190: 27 ef 08 00 45 00 00 3d b8 f5 40 00 7f 06 a0 96  '...E..=..@.....
[  799.317386] skb data:000001a0: c0 a8 01 02 d4 32 0c 52 06 31 00 50 28 3b 73 4a  .....2.R.1.P(;sJ
[  799.417394] skb data:000001b0: 8b 40 32 30 50 18 ff ff a2 5e 00 00 64 65 6e 74  .@20P....^..dent
[  799.517403] skb data:000001c0: 69 74 79 2c 20 74 72 61 69 6c 65 72 73 0d 0a 0d  ity, trailers...
[  799.617411] skb data:000001d0: 0a 92 5f 9e 97 93 9f 98 92 9a 02 14 b2 30 b7 04  .._..........0..
[  799.717420] skb data:000001e0: 89 65 e6 21 94 19 99 a6 02 01 00 00 00 ff ff 03  .e.!............
[  799.817428] skb data:000001f0: 00 cc a7 a3 30 5a 00 00 00 0d 0a 30 0d 0a 0d 0a  ....0Z.....0....
[  800.314252] ------------[ cut here ]------------
[  800.414269] WARNING: at net/core/dev.c:1945 skb_csum_start_bug+0x46/0x133()
[  800.514278] Hardware name: PowerEdge SC440
[  800.614288] Pid: 2960, comm: FahCore_78.exe Tainted: G        W   2.6.36-rc2-FS-00210-geedff42 #1
[  800.714297] Call Trace:
[  800.914315]  [<c102d87c>] ? warn_slowpath_common+0x67/0x8c
[  801.114330]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  801.314345]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  801.514360]  [<c102d8bc>] ? warn_slowpath_null+0x1b/0x1f
[  801.714376]  [<c12adca9>] ? skb_csum_start_bug+0x46/0x133
[  801.914393]  [<c102456f>] ? __wake_up_sync_key+0x3c/0x52
[  802.114408]  [<c12a7bbb>] ? skb_copy_and_csum_dev+0x2a/0xaf
[  802.314424]  [<c1224852>] ? rtl8139_start_xmit+0x51/0x13a
[  802.514439]  [<c122484b>] ? rtl8139_start_xmit+0x4a/0x13a
[  802.714454]  [<c12ae2ee>] ? dev_hard_start_xmit+0x220/0x4cc
[  802.914469]  [<c12bfc3d>] ? sch_direct_xmit+0xac/0x174
[  803.114483]  [<c12c3fb9>] ? nf_iterate+0x69/0x7c
[  803.314498]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  803.514513]  [<c12b013c>] ? dev_queue_xmit+0xc7/0x354
[  803.714528]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  803.914543]  [<c12e8ae2>] ? ip_finish_output+0x11c/0x2b6
[  804.114557]  [<c12e8f61>] ? ip_output+0xa4/0xc3
[  804.314570]  [<c12e89c6>] ? ip_finish_output+0x0/0x2b6
[  804.514583]  [<c12e5049>] ? ip_forward_finish+0x39/0x44
[  804.714596]  [<c12e3a88>] ? ip_rcv_finish+0xe8/0x39f
[  804.914609]  [<c12ad01d>] ? __netif_receive_skb+0x237/0x2b3
[  805.114625]  [<c12ad62b>] ? netif_receive_skb+0x5f/0x64
[  805.314640]  [<c12ad67e>] ? napi_gro_complete+0x4e/0x94
[  805.514654]  [<c12ad9ba>] ? dev_gro_receive+0x158/0x1f5
[  805.714668]  [<c12adba4>] ? napi_gro_receive+0x16/0x1f
[  805.914683]  [<c1217f0b>] ? tg3_poll_work+0x5bc/0xbfb
[  806.114698]  [<c1006e50>] ? nommu_sync_single_for_device+0x0/0x1
[  806.314713]  [<c121ce78>] ? tg3_poll+0x43/0x194
[  806.514726]  [<c12ad7d3>] ? net_rx_action+0xcc/0x15b
[  806.714740]  [<c1031cbd>] ? __do_softirq+0x7f/0xfa
[  806.914755]  [<c1053dd9>] ? handle_IRQ_event+0x48/0xa6
[  807.114769]  [<c1018390>] ? ack_apic_level+0x5f/0x1dd
[  807.314783]  [<c10568ab>] ? move_native_irq+0x9/0x3e
[  807.514796]  [<c1031d5f>] ? do_softirq+0x27/0x2a
[  807.714810]  [<c1031ead>] ? irq_exit+0x63/0x68
[  807.914823]  [<c1003dda>] ? do_IRQ+0x44/0xa1
[  808.114836]  [<c1031e7b>] ? irq_exit+0x31/0x68
[  808.314849]  [<c101654e>] ? smp_apic_timer_interrupt+0x53/0x83
[  808.514863]  [<c1002d29>] ? common_interrupt+0x29/0x30
[  808.714877]  [<c1390000>] ? quirk_ati_exploding_mce+0x46/0x7a
[  808.814885] ---[ end trace df8deff2ad2a9774 ]---
[  808.914893] 100: csum_start 306, offset 16, headroom 390, headlen 179, len 179
[  809.014900] nr_frags=0 gso_size=0
[  809.114906]
[  809.214915] skb data:00000000: 00 a8 05 c7 00 10 00 00 00 00 00 00 00 00 00 00  ................
[  809.314923] skb data:00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  809.414931] skb data:00000020: 00 00 00 00 00 00 00 00 30 30 30 30 30 30 30 39  ........00000009
[  809.514940] skb data:00000030: 30 36 49 30 31 42 32 45 41 32 45 5f 31 2e 66 73  06I01B2EA2E_1.fs
[  809.614947] skb data:00000040: 2c 53 3d 35 31 37 31 3a 32 2c 53 53 53 2c 53 01  ,S=5171:2,SSS,S.
[  809.714956] skb data:00000050: 00 00 00 00 01 69 03 64 69 72 02 62 67 00 00 01  .....i.dir.bg...
[  809.814964] skb data:00000060: 00 01 c0 0c 00 01 00 01 00 00 07 c9 00 04 c2 91  ................
[  809.914972] skb data:00000070: 3f 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ?...............
[  810.014980] skb data:00000080: 00 0e 2e 5c 27 ef 00 0e 2e 5c 27 b2 08 00 45 00  ...\'....\'...E.
[  810.114989] skb data:00000090: 00 2e c2 18 00 00 72 11 58 b7 57 06 15 3f c0 a8  ......r.X.W..?..
[  810.214997] skb data:000000a0: 01 02 85 42 13 02 00 1a e2 be cc 12 02 3e b5 ed  ...B.........>..
[  810.315007] skb data:000000b0: 0b c6 2d 3e a7 ad 70 7c 6c af 14 ab 8e 6e 3d 63  ..->..p|l....n=c
[  810.415015] skb data:000000c0: 12 85 00 68 d3 e5 0a 44 61 74 65 3a 20 57 65 64  ...h...Date: Wed
[  810.515023] skb data:000000d0: 2c 20 32 35 20 41 75 67 20 32 30 31 30 20 30 37  , 25 Aug 2010 07
[  810.615031] skb data:000000e0: 3a 30 30 3a 34 31 20 47 4d 54 0d 0a 53 65 72 76  :00:41 GMT..Serv
[  810.715040] skb data:000000f0: 65 72 3a 20 41 70 61 63 68 65 0d 0a 56 61 72 79  er: Apache..Vary
[  810.815048] skb data:00000100: 00 b8 05 c7 00 10 00 00 00 00 00 00 00 00 00 00  ................
[  810.915056] skb data:00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  811.015064] skb data:00000120: 00 00 00 00 00 00 00 00 30 30 30 30 30 30 30 39  ........00000009
[  811.115072] skb data:00000130: 30 36 49 30 31 42 32 45 41 32 45 5f 31 2e 66 73  06I01B2EA2E_1.fs
[  811.215081] skb data:00000140: 2c 53 3d 35 31 37 31 3a 32 2c 53 53 53 2c 53 01  ,S=5171:2,SSS,S.
[  811.315089] skb data:00000150: 00 00 00 00 01 69 03 64 69 72 02 62 67 00 00 01  .....i.dir.bg...
[  811.415097] skb data:00000160: 00 01 c0 0c 00 01 00 01 00 00 07 c9 00 04 c2 91  ................
[  811.515105] skb data:00000170: 3f 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ?...............
[  811.615113] skb data:00000180: 00 26 9e 0f b0 bf 00 0e 2e 5c 27 b2 00 0e 2e 5c  .&.......\'....\
[  811.715122] skb data:00000190: 27 ef 08 00 45 00 00 a5 04 e8 40 00 7f 06 4a 97  '[email protected].
[  811.815130] skb data:000001a0: c0 a8 01 02 5b 67 8e c2 04 3e 00 50 1b 94 70 c6  ....[g...>.P..p.
[  811.915138] skb data:000001b0: fd ca fd 54 50 18 ff ff ac 6b 00 00 44 25 33 42  ...TP....k..D%3B
[  812.015146] skb data:000001c0: 25 32 34 71 70 70 25 33 44 30 25 33 42 25 32 34  %24qpp%3D0%3B%24
[  812.115154] skb data:000001d0: 71 74 25 33 44 39 25 35 46 30 25 35 46 30 74 3b  qt%3D9%5F0%5F0t;
[  812.215162] skb data:000001e0: 20 41 53 50 53 45 53 53 49 4f 4e 49 44 53 43 43   ASPSESSIONIDSCC
[  812.315170] skb data:000001f0: 44 51 41 42 54 3d 41 45 48 48 44 4e 43 43 50 4c  DQABT=AEHHDNCCPL
[  812.415178] skb data:00000200: 44 42 4a 42 49 4b 46 46 41 42 4f 47 4c 42 3b 20  DBJBIKFFABOGLB;
[  812.515186] skb data:00000210: 42 49 47 69 70 53 65 72 76 65 72 53 4d 41 52 54  BIGipServerSMART
[  812.615194] skb data:00000220: 3d 36 38 38 33 32 35 33 38 36 2e 32 33 30 34 30  =688325386.23040
[  812.715203] skb data:00000230: 2e 30 30 30 30 0d 0a 0d 0a 00 00 00 00 00 00 00  .0000...........
[  812.815211] skb data:00000240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  812.915219] skb data:00000250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  813.015228] skb data:00000260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  813.115237] skb data:00000270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

Current status refresh:

Just tried 2.6.35.4, but without luck.

See the oops at http://picpaste.com/8240d73e92bd2ba25a9b4019010fcabf.jpg

Eric Dumazet and I are continuing to try and find a solution for my problem via private email.

Thanks,
Plamen

It seems we forget to reset skb->ip_summed to CHECKSUM_NONE in various tunnels (gre, ipip, sit, ip6_tunnel), before re-entering stack.

Add ip_summed initialization in skb_tunnel_rx(), and remove it from ipmr / ip6mr

---

include/net/dst.h |    1 +
net/ipv4/ipmr.c   |    1 -
net/ipv6/ip6mr.c  |    1 -
3 files changed, 1 insertion(+), 2 deletions(-)

index 81d1413..2ef5580 100644
@@ -237,6 +237,7 @@ static inline void skb_dst_force(struct sk_buff *skb)
  */
 static inline void skb_tunnel_rx(struct sk_buff *skb, struct net_device *dev)
 {
+	skb->ip_summed = CHECKSUM_NONE;
 	skb->dev = dev;
 	/* TODO : stats should be SMP safe */
 	dev->stats.rx_packets++;

index 179fcab..70fa43b 100644
@@ -1837,7 +1837,6 @@ static int __pim_rcv(struct mr_table *mrt, struct sk_buff *skb,
 	skb_pull(skb, (u8*)encap - skb->data);
 	skb_reset_network_header(skb);
 	skb->protocol = htons(ETH_P_IP);
-	skb->ip_summed = 0;
 	skb->pkt_type = PACKET_HOST;
 
 	skb_tunnel_rx(skb, reg_dev);

index 66078da..c4d0146 100644
@@ -661,7 +661,6 @@ static int pim6_rcv(struct sk_buff *skb)
 	skb_pull(skb, (u8 *)encap - skb->data);
 	skb_reset_network_header(skb);
 	skb->protocol = htons(ETH_P_IPV6);
-	skb->ip_summed = 0;
 	skb->pkt_type = PACKET_HOST;
 
 	skb_tunnel_rx(skb, reg_dev);

This is intentional behavior. For CHECKSUM_COMPLETE packets we subtract off the checksum of the encapsulating headers that we remove so we can still get the benefit of checksum hardware for the inner packet as well. GRE does this using skb_postpull_rcsum(), for IPIP it is implicit because the outer IP header was already removed, etc.