Skip to contentSkip to conversation

gainfar.c : skb_over_panic (kernel-2.6.32.15)

PATCH In review Eran Liberty 3 participants 10 messages Jun 24, 2010 – Oct 3, 2010 Network drivers

lore
Share & tools
lore

Eran Liberty

author
#0 lore raw

PATCH 1/1 gainfar.c : skb_over_panic (kernel-2.6.32.15)

Fix possible skb_over_panic event in Freescale's "gianfar" driver.

The skb_over_panic occurs due to calling skb_put() within gfar_clean_rx_ring(). This happens if (and only if) shortly prior to the event and a few lined above the skb_put(), an skb was queued back to the priv->rx_recycle queue due to RXBD_LAST or RXBD_ERR status. The skb is queued without properly re-setting its state.

The patch properly reset the skb state.

I have tested this patch on MPC8548 based product and asserted it avoided the skb_over_panic event.

Signed-off-by Eran Liberty <[email protected]>

---

---
 drivers/net/gianfar.c |    9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)
drivers/net/gianfar.c +2 -7
@@ -1854,14 +1854,9 @@
 			if (unlikely(!newskb))
 				newskb = skb;
 			else if (skb) {
-				/*
-				 * We need to reset ->data to what it
-				 * was before gfar_new_skb() re-aligned
-				 * it to an RXBUF_ALIGNMENT boundary
-				 * before we put the skb back on the
-				 * recycle list.
-				 */
 				skb->data = skb->head + NET_SKB_PAD;
+				skb->len = 0;
+				skb_reset_tail_pointer(skb);
 				__skb_queue_head(&priv->rx_recycle, skb);
 			}
 		} else {
maintainer
#1 lore raw

From: Eran Liberty <[email protected]>
Date: Thu, 24 Jun 2010 12:53:23 +0300

Quoted 8 lines

Fix possible skb_over_panic event in Freescale's "gianfar" driver.

The skb_over_panic occurs due to calling skb_put() within gfar_clean_rx_ring(). This happens if (and only if) shortly prior to the event and a few lined above the skb_put(), an skb was queued back to the priv->rx_recycle queue due to RXBD_LAST or RXBD_ERR status. The skb is queued without properly re-setting its state.

Quoted 6 lines

The patch properly reset the skb state.

I have tested this patch on MPC8548 based product and asserted it avoided the skb_over_panic event.

Signed-off-by Eran Liberty <[email protected]>

Eran, this seems to be fixed already. The code in the current tree now reads:

/*
* We need to un-reserve() the skb to what it
* was before gfar_new_skb() re-aligned
* it to an RXBUF_ALIGNMENT boundary
* before we put the skb back on the
* recycle list.
*/
skb_reserve(skb, -GFAR_CB(skb)->alignamount); __skb_queue_head(&priv->rx_recycle, skb);

author
#2 lore raw
Quoted 23 lines · David Miller

From: Eran Liberty <[email protected]>
Date: Thu, 24 Jun 2010 12:53:23 +0300

Quoted 8 lines

Fix possible skb_over_panic event in Freescale's "gianfar" driver.

The skb_over_panic occurs due to calling skb_put() within gfar_clean_rx_ring(). This happens if (and only if) shortly prior to the event and a few lined above the skb_put(), an skb was queued back to the priv->rx_recycle queue due to RXBD_LAST or RXBD_ERR status. The skb is queued without properly re-setting its state.

Quoted 7 lines

The patch properly reset the skb state.

I have tested this patch on MPC8548 based product and asserted it avoided the skb_over_panic event.

Signed-off-by Eran Liberty <[email protected]>

Eran, this seems to be fixed already. The code in the current tree now reads:

Quoted 12 lines

/*
* We need to un-reserve() the skb to what it
* was before gfar_new_skb() re-aligned
* it to an RXBUF_ALIGNMENT boundary
* before we put the skb back on the
* recycle list.
*/
skb_reserve(skb, -GFAR_CB(skb)->alignamount); __skb_queue_head(&priv->rx_recycle, skb);

This code has proved to be insufficient and produce skb_over_panic. The proposed patch fix this.

-- Liberty

maintainer
#3 lore raw

From: Eran Liberty <[email protected]>
Date: Mon, 28 Jun 2010 10:57:08 +0300

Quoted 2 lines

This code has proved to be insufficient and produce skb_over_panic. The proposed patch fix this.

Then you have to post a patch relative to the current code, rather than against the code as it was several releases ago.

Your patch didn't apply, so I can't use it.

author
#4 lore raw

PATCH 1/1 gainfar.c : skb_over_panic (kernel-2.6.32.15)

Quoted 4 lines · David Miller

From: Eran Liberty <[email protected]>
Date: Mon, 28 Jun 2010 10:57:08 +0300

Quoted 10 lines
Quoted 3 lines

This code has proved to be insufficient and produce skb_over_panic. The proposed patch fix this.

Then you have to post a patch relative to the current code, rather than against the code as it was several releases ago.

Your patch didn't apply, so I can't use it.

Upon cleaning up my patch for the latest kernel I realized I do not
like: the previous partial fix, the fix in ucc_geth.c, the fix in the current latest kernel, and my own previously proposed patch. They all tried to undo the alignment skb_reserve done in gfar_new_skb() before queuing the skb into the rw_recycle, because upon getting a new one in gfar_new_skb() if the skb is from the rx_recycle pool rather then newly allocated it is reserved twice.

Instead of trying to undo the skb_reserve this proposed patch will make sure the alignment skb_reserve is done once, upon allocating the skb and not when taken out of the rx_recycle pool. Eliminating the need to undo anything before queue skb back to the pool.

This patch will apply cleanly against the 2.6.32.15. Another patch will be submitted separately for the current Linus tree.

-- Liberty

Signed-off-by Eran Liberty <[email protected]>
---
 drivers/net/gianfar.c |   52 +++++++++++++++++++++++++-------------------------
 1 file changed, 27 insertions(+), 25 deletions(-)
drivers/net/gianfar.c +27 -25
@@ -1568,6 +1568,15 @@
 	schedule_work(&priv->reset_task);
 }
 
+static void gfar_align_skb(struct sk_buff *skb)
+{
+	/* We need the data buffer to be aligned properly.  We will reserve
+	 * as many bytes as needed to align the data properly
+	 */
+	skb_reserve(skb, RXBUF_ALIGNMENT -
+		(((unsigned long) skb->data) & (RXBUF_ALIGNMENT - 1)));
+}
+
 /* Interrupt Handler for Transmit complete */
 static int gfar_clean_tx_ring(struct net_device *dev)
 {

@@ -1620,9 +1629,10 @@
 		 */
 		if (skb_queue_len(&priv->rx_recycle) < priv->rx_ring_size &&
 				skb_recycle_check(skb, priv->rx_buffer_size +
-					RXBUF_ALIGNMENT))
+					RXBUF_ALIGNMENT)) {
+			gfar_align_skb(skb);
 			__skb_queue_head(&priv->rx_recycle, skb);
-		else
+		} else
 			dev_kfree_skb_any(skb);
 
 		priv->tx_skbuff[skb_dirtytx] = NULL;

@@ -1696,28 +1706,29 @@
 	bdp->lstatus = lstatus;
 }
 
+static struct sk_buff * gfar_alloc_skb(struct net_device *dev)
+{
+	struct gfar_private *priv = netdev_priv(dev);
+	struct sk_buff *skb = NULL;
+
+	skb = netdev_alloc_skb(dev, priv->rx_buffer_size + RXBUF_ALIGNMENT);
+	if (!skb)
+		return NULL;
+
+	gfar_align_skb(skb);
+
+	return skb;
+}
 
 struct sk_buff * gfar_new_skb(struct net_device *dev)
 {
-	unsigned int alignamount;
 	struct gfar_private *priv = netdev_priv(dev);
 	struct sk_buff *skb = NULL;
 
 	skb = __skb_dequeue(&priv->rx_recycle);
-	if (!skb)
-		skb = netdev_alloc_skb(dev,
-				priv->rx_buffer_size + RXBUF_ALIGNMENT);
 
 	if (!skb)
-		return NULL;
-
-	alignamount = RXBUF_ALIGNMENT -
-		(((unsigned long) skb->data) & (RXBUF_ALIGNMENT - 1));
-
-	/* We need the data buffer to be aligned properly.  We will reserve
-	 * as many bytes as needed to align the data properly
-	 */
-	skb_reserve(skb, alignamount);
+		skb = gfar_alloc_skb(dev);
 
 	return skb;
 }

@@ -1853,17 +1864,8 @@
 
 			if (unlikely(!newskb))
 				newskb = skb;
-			else if (skb) {
-				/*
-				 * We need to reset ->data to what it
-				 * was before gfar_new_skb() re-aligned
-				 * it to an RXBUF_ALIGNMENT boundary
-				 * before we put the skb back on the
-				 * recycle list.
-				 */
-				skb->data = skb->head + NET_SKB_PAD;
+			else if (skb)
 				__skb_queue_head(&priv->rx_recycle, skb);
-			}
 		} else {
 			/* Increment the number of packets */
 			dev->stats.rx_packets++;
author
#5 lore raw

PATCH 1/1 gainfar.c : code cleanup

This patch relates to "[PATCH] gainfar.c : skb_over_panic (kernel-2.6.32.15)"

While in 2.6.32.15 it actually fixed a bug here it merely cleans up the previous attempts to fix the bug with a more coherent code.

Currently before queuing skb into the rx_recycle it is "un-skb_reserve"-ed so when taken out in gfar_new_skb() it wont be reserved twice.

This patch makes sure the alignment skb_reserve is done once, upon allocating the skb and
not when taken out of the rx_recycle pool. Eliminating the need to undo anything before queue skb back to the pool.

NOTE: This patch will compile and is fairly straight forward but I do not have environment to test it as I did with the 2.6.32.15 fix.

-- Liberty

Signed-off-by Eran Liberty <[email protected]>
---
 drivers/net/gianfar.c |   54 ++++++++++++++++++++++++------------------------
 1 files changed, 27 insertions(+), 27 deletions(-)
drivers/net/gianfar.c +27 -27
index 28b53d1..334c8b5 100644
@@ -2342,6 +2342,15 @@ static void gfar_timeout(struct net_device *dev)
 	schedule_work(&priv->reset_task);
 }
 
+static void gfar_align_skb(struct sk_buff *skb)
+{
+	/* We need the data buffer to be aligned properly.  We will reserve
+	 * as many bytes as needed to align the data properly
+	 */
+	skb_reserve(skb, RXBUF_ALIGNMENT -
+		(((unsigned long) skb->data) & (RXBUF_ALIGNMENT - 1)));
+}
+
 /* Interrupt Handler for Transmit complete */
 static int gfar_clean_tx_ring(struct gfar_priv_tx_q *tx_queue)
 {

@@ -2426,9 +2435,10 @@ static int gfar_clean_tx_ring(struct gfar_priv_tx_q *tx_queue)
 		 */
 		if (skb_queue_len(&priv->rx_recycle) < rx_queue->rx_ring_size &&
 				skb_recycle_check(skb, priv->rx_buffer_size +
-					RXBUF_ALIGNMENT))
+					RXBUF_ALIGNMENT)) {
+			gfar_align_skb(skb);
 			__skb_queue_head(&priv->rx_recycle, skb);
-		else
+		} else
 			dev_kfree_skb_any(skb);
 
 		tx_queue->tx_skbuff[skb_dirtytx] = NULL;

@@ -2491,29 +2501,28 @@ static void gfar_new_rxbdp(struct gfar_priv_rx_q *rx_queue, struct rxbd8 *bdp,
 	gfar_init_rxbdp(rx_queue, bdp, buf);
 }
 
-
-struct sk_buff * gfar_new_skb(struct net_device *dev)
+static struct sk_buff * gfar_alloc_skb(struct net_device *dev)
 {
-	unsigned int alignamount;
 	struct gfar_private *priv = netdev_priv(dev);
 	struct sk_buff *skb = NULL;
 
-	skb = __skb_dequeue(&priv->rx_recycle);
-	if (!skb)
-		skb = netdev_alloc_skb(dev,
-				priv->rx_buffer_size + RXBUF_ALIGNMENT);
-
+	skb = netdev_alloc_skb(dev, priv->rx_buffer_size + RXBUF_ALIGNMENT);
 	if (!skb)
 		return NULL;
 
-	alignamount = RXBUF_ALIGNMENT -
-		(((unsigned long) skb->data) & (RXBUF_ALIGNMENT - 1));
+	gfar_align_skb(skb);
 
-	/* We need the data buffer to be aligned properly.  We will reserve
-	 * as many bytes as needed to align the data properly
-	 */
-	skb_reserve(skb, alignamount);
-	GFAR_CB(skb)->alignamount = alignamount;
+	return skb;
+}
+
+struct sk_buff * gfar_new_skb(struct net_device *dev)
+{
+	struct gfar_private *priv = netdev_priv(dev);
+	struct sk_buff *skb = NULL;
+
+	skb = __skb_dequeue(&priv->rx_recycle);
+	if (!skb)
+		skb = gfar_alloc_skb(dev);
 
 	return skb;
 }

@@ -2666,17 +2675,8 @@ int gfar_clean_rx_ring(struct gfar_priv_rx_q *rx_queue, int rx_work_limit)
 
 			if (unlikely(!newskb))
 				newskb = skb;
-			else if (skb) {
-				/*
-				 * We need to un-reserve() the skb to what it
-				 * was before gfar_new_skb() re-aligned
-				 * it to an RXBUF_ALIGNMENT boundary
-				 * before we put the skb back on the
-				 * recycle list.
-				 */
-				skb_reserve(skb, -GFAR_CB(skb)->alignamount);
+			else if (skb) 
 				__skb_queue_head(&priv->rx_recycle, skb);
-			}
 		} else {
 			/* Increment the number of packets */
 			rx_queue->stats.rx_packets++;
Signature · 1 line
1.5.6.5
#6 lore raw
In reply to Eran Liberty

Hi Eran, David
This bug still exits on current both stable and rc kernel releases reported on the mail list.
http://lkml.org/lkml/2010/9/28/178
Is there any merge plan this patch to current kernel. Regards

2010/6/29 Eran Liberty <[email protected]>:

Quoted 39 lines
Quoted 6 lines · David Miller

From: Eran Liberty <[email protected]>
Date: Mon, 28 Jun 2010 10:57:08 +0300

Quoted 1 line
Quoted 10 lines
Quoted 3 lines

This code has proved to be insufficient and produce skb_over_panic. The proposed patch fix this.

Then you have to post a patch relative to the current code, rather than against the code as it was several releases ago.

Your patch didn't apply, so I can't use it.

Upon cleaning up my patch for the latest kernel I realized I do not
like: the previous partial fix, the fix in ucc_geth.c, the fix in the current latest kernel, and my own previously proposed patch. They all tried to undo the alignment skb_reserve done in gfar_new_skb() before queuing the skb into the rw_recycle, because upon getting a new one in gfar_new_skb() if the skb is from the rx_recycle pool rather then newly allocated it is reserved twice.

Instead of trying to undo the skb_reserve this proposed patch will make sure the alignment skb_reserve is done once, upon allocating the skb and not when taken out of the rx_recycle pool. Eliminating the need to undo anything before queue skb back to the pool.

This patch will apply cleanly against the 2.6.32.15. Another patch will be submitted separately for the current Linus tree.

-- Liberty

Signed-off-by Eran Liberty <[email protected]>
author
#7 lore raw

Dear Amin Ak,

A patch has been posted both to the most recent kernel tree at the time and to the long term 2.6.32.X branch. At which point it is out of mine and in the capable hands of the maintainers.

If needs be, I am willing to repost the patches even though they are still out there and probably still fresh enough.

If you apply them and solve your own private overpanic maybe "ack"ing the patch will expedite its trip into the kernel kingdom.

-- Liberty

Quoted 45 lines · emin ak

Hi Eran, David
This bug still exits on current both stable and rc kernel releases reported on the mail list.
http://lkml.org/lkml/2010/9/28/178
Is there any merge plan this patch to current kernel. Regards

2010/6/29 Eran Liberty <[email protected]>:

Quoted 35 lines
Quoted 4 lines · David Miller

From: Eran Liberty <[email protected]>
Date: Mon, 28 Jun 2010 10:57:08 +0300

Quoted 9 lines
Quoted 3 lines

This code has proved to be insufficient and produce skb_over_panic. The proposed patch fix this.

Then you have to post a patch relative to the current code, rather than against the code as it was several releases ago.

Your patch didn't apply, so I can't use it.

Upon cleaning up my patch for the latest kernel I realized I do not
like: the previous partial fix, the fix in ucc_geth.c, the fix in the current latest kernel, and my own previously proposed patch. They all tried to undo the alignment skb_reserve done in gfar_new_skb() before queuing the skb into the rw_recycle, because upon getting a new one in gfar_new_skb() if the skb is from the rx_recycle pool rather then newly allocated it is reserved twice.

Instead of trying to undo the skb_reserve this proposed patch will make sure the alignment skb_reserve is done once, upon allocating the skb and not when taken out of the rx_recycle pool. Eliminating the need to undo anything before queue skb back to the pool.

This patch will apply cleanly against the 2.6.32.15. Another patch will be submitted separately for the current Linus tree.

-- Liberty

Signed-off-by Eran Liberty <[email protected]>
#8 lore raw

2010/10/3 Eran Liberty <[email protected]>:

Quoted 6 lines

Dear Amin Ak,

A patch has been posted both to the most recent kernel tree at the time and to the long term 2.6.32.X branch. At which point it is out of mine and in the capable hands of the maintainers.

Quoted 8 lines

If needs be, I am willing to repost the patches even though they are still out there and probably still fresh enough.

If you apply them and solve your own private overpanic maybe "ack"ing the patch will expedite its trip into the kernel kingdom.

-- Liberty

Hi Eran,
Firstly thanks for your answer. After my previous e-mail, I found that your patch has been applied to linux-2.6.36-rc5 kernel, so that I have tried this kernel with my development card (MPC8572DS) and run my test at full line rate ip traffic. But sadly, it has crashed also. I'am not sure this crashes related with skb recycling or the problem that you solved with your patch, but looks like similar. I have report this bug to the maillist today
(http://permalink.gmane.org/gmane.linux.kernel/1043818), but also attached the crash logs at the end of this mail.
Do you think this crashes related with same problem as yours or caused by another reason?
Thanks.
Emin

Crash type 1:

root@mpc8572ds:~# skb_over_panic: text:c0226280 len:1171 put:1171
head:eed6d000 data:eed63040 tail:0xeed6d4d3 end:0xeed63660 dev:<NULL>
------------[ cut here ]------------
kernel BUG at net/core/skbuff.c:127!
Oops: Exception in kernel mode, sig: 5 [#1]
SMP NR_CPUS=2 MPC8572 DS
last sysfs file: /sys/devices/pci0002:03/0002:03:00.0/subsystem_device
Modules linked in:
NIP: c023bdcc LR: c023bdcc CTR: c01f3ff8
REGS: effe7d70 TRAP: 0700   Not tainted  (2.6.36-rc5)
MSR: 00029000 <EE,ME,CE>  CR: 22028024  XER: 20000000
TASK = ef83e9a0[9] 'ksoftirqd/1' THREAD: ef856000 CPU: 1
GPR00: c023bdcc effe7e20 ef83e9a0 0000007c 00021000 ffffffff c01f7b98 c03ccf1c
GPR08: c03c69d4 c03f94b4 00c4e000 00000004 20028048 1001a108 ef211000 efb52d90
GPR16: efb52e38 efb52870 00000000 ef211800 00000008 00000009 efb52800 00000037
GPR24: ef24e180 ef2be040 00000000 ef211948 efb52b80 00000493 ef015940 ef386600
NIP [c023bdcc] skb_put+0x8c/0x94
LR [c023bdcc] skb_put+0x8c/0x94
Call Trace:
[effe7e20] [c023bdcc] skb_put+0x8c/0x94 (unreliable)
[effe7e30] [c0226280] gfar_clean_rx_ring+0x104/0x4b8
[effe7e90] [c02269dc] gfar_poll+0x3a8/0x60c
[effe7f60] [c024928c] net_rx_action+0xf8/0x1a4
[effe7fa0] [c0042524] __do_softirq+0xe0/0x178
[effe7ff0] [c000e59c] call_do_softirq+0x14/0x24
[ef857f50] [c0004840] do_softirq+0x90/0xa0
[ef857f70] [c00430e4] run_ksoftirqd+0xb4/0x164
[ef857fb0] [c00586b4] kthread+0x7c/0x80
[ef857ff0] [c000e9a8] kernel_thread+0x4c/0x68
Instruction dump:
81030098 2f800000 409e000c 3d20c037 3809a19c 3c60c037 7c8802a6 7d695b78
3863b010 90010008 4cc63182 4be016c5 <0fe00000> 48000000 9421fff0 7c0802a6
Kernel panic - not syncing: Fatal exception in interrupt

Crash type: 2

Faulting instruction address: 0xc026c1dc

Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2 MPC8572 DS
last sysfs file: /sys/devices/pci0002:03/0002:03:00.0/subsystem_device
Modules linked in:
NIP: c026c1dc LR: c026bfac CTR: 00000000
REGS: effebd00 TRAP: 0300   Not tainted  (2.6.36-rc5)
MSR: 00029000 <EE,ME,CE>  CR: 42028042  XER: 00000000
DEAR: 0000cad8, ESR: 00000000

TASK = ef83cde0[3] 'ksoftirqd/0' THREAD: ef84a000 CPU: 0
GPR00: 00000005 effebdb0 ef83cde0 00000000 000001b9 00000000 c1008060 00000000
GPR08: 02c3f605 0000ca00 000005b9 0000ca00 b653a6c7 7af823f0 ef217000 efbab590
GPR16: efbab638 efbab070 00000000 ef217800 00000008 00000018 efbab000 00000028
GPR24: c03f971c c0410000 c0400000 c03f94b4 effea000 ef316e40 00000000 eecb685e NIP [c026c1dc] ip_rcv+0x3f8/0x808
LR [c026bfac] ip_rcv+0x1c8/0x808

Call Trace:
[effebdb0] [c026c204] ip_rcv+0x420/0x808 (unreliable)
[effebde0] [c02482dc] __netif_receive_skb+0x2f8/0x324
[effebe10] [c02483a4] netif_receive_skb+0x9c/0xb0
[effebe30] [c0226308] gfar_clean_rx_ring+0x18c/0x4b8
[effebe90] [c02269dc] gfar_poll+0x3a8/0x60c
[effebf60] [c024928c] net_rx_action+0xf8/0x1a4
[effebfa0] [c0042524] __do_softirq+0xe0/0x178
[effebff0] [c000e59c] call_do_softirq+0x14/0x24
[ef84bf50] [c0004840] do_softirq+0x90/0xa0
[ef84bf70] [c00430e4] run_ksoftirqd+0xb4/0x164
[ef84bfb0] [c00586b4] kthread+0x7c/0x80
[ef84bff0] [c000e9a8] kernel_thread+0x4c/0x68
Instruction dump:
8148003c 318a0001 7d690194 91680038 9188003c 4bfffd78 7fa3eb78 48002a29
2f830000 40beff50 817d0048 5569003c <a00900d8> 2f800005 419e0034 2f800003
Kernel panic - not syncing: Fatal exception in interrupt
author
#9 lore raw

Dear Amin Ak,

Your crash does strike some resemblance to the one I used to experience.

I have looked up the latest kernel in the LXR and it seems my patch was not adopted.
You are welcome to grab it off the mailing list ( https://patchwork.kernel.org/patch/108660/ ), apply it, and if it makes you happy try to re-push it upstream.

-- Liberty

Quoted 105 lines · emin ak

2010/10/3 Eran Liberty <[email protected]>:

Quoted 6 lines

Dear Amin Ak,

A patch has been posted both to the most recent kernel tree at the time and to the long term 2.6.32.X branch. At which point it is out of mine and in the capable hands of the maintainers.

Quoted 9 lines

If needs be, I am willing to repost the patches even though they are still out there and probably still fresh enough.

If you apply them and solve your own private overpanic maybe "ack"ing the patch will expedite its trip into the kernel kingdom.

-- Liberty

Hi Eran,
Firstly thanks for your answer. After my previous e-mail, I found that your patch has been applied to linux-2.6.36-rc5 kernel, so that I have tried this kernel with my development card (MPC8572DS) and run my test at full line rate ip traffic. But sadly, it has crashed also. I'am not sure this crashes related with skb recycling or the problem that you solved with your patch, but looks like similar. I have report this bug to the maillist today
(http://permalink.gmane.org/gmane.linux.kernel/1043818), but also attached the crash logs at the end of this mail.
Do you think this crashes related with same problem as yours or caused by another reason?
Thanks.
Emin

Crash type 1:

root@mpc8572ds:~# skb_over_panic: text:c0226280 len:1171 put:1171
head:eed6d000 data:eed63040 tail:0xeed6d4d3 end:0xeed63660 dev:<NULL>
------------[ cut here ]------------
kernel BUG at net/core/skbuff.c:127!
Oops: Exception in kernel mode, sig: 5 [#1]
SMP NR_CPUS=2 MPC8572 DS
last sysfs file: /sys/devices/pci0002:03/0002:03:00.0/subsystem_device
Modules linked in:
NIP: c023bdcc LR: c023bdcc CTR: c01f3ff8
REGS: effe7d70 TRAP: 0700   Not tainted  (2.6.36-rc5)
MSR: 00029000 <EE,ME,CE>  CR: 22028024  XER: 20000000
TASK = ef83e9a0[9] 'ksoftirqd/1' THREAD: ef856000 CPU: 1
GPR00: c023bdcc effe7e20 ef83e9a0 0000007c 00021000 ffffffff c01f7b98 c03ccf1c
GPR08: c03c69d4 c03f94b4 00c4e000 00000004 20028048 1001a108 ef211000 efb52d90
GPR16: efb52e38 efb52870 00000000 ef211800 00000008 00000009 efb52800 00000037
GPR24: ef24e180 ef2be040 00000000 ef211948 efb52b80 00000493 ef015940 ef386600
NIP [c023bdcc] skb_put+0x8c/0x94
LR [c023bdcc] skb_put+0x8c/0x94
Call Trace:
[effe7e20] [c023bdcc] skb_put+0x8c/0x94 (unreliable)
[effe7e30] [c0226280] gfar_clean_rx_ring+0x104/0x4b8
[effe7e90] [c02269dc] gfar_poll+0x3a8/0x60c
[effe7f60] [c024928c] net_rx_action+0xf8/0x1a4
[effe7fa0] [c0042524] __do_softirq+0xe0/0x178
[effe7ff0] [c000e59c] call_do_softirq+0x14/0x24
[ef857f50] [c0004840] do_softirq+0x90/0xa0
[ef857f70] [c00430e4] run_ksoftirqd+0xb4/0x164
[ef857fb0] [c00586b4] kthread+0x7c/0x80
[ef857ff0] [c000e9a8] kernel_thread+0x4c/0x68
Instruction dump:
81030098 2f800000 409e000c 3d20c037 3809a19c 3c60c037 7c8802a6 7d695b78
3863b010 90010008 4cc63182 4be016c5 <0fe00000> 48000000 9421fff0 7c0802a6
Kernel panic - not syncing: Fatal exception in interrupt

Crash type: 2

Faulting instruction address: 0xc026c1dc

Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2 MPC8572 DS
last sysfs file: /sys/devices/pci0002:03/0002:03:00.0/subsystem_device
Modules linked in:
NIP: c026c1dc LR: c026bfac CTR: 00000000
REGS: effebd00 TRAP: 0300   Not tainted  (2.6.36-rc5)
MSR: 00029000 <EE,ME,CE>  CR: 42028042  XER: 00000000
DEAR: 0000cad8, ESR: 00000000

TASK = ef83cde0[3] 'ksoftirqd/0' THREAD: ef84a000 CPU: 0
GPR00: 00000005 effebdb0 ef83cde0 00000000 000001b9 00000000 c1008060 00000000
GPR08: 02c3f605 0000ca00 000005b9 0000ca00 b653a6c7 7af823f0 ef217000 efbab590
GPR16: efbab638 efbab070 00000000 ef217800 00000008 00000018 efbab000 00000028
GPR24: c03f971c c0410000 c0400000 c03f94b4 effea000 ef316e40 00000000 eecb685e NIP [c026c1dc] ip_rcv+0x3f8/0x808
LR [c026bfac] ip_rcv+0x1c8/0x808

Call Trace:
[effebdb0] [c026c204] ip_rcv+0x420/0x808 (unreliable)
[effebde0] [c02482dc] __netif_receive_skb+0x2f8/0x324
[effebe10] [c02483a4] netif_receive_skb+0x9c/0xb0
[effebe30] [c0226308] gfar_clean_rx_ring+0x18c/0x4b8
[effebe90] [c02269dc] gfar_poll+0x3a8/0x60c
[effebf60] [c024928c] net_rx_action+0xf8/0x1a4
[effebfa0] [c0042524] __do_softirq+0xe0/0x178
[effebff0] [c000e59c] call_do_softirq+0x14/0x24
[ef84bf50] [c0004840] do_softirq+0x90/0xa0
[ef84bf70] [c00430e4] run_ksoftirqd+0xb4/0x164
[ef84bfb0] [c00586b4] kthread+0x7c/0x80
[ef84bff0] [c000e9a8] kernel_thread+0x4c/0x68
Instruction dump:
8148003c 318a0001 7d690194 91680038 9188003c 4bfffd78 7fa3eb78 48002a29
2f830000 40beff50 817d0048 5569003c <a00900d8> 2f800005 419e0034 2f800003
Kernel panic - not syncing: Fatal exception in interrupt