oops in net_rx_action on 64-bit powerpc
Subsystems (2)
I tried booting a post 2.6.27 -git on a Motorola ATCA6101 (very similar to a Maple board). The first time I booted I got the first log below via the serial console. I rebooted and got as far as a login prompt. I was able to log in via the serial console, but then got an almost identical oops again, as shown in the second log below.
I configed out the gigE drivers for the backplane so the only remaining network link was the e100 link used for booting, but the problem remained.
Anyone have any idea what might be causing this?
Thanks,
Chris
Starting xinetd: [ OK ]
Starting cron: [ OK ]
Unable to handle kernel paging request for data at address 0x00100108 Faulting instruction address: 0xc00000000028c1cc Oops: Kernel access of bad area, sig: 11 [#1] SMP NR_CPUS=2 Maple Modules linked in: NIP: c00000000028c1cc LR: c00000000028c13c CTR: 0000000000000000 REGS: c00000000fff7b90 TRAP: 0300 Not tainted (2.6.27-05329-g39076ba) MSR: 9000000000009032 <EE,ME,IR,DR> CR: 22ffff24 XER: 20000000 DAR: 0000000000100108, DSISR: 000000000a000000
TASK = c00000017a061080[0] 'swapper' THREAD: c00000017a078000 CPU: 1
GPR00: 0000000000000000 c00000000fff7e10 c00000000059bfe0 0000000000000020
GPR04: 0000000000000001 c000000178179800 c00000000027fda8 0000000000000000
GPR08: 0000000000000000 0000000000200200 0000000000000001 0000000000100100
GPR12: 0000000022ffff22 c0000000005bc500 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000000000 000000000000000a 0000000000000001 0000000000000001
GPR24: c0000000005a2280 c0000000005f5134 00000000fffd9bbe 00000000000000ec
GPR28: c000000006e30c28 0000000000000020 c000000000543440 c00000017a279b40 NIP [c00000000028c1cc] .net_rx_action+0x1e4/0x26c
LR [c00000000028c13c] .net_rx_action+0x154/0x26c
Call Trace:
[c00000000fff7e10] [c00000000028c13c] .net_rx_action+0x154/0x26c (unreliable)
[c00000000fff7ec0] [c000000000056938] .__do_softirq+0xf8/0x1f4
[c00000000fff7f90] [c000000000024334] .call_do_softirq+0x14/0x24
[c00000017a07b970] [c00000000000bcdc] .do_softirq+0xf0/0x104
[c00000017a07ba10] [c000000000056ae8] .irq_exit+0x70/0x88
[c00000017a07ba90] [c00000000000ba18] .do_IRQ+0x14c/0x244
[c00000017a07bb30] [c000000000004710] hardware_interrupt_entry+0x18/0x1c
--- Exception: 501 at .raw_local_irq_restore+0x38/0x44
LR = .cpu_idle+0xd8/0x154
[c00000017a07be20] [c000000000012068] .cpu_idle+0x118/0x154 (unreliable)
[c00000017a07bec0] [c0000000003d4304] .start_secondary+0x310/0x3e8
[c00000017a07bf90] [c0000000000072b4] .start_secondary_prolog+0x10/0x14Instruction dump:eb61ffd8 eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 7c0803a6 4e800020 e81f0010 7809ffe3 40820038 e93f0008 e97f0000 <f92b0008> f9690000 e95c0008 fb9f0000
root@10:/root> uname -a
Linux 10.41.18.77 2.6.27-05329-g39076ba #1 SMP Tue Oct 21 16:46:06 CST 2008 ppc64 GNU/Linux
root@10:/root> Unable to handle kernel paging request for data at address 0x00100108
Faulting instruction address: 0xc00000000028c1cc
Oops: Kernel access of bad area, sig: 11 [#1] SMP NR_CPUS=2 Maple Modules linked in: NIP: c00000000028c1cc LR: c00000000028c13c CTR: 0000000000000000 REGS: c00000000fff7b90 TRAP: 0300 Not tainted (2.6.27-05329-g39076ba) MSR: 9000000000009032 <EE,ME,IR,DR> CR: 22ffff24 XER: 20000000 DAR: 0000000000100108, DSISR: 000000000a000000
TASK = c00000017a061080[0] 'swapper' THREAD: c00000017a078000 CPU: 1
GPR00: 0000000000000000 c00000000fff7e10 c00000000059bfe0 0000000000000020
GPR04: 0000000000000001 0000000000000001 c00000000027fda8 0000000000000000
GPR08: 0000000000000000 0000000000200200 0000000000000001 0000000000100100
GPR12: 0000000022ffff22 c0000000005bc500 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000000000 000000000000000a 0000000000000001 0000000000000001
GPR24: c0000000005a2280 c0000000005f5134 00000001000387ff 000000000000010c
GPR28: c000000006e30c28 0000000000000020 c000000000543440 c00000017a2b0b40 NIP [c00000000028c1cc] .net_rx_action+0x1e4/0x26c
LR [c00000000028c13c] .net_rx_action+0x154/0x26c
Call Trace:
[c00000000fff7e10] [c00000000028c13c] .net_rx_action+0x154/0x26c (unreliable)
[c00000000fff7ec0] [c000000000056938] .__do_softirq+0xf8/0x1f4
[c00000000fff7f90] [c000000000024334] .call_do_softirq+0x14/0x24
[c00000017a07b970] [c00000000000bcdc] .do_softirq+0xf0/0x104
[c00000017a07ba10] [c000000000056ae8] .irq_exit+0x70/0x88
[c00000017a07ba90] [c00000000000ba18] .do_IRQ+0x14c/0x244
[c00000017a07bb30] [c000000000004710] hardware_interrupt_entry+0x18/0x1c
--- Exception: 501 at .cpu_idle+0xf0/0x154
LR = .cpu_idle+0xd8/0x154
[c00000017a07be20] [c000000000012068] .cpu_idle+0x118/0x154 (unreliable)
[c00000017a07bec0] [c0000000003d4304] .start_secondary+0x310/0x3e8
[c00000017a07bf90] [c0000000000072b4] .start_secondary_prolog+0x10/0x14Instruction dump:eb61ffd8 eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 7c0803a6 4e800020 e81f0010 7809ffe3 40820038 e93f0008 e97f0000 <f92b0008> f9690000 e95c0008 fb9f0000
Quoted 19 lines · Chris Friesen
I tried booting a post 2.6.27 -git on a Motorola ATCA6101 (very similar to a Maple board). The first time I booted I got the first log below via the serial console. I rebooted and got as far as a login prompt. I was able to log in via the serial console, but then got an almost identical oops again, as shown in the second log below. =20
I configed out the gigE drivers for the backplane so the only remaining network link was the e100 link used for booting, but the problem remained.=20
=20
Anyone have any idea what might be causing this?
=20
Thanks,
=20
Chris
=20
=20
Starting xinetd: [ OK ]
Starting cron: [ OK ]
Quoted 1 line
Unable to handle kernel paging request for data at address 0x00100108
that 00100108 pattern looks familiar, I'm not much help here, but I think t= hat had something to do with the list management of the poll_list in a netd= ev struct.
so now you just have to figure out why someone's netdev struct is becoming = NULL. :-)
Quoted 88 lines
Faulting instruction address:
0xc00000000028c1ccOops: Kernel access of bad area, sig: 11 [#1] SMP NR_CPUS=3D2 Maple Modules linked in: NIP: c00000000028c1cc LR: c00000000028c13c CTR: 0000000000000000 REGS: c00000000fff7b90 TRAP: 0300 Not tainted=20(2.6.27-05329-g39076ba)=20
MSR: 9000000000009032 <EE,ME,IR,DR> CR:22ffff24XER: 20000000
DAR: 0000000000100108, DSISR:000000000a000000TASK =3D c00000017a061080[0] 'swapper' THREAD:
c00000017a078000CPU: 1
GPR00: 0000000000000000c00000000fff7e10c00000000059bfe00000000000000020 GPR04: 0000000000000001c000000178179800c00000000027fda80000000000000000 GPR08: 0000000000000000 0000000000200200 0000000000000001 0000000000100100 GPR12:0000000022ffff22c0000000005bc5000000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR20: 0000000000000000000000000000000a0000000000000001 0000000000000001 GPR24:c0000000005a2280c0000000005f513400000000fffd9bbe00000000000000ecGPR28:c000000006e30c280000000000000020c000000000543440c00000017a279b40NIP [c00000000028c1cc] .net_rx_action+0x1e4/0x26c =20
LR [c00000000028c13c] .net_rx_action+0x154/0x26cCall Trace: [c00000000fff7e10] [c00000000028c13c] .net_rx_action+0x154/0x26c (unreliable) [c00000000fff7ec0] [c000000000056938] .__do_softirq+0xf8/0x1f4 [c00000000fff7f90] [c000000000024334] .call_do_softirq+0x14/0x24 [c00000017a07b970] [c00000000000bcdc] .do_softirq+0xf0/0x104 [c00000017a07ba10] [c000000000056ae8] .irq_exit+0x70/0x88 [c00000017a07ba90] [c00000000000ba18] .do_IRQ+0x14c/0x244 [c00000017a07bb30] [c000000000004710] hardware_interrupt_entry+0x18/0x1c --- Exception: 501 at .raw_local_irq_restore+0x38/0x44 LR =3D .cpu_idle+0xd8/0x154 [c00000017a07be20] [c000000000012068] .cpu_idle+0x118/0x154 (unreliable) [c00000017a07bec0] [c0000000003d4304] .start_secondary+0x310/0x3e8 [c00000017a07bf90] [c0000000000072b4] .start_secondary_prolog+0x10/0x14 Instruction dump:
eb61ffd8eb81ffe0eba1ffe8ebc1fff0ebe1fff87c0803a64e800020e81f00107809ffe340820038e93f0008e97f0000<f92b0008>f9690000e95c0008fb9f0000=20
=20
=20
=20
=20
root@10:/root> uname -a
Linux 10.41.18.77 2.6.27-05329-g39076ba #1 SMP Tue Oct 21 16:46:06 CST 2008 ppc64 GNU/Linux
root@10:/root> Unable to handle kernel paging request for data at address0x00100108
Faulting instruction address:0xc00000000028c1ccOops: Kernel access of bad area, sig: 11 [#1] SMP NR_CPUS=3D2 Maple Modules linked in: NIP: c00000000028c1cc LR: c00000000028c13c CTR: 0000000000000000 REGS: c00000000fff7b90 TRAP: 0300 Not tainted=20(2.6.27-05329-g39076ba)=20
MSR: 9000000000009032 <EE,ME,IR,DR> CR:22ffff24XER: 20000000
DAR: 0000000000100108, DSISR:000000000a000000
TASK =3D c00000017a061080[0] 'swapper' THREAD:c00000017a078000CPU: 1
GPR00: 0000000000000000c00000000fff7e10c00000000059bfe00000000000000020 GPR04: 0000000000000001 0000000000000001c00000000027fda80000000000000000 GPR08: 0000000000000000 0000000000200200 0000000000000001 0000000000100100 GPR12:0000000022ffff22c0000000005bc5000000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR20: 0000000000000000000000000000000a0000000000000001 0000000000000001 GPR24:c0000000005a2280c0000000005f513400000001000387ff000000000000010cGPR28:c000000006e30c280000000000000020c000000000543440c00000017a2b0b40NIP [c00000000028c1cc] .net_rx_action+0x1e4/0x26c =20
LR [c00000000028c13c] .net_rx_action+0x154/0x26cCall Trace: [c00000000fff7e10] [c00000000028c13c] .net_rx_action+0x154/0x26c (unreliable) [c00000000fff7ec0] [c000000000056938] .__do_softirq+0xf8/0x1f4 [c00000000fff7f90] [c000000000024334] .call_do_softirq+0x14/0x24 [c00000017a07b970] [c00000000000bcdc] .do_softirq+0xf0/0x104 [c00000017a07ba10] [c000000000056ae8] .irq_exit+0x70/0x88 [c00000017a07ba90] [c00000000000ba18] .do_IRQ+0x14c/0x244 [c00000017a07bb30] [c000000000004710] hardware_interrupt_entry+0x18/0x1c --- Exception: 501 at .cpu_idle+0xf0/0x154 LR =3D .cpu_idle+0xd8/0x154 [c00000017a07be20] [c000000000012068] .cpu_idle+0x118/0x154 (unreliable) [c00000017a07bec0] [c0000000003d4304] .start_secondary+0x310/0x3e8 [c00000017a07bf90] [c0000000000072b4] .start_secondary_prolog+0x10/0x14 Instruction dump:
eb61ffd8eb81ffe0eba1ffe8ebc1fff0ebe1fff87c0803a64e800020e81f00107809ffe340820038e93f0008e97f0000<f92b0008>f9690000e95c0008fb9f0000=20
David Miller
maintainerFrom: "Brandeburg, Jesse" <[email protected]>
Date: Thu, 23 Oct 2008 14:50:06 -0700
Quoted 18 lines
Quoted 17 lines · Chris Friesen
I tried booting a post 2.6.27 -git on a Motorola ATCA6101 (very similar to a Maple board). The first time I booted I got the first log below via the serial console. I rebooted and got as far as a login prompt. I was able to log in via the serial console, but then got an almost identical oops again, as shown in the second log below.
I configed out the gigE drivers for the backplane so the only remaining network link was the e100 link used for booting, but the problem remained.
Anyone have any idea what might be causing this?
Thanks,
Chris
Quoted 7 lines
Quoted 3 lines
Starting xinetd: [ OK ]
Starting cron: [ OK ]Unable to handle kernel paging request for data at address 0x00100108that 00100108 pattern looks familiar, I'm not much help here, but I think that had something to do with the list management of the
poll_listin a netdev struct.so now you just have to figure out why someone's netdev struct is becoming NULL. :-)
Usually this is an indication of returning the wrong value from the driver's ->poll() routine.
Quoted 27 lines · David Miller
From: "Brandeburg, Jesse" <[email protected]> Date: Thu, 23 Oct 2008 14:50:06 -0700
Quoted 24 lines
Quoted 12 lines · Chris Friesen
I tried booting a post 2.6.27 -git on a Motorola ATCA6101 (very similar to a Maple board). The first time I booted I got the first log below via the serial console. I rebooted and got as far as a login prompt. I was able to log in via the serial console, but then got an almost identical oops again, as shown in the second log below.
I configed out the gigE drivers for the backplane so the only remaining network link was the e100 link used for booting, but the problem remained.
Anyone have any idea what might be causing this?
Quoted 7 lines
Thanks,
Chris
Starting xinetd: [ OK ] Starting cron: [ OK ] Unable to handle kernel paging request for data at address
0x00100108that 00100108 pattern looks familiar, I'm not much help here, but I think that had something to do with the list management of the
poll_listin a netdev struct.
Quoted 5 lines
Quoted 2 lines
so now you just have to figure out why someone's netdev struct is becoming NULL. :-)
Usually this is an indication of returning the wrong value from the driver's ->poll() routine.
Looks like I was wrong before...the remaining ethernet link is an AMD-8111, not an e100. Sorry about that.
I backed out 6ba33ac "amd8111e: delete non NAPI code from the driver". With NAPI disabled, the blade appears stable. With NAPI enabled, the original problem recurred.
So...it would appear that the NAPI code is somehow buggy, and 6ba33ac should probably be reverted until the problem is found and fixed.
Chris
David Miller
maintainerFrom: "Chris Friesen" <[email protected]>
Date: Fri, 24 Oct 2008 17:39:00 -0600
Quoted 3 lines
So...it would appear that the NAPI code is somehow buggy, and 6ba33ac should probably be reverted until the problem is found and fixed.
No I think the problem is simple enough that someone should study the ->poll() routine quickly and audit it's return values.
Quoted 9 lines · David Miller
From: "Chris Friesen" <[email protected]>
Date: Fri, 24 Oct 2008 17:39:00 -0600Quoted 3 lines
So...it would appear that the NAPI code is somehow buggy, and 6ba33ac should probably be reverted until the problem is found and fixed.
No I think the problem is simple enough that someone should study the ->poll() routine quickly and audit it's return values.
Assuming that amd8111e_rx_poll() is the proper routine, there is only one exit point, and it returns "num_rx_pkt". This variable is initialized to zero and increments for each packet sent up the stack.
Chris
David Miller
maintainerFrom: "Chris Friesen" <[email protected]>
Date: Fri, 24 Oct 2008 23:42:48 -0600
Quoted 7 lines
Quoted 6 lines · David Miller
From: "Chris Friesen" <[email protected]>
Date: Fri, 24 Oct 2008 17:39:00 -0600Quoted 3 lines
So...it would appear that the NAPI code is somehow buggy, and 6ba33ac should probably be reverted until the problem is found and fixed.
Quoted 7 lines
Quoted 2 lines
No I think the problem is simple enough that someone should study the ->poll() routine quickly and audit it's return values.
Assuming that
amd8111e_rx_poll()is the proper routine, there is only one exit point, and it returns "num_rx_pkt". This variable is initialized to zero and increments for each packet sent up the stack.
The problematic case in this routine seem to be when the driver processes exactly "budget" number of packets.
In this case it should not call __netif_rx_complete(), it should instead use the rx_not_empty label.
Probably the simplest fix is to get rid of the rx_not_empty label and protect the entire:
/* Receive descriptor is empty now */ spin_lock_irqsave(&lp->lock, flags); __netif_rx_complete(dev, napi); writel(VAL0|RINTEN0, mmio + INTEN0); writel(VAL2 | RDMD0, mmio + CMD0); spin_unlock_irqrestore(&lp->lock, flags);
code block with a test such as:
if (rx_pkt_limit > 0)
(yes, greater than zero, not >= 0)
then replace the rx_not_empty goto with a simple break.
Quoted 10 lines · David Miller
Probably the simplest fix is to get rid of the
rx_not_emptylabel and protect the entire:/* Receive descriptor is empty now */ spin_lock_irqsave(&lp->lock, flags); __netif_rx_complete(dev, napi); writel(VAL0|RINTEN0, mmio + INTEN0); writel(VAL2 | RDMD0, mmio + CMD0); spin_unlock_irqrestore(&lp->lock, flags);
Quoted 7 lines
code block with a test such as:
if (rx_pkt_limit > 0)(yes, greater than zero, not >= 0)
then replace the
rx_not_emptygoto with a simple break.
Are you sure about that? Doing that, if we "--rx_pkt_limit < 0" we'll only break out of the inner while loop. We then check then interrupt status register and potentially loop through the do/while loop again (maybe decrementing rx_pkt_limit again) even though we've used up our budget.
If I leave the label and jump and just add the "rx_pkt_limit > 0" test, it seems to work.
Chris
From: Chris Friesen <[email protected]>
Subject: [PATCH] fix amd8111e rx return code
The amd8111e rx poll routine currently mishandles the case when we process exactly the number of packets specified in the budget.
This patch is basically as suggested by David Miller.
drivers/net/amd8111e.c +8 -6
@@ -833,12 +833,14 @@ static int amd8111e_rx_poll(struct napi_struct *napi, int budget) }while (intr0 &RINT0 ); -/* Receive descriptor is empty now */ -spin_lock_irqsave (&lp->lock, flags); -__netif_rx_complete (dev, napi); -writel (VAL0 |RINTEN0 , mmio +INTEN0 ); -writel (VAL2 |RDMD0 , mmio +CMD0 ); -spin_unlock_irqrestore (&lp->lock, flags); +if (rx_pkt_limit >0 ) { +/* Receive descriptor is empty now */ +spin_lock_irqsave (&lp->lock, flags); +__netif_rx_complete (dev, napi); +writel (VAL0 |RINTEN0 , mmio +INTEN0 ); +writel (VAL2 |RDMD0 , mmio +CMD0 ); +spin_unlock_irqrestore (&lp->lock, flags); + } rx_not_empty:return num_rx_pkt;
David Miller
maintainerFrom: "Chris Friesen" <[email protected]>
Date: Mon, 27 Oct 2008 18:13:54 -0600
Quoted 8 lines
[PATCH] fix amd8111e rx return code
The amd8111e rx poll routine currently mishandles the case when we process exactly the number of packets specified in the budget.
This patch is basically as suggested by David Miller.
Signed-off-by Chris Friesen <[email protected]>
I had to apply this by hand because your email client heavily corrupted the patch.
Thanks.
Quoted 9 lines · David Miller
From: "Chris Friesen" <[email protected]>
Date: Mon, 27 Oct 2008 18:13:54 -0600Quoted 5 lines
[PATCH] fix amd8111e rx return code
The amd8111e rx poll routine currently mishandles the case when we process exactly the number of packets specified in the budget.
Quoted 7 lines
Quoted 3 lines
This patch is basically as suggested by David Miller.
Signed-off-by Chris Friesen <[email protected]>I had to apply this by hand because your email client heavily corrupted the patch.
Crap. Sorry about that, it looks like I forgot to set my line-wrap to 0 before creating the email. @#$#% Thunderbird.
Chris