Skip to contentSkip to conversation

oops in net_rx_action on 64-bit powerpc

BUG Chris Friesen 3 participants 10 messages Oct 23, 2008 – Oct 28, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I tried booting a post 2.6.27 -git on a Motorola ATCA6101 (very similar to a Maple board). The first time I booted I got the first log below via the serial console. I rebooted and got as far as a login prompt. I was able to log in via the serial console, but then got an almost identical oops again, as shown in the second log below.

I configed out the gigE drivers for the backplane so the only remaining network link was the e100 link used for booting, but the problem remained.

Anyone have any idea what might be causing this?

Thanks,

Chris

Starting xinetd: [ OK ]
Starting cron: [ OK ]

Unable to handle kernel paging request for data at address 0x00100108
Faulting instruction address: 0xc00000000028c1cc
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2 Maple
Modules linked in:
NIP: c00000000028c1cc LR: c00000000028c13c CTR: 0000000000000000
REGS: c00000000fff7b90 TRAP: 0300   Not tainted  (2.6.27-05329-g39076ba)
MSR: 9000000000009032 <EE,ME,IR,DR>  CR: 22ffff24  XER: 20000000
DAR: 0000000000100108, DSISR: 000000000a000000

TASK = c00000017a061080[0] 'swapper' THREAD: c00000017a078000 CPU: 1
GPR00: 0000000000000000 c00000000fff7e10 c00000000059bfe0 0000000000000020
GPR04: 0000000000000001 c000000178179800 c00000000027fda8 0000000000000000
GPR08: 0000000000000000 0000000000200200 0000000000000001 0000000000100100
GPR12: 0000000022ffff22 c0000000005bc500 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000000000 000000000000000a 0000000000000001 0000000000000001
GPR24: c0000000005a2280 c0000000005f5134 00000000fffd9bbe 00000000000000ec
GPR28: c000000006e30c28 0000000000000020 c000000000543440 c00000017a279b40 NIP [c00000000028c1cc] .net_rx_action+0x1e4/0x26c
LR [c00000000028c13c] .net_rx_action+0x154/0x26c

Call Trace:
[c00000000fff7e10] [c00000000028c13c] .net_rx_action+0x154/0x26c (unreliable)
[c00000000fff7ec0] [c000000000056938] .__do_softirq+0xf8/0x1f4
[c00000000fff7f90] [c000000000024334] .call_do_softirq+0x14/0x24
[c00000017a07b970] [c00000000000bcdc] .do_softirq+0xf0/0x104
[c00000017a07ba10] [c000000000056ae8] .irq_exit+0x70/0x88
[c00000017a07ba90] [c00000000000ba18] .do_IRQ+0x14c/0x244
[c00000017a07bb30] [c000000000004710] hardware_interrupt_entry+0x18/0x1c
--- Exception: 501 at .raw_local_irq_restore+0x38/0x44
     LR = .cpu_idle+0xd8/0x154
[c00000017a07be20] [c000000000012068] .cpu_idle+0x118/0x154 (unreliable)
[c00000017a07bec0] [c0000000003d4304] .start_secondary+0x310/0x3e8
[c00000017a07bf90] [c0000000000072b4] .start_secondary_prolog+0x10/0x14

Instruction dump:
eb61ffd8 eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 7c0803a6 4e800020 e81f0010 7809ffe3 40820038 e93f0008 e97f0000 <f92b0008> f9690000 e95c0008 fb9f0000

root@10:/root> uname -a
Linux 10.41.18.77 2.6.27-05329-g39076ba #1 SMP Tue Oct 21 16:46:06 CST 2008 ppc64 GNU/Linux
root@10:/root> Unable to handle kernel paging request for data at address 0x00100108
Faulting instruction address: 0xc00000000028c1cc

Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2 Maple
Modules linked in:
NIP: c00000000028c1cc LR: c00000000028c13c CTR: 0000000000000000
REGS: c00000000fff7b90 TRAP: 0300   Not tainted  (2.6.27-05329-g39076ba)
MSR: 9000000000009032 <EE,ME,IR,DR>  CR: 22ffff24  XER: 20000000
DAR: 0000000000100108, DSISR: 000000000a000000

TASK = c00000017a061080[0] 'swapper' THREAD: c00000017a078000 CPU: 1
GPR00: 0000000000000000 c00000000fff7e10 c00000000059bfe0 0000000000000020
GPR04: 0000000000000001 0000000000000001 c00000000027fda8 0000000000000000
GPR08: 0000000000000000 0000000000200200 0000000000000001 0000000000100100
GPR12: 0000000022ffff22 c0000000005bc500 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000000000 000000000000000a 0000000000000001 0000000000000001
GPR24: c0000000005a2280 c0000000005f5134 00000001000387ff 000000000000010c
GPR28: c000000006e30c28 0000000000000020 c000000000543440 c00000017a2b0b40 NIP [c00000000028c1cc] .net_rx_action+0x1e4/0x26c
LR [c00000000028c13c] .net_rx_action+0x154/0x26c

Call Trace:
[c00000000fff7e10] [c00000000028c13c] .net_rx_action+0x154/0x26c (unreliable)
[c00000000fff7ec0] [c000000000056938] .__do_softirq+0xf8/0x1f4
[c00000000fff7f90] [c000000000024334] .call_do_softirq+0x14/0x24
[c00000017a07b970] [c00000000000bcdc] .do_softirq+0xf0/0x104
[c00000017a07ba10] [c000000000056ae8] .irq_exit+0x70/0x88
[c00000017a07ba90] [c00000000000ba18] .do_IRQ+0x14c/0x244
[c00000017a07bb30] [c000000000004710] hardware_interrupt_entry+0x18/0x1c
--- Exception: 501 at .cpu_idle+0xf0/0x154
     LR = .cpu_idle+0xd8/0x154
[c00000017a07be20] [c000000000012068] .cpu_idle+0x118/0x154 (unreliable)
[c00000017a07bec0] [c0000000003d4304] .start_secondary+0x310/0x3e8
[c00000017a07bf90] [c0000000000072b4] .start_secondary_prolog+0x10/0x14

Instruction dump:
eb61ffd8 eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 7c0803a6 4e800020 e81f0010 7809ffe3 40820038 e93f0008 e97f0000 <f92b0008> f9690000 e95c0008 fb9f0000

#1 lore raw
Quoted 19 lines · Chris Friesen

I tried booting a post 2.6.27 -git on a Motorola ATCA6101 (very similar to a Maple board). The first time I booted I got the first log below via the serial console. I rebooted and got as far as a login prompt. I was able to log in via the serial console, but then got an almost identical oops again, as shown in the second log below. =20
I configed out the gigE drivers for the backplane so the only remaining network link was the e100 link used for booting, but the problem remained.=20
=20
Anyone have any idea what might be causing this?
=20
Thanks,
=20
Chris
=20
=20
Starting xinetd: [ OK ]
Starting cron: [ OK ]

Quoted 1 line
Unable to handle kernel paging request for data at address 0x00100108

that 00100108 pattern looks familiar, I'm not much help here, but I think t= hat had something to do with the list management of the poll_list in a netd= ev struct.

so now you just have to figure out why someone's netdev struct is becoming = NULL. :-)

Quoted 88 lines

Faulting instruction address: 0xc00000000028c1cc

Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=3D2 Maple
Modules linked in:
NIP: c00000000028c1cc LR: c00000000028c13c CTR: 0000000000000000
REGS: c00000000fff7b90 TRAP: 0300   Not tainted=20

(2.6.27-05329-g39076ba)=20
MSR: 9000000000009032 <EE,ME,IR,DR> CR: 22ffff24 XER: 20000000
DAR: 0000000000100108, DSISR: 000000000a000000

TASK =3D c00000017a061080[0] 'swapper' THREAD: c00000017a078000 CPU: 1
GPR00: 0000000000000000 c00000000fff7e10 c00000000059bfe0 0000000000000020 GPR04: 0000000000000001 c000000178179800 c00000000027fda8 0000000000000000 GPR08: 0000000000000000 0000000000200200 0000000000000001 0000000000100100 GPR12:
0000000022ffff22 c0000000005bc500 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR20: 0000000000000000 000000000000000a 0000000000000001 0000000000000001 GPR24: c0000000005a2280 c0000000005f5134 00000000fffd9bbe 00000000000000ec GPR28:
c000000006e30c28 0000000000000020 c000000000543440 c00000017a279b40 NIP [c00000000028c1cc] .net_rx_action+0x1e4/0x26c =20
LR [c00000000028c13c] .net_rx_action+0x154/0x26c

Call Trace:
[c00000000fff7e10] [c00000000028c13c] .net_rx_action+0x154/0x26c
(unreliable) [c00000000fff7ec0] [c000000000056938]
.__do_softirq+0xf8/0x1f4 [c00000000fff7f90] [c000000000024334]
.call_do_softirq+0x14/0x24 [c00000017a07b970] [c00000000000bcdc]
.do_softirq+0xf0/0x104 [c00000017a07ba10] [c000000000056ae8]
.irq_exit+0x70/0x88 [c00000017a07ba90] [c00000000000ba18]
.do_IRQ+0x14c/0x244 [c00000017a07bb30] [c000000000004710]
hardware_interrupt_entry+0x18/0x1c --- Exception: 501 at
      .raw_local_irq_restore+0x38/0x44 LR =3D .cpu_idle+0xd8/0x154
[c00000017a07be20] [c000000000012068] .cpu_idle+0x118/0x154
(unreliable) [c00000017a07bec0] [c0000000003d4304]
.start_secondary+0x310/0x3e8 [c00000017a07bf90] [c0000000000072b4]
.start_secondary_prolog+0x10/0x14 Instruction dump:

eb61ffd8 eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 7c0803a6 4e800020 e81f0010 7809ffe3 40820038 e93f0008 e97f0000 <f92b0008> f9690000 e95c0008 fb9f0000=20
=20
=20
=20
=20
root@10:/root> uname -a
Linux 10.41.18.77 2.6.27-05329-g39076ba #1 SMP Tue Oct 21 16:46:06 CST 2008 ppc64 GNU/Linux
root@10:/root> Unable to handle kernel paging request for data at address 0x00100108
Faulting instruction address: 0xc00000000028c1cc

Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=3D2 Maple
Modules linked in:
NIP: c00000000028c1cc LR: c00000000028c13c CTR: 0000000000000000
REGS: c00000000fff7b90 TRAP: 0300   Not tainted=20

(2.6.27-05329-g39076ba)=20
MSR: 9000000000009032 <EE,ME,IR,DR> CR: 22ffff24 XER: 20000000
DAR: 0000000000100108, DSISR: 000000000a000000
TASK =3D c00000017a061080[0] 'swapper' THREAD: c00000017a078000 CPU: 1
GPR00: 0000000000000000 c00000000fff7e10 c00000000059bfe0 0000000000000020 GPR04: 0000000000000001 0000000000000001 c00000000027fda8 0000000000000000 GPR08: 0000000000000000 0000000000200200 0000000000000001 0000000000100100 GPR12:
0000000022ffff22 c0000000005bc500 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 GPR20: 0000000000000000 000000000000000a 0000000000000001 0000000000000001 GPR24: c0000000005a2280 c0000000005f5134 00000001000387ff 000000000000010c GPR28:
c000000006e30c28 0000000000000020 c000000000543440 c00000017a2b0b40 NIP [c00000000028c1cc] .net_rx_action+0x1e4/0x26c =20
LR [c00000000028c13c] .net_rx_action+0x154/0x26c

Call Trace:
[c00000000fff7e10] [c00000000028c13c] .net_rx_action+0x154/0x26c
(unreliable) [c00000000fff7ec0] [c000000000056938]
.__do_softirq+0xf8/0x1f4 [c00000000fff7f90] [c000000000024334]
.call_do_softirq+0x14/0x24 [c00000017a07b970] [c00000000000bcdc]
.do_softirq+0xf0/0x104 [c00000017a07ba10] [c000000000056ae8]
.irq_exit+0x70/0x88 [c00000017a07ba90] [c00000000000ba18]
.do_IRQ+0x14c/0x244 [c00000017a07bb30] [c000000000004710]
hardware_interrupt_entry+0x18/0x1c --- Exception: 501 at
      .cpu_idle+0xf0/0x154 LR =3D .cpu_idle+0xd8/0x154
[c00000017a07be20] [c000000000012068] .cpu_idle+0x118/0x154
(unreliable) [c00000017a07bec0] [c0000000003d4304]
.start_secondary+0x310/0x3e8 [c00000017a07bf90] [c0000000000072b4]
.start_secondary_prolog+0x10/0x14 Instruction dump:

eb61ffd8 eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 7c0803a6 4e800020 e81f0010 7809ffe3 40820038 e93f0008 e97f0000 <f92b0008> f9690000 e95c0008 fb9f0000=20

maintainer
#2 lore raw

From: "Brandeburg, Jesse" <[email protected]>
Date: Thu, 23 Oct 2008 14:50:06 -0700

Quoted 18 lines
Quoted 17 lines · Chris Friesen

I tried booting a post 2.6.27 -git on a Motorola ATCA6101 (very similar to a Maple board). The first time I booted I got the first log below via the serial console. I rebooted and got as far as a login prompt. I was able to log in via the serial console, but then got an almost identical oops again, as shown in the second log below.

I configed out the gigE drivers for the backplane so the only remaining network link was the e100 link used for booting, but the problem remained.

Anyone have any idea what might be causing this?

Thanks,

Chris

Quoted 7 lines
Quoted 3 lines

Starting xinetd: [ OK ]
Starting cron: [ OK ]

Unable to handle kernel paging request for data at address 0x00100108

that 00100108 pattern looks familiar, I'm not much help here, but I think that had something to do with the list management of the poll_list in a netdev struct.

so now you just have to figure out why someone's netdev struct is becoming NULL. :-)

Usually this is an indication of returning the wrong value from the driver's ->poll() routine.

#3 lore raw
Quoted 27 lines · David Miller

From: "Brandeburg, Jesse" <[email protected]> Date: Thu, 23 Oct 2008 14:50:06 -0700

Quoted 24 lines
Quoted 12 lines · Chris Friesen

I tried booting a post 2.6.27 -git on a Motorola ATCA6101 (very similar to a Maple board). The first time I booted I got the first log below via the serial console. I rebooted and got as far as a login prompt. I was able to log in via the serial console, but then got an almost identical oops again, as shown in the second log below.

I configed out the gigE drivers for the backplane so the only remaining network link was the e100 link used for booting, but the problem remained.

Anyone have any idea what might be causing this?

Quoted 7 lines

Thanks,

Chris

Starting xinetd: [ OK ] Starting cron: [ OK ] Unable to handle kernel paging request for data at address 0x00100108

that 00100108 pattern looks familiar, I'm not much help here, but I think that had something to do with the list management of the poll_list in a netdev struct.

Quoted 5 lines
Quoted 2 lines

so now you just have to figure out why someone's netdev struct is becoming NULL. :-)

Usually this is an indication of returning the wrong value from the driver's ->poll() routine.

Looks like I was wrong before...the remaining ethernet link is an AMD-8111, not an e100. Sorry about that.

I backed out 6ba33ac "amd8111e: delete non NAPI code from the driver". With NAPI disabled, the blade appears stable. With NAPI enabled, the original problem recurred.

So...it would appear that the NAPI code is somehow buggy, and 6ba33ac should probably be reverted until the problem is found and fixed.

Chris

maintainer
#4 lore raw

From: "Chris Friesen" <[email protected]>
Date: Fri, 24 Oct 2008 17:39:00 -0600

Quoted 3 lines

So...it would appear that the NAPI code is somehow buggy, and 6ba33ac should probably be reverted until the problem is found and fixed.

No I think the problem is simple enough that someone should study the ->poll() routine quickly and audit it's return values.

#5 lore raw
Quoted 9 lines · David Miller

From: "Chris Friesen" <[email protected]>
Date: Fri, 24 Oct 2008 17:39:00 -0600

Quoted 3 lines

So...it would appear that the NAPI code is somehow buggy, and 6ba33ac should probably be reverted until the problem is found and fixed.

No I think the problem is simple enough that someone should study the ->poll() routine quickly and audit it's return values.

Assuming that amd8111e_rx_poll() is the proper routine, there is only one exit point, and it returns "num_rx_pkt". This variable is initialized to zero and increments for each packet sent up the stack.

Chris

maintainer
#6 lore raw

From: "Chris Friesen" <[email protected]>
Date: Fri, 24 Oct 2008 23:42:48 -0600

Quoted 7 lines
Quoted 6 lines · David Miller

From: "Chris Friesen" <[email protected]>
Date: Fri, 24 Oct 2008 17:39:00 -0600

Quoted 3 lines

So...it would appear that the NAPI code is somehow buggy, and 6ba33ac should probably be reverted until the problem is found and fixed.

Quoted 7 lines
Quoted 2 lines

No I think the problem is simple enough that someone should study the ->poll() routine quickly and audit it's return values.

Assuming that amd8111e_rx_poll() is the proper routine, there is only one exit point, and it returns "num_rx_pkt". This variable is initialized to zero and increments for each packet sent up the stack.

The problematic case in this routine seem to be when the driver processes exactly "budget" number of packets.

In this case it should not call __netif_rx_complete(), it should instead use the rx_not_empty label.

Probably the simplest fix is to get rid of the rx_not_empty label and protect the entire:

	/* Receive descriptor is empty now */
	spin_lock_irqsave(&lp->lock, flags);
	__netif_rx_complete(dev, napi);
	writel(VAL0|RINTEN0, mmio + INTEN0);
	writel(VAL2 | RDMD0, mmio + CMD0);
	spin_unlock_irqrestore(&lp->lock, flags);

code block with a test such as:

	if (rx_pkt_limit > 0)

(yes, greater than zero, not >= 0)

then replace the rx_not_empty goto with a simple break.

#7 lore raw
Quoted 10 lines · David Miller

Probably the simplest fix is to get rid of the rx_not_empty label and protect the entire:

	/* Receive descriptor is empty now */
	spin_lock_irqsave(&lp->lock, flags);
	__netif_rx_complete(dev, napi);
	writel(VAL0|RINTEN0, mmio + INTEN0);
	writel(VAL2 | RDMD0, mmio + CMD0);
	spin_unlock_irqrestore(&lp->lock, flags);
Quoted 7 lines

code block with a test such as:

	if (rx_pkt_limit > 0)

(yes, greater than zero, not >= 0)

then replace the rx_not_empty goto with a simple break.

Are you sure about that? Doing that, if we "--rx_pkt_limit < 0" we'll only break out of the inner while loop. We then check then interrupt status register and potentially loop through the do/while loop again (maybe decrementing rx_pkt_limit again) even though we've used up our budget.

If I leave the label and jump and just add the "rx_pkt_limit > 0" test, it seems to work.

Chris

From: Chris Friesen <[email protected]>
Subject: [PATCH] fix amd8111e rx return code

The amd8111e rx poll routine currently mishandles the case when we process exactly the number of packets specified in the budget.

This patch is basically as suggested by David Miller.

drivers/net/amd8111e.c +8 -6
index c54967f..ba1be0b 100644
@@ -833,12 +833,14 @@ static int amd8111e_rx_poll(struct napi_struct *napi, int budget)
 
 	} while(intr0 & RINT0);
 
-	/* Receive descriptor is empty now */
-	spin_lock_irqsave(&lp->lock, flags);
-	__netif_rx_complete(dev, napi);
-	writel(VAL0|RINTEN0, mmio + INTEN0);
-	writel(VAL2 | RDMD0, mmio + CMD0);
-	spin_unlock_irqrestore(&lp->lock, flags);
+	if (rx_pkt_limit > 0) {
+		/* Receive descriptor is empty now */
+		spin_lock_irqsave(&lp->lock, flags);
+		__netif_rx_complete(dev, napi);
+		writel(VAL0|RINTEN0, mmio + INTEN0);
+		writel(VAL2 | RDMD0, mmio + CMD0);
+		spin_unlock_irqrestore(&lp->lock, flags);
+	}
 
 rx_not_empty:
 	return num_rx_pkt;
maintainer
#8 lore raw

From: "Chris Friesen" <[email protected]>
Date: Mon, 27 Oct 2008 18:13:54 -0600

Quoted 8 lines

[PATCH] fix amd8111e rx return code

The amd8111e rx poll routine currently mishandles the case when we process exactly the number of packets specified in the budget.

This patch is basically as suggested by David Miller.

I had to apply this by hand because your email client heavily corrupted the patch.

Thanks.

#9 lore raw
Quoted 9 lines · David Miller

From: "Chris Friesen" <[email protected]>
Date: Mon, 27 Oct 2008 18:13:54 -0600

Quoted 5 lines

[PATCH] fix amd8111e rx return code

The amd8111e rx poll routine currently mishandles the case when we process exactly the number of packets specified in the budget.

Quoted 7 lines
Quoted 3 lines

This patch is basically as suggested by David Miller.

I had to apply this by hand because your email client heavily corrupted the patch.

Crap. Sorry about that, it looks like I forgot to set my line-wrap to 0 before creating the email. @#$#% Thunderbird.

Chris