Skip to contentSkip to conversation

2.6.8 (or 7?) regression: sleep on older tibooks broken

DISCUSSION David N. Welton 3 participants 14 messages Aug 15, 2004 – Aug 22, 2004

lore
Share & tools
lore
#0 lore raw

I was previously using a stock 2.6.6 kernel without problems on my tibook. I upgraded to 2.6.8.1 and compiled using:

gcc (GCC) 3.3.4 (Debian 1:3.3.4-6sarge1)

and "Version: 2.14.90.0.7-8" of binutils, in Debian.

I saw there is a similar problem here:

http://lists.linuxppc.org/linuxppc-dev/200407/msg00092.html

but it's not the same problem... I removed the ohci_hcd module from the kernel (it's present at boot), and sleep still doesn't happen. I don't even get the "breathing" light, and yet the computer still seems warm after some time, seemingly indicative that it's not really asleep or dead. I can only restart it via the Ctrl-Command-Power combination.

Logs don't say anything.

So - what got changed that could have caused this breakage?

Kernel configs and other info available on request.

@ashland [~] $ cat /proc/cpuinfo
processor : 0
cpu : 7410, altivec supported
temperature : 1-76 C (uncalibrated)
clock : 400MHz
revision : 17.2 (pvr 800c 1102)
bogomips : 796.67
machine : PowerBook3,2
motherboard : PowerBook3,2 MacRISC2 MacRISC Power Macintosh
detected as : 71 (PowerBook Titanium)
pmac flags : 0000000b
L2 cache : 1024K unified
memory : 256MB
pmac-generation : NewWorld

Thankyou,

Signature · 5 lines
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/
#1 lore raw
Quoted 6 lines · David N. Welton

but it's not the same problem... I removed the ohci_hcd module from the kernel (it's present at boot), and sleep still doesn't happen. I don't even get the "breathing" light, and yet the computer still seems warm after some time, seemingly indicative that it's not really asleep or dead. I can only restart it via the Ctrl-Command-Power combination.

Best thing at this point is to hack out the sleep code in the video driver to see where it dies during the sleep process...

Ben.

#2 lore raw
Quoted 11 lines · Benjamin Herrenschmidt
Quoted 6 lines · David N. Welton

but it's not the same problem... I removed the ohci_hcd module from the kernel (it's present at boot), and sleep still doesn't happen. I don't even get the "breathing" light, and yet the computer still seems warm after some time, seemingly indicative that it's not really asleep or dead. I can only restart it via the Ctrl-Command-Power combination.

Best thing at this point is to hack out the sleep code in the video driver to see where it dies during the sleep process...

I made the video driver's sleep routing return 0 immediately.

That was enough to at least get a couple of reports from xmon about a vector 200 corresponding to an address in powerbook_sleep_Core99... Still investigating, but this is new territory for me, and it's certainly at a tricky moment in the life of the kernel. Suggestions appreciated as to what might have changed and what to look for.

Thankyou,

Signature · 5 lines
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/
#3 lore raw
Quoted 8 lines

I made the video driver's sleep routing return 0 immediately.

That was enough to at least get a couple of reports from xmon about a vector 200 corresponding to an address in powerbook_sleep_Core99... Still investigating, but this is new territory for me, and it's certainly at a tricky moment in the life of the kernel. Suggestions appreciated as to what might have changed and what to look for.

Ouch, that's pretty bad. 200 is a machine check, looks like the HW shoked which shouldn't happen there. Can you get me the actual xmon output and eventually backtrace ? (you can disable the adb sleep code to get the kbd working in xmon)

Ben.

#4 lore raw
Quoted 11 lines · Benjamin Herrenschmidt
Quoted 8 lines

I made the video driver's sleep routing return 0 immediately.

That was enough to at least get a couple of reports from xmon about a vector 200 corresponding to an address in powerbook_sleep_Core99... Still investigating, but this is new territory for me, and it's certainly at a tricky moment in the life of the kernel. Suggestions appreciated as to what might have changed and what to look for.

Ouch, that's pretty bad. 200 is a machine check, looks like the HW shoked which shouldn't happen there.

Ok, I'll try that. Piddling around myself, I was tracing the crash to on or around this code (via-pmu.c:2550):

	/* Save & disable L2 and L3 caches*/
//	save_l3cr = _get_L3CR();	/* (returns -1 if not available) */
//	save_l2cr = _get_L2CR();	/* (returns -1 if not available) */
/* 	if (save_l3cr != 0xffffffff && (save_l3cr & L3CR_L3E) != 0)
		_set_L3CR(save_l3cr & 0x7fffffff);  */
/* 	if (save_l2cr != 0xffffffff && (save_l2cr & L2CR_L2E) != 0)
		_set_L2CR(save_l2cr & 0x7fffffff);  */

commenting it out lets things go to sleep, although they don't seem to wake up again.

Quoted 2 lines

Can you get me the actual xmon output and eventually backtrace ? (you can disable the adb sleep code to get the kbd working in xmon)

that would be adb_notify_sleep? ... yes, that seems to work ...

Managed to catch this:

Oops: machine check, sig: 7 [#1]
NIP: C025C5D8 LR: C025C768 SP: CD121E50 REGS: cd121da0 TRAP: 0200    Not tainted

MSR: 02043420 EE: 0 PR: 0 FP: 1 ME: 1 IR/DR: 10
TASK = cf4b0660[886] 'pmud' THREAD: cd120000Last syscall: 54
GPR00: FFFFFFFF CD121E50 CF4B0660 39200000 00400000 00000000 FFFFFF10 02003032
GPR08: 8014C0AC C025C768 00000004 00000000 88044428 1001E214 10010000 10000000
GPR16: 10000000 10000000 10000000 7FFFFD48 10000000 00000001 46FFFFFF 00000000
GPR24: FFFFFFFF 00000000 CD121E60 00000000 B9000000 20004200 FFFFFFE7 C0260000 NIP [c025c5d8] 0xc025c5d8
LR [c025c768] 0xc025c768

Call trace:
 [c025cf34] 0xc025cf34
 [c006e0e0] sys_ioctl+0xdc/0x2f4
 [c0005f40] ret_from_syscall+0x0/0x44

So it's just getting the ioctcl and going into sleep_Core99.

The instructions around C025C5D8 look like:

c025c584:       48 00 9c 8d     bl      c0266210 <pmu_request>
c025c588:       7f 43 d3 78     mr      r3,r26
c025c58c:       48 00 9e 51     bl      c02663dc <pmu_wait_complete>
c025c590:       4b db 0e e1     bl      c000d470 <_get_L3CR>
c025c594:       7c 7b 1b 78     mr      r27,r3
c025c598:       7c 78 18 f8     not     r24,r3
c025c59c:       57 77 0f fe     rlwinm  r23,r27,1,31,31
c025c5a0:       4b db 0d d1     bl      c000d370 <_get_L2CR>
c025c5a4:       31 38 ff ff     addic   r9,r24,-1
c025c5a8:       7c 09 c1 10     subfe   r0,r9,r24
c025c5ac:       7c 7c 1b 78     mr      r28,r3
c025c5b0:       7c 09 b8 39     and.    r9,r0,r23
c025c5b4:       40 82 01 b8     bne-    c025c76c <powerbook_sleep_Core99+0x298>
c025c5b8:       7f 96 e0 f8     not     r22,r28
c025c5bc:       57 95 0f fe     rlwinm  r21,r28,1,31,31
c025c5c0:       31 36 ff ff     addic   r9,r22,-1
c025c5c4:       7c 09 b1 10     subfe   r0,r9,r22
c025c5c8:       7c 09 a8 39     and.    r9,r0,r21
c025c5cc:       40 82 01 94     bne-    c025c760 <powerbook_sleep_Core99+0x28c>
c025c5d0:       3f e0 c0 26     lis     r31,-16346
c025c5d4:       80 1f 78 c4     lwz     r0,30916(r31)
c025c5d8:       2c 00 00 00     cmpwi   r0,0
c025c5dc:       41 82 01 54     beq-    c025c730 <powerbook_sleep_Core99+0x25c>
c025c5e0:       3f 20 c0 27     lis     r25,-16345
c025c5e4:       4b ff f8 91     bl      c025be74 <save_via_state>
c025c5e8:       39 39 93 54     addi    r9,r25,-27820
c025c5ec:       80 09 00 94     lwz     r0,148(r9)
c025c5f0:       2c 00 00 00     cmpwi   r0,0
c025c5f4:       40 82 01 1c     bne-    c025c710 <powerbook_sleep_Core99+0x23c>
c025c5f8:       80 1f 78 c4     lwz     r0,30916(r31)
c025c5fc:       2c 00 00 00     cmpwi   r0,0
c025c600:       41 82 01 08     beq-    c025c708 <powerbook_sleep_Core99+0x234>
c025c604:       3f c0 01 21     lis     r30,289
c025c608:       3b e0 13 87     li      r31,4999
c025c60c:       3f a0 c0 23     lis     r29,-16349
c025c610:       63 de ea c0     ori     r30,r30,60096

I don't have too many ideas. If it's useful, I suppose I can try backing of to 2.6.7 to see if it suffers from the same problem...

Thankyou,

Signature · 5 lines
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/
#5 lore raw
Quoted 2 lines · [email protected] (David N. Welton) writes:

If it's useful, I suppose I can try backing of to 2.6.7 to see if it suffers from the same problem...

2.6.7 is also broken. Fresh 2.6.7 compiled, and it doesn't work either.

Kernel config at dedasys.com/kernelconfig

Thanks,

Signature · 5 lines
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/
#13 lore raw
In reply to David N. Welton
Quoted 11 lines

Ok, I'll try that. Piddling around myself, I was tracing the crash to on or around this code (via-pmu.c:2550):

	/* Save & disable L2 and L3 caches*/
//	save_l3cr = _get_L3CR();	/* (returns -1 if not available) */
//	save_l2cr = _get_L2CR();	/* (returns -1 if not available) */
/* 	if (save_l3cr != 0xffffffff && (save_l3cr & L3CR_L3E) != 0)
		_set_L3CR(save_l3cr & 0x7fffffff);  */
/* 	if (save_l2cr != 0xffffffff && (save_l2cr & L2CR_L2E) != 0)
		_set_L2CR(save_l2cr & 0x7fffffff);  */
Quoted 2 lines

commenting it out lets things go to sleep, although they don't seem to wake up again.

Yah, you aren't disabling the cache ... I fail to see how flushing & disabling the L2 would cause a machine check tho, it's really weird.

Yes, try 2.6.7 and let me know.

Ben.

Quoted 76 lines
Quoted 2 lines

Can you get me the actual xmon output and eventually backtrace ? (you can disable the adb sleep code to get the kbd working in xmon)

that would be adb_notify_sleep? ... yes, that seems to work ...

Managed to catch this:

Oops: machine check, sig: 7 [#1]
NIP: C025C5D8 LR: C025C768 SP: CD121E50 REGS: cd121da0 TRAP: 0200    Not tainted

MSR: 02043420 EE: 0 PR: 0 FP: 1 ME: 1 IR/DR: 10
TASK = cf4b0660[886] 'pmud' THREAD: cd120000Last syscall: 54
GPR00: FFFFFFFF CD121E50 CF4B0660 39200000 00400000 00000000 FFFFFF10 02003032
GPR08: 8014C0AC C025C768 00000004 00000000 88044428 1001E214 10010000 10000000
GPR16: 10000000 10000000 10000000 7FFFFD48 10000000 00000001 46FFFFFF 00000000
GPR24: FFFFFFFF 00000000 CD121E60 00000000 B9000000 20004200 FFFFFFE7 C0260000 NIP [c025c5d8] 0xc025c5d8
LR [c025c768] 0xc025c768

Call trace:
 [c025cf34] 0xc025cf34
 [c006e0e0] sys_ioctl+0xdc/0x2f4
 [c0005f40] ret_from_syscall+0x0/0x44

So it's just getting the ioctcl and going into sleep_Core99.

The instructions around C025C5D8 look like:

c025c584:       48 00 9c 8d     bl      c0266210 <pmu_request>
c025c588:       7f 43 d3 78     mr      r3,r26
c025c58c:       48 00 9e 51     bl      c02663dc <pmu_wait_complete>
c025c590:       4b db 0e e1     bl      c000d470 <_get_L3CR>
c025c594:       7c 7b 1b 78     mr      r27,r3
c025c598:       7c 78 18 f8     not     r24,r3
c025c59c:       57 77 0f fe     rlwinm  r23,r27,1,31,31
c025c5a0:       4b db 0d d1     bl      c000d370 <_get_L2CR>
c025c5a4:       31 38 ff ff     addic   r9,r24,-1
c025c5a8:       7c 09 c1 10     subfe   r0,r9,r24
c025c5ac:       7c 7c 1b 78     mr      r28,r3
c025c5b0:       7c 09 b8 39     and.    r9,r0,r23
c025c5b4:       40 82 01 b8     bne-    c025c76c <powerbook_sleep_Core99+0x298>
c025c5b8:       7f 96 e0 f8     not     r22,r28
c025c5bc:       57 95 0f fe     rlwinm  r21,r28,1,31,31
c025c5c0:       31 36 ff ff     addic   r9,r22,-1
c025c5c4:       7c 09 b1 10     subfe   r0,r9,r22
c025c5c8:       7c 09 a8 39     and.    r9,r0,r21
c025c5cc:       40 82 01 94     bne-    c025c760 <powerbook_sleep_Core99+0x28c>
c025c5d0:       3f e0 c0 26     lis     r31,-16346
c025c5d4:       80 1f 78 c4     lwz     r0,30916(r31)
c025c5d8:       2c 00 00 00     cmpwi   r0,0
c025c5dc:       41 82 01 54     beq-    c025c730 <powerbook_sleep_Core99+0x25c>
c025c5e0:       3f 20 c0 27     lis     r25,-16345
c025c5e4:       4b ff f8 91     bl      c025be74 <save_via_state>
c025c5e8:       39 39 93 54     addi    r9,r25,-27820
c025c5ec:       80 09 00 94     lwz     r0,148(r9)
c025c5f0:       2c 00 00 00     cmpwi   r0,0
c025c5f4:       40 82 01 1c     bne-    c025c710 <powerbook_sleep_Core99+0x23c>
c025c5f8:       80 1f 78 c4     lwz     r0,30916(r31)
c025c5fc:       2c 00 00 00     cmpwi   r0,0
c025c600:       41 82 01 08     beq-    c025c708 <powerbook_sleep_Core99+0x234>
c025c604:       3f c0 01 21     lis     r30,289
c025c608:       3b e0 13 87     li      r31,4999
c025c60c:       3f a0 c0 23     lis     r29,-16349
c025c610:       63 de ea c0     ori     r30,r30,60096

I don't have too many ideas. If it's useful, I suppose I can try backing of to 2.6.7 to see if it suffers from the same problem...

Thankyou,
--
David N. Welton
Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
Apache Tcl: http://tcl.apache.org/
Photos: http://www.dedasys.com/photos/

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Signature · 1 line
Benjamin Herrenschmidt <[email protected]>
#6 lore raw
In reply to David N. Welton
Quoted 7 lines · David N. Welton
Quoted 2 lines · [email protected] (David N. Welton) writes:

If it's useful, I suppose I can try backing of to 2.6.7 to see if it suffers from the same problem...

2.6.7 is also broken. Fresh 2.6.7 compiled, and it doesn't work either.

Can you try going backward in time find when it broke ?

Ben.

#7 lore raw
Quoted 10 lines · Benjamin Herrenschmidt
Quoted 7 lines · David N. Welton
Quoted 2 lines · [email protected] (David N. Welton) writes:

If it's useful, I suppose I can try backing of to 2.6.7 to see if it suffers from the same problem...

2.6.7 is also broken. Fresh 2.6.7 compiled, and it doesn't work either.

Can you try going backward in time find when it broke ?

2.6.6 sleeps/wakes just fine. I had a halfhearted look through the 7 2.6.patch, searching on things like 'ppc' and 'macintosh', but didn't 2.6.see much that jumped out at me.

Thanks,

Signature · 5 lines
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/
#8 lore raw
Quoted 4 lines

2.6.6 sleeps/wakes just fine. I had a halfhearted look through the 7 2.6.patch, searching on things like 'ppc' and 'macintosh', but didn't 2.6.see much that jumped out at me.

Best would be if you could try the various 2.6.7-rc patches ...

Ben.

#9 lore raw
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 3 lines

2.6.6 sleeps/wakes just fine. I had a halfhearted look through the 7 2.6.patch, searching on things like 'ppc' and 'macintosh', but didn't 2.6.see much that jumped out at me.

Best would be if you could try the various 2.6.7-rc patches ...

.... patching, compiling ....

rc1 sleeps/wakes, rc2 does not.

Signature · 5 lines
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/
#10 lore raw
Quoted 11 lines · [email protected] (David N. Welton) writes:
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 3 lines

2.6.6 sleeps/wakes just fine. I had a halfhearted look through the 7 2.6.patch, searching on things like 'ppc' and 'macintosh', but didn't 2.6.see much that jumped out at me.

Best would be if you could try the various 2.6.7-rc patches ...

.... patching, compiling ....

rc1 sleeps/wakes, rc2 does not.

The only thing that jumps out at me is that arch/ppc/mm/cachemap.c 'moved' to arch/ppc/kernel/dma-mapping.c and seems to have changed some as well. Other than that, I would need more suggestions for debugging.

Thankyou,

Signature · 5 lines
David N. Welton
     Personal: http://www.dedasys.com/davidw/
Free Software: http://www.dedasys.com/freesoftware/
   Apache Tcl: http://tcl.apache.org/
       Photos: http://www.dedasys.com/photos/
#11 lore raw
Quoted 3 lines · David N. Welton

The only thing that jumps out at me is that arch/ppc/mm/cachemap.c 'moved' to arch/ppc/kernel/dma-mapping.c and seems to have changed some as well.

dma-mapping.c is used only on non-coherent cache CPUs (like 4xx, 8xx). It's not relevant and is not even compiled for powermacs.

Eugene

#12 lore raw
In reply to David N. Welton
Quoted 5 lines

The only thing that jumps out at me is that arch/ppc/mm/cachemap.c 'moved' to arch/ppc/kernel/dma-mapping.c and seems to have changed some as well. Other than that, I would need more suggestions for debugging.

No, that's probably irrelevant. I'll have a look when I'm back, I've been away for the week-end.

Ben.