Skip to contentSkip to conversation

panic during cpu_up

x86-tip Dhaval Giani 2 participants 4 messages Jul 2, 2008 – Jul 3, 2008 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive

[Missed cc'ing the LKML last time around]

Quoted 259 lines · Dhaval Giani

Hi Ingo, Thomas,

I am hitting this on -tip. With 200a86b5d435a217c3d77f3b53cd32cb78c1fde8 as the top level commit. Wondering if it is known?

I am trying to fix it atm.

Thanks,

Red Hat Enterprise Linux AS release 4 (Nahant Update 2) Kernel 2.6.26-rc8-tip on an i686

llm11.in.ibm.com login: root
Password:
Last login: Thu Jul 3 00:30:47 on ttyS0 You have new mail.
cd[root@llm11 ~]# cd /sys/devices/system/cpu/cpu1/ [root@llm11 cpu1]# echo 0 > online Breaking affinity for irq 45
[root@llm11 cpu1]# echo 1 > online
lockdep: fixing up alternatives.

BUG: unable to handle kernel <1>BUG: unable to handle kernel NULL
pointer dereference at 00000000
IP: [<00000000>]
*pde = 00000000 
Oops: 0000 [#1] SMP 
Modules linked in:

Pid: 0, comm: swapper Not tainted (2.6.26-rc8-tip #1)
EIP: 0060:[<00000000>] EFLAGS: 00010002 CPU: 2 EIP is at 0x0
EAX: c0614c00 EBX: 00000002 ECX: 0799d000 EDX: ffff3bdf

BUG: unable to handle kernel NULL pointer dereference at 00000000
IP:ESI: 00000000 EDI: 00000000 EBP: f7cadfac ESP: f7cadfa4
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
 [<00000000>]
BUG: unable to handle kernel NULL pointer dereference*pde = 00000000

<1>BUG: unable to handle kernel at 00000000 IP:<0>Process swapper (pid: 0, ti=f7cac000 task=f7caaee0 task.ti=f7cac000)NULL pointer dereference [<00000000>] *pde = 00000000 <1>BUG: unable to handle kernel

Stack: 
 at 00000000

IP:NULL pointer dereference [<00000000>]
at 00000000
c01020c7 *pde = 00000000 <1>IP:

[<00000000>]
0402080c *pde = 00000000 f7cadfb4
c040b6cf 00000000 00000000 00000000 00000000

       00000000 00000000 00000000 00000000 00000000 00000000 000000d8
00000000 
       00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7cadfa4
Kernel panic - not syncing: Fatal exception
Oops: 0000 [#2] SMP 
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
Modules linked in: [<c01276d7>] 

panic+0x38/0xe0

Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 3
[<c0104bcd>] EIP is at 0x0
EAX: c0614c00 EBX: 00000003 ECX: 079a5000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7cbbfac ESP: f7cbbfa4
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7cba000 task=f7cb8fe0 task.ti=f7cba000)

Stack: die+0x130/0x147
c01020c7 0602080c f7cbbfb4 c040b6cf 00000000  [<c0411795>] 00000000
00000000 00000000 
       00000000 do_page_fault+0x3bd/0x482
00000000  [<c04113d8>] ? 00000000 00000000 00000000 00000000
do_page_fault+0x0/0x482
000000d8 00000000  [<c040fbda>] 
       00000000 error_code+0x72/0x78
00000000 00000000  [<c01020c7>] ? 00000000 00000000 cpu_idle+0x8a/0x9e
00000000 00000000 
Call Trace:
 [<c040b6cf>] <0> [<c01020c7>] start_secondary+0xbb/0xbd
?  =======================
cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7cbbfa4
Oops: 0000 [#3] <0>Kernel panic - not syncing: Fatal exception

SMP Pid: 0, comm: swapper Tainted: G D 2.6.26-rc8-tip #1

[<c01276d7>] Modules linked in:

panic+0x38/0xe0
Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
[<c0104bcd>] EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 7
die+0x130/0x147
EIP is at 0x0
[<c0411795>] EAX: c0614c00 EBX: 00000007 ECX: 079c5000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7d1bfac ESP: f7d1bfa4
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7d1a000 task=f7d18be0 task.ti=f7d1a000)do_page_fault+0x3bd/0x482

Stack:  [<c04113d8>] c01020c7 ? 0702080c f7d1bfb4
do_page_fault+0x0/0x482
c040b6cf  [<c040fbda>] 00000000 00000000 error_code+0x72/0x78
00000000 00000000 
        [<c01020c7>] ? 00000000 00000000 cpu_idle+0x8a/0x9e

00000000 00000000 [<c040b6cf>] 00000000 00000000 000000d8 start_secondary+0xbb/0xbd
00000000 =======================

       00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7d1bfa4
Kernel panic - not syncing: Fatal exception
Oops: 0000 [#4] SMP 
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
Modules linked in: [<c01276d7>] 
panic+0x38/0xe0

[<c0104bcd>] Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip
#1)
EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 4 die+0x130/0x147
EIP is at 0x0
EAX: c0614c00 EBX: 00000004 ECX: 079ad000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7ccbfac ESP: f7ccbfa4
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7cca000 task=f7cc90e0 task.ti=f7cca000) [<c0411795>]

Stack: c01020c7 do_page_fault+0x3bd/0x482
0102080c  [<c04113d8>] f7ccbfb4 c040b6cf 00000000 00000000 00000000
00000000 ? 
       00000000 00000000 00000000 00000000 00000000
do_page_fault+0x0/0x482
00000000 000000d8 00000000 
       00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c040fbda>] <0> [<c01020c7>] ? error_code+0x72/0x78
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
cpu_idle+0x8a/0x9e
 [<c040b6cf>] ?  [<c040b6cf>] start_secondary+0xbb/0xbd
 =======================
start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7ccbfa4
Oops: 0000 [#5] <0>Kernel panic - not syncing: Fatal exception
SMP 
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
Modules linked in: [<c01276d7>] 

panic+0x38/0xe0

Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
[<c0104bcd>] EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 6
die+0x130/0x147
EIP is at 0x0
EAX: c0614c00 EBX: 00000006 ECX: 079bd000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7d0bfac ESP: f7d0bfa4
[<c0411795>] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7d0a000 task=f7d092e0 task.ti=f7d0a000)do_page_fault+0x3bd/0x482

Stack:  [<c04113d8>] ? c01020c7 0502080c do_page_fault+0x0/0x482
f7d0bfb4  [<c040fbda>] c040b6cf error_code+0x72/0x78
00000000  [<c01020c7>] ? 00000000 cpu_idle+0x8a/0x9e
00000000  [<c040b6cf>] 00000000 start_secondary+0xbb/0xbd
        =======================
00000000 00000000 00000000 00000000 00000000 00000000 000000d8 00000000 
       00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7d0bfa4
Kernel panic - not syncing: Fatal exception
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
 [<c01276d7>] panic+0x38/0xe0
 [<c0104bcd>] die+0x130/0x147
 [<c0411795>] do_page_fault+0x3bd/0x482
 [<c04113d8>] ? do_page_fault+0x0/0x482
 [<c040fbda>] error_code+0x72/0x78
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] start_secondary+0xbb/0xbd
 =======================
NULL pointer dereference at 00000000
IP: [<00000000>]
*pde = 00000000 
Oops: 0000 [#6] SMP 
Modules linked in:

Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
EIP: 0060:[<00000000>] EFLAGS: 00010006 CPU: 1
EIP is at 0x0
EAX: c0614c00 EBX: 00000001 ECX: 07995000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7c7ffb4 ESP: f7c7ffac
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7c7e000 task=f7c7cde0 task.ti=f7c7e000)

Stack: c01020c7 0202080c f7c7ffbc c040b6cf 00000000 00000000 00000000
00000000 
       00000000 00000000 00000000 00000000 000000d8 00000000 00000000
00000000 
       00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7c7ffac
Kernel panic - not syncing: Fatal exception
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
 [<c01276d7>] panic+0x38/0xe0
 [<c0104bcd>] die+0x130/0x147
 [<c0411795>] do_page_fault+0x3bd/0x482
 [<c04113d8>] ? do_page_fault+0x0/0x482
 [<c040fbda>] error_code+0x72/0x78
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] start_secondary+0xbb/0xbd
 =======================
BUG: NMI Watchdog detected LOCKUP on CPU6, ip c0111959, registers:
Modules linked in:

Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
EIP: 0060:[<c0111959>] EFLAGS: 00000093 CPU: 6
EIP is at __smp_call_function+0x5d/0x7a
EAX: 0000009e EBX: 00000005 ECX: 00000006 EDX: f7d092e0
ESI: c01047d4 EDI: c0111a5a EBP: f7d0bf00 ESP: f7d0bed0
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7d0a000 task=f7d092e0 task.ti=f7d0a000)

Stack: 00000000 c0111a5a 00000000 00000000 c01047d4 00000000 c029915b
f7d0bf6c 
       00000006 00000001 00000046 00000000 f7d0bf14 c0111acb 00000000
f7d0bf6c 
       00000006 f7d0bf20 c01276ee f7d0bf6c f7d0bf3c c0104bcd c04d922f
c04e06cf 
Call Trace:
 [<c0111a5a>] ? stop_this_cpu+0x0/0x3a
 [<c01047d4>] ? show_trace+0x10/0x12
 [<c029915b>] ? do_unblank_screen+0x2a/0xf9
 [<c0111acb>] ? native_smp_send_stop+0x37/0x6a
 [<c01276ee>] ? panic+0x4f/0xe0
 [<c0104bcd>] ? die+0x130/0x147
 [<c0411795>] ? do_page_fault+0x3bd/0x482
 [<c04113d8>] ? do_page_fault+0x0/0x482
 [<c040fbda>] ? error_code+0x72/0x78
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code: 85 c0 0f 44 75 e0 89 45 e4 8d 45 d4 a3 d4 33 62 c0 89 75 e0 0f ae
Quoted 3 lines

f0 0f 1f 00 8b 15 e0 69 57 c0 b8 fb 00 00 00 ff 52 78 39 5d dc <74> 04 f3 90 eb f7 83 7d 08 00 74 09 39 5d e0 74 04 f3 90 eb f7

So after digging around a bit, it turns out the pm_idle is NULL. For some reason it is not getting set to default_idle if nothing works. I am not sure of the path being followed, and its a bit late for me to be trying anything serious :).

This seems to work as a temporary workaround, but obviously is not the right fix yet.

---
 arch/x86/kernel/process_32.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletion(-)
arch/x86/kernel/process_32.c +4 -1
Index: linux-2.6.26-rc8-tip/arch/x86/kernel/process_32.c
===================================================================
@@ -144,7 +144,10 @@ void cpu_idle(void)
 			__get_cpu_var(irq_stat).idle_timestamp = jiffies;
 			/* Don't trace irqs off for idle */
 			stop_critical_timings();
-			pm_idle();
+			if (pm_idle)
+				pm_idle();
+			else
+				default_idle();
 			start_critical_timings();
 		}
 		tick_nohz_restart_sched_tick();
Signature · 2 lines
regards,
Dhaval
#2 lore raw
Quoted 266 lines · Dhaval Giani

[Missed cc'ing the LKML last time around]

Quoted 259 lines · Dhaval Giani

Hi Ingo, Thomas,

I am hitting this on -tip. With 200a86b5d435a217c3d77f3b53cd32cb78c1fde8 as the top level commit. Wondering if it is known?

I am trying to fix it atm.

Thanks,

Red Hat Enterprise Linux AS release 4 (Nahant Update 2) Kernel 2.6.26-rc8-tip on an i686

llm11.in.ibm.com login: root
Password:
Last login: Thu Jul 3 00:30:47 on ttyS0 You have new mail.
cd[root@llm11 ~]# cd /sys/devices/system/cpu/cpu1/ [root@llm11 cpu1]# echo 0 > online Breaking affinity for irq 45
[root@llm11 cpu1]# echo 1 > online
lockdep: fixing up alternatives.

BUG: unable to handle kernel <1>BUG: unable to handle kernel NULL
pointer dereference at 00000000
IP: [<00000000>]
*pde = 00000000 
Oops: 0000 [#1] SMP 
Modules linked in:

Pid: 0, comm: swapper Not tainted (2.6.26-rc8-tip #1)
EIP: 0060:[<00000000>] EFLAGS: 00010002 CPU: 2 EIP is at 0x0
EAX: c0614c00 EBX: 00000002 ECX: 0799d000 EDX: ffff3bdf

BUG: unable to handle kernel NULL pointer dereference at 00000000
IP:ESI: 00000000 EDI: 00000000 EBP: f7cadfac ESP: f7cadfa4
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
 [<00000000>]
BUG: unable to handle kernel NULL pointer dereference*pde = 00000000

<1>BUG: unable to handle kernel at 00000000 IP:<0>Process swapper (pid: 0, ti=f7cac000 task=f7caaee0 task.ti=f7cac000)NULL pointer dereference [<00000000>] *pde = 00000000 <1>BUG: unable to handle kernel

Stack: 
 at 00000000

IP:NULL pointer dereference [<00000000>]
at 00000000
c01020c7 *pde = 00000000 <1>IP:

[<00000000>]
0402080c *pde = 00000000 f7cadfb4
c040b6cf 00000000 00000000 00000000 00000000

       00000000 00000000 00000000 00000000 00000000 00000000 000000d8
00000000 
       00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7cadfa4
Kernel panic - not syncing: Fatal exception
Oops: 0000 [#2] SMP 
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
Modules linked in: [<c01276d7>] 

panic+0x38/0xe0

Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 3
[<c0104bcd>] EIP is at 0x0
EAX: c0614c00 EBX: 00000003 ECX: 079a5000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7cbbfac ESP: f7cbbfa4
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7cba000 task=f7cb8fe0 task.ti=f7cba000)

Stack: die+0x130/0x147
c01020c7 0602080c f7cbbfb4 c040b6cf 00000000  [<c0411795>] 00000000
00000000 00000000 
       00000000 do_page_fault+0x3bd/0x482
00000000  [<c04113d8>] ? 00000000 00000000 00000000 00000000
do_page_fault+0x0/0x482
000000d8 00000000  [<c040fbda>] 
       00000000 error_code+0x72/0x78
00000000 00000000  [<c01020c7>] ? 00000000 00000000 cpu_idle+0x8a/0x9e
00000000 00000000 
Call Trace:
 [<c040b6cf>] <0> [<c01020c7>] start_secondary+0xbb/0xbd
?  =======================
cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7cbbfa4
Oops: 0000 [#3] <0>Kernel panic - not syncing: Fatal exception

SMP Pid: 0, comm: swapper Tainted: G D 2.6.26-rc8-tip #1

[<c01276d7>] Modules linked in:

panic+0x38/0xe0
Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
[<c0104bcd>] EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 7
die+0x130/0x147
EIP is at 0x0
[<c0411795>] EAX: c0614c00 EBX: 00000007 ECX: 079c5000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7d1bfac ESP: f7d1bfa4
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7d1a000 task=f7d18be0 task.ti=f7d1a000)do_page_fault+0x3bd/0x482

Stack:  [<c04113d8>] c01020c7 ? 0702080c f7d1bfb4
do_page_fault+0x0/0x482
c040b6cf  [<c040fbda>] 00000000 00000000 error_code+0x72/0x78
00000000 00000000 
        [<c01020c7>] ? 00000000 00000000 cpu_idle+0x8a/0x9e

00000000 00000000 [<c040b6cf>] 00000000 00000000 000000d8 start_secondary+0xbb/0xbd
00000000 =======================

       00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7d1bfa4
Kernel panic - not syncing: Fatal exception
Oops: 0000 [#4] SMP 
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
Modules linked in: [<c01276d7>] 
panic+0x38/0xe0

[<c0104bcd>] Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip
#1)
EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 4 die+0x130/0x147
EIP is at 0x0
EAX: c0614c00 EBX: 00000004 ECX: 079ad000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7ccbfac ESP: f7ccbfa4
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7cca000 task=f7cc90e0 task.ti=f7cca000) [<c0411795>]

Stack: c01020c7 do_page_fault+0x3bd/0x482
0102080c  [<c04113d8>] f7ccbfb4 c040b6cf 00000000 00000000 00000000
00000000 ? 
       00000000 00000000 00000000 00000000 00000000
do_page_fault+0x0/0x482
00000000 000000d8 00000000 
       00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c040fbda>] <0> [<c01020c7>] ? error_code+0x72/0x78
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
cpu_idle+0x8a/0x9e
 [<c040b6cf>] ?  [<c040b6cf>] start_secondary+0xbb/0xbd
 =======================
start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7ccbfa4
Oops: 0000 [#5] <0>Kernel panic - not syncing: Fatal exception
SMP 
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
Modules linked in: [<c01276d7>] 

panic+0x38/0xe0

Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
[<c0104bcd>] EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 6
die+0x130/0x147
EIP is at 0x0
EAX: c0614c00 EBX: 00000006 ECX: 079bd000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7d0bfac ESP: f7d0bfa4
[<c0411795>] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7d0a000 task=f7d092e0 task.ti=f7d0a000)do_page_fault+0x3bd/0x482

Stack:  [<c04113d8>] ? c01020c7 0502080c do_page_fault+0x0/0x482
f7d0bfb4  [<c040fbda>] c040b6cf error_code+0x72/0x78
00000000  [<c01020c7>] ? 00000000 cpu_idle+0x8a/0x9e
00000000  [<c040b6cf>] 00000000 start_secondary+0xbb/0xbd
        =======================
00000000 00000000 00000000 00000000 00000000 00000000 000000d8 00000000 
       00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7d0bfa4
Kernel panic - not syncing: Fatal exception
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
 [<c01276d7>] panic+0x38/0xe0
 [<c0104bcd>] die+0x130/0x147
 [<c0411795>] do_page_fault+0x3bd/0x482
 [<c04113d8>] ? do_page_fault+0x0/0x482
 [<c040fbda>] error_code+0x72/0x78
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] start_secondary+0xbb/0xbd
 =======================
NULL pointer dereference at 00000000
IP: [<00000000>]
*pde = 00000000 
Oops: 0000 [#6] SMP 
Modules linked in:

Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
EIP: 0060:[<00000000>] EFLAGS: 00010006 CPU: 1
EIP is at 0x0
EAX: c0614c00 EBX: 00000001 ECX: 07995000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7c7ffb4 ESP: f7c7ffac
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7c7e000 task=f7c7cde0 task.ti=f7c7e000)

Stack: c01020c7 0202080c f7c7ffbc c040b6cf 00000000 00000000 00000000
00000000 
       00000000 00000000 00000000 00000000 000000d8 00000000 00000000
00000000 
       00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7c7ffac
Kernel panic - not syncing: Fatal exception
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
 [<c01276d7>] panic+0x38/0xe0
 [<c0104bcd>] die+0x130/0x147
 [<c0411795>] do_page_fault+0x3bd/0x482
 [<c04113d8>] ? do_page_fault+0x0/0x482
 [<c040fbda>] error_code+0x72/0x78
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] start_secondary+0xbb/0xbd
 =======================
BUG: NMI Watchdog detected LOCKUP on CPU6, ip c0111959, registers:
Modules linked in:

Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
EIP: 0060:[<c0111959>] EFLAGS: 00000093 CPU: 6
EIP is at __smp_call_function+0x5d/0x7a
EAX: 0000009e EBX: 00000005 ECX: 00000006 EDX: f7d092e0
ESI: c01047d4 EDI: c0111a5a EBP: f7d0bf00 ESP: f7d0bed0
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7d0a000 task=f7d092e0 task.ti=f7d0a000)

Stack: 00000000 c0111a5a 00000000 00000000 c01047d4 00000000 c029915b
f7d0bf6c 
       00000006 00000001 00000046 00000000 f7d0bf14 c0111acb 00000000
f7d0bf6c 
       00000006 f7d0bf20 c01276ee f7d0bf6c f7d0bf3c c0104bcd c04d922f
c04e06cf 
Call Trace:
 [<c0111a5a>] ? stop_this_cpu+0x0/0x3a
 [<c01047d4>] ? show_trace+0x10/0x12
 [<c029915b>] ? do_unblank_screen+0x2a/0xf9
 [<c0111acb>] ? native_smp_send_stop+0x37/0x6a
 [<c01276ee>] ? panic+0x4f/0xe0
 [<c0104bcd>] ? die+0x130/0x147
 [<c0411795>] ? do_page_fault+0x3bd/0x482
 [<c04113d8>] ? do_page_fault+0x0/0x482
 [<c040fbda>] ? error_code+0x72/0x78
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code: 85 c0 0f 44 75 e0 89 45 e4 8d 45 d4 a3 d4 33 62 c0 89 75 e0 0f ae
Quoted 3 lines

f0 0f 1f 00 8b 15 e0 69 57 c0 b8 fb 00 00 00 ff 52 78 39 5d dc <74> 04 f3 90 eb f7 83 7d 08 00 74 09 39 5d e0 74 04 f3 90 eb f7

Quoted 7 lines

So after digging around a bit, it turns out the pm_idle is NULL. For some reason it is not getting set to default_idle if nothing works. I am not sure of the path being followed, and its a bit late for me to be trying anything serious :).

This seems to work as a temporary workaround, but obviously is not the right fix yet.

hm, interesting - this is a new bug. I'm wondering where that NULL came from. x86 boot itself does not leave room for pm_idle to be NULL AFAICS.

One suspect would be cpuidle:

void cpuidle_uninstall_idle_handler(void)

 {
         if (enabled_devices && (pm_idle != pm_idle_old)) {
                 pm_idle = pm_idle_old;
                 cpuidle_kick_cpus();

The other suspect would be acpi idle:

        /* Fall back to the default idle loop */
        pm_idle = pm_idle_save;

Could you try latest tip/master or the debug patch below? It should show where the NULL comes from, without crashing.

A third possibility is if pm_idle itself got genuinely corrupted via something else. In that case you should get a warning from process*.c.

Ingo

-------------------->
commit 3d02d03d4974e0776baf2bc03cbb71d4d670aa85

Author: Ingo Molnar <[email protected]>
Date:   Thu Jul 3 08:52:57 2008 +0200
    debug: "[x86-tip] panic during cpu_up"
    
    Dhaval Giani wrote:
    
    > I am hitting this on -tip. With 200a86b5d435a217c3d77f3b53cd32cb78c1fde8
    > as the top level commit. Wondering if it is known?
    >
    > I am trying to fix it atm.
    >
    > Thanks,
    >
    > Red Hat Enterprise Linux AS release 4 (Nahant Update 2)
    > Kernel 2.6.26-rc8-tip on an i686
    >
    > llm11.in.ibm.com login: root
    > Password:
    > Last login: Thu Jul  3 00:30:47 on ttyS0
    > You have new mail.
    > cd[root@llm11 ~]# cd /sys/devices/system/cpu/cpu1/
    > [root@llm11 cpu1]# echo 0 > online
    > Breaking affinity for irq 45
    > [root@llm11 cpu1]# echo 1 > online
    > lockdep: fixing up alternatives.
    > BUG: unable to handle kernel <1>BUG: unable to handle kernel NULL
    > pointer dereference at 00000000
    > IP: [<00000000>]
    > *pde = 00000000
    > Oops: 0000 [#1] SMP
    > Modules linked in:
    >
    > Pid: 0, comm: swapper Not tainted (2.6.26-rc8-tip #1)
    > EIP: 0060:[<00000000>] EFLAGS: 00010002 CPU: 2
    > EIP is at 0x0
    > EAX: c0614c00 EBX: 00000002 ECX: 0799d000 EDX: ffff3bdf
    > BUG: unable to handle kernel NULL pointer dereference at 00000000
    > IP:ESI: 00000000 EDI: 00000000 EBP: f7cadfac ESP: f7cadfa4
    >  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
    >  [<00000000>]
    > BUG: unable to handle kernel NULL pointer dereference*pde = 00000000
    > <1>BUG: unable to handle kernel  at 00000000
    > IP:<0>Process swapper (pid: 0, ti=f7cac000 task=f7caaee0
    > task.ti=f7cac000)NULL pointer dereference [<00000000>]
    > *pde = 00000000 <1>BUG: unable to handle kernel
    > Stack:
    >  at 00000000
    > IP:NULL pointer dereference [<00000000>]
    >  at 00000000
    > c01020c7 *pde = 00000000 <1>IP:
    >
    >  [<00000000>]
    > 0402080c *pde = 00000000 f7cadfb4
    > c040b6cf 00000000 00000000 00000000 00000000
    >        00000000 00000000 00000000 00000000 00000000 00000000 000000d8
    > 00000000
    >        00000000 00000000 00000000 00000000 00000000 00000000 00000000
    > Call Trace:
    >  [<c01020c7>] ? cpu_idle+0x8a/0x9e
    >  [<c040b6cf>] ? start_secondary+0xbb/0xbd
    >  =======================
    > Code:  Bad EIP value.
    > EIP: [<00000000>] 0x0 SS:ESP 0068:f7cadfa4
    > Kernel panic - not syncing: Fatal exception
    > Oops: 0000 [#2] SMP
    > Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
    > Modules linked in: [<c01276d7>]
    >
    > panic+0x38/0xe0
    > Pid: 0, comm: swapper Tainted: G      D   (2.6.26-rc8-tip #1)
    > EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 3
    >  [<c0104bcd>] EIP is at 0x0
    > EAX: c0614c00 EBX: 00000003 ECX: 079a5000 EDX: ffff3bdf
    > ESI: 00000000 EDI: 00000000 EBP: f7cbbfac ESP: f7cbbfa4
    >  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
    > Process swapper (pid: 0, ti=f7cba000 task=f7cb8fe0 task.ti=f7cba000)
    > Stack: die+0x130/0x147
    > c01020c7 0602080c f7cbbfb4 c040b6cf 00000000  [<c0411795>] 00000000
    > 00000000 00000000
    >        00000000 do_page_fault+0x3bd/0x482
    > 00000000  [<c04113d8>] ? 00000000 00000000 00000000 00000000
    > do_page_fault+0x0/0x482
    > 000000d8 00000000  [<c040fbda>]
    >        00000000 error_code+0x72/0x78
    > 00000000 00000000  [<c01020c7>] ? 00000000 00000000 cpu_idle+0x8a/0x9e
    > 00000000 00000000
    > Call Trace:
    >  [<c040b6cf>] <0> [<c01020c7>] start_secondary+0xbb/0xbd
    > ?  =======================
    > cpu_idle+0x8a/0x9e
    >  [<c040b6cf>] ? start_secondary+0xbb/0xbd
    >  =======================
    > Code:  Bad EIP value.
    > EIP: [<00000000>] 0x0 SS:ESP 0068:f7cbbfa4
    > Oops: 0000 [#3] <0>Kernel panic - not syncing: Fatal exception
    > SMP Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
    >
    >  [<c01276d7>] Modules linked in:
    >
    > panic+0x38/0xe0
    > Pid: 0, comm: swapper Tainted: G      D   (2.6.26-rc8-tip #1)
    >  [<c0104bcd>] EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 7
    > die+0x130/0x147
    > EIP is at 0x0
    >  [<c0411795>] EAX: c0614c00 EBX: 00000007 ECX: 079c5000 EDX: ffff3bdf
    > ESI: 00000000 EDI: 00000000 EBP: f7d1bfac ESP: f7d1bfa4
    >  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
    > Process swapper (pid: 0, ti=f7d1a000 task=f7d18be0
    > task.ti=f7d1a000)do_page_fault+0x3bd/0x482
    >
    > Stack:  [<c04113d8>] c01020c7 ? 0702080c f7d1bfb4
    > do_page_fault+0x0/0x482
    > c040b6cf  [<c040fbda>] 00000000 00000000 error_code+0x72/0x78
    > 00000000 00000000
    >         [<c01020c7>] ? 00000000 00000000 cpu_idle+0x8a/0x9e
    > 00000000 00000000  [<c040b6cf>] 00000000 00000000 000000d8
    > start_secondary+0xbb/0xbd
    > 00000000  =======================
    >
    >        00000000 00000000 00000000 00000000 00000000 00000000 00000000
    > Call Trace:
    >  [<c01020c7>] ? cpu_idle+0x8a/0x9e
    >  [<c040b6cf>] ? start_secondary+0xbb/0xbd
    >  =======================
    > Code:  Bad EIP value.
    > EIP: [<00000000>] 0x0 SS:ESP 0068:f7d1bfa4
    > Kernel panic - not syncing: Fatal exception
    > Oops: 0000 [#4] SMP
    > Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
    > Modules linked in: [<c01276d7>]
    > panic+0x38/0xe0
    >
    >  [<c0104bcd>] Pid: 0, comm: swapper Tainted: G      D   (2.6.26-rc8-tip
    > #1)
    > EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 4
    > die+0x130/0x147
    > EIP is at 0x0
    > EAX: c0614c00 EBX: 00000004 ECX: 079ad000 EDX: ffff3bdf
    > ESI: 00000000 EDI: 00000000 EBP: f7ccbfac ESP: f7ccbfa4
    >  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
    > Process swapper (pid: 0, ti=f7cca000 task=f7cc90e0 task.ti=f7cca000)
    > [<c0411795>]
    > Stack: c01020c7 do_page_fault+0x3bd/0x482
    > 0102080c  [<c04113d8>] f7ccbfb4 c040b6cf 00000000 00000000 00000000
    > 00000000 ?
    >        00000000 00000000 00000000 00000000 00000000
    > do_page_fault+0x0/0x482
    > 00000000 000000d8 00000000
    >        00000000 00000000 00000000 00000000 00000000 00000000 00000000
    > Call Trace:
    >  [<c040fbda>] <0> [<c01020c7>] ? error_code+0x72/0x78
    >  [<c01020c7>] ? cpu_idle+0x8a/0x9e
    > cpu_idle+0x8a/0x9e
    >  [<c040b6cf>] ?  [<c040b6cf>] start_secondary+0xbb/0xbd
    >  =======================
    > start_secondary+0xbb/0xbd
    >  =======================
    > Code:  Bad EIP value.
    > EIP: [<00000000>] 0x0 SS:ESP 0068:f7ccbfa4
    > Oops: 0000 [#5] <0>Kernel panic - not syncing: Fatal exception
    > SMP
    > Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
    > Modules linked in: [<c01276d7>]
    >
    > panic+0x38/0xe0
    > Pid: 0, comm: swapper Tainted: G      D   (2.6.26-rc8-tip #1)
    >  [<c0104bcd>] EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 6
    > die+0x130/0x147
    > EIP is at 0x0
    > EAX: c0614c00 EBX: 00000006 ECX: 079bd000 EDX: ffff3bdf
    > ESI: 00000000 EDI: 00000000 EBP: f7d0bfac ESP: f7d0bfa4
    >  [<c0411795>]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
    > Process swapper (pid: 0, ti=f7d0a000 task=f7d092e0
    > task.ti=f7d0a000)do_page_fault+0x3bd/0x482
    >
    > Stack:  [<c04113d8>] ? c01020c7 0502080c do_page_fault+0x0/0x482
    > f7d0bfb4  [<c040fbda>] c040b6cf error_code+0x72/0x78
    > 00000000  [<c01020c7>] ? 00000000 cpu_idle+0x8a/0x9e
    > 00000000  [<c040b6cf>] 00000000 start_secondary+0xbb/0xbd
    >
    >         =======================
    > 00000000 00000000 00000000 00000000 00000000 00000000 000000d8 00000000
    >        00000000 00000000 00000000 00000000 00000000 00000000 00000000
    > Call Trace:
    >  [<c01020c7>] ? cpu_idle+0x8a/0x9e
    >  [<c040b6cf>] ? start_secondary+0xbb/0xbd
    >  =======================
    > Code:  Bad EIP value.
    > EIP: [<00000000>] 0x0 SS:ESP 0068:f7d0bfa4
    > Kernel panic - not syncing: Fatal exception
    > Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
    >  [<c01276d7>] panic+0x38/0xe0
    >  [<c0104bcd>] die+0x130/0x147
    >  [<c0411795>] do_page_fault+0x3bd/0x482
    >  [<c04113d8>] ? do_page_fault+0x0/0x482
    >  [<c040fbda>] error_code+0x72/0x78
    >  [<c01020c7>] ? cpu_idle+0x8a/0x9e
    >  [<c040b6cf>] start_secondary+0xbb/0xbd
    >  =======================
    > NULL pointer dereference at 00000000
    > IP: [<00000000>]
    > *pde = 00000000
    > Oops: 0000 [#6] SMP
    > Modules linked in:
    >
    > Pid: 0, comm: swapper Tainted: G      D   (2.6.26-rc8-tip #1)
    > EIP: 0060:[<00000000>] EFLAGS: 00010006 CPU: 1
    > EIP is at 0x0
    > EAX: c0614c00 EBX: 00000001 ECX: 07995000 EDX: ffff3bdf
    > ESI: 00000000 EDI: 00000000 EBP: f7c7ffb4 ESP: f7c7ffac
    >  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
    > Process swapper (pid: 0, ti=f7c7e000 task=f7c7cde0 task.ti=f7c7e000)
    > Stack: c01020c7 0202080c f7c7ffbc c040b6cf 00000000 00000000 00000000
    > 00000000
    >        00000000 00000000 00000000 00000000 000000d8 00000000 00000000
    > 00000000
    >        00000000 00000000 00000000 00000000 00000000
    > Call Trace:
    >  [<c01020c7>] ? cpu_idle+0x8a/0x9e
    >  [<c040b6cf>] ? start_secondary+0xbb/0xbd
    >  =======================
    > Code:  Bad EIP value.
    > EIP: [<00000000>] 0x0 SS:ESP 0068:f7c7ffac
    > Kernel panic - not syncing: Fatal exception
    > Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
    >  [<c01276d7>] panic+0x38/0xe0
    >  [<c0104bcd>] die+0x130/0x147
    >  [<c0411795>] do_page_fault+0x3bd/0x482
    >  [<c04113d8>] ? do_page_fault+0x0/0x482
    >  [<c040fbda>] error_code+0x72/0x78
    >  [<c01020c7>] ? cpu_idle+0x8a/0x9e
    >  [<c040b6cf>] start_secondary+0xbb/0xbd
    >  =======================
    > BUG: NMI Watchdog detected LOCKUP on CPU6, ip c0111959, registers:
    > Modules linked in:
    >
    > Pid: 0, comm: swapper Tainted: G      D   (2.6.26-rc8-tip #1)
    > EIP: 0060:[<c0111959>] EFLAGS: 00000093 CPU: 6
    > EIP is at __smp_call_function+0x5d/0x7a
    > EAX: 0000009e EBX: 00000005 ECX: 00000006 EDX: f7d092e0
    > ESI: c01047d4 EDI: c0111a5a EBP: f7d0bf00 ESP: f7d0bed0
    >  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
    > Process swapper (pid: 0, ti=f7d0a000 task=f7d092e0 task.ti=f7d0a000)
    > Stack: 00000000 c0111a5a 00000000 00000000 c01047d4 00000000 c029915b
    > f7d0bf6c
    >        00000006 00000001 00000046 00000000 f7d0bf14 c0111acb 00000000
    > f7d0bf6c
    >        00000006 f7d0bf20 c01276ee f7d0bf6c f7d0bf3c c0104bcd c04d922f
    > c04e06cf
    > Call Trace:
    >  [<c0111a5a>] ? stop_this_cpu+0x0/0x3a
    >  [<c01047d4>] ? show_trace+0x10/0x12
    >  [<c029915b>] ? do_unblank_screen+0x2a/0xf9
    >  [<c0111acb>] ? native_smp_send_stop+0x37/0x6a
    >  [<c01276ee>] ? panic+0x4f/0xe0
    >  [<c0104bcd>] ? die+0x130/0x147
    >  [<c0411795>] ? do_page_fault+0x3bd/0x482
    >  [<c04113d8>] ? do_page_fault+0x0/0x482
    >  [<c040fbda>] ? error_code+0x72/0x78
    >  [<c01020c7>] ? cpu_idle+0x8a/0x9e
    >  [<c040b6cf>] ? start_secondary+0xbb/0xbd
    >  =======================
    > Code: 85 c0 0f 44 75 e0 89 45 e4 8d 45 d4 a3 d4 33 62 c0 89 75 e0 0f ae
    > f0 0f 1f 00 8b 15 e0 69 57 c0 b8 fb 00 00 00 ff 52 78 39 5d dc <74> 04
    > f3 90 eb f7 83 7d 08 00 74 09 39 5d e0 74 04 f3 90 eb f7
    >
    >
    > So after digging around a bit, it turns out the pm_idle is NULL. For
    > some reason it is not getting set to default_idle if nothing works.
    
    Signed-off-by: Ingo Molnar <[email protected]>
arch/x86/kernel/process_32.c +2 -0
index 28e77d2..75755e2 100644
@@ -144,6 +144,8 @@ void cpu_idle(void)
 			__get_cpu_var(irq_stat).idle_timestamp = jiffies;
 			/* Don't trace irqs off for idle */
 			stop_critical_timings();
+			if (WARN_ON_ONCE(!pm_idle))
+				pm_idle = default_idle;
 			pm_idle();
 			start_critical_timings();
 		}
arch/x86/kernel/process_64.c +2 -0
index 395a72f..f07278f 100644
@@ -148,6 +148,8 @@ void cpu_idle(void)
 			enter_idle();
 			/* Don't trace irqs off for idle */
 			stop_critical_timings();
+			if (WARN_ON_ONCE(!pm_idle))
+				pm_idle = default_idle;
 			pm_idle();
 			start_critical_timings();
 			/* In many cases the interrupt that ended idle
drivers/acpi/processor_idle.c +6 -2
index 4976e5d..ff57bb2 100644
@@ -1313,7 +1313,9 @@ int acpi_processor_cst_has_changed(struct acpi_processor *pr)
 		return -ENODEV;
 
 	/* Fall back to the default idle loop */
-	pm_idle = pm_idle_save;
+	WARN_ON_ONCE(!pm_idle_save);
+	if (pm_idle_save)
+		pm_idle = pm_idle_save;
 	synchronize_sched();	/* Relies on interrupts forcing exit from idle. */
 
 	pr->flags.power = 0;
@@ -1864,7 +1866,9 @@ int acpi_processor_power_exit(struct acpi_processor *pr,
 
 	/* Unregister the idle handler when processor #0 is removed. */
 	if (pr->id == 0) {
-		pm_idle = pm_idle_save;
+		WARN_ON_ONCE(!pm_idle_save);
+		if (pm_idle_save)
+			pm_idle = pm_idle_save;
 
 		/*
 		 * We are about to unload the current idle thread pm callback
drivers/cpuidle/cpuidle.c +3 -1
index 5405769..bb0ba6e 100644
@@ -95,7 +95,9 @@ void cpuidle_install_idle_handler(void)
 void cpuidle_uninstall_idle_handler(void)
 {
 	if (enabled_devices && (pm_idle != pm_idle_old)) {
-		pm_idle = pm_idle_old;
+		WARN_ON_ONCE(!pm_idle_old);
+		if (pm_idle_old)
+			pm_idle = pm_idle_old;
 		cpuidle_kick_cpus();
 	}
 }
#3 lore raw
Quoted 274 lines · Ingo Molnar
Quoted 271 lines · Dhaval Giani

[Missed cc'ing the LKML last time around]

Quoted 259 lines · Dhaval Giani

Hi Ingo, Thomas,

I am hitting this on -tip. With 200a86b5d435a217c3d77f3b53cd32cb78c1fde8 as the top level commit. Wondering if it is known?

I am trying to fix it atm.

Thanks,

Red Hat Enterprise Linux AS release 4 (Nahant Update 2) Kernel 2.6.26-rc8-tip on an i686

llm11.in.ibm.com login: root
Password:
Last login: Thu Jul 3 00:30:47 on ttyS0 You have new mail.
cd[root@llm11 ~]# cd /sys/devices/system/cpu/cpu1/ [root@llm11 cpu1]# echo 0 > online Breaking affinity for irq 45
[root@llm11 cpu1]# echo 1 > online
lockdep: fixing up alternatives.

BUG: unable to handle kernel <1>BUG: unable to handle kernel NULL
pointer dereference at 00000000
IP: [<00000000>]
*pde = 00000000 
Oops: 0000 [#1] SMP 
Modules linked in:

Pid: 0, comm: swapper Not tainted (2.6.26-rc8-tip #1)
EIP: 0060:[<00000000>] EFLAGS: 00010002 CPU: 2 EIP is at 0x0
EAX: c0614c00 EBX: 00000002 ECX: 0799d000 EDX: ffff3bdf

BUG: unable to handle kernel NULL pointer dereference at 00000000
IP:ESI: 00000000 EDI: 00000000 EBP: f7cadfac ESP: f7cadfa4
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
 [<00000000>]
BUG: unable to handle kernel NULL pointer dereference*pde = 00000000

<1>BUG: unable to handle kernel at 00000000 IP:<0>Process swapper (pid: 0, ti=f7cac000 task=f7caaee0 task.ti=f7cac000)NULL pointer dereference [<00000000>] *pde = 00000000 <1>BUG: unable to handle kernel

Stack: 
 at 00000000

IP:NULL pointer dereference [<00000000>]
at 00000000
c01020c7 *pde = 00000000 <1>IP:

[<00000000>]
0402080c *pde = 00000000 f7cadfb4
c040b6cf 00000000 00000000 00000000 00000000

       00000000 00000000 00000000 00000000 00000000 00000000 000000d8
00000000 
       00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7cadfa4
Kernel panic - not syncing: Fatal exception
Oops: 0000 [#2] SMP 
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
Modules linked in: [<c01276d7>] 

panic+0x38/0xe0

Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 3
[<c0104bcd>] EIP is at 0x0
EAX: c0614c00 EBX: 00000003 ECX: 079a5000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7cbbfac ESP: f7cbbfa4
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7cba000 task=f7cb8fe0 task.ti=f7cba000)

Stack: die+0x130/0x147
c01020c7 0602080c f7cbbfb4 c040b6cf 00000000  [<c0411795>] 00000000
00000000 00000000 
       00000000 do_page_fault+0x3bd/0x482
00000000  [<c04113d8>] ? 00000000 00000000 00000000 00000000
do_page_fault+0x0/0x482
000000d8 00000000  [<c040fbda>] 
       00000000 error_code+0x72/0x78
00000000 00000000  [<c01020c7>] ? 00000000 00000000 cpu_idle+0x8a/0x9e
00000000 00000000 
Call Trace:
 [<c040b6cf>] <0> [<c01020c7>] start_secondary+0xbb/0xbd
?  =======================
cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7cbbfa4
Oops: 0000 [#3] <0>Kernel panic - not syncing: Fatal exception

SMP Pid: 0, comm: swapper Tainted: G D 2.6.26-rc8-tip #1

[<c01276d7>] Modules linked in:

panic+0x38/0xe0
Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
[<c0104bcd>] EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 7
die+0x130/0x147
EIP is at 0x0
[<c0411795>] EAX: c0614c00 EBX: 00000007 ECX: 079c5000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7d1bfac ESP: f7d1bfa4
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7d1a000 task=f7d18be0 task.ti=f7d1a000)do_page_fault+0x3bd/0x482

Stack:  [<c04113d8>] c01020c7 ? 0702080c f7d1bfb4
do_page_fault+0x0/0x482
c040b6cf  [<c040fbda>] 00000000 00000000 error_code+0x72/0x78
00000000 00000000 
        [<c01020c7>] ? 00000000 00000000 cpu_idle+0x8a/0x9e

00000000 00000000 [<c040b6cf>] 00000000 00000000 000000d8 start_secondary+0xbb/0xbd
00000000 =======================

       00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7d1bfa4
Kernel panic - not syncing: Fatal exception
Oops: 0000 [#4] SMP 
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
Modules linked in: [<c01276d7>] 
panic+0x38/0xe0

[<c0104bcd>] Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip
#1)
EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 4 die+0x130/0x147
EIP is at 0x0
EAX: c0614c00 EBX: 00000004 ECX: 079ad000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7ccbfac ESP: f7ccbfa4
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7cca000 task=f7cc90e0 task.ti=f7cca000) [<c0411795>]

Stack: c01020c7 do_page_fault+0x3bd/0x482
0102080c  [<c04113d8>] f7ccbfb4 c040b6cf 00000000 00000000 00000000
00000000 ? 
       00000000 00000000 00000000 00000000 00000000
do_page_fault+0x0/0x482
00000000 000000d8 00000000 
       00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c040fbda>] <0> [<c01020c7>] ? error_code+0x72/0x78
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
cpu_idle+0x8a/0x9e
 [<c040b6cf>] ?  [<c040b6cf>] start_secondary+0xbb/0xbd
 =======================
start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7ccbfa4
Oops: 0000 [#5] <0>Kernel panic - not syncing: Fatal exception
SMP 
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
Modules linked in: [<c01276d7>] 

panic+0x38/0xe0

Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
[<c0104bcd>] EIP: 0060:[<00000000>] EFLAGS: 00010046 CPU: 6
die+0x130/0x147
EIP is at 0x0
EAX: c0614c00 EBX: 00000006 ECX: 079bd000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7d0bfac ESP: f7d0bfa4
[<c0411795>] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7d0a000 task=f7d092e0 task.ti=f7d0a000)do_page_fault+0x3bd/0x482

Stack:  [<c04113d8>] ? c01020c7 0502080c do_page_fault+0x0/0x482
f7d0bfb4  [<c040fbda>] c040b6cf error_code+0x72/0x78
00000000  [<c01020c7>] ? 00000000 cpu_idle+0x8a/0x9e
00000000  [<c040b6cf>] 00000000 start_secondary+0xbb/0xbd
        =======================
00000000 00000000 00000000 00000000 00000000 00000000 000000d8 00000000 
       00000000 00000000 00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7d0bfa4
Kernel panic - not syncing: Fatal exception
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
 [<c01276d7>] panic+0x38/0xe0
 [<c0104bcd>] die+0x130/0x147
 [<c0411795>] do_page_fault+0x3bd/0x482
 [<c04113d8>] ? do_page_fault+0x0/0x482
 [<c040fbda>] error_code+0x72/0x78
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] start_secondary+0xbb/0xbd
 =======================
NULL pointer dereference at 00000000
IP: [<00000000>]
*pde = 00000000 
Oops: 0000 [#6] SMP 
Modules linked in:

Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
EIP: 0060:[<00000000>] EFLAGS: 00010006 CPU: 1
EIP is at 0x0
EAX: c0614c00 EBX: 00000001 ECX: 07995000 EDX: ffff3bdf
ESI: 00000000 EDI: 00000000 EBP: f7c7ffb4 ESP: f7c7ffac
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7c7e000 task=f7c7cde0 task.ti=f7c7e000)

Stack: c01020c7 0202080c f7c7ffbc c040b6cf 00000000 00000000 00000000
00000000 
       00000000 00000000 00000000 00000000 000000d8 00000000 00000000
00000000 
       00000000 00000000 00000000 00000000 00000000 
Call Trace:
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code:  Bad EIP value.
EIP: [<00000000>] 0x0 SS:ESP 0068:f7c7ffac
Kernel panic - not syncing: Fatal exception
Pid: 0, comm: swapper Tainted: G      D   2.6.26-rc8-tip #1
 [<c01276d7>] panic+0x38/0xe0
 [<c0104bcd>] die+0x130/0x147
 [<c0411795>] do_page_fault+0x3bd/0x482
 [<c04113d8>] ? do_page_fault+0x0/0x482
 [<c040fbda>] error_code+0x72/0x78
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] start_secondary+0xbb/0xbd
 =======================
BUG: NMI Watchdog detected LOCKUP on CPU6, ip c0111959, registers:
Modules linked in:

Pid: 0, comm: swapper Tainted: G D (2.6.26-rc8-tip #1)
EIP: 0060:[<c0111959>] EFLAGS: 00000093 CPU: 6
EIP is at __smp_call_function+0x5d/0x7a
EAX: 0000009e EBX: 00000005 ECX: 00000006 EDX: f7d092e0
ESI: c01047d4 EDI: c0111a5a EBP: f7d0bf00 ESP: f7d0bed0
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=f7d0a000 task=f7d092e0 task.ti=f7d0a000)

Stack: 00000000 c0111a5a 00000000 00000000 c01047d4 00000000 c029915b
f7d0bf6c 
       00000006 00000001 00000046 00000000 f7d0bf14 c0111acb 00000000
f7d0bf6c 
       00000006 f7d0bf20 c01276ee f7d0bf6c f7d0bf3c c0104bcd c04d922f
c04e06cf 
Call Trace:
 [<c0111a5a>] ? stop_this_cpu+0x0/0x3a
 [<c01047d4>] ? show_trace+0x10/0x12
 [<c029915b>] ? do_unblank_screen+0x2a/0xf9
 [<c0111acb>] ? native_smp_send_stop+0x37/0x6a
 [<c01276ee>] ? panic+0x4f/0xe0
 [<c0104bcd>] ? die+0x130/0x147
 [<c0411795>] ? do_page_fault+0x3bd/0x482
 [<c04113d8>] ? do_page_fault+0x0/0x482
 [<c040fbda>] ? error_code+0x72/0x78
 [<c01020c7>] ? cpu_idle+0x8a/0x9e
 [<c040b6cf>] ? start_secondary+0xbb/0xbd
 =======================
Code: 85 c0 0f 44 75 e0 89 45 e4 8d 45 d4 a3 d4 33 62 c0 89 75 e0 0f ae
Quoted 3 lines

f0 0f 1f 00 8b 15 e0 69 57 c0 b8 fb 00 00 00 ff 52 78 39 5d dc <74> 04 f3 90 eb f7 83 7d 08 00 74 09 39 5d e0 74 04 f3 90 eb f7

So after digging around a bit, it turns out the pm_idle is NULL. For some reason it is not getting set to default_idle if nothing works. I am not sure of the path being followed, and its a bit late for me to be trying anything serious :).

Quoted 6 lines
Quoted 2 lines

This seems to work as a temporary workaround, but obviously is not the right fix yet.

hm, interesting - this is a new bug. I'm wondering where that NULL came from. x86 boot itself does not leave room for pm_idle to be NULL AFAICS.

Well, it is not null at boot time, since the system does boot up. It is lost (corrupted?) sometime later.

Quoted 8 lines

One suspect would be cpuidle:

void cpuidle_uninstall_idle_handler(void)

 {
         if (enabled_devices && (pm_idle != pm_idle_old)) {
                 pm_idle = pm_idle_old;
                 cpuidle_kick_cpus();
Quoted 8 lines

The other suspect would be acpi idle:

        /* Fall back to the default idle loop */
        pm_idle = pm_idle_save;

Could you try latest tip/master or the debug patch below? It should show where the NULL comes from, without crashing.

Sure, will give it a run sometime soon.

thanks,

Signature · 2 lines
regards,
Dhaval
#4 lore raw
Quoted 6 lines
Quoted 6 lines

The other suspect would be acpi idle:

        /* Fall back to the default idle loop */
        pm_idle = pm_idle_save;

Yep, you are right.

Quoted 6 lines
Quoted 3 lines

Could you try latest tip/master or the debug patch below? It should show where the NULL comes from, without crashing.

Sure, will give it a run sometime soon.

------------[ cut here ]------------
WARNING: at drivers/acpi/processor_idle.c:1316
acpi_processor_cst_has_changed+0x64/0xc0()
Modules linked in:
Pid: 4433, comm: bash Not tainted 2.6.26-rc8-tip #15
 [<c0127a0a>] warn_on_slowpath+0x41/0x60
 [<c0393a00>] ? __cpufreq_set_policy+0x135/0x1bf
 [<c0141c28>] ? mark_held_locks+0x46/0x61
 [<c0141d7d>] ? trace_hardirqs_on+0xb/0xd
 [<c0141d4a>] ? trace_hardirqs_on_caller+0xe9/0x111
 [<c0141d7d>] ? trace_hardirqs_on+0xb/0xd
 [<c027fc1d>] ? acpi_processor_get_platform_limit+0x9f/0xac
 [<c027f285>] acpi_processor_cst_has_changed+0x64/0xc0
 [<c027cadd>] acpi_cpu_soft_notify+0x2a/0x39
 [<c041193c>] notifier_call_chain+0x32/0x64
 [<c013b87f>] __raw_notifier_call_chain+0xe/0x10
 [<c013b88d>] raw_notifier_call_chain+0xc/0xe
 [<c040cb33>] _cpu_up+0xb3/0xdc
 [<c040cb9e>] cpu_up+0x42/0x52
 [<c03fab35>] store_online+0x39/0x5d
 [<c03faafc>] ? store_online+0x0/0x5d
 [<c02c1e6c>] sysdev_store+0x20/0x25
 [<c01b2190>] flush_write_buffer+0x3e/0x53
 [<c01b21e3>] sysfs_write_file+0x3e/0x5d
 [<c017e281>] vfs_write+0x8d/0x105
 [<c017e394>] sys_write+0x3b/0x60
 [<c0103931>] sysenter_past_esp+0x6a/0xa5
 =======================
---[ end trace 987615de2ae7cfa5 ]---

Which corresponds to

        /* Fall back to the default idle loop */
        WARN_ON_ONCE(!pm_idle_save);
        if (pm_idle_save)
                pm_idle = pm_idle_save;
        synchronize_sched();    /* Relies on interrupts forcing exit from idle. */
Signature · 2 lines
regards,
Dhaval