NULL dereference OOPS on SysRq-w
I've experienced the following on a build done today from Linus's linux-2.6 tree. After a fresh boot I do "echo "w" > /proc/sysrq_trigger" and I get the OOPS, every time. I looked at the sched_debug code but I admit I was lost. Not sure why sched.c includes sched_debug.c.
Cheers,
Dan
Via netconsole:
SysRq : Show Blocked State
task PC stack pid father
Sched Debug Version: v0.07, 2.6.25 #12
now at 275558.648467 msecs
.sysctl_sched_latency : 40.000000 .sysctl_sched_min_granularity : 8.000000 .sysctl_sched_wakeup_granularity : 20.000000 .sysctl_sched_child_runs_first : 0.000001 .sysctl_sched_features : 895
cpu#0, 2194.491 MHz
.nr_running : 3 .load : 3145728 .nr_switches : 13171 .nr_load_updates : 6476 .nr_uninterruptible : -94 .jiffies : 4294961186 .next_balance : 4294.961046 .curr->pid : 3078 .clock : 101239.537371 .idle_clock : 0.000000 .prev_clock_raw : 296931.974646 .clock_warps : 0 .clock_overflows : 12189 .clock_underflows : 4657 .clock_deep_idle_events : 4 .clock_max_delta : 499.306905 .cpu_load[0] : 0 .cpu_load[1] : 96 .cpu_load[2] : 14028 .cpu_load[3] : 62524 .cpu_load[4] : 91704
cfs_rq[0]:
.exec_clock : 400.314576 .MIN_vruntime : 17300.019734 .min_vruntime : 2099676.053218 .max_vruntime : 17300.019734 .spread : 0.000000 .spread0 : 0.000000 .nr_running : 1 .load : 1024 .bkl_count : 353 .nr_spread_over : 44 .shares : 1024
cfs_rq[0]:
.exec_clock : 1.281845 .MIN_vruntime : 0.000001 .min_vruntime : 2099676.053218 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : 0.000000 .nr_running : 0 .load : 0 .bkl_count : 353 .nr_spread_over : 2 .shares : 0
cfs_rq[0]:
.exec_clock : 13.306219 .MIN_vruntime : 0.000001 .min_vruntime : 2099676.053218 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : 0.000000 .nr_running : 0 .load : 0 .bkl_count : 353 .nr_spread_over : 3 .shares : 0
cfs_rq[0]:
.exec_clock : 0.380305 .MIN_vruntime : 0.000001 .min_vruntime : 2099676.053218 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : 0.000000 .nr_running : 0 .load : 0 .bkl_count : 353 .nr_spread_over : 1 .shares : 0
cfs_rq[0]:
.exec_clock : 6.341858 .MIN_vruntime : 0.000001 .min_vruntime : 2099676.053218 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : 0.000000 .nr_running : 0 .load : 0 .bkl_count : 353 .nr_spread_over : 1 .shares : 0
cfs_rq[0]:
.exec_clock : 6.341858 .MIN_vruntime : 0.000001 .min_vruntime : 2099676.053218 .max_vruntime : 0.000001 .spread : 0.000000 .spread0 : 0.000000 .nr_running : 0 .load : 0 .bkl_count : 353 .nr_spread_over : 16 .shares : 1024
cfs_rq[0]:
.exec_clock : 12850.277817 .MIN_vruntime : 69710.961030 .min_vruntime : 2099676.053218 .max_vruntime : 69751.539537 .spread : 40.578507 .spread0 : 0.000000 .nr_running : 3 .load : 5169 .bkl_count
: 353 .nr_spread_over : 2038 .shares
: 2048BUG: unable to handle kernel NULL pointer dereference at
0000000000000018IP: [<ffffffff802b9ab6>] seq_printf+0x2e/0xa7
PGD 7dd05067 PUD 7e0fb067 PMD 0 Oops: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOCCPU 0
Modules linked in: netconsole nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs ac battery ipv6 dm_snapshot dm_mirror dm_log dm_mod eeprom coretemp loop parport_pc parport snd_hda_intel rtc iTCO_wdt iTCO_vendor_support i2c_i801 i2c_core snd_pc m snd_timer snd soundcore snd_page_alloc button intel_agp evdev ext3 jbd mbcache sd_mod ata_generic ata_piix pata_acpi libata scsi_mod dock ehci_hcd uhci_hcd ide_pci_generic piix ide_core r8169 usbcore thermal processor fan Pid: 3078, comm: zsh Not tainted 2.6.25 #12 RIP: 0010:[<ffffffff802b9ab6>] [<ffffffff802b9ab6>] seq_printf+0x2e/0xa7 RSP: 0018:ffff81007e387c68 EFLAGS: 00010092 RAX: ffffffff804eec79 RBX: 0000000000000000 RCX: ffffee71806668f8 RDX: ffff810001091f80 RSI: ffffffff804eec79 RDI: 0000000000000000 RBP: ffff81007e387d48 R08: 0000000000000002 R09: ffffffff8024fb76 R10: 00000000d59bb5d6 R11: ffff81007e387968 R12: 0000000000000000 R13: 0000000000000000 R14: ffff810001091740 R15: 0000000000000007 FS: 00007f7dce0896e0(0000) GS:ffffffff8057a000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000018 CR3: 000000007e3ed000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process zsh (pid: 3078, threadinfo ffff81007e386000, task ffff81007e090000) Stack: 0000000000000002 ffffffff8024fb76 ffff81007e090090 ffff81007e090000 ffffffff8022dc31 0000000000000046 ffff810001091f80 ffffee71806668f8 0000000000000002 ffffffff8024fb76 0000000000000096 0000000000000000 Call Trace: [<ffffffff8024fb76>] ? up+0xf/0x39 [<ffffffff8022dc31>] ? print_cfs_rq+0x130/0x465 [<ffffffff8024fb76>] ? up+0xf/0x39 [<ffffffff8041dfa5>] ? _spin_unlock_irqrestore+0x66/0x74 [<ffffffff8022df57>] ? print_cfs_rq+0x456/0x465 [<ffffffff8022e920>] sched_debug_show+0x9ba/0xd28 [<ffffffff8022f978>] ? cpu_clock+0xf0/0xff [<ffffffff802745fe>] ? get_timestamp+0x9/0xf [<ffffffff80274628>] ? touch_softlockup_watchdog+0x24/0x2d [<ffffffff802319ec>] show_state_filter+0x8b/0xaa [<ffffffff80375848>] sysrq_handle_showstate_blocked+0xe/0x10 [<ffffffff803757a3>] __handle_sysrq+0x9e/0x135 [<ffffffff802e2544>] ? write_sysrq_trigger+0x0/0x3a [<ffffffff802e2574>] write_sysrq_trigger+0x30/0x3a [<ffffffff802dc6b2>] proc_reg_write+0x8a/0xa7 [<ffffffff802a0118>] vfs_write+0xa7/0xe1 [<ffffffff802a020c>] sys_write+0x47/0x6d [<ffffffff8020c3ab>] system_call_after_swapgs+0x7b/0x80 Code: f0 48 89 e5 53 48 89 fb 48 81 ec d8 00 00 00 48 89 95 50 ff ff ff 48 89 8d 58 ff ff ff 4c 89 85 60 ff ff ff 4c 89 8d 68 ff ff ff <48> 8b 7f 18 48 3b 7b 08 73 5a 48 8b 73 08 48 8d 55 10 48 8d 8d RIP [<ffffffff802b9ab6>] seq_printf+0x2e/0xa7 RSP <ffff81007e387c68> CR2: 0000000000000018 ---[ end trace ffd04a8c8a72749a ]--- note: zsh[3078] exited with preempt_count 2 BUG: sleeping function called from invalid context at kernel/rwsem.c:21 in_atomic():1, irqs_disabled():1 INFO: lockdep is turned off. irq event stamp: 136430 hardirqs last enabled at (136429): [<ffffffff8041d4c0>] trace_hardirqs_on_thunk+0x35/0x3a hardirqs last disabled at (136430): [<ffffffff8041de72>] _spin_lock_irqsave+0x19/0x73 softirqs last enabled at (131948): [<ffffffff8023dcec>] __do_softirq+0x9e/0xa7 softirqs last disabled at (131911): [<ffffffff8020d68c>] call_softirq+0x1c/0x28 Pid: 3078, comm: zsh Tainted: G D 2.6.25 #12 Call Trace: [<ffffffff80255f3d>] ? print_irqtrace_events+0x110/0x114 [<ffffffff8022f7fc>] __might_sleep+0xda/0xdc [<ffffffff8041c7f6>] down_read+0x20/0x68 [<ffffffff802681eb>] acct_collect+0x42/0x1a7 [<ffffffff8023ba57>] do_exit+0x205/0x713 [<ffffffff803723e2>] ? do_unblank_screen+0x29/0x125 [<ffffffff8020d90d>] oops_begin+0x0/0x8f [<ffffffff80223c64>] do_page_fault+0x69f/0x757 [<ffffffff8041e2ed>] error_exit+0x0/0xa9 [<ffffffff8024fb76>] ? up+0xf/0x39 [<ffffffff802b9ab6>] ? seq_printf+0x2e/0xa7 [<ffffffff8024fb76>] ? up+0xf/0x39 [<ffffffff8022dc31>] ? print_cfs_rq+0x130/0x465 [<ffffffff8024fb76>] ? up+0xf/0x39 [<ffffffff8041dfa5>] ? _spin_unlock_irqrestore+0x66/0x74 [<ffffffff8022df57>] ? print_cfs_rq+0x456/0x465 [<ffffffff8022e920>] ? sched_debug_show+0x9ba/0xd28 [<ffffffff8022f978>] ? cpu_clock+0xf0/0xff [<ffffffff802745fe>] ? get_timestamp+0x9/0xf [<ffffffff80274628>] ? touch_softlockup_watchdog+0x24/0x2d [<ffffffff802319ec>] ? show_state_filter+0x8b/0xaa [<ffffffff80375848>] ? sysrq_handle_showstate_blocked+0xe/0x10 [<ffffffff803757a3>] ? __handle_sysrq+0x9e/0x135 [<ffffffff802e2544>] ? write_sysrq_trigger+0x0/0x3a [<ffffffff802e2574>] ? write_sysrq_trigger+0x30/0x3a [<ffffffff802dc6b2>] ? proc_reg_write+0x8a/0xa7 [<ffffffff802a0118>] ? vfs_write+0xa7/0xe1 [<ffffffff802a020c>] ? sys_write+0x47/0x6d [<ffffffff8020c3ab>] ? system_call_after_swapgs+0x7b/0x80
Signature · 3 lines
/--------------- - - - - - -
| Dan Noe
| http://isomerica.net/~dpn/Quoted 14 lines · Dan Noe
I've experienced the following on a build done today from Linus's linux-2.6 tree. After a fresh boot I do "echo "w" >
/proc/sysrq_trigger" and I get the OOPS, every time. I looked at thesched_debugcode but I admit I was lost. Not sure whysched.cincludessched_debug.c.Cheers,
DanVia netconsole:
SysRq : Show Blocked State
task PC stack pid father
Sched Debug Version: v0.07, 2.6.25 #12
now at 275558.648467 msecs
[snip]
Quoted 49 lines
: 2038 .shares : 2048BUG: unable to handle kernel
NULL pointer dereference at 0000000000000018IP: [<ffffffff802b9ab6>]seq_printf+0x2e/0xa7
PGD7dd05067PUD7e0fb067PMD 0 Oops: 0000 [1] PREEMPT SMPDEBUG_PAGEALLOCCPU0Modules linked in: netconsole nfsd lockd nfs_acl auth_rpcgss sunrpcexportfs ac battery ipv6
dm_snapshotdm_mirrordm_logdm_modeeprom coretemp loopparport_pcparportsnd_hda_intelrtciTCO_wdtiTCO_vendor_supporti2c_i801i2c_coresnd_pc
msnd_timersnd soundcoresnd_page_allocbuttonintel_agpevdev ext3 jbd mbcachesd_modata_genericata_piixpata_acpilibatascsi_moddockehci_hcduhci_hcdide_pci_genericpiixide_corer8169 usbcore thermal processor fan
Pid: 3078, comm: zsh Not tainted 2.6.25 #12RIP: 0010:[<ffffffff802b9ab6>] [<ffffffff802b9ab6>] seq_printf+0x2e/0xa7 RSP: 0018:ffff81007e387c68 EFLAGS: 00010092 RAX: ffffffff804eec79 RBX: 0000000000000000 RCX: ffffee71806668f8 RDX: ffff810001091f80 RSI: ffffffff804eec79 RDI: 0000000000000000 RBP: ffff81007e387d48 R08: 0000000000000002 R09: ffffffff8024fb76 R10: 00000000d59bb5d6 R11: ffff81007e387968 R12: 0000000000000000 R13: 0000000000000000 R14: ffff810001091740 R15: 0000000000000007 FS: 00007f7dce0896e0(0000) GS:ffffffff8057a000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000018 CR3: 000000007e3ed000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process zsh (pid: 3078, threadinfo ffff81007e386000, task ffff81007e090000) Stack: 0000000000000002 ffffffff8024fb76 ffff81007e090090 ffff81007e090000 ffffffff8022dc31 0000000000000046 ffff810001091f80 ffffee71806668f8 0000000000000002 ffffffff8024fb76 0000000000000096 0000000000000000 Call Trace: [<ffffffff8024fb76>] ? up+0xf/0x39 [<ffffffff8022dc31>] ? print_cfs_rq+0x130/0x465 [<ffffffff8024fb76>] ? up+0xf/0x39 [<ffffffff8041dfa5>] ? _spin_unlock_irqrestore+0x66/0x74 [<ffffffff8022df57>] ? print_cfs_rq+0x456/0x465 [<ffffffff8022e920>] sched_debug_show+0x9ba/0xd28 [<ffffffff8022f978>] ? cpu_clock+0xf0/0xff [<ffffffff802745fe>] ? get_timestamp+0x9/0xf [<ffffffff80274628>] ? touch_softlockup_watchdog+0x24/0x2d [<ffffffff802319ec>] show_state_filter+0x8b/0xaa [<ffffffff80375848>] sysrq_handle_showstate_blocked+0xe/0x10 [<ffffffff803757a3>] __handle_sysrq+0x9e/0x135 [<ffffffff802e2544>] ? write_sysrq_trigger+0x0/0x3a [<ffffffff802e2574>] write_sysrq_trigger+0x30/0x3a [<ffffffff802dc6b2>] proc_reg_write+0x8a/0xa7 [<ffffffff802a0118>] vfs_write+0xa7/0xe1 [<ffffffff802a020c>] sys_write+0x47/0x6d [<ffffffff8020c3ab>] system_call_after_swapgs+0x7b/0x80 Code: f0 48 89 e5 53 48 89 fb 48 81 ec d8 00 00 00 48 89 95 50 ff ff ff 48
Quoted 6 lines
89 8d 58 ff ff ff 4c 89 85 60 ff ff ff 4c 89 8d 68 ff ff ff <48> 8b 7f 18 48 3b 7b 08 73 5a 48 8b 73 08 48 8d 55 10 48 8d 8d
RIP [<ffffffff802b9ab6>]seq_printf+0x2e/0xa7
RSP <ffff81007e387c68>
CR2: 0000000000000018---[ end trace ffd04a8c8a72749a ]---
[snip]
I have a fix for the NULL pointer reference BUG, tested on my pc.
---
"m" will be NULL if seq_printf & seq_puts is called from sched_debug_show Use SEQ_* macros to fix it
---
include/linux/sched.h | 22 ++++++++++++++++++++++ kernel/sched_debug.c | 12 ------------ kernel/sched_fair.c | 6 +++--- 3 files changed, 25 insertions(+), 15 deletions(-)
include/linux/sched.h +22 -0
@@ -2145,6 +2145,28 @@ static inline void migration_init(void)#define TASK_SIZE_OF (tsk) TASK_SIZE#endif +#ifdef CONFIG_SCHED_DEBUG +/* +* This allows printing both to /proc/sched_debug and +* to the console +*/ +#define SEQ_printf(m, x...) \ + do { \ + if (m) \ + seq_printf(m, x); \ + else \ + printk(x); \ + } while (0) + +#define SEQ_puts(m, x...) \ + do { \ + if (m) \ + seq_puts(m, x); \ + else \ + printk(x); \ + } while (0) +#endif +#endif /* __KERNEL__ */ #endif
kernel/sched_debug.c +0 -12
@@ -17,18 +17,6 @@#include <linux/utsname.h> /* -* This allows printing both to /proc/sched_debug and -* to the console -*/ -#define SEQ_printf(m, x...) \ - do { \ - if (m) \ - seq_printf(m, x); \ - else \ - printk(x); \ - } while (0) - -/* * Ease the printing of nsec fields: */ static long long nsec_high (unsigned long long nsec)
kernel/sched_fair.c +3 -3
@@ -1623,9 +1623,9 @@ print_cfs_rq_tasks(struct seq_file *m, sint i;for (i = depth; i; i--) -seq_puts (m," " ); +SEQ_puts (m," " ); -seq_printf (m,"%lu %s %lu\n" , +SEQ_printf (m,"%lu %s %lu\n" , se->load.weight,entity_is_task (se) ?"T" :"G" , calc_delta_weight(SCHED_LOAD_SCALE, se)
@@ -1643,7 +1643,7 @@ static void print_cfs_stats(struct seq_ffor_each_leaf_cfs_rq (cpu_rq (cpu), cfs_rq)print_cfs_rq (m, cpu, cfs_rq); -seq_printf (m,"\nWeight tree:\n" ); +SEQ_printf (m,"\nWeight tree:\n" );print_cfs_rq_tasks (m, &cpu_rq (cpu)->cfs,1 );rcu_read_unlock (); }
Quoted 9 lines · Dave Young
[snip]
I have a fix for the NULL pointer reference BUG, tested on my pc.
---"m" will be NULL if
seq_printf&seq_putsis called fromsched_debug_showUse SEQ_* macros to fix itSigned-off-by Dave Young <[email protected]>
There is actually already a patch out for this which simply hasn't made it upstream yet - Mike Galbraith emailed me with it and it indeed fixes the problem. I hadn't realized his email didn't go to lkml.. otherwise I would have echoed that here to avoid wasting people's time. My apologies.
Cheers,
Dan
===
Here is his message:
From: [email protected]
Peter's fix didn't make it up-stream.
Subject: sched: fix oops
sched_debug uses SEQ_printf to use printk when the seqfile 'm' is NULL. Instead of doing that here too; choose to not output the weight tree to cut back on output.
---
kernel/sched_fair.c +4 -2
@@ -1643,8 +1643,10 @@ static void print_cfs_stats(struct seq_file *m, int cpu)for_each_leaf_cfs_rq (cpu_rq (cpu), cfs_rq)print_cfs_rq (m, cpu, cfs_rq); -seq_printf (m,"\nWeight tree:\n" ); -print_cfs_rq_tasks (m, &cpu_rq (cpu)->cfs,1 ); +if (m) { +seq_printf (m,"\nWeight tree:\n" ); +print_cfs_rq_tasks (m, &cpu_rq (cpu)->cfs,1 ); + } rcu_read_unlock(); }#endif
Signature · 3 lines
/--------------- - - - - - -
| Dan Noe
| http://isomerica.net/~dpn/Quoted 12 lines · Dan Noe
Quoted 9 lines · Dave Young
[snip]
I have a fix for the NULL pointer reference BUG, tested on my pc.
---"m" will be NULL if
seq_printf&seq_putsis called fromsched_debug_showUse SEQ_* macros to fix itSigned-off-by Dave Young <[email protected]>
Quoted 4 lines
There is actually already a patch out for this which simply hasn't made it
upstream yet - Mike Galbraith emailed me with it and it indeed fixes the problem. I hadn't realized his email didn't go to lkml.. otherwise I would have echoed that here to avoid wasting people's time. My apologies.
Ok, thanks.
While debuging this issue I found another BUG about sysrq,
echo w >/proc/sys/kernel/sysrq
My system hangs, there's no response from then on.
Quoted 49 lines
Cheers,
Dan===
Here is his message:
From: [email protected]
Peter's fix didn't make it up-stream.
Subject: sched: fix oops
sched_debugusesSEQ_printfto use printk when the seqfile 'm' is NULL. Instead of doing that here too; choose to not output the weight tree to cut back on output.Signed-off-by Peter Zijlstra <[email protected]>---
diff --gita/kernel/sched_fair.cb/kernel/sched_fair.c
index 89fa32b..353a481 100644
---a/kernel/sched_fair.c
+++b/kernel/sched_fair.c
@@ -1643,8 +1643,10 @@ static void print_cfs_stats(structseq_file*m, int
cpu)for_each_leaf_cfs_rq(cpu_rq(cpu), cfs_rq) print_cfs_rq(m, cpu, cfs_rq);- seq_printf(m, "\nWeight tree:\n"); - print_cfs_rq_tasks(m, &cpu_rq(cpu)->cfs, 1); + if (m) {+
seq_printf(m, "\nWeight tree:\n");+
print_cfs_rq_tasks(m, &cpu_rq(cpu)->cfs, 1);
+ }rcu_read_unlock();}
#endif--
/--------------- - - - - - - | Dan Noe | http://isomerica.net/~dpn/
Quoted 18 lines · Dave Young
Quoted 12 lines · Dan Noe
Quoted 9 lines · Dave Young
[snip]
I have a fix for the NULL pointer reference BUG, tested on my pc.
---"m" will be NULL if
seq_printf&seq_putsis called fromsched_debug_showUse SEQ_* macros to fix itSigned-off-by Dave Young <[email protected]>Quoted 4 lines
There is actually already a patch out for this which simply hasn't made it
upstream yet - Mike Galbraith emailed me with it and it indeed fixes the problem. I hadn't realized his email didn't go to lkml.. otherwise I would have echoed that here to avoid wasting people's time. My apologies.
Quoted 6 lines
Ok, thanks.
While debuging this issue I found another BUG about sysrq,
echo w >/proc/sys/kernel/sysrq
My system hangs, there's no response from then on.
Strange, the bug can not be reproduced now. Now I only get the expected result "Invalid argument".