general protection fault: 0000 [#1] PREEMPT SMP

PATCH In review Sergey Senozhatsky 4 participants 8 messages Aug 13, 2010 – Aug 24, 2010

Subsystems (2)

Library code Per-CPU memory allocator

lore

Share & tools

lore

Tejun Heo

maintainer

2010-08-24 09:03 UTC #1 lore raw

Hello,

Quoted 1 line · Sergey Senozhatsky

[   29.943019]  [<ffffffff81065893>] lock_acquire+0x97/0xb6

Quoted 1 line

[   29.943257]  [<ffffffff8137568d>] _raw_spin_lock_irqsave+0x4e/0x60

Quoted 1 line

[   29.943376]  [<ffffffff8137145b>] percpu_counter_hotcpu_callback+0x3e/0x93

It's getting gpf at spin_lock_irqsave(&fbc->lock) in percpu_counter_hotplug_callback(). percpu_counter keeps track of all the allocated percpu counters and walk them on cpu up/down events. It's most likely one of its users freed or corrupted the percpu counter structure without properly destroying it. Adding debugobj is probably the best way to track down the offending user.

Thanks.

Signature · 1 line

tejun

Sergey Senozhatsky

author

2010-08-24 09:15 UTC #2 lore raw

Quoted 9 lines · Tejun Heo

Hello,

Quoted 1 line · Sergey Senozhatsky

[   29.943019]  [<ffffffff81065893>] lock_acquire+0x97/0xb6

Quoted 1 line

[   29.943257]  [<ffffffff8137568d>] _raw_spin_lock_irqsave+0x4e/0x60

Quoted 1 line

[   29.943376]  [<ffffffff8137145b>] percpu_counter_hotcpu_callback+0x3e/0x93

Quoted 12 lines

It's getting gpf at spin_lock_irqsave(&fbc->lock) in percpu_counter_hotplug_callback(). percpu_counter keeps track of all the allocated percpu counters and walk them on cpu up/down events. It's most likely one of its users freed or corrupted the percpu counter structure without properly destroying it. Adding debugobj is probably the best way to track down the offending user.
Thanks.
Signature · 1 line
tejun

Hello,

Can't reproduce it so far.

Sergey

Tejun Heo

maintainer

2010-08-24 10:17 UTC #3 lore raw

percpu_counter: add debugobj support

All percpu counters are linked to a global list on initialization and removed from it on destruction. The list is walked during CPU up/down. If a percpu counter is freed without being properly destroyed, the system will oops only on the next CPU up/down making it pretty nasty to track down. This patch adds debugobj support for percpu counters so that such problems can be found easily.

As percpu counters don't make sense on stack and can't be statically initialized, debugobj support is pretty simple. It's initialized and activated on counter initialization, and deactivatd and destroyed on counter destruction. With this patch applied, the bug fixed by commit 602586a83b719df0fbd94196a1359ed35aeb2df3 (shmem: put_super must percpu_counter_destroy) triggers the following warning on tmpfs unmount and the system won't oops on the next cpu up/down operation.

 ------------[ cut here ]------------
 WARNING: at /devel/tj/os/work/lib/debugobjects.c:259 debug_print_object+0x5c/0x70()
 Hardware name: Bochs
 ODEBUG: free active (active state 0) object type: percpu_counter
 Modules linked in:
 Pid: 3999, comm: umount Not tainted 2.6.36-rc2-work+ #5
 Call Trace:
  [<ffffffff81083f7f>] warn_slowpath_common+0x7f/0xc0
  [<ffffffff81084076>] warn_slowpath_fmt+0x46/0x50
  [<ffffffff813b45cc>] debug_print_object+0x5c/0x70
  [<ffffffff813b50e5>] debug_check_no_obj_freed+0x125/0x210
  [<ffffffff811577d3>] kfree+0xb3/0x2f0
  [<ffffffff81132edd>] shmem_put_super+0x1d/0x30
  [<ffffffff81162e96>] generic_shutdown_super+0x56/0xe0
  [<ffffffff81162f86>] kill_anon_super+0x16/0x60
  [<ffffffff81162ff7>] kill_litter_super+0x27/0x30
  [<ffffffff81163295>] deactivate_locked_super+0x45/0x60
  [<ffffffff81163cfa>] deactivate_super+0x4a/0x70
  [<ffffffff8117d446>] mntput_no_expire+0x86/0xe0
  [<ffffffff8117df7f>] sys_umount+0x6f/0x360
  [<ffffffff8103f01b>] system_call_fastpath+0x16/0x1b
 ---[ end trace cce2a341ba3611a7 ]---

Signed-off-by Tejun Heo <[email protected]>

---
Andrew, if there's no objection, can you please route this through your tree?

Thanks.

 lib/Kconfig.debug    |    8 ++++++++
 lib/percpu_counter.c |   48 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)

lib/Kconfig.debug +8 -0

index 1b4afd2..651d794 100644
@@ -317,6 +317,14 @@ config DEBUG_OBJECTS_RCU_HEAD
 	help
 	  Enable this to turn on debugging of RCU list heads (call_rcu() usage).
 
+config DEBUG_OBJECTS_PERCPU_COUNTER
+	bool "Debug percpu counter objects"
+	depends on DEBUG_OBJECTS
+	help
+	  If you say Y here, additional code will be inserted into the
+	  percpu counter routines to track the life time of percpu counter
+	  objects and validate the percpu counter operations.
+
 config DEBUG_OBJECTS_ENABLE_DEFAULT
 	int "debug_objects bootup default value (0-1)"
         range 0 1

lib/percpu_counter.c +48 -0

70 lines not on this page · raw message

Tejun Heo

maintainer

2010-08-24 10:18 UTC #4 lore raw

PATCH 1/1 percpu_counter: add debugobj support

In reply to Sergey Senozhatsky

 ------------[ cut here ]------------
 WARNING: at /devel/tj/os/work/lib/debugobjects.c:259 debug_print_object+0x5c/0x70()
 Hardware name: Bochs
 ODEBUG: free active (active state 0) object type: percpu_counter
 Modules linked in:
 Pid: 3999, comm: umount Not tainted 2.6.36-rc2-work+ #5
 Call Trace:
  [<ffffffff81083f7f>] warn_slowpath_common+0x7f/0xc0
  [<ffffffff81084076>] warn_slowpath_fmt+0x46/0x50
  [<ffffffff813b45cc>] debug_print_object+0x5c/0x70
  [<ffffffff813b50e5>] debug_check_no_obj_freed+0x125/0x210
  [<ffffffff811577d3>] kfree+0xb3/0x2f0
  [<ffffffff81132edd>] shmem_put_super+0x1d/0x30
  [<ffffffff81162e96>] generic_shutdown_super+0x56/0xe0
  [<ffffffff81162f86>] kill_anon_super+0x16/0x60
  [<ffffffff81162ff7>] kill_litter_super+0x27/0x30
  [<ffffffff81163295>] deactivate_locked_super+0x45/0x60
  [<ffffffff81163cfa>] deactivate_super+0x4a/0x70
  [<ffffffff8117d446>] mntput_no_expire+0x86/0xe0
  [<ffffffff8117df7f>] sys_umount+0x6f/0x360
  [<ffffffff8103f01b>] system_call_fastpath+0x16/0x1b
 ---[ end trace cce2a341ba3611a7 ]---

Signed-off-by Tejun Heo <[email protected]>

---
Andrew, if there's no objection, can you please route this through your tree?

Oops, reposted with [PATCH] prefix.

Thanks.

 lib/Kconfig.debug    |    8 ++++++++
 lib/percpu_counter.c |   48 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)

lib/Kconfig.debug +8 -0

index 1b4afd2..651d794 100644
@@ -317,6 +317,14 @@ config DEBUG_OBJECTS_RCU_HEAD
 	help
 	  Enable this to turn on debugging of RCU list heads (call_rcu() usage).
 
+config DEBUG_OBJECTS_PERCPU_COUNTER
+	bool "Debug percpu counter objects"
+	depends on DEBUG_OBJECTS
+	help
+	  If you say Y here, additional code will be inserted into the
+	  percpu counter routines to track the life time of percpu counter
+	  objects and validate the percpu counter operations.
+
 config DEBUG_OBJECTS_ENABLE_DEFAULT
 	int "debug_objects bootup default value (0-1)"
         range 0 1

lib/percpu_counter.c +48 -0

70 lines not on this page · raw message

Thomas Gleixner

Acked-by on v1 1/1

2010-08-24 13:27 UTC #5 lore raw

Re: [PATCH REPOST 2.6.36-rc2] percpu_counter: add debugobj support

Quoted 38 lines · Tejun Heo

All percpu counters are linked to a global list on initialization and removed from it on destruction. The list is walked during CPU up/down. If a percpu counter is freed without being properly destroyed, the system will oops only on the next CPU up/down making it pretty nasty to track down. This patch adds debugobj support for percpu counters so that such problems can be found easily.
As percpu counters don't make sense on stack and can't be statically initialized, debugobj support is pretty simple. It's initialized and activated on counter initialization, and deactivatd and destroyed on counter destruction. With this patch applied, the bug fixed by commit 602586a83b719df0fbd94196a1359ed35aeb2df3 (shmem: put_super must percpu_counter_destroy) triggers the following warning on tmpfs unmount and the system won't oops on the next cpu up/down operation.
 ------------[ cut here ]------------
 WARNING: at /devel/tj/os/work/lib/debugobjects.c:259 debug_print_object+0x5c/0x70()
 Hardware name: Bochs
 ODEBUG: free active (active state 0) object type: percpu_counter
 Modules linked in:
 Pid: 3999, comm: umount Not tainted 2.6.36-rc2-work+ #5
 Call Trace:
  [<ffffffff81083f7f>] warn_slowpath_common+0x7f/0xc0
  [<ffffffff81084076>] warn_slowpath_fmt+0x46/0x50
  [<ffffffff813b45cc>] debug_print_object+0x5c/0x70
  [<ffffffff813b50e5>] debug_check_no_obj_freed+0x125/0x210
  [<ffffffff811577d3>] kfree+0xb3/0x2f0
  [<ffffffff81132edd>] shmem_put_super+0x1d/0x30
  [<ffffffff81162e96>] generic_shutdown_super+0x56/0xe0
  [<ffffffff81162f86>] kill_anon_super+0x16/0x60
  [<ffffffff81162ff7>] kill_litter_super+0x27/0x30
  [<ffffffff81163295>] deactivate_locked_super+0x45/0x60
  [<ffffffff81163cfa>] deactivate_super+0x4a/0x70
  [<ffffffff8117d446>] mntput_no_expire+0x86/0xe0
  [<ffffffff8117df7f>] sys_umount+0x6f/0x360
  [<ffffffff8103f01b>] system_call_fastpath+0x16/0x1b
 ---[ end trace cce2a341ba3611a7 ]---

Quoted 1 line

Signed-off-by Tejun Heo <[email protected]>

Acked-by Thomas Gleixner <tglxlinutronix.de>

Yong Zhang

2010-08-24 09:08 UTC #6 lore raw

In reply to Sergey Senozhatsky

Quoted 94 lines · Sergey Senozhatsky

Hello,
Got this traces today:
[ 29.940248] CPU 1 is now offline
[ 29.941025] general protection fault: 0000 [#1] PREEMPT SMP
[ 29.941103] last sysfs file: /sys/devices/system/cpu/cpu1/online
[ 29.941157] CPU 0
[ 29.941178] Modules linked in: snd_hwdep snd_hda_codec_atihdmi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device battery ac wmi snd_pcm_oss snd_mixer_oss button snd_hda_codec_realtek radeon broadcom snd_hda_intel snd_hda_codec snd_pcm snd_timer snd soundcore snd_page_alloc usbhid hid tg3 libphy psmouse serio_raw evdev ttm drm_kms_helper ehci_hcd sr_mod usbcore cdrom sd_mod ahci libahci
[ 29.941679]
[ 29.941699] Pid: 5208, comm: bash Not tainted 2.6.36-rc0-git12-07921-g60bf26a-dirty #124 Aspire 5741G /Aspire 5741G
[ 29.941792] RIP: 0010:[<ffffffff810640a1>] [<ffffffff810640a1>] __lock_acquire+0x4e9/0x17fd
[ 29.941878] RSP: 0018:ffff88015751dbc8 EFLAGS: 00010082
[ 29.941926] RAX: 0000000000000001 RBX: ffff880152284920 RCX: 0000000000000000
[ 29.941988] RDX: dead4ead00000202 RSI: 0000000000000000 RDI: ffff880152284920
[ 29.942049] RBP: ffff88015751dca8 R08: 0000000000000002 R09: 0000000000000001
[ 29.942111] R10: 0000000000000000 R11: 0000000000000005 R12: ffff8801504c3f60
[ 29.942171] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000002
[ 29.942233] FS: 00007f53580ca700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
[ 29.942303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 29.942353] CR2: 00007fb030079000 CR3: 0000000151b5a000 CR4: 00000000000006f0
[ 29.942414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 29.942475] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 29.942536] Process bash (pid: 5208, threadinfo ffff88015751c000, task ffff8801504c3f60)
[ 29.942604] Stack:
[ 29.942625] ffff8801569a4888 ffff88015751dc48 ffff88015751dcb8 ffffffff81132b2a
[ 29.942702] <0> ffff8801504c3f60 0000000000000001 ffff880100000000 ffffffff8186c0a0
[ 29.942789] <0> ffffffff00000000 0000000000000004 ffff8801504c3f60 ffff8801504c3f60
[ 29.942880] Call Trace:
[ 29.942910] [<ffffffff81132b2a>] ? sysfs_deactivate+0x3e/0xec
[ 29.942966] [<ffffffff81062ddd>] ? mark_held_locks+0x50/0x72
[ 29.943019] [<ffffffff81065893>] lock_acquire+0x97/0xb6
[ 29.943072] [<ffffffff8137145b>] ? percpu_counter_hotcpu_callback+0x3e/0x93
[ 29.943136] [<ffffffff81374321>] ? mutex_lock_nested+0x2f3/0x31b
[ 29.943192] [<ffffffff81371446>] ? percpu_counter_hotcpu_callback+0x29/0x93
[ 29.943257] [<ffffffff8137568d>] _raw_spin_lock_irqsave+0x4e/0x60
[ 29.943313] [<ffffffff8137145b>] ? percpu_counter_hotcpu_callback+0x3e/0x93
[ 29.943376] [<ffffffff8137145b>] percpu_counter_hotcpu_callback+0x3e/0x93
[ 29.943441] [<ffffffff81057344>] notifier_call_chain+0x32/0x5e
[ 29.943494] [<ffffffff8105738f>] __raw_notifier_call_chain+0x9/0xb
[ 29.943552] [<ffffffff8103c6e3>] __cpu_notify+0x1b/0x2d
[ 29.943602] [<ffffffff8103c703>] cpu_notify+0xe/0x10
[ 29.943649] [<ffffffff8103c70e>] cpu_notify_nofail+0x9/0x11
[ 29.943703] [<ffffffff81362d82>] _cpu_down+0x151/0x206
[ 29.943751] [<ffffffff81362ea8>] cpu_down+0x28/0x35
[ 29.943798] [<ffffffff8136430d>] store_online+0x27/0x6e
[ 29.943850] [<ffffffff812923ab>] sysdev_store+0x1b/0x1d
[ 29.943899] [<ffffffff811321b2>] sysfs_write_file+0x103/0x13f
[ 29.943955] [<ffffffff810daf92>] vfs_write+0xb0/0x14f
[ 29.944003] [<ffffffff810db22e>] sys_write+0x45/0x6c
[ 29.944054] [<ffffffff81002002>] system_call_fastpath+0x16/0x1b
[ 29.946132] Code: 85 c0 0f 84 a4 12 00 00 be 0b 03 00 00 83 3d 5e c4 f5 00 00 0f 85 92 12 00 00 e9 a4 11 00 00 45 31 f6 48 85 d2 0f 84 81 12 00 00 <f0> ff 82 98 01 00 00 45 8b 84 24 20 07 00 00 83 3d 79 ce 6a 00 [ 29.951109] RIP [<ffffffff810640a1>] __lock_acquire+0x4e9/0x17fd
[ 29.953421] RSP <ffff88015751dbc8>
[ 29.965605] ---[ end trace 34832156140843b2 ]---
[ 29.967758] note: bash[5208] exited with preempt_count 1
[ 29.969997] BUG: scheduling while atomic: bash/5208/0x10000002
[ 29.972098] INFO: lockdep is turned off.
[ 29.974167] Modules linked in: snd_hwdep snd_hda_codec_atihdmi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device battery ac wmi snd_pcm_oss snd_mixer_oss button snd_hda_codec_realtek radeon broadcom snd_hda_intel snd_hda_codec snd_pcm snd_timer snd soundcore snd_page_alloc usbhid hid tg3 libphy psmouse serio_raw evdev ttm drm_kms_helper ehci_hcd sr_mod usbcore cdrom sd_mod ahci libahci
[ 29.981076] Pid: 5208, comm: bash Tainted: G D 2.6.36-rc0-git12-07921-g60bf26a-dirty #124
[ 29.983375] Call Trace:
[ 29.985672] [<ffffffff8102dd6a>] __schedule_bug+0x72/0x77
[ 29.987996] [<ffffffff81372790>] schedule+0xdc/0x8f2
[ 29.990342] [<ffffffff810360f9>] __cond_resched+0x13/0x1f
[ 29.992607] [<ffffffff813730c5>] _cond_resched+0x29/0x30
[ 29.994912] [<ffffffff810bbe97>] unmap_vmas+0x747/0x92b
[ 29.997198] [<ffffffff810c15cd>] exit_mmap+0xda/0x184
[ 29.999410] [<ffffffff8103888e>] mmput+0x28/0xcf
[ 30.001623] [<ffffffff8103cd21>] exit_mm+0x109/0x116
[ 30.003791] [<ffffffff81375db0>] ? _raw_spin_unlock_irq+0x55/0x59
[ 30.005931] [<ffffffff8103e387>] do_exit+0x1fe/0x6c0
[ 30.008070] [<ffffffff8103c6a5>] ? kmsg_dump+0x14f/0x16a
[ 30.010205] [<ffffffff810060c4>] oops_end+0x8f/0x94
[ 30.012285] [<ffffffff81006206>] die+0x55/0x5e
[ 30.014369] [<ffffffff8100371c>] do_general_protection+0x135/0x13d
[ 30.016453] [<ffffffff81376304>] ? irq_return+0x0/0xc
[ 30.018546] [<ffffffff813764e5>] general_protection+0x25/0x30
[ 30.020643] [<ffffffff810640a1>] ? __lock_acquire+0x4e9/0x17fd
[ 30.022719] [<ffffffff81132b2a>] ? sysfs_deactivate+0x3e/0xec
[ 30.024804] [<ffffffff81062ddd>] ? mark_held_locks+0x50/0x72
[ 30.026883] [<ffffffff81065893>] lock_acquire+0x97/0xb6
[ 30.028932] [<ffffffff8137145b>] ? percpu_counter_hotcpu_callback+0x3e/0x93
[ 30.031017] [<ffffffff81374321>] ? mutex_lock_nested+0x2f3/0x31b
[ 30.033076] [<ffffffff81371446>] ? percpu_counter_hotcpu_callback+0x29/0x93
[ 30.035171] [<ffffffff8137568d>] _raw_spin_lock_irqsave+0x4e/0x60
[ 30.037272] [<ffffffff8137145b>] ? percpu_counter_hotcpu_callback+0x3e/0x93
[ 30.039372] [<ffffffff8137145b>] percpu_counter_hotcpu_callback+0x3e/0x93
[ 30.041515] [<ffffffff81057344>] notifier_call_chain+0x32/0x5e
[ 30.043653] [<ffffffff8105738f>] __raw_notifier_call_chain+0x9/0xb
[ 30.045777] [<ffffffff8103c6e3>] __cpu_notify+0x1b/0x2d
[ 30.047927] [<ffffffff8103c703>] cpu_notify+0xe/0x10
[ 30.050049] [<ffffffff8103c70e>] cpu_notify_nofail+0x9/0x11

Quoted 8 lines

[ 30.052172] [<ffffffff81362d82>] _cpu_down+0x151/0x206 [ 30.054308] [<ffffffff81362ea8>] cpu_down+0x28/0x35 [ 30.056413] [<ffffffff8136430d>] store_online+0x27/0x6e [ 30.058546] [<ffffffff812923ab>] sysdev_store+0x1b/0x1d [ 30.060537] [<ffffffff811321b2>] sysfs_write_file+0x103/0x13f [ 30.062357] [<ffffffff810daf92>] vfs_write+0xb0/0x14f [ 30.064167] [<ffffffff810db22e>] sys_write+0x45/0x6c [ 30.065943] [<ffffffff81002002>] system_call_fastpath+0x16/0x1b

Seems this is resolved by commit 602586a8, can you try it?

Thanks,
Yong

Sergey Senozhatsky

author

2010-08-24 09:17 UTC #7 lore raw

Quoted 103 lines · Yong Zhang

Quoted 94 lines
Hello,
Got this traces today:
[ 29.940248] CPU 1 is now offline
[ 29.941025] general protection fault: 0000 [#1] PREEMPT SMP
[ 29.941103] last sysfs file: /sys/devices/system/cpu/cpu1/online
[ 29.941157] CPU 0
[ 29.941178] Modules linked in: snd_hwdep snd_hda_codec_atihdmi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device battery ac wmi snd_pcm_oss snd_mixer_oss button snd_hda_codec_realtek radeon broadcom snd_hda_intel snd_hda_codec snd_pcm snd_timer snd soundcore snd_page_alloc usbhid hid tg3 libphy psmouse serio_raw evdev ttm drm_kms_helper ehci_hcd sr_mod usbcore cdrom sd_mod ahci libahci
[ 29.941679]
[ 29.941699] Pid: 5208, comm: bash Not tainted 2.6.36-rc0-git12-07921-g60bf26a-dirty #124 Aspire 5741G /Aspire 5741G
[ 29.941792] RIP: 0010:[<ffffffff810640a1>] [<ffffffff810640a1>] __lock_acquire+0x4e9/0x17fd
[ 29.941878] RSP: 0018:ffff88015751dbc8 EFLAGS: 00010082
[ 29.941926] RAX: 0000000000000001 RBX: ffff880152284920 RCX: 0000000000000000
[ 29.941988] RDX: dead4ead00000202 RSI: 0000000000000000 RDI: ffff880152284920
[ 29.942049] RBP: ffff88015751dca8 R08: 0000000000000002 R09: 0000000000000001
[ 29.942111] R10: 0000000000000000 R11: 0000000000000005 R12: ffff8801504c3f60
[ 29.942171] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000002
[ 29.942233] FS: 00007f53580ca700(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
[ 29.942303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 29.942353] CR2: 00007fb030079000 CR3: 0000000151b5a000 CR4: 00000000000006f0
[ 29.942414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 29.942475] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 29.942536] Process bash (pid: 5208, threadinfo ffff88015751c000, task ffff8801504c3f60)
[ 29.942604] Stack:
[ 29.942625] ffff8801569a4888 ffff88015751dc48 ffff88015751dcb8 ffffffff81132b2a
[ 29.942702] <0> ffff8801504c3f60 0000000000000001 ffff880100000000 ffffffff8186c0a0
[ 29.942789] <0> ffffffff00000000 0000000000000004 ffff8801504c3f60 ffff8801504c3f60
[ 29.942880] Call Trace:
[ 29.942910] [<ffffffff81132b2a>] ? sysfs_deactivate+0x3e/0xec
[ 29.942966] [<ffffffff81062ddd>] ? mark_held_locks+0x50/0x72
[ 29.943019] [<ffffffff81065893>] lock_acquire+0x97/0xb6
[ 29.943072] [<ffffffff8137145b>] ? percpu_counter_hotcpu_callback+0x3e/0x93
[ 29.943136] [<ffffffff81374321>] ? mutex_lock_nested+0x2f3/0x31b
[ 29.943192] [<ffffffff81371446>] ? percpu_counter_hotcpu_callback+0x29/0x93
[ 29.943257] [<ffffffff8137568d>] _raw_spin_lock_irqsave+0x4e/0x60
[ 29.943313] [<ffffffff8137145b>] ? percpu_counter_hotcpu_callback+0x3e/0x93
[ 29.943376] [<ffffffff8137145b>] percpu_counter_hotcpu_callback+0x3e/0x93
[ 29.943441] [<ffffffff81057344>] notifier_call_chain+0x32/0x5e
[ 29.943494] [<ffffffff8105738f>] __raw_notifier_call_chain+0x9/0xb
[ 29.943552] [<ffffffff8103c6e3>] __cpu_notify+0x1b/0x2d
[ 29.943602] [<ffffffff8103c703>] cpu_notify+0xe/0x10
[ 29.943649] [<ffffffff8103c70e>] cpu_notify_nofail+0x9/0x11
[ 29.943703] [<ffffffff81362d82>] _cpu_down+0x151/0x206
[ 29.943751] [<ffffffff81362ea8>] cpu_down+0x28/0x35
[ 29.943798] [<ffffffff8136430d>] store_online+0x27/0x6e
[ 29.943850] [<ffffffff812923ab>] sysdev_store+0x1b/0x1d
[ 29.943899] [<ffffffff811321b2>] sysfs_write_file+0x103/0x13f
[ 29.943955] [<ffffffff810daf92>] vfs_write+0xb0/0x14f
[ 29.944003] [<ffffffff810db22e>] sys_write+0x45/0x6c
[ 29.944054] [<ffffffff81002002>] system_call_fastpath+0x16/0x1b
[ 29.946132] Code: 85 c0 0f 84 a4 12 00 00 be 0b 03 00 00 83 3d 5e c4 f5 00 00 0f 85 92 12 00 00 e9 a4 11 00 00 45 31 f6 48 85 d2 0f 84 81 12 00 00 <f0> ff 82 98 01 00 00 45 8b 84 24 20 07 00 00 83 3d 79 ce 6a 00 [ 29.951109] RIP [<ffffffff810640a1>] __lock_acquire+0x4e9/0x17fd
[ 29.953421] RSP <ffff88015751dbc8>
[ 29.965605] ---[ end trace 34832156140843b2 ]---
[ 29.967758] note: bash[5208] exited with preempt_count 1
[ 29.969997] BUG: scheduling while atomic: bash/5208/0x10000002
[ 29.972098] INFO: lockdep is turned off.
[ 29.974167] Modules linked in: snd_hwdep snd_hda_codec_atihdmi snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device battery ac wmi snd_pcm_oss snd_mixer_oss button snd_hda_codec_realtek radeon broadcom snd_hda_intel snd_hda_codec snd_pcm snd_timer snd soundcore snd_page_alloc usbhid hid tg3 libphy psmouse serio_raw evdev ttm drm_kms_helper ehci_hcd sr_mod usbcore cdrom sd_mod ahci libahci
[ 29.981076] Pid: 5208, comm: bash Tainted: G D 2.6.36-rc0-git12-07921-g60bf26a-dirty #124
[ 29.983375] Call Trace:
[ 29.985672] [<ffffffff8102dd6a>] __schedule_bug+0x72/0x77
[ 29.987996] [<ffffffff81372790>] schedule+0xdc/0x8f2
[ 29.990342] [<ffffffff810360f9>] __cond_resched+0x13/0x1f
[ 29.992607] [<ffffffff813730c5>] _cond_resched+0x29/0x30
[ 29.994912] [<ffffffff810bbe97>] unmap_vmas+0x747/0x92b
[ 29.997198] [<ffffffff810c15cd>] exit_mmap+0xda/0x184
[ 29.999410] [<ffffffff8103888e>] mmput+0x28/0xcf
[ 30.001623] [<ffffffff8103cd21>] exit_mm+0x109/0x116
[ 30.003791] [<ffffffff81375db0>] ? _raw_spin_unlock_irq+0x55/0x59
[ 30.005931] [<ffffffff8103e387>] do_exit+0x1fe/0x6c0
[ 30.008070] [<ffffffff8103c6a5>] ? kmsg_dump+0x14f/0x16a
[ 30.010205] [<ffffffff810060c4>] oops_end+0x8f/0x94
[ 30.012285] [<ffffffff81006206>] die+0x55/0x5e
[ 30.014369] [<ffffffff8100371c>] do_general_protection+0x135/0x13d
[ 30.016453] [<ffffffff81376304>] ? irq_return+0x0/0xc
[ 30.018546] [<ffffffff813764e5>] general_protection+0x25/0x30
[ 30.020643] [<ffffffff810640a1>] ? __lock_acquire+0x4e9/0x17fd
[ 30.022719] [<ffffffff81132b2a>] ? sysfs_deactivate+0x3e/0xec
[ 30.024804] [<ffffffff81062ddd>] ? mark_held_locks+0x50/0x72
[ 30.026883] [<ffffffff81065893>] lock_acquire+0x97/0xb6
[ 30.028932] [<ffffffff8137145b>] ? percpu_counter_hotcpu_callback+0x3e/0x93
[ 30.031017] [<ffffffff81374321>] ? mutex_lock_nested+0x2f3/0x31b
[ 30.033076] [<ffffffff81371446>] ? percpu_counter_hotcpu_callback+0x29/0x93
[ 30.035171] [<ffffffff8137568d>] _raw_spin_lock_irqsave+0x4e/0x60
[ 30.037272] [<ffffffff8137145b>] ? percpu_counter_hotcpu_callback+0x3e/0x93
[ 30.039372] [<ffffffff8137145b>] percpu_counter_hotcpu_callback+0x3e/0x93
[ 30.041515] [<ffffffff81057344>] notifier_call_chain+0x32/0x5e
[ 30.043653] [<ffffffff8105738f>] __raw_notifier_call_chain+0x9/0xb
[ 30.045777] [<ffffffff8103c6e3>] __cpu_notify+0x1b/0x2d
[ 30.047927] [<ffffffff8103c703>] cpu_notify+0xe/0x10
[ 30.050049] [<ffffffff8103c70e>] cpu_notify_nofail+0x9/0x11
Quoted 8 lines
[ 30.052172] [<ffffffff81362d82>] _cpu_down+0x151/0x206 [ 30.054308] [<ffffffff81362ea8>] cpu_down+0x28/0x35 [ 30.056413] [<ffffffff8136430d>] store_online+0x27/0x6e [ 30.058546] [<ffffffff812923ab>] sysdev_store+0x1b/0x1d [ 30.060537] [<ffffffff811321b2>] sysfs_write_file+0x103/0x13f [ 30.062357] [<ffffffff810daf92>] vfs_write+0xb0/0x14f [ 30.064167] [<ffffffff810db22e>] sys_write+0x45/0x6c [ 30.065943] [<ffffffff81002002>] system_call_fastpath+0x16/0x1b

Quoted 5 lines

Seems this is resolved by commit 602586a8, can you try it?
Thanks,
Yong

commit 602586a83b719df0fbd94196a1359ed35aeb2df3
Author: Hugh Dickins <[email protected]>
Date: Tue Aug 17 15:23:56 2010 -0700
shmem: put_super must percpu_counter_destroy

Already compiled. Seems to be fixed.

Sergey