WARNING: at kernel/lockdep.c:690 __lock_acquire+0x168/0x164b()
Hello,
3.1-rc9
[10172.218213] ------------[ cut here ]------------ [10172.218233] WARNING: at kernel/lockdep.c:690 __lock_acquire+0x168/0x164b() [10172.218242] Hardware name: Aspire 5741G [10172.218248] Modules linked in: ipv6 usb_storage uas microcode snd_hda_codec_hdmi snd_hda_codec_realtek broadcom tg3 snd_hda_intel snd_hda_codec snd_pcm snd_timer snd rndis_host cdc_ether usbnet evdev psmouse soundcore pcspkr mii snd_page_alloc libphy ac battery wmi button ehci_hcd sr_mod cdrom usbcore sd_mod ahci [10172.218330] Pid: 22953, comm: kworker/0:2 Not tainted 3.1.0-rc9-dbg-00681-gec325b2 #730 [10172.218335] Call Trace: [10172.218346] [<ffffffff8103e7c8>] warn_slowpath_common+0x7e/0x96 [10172.218353] [<ffffffff8103e7f5>] warn_slowpath_null+0x15/0x17 [10172.218361] [<ffffffff8106fee5>] __lock_acquire+0x168/0x164b [10172.218370] [<ffffffff81034645>] ? find_busiest_group+0x7b6/0x941 [10172.218381] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218389] [<ffffffff8107197e>] lock_acquire+0x138/0x1ac [10172.218397] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218404] [<ffffffff8102a5c4>] ? double_rq_lock+0x2e/0x52 [10172.218414] [<ffffffff8148fb49>] _raw_spin_lock_nested+0x3a/0x49 [10172.218421] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218428] [<ffffffff8148fabe>] ? _raw_spin_lock+0x3e/0x45 [10172.218435] [<ffffffff8102a5c4>] ? double_rq_lock+0x2e/0x52 [10172.218442] [<ffffffff8102a5e3>] double_rq_lock+0x4d/0x52 [10172.218449] [<ffffffff810349cc>] load_balance+0x1fc/0x769 [10172.218458] [<ffffffff810075c5>] ? native_sched_clock+0x38/0x65 [10172.218466] [<ffffffff8148ca17>] ? __schedule+0x2f5/0xa2d [10172.218474] [<ffffffff8148caf5>] __schedule+0x3d3/0xa2d [10172.218480] [<ffffffff8148ca17>] ? __schedule+0x2f5/0xa2d [10172.218490] [<ffffffff8104db06>] ? add_timer_on+0xd/0x196 [10172.218497] [<ffffffff8148fc02>] ? _raw_spin_lock_irq+0x4a/0x51 [10172.218505] [<ffffffff8105907b>] ? process_one_work+0x3ed/0x54c [10172.218512] [<ffffffff81059126>] ? process_one_work+0x498/0x54c [10172.218518] [<ffffffff81058e1b>] ? process_one_work+0x18d/0x54c [10172.218526] [<ffffffff814902d0>] ? _raw_spin_unlock_irq+0x28/0x56 [10172.218533] [<ffffffff81033950>] ? get_parent_ip+0xe/0x3e [10172.218540] [<ffffffff8148d26e>] schedule+0x55/0x57 [10172.218547] [<ffffffff8105970f>] worker_thread+0x217/0x21c [10172.218554] [<ffffffff810594f8>] ? manage_workers.isra.21+0x16c/0x16c [10172.218564] [<ffffffff8105d4de>] kthread+0x9a/0xa2 [10172.218573] [<ffffffff81497984>] kernel_thread_helper+0x4/0x10 [10172.218580] [<ffffffff8102d6d2>] ? finish_task_switch+0x76/0xf3 [10172.218587] [<ffffffff81490778>] ? retint_restore_args+0x13/0x13 [10172.218595] [<ffffffff8105d444>] ? __init_kthread_worker+0x53/0x53 [10172.218602] [<ffffffff81497980>] ? gs_change+0x13/0x13 [10172.218607] ---[ end trace 9d11d6b5e4b96730 ]---
Really not sure what's going on and how to reproduce.
Sergey
Quoted 34 lines · Sergey Senozhatsky
[10172.218213] ------------[ cut here ]------------ [10172.218233] WARNING: at kernel/lockdep.c:690 __lock_acquire+0x168/0x164b() [10172.218242] Hardware name: Aspire 5741G [10172.218248] Modules linked in: ipv6 usb_storage uas microcode snd_hda_codec_hdmi snd_hda_codec_realtek broadcom tg3 snd_hda_intel snd_hda_codec snd_pcm snd_timer snd rndis_host cdc_ether usbnet evdev psmouse soundcore pcspkr mii snd_page_alloc libphy ac battery wmi button ehci_hcd sr_mod cdrom usbcore sd_mod ahci [10172.218330] Pid: 22953, comm: kworker/0:2 Not tainted 3.1.0-rc9-dbg-00681-gec325b2 #730 [10172.218335] Call Trace: [10172.218346] [<ffffffff8103e7c8>] warn_slowpath_common+0x7e/0x96 [10172.218353] [<ffffffff8103e7f5>] warn_slowpath_null+0x15/0x17 [10172.218361] [<ffffffff8106fee5>] __lock_acquire+0x168/0x164b [10172.218370] [<ffffffff81034645>] ? find_busiest_group+0x7b6/0x941 [10172.218381] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218389] [<ffffffff8107197e>] lock_acquire+0x138/0x1ac [10172.218397] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218404] [<ffffffff8102a5c4>] ? double_rq_lock+0x2e/0x52 [10172.218414] [<ffffffff8148fb49>] _raw_spin_lock_nested+0x3a/0x49 [10172.218421] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218428] [<ffffffff8148fabe>] ? _raw_spin_lock+0x3e/0x45 [10172.218435] [<ffffffff8102a5c4>] ? double_rq_lock+0x2e/0x52 [10172.218442] [<ffffffff8102a5e3>] double_rq_lock+0x4d/0x52 [10172.218449] [<ffffffff810349cc>] load_balance+0x1fc/0x769 [10172.218458] [<ffffffff810075c5>] ? native_sched_clock+0x38/0x65 [10172.218466] [<ffffffff8148ca17>] ? __schedule+0x2f5/0xa2d [10172.218474] [<ffffffff8148caf5>] __schedule+0x3d3/0xa2d [10172.218480] [<ffffffff8148ca17>] ? __schedule+0x2f5/0xa2d [10172.218490] [<ffffffff8104db06>] ? add_timer_on+0xd/0x196 [10172.218497] [<ffffffff8148fc02>] ? _raw_spin_lock_irq+0x4a/0x51 [10172.218505] [<ffffffff8105907b>] ? process_one_work+0x3ed/0x54c [10172.218512] [<ffffffff81059126>] ? process_one_work+0x498/0x54c [10172.218518] [<ffffffff81058e1b>] ? process_one_work+0x18d/0x54c [10172.218526] [<ffffffff814902d0>] ? _raw_spin_unlock_irq+0x28/0x56 [10172.218533] [<ffffffff81033950>] ? get_parent_ip+0xe/0x3e [10172.218540] [<ffffffff8148d26e>] schedule+0x55/0x57 [10172.218547] [<ffffffff8105970f>] worker_thread+0x217/0x21c
Quoted 8 lines
[10172.218554] [<ffffffff810594f8>] ? manage_workers.isra.21+0x16c/0x16c [10172.218564] [<ffffffff8105d4de>] kthread+0x9a/0xa2 [10172.218573] [<ffffffff81497984>] kernel_thread_helper+0x4/0x10 [10172.218580] [<ffffffff8102d6d2>] ? finish_task_switch+0x76/0xf3 [10172.218587] [<ffffffff81490778>] ? retint_restore_args+0x13/0x13 [10172.218595] [<ffffffff8105d444>] ? __init_kthread_worker+0x53/0x53 [10172.218602] [<ffffffff81497980>] ? gs_change+0x13/0x13 [10172.218607] ---[ end trace 9d11d6b5e4b96730 ]---
I think this is a problem with lockdep itself, could you try reverting f59de8992aa6 ("lockdep: Clear whole lockdep_map on initialization") if this reliably happens everytime you reboot (lockdep will only emit this once and then will suppress future warnings until the next boot)?
I think the new memset() is inadvertently clearing the name for double_unlock_balance().
Quoted 45 lines · David Rientjes
Quoted 34 lines · Sergey Senozhatsky
[10172.218213] ------------[ cut here ]------------ [10172.218233] WARNING: at kernel/lockdep.c:690 __lock_acquire+0x168/0x164b() [10172.218242] Hardware name: Aspire 5741G [10172.218248] Modules linked in: ipv6 usb_storage uas microcode snd_hda_codec_hdmi snd_hda_codec_realtek broadcom tg3 snd_hda_intel snd_hda_codec snd_pcm snd_timer snd rndis_host cdc_ether usbnet evdev psmouse soundcore pcspkr mii snd_page_alloc libphy ac battery wmi button ehci_hcd sr_mod cdrom usbcore sd_mod ahci [10172.218330] Pid: 22953, comm: kworker/0:2 Not tainted 3.1.0-rc9-dbg-00681-gec325b2 #730 [10172.218335] Call Trace: [10172.218346] [<ffffffff8103e7c8>] warn_slowpath_common+0x7e/0x96 [10172.218353] [<ffffffff8103e7f5>] warn_slowpath_null+0x15/0x17 [10172.218361] [<ffffffff8106fee5>] __lock_acquire+0x168/0x164b [10172.218370] [<ffffffff81034645>] ? find_busiest_group+0x7b6/0x941 [10172.218381] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218389] [<ffffffff8107197e>] lock_acquire+0x138/0x1ac [10172.218397] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218404] [<ffffffff8102a5c4>] ? double_rq_lock+0x2e/0x52 [10172.218414] [<ffffffff8148fb49>] _raw_spin_lock_nested+0x3a/0x49 [10172.218421] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218428] [<ffffffff8148fabe>] ? _raw_spin_lock+0x3e/0x45 [10172.218435] [<ffffffff8102a5c4>] ? double_rq_lock+0x2e/0x52 [10172.218442] [<ffffffff8102a5e3>] double_rq_lock+0x4d/0x52 [10172.218449] [<ffffffff810349cc>] load_balance+0x1fc/0x769 [10172.218458] [<ffffffff810075c5>] ? native_sched_clock+0x38/0x65 [10172.218466] [<ffffffff8148ca17>] ? __schedule+0x2f5/0xa2d [10172.218474] [<ffffffff8148caf5>] __schedule+0x3d3/0xa2d [10172.218480] [<ffffffff8148ca17>] ? __schedule+0x2f5/0xa2d [10172.218490] [<ffffffff8104db06>] ? add_timer_on+0xd/0x196 [10172.218497] [<ffffffff8148fc02>] ? _raw_spin_lock_irq+0x4a/0x51 [10172.218505] [<ffffffff8105907b>] ? process_one_work+0x3ed/0x54c [10172.218512] [<ffffffff81059126>] ? process_one_work+0x498/0x54c [10172.218518] [<ffffffff81058e1b>] ? process_one_work+0x18d/0x54c [10172.218526] [<ffffffff814902d0>] ? _raw_spin_unlock_irq+0x28/0x56 [10172.218533] [<ffffffff81033950>] ? get_parent_ip+0xe/0x3e [10172.218540] [<ffffffff8148d26e>] schedule+0x55/0x57 [10172.218547] [<ffffffff8105970f>] worker_thread+0x217/0x21cQuoted 8 lines
[10172.218554] [<ffffffff810594f8>] ? manage_workers.isra.21+0x16c/0x16c [10172.218564] [<ffffffff8105d4de>] kthread+0x9a/0xa2 [10172.218573] [<ffffffff81497984>] kernel_thread_helper+0x4/0x10 [10172.218580] [<ffffffff8102d6d2>] ? finish_task_switch+0x76/0xf3 [10172.218587] [<ffffffff81490778>] ? retint_restore_args+0x13/0x13 [10172.218595] [<ffffffff8105d444>] ? __init_kthread_worker+0x53/0x53 [10172.218602] [<ffffffff81497980>] ? gs_change+0x13/0x13 [10172.218607] ---[ end trace 9d11d6b5e4b96730 ]---
Quoted 7 lines
I think this is a problem with lockdep itself, could you try reverting
f59de8992aa6("lockdep: Clear wholelockdep_mapon initialization") if this reliably happens everytime you reboot (lockdep will only emit this once and then will suppress future warnings until the next boot)?I think the new
memset()is inadvertently clearing the name fordouble_unlock_balance().
Great,
so I'm not the only one seeing the above:
http://marc.info/?l=linux-kernel&m=131468805610527
Due to it being very hard to reproduce, we dismissed it then as a possible hw corruption.
But yeah, it looks like I have triggered it on -rc9 too, just the other day. Oh, and I see -rc6 and -rc8 warnings in the logs too. Ok, correction, not that hard to trigger.
Oct 11 09:08:11 liondog kernel: [15367.473110] ------------[ cut here ]------------ Oct 11 09:08:11 liondog kernel: [15367.473135] WARNING: at kernel/lockdep.c:690 __lock_acquire+0x173/0x17b5() Oct 11 09:08:11 liondog kernel: [15367.473145] Hardware name: System Product Name Oct 11 09:08:11 liondog kernel: [15367.473152] Modules linked in: cryptd aes_x86_64 aes_generic nls_iso8859_15 nls_cp437 tun cpufreq_powersave cpufreq_userspace cpufreq_conservative powernow_k8 mperf cpufreq_stats binfmt_misc fuse dm_crypt dm_mod ipv6 kvm_amd kvm vfat fat radeon 8250_pnp 8250 ttm drm_kms_helper cfbcopyarea edac_core serial_core cfbimgblt cfbfillrect k10temp Oct 11 09:08:11 liondog kernel: [15367.473256] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc9-00005-g538d2882213e #5 Oct 11 09:08:11 liondog kernel: [15367.473264] Call Trace: Oct 11 09:08:11 liondog kernel: [15367.473270] <IRQ> [<ffffffff810367ff>] warn_slowpath_common+0x83/0x9b Oct 11 09:08:11 liondog kernel: [15367.473298] [<ffffffff81036831>] warn_slowpath_null+0x1a/0x1c Oct 11 09:08:11 liondog kernel: [15367.473309] [<ffffffff810691fc>] __lock_acquire+0x173/0x17b5 Oct 11 09:08:11 liondog kernel: [15367.473321] [<ffffffff8106a82c>] ? __lock_acquire+0x17a3/0x17b5 Oct 11 09:08:11 liondog kernel: [15367.473334] [<ffffffff81029c53>] ? double_rq_lock+0x4d/0x52 Oct 11 09:08:11 liondog kernel: [15367.473346] [<ffffffff8106ae67>] lock_acquire+0x154/0x198 Oct 11 09:08:11 liondog kernel: [15367.473356] [<ffffffff81029c53>] ? double_rq_lock+0x4d/0x52 Oct 11 09:08:11 liondog kernel: [15367.473368] [<ffffffff810659de>] ? put_lock_stats.isra.15+0xe/0x29 Oct 11 09:08:11 liondog kernel: [15367.473382] [<ffffffff813c1c49>] _raw_spin_lock_nested+0x44/0x79 Oct 11 09:08:11 liondog kernel: [15367.473392] [<ffffffff81029c53>] ? double_rq_lock+0x4d/0x52 Oct 11 09:08:11 liondog kernel: [15367.473403] [<ffffffff813c1b84>] ? _raw_spin_lock+0x6c/0x73 Oct 11 09:08:11 liondog kernel: [15367.473413] [<ffffffff81029c34>] ? double_rq_lock+0x2e/0x52 Oct 11 09:08:11 liondog kernel: [15367.473423] [<ffffffff81029c53>] double_rq_lock+0x4d/0x52 Oct 11 09:08:11 liondog kernel: [15367.473434] [<ffffffff8102e7d5>] load_balance+0x1b7/0x4f7 Oct 11 09:08:11 liondog kernel: [15367.473447] [<ffffffff8102ec79>] rebalance_domains+0x164/0x1f9 Oct 11 09:08:11 liondog kernel: [15367.473458] [<ffffffff8102eb15>] ? load_balance+0x4f7/0x4f7 Oct 11 09:08:11 liondog kernel: [15367.473470] [<ffffffff8102edcb>] run_rebalance_domains+0xbd/0x12a Oct 11 09:08:11 liondog kernel: [15367.473487] [<ffffffff8103d180>] __do_softirq+0x165/0x2eb Oct 11 09:08:11 liondog kernel: [15367.473499] [<ffffffff81070ff4>] ? generic_smp_call_function_single_interrupt+0x9f/0xd8 Oct 11 09:08:11 liondog kernel: [15367.473512] [<ffffffff813c472c>] call_softirq+0x1c/0x30 Oct 11 09:08:11 liondog kernel: [15367.473525] [<ffffffff810036bb>] do_softirq+0x3d/0x86 Oct 11 09:08:11 liondog kernel: [15367.473535] [<ffffffff8103d561>] irq_exit+0x53/0xbd Oct 11 09:08:11 liondog kernel: [15367.473548] [<ffffffff81018155>] smp_call_function_single_interrupt+0x34/0x37 Oct 11 09:08:11 liondog kernel: [15367.473560] [<ffffffff813c41b0>] call_function_single_interrupt+0x70/0x80 Oct 11 09:08:11 liondog kernel: [15367.473567] <EOI> [<ffffffff8105b7f3>] ? local_clock+0xf/0x3b Oct 11 09:08:11 liondog kernel: [15367.473586] [<ffffffff8105b7f3>] ? local_clock+0xf/0x3b Oct 11 09:08:11 liondog kernel: [15367.473598] [<ffffffff810090b7>] ? default_idle+0xf1/0x1fd Oct 11 09:08:11 liondog kernel: [15367.473610] [<ffffffff810090b5>] ? default_idle+0xef/0x1fd Oct 11 09:08:11 liondog kernel: [15367.473621] [<ffffffff8100931a>] amd_e400_idle+0xc4/0xe7 Oct 11 09:08:11 liondog kernel: [15367.473632] [<ffffffff8100074c>] cpu_idle+0x67/0xbe Oct 11 09:08:11 liondog kernel: [15367.473645] [<ffffffff813b54af>] start_secondary+0x1ad/0x1b2 Oct 11 09:08:11 liondog kernel: [15367.473655] ---[ end trace 63070f7e22365bb6 ]---
Signature · 2 lines
Regards/Gruss,
Boris.Quoted 13 lines · Borislav Petkov
Quoted 7 lines
I think this is a problem with lockdep itself, could you try reverting
f59de8992aa6("lockdep: Clear wholelockdep_mapon initialization") if this reliably happens everytime you reboot (lockdep will only emit this once and then will suppress future warnings until the next boot)?I think the new
memset()is inadvertently clearing the name fordouble_unlock_balance().Great,
so I'm not the only one seeing the above:
http://marc.info/?l=linux-kernel&m=131468805610527
Quoted 7 lines
Due to it being very hard to reproduce, we dismissed it then as a possible hw corruption.
But yeah, it looks like I have triggered it on -rc9 too, just the other day. Oh, and I see -rc6 and -rc8 warnings in the logs too. Ok, correction, not that hard to trigger.
Could you try to revert f59de8992aa6 ("lockdep: Clear whole lockdep_map on initialization") with this patch and see if it helps? Thanks.
---
kernel/lockdep.c +4 -1
@@ -2874,7 +2874,10 @@ static int mark_lock(struct task_struct *curr, struct held_lock *this,void lockdep_init_map (struct lockdep_map *lock,const char *name,struct lock_class_key *key,int subclass) { -memset (lock,0 ,sizeof (*lock)); +int i; + +for (i =0 ; i <NR_LOCKDEP_CACHING_CLASSES ; i++) + lock->class_cache[i] =NULL ;#ifdef CONFIG_LOCK_STAT lock->cpu =raw_smp_processor_id ();
Quoted 16 lines · David Rientjes
Quoted 16 lines
Quoted 7 lines
I think this is a problem with lockdep itself, could you try reverting
f59de8992aa6("lockdep: Clear wholelockdep_mapon initialization") if this reliably happens everytime you reboot (lockdep will only emit this once and then will suppress future warnings until the next boot)?I think the new
memset()is inadvertently clearing the name fordouble_unlock_balance().Great,
so I'm not the only one seeing the above:
http://marc.info/?l=linux-kernel&m=131468805610527Due to it being very hard to reproduce, we dismissed it then as a possible hw corruption.
Quoted 7 lines
Quoted 4 lines
But yeah, it looks like I have triggered it on -rc9 too, just the other day. Oh, and I see -rc6 and -rc8 warnings in the logs too. Ok, correction, not that hard to trigger.
Could you try to revert
f59de8992aa6("lockdep: Clear wholelockdep_mapon initialization") with this patch and see if it helps? Thanks.
Sure, I'd love to and will do, it's just I'm not sure I can easily reproduce it.
Quoted 17 lines
---
kernel/lockdep.c +4
-1@@ -2874,7 +2874,10 @@ static int mark_lock(struct task_struct *curr, struct held_lock *this,void lockdep_init_map (struct lockdep_map *lock,const char *name,struct lock_class_key *key,int subclass) { -memset (lock,0 ,sizeof (*lock)); +int i; + +for (i =0 ; i <NR_LOCKDEP_CACHING_CLASSES ; i++) + lock->class_cache[i] =NULL ;#ifdef CONFIG_LOCK_STAT lock->cpu =raw_smp_processor_id ();
Quoted 8 lines · David Rientjes
Could you try to revert
f59de8992aa6("lockdep: Clear wholelockdep_mapon initialization") with this patch and see if it helps? Thanks.
---kernel/lockdep.c +0
-0@@ -2874,7 +2874,10 @@ static int mark_lock(struct task_struct *curr, struct held_lock *this, void lockdep_init_map(struct lockdep_map *lock, const char *name,
kernel/lockdep.c static int mark_lock(struct task_struct *curr, struct held_lock *this, · line 2883
Quoted 10 lines
kernel/lockdep.c +4
-1struct lock_class_key *key, int subclass) { - memset(lock, 0, sizeof(*lock)); + int i; + +for (i =0 ; i <NR_LOCKDEP_CACHING_CLASSES ; i++) + lock->class_cache[i] =NULL ;#ifdef CONFIG_LOCK_STAT lock->cpu =raw_smp_processor_id ();
FWIW,
the box has been running here with f59de8992aa6 reverted for a couple of days now and no sign of the warning. I'll keep watching it but it looks ok so far, so David, you could've nailed it.
Thanks.
Signature · 2 lines
Regards/Gruss,
Boris.Quoted 17 lines · Borislav Petkov
Quoted 16 lines · David Rientjes
Could you try to revert
f59de8992aa6("lockdep: Clear wholelockdep_mapon initialization") with this patch and see if it helps? Thanks.
---kernel/lockdep.c +4
-1@@ -2874,7 +2874,10 @@ static int mark_lock(struct task_struct *curr, struct held_lock *this,void lockdep_init_map (struct lockdep_map *lock,const char *name,struct lock_class_key *key,int subclass) { -memset (lock,0 ,sizeof (*lock)); +int i; + +for (i =0 ; i <NR_LOCKDEP_CACHING_CLASSES ; i++) + lock->class_cache[i] =NULL ;
Quoted 9 lines
kernel/lockdep.c static int mark_lock(struct task_struct *curr, struct held_lock *this, · line 2883
Quoted 2 lines
kernel/lockdep.c +0
-0#ifdef CONFIG_LOCK_STAT lock->cpu =raw_smp_processor_id ();FWIW,
the box has been running here with
f59de8992aa6reverted for a couple of days now and no sign of the warning. I'll keep watching it but it looks ok so far, so David, you could've nailed it.
Hello,
Well, the same with me. My laptop has been running with reverted f59de8992aa6 without any problems so far. Yet, I'm not sure I understand how memset() and loop could produce different results.
commit in question (f59de8992aa6dc85e81aadc26b0f69e17809721d) has been merge on
Jul 14 15:19:09 2011 +0200, so, Borislav, you probably should have seen it
not only on 3.1-rc5, 3.1-rc6,..., but even on 3.0.
Sergey
Quoted 37 lines · Sergey Senozhatsky
Date: Thu, 20 Oct 2011 21:53:29 +0300
From: Sergey Senozhatsky <[email protected]>
To: Borislav Petkov <[email protected]>, David Rientjes <[email protected]>,
Tejun Heo <[email protected]>, Tejun Heo <[email protected]>, Peter Zijlstra <[email protected]>, Ingo Molnar <[email protected]>, [email protected], Andrew Morton <[email protected]>
Subject: Re: WARNING: atkernel/lockdep.c:690__lock_acquire+0x168/0x164b()
User-Agent: Mutt/1.5.21 (2010-09-15)Quoted 17 lines · Borislav Petkov
Quoted 16 lines · David Rientjes
Could you try to revert
f59de8992aa6("lockdep: Clear wholelockdep_mapon initialization") with this patch and see if it helps? Thanks.
---kernel/lockdep.c +4
-1@@ -2874,7 +2874,10 @@ static int mark_lock(struct task_struct *curr, struct held_lock *this,void lockdep_init_map (struct lockdep_map *lock,const char *name,struct lock_class_key *key,int subclass) { -memset (lock,0 ,sizeof (*lock)); +int i; + +for (i =0 ; i <NR_LOCKDEP_CACHING_CLASSES ; i++) + lock->class_cache[i] =NULL ;Quoted 9 lines
kernel/lockdep.c static int mark_lock(struct task_struct *curr, struct held_lock *this, · line 2883
Quoted 2 lines
kernel/lockdep.c +0
-0#ifdef CONFIG_LOCK_STAT lock->cpu =raw_smp_processor_id ();FWIW,
the box has been running here with
f59de8992aa6reverted for a couple of days now and no sign of the warning. I'll keep watching it but it looks ok so far, so David, you could've nailed it.
Quoted 5 lines
Hello,
Well, the same with me. My laptop has been running with revertedf59de8992aa6without any problems so far. Yet, I'm not sure I understand howmemset()and loop could produce different results.
Oh, well, nevermind I think I get it.
Reverting opens https://bugzilla.kernel.org/show_bug.cgi?id=35532 again.
Quoted 5 lines
commit in question (
f59de8992aa6dc85e81aadc26b0f69e17809721d) has been merge onJul 14 15:19:09 2011 +0200, so, Borislav, you probably should have seen itnot only on 3.1-rc5, 3.1-rc6,..., but even on 3.0.
Sergey
Quoted 13 lines · Sergey Senozhatsky
Quoted 7 lines
Quoted 6 lines
FWIW,
the box has been running here with
f59de8992aa6reverted for a couple of days now and no sign of the warning. I'll keep watching it but it looks ok so far, so David, you could've nailed it.Quoted 5 lines
Hello,
Well, the same with me. My laptop has been running with revertedf59de8992aa6without any problems so far. Yet, I'm not sure I understand howmemset()and loop could produce different results.
Quoted 4 lines
Oh, well, nevermind I think I get it.
Reverting opens https://bugzilla.kernel.org/show_bug.cgi?id=35532 again.
I don't know what that is since bugzilla.kernel.org is down :) The problem is that the memset(), in addition to all the other fields in lockdep_map, clears the "name" field, which is what the scheduler uses via lock_set_sublcass() to prevent this lockdep warning. My initial speculation seems to be confirmed since either you or Borislav have been able to reproduce the warning since removing the memset().
Tejun, would you like to revert f59de8992aa6 ("lockdep: Clear whole lockdep_map on initialization") since it fixes this lockdep warning?
Hello,
Quoted 27 lines · David Rientjes
Quoted 13 lines · Sergey Senozhatsky
Quoted 7 lines
Quoted 6 lines
FWIW,
the box has been running here with
f59de8992aa6reverted for a couple of days now and no sign of the warning. I'll keep watching it but it looks ok so far, so David, you could've nailed it.Quoted 5 lines
Hello,
Well, the same with me. My laptop has been running with revertedf59de8992aa6without any problems so far. Yet, I'm not sure I understand howmemset()and loop could produce different results.Quoted 4 lines
Oh, well, nevermind I think I get it.
Reverting opens https://bugzilla.kernel.org/show_bug.cgi?id=35532 again.
I don't know what that is since bugzilla.kernel.org is down :) The problem is that the
memset(), in addition to all the other fields inlockdep_map, clears the "name" field, which is what the scheduler uses vialock_set_sublcass()to prevent this lockdep warning. My initial speculation seems to be confirmed since either you or Borislav have been able to reproduce the warning since removing thememset().
Quoted 2 lines
Tejun, would you like to revert
f59de8992aa6("lockdep: Clear wholelockdep_mapon initialization") since it fixes this lockdep warning?
Hmmm... the issue was that kmemcheck noticed that memory regions in lockdep_map are accessed before being set to any value. I'm feeling dim as usual and don't understand what's going on here. The function looks like the following.
void lockdep_init_map(struct lockdep_map *lock, const char *name,
struct lock_class_key *key, int subclass)
{
memset(lock, 0, sizeof(*lock)); #ifdef CONFIG_LOCK_STAT
lock->cpu = raw_smp_processor_id();
#endif
if (DEBUG_LOCKS_WARN_ON(!name)) {
lock->name = "NULL";
return;
}
lock->name = name;So, according to this thread, the problem is that the memset() clears lock->name field, right? But how can that be a problem? lock->name is always set to either "NULL" or @name. Why would clearing it before setting make any difference? What am I missing?
Thanks.
Signature · 1 line
tejun
Quoted 25 lines · Tejun Heo
Quoted 2 lines
Tejun, would you like to revert
f59de8992aa6("lockdep: Clear wholelockdep_mapon initialization") since it fixes this lockdep warning?Hmmm... the issue was that kmemcheck noticed that memory regions in
lockdep_mapare accessed before being set to any value. I'm feeling dim as usual and don't understand what's going on here. The function looks like the following.void lockdep_init_map(struct
lockdep_map*lock, const char *name,
structlock_class_key*key, int subclass){ memset(lock, 0, sizeof(*lock));#ifdef CONFIG_LOCK_STAT lock->cpu = raw_smp_processor_id(); #endif if (DEBUG_LOCKS_WARN_ON(!name)) { lock->name = "NULL"; return; } lock->name = name;
Quoted 2 lines
So, according to this thread, the problem is that the
memset()clearslock->namefield, right?
Right, and reverting f59de8992aa6 ("lockdep: Clear whole lockdep_map on initialization") seems to fix the lockdep warning.
Quoted 4 lines
But how can that be a problem?
lock->name
is always set to either "NULL" or @name. Why would clearing it before setting make any difference? What am I missing?
The scheduler (in sched_fair and sched_rt) calls lock_set_subclass() which sets the name in double_unlock_balance() to set the name but there's a race between when that is cleared with the memset() and setting of lock->name where lockdep can find them to match.
Hello,
Quoted 11 lines · David Rientjes
Quoted 2 lines
So, according to this thread, the problem is that the
memset()clearslock->namefield, right?Right, and reverting
f59de8992aa6("lockdep: Clear wholelockdep_mapon initialization") seems to fix the lockdep warning.Quoted 4 lines
But how can that be a problem?
lock->name
is always set to either "NULL" or @name. Why would clearing it before setting make any difference? What am I missing?
Quoted 4 lines
The scheduler (in
sched_fairandsched_rt) callslock_set_subclass()which sets the name indouble_unlock_balance()to set the name but there's a race between when that is cleared with thememset()and setting oflock->namewhere lockdep can find them to match.
Hmmm... so lock_set_subclass() is racing against lockdep_init()? That sounds very fishy and probably needs better fix. Anyways, if someone can't come up with proper solution, please feel free to revert the commit.
Thanks.
Signature · 1 line
tejun
Quoted 19 lines · Tejun Heo
Hello,
Quoted 11 lines · David Rientjes
Quoted 2 lines
So, according to this thread, the problem is that the
memset()clearslock->namefield, right?Right, and reverting
f59de8992aa6("lockdep: Clear wholelockdep_mapon initialization") seems to fix the lockdep warning.Quoted 4 lines
But how can that be a problem?
lock->name
is always set to either "NULL" or @name. Why would clearing it before setting make any difference? What am I missing?Quoted 4 lines
The scheduler (in
sched_fairandsched_rt) callslock_set_subclass()which sets the name indouble_unlock_balance()to set the name but there's a race between when that is cleared with thememset()and setting oflock->namewhere lockdep can find them to match.
Quoted 5 lines
Hmmm... so
lock_set_subclass()is racing againstlockdep_init()? That sounds very fishy and probably needs better fix. Anyways, if someone can't come up with proper solution, please feel free to revert the commit.
I thought I've started understand this, but it was wrong feeling.
The error indeed is that class name and lock name are mismatch
689 if (class->key == key) {
690 WARN_ON_ONCE(class->name != lock->name);
691 return class;
692 }
And the problem as far as I understand only shows up when active_load_balance_cpu_stop() gets called on rq with active_balance.
double_unlock_balance() is called with busiest_rq spin lock held and I don't see who calls lockdep_init_map() on busiest_rq somewhere around. work_struct has its own lockdep_map touched after __queue_work(cpu, wq, work).
I'm not sure that reverting is the best option we have, since it's not fixing the possible race condition it's just mask it.
I'm not very lucky at reproducing issue, in fact I had only one trace so far.
[10172.218213] ------------[ cut here ]------------ [10172.218233] WARNING: at kernel/lockdep.c:690 __lock_acquire+0x168/0x164b() [10172.218346] [<ffffffff8103e7c8>] warn_slowpath_common+0x7e/0x96 [10172.218353] [<ffffffff8103e7f5>] warn_slowpath_null+0x15/0x17 [10172.218361] [<ffffffff8106fee5>] __lock_acquire+0x168/0x164b [10172.218370] [<ffffffff81034645>] ? find_busiest_group+0x7b6/0x941 [10172.218381] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218389] [<ffffffff8107197e>] lock_acquire+0x138/0x1ac [10172.218397] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218404] [<ffffffff8102a5c4>] ? double_rq_lock+0x2e/0x52 [10172.218414] [<ffffffff8148fb49>] _raw_spin_lock_nested+0x3a/0x49 [10172.218421] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218428] [<ffffffff8148fabe>] ? _raw_spin_lock+0x3e/0x45 [10172.218435] [<ffffffff8102a5c4>] ? double_rq_lock+0x2e/0x52 [10172.218442] [<ffffffff8102a5e3>] double_rq_lock+0x4d/0x52 [10172.218449] [<ffffffff810349cc>] load_balance+0x1fc/0x769 [10172.218458] [<ffffffff810075c5>] ? native_sched_clock+0x38/0x65 [10172.218466] [<ffffffff8148ca17>] ? __schedule+0x2f5/0xa2d [10172.218474] [<ffffffff8148caf5>] __schedule+0x3d3/0xa2d [10172.218480] [<ffffffff8148ca17>] ? __schedule+0x2f5/0xa2d [10172.218490] [<ffffffff8104db06>] ? add_timer_on+0xd/0x196 [10172.218497] [<ffffffff8148fc02>] ? _raw_spin_lock_irq+0x4a/0x51 [10172.218505] [<ffffffff8105907b>] ? process_one_work+0x3ed/0x54c [10172.218512] [<ffffffff81059126>] ? process_one_work+0x498/0x54c [10172.218518] [<ffffffff81058e1b>] ? process_one_work+0x18d/0x54c [10172.218526] [<ffffffff814902d0>] ? _raw_spin_unlock_irq+0x28/0x56 [10172.218533] [<ffffffff81033950>] ? get_parent_ip+0xe/0x3e [10172.218540] [<ffffffff8148d26e>] schedule+0x55/0x57 [10172.218547] [<ffffffff8105970f>] worker_thread+0x217/0x21c [10172.218554] [<ffffffff810594f8>] ? manage_workers.isra.21+0x16c/0x16c [10172.218564] [<ffffffff8105d4de>] kthread+0x9a/0xa2 [10172.218573] [<ffffffff81497984>] kernel_thread_helper+0x4/0x10 [10172.218580] [<ffffffff8102d6d2>] ? finish_task_switch+0x76/0xf3 [10172.218587] [<ffffffff81490778>] ? retint_restore_args+0x13/0x13 [10172.218595] [<ffffffff8105d444>] ? __init_kthread_worker+0x53/0x53 [10172.218602] [<ffffffff81497980>] ? gs_change+0x13/0x13 [10172.218607] ---[ end trace 9d11d6b5e4b96730 ]---
Sergey
Quoted 12 lines · Sergey Senozhatsky
I thought I've started understand this, but it was wrong feeling.
The error indeed is that class name and lock name are mismatch
689 if (
class->key== key) {
690WARN_ON_ONCE(class->name != lock->name);
691 return class;
692 }And the problem as far as I understand only shows up when
active_load_balance_cpu_stop()gets called on rq withactive_balance.
Quoted 7 lines
double_unlock_balance()is called withbusiest_rqspin lock held and I don't see who callslockdep_init_map()onbusiest_rqsomewhere around.work_structhas its ownlockdep_maptouched after__queue_work(cpu, wq, work).I'm not sure that reverting is the best option we have, since it's not fixing the possible race condition it's just mask it.
How does it mask the race condition? Before the memset(), the ->name field was never _cleared_ in lockdep_init_map() like it is now, it was only stored.
Quoted 31 lines
I'm not very lucky at reproducing issue, in fact I had only one trace so far.
[10172.218213] ------------[ cut here ]------------ [10172.218233] WARNING: at kernel/lockdep.c:690 __lock_acquire+0x168/0x164b() [10172.218346] [<ffffffff8103e7c8>] warn_slowpath_common+0x7e/0x96 [10172.218353] [<ffffffff8103e7f5>] warn_slowpath_null+0x15/0x17 [10172.218361] [<ffffffff8106fee5>] __lock_acquire+0x168/0x164b [10172.218370] [<ffffffff81034645>] ? find_busiest_group+0x7b6/0x941 [10172.218381] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218389] [<ffffffff8107197e>] lock_acquire+0x138/0x1ac [10172.218397] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218404] [<ffffffff8102a5c4>] ? double_rq_lock+0x2e/0x52 [10172.218414] [<ffffffff8148fb49>] _raw_spin_lock_nested+0x3a/0x49 [10172.218421] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218428] [<ffffffff8148fabe>] ? _raw_spin_lock+0x3e/0x45 [10172.218435] [<ffffffff8102a5c4>] ? double_rq_lock+0x2e/0x52 [10172.218442] [<ffffffff8102a5e3>] double_rq_lock+0x4d/0x52 [10172.218449] [<ffffffff810349cc>] load_balance+0x1fc/0x769 [10172.218458] [<ffffffff810075c5>] ? native_sched_clock+0x38/0x65 [10172.218466] [<ffffffff8148ca17>] ? __schedule+0x2f5/0xa2d [10172.218474] [<ffffffff8148caf5>] __schedule+0x3d3/0xa2d [10172.218480] [<ffffffff8148ca17>] ? __schedule+0x2f5/0xa2d [10172.218490] [<ffffffff8104db06>] ? add_timer_on+0xd/0x196 [10172.218497] [<ffffffff8148fc02>] ? _raw_spin_lock_irq+0x4a/0x51 [10172.218505] [<ffffffff8105907b>] ? process_one_work+0x3ed/0x54c [10172.218512] [<ffffffff81059126>] ? process_one_work+0x498/0x54c [10172.218518] [<ffffffff81058e1b>] ? process_one_work+0x18d/0x54c [10172.218526] [<ffffffff814902d0>] ? _raw_spin_unlock_irq+0x28/0x56 [10172.218533] [<ffffffff81033950>] ? get_parent_ip+0xe/0x3e [10172.218540] [<ffffffff8148d26e>] schedule+0x55/0x57 [10172.218547] [<ffffffff8105970f>] worker_thread+0x217/0x21c
Quoted 9 lines
[10172.218554] [<ffffffff810594f8>] ? manage_workers.isra.21+0x16c/0x16c [10172.218564] [<ffffffff8105d4de>] kthread+0x9a/0xa2 [10172.218573] [<ffffffff81497984>] kernel_thread_helper+0x4/0x10 [10172.218580] [<ffffffff8102d6d2>] ? finish_task_switch+0x76/0xf3 [10172.218587] [<ffffffff81490778>] ? retint_restore_args+0x13/0x13 [10172.218595] [<ffffffff8105d444>] ? __init_kthread_worker+0x53/0x53 [10172.218602] [<ffffffff81497980>] ? gs_change+0x13/0x13 [10172.218607] ---[ end trace 9d11d6b5e4b96730 ]---
This is with the revert?
Quoted 16 lines · David Rientjes
Quoted 16 lines
I thought I've started understand this, but it was wrong feeling.
The error indeed is that class name and lock name are mismatch
689 if (
class->key== key) {
690WARN_ON_ONCE(class->name != lock->name);
691 return class;
692 }And the problem as far as I understand only shows up when
active_load_balance_cpu_stop()gets called on rq withactive_balance.
double_unlock_balance()is called withbusiest_rqspin lock held and I don't see who callslockdep_init_map()onbusiest_rqsomewhere around.work_structhas its ownlockdep_maptouched after__queue_work(cpu, wq, work).
Quoted 8 lines
Quoted 3 lines
I'm not sure that reverting is the best option we have, since it's not fixing the possible race condition it's just mask it.
How does it mask the race condition? Before the
memset(), the ->name field was never _cleared_ inlockdep_init_map()like it is now, it was only stored.
Well, if we have race condition between reader' and writer', then it's our luck that we only hit it with ->name modification. It could be ->cpu = raw_smp_processor_id' or while iterating thr' class_cache' to NULL it. Current implementation may only race with `->name' but in theory we have the whole bunch of opportunities. Of course I may be wrong.
Quoted 41 lines
Quoted 31 lines
I'm not very lucky at reproducing issue, in fact I had only one trace so far.
[10172.218213] ------------[ cut here ]------------ [10172.218233] WARNING: at kernel/lockdep.c:690 __lock_acquire+0x168/0x164b() [10172.218346] [<ffffffff8103e7c8>] warn_slowpath_common+0x7e/0x96 [10172.218353] [<ffffffff8103e7f5>] warn_slowpath_null+0x15/0x17 [10172.218361] [<ffffffff8106fee5>] __lock_acquire+0x168/0x164b [10172.218370] [<ffffffff81034645>] ? find_busiest_group+0x7b6/0x941 [10172.218381] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218389] [<ffffffff8107197e>] lock_acquire+0x138/0x1ac [10172.218397] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218404] [<ffffffff8102a5c4>] ? double_rq_lock+0x2e/0x52 [10172.218414] [<ffffffff8148fb49>] _raw_spin_lock_nested+0x3a/0x49 [10172.218421] [<ffffffff8102a5e3>] ? double_rq_lock+0x4d/0x52 [10172.218428] [<ffffffff8148fabe>] ? _raw_spin_lock+0x3e/0x45 [10172.218435] [<ffffffff8102a5c4>] ? double_rq_lock+0x2e/0x52 [10172.218442] [<ffffffff8102a5e3>] double_rq_lock+0x4d/0x52 [10172.218449] [<ffffffff810349cc>] load_balance+0x1fc/0x769 [10172.218458] [<ffffffff810075c5>] ? native_sched_clock+0x38/0x65 [10172.218466] [<ffffffff8148ca17>] ? __schedule+0x2f5/0xa2d [10172.218474] [<ffffffff8148caf5>] __schedule+0x3d3/0xa2d [10172.218480] [<ffffffff8148ca17>] ? __schedule+0x2f5/0xa2d [10172.218490] [<ffffffff8104db06>] ? add_timer_on+0xd/0x196 [10172.218497] [<ffffffff8148fc02>] ? _raw_spin_lock_irq+0x4a/0x51 [10172.218505] [<ffffffff8105907b>] ? process_one_work+0x3ed/0x54c [10172.218512] [<ffffffff81059126>] ? process_one_work+0x498/0x54c [10172.218518] [<ffffffff81058e1b>] ? process_one_work+0x18d/0x54c [10172.218526] [<ffffffff814902d0>] ? _raw_spin_unlock_irq+0x28/0x56 [10172.218533] [<ffffffff81033950>] ? get_parent_ip+0xe/0x3e [10172.218540] [<ffffffff8148d26e>] schedule+0x55/0x57 [10172.218547] [<ffffffff8105970f>] worker_thread+0x217/0x21cQuoted 9 lines
[10172.218554] [<ffffffff810594f8>] ? manage_workers.isra.21+0x16c/0x16c [10172.218564] [<ffffffff8105d4de>] kthread+0x9a/0xa2 [10172.218573] [<ffffffff81497984>] kernel_thread_helper+0x4/0x10 [10172.218580] [<ffffffff8102d6d2>] ? finish_task_switch+0x76/0xf3 [10172.218587] [<ffffffff81490778>] ? retint_restore_args+0x13/0x13 [10172.218595] [<ffffffff8105d444>] ? __init_kthread_worker+0x53/0x53 [10172.218602] [<ffffffff81497980>] ? gs_change+0x13/0x13 [10172.218607] ---[ end trace 9d11d6b5e4b96730 ]---
Quoted 2 lines
This is with the revert?
Nope, sorry for being unclear, this is the only trace I got.
Sergey
Quoted 3 lines · David Rientjes
How does it mask the race condition? Before the
memset(), the ->name field was never _cleared_ inlockdep_init_map()like it is now, it was only stored.
A typcal race condition will like this:
CPU A CPU B
lock_set_subclass(lockA);
lock_set_class(lockA);
lockdep_init_map(lockA);
/* lockA->name is cleared */
memset(lockA);
__lock_acquire(lockA);
/* lockA->class_cache[] is cleared */
register_lock_class(lockA);
look_up_lock_class(lockA);
WARN_ON_ONCE(class->name !=
lock->name);
lock->name = name;And a untested patch is below:
BTW, now the patch could cure (I guess) the very issue reported in this thread.
But it don't cover the case which change the key and the relevant lock_class has existed, I don't think out a way how to fix it yet :) But the fact is we have no such caller yet, the only call site of lock_set_subclass() is double_unlock_balance().
Thanks,
Yong
---
From: Yong Zhang <[email protected]>
Subject: [PATCH] lockdep: On-demand initialization for lock_set_class()
Since commit f59de89 [lockdep: Clear whole lockdep_map on initialization], lockdep_init_map() will clear all the struct. But it will break lock_set_class()/lock_set_subclass(). A typical race condition is like below:
CPU A CPU B
lock_set_subclass(lockA);
lock_set_class(lockA);
lockdep_init_map(lockA);
/* lockA->name is cleared */
memset(lockA);
__lock_acquire(lockA);
/* lockA->class_cache[] is cleared */
register_lock_class(lockA);
look_up_lock_class(lockA);
WARN_ON_ONCE(class->name !=
lock->name);
lock->name = name;--- kernel/lockdep.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)
kernel/lockdep.c +4 -1
@@ -3160,7 +3160,10 @@ __lock_set_class(struct lockdep_map *lock, const char *name,return print_unlock_inbalance_bug (curr, lock, ip); found_it: -lockdep_init_map (lock, name, key,0 ); +/* only changing lock->name make no sense */ +WARN_ON (lock->key == key && lock->name != name); +if (lock->key != key) +lockdep_init_map (lock, name, key,0 ); class =register_lock_class (lock, subclass,0 ); hlock->class_idx = class - lock_classes +1 ;
Signature · 1 line
1.7.5.4
Quoted 22 lines · Yong Zhang
Quoted 3 lines · David Rientjes
How does it mask the race condition? Before the
memset(), the ->name field was never _cleared_ inlockdep_init_map()like it is now, it was only stored.A typcal race condition will like this:
CPU A CPU B lock_set_subclass(lockA); lock_set_class(lockA); lockdep_init_map(lockA); /* lockA->name is cleared */ memset(lockA); __lock_acquire(lockA); /* lockA->class_cache[] is cleared */ register_lock_class(lockA); look_up_lock_class(lockA); WARN_ON_ONCE(class->name != lock->name); lock->name = name;
Quoted 8 lines
And a untested patch is below:
BTW, now the patch could cure (I guess) the very issue reported in this thread.
But it don't cover the case which change the key and the relevantlock_classhas existed, I don't think out a way how to fix it yet :) But the fact is we have no such caller yet, the only call site oflock_set_subclass()isdouble_unlock_balance().
Hello,
Any news on this patch? Do you like it or hate it? With recent kernels I'm able to hit this problem more often (several time a day) so if any testing is required I'm willing to help.
Sergey
Quoted 56 lines
---
From: Yong Zhang <[email protected]>
Subject: [PATCH] lockdep: On-demand initialization forlock_set_class()Since commit f59de89 [lockdep: Clear whole
lockdep_mapon initialization],lockdep_init_map()will clear all the struct. But it will break lock_set_class()/lock_set_subclass(). A typical race condition is like below:CPU A CPU B lock_set_subclass(lockA); lock_set_class(lockA); lockdep_init_map(lockA); /* lockA->name is cleared */ memset(lockA); __lock_acquire(lockA); /* lockA->class_cache[] is cleared */ register_lock_class(lockA); look_up_lock_class(lockA); WARN_ON_ONCE(class->name != lock->name); lock->name = name;Reported-by Sergey Senozhatsky <[email protected]>Reported-by Borislav Petkov <[email protected]>Signed-off-by Yong Zhang <[email protected]>Cc Tejun Heo <[email protected]>Cc David Rientjes <[email protected]>Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>--- kernel/lockdep.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)kernel/lockdep.c +4
-1@@ -3160,7 +3160,10 @@ __lock_set_class(struct lockdep_map *lock, const char *name,return print_unlock_inbalance_bug (curr, lock, ip); found_it: -lockdep_init_map (lock, name, key,0 ); +/* only changing lock->name make no sense */ +WARN_ON (lock->key == key && lock->name != name); +if (lock->key != key) +lockdep_init_map (lock, name, key,0 ); class =register_lock_class (lock, subclass,0 ); hlock->class_idx = class - lock_classes +1 ;Signature · 1 line
1.7.5.4
Quoted 32 lines · Sergey Senozhatsky
Quoted 22 lines · Yong Zhang
Quoted 3 lines · David Rientjes
How does it mask the race condition? Before the
memset(), the ->name field was never _cleared_ inlockdep_init_map()like it is now, it was only stored.A typcal race condition will like this:
CPU A CPU B lock_set_subclass(lockA); lock_set_class(lockA); lockdep_init_map(lockA); /* lockA->name is cleared */ memset(lockA); __lock_acquire(lockA); /* lockA->class_cache[] is cleared */ register_lock_class(lockA); look_up_lock_class(lockA); WARN_ON_ONCE(class->name != lock->name); lock->name = name;Quoted 8 lines
And a untested patch is below:
BTW, now the patch could cure (I guess) the very issue reported in this thread.
But it don't cover the case which change the key and the relevantlock_classhas existed, I don't think out a way how to fix it yet :) But the fact is we have no such caller yet, the only call site oflock_set_subclass()isdouble_unlock_balance().
Quoted 4 lines
Hello,
Any news on this patch? Do you like it or hate it? With recent kernels I'm able to hit this problem more often (several time a day) so if any testing is required I'm willing to help.
Did you have tried it? Though I don't find time to polish it yet but I think will smooth your concern.
Thanks,
Yong
Quoted 26 lines · Yong Zhang
Quoted 26 lines
Quoted 25 lines
A typcal race condition will like this:
CPU A CPU B lock_set_subclass(lockA); lock_set_class(lockA); lockdep_init_map(lockA); /* lockA->name is cleared */ memset(lockA); __lock_acquire(lockA); /* lockA->class_cache[] is cleared */ register_lock_class(lockA); look_up_lock_class(lockA); WARN_ON_ONCE(class->name != lock->name); lock->name = name;And a untested patch is below:
BTW, now the patch could cure (I guess) the very issue reported in this thread.
But it don't cover the case which change the key and the relevantlock_classhas existed, I don't think out a way how to fix it yet :) But the fact is we have no such caller yet, the only call site oflock_set_subclass()isdouble_unlock_balance().
Quoted 8 lines
Quoted 4 lines
Hello,
Any news on this patch? Do you like it or hate it? With recent kernels I'm able to hit this problem more often (several time a day) so if any testing is required I'm willing to help.Did you have tried it? Though I don't find time to polish it yet but I think will smooth your concern.
I'm compiling the kernel with you patch right now. The whole point was just for case if someone has different approach or whatsoever.
Sergey
Quoted 8 lines · Sergey Senozhatsky
Quoted 3 lines · Yong Zhang
Did you have tried it? Though I don't find time to polish it yet but I think will smooth your concern.
I'm compiling the kernel with you patch right now. The whole point was just for
case if someone has different approach or whatsoever.
Understood. If someone can come up with a simple patch which could cover the case I mentioned before, that would be great. /me goes to poke at it.
Thanks,
Yong
Quoted 6 lines · Yong Zhang
Quoted 5 lines · Sergey Senozhatsky
Quoted 3 lines · Yong Zhang
Did you have tried it? Though I don't find time to polish it yet but I think will smooth your concern.
Quoted 7 lines
Quoted 3 lines
I'm compiling the kernel with you patch right now. The whole point was just for
case if someone has different approach or whatsoever.Understood. If someone can come up with a simple patch which could cover the case I mentioned before, that would be great. /me goes to poke at it.
I dunno whether this is related but I get the following on 3.1:
[ 5499.537074] INFO: trying to register non-static key. [ 5499.537080] the code is fine but needs lockdep annotation. [ 5499.537083] turning off the locking correctness validator. [ 5499.537088] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0 #1 [ 5499.537091] Call Trace: [ 5499.537094] <IRQ> [<ffffffff8107beed>] __lock_acquire+0x165d/0x1e30 [ 5499.537109] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537115] [<ffffffff8107ccd3>] lock_acquire+0x93/0x160 [ 5499.537120] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537126] [<ffffffff814d9866>] _raw_spin_lock+0x36/0x50 [ 5499.537130] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537135] [<ffffffff810321fc>] double_rq_lock+0x2c/0x80 [ 5499.537140] [<ffffffff81039195>] load_balance+0x215/0x6c0 [ 5499.537146] [<ffffffff81039640>] ? load_balance+0x6c0/0x6c0 [ 5499.537151] [<ffffffff810396fd>] rebalance_domains+0xbd/0x1d0 [ 5499.537155] [<ffffffff81039640>] ? load_balance+0x6c0/0x6c0 [ 5499.537161] [<ffffffff810398ec>] run_rebalance_domains+0xdc/0x130 [ 5499.537166] [<ffffffff81048dcd>] __do_softirq+0xbd/0x290 [ 5499.537173] [<ffffffff814dc42c>] call_softirq+0x1c/0x30 [ 5499.537178] [<ffffffff81003eb5>] do_softirq+0x85/0xc0 [ 5499.537183] [<ffffffff810492ce>] irq_exit+0x9e/0xc0 [ 5499.537189] [<ffffffff8101ca9f>] smp_call_function_single_interrupt+0x2f/0x40 [ 5499.537195] [<ffffffff814dbeb0>] call_function_single_interrupt+0x70/0x80 [ 5499.537199] <EOI> [<ffffffff810096e6>] ? native_sched_clock+0x26/0x70 [ 5499.537212] [<ffffffffa0038e1a>] ? acpi_idle_enter_simple+0xee/0x11f [processor] [ 5499.537221] [<ffffffffa0038e15>] ? acpi_idle_enter_simple+0xe9/0x11f [processor] [ 5499.537227] [<ffffffff813f8b1d>] cpuidle_idle_call+0xdd/0x350 [ 5499.537233] [<ffffffff8100081f>] cpu_idle+0x6f/0xd0 [ 5499.537238] [<ffffffff814cc665>] start_secondary+0x1ae/0x1b3
Signature · 2 lines
Regards/Gruss, Boris.
Quoted 6 lines · Borislav Petkov
Quoted 3 lines
Understood. If someone can come up with a simple patch which could cover the case I mentioned before, that would be great. /me goes to poke at it.
I dunno whether this is related but I get the following on 3.1:
I think this is different problem. Failed check that lockdep key is marked as `static'.
Sergey
Quoted 34 lines
[ 5499.537074] INFO: trying to register non-static key. [ 5499.537080] the code is fine but needs lockdep annotation. [ 5499.537083] turning off the locking correctness validator. [ 5499.537088] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0 #1 [ 5499.537091] Call Trace: [ 5499.537094] <IRQ> [<ffffffff8107beed>] __lock_acquire+0x165d/0x1e30 [ 5499.537109] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537115] [<ffffffff8107ccd3>] lock_acquire+0x93/0x160 [ 5499.537120] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537126] [<ffffffff814d9866>] _raw_spin_lock+0x36/0x50 [ 5499.537130] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537135] [<ffffffff810321fc>] double_rq_lock+0x2c/0x80 [ 5499.537140] [<ffffffff81039195>] load_balance+0x215/0x6c0 [ 5499.537146] [<ffffffff81039640>] ? load_balance+0x6c0/0x6c0 [ 5499.537151] [<ffffffff810396fd>] rebalance_domains+0xbd/0x1d0 [ 5499.537155] [<ffffffff81039640>] ? load_balance+0x6c0/0x6c0 [ 5499.537161] [<ffffffff810398ec>] run_rebalance_domains+0xdc/0x130 [ 5499.537166] [<ffffffff81048dcd>] __do_softirq+0xbd/0x290 [ 5499.537173] [<ffffffff814dc42c>] call_softirq+0x1c/0x30 [ 5499.537178] [<ffffffff81003eb5>] do_softirq+0x85/0xc0 [ 5499.537183] [<ffffffff810492ce>] irq_exit+0x9e/0xc0 [ 5499.537189] [<ffffffff8101ca9f>] smp_call_function_single_interrupt+0x2f/0x40 [ 5499.537195] [<ffffffff814dbeb0>] call_function_single_interrupt+0x70/0x80 [ 5499.537199] <EOI> [<ffffffff810096e6>] ? native_sched_clock+0x26/0x70 [ 5499.537212] [<ffffffffa0038e1a>] ? acpi_idle_enter_simple+0xee/0x11f [processor] [ 5499.537221] [<ffffffffa0038e15>] ? acpi_idle_enter_simple+0xe9/0x11f [processor] [ 5499.537227] [<ffffffff813f8b1d>] cpuidle_idle_call+0xdd/0x350 [ 5499.537233] [<ffffffff8100081f>] cpu_idle+0x6f/0xd0 [ 5499.537238] [<ffffffff814cc665>] start_secondary+0x1ae/0x1b3Signature · 2 lines
Regards/Gruss, Boris.
Quoted 11 lines · Borislav Petkov
Quoted 10 lines · Yong Zhang
Quoted 8 lines · Sergey Senozhatsky
Quoted 3 lines · Yong Zhang
Did you have tried it? Though I don't find time to polish it yet but I think will smooth your concern.
I'm compiling the kernel with you patch right now. The whole point was just for
case if someone has different approach or whatsoever.
Quoted 5 lines
Quoted 3 lines
Understood. If someone can come up with a simple patch which could cover the case I mentioned before, that would be great. /me goes to poke at it.
I dunno whether this is related but I get the following on 3.1:
Maybe, so could you try my patches just sent out?
http://marc.info/?l=linux-kernel&m=132039886826672
Thanks,
Yong
Quoted 39 lines
[ 5499.537074] INFO: trying to register non-static key. [ 5499.537080] the code is fine but needs lockdep annotation. [ 5499.537083] turning off the locking correctness validator. [ 5499.537088] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0 #1 [ 5499.537091] Call Trace: [ 5499.537094] <IRQ> [<ffffffff8107beed>] __lock_acquire+0x165d/0x1e30 [ 5499.537109] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537115] [<ffffffff8107ccd3>] lock_acquire+0x93/0x160 [ 5499.537120] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537126] [<ffffffff814d9866>] _raw_spin_lock+0x36/0x50 [ 5499.537130] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537135] [<ffffffff810321fc>] double_rq_lock+0x2c/0x80 [ 5499.537140] [<ffffffff81039195>] load_balance+0x215/0x6c0 [ 5499.537146] [<ffffffff81039640>] ? load_balance+0x6c0/0x6c0 [ 5499.537151] [<ffffffff810396fd>] rebalance_domains+0xbd/0x1d0 [ 5499.537155] [<ffffffff81039640>] ? load_balance+0x6c0/0x6c0 [ 5499.537161] [<ffffffff810398ec>] run_rebalance_domains+0xdc/0x130 [ 5499.537166] [<ffffffff81048dcd>] __do_softirq+0xbd/0x290 [ 5499.537173] [<ffffffff814dc42c>] call_softirq+0x1c/0x30 [ 5499.537178] [<ffffffff81003eb5>] do_softirq+0x85/0xc0 [ 5499.537183] [<ffffffff810492ce>] irq_exit+0x9e/0xc0 [ 5499.537189] [<ffffffff8101ca9f>] smp_call_function_single_interrupt+0x2f/0x40 [ 5499.537195] [<ffffffff814dbeb0>] call_function_single_interrupt+0x70/0x80 [ 5499.537199] <EOI> [<ffffffff810096e6>] ? native_sched_clock+0x26/0x70 [ 5499.537212] [<ffffffffa0038e1a>] ? acpi_idle_enter_simple+0xee/0x11f [processor] [ 5499.537221] [<ffffffffa0038e15>] ? acpi_idle_enter_simple+0xe9/0x11f [processor] [ 5499.537227] [<ffffffff813f8b1d>] cpuidle_idle_call+0xdd/0x350 [ 5499.537233] [<ffffffff8100081f>] cpu_idle+0x6f/0xd0 [ 5499.537238] [<ffffffff814cc665>] start_secondary+0x1ae/0x1b3Signature · 7 lines
Regards/Gruss, Boris. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Signature · 1 line
Only stand for myself
Quoted 4 lines · Yong Zhang
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
I'm compiling the kernel with you patch right now. The whole point was just for
case if someone has different approach or whatsoever.
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
Understood. If someone can come up with a simple patch which could cover the case I mentioned before, that would be great. /me goes to poke at it.
I dunno whether this is related but I get the following on 3.1:
Maybe, so could you try my patches just sent out?
http://marc.info/?l=linux-kernel&m=132039886826672
Sure I'll try you patches. Later today or (most likely) during the upcoming weekend, since now I'm extremly busy at work.
Thanks,
Sergey
Quoted 46 lines
Thanks,
YongQuoted 39 lines
[ 5499.537074] INFO: trying to register non-static key. [ 5499.537080] the code is fine but needs lockdep annotation. [ 5499.537083] turning off the locking correctness validator. [ 5499.537088] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0 #1 [ 5499.537091] Call Trace: [ 5499.537094] <IRQ> [<ffffffff8107beed>] __lock_acquire+0x165d/0x1e30 [ 5499.537109] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537115] [<ffffffff8107ccd3>] lock_acquire+0x93/0x160 [ 5499.537120] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537126] [<ffffffff814d9866>] _raw_spin_lock+0x36/0x50 [ 5499.537130] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537135] [<ffffffff810321fc>] double_rq_lock+0x2c/0x80 [ 5499.537140] [<ffffffff81039195>] load_balance+0x215/0x6c0 [ 5499.537146] [<ffffffff81039640>] ? load_balance+0x6c0/0x6c0 [ 5499.537151] [<ffffffff810396fd>] rebalance_domains+0xbd/0x1d0 [ 5499.537155] [<ffffffff81039640>] ? load_balance+0x6c0/0x6c0 [ 5499.537161] [<ffffffff810398ec>] run_rebalance_domains+0xdc/0x130 [ 5499.537166] [<ffffffff81048dcd>] __do_softirq+0xbd/0x290 [ 5499.537173] [<ffffffff814dc42c>] call_softirq+0x1c/0x30 [ 5499.537178] [<ffffffff81003eb5>] do_softirq+0x85/0xc0 [ 5499.537183] [<ffffffff810492ce>] irq_exit+0x9e/0xc0 [ 5499.537189] [<ffffffff8101ca9f>] smp_call_function_single_interrupt+0x2f/0x40 [ 5499.537195] [<ffffffff814dbeb0>] call_function_single_interrupt+0x70/0x80 [ 5499.537199] <EOI> [<ffffffff810096e6>] ? native_sched_clock+0x26/0x70 [ 5499.537212] [<ffffffffa0038e1a>] ? acpi_idle_enter_simple+0xee/0x11f [processor] [ 5499.537221] [<ffffffffa0038e15>] ? acpi_idle_enter_simple+0xe9/0x11f [processor] [ 5499.537227] [<ffffffff813f8b1d>] cpuidle_idle_call+0xdd/0x350 [ 5499.537233] [<ffffffff8100081f>] cpu_idle+0x6f/0xd0 [ 5499.537238] [<ffffffff814cc665>] start_secondary+0x1ae/0x1b3Signature · 7 lines
Regards/Gruss, Boris. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/Signature · 1 line
Only stand for myself
Quoted 9 lines · Sergey Senozhatsky
Quoted 6 lines · Borislav Petkov
Quoted 3 lines
Understood. If someone can come up with a simple patch which could cover the case I mentioned before, that would be great. /me goes to poke at it.
I dunno whether this is related but I get the following on 3.1:
I think this is different problem. Failed check that lockdep key is marked as `static'.
Actually the lockdep_init_map() in __lock_set_class could lead to more problem, such as: certain rq->lock could have different 'key' with what we give them in sched_init() because rq is defined staticly.
Given that, we could have another typical race:
CPU A CPU B
lock_set_subclass(lockA);
lock_set_class(lockA);
/* lockA->class_cache[] is not set */
register_lock_class(lockA);
look_up_lock_class(lockA); /* retrun NULL */
lockdep_init_map(lockA);
/* lockA->name is cleared */
memset(lockA);
if (!static_obj(lock->key))
/* we get warning here */
lock->name = name;So memset() in lockdep_init_map() is still the culprit IMHO.
Thanks,
Yong
Quoted 42 lines
Sergey
Quoted 21 lines
[ 5499.537074] INFO: trying to register non-static key. [ 5499.537080] the code is fine but needs lockdep annotation. [ 5499.537083] turning off the locking correctness validator. [ 5499.537088] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0 #1 [ 5499.537091] Call Trace: [ 5499.537094] <IRQ> [<ffffffff8107beed>] __lock_acquire+0x165d/0x1e30 [ 5499.537109] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537115] [<ffffffff8107ccd3>] lock_acquire+0x93/0x160 [ 5499.537120] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537126] [<ffffffff814d9866>] _raw_spin_lock+0x36/0x50 [ 5499.537130] [<ffffffff810321fc>] ? double_rq_lock+0x2c/0x80 [ 5499.537135] [<ffffffff810321fc>] double_rq_lock+0x2c/0x80 [ 5499.537140] [<ffffffff81039195>] load_balance+0x215/0x6c0 [ 5499.537146] [<ffffffff81039640>] ? load_balance+0x6c0/0x6c0 [ 5499.537151] [<ffffffff810396fd>] rebalance_domains+0xbd/0x1d0 [ 5499.537155] [<ffffffff81039640>] ? load_balance+0x6c0/0x6c0 [ 5499.537161] [<ffffffff810398ec>] run_rebalance_domains+0xdc/0x130 [ 5499.537166] [<ffffffff81048dcd>] __do_softirq+0xbd/0x290 [ 5499.537173] [<ffffffff814dc42c>] call_softirq+0x1c/0x30 [ 5499.537178] [<ffffffff81003eb5>] do_softirq+0x85/0xc0 [ 5499.537183] [<ffffffff810492ce>] irq_exit+0x9e/0xc0Quoted 13 lines
[ 5499.537189] [<ffffffff8101ca9f>] smp_call_function_single_interrupt+0x2f/0x40 [ 5499.537195] [<ffffffff814dbeb0>] call_function_single_interrupt+0x70/0x80 [ 5499.537199] <EOI> [<ffffffff810096e6>] ? native_sched_clock+0x26/0x70 [ 5499.537212] [<ffffffffa0038e1a>] ? acpi_idle_enter_simple+0xee/0x11f [processor] [ 5499.537221] [<ffffffffa0038e15>] ? acpi_idle_enter_simple+0xe9/0x11f [processor] [ 5499.537227] [<ffffffff813f8b1d>] cpuidle_idle_call+0xdd/0x350 [ 5499.537233] [<ffffffff8100081f>] cpu_idle+0x6f/0xd0 [ 5499.537238] [<ffffffff814cc665>] start_secondary+0x1ae/0x1b3Signature · 2 lines
Regards/Gruss, Boris.--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 1 line
Only stand for myself
Quoted 29 lines · Yong Zhang
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Understood. If someone can come up with a simple patch which could cover the case I mentioned before, that would be great. /me goes to poke at it.
I dunno whether this is related but I get the following on 3.1:
I think this is different problem. Failed check that lockdep key is marked as `static'.
Actually the
lockdep_init_map()in__lock_set_classcould lead to more problem, such as: certainrq->lockcould have different 'key' with what we give them insched_init()because rq is defined staticly.Given that, we could have another typical race:
CPU A CPU B lock_set_subclass(lockA); lock_set_class(lockA); /* lockA->class_cache[] is not set */ register_lock_class(lockA); look_up_lock_class(lockA); /* retrun NULL */ lockdep_init_map(lockA); /* lockA->name is cleared */ memset(lockA); if (!static_obj(lock->key)) /* we get warning here */ lock->name = name;
Quoted 2 lines
So
memset()inlockdep_init_map()is still the culprit IMHO.
Hm, agreed, that still could be the reason.
I guess a little more information may be helpful in some cases.
---
Print key address when attempt to register non-static lock key detected.
---
kernel/lockdep.c +1 -1
@@ -729,7 +729,7 @@ register_lock_class(struct lockdep_map *lock, unsigned int subclass, int force)*/ if (!static_obj (lock->key)) {debug_locks_off (); -printk ("INFO: trying to register non-static key.\n" ); +printk ("INFO: trying to register non-static key at address %p.\n" , lock->key);printk ("the code is fine but needs lockdep annotation.\n" );printk ("turning off the locking correctness validator.\n" );dump_stack ();