kernel: smp: WARNING at kernel/smp.c:673
Hi all,
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:
[ 1226.701012] WARNING: CPU: 6 PID: 8624 at kernel/smp.c:673 on_each_cpu_cond+0x27f/0x2f0() [ 1226.703128] Modules linked in: [ 1226.703955] CPU: 6 PID: 8624 Comm: trinity-c26 Tainted: G W 3.16.0-rc7-next-20140801-sasha-00047-gd6ce559 #991 [ 1226.706792] ffff881b2e8a3000 0000000023830039 0000000000000000 ffff881b2e8a3000 [ 1226.708179] ffff881ada5b3978 ffffffffa1ee96b4 0000000000000000 ffff881ada5b39b8 [ 1226.709610] ffffffff9d1ce5ba ffffffff9d2c0ddf 000000000000001b ffff8807d7608000 [ 1226.710966] Call Trace: [ 1226.711417] dump_stack (lib/dump_stack.c:52) [ 1226.712311] warn_slowpath_common (kernel/panic.c:432) [ 1226.715425] warn_slowpath_null (kernel/panic.c:466) [ 1226.716436] on_each_cpu_cond (kernel/smp.c:673 (discriminator 3)) [ 1226.720821] __kmem_cache_shrink (mm/slub.c:3420) [ 1226.722711] slab_memory_callback (mm/slub.c:3464 mm/slub.c:3556) [ 1226.723925] notifier_call_chain (kernel/notifier.c:93) [ 1226.725244] __blocking_notifier_call_chain (kernel/notifier.c:319) [ 1226.726310] blocking_notifier_call_chain (kernel/notifier.c:329) [ 1226.727263] memory_notify (drivers/base/memory.c:177) [ 1226.728210] __offline_pages.constprop.23 (include/linux/notifier.h:179 mm/memory_hotplug.c:1682) [ 1226.731896] offline_pages (mm/memory_hotplug.c:1787) [ 1226.733095] memory_subsys_offline (drivers/base/memory.c:267 drivers/base/memory.c:304) [ 1226.735208] device_offline (drivers/base/core.c:1428) [ 1226.736966] online_store (drivers/base/core.c:451 (discriminator 2)) [ 1226.737720] dev_attr_store (drivers/base/core.c:138) [ 1226.739986] sysfs_kf_write (fs/sysfs/file.c:116) [ 1226.742105] kernfs_fop_write (fs/kernfs/file.c:308) [ 1226.743058] do_loop_readv_writev (fs/read_write.c:708) [ 1226.745095] do_readv_writev (fs/read_write.c:840) [ 1226.752769] vfs_writev (fs/read_write.c:879) [ 1226.753755] SyS_writev (fs/read_write.c:911 fs/read_write.c:903) [ 1226.754660] tracesys (arch/x86/kernel/entry_64.S:541)
No time to dig too much into this today, will work on it tomorrow.
Thanks,
Sasha
Quoted 37 lines · Sasha Levin
Hi all,
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:
[ 1226.701012] WARNING: CPU: 6 PID: 8624 at kernel/smp.c:673 on_each_cpu_cond+0x27f/0x2f0() [ 1226.703128] Modules linked in: [ 1226.703955] CPU: 6 PID: 8624 Comm: trinity-c26 Tainted: G W 3.16.0-rc7-next-20140801-sasha-00047-gd6ce559 #991 [ 1226.706792] ffff881b2e8a3000 0000000023830039 0000000000000000 ffff881b2e8a3000 [ 1226.708179] ffff881ada5b3978 ffffffffa1ee96b4 0000000000000000 ffff881ada5b39b8 [ 1226.709610] ffffffff9d1ce5ba ffffffff9d2c0ddf 000000000000001b ffff8807d7608000 [ 1226.710966] Call Trace: [ 1226.711417] dump_stack (lib/dump_stack.c:52) [ 1226.712311] warn_slowpath_common (kernel/panic.c:432) [ 1226.715425] warn_slowpath_null (kernel/panic.c:466) [ 1226.716436] on_each_cpu_cond (kernel/smp.c:673 (discriminator 3)) [ 1226.720821] __kmem_cache_shrink (mm/slub.c:3420) [ 1226.722711] slab_memory_callback (mm/slub.c:3464 mm/slub.c:3556) [ 1226.723925] notifier_call_chain (kernel/notifier.c:93) [ 1226.725244] __blocking_notifier_call_chain (kernel/notifier.c:319) [ 1226.726310] blocking_notifier_call_chain (kernel/notifier.c:329) [ 1226.727263] memory_notify (drivers/base/memory.c:177) [ 1226.728210] __offline_pages.constprop.23 (include/linux/notifier.h:179 mm/memory_hotplug.c:1682) [ 1226.731896] offline_pages (mm/memory_hotplug.c:1787) [ 1226.733095] memory_subsys_offline (drivers/base/memory.c:267 drivers/base/memory.c:304) [ 1226.735208] device_offline (drivers/base/core.c:1428) [ 1226.736966] online_store (drivers/base/core.c:451 (discriminator 2)) [ 1226.737720] dev_attr_store (drivers/base/core.c:138) [ 1226.739986] sysfs_kf_write (fs/sysfs/file.c:116) [ 1226.742105] kernfs_fop_write (fs/kernfs/file.c:308) [ 1226.743058] do_loop_readv_writev (fs/read_write.c:708) [ 1226.745095] do_readv_writev (fs/read_write.c:840) [ 1226.752769] vfs_writev (fs/read_write.c:879) [ 1226.753755] SyS_writev (fs/read_write.c:911 fs/read_write.c:903) [ 1226.754660] tracesys (arch/x86/kernel/entry_64.S:541)
Quoted 1 line
No time to dig too much into this today, will work on it tomorrow.
I think that that WARN_ON_ONCE() in kernel/smp.c:673 is wrong. smp_call_function_single() returns 0 if success so it should be WARN_ON_ONCE(ret) rather than WARN_ON_ONCE(!ret).
Thanks.
Hi Sasha and Kim,
I believe Kim is right and the WARN_ON_ONCE condition should be negated. Furthermore, I'm pretty sure I'm the one to blame for this bug.
Good catch and thank you both.
(Un)fortunately, I'm in a vacation in an island resort in Thailand at the moment and the beer is much better than the Internet connection.
I'll do my best to prepare, test and send a patch, but if someone beats me to it before my return in 10 days I will be happy to review.
Many thanks,
Gilad
Quoted 46 lines · Joonsoo Kim
Quoted 37 lines · Sasha Levin
Hi all,
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:
[ 1226.701012] WARNING: CPU: 6 PID: 8624 at kernel/smp.c:673 on_each_cpu_cond+0x27f/0x2f0() [ 1226.703128] Modules linked in: [ 1226.703955] CPU: 6 PID: 8624 Comm: trinity-c26 Tainted: G W 3.16.0-rc7-next-20140801-sasha-00047-gd6ce559 #991 [ 1226.706792] ffff881b2e8a3000 0000000023830039 0000000000000000 ffff881b2e8a3000 [ 1226.708179] ffff881ada5b3978 ffffffffa1ee96b4 0000000000000000 ffff881ada5b39b8 [ 1226.709610] ffffffff9d1ce5ba ffffffff9d2c0ddf 000000000000001b ffff8807d7608000 [ 1226.710966] Call Trace: [ 1226.711417] dump_stack (lib/dump_stack.c:52) [ 1226.712311] warn_slowpath_common (kernel/panic.c:432) [ 1226.715425] warn_slowpath_null (kernel/panic.c:466) [ 1226.716436] on_each_cpu_cond (kernel/smp.c:673 (discriminator 3)) [ 1226.720821] __kmem_cache_shrink (mm/slub.c:3420) [ 1226.722711] slab_memory_callback (mm/slub.c:3464 mm/slub.c:3556) [ 1226.723925] notifier_call_chain (kernel/notifier.c:93) [ 1226.725244] __blocking_notifier_call_chain (kernel/notifier.c:319) [ 1226.726310] blocking_notifier_call_chain (kernel/notifier.c:329) [ 1226.727263] memory_notify (drivers/base/memory.c:177) [ 1226.728210] __offline_pages.constprop.23 (include/linux/notifier.h:179 mm/memory_hotplug.c:1682) [ 1226.731896] offline_pages (mm/memory_hotplug.c:1787) [ 1226.733095] memory_subsys_offline (drivers/base/memory.c:267 drivers/base/memory.c:304) [ 1226.735208] device_offline (drivers/base/core.c:1428) [ 1226.736966] online_store (drivers/base/core.c:451 (discriminator 2)) [ 1226.737720] dev_attr_store (drivers/base/core.c:138) [ 1226.739986] sysfs_kf_write (fs/sysfs/file.c:116) [ 1226.742105] kernfs_fop_write (fs/kernfs/file.c:308) [ 1226.743058] do_loop_readv_writev (fs/read_write.c:708) [ 1226.745095] do_readv_writev (fs/read_write.c:840) [ 1226.752769] vfs_writev (fs/read_write.c:879) [ 1226.753755] SyS_writev (fs/read_write.c:911 fs/read_write.c:903) [ 1226.754660] tracesys (arch/x86/kernel/entry_64.S:541)Quoted 1 line
No time to dig too much into this today, will work on it tomorrow.
I think that that
WARN_ON_ONCE()inkernel/smp.c:673is wrong.smp_call_function_single()returns 0 if success so it should beWARN_ON_ONCE(ret)rather thanWARN_ON_ONCE(!ret).Thanks.
Quoted 2 lines · Sasha Levin
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:
Hmmm... Is this the new cpuset code that calling kmem_cache_shrink from the cpu notifier?
TJ?
Quoted 7 lines · Christoph Lameter
Quoted 2 lines · Sasha Levin
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:
Hmmm... Is this the new cpuset code that calling
kmem_cache_shrinkfrom the cpu notifier?
Hah? I don't see anything cpuset related there.
Signature · 1 line
tejun
Quoted 10 lines · Tejun Heo
Quoted 7 lines · Christoph Lameter
Quoted 2 lines · Sasha Levin
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:
Hmmm... Is this the new cpuset code that calling
kmem_cache_shrinkfrom the cpu notifier?Hah? I don't see anything cpuset related there.
Yuck. I meant cgroup code. There were some patches that were suggesting a shrink operation when taking down a cgroup. Does that also apply to per cpu downing?
Quoted 8 lines · Christoph Lameter
Quoted 6 lines · Tejun Heo
Quoted 5 lines · Christoph Lameter
Quoted 2 lines · Sasha Levin
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Hmmm... Is this the new cpuset code that calling
kmem_cache_shrinkfrom the cpu notifier?Hah? I don't see anything cpuset related there.
Yuck. I meant cgroup code. There were some patches that were suggesting a shrink operation when taking down a cgroup. Does that also apply to per cpu downing?
Heh, I have no idea what you're talking about. cgroup currently doesn't do anything like that. It doesn't even make use of the cpu hotplug hooks.
Thanks.
Signature · 1 line
tejun
Quoted 7 lines · Tejun Heo
Quoted 3 lines
Yuck. I meant cgroup code. There were some patches that were suggesting a shrink operation when taking down a cgroup. Does that also apply to per cpu downing?
Heh, I have no idea what you're talking about. cgroup currently doesn't do anything like that. It doesn't even make use of the cpu hotplug hooks.
Something is running a kmem_cache_shrink from cpu down hooks. Thats weird.
Quoted 6 lines · Christoph Lameter
Quoted 4 lines · Tejun Heo
Quoted 3 lines
Yuck. I meant cgroup code. There were some patches that were suggesting a shrink operation when taking down a cgroup. Does that also apply to per cpu downing?
Quoted 6 lines
Quoted 3 lines
Heh, I have no idea what you're talking about. cgroup currently doesn't do anything like that. It doesn't even make use of the cpu hotplug hooks.
Something is running a
kmem_cache_shrinkfrom cpu down hooks. Thats weird.
That's not cpu down, that's mem down.
Thanks,
Sasha
Quoted 35 lines · On Mon, 4 Aug 2014 15:48:23 +0900 Joonsoo Kim <[email protected]> wrote:
Quoted 35 lines
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:
[ 1226.701012] WARNING: CPU: 6 PID: 8624 at kernel/smp.c:673 on_each_cpu_cond+0x27f/0x2f0() [ 1226.703128] Modules linked in: [ 1226.703955] CPU: 6 PID: 8624 Comm: trinity-c26 Tainted: G W 3.16.0-rc7-next-20140801-sasha-00047-gd6ce559 #991 [ 1226.706792] ffff881b2e8a3000 0000000023830039 0000000000000000 ffff881b2e8a3000 [ 1226.708179] ffff881ada5b3978 ffffffffa1ee96b4 0000000000000000 ffff881ada5b39b8 [ 1226.709610] ffffffff9d1ce5ba ffffffff9d2c0ddf 000000000000001b ffff8807d7608000 [ 1226.710966] Call Trace: [ 1226.711417] dump_stack (lib/dump_stack.c:52) [ 1226.712311] warn_slowpath_common (kernel/panic.c:432) [ 1226.715425] warn_slowpath_null (kernel/panic.c:466) [ 1226.716436] on_each_cpu_cond (kernel/smp.c:673 (discriminator 3)) [ 1226.720821] __kmem_cache_shrink (mm/slub.c:3420) [ 1226.722711] slab_memory_callback (mm/slub.c:3464 mm/slub.c:3556) [ 1226.723925] notifier_call_chain (kernel/notifier.c:93) [ 1226.725244] __blocking_notifier_call_chain (kernel/notifier.c:319) [ 1226.726310] blocking_notifier_call_chain (kernel/notifier.c:329) [ 1226.727263] memory_notify (drivers/base/memory.c:177) [ 1226.728210] __offline_pages.constprop.23 (include/linux/notifier.h:179 mm/memory_hotplug.c:1682) [ 1226.731896] offline_pages (mm/memory_hotplug.c:1787) [ 1226.733095] memory_subsys_offline (drivers/base/memory.c:267 drivers/base/memory.c:304) [ 1226.735208] device_offline (drivers/base/core.c:1428) [ 1226.736966] online_store (drivers/base/core.c:451 (discriminator 2)) [ 1226.737720] dev_attr_store (drivers/base/core.c:138) [ 1226.739986] sysfs_kf_write (fs/sysfs/file.c:116) [ 1226.742105] kernfs_fop_write (fs/kernfs/file.c:308) [ 1226.743058] do_loop_readv_writev (fs/read_write.c:708) [ 1226.745095] do_readv_writev (fs/read_write.c:840) [ 1226.752769] vfs_writev (fs/read_write.c:879) [ 1226.753755] SyS_writev (fs/read_write.c:911 fs/read_write.c:903) [ 1226.754660] tracesys (arch/x86/kernel/entry_64.S:541)
Quoted 5 lines
Quoted 1 line
No time to dig too much into this today, will work on it tomorrow.
I think that that
WARN_ON_ONCE()inkernel/smp.c:673is wrong.smp_call_function_single()returns 0 if success so it should beWARN_ON_ONCE(ret)rather thanWARN_ON_ONCE(!ret).
Surely true.
From: Andrew Morton <[email protected]>
Subject: kernel/smp.c:on_each_cpu_cond(): fix warning in fallback path
The rarely-executed memry-allocation-failed callback path generates a WARN_ON_ONCE() when smp_call_function_single() succeeds. Presumably it's supposed to warn on failures.
---
kernel/smp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/smp.c renamed from kernel/smp.c~a +1 -1
@@ -661,7 +661,7 @@ void on_each_cpu_cond(bool (*cond_func)(if (cond_func (cpu, info)) { ret =smp_call_function_single (cpu, func, info, wait); -WARN_ON_ONCE (!ret); +WARN_ON_ONCE (ret); }preempt_enable (); }
_