Skip to contentSkip to conversation

kernel: smp: WARNING at kernel/smp.c:673

DISCUSSION Sasha Levin 6 participants 10 messages Aug 4, 2014

lore
Share & tools
lore
#0 lore raw

Hi all,

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:

[ 1226.701012] WARNING: CPU: 6 PID: 8624 at kernel/smp.c:673 on_each_cpu_cond+0x27f/0x2f0()
[ 1226.703128] Modules linked in:
[ 1226.703955] CPU: 6 PID: 8624 Comm: trinity-c26 Tainted: G        W      3.16.0-rc7-next-20140801-sasha-00047-gd6ce559 #991
[ 1226.706792]  ffff881b2e8a3000 0000000023830039 0000000000000000 ffff881b2e8a3000
[ 1226.708179]  ffff881ada5b3978 ffffffffa1ee96b4 0000000000000000 ffff881ada5b39b8
[ 1226.709610]  ffffffff9d1ce5ba ffffffff9d2c0ddf 000000000000001b ffff8807d7608000
[ 1226.710966] Call Trace:
[ 1226.711417] dump_stack (lib/dump_stack.c:52)
[ 1226.712311] warn_slowpath_common (kernel/panic.c:432)
[ 1226.715425] warn_slowpath_null (kernel/panic.c:466)
[ 1226.716436] on_each_cpu_cond (kernel/smp.c:673 (discriminator 3))
[ 1226.720821] __kmem_cache_shrink (mm/slub.c:3420)
[ 1226.722711] slab_memory_callback (mm/slub.c:3464 mm/slub.c:3556)
[ 1226.723925] notifier_call_chain (kernel/notifier.c:93)
[ 1226.725244] __blocking_notifier_call_chain (kernel/notifier.c:319)
[ 1226.726310] blocking_notifier_call_chain (kernel/notifier.c:329)
[ 1226.727263] memory_notify (drivers/base/memory.c:177)
[ 1226.728210] __offline_pages.constprop.23 (include/linux/notifier.h:179 mm/memory_hotplug.c:1682)
[ 1226.731896] offline_pages (mm/memory_hotplug.c:1787)
[ 1226.733095] memory_subsys_offline (drivers/base/memory.c:267 drivers/base/memory.c:304)
[ 1226.735208] device_offline (drivers/base/core.c:1428)
[ 1226.736966] online_store (drivers/base/core.c:451 (discriminator 2))
[ 1226.737720] dev_attr_store (drivers/base/core.c:138)
[ 1226.739986] sysfs_kf_write (fs/sysfs/file.c:116)
[ 1226.742105] kernfs_fop_write (fs/kernfs/file.c:308)
[ 1226.743058] do_loop_readv_writev (fs/read_write.c:708)
[ 1226.745095] do_readv_writev (fs/read_write.c:840)
[ 1226.752769] vfs_writev (fs/read_write.c:879)
[ 1226.753755] SyS_writev (fs/read_write.c:911 fs/read_write.c:903)
[ 1226.754660] tracesys (arch/x86/kernel/entry_64.S:541)

No time to dig too much into this today, will work on it tomorrow.

Thanks,
Sasha

#1 lore raw
Quoted 37 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:

[ 1226.701012] WARNING: CPU: 6 PID: 8624 at kernel/smp.c:673 on_each_cpu_cond+0x27f/0x2f0()
[ 1226.703128] Modules linked in:
[ 1226.703955] CPU: 6 PID: 8624 Comm: trinity-c26 Tainted: G        W      3.16.0-rc7-next-20140801-sasha-00047-gd6ce559 #991
[ 1226.706792]  ffff881b2e8a3000 0000000023830039 0000000000000000 ffff881b2e8a3000
[ 1226.708179]  ffff881ada5b3978 ffffffffa1ee96b4 0000000000000000 ffff881ada5b39b8
[ 1226.709610]  ffffffff9d1ce5ba ffffffff9d2c0ddf 000000000000001b ffff8807d7608000
[ 1226.710966] Call Trace:
[ 1226.711417] dump_stack (lib/dump_stack.c:52)
[ 1226.712311] warn_slowpath_common (kernel/panic.c:432)
[ 1226.715425] warn_slowpath_null (kernel/panic.c:466)
[ 1226.716436] on_each_cpu_cond (kernel/smp.c:673 (discriminator 3))
[ 1226.720821] __kmem_cache_shrink (mm/slub.c:3420)
[ 1226.722711] slab_memory_callback (mm/slub.c:3464 mm/slub.c:3556)
[ 1226.723925] notifier_call_chain (kernel/notifier.c:93)
[ 1226.725244] __blocking_notifier_call_chain (kernel/notifier.c:319)
[ 1226.726310] blocking_notifier_call_chain (kernel/notifier.c:329)
[ 1226.727263] memory_notify (drivers/base/memory.c:177)
[ 1226.728210] __offline_pages.constprop.23 (include/linux/notifier.h:179 mm/memory_hotplug.c:1682)
[ 1226.731896] offline_pages (mm/memory_hotplug.c:1787)
[ 1226.733095] memory_subsys_offline (drivers/base/memory.c:267 drivers/base/memory.c:304)
[ 1226.735208] device_offline (drivers/base/core.c:1428)
[ 1226.736966] online_store (drivers/base/core.c:451 (discriminator 2))
[ 1226.737720] dev_attr_store (drivers/base/core.c:138)
[ 1226.739986] sysfs_kf_write (fs/sysfs/file.c:116)
[ 1226.742105] kernfs_fop_write (fs/kernfs/file.c:308)
[ 1226.743058] do_loop_readv_writev (fs/read_write.c:708)
[ 1226.745095] do_readv_writev (fs/read_write.c:840)
[ 1226.752769] vfs_writev (fs/read_write.c:879)
[ 1226.753755] SyS_writev (fs/read_write.c:911 fs/read_write.c:903)
[ 1226.754660] tracesys (arch/x86/kernel/entry_64.S:541)
Quoted 1 line

No time to dig too much into this today, will work on it tomorrow.

I think that that WARN_ON_ONCE() in kernel/smp.c:673 is wrong. smp_call_function_single() returns 0 if success so it should be WARN_ON_ONCE(ret) rather than WARN_ON_ONCE(!ret).

Thanks.

#2 lore raw

Hi Sasha and Kim,

I believe Kim is right and the WARN_ON_ONCE condition should be negated. Furthermore, I'm pretty sure I'm the one to blame for this bug.

Good catch and thank you both.

(Un)fortunately, I'm in a vacation in an island resort in Thailand at the moment and the beer is much better than the Internet connection.

I'll do my best to prepare, test and send a patch, but if someone beats me to it before my return in 10 days I will be happy to review.

Many thanks,
Gilad

Quoted 46 lines · Joonsoo Kim
Quoted 37 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:

[ 1226.701012] WARNING: CPU: 6 PID: 8624 at kernel/smp.c:673 on_each_cpu_cond+0x27f/0x2f0()
[ 1226.703128] Modules linked in:
[ 1226.703955] CPU: 6 PID: 8624 Comm: trinity-c26 Tainted: G        W      3.16.0-rc7-next-20140801-sasha-00047-gd6ce559 #991
[ 1226.706792]  ffff881b2e8a3000 0000000023830039 0000000000000000 ffff881b2e8a3000
[ 1226.708179]  ffff881ada5b3978 ffffffffa1ee96b4 0000000000000000 ffff881ada5b39b8
[ 1226.709610]  ffffffff9d1ce5ba ffffffff9d2c0ddf 000000000000001b ffff8807d7608000
[ 1226.710966] Call Trace:
[ 1226.711417] dump_stack (lib/dump_stack.c:52)
[ 1226.712311] warn_slowpath_common (kernel/panic.c:432)
[ 1226.715425] warn_slowpath_null (kernel/panic.c:466)
[ 1226.716436] on_each_cpu_cond (kernel/smp.c:673 (discriminator 3))
[ 1226.720821] __kmem_cache_shrink (mm/slub.c:3420)
[ 1226.722711] slab_memory_callback (mm/slub.c:3464 mm/slub.c:3556)
[ 1226.723925] notifier_call_chain (kernel/notifier.c:93)
[ 1226.725244] __blocking_notifier_call_chain (kernel/notifier.c:319)
[ 1226.726310] blocking_notifier_call_chain (kernel/notifier.c:329)
[ 1226.727263] memory_notify (drivers/base/memory.c:177)
[ 1226.728210] __offline_pages.constprop.23 (include/linux/notifier.h:179 mm/memory_hotplug.c:1682)
[ 1226.731896] offline_pages (mm/memory_hotplug.c:1787)
[ 1226.733095] memory_subsys_offline (drivers/base/memory.c:267 drivers/base/memory.c:304)
[ 1226.735208] device_offline (drivers/base/core.c:1428)
[ 1226.736966] online_store (drivers/base/core.c:451 (discriminator 2))
[ 1226.737720] dev_attr_store (drivers/base/core.c:138)
[ 1226.739986] sysfs_kf_write (fs/sysfs/file.c:116)
[ 1226.742105] kernfs_fop_write (fs/kernfs/file.c:308)
[ 1226.743058] do_loop_readv_writev (fs/read_write.c:708)
[ 1226.745095] do_readv_writev (fs/read_write.c:840)
[ 1226.752769] vfs_writev (fs/read_write.c:879)
[ 1226.753755] SyS_writev (fs/read_write.c:911 fs/read_write.c:903)
[ 1226.754660] tracesys (arch/x86/kernel/entry_64.S:541)
Quoted 1 line

No time to dig too much into this today, will work on it tomorrow.

I think that that WARN_ON_ONCE() in kernel/smp.c:673 is wrong. smp_call_function_single() returns 0 if success so it should be WARN_ON_ONCE(ret) rather than WARN_ON_ONCE(!ret).

Thanks.

#4 lore raw
In reply to Sasha Levin
Quoted 2 lines · Sasha Levin

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:

Hmmm... Is this the new cpuset code that calling kmem_cache_shrink from the cpu notifier?

TJ?

#5 lore raw
Quoted 7 lines · Christoph Lameter
Quoted 2 lines · Sasha Levin

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:

Hmmm... Is this the new cpuset code that calling kmem_cache_shrink from the cpu notifier?

Hah? I don't see anything cpuset related there.

Signature · 1 line
tejun
#6 lore raw
Quoted 10 lines · Tejun Heo
Quoted 7 lines · Christoph Lameter
Quoted 2 lines · Sasha Levin

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:

Hmmm... Is this the new cpuset code that calling kmem_cache_shrink from the cpu notifier?

Hah? I don't see anything cpuset related there.

Yuck. I meant cgroup code. There were some patches that were suggesting a shrink operation when taking down a cgroup. Does that also apply to per cpu downing?

#7 lore raw
Quoted 8 lines · Christoph Lameter
Quoted 6 lines · Tejun Heo
Quoted 5 lines · Christoph Lameter
Quoted 2 lines · Sasha Levin

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Hmmm... Is this the new cpuset code that calling kmem_cache_shrink from the cpu notifier?

Hah? I don't see anything cpuset related there.

Yuck. I meant cgroup code. There were some patches that were suggesting a shrink operation when taking down a cgroup. Does that also apply to per cpu downing?

Heh, I have no idea what you're talking about. cgroup currently doesn't do anything like that. It doesn't even make use of the cpu hotplug hooks.

Thanks.

Signature · 1 line
tejun
#8 lore raw
Quoted 7 lines · Tejun Heo
Quoted 3 lines

Yuck. I meant cgroup code. There were some patches that were suggesting a shrink operation when taking down a cgroup. Does that also apply to per cpu downing?

Heh, I have no idea what you're talking about. cgroup currently doesn't do anything like that. It doesn't even make use of the cpu hotplug hooks.

Something is running a kmem_cache_shrink from cpu down hooks. Thats weird.

#9 lore raw
Quoted 6 lines · Christoph Lameter
Quoted 4 lines · Tejun Heo
Quoted 3 lines

Yuck. I meant cgroup code. There were some patches that were suggesting a shrink operation when taking down a cgroup. Does that also apply to per cpu downing?

Quoted 6 lines
Quoted 3 lines

Heh, I have no idea what you're talking about. cgroup currently doesn't do anything like that. It doesn't even make use of the cpu hotplug hooks.

Something is running a kmem_cache_shrink from cpu down hooks. Thats weird.

That's not cpu down, that's mem down.

Thanks,
Sasha

#3 lore raw
In reply to Joonsoo Kim
Quoted 35 lines · On Mon, 4 Aug 2014 15:48:23 +0900 Joonsoo Kim <[email protected]> wrote:
Quoted 35 lines

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:

[ 1226.701012] WARNING: CPU: 6 PID: 8624 at kernel/smp.c:673 on_each_cpu_cond+0x27f/0x2f0()
[ 1226.703128] Modules linked in:
[ 1226.703955] CPU: 6 PID: 8624 Comm: trinity-c26 Tainted: G        W      3.16.0-rc7-next-20140801-sasha-00047-gd6ce559 #991
[ 1226.706792]  ffff881b2e8a3000 0000000023830039 0000000000000000 ffff881b2e8a3000
[ 1226.708179]  ffff881ada5b3978 ffffffffa1ee96b4 0000000000000000 ffff881ada5b39b8
[ 1226.709610]  ffffffff9d1ce5ba ffffffff9d2c0ddf 000000000000001b ffff8807d7608000
[ 1226.710966] Call Trace:
[ 1226.711417] dump_stack (lib/dump_stack.c:52)
[ 1226.712311] warn_slowpath_common (kernel/panic.c:432)
[ 1226.715425] warn_slowpath_null (kernel/panic.c:466)
[ 1226.716436] on_each_cpu_cond (kernel/smp.c:673 (discriminator 3))
[ 1226.720821] __kmem_cache_shrink (mm/slub.c:3420)
[ 1226.722711] slab_memory_callback (mm/slub.c:3464 mm/slub.c:3556)
[ 1226.723925] notifier_call_chain (kernel/notifier.c:93)
[ 1226.725244] __blocking_notifier_call_chain (kernel/notifier.c:319)
[ 1226.726310] blocking_notifier_call_chain (kernel/notifier.c:329)
[ 1226.727263] memory_notify (drivers/base/memory.c:177)
[ 1226.728210] __offline_pages.constprop.23 (include/linux/notifier.h:179 mm/memory_hotplug.c:1682)
[ 1226.731896] offline_pages (mm/memory_hotplug.c:1787)
[ 1226.733095] memory_subsys_offline (drivers/base/memory.c:267 drivers/base/memory.c:304)
[ 1226.735208] device_offline (drivers/base/core.c:1428)
[ 1226.736966] online_store (drivers/base/core.c:451 (discriminator 2))
[ 1226.737720] dev_attr_store (drivers/base/core.c:138)
[ 1226.739986] sysfs_kf_write (fs/sysfs/file.c:116)
[ 1226.742105] kernfs_fop_write (fs/kernfs/file.c:308)
[ 1226.743058] do_loop_readv_writev (fs/read_write.c:708)
[ 1226.745095] do_readv_writev (fs/read_write.c:840)
[ 1226.752769] vfs_writev (fs/read_write.c:879)
[ 1226.753755] SyS_writev (fs/read_write.c:911 fs/read_write.c:903)
[ 1226.754660] tracesys (arch/x86/kernel/entry_64.S:541)
Quoted 5 lines
Quoted 1 line

No time to dig too much into this today, will work on it tomorrow.

I think that that WARN_ON_ONCE() in kernel/smp.c:673 is wrong. smp_call_function_single() returns 0 if success so it should be WARN_ON_ONCE(ret) rather than WARN_ON_ONCE(!ret).

Surely true.

From: Andrew Morton <[email protected]>
Subject: kernel/smp.c:on_each_cpu_cond(): fix warning in fallback path

The rarely-executed memry-allocation-failed callback path generates a WARN_ON_ONCE() when smp_call_function_single() succeeds. Presumably it's supposed to warn on failures.

Cc Christoph Lameter <[email protected]>
Cc Gilad Ben-Yossef <[email protected]>
Cc David Rientjes <[email protected]>
Cc Joonsoo Kim <[email protected]>
Cc Tejun Heo <[email protected]>

---

 kernel/smp.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/smp.c renamed from kernel/smp.c~a +1 -1
diff -puN kernel/smp.c~a kernel/smp.c
@@ -661,7 +661,7 @@ void on_each_cpu_cond(bool (*cond_func)(
 			if (cond_func(cpu, info)) {
 				ret = smp_call_function_single(cpu, func,
 								info, wait);
-				WARN_ON_ONCE(!ret);
+				WARN_ON_ONCE(ret);
 			}
 		preempt_enable();
 	}

_