2.6.19-rc5-mm2 (end earlier): WARNING at lib/kobject.c:172 kobject_init() on resume from disk
Hi,
I get similar traces on every resume from disk on SMP systems:
WARNING at lib/kobject.c:172 kobject_init()
Call Trace: [<ffffffff80265559>] dump_trace+0xaa/0x3fd [<ffffffff802658e8>] show_trace+0x3c/0x52 [<ffffffff80265913>] dump_stack+0x15/0x17 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a [<ffffffff8031c298>] kobject_register+0x1a/0x3e [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec [<ffffffff8026af39>] mce_create_device+0x79/0x103 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb [<ffffffff80299f18>] _cpu_up+0xc2/0xd5 [<ffffffff80299f56>] cpu_up+0x2b/0x42 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82 [<ffffffff8025811e>] system_call+0x7e/0x83 DWARF2 unwinder stuck at system_call+0x7e/0x83
Leftover inexact backtrace:
False positive?
Signature · 2 lines
You never change things by fighting the existing reality. R. Buckminster Fuller
Quoted 26 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:
Hi,
I get similar traces on every resume from disk on SMP systems:
WARNING at
lib/kobject.c:172kobject_init()Call Trace: [<ffffffff80265559>] dump_trace+0xaa/0x3fd [<ffffffff802658e8>] show_trace+0x3c/0x52 [<ffffffff80265913>] dump_stack+0x15/0x17 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a [<ffffffff8031c298>] kobject_register+0x1a/0x3e [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec [<ffffffff8026af39>] mce_create_device+0x79/0x103 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb [<ffffffff80299f18>] _cpu_up+0xc2/0xd5 [<ffffffff80299f56>] cpu_up+0x2b/0x42 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82 [<ffffffff8025811e>] system_call+0x7e/0x83 DWARF2 unwinder stuck at system_call+0x7e/0x83
Quoted 4 lines
Leftover inexact backtrace:
False positive?
Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.
Perhaps mce_remove_device() isn't being called.
<wonders why mce_remove_device() isn't __cpuinit too>
Quoted 31 lines · Andrew Morton
Quoted 28 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:
Hi,
I get similar traces on every resume from disk on SMP systems:
WARNING at
lib/kobject.c:172kobject_init()Call Trace: [<ffffffff80265559>] dump_trace+0xaa/0x3fd [<ffffffff802658e8>] show_trace+0x3c/0x52 [<ffffffff80265913>] dump_stack+0x15/0x17 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a [<ffffffff8031c298>] kobject_register+0x1a/0x3e [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec [<ffffffff8026af39>] mce_create_device+0x79/0x103 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb [<ffffffff80299f18>] _cpu_up+0xc2/0xd5 [<ffffffff80299f56>] cpu_up+0x2b/0x42 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82 [<ffffffff8025811e>] system_call+0x7e/0x83 DWARF2 unwinder stuck at system_call+0x7e/0x83Leftover inexact backtrace:
Quoted 6 lines
Quoted 2 lines
False positive?
Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.
Heh, I would think that it's a big "obvious", oh well.
There are 2 changes in this patch.
First one is to ensure that the kobject is properly initialized _before_ kobject_init() is called. Yeah, seems funny, right? Turns out this has caught a lot of issues where kobject_init() is called twice on the same object, not a good thing at all.
And this looks like that is exactly what is happening here, as you mention:
Quoted 1 line
Perhaps
mce_remove_device()isn't being called.
That's probably the issue.
The second change in that patch tries to enforce the "everything needs a release() function" rule for kobjects, but it turns out, a lot of static kobjects trigger this inproperly (struct bus and friends), so that can't go to mainline, and it only shows up if you enable CONFIG_KOBJECT_DEBUG.
Hope this helps explain things, I'll go off and edit the text there to make it clearer.
thanks,
greg k-h
Quoted 51 lines · Greg KH
Quoted 31 lines · Andrew Morton
Quoted 28 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:
Hi,
I get similar traces on every resume from disk on SMP systems:
WARNING at
lib/kobject.c:172kobject_init()Call Trace: [<ffffffff80265559>] dump_trace+0xaa/0x3fd [<ffffffff802658e8>] show_trace+0x3c/0x52 [<ffffffff80265913>] dump_stack+0x15/0x17 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a [<ffffffff8031c298>] kobject_register+0x1a/0x3e [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec [<ffffffff8026af39>] mce_create_device+0x79/0x103 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb [<ffffffff80299f18>] _cpu_up+0xc2/0xd5 [<ffffffff80299f56>] cpu_up+0x2b/0x42 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82 [<ffffffff8025811e>] system_call+0x7e/0x83 DWARF2 unwinder stuck at system_call+0x7e/0x83Leftover inexact backtrace:
Quoted 6 lines
Quoted 2 lines
False positive?
Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.Heh, I would think that it's a big "obvious", oh well.
There are 2 changes in this patch.
First one is to ensure that the kobject is properly initialized _before_
kobject_init()is called. Yeah, seems funny, right? Turns out this has caught a lot of issues wherekobject_init()is called twice on the same object, not a good thing at all.And this looks like that is exactly what is happening here, as you mention:
Quoted 8 lines
Quoted 1 line
Perhaps
mce_remove_device()isn't being called.That's probably the issue.
The second change in that patch tries to enforce the "everything needs a
release()function" rule for kobjects, but it turns out, a lot of static kobjects trigger this inproperly (struct bus and friends), so that can't go to mainline, and it only shows up if you enableCONFIG_KOBJECT_DEBUG.
CONFIG_DEBUG_KOBJECT is not set here, so it looks like kobject_init() is being called for the second time on the same object.
Greetings,
Rafael
Signature · 2 lines
You never change things by fighting the existing reality. R. Buckminster Fuller
Quoted 31 lines · Andrew Morton
Quoted 28 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:
Hi,
I get similar traces on every resume from disk on SMP systems:
WARNING at
lib/kobject.c:172kobject_init()Call Trace: [<ffffffff80265559>] dump_trace+0xaa/0x3fd [<ffffffff802658e8>] show_trace+0x3c/0x52 [<ffffffff80265913>] dump_stack+0x15/0x17 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a [<ffffffff8031c298>] kobject_register+0x1a/0x3e [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec [<ffffffff8026af39>] mce_create_device+0x79/0x103 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb [<ffffffff80299f18>] _cpu_up+0xc2/0xd5 [<ffffffff80299f56>] cpu_up+0x2b/0x42 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82 [<ffffffff8025811e>] system_call+0x7e/0x83 DWARF2 unwinder stuck at system_call+0x7e/0x83Leftover inexact backtrace:
Quoted 8 lines
Quoted 2 lines
False positive?
Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.Perhaps
mce_remove_device()isn't being called.
I've added some debugging code into mce_remove_device() which shows that it is being called when the CPU is removed.
Investigation continues.
Quoted 39 lines · Rafael J. Wysocki
Quoted 38 lines · Andrew Morton
Quoted 26 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:
Hi,
I get similar traces on every resume from disk on SMP systems:
WARNING at
lib/kobject.c:172kobject_init()Call Trace: [<ffffffff80265559>] dump_trace+0xaa/0x3fd [<ffffffff802658e8>] show_trace+0x3c/0x52 [<ffffffff80265913>] dump_stack+0x15/0x17 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a [<ffffffff8031c298>] kobject_register+0x1a/0x3e [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec [<ffffffff8026af39>] mce_create_device+0x79/0x103 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb [<ffffffff80299f18>] _cpu_up+0xc2/0xd5 [<ffffffff80299f56>] cpu_up+0x2b/0x42 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82 [<ffffffff8025811e>] system_call+0x7e/0x83 DWARF2 unwinder stuck at system_call+0x7e/0x83Quoted 4 lines
Leftover inexact backtrace:
False positive?
Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.
Quoted 6 lines
Quoted 1 line
Perhaps
mce_remove_device()isn't being called.I've added some debugging code into
mce_remove_device()which shows that it is being called when the CPU is removed.Investigation continues.
Ah, I think the problem is that the last user of a kobject doesn't decrease the refcount in kref_put(), so if the same kobject is registered for the second time, the refcount is still one and the warning triggers.
So, it seems, this is a false positive and I think we can get rid of it in the following way (tested and works):
---
Make mce_remove_device() clean up the kobject in per_cpu(device_mce, cpu) after it has been unregistered.
--- arch/x86_64/kernel/mce.c | 1 + 1 file changed, 1 insertion(+)
arch/x86_64/kernel/mce.c +1 -0
@@ -651,6 +651,7 @@ static void mce_remove_device(unsigned isysdev_remove_file (&per_cpu (device_mce,cpu), &attr_tolerant);sysdev_remove_file (&per_cpu (device_mce,cpu), &attr_check_interval);sysdev_unregister (&per_cpu (device_mce,cpu)); +per_cpu (device_mce, cpu).kobj = (struct kobject ){0 }; }/* Get notified when a cpu comes on/off. Be hotplug friendly. */
Quoted 42 lines · Rafael J. Wysocki
Quoted 41 lines · Rafael J. Wysocki
Quoted 39 lines · Andrew Morton
Quoted 26 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:
Hi,
I get similar traces on every resume from disk on SMP systems:
WARNING at
lib/kobject.c:172kobject_init()Call Trace: [<ffffffff80265559>] dump_trace+0xaa/0x3fd [<ffffffff802658e8>] show_trace+0x3c/0x52 [<ffffffff80265913>] dump_stack+0x15/0x17 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a [<ffffffff8031c298>] kobject_register+0x1a/0x3e [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec [<ffffffff8026af39>] mce_create_device+0x79/0x103 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb [<ffffffff80299f18>] _cpu_up+0xc2/0xd5 [<ffffffff80299f56>] cpu_up+0x2b/0x42 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82 [<ffffffff8025811e>] system_call+0x7e/0x83 DWARF2 unwinder stuck at system_call+0x7e/0x83Quoted 4 lines
Leftover inexact backtrace:
False positive?
Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.Perhaps
mce_remove_device()isn't being called.
Quoted 8 lines
Quoted 4 lines
I've added some debugging code into
mce_remove_device()which shows that it is being called when the CPU is removed.Investigation continues.
Ah, I think the problem is that the last user of a kobject doesn't decrease the refcount in
kref_put(), so if the same kobject is registered for the second time, the refcount is still one and the warning triggers.
But the last user of the kobject should cause the kobject to be freed and disappear. It should not hang around, right?
Oh yuck, this is a static struct device, one per cpu :(
Quoted 17 lines
So, it seems, this is a false positive and I think we can get rid of it in the following way (tested and works):
---
Makemce_remove_device()clean up the kobject inper_cpu(device_mce, cpu)after it has been unregistered.Signed-off-by Rafael J. Wysocki <[email protected]>--- arch/x86_64/kernel/mce.c | 1 + 1 file changed, 1 insertion(+)Index:
linux-2.6.19-rc6-mm1/arch/x86_64/kernel/mce.c
===================================================================
arch/x86_64/kernel/mce.c static void mce_remove_device(unsigned i · line 654
Quoted 4 lines
arch/x86_64/kernel/mce.c +1
-0@@ -651,6 +651,7 @@ static void mce_remove_device(unsigned isysdev_remove_file (&per_cpu (device_mce,cpu), &attr_tolerant);sysdev_remove_file (&per_cpu (device_mce,cpu), &attr_check_interval);sysdev_unregister (&per_cpu (device_mce,cpu)); +per_cpu (device_mce, cpu).kobj = (struct kobject ){0 };
memset the kobj instead perhaps? Yeah, I guess this copy will work, as the compiler turns it into a memset.
But overall, ick :(
thanks,
greg k-h
Quoted 69 lines · Greg KH
Quoted 42 lines · Rafael J. Wysocki
Quoted 41 lines · Rafael J. Wysocki
Quoted 39 lines · Andrew Morton
Quoted 26 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:
Hi,
I get similar traces on every resume from disk on SMP systems:
WARNING at
lib/kobject.c:172kobject_init()Call Trace: [<ffffffff80265559>] dump_trace+0xaa/0x3fd [<ffffffff802658e8>] show_trace+0x3c/0x52 [<ffffffff80265913>] dump_stack+0x15/0x17 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a [<ffffffff8031c298>] kobject_register+0x1a/0x3e [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec [<ffffffff8026af39>] mce_create_device+0x79/0x103 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb [<ffffffff80299f18>] _cpu_up+0xc2/0xd5 [<ffffffff80299f56>] cpu_up+0x2b/0x42 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82 [<ffffffff8025811e>] system_call+0x7e/0x83 DWARF2 unwinder stuck at system_call+0x7e/0x83Quoted 4 lines
Leftover inexact backtrace:
False positive?
Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.Perhaps
mce_remove_device()isn't being called.Quoted 8 lines
Quoted 4 lines
I've added some debugging code into
mce_remove_device()which shows that it is being called when the CPU is removed.Investigation continues.
Ah, I think the problem is that the last user of a kobject doesn't decrease the refcount in
kref_put(), so if the same kobject is registered for the second time, the refcount is still one and the warning triggers.But the last user of the kobject should cause the kobject to be freed and disappear. It should not hang around, right?
Oh yuck, this is a static struct device, one per cpu :(
Quoted 12 lines
So, it seems, this is a false positive and I think we can get rid of it in the following way (tested and works):
---
Makemce_remove_device()clean up the kobject inper_cpu(device_mce, cpu)after it has been unregistered.Signed-off-by Rafael J. Wysocki <[email protected]>--- arch/x86_64/kernel/mce.c | 1 + 1 file changed, 1 insertion(+)
Quoted 12 lines
arch/x86_64/kernel/mce.c static void mce_remove_device(unsigned i · line 654
Quoted 9 lines
Index:
linux-2.6.19-rc6-mm1/arch/x86_64/kernel/mce.c
===================================================================arch/x86_64/kernel/mce.c +1
-0@@ -651,6 +651,7 @@ static void mce_remove_device(unsigned isysdev_remove_file (&per_cpu (device_mce,cpu), &attr_tolerant);sysdev_remove_file (&per_cpu (device_mce,cpu), &attr_check_interval);sysdev_unregister (&per_cpu (device_mce,cpu)); +per_cpu (device_mce, cpu).kobj = (struct kobject ){0 };memset the kobj instead perhaps? Yeah, I guess this copy will work, as the compiler turns it into a memset.
Patch with the memset follows.
BTW, it seems to me that the WARN_ON in kref_get will never trigger, will it?
Greetings,
Rafael
---
Make mce_remove_device() clean up the kobject in per_cpu(device_mce, cpu) after it has been unregistered.
--- arch/x86_64/kernel/mce.c | 1 + 1 file changed, 1 insertion(+)
arch/x86_64/kernel/mce.c +1 -0
@@ -651,6 +651,7 @@ static void mce_remove_device(unsigned isysdev_remove_file (&per_cpu (device_mce,cpu), &attr_tolerant);sysdev_remove_file (&per_cpu (device_mce,cpu), &attr_check_interval);sysdev_unregister (&per_cpu (device_mce,cpu)); +memset (&per_cpu (device_mce, cpu).kobj,0 ,sizeof (struct kobject )); }/* Get notified when a cpu comes on/off. Be hotplug friendly. */
Quoted 2 lines
Patch with the memset follows.
Added thanks
-Andi