Skip to contentSkip to conversation

2.6.19-rc5-mm2 (end earlier): WARNING at lib/kobject.c:172 kobject_init() on resume from disk

DISCUSSION Rafael J. Wysocki 4 participants 9 messages Nov 22, 2006 – Nov 26, 2006

lore
Share & tools
lore
#0 lore raw

Hi,

I get similar traces on every resume from disk on SMP systems:

WARNING at lib/kobject.c:172 kobject_init()

Call Trace:
 [<ffffffff80265559>] dump_trace+0xaa/0x3fd
 [<ffffffff802658e8>] show_trace+0x3c/0x52
 [<ffffffff80265913>] dump_stack+0x15/0x17
 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a
 [<ffffffff8031c298>] kobject_register+0x1a/0x3e
 [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec
 [<ffffffff8026af39>] mce_create_device+0x79/0x103
 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd
 [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e
 [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb
 [<ffffffff80299f18>] _cpu_up+0xc2/0xd5
 [<ffffffff80299f56>] cpu_up+0x2b/0x42
 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b
 [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2
 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77
 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273
 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82
 [<ffffffff8025811e>] system_call+0x7e/0x83
DWARF2 unwinder stuck at system_call+0x7e/0x83

Leftover inexact backtrace:

False positive?

Signature · 2 lines
You never change things by fighting the existing reality.
		R. Buckminster Fuller
#1 lore raw
Quoted 26 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Hi,

I get similar traces on every resume from disk on SMP systems:

WARNING at lib/kobject.c:172 kobject_init()

Call Trace:
 [<ffffffff80265559>] dump_trace+0xaa/0x3fd
 [<ffffffff802658e8>] show_trace+0x3c/0x52
 [<ffffffff80265913>] dump_stack+0x15/0x17
 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a
 [<ffffffff8031c298>] kobject_register+0x1a/0x3e
 [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec
 [<ffffffff8026af39>] mce_create_device+0x79/0x103
 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd
 [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e
 [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb
 [<ffffffff80299f18>] _cpu_up+0xc2/0xd5
 [<ffffffff80299f56>] cpu_up+0x2b/0x42
 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b
 [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2
 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77
 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273
 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82
 [<ffffffff8025811e>] system_call+0x7e/0x83
DWARF2 unwinder stuck at system_call+0x7e/0x83
Quoted 4 lines

Leftover inexact backtrace:

False positive?

Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.

Perhaps mce_remove_device() isn't being called.

<wonders why mce_remove_device() isn't __cpuinit too>

#2 lore raw
Quoted 31 lines · Andrew Morton
Quoted 28 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Hi,

I get similar traces on every resume from disk on SMP systems:

WARNING at lib/kobject.c:172 kobject_init()

Call Trace:
 [<ffffffff80265559>] dump_trace+0xaa/0x3fd
 [<ffffffff802658e8>] show_trace+0x3c/0x52
 [<ffffffff80265913>] dump_stack+0x15/0x17
 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a
 [<ffffffff8031c298>] kobject_register+0x1a/0x3e
 [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec
 [<ffffffff8026af39>] mce_create_device+0x79/0x103
 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd
 [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e
 [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb
 [<ffffffff80299f18>] _cpu_up+0xc2/0xd5
 [<ffffffff80299f56>] cpu_up+0x2b/0x42
 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b
 [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2
 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77
 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273
 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82
 [<ffffffff8025811e>] system_call+0x7e/0x83
DWARF2 unwinder stuck at system_call+0x7e/0x83

Leftover inexact backtrace:

Quoted 6 lines
Quoted 2 lines

False positive?

Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.

Heh, I would think that it's a big "obvious", oh well.

There are 2 changes in this patch.

First one is to ensure that the kobject is properly initialized _before_ kobject_init() is called. Yeah, seems funny, right? Turns out this has caught a lot of issues where kobject_init() is called twice on the same object, not a good thing at all.

And this looks like that is exactly what is happening here, as you mention:

Quoted 1 line

Perhaps mce_remove_device() isn't being called.

That's probably the issue.

The second change in that patch tries to enforce the "everything needs a release() function" rule for kobjects, but it turns out, a lot of static kobjects trigger this inproperly (struct bus and friends), so that can't go to mainline, and it only shows up if you enable CONFIG_KOBJECT_DEBUG.

Hope this helps explain things, I'll go off and edit the text there to make it clearer.

thanks,

greg k-h

#3 lore raw
Quoted 51 lines · Greg KH
Quoted 31 lines · Andrew Morton
Quoted 28 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Hi,

I get similar traces on every resume from disk on SMP systems:

WARNING at lib/kobject.c:172 kobject_init()

Call Trace:
 [<ffffffff80265559>] dump_trace+0xaa/0x3fd
 [<ffffffff802658e8>] show_trace+0x3c/0x52
 [<ffffffff80265913>] dump_stack+0x15/0x17
 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a
 [<ffffffff8031c298>] kobject_register+0x1a/0x3e
 [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec
 [<ffffffff8026af39>] mce_create_device+0x79/0x103
 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd
 [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e
 [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb
 [<ffffffff80299f18>] _cpu_up+0xc2/0xd5
 [<ffffffff80299f56>] cpu_up+0x2b/0x42
 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b
 [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2
 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77
 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273
 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82
 [<ffffffff8025811e>] system_call+0x7e/0x83
DWARF2 unwinder stuck at system_call+0x7e/0x83

Leftover inexact backtrace:

Quoted 6 lines
Quoted 2 lines

False positive?

Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.

Heh, I would think that it's a big "obvious", oh well.

There are 2 changes in this patch.

First one is to ensure that the kobject is properly initialized _before_ kobject_init() is called. Yeah, seems funny, right? Turns out this has caught a lot of issues where kobject_init() is called twice on the same object, not a good thing at all.

And this looks like that is exactly what is happening here, as you mention:

Quoted 8 lines
Quoted 1 line

Perhaps mce_remove_device() isn't being called.

That's probably the issue.

The second change in that patch tries to enforce the "everything needs a release() function" rule for kobjects, but it turns out, a lot of static kobjects trigger this inproperly (struct bus and friends), so that can't go to mainline, and it only shows up if you enable CONFIG_KOBJECT_DEBUG.

CONFIG_DEBUG_KOBJECT is not set here, so it looks like kobject_init() is being called for the second time on the same object.

Greetings,
Rafael

Signature · 2 lines
You never change things by fighting the existing reality.
		R. Buckminster Fuller
#4 lore raw
In reply to Andrew Morton
Quoted 31 lines · Andrew Morton
Quoted 28 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Hi,

I get similar traces on every resume from disk on SMP systems:

WARNING at lib/kobject.c:172 kobject_init()

Call Trace:
 [<ffffffff80265559>] dump_trace+0xaa/0x3fd
 [<ffffffff802658e8>] show_trace+0x3c/0x52
 [<ffffffff80265913>] dump_stack+0x15/0x17
 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a
 [<ffffffff8031c298>] kobject_register+0x1a/0x3e
 [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec
 [<ffffffff8026af39>] mce_create_device+0x79/0x103
 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd
 [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e
 [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb
 [<ffffffff80299f18>] _cpu_up+0xc2/0xd5
 [<ffffffff80299f56>] cpu_up+0x2b/0x42
 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b
 [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2
 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77
 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273
 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82
 [<ffffffff8025811e>] system_call+0x7e/0x83
DWARF2 unwinder stuck at system_call+0x7e/0x83

Leftover inexact backtrace:

Quoted 8 lines
Quoted 2 lines

False positive?

Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.

Perhaps mce_remove_device() isn't being called.

I've added some debugging code into mce_remove_device() which shows that it is being called when the CPU is removed.

Investigation continues.

#5 lore raw
Quoted 39 lines · Rafael J. Wysocki
Quoted 38 lines · Andrew Morton
Quoted 26 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Hi,

I get similar traces on every resume from disk on SMP systems:

WARNING at lib/kobject.c:172 kobject_init()

Call Trace:
 [<ffffffff80265559>] dump_trace+0xaa/0x3fd
 [<ffffffff802658e8>] show_trace+0x3c/0x52
 [<ffffffff80265913>] dump_stack+0x15/0x17
 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a
 [<ffffffff8031c298>] kobject_register+0x1a/0x3e
 [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec
 [<ffffffff8026af39>] mce_create_device+0x79/0x103
 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd
 [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e
 [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb
 [<ffffffff80299f18>] _cpu_up+0xc2/0xd5
 [<ffffffff80299f56>] cpu_up+0x2b/0x42
 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b
 [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2
 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77
 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273
 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82
 [<ffffffff8025811e>] system_call+0x7e/0x83
DWARF2 unwinder stuck at system_call+0x7e/0x83
Quoted 4 lines

Leftover inexact backtrace:

False positive?

Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.

Quoted 6 lines
Quoted 1 line

Perhaps mce_remove_device() isn't being called.

I've added some debugging code into mce_remove_device() which shows that it is being called when the CPU is removed.

Investigation continues.

Ah, I think the problem is that the last user of a kobject doesn't decrease the refcount in kref_put(), so if the same kobject is registered for the second time, the refcount is still one and the warning triggers.

So, it seems, this is a false positive and I think we can get rid of it in the following way (tested and works):

---
Make mce_remove_device() clean up the kobject in per_cpu(device_mce, cpu) after it has been unregistered.

---
 arch/x86_64/kernel/mce.c |    1 +
 1 file changed, 1 insertion(+)
arch/x86_64/kernel/mce.c +1 -0
Index: linux-2.6.19-rc6-mm1/arch/x86_64/kernel/mce.c
===================================================================
@@ -651,6 +651,7 @@ static void mce_remove_device(unsigned i
 	sysdev_remove_file(&per_cpu(device_mce,cpu), &attr_tolerant);
 	sysdev_remove_file(&per_cpu(device_mce,cpu), &attr_check_interval);
 	sysdev_unregister(&per_cpu(device_mce,cpu));
+	per_cpu(device_mce, cpu).kobj = (struct kobject){ 0 };
 }
 
 /* Get notified when a cpu comes on/off. Be hotplug friendly. */
#6 lore raw
Quoted 42 lines · Rafael J. Wysocki
Quoted 41 lines · Rafael J. Wysocki
Quoted 39 lines · Andrew Morton
Quoted 26 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Hi,

I get similar traces on every resume from disk on SMP systems:

WARNING at lib/kobject.c:172 kobject_init()

Call Trace:
 [<ffffffff80265559>] dump_trace+0xaa/0x3fd
 [<ffffffff802658e8>] show_trace+0x3c/0x52
 [<ffffffff80265913>] dump_stack+0x15/0x17
 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a
 [<ffffffff8031c298>] kobject_register+0x1a/0x3e
 [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec
 [<ffffffff8026af39>] mce_create_device+0x79/0x103
 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd
 [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e
 [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb
 [<ffffffff80299f18>] _cpu_up+0xc2/0xd5
 [<ffffffff80299f56>] cpu_up+0x2b/0x42
 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b
 [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2
 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77
 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273
 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82
 [<ffffffff8025811e>] system_call+0x7e/0x83
DWARF2 unwinder stuck at system_call+0x7e/0x83
Quoted 4 lines

Leftover inexact backtrace:

False positive?

Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.

Perhaps mce_remove_device() isn't being called.

Quoted 8 lines
Quoted 4 lines

I've added some debugging code into mce_remove_device() which shows that it is being called when the CPU is removed.

Investigation continues.

Ah, I think the problem is that the last user of a kobject doesn't decrease the refcount in kref_put(), so if the same kobject is registered for the second time, the refcount is still one and the warning triggers.

But the last user of the kobject should cause the kobject to be freed and disappear. It should not hang around, right?

Oh yuck, this is a static struct device, one per cpu :(

Quoted 17 lines

So, it seems, this is a false positive and I think we can get rid of it in the following way (tested and works):

---
Make mce_remove_device() clean up the kobject in per_cpu(device_mce, cpu) after it has been unregistered.

---
 arch/x86_64/kernel/mce.c |    1 +
 1 file changed, 1 insertion(+)

Index: linux-2.6.19-rc6-mm1/arch/x86_64/kernel/mce.c
===================================================================

arch/x86_64/kernel/mce.c static void mce_remove_device(unsigned i · line 654

Quoted 4 lines
arch/x86_64/kernel/mce.c +1 -0
@@ -651,6 +651,7 @@ static void mce_remove_device(unsigned i
 	sysdev_remove_file(&per_cpu(device_mce,cpu), &attr_tolerant);
 	sysdev_remove_file(&per_cpu(device_mce,cpu), &attr_check_interval);
 	sysdev_unregister(&per_cpu(device_mce,cpu));
+	per_cpu(device_mce, cpu).kobj = (struct kobject){ 0 };

memset the kobj instead perhaps? Yeah, I guess this copy will work, as the compiler turns it into a memset.

But overall, ick :(

thanks,

greg k-h

#7 lore raw
Quoted 69 lines · Greg KH
Quoted 42 lines · Rafael J. Wysocki
Quoted 41 lines · Rafael J. Wysocki
Quoted 39 lines · Andrew Morton
Quoted 26 lines · On Wed, 22 Nov 2006 22:07:06 +0100 "Rafael J. Wysocki" <[email protected]> wrote:

Hi,

I get similar traces on every resume from disk on SMP systems:

WARNING at lib/kobject.c:172 kobject_init()

Call Trace:
 [<ffffffff80265559>] dump_trace+0xaa/0x3fd
 [<ffffffff802658e8>] show_trace+0x3c/0x52
 [<ffffffff80265913>] dump_stack+0x15/0x17
 [<ffffffff8031c1ad>] kobject_init+0x3f/0x8a
 [<ffffffff8031c298>] kobject_register+0x1a/0x3e
 [<ffffffff8038e5b4>] sysdev_register+0x5f/0xec
 [<ffffffff8026af39>] mce_create_device+0x79/0x103
 [<ffffffff8026afed>] mce_cpu_callback+0x2a/0xbd
 [<ffffffff8026112f>] notifier_call_chain+0x29/0x3e
 [<ffffffff8028e809>] raw_notifier_call_chain+0x9/0xb
 [<ffffffff80299f18>] _cpu_up+0xc2/0xd5
 [<ffffffff80299f56>] cpu_up+0x2b/0x42
 [<ffffffff80299fbb>] enable_nonboot_cpus+0x4e/0x9b
 [<ffffffff802a35da>] snapshot_ioctl+0x1a0/0x5d2
 [<ffffffff8023d9cd>] do_ioctl+0x5e/0x77
 [<ffffffff8022d785>] vfs_ioctl+0x256/0x273
 [<ffffffff8024770b>] sys_ioctl+0x5f/0x82
 [<ffffffff8025811e>] system_call+0x7e/0x83
DWARF2 unwinder stuck at system_call+0x7e/0x83
Quoted 4 lines

Leftover inexact backtrace:

False positive?

Don't know. The changelog in
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-01-driver/kobject-warn.patch
is pretty pathetic.

Perhaps mce_remove_device() isn't being called.

Quoted 8 lines
Quoted 4 lines

I've added some debugging code into mce_remove_device() which shows that it is being called when the CPU is removed.

Investigation continues.

Ah, I think the problem is that the last user of a kobject doesn't decrease the refcount in kref_put(), so if the same kobject is registered for the second time, the refcount is still one and the warning triggers.

But the last user of the kobject should cause the kobject to be freed and disappear. It should not hang around, right?

Oh yuck, this is a static struct device, one per cpu :(

Quoted 12 lines

So, it seems, this is a false positive and I think we can get rid of it in the following way (tested and works):

---
Make mce_remove_device() clean up the kobject in per_cpu(device_mce, cpu) after it has been unregistered.

---
 arch/x86_64/kernel/mce.c |    1 +
 1 file changed, 1 insertion(+)
Quoted 12 lines

arch/x86_64/kernel/mce.c static void mce_remove_device(unsigned i · line 654

Quoted 9 lines

Index: linux-2.6.19-rc6-mm1/arch/x86_64/kernel/mce.c
===================================================================

arch/x86_64/kernel/mce.c +1 -0
@@ -651,6 +651,7 @@ static void mce_remove_device(unsigned i
 	sysdev_remove_file(&per_cpu(device_mce,cpu), &attr_tolerant);
 	sysdev_remove_file(&per_cpu(device_mce,cpu), &attr_check_interval);
 	sysdev_unregister(&per_cpu(device_mce,cpu));
+	per_cpu(device_mce, cpu).kobj = (struct kobject){ 0 };

memset the kobj instead perhaps? Yeah, I guess this copy will work, as the compiler turns it into a memset.

Patch with the memset follows.

BTW, it seems to me that the WARN_ON in kref_get will never trigger, will it?

Greetings,
Rafael

---
Make mce_remove_device() clean up the kobject in per_cpu(device_mce, cpu) after it has been unregistered.

---
 arch/x86_64/kernel/mce.c |    1 +
 1 file changed, 1 insertion(+)
arch/x86_64/kernel/mce.c +1 -0
Index: linux-2.6.19-rc6-mm1/arch/x86_64/kernel/mce.c
===================================================================
@@ -651,6 +651,7 @@ static void mce_remove_device(unsigned i
 	sysdev_remove_file(&per_cpu(device_mce,cpu), &attr_tolerant);
 	sysdev_remove_file(&per_cpu(device_mce,cpu), &attr_check_interval);
 	sysdev_unregister(&per_cpu(device_mce,cpu));
+	memset(&per_cpu(device_mce, cpu).kobj, 0, sizeof(struct kobject));
 }
 
 /* Get notified when a cpu comes on/off. Be hotplug friendly. */
#8 lore raw
Quoted 2 lines

Patch with the memset follows.

Added thanks

-Andi