kobject: delayed kobject release: help find buggy drivers
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 26 lines · Linux Kernel
Gitweb: http://git.kernel.org/linus/;a=commit;h=c817a67ecba7c3c2aaa104796d78f160af60920d Commit: c817a67ecba7c3c2aaa104796d78f160af60920d Parent: 7c42721fe0c58a848849b43ff558cf2fb86aa35a Author: Russell King <[email protected]> AuthorDate: Thu Jun 27 15:06:14 2013 +0100 Committer: Greg Kroah-Hartman <[email protected]> CommitDate: Thu Jul 25 15:39:04 2013 -0700kobject: delayed kobject release: help find buggy drivers
Implement debugging for kobject release functions. kobjects are reference counted, so the drop of the last reference to them is not predictable. However, the common case is for the last reference to be the kobject's removal from a subsystem, which results in the release function being immediately called.
This can hide subtle bugs, which can occur when another thread holds a reference to the kobject at the same time that a kobject is removed. This results in the release method being delayed.
In order to make these kinds of problems more visible, the following patch implements a delayed release; this has the effect that the release function will be out of order with respect to the removal of the kobject in the same manner that it would be if a reference was being held.
Quoted 5 lines
This provides us with an easy way to allow driver writers to debug their drivers and fix otherwise hidden problems.
Signed-off-by Russell King <[email protected]>Signed-off-by Greg Kroah-Hartman <[email protected]>
With this enabled, I get a bunch of scrolling oopses immediately after exiting the bootloader. It happens so early I can't even capture them over usb-serial, or earlyprintk=dbgp.
And for whatever reason, the printk path while oopsing ignores boot_delay parameter, so I can't even use that.
any ideas ?
Dave
Greg KH
maintainerQuoted 33 lines · Dave Jones
Quoted 26 lines · Linux Kernel
Gitweb: http://git.kernel.org/linus/;a=commit;h=c817a67ecba7c3c2aaa104796d78f160af60920d Commit: c817a67ecba7c3c2aaa104796d78f160af60920d Parent: 7c42721fe0c58a848849b43ff558cf2fb86aa35a Author: Russell King <[email protected]> AuthorDate: Thu Jun 27 15:06:14 2013 +0100 Committer: Greg Kroah-Hartman <[email protected]> CommitDate: Thu Jul 25 15:39:04 2013 -0700kobject: delayed kobject release: help find buggy drivers
Implement debugging for kobject release functions. kobjects are reference counted, so the drop of the last reference to them is not predictable. However, the common case is for the last reference to be the kobject's removal from a subsystem, which results in the release function being immediately called.
This can hide subtle bugs, which can occur when another thread holds a reference to the kobject at the same time that a kobject is removed. This results in the release method being delayed.
In order to make these kinds of problems more visible, the following patch implements a delayed release; this has the effect that the release function will be out of order with respect to the removal of the kobject in the same manner that it would be if a reference was being held.
Quoted 5 lines
This provides us with an easy way to allow driver writers to debug their drivers and fix otherwise hidden problems.
Signed-off-by Russell King <[email protected]>Signed-off-by Greg Kroah-Hartman <[email protected]>
Quoted 8 lines
With this enabled, I get a bunch of scrolling oopses immediately after exiting the bootloader. It happens so early I can't even capture them over usb-serial, or earlyprintk=dbgp.
And for whatever reason, the printk path while oopsing ignores
boot_delayparameter, so I can't even use that.any ideas ?
The first big bug found with this was with the module kobject code, and a fix for that should be going in through Rusty's tree to Linus for this merge window (right Rusty?)
After that, things should settle down and boot properly, although a few bugs have been reported already to their respective subsystem owners, I don't know the status of them yet.
thanks,
greg k-h
Quoted 38 lines · Greg KH
Quoted 37 lines · Dave Jones
Quoted 26 lines · Linux Kernel
Gitweb: http://git.kernel.org/linus/;a=commit;h=c817a67ecba7c3c2aaa104796d78f160af60920d Commit: c817a67ecba7c3c2aaa104796d78f160af60920d Parent: 7c42721fe0c58a848849b43ff558cf2fb86aa35a Author: Russell King <[email protected]> AuthorDate: Thu Jun 27 15:06:14 2013 +0100 Committer: Greg Kroah-Hartman <[email protected]> CommitDate: Thu Jul 25 15:39:04 2013 -0700kobject: delayed kobject release: help find buggy drivers
Implement debugging for kobject release functions. kobjects are reference counted, so the drop of the last reference to them is not predictable. However, the common case is for the last reference to be the kobject's removal from a subsystem, which results in the release function being immediately called.
This can hide subtle bugs, which can occur when another thread holds a reference to the kobject at the same time that a kobject is removed. This results in the release method being delayed.
In order to make these kinds of problems more visible, the following patch implements a delayed release; this has the effect that the release function will be out of order with respect to the removal of the kobject in the same manner that it would be if a reference was being held.
Quoted 5 lines
This provides us with an easy way to allow driver writers to debug their drivers and fix otherwise hidden problems.
Signed-off-by Russell King <[email protected]>Signed-off-by Greg Kroah-Hartman <[email protected]>With this enabled, I get a bunch of scrolling oopses immediately after exiting the bootloader. It happens so early I can't even capture them over usb-serial, or earlyprintk=dbgp.
Quoted 8 lines
Quoted 4 lines
And for whatever reason, the printk path while oopsing ignores
boot_delayparameter, so I can't even use that.any ideas ?
The first big bug found with this was with the module kobject code, and a fix for that should be going in through Rusty's tree to Linus for this merge window (right Rusty?)
Yep, just sent pull request. The fix was the final commit there...
Cheers,
Rusty.
Quoted 7 lines · Rusty Russell
Quoted 7 lines
Quoted 7 lines
With this enabled, I get a bunch of scrolling oopses immediately after exiting the bootloader. It happens so early I can't even capture them over usb-serial, or earlyprintk=dbgp.
And for whatever reason, the printk path while oopsing ignores
boot_delayparameter, so I can't even use that.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
any ideas ?
The first big bug found with this was with the module kobject code, and a fix for that should be going in through Rusty's tree to Linus for this merge window (right Rusty?)
Yep, just sent pull request. The fix was the final commit there...
Crap, that isn't the bug I'm hitting.
Dave
Greg KH
maintainerQuoted 11 lines · Dave Jones
Quoted 9 lines · Rusty Russell
Quoted 9 lines
Quoted 8 lines
With this enabled, I get a bunch of scrolling oopses immediately after exiting the bootloader. It happens so early I can't even capture them over usb-serial, or earlyprintk=dbgp.
And for whatever reason, the printk path while oopsing ignores
boot_delayparameter, so I can't even use that.any ideas ?
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
The first big bug found with this was with the module kobject code, and a fix for that should be going in through Rusty's tree to Linus for this merge window (right Rusty?)
Yep, just sent pull request. The fix was the final commit there...
Crap, that isn't the bug I'm hitting.
Any chance to get a oops traceback?
greg k-h
Quoted 16 lines · Greg KH
Quoted 15 lines · Dave Jones
Quoted 13 lines · Rusty Russell
Quoted 12 lines
Quoted 8 lines
With this enabled, I get a bunch of scrolling oopses immediately after exiting the bootloader. It happens so early I can't even capture them over usb-serial, or earlyprintk=dbgp.
And for whatever reason, the printk path while oopsing ignores
boot_delayparameter, so I can't even use that.any ideas ?
The first big bug found with this was with the module kobject code, and a fix for that should be going in through Rusty's tree to Linus for this merge window (right Rusty?)
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Yep, just sent pull request. The fix was the final commit there...
Crap, that isn't the bug I'm hitting.
Any chance to get a oops traceback?
Trying to figure out where to stick a while (1) to stop it scrolling away. The end of dump_stack apparently wasn't enough.
Dave
Quoted 16 lines · Greg KH
Quoted 15 lines · Dave Jones
Quoted 13 lines · Rusty Russell
Quoted 12 lines
Quoted 8 lines
With this enabled, I get a bunch of scrolling oopses immediately after exiting the bootloader. It happens so early I can't even capture them over usb-serial, or earlyprintk=dbgp.
And for whatever reason, the printk path while oopsing ignores
boot_delayparameter, so I can't even use that.any ideas ?
The first big bug found with this was with the module kobject code, and a fix for that should be going in through Rusty's tree to Linus for this merge window (right Rusty?)
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Yep, just sent pull request. The fix was the final commit there...
Crap, that isn't the bug I'm hitting.
Any chance to get a oops traceback?
Got it. (I think I was getting tracebacks from multiple cpus, hence the spewing). Adding a check for tainted() = infinite loop to show_backtrace() combined with boot_delay gave me a really long oops that I had to grab video to record.
RIP: <null>
Trace:
<IRQ>
run_timer_softirq__do_softirqirq_exitsmp_apic_timer_interruptapic_timer_interrupt
<EOI>vprintk_emitdev_vprintk_emitdev_vprint_emit__dev_printk_dev_infoahci_print_infoahci_init_onelocal_pci_probepci_device_probedriver_probe_device__driver_attachbus_for_each_devdriver_attachbus_add_driverdriver_register__pci_register_driverahci_pci_driver_initdo_one_initcall
...
Dave
Greg KH
maintainerQuoted 19 lines · Dave Jones
Quoted 18 lines · Greg KH
Quoted 17 lines · Dave Jones
Quoted 14 lines · Rusty Russell
Quoted 4 lines
Quoted 4 lines
With this enabled, I get a bunch of scrolling oopses immediately after exiting the bootloader. It happens so early I can't even capture them over usb-serial, or earlyprintk=dbgp.
Quoted 8 lines
Quoted 4 lines
And for whatever reason, the printk path while oopsing ignores
boot_delayparameter, so I can't even use that.any ideas ?
The first big bug found with this was with the module kobject code, and a fix for that should be going in through Rusty's tree to Linus for this merge window (right Rusty?)
Yep, just sent pull request. The fix was the final commit there...
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Crap, that isn't the bug I'm hitting.
Any chance to get a oops traceback?
Got it. (I think I was getting tracebacks from multiple cpus, hence the spewing). Adding a check for
tainted()= infinite loop toshow_backtrace()combined withboot_delaygave me a really long oops that I had to grab video to record.
Odd, but thanks for going through all of that.
Quoted 19 lines
RIP: <null>Trace:
<IRQ>
run_timer_softirq__do_softirqirq_exitsmp_apic_timer_interruptapic_timer_interrupt
<EOI>vprintk_emitdev_vprintk_emitdev_vprint_emit__dev_printk_dev_infoahci_print_infoahci_init_onelocal_pci_probepci_device_probe
Quoted 8 lines
driver_probe_device__driver_attachbus_for_each_devdriver_attachbus_add_driverdriver_register__pci_register_driverahci_pci_driver_init
So we are loading a ahci driver here. Any hint as to which driver/device this is? Is it a device that fails for this driver, and then falls back to another more "specific" one?
And dieing in ahci_print_info()? That's very strange.
Russell (not Rusty), any thoughts about this one? You've been able to debug a bunch of these tracebacks recently really well.
thanks,
greg k-h
Quoted 26 lines · Greg KH
Quoted 17 lines
Trace:
<IRQ>
run_timer_softirq__do_softirqirq_exitsmp_apic_timer_interruptapic_timer_interrupt
<EOI>vprintk_emitdev_vprintk_emitdev_vprint_emit__dev_printk_dev_infoahci_print_infoahci_init_onelocal_pci_probepci_device_probeQuoted 8 lines
driver_probe_device__driver_attachbus_for_each_devdriver_attachbus_add_driverdriver_register__pci_register_driverahci_pci_driver_init
Quoted 3 lines
So we are loading a ahci driver here. Any hint as to which driver/device this is? Is it a device that fails for this driver, and then falls back to another more "specific" one?
It's "ahci.ko" (though it's built in).
For a boring onboard intel sata controller...
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04)
When I disable the debug option, usually it looks like..
[ 6.382970] ahci 0000:00:1f.2: version 3.0 [ 6.384753] ahci 0000:00:1f.2: irq 43 for MSI/MSI-X [ 6.385039] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x6 impl SATA mode [ 6.385067] ahci 0000:00:1f.2: flags: 64bit ncq pm led clo pio slum part ems apst [ 6.385101] ahci 0000:00:1f.2: setting latency timer to 64 [ 6.402536] scsi0 : ahci [ 6.403669] scsi1 : ahci [ 6.404208] scsi2 : ahci [ 6.404756] scsi3 : ahci [ 6.405342] scsi4 : ahci [ 6.405951] scsi5 : ahci [ 6.406345] ata1: DUMMY [ 6.406364] ata2: SATA max UDMA/133 abar m2048@0xc043b000 port 0xc043b180 irq 43 [ 6.406393] ata3: SATA max UDMA/133 abar m2048@0xc043b000 port 0xc043b200 irq 43 [ 6.406416] ata4: DUMMY [ 6.406427] ata5: DUMMY [ 6.406438] ata6: DUMMY
Dave
Quoted 4 lines · Dave Jones
Got it. (I think I was getting tracebacks from multiple cpus, hence the spewing). Adding a check for
tainted()= infinite loop toshow_backtrace()combined withboot_delaygave me a really long oops that I had to grab video to record.
How complete is this trace? If accurate, I suspect that this is an oops caused by the timer expiring, but the memory associated with the timer has already been overwritten. That would make it particularly hard to debug.
Given that we run the delayed release 1 second after the object release, I'm thinking that this has nothing to do with AHCI.
So, there's not a lot to go on here.
I'm wondering if there's any milage to allocating the kobject release information inside kobject_add_internal(), so that we can at least run the timer list and delayed work queues safely, and add debugging to print the kobject pointer before we end up oopsing. If we also add a magic value to the beginning of the kobject, we can check whether it looks like it may still be valid before we try and treat it as a kobject, which may save us from oopsing on boot.
With the kobject debugging already there, it should be possible to match the pointer value with previous debugging from kobject_add_internal() and work out what kobject causes this.
Maybe the patch below will allow Dave to boot to a prompt and capture the full dmesg - I've only just cooked up this patch so it may need a bit of work to get it to build. Please ensure that kobject debugging is enabled, so we can translate from kobject addresses to names.
Dave - thanks for your patience and effort in trying to track this down. Hopefully the patch below will make it easier.
include/linux/kobject.h +6 -3
@@ -58,7 +58,13 @@ enum kobject_action {KOBJ_MAX }; +struct kobject_release ; +struct kobject { +#ifdef CONFIG_DEBUG_KOBJECT_RELEASE +unsigned int magic; +struct kobject_release *release; +#endif const char *name;struct list_head entry;struct kobject *parent;
@@ -66,9 +72,6 @@ struct kobject {struct kobj_type *ktype;struct sysfs_dirent *sd;struct kref kref; -#ifdef CONFIG_DEBUG_KOBJECT_RELEASE -struct delayed_work release; -#endif unsigned int state_initialized:1; unsigned int state_in_sysfs:1; unsigned int state_add_uevent_sent:1;
lib/kobject.c +34 -4
@@ -153,6 +153,13 @@ static void kobject_init_internal(struct kobject *kobj) } +#ifdef CONFIG_DEBUG_KOBJECT_RELEASE +#define KOBJ_MAGIC ('k' | 'o' << 8 | 'b' << 16 | 'j' << 24) +struct kobject_release { +struct delayed_work work; +struct kobject *kobj; +}; +#endif static int kobject_add_internal (struct kobject *kobj) {int error =0 ;
@@ -169,6 +176,14 @@ static int kobject_add_internal(struct kobject *kobj) parent =kobject_get (kobj->parent); +#ifdef CONFIG_DEBUG_KOBJECT_RELEASE + kobj->release =kmalloc (sizeof (*kobj->release),GFP_KERNEL ); +if (!kobj->release) +return -ENOMEM ; + kobj->release->kobj = kobj; + kobj->magic =KOBJ_MAGIC ; +#endif +/* join kset if set, use it as parent if we do not already have one */ if (kobj->kset) {if (!parent)
@@ -198,6 +213,9 @@ static int kobject_add_internal(struct kobject *kobj)WARN (1 ,"%s failed for %s (error: %d parent: %s)\n" , __func__,kobject_name (kobj), error, parent ?kobject_name (parent) :"'none'" ); +kfree (kobj->release); + kobj->release =NULL ; + kobj->magic =0 ; }else kobj->state_in_sysfs =1 ;
@@ -583,8 +601,20 @@ static void kobject_cleanup(struct kobject *kobj)#ifdef CONFIG_DEBUG_KOBJECT_RELEASE static void kobject_delayed_cleanup (struct work_struct *work) { -kobject_cleanup (container_of(to_delayed_work (work ), -struct kobject ,release )); +struct kobject_release *release =container_of (to_delayed_work (work), + struct kobject_release, work); +struct kobject *kobj = release->kobj; + +kfree (release); +if (kobj->magic ==KOBJ_MAGIC ) { +pr_debug ("kobject: '%s' (%p): %s\n" , +kobject_name (kobj), kobj, __func__); +kobject_cleanup (kobj); + }else { +pr_err ("kobject %p has been corrupted (magic 0x%08x). Please enable kobject debugging for full debug.\n" , + kobj, kobj->magic); +/* We end up leaking the kobject */ + } }#endif
@@ -594,8 +624,8 @@ static void kobject_release(struct kref *kref)#ifdef CONFIG_DEBUG_KOBJECT_RELEASE pr_debug ("kobject: '%s' (%p): %s, parent %p (delayed)\n" ,kobject_name (kobj), kobj, __func__, kobj->parent); -INIT_DELAYED_WORK (&kobj->release, kobject_delayed_cleanup); -schedule_delayed_work (&kobj->release,HZ ); +INIT_DELAYED_WORK (&kobj->release->work, kobject_delayed_cleanup); +schedule_delayed_work (&kobj->release->work,HZ );#else kobject_cleanup (kobj);#endif
Quoted 10 lines · Russell King - ARM Linux
Quoted 4 lines · Dave Jones
Got it. (I think I was getting tracebacks from multiple cpus, hence the spewing). Adding a check for
tainted()= infinite loop toshow_backtrace()combined withboot_delaygave me a really long oops that I had to grab video to record.How complete is this trace? If accurate, I suspect that this is an oops caused by the timer expiring, but the memory associated with the timer has already been overwritten. That would make it particularly hard to debug.
There were some '?' symbols in the trace too which I omitted last night. (offsets left out, because I'm hand-transcribing from video)
<IRQ>
? call_timer_fn
? call_timer_fnrun_timer_softirq__do_softirqirq_exitsmp_apic_timer_interruptapic_timer_interrupt
<EOI>
? touch_nmi_watchdog
? touch_nmi_watchdogvprintk_emit
? create_syslog_headerdev_vprintk_emit
? dump_trace
? trace_hardirqs_off_caller
? trace_hardirqs_off_callerdev_print_emit
? trace_hardirqs_off_caller__dev_printk_dev_info
? trace_hardirqs_offahci_print_infoahci_init_onelocal_pci_probepci_device_probedriver_probe_device__driver_attach
? __device_attachbus_for_each_devdriver_attachbus_add_driver
? ata_sff_initdriver_register__pci_register_driverahci_pci_driver_initdo_one_initcall
? parse_argskernel_init_freeable
? do_early_param
? rest_initkernel_initret_from_fork
? rest_init
Quoted 7 lines
Maybe the patch below will allow Dave to boot to a prompt and capture the full dmesg - I've only just cooked up this patch so it may need a bit of work to get it to build. Please ensure that kobject debugging is enabled, so we can translate from kobject addresses to names.
Dave - thanks for your patience and effort in trying to track this down. Hopefully the patch below will make it easier.
building now.
Thanks,
Dave
Quoted 2 lines · Russell King - ARM Linux
Dave - thanks for your patience and effort in trying to track this down. Hopefully the patch below will make it easier.
I'm not sure if this is happening because I enabled DEBUG_KOBJECT this time around or not, but now I'm seeing a different trace. (same hang though)
serial8250: ttyS0 at I/O 0x3f8 (ieq =4, base_baud = 115200) is 16550A Oops
RIP: kobject_release
trace:
kobject_putcdev_deltty_unregister_deviceuart_remove_one_portserial8250_register_8250_portpciserial_init_portspciserial_init_onelocal_pci_probepci_device_probedriver_probe_device__driver_attach
? __device_attachbus_for_each_devdriver_attachbus_add_driver
? early_serial_setupdriver_register
? early_serial_setup__pci_register_driverserial_pci_driver_initdo_one_initcall
Trying without serial console next..
Quoted 36 lines · Dave Jones
Quoted 2 lines · Russell King - ARM Linux
Dave - thanks for your patience and effort in trying to track this down. Hopefully the patch below will make it easier.
I'm not sure if this is happening because I enabled
DEBUG_KOBJECTthis time around or not, but now I'm seeing a different trace. (same hang though)serial8250: ttyS0 at I/O
0x3f8(ieq =4,base_baud= 115200) is 16550A OopsRIP: kobject_releasetrace:
kobject_putcdev_deltty_unregister_deviceuart_remove_one_portserial8250_register_8250_portpciserial_init_portspciserial_init_onelocal_pci_probepci_device_probedriver_probe_device__driver_attach
?__device_attachbus_for_each_devdriver_attachbus_add_driver
?early_serial_setupdriver_register
?early_serial_setup__pci_register_driverserial_pci_driver_initdo_one_initcall
Quoted 1 line
Trying without serial console next..
rebuilt with all serial turned off.
no luck, then it oopses somewhere else. I'm suspecting something isn't right with that debug patch, as the next trace is also in kobject_release
Dave
Quoted 6 lines · Dave Jones
Quoted 1 line
Trying without serial console next..
rebuilt with all serial turned off.
no luck, then it oopses somewhere else. I'm suspecting something isn't right with that debug patch, as the next trace is also in
kobject_release
I managed to get to userspace on one boot, and got this.
kobject ffff88023d93f518 has been corrupted (magic 0x6b6b6b6b). Please enable kobject debugging for full debug.
That looks like SLAB_POISON. Incompatibility between the two options ?
For some reason, even though I have DEBUG_KOBJECT on, I didn't get extra messages output.
Dave
Quoted 7 lines · Dave Jones
Quoted 6 lines · Dave Jones
Quoted 1 line
Trying without serial console next..
rebuilt with all serial turned off.
no luck, then it oopses somewhere else. I'm suspecting something isn't right with that debug patch, as the next trace is also in
kobject_release
You're right about that - I had assumed that it was necessary for all kobjects to be 'added' before they're released, but that is not so. Mea culpa. They just need to be initialised - the problem with putting a kmalloc into kobject_init() is that if it fails, we have no way to report that failure...
Updated patch attached... though it sounds like you got it working anyway.
Quoted 4 lines
I managed to get to userspace on one boot, and got this.
kobject
ffff88023d93f518has been corrupted (magic0x6b6b6b6b). Please enable kobject debugging for full debug.
Okay, so this is definitely a case that someone has kfree'd the kobject without waiting for the ->release function to be called.
Quoted 4 lines
That looks like
SLAB_POISON. Incompatibility between the two options ?For some reason, even though I have
DEBUG_KOBJECTon, I didn't get extra messages output.
Hmm.
ifeq ($(CONFIG_DEBUG_KOBJECT),y)
CFLAGS_kobject.o += -DDEBUG
CFLAGS_kobject_uevent.o += -DDEBUG
endif
should enable the pr_debug()'s in lib/kobject.c... which should at least appear in the dmesg log. Being debug level, of course, they won't appear during normal kernel boot unless 'debug' is passed on the kernel command line.
It seems to work for me - but... produces rather a lot of debug messages, so you may also wish to ensure that you have LOG_BUF_SHIFT set to something large.
include/linux/kobject.h +6 -3
@@ -58,7 +58,13 @@ enum kobject_action {KOBJ_MAX }; +struct kobject_release ; +struct kobject { +#ifdef CONFIG_DEBUG_KOBJECT_RELEASE +unsigned int magic; +struct kobject_release *release; +#endif const char *name;struct list_head entry;struct kobject *parent;
@@ -66,9 +72,6 @@ struct kobject {struct kobj_type *ktype;struct sysfs_dirent *sd;struct kref kref; -#ifdef CONFIG_DEBUG_KOBJECT_RELEASE -struct delayed_work release; -#endif unsigned int state_initialized:1; unsigned int state_in_sysfs:1; unsigned int state_add_uevent_sent:1;
lib/kobject.c +32 -4
@@ -153,6 +153,13 @@ static void kobject_init_internal(struct kobject *kobj) } +#ifdef CONFIG_DEBUG_KOBJECT_RELEASE +#define KOBJ_MAGIC ('k' | 'o' << 8 | 'b' << 16 | 'j' << 24) +struct kobject_release { +struct delayed_work work; +struct kobject *kobj; +}; +#endif static int kobject_add_internal (struct kobject *kobj) {int error =0 ;
@@ -283,6 +290,15 @@ void kobject_init(struct kobject *kobj, struct kobj_type *ktype) "object, something is seriously wrong.\n", kobj);dump_stack (); } +#ifdef CONFIG_DEBUG_KOBJECT_RELEASE + kobj->release =kmalloc (sizeof (*kobj->release),GFP_KERNEL ); +WARN_ON (!kobj->release); +if (kobj->release) { + kobj->release->kobj = kobj; + kobj->magic =KOBJ_MAGIC ; + } +#endif +kobject_init_internal (kobj); kobj->ktype = ktype;
@@ -583,8 +599,20 @@ static void kobject_cleanup(struct kobject *kobj)#ifdef CONFIG_DEBUG_KOBJECT_RELEASE static void kobject_delayed_cleanup (struct work_struct *work) { -kobject_cleanup (container_of(to_delayed_work (work ), -struct kobject ,release )); +struct kobject_release *release =container_of (to_delayed_work (work), + struct kobject_release, work); +struct kobject *kobj = release->kobj; + +kfree (release); +if (kobj->magic ==KOBJ_MAGIC ) { +pr_debug ("kobject: '%s' (%p): %s\n" , +kobject_name (kobj), kobj, __func__); +kobject_cleanup (kobj); + }else { +pr_err ("kobject %p has been corrupted (magic 0x%08x). Please enable kobject debugging for full debug.\n" , + kobj, kobj->magic); +/* We end up leaking the kobject */ + } }#endif
@@ -594,8 +622,8 @@ static void kobject_release(struct kref *kref)#ifdef CONFIG_DEBUG_KOBJECT_RELEASE pr_debug ("kobject: '%s' (%p): %s, parent %p (delayed)\n" ,kobject_name (kobj), kobj, __func__, kobj->parent); -INIT_DELAYED_WORK (&kobj->release, kobject_delayed_cleanup); -schedule_delayed_work (&kobj->release,HZ ); +INIT_DELAYED_WORK (&kobj->release->work, kobject_delayed_cleanup); +schedule_delayed_work (&kobj->release->work,HZ );#else kobject_cleanup (kobj);#endif
Quoted 38 lines · Russell King - ARM Linux
Quoted 7 lines · Dave Jones
Quoted 6 lines · Dave Jones
Quoted 1 line
Trying without serial console next..
rebuilt with all serial turned off.
no luck, then it oopses somewhere else. I'm suspecting something isn't right with that debug patch, as the next trace is also in
kobject_releaseYou're right about that - I had assumed that it was necessary for all kobjects to be 'added' before they're released, but that is not so. Mea culpa. They just need to be initialised - the problem with putting a kmalloc into
kobject_init()is that if it fails, we have no way to report that failure...Updated patch attached... though it sounds like you got it working anyway.
Quoted 4 lines
I managed to get to userspace on one boot, and got this.
kobject
ffff88023d93f518has been corrupted (magic0x6b6b6b6b). Please enable kobject debugging for full debug.Okay, so this is definitely a case that someone has kfree'd the kobject without waiting for the ->release function to be called.
Quoted 4 lines
That looks like
SLAB_POISON. Incompatibility between the two options ?For some reason, even though I have
DEBUG_KOBJECTon, I didn't get extra messages output.Hmm.
ifeq ($(CONFIG_DEBUG_KOBJECT),y)
CFLAGS_kobject.o += -DDEBUG
CFLAGS_kobject_uevent.o += -DDEBUG
endif
Quoted 8 lines
should enable the pr_debug()'s in
lib/kobject.c... which should at least appear in the dmesg log. Being debug level, of course, they won't appear during normal kernel boot unless 'debug' is passed on the kernel command line.It seems to work for me - but... produces rather a lot of debug messages, so you may also wish to ensure that you have
LOG_BUF_SHIFTset to something large.
As discussed on irc, with this updated patch it hangs *really* early in boot.
With earlyprintk=vga I can see some of the trace..
kobject_initfirmware_map_add_entryfirmware_map_add_earlye820_reserve_resourcessetup_archstart_kernelx86_64_start_reservationsx86_64_start_kernel
<RIP> kmem_cache_alloc_trace
Dave
Quoted 13 lines · Dave Jones
As discussed on irc, with this updated patch it hangs *really* early in boot.
With earlyprintk=vga I can see some of the trace..
kobject_initfirmware_map_add_entryfirmware_map_add_earlye820_reserve_resourcessetup_archstart_kernelx86_64_start_reservationsx86_64_start_kernel
Quoted 1 line
<RIP>
kmem_cache_alloc_trace
Argh, so kobject_init() gets used before the memory allocators are setup. This is starting to look like we can't separate the delayed_work from the kobject struct to provide a layer of isolation. I'm running out of ideas here.
Do we have a way to tell if the memory allocators have been initialised? I can't see anything that would, with the exception that the scheduler initialisation requires memory allocators are up and running, and it has its own scheduler_running state, but that's buried in the scheduler.
system_state is useless for this because it gets set to the running state after device initialisation, and just before we drop into userspace.
We could go back to doing the allocation at kobject_add() time, that seemed to be marginally successful for you - but I'm fairly certain it won't get all kobjects that are later released (because its possible to init a kobject and then put it once to discard it.)