call_usermodehelper hang
Subsystems (6)
I have been running into some kernel hangs due to call_usermodehelper. Looking at the backtrace, it looks to me like there are deadlock issues with adding devices from work queues. Attached is a sample backtrace from one of the hangs I experienced. My question is why does call_usermodehelper do 2 different things depending on whether or not it is called from the kevent task? It appears that the simple way to fix the hang would be to never have call_usermodehelper use a work_queue since it must be called from process context anyway, or am I missing something?
0xc0000000017df300 1 0 0 0 D 0xc0000000017df7b0 swapper
SP(esp) PC(eip) Function(args)
0xc00000003fc9f460 0x0000000000000000 NO_SYMBOL or Userspace 0xc00000003fc9f4f0 0xc000000000058c40 .schedule +0xb4 0xc00000003fc9f5c0 0xc00000000005a464 .wait_for_completion +0x138 0xc00000003fc9f6c0 0xc00000000007c594 .call_usermodehelper +0x104 0xc00000003fc9f810 0xc00000000022d3e8 .kobject_hotplug +0x3c4 0xc00000003fc9f900 0xc00000000022d67c .kobject_add +0x134 0xc00000003fc9f9a0 0xc00000000012b3d8 .register_disk +0x70 0xc00000003fc9fa40 0xc00000000027dfe4 .add_disk +0x60 0xc00000003fc9fad0 0xc0000000002dc7dc .sd_probe +0x290 0xc00000003fc9fb80 0xc00000000026fbe8 .bus_match +0x94 0xc00000003fc9fc10 0xc00000000026ff70 .driver_attach +0x8c 0xc00000003fc9fca0 0xc000000000270104 .bus_add_driver +0x110 0xc00000003fc9fd50 0xc000000000270a18 .driver_register +0x38 0xc00000003fc9fdd0 0xc0000000002cd8f8 .scsi_register_driver +0x28 0xc00000003fc9fe50 0xc0000000004941d8 .init_sd +0x8c 0xc00000003fc9fee0 0xc00000000000c720 .init +0x25c 0xc00000003fc9ff90 0xc0000000000183ec .kernel_thread +0x4c
0xc00000003fab3380 4 1 0 0 D 0xc00000003fab3830 events/0
SP(esp) PC(eip) Function(args)
0xc00000003faaf6e0 0x0000000000000000 NO_SYMBOL or Userspace 0xc00000003faaf770 0xc000000000058c40 .schedule +0xb4 0xc00000003faaf840 0xc00000000022fa20 .rwsem_down_write_failed +0x14c 0xc00000003faaf910 0xc00000000026fed0 .bus_add_device +0x11c 0xc00000003faaf9b0 0xc00000000026e288 .device_add +0xd0 0xc00000003faafa50 0xc0000000002cdb00 .scsi_sysfs_add_sdev +0x8c 0xc00000003faafb00 0xc0000000002cbff8 .scsi_probe_and_add_lun +0xb04 0xc00000003faafc00 0xc0000000002ccca0 .scsi_add_device +0x90 0xc00000003faafcb0 0xc0000000002d9458 .ipr_worker_thread +0xc60 0xc00000003faafdc0 0xc00000000007cd9c .worker_thread +0x268 0xc00000003faafee0 0xc0000000000839cc .kthread +0x160 0xc00000003faaff90 0xc0000000000183ec .kernel_thread +0x4c
Signature · 10 lines
Brian King eServer Storage I/O IBM Linux Technology Center -- Brian King eServer Storage I/O IBM Linux Technology Center
Quoted 10 lines · Brian King
I have been running into some kernel hangs due to
call_usermodehelper. Looking at the backtrace, it looks to me like there are deadlock issues with adding devices from work queues. Attached is a sample backtrace from one of the hangs I experienced. My question is why doescall_usermodehelperdo 2 different things depending on whether or not it is called from the kevent task? It appears that the simple way to fix the hang would be to never havecall_usermodehelperuse awork_queuesince it must be called from process context anyway, or am I missing something?
swapper is running call_usermodehelper() while holding down_write(&bus->subsys.rwsem); via bus_add_driver().
Meanwhile, keventd is blocked on the same lock in bus_add_device().
I'd say that the bug lies in the kobject code - we should not call call_usermodehelper() while holding any locks which keventd may ever acquire.
Quoted 34 lines
0xc0000000017df3001 0 0 0 D0xc0000000017df7b0swapper
SP(esp) PC(eip) Function(args)0xc00000003fc9f460 0x0000000000000000 NO_SYMBOL or Userspace 0xc00000003fc9f4f0 0xc000000000058c40 .schedule +0xb4 0xc00000003fc9f5c0 0xc00000000005a464 .wait_for_completion +0x138 0xc00000003fc9f6c0 0xc00000000007c594 .call_usermodehelper +0x104 0xc00000003fc9f810 0xc00000000022d3e8 .kobject_hotplug +0x3c4 0xc00000003fc9f900 0xc00000000022d67c .kobject_add +0x134 0xc00000003fc9f9a0 0xc00000000012b3d8 .register_disk +0x70 0xc00000003fc9fa40 0xc00000000027dfe4 .add_disk +0x60 0xc00000003fc9fad0 0xc0000000002dc7dc .sd_probe +0x290 0xc00000003fc9fb80 0xc00000000026fbe8 .bus_match +0x94 0xc00000003fc9fc10 0xc00000000026ff70 .driver_attach +0x8c 0xc00000003fc9fca0 0xc000000000270104 .bus_add_driver +0x110 0xc00000003fc9fd50 0xc000000000270a18 .driver_register +0x38 0xc00000003fc9fdd0 0xc0000000002cd8f8 .scsi_register_driver +0x28 0xc00000003fc9fe50 0xc0000000004941d8 .init_sd +0x8c 0xc00000003fc9fee0 0xc00000000000c720 .init +0x25c 0xc00000003fc9ff90 0xc0000000000183ec .kernel_thread +0x4c
0xc00000003fab33804 1 0 0 D0xc00000003fab3830events/0
SP(esp) PC(eip) Function(args)0xc00000003faaf6e0 0x0000000000000000 NO_SYMBOL or Userspace 0xc00000003faaf770 0xc000000000058c40 .schedule +0xb4 0xc00000003faaf840 0xc00000000022fa20 .rwsem_down_write_failed +0x14c 0xc00000003faaf910 0xc00000000026fed0 .bus_add_device +0x11c 0xc00000003faaf9b0 0xc00000000026e288 .device_add +0xd0 0xc00000003faafa50 0xc0000000002cdb00 .scsi_sysfs_add_sdev +0x8c 0xc00000003faafb00 0xc0000000002cbff8 .scsi_probe_and_add_lun +0xb04 0xc00000003faafc00 0xc0000000002ccca0 .scsi_add_device +0x90 0xc00000003faafcb0 0xc0000000002d9458 .ipr_worker_thread +0xc60 0xc00000003faafdc0 0xc00000000007cd9c .worker_thread +0x268 0xc00000003faafee0 0xc0000000000839cc .kthread +0x160 0xc00000003faaff90 0xc0000000000183ec .kernel_thread +0x4c
Quoted 12 lines · Andrew Morton
Quoted 10 lines · Brian King
I have been running into some kernel hangs due to
call_usermodehelper. Looking at the backtrace, it looks to me like there are deadlock issues with adding devices from work queues. Attached is a sample backtrace from one of the hangs I experienced. My question is why doescall_usermodehelperdo 2 different things depending on whether or not it is called from the kevent task? It appears that the simple way to fix the hang would be to never havecall_usermodehelperuse awork_queuesince it must be called from process context anyway, or am I missing something?
Quoted 8 lines
swapper is running
call_usermodehelper()while holdingdown_write(&bus->subsys.rwsem); viabus_add_driver().Meanwhile, keventd is blocked on the same lock in
bus_add_device().I'd say that the bug lies in the kobject code - we should not call
call_usermodehelper()while holding any locks which keventd may ever acquire.
How is keventd calling sysfs code? Is scsi using it to drive device detection somehow? I don't see how the kobject core code itself can do this on its own.
thanks,
greg k-h
Quoted 19 lines · Greg KH
Quoted 17 lines · Andrew Morton
Quoted 9 lines · Brian King
I have been running into some kernel hangs due to
call_usermodehelper. Looking at the backtrace, it looks to me like there are deadlock issues with adding devices from work queues. Attached is a sample backtrace from one of the hangs I experienced. My question is why doescall_usermodehelperdo 2 different things depending on whether or not it is called from the kevent task? It appears that the simple way to fix the hang would be to never havecall_usermodehelperuse awork_queuesince it must be called from process context anyway, or am I missing something?swapper is running
call_usermodehelper()while holdingdown_write(&bus->subsys.rwsem); viabus_add_driver().Meanwhile, keventd is blocked on the same lock in
bus_add_device().
Quoted 8 lines
Quoted 3 lines
I'd say that the bug lies in the kobject code - we should not call
call_usermodehelper()while holding any locks which keventd may ever acquire.How is keventd calling sysfs code? Is scsi using it to drive device detection somehow? I don't see how the kobject core code itself can do this on its own.
scsi_add_device is being called from keventd by the ipr LLD. I am assuming this is a reasonable thing to do. If not, then I would argue the scsi_add_device API is broken as it must be called from task level, and if a LLD cannot use keventd, then every LLD that wants to use it would need to create its own workqueue thread.
Signature · 3 lines
Brian King eServer Storage I/O IBM Linux Technology Center
Quoted 19 lines · Greg KH
Quoted 17 lines · Andrew Morton
Quoted 9 lines · Brian King
I have been running into some kernel hangs due to
call_usermodehelper. Looking at the backtrace, it looks to me like there are deadlock issues with adding devices from work queues. Attached is a sample backtrace from one of the hangs I experienced. My question is why doescall_usermodehelperdo 2 different things depending on whether or not it is called from the kevent task? It appears that the simple way to fix the hang would be to never havecall_usermodehelperuse awork_queuesince it must be called from process context anyway, or am I missing something?swapper is running
call_usermodehelper()while holdingdown_write(&bus->subsys.rwsem); viabus_add_driver().Meanwhile, keventd is blocked on the same lock in
bus_add_device().
Quoted 8 lines
Quoted 3 lines
I'd say that the bug lies in the kobject code - we should not call
call_usermodehelper()while holding any locks which keventd may ever acquire.How is keventd calling sysfs code? Is scsi using it to drive device detection somehow? I don't see how the kobject core code itself can do this on its own.
Here is a patch which fixes the problem on my system.
Signature · 3 lines
Brian King eServer Storage I/O IBM Linux Technology Center
The following patch fixes a deadlock experienced when devices are being added to a bus both from a user process and eventd process. The eventd process was hung waiting on dev->bus->subsys.rwsem which was held by another process, which was hung since it was calling call_usermodehelper directly which was hung waiting for work scheduled on the eventd workqueue to complete. The patch fixes this by delaying the kobject_hotplug work, running it from eventd if possible.
Backtraces of the hang:
0xc0000000017df300 1 0 0 0 D 0xc0000000017df7b0
swapper
SP(esp) PC(eip) Function(args)
0xc00000003fc9f460 0x0000000000000000 NO_SYMBOL or Userspace 0xc00000003fc9f4f0 0xc000000000058c40 .schedule +0xb4 0xc00000003fc9f5c0 0xc00000000005a464 .wait_for_completion +0x138 0xc00000003fc9f6c0 0xc00000000007c594 .call_usermodehelper +0x104 0xc00000003fc9f810 0xc00000000022d3e8 .kobject_hotplug +0x3c4 0xc00000003fc9f900 0xc00000000022d67c .kobject_add +0x134 0xc00000003fc9f9a0 0xc00000000012b3d8 .register_disk +0x70 0xc00000003fc9fa40 0xc00000000027dfe4 .add_disk +0x60 0xc00000003fc9fad0 0xc0000000002dc7dc .sd_probe +0x290 0xc00000003fc9fb80 0xc00000000026fbe8 .bus_match +0x94 0xc00000003fc9fc10 0xc00000000026ff70 .driver_attach +0x8c 0xc00000003fc9fca0 0xc000000000270104 .bus_add_driver +0x110 0xc00000003fc9fd50 0xc000000000270a18 .driver_register +0x38 0xc00000003fc9fdd0 0xc0000000002cd8f8 .scsi_register_driver +0x28 0xc00000003fc9fe50 0xc0000000004941d8 .init_sd +0x8c 0xc00000003fc9fee0 0xc00000000000c720 .init +0x25c 0xc00000003fc9ff90 0xc0000000000183ec .kernel_thread +0x4c
0xc00000003fab3380 4 1 0 0 D 0xc00000003fab3830
events/0
SP(esp) PC(eip) Function(args)
0xc00000003faaf6e0 0x0000000000000000 NO_SYMBOL or Userspace 0xc00000003faaf770 0xc000000000058c40 .schedule +0xb4 0xc00000003faaf840 0xc00000000022fa20 .rwsem_down_write_failed +0x14c 0xc00000003faaf910 0xc00000000026fed0 .bus_add_device +0x11c 0xc00000003faaf9b0 0xc00000000026e288 .device_add +0xd0 0xc00000003faafa50 0xc0000000002cdb00 .scsi_sysfs_add_sdev +0x8c 0xc00000003faafb00 0xc0000000002cbff8 .scsi_probe_and_add_lun +0xb04 0xc00000003faafc00 0xc0000000002ccca0 .scsi_add_device +0x90 0xc00000003faafcb0 0xc0000000002d9458 .ipr_worker_thread +0xc60 0xc00000003faafdc0 0xc00000000007cd9c .worker_thread +0x268 0xc00000003faafee0 0xc0000000000839cc .kthread +0x160 0xc00000003faaff90 0xc0000000000183ec .kernel_thread +0x4c
---
lib/kobject.c renamed from lib/kobject.c~kobject_hotplug_hang +49 -9
@@ -103,8 +103,14 @@ static void fill_kobj_path(struct kset *static unsigned long sequence_num;static spinlock_t sequence_lock =SPIN_LOCK_UNLOCKED ; -static void kset_hotplug (const char *action,struct kset *kset, -struct kobject *kobj) +struct hotplug_work { +struct work_struct work; +const char *action; +struct kset *kset; +struct kobject *kobj; +}; + +static void kset_hotplug_work (void *data) {char *argv [3 ];char **envp =NULL ;
@@ -116,22 +122,26 @@ static void kset_hotplug(const char *actchar *kobj_path =NULL ;char *name =NULL ;unsigned long seq; +struct hotplug_work *work = (struct hotplug_work *)data; +const char *action = work->action; +struct kset *kset = work->kset; +struct kobject *kobj = work->kobj;/* If the kset has a filter operation, call it. If it returns -failure, no hotplug event is required. */ +failure, no hotplug event is required. */ if (kset->hotplug_ops->filter) {if (!kset->hotplug_ops->filter (kset, kobj)) -return ; + goto exit; }pr_debug ("%s\n" , __FUNCTION__);if (!hotplug_path[0 ]) -return ; + goto exit; envp =kmalloc (NUM_ENVP *sizeof (char *),GFP_KERNEL );if (!envp) -return ; + goto exit;memset (envp,0x00 ,NUM_ENVP *sizeof (char *)); buffer =kmalloc (BUFFER_SIZE ,GFP_KERNEL );
@@ -176,8 +186,8 @@ static void kset_hotplug(const char *actif (kset->hotplug_ops->hotplug) {/* have the kset specific function add its stuff */ retval = kset->hotplug_ops->hotplug (kset, kobj, - &envp[i],NUM_ENVP - i, scratch, -BUFFER_SIZE - (scratch - buffer)); + &envp[i],NUM_ENVP - i, scratch, +BUFFER_SIZE - (scratch - buffer));if (retval) {pr_debug ("%s - hotplug() returned %d\n" , __FUNCTION__, retval);
@@ -193,10 +203,40 @@ static void kset_hotplug(const char *act __FUNCTION__, retval); exit: +kset_put (kset); +kobject_put (kobj);kfree (kobj_path); +kfree (work);kfree (buffer);kfree (envp); -return ; +} + +static void kset_hotplug (const char *action,struct kset *kset, +struct kobject *kobj) +{ +struct hotplug_work *work; + +if (!(work =kmalloc (sizeof (*work),GFP_KERNEL ))) +return ; + + work->action = action; +if (!(work->kset =kset_get (kset))) { +kfree (work); +return ; + } + +if (!(work->kobj =kobject_get (kobj))) { +kset_put (kset); +kfree (work); +return ; + } + +INIT_WORK (&work->work, kset_hotplug_work, work); + +if (keventd_up ()) +schedule_work (&work->work); +else +kset_hotplug_work (work); }void kobject_hotplug (const char *action,struct kobject *kobj)
_
Quoted 7 lines · Brian King
The following patch fixes a deadlock experienced when devices are being added to a bus both from a user process and eventd process. The eventd process was hung waiting on
dev->bus->subsys.rwsemwhich was held by another process, which was hung since it was callingcall_usermodehelperdirectly which was hung waiting for work scheduled on the eventd workqueue to complete. The patch fixes this by delaying thekobject_hotplugwork, running it from eventd if possible.
But why? Will this not still cause the same deadlock eventually? The call_usermodehelper function uses keventd, so what about users who call that function directly?
Also, you gratitously changed some of the whitespace in the file you were modifying, which isn't a nice thing to do :)
thanks,
greg k-h
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
Quoted 10 lines · Greg KH
Quoted 6 lines · Brian King
The following patch fixes a deadlock experienced when devices are
being added to a bus both from a user process and eventd process.
The eventd process was hung waiting ondev->bus->subsys.rwsemwhich
was held by another process, which was hung since it was callingcall_usermodehelperdirectly which was hung waiting for work scheduled on the eventd workqueue to complete. The patch fixes this by delaying
thekobject_hotplugwork, running it from eventd if possible.
Quoted 3 lines
But why? Will this not still cause the same deadlock eventually? The
call_usermodehelperfunction uses keventd, so what about users who call that function directly?
It fixes the problem as long as the rule of not holding locks/semaphores when calling call_usermodehelper holds. I don't see how the deadlock can occur for the scenario I hit with the fix, since the hotplug action runs completely asynchronously without the semaphore that was causing the deadlock. Now, could there be other places in the kernel today that call call_usermodehelper with locks that could also have deadlock issues? Probably.
Would you prefer a fix in call_usermodehelper itself? It could certainly be argued that calling call_usermodehelper with wait=0 should be allowed even when holding locks. Although, fixing it here is less obvious to me how to do because of the arguments to call_usermodehelper. I would imagine it would consist of creating a kernel_thread to preserve the caller's stack.
Quoted 2 lines
Also, you gratitously changed some of the whitespace in the file you were modifying, which isn't a nice thing to do :)
Sorry about that. Attached is a patch which fixes this. If call_usermodehelper is the proper place to fix this, then I'll look into rolling a new patch.
Thanks
-Brian
Signature · 3 lines
Brian King eServer Storage I/O IBM Linux Technology Center
The following patch fixes a deadlock experienced when devices are being added to a bus both from a user process and eventd process. The eventd process was hung waiting on dev->bus->subsys.rwsem which was held by another process, which was hung since it was calling call_usermodehelper directly which was hung waiting for work scheduled on the eventd workqueue to complete. The patch fixes this by delaying the kobject_hotplug work, running it from eventd if possible.
Backtraces of the hang:
0xc0000000017df300 1 0 0 0 D 0xc0000000017df7b0
swapper
SP(esp) PC(eip) Function(args)
0xc00000003fc9f460 0x0000000000000000 NO_SYMBOL or Userspace 0xc00000003fc9f4f0 0xc000000000058c40 .schedule +0xb4 0xc00000003fc9f5c0 0xc00000000005a464 .wait_for_completion +0x138 0xc00000003fc9f6c0 0xc00000000007c594 .call_usermodehelper +0x104 0xc00000003fc9f810 0xc00000000022d3e8 .kobject_hotplug +0x3c4 0xc00000003fc9f900 0xc00000000022d67c .kobject_add +0x134 0xc00000003fc9f9a0 0xc00000000012b3d8 .register_disk +0x70 0xc00000003fc9fa40 0xc00000000027dfe4 .add_disk +0x60 0xc00000003fc9fad0 0xc0000000002dc7dc .sd_probe +0x290 0xc00000003fc9fb80 0xc00000000026fbe8 .bus_match +0x94 0xc00000003fc9fc10 0xc00000000026ff70 .driver_attach +0x8c 0xc00000003fc9fca0 0xc000000000270104 .bus_add_driver +0x110 0xc00000003fc9fd50 0xc000000000270a18 .driver_register +0x38 0xc00000003fc9fdd0 0xc0000000002cd8f8 .scsi_register_driver +0x28 0xc00000003fc9fe50 0xc0000000004941d8 .init_sd +0x8c 0xc00000003fc9fee0 0xc00000000000c720 .init +0x25c 0xc00000003fc9ff90 0xc0000000000183ec .kernel_thread +0x4c
0xc00000003fab3380 4 1 0 0 D 0xc00000003fab3830
events/0
SP(esp) PC(eip) Function(args)
0xc00000003faaf6e0 0x0000000000000000 NO_SYMBOL or Userspace 0xc00000003faaf770 0xc000000000058c40 .schedule +0xb4 0xc00000003faaf840 0xc00000000022fa20 .rwsem_down_write_failed +0x14c 0xc00000003faaf910 0xc00000000026fed0 .bus_add_device +0x11c 0xc00000003faaf9b0 0xc00000000026e288 .device_add +0xd0 0xc00000003faafa50 0xc0000000002cdb00 .scsi_sysfs_add_sdev +0x8c 0xc00000003faafb00 0xc0000000002cbff8 .scsi_probe_and_add_lun +0xb04 0xc00000003faafc00 0xc0000000002ccca0 .scsi_add_device +0x90 0xc00000003faafcb0 0xc0000000002d9458 .ipr_worker_thread +0xc60 0xc00000003faafdc0 0xc00000000007cd9c .worker_thread +0x268 0xc00000003faafee0 0xc0000000000839cc .kthread +0x160 0xc00000003faaff90 0xc0000000000183ec .kernel_thread +0x4c
---
lib/kobject.c renamed from lib/kobject.c~kobject_hotplug_hang +46 -6
@@ -103,8 +103,14 @@ static void fill_kobj_path(struct kset *static unsigned long sequence_num;static spinlock_t sequence_lock =SPIN_LOCK_UNLOCKED ; -static void kset_hotplug (const char *action,struct kset *kset, -struct kobject *kobj) +struct hotplug_work { +struct work_struct work; +const char *action; +struct kset *kset; +struct kobject *kobj; +}; + +static void kset_hotplug_work (void *data) {char *argv [3 ];char **envp =NULL ;
@@ -116,22 +122,26 @@ static void kset_hotplug(const char *actchar *kobj_path =NULL ;char *name =NULL ;unsigned long seq; +struct hotplug_work *work = (struct hotplug_work *)data; +const char *action = work->action; +struct kset *kset = work->kset; +struct kobject *kobj = work->kobj;/* If the kset has a filter operation, call it. If it returns failure, no hotplug event is required. */ if (kset->hotplug_ops->filter) {if (!kset->hotplug_ops->filter (kset, kobj)) -return ; + goto exit; }pr_debug ("%s\n" , __FUNCTION__);if (!hotplug_path[0 ]) -return ; + goto exit; envp =kmalloc (NUM_ENVP *sizeof (char *),GFP_KERNEL );if (!envp) -return ; + goto exit;memset (envp,0x00 ,NUM_ENVP *sizeof (char *)); buffer =kmalloc (BUFFER_SIZE ,GFP_KERNEL );
@@ -193,10 +203,40 @@ static void kset_hotplug(const char *act __FUNCTION__, retval); exit: +kset_put (kset); +kobject_put (kobj);kfree (kobj_path); +kfree (work);kfree (buffer);kfree (envp); -return ; +} + +static void kset_hotplug (const char *action,struct kset *kset, +struct kobject *kobj) +{ +struct hotplug_work *work; + +if (!(work =kmalloc (sizeof (*work),GFP_KERNEL ))) +return ; + + work->action = action; +if (!(work->kset =kset_get (kset))) { +kfree (work); +return ; + } + +if (!(work->kobj =kobject_get (kobj))) { +kset_put (kset); +kfree (work); +return ; + } + +INIT_WORK (&work->work, kset_hotplug_work, work); + +if (keventd_up ()) +schedule_work (&work->work); +else +kset_hotplug_work (work); }void kobject_hotplug (const char *action,struct kobject *kobj)
_
Quoted 5 lines · Brian King
Would you prefer a fix in
call_usermodehelperitself? It could certainly be argued that callingcall_usermodehelperwith wait=0 should be allowed even when holding locks. Although, fixing it here is less obvious to me how to do because of the arguments tocall_usermodehelper. I would imagine it would consist of creating akernel_threadto preserve the caller's stack.
Yes, I think call_usermodehelper should be changed to create a new kernel thread for every call. That would solve this problem, and any future races that might happen. Care to work on that?
thanks,
greg k-h
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
Quoted 9 lines · Greg KH
Quoted 5 lines · Brian King
Would you prefer a fix in
call_usermodehelperitself? It could certainly be argued that callingcall_usermodehelperwith wait=0 should be allowed even when holding locks. Although, fixing it here is less obvious to me how to do because of the arguments tocall_usermodehelper. I would imagine it would consist of creating akernel_threadto preserve the caller's stack.
Quoted 3 lines
Yes, I think
call_usermodehelpershould be changed to create a new kernel thread for every call. That would solve this problem, and any future races that might happen. Care to work on that?
I'll give it a shot.
-Brian
Signature · 15 lines
Brian King eServer Storage I/O IBM Linux Technology Center ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click _______________________________________________ Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net [email protected] https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
Quoted 10 lines · Greg KH
Quoted 5 lines · Brian King
Would you prefer a fix in
call_usermodehelperitself? It could certainly be argued that callingcall_usermodehelperwith wait=0 should be allowed even when holding locks. Although, fixing it here is less obvious to me how to do because of the arguments tocall_usermodehelper. I would imagine it would consist of creating akernel_threadto preserve the caller's stack.Yes, I think
call_usermodehelpershould be changed to create a new kernel thread for every call.
It does that already. But that thread is parented by keventd. This was done to avoid all the various nasty things which can happen when you have a kernel thread and a hotplug helper which are parented by a random userspace process. All the crap which it might have inherited: uid? gid? signals? nice? rtprio? rlimits? namespace?
The deadlock opportunity occurs during the call_usermodehelper() handoff to keventd, which is synchronous.
2-3 years back I did have a call_usermodehelper() which was fully async. It was pretty unpleasant because of the need to atomically allocate arbitrary amounts of memory to hold the argv[] and endp[] arrays, to pass them between a couple of threads and to then correctly free it all up again.
Quoted 9 lines · Andrew Morton
Quoted 8 lines · Greg KH
Quoted 5 lines · Brian King
Would you prefer a fix in
call_usermodehelperitself? It could certainly be argued that callingcall_usermodehelperwith wait=0 should be allowed even when holding locks. Although, fixing it here is less obvious to me how to do because of the arguments tocall_usermodehelper. I would imagine it would consist of creating akernel_threadto preserve the caller's stack.
Quoted 8 lines
Quoted 2 lines
Yes, I think
call_usermodehelpershould be changed to create a new kernel thread for every call.It does that already. But that thread is parented by keventd. This was done to avoid all the various nasty things which can happen when you have a kernel thread and a hotplug helper which are parented by a random userspace process. All the crap which it might have inherited: uid? gid? signals? nice? rtprio? rlimits? namespace?
Yeah, good point.
Quoted 8 lines
The deadlock opportunity occurs during the
call_usermodehelper()handoff to keventd, which is synchronous.2-3 years back I did have a
call_usermodehelper()which was fully async. It was pretty unpleasant because of the need to atomically allocate arbitrary amounts of memory to hold the argv[] and endp[] arrays, to pass them between a couple of threads and to then correctly free it all up again.
Ok, you've convinced me of the mess that would cause. So what should we do to help fix this? Serialize call_usermodehelper()?
thanks,
greg k-h
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
Quoted 4 lines · Greg KH
Quoted 4 lines
The deadlock opportunity occurs during the
call_usermodehelper()handoff to keventd, which is synchronous.
Quoted 8 lines
Quoted 5 lines
2-3 years back I did have a
call_usermodehelper()which was fully async. It was pretty unpleasant because of the need to atomically allocate arbitrary amounts of memory to hold the argv[] and endp[] arrays, to pass them between a couple of threads and to then correctly free it all up again.Ok, you've convinced me of the mess that would cause. So what should we do to help fix this? Serialize
call_usermodehelper()?
May as well bring back call_usermodehelper_async() I guess.
There are two patches here, and they are totally untested...
Add some generally-useful string replication functions which are required by call_usermodehelper_async():
void *kzmalloc(size_t size, int gfp_flags);
kmalloc() then bzero().
char *kstrdup(char *p, int gfp_flags);
kmalloc() then strcpy()
char **kstrdup_vec(char **vec, int gfp_flags);
duplicate an argv[]-style string array
void kfree_strvec(char **vec);
free up the result of a previous kstrdup_vec()
---
25-akpm/drivers/md/dm-ioctl.c | 17 ++---------- 25-akpm/include/linux/slab.h | 5 +++ 25-akpm/mm/slab.c | 57 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 65 insertions(+), 14 deletions(-)
drivers/md/dm-ioctl.c renamed from drivers/md/dm-ioctl.c~kstrdup-and-friends +3 -14
@@ -118,17 +118,6 @@ static struct hash_cell *__get_uuid_cellreturn NULL ; } -/*----------------------------------------------------------------- -* Inserting, removing and renaming a device. -*---------------------------------------------------------------*/ -static inline char *kstrdup (const char *str) -{ -char *r =kmalloc (strlen (str) +1 ,GFP_KERNEL ); -if (r) -strcpy (r, str); -return r; -} -static struct hash_cell *alloc_cell (const char *name,const char *uuid,struct mapped_device *md) {
@@ -138,7 +127,7 @@ static struct hash_cell *alloc_cell(consif (!hc)return NULL ; - hc->name =kstrdup (name); + hc->name =kstrdup (name,GFP_KERNEL );if (!hc->name) {kfree (hc);return NULL ;
@@ -148,7 +137,7 @@ static struct hash_cell *alloc_cell(cons hc->uuid =NULL ; else { - hc->uuid =kstrdup (uuid); + hc->uuid =kstrdup (uuid,GFP_KERNEL );if (!hc->uuid) {kfree (hc->name);kfree (hc);
@@ -270,7 +259,7 @@ int dm_hash_rename(const char *old, cons/* * duplicate new. */ - new_name =kstrdup (new); + new_name =kstrdup (new,GFP_KERNEL );if (!new_name)return -ENOMEM ;
include/linux/slab.h renamed from include/linux/slab.h~kstrdup-and-friends +5 -0
@@ -117,6 +117,11 @@ void ptrinfo(unsigned long addr);extern atomic_t slab_reclaim_pages; +void *kzmalloc (size_t size,int gfp_flags); +char *kstrdup (const char *p,int gfp_flags); +char **kstrdup_vec (char **vec,int gfp_flags); +void kfree_strvec (char **vec); +#endif /* __KERNEL__ */ #endif /* _LINUX_SLAB_H */
mm/slab.c renamed from mm/slab.c~kstrdup-and-friends +57 -0
60 lines not on this page · raw message
_
call_usermodehelper() has to synchronously wait until it has handed its local variables off to keventd. This can lead to deadlocks when the caller holds semphores which keventd may also want.
To fix this we introduce call_usermodehelper_async(), which does not block on a keventd response.
---
25-akpm/include/linux/kmod.h | 5 ++- 25-akpm/kernel/kmod.c | 69 +++++++++++++++++++++++++++++++++++++------ 2 files changed, 64 insertions(+), 10 deletions(-)
include/linux/kmod.h renamed from include/linux/kmod.h~call_usermodehelper_async +4 -1
@@ -32,7 +32,10 @@ static inline int request_module(const c#endif #define try_then_request_module(x, mod...) ((x) ?: (request_module(mod), (x))) -extern int call_usermodehelper (char *path,char *argv[],char *envp[],int wait); + +int call_usermodehelper (char *path,char **argv,char **envp,int wait); +int call_usermodehelper_async (char *path,char **argv, +char **envp,int gfp_flags);#ifdef CONFIG_HOTPLUG extern char hotplug_path [];
kernel/kmod.c renamed from kernel/kmod.c~call_usermodehelper_async +60 -9
100 lines not on this page · raw message
_
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
Quoted 11 lines · Andrew Morton
Quoted 9 lines · Greg KH
Quoted 8 lines
The deadlock opportunity occurs during the
call_usermodehelper()handoff tokeventd, which is synchronous.
2-3 years back I did have a
call_usermodehelper()which was fully async. It was pretty unpleasant because of the need to atomically allocate arbitrary amounts of memory to hold the argv[] and endp[] arrays, to pass
them between a couple of threads and to then correctly free it all up
again.
Quoted 8 lines
Quoted 2 lines
Ok, you've convinced me of the mess that would cause. So what should we do to help fix this? Serialize
call_usermodehelper()?May as well bring back
call_usermodehelper_async()I guess.There are two patches here, and they are totally untested...
I loaded the patches on my ppc64 box and they worked fine after I fixed a compile bug. The attached patch fixes the compile bug and changes the call_usermodehelper call in kset_hotplug to call_usermodehelper_async.
-Brian
Signature · 3 lines
Brian King eServer Storage I/O IBM Linux Technology Center
Fixes a compile error in call_usermodehelper_async and changes kset_hotplug to use call_usermodehelper_async, since it is called with a semaphore held, which can result in a deadlock.
---
kernel/kmod.c renamed from kernel/kmod.c~call_usermodehelper_kobject +1 -1
@@ -296,7 +296,7 @@ int call_usermodehelper_async(char *path {struct subprocess_info *sub_info; -if (system_state !=SYSTEM_RUNNING ) +if (!system_running)return -EBUSY ;if (path[0 ] =='\0' ) goto out;
lib/kobject.c renamed from lib/kobject.c~call_usermodehelper_kobject +1 -1
@@ -187,7 +187,7 @@ static void kset_hotplug(const char *actpr_debug ("%s: %s %s %s %s %s %s %s\n" , __FUNCTION__, argv[0 ], argv[1 ], envp[0 ], envp[1 ], envp[2 ], envp[3 ], envp[4 ]); - retval =call_usermodehelper (argv[0 ], argv, envp,0 ); + retval =call_usermodehelper_async (argv[0 ], argv, envp,GFP_KERNEL );if (retval)pr_debug ("%s - call_usermodehelper returned %d\n" , __FUNCTION__, retval);
_
Quoted 5 lines · Brian King
Quoted 5 lines
Quoted 3 lines
Ok, you've convinced me of the mess that would cause. So what should we do to help fix this? Serialize
call_usermodehelper()?
Quoted 8 lines
Quoted 4 lines
May as well bring back
call_usermodehelper_async()I guess.There are two patches here, and they are totally untested...
I loaded the patches on my ppc64 box and they worked fine after I fixed a compile bug. The attached patch fixes the compile bug and changes the
call_usermodehelpercall inkset_hotplugtocall_usermodehelper_async.
yup, thanks. I've changed the patch in my tree so that we always perform the fully-async operation if call_usermodehelper() is passed "wait=0". It gets the new code some more testing and keeps the kernel API simpler.
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
I applied and tested the new patches in the mm tree and it works fine for me.
Thanks
-Brian
Quoted 22 lines · Andrew Morton
Quoted 5 lines · Brian King
Quoted 5 lines
Quoted 3 lines
Ok, you've convinced me of the mess that would cause. So what should we
do to help fix this? Serialize
call_usermodehelper()?Quoted 8 lines
Quoted 4 lines
May as well bring back
call_usermodehelper_async()I guess.There are two patches here, and they are totally untested...
I loaded the patches on my ppc64 box and they worked fine after I fixed a compile bug. The attached patch fixes the compile bug and changes the
call_usermodehelpercall inkset_hotplugtocall_usermodehelper_async.yup, thanks. I've changed the patch in my tree so that we always perform the fully-async operation if
call_usermodehelper()is passed "wait=0". It gets the new code some more testing and keeps the kernel API simpler.
Signature · 15 lines
Brian King eServer Storage I/O IBM Linux Technology Center ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click _______________________________________________ Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net [email protected] https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
Quoted 4 lines · Andrew Morton
void *kzmalloc(size_t size, int
gfp_flags);
kmalloc()thenbzero().
Sure, make the kernel-janitor's list even longer now :)
Quoted 3 lines
char *kstrdup(char *p, int
gfp_flags);
kmalloc()thenstrcpy()
Haha, that's Rusty's "Is Linus Awake" patch he tries to send every 6 months or so...
greg k-h
-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click
_______________________________________________
Linux-hotplug-devel mailing list http://linux-hotplug.sourceforge.net
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel
Quoted 7 lines · * Brian King ([email protected]) wrote:
The following patch fixes a deadlock experienced when devices are being added to a bus both from a user process and eventd process. The eventd process was hung waiting on
dev->bus->subsys.rwsemwhich was held by another process, which was hung since it was callingcall_usermodehelperdirectly which was hung waiting for work scheduled on the eventd workqueue to complete. The patch fixes this by delaying thekobject_hotplugwork, running it from eventd if possible.
Couple of gratuitous formatting changes.
lib/kobject.c static void kset_hotplug(const char *act · line 131
Quoted 2 lines
lib/kobject.c renamed from lib/kobject.c~kobject_hotplug_hang +1
-1-failure, no hotplug event is required. */ +failure, no hotplug event is required. */
here
lib/kobject.c static void kset_hotplug(const char *act · line 190
Quoted 4 lines
lib/kobject.c renamed from lib/kobject.c~kobject_hotplug_hang +2
-2- &envp[i],NUM_ENVP - i, scratch, - BUFFER_SIZE - (scratch - buffer)); + &envp[i],NUM_ENVP - i, scratch, + BUFFER_SIZE - (scratch - buffer));
and here.
Overall, why does it seem to just be pushing the problem around? Similarly, if you did your work in a child of keventd the problem would move away.
thanks,
-chris
Signature · 1 line
Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net
Quoted 8 lines · * Brian King ([email protected]) wrote:
I have been running into some kernel hangs due to
call_usermodehelper. Looking at the backtrace, it looks to me like there are deadlock issues with adding devices from work queues. Attached is a sample backtrace from one of the hangs I experienced. My question is why doescall_usermodehelperdo 2 different things depending on whether or not it is called from the kevent task? It appears that the simple way to fix the hang would be to never havecall_usermodehelperuse awork_queuesince it must be called from process context anyway, or am I missing something?
It does two different things because it's trying to run from keventd. In the case that current is not keventd, it schedules the work, so keventd will pick that work up later to run in it's process context.
How early is this hang? It looks like init thread adds work and waits for it's completion while holding a semaphore. It is never woken up by keventd which is sleeping waiting for wakeup from semaphore that init thread took.
Seems troubling to hold the sem while calling call_usermodehelper, as that could go off for a long time.
swapper events/0
------- --------bus_add_driver() ipr_worker_thread()
down_write(&bus->subsys.rwsem) /* here */ scsi_add_device()
driver_attach() bus_add_device()
... /* sem is already taken */
sd_probe() down_write(&bus->subsys.rwsem)
add_disk() schedule() /* bye! */
...
call_usermodehelper()
wait_for_completion() /* never awoken by keventd */thanks,
-chris
Signature · 1 line
Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net
Quoted 12 lines · Chris Wright
Quoted 8 lines · * Brian King ([email protected]) wrote:
I have been running into some kernel hangs due to
call_usermodehelper. Looking at the backtrace, it looks to me like there are deadlock issues with adding devices from work queues. Attached is a sample backtrace from one of the hangs I experienced. My question is why doescall_usermodehelperdo 2 different things depending on whether or not it is called from the kevent task? It appears that the simple way to fix the hang would be to never havecall_usermodehelperuse awork_queuesince it must be called from process context anyway, or am I missing something?
Quoted 5 lines
It does two different things because it's trying to run from keventd. In the case that current is not keventd, it schedules the work, so keventd will pick that work up later to run in it's process context.
How early is this hang?
Pretty early. Boot time, loading scsi drivers. While initializing the third scsi adapter on the system a device shows up dynamically on the first adapter, the LLD schedules work to call scsi_add_device and we end up with the hang.
It looks like init thread adds work and waits
Quoted 6 lines
for it's completion while holding a semaphore. It is never woken up by keventd which is sleeping waiting for wakeup from semaphore that init thread took.
Seems troubling to hold the sem while calling
call_usermodehelper, as that could go off for a long time.
I agree. There is another similar hang I ran into recently with the scsi core in that I was calling scsi_add_device from keventd, which ended up sleeping on the scan_mutex since the module load process was calling scsi_scan_host. scsi_scan_host was in the same kobject hotplug code, calling call_usermodehelper. I figured my LLD shouldn't be doing that, so I synchronized the events. Perhaps that was the wrong fix... I suppose I could have changed the LLD to always call scsi_scan_host/scsi_add_device/etc. from keventd...
-Brian