+ kmod-avoid-deadlock-by-recursive-kmod-call.patch added to -mm tree
Quoted 4 lines
(quoted diff) +0
-0@@ -449,6 +460,16 @@ int call_usermodehelper_exec(struct subp retval = -EBUSY; goto out; }
Review int call_usermodehelper_exec(struct subp · line 472
Quoted 10 lines
(quoted diff) +10
-0+ /* + * Worker thread must not wait for khelper thread at below + * wait_for_completion() if the thread was created with CLONE_VFORK + * flag, for khelper thread is already waiting for the thread at + * wait_for_completion() in do_fork(). + */ + if (wait != UMH_NO_WAIT && current == kmod_thread_locker) { + retval = -EBUSY; + goto out; + }
So, this is because khelper_wq's max_active == 1.
Can't we simply kill khelper_wq and use system_unbound_wq instead?
Oleg.
Quoted 17 lines · Oleg Nesterov
Quoted 4 lines
(quoted diff) +0
-0@@ -449,6 +460,16 @@ int call_usermodehelper_exec(struct subp retval = -EBUSY; goto out; }Review int call_usermodehelper_exec(struct subp · line 472
Quoted 10 lines
(quoted diff) +10
-0+ /* + * Worker thread must not wait for khelper thread at below + * wait_for_completion() if the thread was created with CLONE_VFORK + * flag, for khelper thread is already waiting for the thread at + * wait_for_completion() in do_fork(). + */ + if (wait != UMH_NO_WAIT && current == kmod_thread_locker) { + retval = -EBUSY; + goto out; + }So, this is because khelper_wq's
max_active== 1.
Quoted 1 line
Can't we simply kill
khelper_wqand usesystem_unbound_wqinstead?
I'd prefer that, because then we'd hit the existing "too many modprobes" check.
Thanks,
Rusty.
Quoted 19 lines · Rusty Russell
Quoted 17 lines · Oleg Nesterov
Quoted 4 lines
(quoted diff) +0
-0@@ -449,6 +460,16 @@ int call_usermodehelper_exec(struct subp retval = -EBUSY; goto out; }Review int call_usermodehelper_exec(struct subp · line 472
Quoted 10 lines
(quoted diff) +10
-0+ /* + * Worker thread must not wait for khelper thread at below + * wait_for_completion() if the thread was created with CLONE_VFORK + * flag, for khelper thread is already waiting for the thread at + * wait_for_completion() in do_fork(). + */ + if (wait != UMH_NO_WAIT && current == kmod_thread_locker) { + retval = -EBUSY; + goto out; + }So, this is because khelper_wq's
max_active== 1.
Quoted 4 lines
Quoted 1 line
Can't we simply kill
khelper_wqand usesystem_unbound_wqinstead?I'd prefer that, because then we'd hit the existing "too many modprobes" check.
Hmm. Why? I mean, why do you think that s/khelper_wq/system_unbound_wq/ leads to recursive __request_module's ?
Note that that this patch (which adds kmod_thread_locker) can not limit the recursive modprobe loop.
OK, yes, with system_unbound_wq we can hit this warning if we have max_modprobes UMH_WAIT_EXEC's resulting in __request_module at the same time, but probably this is good?
I guess I missed something, could you explain? Just curious.
Oleg.
Quoted 10 lines · Oleg Nesterov
Quoted 5 lines · Rusty Russell
Quoted 1 line · Oleg Nesterov
Can't we simply kill
khelper_wqand usesystem_unbound_wqinstead?I'd prefer that, because then we'd hit the existing "too many modprobes" check.
Hmm. Why? I mean, why do you think that s/khelper_wq/system_unbound_wq/ leads to recursive __request_module's ?
Quoted 7 lines
Note that that this patch (which adds
kmod_thread_locker) can not limit the recursive modprobe loop.OK, yes, with
system_unbound_wqwe can hit this warning if we havemax_modprobesUMH_WAIT_EXEC's resulting in__request_moduleat the same time, but probably this is good?
Yes, that's what I'm saying.
We already have a check against too many modprobes, it might be best to use it.
Cheers,
Rusty.
Quoted 16 lines · Rusty Russell
Quoted 14 lines · Oleg Nesterov
Quoted 5 lines · Rusty Russell
Quoted 1 line · Oleg Nesterov
Can't we simply kill
khelper_wqand usesystem_unbound_wqinstead?I'd prefer that, because then we'd hit the existing "too many modprobes" check.
Hmm. Why? I mean, why do you think that s/khelper_wq/system_unbound_wq/ leads to recursive __request_module's ?
Note that that this patch (which adds
kmod_thread_locker) can not limit the recursive modprobe loop.
Quoted 8 lines
Quoted 3 lines
OK, yes, with
system_unbound_wqwe can hit this warning if we havemax_modprobesUMH_WAIT_EXEC's resulting in__request_moduleat the same time, but probably this is good?Yes, that's what I'm saying.
We already have a check against too many modprobes, it might be best to use it.
Confused... in this case I do not understand why do you dislike the idea to kill khelper_wq.
Help!
Oleg.
Quoted 2 lines · Oleg Nesterov
Confused... in this case I do not understand why do you dislike the idea to kill
khelper_wq.
Yes, you are confused. I was agreeing with you:
Quoted 1 line · Oleg Nesterov
Can't we simply kill
khelper_wqand usesystem_unbound_wqinstead?
I'd prefer that, because then we'd hit the existing "too many modprobes" check.
Thanks,
Rusty.
I thought "I'd prefer that" was pretty clear??
Thanks,
Rusty.
Quoted 8 lines · Rusty Russell
Quoted 2 lines · Oleg Nesterov
Confused... in this case I do not understand why do you dislike the idea to kill
khelper_wq.Yes, you are confused. I was agreeing with you:
Quoted 1 line · Oleg Nesterov
Can't we simply kill
khelper_wqand usesystem_unbound_wqinstead?
BTW, why does it have to be unbound_wq? Is it expected consume large amount of CPU cycles?
Thanks.
Signature · 1 line
tejun
Quoted 6 lines · Rusty Russell
Quoted 2 lines · Oleg Nesterov
Confused... in this case I do not understand why do you dislike the idea to kill
khelper_wq.Yes, you are confused. I was agreeing with you:
Ah...
Quoted 1 line
I thought "I'd prefer that" was pretty clear??
It was! Looking back, I do not understand how did I manage to misread your email.
Thanks Rusty,
Oleg.
Quoted 2 lines · Tejun Heo
BTW, why does it have to be
unbound_wq?
Perhaps we can use another system_wq, but afaics WQ_UNBOUND makes sense in this case. I mean, there is no reason to bind this work to any CPU. See also below.
Quoted 2 lines
Is it expected consume large
amount of CPU cycles?
Currently __call_usermodehelper() does kernel_thread(), this is almost all. But it can block waiting for kernel_execve().
Not sure this really makes sense, but if we kill khelper_wq perhaps we can simplify this code a bit. We can change __call_usermodehelper()
if (wait == UMH_WAIT_PROC) - pid = kernel_thread(wait_for_helper, sub_info, - CLONE_FS | CLONE_FILES | SIGCHLD); + wait_for_helper(...); else
IOW, the worker thread itself can do the UMH_WAIT_PROC work. This makes this work really "long running", but then we can kill sub_info->complete and use flush_work().
Oleg.
Hello,
Quoted 3 lines · Oleg Nesterov
Perhaps we can use another
system_wq, but afaicsWQ_UNBOUNDmakes sense in this case. I mean, there is no reason to bind this work to any CPU. See also below.
I've been trying to nudge people away from using special wqs or flags unless really necessary. Other than non-reentrancy and strict ordering, all behaviors are mostly for optimization and using them incorrectly / spuriously usually doesn't cause any visible failure, making it very easy to get them wrong and if you have enough of wrong / unnecessary usages in tree, the whole thing gets really confusing and difficult to update in the future.
Quoted 5 lines
Quoted 2 lines
Is it expected consume large
amount of CPU cycles?Currently
__call_usermodehelper()doeskernel_thread(), this is almost all. But it can block waiting forkernel_execve().
Blocking is completely fine on any workqueue. The only reason to require the use of unbound_wq is if work items would burn a lot of CPU cycles. In such cases, we want to let the scheduler have full jurisdiction instead of wq regulating concurrency.
Quoted 9 lines
Not sure this really makes sense, but if we kill
khelper_wqperhaps we can simplify this code a bit. We can change__call_usermodehelper()if (wait == UMH_WAIT_PROC) - pid = kernel_thread(wait_for_helper, sub_info, - CLONE_FS | CLONE_FILES | SIGCHLD); + wait_for_helper(...); else
Quoted 3 lines
IOW, the worker thread itself can do the
UMH_WAIT_PROCwork. This makes this work really "long running", but then we can killsub_info->completeand useflush_work().
Again, long-running in the sense that the work item spending a lot of time sleeping should be fine on system_wq or any other wq with default attributes. AFAICS, the things to consider here are...
* If work items are expected to consume large amount of CPU cycles (as in crypto work items), consider using system_unbound_wq / WQ_UNBOUND.
* If per-domain concurrency limit is necessary (ie. the number of concurrent work items doing this particular task should be limited rather than consuming global system_wq limit), a dedicated workqueue would be better.
Thanks.
Signature · 1 line
tejun
Hi Tejun,
Quoted 6 lines · Tejun Heo
Quoted 3 lines · Oleg Nesterov
Perhaps we can use another
system_wq, but afaicsWQ_UNBOUNDmakes sense in this case. I mean, there is no reason to bind this work to any CPU. See also below.
Quoted 7 lines
I've been trying to nudge people away from using special wqs or flags unless really necessary. Other than non-reentrancy and strict ordering, all behaviors are mostly for optimization and using them incorrectly / spuriously usually doesn't cause any visible failure, making it very easy to get them wrong and if you have enough of wrong / unnecessary usages in tree, the whole thing gets really confusing and difficult to update in the future.
You know, I am a bit suprized. To me, it is the !WQ_UNBOUND case is "special". IOW, I think we need some reason to bind the work to the specific CPU.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Is it expected consume large
amount of CPU cycles?Currently
__call_usermodehelper()doeskernel_thread(), this is almost all. But it can block waiting forkernel_execve().Blocking is completely fine on any workqueue.
I understand. But, the blocked worker "consumes" nr_active/worker.
Quoted 4 lines
The only reason to
require the use ofunbound_wqis if work items would burn a lot of CPU cycles. In such cases, we want to let the scheduler have full jurisdiction instead of wq regulating concurrency.
I am starting to think I do not understand this code at all. OK, perhaps unbound_wq should be used for cpu-intensive works only.
But why do you think that we should use a !WQ_UNBOUND workque instead of khelper_wq? And why "a lot of CPU" is the only reason for WQ_UNBOUND?
Quoted 7 lines
* If work items are expected to consume large amount of CPU cycles (as in crypto work items), consider using
system_unbound_wq/WQ_UNBOUND.* If per-domain concurrency limit is necessary (ie. the number of concurrent work items doing this particular task should be limited rather than consuming global
system_wqlimit), a dedicated workqueue would be better.
So I don't understand whether you like the idea to kill khelper_wq and use some system_ wq or not (and fix the bug).
I do not really like the current patch. If nothing else, what if UMH_WAIT_EXEC request actually needs another UMH_WAIT_EXEC/PROC request to succeed?
Tetsuo, we spent a lot of time discussing other problems. What do you think about s/khelper/system/ instead of this patch?
Oleg.
Hello, Oleg.
Quoted 11 lines · Oleg Nesterov
Quoted 7 lines
I've been trying to nudge people away from using special wqs or flags unless really necessary. Other than non-reentrancy and strict ordering, all behaviors are mostly for optimization and using them incorrectly / spuriously usually doesn't cause any visible failure, making it very easy to get them wrong and if you have enough of wrong / unnecessary usages in tree, the whole thing gets really confusing and difficult to update in the future.
You know, I am a bit suprized. To me, it is the !WQ_UNBOUND case is "special". IOW, I think we need some reason to bind the work to the specific CPU.
It's part historical due to the way workqueue was originally implemented and part because work items are expected to consume little CPU cycles and access data which the issuer accessed, but the question really isn't about whether unbound would be marginally better or not. We may change the default behavior in the future as implementation evolves or hardware environment changes and having more things which claim special treatment without clear reason makes things needlessly complicated, so the question to ask is whether the default behavior doesn't fit enough that the extra flag is really required.
Quoted 3 lines
Quoted 1 line
Blocking is completely fine on any workqueue.
I understand. But, the blocked worker "consumes" nr_active/worker.
If it's expected to consume high number of nr_active and needs limiting, creating a dedicated wq as an isolation domain would be a good idea. It's not like wqs are expensive or complex to use after all.
Quoted 11 lines
Quoted 4 lines
The only reason to
require the use ofunbound_wqis if work items would burn a lot of CPU cycles. In such cases, we want to let the scheduler have full jurisdiction instead of wq regulating concurrency.I am starting to think I do not understand this code at all. OK, perhaps
unbound_wqshould be used for cpu-intensive works only.But why do you think that we should use a !WQ_UNBOUND workque instead of
khelper_wq? And why "a lot of CPU" is the only reason forWQ_UNBOUND?
As I wrote, it's not about "should use bound", it's about having enough justification for deviating from the default.
Thanks.
Signature · 1 line
tejun
Re: + kmod-avoid-deadlock-by-recursive-kmod-call.patch added to-mm tree
In reply to Oleg NesterovQuoted 1 line · Oleg Nesterov
What do you think about s/khelper/system/ instead of this patch?
I'm not catching up with this topic. But I'm fine with any solution provided that it can handle
some kernel function calls request_module()
request_module() triggers call_usermodehelper("/sbin/modprobe", UMH_WAIT_PROC)
/sbin/modprobe loads a kernel module
loading a kernel module triggers kobject_uevent_env()
kobject_uevent_env() calls call_usermodehelper("/sbin/hotplug", UMH_WAIT_EXEC)
do_execve("/sbin/hotplug") calls request_module("binfmt-0000")
request_module("binfmt-0000") triggers call_usermodehelper("/sbin/modprobe", UMH_WAIT_PROC)
/sbin/modprobe fails to load binfmt-0000 module
call trace on an UP machine.
kmod-avoid-deadlock-by-recursive-kmod-call.patch is for avoiding call_usermodehelper("/sbin/modprobe", UMH_WAIT_PROC) called from call_usermodehelper("/sbin/hotplug", UMH_WAIT_EXEC).
Re: + kmod-avoid-deadlock-by-recursive-kmod-call.patch added to-mm tree
Quoted 6 lines · Tetsuo Handa
Quoted 1 line · Oleg Nesterov
What do you think about s/khelper/system/ instead of this patch?
I'm not catching up with this topic. But I'm fine with any solution provided that it can handle
I think it should...
Tetsuo, all, sorry I am busy today. I'll try to return tomorrow to continue this (and off-list) discussion.
Oleg.