Skip to contentSkip to conversation

+ kmod-avoid-deadlock-by-recursive-kmod-call.patch added to -mm tree

DISCUSSION Oleg Nesterov 4 participants 14 messages Jan 26, 2012 – Feb 6, 2012

lore
Share & tools
lore
#0 lore raw
Quoted 4 lines
(quoted diff) +0 -0
@@ -449,6 +460,16 @@ int call_usermodehelper_exec(struct subp
 		retval = -EBUSY;
 		goto out;
 	}

Review int call_usermodehelper_exec(struct subp · line 472

Quoted 10 lines
(quoted diff) +10 -0
+	/*
+	 * Worker thread must not wait for khelper thread at below
+	 * wait_for_completion() if the thread was created with CLONE_VFORK
+	 * flag, for khelper thread is already waiting for the thread at
+	 * wait_for_completion() in do_fork().
+	 */
+	if (wait != UMH_NO_WAIT && current == kmod_thread_locker) {
+		retval = -EBUSY;
+		goto out;
+	}

So, this is because khelper_wq's max_active == 1.

Can't we simply kill khelper_wq and use system_unbound_wq instead?

Oleg.

#1 lore raw
Quoted 17 lines · Oleg Nesterov
Quoted 4 lines
(quoted diff) +0 -0
@@ -449,6 +460,16 @@ int call_usermodehelper_exec(struct subp
 		retval = -EBUSY;
 		goto out;
 	}

Review int call_usermodehelper_exec(struct subp · line 472

Quoted 10 lines
(quoted diff) +10 -0
+	/*
+	 * Worker thread must not wait for khelper thread at below
+	 * wait_for_completion() if the thread was created with CLONE_VFORK
+	 * flag, for khelper thread is already waiting for the thread at
+	 * wait_for_completion() in do_fork().
+	 */
+	if (wait != UMH_NO_WAIT && current == kmod_thread_locker) {
+		retval = -EBUSY;
+		goto out;
+	}

So, this is because khelper_wq's max_active == 1.

Quoted 1 line

Can't we simply kill khelper_wq and use system_unbound_wq instead?

I'd prefer that, because then we'd hit the existing "too many modprobes" check.

Thanks,
Rusty.

#2 lore raw
Quoted 19 lines · Rusty Russell
Quoted 17 lines · Oleg Nesterov
Quoted 4 lines
(quoted diff) +0 -0
@@ -449,6 +460,16 @@ int call_usermodehelper_exec(struct subp
 		retval = -EBUSY;
 		goto out;
 	}

Review int call_usermodehelper_exec(struct subp · line 472

Quoted 10 lines
(quoted diff) +10 -0
+	/*
+	 * Worker thread must not wait for khelper thread at below
+	 * wait_for_completion() if the thread was created with CLONE_VFORK
+	 * flag, for khelper thread is already waiting for the thread at
+	 * wait_for_completion() in do_fork().
+	 */
+	if (wait != UMH_NO_WAIT && current == kmod_thread_locker) {
+		retval = -EBUSY;
+		goto out;
+	}

So, this is because khelper_wq's max_active == 1.

Quoted 4 lines
Quoted 1 line

Can't we simply kill khelper_wq and use system_unbound_wq instead?

I'd prefer that, because then we'd hit the existing "too many modprobes" check.

Hmm. Why? I mean, why do you think that s/khelper_wq/system_unbound_wq/ leads to recursive __request_module's ?

Note that that this patch (which adds kmod_thread_locker) can not limit the recursive modprobe loop.

OK, yes, with system_unbound_wq we can hit this warning if we have max_modprobes UMH_WAIT_EXEC's resulting in __request_module at the same time, but probably this is good?

I guess I missed something, could you explain? Just curious.

Oleg.

#3 lore raw
Quoted 10 lines · Oleg Nesterov
Quoted 5 lines · Rusty Russell
Quoted 1 line · Oleg Nesterov

Can't we simply kill khelper_wq and use system_unbound_wq instead?

I'd prefer that, because then we'd hit the existing "too many modprobes" check.

Hmm. Why? I mean, why do you think that s/khelper_wq/system_unbound_wq/ leads to recursive __request_module's ?

Quoted 7 lines

Note that that this patch (which adds kmod_thread_locker) can not limit the recursive modprobe loop.

OK, yes, with system_unbound_wq we can hit this warning if we have max_modprobes UMH_WAIT_EXEC's resulting in __request_module at the same time, but probably this is good?

Yes, that's what I'm saying.

We already have a check against too many modprobes, it might be best to use it.

Cheers,
Rusty.

#4 lore raw
Quoted 16 lines · Rusty Russell
Quoted 14 lines · Oleg Nesterov
Quoted 5 lines · Rusty Russell
Quoted 1 line · Oleg Nesterov

Can't we simply kill khelper_wq and use system_unbound_wq instead?

I'd prefer that, because then we'd hit the existing "too many modprobes" check.

Hmm. Why? I mean, why do you think that s/khelper_wq/system_unbound_wq/ leads to recursive __request_module's ?

Note that that this patch (which adds kmod_thread_locker) can not limit the recursive modprobe loop.

Quoted 8 lines
Quoted 3 lines

OK, yes, with system_unbound_wq we can hit this warning if we have max_modprobes UMH_WAIT_EXEC's resulting in __request_module at the same time, but probably this is good?

Yes, that's what I'm saying.

We already have a check against too many modprobes, it might be best to use it.

Confused... in this case I do not understand why do you dislike the idea to kill khelper_wq.

Help!

Oleg.

#5 lore raw
Quoted 2 lines · Oleg Nesterov

Confused... in this case I do not understand why do you dislike the idea to kill khelper_wq.

Yes, you are confused. I was agreeing with you:

Quoted 1 line · Oleg Nesterov

Can't we simply kill khelper_wq and use system_unbound_wq instead?

I'd prefer that, because then we'd hit the existing "too many modprobes" check.

Thanks,
Rusty.

I thought "I'd prefer that" was pretty clear??

Thanks,
Rusty.

#6 lore raw
Quoted 8 lines · Rusty Russell
Quoted 2 lines · Oleg Nesterov

Confused... in this case I do not understand why do you dislike the idea to kill khelper_wq.

Yes, you are confused. I was agreeing with you:

Quoted 1 line · Oleg Nesterov

Can't we simply kill khelper_wq and use system_unbound_wq instead?

BTW, why does it have to be unbound_wq? Is it expected consume large amount of CPU cycles?

Thanks.

Signature · 1 line
tejun
#13 lore raw
In reply to Rusty Russell
Quoted 6 lines · Rusty Russell
Quoted 2 lines · Oleg Nesterov

Confused... in this case I do not understand why do you dislike the idea to kill khelper_wq.

Yes, you are confused. I was agreeing with you:

Ah...

Quoted 1 line

I thought "I'd prefer that" was pretty clear??

It was! Looking back, I do not understand how did I manage to misread your email.

Thanks Rusty,

Oleg.

#7 lore raw
In reply to Tejun Heo
Quoted 2 lines · Tejun Heo

BTW, why does it have to be unbound_wq?

Perhaps we can use another system_wq, but afaics WQ_UNBOUND makes sense in this case. I mean, there is no reason to bind this work to any CPU. See also below.

Quoted 2 lines

Is it expected consume large
amount of CPU cycles?

Currently __call_usermodehelper() does kernel_thread(), this is almost all. But it can block waiting for kernel_execve().

Not sure this really makes sense, but if we kill khelper_wq perhaps we can simplify this code a bit. We can change __call_usermodehelper()

		if (wait == UMH_WAIT_PROC)
	-		pid = kernel_thread(wait_for_helper, sub_info,
	-				    CLONE_FS | CLONE_FILES | SIGCHLD);
	+		wait_for_helper(...);
		else

IOW, the worker thread itself can do the UMH_WAIT_PROC work. This makes this work really "long running", but then we can kill sub_info->complete and use flush_work().

Oleg.

#8 lore raw

Hello,

Quoted 3 lines · Oleg Nesterov

Perhaps we can use another system_wq, but afaics WQ_UNBOUND makes sense in this case. I mean, there is no reason to bind this work to any CPU. See also below.

I've been trying to nudge people away from using special wqs or flags unless really necessary. Other than non-reentrancy and strict ordering, all behaviors are mostly for optimization and using them incorrectly / spuriously usually doesn't cause any visible failure, making it very easy to get them wrong and if you have enough of wrong / unnecessary usages in tree, the whole thing gets really confusing and difficult to update in the future.

Quoted 5 lines
Quoted 2 lines

Is it expected consume large
amount of CPU cycles?

Currently __call_usermodehelper() does kernel_thread(), this is almost all. But it can block waiting for kernel_execve().

Blocking is completely fine on any workqueue. The only reason to require the use of unbound_wq is if work items would burn a lot of CPU cycles. In such cases, we want to let the scheduler have full jurisdiction instead of wq regulating concurrency.

Quoted 9 lines

Not sure this really makes sense, but if we kill khelper_wq perhaps we can simplify this code a bit. We can change __call_usermodehelper()

		if (wait == UMH_WAIT_PROC)
	-		pid = kernel_thread(wait_for_helper, sub_info,
	-				    CLONE_FS | CLONE_FILES | SIGCHLD);
	+		wait_for_helper(...);
		else
Quoted 3 lines

IOW, the worker thread itself can do the UMH_WAIT_PROC work. This makes this work really "long running", but then we can kill sub_info->complete and use flush_work().

Again, long-running in the sense that the work item spending a lot of time sleeping should be fine on system_wq or any other wq with default attributes. AFAICS, the things to consider here are...

* If work items are expected to consume large amount of CPU cycles (as in crypto work items), consider using system_unbound_wq / WQ_UNBOUND.

* If per-domain concurrency limit is necessary (ie. the number of concurrent work items doing this particular task should be limited rather than consuming global system_wq limit), a dedicated workqueue would be better.

Thanks.

Signature · 1 line
tejun
#9 lore raw

Hi Tejun,

Quoted 6 lines · Tejun Heo
Quoted 3 lines · Oleg Nesterov

Perhaps we can use another system_wq, but afaics WQ_UNBOUND makes sense in this case. I mean, there is no reason to bind this work to any CPU. See also below.

Quoted 7 lines

I've been trying to nudge people away from using special wqs or flags unless really necessary. Other than non-reentrancy and strict ordering, all behaviors are mostly for optimization and using them incorrectly / spuriously usually doesn't cause any visible failure, making it very easy to get them wrong and if you have enough of wrong / unnecessary usages in tree, the whole thing gets really confusing and difficult to update in the future.

You know, I am a bit suprized. To me, it is the !WQ_UNBOUND case is "special". IOW, I think we need some reason to bind the work to the specific CPU.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Is it expected consume large
amount of CPU cycles?

Currently __call_usermodehelper() does kernel_thread(), this is almost all. But it can block waiting for kernel_execve().

Blocking is completely fine on any workqueue.

I understand. But, the blocked worker "consumes" nr_active/worker.

Quoted 4 lines

The only reason to
require the use of unbound_wq is if work items would burn a lot of CPU cycles. In such cases, we want to let the scheduler have full jurisdiction instead of wq regulating concurrency.

I am starting to think I do not understand this code at all. OK, perhaps unbound_wq should be used for cpu-intensive works only.

But why do you think that we should use a !WQ_UNBOUND workque instead of khelper_wq? And why "a lot of CPU" is the only reason for WQ_UNBOUND?

Quoted 7 lines

* If work items are expected to consume large amount of CPU cycles (as in crypto work items), consider using system_unbound_wq / WQ_UNBOUND.

* If per-domain concurrency limit is necessary (ie. the number of concurrent work items doing this particular task should be limited rather than consuming global system_wq limit), a dedicated workqueue would be better.

So I don't understand whether you like the idea to kill khelper_wq and use some system_ wq or not (and fix the bug).

I do not really like the current patch. If nothing else, what if UMH_WAIT_EXEC request actually needs another UMH_WAIT_EXEC/PROC request to succeed?

Tetsuo, we spent a lot of time discussing other problems. What do you think about s/khelper/system/ instead of this patch?

Oleg.

#10 lore raw

Hello, Oleg.

Quoted 11 lines · Oleg Nesterov
Quoted 7 lines

I've been trying to nudge people away from using special wqs or flags unless really necessary. Other than non-reentrancy and strict ordering, all behaviors are mostly for optimization and using them incorrectly / spuriously usually doesn't cause any visible failure, making it very easy to get them wrong and if you have enough of wrong / unnecessary usages in tree, the whole thing gets really confusing and difficult to update in the future.

You know, I am a bit suprized. To me, it is the !WQ_UNBOUND case is "special". IOW, I think we need some reason to bind the work to the specific CPU.

It's part historical due to the way workqueue was originally implemented and part because work items are expected to consume little CPU cycles and access data which the issuer accessed, but the question really isn't about whether unbound would be marginally better or not. We may change the default behavior in the future as implementation evolves or hardware environment changes and having more things which claim special treatment without clear reason makes things needlessly complicated, so the question to ask is whether the default behavior doesn't fit enough that the extra flag is really required.

Quoted 3 lines
Quoted 1 line

Blocking is completely fine on any workqueue.

I understand. But, the blocked worker "consumes" nr_active/worker.

If it's expected to consume high number of nr_active and needs limiting, creating a dedicated wq as an isolation domain would be a good idea. It's not like wqs are expensive or complex to use after all.

Quoted 11 lines
Quoted 4 lines

The only reason to
require the use of unbound_wq is if work items would burn a lot of CPU cycles. In such cases, we want to let the scheduler have full jurisdiction instead of wq regulating concurrency.

I am starting to think I do not understand this code at all. OK, perhaps unbound_wq should be used for cpu-intensive works only.

But why do you think that we should use a !WQ_UNBOUND workque instead of khelper_wq? And why "a lot of CPU" is the only reason for WQ_UNBOUND?

As I wrote, it's not about "should use bound", it's about having enough justification for deviating from the default.

Thanks.

Signature · 1 line
tejun
#11 lore raw

Re: + kmod-avoid-deadlock-by-recursive-kmod-call.patch added to-mm tree

In reply to Oleg Nesterov
Quoted 1 line · Oleg Nesterov

What do you think about s/khelper/system/ instead of this patch?

I'm not catching up with this topic. But I'm fine with any solution provided that it can handle

some kernel function calls request_module()

request_module() triggers call_usermodehelper("/sbin/modprobe", UMH_WAIT_PROC)

/sbin/modprobe loads a kernel module

loading a kernel module triggers kobject_uevent_env()

kobject_uevent_env() calls call_usermodehelper("/sbin/hotplug", UMH_WAIT_EXEC)

do_execve("/sbin/hotplug") calls request_module("binfmt-0000")

request_module("binfmt-0000") triggers call_usermodehelper("/sbin/modprobe", UMH_WAIT_PROC)

/sbin/modprobe fails to load binfmt-0000 module

call trace on an UP machine.

kmod-avoid-deadlock-by-recursive-kmod-call.patch is for avoiding call_usermodehelper("/sbin/modprobe", UMH_WAIT_PROC) called from call_usermodehelper("/sbin/hotplug", UMH_WAIT_EXEC).

#12 lore raw

Re: + kmod-avoid-deadlock-by-recursive-kmod-call.patch added to-mm tree

Quoted 6 lines · Tetsuo Handa
Quoted 1 line · Oleg Nesterov

What do you think about s/khelper/system/ instead of this patch?

I'm not catching up with this topic. But I'm fine with any solution provided that it can handle

I think it should...

Tetsuo, all, sorry I am busy today. I'll try to return tomorrow to continue this (and off-list) discussion.

Oleg.