Skip to contentSkip to conversation

calling flush_scheduled_work()

DISCUSSION Tim Hockin 4 participants 10 messages Mar 12, 2004 – Mar 13, 2004 Workqueue

lore
Share & tools
lore

Tim Hockin

author
#0 lore raw

We've recently bumped into an issue, and I'm not sure which is the real bug.

In short we have a case where mntput() is called from the kevetd workqueue. When that mntput() hit an NFS mount, we got a deadlock. It turns out that deep in the RPC code, someone calls flush_scheduled_work(). Deadlock.

So what is the real bug?

Is it verboten to call mntput() from keventd? What other things might lead to a flush_scheduled_work() and must therefore be avoided?

Should callers of flush_scheduled_work() be changed to use private workqueues? There are 31 calls that I got from grep. 25 are in drivers/, 1 in ncpfs, 3 in nfs4, 2 in sunrpc. The drivers/ are *probably* ok. Should those other 6 be changed?

Either way, it seems like there should maybe be a check and a badness warning if flush_workqueue is called from that workqueue.

Which avenue should we follow? Our own problem can be fixed differently, but I didn't want to just ignore this unstated assumption that it is safe to call flush_scheduled_work() anywhere.

Tim

Signature · 4 lines
Tim Hockin
Sun Microsystems, Linux Software Engineering
[email protected]
All opinions are my own, not Sun's
#1 lore raw

På fr , 12/03/2004 klokka 15:58, skreiv Tim Hockin:

Quoted 8 lines

We've recently bumped into an issue, and I'm not sure which is the real bug.

In short we have a case where mntput() is called from the kevetd workqueue. When that mntput() hit an NFS mount, we got a deadlock. It turns out that deep in the RPC code, someone calls flush_scheduled_work(). Deadlock.

So what is the real bug?

Quoted 7 lines

Is it verboten to call mntput() from keventd? What other things might lead to a flush_scheduled_work() and must therefore be avoided?

Should callers of flush_scheduled_work() be changed to use private workqueues? There are 31 calls that I got from grep. 25 are in drivers/, 1 in ncpfs, 3 in nfs4, 2 in sunrpc. The drivers/ are *probably* ok. Should those other 6 be changed?

It would be dead easy to change RPC+NFS to use their own, workqueue. In fact I've already considered that several times in the places where the NFSv4 state stuff gets hairy. ... but it might be nice not to have to set up all these (mostly) idle threads for exclusive use by NFS just in order to catch a corner case.

Could we therefore perhaps consider setting up one or more global workqueues that would be alternatives to keventd?

Cheers,
Trond

#2 lore raw
Quoted 8 lines · Trond Myklebust

It would be dead easy to change RPC+NFS to use their own, workqueue. In fact I've already considered that several times in the places where the NFSv4 state stuff gets hairy. ... but it might be nice not to have to set up all these (mostly) idle threads for exclusive use by NFS just in order to catch a corner case.

Could we therefore perhaps consider setting up one or more global workqueues that would be alternatives to keventd?

That just divides up the problem, but doesn't solve it. The simplest would be to print a badness and somehow get out of the flush. Then anyone who is triggering the corner case (us, in this case) can just solve it ourselves. It's not pretty.

In short, it's dubious that ANYONE call flush_scheduled_work() on a workqueue that they don't own.

Tim

#3 lore raw

På fr , 12/03/2004 klokka 17:38, skreiv Tim Hockin:

Quoted 4 lines

That just divides up the problem, but doesn't solve it. The simplest would be to print a badness and somehow get out of the flush. Then anyone who is triggering the corner case (us, in this case) can just solve it ourselves. It's not pretty.

Ewww.... That's saying "I'm just going to ignore the problem and pretend I can continue". At least the hang will tell you that there *is* a problem and points to where it is happening.

If I understood your description of where it is happening, then in this case, continuing is likely to lead to an Oops later when all those open RPC up/downcall pipes find that their parent directory has disappeared from underneath them.

Quoted 2 lines

In short, it's dubious that ANYONE call flush_scheduled_work() on a workqueue that they don't own.

Huh? You're saying that just because work has been scheduled on some third party thread, you should not be allowed to wait on completion of that work? That is a clearly unreasonable statement... Imagine doing even memory allocation in such an environment...

I agree that we need to identify and solve the deadlocking problems, but let's not start constricting developers with completely arbitrary rules.

The *real* problem here is that mput() is being run from keventd, and is triggering code that was assumed to be running from an ordinary process context. Let's concentrate on fixing that...

Cheers,
Trond

#5 lore raw
In reply to Tim Hockin
Quoted 6 lines · Tim Hockin

We've recently bumped into an issue, and I'm not sure which is the real bug.

In short we have a case where mntput() is called from the kevetd workqueue. When that mntput() hit an NFS mount, we got a deadlock. It turns out that deep in the RPC code, someone calls flush_scheduled_work(). Deadlock.

Seems simple enough to fix the workqueue code to handle this situation.

Wanna test this for me?

---

 25-akpm/kernel/workqueue.c |    8 ++++++++
 1 files changed, 8 insertions(+)
kernel/workqueue.c renamed from kernel/workqueue.c~flush_scheduled_work-deadlock-fix +8 -0
diff -puN kernel/workqueue.c~flush_scheduled_work-deadlock-fix kernel/workqueue.c
@@ -229,6 +229,14 @@ void fastcall flush_workqueue(struct wor
 			continue;
 		cwq = wq->cpu_wq + cpu;
 
+		if (cwq->thread == current) {
+			/*
+			 * Probably keventd trying to flush its own queue.
+			 * So just run it by hand rather than deadlocking
+			 */
+			run_workqueue(cwq);
+			continue;
+		}
 		spin_lock_irq(&cwq->lock);
 		sequence_needed = cwq->insert_sequence;
 

_

author
#4 lore raw
In reply to Trond Myklebust
Quoted 3 lines · Trond Myklebust

Ewww.... That's saying "I'm just going to ignore the problem and pretend I can continue". At least the hang will tell you that there *is* a problem and points to where it is happening.

Well, it needs to be non-silent. Whether that is a BUG() or a badness or a panic..

Quoted 7 lines
Quoted 2 lines

In short, it's dubious that ANYONE call flush_scheduled_work() on a workqueue that they don't own.

Huh? You're saying that just because work has been scheduled on some third party thread, you should not be allowed to wait on completion of that work? That is a clearly unreasonable statement... Imagine doing even memory allocation in such an environment...

Waiting for completion of your work is one thing. But you can't know what else has been scheduled. In this case RPC doesn't know that our work is calling it. By waiting for ALL work, it is assuming (silently) that it will never be called from a keventd.

Either that assumption is true, in which case there needs to be a BUG and a way out, or that assumption is false. I'd like to believe that the assumption is false.

Quoted 3 lines

The *real* problem here is that mput() is being run from keventd, and is triggering code that was assumed to be running from an ordinary process context. Let's concentrate on fixing that...

Well, we did fix it in a different way. I just wanted to bring this up as something that warrants at least a BUG(). But a BUG() and deadlock is almost a panic().

So if mntput() just CAN NOT be called from keventd, that is a fine answer, but something should be written about that. And all the other things that can not be called from keventd.

Signature · 4 lines
Tim Hockin
Sun Microsystems, Linux Software Engineering
[email protected]
All opinions are my own, not Sun's
author
#6 lore raw
In reply to Andrew Morton
Quoted 1 line · Andrew Morton

Seems simple enough to fix the workqueue code to handle this situation.

Indeed. I should have done that myself. Thanks.

Quoted 1 line

Wanna test this for me?

Works for me.

Signature · 4 lines
Tim Hockin
Sun Microsystems, Linux Software Engineering
[email protected]
All opinions are my own, not Sun's
#7 lore raw
In reply to Andrew Morton

På fr , 12/03/2004 klokka 18:27, skreiv Andrew Morton:

Quoted 3 lines

Seems simple enough to fix the workqueue code to handle this situation.

Wanna test this for me?

What if one of the workloads on the queue has another call to flush_scheduled_work()? (which again finds itself calling another instance, ...)
This fix doesn't really resolve the deadlock issue. It just reduces the possibilities of it having any consequences...

Could you please at least include a dump_stack() before the call to run_workqueue()) so that we can catch the bug in the act, and attempt to fix it.

Cheers,
Trond

#8 lore raw
Quoted 9 lines · Trond Myklebust

På fr , 12/03/2004 klokka 18:27, skreiv Andrew Morton:

Quoted 3 lines

Seems simple enough to fix the workqueue code to handle this situation.

Wanna test this for me?

What if one of the workloads on the queue has another call to flush_scheduled_work()? (which again finds itself calling another instance, ...)

I fixed that with the old solution-via-changelog:

Because keventd is a resource which is shared between unrelated parts of the kernel it is possible for one person's workqueue handler to accidentally call another person's flush_scheduled_work(). thockin managed it by calling mntput() from a workqueue handler. It deadlocks.

It's simple enough to fix: teach flush_scheduled_work() to go direct when it discovers that the calling thread is the one which should be running the work.

Note that this can cause recursion. The depth of that recursion is equal to the number of currently-queued works which themselves want to call flush_scheduled_work(). If this ever exceeds three I'll eat my hat.

Really, it's so unlikely it's hardly worth bothering about.

Quoted 6 lines

This fix doesn't really resolve the deadlock issue. It just reduces the possibilities of it having any consequences...

Could you please at least include a dump_stack() before the call to run_workqueue()) so that we can catch the bug in the act, and attempt to fix it.

How's this?

Add a debug check for workqueues nested more than three deep via the direct-run-workqueue() path.

---

 25-akpm/kernel/workqueue.c |   10 ++++++++++
 1 files changed, 10 insertions(+)
kernel/workqueue.c renamed from kernel/workqueue.c~flush_scheduled_work-recursion-detect +10 -0
diff -puN kernel/workqueue.c~flush_scheduled_work-recursion-detect kernel/workqueue.c
@@ -47,6 +47,7 @@ struct cpu_workqueue_struct {
 	struct workqueue_struct *wq;
 	task_t *thread;
 
+	int run_depth;		/* Detect run_workqueue() recursion depth */
 } ____cacheline_aligned;
 
 /*
@@ -140,6 +141,13 @@ static inline void run_workqueue(struct 
 	 * done.
 	 */
 	spin_lock_irqsave(&cwq->lock, flags);
+	cwq->run_depth++;
+	if (cwq->run_depth > 3) {
+		/* morton gets to eat his hat */
+		printk("%s: recusrion depth exceeded: %d\n",
+			__FUNCTION__, cwq->run_depth);
+		dump_stack();
+	}
 	while (!list_empty(&cwq->worklist)) {
 		struct work_struct *work = list_entry(cwq->worklist.next,
 						struct work_struct, entry);
@@ -157,6 +165,7 @@ static inline void run_workqueue(struct 
 		cwq->remove_sequence++;
 		wake_up(&cwq->work_done);
 	}
+	cwq->run_depth--;
 	spin_unlock_irqrestore(&cwq->lock, flags);
 }
 
@@ -286,6 +295,7 @@ struct workqueue_struct *create_workqueu
 	wq = kmalloc(sizeof(*wq), GFP_KERNEL);
 	if (!wq)
 		return NULL;
+	memset(wq, 0, sizeof(*wq));
 
 	for (cpu = 0; cpu < NR_CPUS; cpu++) {
 		if (!cpu_online(cpu))

_

#9 lore raw

På fr , 12/03/2004 klokka 21:12, skreiv Andrew Morton:

Quoted 5 lines
Quoted 3 lines

Could you please at least include a dump_stack() before the call to run_workqueue()) so that we can catch the bug in the act, and attempt to fix it.

How's this?

Excellent (aside from the spelling error)... Just for the record:
chocolate hats will not be acceptable as a forfeit. 8-)

Cheers,
Trond