Skip to contentSkip to conversation

Fix SUNRPC wakeup/execute race condition

PATCH In review Christophe Saout 2 participants 6 messages Nov 5, 2006 – Nov 8, 2006 10 messages not in the archive

Subsystems (3)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 1157576316.3292.13.camel@dyn9047022153

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 1157731084.3292.25.camel@dyn9047022153

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw

PATCH 1/1 Fix SUNRPC wakeup/execute race condition

The sunrpc scheduler contains a race condition that can let an RPC task end up being neither running nor on any wait queue. The race takes place between rpc_make_runnable (called from rpc_wake_up_task) and __rpc_execute under the following condition:

First __rpc_execute calls tk_action which puts the task on some wait queue. The task is dequeued by another process before __rpc_execute continues its execution. While executing rpc_make_runnable exactly after setting the task running' bit and before clearing the queued' bit __rpc_execute picks up execution, clears `running' and subsequently both functions fall through, both under the false assumption somebody else took the job.

Swapping rpc_test_and_set_running with rpc_clear_queued in rpc_make_runnable fixes that hole. The reordering hopefully doesn't introduce some new race condition, in fact the only possible one is already correctly detected and handled in __rpc_execute.

Bug noticed on a 4-way x86_64 system under XEN with an NFSv4 server on the same physical machine, apparently one of the few ways to hit this race condition at all.

Cc Trond Myklebust <[email protected]>
Cc J. Bruce Fields <[email protected]>
Cc Olaf Kirch <[email protected]>
net/sunrpc/sched.c +1 -4
@@ -302,12 +302,9 @@ EXPORT_SYMBOL(__rpc_wait_for_completion_
  */
 static void rpc_make_runnable(struct rpc_task *task)
 {
-	int do_ret;
-
 	BUG_ON(task->tk_timeout_fn);
-	do_ret = rpc_test_and_set_running(task);
 	rpc_clear_queued(task);
-	if (do_ret)
+	if (rpc_test_and_set_running(task))
 		return;
 	if (RPC_IS_ASYNC(task)) {
 		int status;
#11 lore raw
Quoted 33 lines · Christophe Saout

The sunrpc scheduler contains a race condition that can let an RPC task end up being neither running nor on any wait queue. The race takes place between rpc_make_runnable (called from rpc_wake_up_task) and __rpc_execute under the following condition:

First __rpc_execute calls tk_action which puts the task on some wait queue. The task is dequeued by another process before __rpc_execute continues its execution. While executing rpc_make_runnable exactly after setting the task running' bit and before clearing the queued' bit __rpc_execute picks up execution, clears `running' and subsequently both functions fall through, both under the false assumption somebody else took the job.

Swapping rpc_test_and_set_running with rpc_clear_queued in rpc_make_runnable fixes that hole. The reordering hopefully doesn't introduce some new race condition, in fact the only possible one is already correctly detected and handled in __rpc_execute.

Bug noticed on a 4-way x86_64 system under XEN with an NFSv4 server on the same physical machine, apparently one of the few ways to hit this race condition at all.

Cc Trond Myklebust <[email protected]>
Cc J. Bruce Fields <[email protected]>
Cc Olaf Kirch <[email protected]>
net/sunrpc/sched.c +0 -0
@@ -302,12 +302,9 @@ EXPORT_SYMBOL(__rpc_wait_for_completion_
  */
 static void rpc_make_runnable(struct rpc_task *task)
 {

net/sunrpc/sched.c EXPORT_SYMBOL(__rpc_wait_for_completion_ · line 310

Quoted 10 lines
net/sunrpc/sched.c +1 -4
-	int do_ret;
-
 	BUG_ON(task->tk_timeout_fn);
-	do_ret = rpc_test_and_set_running(task);
 	rpc_clear_queued(task);
-	if (do_ret)
+	if (rpc_test_and_set_running(task))
 		return;
 	if (RPC_IS_ASYNC(task)) {
 		int status;

This fix looks wrong to me. If we've made it to 'rpc_make_runnable', then the rpc_task will have already been removed from the rpc_wait_queue.
The only question left is "who is responsible for waking up the synchronous task / setting up the asynchronous workqueue item?".

Cheers,
Trond

#12 lore raw

Am Sonntag, den 05.11.2006, 01:50 -0500 schrieb Trond Myklebust:

Quoted 12 lines
Quoted 12 lines
net/sunrpc/sched.c +0 -4
@@ -302,12 +302,9 @@ EXPORT_SYMBOL(__rpc_wait_for_completion_
  */
 static void rpc_make_runnable(struct rpc_task *task)
 {
-	int do_ret;
-
 	BUG_ON(task->tk_timeout_fn);
-	do_ret = rpc_test_and_set_running(task);
 	rpc_clear_queued(task);
-	if (do_ret)
Quoted 8 lines

net/sunrpc/sched.c EXPORT_SYMBOL(__rpc_wait_for_completion_ · line 310

Quoted 4 lines
net/sunrpc/sched.c +1 -0
+	if (rpc_test_and_set_running(task))
 		return;
 	if (RPC_IS_ASYNC(task)) {
 		int status;

This fix looks wrong to me. If we've made it to 'rpc_make_runnable', then the rpc_task will have already been removed from the rpc_wait_queue.

I just flipped the two lines, changed nothing else. Why exactly do you think that's wrong, I don't see anything particular that could be broken by chaning the ordering. Anyway, the fsstress has been running for 18 hours straight now without showing any signs of problems.

#13 lore raw
Quoted 23 lines · Christophe Saout

Am Sonntag, den 05.11.2006, 01:50 -0500 schrieb Trond Myklebust:

Quoted 12 lines
Quoted 12 lines
net/sunrpc/sched.c +0 -4
@@ -302,12 +302,9 @@ EXPORT_SYMBOL(__rpc_wait_for_completion_
  */
 static void rpc_make_runnable(struct rpc_task *task)
 {
-	int do_ret;
-
 	BUG_ON(task->tk_timeout_fn);
-	do_ret = rpc_test_and_set_running(task);
 	rpc_clear_queued(task);
-	if (do_ret)
Quoted 8 lines

net/sunrpc/sched.c EXPORT_SYMBOL(__rpc_wait_for_completion_ · line 310

Quoted 4 lines
net/sunrpc/sched.c +1 -0
+	if (rpc_test_and_set_running(task))
 		return;
 	if (RPC_IS_ASYNC(task)) {
 		int status;

This fix looks wrong to me. If we've made it to 'rpc_make_runnable', then the rpc_task will have already been removed from the rpc_wait_queue.

Quoted 4 lines

I just flipped the two lines, changed nothing else. Why exactly do you think that's wrong, I don't see anything particular that could be broken by chaning the ordering. Anyway, the fsstress has been running for 18 hours straight now without showing any signs of problems.

OK. I finally see the bug that you've spotted. The problem occurs when __rpc_execute clears RPC_TASK_RUNNING after rpc_make_runnable has called rpc_test_and_set_running, but before it has called rpc_clear_queued.

However if you just swap the two lines, you run into a new race:
__rpc_execute() may just put the rpc_task back to sleep before your call to rpc_test_and_set_running() finishes executing.
We therefore need an extra test for RPC_IS_QUEUED() in rpc_make_runnable().

Cheers,
Trond

#14 lore raw
Quoted 24 lines · Trond Myklebust
Quoted 23 lines · Christophe Saout

Am Sonntag, den 05.11.2006, 01:50 -0500 schrieb Trond Myklebust:

Quoted 20 lines
Quoted 6 lines
net/sunrpc/sched.c +0 -0
@@ -302,12 +302,9 @@ EXPORT_SYMBOL(__rpc_wait_for_completion_
  */
 static void rpc_make_runnable(struct rpc_task *task)
 {

net/sunrpc/sched.c EXPORT_SYMBOL(__rpc_wait_for_completion_ · line 310

Quoted 10 lines
net/sunrpc/sched.c +1 -4
-	int do_ret;
-
 	BUG_ON(task->tk_timeout_fn);
-	do_ret = rpc_test_and_set_running(task);
 	rpc_clear_queued(task);
-	if (do_ret)
+	if (rpc_test_and_set_running(task))
 		return;
 	if (RPC_IS_ASYNC(task)) {
 		int status;

This fix looks wrong to me. If we've made it to 'rpc_make_runnable', then the rpc_task will have already been removed from the rpc_wait_queue.

Quoted 8 lines
Quoted 4 lines

I just flipped the two lines, changed nothing else. Why exactly do you think that's wrong, I don't see anything particular that could be broken by chaning the ordering. Anyway, the fsstress has been running for 18 hours straight now without showing any signs of problems.

OK. I finally see the bug that you've spotted. The problem occurs when __rpc_execute clears RPC_TASK_RUNNING after rpc_make_runnable has called rpc_test_and_set_running, but before it has called rpc_clear_queued.

Yes, exactly.

Quoted 5 lines

However if you just swap the two lines, you run into a new race:
__rpc_execute() may just put the rpc_task back to sleep before your call to rpc_test_and_set_running() finishes executing.
We therefore need an extra test for RPC_IS_QUEUED() in rpc_make_runnable().

Damn, you're right. I missed that one. What about that:

----
The sunrpc scheduler contains a race condition that can let an RPC task end up being neither running nor on any wait queue. The race takes place between rpc_make_runnable (called from rpc_wake_up_task) and __rpc_execute under the following condition:

First __rpc_execute calls tk_action which puts the task on some wait queue. The task is dequeued by another process before __rpc_execute continues its execution. While executing rpc_make_runnable exactly after setting the task running' bit and before clearing the queued' bit __rpc_execute picks up execution, clears `running' and subsequently both functions fall through, both under the false assumption somebody else took the job.

Swapping rpc_test_and_set_running with rpc_clear_queued in rpc_make_runnable fixes that hole. This introduces another possible race condition that can be handled by checking for queued' after setting the running' bit.

Bug noticed on a 4-way x86_64 system under XEN with an NFSv4 server on the same physical machine, apparently one of the few ways to hit this race condition at all.

Cc Trond Myklebust <[email protected]>
Cc J. Bruce Fields <[email protected]>
net/sunrpc/sched.c +6 -4
@@ -302,12 +302,15 @@ EXPORT_SYMBOL(__rpc_wait_for_completion_
  */
 static void rpc_make_runnable(struct rpc_task *task)
 {
-	int do_ret;
-
 	BUG_ON(task->tk_timeout_fn);
-	do_ret = rpc_test_and_set_running(task);
 	rpc_clear_queued(task);
-	if (do_ret)
+	if (rpc_test_and_set_running(task))
 		return;
+	/* We might have raced */
+	if (RPC_IS_QUEUED(task)) {
+		rpc_clear_running(task);
+		return;
+	}
 	if (RPC_IS_ASYNC(task)) {
 		int status;

- int do_ret;
-
BUG_ON(task->tk_timeout_fn);
- do_ret = rpc_test_and_set_running(task);
rpc_clear_queued(task);
- if (do_ret)
+ if (rpc_test_and_set_running(task))
return;
+ /* We might have raced with __rpc_execute */
+ if (RPC_IS_QUEUED(task)) {
+ rpc_clear_running(task);
+ return;
+ }
if (RPC_IS_ASYNC(task)) {
int status;

#15 lore raw
Quoted 82 lines · Christophe Saout
Quoted 24 lines · Trond Myklebust
Quoted 23 lines · Christophe Saout

Am Sonntag, den 05.11.2006, 01:50 -0500 schrieb Trond Myklebust:

Quoted 20 lines
Quoted 6 lines
net/sunrpc/sched.c +0 -0
@@ -302,12 +302,9 @@ EXPORT_SYMBOL(__rpc_wait_for_completion_
  */
 static void rpc_make_runnable(struct rpc_task *task)
 {

net/sunrpc/sched.c EXPORT_SYMBOL(__rpc_wait_for_completion_ · line 310

Quoted 10 lines
net/sunrpc/sched.c +1 -4
-	int do_ret;
-
 	BUG_ON(task->tk_timeout_fn);
-	do_ret = rpc_test_and_set_running(task);
 	rpc_clear_queued(task);
-	if (do_ret)
+	if (rpc_test_and_set_running(task))
 		return;
 	if (RPC_IS_ASYNC(task)) {
 		int status;

This fix looks wrong to me. If we've made it to 'rpc_make_runnable', then the rpc_task will have already been removed from the rpc_wait_queue.

Quoted 8 lines
Quoted 4 lines

I just flipped the two lines, changed nothing else. Why exactly do you think that's wrong, I don't see anything particular that could be broken by chaning the ordering. Anyway, the fsstress has been running for 18 hours straight now without showing any signs of problems.

OK. I finally see the bug that you've spotted. The problem occurs when __rpc_execute clears RPC_TASK_RUNNING after rpc_make_runnable has called rpc_test_and_set_running, but before it has called rpc_clear_queued.

Yes, exactly.

Quoted 5 lines

However if you just swap the two lines, you run into a new race:
__rpc_execute() may just put the rpc_task back to sleep before your call to rpc_test_and_set_running() finishes executing.
We therefore need an extra test for RPC_IS_QUEUED() in rpc_make_runnable().

Damn, you're right. I missed that one. What about that:

----
The sunrpc scheduler contains a race condition that can let an RPC task end up being neither running nor on any wait queue. The race takes place between rpc_make_runnable (called from rpc_wake_up_task) and __rpc_execute under the following condition:

First __rpc_execute calls tk_action which puts the task on some wait queue. The task is dequeued by another process before __rpc_execute continues its execution. While executing rpc_make_runnable exactly after setting the task running' bit and before clearing the queued' bit __rpc_execute picks up execution, clears `running' and subsequently both functions fall through, both under the false assumption somebody else took the job.

Swapping rpc_test_and_set_running with rpc_clear_queued in rpc_make_runnable fixes that hole. This introduces another possible race condition that can be handled by checking for queued' after setting the running' bit.

Bug noticed on a 4-way x86_64 system under XEN with an NFSv4 server on the same physical machine, apparently one of the few ways to hit this race condition at all.

Cc Trond Myklebust <[email protected]>
Cc J. Bruce Fields <[email protected]>
net/sunrpc/sched.c +0 -3
@@ -302,12 +302,15 @@ EXPORT_SYMBOL(__rpc_wait_for_completion_
  */
 static void rpc_make_runnable(struct rpc_task *task)
 {
-	int do_ret;
-
 	BUG_ON(task->tk_timeout_fn);
-	do_ret = rpc_test_and_set_running(task);
 	rpc_clear_queued(task);

net/sunrpc/sched.c EXPORT_SYMBOL(__rpc_wait_for_completion_ · line 315

Quoted 10 lines
net/sunrpc/sched.c +6 -1
-	if (do_ret)
+	if (rpc_test_and_set_running(task))
 		return;
+	/* We might have raced */
+	if (RPC_IS_QUEUED(task)) {
+		rpc_clear_running(task);
+		return;
+	}
 	if (RPC_IS_ASYNC(task)) {
 		int status;

This looks OK.

Quoted 5 lines
net/sunrpc/sched.c +0 -3
-	int do_ret;
-
 	BUG_ON(task->tk_timeout_fn);
-	do_ret = rpc_test_and_set_running(task);
 	rpc_clear_queued(task);

net/sunrpc/sched.c EXPORT_SYMBOL(__rpc_wait_for_completion_ · line 315

Quoted 10 lines
net/sunrpc/sched.c +6 -1
-	if (do_ret)
+	if (rpc_test_and_set_running(task))
 		return;
+	/* We might have raced with __rpc_execute */
+	if (RPC_IS_QUEUED(task)) {
+		rpc_clear_running(task);
+		return;
+	}
 	if (RPC_IS_ASYNC(task)) {
 		int status;

A cut'n paste error?

Cheers,
Trond