Skip to contentSkip to conversation

nfsd: use a multithreaded workqueue for nfsd4_callbacks

PATCH In review Jeff Layton 2 participants 8 messages Oct 3, 2015 – Oct 12, 2015

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nfsd: use a multithreaded workqueue for nfsd4_callbacks

I don't see any need to order callbacks with respect to one another. Also, these are generally not involved in memory reclaim, so I don't see the need for a rescuer thread here either.

---
 fs/nfsd/nfs4callback.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfsd/nfs4callback.c +1 -1
index e7f50c4081d6..7dabbb436290 100644
@@ -1017,7 +1017,7 @@ static const struct rpc_call_ops nfsd4_cb_ops = {
 
 int nfsd4_create_callback_queue(void)
 {
-	callback_wq = create_singlethread_workqueue("nfsd4_callbacks");
+	callback_wq = alloc_workqueue("%s", WQ_UNBOUND, 0, "nfsd4_callbacks");
 	if (!callback_wq)
 		return -ENOMEM;
 	return 0;
Signature · 1 line
2.4.3
#1 lore raw
Quoted 1 line · Jeff Layton

I don't see any need to order callbacks with respect to one another.

I thought the code in nfsd4_process_cb_update really depended on this. The locking it has is against nfsd threads, it probably assumes it's only run from a cb thread and that it's the only one running at a time.

But I haven't reviewed it lately.

--b.

Quoted 23 lines

Also, these are generally not involved in memory reclaim, so I don't see the need for a rescuer thread here either.

---
 fs/nfsd/nfs4callback.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfsd/nfs4callback.c +1 -1
index e7f50c4081d6..7dabbb436290 100644
@@ -1017,7 +1017,7 @@ static const struct rpc_call_ops nfsd4_cb_ops = {
 
 int nfsd4_create_callback_queue(void)
 {
-	callback_wq = create_singlethread_workqueue("nfsd4_callbacks");
+	callback_wq = alloc_workqueue("%s", WQ_UNBOUND, 0, "nfsd4_callbacks");
 	if (!callback_wq)
 		return -ENOMEM;
 	return 0;
Signature · 1 line
2.4.3
#2 lore raw
Quoted 11 lines · On Fri, 9 Oct 2015 17:24:59 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 1 line · Jeff Layton

I don't see any need to order callbacks with respect to one another.

I thought the code in nfsd4_process_cb_update really depended on this. The locking it has is against nfsd threads, it probably assumes it's only run from a cb thread and that it's the only one running at a time.

But I haven't reviewed it lately.

--b.

Yikes -- ok. That's not at all obvious. That should prob be documented if so.

Yeah, ok...I guess you could end up with multiple threads racing to tear down the cb_client and xprt and create a new one, and it looks like the cl_cb_client and cl_cred pointers could get clobbered by new ones in that case.

Shouldn't be too hard to fix protecting those pointers with the cl_lock. That said, I prob won't be able to spend time on it right now. You can go ahead and drop that patch and I'll resend if/when I get around to fixing that.

Thanks for having a look...

Quoted 23 lines
Quoted 23 lines

Also, these are generally not involved in memory reclaim, so I don't see the need for a rescuer thread here either.

---
 fs/nfsd/nfs4callback.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfsd/nfs4callback.c +1 -1
index e7f50c4081d6..7dabbb436290 100644
@@ -1017,7 +1017,7 @@ static const struct rpc_call_ops nfsd4_cb_ops = {
 
 int nfsd4_create_callback_queue(void)
 {
-	callback_wq = create_singlethread_workqueue("nfsd4_callbacks");
+	callback_wq = alloc_workqueue("%s", WQ_UNBOUND, 0, "nfsd4_callbacks");
 	if (!callback_wq)
 		return -ENOMEM;
 	return 0;
Signature · 1 line
2.4.3
Signature · 1 line
Jeff Layton <[email protected]>
#3 lore raw
Quoted 10 lines · Jeff Layton
Quoted 7 lines · On Fri, 9 Oct 2015 17:24:59 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 1 line · Jeff Layton

I don't see any need to order callbacks with respect to one another.

I thought the code in nfsd4_process_cb_update really depended on this. The locking it has is against nfsd threads, it probably assumes it's only run from a cb thread and that it's the only one running at a time.

Quoted 7 lines
Quoted 4 lines

But I haven't reviewed it lately.

--b.

Yikes -- ok. That's not at all obvious. That should prob be documented if so.

Yes, my bad.

Quoted 9 lines

Yeah, ok...I guess you could end up with multiple threads racing to tear down the cb_client and xprt and create a new one, and it looks like the cl_cb_client and cl_cred pointers could get clobbered by new ones in that case.

Shouldn't be too hard to fix protecting those pointers with the cl_lock. That said, I prob won't be able to spend time on it right now. You can go ahead and drop that patch and I'll resend if/when I get around to fixing that.

What's the problem that this fixes exactly?

--b.

#4 lore raw
Quoted 26 lines · On Mon, 12 Oct 2015 14:11:17 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 10 lines · Jeff Layton
Quoted 7 lines · On Fri, 9 Oct 2015 17:24:59 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 1 line · Jeff Layton

I don't see any need to order callbacks with respect to one another.

I thought the code in nfsd4_process_cb_update really depended on this. The locking it has is against nfsd threads, it probably assumes it's only run from a cb thread and that it's the only one running at a time.

Quoted 7 lines
Quoted 4 lines

But I haven't reviewed it lately.

--b.

Yikes -- ok. That's not at all obvious. That should prob be documented if so.

Yes, my bad.

Quoted 5 lines

Yeah, ok...I guess you could end up with multiple threads racing to tear down the cb_client and xprt and create a new one, and it looks like the cl_cb_client and cl_cred pointers could get clobbered by new ones in that case.

Quoted 7 lines
Quoted 4 lines

Shouldn't be too hard to fix protecting those pointers with the cl_lock. That said, I prob won't be able to spend time on it right now. You can go ahead and drop that patch and I'll resend if/when I get around to fixing that.

What's the problem that this fixes exactly?

Unnecessary serialization of callbacks?

Not that that's necessarily a problem, but with the newer workqueue implementation there is more overhead in running a single threaded workqueue since it implies a rescuer thread and has to do extra work to ensure that the jobs are run in sequential order.

If that serialization is not actually needed, then it's better to move it to a multithreaded workqueue.

Signature · 1 line
Jeff Layton <[email protected]>
#5 lore raw
Quoted 39 lines · Jeff Layton
Quoted 26 lines · On Mon, 12 Oct 2015 14:11:17 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 10 lines · Jeff Layton
Quoted 7 lines · On Fri, 9 Oct 2015 17:24:59 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 1 line · Jeff Layton

I don't see any need to order callbacks with respect to one another.

I thought the code in nfsd4_process_cb_update really depended on this. The locking it has is against nfsd threads, it probably assumes it's only run from a cb thread and that it's the only one running at a time.

Quoted 7 lines
Quoted 4 lines

But I haven't reviewed it lately.

--b.

Yikes -- ok. That's not at all obvious. That should prob be documented if so.

Yes, my bad.

Quoted 5 lines

Yeah, ok...I guess you could end up with multiple threads racing to tear down the cb_client and xprt and create a new one, and it looks like the cl_cb_client and cl_cred pointers could get clobbered by new ones in that case.

Quoted 7 lines
Quoted 4 lines

Shouldn't be too hard to fix protecting those pointers with the cl_lock. That said, I prob won't be able to spend time on it right now. You can go ahead and drop that patch and I'll resend if/when I get around to fixing that.

What's the problem that this fixes exactly?

Unnecessary serialization of callbacks?

Quoted 7 lines

Not that that's necessarily a problem, but with the newer workqueue implementation there is more overhead in running a single threaded workqueue since it implies a rescuer thread and has to do extra work to ensure that the jobs are run in sequential order.

If that serialization is not actually needed, then it's better to move it to a multithreaded workqueue.

We'd still be serializing a lot of the same code, just using locks instead, wouldn't we?

--b.

#6 lore raw
Quoted 45 lines · On Mon, 12 Oct 2015 16:20:44 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 44 lines · Jeff Layton
Quoted 26 lines · On Mon, 12 Oct 2015 14:11:17 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 10 lines · Jeff Layton
Quoted 7 lines · On Fri, 9 Oct 2015 17:24:59 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 1 line · Jeff Layton

I don't see any need to order callbacks with respect to one another.

I thought the code in nfsd4_process_cb_update really depended on this. The locking it has is against nfsd threads, it probably assumes it's only run from a cb thread and that it's the only one running at a time.

Quoted 7 lines
Quoted 4 lines

But I haven't reviewed it lately.

--b.

Yikes -- ok. That's not at all obvious. That should prob be documented if so.

Yes, my bad.

Quoted 5 lines

Yeah, ok...I guess you could end up with multiple threads racing to tear down the cb_client and xprt and create a new one, and it looks like the cl_cb_client and cl_cred pointers could get clobbered by new ones in that case.

Quoted 7 lines
Quoted 4 lines

Shouldn't be too hard to fix protecting those pointers with the cl_lock. That said, I prob won't be able to spend time on it right now. You can go ahead and drop that patch and I'll resend if/when I get around to fixing that.

What's the problem that this fixes exactly?

Unnecessary serialization of callbacks?

Not that that's necessarily a problem, but with the newer workqueue implementation there is more overhead in running a single threaded workqueue since it implies a rescuer thread and has to do extra work to ensure that the jobs are run in sequential order.

Quoted 7 lines
Quoted 2 lines

If that serialization is not actually needed, then it's better to move it to a multithreaded workqueue.

We'd still be serializing a lot of the same code, just using locks instead, wouldn't we?

--b.

Any serialization would be per-clnt instead of global like it is today.

Also, we'd likely only serialize changes to some of the pointers instead of the entire operation.

Signature · 1 line
Jeff Layton <[email protected]>
#7 lore raw
Quoted 54 lines · Jeff Layton
Quoted 51 lines · On Mon, 12 Oct 2015 16:20:44 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 39 lines · Jeff Layton
Quoted 26 lines · On Mon, 12 Oct 2015 14:11:17 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 10 lines · Jeff Layton
Quoted 7 lines · On Fri, 9 Oct 2015 17:24:59 -0400 "J. Bruce Fields" <[email protected]>…
Quoted 1 line · Jeff Layton

I don't see any need to order callbacks with respect to one another.

I thought the code in nfsd4_process_cb_update really depended on this. The locking it has is against nfsd threads, it probably assumes it's only run from a cb thread and that it's the only one running at a time.

Quoted 7 lines
Quoted 4 lines

But I haven't reviewed it lately.

--b.

Yikes -- ok. That's not at all obvious. That should prob be documented if so.

Yes, my bad.

Quoted 5 lines

Yeah, ok...I guess you could end up with multiple threads racing to tear down the cb_client and xprt and create a new one, and it looks like the cl_cb_client and cl_cred pointers could get clobbered by new ones in that case.

Quoted 7 lines
Quoted 4 lines

Shouldn't be too hard to fix protecting those pointers with the cl_lock. That said, I prob won't be able to spend time on it right now. You can go ahead and drop that patch and I'll resend if/when I get around to fixing that.

What's the problem that this fixes exactly?

Unnecessary serialization of callbacks?

Quoted 7 lines

Not that that's necessarily a problem, but with the newer workqueue implementation there is more overhead in running a single threaded workqueue since it implies a rescuer thread and has to do extra work to ensure that the jobs are run in sequential order.

If that serialization is not actually needed, then it's better to move it to a multithreaded workqueue.

We'd still be serializing a lot of the same code, just using locks instead, wouldn't we?

Quoted 6 lines
Quoted 1 line

--b.

Any serialization would be per-clnt instead of global like it is today.

Also, we'd likely only serialize changes to some of the pointers instead of the entire operation.

OK, makes sense. Probably not a priority for now, though.

--b.