Skip to contentSkip to conversation

dlm: Use cmwq for send and receive workqueues

PATCH Steven Whitehouse 2 participants 4 messages Nov 12, 2010 Distributed lock manager (DLM)

lore
Share & tools
lore
#0 lore raw

So far as I can tell, there is no reason to use a single-threaded send workqueue for dlm, since it may need to send to several sockets concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid any possible deadlocks, WQ_HIGHPRI since locking traffic is highly latency sensitive (and to avoid a priority inversion wrt GFS2's glock_workqueue) and WQ_FREEZABLE just in case someone needs to do that (even though with current cluster infrastructure, it doesn't make sense as the node will most likely land up ejected from the cluster) in the future.

fs/dlm/lowcomms.c +4 -2
index 37a34c2..0893b30 100644
@@ -1431,14 +1431,16 @@ static void work_stop(void)
 static int work_start(void)
 {
 	int error;
-	recv_workqueue = create_workqueue("dlm_recv");
+	recv_workqueue = alloc_workqueue("dlm_recv", WQ_MEM_RECLAIM |
+					 WQ_HIGHPRI | WQ_FREEZEABLE, 0);
 	error = IS_ERR(recv_workqueue);
 	if (error) {
 		log_print("can't start dlm_recv %d", error);
 		return error;
 	}
 
-	send_workqueue = create_singlethread_workqueue("dlm_send");
+	send_workqueue = alloc_workqueue("dlm_send", WQ_MEM_RECLAIM |
+					 WQ_HIGHPRI | WQ_FREEZEABLE, 0);
 	error = IS_ERR(send_workqueue);
 	if (error) {
 		log_print("can't start dlm_send %d", error);
maintainer
#1 lore raw
Quoted 10 lines · Steven Whitehouse

So far as I can tell, there is no reason to use a single-threaded send workqueue for dlm, since it may need to send to several sockets concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid any possible deadlocks, WQ_HIGHPRI since locking traffic is highly latency sensitive (and to avoid a priority inversion wrt GFS2's glock_workqueue) and WQ_FREEZABLE just in case someone needs to do that (even though with current cluster infrastructure, it doesn't make sense as the node will most likely land up ejected from the cluster) in the future.

Thanks, I'll want to do some testing with this, but my test machines do not seem to create more than one dlm_recv workqueue thread (prior to this patch). Have you tested in any cases where many threads end up being created? I've noticed while debugging some many-cpu machines a huge number of dlm_recv threads, which is just excessive. Does this patch address that?

Quoted 28 lines
fs/dlm/lowcomms.c +4 -2
index 37a34c2..0893b30 100644
@@ -1431,14 +1431,16 @@ static void work_stop(void)
 static int work_start(void)
 {
 	int error;
-	recv_workqueue = create_workqueue("dlm_recv");
+	recv_workqueue = alloc_workqueue("dlm_recv", WQ_MEM_RECLAIM |
+					 WQ_HIGHPRI | WQ_FREEZEABLE, 0);
 	error = IS_ERR(recv_workqueue);
 	if (error) {
 		log_print("can't start dlm_recv %d", error);
 		return error;
 	}
 
-	send_workqueue = create_singlethread_workqueue("dlm_send");
+	send_workqueue = alloc_workqueue("dlm_send", WQ_MEM_RECLAIM |
+					 WQ_HIGHPRI | WQ_FREEZEABLE, 0);
 	error = IS_ERR(send_workqueue);
 	if (error) {
 		log_print("can't start dlm_send %d", error);
#2 lore raw

Hi,

Quoted 12 lines · David Teigland
Quoted 10 lines · Steven Whitehouse

So far as I can tell, there is no reason to use a single-threaded send workqueue for dlm, since it may need to send to several sockets concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid any possible deadlocks, WQ_HIGHPRI since locking traffic is highly latency sensitive (and to avoid a priority inversion wrt GFS2's glock_workqueue) and WQ_FREEZABLE just in case someone needs to do that (even though with current cluster infrastructure, it doesn't make sense as the node will most likely land up ejected from the cluster) in the future.

Quoted 8 lines

Thanks, I'll want to do some testing with this, but my test machines do not seem to create more than one dlm_recv workqueue thread (prior to this patch). Have you tested in any cases where many threads end up being created? I've noticed while debugging some many-cpu machines a huge number of dlm_recv threads, which is just excessive. Does this patch address that?

Yes, one of the features of the cmwq is that you land up with only as many threads as required. When threads block, new ones are created to avoid stalling the workqueue. Workqueues marked with WQ_MEM_RECLAIM create a single rescuer thread, otherwise the threads are shared with all other users of cmwq,

Steve.

Quoted 28 lines
Quoted 28 lines
fs/dlm/lowcomms.c +4 -2
index 37a34c2..0893b30 100644
@@ -1431,14 +1431,16 @@ static void work_stop(void)
 static int work_start(void)
 {
 	int error;
-	recv_workqueue = create_workqueue("dlm_recv");
+	recv_workqueue = alloc_workqueue("dlm_recv", WQ_MEM_RECLAIM |
+					 WQ_HIGHPRI | WQ_FREEZEABLE, 0);
 	error = IS_ERR(recv_workqueue);
 	if (error) {
 		log_print("can't start dlm_recv %d", error);
 		return error;
 	}
 
-	send_workqueue = create_singlethread_workqueue("dlm_send");
+	send_workqueue = alloc_workqueue("dlm_send", WQ_MEM_RECLAIM |
+					 WQ_HIGHPRI | WQ_FREEZEABLE, 0);
 	error = IS_ERR(send_workqueue);
 	if (error) {
 		log_print("can't start dlm_send %d", error);
maintainer
#3 lore raw
Quoted 23 lines · Steven Whitehouse

Hi,

Quoted 12 lines · David Teigland
Quoted 10 lines · Steven Whitehouse

So far as I can tell, there is no reason to use a single-threaded send workqueue for dlm, since it may need to send to several sockets concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid any possible deadlocks, WQ_HIGHPRI since locking traffic is highly latency sensitive (and to avoid a priority inversion wrt GFS2's glock_workqueue) and WQ_FREEZABLE just in case someone needs to do that (even though with current cluster infrastructure, it doesn't make sense as the node will most likely land up ejected from the cluster) in the future.

Quoted 8 lines

Thanks, I'll want to do some testing with this, but my test machines do not seem to create more than one dlm_recv workqueue thread (prior to this patch). Have you tested in any cases where many threads end up being created? I've noticed while debugging some many-cpu machines a huge number of dlm_recv threads, which is just excessive. Does this patch address that?

Quoted 5 lines

Yes, one of the features of the cmwq is that you land up with only as many threads as required. When threads block, new ones are created to avoid stalling the workqueue. Workqueues marked with WQ_MEM_RECLAIM create a single rescuer thread, otherwise the threads are shared with all other users of cmwq,

Thanks, pushed to
git://git.kernel.org/pub/scm/linux/kernel/git/teigland/dlm.git#next