dlm: Use cmwq for send and receive workqueues
So far as I can tell, there is no reason to use a single-threaded send workqueue for dlm, since it may need to send to several sockets concurrently. Both workqueues are set to WQ_MEM_RECLAIM to avoid any possible deadlocks, WQ_HIGHPRI since locking traffic is highly latency sensitive (and to avoid a priority inversion wrt GFS2's glock_workqueue) and WQ_FREEZABLE just in case someone needs to do that (even though with current cluster infrastructure, it doesn't make sense as the node will most likely land up ejected from the cluster) in the future.
fs/dlm/lowcomms.c +4 -2
@@ -1431,14 +1431,16 @@ static void work_stop(void)static int work_start (void ) {int error; - recv_workqueue =create_workqueue ("dlm_recv" ); + recv_workqueue =alloc_workqueue ("dlm_recv" ,WQ_MEM_RECLAIM | +WQ_HIGHPRI |WQ_FREEZEABLE ,0 ); error =IS_ERR (recv_workqueue);if (error) {log_print ("can't start dlm_recv %d" , error);return error; } - send_workqueue =create_singlethread_workqueue ("dlm_send" ); + send_workqueue =alloc_workqueue ("dlm_send" ,WQ_MEM_RECLAIM | +WQ_HIGHPRI |WQ_FREEZEABLE ,0 ); error =IS_ERR (send_workqueue);if (error) {log_print ("can't start dlm_send %d" , error);
David Teigland
maintainerQuoted 10 lines · Steven Whitehouse
So far as I can tell, there is no reason to use a single-threaded send workqueue for dlm, since it may need to send to several sockets concurrently. Both workqueues are set to
WQ_MEM_RECLAIMto avoid any possible deadlocks,WQ_HIGHPRIsince locking traffic is highly latency sensitive (and to avoid a priority inversion wrt GFS2'sglock_workqueue) andWQ_FREEZABLEjust in case someone needs to do that (even though with current cluster infrastructure, it doesn't make sense as the node will most likely land up ejected from the cluster) in the future.
Thanks, I'll want to do some testing with this, but my test machines do not seem to create more than one dlm_recv workqueue thread (prior to this patch). Have you tested in any cases where many threads end up being created? I've noticed while debugging some many-cpu machines a huge number of dlm_recv threads, which is just excessive. Does this patch address that?
Quoted 28 lines
Signed-off-by Steven Whitehouse <[email protected]>Cc Tejun Heo <[email protected]>fs/dlm/lowcomms.c +4
-2@@ -1431,14 +1431,16 @@ static void work_stop(void)static int work_start (void ) {int error; - recv_workqueue =create_workqueue ("dlm_recv" ); + recv_workqueue =alloc_workqueue ("dlm_recv" ,WQ_MEM_RECLAIM | +WQ_HIGHPRI |WQ_FREEZEABLE ,0 ); error =IS_ERR (recv_workqueue);if (error) {log_print ("can't start dlm_recv %d" , error);return error; } - send_workqueue =create_singlethread_workqueue ("dlm_send" ); + send_workqueue =alloc_workqueue ("dlm_send" ,WQ_MEM_RECLAIM | +WQ_HIGHPRI |WQ_FREEZEABLE ,0 ); error =IS_ERR (send_workqueue);if (error) {log_print ("can't start dlm_send %d" , error);
Hi,
Quoted 12 lines · David Teigland
Quoted 10 lines · Steven Whitehouse
So far as I can tell, there is no reason to use a single-threaded send workqueue for dlm, since it may need to send to several sockets concurrently. Both workqueues are set to
WQ_MEM_RECLAIMto avoid any possible deadlocks,WQ_HIGHPRIsince locking traffic is highly latency sensitive (and to avoid a priority inversion wrt GFS2'sglock_workqueue) andWQ_FREEZABLEjust in case someone needs to do that (even though with current cluster infrastructure, it doesn't make sense as the node will most likely land up ejected from the cluster) in the future.
Quoted 8 lines
Thanks, I'll want to do some testing with this, but my test machines do not seem to create more than one
dlm_recvworkqueue thread (prior to this patch). Have you tested in any cases where many threads end up being created? I've noticed while debugging some many-cpu machines a huge number ofdlm_recvthreads, which is just excessive. Does this patch address that?
Yes, one of the features of the cmwq is that you land up with only as many threads as required. When threads block, new ones are created to avoid stalling the workqueue. Workqueues marked with WQ_MEM_RECLAIM create a single rescuer thread, otherwise the threads are shared with all other users of cmwq,
Steve.
Quoted 28 lines
Quoted 28 lines
Signed-off-by Steven Whitehouse <[email protected]>Cc Tejun Heo <[email protected]>fs/dlm/lowcomms.c +4
-2@@ -1431,14 +1431,16 @@ static void work_stop(void)static int work_start (void ) {int error; - recv_workqueue =create_workqueue ("dlm_recv" ); + recv_workqueue =alloc_workqueue ("dlm_recv" ,WQ_MEM_RECLAIM | +WQ_HIGHPRI |WQ_FREEZEABLE ,0 ); error =IS_ERR (recv_workqueue);if (error) {log_print ("can't start dlm_recv %d" , error);return error; } - send_workqueue =create_singlethread_workqueue ("dlm_send" ); + send_workqueue =alloc_workqueue ("dlm_send" ,WQ_MEM_RECLAIM | +WQ_HIGHPRI |WQ_FREEZEABLE ,0 ); error =IS_ERR (send_workqueue);if (error) {log_print ("can't start dlm_send %d" , error);
David Teigland
maintainerQuoted 23 lines · Steven Whitehouse
Hi,
Quoted 12 lines · David Teigland
Quoted 10 lines · Steven Whitehouse
So far as I can tell, there is no reason to use a single-threaded send workqueue for dlm, since it may need to send to several sockets concurrently. Both workqueues are set to
WQ_MEM_RECLAIMto avoid any possible deadlocks,WQ_HIGHPRIsince locking traffic is highly latency sensitive (and to avoid a priority inversion wrt GFS2'sglock_workqueue) andWQ_FREEZABLEjust in case someone needs to do that (even though with current cluster infrastructure, it doesn't make sense as the node will most likely land up ejected from the cluster) in the future.Quoted 8 lines
Thanks, I'll want to do some testing with this, but my test machines do not seem to create more than one
dlm_recvworkqueue thread (prior to this patch). Have you tested in any cases where many threads end up being created? I've noticed while debugging some many-cpu machines a huge number ofdlm_recvthreads, which is just excessive. Does this patch address that?
Quoted 5 lines
Yes, one of the features of the cmwq is that you land up with only as many threads as required. When threads block, new ones are created to avoid stalling the workqueue. Workqueues marked with
WQ_MEM_RECLAIMcreate a single rescuer thread, otherwise the threads are shared with all other users of cmwq,
Thanks, pushed togit://git.kernel.org/pub/scm/linux/kernel/git/teigland/dlm.git#next