dlm: md: introduce DLM_LSFL_SOFTIRQ_SAFE
Subsystems (2)
Alexander Aring
maintainercover dlm: md: introduce DLM_LSFL_SOFTIRQ_SAFE
Hi,
the first three patches are resend with an updated commit message and additional comment e.g. to explain the rcu_read_lock() handling to avoid a lookup twice.
The main part are the rest of the patches introducing the DLM_LSFL_SOFTIRQ_SAFE flag to signal that ast/bast callbacks can be handled in softirq context. md-cluster is the first user here to take advantage about this flag as their callback handling is simple enough and looks okay to be called from softirq context. To be honest, I also just need a upstream user for the new flag. Other users, e.g. gfs2 can take also advantage of it.
I also gave it a try with a debug kernel and used md-cluster with 2 block devices and level mirrored. I used gfs2 on top of it and it seems to run without any problems.
This patch series is based on:
https://git.kernel.org/pub/scm/linux/kernel/git/teigland/linux-dlm.git/log/?h=next
in case of md-cluster people want to try it out.
Thanks.
- Alex
Alexander Aring (8):
dlm: using rcu to avoid rsb lookup again
dlm: remove struct field with the same meaning
dlm: use is_master() on checks if we are the master
dlm: use LSFL_FS to check if it's a kernel lockspace
dlm: introduce DLM_LSFL_SOFTIRQ_SAFE
dlm: implement LSFL_SOFTIRQ_SAFE
dlm: convert ls_cb_lock to rwlock
md-cluster: use DLM_LSFL_SOFTIRQ for dlm_new_lockspace()
drivers/md/md-cluster.c | 2 +- fs/dlm/ast.c | 185 ++++++++++++++++++--------- fs/dlm/ast.h | 11 +- fs/dlm/debug_fs.c | 33 ++--- fs/dlm/dlm_internal.h | 40 +++++- fs/dlm/lock.c | 268 +++++++++++++++++++++++---------------- fs/dlm/lock.h | 5 +- fs/dlm/lockspace.c | 23 ++-- fs/dlm/memory.c | 15 ++- fs/dlm/rcom.c | 4 +- fs/dlm/recover.c | 20 +-- fs/dlm/recoverd.c | 2 +- fs/dlm/user.c | 38 +++--- include/linux/dlm.h | 17 ++- include/uapi/linux/dlm.h | 2 + 15 files changed, 413 insertions(+), 252 deletions(-)
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH 1/8 dlm: using rcu to avoid rsb lookup again
This patch converts the rsb to be protected under rcu. When the rcu lock is held it cannot be freed. In combination with the new introduced flag RSB_HASHED we can check if the rsb is still part of the ls_rsbtbl hashtable, it cannot be part of another table. If its still part of the ls_rsbtbl we can avoid a second dlm_search_rsb_tree() call.
--- fs/dlm/dlm_internal.h | 3 ++ fs/dlm/lock.c | 117 ++++++++++++++++++++++++++++++++++-------- fs/dlm/memory.c | 15 +++++- 3 files changed, 112 insertions(+), 23 deletions(-)
fs/dlm/dlm_internal.h +3 -0
@@ -333,6 +333,7 @@ struct dlm_rsb {struct list_head res_recover_list;/* used for recovery */ struct list_head res_toss_q_list;int res_recover_locks_count; +struct rcu_head rcu;char *res_lvbptr;char res_name[DLM_RESNAME_MAXLEN +1 ];
@@ -366,6 +367,8 @@ enum rsb_flags {RSB_RECOVER_GRANT ,RSB_RECOVER_LVB_INVAL ,RSB_TOSS , +/* if rsb is part of ls_rsbtbl or not */ + RSB_HASHED, };static inline void rsb_set_flag (struct dlm_rsb *r,enum rsb_flags flag)
fs/dlm/lock.c +95 -22
194 lines not on this page · raw message
fs/dlm/memory.c +14 -1
@@ -101,13 +101,26 @@ struct dlm_rsb *dlm_allocate_rsb(struct dlm_ls *ls)return r; } -void dlm_free_rsb (struct dlm_rsb *r) +static void __dlm_free_rsb (struct rcu_head *rcu) { + struct dlm_rsb *r = container_of(rcu, struct dlm_rsb, rcu); +if (r->res_lvbptr)dlm_free_lvb (r->res_lvbptr);kmem_cache_free (rsb_cache, r); } +void dlm_free_rsb (struct dlm_rsb *r) +{ +/* wait until all current rcu read critical sections +* are done until calling __dlm_free_rsb. Currently +* this is used to avoid an use after free when checking +* of the rsb is still part of ls_rsbtbl by testing on +* RSB_HASHED flag. +*/ +call_rcu (&r->rcu, __dlm_free_rsb); +} +struct dlm_lkb *dlm_allocate_lkb (struct dlm_ls *ls) {struct dlm_lkb *lkb;
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH 2/8 dlm: remove struct field with the same meaning
In reply to Alexander AringThere is currently "res_nodeid" and "res_master_nodeid" fields in "struct dlm_rsb". At some point a developer does not know when to use which one or forget to update one and they are out of sync. This patch removes the "res_nodeid" and allows "res_master_nodeid" only. They have different representation values about their invalid values, the "res_master_nodeid" seems to be the modern way of represent the actual master nodeid and actually use the nodeid value when the own nodeid is the master (on res_nodeid the 0 represented this value). Also the modern nodeid representation fits into a "unsigned" range as this avoids to convert negative values over the network. The old value representation is still part of the DLM networking protocol that's why the conversion functions dlm_res_nodeid() and dlm_res_master_nodeid() are still present. On a new major DLM version bump protocol the nodeid representation should be updated to the modern value representation. These conversion functions also applies for existing UAPI and the user space still assumes the old "res_nodeid" value representation.
The same arguments applies to "lkb_nodeid" and "lkb_master_nodeid" wheras this is also only a copied value from another lkb related field "lkb_resource" and it's "res_master_nodeid" value. In this case it requires more code review because "lkb_resource" is not set sometimes.
This patch so far makes the code easier to read and understandable because we don't have several fields with the same meaning in some structs. In case of the previously "res_nodeid" value, sometimes an additional check on our_nodeid() is required to set the value to 0 that represents in this value representation that we are the master node.
--- fs/dlm/debug_fs.c | 33 +++++----- fs/dlm/dlm_internal.h | 33 ++++++++-- fs/dlm/lock.c | 144 ++++++++++++++++++------------------------ fs/dlm/lock.h | 5 +- fs/dlm/lockspace.c | 2 +- fs/dlm/rcom.c | 4 +- fs/dlm/recover.c | 20 ++---- fs/dlm/recoverd.c | 2 +- 8 files changed, 119 insertions(+), 124 deletions(-)
fs/dlm/debug_fs.c +17 -16
@@ -58,9 +58,10 @@ static void print_format1_lock(struct seq_file *s, struct dlm_lkb *lkb, lkb->lkb_status == DLM_LKSTS_WAITING)seq_printf (s," (%s)" ,print_lockmode (lkb->lkb_rqmode)); -if (lkb->lkb_nodeid) { -if (lkb->lkb_nodeid != res->res_nodeid) -seq_printf (s," Remote: %3d %08x" , lkb->lkb_nodeid, +if (lkb->lkb_master_nodeid !=dlm_our_nodeid ()) { +if (lkb->lkb_master_nodeid != res->res_master_nodeid) +seq_printf (s," Remote: %3d %08x" , +dlm_res_nodeid (lkb->lkb_master_nodeid), lkb->lkb_remid);else seq_printf (s," Master: %08x" , lkb->lkb_remid);
@@ -88,16 +89,15 @@ static void print_format1(struct dlm_rsb *res, struct seq_file *s)seq_printf (s,"%c" ,'.' ); } -if (res->res_nodeid >0 ) -seq_printf (s,"\"\nLocal Copy, Master is node %d\n" , - res->res_nodeid); -else if (res->res_nodeid ==0 ) -seq_puts (s,"\"\nMaster Copy\n" ); -else if (res->res_nodeid ==-1 ) + +if (res->res_master_nodeid ==0 )seq_printf (s,"\"\nLooking up master (lkid %x)\n" , res->res_first_lkid); +else if (res->res_master_nodeid ==dlm_our_nodeid ()) +seq_puts (s,"\"\nMaster Copy\n" );else -seq_printf (s,"\"\nInvalid master %d\n" , res->res_nodeid); +seq_printf (s,"\"\nLocal Copy, Master is node %d\n" , + res->res_master_nodeid);if (seq_has_overflowed (s)) goto out;
@@ -184,7 +184,7 @@ static void print_format2_lock(struct seq_file *s, struct dlm_lkb *lkb, seq_printf(s, "%x %d %x %u %llu %x %x %d %d %d %llu %u %d \"%s\"\n", lkb->lkb_id, - lkb->lkb_nodeid, + dlm_res_nodeid(lkb->lkb_master_nodeid), lkb->lkb_remid, lkb->lkb_ownpid, (unsigned long long)xid,
@@ -194,7 +194,7 @@ static void print_format2_lock(struct seq_file *s, struct dlm_lkb *lkb, lkb->lkb_grmode, lkb->lkb_rqmode, (unsigned long long )us, - r->res_nodeid, +dlm_res_nodeid (r->res_master_nodeid), r->res_length, r->res_name); }
@@ -238,7 +238,7 @@ static void print_format3_lock(struct seq_file *s, struct dlm_lkb *lkb, seq_printf(s, "lkb %x %d %x %u %llu %x %x %d %d %d %d %d %d %u %llu %llu\n", lkb->lkb_id, - lkb->lkb_nodeid, + dlm_res_nodeid(lkb->lkb_master_nodeid), lkb->lkb_remid, lkb->lkb_ownpid, (unsigned long long)xid,
@@ -265,7 +265,7 @@ static void print_format3(struct dlm_rsb *r, struct seq_file *s) seq_printf(s, "rsb %p %d %x %lx %d %d %u %d ", r, - r->res_nodeid, + dlm_res_nodeid(r->res_master_nodeid), r->res_first_lkid, r->res_flags, !list_empty(&r->res_root_list),
@@ -341,7 +341,7 @@ static void print_format4(struct dlm_rsb *r, struct seq_file *s) seq_printf(s, "rsb %p %d %d %d %d %lu %lx %d ", r, - r->res_nodeid, + dlm_res_nodeid(r->res_master_nodeid), r->res_master_nodeid, r->res_dir_nodeid, our_nodeid,
@@ -611,7 +611,8 @@ static ssize_t waiters_read(struct file *file, char __user *userbuf,list_for_each_entry (lkb, &ls->ls_waiters, lkb_wait_reply) { ret =snprintf (debug_buf + pos, len - pos,"%x %d %d %s\n" , lkb->lkb_id, lkb->lkb_wait_type, - lkb->lkb_nodeid, lkb->lkb_resource->res_name); +dlm_res_nodeid (lkb->lkb_master_nodeid), + lkb->lkb_resource->res_name);if (ret >= len - pos)break ; pos += ret;
fs/dlm/dlm_internal.h +28 -5
@@ -253,7 +253,7 @@ struct dlm_callback {struct dlm_lkb {struct dlm_rsb *lkb_resource;/* the rsb */ struct kref lkb_ref; -int lkb_nodeid;/* copied from rsb */ +unsigned int lkb_master_nodeid;/* copied from rsb */ int lkb_ownpid;/* pid of lock owner */ uint32_t lkb_id;/* our lock ID */ uint32_t lkb_remid;/* lock ID on remote partner */
@@ -303,18 +303,41 @@ struct dlm_lkb {* * res_nodeid is "odd": -1 is unset/invalid, zero means our_nodeid, * greater than zero when another nodeid. -* -* (TODO: remove res_nodeid and only use res_master_nodeid) */ +/* For backwards compatibility, see above. +* The protocol is still using res_nodeid. +*/ +static inline int dlm_res_nodeid (unsigned int res_master_nodeid) +{ +if (res_master_nodeid ==0 ) +return -1 ; +else if (res_master_nodeid ==dlm_our_nodeid ()) +return 0 ; +else +return res_master_nodeid; +} + +/* For backwards compatibility, see above. +* The protocol is still using res_nodeid. +*/ +static inline unsigned int dlm_res_master_nodeid (int res_nodeid) +{ +if (res_nodeid ==-1 ) +return 0 ; +else if (res_nodeid ==0 ) +return dlm_our_nodeid (); +else +return res_nodeid; +} +struct dlm_rsb {struct dlm_ls *res_ls;/* the lockspace */ struct kref res_ref;spinlock_t res_lock;unsigned long res_flags;int res_length;/* length of rsb name */ -int res_nodeid; -int res_master_nodeid; +unsigned int res_master_nodeid;int res_dir_nodeid;unsigned long res_id;/* for ls_recover_xa */ uint32_t res_lvbseq;
fs/dlm/lock.c +61 -83
455 lines not on this page · raw message
fs/dlm/lock.h +3 -2
@@ -61,13 +61,14 @@ int dlm_user_purge(struct dlm_ls *ls, struct dlm_user_proc *proc,int dlm_user_deadlock (struct dlm_ls *ls,uint32_t flags,uint32_t lkid);void dlm_clear_proc_locks (struct dlm_ls *ls,struct dlm_user_proc *proc);int dlm_debug_add_lkb (struct dlm_ls *ls,uint32_t lkb_id,char *name,int len, -int lkb_nodeid,unsigned int lkb_flags,int lkb_status); +int lkb_master_nodeid,unsigned int lkb_flags, +int lkb_status);int dlm_debug_add_lkb_to_waiters (struct dlm_ls *ls,uint32_t lkb_id,int mstype,int to_nodeid);static inline int is_master (struct dlm_rsb *r) { -return !r->res_nodeid; +return r->res_master_nodeid ==dlm_our_nodeid (); }static inline void lock_rsb (struct dlm_rsb *r)
fs/dlm/lockspace.c +1 -1
@@ -660,7 +660,7 @@ static int lockspace_busy(struct dlm_ls *ls, int force) } }else if (force ==1 ) {xa_for_each (&ls->ls_lkbxa, id, lkb) { -if (lkb->lkb_nodeid ==0 && +if (lkb->lkb_master_nodeid ==dlm_our_nodeid () && lkb->lkb_grmode !=DLM_LOCK_IV ) { rv =1 ;break ;
fs/dlm/rcom.c +2 -2
@@ -455,8 +455,8 @@ int dlm_send_rcom_lock(struct dlm_rsb *r, struct dlm_lkb *lkb, uint64_t seq)if (lkb->lkb_lvbptr) len += ls->ls_lvblen; - error =create_rcom (ls, r->res_nodeid,DLM_RCOM_LOCK , len, &rc, &mh, - seq); + error =create_rcom (ls, r->res_master_nodeid,DLM_RCOM_LOCK , len, &rc, + &mh, seq);if (error) goto out;
fs/dlm/recover.c +6 -14
@@ -408,7 +408,7 @@ static void set_lock_master(struct list_head *queue, int nodeid)list_for_each_entry (lkb, queue, lkb_statequeue) {if (!test_bit (DLM_IFL_MSTCPY_BIT , &lkb->lkb_iflags)) { - lkb->lkb_nodeid = nodeid; + lkb->lkb_master_nodeid = nodeid; lkb->lkb_remid =0 ; } }
@@ -416,9 +416,9 @@ static void set_lock_master(struct list_head *queue, int nodeid)static void set_master_lkbs (struct dlm_rsb *r) { -set_lock_master (&r->res_grantqueue, r->res_nodeid); -set_lock_master (&r->res_convertqueue, r->res_nodeid); -set_lock_master (&r->res_waitqueue, r->res_nodeid); +set_lock_master (&r->res_grantqueue, r->res_master_nodeid); +set_lock_master (&r->res_convertqueue, r->res_master_nodeid); +set_lock_master (&r->res_waitqueue, r->res_master_nodeid); }/*
@@ -455,7 +455,7 @@ static int recover_master(struct dlm_rsb *r, unsigned int *count, uint64_t seq)if (is_master (r))return 0 ; - is_removed =dlm_is_removed (ls, r->res_nodeid); + is_removed =dlm_is_removed (ls, r->res_master_nodeid);if (!is_removed && !rsb_flag (r,RSB_NEW_MASTER ))return 0 ;
@@ -464,10 +464,8 @@ static int recover_master(struct dlm_rsb *r, unsigned int *count, uint64_t seq) dir_nodeid =dlm_dir_nodeid (r);if (dir_nodeid == our_nodeid) { -if (is_removed) { +if (is_removed) r->res_master_nodeid = our_nodeid; - r->res_nodeid =0 ; - }/* set master of lkbs to ourself when is_removed, or to another new master which we set along with NEW_MASTER
@@ -501,14 +499,9 @@ static int recover_master(struct dlm_rsb *r, unsigned int *count, uint64_t seq)static int recover_master_static (struct dlm_rsb *r,unsigned int *count) {int dir_nodeid =dlm_dir_nodeid (r); -int new_master = dir_nodeid; - -if (dir_nodeid ==dlm_our_nodeid ()) - new_master =0 ;dlm_purge_mstcpy_locks (r); r->res_master_nodeid = dir_nodeid; - r->res_nodeid = new_master;set_new_master (r); (*count)++;return 0 ;
@@ -584,7 +577,6 @@ int dlm_recover_master_reply(struct dlm_ls *ls, const struct dlm_rcom *rc)lock_rsb (r); r->res_master_nodeid = ret_nodeid; - r->res_nodeid = new_master;set_new_master (r);unlock_rsb (r);recover_xa_del (r);
fs/dlm/recoverd.c +1 -1
@@ -34,7 +34,7 @@ static int dlm_create_masters_list(struct dlm_ls *ls)read_lock_bh (&ls->ls_rsbtbl_lock);list_for_each_entry (r, &ls->ls_keep, res_rsbs_list) { -if (r->res_nodeid) +if (r->res_master_nodeid !=dlm_our_nodeid ())continue ;list_add (&r->res_masters_list, &ls->ls_masters_list);
Signature · 1 line
2.43.0
Alexander Aring
maintainerRe: [PATCH dlm/next 2/8] dlm: remove struct field with the same meaning
Hi,
Quoted 23 lines · Alexander Aring
There is currently "
res_nodeid" and "res_master_nodeid" fields in "structdlm_rsb". At some point a developer does not know when to use which one or forget to update one and they are out of sync. This patch removes the "res_nodeid" and allows "res_master_nodeid" only. They have different representation values about their invalid values, the "res_master_nodeid" seems to be the modern way of represent the actual master nodeid and actually use the nodeid value when the own nodeid is the master (onres_nodeidthe 0 represented this value). Also the modern nodeid representation fits into a "unsigned" range as this avoids to convert negative values over the network. The old value representation is still part of the DLM networking protocol that's why the conversion functionsdlm_res_nodeid()anddlm_res_master_nodeid()are still present. On a new major DLM version bump protocol the nodeid representation should be updated to the modern value representation. These conversion functions also applies for existing UAPI and the user space still assumes the old "res_nodeid" value representation.The same arguments applies to "
lkb_nodeid" and "lkb_master_nodeid" wheras this is also only a copied value from another lkb related field "lkb_resource" and it's "res_master_nodeid" value. In this case it requires more code review because "lkb_resource" is not set sometimes.
Quoted 7 lines
This patch so far makes the code easier to read and understandable because we don't have several fields with the same meaning in some structs. In case of the previously "
res_nodeid" value, sometimes an additional check onour_nodeid()is required to set the value to 0 that represents in this value representation that we are the master node.Signed-off-by Alexander Aring <[email protected]>
__dlm_master_lookup() returns "-1" (the old value representation for invalid nodeid) and it should return 0 for invalid now. Also we forgot dlm_send_rcom_lookup()/receive_rcom_lookup() to update using the conversion functions.
- Alex
Alexander Aring
maintainerPATCH 3/8 dlm: use is_master() on checks if we are the master
In reply to Alexander AringThere are checks if we are the master or not done by "r->res_master_nodeid == dlm_our_nodeid()" or unequal to check if we are not the master. There is a helper function is_master() for doing this kind of check. This patch replaces several checks of those by using the helper instead of using the mentioned condition check.
--- fs/dlm/lock.c | 9 ++++----- fs/dlm/recoverd.c | 2 +- 2 files changed, 5 insertions(+), 6 deletions(-)
fs/dlm/lock.c +4 -5
@@ -242,7 +242,7 @@ static inline int is_granted(struct dlm_lkb *lkb)static inline int is_remote (struct dlm_rsb *r) { DLM_ASSERT(r->res_master_nodeid != 0, dlm_print_rsb(r);); -return r->res_master_nodeid !=dlm_our_nodeid (); +return !is_master (r); }static inline int is_process_copy (struct dlm_lkb *lkb)
@@ -4025,7 +4025,7 @@ static int receive_request(struct dlm_ls *ls, const struct dlm_message *ms)lock_rsb (r); -if (r->res_master_nodeid !=dlm_our_nodeid ()) { +if (!is_master (r)) { error =validate_master_nodeid (ls, r, from_nodeid);if (error) {unlock_rsb (r);
@@ -4447,8 +4447,7 @@ static int receive_request_reply(struct dlm_ls *ls, from_nodeid, result, r->res_master_nodeid, r->res_dir_nodeid, r->res_first_lkid, r->res_name); -if (r->res_dir_nodeid !=dlm_our_nodeid () && - r->res_master_nodeid !=dlm_our_nodeid ()) { +if (r->res_dir_nodeid !=dlm_our_nodeid () && !is_master (r)) {/* cause _request_lock->set_master->send_lookup */ r->res_master_nodeid =0 ; lkb->lkb_master_nodeid =0 ;
@@ -4462,7 +4461,7 @@ static int receive_request_reply(struct dlm_ls *ls, }else {_request_lock (r, lkb); -if (r->res_master_nodeid ==dlm_our_nodeid ()) +if (is_master (r))confirm_master (r,0 ); }break ;
fs/dlm/recoverd.c +1 -1
@@ -34,7 +34,7 @@ static int dlm_create_masters_list(struct dlm_ls *ls)read_lock_bh (&ls->ls_rsbtbl_lock);list_for_each_entry (r, &ls->ls_keep, res_rsbs_list) { -if (r->res_master_nodeid !=dlm_our_nodeid ()) +if (!is_master (r))continue ;list_add (&r->res_masters_list, &ls->ls_masters_list);
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH 4/8 dlm: use LSFL_FS to check if it's a kernel lockspace
In reply to Alexander AringThis patch sets and uses the internal flag LSFL_FS to check in workqueue handling if it's a kernel lockspace or not in the callback workqueue handling. Currently user space lockspaces don't require the callback workqueue. Upcoming changes will make it possible to remove the the callback workqueue context switch when a specific lockspace flag is passed. This patch prepares for such handling as some handling like setting of LSFL_CB_DELAY is still required for those kernel lockspaces but they don't have a callback workqueue.
--- fs/dlm/ast.c | 17 +++++++++++------ fs/dlm/dlm_internal.h | 1 + fs/dlm/lockspace.c | 13 +++++++------ 3 files changed, 19 insertions(+), 12 deletions(-)
fs/dlm/ast.c +11 -6
@@ -161,6 +161,9 @@ void dlm_add_cb(struct dlm_lkb *lkb, uint32_t flags, int mode, int status,int dlm_callback_start (struct dlm_ls *ls) { +if (!test_bit (LSFL_FS , &ls->ls_flags)) +return 0 ; + ls->ls_callback_wq =alloc_ordered_workqueue ("dlm_callback" ,WQ_HIGHPRI |WQ_MEM_RECLAIM );if (!ls->ls_callback_wq) {
@@ -178,13 +181,15 @@ void dlm_callback_stop(struct dlm_ls *ls)void dlm_callback_suspend (struct dlm_ls *ls) { -if (ls->ls_callback_wq) { -spin_lock_bh (&ls->ls_cb_lock); -set_bit (LSFL_CB_DELAY , &ls->ls_flags); -spin_unlock_bh (&ls->ls_cb_lock); +if (!test_bit (LSFL_FS , &ls->ls_flags)) +return ; +spin_lock_bh (&ls->ls_cb_lock); +set_bit (LSFL_CB_DELAY , &ls->ls_flags); +spin_unlock_bh (&ls->ls_cb_lock); + +if (ls->ls_callback_wq)flush_workqueue (ls->ls_callback_wq); - } }#define MAX_CB_QUEUE 25
@@ -195,7 +200,7 @@ void dlm_callback_resume(struct dlm_ls *ls)int count =0 , sum =0 ;bool empty; -if (!ls->ls_callback_wq) +if (!test_bit (LSFL_FS , &ls->ls_flags))return ; more:
fs/dlm/dlm_internal.h +1 -0
@@ -726,6 +726,7 @@ struct dlm_ls {#define LSFL_CB_DELAY 9#define LSFL_NODIR 10#define LSFL_RECV_MSG_BLOCKED 11 +#define LSFL_FS 12#define DLM_PROC_FLAGS_CLOSING 1#define DLM_PROC_FLAGS_COMPAT 2
fs/dlm/lockspace.c +7 -6
@@ -500,12 +500,13 @@ static int new_lockspace(const char *name, const char *cluster,list_add (&ls->ls_list, &lslist);spin_unlock_bh (&lslist_lock); -if (flags &DLM_LSFL_FS ) { - error =dlm_callback_start (ls); -if (error) { -log_error (ls,"can't start dlm_callback %d" , error); - goto out_delist; - } +if (flags &DLM_LSFL_FS ) +set_bit (LSFL_FS , &ls->ls_flags); + + error =dlm_callback_start (ls); +if (error) { +log_error (ls,"can't start dlm_callback %d" , error); + goto out_delist; }init_waitqueue_head (&ls->ls_recover_lock_wait);
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH 5/8 dlm: introduce DLM_LSFL_SOFTIRQ_SAFE
In reply to Alexander AringThis patch introduces the per lockspace DLM_LSFL_SOFTIRQ_SAFE flag to signal that the user is ready to handle ast/bast callbacks in softirq context. It is a reserved dlm kernel lockspace flag as seen from the user space yet. We not chose available 0x4 flag as this was recently flag that was dropped and should be not reused now. In future, if all users changed to use DLM_LSFL_SOFTIRQ_SAFE, we can hopefully drop this flag and dlm code functionality (dlm_callback workqueue) which belongs to it.
--- fs/dlm/lockspace.c | 3 +++ include/linux/dlm.h | 17 ++++++++++++++++- include/uapi/linux/dlm.h | 2 ++ 3 files changed, 21 insertions(+), 1 deletion(-)
fs/dlm/lockspace.c +3 -0
@@ -630,6 +630,9 @@ int dlm_new_user_lockspace(const char *name, const char *cluster, void *ops_arg, int *ops_result, dlm_lockspace_t **lockspace) { + if (flags & DLM_LSFL_SOFTIRQ) + return -EINVAL; + return __dlm_new_lockspace(name, cluster, flags, lvblen, ops,ops_arg ,ops_result ,lockspace ); }
include/linux/dlm.h +16 -1
@@ -35,6 +35,9 @@ struct dlm_lockspace_ops { int num_slots, int our_slot, uint32_t generation); }; +/* only relevant for kernel lockspaces, will be removed in future */ +#define DLM_LSFL_SOFTIRQ __DLM_LSFL_RESERVED0 +/* * dlm_new_lockspace *
@@ -55,6 +58,11 @@ struct dlm_lockspace_ops {* used to select the directory node. Must be the same on all nodes. * DLM_LSFL_NEWEXCL * dlm_new_lockspace() should return -EEXIST if the lockspace exists. +* DLM_LSFL_SOFTIRQ +* dlm request callbacks (ast, bast) are softirq safe. Flag should be +* preferred by users. Will be default in some future. If set the +* strongest context for ast, bast callback is softirq as it avoids +* an additional context switch. * * lvblen: length of lvb in bytes. Must be multiple of 8. * dlm_new_lockspace() returns an error if this does not match
@@ -121,7 +129,14 @@ int dlm_release_lockspace(dlm_lockspace_t *lockspace, int force);* call. * * AST routines should not block (at least not for long), but may make -* any locking calls they please. +* any locking calls they please. If DLM_LSFL_SOFTIRQ for kernel +* users of dlm_new_lockspace() is passed the ast and bast callbacks +* can be processed in softirq context. Also some of the callback +* contexts are in the same context as the DLM lock request API, users +* must not hold locks while calling dlm lock request API and trying +* to acquire this lock in the callback again, this will end in a +* lock recursion. For newer implementation the DLM_LSFL_SOFTIRQ +* should be used. */ int dlm_lock(dlm_lockspace_t *lockspace,
include/uapi/linux/dlm.h +2 -0
@@ -71,6 +71,8 @@ struct dlm_lksb {/* DLM_LSFL_TIMEWARN is deprecated and reserved. DO NOT USE! */ #define DLM_LSFL_TIMEWARN 0x00000002#define DLM_LSFL_NEWEXCL 0x00000008 +/* currently reserved due in-kernel use */ +#define __DLM_LSFL_RESERVED0 0x00000010#endif /* _UAPI__DLM_DOT_H__ */
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH 6/8 dlm: implement LSFL_SOFTIRQ_SAFE
In reply to Alexander AringThis patch implements to allow to directly call ast and bast callbacks in the context as soon they arrive. Currently every ast and bast callback of a kernel lockspace callback is queued through a workqueue to provide the dlm user a different context. However some users can't currently handle the strongest context of ast/bast which is a softirq context but new users should directly implement their dlm application to run in softirq context. Another mentioned requirement that is more unlikely that a current DLM runs into is a lock recursion if a lock if the same lock is held in the callback and when the DLM API is called.
--- fs/dlm/ast.c | 157 +++++++++++++++++++++++++++--------------- fs/dlm/ast.h | 11 ++- fs/dlm/dlm_internal.h | 1 + fs/dlm/lockspace.c | 3 + fs/dlm/user.c | 38 +++++----- 5 files changed, 126 insertions(+), 84 deletions(-)
fs/dlm/ast.c +101 -56
@@ -18,35 +18,52 @@#include "user.h" #include "ast.h" -static void dlm_callback_work (struct work_struct *work) +static void dlm_run_callback (uint32_t ls_id,uint32_t lkb_id,int8_t mode, +uint32_t flags,uint8_t sb_flags,int sb_status, +struct dlm_lksb *lksb, +void (*astfn)(void *astparam), +void (*bastfn)(void *astparam,int mode), +void *astparam,const char *res_name, +size_t res_length) { - struct dlm_callback *cb = container_of(work, struct dlm_callback, work); - -if (cb->flags &DLM_CB_BAST ) { -trace_dlm_bast (cb->ls_id, cb->lkb_id, cb->mode, cb->res_name, - cb->res_length); - cb->bastfn (cb->astparam, cb->mode); - }else if (cb->flags &DLM_CB_CAST ) { -trace_dlm_ast (cb->ls_id, cb->lkb_id, cb->sb_status, - cb->sb_flags, cb->res_name, cb->res_length); - cb->lkb_lksb->sb_status = cb->sb_status; - cb->lkb_lksb->sb_flags = cb->sb_flags; - cb->astfn (cb->astparam); +if (flags &DLM_CB_BAST ) { +trace_dlm_bast (ls_id, lkb_id, mode, res_name, res_length); +bastfn (astparam, mode); + }else if (flags &DLM_CB_CAST ) { +trace_dlm_ast (ls_id, lkb_id, sb_status, sb_flags, res_name, + res_length); + lksb->sb_status = sb_status; + lksb->sb_flags = sb_flags; +astfn (astparam); } +} +static void dlm_do_callback (struct dlm_callback *cb) +{ +dlm_run_callback (cb->ls_id, cb->lkb_id, cb->mode, cb->flags, + cb->sb_flags, cb->sb_status, cb->lkb_lksb, + cb->astfn, cb->bastfn, cb->astparam, + cb->res_name, cb->res_length);dlm_free_cb (cb); } -int dlm_queue_lkb_callback (struct dlm_lkb *lkb,uint32_t flags,int mode, -int status,uint32_t sbflags, -struct dlm_callback **cb) +static void dlm_callback_work (struct work_struct *work) +{ + struct dlm_callback *cb = container_of(work, struct dlm_callback, work); + +dlm_do_callback (cb); +} + +bool dlm_may_skip_callback (struct dlm_lkb *lkb,uint32_t flags,int mode, +int status,uint32_t sbflags,int *copy_lvb) {struct dlm_rsb *rsb = lkb->lkb_resource; -int rv =DLM_ENQUEUE_CALLBACK_SUCCESS ;struct dlm_ls *ls = rsb->res_ls; -int copy_lvb =0 ;int prev_mode; +if (copy_lvb) + *copy_lvb =0 ; +if (flags &DLM_CB_BAST ) {/* if cb is a bast, it should be skipped if the blocking mode is * compatible with the last granted mode
@@ -56,7 +73,7 @@ int dlm_queue_lkb_callback(struct dlm_lkb *lkb, uint32_t flags, int mode,log_debug (ls,"skip %x bast mode %d for cast mode %d" , lkb->lkb_id, mode, lkb->lkb_last_cast_cb_mode); - goto out; +return true; } }
@@ -74,7 +91,7 @@ int dlm_queue_lkb_callback(struct dlm_lkb *lkb, uint32_t flags, int mode, (prev_mode > mode && prev_mode > DLM_LOCK_PR)) {log_debug (ls,"skip %x add bast mode %d for bast mode %d" , lkb->lkb_id, mode, prev_mode); - goto out; +return true; } }
@@ -85,8 +102,10 @@ int dlm_queue_lkb_callback(struct dlm_lkb *lkb, uint32_t flags, int mode, prev_mode = lkb->lkb_last_cast_cb_mode;if (!status && lkb->lkb_lksb->sb_lvbptr && - dlm_lvb_operations[prev_mode +1 ][mode +1 ]) - copy_lvb =1 ; + dlm_lvb_operations[prev_mode +1 ][mode +1 ]) { +if (copy_lvb) + *copy_lvb =1 ; + } } lkb->lkb_last_cast_cb_mode = mode;
@@ -96,11 +115,19 @@ int dlm_queue_lkb_callback(struct dlm_lkb *lkb, uint32_t flags, int mode, lkb->lkb_last_cb_mode = mode; lkb->lkb_last_cb_flags = flags; +return false; +} + +int dlm_get_cb (struct dlm_lkb *lkb,uint32_t flags,int mode, +int status,uint32_t sbflags, +struct dlm_callback **cb) +{ +struct dlm_rsb *rsb = lkb->lkb_resource; +struct dlm_ls *ls = rsb->res_ls; + *cb =dlm_allocate_cb (); -if (!*cb) { - rv =DLM_ENQUEUE_CALLBACK_FAILURE ; - goto out; - } +if (WARN_ON_ONCE (!*cb)) +return -ENOMEM ;/* for tracing */ (*cb)->lkb_id = lkb->lkb_id;
@@ -112,19 +139,34 @@ int dlm_queue_lkb_callback(struct dlm_lkb *lkb, uint32_t flags, int mode, (*cb)->mode = mode; (*cb)->sb_status = status; (*cb)->sb_flags = (sbflags &0x000000FF ); - (*cb)->copy_lvb = copy_lvb; (*cb)->lkb_lksb = lkb->lkb_lksb; - rv =DLM_ENQUEUE_CALLBACK_NEED_SCHED ; +return 0 ; +} + +static int dlm_get_queue_cb (struct dlm_lkb *lkb,uint32_t flags,int mode, +int status,uint32_t sbflags, +struct dlm_callback **cb) +{ +int rv; -out: -return rv; + rv =dlm_get_cb (lkb, flags, mode, status, sbflags, cb); +if (rv) +return rv; + + (*cb)->astfn = lkb->lkb_astfn; + (*cb)->bastfn = lkb->lkb_bastfn; + (*cb)->astparam = lkb->lkb_astparam; +INIT_WORK (&(*cb)->work, dlm_callback_work); + +return 0 ; }void dlm_add_cb (struct dlm_lkb *lkb,uint32_t flags,int mode,int status, -uint32_t sbflags) +uint32_t sbflags) { -struct dlm_ls *ls = lkb->lkb_resource->res_ls; +struct dlm_rsb *rsb = lkb->lkb_resource; +struct dlm_ls *ls = rsb->res_ls;struct dlm_callback *cb;int rv;
@@ -133,35 +175,34 @@ void dlm_add_cb(struct dlm_lkb *lkb, uint32_t flags, int mode, int status,return ; } - rv =dlm_queue_lkb_callback (lkb, flags, mode, status, sbflags, - &cb); -switch (rv) { -case DLM_ENQUEUE_CALLBACK_NEED_SCHED : - cb->astfn = lkb->lkb_astfn; - cb->bastfn = lkb->lkb_bastfn; - cb->astparam = lkb->lkb_astparam; -INIT_WORK (&cb->work, dlm_callback_work); - -spin_lock_bh (&ls->ls_cb_lock); -if (test_bit (LSFL_CB_DELAY , &ls->ls_flags)) +if (dlm_may_skip_callback (lkb, flags, mode, status, sbflags,NULL )) +return ; + +spin_lock_bh (&ls->ls_cb_lock); +if (test_bit (LSFL_CB_DELAY , &ls->ls_flags)) { + rv =dlm_get_queue_cb (lkb, flags, mode, status, sbflags, &cb); +if (!rv)list_add (&cb->list, &ls->ls_cb_delay); -else -queue_work (ls->ls_callback_wq, &cb->work); -spin_unlock_bh (&ls->ls_cb_lock); -break ; -case DLM_ENQUEUE_CALLBACK_SUCCESS : -break ; -case DLM_ENQUEUE_CALLBACK_FAILURE : - fallthrough; -default : -WARN_ON_ONCE (1 ); -break ; + }else { +if (test_bit (LSFL_SOFTIRQ , &ls->ls_flags)) { +dlm_run_callback (ls->ls_global_id, lkb->lkb_id, mode, flags, + sbflags, status, lkb->lkb_lksb, + lkb->lkb_astfn, lkb->lkb_bastfn, + lkb->lkb_astparam, rsb->res_name, + rsb->res_length); + }else { + rv =dlm_get_queue_cb (lkb, flags, mode, status, sbflags, &cb); +if (!rv) +queue_work (ls->ls_callback_wq, &cb->work); + } } +spin_unlock_bh (&ls->ls_cb_lock); }int dlm_callback_start (struct dlm_ls *ls) { -if (!test_bit (LSFL_FS , &ls->ls_flags)) +if (!test_bit (LSFL_FS , &ls->ls_flags) || +test_bit (LSFL_SOFTIRQ , &ls->ls_flags))return 0 ; ls->ls_callback_wq = alloc_ordered_workqueue("dlm_callback",
@@ -207,7 +248,11 @@ void dlm_callback_resume(struct dlm_ls *ls)spin_lock_bh (&ls->ls_cb_lock);list_for_each_entry_safe (cb, safe, &ls->ls_cb_delay, list) {list_del (&cb->list); -queue_work (ls->ls_callback_wq, &cb->work); +if (test_bit (LSFL_SOFTIRQ , &ls->ls_flags)) +dlm_do_callback (cb); +else +queue_work (ls->ls_callback_wq, &cb->work); + count++;if (count ==MAX_CB_QUEUE )break ;
fs/dlm/ast.h +5 -6
@@ -11,12 +11,11 @@#ifndef __ASTD_DOT_H__#define __ASTD_DOT_H__ -#define DLM_ENQUEUE_CALLBACK_NEED_SCHED 1 -#define DLM_ENQUEUE_CALLBACK_SUCCESS 0 -#define DLM_ENQUEUE_CALLBACK_FAILURE -1 -int dlm_queue_lkb_callback (struct dlm_lkb *lkb,uint32_t flags,int mode, -int status,uint32_t sbflags, -struct dlm_callback **cb); +bool dlm_may_skip_callback (struct dlm_lkb *lkb,uint32_t flags,int mode, +int status,uint32_t sbflags,int *copy_lvb); +int dlm_get_cb (struct dlm_lkb *lkb,uint32_t flags,int mode, +int status,uint32_t sbflags, +struct dlm_callback **cb);void dlm_add_cb (struct dlm_lkb *lkb,uint32_t flags,int mode,int status,uint32_t sbflags);
fs/dlm/dlm_internal.h +1 -0
@@ -727,6 +727,7 @@ struct dlm_ls {#define LSFL_NODIR 10#define LSFL_RECV_MSG_BLOCKED 11#define LSFL_FS 12 +#define LSFL_SOFTIRQ 13#define DLM_PROC_FLAGS_CLOSING 1#define DLM_PROC_FLAGS_COMPAT 2
fs/dlm/lockspace.c +3 -0
@@ -407,6 +407,9 @@ static int new_lockspace(const char *name, const char *cluster, ls->ls_ops_arg = ops_arg; } +if (flags &DLM_LSFL_SOFTIRQ ) +set_bit (LSFL_SOFTIRQ , &ls->ls_flags); +/* ls_exflags are forced to match among nodes, and we don't * need to require all nodes to have some flags set */
fs/dlm/user.c +16 -22
@@ -182,7 +182,7 @@ void dlm_user_add_ast(struct dlm_lkb *lkb, uint32_t flags, int mode,struct dlm_user_args *ua;struct dlm_user_proc *proc;struct dlm_callback *cb; -int rv; +int rv, copy_lvb;if (test_bit (DLM_DFL_ORPHAN_BIT , &lkb->lkb_dflags) ||test_bit (DLM_IFL_DEAD_BIT , &lkb->lkb_iflags))
@@ -213,28 +213,22 @@ void dlm_user_add_ast(struct dlm_lkb *lkb, uint32_t flags, int mode,spin_lock_bh (&proc->asts_spin); - rv =dlm_queue_lkb_callback (lkb, flags, mode, status, sbflags, &cb); -switch (rv) { -case DLM_ENQUEUE_CALLBACK_NEED_SCHED : - cb->ua = *ua; - cb->lkb_lksb = &cb->ua.lksb; -if (cb->copy_lvb) { -memcpy (cb->lvbptr, ua->lksb.sb_lvbptr, -DLM_USER_LVB_LEN ); - cb->lkb_lksb->sb_lvbptr = cb->lvbptr; +if (!dlm_may_skip_callback (lkb, flags, mode, status, sbflags, + ©_lvb)) { + rv =dlm_get_cb (lkb, flags, mode, status, sbflags, &cb); +if (!rv) { + cb->copy_lvb = copy_lvb; + cb->ua = *ua; + cb->lkb_lksb = &cb->ua.lksb; +if (copy_lvb) { +memcpy (cb->lvbptr, ua->lksb.sb_lvbptr, +DLM_USER_LVB_LEN ); + cb->lkb_lksb->sb_lvbptr = cb->lvbptr; + } + +list_add_tail (&cb->list, &proc->asts); +wake_up_interruptible (&proc->wait); } - -list_add_tail (&cb->list, &proc->asts); -wake_up_interruptible (&proc->wait); -break ; -case DLM_ENQUEUE_CALLBACK_SUCCESS : -break ; -case DLM_ENQUEUE_CALLBACK_FAILURE : - fallthrough; -default : -spin_unlock_bh (&proc->asts_spin); -WARN_ON_ONCE (1 ); - goto out; }spin_unlock_bh (&proc->asts_spin);
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH 7/8 dlm: convert ls_cb_lock to rwlock
In reply to Alexander AringCurrently a parallel call of dlm_add_cb() can happen either from the DLM API call or from the DLM message receive context. In this case a rwlock could have a benefit when both context running into the same critical section at the same time. In future more parallel message receive context can happen at the same time so that this conversion of a per lockspace lock to a rwlock is more useful. In far future the whole delayed callbacks might not be necessary when the synchronization happens in a different way than it is now done.
--- fs/dlm/ast.c | 21 +++++++++++++++------ fs/dlm/dlm_internal.h | 2 +- fs/dlm/lockspace.c | 2 +- 3 files changed, 17 insertions(+), 8 deletions(-)
fs/dlm/ast.c +15 -6
@@ -178,11 +178,20 @@ void dlm_add_cb(struct dlm_lkb *lkb, uint32_t flags, int mode, int status,if (dlm_may_skip_callback (lkb, flags, mode, status, sbflags,NULL ))return ; -spin_lock_bh (&ls->ls_cb_lock); +retry: +read_lock_bh (&ls->ls_cb_lock);if (test_bit (LSFL_CB_DELAY , &ls->ls_flags)) { +read_unlock_bh (&ls->ls_cb_lock); +write_lock_bh (&ls->ls_cb_lock); +if (!test_bit (LSFL_CB_DELAY , &ls->ls_flags)) { +write_unlock_bh (&ls->ls_cb_lock); + goto retry; + } + rv =dlm_get_queue_cb (lkb, flags, mode, status, sbflags, &cb);if (!rv)list_add (&cb->list, &ls->ls_cb_delay); +write_unlock_bh (&ls->ls_cb_lock); }else {if (test_bit (LSFL_SOFTIRQ , &ls->ls_flags)) { dlm_run_callback(ls->ls_global_id, lkb->lkb_id, mode, flags,
@@ -195,8 +204,8 @@ void dlm_add_cb(struct dlm_lkb *lkb, uint32_t flags, int mode, int status,if (!rv)queue_work (ls->ls_callback_wq, &cb->work); } +read_unlock_bh (&ls->ls_cb_lock); } -spin_unlock_bh (&ls->ls_cb_lock); }int dlm_callback_start (struct dlm_ls *ls)
@@ -225,9 +234,9 @@ void dlm_callback_suspend(struct dlm_ls *ls)if (!test_bit (LSFL_FS , &ls->ls_flags))return ; -spin_lock_bh (&ls->ls_cb_lock); +write_lock_bh (&ls->ls_cb_lock);set_bit (LSFL_CB_DELAY , &ls->ls_flags); -spin_unlock_bh (&ls->ls_cb_lock); +write_unlock_bh (&ls->ls_cb_lock);if (ls->ls_callback_wq)flush_workqueue (ls->ls_callback_wq);
@@ -245,7 +254,7 @@ void dlm_callback_resume(struct dlm_ls *ls)return ; more: -spin_lock_bh (&ls->ls_cb_lock); +write_lock_bh (&ls->ls_cb_lock);list_for_each_entry_safe (cb, safe, &ls->ls_cb_delay, list) {list_del (&cb->list);if (test_bit (LSFL_SOFTIRQ , &ls->ls_flags))
@@ -260,7 +269,7 @@ void dlm_callback_resume(struct dlm_ls *ls) empty =list_empty (&ls->ls_cb_delay);if (empty)clear_bit (LSFL_CB_DELAY , &ls->ls_flags); -spin_unlock_bh (&ls->ls_cb_lock); +write_unlock_bh (&ls->ls_cb_lock); sum += count;if (!empty) {
fs/dlm/dlm_internal.h +1 -1
@@ -653,7 +653,7 @@ struct dlm_ls {/* recovery related */ -spinlock_t ls_cb_lock; +rwlock_t ls_cb_lock;struct list_head ls_cb_delay;/* save for queue_work later */ struct task_struct *ls_recoverd_task;struct mutex ls_recoverd_active;
fs/dlm/lockspace.c +1 -1
@@ -449,7 +449,7 @@ static int new_lockspace(const char *name, const char *cluster,init_completion (&ls->ls_recovery_done); ls->ls_recovery_result =-1 ; -spin_lock_init (&ls->ls_cb_lock); +rwlock_init (&ls->ls_cb_lock);INIT_LIST_HEAD (&ls->ls_cb_delay); ls->ls_recoverd_task =NULL ;
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH 8/8 md-cluster: use DLM_LSFL_SOFTIRQ for dlm_new_lockspace()
In reply to Alexander AringRecently the DLM subsystem introduced the flag DLM_LSFL_SOFTIRQ for dlm_new_lockspace() to signal the capability to handle DLM ast/bast callbacks in softirq context to avoid an additional context switch due the DLM callback workqueue.
The md-cluster implementation only does synchronized calls above the async DLM API. That synchronized API should may be also offered by DLM, however it is very simple as md-cluster callbacks only does a complete() call for their wait_for_completion() wait that is occurred after the async DLM API call. This patch activates the recently introduced DLM_LSFL_SOFTIRQ flag that allows that the DLM callbacks are executed in a softirq context that md-cluster can handle. It is reducing a unnecessary context workqueue switch and should speed up DLM in some circumstance.
In future other DLM users e.g. gfs2 will also take usage of this flag to avoid the additional context switch due the DLM callback workqueue. In far future hopefully we can remove this kernel lockspace flag only and remove the callback workqueue at all.
--- drivers/md/md-cluster.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/md/md-cluster.c +1 -1
@@ -887,7 +887,7 @@ static int join(struct mddev *mddev, int nodes)memset (str,0 ,64 );sprintf (str,"%pU" , mddev->uuid); ret =dlm_new_lockspace (str, mddev->bitmap_info.cluster_name, -0 ,LVB_SIZE , &md_ls_ops, mddev, +DLM_LSFL_SOFTIRQ ,LVB_SIZE , &md_ls_ops, mddev, &ops_rv, &cinfo->lockspace);if (ret) goto err;
Signature · 1 line
2.43.0
Alexander Aring
maintainerRe: [PATCH dlm/next 8/8] md-cluster: use DLM_LSFL_SOFTIRQ for dlm_new_lockspace()
Hi,
Quoted 6 lines · Alexander Aring
Recently the DLM subsystem introduced the flag
DLM_LSFL_SOFTIRQfordlm_new_lockspace()to signal the capability to handle DLM ast/bast callbacks in softirq context to avoid an additional context switch due the DLM callback workqueue.
Quoted 10 lines
The md-cluster implementation only does synchronized calls above the async DLM API. That synchronized API should may be also offered by DLM, however it is very simple as md-cluster callbacks only does a
complete()call for theirwait_for_completion()wait that is occurred after the async DLM API call. This patch activates the recently introducedDLM_LSFL_SOFTIRQflag that allows that the DLM callbacks are executed in a softirq context that md-cluster can handle. It is reducing a unnecessary context workqueue switch and should speed up DLM in some circumstance.
Can somebody with a md-cluster environment test it as well? All I was doing was (with a working dlm_controld cluster env):
mdadm --create /dev/md0 --bitmap=clustered --metadata=1.2 --raid-devices=2 --level=mirror /dev/sda /dev/sdb
sda and sdb are shared block devices on each node.
Create a /etc/mdadm.conf with the content mostly out of:
mdadm --detail --scan
on each node.
Then call mdadm --assemble on all nodes where not "mdadm --create ..." happened. I hope that is the right thing to do and I had with "dlm_tool ls" a UUID as a lockspace name with some md-cluster locks being around.
To bring this new flag upstream, would it be okay to get this through dlm-tree? I am requesting here for an "Acked-by" tag from the md maintainers.
Thanks.
- Alex
Re: [PATCH dlm/next 8/8] md-cluster: use DLM_LSFL_SOFTIRQ for dlm_new_lockspace()
Quoted 30 lines · Alexander Aring
Hi,
Quoted 6 lines · Alexander Aring
Recently the DLM subsystem introduced the flag
DLM_LSFL_SOFTIRQfordlm_new_lockspace()to signal the capability to handle DLM ast/bast callbacks in softirq context to avoid an additional context switch due the DLM callback workqueue.Quoted 10 lines
The md-cluster implementation only does synchronized calls above the async DLM API. That synchronized API should may be also offered by DLM, however it is very simple as md-cluster callbacks only does a
complete()call for theirwait_for_completion()wait that is occurred after the async DLM API call. This patch activates the recently introducedDLM_LSFL_SOFTIRQflag that allows that the DLM callbacks are executed in a softirq context that md-cluster can handle. It is reducing a unnecessary context workqueue switch and should speed up DLM in some circumstance.Can somebody with a md-cluster environment test it as well? All I was doing was (with a working
dlm_controldcluster env):mdadm --create
/dev/md0--bitmap=clustered --metadata=1.2 --raid-devices=2 --level=mirror/dev/sda/dev/sdbsda and sdb are shared block devices on each node.
Create a
/etc/mdadm.confwith the content mostly out of:
Quoted 7 lines
mdadm --detail --scan
on each node.
Then call mdadm --assemble on all nodes where not "mdadm --create ..." happened. I hope that is the right thing to do and I had with "
dlm_toolls" a UUID as a lockspace name with some md-cluster locks being around.
The above setup method is correct.
SUSE doc [1] provides more details on assembling the clustered array.
Quoted 5 lines
To bring this new flag upstream, would it be okay to get this through dlm-tree? I am requesting here for an "Acked-by" tag from the md maintainers.
I compiled & tested the dlm-tree [2] with SUSE CI env, and didn't see these patches introduce new issue.
[2]: https://git.kernel.org/pub/scm/linux/kernel/git/teigland/linux-dlm.git/log/?h=next
Thanks,
Heming
Alexander Aring
maintainerRe: [PATCH dlm/next 8/8] md-cluster: use DLM_LSFL_SOFTIRQ for dlm_new_lockspace()
Hi Heming,
Quoted 34 lines · Heming Zhao
Quoted 32 lines · Alexander Aring
Hi,
Quoted 6 lines · Alexander Aring
Recently the DLM subsystem introduced the flag
DLM_LSFL_SOFTIRQfordlm_new_lockspace()to signal the capability to handle DLM ast/bast callbacks in softirq context to avoid an additional context switch due the DLM callback workqueue.Quoted 10 lines
The md-cluster implementation only does synchronized calls above the async DLM API. That synchronized API should may be also offered by DLM, however it is very simple as md-cluster callbacks only does a
complete()call for theirwait_for_completion()wait that is occurred after the async DLM API call. This patch activates the recently introducedDLM_LSFL_SOFTIRQflag that allows that the DLM callbacks are executed in a softirq context that md-cluster can handle. It is reducing a unnecessary context workqueue switch and should speed up DLM in some circumstance.Can somebody with a md-cluster environment test it as well? All I was doing was (with a working
dlm_controldcluster env):mdadm --create
/dev/md0--bitmap=clustered --metadata=1.2 --raid-devices=2 --level=mirror/dev/sda/dev/sdbsda and sdb are shared block devices on each node.
Create a
/etc/mdadm.confwith the content mostly out of:mdadm --detail --scan
Quoted 9 lines
Quoted 5 lines
on each node.
Then call mdadm --assemble on all nodes where not "mdadm --create ..." happened. I hope that is the right thing to do and I had with "
dlm_toolls" a UUID as a lockspace name with some md-cluster locks being around.The above setup method is correct.
SUSE doc [1] provides more details on assembling the clustered array.
yea, I saw that and mostly cut it down into the necessary steps in my development setup.
Thanks for confirming I did something right here.
Quoted 11 lines
Quoted 5 lines
To bring this new flag upstream, would it be okay to get this through dlm-tree? I am requesting here for an "Acked-by" tag from the md maintainers.
I compiled & tested the dlm-tree [2] with SUSE CI env, and didn't see these patches introduce new issue.
Thanks for doing that. So that means you tried the dlm-tree with this patch series applied?
Song or Yu, can I get an "Acked-by" from you and an answer if it is okay that this md-cluster.c patch goes upstream via dlm-tree?
Thanks.
- Alex
Re: [PATCH dlm/next 8/8] md-cluster: use DLM_LSFL_SOFTIRQ for dlm_new_lockspace()
Quoted 60 lines · Alexander Aring
Hi Heming,
Quoted 34 lines · Heming Zhao
Quoted 32 lines · Alexander Aring
Hi,
Quoted 6 lines · Alexander Aring
Recently the DLM subsystem introduced the flag
DLM_LSFL_SOFTIRQfordlm_new_lockspace()to signal the capability to handle DLM ast/bast callbacks in softirq context to avoid an additional context switch due the DLM callback workqueue.Quoted 10 lines
The md-cluster implementation only does synchronized calls above the async DLM API. That synchronized API should may be also offered by DLM, however it is very simple as md-cluster callbacks only does a
complete()call for theirwait_for_completion()wait that is occurred after the async DLM API call. This patch activates the recently introducedDLM_LSFL_SOFTIRQflag that allows that the DLM callbacks are executed in a softirq context that md-cluster can handle. It is reducing a unnecessary context workqueue switch and should speed up DLM in some circumstance.Can somebody with a md-cluster environment test it as well? All I was doing was (with a working
dlm_controldcluster env):mdadm --create
/dev/md0--bitmap=clustered --metadata=1.2 --raid-devices=2 --level=mirror/dev/sda/dev/sdbsda and sdb are shared block devices on each node.
Create a
/etc/mdadm.confwith the content mostly out of:mdadm --detail --scan
Quoted 9 lines
Quoted 5 lines
on each node.
Then call mdadm --assemble on all nodes where not "mdadm --create ..." happened. I hope that is the right thing to do and I had with "
dlm_toolls" a UUID as a lockspace name with some md-cluster locks being around.The above setup method is correct.
SUSE doc [1] provides more details on assembling the clustered array.yea, I saw that and mostly cut it down into the necessary steps in my development setup.
Thanks for confirming I did something right here.
Quoted 8 lines
Quoted 5 lines
To bring this new flag upstream, would it be okay to get this through dlm-tree? I am requesting here for an "Acked-by" tag from the md maintainers.
Quoted 6 lines
Quoted 3 lines
I compiled & tested the dlm-tree [2] with SUSE CI env, and didn't see these patches introduce new issue.
Thanks for doing that. So that means you tried the dlm-tree with this patch series applied?
Yes.
-Heming
Quoted 8 lines
Song or Yu, can I get an "Acked-by" from you and an answer if it is okay that this
md-cluster.cpatch goes upstream via dlm-tree?Thanks.
- Alex
Re: [PATCH dlm/next 8/8] md-cluster: use DLM_LSFL_SOFTIRQ for dlm_new_lockspace()
In reply to Alexander AringQuoted 65 lines · Alexander Aring
Hi Heming,
Quoted 34 lines · Heming Zhao
Quoted 32 lines · Alexander Aring
Hi,
Quoted 6 lines · Alexander Aring
Recently the DLM subsystem introduced the flag
DLM_LSFL_SOFTIRQfordlm_new_lockspace()to signal the capability to handle DLM ast/bast callbacks in softirq context to avoid an additional context switch due the DLM callback workqueue.Quoted 10 lines
The md-cluster implementation only does synchronized calls above the async DLM API. That synchronized API should may be also offered by DLM, however it is very simple as md-cluster callbacks only does a
complete()call for theirwait_for_completion()wait that is occurred after the async DLM API call. This patch activates the recently introducedDLM_LSFL_SOFTIRQflag that allows that the DLM callbacks are executed in a softirq context that md-cluster can handle. It is reducing a unnecessary context workqueue switch and should speed up DLM in some circumstance.Can somebody with a md-cluster environment test it as well? All I was doing was (with a working
dlm_controldcluster env):mdadm --create
/dev/md0--bitmap=clustered --metadata=1.2 --raid-devices=2 --level=mirror/dev/sda/dev/sdbsda and sdb are shared block devices on each node.
Create a
/etc/mdadm.confwith the content mostly out of:mdadm --detail --scan
Quoted 9 lines
Quoted 5 lines
on each node.
Then call mdadm --assemble on all nodes where not "mdadm --create ..." happened. I hope that is the right thing to do and I had with "
dlm_toolls" a UUID as a lockspace name with some md-cluster locks being around.The above setup method is correct.
SUSE doc [1] provides more details on assembling the clustered array.yea, I saw that and mostly cut it down into the necessary steps in my development setup.
Thanks for confirming I did something right here.
Quoted 11 lines
Quoted 5 lines
To bring this new flag upstream, would it be okay to get this through dlm-tree? I am requesting here for an "Acked-by" tag from the md maintainers.
I compiled & tested the dlm-tree [2] with SUSE CI env, and didn't see these patches introduce new issue.
Quoted 5 lines
Thanks for doing that. So that means you tried the dlm-tree with this patch series applied?
Song or Yu, can I get an "Acked-by" from you and an answer if it is okay that this
md-cluster.cpatch goes upstream via dlm-tree?
LGTM.
Yes, let's route this via dlm-tree.
Thanks,
Song