dlm: sand fix, rhashtable, timers and lookup hotpath speedup
Alexander Aring
maintainercover dlm: sand fix, rhashtable, timers and lookup hotpath speedup
Hi,
this is a RFC for an immense change in DLM rsb hashtable logic. It removes the double lookup functionality for rsb hashtables, convert to rhashtable instead of own bucket hlist hash implementation.
At first there is a fix for scand that I detected while I was implementing this patch series. It could be that remove messages are still send when the lockspace is releasing the resource that could occur into a use after free.
There is a conversion to use lists (keep/toss lists) instead of iterating over the hash bucket. As we do a transition to rhashtable, they don't like to being iterated regarding to their own bucket sizing implementation that is sitting in the rhashtable implementation. We just introduce the lists to do the iteration, as advantage we have a huge reduce of code in the debugfs dump functionality as we use the dump list helpers of debugfs. There is also a potential refcount bug when holding rsb references of rsbs in toss state as receive remove message requires no rsb references being hold. Another issue is also holding rsb in keep state as they are not going into toss state when they required to.
It is now forbidden to hold references of rsbs in toss state. The refcounter must be only functional in rsb keep state. That hopefully will show is more invalid usage of the rsb refcounter if the rsb is in toss state.
The scand was being fixed but now also it's removed. The scand process was holding the hashtable/hash bucket lock for a longer timer because it iterated over the whole hash. We use timers now to reduce the held time of the hashtable lock. We take here the advantage of the timers data structure that the rsb timer will delete the rsb itself from the hash. No lookup required as this is part of the timer subsystem.
As end we move very likely lookup hotpath to read lock mostly. This should for sure avoid contention in the most cases. Unlikely path need to still hold the write lock and do some extra relookup and check if the state of an rsb changed. However I think we hit over 90% the likely path that we only need to hold the read lock and avoid contention between processing DLM messages and the user triggers new DLM requests.
- Alex
Alexander Aring (9):
dlm: increment ls_count on find_ls_to_scan()
dlm: change to non per bucket hashtable lock
dlm: merge toss and keep hash into one
dlm: fix avoid rsb hold during debugfs dump
dlm: switch to use rhashtable for rsbs
dlm: remove refcounting if rsb is on toss
dlm: drop scand kthread and use timers
dlm: likely read lock path for rsb lookup
dlm: convert lkbidr to rwlock
fs/dlm/config.c | 8 + fs/dlm/config.h | 2 + fs/dlm/debug_fs.c | 212 ++---------- fs/dlm/dir.c | 14 +- fs/dlm/dlm_internal.h | 39 +-- fs/dlm/lock.c | 774 ++++++++++++++++++++++-------------------- fs/dlm/lock.h | 3 +- fs/dlm/lockspace.c | 151 ++------ fs/dlm/recover.c | 38 ++- fs/dlm/recoverd.c | 40 +-- 10 files changed, 541 insertions(+), 740 deletions(-)
Signature · 1 line
2.43.0
Alexander Aring
maintainerRFC 1/9 dlm: increment ls_count on find_ls_to_scan()
This patch increments the ls_count atomic value that is used as a reference counter in find_ls_to_scan(). An additional dlm_put_lockspace() is necessary in dlm_scand() to drop this reference again.
This handling is necessary because release_lockspace() calls remove_lockspace() that removes the lockspace from the global lockspace list. It only does that when the lockspace is not in use anymore signaled by the menitoned ls_count atomic value. Currently dlm_scand() can be executed when release_lockspace() is freeing all lockspace resources. As example:
1. scand timeout occurs
2. find_ls_to_scan() returns ls A - lslist_lock is not held anymore
3. dlm_scan_rsbs(A) does its job
4. release_lockspace(A, 2) is called by the user. User has no control of dlm_scan_rsbs(A)
5. lockspace_busy(A, 2) returns 0, it's not busy
6. dlm_scan_rsbs(A) iterates over the whole rsb hash bucket, sometimes triggers send_remove()
7. release_lockspace(A, 2) does not stop scand because ls_count was >= 2
8. remove_lockspace() will ignore ongoing scand because scand does not hold a reference as ls_count is intended to do so
9. release_lockspace(A, 2) cleans up everything while 6. can occur
Point 8 is a problem because both access the same resources and release_lockspace() frees them.
--- fs/dlm/lockspace.c | 3 +++ 1 file changed, 3 insertions(+)
fs/dlm/lockspace.c +3 -0
@@ -255,6 +255,7 @@ static struct dlm_ls *find_ls_to_scan(void)list_for_each_entry (ls, &lslist, ls_list) {if (time_after_eq (jiffies, ls->ls_scan_time + dlm_config.ci_scan_secs *HZ )) { +atomic_inc (&ls->ls_count);spin_unlock_bh (&lslist_lock);return ls; }
@@ -277,6 +278,8 @@ static int dlm_scand(void *data) }else { ls->ls_scan_time +=HZ ; } + +dlm_put_lockspace (ls);continue ; }schedule_timeout_interruptible (dlm_config.ci_scan_secs *HZ );
Signature · 1 line
2.43.0
Alexander Aring
maintainerRFC 2/9 dlm: change to non per bucket hashtable lock
In reply to Alexander AringThis patch prepares to switch to use the resizable, scalable, concurrent hash table (rhashtable) implementation instead of having a own per bucket implementation. The rhashtable implementation deals with the buckets internally for that reason we use a per lockspace rhashtable lock to protect mutual rhashtable access. This might reduce the perfromance of the hash, depends on how rhashtable lookup performs to the current own DLM hash bucket implementation. However we switch to the rhashtable implementation and a potential lockless access can be part of future changes.
--- fs/dlm/debug_fs.c | 24 +++++++------- fs/dlm/dir.c | 4 +-- fs/dlm/dlm_internal.h | 2 +- fs/dlm/lock.c | 77 +++++++++++++++++++++---------------------- fs/dlm/lockspace.c | 2 +- fs/dlm/recover.c | 4 +-- fs/dlm/recoverd.c | 8 ++--- 7 files changed, 60 insertions(+), 61 deletions(-)
fs/dlm/debug_fs.c +12 -12
@@ -452,7 +452,7 @@ static void *table_seq_start(struct seq_file *seq, loff_t *pos) tree = toss ? &ls->ls_rsbtbl[bucket].toss : &ls->ls_rsbtbl[bucket].keep; -spin_lock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_lock_bh (&ls->ls_rsbtbl_lock);if (!RB_EMPTY_ROOT (tree)) {for (node =rb_first (tree); node; node =rb_next (node)) { r = rb_entry(node, struct dlm_rsb, res_hashnode);
@@ -460,12 +460,12 @@ static void *table_seq_start(struct seq_file *seq, loff_t *pos)dlm_hold_rsb (r); ri->rsb = r; ri->bucket = bucket; -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock);return ri; } } } -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock);/* * move to the first rsb in the next non-empty bucket
@@ -484,18 +484,18 @@ static void *table_seq_start(struct seq_file *seq, loff_t *pos) } tree = toss ? &ls->ls_rsbtbl[bucket].toss : &ls->ls_rsbtbl[bucket].keep; -spin_lock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_lock_bh (&ls->ls_rsbtbl_lock);if (!RB_EMPTY_ROOT (tree)) { node =rb_first (tree); r = rb_entry(node, struct dlm_rsb, res_hashnode);dlm_hold_rsb (r); ri->rsb = r; ri->bucket = bucket; -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock); *pos = n;return ri; } -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock); } }
@@ -516,7 +516,7 @@ static void *table_seq_next(struct seq_file *seq, void *iter_ptr, loff_t *pos)* move to the next rsb in the same bucket */ -spin_lock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_lock_bh (&ls->ls_rsbtbl_lock); rp = ri->rsb; next =rb_next (&rp->res_hashnode);
@@ -524,12 +524,12 @@ static void *table_seq_next(struct seq_file *seq, void *iter_ptr, loff_t *pos) r = rb_entry(next, struct dlm_rsb, res_hashnode);dlm_hold_rsb (r); ri->rsb = r; -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock);dlm_put_rsb (rp); ++*pos;return ri; } -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock);dlm_put_rsb (rp);/*
@@ -550,18 +550,18 @@ static void *table_seq_next(struct seq_file *seq, void *iter_ptr, loff_t *pos) } tree = toss ? &ls->ls_rsbtbl[bucket].toss : &ls->ls_rsbtbl[bucket].keep; -spin_lock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_lock_bh (&ls->ls_rsbtbl_lock);if (!RB_EMPTY_ROOT (tree)) { next =rb_first (tree); r = rb_entry(next, struct dlm_rsb, res_hashnode);dlm_hold_rsb (r); ri->rsb = r; ri->bucket = bucket; -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock); *pos = n;return ri; } -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock); } }
fs/dlm/dir.c +2 -2
@@ -204,12 +204,12 @@ static struct dlm_rsb *find_rsb_root(struct dlm_ls *ls, const char *name, hash =jhash (name, len,0 ); bucket = hash & (ls->ls_rsbtbl_size -1 ); -spin_lock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_lock_bh (&ls->ls_rsbtbl_lock); rv =dlm_search_rsb_tree (&ls->ls_rsbtbl[bucket].keep, name, len, &r);if (rv) rv =dlm_search_rsb_tree (&ls->ls_rsbtbl[bucket].toss, name, len, &r); -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock);if (!rv)return r;
fs/dlm/dlm_internal.h +1 -1
@@ -105,7 +105,6 @@ do { \struct dlm_rsbtable {struct rb_root keep;struct rb_root toss; -spinlock_t lock;unsigned long flags; };
@@ -593,6 +592,7 @@ struct dlm_ls {spinlock_t ls_lkbidr_spin;struct dlm_rsbtable *ls_rsbtbl; +spinlock_t ls_rsbtbl_lock;uint32_t ls_rsbtbl_size;spinlock_t ls_waiters_lock;
fs/dlm/lock.c +38 -39
289 lines not on this page · raw message
fs/dlm/lockspace.c +1 -1
@@ -495,6 +495,7 @@ static int new_lockspace(const char *name, const char *cluster,*/ ls->ls_exflags = (flags & ~(DLM_LSFL_FS |DLM_LSFL_NEWEXCL )); +spin_lock_init (&ls->ls_rsbtbl_lock); size =READ_ONCE (dlm_config.ci_rsbtbl_size); ls->ls_rsbtbl_size = size;
@@ -504,7 +505,6 @@ static int new_lockspace(const char *name, const char *cluster,for (i =0 ; i < size; i++) { ls->ls_rsbtbl[i].keep.rb_node =NULL ; ls->ls_rsbtbl[i].toss.rb_node =NULL ; -spin_lock_init (&ls->ls_rsbtbl[i].lock); }for (i =0 ; i <DLM_REMOVE_NAMES_MAX ; i++) {
fs/dlm/recover.c +2 -2
@@ -886,8 +886,8 @@ void dlm_clear_toss(struct dlm_ls *ls)unsigned int count =0 ;int i; +spin_lock (&ls->ls_rsbtbl_lock);for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -spin_lock_bh (&ls->ls_rsbtbl[i].lock);for (n =rb_first (&ls->ls_rsbtbl[i].toss); n; n = next) { next =rb_next (n); r = rb_entry(n, struct dlm_rsb, res_hashnode);
@@ -895,8 +895,8 @@ void dlm_clear_toss(struct dlm_ls *ls)dlm_free_rsb (r); count++; } -spin_unlock_bh (&ls->ls_rsbtbl[i].lock); } +spin_unlock_bh (&ls->ls_rsbtbl_lock);if (count)log_rinfo (ls,"dlm_clear_toss %u done" , count);
fs/dlm/recoverd.c +4 -4
@@ -33,8 +33,8 @@ static int dlm_create_masters_list(struct dlm_ls *ls) goto out; } +spin_lock_bh (&ls->ls_rsbtbl_lock);for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -spin_lock_bh (&ls->ls_rsbtbl[i].lock);for (n =rb_first (&ls->ls_rsbtbl[i].keep); n; n =rb_next (n)) { r = rb_entry(n, struct dlm_rsb, res_hashnode);if (r->res_nodeid)
@@ -43,8 +43,8 @@ static int dlm_create_masters_list(struct dlm_ls *ls)list_add (&r->res_masters_list, &ls->ls_masters_list);dlm_hold_rsb (r); } -spin_unlock_bh (&ls->ls_rsbtbl[i].lock); } +spin_unlock_bh (&ls->ls_rsbtbl_lock); out:write_unlock_bh (&ls->ls_masters_lock);return error;
@@ -68,8 +68,8 @@ static void dlm_create_root_list(struct dlm_ls *ls, struct list_head *root_list)struct dlm_rsb *r;int i; +spin_lock_bh (&ls->ls_rsbtbl_lock);for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -spin_lock_bh (&ls->ls_rsbtbl[i].lock);for (n =rb_first (&ls->ls_rsbtbl[i].keep); n; n =rb_next (n)) { r = rb_entry(n, struct dlm_rsb, res_hashnode);list_add (&r->res_root_list, root_list);
@@ -78,8 +78,8 @@ static void dlm_create_root_list(struct dlm_ls *ls, struct list_head *root_list)if (!RB_EMPTY_ROOT (&ls->ls_rsbtbl[i].toss))log_error (ls,"%s toss not empty" , __func__); -spin_unlock_bh (&ls->ls_rsbtbl[i].lock); } +spin_unlock_bh (&ls->ls_rsbtbl_lock); }static void dlm_release_root_list (struct list_head *root_list)
Signature · 1 line
2.43.0
Alexander Aring
maintainerRFC 3/9 dlm: merge toss and keep hash into one
In reply to Alexander AringThere are several places in current DLM that can end in two lookups, e.g. first look into the keep hash, then into the toss hash if its not in the keep hash. However there can't be never the same rsb into both of these hashs. This patch introduce a new rsb state flag "RSB_TOSS" that signals if the rsb would be on the toss hash or not, which then would be on the keep hash. Instead of those multiple lookups there is only one now that reduce the amount of hash lookups in cases where the rsb wasn't on keep hash. The move of an rsb from rsb toss to keep hash or vice versa need to be done automatically anyway.
--- fs/dlm/debug_fs.c | 29 ++++++++++-- fs/dlm/dir.c | 6 +-- fs/dlm/dlm_internal.h | 4 +- fs/dlm/lock.c | 103 ++++++++++++++++++++++-------------------- fs/dlm/lockspace.c | 13 ++---- fs/dlm/recover.c | 7 ++- fs/dlm/recoverd.c | 12 ++--- 7 files changed, 98 insertions(+), 76 deletions(-)
fs/dlm/debug_fs.c +26 -3
@@ -450,12 +450,20 @@ static void *table_seq_start(struct seq_file *seq, loff_t *pos)if (seq->op == &format4_seq_ops) ri->format =4 ; - tree = toss ? &ls->ls_rsbtbl[bucket].toss : &ls->ls_rsbtbl[bucket].keep; + tree = &ls->ls_rsbtbl[bucket].r;spin_lock_bh (&ls->ls_rsbtbl_lock);if (!RB_EMPTY_ROOT (tree)) {for (node =rb_first (tree); node; node =rb_next (node)) { r = rb_entry(node, struct dlm_rsb, res_hashnode); +if (toss) { +if (!rsb_flag (r,RSB_TOSS )) +continue ; + }else { +if (rsb_flag (r,RSB_TOSS )) +continue ; + } +if (!entry--) {dlm_hold_rsb (r); ri->rsb = r;
@@ -482,12 +490,20 @@ static void *table_seq_start(struct seq_file *seq, loff_t *pos)kfree (ri);return NULL ; } - tree = toss ? &ls->ls_rsbtbl[bucket].toss : &ls->ls_rsbtbl[bucket].keep; + tree = &ls->ls_rsbtbl[bucket].r;spin_lock_bh (&ls->ls_rsbtbl_lock);if (!RB_EMPTY_ROOT (tree)) { node =rb_first (tree); r = rb_entry(node, struct dlm_rsb, res_hashnode); +if (toss) { +if (!rsb_flag (r,RSB_TOSS )) +continue ; + }else { +if (rsb_flag (r,RSB_TOSS )) +continue ; + } +dlm_hold_rsb (r); ri->rsb = r; ri->bucket = bucket;
@@ -548,12 +564,19 @@ static void *table_seq_next(struct seq_file *seq, void *iter_ptr, loff_t *pos) ++*pos;return NULL ; } - tree = toss ? &ls->ls_rsbtbl[bucket].toss : &ls->ls_rsbtbl[bucket].keep; + tree = &ls->ls_rsbtbl[bucket].r;spin_lock_bh (&ls->ls_rsbtbl_lock);if (!RB_EMPTY_ROOT (tree)) { next =rb_first (tree); r = rb_entry(next, struct dlm_rsb, res_hashnode); +if (toss) { +if (!rsb_flag (r,RSB_TOSS )) +continue ; + }else { +if (rsb_flag (r,RSB_TOSS )) +continue ; + }dlm_hold_rsb (r); ri->rsb = r; ri->bucket = bucket;
fs/dlm/dir.c +1 -5
@@ -205,12 +205,8 @@ static struct dlm_rsb *find_rsb_root(struct dlm_ls *ls, const char *name, bucket = hash & (ls->ls_rsbtbl_size -1 );spin_lock_bh (&ls->ls_rsbtbl_lock); - rv =dlm_search_rsb_tree (&ls->ls_rsbtbl[bucket].keep, name, len, &r); -if (rv) - rv =dlm_search_rsb_tree (&ls->ls_rsbtbl[bucket].toss, - name, len, &r); + rv =dlm_search_rsb_tree (&ls->ls_rsbtbl[bucket].r, name, len, &r);spin_unlock_bh (&ls->ls_rsbtbl_lock); -if (!rv)return r;
fs/dlm/dlm_internal.h +2 -2
@@ -103,8 +103,7 @@ do { \#define DLM_RTF_SHRINK_BIT 0struct dlm_rsbtable { -struct rb_root keep; -struct rb_root toss; +struct rb_root r;unsigned long flags; };
@@ -376,6 +375,7 @@ enum rsb_flags {RSB_RECOVER_CONVERT ,RSB_RECOVER_GRANT ,RSB_RECOVER_LVB_INVAL , + RSB_TOSS, };static inline void rsb_set_flag (struct dlm_rsb *r,enum rsb_flags flag)
fs/dlm/lock.c +55 -48
247 lines not on this page · raw message
fs/dlm/lockspace.c +3 -10
@@ -503,8 +503,7 @@ static int new_lockspace(const char *name, const char *cluster,if (!ls->ls_rsbtbl) goto out_lsfree;for (i =0 ; i < size; i++) { - ls->ls_rsbtbl[i].keep.rb_node =NULL ; - ls->ls_rsbtbl[i].toss.rb_node =NULL ; + ls->ls_rsbtbl[i].r.rb_node =NULL ; }for (i =0 ; i <DLM_REMOVE_NAMES_MAX ; i++) {
@@ -837,15 +836,9 @@ static int release_lockspace(struct dlm_ls *ls, int force)*/ for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -while ((n =rb_first (&ls->ls_rsbtbl[i].keep))) { +while ((n =rb_first (&ls->ls_rsbtbl[i].r))) { rsb = rb_entry(n, struct dlm_rsb, res_hashnode); -rb_erase (n, &ls->ls_rsbtbl[i].keep); -dlm_free_rsb (rsb); - } - -while ((n =rb_first (&ls->ls_rsbtbl[i].toss))) { - rsb = rb_entry(n, struct dlm_rsb, res_hashnode); -rb_erase (n, &ls->ls_rsbtbl[i].toss); +rb_erase (n, &ls->ls_rsbtbl[i].r);dlm_free_rsb (rsb); } }
fs/dlm/recover.c +5 -2
@@ -888,10 +888,13 @@ void dlm_clear_toss(struct dlm_ls *ls)spin_lock (&ls->ls_rsbtbl_lock);for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -for (n =rb_first (&ls->ls_rsbtbl[i].toss); n; n = next) { +for (n =rb_first (&ls->ls_rsbtbl[i].r); n; n = next) { next =rb_next (n); r = rb_entry(n, struct dlm_rsb, res_hashnode); -rb_erase (n, &ls->ls_rsbtbl[i].toss); +if (!rsb_flag (r,RSB_TOSS )) +continue ; + +rb_erase (n, &ls->ls_rsbtbl[i].r);dlm_free_rsb (r); count++; }
fs/dlm/recoverd.c +6 -6
@@ -35,9 +35,9 @@ static int dlm_create_masters_list(struct dlm_ls *ls)spin_lock_bh (&ls->ls_rsbtbl_lock);for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -for (n =rb_first (&ls->ls_rsbtbl[i].keep); n; n =rb_next (n)) { +for (n =rb_first (&ls->ls_rsbtbl[i].r); n; n =rb_next (n)) { r = rb_entry(n, struct dlm_rsb, res_hashnode); -if (r->res_nodeid) +if (rsb_flag (r,RSB_TOSS ) || r->res_nodeid)continue ;list_add (&r->res_masters_list, &ls->ls_masters_list);
@@ -70,14 +70,14 @@ static void dlm_create_root_list(struct dlm_ls *ls, struct list_head *root_list)spin_lock_bh (&ls->ls_rsbtbl_lock);for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -for (n =rb_first (&ls->ls_rsbtbl[i].keep); n; n =rb_next (n)) { +for (n =rb_first (&ls->ls_rsbtbl[i].r); n; n =rb_next (n)) { r = rb_entry(n, struct dlm_rsb, res_hashnode); +if (WARN_ON_ONCE (rsb_flag (r,RSB_TOSS ))) +continue ; +list_add (&r->res_root_list, root_list);dlm_hold_rsb (r); } - -if (!RB_EMPTY_ROOT (&ls->ls_rsbtbl[i].toss)) -log_error (ls,"%s toss not empty" , __func__); }spin_unlock_bh (&ls->ls_rsbtbl_lock); }
Signature · 1 line
2.43.0
Alexander Aring
maintainerRFC 4/9 dlm: fix avoid rsb hold during debugfs dump
In reply to Alexander AringCurrently the debugfs calls dlm_hold_rsb() while iterating the ls_rsbtbl hash and release the ls_rsbtbl_lock during dumping. We can't do that in every situation because some rsbs don't get into the toss state when they should be on the toss state e.g. when receive_remove() is being called. To avoid this problem we don't use the hash as iteratable datastructure. Moving to rhashtable implementation it should be avoided to use the rhashtable to iterate over it due dynamic rehash background operations that results into a whole new iteration of the rhashtable datastructure. Instead of iterating over the hash we introduce two lists ls_toss and ls_keep as DLM cares about the RSB_TOSS state if iterate over all rsbs. On ls_toss are only rsbs that are on RSB_TOSS state and on ls_keep the rsbs they are not in RSB_TOSS state. There are only few situation to bring a rsb into toss state and back which will be just a list_move() between those datastructures protected by the ls_rsbtbl_lock to keep in sync with the hashtable and RSB_TOSS flag.
The advantage in debugfs is that we can simple use the predefined debugfs dump functionality for linux lists. Instead of calling dlm_hold_rsb() while dumping rsbs, we hold the ls_rsbtbl_lock the whole debugfs dump.
--- fs/dlm/debug_fs.c | 233 +++++++----------------------------------- fs/dlm/dlm_internal.h | 4 + fs/dlm/lock.c | 47 ++++----- fs/dlm/lockspace.c | 3 + fs/dlm/recover.c | 24 ++--- fs/dlm/recoverd.c | 34 +++--- 6 files changed, 84 insertions(+), 261 deletions(-)
fs/dlm/debug_fs.c +36 -197
@@ -366,12 +366,10 @@ static void print_format4(struct dlm_rsb *r, struct seq_file *s)unlock_rsb (r); } -struct rsbtbl_iter { -struct dlm_rsb *rsb; -unsigned bucket; -int format; -int header; -}; +static const struct seq_operations format1_seq_ops; +static const struct seq_operations format2_seq_ops; +static const struct seq_operations format3_seq_ops; +static const struct seq_operations format4_seq_ops;/* * If the buffer is full, seq_printf can be called again, but it
@@ -382,220 +380,61 @@ struct rsbtbl_iter {static int table_seq_show (struct seq_file *seq,void *iter_ptr) { -struct rsbtbl_iter *ri = iter_ptr; - -switch (ri->format) { -case 1 : -print_format1 (ri->rsb, seq); -break ; -case 2 : -if (ri->header) { -seq_puts (seq,"id nodeid remid pid xid exflags flags sts grmode rqmode time_ms r_nodeid r_len r_name\n" ); - ri->header =0 ; - } -print_format2 (ri->rsb, seq); -break ; -case 3 : -if (ri->header) { -seq_puts (seq,"rsb ptr nodeid first_lkid flags !root_list_empty !recover_list_empty recover_locks_count len\n" ); - ri->header =0 ; - } -print_format3 (ri->rsb, seq); -break ; -case 4 : -if (ri->header) { -seq_puts (seq,"rsb ptr nodeid master_nodeid dir_nodeid our_nodeid toss_time flags len str|hex name\n" ); - ri->header =0 ; - } -print_format4 (ri->rsb, seq); -break ; - } + struct dlm_rsb *rsb = list_entry(iter_ptr, struct dlm_rsb, res_rsbs_list); + +if (seq->op == &format1_seq_ops) +print_format1 (rsb, seq); +else if (seq->op == &format2_seq_ops) +print_format2 (rsb, seq); +else if (seq->op == &format3_seq_ops) +print_format3 (rsb, seq); +else if (seq->op == &format4_seq_ops) +print_format4 (rsb, seq);return 0 ; } -static const struct seq_operations format1_seq_ops; -static const struct seq_operations format2_seq_ops; -static const struct seq_operations format3_seq_ops; -static const struct seq_operations format4_seq_ops; -static void *table_seq_start (struct seq_file *seq,loff_t *pos) { -struct rb_root *tree; -struct rb_node *node;struct dlm_ls *ls = seq->private; -struct rsbtbl_iter *ri; -struct dlm_rsb *r; -loff_t n = *pos; -unsigned bucket, entry; -int toss = (seq->op == &format4_seq_ops); - - bucket = n >>32 ; - entry = n & ((1LL <<32 ) -1 ); - -if (bucket >= ls->ls_rsbtbl_size) -return NULL ; - - ri =kzalloc (sizeof (*ri),GFP_NOFS ); -if (!ri) -return NULL ; -if (n ==0 ) - ri->header =1 ; -if (seq->op == &format1_seq_ops) - ri->format =1 ; -if (seq->op == &format2_seq_ops) - ri->format =2 ; -if (seq->op == &format3_seq_ops) - ri->format =3 ; -if (seq->op == &format4_seq_ops) - ri->format =4 ; - - tree = &ls->ls_rsbtbl[bucket].r; +struct list_head *list; -spin_lock_bh (&ls->ls_rsbtbl_lock); -if (!RB_EMPTY_ROOT (tree)) { -for (node =rb_first (tree); node; node =rb_next (node)) { - r = rb_entry(node, struct dlm_rsb, res_hashnode); -if (toss) { -if (!rsb_flag (r,RSB_TOSS )) -continue ; - }else { -if (rsb_flag (r,RSB_TOSS )) -continue ; - } - -if (!entry--) { -dlm_hold_rsb (r); - ri->rsb = r; - ri->bucket = bucket; -spin_unlock_bh (&ls->ls_rsbtbl_lock); -return ri; - } - } +if (!*pos) { +if (seq->op == &format2_seq_ops) +seq_puts (seq,"id nodeid remid pid xid exflags flags sts grmode rqmode time_ms r_nodeid r_len r_name\n" ); +else if (seq->op == &format3_seq_ops) +seq_puts (seq,"rsb ptr nodeid first_lkid flags !root_list_empty !recover_list_empty recover_locks_count len\n" ); +else if (seq->op == &format4_seq_ops) +seq_puts (seq,"rsb ptr nodeid master_nodeid dir_nodeid our_nodeid toss_time flags len str|hex name\n" ); } -spin_unlock_bh (&ls->ls_rsbtbl_lock); - -/* -* move to the first rsb in the next non-empty bucket -*/ - -/* zero the entry */ - n &= ~((1LL <<32 ) -1 ); -while (1 ) { - bucket++; - n +=1LL <<32 ; +if (seq->op == &format4_seq_ops) + list = &ls->ls_toss; +else + list = &ls->ls_keep; -if (bucket >= ls->ls_rsbtbl_size) { -kfree (ri); -return NULL ; - } - tree = &ls->ls_rsbtbl[bucket].r; - -spin_lock_bh (&ls->ls_rsbtbl_lock); -if (!RB_EMPTY_ROOT (tree)) { - node =rb_first (tree); - r = rb_entry(node, struct dlm_rsb, res_hashnode); -if (toss) { -if (!rsb_flag (r,RSB_TOSS )) -continue ; - }else { -if (rsb_flag (r,RSB_TOSS )) -continue ; - } - -dlm_hold_rsb (r); - ri->rsb = r; - ri->bucket = bucket; -spin_unlock_bh (&ls->ls_rsbtbl_lock); - *pos = n; -return ri; - } -spin_unlock_bh (&ls->ls_rsbtbl_lock); - } +spin_lock_bh (&ls->ls_rsbtbl_lock); +return seq_list_start (list, *pos); }static void *table_seq_next (struct seq_file *seq,void *iter_ptr,loff_t *pos) {struct dlm_ls *ls = seq->private; -struct rsbtbl_iter *ri = iter_ptr; -struct rb_root *tree; -struct rb_node *next; -struct dlm_rsb *r, *rp; -loff_t n = *pos; -unsigned bucket; -int toss = (seq->op == &format4_seq_ops); - - bucket = n >>32 ; - -/* -* move to the next rsb in the same bucket -*/ - -spin_lock_bh (&ls->ls_rsbtbl_lock); - rp = ri->rsb; - next =rb_next (&rp->res_hashnode); - -if (next) { - r = rb_entry(next, struct dlm_rsb, res_hashnode); -dlm_hold_rsb (r); - ri->rsb = r; -spin_unlock_bh (&ls->ls_rsbtbl_lock); -dlm_put_rsb (rp); - ++*pos; -return ri; - } -spin_unlock_bh (&ls->ls_rsbtbl_lock); -dlm_put_rsb (rp); - -/* -* move to the first rsb in the next non-empty bucket -*/ +struct list_head *list; -/* zero the entry */ - n &= ~((1LL <<32 ) -1 ); - -while (1 ) { - bucket++; - n +=1LL <<32 ; +if (seq->op == &format4_seq_ops) + list = &ls->ls_toss; +else + list = &ls->ls_keep; -if (bucket >= ls->ls_rsbtbl_size) { -kfree (ri); - ++*pos; -return NULL ; - } - tree = &ls->ls_rsbtbl[bucket].r; - -spin_lock_bh (&ls->ls_rsbtbl_lock); -if (!RB_EMPTY_ROOT (tree)) { - next =rb_first (tree); - r = rb_entry(next, struct dlm_rsb, res_hashnode); -if (toss) { -if (!rsb_flag (r,RSB_TOSS )) -continue ; - }else { -if (rsb_flag (r,RSB_TOSS )) -continue ; - } -dlm_hold_rsb (r); - ri->rsb = r; - ri->bucket = bucket; -spin_unlock_bh (&ls->ls_rsbtbl_lock); - *pos = n; -return ri; - } -spin_unlock_bh (&ls->ls_rsbtbl_lock); - } +return seq_list_next (iter_ptr, list, pos); }static void table_seq_stop (struct seq_file *seq,void *iter_ptr) { -struct rsbtbl_iter *ri = iter_ptr; +struct dlm_ls *ls = seq->private; -if (ri) { -dlm_put_rsb (ri->rsb); -kfree (ri); - } +spin_unlock_bh (&ls->ls_rsbtbl_lock); }static const struct seq_operations format1_seq_ops = {
fs/dlm/dlm_internal.h +4 -0
@@ -339,6 +339,7 @@ struct dlm_rsb {struct list_head res_convertqueue;struct list_head res_waitqueue; +struct list_head res_rsbs_list;struct list_head res_root_list;/* used for recovery */ struct list_head res_masters_list;/* used for recovery */ struct list_head res_recover_list;/* used for recovery */
@@ -595,6 +596,9 @@ struct dlm_ls {spinlock_t ls_rsbtbl_lock;uint32_t ls_rsbtbl_size; +struct list_head ls_toss; +struct list_head ls_keep; +spinlock_t ls_waiters_lock;struct list_head ls_waiters;/* lkbs needing a reply */
fs/dlm/lock.c +21 -26
131 lines not on this page · raw message
fs/dlm/lockspace.c +3 -0
@@ -495,6 +495,8 @@ static int new_lockspace(const char *name, const char *cluster,*/ ls->ls_exflags = (flags & ~(DLM_LSFL_FS |DLM_LSFL_NEWEXCL )); +INIT_LIST_HEAD (&ls->ls_toss); +INIT_LIST_HEAD (&ls->ls_keep);spin_lock_init (&ls->ls_rsbtbl_lock); size =READ_ONCE (dlm_config.ci_rsbtbl_size); ls->ls_rsbtbl_size = size;
@@ -838,6 +840,7 @@ static int release_lockspace(struct dlm_ls *ls, int force)for (i =0 ; i < ls->ls_rsbtbl_size; i++) {while ((n =rb_first (&ls->ls_rsbtbl[i].r))) { rsb = rb_entry(n, struct dlm_rsb, res_hashnode); +list_del (&rsb->res_rsbs_list);rb_erase (n, &ls->ls_rsbtbl[i].r);dlm_free_rsb (rsb); }
fs/dlm/recover.c +8 -16
@@ -881,23 +881,15 @@ void dlm_recover_rsbs(struct dlm_ls *ls, const struct list_head *root_list)void dlm_clear_toss (struct dlm_ls *ls) { -struct rb_node *n, *next; -struct dlm_rsb *r; +struct dlm_rsb *r, *safe;unsigned int count =0 ; -int i; - -spin_lock (&ls->ls_rsbtbl_lock); -for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -for (n =rb_first (&ls->ls_rsbtbl[i].r); n; n = next) { - next =rb_next (n); - r = rb_entry(n, struct dlm_rsb, res_hashnode); -if (!rsb_flag (r,RSB_TOSS )) -continue ; - -rb_erase (n, &ls->ls_rsbtbl[i].r); -dlm_free_rsb (r); - count++; - } + +spin_lock_bh (&ls->ls_rsbtbl_lock); +list_for_each_entry_safe (r, safe, &ls->ls_toss, res_rsbs_list) { +list_del (&r->res_rsbs_list); +rb_erase (&r->res_hashnode, &ls->ls_rsbtbl[r->res_bucket].r); +dlm_free_rsb (r); + count++; }spin_unlock_bh (&ls->ls_rsbtbl_lock);
fs/dlm/recoverd.c +12 -22
@@ -22,9 +22,8 @@static int dlm_create_masters_list (struct dlm_ls *ls) { -struct rb_node *n;struct dlm_rsb *r; -int i, error =0 ; +int error =0 ;write_lock_bh (&ls->ls_masters_lock);if (!list_empty (&ls->ls_masters_list)) {
@@ -34,15 +33,12 @@ static int dlm_create_masters_list(struct dlm_ls *ls) }spin_lock_bh (&ls->ls_rsbtbl_lock); -for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -for (n =rb_first (&ls->ls_rsbtbl[i].r); n; n =rb_next (n)) { - r = rb_entry(n, struct dlm_rsb, res_hashnode); -if (rsb_flag (r,RSB_TOSS ) || r->res_nodeid) -continue ; - -list_add (&r->res_masters_list, &ls->ls_masters_list); -dlm_hold_rsb (r); - } +list_for_each_entry (r, &ls->ls_keep, res_rsbs_list) { +if (r->res_nodeid) +continue ; + +list_add (&r->res_masters_list, &ls->ls_masters_list); +dlm_hold_rsb (r); }spin_unlock_bh (&ls->ls_rsbtbl_lock); out:
@@ -64,21 +60,15 @@ static void dlm_release_masters_list(struct dlm_ls *ls)static void dlm_create_root_list (struct dlm_ls *ls,struct list_head *root_list) { -struct rb_node *n;struct dlm_rsb *r; -int i;spin_lock_bh (&ls->ls_rsbtbl_lock); -for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -for (n =rb_first (&ls->ls_rsbtbl[i].r); n; n =rb_next (n)) { - r = rb_entry(n, struct dlm_rsb, res_hashnode); -if (WARN_ON_ONCE (rsb_flag (r,RSB_TOSS ))) -continue ; - -list_add (&r->res_root_list, root_list); -dlm_hold_rsb (r); - } +list_for_each_entry (r, &ls->ls_keep, res_rsbs_list) { +list_add (&r->res_root_list, root_list); +dlm_hold_rsb (r); } + +WARN_ON_ONCE (!list_empty (&ls->ls_toss));spin_unlock_bh (&ls->ls_rsbtbl_lock); }
Signature · 1 line
2.43.0
Alexander Aring
maintainerRFC 5/9 dlm: switch to use rhashtable for rsbs
In reply to Alexander AringThis patch switches to use the rhashtable implementation to lookup rsbs instead doing an own bucket hash implementation. The rhashtable implementation deals internally to handle the hashbuckets depending on the amount of the entries it stores. Currently ls_rsbtbl_size to set the bucket size is only a guess, in reality we don't know how the user is using DLM. We move this decision to the rhashtable handling that does this dynamically during runtime.
--- fs/dlm/config.c | 8 ++ fs/dlm/config.h | 2 + fs/dlm/dir.c | 6 +- fs/dlm/dlm_internal.h | 18 ++--- fs/dlm/lock.c | 172 +++++++++++++----------------------------- fs/dlm/lock.h | 2 +- fs/dlm/lockspace.c | 35 ++++----- fs/dlm/recover.c | 3 +- 8 files changed, 86 insertions(+), 160 deletions(-)
fs/dlm/config.c +8 -0
@@ -63,6 +63,14 @@ static void release_node(struct config_item *);static struct configfs_attribute *comm_attrs[];static struct configfs_attribute *node_attrs[]; +const struct rhashtable_params dlm_rhash_rsb_params = { + .nelem_hint =3 ,/* start small */ + .key_len =DLM_RESNAME_MAXLEN , + .key_offset = offsetof(struct dlm_rsb , res_name), + .head_offset = offsetof(struct dlm_rsb , res_node), + .automatic_shrinking = true, +}; +struct dlm_cluster {struct config_group group;unsigned int cl_tcp_port;
fs/dlm/config.h +2 -0
@@ -21,6 +21,8 @@ struct dlm_config_node {uint32_t comm_seq; }; +extern const struct rhashtable_params dlm_rhash_rsb_params; +#define DLM_MAX_ADDR_COUNT 3#define DLM_PROTO_TCP 0
fs/dlm/dir.c +1 -5
@@ -198,14 +198,10 @@ static struct dlm_rsb *find_rsb_root(struct dlm_ls *ls, const char *name, int len) {struct dlm_rsb *r; -uint32_t hash, bucket;int rv; - hash =jhash (name, len,0 ); - bucket = hash & (ls->ls_rsbtbl_size -1 ); -spin_lock_bh (&ls->ls_rsbtbl_lock); - rv =dlm_search_rsb_tree (&ls->ls_rsbtbl[bucket].r, name, len, &r); + rv =dlm_search_rsb_tree (&ls->ls_rsbtbl, name, len, &r);spin_unlock_bh (&ls->ls_rsbtbl_lock);if (!rv)return r;
fs/dlm/dlm_internal.h +5 -13
@@ -34,6 +34,7 @@#include <linux/kernel.h> #include <linux/jhash.h> #include <linux/miscdevice.h> +#include <linux/rhashtable.h> #include <linux/mutex.h> #include <linux/idr.h> #include <linux/ratelimit.h>
@@ -99,15 +100,6 @@ do { \ } \ } - -#define DLM_RTF_SHRINK_BIT 0 - -struct dlm_rsbtable { -struct rb_root r; -unsigned long flags; -}; - -/* * Lockspace member (per node in a ls) */
@@ -327,13 +319,12 @@ struct dlm_rsb {int res_id;/* for ls_recover_idr */ uint32_t res_lvbseq;uint32_t res_hash; -uint32_t res_bucket;/* rsbtbl */ unsigned long res_toss_time;uint32_t res_first_lkid;struct list_head res_lookup;/* lkbs waiting on first */ union {struct list_head res_hashchain; -struct rb_node res_hashnode;/* rsbtbl */ +struct rhash_head res_node;/* rsbtbl */ };struct list_head res_grantqueue;struct list_head res_convertqueue;
@@ -592,9 +583,10 @@ struct dlm_ls {struct idr ls_lkbidr;spinlock_t ls_lkbidr_spin; -struct dlm_rsbtable *ls_rsbtbl; +struct rhashtable ls_rsbtbl; +#define DLM_RTF_SHRINK_BIT 0 +unsigned long ls_rsbtbl_flags;spinlock_t ls_rsbtbl_lock; -uint32_t ls_rsbtbl_size;struct list_head ls_toss;struct list_head ls_keep;
fs/dlm/lock.c +54 -118
373 lines not on this page · raw message
fs/dlm/lock.h +1 -1
@@ -29,7 +29,7 @@ void dlm_unlock_recovery(struct dlm_ls *ls);int dlm_master_lookup (struct dlm_ls *ls,int from_nodeid,const char *name,int len,unsigned int flags,int *r_nodeid,int *result); -int dlm_search_rsb_tree (struct rb_root *tree,const void *name,int len, +int dlm_search_rsb_tree (struct rhashtable *rhash,const void *name,int len,struct dlm_rsb **r_ret);void dlm_recover_purge (struct dlm_ls *ls,const struct list_head *root_list);
fs/dlm/lockspace.c +13 -22
@@ -410,9 +410,9 @@ static int new_lockspace(const char *name, const char *cluster, int *ops_result, dlm_lockspace_t **lockspace) { struct dlm_ls *ls; -int i, size, error;int do_unreg =0 ;int namelen =strlen (name); +int i, error;if (namelen >DLM_LOCKSPACE_LEN || namelen ==0 )return -EINVAL ;
@@ -498,15 +498,10 @@ static int new_lockspace(const char *name, const char *cluster,INIT_LIST_HEAD (&ls->ls_toss);INIT_LIST_HEAD (&ls->ls_keep);spin_lock_init (&ls->ls_rsbtbl_lock); - size =READ_ONCE (dlm_config.ci_rsbtbl_size); - ls->ls_rsbtbl_size = size; - ls->ls_rsbtbl =vmalloc (array_size (size,sizeof (struct dlm_rsbtable ))); -if (!ls->ls_rsbtbl) + error =rhashtable_init (&ls->ls_rsbtbl, &dlm_rhash_rsb_params); +if (error) goto out_lsfree; -for (i =0 ; i < size; i++) { - ls->ls_rsbtbl[i].r.rb_node =NULL ; - }for (i =0 ; i <DLM_REMOVE_NAMES_MAX ; i++) { ls->ls_remove_names[i] = kzalloc(DLM_RESNAME_MAXLEN+1,
@@ -669,7 +664,7 @@ static int new_lockspace(const char *name, const char *cluster, out_rsbtbl:for (i =0 ; i <DLM_REMOVE_NAMES_MAX ; i++)kfree (ls->ls_remove_names[i]); -vfree (ls->ls_rsbtbl); +rhashtable_destroy (&ls->ls_rsbtbl); out_lsfree:if (do_unreg)kobject_put (&ls->ls_kobj);
@@ -772,10 +767,16 @@ static int lockspace_busy(struct dlm_ls *ls, int force)return rv; } +static void rhash_free_rsb (void *ptr,void *arg) +{ +struct dlm_rsb *rsb = ptr; + +dlm_free_rsb (rsb); +} +static int release_lockspace (struct dlm_ls *ls,int force) {struct dlm_rsb *rsb; -struct rb_node *n;int i, busy, rv; busy =lockspace_busy (ls, force);
@@ -834,19 +835,9 @@ static int release_lockspace(struct dlm_ls *ls, int force)idr_destroy (&ls->ls_lkbidr);/* -* Free all rsb's on rsbtbl[] lists +* Free all rsb's on rsbtbl */ - -for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -while ((n =rb_first (&ls->ls_rsbtbl[i].r))) { - rsb = rb_entry(n, struct dlm_rsb, res_hashnode); -list_del (&rsb->res_rsbs_list); -rb_erase (n, &ls->ls_rsbtbl[i].r); -dlm_free_rsb (rsb); - } - } - -vfree (ls->ls_rsbtbl); +rhashtable_free_and_destroy (&ls->ls_rsbtbl, rhash_free_rsb,NULL );for (i =0 ; i <DLM_REMOVE_NAMES_MAX ; i++)kfree (ls->ls_remove_names[i]);
fs/dlm/recover.c +2 -1
@@ -887,7 +887,8 @@ void dlm_clear_toss(struct dlm_ls *ls)spin_lock_bh (&ls->ls_rsbtbl_lock);list_for_each_entry_safe (r, safe, &ls->ls_toss, res_rsbs_list) {list_del (&r->res_rsbs_list); -rb_erase (&r->res_hashnode, &ls->ls_rsbtbl[r->res_bucket].r); +rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, + dlm_rhash_rsb_params);dlm_free_rsb (r); count++; }
Signature · 1 line
2.43.0
Alexander Aring
maintainerRFC 6/9 dlm: remove refcounting if rsb is on toss
In reply to Alexander AringIn the past we had problems when rsb have a reference counter greater than one when they are in the toss state. A rsb in the toss state is not "alive" anymore and should not have any other references holding than just the last one keeping it on the rsb hash. It is a delayed free as kind of garbage collection with an additional handling to move the rsb into the keep state thats alive again.
The reference counter should only be used when the rsb is not in toss state (keep state), if the rsb isn't used anymore in keep state it will be moved to toss state where it can be freed on the right time. The release() function of the keep rsb will therefore set the RSB_TOSS state and does not kref_init() at this state as toss rsbs should never deal with any reference counting. If the rsb got alive again then the RSB_TOSS flag will be cleared and the kref_init() will be called to use reference counters when the rsb is in keep state.
There are warnings introduced now that will be triggered if we ever use any reference counting when a rsb is in toss state. All before kref_put() in toss state was only a sanity check if it ever wasn't 1. This will be removed and we directly free the rsb when it should be freed in the toss state. If there are still references around it is a bug.
This WARN_ON_ONCE() should hopefully find maybe more cases where we hold references of a toss rsb which we should not have as receive_remove() in the most cases requires to have the rsb in toss state. Otherwise we can run into problems when e.g. the directory rsb wasn't removed and the directory node tells the master node in case of a lookup message.
--- fs/dlm/lock.c | 61 ++++++++++++++++++++++++------------------------ fs/dlm/lock.h | 1 + fs/dlm/recover.c | 2 +- 3 files changed, 32 insertions(+), 32 deletions(-)
fs/dlm/lock.c +30 -31
@@ -325,6 +325,8 @@ static void queue_bast(struct dlm_rsb *r, struct dlm_lkb *lkb, int rqmode)static inline void hold_rsb (struct dlm_rsb *r) { +/* rsbs in toss state never get referenced */ +WARN_ON (rsb_flag (r,RSB_TOSS ));kref_get (&r->res_ref); }
@@ -631,6 +633,11 @@ static int find_rsb_dir(struct dlm_ls *ls, const void *name, int len,list_move (&r->res_rsbs_list, &ls->ls_keep);rsb_clear_flag (r,RSB_TOSS ); +/* rsb got out of toss state, it becomes alive again +* and we reinit the reference counter that is only +* valid for keep state rsbs +*/ +kref_init (&r->res_ref); goto out_unlock;
@@ -765,6 +772,11 @@ static int find_rsb_nodir(struct dlm_ls *ls, const void *name, int len,list_move (&r->res_rsbs_list, &ls->ls_keep);rsb_clear_flag (r,RSB_TOSS ); +/* rsb got out of toss state, it becomes alive again +* and we reinit the reference counter that is only +* valid for keep state rsbs +*/ +kref_init (&r->res_ref); goto out_unlock;
@@ -1112,8 +1124,6 @@ static void toss_rsb(struct kref *kref)struct dlm_ls *ls = r->res_ls; DLM_ASSERT(list_empty(&r->res_root_list), dlm_print_rsb(r);); -kref_init (&r->res_ref); -WARN_ON (rsb_flag (r,RSB_TOSS ));rsb_set_flag (r,RSB_TOSS );list_move (&r->res_rsbs_list, &ls->ls_toss); r->res_toss_time = jiffies;
@@ -1129,16 +1139,20 @@ static void toss_rsb(struct kref *kref)static void unhold_rsb (struct dlm_rsb *r) {int rv; + +/* rsbs in toss state never get referenced */ +WARN_ON (rsb_flag (r,RSB_TOSS )); rv =kref_put (&r->res_ref, toss_rsb); DLM_ASSERT(!rv, dlm_dump_rsb(r);); } -static void kill_rsb (struct kref *kref) +void free_toss_rsb (struct dlm_rsb *r) { - struct dlm_rsb *r = container_of(kref, struct dlm_rsb, res_ref); +WARN_ON_ONCE (!rsb_flag (r,RSB_TOSS )); -/* All work is done after the return from kref_put() so we -can release the write_lock before the remove and free. */ +/* check if all work is done after the rsb is on toss list +* and it can be freed. +*/ DLM_ASSERT(list_empty(&r->res_lookup), dlm_dump_rsb(r);); DLM_ASSERT(list_empty(&r->res_grantqueue), dlm_dump_rsb(r););
@@ -1147,6 +1161,8 @@ static void kill_rsb(struct kref *kref) DLM_ASSERT(list_empty(&r->res_root_list), dlm_dump_rsb(r);); DLM_ASSERT(list_empty(&r->res_recover_list), dlm_dump_rsb(r);); DLM_ASSERT(list_empty(&r->res_masters_list), dlm_dump_rsb(r);); + +dlm_free_rsb (r); }/* Attaching/detaching lkb's from rsb's is for rsb reference counting.
@@ -1611,15 +1627,10 @@ static void shrink_bucket(struct dlm_ls *ls)continue ; } -if (!kref_put (&r->res_ref, kill_rsb)) { -log_error (ls,"tossed rsb in use %s" , r->res_name); -continue ; - } -list_del (&r->res_rsbs_list);rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, dlm_rhash_rsb_params); -dlm_free_rsb (r); +free_toss_rsb (r); }if (need_shrink)
@@ -1680,19 +1691,13 @@ static void shrink_bucket(struct dlm_ls *ls)continue ; } -if (!kref_put (&r->res_ref, kill_rsb)) { -spin_unlock_bh (&ls->ls_rsbtbl_lock); -log_error (ls,"remove_name in use %s" , name); -continue ; - } -list_del (&r->res_rsbs_list);rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, dlm_rhash_rsb_params);send_remove (r);spin_unlock_bh (&ls->ls_rsbtbl_lock); -dlm_free_rsb (r); +free_toss_rsb (r); } }
@@ -4180,18 +4185,12 @@ static void receive_remove(struct dlm_ls *ls, const struct dlm_message *ms)return ; } -if (kref_put (&r->res_ref, kill_rsb)) { -list_del (&r->res_rsbs_list); -rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, - dlm_rhash_rsb_params); -spin_unlock_bh (&ls->ls_rsbtbl_lock); -dlm_free_rsb (r); - }else { -log_error (ls,"receive_remove from %d rsb ref error" , - from_nodeid); -dlm_print_rsb (r); -spin_unlock_bh (&ls->ls_rsbtbl_lock); - } +list_del (&r->res_rsbs_list); +rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, + dlm_rhash_rsb_params); +spin_unlock_bh (&ls->ls_rsbtbl_lock); + +free_toss_rsb (r); }static void receive_purge (struct dlm_ls *ls,const struct dlm_message *ms)
fs/dlm/lock.h +1 -0
@@ -18,6 +18,7 @@ void dlm_receive_message_saved(struct dlm_ls *ls, const struct dlm_message *ms, uint32_t saved_seq);void dlm_receive_buffer (const union dlm_packet *p,int nodeid);int dlm_modes_compat (int mode1,int mode2); +void free_toss_rsb (struct dlm_rsb *r);void dlm_put_rsb (struct dlm_rsb *r);void dlm_hold_rsb (struct dlm_rsb *r);int dlm_put_lkb (struct dlm_lkb *lkb);
fs/dlm/recover.c +1 -1
@@ -889,7 +889,7 @@ void dlm_clear_toss(struct dlm_ls *ls)list_del (&r->res_rsbs_list);rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, dlm_rhash_rsb_params); -dlm_free_rsb (r); +free_toss_rsb (r); count++; }spin_unlock_bh (&ls->ls_rsbtbl_lock);
Signature · 1 line
2.43.0
Alexander Aring
maintainerRFC 7/9 dlm: drop scand kthread and use timers
In reply to Alexander AringCurrently the scand kthread acts like a garbage collection for expired rsbs on toss list to clean them up after a certain timeout. It triggers every couple of seconds and iterates over the toss list while holding ls_rsbtbl_lock for the whole hash bucket iteration. To reduce the amount of holding the ls_rsbtbl_lock time we handle free of cached rsbs (tossed rsbs) on a per rsb timer. If it expires the rsb deletes itseld out of rsb data structures. Addtional the timer can run on any cpu instead of scand that was only running on one cpu, so there should be a performance speedup handling such freeing of resources.
--- fs/dlm/dlm_internal.h | 15 ++- fs/dlm/lock.c | 272 +++++++++++++++++++++--------------------- fs/dlm/lockspace.c | 105 +++------------- fs/dlm/recover.c | 24 +++- 4 files changed, 178 insertions(+), 238 deletions(-)
fs/dlm/dlm_internal.h +10 -5
@@ -320,6 +320,7 @@ struct dlm_rsb {uint32_t res_lvbseq;uint32_t res_hash;unsigned long res_toss_time; +struct timer_list res_toss_timer;uint32_t res_first_lkid;struct list_head res_lookup;/* lkbs waiting on first */ union {
@@ -368,6 +369,7 @@ enum rsb_flags {RSB_RECOVER_GRANT ,RSB_RECOVER_LVB_INVAL ,RSB_TOSS , + RSB_TIMER_KILLED, };static inline void rsb_set_flag (struct dlm_rsb *r,enum rsb_flags flag)
@@ -584,8 +586,6 @@ struct dlm_ls {spinlock_t ls_lkbidr_spin;struct rhashtable ls_rsbtbl; -#define DLM_RTF_SHRINK_BIT 0 -unsigned long ls_rsbtbl_flags;spinlock_t ls_rsbtbl_lock;struct list_head ls_toss;
@@ -601,9 +601,6 @@ struct dlm_ls {int ls_new_rsb_count;struct list_head ls_new_rsb;/* new rsb structs */ -char *ls_remove_names[DLM_REMOVE_NAMES_MAX ]; -int ls_remove_lens[DLM_REMOVE_NAMES_MAX ]; -struct list_head ls_nodes;/* current nodes in ls */ struct list_head ls_nodes_gone;/* dead node list, recovery */ int ls_num_nodes;/* number of nodes in ls */
@@ -698,6 +695,13 @@ struct dlm_ls {* dlm_ls_stop() clears this to tell dlm locking routines that they should * quit what they are doing so recovery can run. dlm_recoverd sets * this after recovery is finished. +* +* LSFL_TIMER_KILLED - a per lockspace switch that will kill all timers in +* sense of they don't do anything anymore. To set this flag the caller +* need to held the ls_in_recovery write lock. It's useful to just don't iterate +* over all timers and call timer_shutdown_sync() to stop every res_toss_timer +* that is currently ongoing and queued e.g. when release_lockspace() is called +* and we need to force to stop everything. */ #define LSFL_RECOVER_STOP 0
@@ -712,6 +716,7 @@ struct dlm_ls {#define LSFL_CB_DELAY 9#define LSFL_NODIR 10#define LSFL_RECV_MSG_BLOCKED 11 +#define LSFL_TIMER_KILLED 12#define DLM_PROC_FLAGS_CLOSING 1#define DLM_PROC_FLAGS_COMPAT 2
fs/dlm/lock.c +134 -138
338 lines not on this page · raw message
fs/dlm/lockspace.c +14 -91
174 lines not on this page · raw message
fs/dlm/recover.c +20 -4
@@ -881,18 +881,34 @@ void dlm_recover_rsbs(struct dlm_ls *ls, const struct list_head *root_list)void dlm_clear_toss (struct dlm_ls *ls) { -struct dlm_rsb *r, *safe;unsigned int count =0 ; +struct dlm_rsb *r; + +while (1 ) { +spin_lock_bh (&ls->ls_rsbtbl_lock); + r = list_first_entry_or_null(&ls->ls_toss, struct dlm_rsb, + res_rsbs_list); +if (!r) { +spin_unlock_bh (&ls->ls_rsbtbl_lock); +break ; + } -spin_lock_bh (&ls->ls_rsbtbl_lock); -list_for_each_entry_safe (r, safe, &ls->ls_toss, res_rsbs_list) {list_del (&r->res_rsbs_list);rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, dlm_rhash_rsb_params); +rsb_set_flag (r,RSB_TIMER_KILLED ); +spin_unlock_bh (&ls->ls_rsbtbl_lock); + +/* Wait until all killed currently running timer +* callbacks are done. Killed as RSB_TIMER_KILLED +* is set and the timer callback does nothing +* anymore and dlm_clear_toss() takes over. +*/ +timer_shutdown_sync (&r->res_toss_timer); +free_toss_rsb (r); count++; } -spin_unlock_bh (&ls->ls_rsbtbl_lock);if (count)log_rinfo (ls,"dlm_clear_toss %u done" , count);
Signature · 1 line
2.43.0
Alexander Aring
maintainerRe: [RFC dlm/next 7/9] dlm: drop scand kthread and use timers
Hi,
Quoted 11 lines · Alexander Aring
Currently the scand kthread acts like a garbage collection for expired rsbs on toss list to clean them up after a certain timeout. It triggers every couple of seconds and iterates over the toss list while holding
ls_rsbtbl_lockfor the whole hash bucket iteration. To reduce the amount of holding thels_rsbtbl_locktime we handle free of cached rsbs (tossed rsbs) on a per rsb timer. If it expires the rsb deletes itseld out of rsb data structures. Addtional the timer can run on any cpu instead of scand that was only running on one cpu, so there should be a performance speedup handling such freeing of resources.
I currently try to work on a per lockspace timer that gets rearmed when the next rsb is expired instead of spamming the timer subsystems with a lot of timers.
That has also the advantage to simply shutdown the per ls timer, instead iterating over all timers which is done by some flags in this version.
- Alex
Alexander Aring
maintainerRFC 8/9 dlm: likely read lock path for rsb lookup
In reply to Alexander AringAs the conversion to rhashtable introduced a hash lock per lockspace instead of doing a per bucket hash, this patch will change the hash lock handling as holding only a read lock in a very likely hot path in DLM.
This hot path is to lookup rsbs and they are in keep state. Keep state means that the RSB_TOSS flag isn't set. If this likely read lock path is the case we don't need to hold the ls_rsbtbl_lock in write lock at all. If it's on toss state holding the ls_rsbtbl_lock in write lock is required. If we see that the rsb is in toss state, we hold the ls_rsbtbl_lock in write state and relookup (because the rsb could get removed into the transition from read to write state) and check if the rsb is still in toss state. However this is an unlikely path as toss rsbs are cached rsbs. Having the rsb in keep state is very likely.
--- fs/dlm/debug_fs.c | 4 +- fs/dlm/dir.c | 4 +- fs/dlm/dlm_internal.h | 2 +- fs/dlm/lock.c | 260 ++++++++++++++++++++++++++++++------------ fs/dlm/lockspace.c | 2 +- fs/dlm/recover.c | 6 +- fs/dlm/recoverd.c | 8 +- 7 files changed, 202 insertions(+), 84 deletions(-)
fs/dlm/debug_fs.c +2 -2
@@ -413,7 +413,7 @@ static void *table_seq_start(struct seq_file *seq, loff_t *pos)else list = &ls->ls_keep; -spin_lock_bh (&ls->ls_rsbtbl_lock); +read_lock_bh (&ls->ls_rsbtbl_lock);return seq_list_start (list, *pos); }
@@ -434,7 +434,7 @@ static void table_seq_stop(struct seq_file *seq, void *iter_ptr) {struct dlm_ls *ls = seq->private; -spin_unlock_bh (&ls->ls_rsbtbl_lock); +read_unlock_bh (&ls->ls_rsbtbl_lock); }static const struct seq_operations format1_seq_ops = {
fs/dlm/dir.c +2 -2
@@ -200,9 +200,9 @@ static struct dlm_rsb *find_rsb_root(struct dlm_ls *ls, const char *name,struct dlm_rsb *r;int rv; -spin_lock_bh (&ls->ls_rsbtbl_lock); +read_lock_bh (&ls->ls_rsbtbl_lock); rv =dlm_search_rsb_tree (&ls->ls_rsbtbl, name, len, &r); -spin_unlock_bh (&ls->ls_rsbtbl_lock); +read_unlock_bh (&ls->ls_rsbtbl_lock);if (!rv)return r;
fs/dlm/dlm_internal.h +1 -1
@@ -586,7 +586,7 @@ struct dlm_ls {spinlock_t ls_lkbidr_spin;struct rhashtable ls_rsbtbl; -spinlock_t ls_rsbtbl_lock; +rwlock_t ls_rsbtbl_lock;struct list_head ls_toss;struct list_head ls_keep;
fs/dlm/lock.c +189 -71
550 lines not on this page · raw message
fs/dlm/lockspace.c +1 -1
@@ -424,7 +424,7 @@ static int new_lockspace(const char *name, const char *cluster,INIT_LIST_HEAD (&ls->ls_toss);INIT_LIST_HEAD (&ls->ls_keep); -spin_lock_init (&ls->ls_rsbtbl_lock); +rwlock_init (&ls->ls_rsbtbl_lock); error =rhashtable_init (&ls->ls_rsbtbl, &dlm_rhash_rsb_params);if (error)
fs/dlm/recover.c +3 -3
@@ -885,11 +885,11 @@ void dlm_clear_toss(struct dlm_ls *ls)struct dlm_rsb *r;while (1 ) { -spin_lock_bh (&ls->ls_rsbtbl_lock); +write_lock_bh (&ls->ls_rsbtbl_lock); r = list_first_entry_or_null(&ls->ls_toss, struct dlm_rsb, res_rsbs_list);if (!r) { -spin_unlock_bh (&ls->ls_rsbtbl_lock); +write_unlock_bh (&ls->ls_rsbtbl_lock);break ; }
@@ -897,7 +897,7 @@ void dlm_clear_toss(struct dlm_ls *ls)rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, dlm_rhash_rsb_params);rsb_set_flag (r,RSB_TIMER_KILLED ); -spin_unlock_bh (&ls->ls_rsbtbl_lock); +write_unlock_bh (&ls->ls_rsbtbl_lock);/* Wait until all killed currently running timer * callbacks are done. Killed as RSB_TIMER_KILLED
fs/dlm/recoverd.c +4 -4
@@ -32,7 +32,7 @@ static int dlm_create_masters_list(struct dlm_ls *ls) goto out; } -spin_lock_bh (&ls->ls_rsbtbl_lock); +read_lock_bh (&ls->ls_rsbtbl_lock);list_for_each_entry (r, &ls->ls_keep, res_rsbs_list) {if (r->res_nodeid)continue ;
@@ -40,7 +40,7 @@ static int dlm_create_masters_list(struct dlm_ls *ls)list_add (&r->res_masters_list, &ls->ls_masters_list);dlm_hold_rsb (r); } -spin_unlock_bh (&ls->ls_rsbtbl_lock); +read_unlock_bh (&ls->ls_rsbtbl_lock); out:write_unlock_bh (&ls->ls_masters_lock);return error;
@@ -62,14 +62,14 @@ static void dlm_create_root_list(struct dlm_ls *ls, struct list_head *root_list) {struct dlm_rsb *r; -spin_lock_bh (&ls->ls_rsbtbl_lock); +read_lock_bh (&ls->ls_rsbtbl_lock);list_for_each_entry (r, &ls->ls_keep, res_rsbs_list) {list_add (&r->res_root_list, root_list);dlm_hold_rsb (r); }WARN_ON_ONCE (!list_empty (&ls->ls_toss)); -spin_unlock_bh (&ls->ls_rsbtbl_lock); +read_unlock_bh (&ls->ls_rsbtbl_lock); }static void dlm_release_root_list (struct list_head *root_list)
Signature · 1 line
2.43.0
Alexander Aring
maintainerRe: [RFC dlm/next 8/9] dlm: likely read lock path for rsb lookup
Hi,
Quoted 15 lines · Alexander Aring
As the conversion to rhashtable introduced a hash lock per lockspace instead of doing a per bucket hash, this patch will change the hash lock handling as holding only a read lock in a very likely hot path in DLM.
This hot path is to lookup rsbs and they are in keep state. Keep state means that the
RSB_TOSSflag isn't set. If this likely read lock path is the case we don't need to hold thels_rsbtbl_lockin write lock at all. If it's on toss state holding thels_rsbtbl_lockin write lock is required. If we see that the rsb is in toss state, we hold thels_rsbtbl_lockin write state and relookup (because the rsb could get removed into the transition from read to write state) and check if the rsb is still in toss state. However this is an unlikely path as toss rsbs are cached rsbs. Having the rsb in keep state is very likely.
...
fs/dlm/lock.c static int find_rsb_nodir(struct dlm_ls *ls, const void *name, int len, · line 893
Quoted 11 lines
fs/dlm/lock.c +3
-1@@ -858,7 +885,9 @@ static int find_rsb_nodir(struct dlm_ls *ls, const void *name, int len,*/ kref_get (&r->res_ref); - goto out_unlock; +read_unlock_bh (&ls->ls_rsbtbl_lock); + + goto out; do_toss:
we need to start to get the write lock here and then do the relookup and test if the state has been changed (again do_toss is an unlikely path as the first lookup that can be done under read lock only). The reason is that there can be a remove and add again and we check on some fields in the toss path that can be different then, the whole do_toss path needs to be under write lock.
It makes more sense that the first lookup is in toss state and drops the read lock, in do_toss we take the write lock and check if the lookup and the states are still the same under this condition, if so we go ahead with the toss lookup path under the write lock.
Same for the other lookup functions that look for the rsb being in toss state and bring it to the keep state (unlikely case), because it is mostly on the keep list that can be done under read lock.
- Alex
Alexander Aring
maintainerRFC 9/9 dlm: convert lkbidr to rwlock
In reply to Alexander AringThis patch converts lkbidr lock to rwlock. The most time we doing lookups of idr only this can be done by holding the read lock only.
--- fs/dlm/dlm_internal.h | 2 +- fs/dlm/lock.c | 44 +++++++------------------------------------ fs/dlm/lockspace.c | 6 +++--- 3 files changed, 11 insertions(+), 41 deletions(-)
fs/dlm/dlm_internal.h +1 -1
@@ -583,7 +583,7 @@ struct dlm_ls {struct kobject ls_kobj;struct idr ls_lkbidr; -spinlock_t ls_lkbidr_spin; +rwlock_t ls_lkbidr_lock;struct rhashtable ls_rsbtbl;rwlock_t ls_rsbtbl_lock;
fs/dlm/lock.c +7 -37
76 lines not on this page · raw message
fs/dlm/lockspace.c +3 -3
@@ -431,7 +431,7 @@ static int new_lockspace(const char *name, const char *cluster, goto out_lsfree;idr_init (&ls->ls_lkbidr); -spin_lock_init (&ls->ls_lkbidr_spin); +rwlock_init (&ls->ls_lkbidr_lock);INIT_LIST_HEAD (&ls->ls_waiters);spin_lock_init (&ls->ls_waiters_lock);
@@ -671,7 +671,7 @@ static int lockspace_busy(struct dlm_ls *ls, int force) {int rv; -spin_lock_bh (&ls->ls_lkbidr_spin); +read_lock_bh (&ls->ls_lkbidr_lock);if (force ==0 ) { rv =idr_for_each (&ls->ls_lkbidr, lkb_idr_is_any, ls); }else if (force ==1 ) {
@@ -679,7 +679,7 @@ static int lockspace_busy(struct dlm_ls *ls, int force) }else { rv =0 ; } -spin_unlock_bh (&ls->ls_lkbidr_spin); +read_unlock_bh (&ls->ls_lkbidr_lock);return rv; }
Signature · 1 line
2.43.0