dlm: sand fix, rhashtable, timers and lookup hotpath speedup
Alexander Aring
maintainerv2 cover dlm: sand fix, rhashtable, timers and lookup hotpath speedup
Hi,
this is a patch series for an immense change in DLM rsb hashtable logic. It removes the double lookup functionality for rsb hashtables, convert to rhashtable instead of own bucket hlist hash implementation.
At first there is a fix for scand that I detected while I was implementing this patch series. It could be that remove messages are still send when the lockspace is releasing the resource that could occur into a use after free.
There is a conversion to use lists (keep/toss lists) instead of iterating over the hash bucket. As we do a transition to rhashtable, they don't like to being iterated regarding to their own bucket sizing implementation that is sitting in the rhashtable implementation. We just introduce the lists to do the iteration, as advantage we have a huge reduce of code in the debugfs dump functionality as we use the dump list helpers of debugfs. There is also a potential refcount bug when holding rsb references of rsbs in toss state as receive remove message requires no rsb references being hold. Another issue is also holding rsb in keep state as they are not going into toss state when they required to.
It is now forbidden to hold references of rsbs in toss state. The refcounter must be only functional in rsb keep state. That hopefully will show is more invalid usage of the rsb refcounter if the rsb is in toss state.
The scand was being fixed but now also it's removed. The scand process was holding the hashtable/hash bucket lock for a longer timer because it iterated over the whole hash. We use timers now to reduce the held time of the hashtable lock. We introduce a per lockspace toss queue with tossed timer rsb and the first item is the earliest rsb that will be expired by the timer vice versa the last item. This makes it easy to change the timer expiration to the next one in the queue.
The last two patches we move very likely lookup hotpath to read lock mostly. This should for sure avoid contention in the most cases. Unlikely path need to still hold the write lock and do some extra relookup and check if the state of an rsb changed. However I think we hit over 90% the likely path that we only need to hold the read lock and avoid contention between processing DLM messages and the user triggers new DLM requests.
- Alex
changes since v2:
- introduce dlm_timer_resume() and call it in between LSFL_RUNNING and ls_in_recovery lock. Comment this function and some rare cases that it is only a "try" resume.
- comment more why holding ls_rsbtbl_lock lock in timer and correct the retry comment in case of timer hit contention. It is only a "try" as well, as others can set new timer expirations.
changes since RFC:
- hold the write_lock in find_rsb_dir/nodir when hitting the do_toss path and then do the lookup and check on the do_toss rsb fields
- move from a per rsb timer to a per lockspace timer and introduce a per rsb toss queue.
Alexander Aring (9):
dlm: increment ls_count on find_ls_to_scan()
dlm: change to non per bucket hashtable lock
dlm: merge toss and keep hash into one
dlm: fix avoid rsb hold during debugfs dump
dlm: switch to use rhashtable for rsbs
dlm: remove refcounting if rsb is on toss
dlm: drop scand kthread and use timers
dlm: likely read lock path for rsb lookup
dlm: convert lkbidr to rwlock
fs/dlm/config.c | 8 + fs/dlm/config.h | 2 + fs/dlm/debug_fs.c | 212 ++--------- fs/dlm/dir.c | 14 +- fs/dlm/dlm_internal.h | 40 +- fs/dlm/lock.c | 867 ++++++++++++++++++++++++------------------ fs/dlm/lock.h | 5 +- fs/dlm/lockspace.c | 151 ++------ fs/dlm/member.c | 2 + fs/dlm/recover.c | 29 +- fs/dlm/recoverd.c | 50 +-- 11 files changed, 643 insertions(+), 737 deletions(-)
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH v2 1/9 dlm: increment ls_count on find_ls_to_scan()
This patch increments the ls_count atomic value that is used as a reference counter in find_ls_to_scan(). An additional dlm_put_lockspace() is necessary in dlm_scand() to drop this reference again.
This handling is necessary because release_lockspace() calls remove_lockspace() that removes the lockspace from the global lockspace list. It only does that when the lockspace is not in use anymore signaled by the menitoned ls_count atomic value. Currently dlm_scand() can be executed when release_lockspace() is freeing all lockspace resources. As example:
1. scand timeout occurs
2. find_ls_to_scan() returns ls A - lslist_lock is not held anymore
3. dlm_scan_rsbs(A) does its job
4. release_lockspace(A, 2) is called by the user. User has no control of dlm_scan_rsbs(A)
5. lockspace_busy(A, 2) returns 0, it's not busy
6. dlm_scan_rsbs(A) iterates over the whole rsb hash bucket, sometimes triggers send_remove()
7. release_lockspace(A, 2) does not stop scand because ls_count was >= 2
8. remove_lockspace() will ignore ongoing scand because scand does not hold a reference as ls_count is intended to do so
9. release_lockspace(A, 2) cleans up everything while 6. can occur
Point 8 is a problem because both access the same resources and release_lockspace() frees them.
--- fs/dlm/lockspace.c | 3 +++ 1 file changed, 3 insertions(+)
fs/dlm/lockspace.c +3 -0
@@ -255,6 +255,7 @@ static struct dlm_ls *find_ls_to_scan(void)list_for_each_entry (ls, &lslist, ls_list) {if (time_after_eq (jiffies, ls->ls_scan_time + dlm_config.ci_scan_secs *HZ )) { +atomic_inc (&ls->ls_count);spin_unlock_bh (&lslist_lock);return ls; }
@@ -277,6 +278,8 @@ static int dlm_scand(void *data) }else { ls->ls_scan_time +=HZ ; } + +dlm_put_lockspace (ls);continue ; }schedule_timeout_interruptible (dlm_config.ci_scan_secs *HZ );
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH v2 2/9 dlm: change to non per bucket hashtable lock
In reply to Alexander AringThis patch prepares to switch to use the resizable, scalable, concurrent hash table (rhashtable) implementation instead of having a own per bucket implementation. The rhashtable implementation deals with the buckets internally for that reason we use a per lockspace rhashtable lock to protect mutual rhashtable access. This might reduce the perfromance of the hash, depends on how rhashtable lookup performs to the current own DLM hash bucket implementation. However we switch to the rhashtable implementation and a potential lockless access can be part of future changes.
--- fs/dlm/debug_fs.c | 24 +++++++------- fs/dlm/dir.c | 4 +-- fs/dlm/dlm_internal.h | 2 +- fs/dlm/lock.c | 77 +++++++++++++++++++++---------------------- fs/dlm/lockspace.c | 2 +- fs/dlm/recover.c | 4 +-- fs/dlm/recoverd.c | 8 ++--- 7 files changed, 60 insertions(+), 61 deletions(-)
fs/dlm/debug_fs.c +12 -12
@@ -452,7 +452,7 @@ static void *table_seq_start(struct seq_file *seq, loff_t *pos) tree = toss ? &ls->ls_rsbtbl[bucket].toss : &ls->ls_rsbtbl[bucket].keep; -spin_lock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_lock_bh (&ls->ls_rsbtbl_lock);if (!RB_EMPTY_ROOT (tree)) {for (node =rb_first (tree); node; node =rb_next (node)) { r = rb_entry(node, struct dlm_rsb, res_hashnode);
@@ -460,12 +460,12 @@ static void *table_seq_start(struct seq_file *seq, loff_t *pos)dlm_hold_rsb (r); ri->rsb = r; ri->bucket = bucket; -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock);return ri; } } } -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock);/* * move to the first rsb in the next non-empty bucket
@@ -484,18 +484,18 @@ static void *table_seq_start(struct seq_file *seq, loff_t *pos) } tree = toss ? &ls->ls_rsbtbl[bucket].toss : &ls->ls_rsbtbl[bucket].keep; -spin_lock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_lock_bh (&ls->ls_rsbtbl_lock);if (!RB_EMPTY_ROOT (tree)) { node =rb_first (tree); r = rb_entry(node, struct dlm_rsb, res_hashnode);dlm_hold_rsb (r); ri->rsb = r; ri->bucket = bucket; -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock); *pos = n;return ri; } -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock); } }
@@ -516,7 +516,7 @@ static void *table_seq_next(struct seq_file *seq, void *iter_ptr, loff_t *pos)* move to the next rsb in the same bucket */ -spin_lock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_lock_bh (&ls->ls_rsbtbl_lock); rp = ri->rsb; next =rb_next (&rp->res_hashnode);
@@ -524,12 +524,12 @@ static void *table_seq_next(struct seq_file *seq, void *iter_ptr, loff_t *pos) r = rb_entry(next, struct dlm_rsb, res_hashnode);dlm_hold_rsb (r); ri->rsb = r; -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock);dlm_put_rsb (rp); ++*pos;return ri; } -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock);dlm_put_rsb (rp);/*
@@ -550,18 +550,18 @@ static void *table_seq_next(struct seq_file *seq, void *iter_ptr, loff_t *pos) } tree = toss ? &ls->ls_rsbtbl[bucket].toss : &ls->ls_rsbtbl[bucket].keep; -spin_lock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_lock_bh (&ls->ls_rsbtbl_lock);if (!RB_EMPTY_ROOT (tree)) { next =rb_first (tree); r = rb_entry(next, struct dlm_rsb, res_hashnode);dlm_hold_rsb (r); ri->rsb = r; ri->bucket = bucket; -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock); *pos = n;return ri; } -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock); } }
fs/dlm/dir.c +2 -2
@@ -204,12 +204,12 @@ static struct dlm_rsb *find_rsb_root(struct dlm_ls *ls, const char *name, hash =jhash (name, len,0 ); bucket = hash & (ls->ls_rsbtbl_size -1 ); -spin_lock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_lock_bh (&ls->ls_rsbtbl_lock); rv =dlm_search_rsb_tree (&ls->ls_rsbtbl[bucket].keep, name, len, &r);if (rv) rv =dlm_search_rsb_tree (&ls->ls_rsbtbl[bucket].toss, name, len, &r); -spin_unlock_bh (&ls->ls_rsbtbl[bucket].lock); +spin_unlock_bh (&ls->ls_rsbtbl_lock);if (!rv)return r;
fs/dlm/dlm_internal.h +1 -1
@@ -105,7 +105,6 @@ do { \struct dlm_rsbtable {struct rb_root keep;struct rb_root toss; -spinlock_t lock;unsigned long flags; };
@@ -593,6 +592,7 @@ struct dlm_ls {spinlock_t ls_lkbidr_spin;struct dlm_rsbtable *ls_rsbtbl; +spinlock_t ls_rsbtbl_lock;uint32_t ls_rsbtbl_size;spinlock_t ls_waiters_lock;
fs/dlm/lock.c +38 -39
289 lines not on this page · raw message
fs/dlm/lockspace.c +1 -1
@@ -495,6 +495,7 @@ static int new_lockspace(const char *name, const char *cluster,*/ ls->ls_exflags = (flags & ~(DLM_LSFL_FS |DLM_LSFL_NEWEXCL )); +spin_lock_init (&ls->ls_rsbtbl_lock); size =READ_ONCE (dlm_config.ci_rsbtbl_size); ls->ls_rsbtbl_size = size;
@@ -504,7 +505,6 @@ static int new_lockspace(const char *name, const char *cluster,for (i =0 ; i < size; i++) { ls->ls_rsbtbl[i].keep.rb_node =NULL ; ls->ls_rsbtbl[i].toss.rb_node =NULL ; -spin_lock_init (&ls->ls_rsbtbl[i].lock); }for (i =0 ; i <DLM_REMOVE_NAMES_MAX ; i++) {
fs/dlm/recover.c +2 -2
@@ -886,8 +886,8 @@ void dlm_clear_toss(struct dlm_ls *ls)unsigned int count =0 ;int i; +spin_lock (&ls->ls_rsbtbl_lock);for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -spin_lock_bh (&ls->ls_rsbtbl[i].lock);for (n =rb_first (&ls->ls_rsbtbl[i].toss); n; n = next) { next =rb_next (n); r = rb_entry(n, struct dlm_rsb, res_hashnode);
@@ -895,8 +895,8 @@ void dlm_clear_toss(struct dlm_ls *ls)dlm_free_rsb (r); count++; } -spin_unlock_bh (&ls->ls_rsbtbl[i].lock); } +spin_unlock_bh (&ls->ls_rsbtbl_lock);if (count)log_rinfo (ls,"dlm_clear_toss %u done" , count);
fs/dlm/recoverd.c +4 -4
@@ -33,8 +33,8 @@ static int dlm_create_masters_list(struct dlm_ls *ls) goto out; } +spin_lock_bh (&ls->ls_rsbtbl_lock);for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -spin_lock_bh (&ls->ls_rsbtbl[i].lock);for (n =rb_first (&ls->ls_rsbtbl[i].keep); n; n =rb_next (n)) { r = rb_entry(n, struct dlm_rsb, res_hashnode);if (r->res_nodeid)
@@ -43,8 +43,8 @@ static int dlm_create_masters_list(struct dlm_ls *ls)list_add (&r->res_masters_list, &ls->ls_masters_list);dlm_hold_rsb (r); } -spin_unlock_bh (&ls->ls_rsbtbl[i].lock); } +spin_unlock_bh (&ls->ls_rsbtbl_lock); out:write_unlock_bh (&ls->ls_masters_lock);return error;
@@ -68,8 +68,8 @@ static void dlm_create_root_list(struct dlm_ls *ls, struct list_head *root_list)struct dlm_rsb *r;int i; +spin_lock_bh (&ls->ls_rsbtbl_lock);for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -spin_lock_bh (&ls->ls_rsbtbl[i].lock);for (n =rb_first (&ls->ls_rsbtbl[i].keep); n; n =rb_next (n)) { r = rb_entry(n, struct dlm_rsb, res_hashnode);list_add (&r->res_root_list, root_list);
@@ -78,8 +78,8 @@ static void dlm_create_root_list(struct dlm_ls *ls, struct list_head *root_list)if (!RB_EMPTY_ROOT (&ls->ls_rsbtbl[i].toss))log_error (ls,"%s toss not empty" , __func__); -spin_unlock_bh (&ls->ls_rsbtbl[i].lock); } +spin_unlock_bh (&ls->ls_rsbtbl_lock); }static void dlm_release_root_list (struct list_head *root_list)
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH v2 3/9 dlm: merge toss and keep hash into one
In reply to Alexander AringThere are several places in current DLM that can end in two lookups, e.g. first look into the keep hash, then into the toss hash if its not in the keep hash. However there can't be never the same rsb into both of these hashs. This patch introduce a new rsb state flag "RSB_TOSS" that signals if the rsb would be on the toss hash or not, which then would be on the keep hash. Instead of those multiple lookups there is only one now that reduce the amount of hash lookups in cases where the rsb wasn't on keep hash. The move of an rsb from rsb toss to keep hash or vice versa need to be done automatically anyway.
--- fs/dlm/debug_fs.c | 29 ++++++++++-- fs/dlm/dir.c | 6 +-- fs/dlm/dlm_internal.h | 4 +- fs/dlm/lock.c | 103 ++++++++++++++++++++++-------------------- fs/dlm/lockspace.c | 13 ++---- fs/dlm/recover.c | 7 ++- fs/dlm/recoverd.c | 12 ++--- 7 files changed, 98 insertions(+), 76 deletions(-)
fs/dlm/debug_fs.c +26 -3
@@ -450,12 +450,20 @@ static void *table_seq_start(struct seq_file *seq, loff_t *pos)if (seq->op == &format4_seq_ops) ri->format =4 ; - tree = toss ? &ls->ls_rsbtbl[bucket].toss : &ls->ls_rsbtbl[bucket].keep; + tree = &ls->ls_rsbtbl[bucket].r;spin_lock_bh (&ls->ls_rsbtbl_lock);if (!RB_EMPTY_ROOT (tree)) {for (node =rb_first (tree); node; node =rb_next (node)) { r = rb_entry(node, struct dlm_rsb, res_hashnode); +if (toss) { +if (!rsb_flag (r,RSB_TOSS )) +continue ; + }else { +if (rsb_flag (r,RSB_TOSS )) +continue ; + } +if (!entry--) {dlm_hold_rsb (r); ri->rsb = r;
@@ -482,12 +490,20 @@ static void *table_seq_start(struct seq_file *seq, loff_t *pos)kfree (ri);return NULL ; } - tree = toss ? &ls->ls_rsbtbl[bucket].toss : &ls->ls_rsbtbl[bucket].keep; + tree = &ls->ls_rsbtbl[bucket].r;spin_lock_bh (&ls->ls_rsbtbl_lock);if (!RB_EMPTY_ROOT (tree)) { node =rb_first (tree); r = rb_entry(node, struct dlm_rsb, res_hashnode); +if (toss) { +if (!rsb_flag (r,RSB_TOSS )) +continue ; + }else { +if (rsb_flag (r,RSB_TOSS )) +continue ; + } +dlm_hold_rsb (r); ri->rsb = r; ri->bucket = bucket;
@@ -548,12 +564,19 @@ static void *table_seq_next(struct seq_file *seq, void *iter_ptr, loff_t *pos) ++*pos;return NULL ; } - tree = toss ? &ls->ls_rsbtbl[bucket].toss : &ls->ls_rsbtbl[bucket].keep; + tree = &ls->ls_rsbtbl[bucket].r;spin_lock_bh (&ls->ls_rsbtbl_lock);if (!RB_EMPTY_ROOT (tree)) { next =rb_first (tree); r = rb_entry(next, struct dlm_rsb, res_hashnode); +if (toss) { +if (!rsb_flag (r,RSB_TOSS )) +continue ; + }else { +if (rsb_flag (r,RSB_TOSS )) +continue ; + }dlm_hold_rsb (r); ri->rsb = r; ri->bucket = bucket;
fs/dlm/dir.c +1 -5
@@ -205,12 +205,8 @@ static struct dlm_rsb *find_rsb_root(struct dlm_ls *ls, const char *name, bucket = hash & (ls->ls_rsbtbl_size -1 );spin_lock_bh (&ls->ls_rsbtbl_lock); - rv =dlm_search_rsb_tree (&ls->ls_rsbtbl[bucket].keep, name, len, &r); -if (rv) - rv =dlm_search_rsb_tree (&ls->ls_rsbtbl[bucket].toss, - name, len, &r); + rv =dlm_search_rsb_tree (&ls->ls_rsbtbl[bucket].r, name, len, &r);spin_unlock_bh (&ls->ls_rsbtbl_lock); -if (!rv)return r;
fs/dlm/dlm_internal.h +2 -2
@@ -103,8 +103,7 @@ do { \#define DLM_RTF_SHRINK_BIT 0struct dlm_rsbtable { -struct rb_root keep; -struct rb_root toss; +struct rb_root r;unsigned long flags; };
@@ -376,6 +375,7 @@ enum rsb_flags {RSB_RECOVER_CONVERT ,RSB_RECOVER_GRANT ,RSB_RECOVER_LVB_INVAL , + RSB_TOSS, };static inline void rsb_set_flag (struct dlm_rsb *r,enum rsb_flags flag)
fs/dlm/lock.c +55 -48
247 lines not on this page · raw message
fs/dlm/lockspace.c +3 -10
@@ -503,8 +503,7 @@ static int new_lockspace(const char *name, const char *cluster,if (!ls->ls_rsbtbl) goto out_lsfree;for (i =0 ; i < size; i++) { - ls->ls_rsbtbl[i].keep.rb_node =NULL ; - ls->ls_rsbtbl[i].toss.rb_node =NULL ; + ls->ls_rsbtbl[i].r.rb_node =NULL ; }for (i =0 ; i <DLM_REMOVE_NAMES_MAX ; i++) {
@@ -837,15 +836,9 @@ static int release_lockspace(struct dlm_ls *ls, int force)*/ for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -while ((n =rb_first (&ls->ls_rsbtbl[i].keep))) { +while ((n =rb_first (&ls->ls_rsbtbl[i].r))) { rsb = rb_entry(n, struct dlm_rsb, res_hashnode); -rb_erase (n, &ls->ls_rsbtbl[i].keep); -dlm_free_rsb (rsb); - } - -while ((n =rb_first (&ls->ls_rsbtbl[i].toss))) { - rsb = rb_entry(n, struct dlm_rsb, res_hashnode); -rb_erase (n, &ls->ls_rsbtbl[i].toss); +rb_erase (n, &ls->ls_rsbtbl[i].r);dlm_free_rsb (rsb); } }
fs/dlm/recover.c +5 -2
@@ -888,10 +888,13 @@ void dlm_clear_toss(struct dlm_ls *ls)spin_lock (&ls->ls_rsbtbl_lock);for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -for (n =rb_first (&ls->ls_rsbtbl[i].toss); n; n = next) { +for (n =rb_first (&ls->ls_rsbtbl[i].r); n; n = next) { next =rb_next (n); r = rb_entry(n, struct dlm_rsb, res_hashnode); -rb_erase (n, &ls->ls_rsbtbl[i].toss); +if (!rsb_flag (r,RSB_TOSS )) +continue ; + +rb_erase (n, &ls->ls_rsbtbl[i].r);dlm_free_rsb (r); count++; }
fs/dlm/recoverd.c +6 -6
@@ -35,9 +35,9 @@ static int dlm_create_masters_list(struct dlm_ls *ls)spin_lock_bh (&ls->ls_rsbtbl_lock);for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -for (n =rb_first (&ls->ls_rsbtbl[i].keep); n; n =rb_next (n)) { +for (n =rb_first (&ls->ls_rsbtbl[i].r); n; n =rb_next (n)) { r = rb_entry(n, struct dlm_rsb, res_hashnode); -if (r->res_nodeid) +if (rsb_flag (r,RSB_TOSS ) || r->res_nodeid)continue ;list_add (&r->res_masters_list, &ls->ls_masters_list);
@@ -70,14 +70,14 @@ static void dlm_create_root_list(struct dlm_ls *ls, struct list_head *root_list)spin_lock_bh (&ls->ls_rsbtbl_lock);for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -for (n =rb_first (&ls->ls_rsbtbl[i].keep); n; n =rb_next (n)) { +for (n =rb_first (&ls->ls_rsbtbl[i].r); n; n =rb_next (n)) { r = rb_entry(n, struct dlm_rsb, res_hashnode); +if (WARN_ON_ONCE (rsb_flag (r,RSB_TOSS ))) +continue ; +list_add (&r->res_root_list, root_list);dlm_hold_rsb (r); } - -if (!RB_EMPTY_ROOT (&ls->ls_rsbtbl[i].toss)) -log_error (ls,"%s toss not empty" , __func__); }spin_unlock_bh (&ls->ls_rsbtbl_lock); }
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH v2 4/9 dlm: fix avoid rsb hold during debugfs dump
In reply to Alexander AringCurrently the debugfs calls dlm_hold_rsb() while iterating the ls_rsbtbl hash and release the ls_rsbtbl_lock during dumping. We can't do that in every situation because some rsbs don't get into the toss state when they should be on the toss state e.g. when receive_remove() is being called. To avoid this problem we don't use the hash as iteratable datastructure. Moving to rhashtable implementation it should be avoided to use the rhashtable to iterate over it due dynamic rehash background operations that results into a whole new iteration of the rhashtable datastructure. Instead of iterating over the hash we introduce two lists ls_toss and ls_keep as DLM cares about the RSB_TOSS state if iterate over all rsbs. On ls_toss are only rsbs that are on RSB_TOSS state and on ls_keep the rsbs they are not in RSB_TOSS state. There are only few situation to bring a rsb into toss state and back which will be just a list_move() between those datastructures protected by the ls_rsbtbl_lock to keep in sync with the hashtable and RSB_TOSS flag.
The advantage in debugfs is that we can simple use the predefined debugfs dump functionality for linux lists. Instead of calling dlm_hold_rsb() while dumping rsbs, we hold the ls_rsbtbl_lock the whole debugfs dump.
--- fs/dlm/debug_fs.c | 233 +++++++----------------------------------- fs/dlm/dlm_internal.h | 4 + fs/dlm/lock.c | 47 ++++----- fs/dlm/lockspace.c | 3 + fs/dlm/recover.c | 24 ++--- fs/dlm/recoverd.c | 34 +++--- 6 files changed, 84 insertions(+), 261 deletions(-)
fs/dlm/debug_fs.c +36 -197
@@ -366,12 +366,10 @@ static void print_format4(struct dlm_rsb *r, struct seq_file *s)unlock_rsb (r); } -struct rsbtbl_iter { -struct dlm_rsb *rsb; -unsigned bucket; -int format; -int header; -}; +static const struct seq_operations format1_seq_ops; +static const struct seq_operations format2_seq_ops; +static const struct seq_operations format3_seq_ops; +static const struct seq_operations format4_seq_ops;/* * If the buffer is full, seq_printf can be called again, but it
@@ -382,220 +380,61 @@ struct rsbtbl_iter {static int table_seq_show (struct seq_file *seq,void *iter_ptr) { -struct rsbtbl_iter *ri = iter_ptr; - -switch (ri->format) { -case 1 : -print_format1 (ri->rsb, seq); -break ; -case 2 : -if (ri->header) { -seq_puts (seq,"id nodeid remid pid xid exflags flags sts grmode rqmode time_ms r_nodeid r_len r_name\n" ); - ri->header =0 ; - } -print_format2 (ri->rsb, seq); -break ; -case 3 : -if (ri->header) { -seq_puts (seq,"rsb ptr nodeid first_lkid flags !root_list_empty !recover_list_empty recover_locks_count len\n" ); - ri->header =0 ; - } -print_format3 (ri->rsb, seq); -break ; -case 4 : -if (ri->header) { -seq_puts (seq,"rsb ptr nodeid master_nodeid dir_nodeid our_nodeid toss_time flags len str|hex name\n" ); - ri->header =0 ; - } -print_format4 (ri->rsb, seq); -break ; - } + struct dlm_rsb *rsb = list_entry(iter_ptr, struct dlm_rsb, res_rsbs_list); + +if (seq->op == &format1_seq_ops) +print_format1 (rsb, seq); +else if (seq->op == &format2_seq_ops) +print_format2 (rsb, seq); +else if (seq->op == &format3_seq_ops) +print_format3 (rsb, seq); +else if (seq->op == &format4_seq_ops) +print_format4 (rsb, seq);return 0 ; } -static const struct seq_operations format1_seq_ops; -static const struct seq_operations format2_seq_ops; -static const struct seq_operations format3_seq_ops; -static const struct seq_operations format4_seq_ops; -static void *table_seq_start (struct seq_file *seq,loff_t *pos) { -struct rb_root *tree; -struct rb_node *node;struct dlm_ls *ls = seq->private; -struct rsbtbl_iter *ri; -struct dlm_rsb *r; -loff_t n = *pos; -unsigned bucket, entry; -int toss = (seq->op == &format4_seq_ops); - - bucket = n >>32 ; - entry = n & ((1LL <<32 ) -1 ); - -if (bucket >= ls->ls_rsbtbl_size) -return NULL ; - - ri =kzalloc (sizeof (*ri),GFP_NOFS ); -if (!ri) -return NULL ; -if (n ==0 ) - ri->header =1 ; -if (seq->op == &format1_seq_ops) - ri->format =1 ; -if (seq->op == &format2_seq_ops) - ri->format =2 ; -if (seq->op == &format3_seq_ops) - ri->format =3 ; -if (seq->op == &format4_seq_ops) - ri->format =4 ; - - tree = &ls->ls_rsbtbl[bucket].r; +struct list_head *list; -spin_lock_bh (&ls->ls_rsbtbl_lock); -if (!RB_EMPTY_ROOT (tree)) { -for (node =rb_first (tree); node; node =rb_next (node)) { - r = rb_entry(node, struct dlm_rsb, res_hashnode); -if (toss) { -if (!rsb_flag (r,RSB_TOSS )) -continue ; - }else { -if (rsb_flag (r,RSB_TOSS )) -continue ; - } - -if (!entry--) { -dlm_hold_rsb (r); - ri->rsb = r; - ri->bucket = bucket; -spin_unlock_bh (&ls->ls_rsbtbl_lock); -return ri; - } - } +if (!*pos) { +if (seq->op == &format2_seq_ops) +seq_puts (seq,"id nodeid remid pid xid exflags flags sts grmode rqmode time_ms r_nodeid r_len r_name\n" ); +else if (seq->op == &format3_seq_ops) +seq_puts (seq,"rsb ptr nodeid first_lkid flags !root_list_empty !recover_list_empty recover_locks_count len\n" ); +else if (seq->op == &format4_seq_ops) +seq_puts (seq,"rsb ptr nodeid master_nodeid dir_nodeid our_nodeid toss_time flags len str|hex name\n" ); } -spin_unlock_bh (&ls->ls_rsbtbl_lock); - -/* -* move to the first rsb in the next non-empty bucket -*/ - -/* zero the entry */ - n &= ~((1LL <<32 ) -1 ); -while (1 ) { - bucket++; - n +=1LL <<32 ; +if (seq->op == &format4_seq_ops) + list = &ls->ls_toss; +else + list = &ls->ls_keep; -if (bucket >= ls->ls_rsbtbl_size) { -kfree (ri); -return NULL ; - } - tree = &ls->ls_rsbtbl[bucket].r; - -spin_lock_bh (&ls->ls_rsbtbl_lock); -if (!RB_EMPTY_ROOT (tree)) { - node =rb_first (tree); - r = rb_entry(node, struct dlm_rsb, res_hashnode); -if (toss) { -if (!rsb_flag (r,RSB_TOSS )) -continue ; - }else { -if (rsb_flag (r,RSB_TOSS )) -continue ; - } - -dlm_hold_rsb (r); - ri->rsb = r; - ri->bucket = bucket; -spin_unlock_bh (&ls->ls_rsbtbl_lock); - *pos = n; -return ri; - } -spin_unlock_bh (&ls->ls_rsbtbl_lock); - } +spin_lock_bh (&ls->ls_rsbtbl_lock); +return seq_list_start (list, *pos); }static void *table_seq_next (struct seq_file *seq,void *iter_ptr,loff_t *pos) {struct dlm_ls *ls = seq->private; -struct rsbtbl_iter *ri = iter_ptr; -struct rb_root *tree; -struct rb_node *next; -struct dlm_rsb *r, *rp; -loff_t n = *pos; -unsigned bucket; -int toss = (seq->op == &format4_seq_ops); - - bucket = n >>32 ; - -/* -* move to the next rsb in the same bucket -*/ - -spin_lock_bh (&ls->ls_rsbtbl_lock); - rp = ri->rsb; - next =rb_next (&rp->res_hashnode); - -if (next) { - r = rb_entry(next, struct dlm_rsb, res_hashnode); -dlm_hold_rsb (r); - ri->rsb = r; -spin_unlock_bh (&ls->ls_rsbtbl_lock); -dlm_put_rsb (rp); - ++*pos; -return ri; - } -spin_unlock_bh (&ls->ls_rsbtbl_lock); -dlm_put_rsb (rp); - -/* -* move to the first rsb in the next non-empty bucket -*/ +struct list_head *list; -/* zero the entry */ - n &= ~((1LL <<32 ) -1 ); - -while (1 ) { - bucket++; - n +=1LL <<32 ; +if (seq->op == &format4_seq_ops) + list = &ls->ls_toss; +else + list = &ls->ls_keep; -if (bucket >= ls->ls_rsbtbl_size) { -kfree (ri); - ++*pos; -return NULL ; - } - tree = &ls->ls_rsbtbl[bucket].r; - -spin_lock_bh (&ls->ls_rsbtbl_lock); -if (!RB_EMPTY_ROOT (tree)) { - next =rb_first (tree); - r = rb_entry(next, struct dlm_rsb, res_hashnode); -if (toss) { -if (!rsb_flag (r,RSB_TOSS )) -continue ; - }else { -if (rsb_flag (r,RSB_TOSS )) -continue ; - } -dlm_hold_rsb (r); - ri->rsb = r; - ri->bucket = bucket; -spin_unlock_bh (&ls->ls_rsbtbl_lock); - *pos = n; -return ri; - } -spin_unlock_bh (&ls->ls_rsbtbl_lock); - } +return seq_list_next (iter_ptr, list, pos); }static void table_seq_stop (struct seq_file *seq,void *iter_ptr) { -struct rsbtbl_iter *ri = iter_ptr; +struct dlm_ls *ls = seq->private; -if (ri) { -dlm_put_rsb (ri->rsb); -kfree (ri); - } +spin_unlock_bh (&ls->ls_rsbtbl_lock); }static const struct seq_operations format1_seq_ops = {
fs/dlm/dlm_internal.h +4 -0
@@ -339,6 +339,7 @@ struct dlm_rsb {struct list_head res_convertqueue;struct list_head res_waitqueue; +struct list_head res_rsbs_list;struct list_head res_root_list;/* used for recovery */ struct list_head res_masters_list;/* used for recovery */ struct list_head res_recover_list;/* used for recovery */
@@ -595,6 +596,9 @@ struct dlm_ls {spinlock_t ls_rsbtbl_lock;uint32_t ls_rsbtbl_size; +struct list_head ls_toss; +struct list_head ls_keep; +spinlock_t ls_waiters_lock;struct list_head ls_waiters;/* lkbs needing a reply */
fs/dlm/lock.c +21 -26
131 lines not on this page · raw message
fs/dlm/lockspace.c +3 -0
@@ -495,6 +495,8 @@ static int new_lockspace(const char *name, const char *cluster,*/ ls->ls_exflags = (flags & ~(DLM_LSFL_FS |DLM_LSFL_NEWEXCL )); +INIT_LIST_HEAD (&ls->ls_toss); +INIT_LIST_HEAD (&ls->ls_keep);spin_lock_init (&ls->ls_rsbtbl_lock); size =READ_ONCE (dlm_config.ci_rsbtbl_size); ls->ls_rsbtbl_size = size;
@@ -838,6 +840,7 @@ static int release_lockspace(struct dlm_ls *ls, int force)for (i =0 ; i < ls->ls_rsbtbl_size; i++) {while ((n =rb_first (&ls->ls_rsbtbl[i].r))) { rsb = rb_entry(n, struct dlm_rsb, res_hashnode); +list_del (&rsb->res_rsbs_list);rb_erase (n, &ls->ls_rsbtbl[i].r);dlm_free_rsb (rsb); }
fs/dlm/recover.c +8 -16
@@ -881,23 +881,15 @@ void dlm_recover_rsbs(struct dlm_ls *ls, const struct list_head *root_list)void dlm_clear_toss (struct dlm_ls *ls) { -struct rb_node *n, *next; -struct dlm_rsb *r; +struct dlm_rsb *r, *safe;unsigned int count =0 ; -int i; - -spin_lock (&ls->ls_rsbtbl_lock); -for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -for (n =rb_first (&ls->ls_rsbtbl[i].r); n; n = next) { - next =rb_next (n); - r = rb_entry(n, struct dlm_rsb, res_hashnode); -if (!rsb_flag (r,RSB_TOSS )) -continue ; - -rb_erase (n, &ls->ls_rsbtbl[i].r); -dlm_free_rsb (r); - count++; - } + +spin_lock_bh (&ls->ls_rsbtbl_lock); +list_for_each_entry_safe (r, safe, &ls->ls_toss, res_rsbs_list) { +list_del (&r->res_rsbs_list); +rb_erase (&r->res_hashnode, &ls->ls_rsbtbl[r->res_bucket].r); +dlm_free_rsb (r); + count++; }spin_unlock_bh (&ls->ls_rsbtbl_lock);
fs/dlm/recoverd.c +12 -22
@@ -22,9 +22,8 @@static int dlm_create_masters_list (struct dlm_ls *ls) { -struct rb_node *n;struct dlm_rsb *r; -int i, error =0 ; +int error =0 ;write_lock_bh (&ls->ls_masters_lock);if (!list_empty (&ls->ls_masters_list)) {
@@ -34,15 +33,12 @@ static int dlm_create_masters_list(struct dlm_ls *ls) }spin_lock_bh (&ls->ls_rsbtbl_lock); -for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -for (n =rb_first (&ls->ls_rsbtbl[i].r); n; n =rb_next (n)) { - r = rb_entry(n, struct dlm_rsb, res_hashnode); -if (rsb_flag (r,RSB_TOSS ) || r->res_nodeid) -continue ; - -list_add (&r->res_masters_list, &ls->ls_masters_list); -dlm_hold_rsb (r); - } +list_for_each_entry (r, &ls->ls_keep, res_rsbs_list) { +if (r->res_nodeid) +continue ; + +list_add (&r->res_masters_list, &ls->ls_masters_list); +dlm_hold_rsb (r); }spin_unlock_bh (&ls->ls_rsbtbl_lock); out:
@@ -64,21 +60,15 @@ static void dlm_release_masters_list(struct dlm_ls *ls)static void dlm_create_root_list (struct dlm_ls *ls,struct list_head *root_list) { -struct rb_node *n;struct dlm_rsb *r; -int i;spin_lock_bh (&ls->ls_rsbtbl_lock); -for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -for (n =rb_first (&ls->ls_rsbtbl[i].r); n; n =rb_next (n)) { - r = rb_entry(n, struct dlm_rsb, res_hashnode); -if (WARN_ON_ONCE (rsb_flag (r,RSB_TOSS ))) -continue ; - -list_add (&r->res_root_list, root_list); -dlm_hold_rsb (r); - } +list_for_each_entry (r, &ls->ls_keep, res_rsbs_list) { +list_add (&r->res_root_list, root_list); +dlm_hold_rsb (r); } + +WARN_ON_ONCE (!list_empty (&ls->ls_toss));spin_unlock_bh (&ls->ls_rsbtbl_lock); }
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH v2 5/9 dlm: switch to use rhashtable for rsbs
In reply to Alexander AringThis patch switches to use the rhashtable implementation to lookup rsbs instead doing an own bucket hash implementation. The rhashtable implementation deals internally to handle the hashbuckets depending on the amount of the entries it stores. Currently ls_rsbtbl_size to set the bucket size is only a guess, in reality we don't know how the user is using DLM. We move this decision to the rhashtable handling that does this dynamically during runtime.
--- fs/dlm/config.c | 8 ++ fs/dlm/config.h | 2 + fs/dlm/dir.c | 6 +- fs/dlm/dlm_internal.h | 18 ++--- fs/dlm/lock.c | 172 +++++++++++++----------------------------- fs/dlm/lock.h | 2 +- fs/dlm/lockspace.c | 35 ++++----- fs/dlm/recover.c | 3 +- 8 files changed, 86 insertions(+), 160 deletions(-)
fs/dlm/config.c +8 -0
@@ -63,6 +63,14 @@ static void release_node(struct config_item *);static struct configfs_attribute *comm_attrs[];static struct configfs_attribute *node_attrs[]; +const struct rhashtable_params dlm_rhash_rsb_params = { + .nelem_hint =3 ,/* start small */ + .key_len =DLM_RESNAME_MAXLEN , + .key_offset = offsetof(struct dlm_rsb , res_name), + .head_offset = offsetof(struct dlm_rsb , res_node), + .automatic_shrinking = true, +}; +struct dlm_cluster {struct config_group group;unsigned int cl_tcp_port;
fs/dlm/config.h +2 -0
@@ -21,6 +21,8 @@ struct dlm_config_node {uint32_t comm_seq; }; +extern const struct rhashtable_params dlm_rhash_rsb_params; +#define DLM_MAX_ADDR_COUNT 3#define DLM_PROTO_TCP 0
fs/dlm/dir.c +1 -5
@@ -198,14 +198,10 @@ static struct dlm_rsb *find_rsb_root(struct dlm_ls *ls, const char *name, int len) {struct dlm_rsb *r; -uint32_t hash, bucket;int rv; - hash =jhash (name, len,0 ); - bucket = hash & (ls->ls_rsbtbl_size -1 ); -spin_lock_bh (&ls->ls_rsbtbl_lock); - rv =dlm_search_rsb_tree (&ls->ls_rsbtbl[bucket].r, name, len, &r); + rv =dlm_search_rsb_tree (&ls->ls_rsbtbl, name, len, &r);spin_unlock_bh (&ls->ls_rsbtbl_lock);if (!rv)return r;
fs/dlm/dlm_internal.h +5 -13
@@ -34,6 +34,7 @@#include <linux/kernel.h> #include <linux/jhash.h> #include <linux/miscdevice.h> +#include <linux/rhashtable.h> #include <linux/mutex.h> #include <linux/idr.h> #include <linux/ratelimit.h>
@@ -99,15 +100,6 @@ do { \ } \ } - -#define DLM_RTF_SHRINK_BIT 0 - -struct dlm_rsbtable { -struct rb_root r; -unsigned long flags; -}; - -/* * Lockspace member (per node in a ls) */
@@ -327,13 +319,12 @@ struct dlm_rsb {int res_id;/* for ls_recover_idr */ uint32_t res_lvbseq;uint32_t res_hash; -uint32_t res_bucket;/* rsbtbl */ unsigned long res_toss_time;uint32_t res_first_lkid;struct list_head res_lookup;/* lkbs waiting on first */ union {struct list_head res_hashchain; -struct rb_node res_hashnode;/* rsbtbl */ +struct rhash_head res_node;/* rsbtbl */ };struct list_head res_grantqueue;struct list_head res_convertqueue;
@@ -592,9 +583,10 @@ struct dlm_ls {struct idr ls_lkbidr;spinlock_t ls_lkbidr_spin; -struct dlm_rsbtable *ls_rsbtbl; +struct rhashtable ls_rsbtbl; +#define DLM_RTF_SHRINK_BIT 0 +unsigned long ls_rsbtbl_flags;spinlock_t ls_rsbtbl_lock; -uint32_t ls_rsbtbl_size;struct list_head ls_toss;struct list_head ls_keep;
fs/dlm/lock.c +54 -118
373 lines not on this page · raw message
fs/dlm/lock.h +1 -1
@@ -29,7 +29,7 @@ void dlm_unlock_recovery(struct dlm_ls *ls);int dlm_master_lookup (struct dlm_ls *ls,int from_nodeid,const char *name,int len,unsigned int flags,int *r_nodeid,int *result); -int dlm_search_rsb_tree (struct rb_root *tree,const void *name,int len, +int dlm_search_rsb_tree (struct rhashtable *rhash,const void *name,int len,struct dlm_rsb **r_ret);void dlm_recover_purge (struct dlm_ls *ls,const struct list_head *root_list);
fs/dlm/lockspace.c +13 -22
@@ -410,9 +410,9 @@ static int new_lockspace(const char *name, const char *cluster, int *ops_result, dlm_lockspace_t **lockspace) { struct dlm_ls *ls; -int i, size, error;int do_unreg =0 ;int namelen =strlen (name); +int i, error;if (namelen >DLM_LOCKSPACE_LEN || namelen ==0 )return -EINVAL ;
@@ -498,15 +498,10 @@ static int new_lockspace(const char *name, const char *cluster,INIT_LIST_HEAD (&ls->ls_toss);INIT_LIST_HEAD (&ls->ls_keep);spin_lock_init (&ls->ls_rsbtbl_lock); - size =READ_ONCE (dlm_config.ci_rsbtbl_size); - ls->ls_rsbtbl_size = size; - ls->ls_rsbtbl =vmalloc (array_size (size,sizeof (struct dlm_rsbtable ))); -if (!ls->ls_rsbtbl) + error =rhashtable_init (&ls->ls_rsbtbl, &dlm_rhash_rsb_params); +if (error) goto out_lsfree; -for (i =0 ; i < size; i++) { - ls->ls_rsbtbl[i].r.rb_node =NULL ; - }for (i =0 ; i <DLM_REMOVE_NAMES_MAX ; i++) { ls->ls_remove_names[i] = kzalloc(DLM_RESNAME_MAXLEN+1,
@@ -669,7 +664,7 @@ static int new_lockspace(const char *name, const char *cluster, out_rsbtbl:for (i =0 ; i <DLM_REMOVE_NAMES_MAX ; i++)kfree (ls->ls_remove_names[i]); -vfree (ls->ls_rsbtbl); +rhashtable_destroy (&ls->ls_rsbtbl); out_lsfree:if (do_unreg)kobject_put (&ls->ls_kobj);
@@ -772,10 +767,16 @@ static int lockspace_busy(struct dlm_ls *ls, int force)return rv; } +static void rhash_free_rsb (void *ptr,void *arg) +{ +struct dlm_rsb *rsb = ptr; + +dlm_free_rsb (rsb); +} +static int release_lockspace (struct dlm_ls *ls,int force) {struct dlm_rsb *rsb; -struct rb_node *n;int i, busy, rv; busy =lockspace_busy (ls, force);
@@ -834,19 +835,9 @@ static int release_lockspace(struct dlm_ls *ls, int force)idr_destroy (&ls->ls_lkbidr);/* -* Free all rsb's on rsbtbl[] lists +* Free all rsb's on rsbtbl */ - -for (i =0 ; i < ls->ls_rsbtbl_size; i++) { -while ((n =rb_first (&ls->ls_rsbtbl[i].r))) { - rsb = rb_entry(n, struct dlm_rsb, res_hashnode); -list_del (&rsb->res_rsbs_list); -rb_erase (n, &ls->ls_rsbtbl[i].r); -dlm_free_rsb (rsb); - } - } - -vfree (ls->ls_rsbtbl); +rhashtable_free_and_destroy (&ls->ls_rsbtbl, rhash_free_rsb,NULL );for (i =0 ; i <DLM_REMOVE_NAMES_MAX ; i++)kfree (ls->ls_remove_names[i]);
fs/dlm/recover.c +2 -1
@@ -887,7 +887,8 @@ void dlm_clear_toss(struct dlm_ls *ls)spin_lock_bh (&ls->ls_rsbtbl_lock);list_for_each_entry_safe (r, safe, &ls->ls_toss, res_rsbs_list) {list_del (&r->res_rsbs_list); -rb_erase (&r->res_hashnode, &ls->ls_rsbtbl[r->res_bucket].r); +rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, + dlm_rhash_rsb_params);dlm_free_rsb (r); count++; }
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH v2 6/9 dlm: remove refcounting if rsb is on toss
In reply to Alexander AringIn the past we had problems when rsb have a reference counter greater than one when they are in the toss state. A rsb in the toss state is not "alive" anymore and should not have any other references holding than just the last one keeping it on the rsb hash. It is a delayed free as kind of garbage collection with an additional handling to move the rsb into the keep state thats alive again.
The reference counter should only be used when the rsb is not in toss state (keep state), if the rsb isn't used anymore in keep state it will be moved to toss state where it can be freed on the right time. The release() function of the keep rsb will therefore set the RSB_TOSS state and does not kref_init() at this state as toss rsbs should never deal with any reference counting. If the rsb got alive again then the RSB_TOSS flag will be cleared and the kref_init() will be called to use reference counters when the rsb is in keep state.
There are warnings introduced now that will be triggered if we ever use any reference counting when a rsb is in toss state. All before kref_put() in toss state was only a sanity check if it ever wasn't 1. This will be removed and we directly free the rsb when it should be freed in the toss state. If there are still references around it is a bug.
This WARN_ON_ONCE() should hopefully find maybe more cases where we hold references of a toss rsb which we should not have as receive_remove() in the most cases requires to have the rsb in toss state. Otherwise we can run into problems when e.g. the directory rsb wasn't removed and the directory node tells the master node in case of a lookup message.
--- fs/dlm/lock.c | 61 ++++++++++++++++++++++++------------------------ fs/dlm/lock.h | 1 + fs/dlm/recover.c | 2 +- 3 files changed, 32 insertions(+), 32 deletions(-)
fs/dlm/lock.c +30 -31
@@ -325,6 +325,8 @@ static void queue_bast(struct dlm_rsb *r, struct dlm_lkb *lkb, int rqmode)static inline void hold_rsb (struct dlm_rsb *r) { +/* rsbs in toss state never get referenced */ +WARN_ON (rsb_flag (r,RSB_TOSS ));kref_get (&r->res_ref); }
@@ -631,6 +633,11 @@ static int find_rsb_dir(struct dlm_ls *ls, const void *name, int len,list_move (&r->res_rsbs_list, &ls->ls_keep);rsb_clear_flag (r,RSB_TOSS ); +/* rsb got out of toss state, it becomes alive again +* and we reinit the reference counter that is only +* valid for keep state rsbs +*/ +kref_init (&r->res_ref); goto out_unlock;
@@ -765,6 +772,11 @@ static int find_rsb_nodir(struct dlm_ls *ls, const void *name, int len,list_move (&r->res_rsbs_list, &ls->ls_keep);rsb_clear_flag (r,RSB_TOSS ); +/* rsb got out of toss state, it becomes alive again +* and we reinit the reference counter that is only +* valid for keep state rsbs +*/ +kref_init (&r->res_ref); goto out_unlock;
@@ -1112,8 +1124,6 @@ static void toss_rsb(struct kref *kref)struct dlm_ls *ls = r->res_ls; DLM_ASSERT(list_empty(&r->res_root_list), dlm_print_rsb(r);); -kref_init (&r->res_ref); -WARN_ON (rsb_flag (r,RSB_TOSS ));rsb_set_flag (r,RSB_TOSS );list_move (&r->res_rsbs_list, &ls->ls_toss); r->res_toss_time = jiffies;
@@ -1129,16 +1139,20 @@ static void toss_rsb(struct kref *kref)static void unhold_rsb (struct dlm_rsb *r) {int rv; + +/* rsbs in toss state never get referenced */ +WARN_ON (rsb_flag (r,RSB_TOSS )); rv =kref_put (&r->res_ref, toss_rsb); DLM_ASSERT(!rv, dlm_dump_rsb(r);); } -static void kill_rsb (struct kref *kref) +void free_toss_rsb (struct dlm_rsb *r) { - struct dlm_rsb *r = container_of(kref, struct dlm_rsb, res_ref); +WARN_ON_ONCE (!rsb_flag (r,RSB_TOSS )); -/* All work is done after the return from kref_put() so we -can release the write_lock before the remove and free. */ +/* check if all work is done after the rsb is on toss list +* and it can be freed. +*/ DLM_ASSERT(list_empty(&r->res_lookup), dlm_dump_rsb(r);); DLM_ASSERT(list_empty(&r->res_grantqueue), dlm_dump_rsb(r););
@@ -1147,6 +1161,8 @@ static void kill_rsb(struct kref *kref) DLM_ASSERT(list_empty(&r->res_root_list), dlm_dump_rsb(r);); DLM_ASSERT(list_empty(&r->res_recover_list), dlm_dump_rsb(r);); DLM_ASSERT(list_empty(&r->res_masters_list), dlm_dump_rsb(r);); + +dlm_free_rsb (r); }/* Attaching/detaching lkb's from rsb's is for rsb reference counting.
@@ -1611,15 +1627,10 @@ static void shrink_bucket(struct dlm_ls *ls)continue ; } -if (!kref_put (&r->res_ref, kill_rsb)) { -log_error (ls,"tossed rsb in use %s" , r->res_name); -continue ; - } -list_del (&r->res_rsbs_list);rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, dlm_rhash_rsb_params); -dlm_free_rsb (r); +free_toss_rsb (r); }if (need_shrink)
@@ -1680,19 +1691,13 @@ static void shrink_bucket(struct dlm_ls *ls)continue ; } -if (!kref_put (&r->res_ref, kill_rsb)) { -spin_unlock_bh (&ls->ls_rsbtbl_lock); -log_error (ls,"remove_name in use %s" , name); -continue ; - } -list_del (&r->res_rsbs_list);rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, dlm_rhash_rsb_params);send_remove (r);spin_unlock_bh (&ls->ls_rsbtbl_lock); -dlm_free_rsb (r); +free_toss_rsb (r); } }
@@ -4180,18 +4185,12 @@ static void receive_remove(struct dlm_ls *ls, const struct dlm_message *ms)return ; } -if (kref_put (&r->res_ref, kill_rsb)) { -list_del (&r->res_rsbs_list); -rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, - dlm_rhash_rsb_params); -spin_unlock_bh (&ls->ls_rsbtbl_lock); -dlm_free_rsb (r); - }else { -log_error (ls,"receive_remove from %d rsb ref error" , - from_nodeid); -dlm_print_rsb (r); -spin_unlock_bh (&ls->ls_rsbtbl_lock); - } +list_del (&r->res_rsbs_list); +rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, + dlm_rhash_rsb_params); +spin_unlock_bh (&ls->ls_rsbtbl_lock); + +free_toss_rsb (r); }static void receive_purge (struct dlm_ls *ls,const struct dlm_message *ms)
fs/dlm/lock.h +1 -0
@@ -18,6 +18,7 @@ void dlm_receive_message_saved(struct dlm_ls *ls, const struct dlm_message *ms, uint32_t saved_seq);void dlm_receive_buffer (const union dlm_packet *p,int nodeid);int dlm_modes_compat (int mode1,int mode2); +void free_toss_rsb (struct dlm_rsb *r);void dlm_put_rsb (struct dlm_rsb *r);void dlm_hold_rsb (struct dlm_rsb *r);int dlm_put_lkb (struct dlm_lkb *lkb);
fs/dlm/recover.c +1 -1
@@ -889,7 +889,7 @@ void dlm_clear_toss(struct dlm_ls *ls)list_del (&r->res_rsbs_list);rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, dlm_rhash_rsb_params); -dlm_free_rsb (r); +free_toss_rsb (r); count++; }spin_unlock_bh (&ls->ls_rsbtbl_lock);
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH v2 7/9 dlm: drop scand kthread and use timers
In reply to Alexander AringCurrently the scand kthread acts like a garbage collection for expired rsbs on toss list to clean them up after a certain timeout. It triggers every couple of seconds and iterates over the toss list while holding ls_rsbtbl_lock for the whole hash bucket iteration. To reduce the amount of holding the ls_rsbtbl_lock time we handle free of cached rsbs (tossed rsbs) on a per rsb timer. If it expires the rsb deletes itself out of the rsb toss queue that is an ordered queue of tossed rsbs. First entry is the earliest rsb to expire the last is the lastest rsb that will expire. The res_toss_time is now an absolute value of when it will be expired before it was the time when the rsb was tossed. It was an debug functionality for debugfs anyway, we change a little bit the semantic here.
--- fs/dlm/dlm_internal.h | 16 +- fs/dlm/lock.c | 382 ++++++++++++++++++++++++++---------------- fs/dlm/lock.h | 2 + fs/dlm/lockspace.c | 105 ++---------- fs/dlm/member.c | 2 + fs/dlm/recover.c | 5 + fs/dlm/recoverd.c | 10 ++ 7 files changed, 283 insertions(+), 239 deletions(-)
fs/dlm/dlm_internal.h +10 -6
@@ -334,6 +334,7 @@ struct dlm_rsb {struct list_head res_root_list;/* used for recovery */ struct list_head res_masters_list;/* used for recovery */ struct list_head res_recover_list;/* used for recovery */ +struct list_head res_toss_q_list;int res_recover_locks_count;char *res_lvbptr;
@@ -584,13 +585,20 @@ struct dlm_ls {spinlock_t ls_lkbidr_spin;struct rhashtable ls_rsbtbl; -#define DLM_RTF_SHRINK_BIT 0 -unsigned long ls_rsbtbl_flags;spinlock_t ls_rsbtbl_lock;struct list_head ls_toss;struct list_head ls_keep; +struct timer_list ls_timer; +/* this queue is ordered according the +* absolute res_toss_time jiffies time +* to mod_timer() with the first element +* if necessary. +*/ +struct list_head ls_toss_q; +spinlock_t ls_toss_q_lock; +spinlock_t ls_waiters_lock;struct list_head ls_waiters;/* lkbs needing a reply */
@@ -601,9 +609,6 @@ struct dlm_ls {int ls_new_rsb_count;struct list_head ls_new_rsb;/* new rsb structs */ -char *ls_remove_names[DLM_REMOVE_NAMES_MAX ]; -int ls_remove_lens[DLM_REMOVE_NAMES_MAX ]; -struct list_head ls_nodes;/* current nodes in ls */ struct list_head ls_nodes_gone;/* dead node list, recovery */ int ls_num_nodes;/* number of nodes in ls */
@@ -640,7 +645,6 @@ struct dlm_ls {spinlock_t ls_cb_lock;struct list_head ls_cb_delay;/* save for queue_work later */ -struct timer_list ls_timer;struct task_struct *ls_recoverd_task;struct mutex ls_recoverd_active;spinlock_t ls_recover_lock;
fs/dlm/lock.c +240 -142
453 lines not on this page · raw message
fs/dlm/lock.h +2 -0
@@ -11,6 +11,7 @@#ifndef __LOCK_DOT_H__#define __LOCK_DOT_H__ +void dlm_rsb_toss_timer (struct timer_list *timer);void dlm_dump_rsb (struct dlm_rsb *r);void dlm_dump_rsb_name (struct dlm_ls *ls,const char *name,int len);void dlm_print_lkb (struct dlm_lkb *lkb);
@@ -26,6 +27,7 @@ void dlm_scan_rsbs(struct dlm_ls *ls);int dlm_lock_recovery_try (struct dlm_ls *ls);void dlm_lock_recovery (struct dlm_ls *ls);void dlm_unlock_recovery (struct dlm_ls *ls); +void dlm_timer_resume (struct dlm_ls *ls);int dlm_master_lookup (struct dlm_ls *ls,int from_nodeid,const char *name,int len,unsigned int flags,int *r_nodeid,int *result);
fs/dlm/lockspace.c +14 -91
174 lines not on this page · raw message
fs/dlm/member.c +2 -0
@@ -641,6 +641,8 @@ int dlm_ls_stop(struct dlm_ls *ls)spin_lock_bh (&ls->ls_recover_lock);set_bit (LSFL_RECOVER_STOP , &ls->ls_flags); new =test_and_clear_bit (LSFL_RUNNING , &ls->ls_flags); +if (new) +timer_delete_sync (&ls->ls_timer); ls->ls_recover_seq++;/* activate requestqueue and stop processing */
fs/dlm/recover.c +5 -0
@@ -889,6 +889,11 @@ void dlm_clear_toss(struct dlm_ls *ls)list_del (&r->res_rsbs_list);rhashtable_remove_fast (&ls->ls_rsbtbl, &r->res_node, dlm_rhash_rsb_params); + +/* remove it from the toss queue if its part of it */ +if (!list_empty (&r->res_toss_q_list)) +list_del_init (&r->res_toss_q_list); +free_toss_rsb (r); count++; }
fs/dlm/recoverd.c +10 -0
@@ -98,6 +98,16 @@ static int enable_locking(struct dlm_ls *ls, uint64_t seq)spin_lock_bh (&ls->ls_recover_lock);if (ls->ls_recover_seq == seq) {set_bit (LSFL_RUNNING , &ls->ls_flags); +/* Schedule next timer if recovery put something on toss. +* +* The rsbs that was queued while recovery on toss hasn't +* started yet because LSFL_RUNNING was set everything +* else recovery hasn't started as well because ls_in_recovery +* is still hold. So we should not run into the case that +* dlm_timer_resume() queues a timer that can occur in +* a no op. +*/ +dlm_timer_resume (ls);/* unblocks processes waiting to enter the dlm */ up_write (&ls->ls_in_recovery);clear_bit (LSFL_RECOVER_LOCK , &ls->ls_flags);
Signature · 1 line
2.43.0
Alexander Aring
maintainerRe: [PATCHv2 dlm/next 7/9] dlm: drop scand kthread and use timers
Hi,
Quoted 65 lines · Alexander Aring
Currently the scand kthread acts like a garbage collection for expired rsbs on toss list to clean them up after a certain timeout. It triggers every couple of seconds and iterates over the toss list while holding
ls_rsbtbl_lockfor the whole hash bucket iteration. To reduce the amount of holding thels_rsbtbl_locktime we handle free of cached rsbs (tossed rsbs) on a per rsb timer. If it expires the rsb deletes itself out of the rsb toss queue that is an ordered queue of tossed rsbs. First entry is the earliest rsb to expire the last is the lastest rsb that will expire. Theres_toss_timeis now an absolute value of when it will be expired before it was the time when the rsb was tossed. It was an debug functionality for debugfs anyway, we change a little bit the semantic here.Signed-off-by Alexander Aring <[email protected]>--- fs/dlm/dlm_internal.h | 16 +- fs/dlm/lock.c | 382 ++++++++++++++++++++++++++---------------- fs/dlm/lock.h | 2 + fs/dlm/lockspace.c | 105 ++---------- fs/dlm/member.c | 2 + fs/dlm/recover.c | 5 + fs/dlm/recoverd.c | 10 ++ 7 files changed, 283 insertions(+), 239 deletions(-)fs/dlm/dlm_internal.h +10
-3@@ -334,6 +334,7 @@ struct dlm_rsb {struct list_head res_root_list;/* used for recovery */ struct list_head res_masters_list;/* used for recovery */ struct list_head res_recover_list;/* used for recovery */ +struct list_head res_toss_q_list;int res_recover_locks_count;char *res_lvbptr;@@ -584,13 +585,20 @@ struct dlm_ls {spinlock_t ls_lkbidr_spin;struct rhashtable ls_rsbtbl; -#define DLM_RTF_SHRINK_BIT 0 -unsigned long ls_rsbtbl_flags;spinlock_t ls_rsbtbl_lock;struct list_head ls_toss;struct list_head ls_keep; +struct timer_list ls_timer; +/* this queue is ordered according the +* absolute res_toss_time jiffies time +* to mod_timer() with the first element +* if necessary. +*/ +struct list_head ls_toss_q; +spinlock_t ls_toss_q_lock; +spinlock_t ls_waiters_lock;struct list_head ls_waiters;/* lkbs needing a reply */ @@ -601,9 +609,6 @@ struct dlm_ls {int ls_new_rsb_count;struct list_head ls_new_rsb;/* new rsb structs */ -char
...
fs/dlm/dlm_internal.h static int find_rsb_nodir(struct dlm_ls *ls, const void *name, int len, · line 1015
Quoted 13 lines
fs/dlm/dlm_internal.h +5
-0+spin_unlock_bh (&ls->ls_rsbtbl_lock); + goto out_unlock;@@ -777,6 +1009,9 @@ static int find_rsb_nodir(struct dlm_ls *ls, const void *name, int len,* valid for keep state rsbs */ kref_init (&r->res_ref); +rsb_delete_toss_timer (ls, r); +spin_unlock_bh (&ls->ls_rsbtbl_lock); + goto out_unlock;
those spin_unlock_bh() are twice there... just for note I realized it now, it was fixed in the next patch. Probably because I was doing some git rebase things and tried to solve conflicts and made this mistake...
- Alex
Alexander Aring
maintainerPATCH v2 8/9 dlm: likely read lock path for rsb lookup
In reply to Alexander AringAs the conversion to rhashtable introduced a hash lock per lockspace instead of doing a per bucket hash, this patch will change the hash lock handling as holding only a read lock in a very likely hot path in DLM.
This hot path is to lookup rsbs and they are in keep state. Keep state means that the RSB_TOSS flag isn't set. If this likely read lock path is the case we don't need to hold the ls_rsbtbl_lock in write lock at all. If it's on toss state holding the ls_rsbtbl_lock in write lock is required. If we see that the rsb is in toss state, we hold the ls_rsbtbl_lock in write state and relookup (because the rsb could get removed into the transition from read to write state) and check if the rsb is still in toss state. However this is an unlikely path as toss rsbs are cached rsbs. Having the rsb in keep state is very likely.
--- fs/dlm/debug_fs.c | 4 +- fs/dlm/dir.c | 4 +- fs/dlm/dlm_internal.h | 2 +- fs/dlm/lock.c | 269 ++++++++++++++++++++++++++++++------------ fs/dlm/lockspace.c | 2 +- fs/dlm/recover.c | 4 +- fs/dlm/recoverd.c | 8 +- 7 files changed, 206 insertions(+), 87 deletions(-)
fs/dlm/debug_fs.c +2 -2
@@ -413,7 +413,7 @@ static void *table_seq_start(struct seq_file *seq, loff_t *pos)else list = &ls->ls_keep; -spin_lock_bh (&ls->ls_rsbtbl_lock); +read_lock_bh (&ls->ls_rsbtbl_lock);return seq_list_start (list, *pos); }
@@ -434,7 +434,7 @@ static void table_seq_stop(struct seq_file *seq, void *iter_ptr) {struct dlm_ls *ls = seq->private; -spin_unlock_bh (&ls->ls_rsbtbl_lock); +read_unlock_bh (&ls->ls_rsbtbl_lock); }static const struct seq_operations format1_seq_ops = {
fs/dlm/dir.c +2 -2
@@ -200,9 +200,9 @@ static struct dlm_rsb *find_rsb_root(struct dlm_ls *ls, const char *name,struct dlm_rsb *r;int rv; -spin_lock_bh (&ls->ls_rsbtbl_lock); +read_lock_bh (&ls->ls_rsbtbl_lock); rv =dlm_search_rsb_tree (&ls->ls_rsbtbl, name, len, &r); -spin_unlock_bh (&ls->ls_rsbtbl_lock); +read_unlock_bh (&ls->ls_rsbtbl_lock);if (!rv)return r;
fs/dlm/dlm_internal.h +1 -1
@@ -585,7 +585,7 @@ struct dlm_ls {spinlock_t ls_lkbidr_spin;struct rhashtable ls_rsbtbl; -spinlock_t ls_rsbtbl_lock; +rwlock_t ls_rsbtbl_lock;struct list_head ls_toss;struct list_head ls_keep;
fs/dlm/lock.c +194 -75
556 lines not on this page · raw message
fs/dlm/lockspace.c +1 -1
@@ -424,7 +424,7 @@ static int new_lockspace(const char *name, const char *cluster,INIT_LIST_HEAD (&ls->ls_toss);INIT_LIST_HEAD (&ls->ls_keep); -spin_lock_init (&ls->ls_rsbtbl_lock); +rwlock_init (&ls->ls_rsbtbl_lock); error =rhashtable_init (&ls->ls_rsbtbl, &dlm_rhash_rsb_params);if (error)
fs/dlm/recover.c +2 -2
@@ -884,7 +884,7 @@ void dlm_clear_toss(struct dlm_ls *ls)struct dlm_rsb *r, *safe;unsigned int count =0 ; -spin_lock_bh (&ls->ls_rsbtbl_lock); +write_lock_bh (&ls->ls_rsbtbl_lock);list_for_each_entry_safe (r, safe, &ls->ls_toss, res_rsbs_list) {list_del (&r->res_rsbs_list); rhashtable_remove_fast(&ls->ls_rsbtbl, &r->res_node,
@@ -897,7 +897,7 @@ void dlm_clear_toss(struct dlm_ls *ls)free_toss_rsb (r); count++; } -spin_unlock_bh (&ls->ls_rsbtbl_lock); +write_unlock_bh (&ls->ls_rsbtbl_lock);if (count)log_rinfo (ls,"dlm_clear_toss %u done" , count);
fs/dlm/recoverd.c +4 -4
@@ -32,7 +32,7 @@ static int dlm_create_masters_list(struct dlm_ls *ls) goto out; } -spin_lock_bh (&ls->ls_rsbtbl_lock); +read_lock_bh (&ls->ls_rsbtbl_lock);list_for_each_entry (r, &ls->ls_keep, res_rsbs_list) {if (r->res_nodeid)continue ;
@@ -40,7 +40,7 @@ static int dlm_create_masters_list(struct dlm_ls *ls)list_add (&r->res_masters_list, &ls->ls_masters_list);dlm_hold_rsb (r); } -spin_unlock_bh (&ls->ls_rsbtbl_lock); +read_unlock_bh (&ls->ls_rsbtbl_lock); out:write_unlock_bh (&ls->ls_masters_lock);return error;
@@ -62,14 +62,14 @@ static void dlm_create_root_list(struct dlm_ls *ls, struct list_head *root_list) {struct dlm_rsb *r; -spin_lock_bh (&ls->ls_rsbtbl_lock); +read_lock_bh (&ls->ls_rsbtbl_lock);list_for_each_entry (r, &ls->ls_keep, res_rsbs_list) {list_add (&r->res_root_list, root_list);dlm_hold_rsb (r); }WARN_ON_ONCE (!list_empty (&ls->ls_toss)); -spin_unlock_bh (&ls->ls_rsbtbl_lock); +read_unlock_bh (&ls->ls_rsbtbl_lock); }static void dlm_release_root_list (struct list_head *root_list)
Signature · 1 line
2.43.0
Alexander Aring
maintainerPATCH v2 9/9 dlm: convert lkbidr to rwlock
In reply to Alexander AringThis patch converts lkbidr lock to rwlock. The most time we doing lookups of idr only this can be done by holding the read lock only.
--- fs/dlm/dlm_internal.h | 2 +- fs/dlm/lock.c | 44 +++++++------------------------------------ fs/dlm/lockspace.c | 6 +++--- 3 files changed, 11 insertions(+), 41 deletions(-)
fs/dlm/dlm_internal.h +1 -1
@@ -582,7 +582,7 @@ struct dlm_ls {struct kobject ls_kobj;struct idr ls_lkbidr; -spinlock_t ls_lkbidr_spin; +rwlock_t ls_lkbidr_lock;struct rhashtable ls_rsbtbl;rwlock_t ls_rsbtbl_lock;
fs/dlm/lock.c +7 -37
76 lines not on this page · raw message
fs/dlm/lockspace.c +3 -3
@@ -431,7 +431,7 @@ static int new_lockspace(const char *name, const char *cluster, goto out_lsfree;idr_init (&ls->ls_lkbidr); -spin_lock_init (&ls->ls_lkbidr_spin); +rwlock_init (&ls->ls_lkbidr_lock);INIT_LIST_HEAD (&ls->ls_waiters);spin_lock_init (&ls->ls_waiters_lock);
@@ -676,7 +676,7 @@ static int lockspace_busy(struct dlm_ls *ls, int force) {int rv; -spin_lock_bh (&ls->ls_lkbidr_spin); +read_lock_bh (&ls->ls_lkbidr_lock);if (force ==0 ) { rv =idr_for_each (&ls->ls_lkbidr, lkb_idr_is_any, ls); }else if (force ==1 ) {
@@ -684,7 +684,7 @@ static int lockspace_busy(struct dlm_ls *ls, int force) }else { rv =0 ; } -spin_unlock_bh (&ls->ls_lkbidr_spin); +read_unlock_bh (&ls->ls_lkbidr_lock);return rv; }
Signature · 1 line
2.43.0