ocfs2/dlm: resend deref to new master if recovery occures
PATCH 1/1 ocfs2/dlm: resend deref to new master if recovery occures
thanks for doing this patch. I have a little comment, wondering if there could be a window where node B sent the lock info to node C as part of recovery and removed flag DLM_LOCK_RES_RECOVERING while dlm_thread was still purging it. In that case dlm_thread will still continue to remove it from hash list.
Also, this patch puts dlm_thread to sleep ... may be it's ok, but wondering if we can avoid that.
delay deref message if DLM_LOCK_RES_RECOVERING is set (which means recovery got to the lockres before dlm_thread could), move the lockres to the end of the purgelist and retry later.
do not inform recovery master if DLM_LOCK_RES_DROPPING_REF is set (which means dlm_thread got to the lockres before recovery). So in the case you described, node C will not know about node B dropping the dereference and node B will just go ahead and remove it from hash list and free it.
Quoted 110 lines · Wengang Wang
When purge a lockres, we unhash the lockres ignore the result of deref request and ignore the lockres state.
There is a problem that rarely happen. It can happen when recovery take places. Say node A is the master of the lockres with node B wants to deref and there is a node C. If things happen in the following order, the bug is triggered.1) node B send DEREF to node A for lockres A and waiting for result.
2) node A crashed, node C become the recovery master.
3) node C mastered lockres A with node B has a ref on it.
4) node B goes to unhashes the lockres A with a ref on node C.After step 4), if a umount comes on node C, it will hang at migrate lockres A since node B has a ref on it.
The fix is that we check if recovery happened on lockres A after sending DEREF request. If that happened, we keep lockres A in hash and in purge list for another try to send DEREF to the new master(node C). So that node C can clear the incorrect refbit.
Signed-off-by Wengang Wang <[email protected]>--- fs/ocfs2/dlm/dlmthread.c | 44 +++++++++++++++++++++++++++++++++++++++----- 1 files changed, 39 insertions(+), 5 deletions(-)fs/ocfs2/dlm/dlmthread.c +39
-5@@ -158,6 +158,8 @@ static int dlm_purge_lockres(struct dlm_ctxt *dlm,int master;int ret =0 ; +assert_spin_locked (&dlm->spinlock); +spin_lock (&res->spinlock);if (!__dlm_lockres_unused (res)) { mlog(0, "%s:%.*s: tried to purge but not unused\n",@@ -211,18 +213,30 @@ static int dlm_purge_lockres(struct dlm_ctxt *dlm, }spin_lock (&res->spinlock); +/* +* we dropped dlm->spinlock and res->spinlock when sending the DEREF +* request, there is a chance that a recovery happend on this lockres. +* in that case, we have to DEREF to the new master when recovery +* finished. otherwise, there can be an incorrect ref on the lockres +* on the new master on behalf of this node. +*/ +if (unlikely (res->state &DLM_LOCK_RES_RECOVERING )) { +spin_unlock (&res->spinlock); +return -EAGAIN ; + } +if (!list_empty (&res->purge)) {mlog (0 ,"removing lockres %.*s:%p from purgelist, " "master = %d\n" , res->lockname.len, res->lockname.name, res, master);list_del_init (&res->purge); -spin_unlock (&res->spinlock); +/* not the last ref */ dlm_lockres_put (res); dlm->purge_count--; - }else -spin_unlock (&res->spinlock); + }__dlm_unhash_lockres (res); +spin_unlock (&res->spinlock);/* lockres is not in the hash now. drop the flag and wake up * any processes waiting in dlm_get_lock_resource. */ @@ -241,6 +255,9 @@ static void dlm_run_purge_list(struct dlm_ctxt *dlm,unsigned int run_max, unused;unsigned long purge_jiffies;struct dlm_lock_resource *lockres; +int ret; + +#define DLM_WAIT_RECOVERY_FINISH_MS 500spin_lock (&dlm->spinlock); run_max = dlm->purge_count;@@ -280,8 +297,25 @@ static void dlm_run_purge_list(struct dlm_ctxt *dlm,/* This may drop and reacquire the dlm spinlock if it * has to do migration. */ -if (dlm_purge_lockres (dlm, lockres)) -BUG (); + ret =dlm_purge_lockres (dlm, lockres); +if (ret) { +if (ret == -EAGAIN ) { +/* +* recovery occured on this lockres. try to +* DEREF to the new master. +*/ +dlm_lockres_put (lockres); +spin_unlock (&dlm->spinlock); +mlog (ML_ERROR ,"retry to purge %.*s after %d" +"ms\n" , lockres->lockname.len, + lockres->lockname.name, +DLM_WAIT_RECOVERY_FINISH_MS ); +msleep (DLM_WAIT_RECOVERY_FINISH_MS ); +spin_lock (&dlm->spinlock); +continue ; + }else +BUG (); + }dlm_lockres_put (lockres);
PATCH 1/1 ocfs2/dlm: resend deref to new master if recovery occures
Quoted 5 lines · Srinivas Eeda
thanks for doing this patch. I have a little comment, wondering if there could be a window where node B sent the lock info to node C as part of recovery and removed flag
DLM_LOCK_RES_RECOVERINGwhiledlm_threadwas still purging it. In that casedlm_threadwill still continue to remove it from hash list.
Yes, you are right. There do is such a window. I missed that.
Quoted 3 lines
Also, this patch puts
dlm_threadto sleep ... may be it's ok, but wondering if we can avoid that.
Yes. I considered about that too but failed at finding a simple way to avoid that.
Quoted 3 lines
delay deref message if
DLM_LOCK_RES_RECOVERINGis set (which means recovery got to the lockres beforedlm_threadcould), move the lockres to the end of the purgelist and retry later.
Good point! I meant that but the patch deosn't prove that :P.
Quoted 5 lines
do not inform recovery master if
DLM_LOCK_RES_DROPPING_REFis set (which meansdlm_threadgot to the lockres before recovery). So in the case you described, node C will not know about node B dropping the dereference and node B will just go ahead and remove it from hash list and free it.
Cool idea! let me try and re-create the patch!
thanks much Srini.
wengang.
Quoted 19 lines
Quoted 18 lines · Wengang Wang
When purge a lockres, we unhash the lockres ignore the result of deref request and ignore the lockres state.
There is a problem that rarely happen. It can happen when recovery take places. Say node A is the master of the lockres with node B wants to deref and there is a node C. If things happen in the following order, the bug is triggered.1) node B send DEREF to node A for lockres A and waiting for result.
2) node A crashed, node C become the recovery master.
3) node C mastered lockres A with node B has a ref on it.
4) node B goes to unhashes the lockres A with a ref on node C. After step 4), if a umount comes on node C, it will hang at
migrate lockres A since node B has a ref on it.The fix is that we check if recovery happened on lockres A after sending DEREF request. If that happened, we keep lockres A in hash and in purge list for another try to send DEREF to the new master(node C). So that node C can clear the incorrect refbit.
PATCH 1/1 ocfs2/dlm: resend deref to new master if recovery occures
Quoted 3 lines
Quoted 3 lines
delay deref message if
DLM_LOCK_RES_RECOVERINGis set (which means recovery got to the lockres beforedlm_threadcould), move the lockres to the end of the purgelist and retry later.
If you meant checking before sending DEREF, it could cause a high cpu useage when there are only the ones in recovery left in the purge list until they are recovered. --Is that acceptable?
regards,
wengang.
PATCH 1/1 ocfs2/dlm: resend deref to new master if recovery occures
Quoted 9 lines · Wengang Wang
Quoted 4 lines
Quoted 4 lines
delay deref message if
DLM_LOCK_RES_RECOVERINGis set (which means recovery got to the lockres beforedlm_threadcould), move the lockres to the end of the purgelist and retry later.If you meant checking before sending DEREF, it could cause a high cpu useage when there are only the ones in recovery left in the purge list until they are recovered. --Is that acceptable?
Yea, you are right about the cpu usage. I think the right thing would be to exit the loop after all locks are traversed once.
Quoted 3 lines
regards,
wengang.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20100524/d283fc49/attachment.html
PATCH 1/1 ocfs2/dlm: cancel the migration or redo deref to recovery master
Changes to V1:
1 move the msleep to the second runs when the lockres is in recovery so the
purging work on other lockres' can go.
2 do not inform recovery master if DLM_LOCK_RES_DROPPING_REF is set and don't
resend deref in this case.
--- fs/ocfs2/dlm/dlmcommon.h | 1 + fs/ocfs2/dlm/dlmrecovery.c | 25 +++++++++++++++ fs/ocfs2/dlm/dlmthread.c | 73 ++++++++++++++++++++++++++++++++++++++----- 3 files changed, 90 insertions(+), 9 deletions(-)
fs/ocfs2/dlm/dlmcommon.h +1 -0
@@ -280,6 +280,7 @@ static inline void __dlm_set_joining_node(struct dlm_ctxt *dlm,#define DLM_LOCK_RES_IN_PROGRESS 0x00000010#define DLM_LOCK_RES_MIGRATING 0x00000020#define DLM_LOCK_RES_DROPPING_REF 0x00000040 +#define DLM_LOCK_RES_DE_DROP_REF 0x00000080#define DLM_LOCK_RES_BLOCK_DIRTY 0x00001000#define DLM_LOCK_RES_SETREF_INPROG 0x00002000
fs/ocfs2/dlm/dlmrecovery.c +25 -0
@@ -913,6 +913,27 @@ static void dlm_request_all_locks_worker(struct dlm_work_item *item, void *data)/* any errors returned will be due to the new_master dying, * the dlm_reco_thread should detect this */ list_for_each_entry (res, &resources, recovering) { +int ignore_mig =0 ; +spin_lock (&res->spinlock); +/* +* if we are dropping the lockres, no need to let the new master +* know the reference of this node. That is don't migrate the +* lockres to the new master. Also make sure we don't send DEREF +* request for the same lockres to the new master either. +*/ +if (unlikely (res->state &DLM_LOCK_RES_DROPPING_REF )) { + ignore_mig =1 ; + res->state |=DLM_LOCK_RES_DE_DROP_REF ; + } +spin_unlock (&res->spinlock); +if (ignore_mig) { +mlog (ML_NOTICE ,"ignore migrating %.*s to recovery " +"master %u as we are dropping it\n" , + res->lockname.len, res->lockname.name, + reco_master); +continue ; + } + ret =dlm_send_one_lockres (dlm, res, mres, reco_master,DLM_MRES_RECOVERY );if (ret <0 ) {
@@ -1997,7 +2018,11 @@ void dlm_move_lockres_to_recovery_list(struct dlm_ctxt *dlm,struct list_head *queue;struct dlm_lock *lock, *next; +assert_spin_locked (&res->spinlock); + res->state |=DLM_LOCK_RES_RECOVERING ; + res->state &= ~DLM_LOCK_RES_DE_DROP_REF ; +if (!list_empty (&res->recovering)) { mlog(0, "Recovering res %s:%.*s, is already on recovery list!\n",
fs/ocfs2/dlm/dlmthread.c +64 -9
124 lines not on this page · raw message
Signature · 1 line
1.6.6.1