ocfs2/dlm: delay the migration when the lockres is in recovery
PATCH 1/1 ocfs2/dlm: delay the migration when the lockres is in recovery
We shouldn't migrate a lockres in recovery state.
Otherwise, it has the following problem:
1) Recovery happened as recovery master on a node(node A) which is in umount migrating all lockres' it owned(master is node A) to other nodes, say a node B.
2) So node A wants to take over all the lockres' those are mastered by the crashed node C.
3) Receiving request_locks request from node A, node B send mig_lockres requests(for recovery) to node A for all lockres' that was mastered by the crashed node C. It can also send the request for a lockres(lockres A) which is not in node A's hashtable.
4) Receiving the mig_lockres request for lockres A from node B, a new lockres object lockres A', with INRECOVERING flag set, is created and inserted to hash table.
5) The recovery for lockres A' is going on on node A, it finally mastered the lockres A'. And now, RECOVERING flag is not cleared from lockres A' nor from lockres A on node B.
6) The migration for lockres A' goes since now node A mastered lockres A' already. the mig_lockres request(for migration) is sent to node B.
7) Node B responsed with -EFAULT because now lockres A is still in recovery state.
8) Node A BUG() on the -EFAULT.
fix:
The recovery state is cleared on node A(recovery master) after it's cleared on node B. We wait until the in recovery state is cleared from node A and migrate it to node B.
--- fs/ocfs2/dlm/dlmmaster.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
fs/ocfs2/dlm/dlmmaster.c +3 -0
@@ -2371,6 +2371,9 @@ static int dlm_is_lockres_migrateable(struct dlm_ctxt *dlm, goto leave; } +if (unlikely (res->state &DLM_LOCK_RES_RECOVERING )) + goto leave; + ret =0 ; queue = &res->granted;for (i =0 ; i <3 ; i++) {
Signature · 1 line
1.6.6.1
PATCH 1/1 ocfs2/dlm: delay the migration when the lockres is in recovery
Any comment on this patch?
regards,
wengang.
Quoted 54 lines · Wengang Wang
We shouldn't migrate a lockres in recovery state.
Otherwise, it has the following problem:1) Recovery happened as recovery master on a node(node A) which is in umount migrating all lockres' it owned(master is node A) to other nodes, say a node B.
2) So node A wants to take over all the lockres' those are mastered by the crashed node C.
3) Receivingrequest_locksrequest from node A, node B sendmig_lockresrequests(for recovery) to node A for all lockres' that was mastered by the crashed node C. It can also send the request for a lockres(lockres A) which is not in node A's hashtable.
4) Receiving themig_lockresrequest for lockres A from node B, a new lockres object lockres A', with INRECOVERING flag set, is created and inserted to hash table.
5) The recovery for lockres A' is going on on node A, it finally mastered the lockres A'. And now, RECOVERING flag is not cleared from lockres A' nor from lockres A on node B.
6) The migration for lockres A' goes since now node A mastered lockres A' already. themig_lockresrequest(for migration) is sent to node B.
7) Node B responsed with-EFAULTbecause now lockres A is still in recovery state.
8) Node ABUG()on the-EFAULT.fix:
The recovery state is cleared on node A(recovery master) after it's cleared on node B. We wait until the in recovery state is cleared from node A and migrate it to node B.Signed-off-by Wengang Wang <[email protected]>--- fs/ocfs2/dlm/dlmmaster.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)fs/ocfs2/dlm/dlmmaster.c +3
-0@@ -2371,6 +2371,9 @@ static int dlm_is_lockres_migrateable(struct dlm_ctxt *dlm, goto leave; } +if (unlikely (res->state &DLM_LOCK_RES_RECOVERING )) + goto leave; + ret =0 ; queue = &res->granted;for (i =0 ; i <3 ; i++) {Signature · 7 lines
1.6.6.1 _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel at oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-devel
PATCH 1/1 ocfs2/dlm: delay the migration when the lockres is in recovery
patch looks good, it fixes the umount code path which prevents a lockres from migrating if it needs to be recovered. I have few comments on the scenario you described.
Quoted 18 lines · Wengang Wang
Any comment on this patch?
regards,
wengang.Quoted 12 lines · Wengang Wang
We shouldn't migrate a lockres in recovery state.
Otherwise, it has the following problem:1) Recovery happened as recovery master on a node(node A) which is in umount migrating all lockres' it owned(master is node A) to other nodes, say a node B.
2) So node A wants to take over all the lockres' those are mastered by the crashed node C.
3) Receivingrequest_locksrequest from node A, node B sendmig_lockresrequests(for recovery) to node A for all lockres' that was mastered by the crashed node C. It can also send the request for a lockres(lockres A) which is not in node A's hashtable.
why wouldn't lockres A be in node A's hashtable? if it's not in hash table, then it won't be migratable
Quoted 39 lines
Quoted 39 lines
4) Receiving the
mig_lockresrequest for lockres A from node B, a new lockres object lockres A', with INRECOVERING flag set, is created and inserted to hash table.
5) The recovery for lockres A' is going on on node A, it finally mastered the lockres A'. And now, RECOVERING flag is not cleared from lockres A' nor from lockres A on node B.
6) The migration for lockres A' goes since now node A mastered lockres A' already. themig_lockresrequest(for migration) is sent to node B.
7) Node B responsed with-EFAULTbecause now lockres A is still in recovery state.
8) Node ABUG()on the-EFAULT.fix:
The recovery state is cleared on node A(recovery master) after it's cleared on node B. We wait until the in recovery state is cleared from node A and migrate it to node B.Signed-off-by Wengang Wang <[email protected]>--- fs/ocfs2/dlm/dlmmaster.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)fs/ocfs2/dlm/dlmmaster.c +3
-0@@ -2371,6 +2371,9 @@ static int dlm_is_lockres_migrateable(struct dlm_ctxt *dlm, goto leave; } +if (unlikely (res->state &DLM_LOCK_RES_RECOVERING )) + goto leave; + ret =0 ; queue = &res->granted;for (i =0 ; i <3 ; i++) {Signature · 1 line
1.6.6.1
Quoted 11 lines
Quoted 5 lines
_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-devel_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20100615/3a7a0ef6/attachment.html
PATCH 1/1 ocfs2/dlm: delay the migration when the lockres is in recovery
Quoted 24 lines · Srinivas Eeda
patch looks good, it fixes the umount code path which prevents a lockres from migrating if it needs to be recovered. I have few comments on the scenario you described.
Quoted 18 lines · Wengang Wang
We shouldn't migrate a lockres in recovery state.
Otherwise, it has the following problem:1) Recovery happened as recovery master on a node(node A) which is in umount
migrating all lockres' it owned(master is node A) to other nodes, say a node B.
2) So node A wants to take over all the lockres' those are mastered by the
crashed node C.
3) Receivingrequest_locksrequest from node A, node B sendmig_lockres
requests(for recovery) to node A for all lockres' that was mastered by the
crashed node C. It can also send the request for a lockres(lockres A) which is
not in node A's hashtable.
Quoted 2 lines
why wouldn't lockres A be in node A's hashtable? if it's not in hash table, then it won't be migratable
In step 3), I am talking about the recovery process of node A(not the migration for umount).
Before node C crashes, the master of lockres A is node B. Node C has a ref on lockres A, but node A doesn't know anything about lockres A. Now node C crashed, node A is requesting(from node B) all lockres' that's owned by the crashed node(node C) and those which is in progress of migrating with the crashed node. Here lockres A is such a lockres that is in progress of migration from node B to node C when node C crashed. So node B can send lockres A, which is not in node A's hashtable before, to node A.
see dlm_clean_master_list(). It's moving lockres A to recovery list. mle->new_master == dead_node
Hope my description is clear :)
regards,
wengang.
PATCH 1/1 ocfs2/dlm: delay the migration when the lockres is in recovery
In reply to Wengang WangAny comment on this?
wengang.
Quoted 54 lines · Wengang Wang
We shouldn't migrate a lockres in recovery state.
Otherwise, it has the following problem:1) Recovery happened as recovery master on a node(node A) which is in umount migrating all lockres' it owned(master is node A) to other nodes, say a node B.
2) So node A wants to take over all the lockres' those are mastered by the crashed node C.
3) Receivingrequest_locksrequest from node A, node B sendmig_lockresrequests(for recovery) to node A for all lockres' that was mastered by the crashed node C. It can also send the request for a lockres(lockres A) which is not in node A's hashtable.
4) Receiving themig_lockresrequest for lockres A from node B, a new lockres object lockres A', with INRECOVERING flag set, is created and inserted to hash table.
5) The recovery for lockres A' is going on on node A, it finally mastered the lockres A'. And now, RECOVERING flag is not cleared from lockres A' nor from lockres A on node B.
6) The migration for lockres A' goes since now node A mastered lockres A' already. themig_lockresrequest(for migration) is sent to node B.
7) Node B responsed with-EFAULTbecause now lockres A is still in recovery state.
8) Node ABUG()on the-EFAULT.fix:
The recovery state is cleared on node A(recovery master) after it's cleared on node B. We wait until the in recovery state is cleared from node A and migrate it to node B.Signed-off-by Wengang Wang <[email protected]>--- fs/ocfs2/dlm/dlmmaster.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)fs/ocfs2/dlm/dlmmaster.c +3
-0@@ -2371,6 +2371,9 @@ static int dlm_is_lockres_migrateable(struct dlm_ctxt *dlm, goto leave; } +if (unlikely (res->state &DLM_LOCK_RES_RECOVERING )) + goto leave; + ret =0 ; queue = &res->granted;for (i =0 ; i <3 ; i++) {Signature · 7 lines
1.6.6.1 _______________________________________________ Ocfs2-devel mailing list Ocfs2-devel at oss.oracle.com http://oss.oracle.com/mailman/listinfo/ocfs2-devel