dlm: fix a race in lockres mastery
PATCH 1/1 dlm: fix a race in lockres mastery
In dlm_assert_master_handler, spinlock is released little sooner which creates a window for two nodes to race during mastery.
In a scenario where node A had a head start during lock mastery and dlm spinlock is just released on node B in dlm_assert_master_handler. Right then a process on node B started to master the resource. It finds the mle but doesn't find lockres. Since mle *master* is not set it creates a lockres and waits for the mastery and doesn't send a master request.
dlm_assert_master_handler doesn't know about this and doesn't set have_lockres_ref(doesn't set DLM_ASSERT_RESPONSE_MASTERY_REF) which creates a hole that results in loss of refmap bit on the master node.
--- fs/ocfs2/dlm/dlmmaster.c | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-)
fs/ocfs2/dlm/dlmmaster.c +1 -3
@@ -1875,7 +1875,6 @@ int dlm_assert_master_handler(struct o2net_msg *msg, u32 len, void *data, ok:spin_unlock (&res->spinlock); } -spin_unlock (&dlm->spinlock);// mlog(0, "woo! got an assert_master from node %u!\n", // assert->node_idx);
@@ -1926,7 +1925,6 @@ ok:/* master is known, detach if not already detached. * ensures that only one assert_master call will happen * on this mle. */ -spin_lock (&dlm->spinlock);spin_lock (&dlm->master_lock); rr =atomic_read (&mle->mle_refs.refcount);
@@ -1959,7 +1957,6 @@ ok:__dlm_put_mle (mle); }spin_unlock (&dlm->master_lock); -spin_unlock (&dlm->spinlock); }else if (res) {if (res->owner != assert->node_idx) { mlog(0, "assert_master from %u, but current "
@@ -1967,6 +1964,7 @@ ok: res->owner, namelen, name); } } +spin_unlock (&dlm->spinlock); done: ret =0 ;
Signature · 1 line
1.5.6.5
PATCH 1/1 dlm: fix a race in lockres mastery
=================================================================
In o2dlm, the resource master maintains a (ref)map of all nodes that are informed of the fact that that node masters that resource. This is done to prevent the master from purging the resource before the other node can create a lock.
This patch plugs a race between the mastery handler thread and the mastery thread that allows the node to discover the master of the resource without informing the master node.
Fixes ossbz#1012
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1012
=================================================================
Add my sob and change the comment to the above.
Quoted 54 lines · Srinivas Eeda
In
dlm_assert_master_handler, spinlock is released little sooner which creates a window for two nodes to race during mastery.In a scenario where node A had a head start during lock mastery and dlm spinlock is just released on node B in
dlm_assert_master_handler. Right then a process on node B started to master the resource. It finds the mle but doesn't find lockres. Since mle *master* is not set it creates a lockres and waits for the mastery and doesn't send a master request.
dlm_assert_master_handlerdoesn't know about this and doesn't sethave_lockres_ref(doesn't set DLM_ASSERT_RESPONSE_MASTERY_REF)which creates a hole that results in loss of refmap bit on the master node.Signed-off-by Srinivas Eeda <[email protected]>--- fs/ocfs2/dlm/dlmmaster.c | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-)fs/ocfs2/dlm/dlmmaster.c +1
-3@@ -1875,7 +1875,6 @@ int dlm_assert_master_handler(struct o2net_msg *msg, u32 len, void *data, ok:spin_unlock (&res->spinlock); } -spin_unlock (&dlm->spinlock);// mlog(0, "woo! got an assert_master from node %u!\n", // assert->node_idx); @@ -1926,7 +1925,6 @@ ok:/* master is known, detach if not already detached. * ensures that only one assert_master call will happen * on this mle. */ -spin_lock (&dlm->spinlock);spin_lock (&dlm->master_lock); rr =atomic_read (&mle->mle_refs.refcount);@@ -1959,7 +1957,6 @@ ok:__dlm_put_mle (mle); }spin_unlock (&dlm->master_lock); -spin_unlock (&dlm->spinlock); }else if (res) {if (res->owner != assert->node_idx) { mlog(0, "assert_master from %u, but current "@@ -1967,6 +1964,7 @@ ok: res->owner, namelen, name); } } +spin_unlock (&dlm->spinlock); done: ret =0 ;
PATCH 1/1 dlm: fix a race in lockres mastery
Quoted 10 lines · Sunil Mushran
=================================================================
In o2dlm, the resource master maintains a (ref)map of all nodes that are informed of the fact that that node masters that resource. This is done to prevent the master from purging the resource before the other node can create a lock.This patch plugs a race between the mastery handler thread and the mastery thread that allows the node to discover the master of the resource without informing the master node.
Quoted 6 lines
Fixes ossbz#1012
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1012
=================================================================Add my sob and change the comment to the above.
Signed-off-by Sunil Mushran <[email protected]>
The comment above makes even less sense to me than Srini's.
Hiss, while a little confusing to read, made me feel like I would get it if I just looked at the code. This text is...mumbo-jumbo. "the resource master maintains a map of all nodes that are informed of the fact that that node masters that resource" indeed.
Let me see if I can parse it:
------------------------------------------
In o2dlm, the master of a lock resource keeps a map of all interested nodes. This prevents the master from purging the resource before an interested node can create a lock.
A race between the mastery thread and the mastery handler allowed an interested node to discover the who the master is without informing the master directly. This is easily fixed by holding the dlm spinlock a little longer in the mastery handler.
------------------------------------------
How's that?
I also had a question about the patch. It is safe to hold the
dlm spinlock across all the other spinlocks, right? I believe that the dlm spinlock is toplevel, so I'm pretty sure it is, but I wanted to verify.
Joel
Signature · 8 lines
"Friends may come and go, but enemies accumulate."
- Thomas Jones
Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127PATCH 1/1 dlm: fix a race in lockres mastery
yes, your wording is better. and yes, dlm->spinlock is the top level lock.
Quoted 47 lines · Joel Becker
Quoted 10 lines · Sunil Mushran
=================================================================
In o2dlm, the resource master maintains a (ref)map of all nodes that are informed of the fact that that node masters that resource. This is done to prevent the master from purging the resource before the other node can create a lock.This patch plugs a race between the mastery handler thread and the mastery thread that allows the node to discover the master of the resource without informing the master node.
Quoted 7 lines
Fixes ossbz#1012
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1012
=================================================================Add my sob and change the comment to the above.
Signed-off-by Sunil Mushran <[email protected]>The comment above makes even less sense to me than Srini's.
Hiss, while a little confusing to read, made me feel like I would get it if I just looked at the code. This text is...mumbo-jumbo. "the resource master maintains a map of all nodes that are informed of the fact that that node masters that resource" indeed.
Let me see if I can parse it:------------------------------------------
In o2dlm, the master of a lock resource keeps a map of all interested nodes. This prevents the master from purging the resource before an interested node can create a lock.A race between the mastery thread and the mastery handler allowed an interested node to discover the who the master is without informing the master directly. This is easily fixed by holding the dlm spinlock a little longer in the mastery handler.
------------------------------------------How's that?
I also had a question about the patch. It is safe to hold the
dlm spinlock across all the other spinlocks, right? I believe that the dlm spinlock is toplevel, so I'm pretty sure it is, but I wanted to verify.Joel
PATCH 1/1 dlm: fix a race in lockres mastery
Quoted 2 lines · Sunil Mushran
yes, your wording is better. and yes,
dlm->spinlockis the top level lock.
This patch is now in the 'fixes' branch of ocfs2.git.
Joel
Signature · 9 lines
Life's Little Instruction Book #314 "Never underestimate the power of forgiveness." Joel Becker Principal Software Developer Oracle E-mail: joel.becker at oracle.com Phone: (650) 506-8127
PATCH 1/1 dlm: fix a race in lockres mastery
Sunil, Joel, thanks for modifying the comments :)
Quoted 11 lines · Joel Becker
Quoted 3 lines · Sunil Mushran
yes, your wording is better. and yes,
dlm->spinlockis the top level lock.This patch is now in the 'fixes' branch of ocfs2.git.
Joel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20100322/c2897bf0/attachment.html