Skip to contentSkip to conversation

dlm: fix a race in lockres mastery

PATCH New Srinivas Eeda 3 participants 6 messages Mar 22, 2010 – Mar 23, 2010 OCFS2

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 dlm: fix a race in lockres mastery

In dlm_assert_master_handler, spinlock is released little sooner which creates a window for two nodes to race during mastery.

In a scenario where node A had a head start during lock mastery and dlm spinlock is just released on node B in dlm_assert_master_handler. Right then a process on node B started to master the resource. It finds the mle but doesn't find lockres. Since mle *master* is not set it creates a lockres and waits for the mastery and doesn't send a master request.

dlm_assert_master_handler doesn't know about this and doesn't set have_lockres_ref(doesn't set DLM_ASSERT_RESPONSE_MASTERY_REF) which creates a hole that results in loss of refmap bit on the master node.

---
 fs/ocfs2/dlm/dlmmaster.c |    4 +---
 1 files changed, 1 insertions(+), 3 deletions(-)
fs/ocfs2/dlm/dlmmaster.c +1 -3
index 83bcaf2..6e694b6 100644
@@ -1875,7 +1875,6 @@ int dlm_assert_master_handler(struct o2net_msg *msg, u32 len, void *data,
 ok:
 		spin_unlock(&res->spinlock);
 	}
-	spin_unlock(&dlm->spinlock);
 
 	// mlog(0, "woo!  got an assert_master from node %u!\n",
 	// 	     assert->node_idx);
@@ -1926,7 +1925,6 @@ ok:
 		/* master is known, detach if not already detached.
 		 * ensures that only one assert_master call will happen
 		 * on this mle. */
-		spin_lock(&dlm->spinlock);
 		spin_lock(&dlm->master_lock);
 
 		rr = atomic_read(&mle->mle_refs.refcount);
@@ -1959,7 +1957,6 @@ ok:
 			__dlm_put_mle(mle);
 		}
 		spin_unlock(&dlm->master_lock);
-		spin_unlock(&dlm->spinlock);
 	} else if (res) {
 		if (res->owner != assert->node_idx) {
 			mlog(0, "assert_master from %u, but current "
@@ -1967,6 +1964,7 @@ ok:
 			     res->owner, namelen, name);
 		}
 	}
+	spin_unlock(&dlm->spinlock);
 
 done:
 	ret = 0;
Signature · 1 line
1.5.6.5
#1 lore raw

PATCH 1/1 dlm: fix a race in lockres mastery

=================================================================
In o2dlm, the resource master maintains a (ref)map of all nodes that are informed of the fact that that node masters that resource. This is done to prevent the master from purging the resource before the other node can create a lock.

This patch plugs a race between the mastery handler thread and the mastery thread that allows the node to discover the master of the resource without informing the master node.

Fixes ossbz#1012
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1012
=================================================================

Add my sob and change the comment to the above.

Quoted 54 lines · Srinivas Eeda

In dlm_assert_master_handler, spinlock is released little sooner which creates a window for two nodes to race during mastery.

In a scenario where node A had a head start during lock mastery and dlm spinlock is just released on node B in dlm_assert_master_handler. Right then a process on node B started to master the resource. It finds the mle but doesn't find lockres. Since mle *master* is not set it creates a lockres and waits for the mastery and doesn't send a master request.

dlm_assert_master_handler doesn't know about this and doesn't set have_lockres_ref(doesn't set DLM_ASSERT_RESPONSE_MASTERY_REF) which creates a hole that results in loss of refmap bit on the master node.

---
 fs/ocfs2/dlm/dlmmaster.c |    4 +---
 1 files changed, 1 insertions(+), 3 deletions(-)
fs/ocfs2/dlm/dlmmaster.c +1 -3
index 83bcaf2..6e694b6 100644
@@ -1875,7 +1875,6 @@ int dlm_assert_master_handler(struct o2net_msg *msg, u32 len, void *data,
 ok:
 		spin_unlock(&res->spinlock);
 	}
-	spin_unlock(&dlm->spinlock);
 
 	// mlog(0, "woo!  got an assert_master from node %u!\n",
 	// 	     assert->node_idx);
@@ -1926,7 +1925,6 @@ ok:
 		/* master is known, detach if not already detached.
 		 * ensures that only one assert_master call will happen
 		 * on this mle. */
-		spin_lock(&dlm->spinlock);
 		spin_lock(&dlm->master_lock);
 
 		rr = atomic_read(&mle->mle_refs.refcount);
@@ -1959,7 +1957,6 @@ ok:
 			__dlm_put_mle(mle);
 		}
 		spin_unlock(&dlm->master_lock);
-		spin_unlock(&dlm->spinlock);
 	} else if (res) {
 		if (res->owner != assert->node_idx) {
 			mlog(0, "assert_master from %u, but current "
@@ -1967,6 +1964,7 @@ ok:
 			     res->owner, namelen, name);
 		}
 	}
+	spin_unlock(&dlm->spinlock);
 
 done:
 	ret = 0;
#2 lore raw

PATCH 1/1 dlm: fix a race in lockres mastery

Quoted 10 lines · Sunil Mushran

=================================================================
In o2dlm, the resource master maintains a (ref)map of all nodes that are informed of the fact that that node masters that resource. This is done to prevent the master from purging the resource before the other node can create a lock.

This patch plugs a race between the mastery handler thread and the mastery thread that allows the node to discover the master of the resource without informing the master node.

Quoted 6 lines

Fixes ossbz#1012
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1012
=================================================================

Add my sob and change the comment to the above.

The comment above makes even less sense to me than Srini's.
Hiss, while a little confusing to read, made me feel like I would get it if I just looked at the code. This text is...mumbo-jumbo. "the resource master maintains a map of all nodes that are informed of the fact that that node masters that resource" indeed.
Let me see if I can parse it:

------------------------------------------
In o2dlm, the master of a lock resource keeps a map of all interested nodes. This prevents the master from purging the resource before an interested node can create a lock.

A race between the mastery thread and the mastery handler allowed an interested node to discover the who the master is without informing the master directly. This is easily fixed by holding the dlm spinlock a little longer in the mastery handler.
------------------------------------------

How's that?
I also had a question about the patch. It is safe to hold the
dlm spinlock across all the other spinlocks, right? I believe that the dlm spinlock is toplevel, so I'm pretty sure it is, but I wanted to verify.

Joel

Signature · 8 lines
"Friends may come and go, but enemies accumulate." 
        - Thomas Jones

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127
#3 lore raw

PATCH 1/1 dlm: fix a race in lockres mastery

yes, your wording is better. and yes, dlm->spinlock is the top level lock.

Quoted 47 lines · Joel Becker
Quoted 10 lines · Sunil Mushran

=================================================================
In o2dlm, the resource master maintains a (ref)map of all nodes that are informed of the fact that that node masters that resource. This is done to prevent the master from purging the resource before the other node can create a lock.

This patch plugs a race between the mastery handler thread and the mastery thread that allows the node to discover the master of the resource without informing the master node.

Quoted 7 lines

Fixes ossbz#1012
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1012
=================================================================

Add my sob and change the comment to the above.

The comment above makes even less sense to me than Srini's.
Hiss, while a little confusing to read, made me feel like I would get it if I just looked at the code. This text is...mumbo-jumbo. "the resource master maintains a map of all nodes that are informed of the fact that that node masters that resource" indeed.
Let me see if I can parse it:

------------------------------------------
In o2dlm, the master of a lock resource keeps a map of all interested nodes. This prevents the master from purging the resource before an interested node can create a lock.

A race between the mastery thread and the mastery handler allowed an interested node to discover the who the master is without informing the master directly. This is easily fixed by holding the dlm spinlock a little longer in the mastery handler.
------------------------------------------

How's that?
I also had a question about the patch. It is safe to hold the
dlm spinlock across all the other spinlocks, right? I believe that the dlm spinlock is toplevel, so I'm pretty sure it is, but I wanted to verify.

Joel

#4 lore raw

PATCH 1/1 dlm: fix a race in lockres mastery

Quoted 2 lines · Sunil Mushran

yes, your wording is better. and yes, dlm->spinlock is the top level lock.

This patch is now in the 'fixes' branch of ocfs2.git.

Joel

Signature · 9 lines
Life's Little Instruction Book #314

	"Never underestimate the power of forgiveness."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127
#5 lore raw

PATCH 1/1 dlm: fix a race in lockres mastery

Sunil, Joel, thanks for modifying the comments :)

Quoted 11 lines · Joel Becker
Quoted 3 lines · Sunil Mushran

yes, your wording is better. and yes, dlm->spinlock is the top level lock.

This patch is now in the 'fixes' branch of ocfs2.git.

Joel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-devel/attachments/20100322/c2897bf0/attachment.html