ocfs2: fix __ocfs2_cluster_lock() dead lock

there is deadlock possibility in __ocfs2_cluster_lock(). the case is like following.

#in time order
1) node A(the lock owner) is doing an up-convert(UC, PR->EX). it got the lock but didn't check BUSY flag again.

2) node B requested an UC on the same lock resource. since node A has an EX on the lockres, a bast is issued and OCFS2_LOCK_BLOCKED flag is set to the lockres meanting that a DC is should be done. the DC asks for agreement of node A to release EX. and for now node A refuses that(still in process of getting that lock:) ).

3) the UC on node A runs into the phrase of checking OCFS2_LOCK_BLOCKED flag and if it can continue without waiting for any DC. code is:

        if (lockres->l_flags & OCFS2_LOCK_BLOCKED &&
            !ocfs2_may_continue_on_blocked_lock(lockres, level)) {

analysis:
the BLOCKED flag is set in 2), and the UC can't continue to get the lock( ocfs2_may_continue_on_blocked_lock() returns false), so it the DC to finish.

so both UC and DC are waiting for each other --a dead lock.

Quoted 19 lines · Wengang Wang

there is deadlock possibility in __ocfs2_cluster_lock(). the case is like following.

#in time order
1) node A(the lock owner) is doing an up-convert(UC, PR->EX). it got the lock but didn't check BUSY flag again.

2) node B requested an UC on the same lock resource. since node A has an EX on the lockres, a bast is issued and OCFS2_LOCK_BLOCKED flag is set to the lockres meanting that a DC is should be done. the DC asks for agreement of node A to release EX. and for now node A refuses that(still in process of getting that lock:) ).

3) the UC on node A runs into the phrase of checking OCFS2_LOCK_BLOCKED flag and if it can continue without waiting for any DC. code is:

        if (lockres->l_flags & OCFS2_LOCK_BLOCKED &&
            !ocfs2_may_continue_on_blocked_lock(lockres, level)) {

Quoted 5 lines

analysis:
the BLOCKED flag is set in 2), and the UC can't continue to get the lock( ocfs2_may_continue_on_blocked_lock() returns false), so it the DC to finish.

so both UC and DC are waiting for each other --a dead lock.

1) Do you have a test case? Have you seen this happen, or are you just postulating from reading the code?

2) It sure looks like you're right.
3) I hate our locking spaghetti!

Quoted 23 lines · Wengang Wang

there is deadlock possibility in __ocfs2_cluster_lock(). the case is like following.

#in time order
1) node A(the lock owner) is doing an up-convert(UC, PR->EX). it got the lock but didn't check BUSY flag again.

2) node B requested an UC on the same lock resource. since node A has an EX on the lockres, a bast is issued and OCFS2_LOCK_BLOCKED flag is set to the lockres meanting that a DC is should be done. the DC asks for agreement of node A to release EX. and for now node A refuses that(still in process of getting that lock:) ).

3) the UC on node A runs into the phrase of checking OCFS2_LOCK_BLOCKED flag and if it can continue without waiting for any DC. code is:

        if (lockres->l_flags & OCFS2_LOCK_BLOCKED &&
            !ocfs2_may_continue_on_blocked_lock(lockres, level)) {

analysis:
the BLOCKED flag is set in 2), and the UC can't continue to get the lock( ocfs2_may_continue_on_blocked_lock() returns false), so it the DC to finish.

Quoted 1 line

so both UC and DC are waiting for each other --a dead lock.

1) Do you have a test case? Have you seen this happen, or are you just postulating from reading the code?
2) It sure looks like you're right.

Quoted 32 lines · Joel Becker

Quoted 31 lines · Wengang Wang

there is deadlock possibility in __ocfs2_cluster_lock(). the case is like following.

#in time order
1) node A(the lock owner) is doing an up-convert(UC, PR->EX). it got the lock
but didn't check BUSY flag again.

2) node B requested an UC on the same lock resource. since node A has an EX on
the lockres, a bast is issued and OCFS2_LOCK_BLOCKED flag is set to the lockres
meanting that a DC is should be done. the DC asks for agreement of node A to
release EX. and for now node A refuses that(still in process of getting that
lock:) ).

3) the UC on node A runs into the phrase of checking OCFS2_LOCK_BLOCKED flag and
if it can continue without waiting for any DC. code is:

       if (lockres->l_flags & OCFS2_LOCK_BLOCKED &&
           !ocfs2_may_continue_on_blocked_lock(lockres, level)) {

analysis:
the BLOCKED flag is set in 2), and the UC can't continue to get the lock(
ocfs2_may_continue_on_blocked_lock() returns false), so it the DC to finish.

Quoted 6 lines

Quoted 1 line

so both UC and DC are waiting for each other --a dead lock.

1) Do you have a test case? Have you seen this happen, or are you just
postulating from reading the code?
2) It sure looks like you're right.

well, this doesn't sounds like a sob :)

regards,
wengang.

_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-devel

This needs testing. Joel has forwarded it to Fabbione who has a testcase that we believe triggers this issue. If that works, I'll push this in the 1.6 testing tree. That way we'll know (a) the patch fixes the problem, and (b) does not break other things. This is a tricky function. We are just being careful.

Quoted 5 lines · Sunil Mushran

This needs testing. Joel has forwarded it to Fabbione who has a testcase that we believe triggers this issue. If that works, I'll push this in the 1.6 testing tree. That way we'll know (a) the patch fixes the problem, and (b) does not break other things. This is a tricky function. We are just being careful.

Wengang,
Like Sunil said, this is a very critical part of the code. Our
locking code is the biggest place for lockups and corruptions to happen. We always take the safe route here. We're working to test both of your patches.

Joel

Signature · 9 lines

Life's Little Instruction Book #198

	"Feed a stranger's expired parking meter."

Joel Becker
Principal Software Developer
Oracle
E-mail: joel.becker at oracle.com
Phone: (650) 506-8127

Quoted 1 line

2) It sure looks like you're right.

well, this doesn't sounds like a sob :)

[Cc'd Mark and Dave]

Quoted 2 lines · Wengang Wang

Quoted 1 line

2) It sure looks like you're right.

well, this doesn't sounds like a sob :)

With good reason, too. You've got a bug. Dave Teigland (copied
now) found it while testing with fs/dlm.
The short description is that you didn't notice the handling of
async -EAGAIN. Not really your fault, there are no obvious comments. With o2dlm, you get -EAGAIN from the call to ocfs2_dlm_lock(). That's safe with your patch. However, with fs/dlm, EAGAIN comes asynchronously. ocfs2_dlm_lock() will return 0. When the ast fires, the ast notices the -EAGAIN and clears the locking state for a retry. In your patch, however, you take the 'lock_done' + !BUSY and immediately increment the holders - even though the lock hasn't been acquired!

The fix is pretty simple, though - when lock_done is set, you
know we've at least tried to upconvert. If the locking level is good, we know the upconvert succeeded. We can then go to inc_holders. I've attached a new version of your patch that does this.
I've also modified the patch to reflect that the original
problem you found is actually a livelock, not a deadlock. And I added a little comment about the async -EAGAIN.
I've attached the full patch with my changes. Dave, please test
my version (the attached one) instead of Wengang's. Mark, can you do a sanity check here?
Here's the diff between your patch and mine:

Date: Wed, 6 Jan 2010 16:34:44 +0800
Subject: [PATCH] ocfs2: fix __ocfs2_cluster_lock() livelock

There is livelock possibility in __ocfs2_cluster_lock(). Here's what happens:

1) node A(the lock owner) is doing an up-convert(UC, PR->EX). it gets the lock but doesn't check BUSY again because the level is right.

2) node B requested an UC on the same lock resource. since node A has an EX on the lockres, a bast is issued and OCFS2_LOCK_BLOCKED flag is set to the lockres meanting that a DC is should be done. the DC asks for agreement of node A to release EX. The downconvert starts because BUSY is cleared by the ast.

3) the UC on node A runs into the check of OCFS2_LOCK_BLOCKED:

        if (lockres->l_flags & OCFS2_LOCK_BLOCKED &&
            !ocfs2_may_continue_on_blocked_lock(lockres, level)) {

analysis:
the BLOCKED flag is set in 2), and the UC can't continue to get the lock (ocfs2_may_continue_on_blocked_lock() returns false), so it waits on the DC to finish.

4) The DC finishes, and BLOCKED is cleared. The UC on node A starts over getting the UC, now from NL->EX. It asks node B for the lock.

Quoted 25 lines · Joel Becker

Date: Wed, 6 Jan 2010 16:34:44 +0800
Subject: [PATCH] ocfs2: fix __ocfs2_cluster_lock() livelock

There is livelock possibility in __ocfs2_cluster_lock(). Here's what happens:

1) node A(the lock owner) is doing an up-convert(UC, PR->EX). it gets the lock but doesn't check BUSY again because the level is right.

2) node B requested an UC on the same lock resource. since node A has an EX on the lockres, a bast is issued and OCFS2_LOCK_BLOCKED flag is set to the lockres meanting that a DC is should be done. the DC asks for agreement of node A to release EX. The downconvert starts because BUSY is cleared by the ast.

3) the UC on node A runs into the check of OCFS2_LOCK_BLOCKED:

        if (lockres->l_flags & OCFS2_LOCK_BLOCKED &&
            !ocfs2_may_continue_on_blocked_lock(lockres, level)) {

analysis:
the BLOCKED flag is set in 2), and the UC can't continue to get the lock (ocfs2_may_continue_on_blocked_lock() returns false), so it waits on the DC to finish.

Quoted 2 lines

4) The DC finishes, and BLOCKED is cleared. The UC on node A starts over getting the UC, now from NL->EX. It asks node B for the lock.

I am wondering why DC can finish.
although BUSY is cleared. I think the check_downconvert() should return false since we have not done whatever we want to do. and then the DC should be requeued.

#don't hate me if I am wrong/stupid :)

Quoted 28 lines

Quoted 27 lines

Date: Wed, 6 Jan 2010 16:34:44 +0800
Subject: [PATCH] ocfs2: fix __ocfs2_cluster_lock() livelock

There is livelock possibility in __ocfs2_cluster_lock(). Here's what happens:

1) node A(the lock owner) is doing an up-convert(UC, PR->EX). it gets the lock but doesn't check BUSY again because the level is right.

2) node B requested an UC on the same lock resource. since node A has an EX on the lockres, a bast is issued and OCFS2_LOCK_BLOCKED flag is set to the lockres meanting that a DC is should be done. the DC asks for agreement of node A to release EX. The downconvert starts because BUSY is cleared by the ast.

3) the UC on node A runs into the check of OCFS2_LOCK_BLOCKED:

        if (lockres->l_flags & OCFS2_LOCK_BLOCKED &&
            !ocfs2_may_continue_on_blocked_lock(lockres, level)) {

analysis:
the BLOCKED flag is set in 2), and the UC can't continue to get the lock (ocfs2_may_continue_on_blocked_lock() returns false), so it waits on the DC to finish.

4) The DC finishes, and BLOCKED is cleared. The UC on node A starts over getting the UC, now from NL->EX. It asks node B for the lock.

Quoted 4 lines

I am wondering why DC can finish.
although BUSY is cleared. I think the check_downconvert() should return false since we have not done whatever we want to do. and then the DC should be requeued.

DC thread only waits on PENDING and holders. If BUSY, it will
cancel the upconvert. If not BUSY, it will schedule a downconvert. There's nothing stopping the downconvert, in other words.

Quoted 1 line

#don't hate me if I am wrong/stupid :)

Won't hate you at all. And you're not stupid even if you are
wrong. It's not easy code. Heck, you may be right and I might be wrong in the end. Remember, you found the bug in the first place!

Quoted 3 lines · Joel Becker

DC thread only waits on PENDING and holders. If BUSY, it will
cancel the upconvert. If not BUSY, it will schedule a downconvert. There's nothing stopping the downconvert, in other words.

in ocfs2_unblock_lock() there are lines after checking for BUSY flag:

        if (lockres->l_ops->check_downconvert
            && !lockres->l_ops->check_downconvert(lockres, new_level))
                goto leave_requeue;

if ctl->requeue is set, ocfs2_schedule_blocked_lock() will add the lockres to blocked_lock_list. the downconvert will then be rescheduled only if ocfs2_wake_downconvert_thread() is called somewhere.
in this simple case:
the "somewhere", I think, is in __ocfs2_cluster_unlock().
so before __ocfs2_cluster_unlock() is called(surely after
__ocfs2_cluster_lock() returns), the downconvert couldn't
be rescheduled.
thus __ocfs2_cluster_lock() is waiting for DC to finish and the DC is waiting __ocfs2_cluster_unlock() to be called. --deadlock.

so maybe you don't notice the lockres->l_ops->check_downconvert()?

Quoted 1 line

so maybe you don't notice the lockres->l_ops->check_downconvert()?

I notice it. I know what it checks. If the lock is not
currently taken it will return "go ahead and downconvert". So we won't requeue, we'll downconvert. Specifically, look at ocfs2_check_meta_downconvert(). It merely checks the caching info. We can't have any cached data if we don't have the EX yet.

Sunil has observed the livelock behavior (pinging between UC and
DC) in the wild via traces. The UC process never leaves ocfs2_cluster_lock(). Your change fixes that.

Is it a rule that If the lock is not currently taken it will return "go ahead and downconvert"?

by checking ocfs2_check_meta_downconvert(), I think it ensures cache is checkpointed by JBD(2) before returning "go ahead and downconvert". maybe yes that you noticed that we can't have cache if we don't have EX, I think the main purpose is that it ensures that we finish things which we should finish before we release the lock. another evidence is that it doesn't check the ex_holders.

another question is that you can find out ocfs2_check_meta_downconvert() checks if lock is taken(not explicitly), can you make such a conclusion that all check_downconvert() follows it? --sorry, I didn't check them and pushed it to you :P --I will check them too later.

so still the question, is it a rule that check_downconvert() should return "go ahead and downconvert" if the lock is not currently taken? and also, what means "currently taken"? --ocfs2 layer(after ocfs2_cluster_lock() returns) or dlm level? though I guess you meant ocfs2 layer.

Quoted 2 lines · Wengang Wang

Is it a rule that If the lock is not currently taken it will return "go ahead and downconvert"?

In short, yes, because of fairness.

Not having holders is one of the checks. If there are incompatible holders then it will requeue the lockres in the blocked queue. The lockres will be re-processed when the dc thread is kicked the next time. If there are no holders and the journal is checkpointed (or check_downconvert succeeds), then yes, it will be downconvert.

Quoted 6 lines

by checking ocfs2_check_meta_downconvert(), I think it ensures cache is checkpointed by JBD(2) before returning "go ahead and downconvert". maybe yes that you noticed that we can't have cache if we don't have EX, I think the main purpose is that it ensures that we finish things which we should finish before we release the lock. another evidence is that it doesn't check the ex_holders.

check_downconvert() is an additional check. The holders check tells the fs whether it can and whether it needs to down convert or not. If answer to both qs is yes, then it sees whether there are any lock type specific checks. M and T lock types want the journal check pointed before a dc.

Quoted 11 lines

another question is that you can find out ocfs2_check_meta_downconvert() checks if lock is taken(not explicitly), can you make such a conclusion that all check_downconvert() follows it? --sorry, I didn't check them and pushed it to you :P --I will check them too later.

so still the question, is it a rule that check_downconvert() should return "go ahead and downconvert" if the lock is not currently taken? and also, what means "currently taken"? --ocfs2 layer(after ocfs2_cluster_lock() returns) or dlm level? though I guess you meant ocfs2 layer.

I am confused by your qs. Have you read the description of check_downconvert()?

/*
* Allow a lock type to add checks to determine whether it is
* safe to downconvert a lock. Return 0 to re-queue the
* downconvert at a later time, nonzero to continue. *
* For most locks, the default checks that there are no
* incompatible holders are sufficient.
*
* Called with the lockres spinlock held.
*/
int (*check_downconvert)(struct ocfs2_lock_res *, int);

Sunil

I've attached the full patch with my changes. Dave, please test
my version (the attached one) instead of Wengang's.

[Cc'd Mark and Dave]

Signed-off-by Joel Becker <[email protected]>

---
 fs/ocfs2/dlmglue.c |   29 +++++++++++++++++++++++++++++
 1 files changed, 29 insertions(+), 0 deletions(-)

fs/ocfs2/dlmglue.c +3 -0

index 90682a0..eabe53e 100644
@@ -1313,6 +1313,7 @@ static int __ocfs2_cluster_lock(struct ocfs2_super *osb,
 	unsigned long flags;
 	unsigned int gen;
 	int noqueue_attempted = 0;
+	int lock_attempted = 0;
 
 	mlog_entry_void();
 

@@ -1347,6 +1348,25 @@ again:
 		goto unlock;
 	}
 
+	if (lock_attempted && (lockres->l_level >= level)) {
+		/*

Hi Joel and David,

I think the patch to fix the deadlock(or livelock) is not good enough yet.
my original patch is based on that when we have requested the lock, we don't allow to downconvert it before we release it.
but per the communications with Joel and Sunil, I found my base is wrong. so my original patch is not good. and Joel's patch has problem either.

Quoted 24 lines · Joel Becker

[Cc'd Mark and Dave]

Signed-off-by Joel Becker <[email protected]>

---
 fs/ocfs2/dlmglue.c |   29 +++++++++++++++++++++++++++++
 1 files changed, 29 insertions(+), 0 deletions(-)

fs/ocfs2/dlmglue.c +3 -0

index 90682a0..eabe53e 100644
@@ -1313,6 +1313,7 @@ static int __ocfs2_cluster_lock(struct ocfs2_super *osb,
 	unsigned long flags;
 	unsigned int gen;
 	int noqueue_attempted = 0;
+	int lock_attempted = 0;
 
 	mlog_entry_void();
 

@@ -1347,6 +1348,25 @@ again:
 		goto unlock;
 	}
 
+	if (lock_attempted && (lockres->l_level >= level)) {
+		/*

fs/ocfs2/dlmglue.c again: · line 1362

Quoted 10 lines

fs/ocfs2/dlmglue.c +10 -0

+		 * We've attempted to upconvert, and the lock now has
+		 * a level we can work with.  If we fell through to the
+		 * next checks, we could spin in an upconvert->downconvert
+		 * cycle as other nodes pounded us.  Instead, we jump
+		 * out and let the caller do some work.  If a downconvert
+		 * has come in, it will do its thing as soon as the caller
+		 * is done.
+		 */
+		goto update_holders;
+	}

before update_holders, the lock could be DCed(since no BUSY flag set by here).

and even after update_holders, the lock could be DCed too.

so that we get ocfs2_cluster_lock()(with holders increased) returned sucessfully but actually we don't hold the dlm lock. --thus more than one node is considering that they have the (EX) lock.

that's could be the cause of what is happening observed by David.

before update_holders, the lock could be DCed(since no BUSY flag set by here).

and even after update_holders, the lock could be DCed too.

so that we get ocfs2_cluster_lock()(with holders increased) returned sucessfully but actually we don't hold the dlm lock. --thus more than one node is considering that they have the (EX) lock.

Quoted 5 lines · Wengang Wang

before update_holders, the lock could be DCed(since no BUSY flag set by here).

and even after update_holders, the lock could be DCed too.

Quoted 3 lines

so that we get ocfs2_cluster_lock()(with holders increased) returned sucessfully but actually we don't hold the dlm lock. --thus more than one node is considering that they have the (EX) lock.

You make a good point. I don't like the solution you propose,
though. Another flag that's almost the same? Eww! There's got to be a better way.

Quoted 6 lines · Joel Becker

Quoted 5 lines · Wengang Wang

before update_holders, the lock could be DCed(since no BUSY flag set by here).

and even after update_holders, the lock could be DCed too.

Quoted 7 lines

Quoted 3 lines

so that we get ocfs2_cluster_lock()(with holders increased) returned sucessfully but actually we don't hold the dlm lock. --thus more than one node is considering that they have the (EX) lock.

You make a good point. I don't like the solution you propose,
though. Another flag that's almost the same? Eww! There's got to be a better way.

cool! expecting your patch...

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

before update_holders, the lock could be DCed(since no BUSY flag set by here).

and even after update_holders, the lock could be DCed too.

so that we get ocfs2_cluster_lock()(with holders increased) returned sucessfully but actually we don't hold the dlm lock. --thus more than one node is considering that they have the (EX) lock.

Quoted 4 lines

Quoted 3 lines

You make a good point. I don't like the solution you propose,
though. Another flag that's almost the same? Eww! There's got to be a better way.

cool! expecting your patch...

Don't have one yet. We can't rely on the spinlock, because we
have to sleep on BUSY. We relied on the spinblock before because we allowed the UC->DC->UC->DC livelock to happen.

Quoted 5 lines · Wengang Wang

before update_holders, the lock could be DCed(since no BUSY flag set by here).

and even after update_holders, the lock could be DCed too.

Quoted 3 lines

so that we get ocfs2_cluster_lock()(with holders increased) returned sucessfully but actually we don't hold the dlm lock. --thus more than one node is considering that they have the (EX) lock.

You make a good point. I don't like the solution you propose,
though. Another flag that's almost the same? Eww! There's got to be a better way.

Quoted 6 lines · Joel Becker

Quoted 5 lines · Wengang Wang

before update_holders, the lock could be DCed(since no BUSY flag set by here).

and even after update_holders, the lock could be DCed too.

Quoted 7 lines

Quoted 3 lines

so that we get ocfs2_cluster_lock()(with holders increased) returned sucessfully but actually we don't hold the dlm lock. --thus more than one node is considering that they have the (EX) lock.

You make a good point. I don't like the solution you propose,
though. Another flag that's almost the same? Eww! There's got to be a better way.

I don't see how we can resolve this without another flag. It could be a mirror image of the PENDING flag. PENDING takes care of the gap between BUSY set and calling dlm_lock(). This flag has to take care of the gap between BUSY clear and inc_holders.

Am working on improving Wengang's patch.

Mark is strangely silent.

Quoted 15 lines · Sunil Mushran

Quoted 13 lines · Joel Becker

Quoted 8 lines · Wengang Wang

before update_holders, the lock could be DCed(since no BUSY flag set by here).

and even after update_holders, the lock could be DCed too.

so that we get ocfs2_cluster_lock()(with holders increased) returned sucessfully but actually we don't hold the dlm lock. --thus more than one node is considering that they have the (EX) lock.

You make a good point. I don't like the solution you propose,
though. Another flag that's almost the same? Eww! There's got to be a better way.

Quoted 6 lines

I don't see how we can resolve this without another flag. It could be a mirror image of the PENDING flag. PENDING takes care of the gap between BUSY set and calling dlm_lock(). This flag has to take care of the gap between BUSY clear and inc_holders.

Am working on improving Wengang's patch.

I don't see anything yet, but PENDING is a complex enough flag
as-is. Another flag and we're even less understandable ;-( If we have to do it, and I assume you'll set the flag in the ast when you clear BUSY, I say call it UPCONVERT_FINISHING.
Oh, wait. This means that we can drop the
lock_attemped/lock_done state variable from wengang's original change. Right before ocfs2_cluster_lock checks for BLOCKING, it can just check flags&UPCONVERT_FINISHING and jump to inc_holders. Maybe this is in wengang's update and I didn't notice.

Quoted 6 lines · Joel Becker

Quoted 5 lines · Wengang Wang

before update_holders, the lock could be DCed(since no BUSY flag set by here).

and even after update_holders, the lock could be DCed too.

Quoted 8 lines

Quoted 4 lines

so that we get ocfs2_cluster_lock()(with holders increased) returned sucessfully
but actually we don't hold the dlm lock. --thus more than one node is considering that they have the (EX) lock.

You make a good point. I don't like the solution you propose,
though. Another flag that's almost the same? Eww! There's got to be a better way.

I don't see how we can resolve this without another flag. It could be a mirror image of the PENDING flag. PENDING takes care of the gap between BUSY set and calling dlm_lock(). This flag has to take care of the gap between BUSY clear and inc_holders.

Am working on improving Wengang's patch.

Quoted 20 lines · Sunil Mushran

Quoted 14 lines · Joel Becker

Quoted 4 lines · Wengang Wang

before update_holders, the lock could be DCed(since no BUSY flag set by
here).

Quoted 8 lines

and even after update_holders, the lock could be DCed too.

so that we get ocfs2_cluster_lock()(with holders increased) returned
sucessfully
but actually we don't hold the dlm lock. --thus more than one node is
considering that they have the (EX) lock.

Quoted 4 lines

You make a good point. I don't like the solution you propose,
though. Another flag that's almost the same? Eww! There's got to be a
better way.

Quoted 6 lines

I don't see how we can resolve this without another flag. It could be a mirror image of the PENDING flag. PENDING takes care of the gap between BUSY set and calling dlm_lock(). This flag has to take care of the gap between BUSY clear and inc_holders.

Am working on improving Wengang's patch.

I'm afraid I lost track of which patches I should be trying. Is there something that we expect should solve the known problems, even if it's not
a final version?
Dave