Skip to contentSkip to conversation

ocfs2: ocfs2_downconvert_lock failure results in deadlock

PATCH Superseded Gang He 2 participants 5 messages Aug 26, 2021 – Aug 27, 2021 OCFS2

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Gang He

author
#0 lore raw

PATCH 1/1 ocfs2: ocfs2_downconvert_lock failure results in deadlock

Usually, ocfs2_downconvert_lock() function always downconverts dlm lock to the expected level for satisfy dlm bast requests from the other nodes.
But there is a rare situation. When dlm lock conversion is being canceled, ocfs2_downconvert_lock() function will return -EBUSY. You need to be aware that ocfs2_cancel_convert() function is asynchronous in fsdlm implementation.
If we does not requeue this lockres entry, ocfs2 downconvert thread no longer handles this dlm lock bast request. Then, the other nodes will not get the dlm lock again, the current node's process will be blocked when acquire this dlm lock again.

---
 fs/ocfs2/dlmglue.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
fs/ocfs2/dlmglue.c +9 -4
index 48fd369c29a4..c454c218fbfe 100644
@@ -3671,13 +3671,11 @@ static int ocfs2_downconvert_lock(struct ocfs2_super *osb,
 			     OCFS2_LOCK_ID_MAX_LEN - 1);
 	lockres_clear_pending(lockres, generation, osb);
 	if (ret) {
-		ocfs2_log_dlm_error("ocfs2_dlm_lock", ret, lockres);
+		if (ret != -EBUSY)
+			ocfs2_log_dlm_error("ocfs2_dlm_lock", ret, lockres);
 		ocfs2_recover_from_dlm_error(lockres, 1);
-		goto bail;
 	}
 
-	ret = 0;
-bail:
 	return ret;
 }
 
@@ -3912,6 +3910,13 @@ static int ocfs2_unblock_lock(struct ocfs2_super *osb,
 	spin_unlock_irqrestore(&lockres->l_lock, flags);
 	ret = ocfs2_downconvert_lock(osb, lockres, new_level, set_lvb,
 				     gen);
+	/* ocfs2_cancel_convert() is in progress, try again later */
+	if (ret == -EBUSY) {
+		ctl->requeue = 1;
+		mlog(ML_BASTS, "lockres %s, ReQ: Downconvert busy\n",
+		     lockres->l_name);
+		ret = 0;
+	}
 
 leave:
 	if (ret)
Signature · 1 line
2.12.3
maintainer
#1 lore raw
Quoted 2 lines · Gang He

Usually, ocfs2_downconvert_lock() function always downconverts dlm lock to the expected level for satisfy dlm bast requests

s/for/to

Quoted 20 lines

from the other nodes.
But there is a rare situation. When dlm lock conversion is being canceled, ocfs2_downconvert_lock() function will return -EBUSY. You need to be aware that ocfs2_cancel_convert() function is asynchronous in fsdlm implementation.
If we does not requeue this lockres entry, ocfs2 downconvert thread no longer handles this dlm lock bast request. Then, the other nodes will not get the dlm lock again, the current node's process will be blocked when acquire this dlm lock again.

---
 fs/ocfs2/dlmglue.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
fs/ocfs2/dlmglue.c +0 -0
index 48fd369c29a4..c454c218fbfe 100644

fs/ocfs2/dlmglue.c static int ocfs2_downconvert_lock(struct ocfs2_super *osb, · line 3675

Quoted 6 lines
fs/ocfs2/dlmglue.c +2 -1
@@ -3671,13 +3671,11 @@ static int ocfs2_downconvert_lock(struct ocfs2_super *osb,
 			     OCFS2_LOCK_ID_MAX_LEN - 1);
 	lockres_clear_pending(lockres, generation, osb);
 	if (ret) {
-		ocfs2_log_dlm_error("ocfs2_dlm_lock", ret, lockres);
+		if (ret != -EBUSY)
+			ocfs2_log_dlm_error("ocfs2_dlm_lock", ret, lockres);

Do we have to treat EBUSY as a normal case?

Quoted 10 lines
fs/ocfs2/dlmglue.c +0 -3
 		ocfs2_recover_from_dlm_error(lockres, 1);
-		goto bail;
 	}
 
-	ret = 0;
-bail:
 	return ret;
 }
 

fs/ocfs2/dlmglue.c static int ocfs2_unblock_lock(struct ocfs2_super *osb, · line 3918

Quoted 9 lines
fs/ocfs2/dlmglue.c +6 -0
@@ -3912,6 +3910,13 @@ static int ocfs2_unblock_lock(struct ocfs2_super *osb,
 	spin_unlock_irqrestore(&lockres->l_lock, flags);
 	ret = ocfs2_downconvert_lock(osb, lockres, new_level, set_lvb,
 				     gen);
+	/* ocfs2_cancel_convert() is in progress, try again later */
+	if (ret == -EBUSY) {
+		ctl->requeue = 1;
+		mlog(ML_BASTS, "lockres %s, ReQ: Downconvert busy\n",
+		     lockres->l_name);
+		ret = 0;

Ditto. If EBUSY is not a normal case, I'd like just requeue it but not change it to normal return code.
You know ML_BASTS is always switched off in production environment.

Thanks,
Joseph

Quoted 5 lines
fs/ocfs2/dlmglue.c +1 -0
+	}
 
 leave:
 	if (ret)
author
#2 lore raw

Hi Joseph,

Quoted 23 lines · Joseph Qi
Quoted 2 lines · Gang He

Usually, ocfs2_downconvert_lock() function always downconverts dlm lock to the expected level for satisfy dlm bast requests

s/for/to

Quoted 15 lines

from the other nodes.
But there is a rare situation. When dlm lock conversion is being canceled, ocfs2_downconvert_lock() function will return -EBUSY. You need to be aware that ocfs2_cancel_convert() function is asynchronous in fsdlm implementation.
If we does not requeue this lockres entry, ocfs2 downconvert thread no longer handles this dlm lock bast request. Then, the other nodes will not get the dlm lock again, the current node's process will be blocked when acquire this dlm lock again.

---
  fs/ocfs2/dlmglue.c | 13 +++++++++----
  1 file changed, 9 insertions(+), 4 deletions(-)
Quoted 13 lines

fs/ocfs2/dlmglue.c static int ocfs2_downconvert_lock(struct ocfs2_super *osb, · line 3675

Quoted 11 lines
fs/ocfs2/dlmglue.c +2 -1
index 48fd369c29a4..c454c218fbfe 100644
@@ -3671,13 +3671,11 @@ static int ocfs2_downconvert_lock(struct ocfs2_super *osb,
  			     OCFS2_LOCK_ID_MAX_LEN - 1);
  	lockres_clear_pending(lockres, generation, osb);
  	if (ret) {
-		ocfs2_log_dlm_error("ocfs2_dlm_lock", ret, lockres);
+		if (ret != -EBUSY)
+			ocfs2_log_dlm_error("ocfs2_dlm_lock", ret, lockres);

Do we have to treat EBUSY as a normal case?

Yes, this return code is expected when call dlm_lock() to convert a dlm lock to another level, but this dlm lock is being cancelled.
As I said in another mail, for fsdlm implementation,ocfs2_cancel_convert
will return immediately, but the related dlm lock will(is) be cancelled in background. For o2dlm implementation,ocfs2_cancel_convert will return after the dlm lock is cancelled and it's ast is invoked, that is why o2cb based ocfs2 does not encounter -EBUSY return code in my test script, of course, this kind of implementation will block other lockres entries to be handled for a little time in down-convert thread.

Quoted 10 lines
Quoted 10 lines
fs/ocfs2/dlmglue.c +0 -3
  		ocfs2_recover_from_dlm_error(lockres, 1);
-		goto bail;
  	}
 
-	ret = 0;
-bail:
  	return ret;
  }
 
Quoted 14 lines

fs/ocfs2/dlmglue.c static int ocfs2_unblock_lock(struct ocfs2_super *osb, · line 3918

Quoted 10 lines
fs/ocfs2/dlmglue.c +6 -0
@@ -3912,6 +3910,13 @@ static int ocfs2_unblock_lock(struct ocfs2_super *osb,
  	spin_unlock_irqrestore(&lockres->l_lock, flags);
  	ret = ocfs2_downconvert_lock(osb, lockres, new_level, set_lvb,
  				     gen);
+	/* ocfs2_cancel_convert() is in progress, try again later */
+	if (ret == -EBUSY) {
+		ctl->requeue = 1;
+		mlog(ML_BASTS, "lockres %s, ReQ: Downconvert busy\n",
+		     lockres->l_name);
+		ret = 0;

Ditto. If EBUSY is not a normal case, I'd like just requeue it but not change it to normal return code.
You know ML_BASTS is always switched off in production environment.

Since this case should be considered as a normal case, although it's rare. We should not print any error message to kernel journal, but if the user turn on the BASTS trace, he should watch this trace for debugging.

Thanks
Gang

Quoted 10 lines

Thanks,
Joseph

Quoted 5 lines
fs/ocfs2/dlmglue.c +1 -0
+	}
 
  leave:
  	if (ret)
maintainer
#3 lore raw
Quoted 39 lines · Gang He

Hi Joseph,

Quoted 23 lines · Joseph Qi
Quoted 2 lines · Gang He

Usually, ocfs2_downconvert_lock() function always downconverts dlm lock to the expected level for satisfy dlm bast requests

s/for/to

Quoted 15 lines

from the other nodes.
But there is a rare situation. When dlm lock conversion is being canceled, ocfs2_downconvert_lock() function will return -EBUSY. You need to be aware that ocfs2_cancel_convert() function is asynchronous in fsdlm implementation.
If we does not requeue this lockres entry, ocfs2 downconvert thread no longer handles this dlm lock bast request. Then, the other nodes will not get the dlm lock again, the current node's process will be blocked when acquire this dlm lock again.

---
  fs/ocfs2/dlmglue.c | 13 +++++++++----

  1 file changed, 9 insertions(+), 4 deletions(-)
Quoted 13 lines

fs/ocfs2/dlmglue.c static int ocfs2_downconvert_lock(struct ocfs2_super *osb, · line 3675

Quoted 11 lines
fs/ocfs2/dlmglue.c +2 -1
index 48fd369c29a4..c454c218fbfe 100644
@@ -3671,13 +3671,11 @@ static int ocfs2_downconvert_lock(struct ocfs2_super *osb,
                    OCFS2_LOCK_ID_MAX_LEN - 1);
       lockres_clear_pending(lockres, generation, osb);
       if (ret) {
-        ocfs2_log_dlm_error("ocfs2_dlm_lock", ret, lockres);
+        if (ret != -EBUSY)
+            ocfs2_log_dlm_error("ocfs2_dlm_lock", ret, lockres);

Do we have to treat EBUSY as a normal case?

Quoted 3 lines

Yes, this return code is expected when call dlm_lock() to convert a dlm lock to another level, but this dlm lock is being cancelled.
As I said in another mail, for fsdlm implementation,ocfs2_cancel_convert
will return immediately, but the related dlm lock will(is) be cancelled in background. For o2dlm implementation,ocfs2_cancel_convert will return after the dlm lock is cancelled and it's ast is invoked, that is why o2cb based ocfs2 does not encounter -EBUSY return code in my test script, of course, this kind of implementation will block other lockres entries to be handled for a little time in down-convert thread.

Better to leave this log for later issue tracking. I'm worrying about if there are other cases here.

Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
fs/ocfs2/dlmglue.c +0 -2
           ocfs2_recover_from_dlm_error(lockres, 1);
-        goto bail;
       }
   -    ret = 0;
-bail:
       return ret;
   }
   @@ -3912,6 +3910,13 @@ static int ocfs2_unblock_lock(struct ocfs2_super *osb,

      spin_unlock_irqrestore(&lockres->l_lock, flags); ret = ocfs2_downconvert_lock(osb, lockres, new_level, set_lvb, gen);
+    /* ocfs2_cancel_convert() is in progress, try again later */ +    if (ret == -EBUSY) {
+        ctl->requeue = 1;
+        mlog(ML_BASTS, "lockres %s, ReQ: Downconvert busy\n", +             lockres->l_name);
+        ret = 0;

Quoted 7 lines
Quoted 3 lines

Ditto. If EBUSY is not a normal case, I'd like just requeue it but not change it to normal return code.
You know ML_BASTS is always switched off in production environment.

Since this case should be considered as a normal case, although it's rare. We should not print any error message to kernel journal, but if the user turn on the BASTS trace, he should watch this trace for debugging.

Okay, since we leave an error message above, we can return normal to caller. And now caller only print a simple error which doesn't have much meaning.

BTW, could we change it like:

ret = ocfs2_downconvert_lock();

if (ret == -EBUSY) {
	mlog(ML_BASTS, ...);
	/* Describe the case why we have to requeue */
	goto requeue;
}

...
requeue:

	ctl->requeue = 1;
	return 0;
author
#4 lore raw
Quoted 42 lines · Joseph Qi
Quoted 39 lines · Gang He

Hi Joseph,

Quoted 36 lines · Joseph Qi
Quoted 2 lines · Gang He

Usually, ocfs2_downconvert_lock() function always downconverts dlm lock to the expected level for satisfy dlm bast requests

s/for/to

Quoted 20 lines

from the other nodes.
But there is a rare situation. When dlm lock conversion is being canceled, ocfs2_downconvert_lock() function will return -EBUSY. You need to be aware that ocfs2_cancel_convert() function is asynchronous in fsdlm implementation.
If we does not requeue this lockres entry, ocfs2 downconvert thread no longer handles this dlm lock bast request. Then, the other nodes will not get the dlm lock again, the current node's process will be blocked when acquire this dlm lock again.

---
   fs/ocfs2/dlmglue.c | 13 +++++++++----
   1 file changed, 9 insertions(+), 4 deletions(-)
fs/ocfs2/dlmglue.c +0 -0
index 48fd369c29a4..c454c218fbfe 100644

fs/ocfs2/dlmglue.c static int ocfs2_downconvert_lock(struct ocfs2_super *osb, · line 3675

Quoted 6 lines
fs/ocfs2/dlmglue.c +2 -1
@@ -3671,13 +3671,11 @@ static int ocfs2_downconvert_lock(struct ocfs2_super *osb,
                    OCFS2_LOCK_ID_MAX_LEN - 1);
       lockres_clear_pending(lockres, generation, osb);
       if (ret) {
-        ocfs2_log_dlm_error("ocfs2_dlm_lock", ret, lockres);
+        if (ret != -EBUSY)
+            ocfs2_log_dlm_error("ocfs2_dlm_lock", ret, lockres);

Do we have to treat EBUSY as a normal case?

Quoted 6 lines
Quoted 3 lines

Yes, this return code is expected when call dlm_lock() to convert a dlm lock to another level, but this dlm lock is being cancelled.
As I said in another mail, for fsdlm implementation,ocfs2_cancel_convert
will return immediately, but the related dlm lock will(is) be cancelled in background. For o2dlm implementation,ocfs2_cancel_convert will return after the dlm lock is cancelled and it's ast is invoked, that is why o2cb based ocfs2 does not encounter -EBUSY return code in my test script, of course, this kind of implementation will block other lockres entries to be handled for a little time in down-convert thread.

Better to leave this log for later issue tracking. I'm worrying about if there are other cases here.

OK

Quoted 33 lines
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
fs/ocfs2/dlmglue.c +6 -2
           ocfs2_recover_from_dlm_error(lockres, 1);
-        goto bail;
       }
   -    ret = 0;
-bail:
       return ret;
   }
   @@ -3912,6 +3910,13 @@ static int ocfs2_unblock_lock(struct ocfs2_super *osb,
       spin_unlock_irqrestore(&lockres->l_lock, flags);
       ret = ocfs2_downconvert_lock(osb, lockres, new_level, set_lvb,
                        gen);
+    /* ocfs2_cancel_convert() is in progress, try again later */
+    if (ret == -EBUSY) {
+        ctl->requeue = 1;
+        mlog(ML_BASTS, "lockres %s, ReQ: Downconvert busy\n",
+             lockres->l_name);
+        ret = 0;
Quoted 7 lines
Quoted 3 lines

Ditto. If EBUSY is not a normal case, I'd like just requeue it but not change it to normal return code.
You know ML_BASTS is always switched off in production environment.

Since this case should be considered as a normal case, although it's rare. We should not print any error message to kernel journal, but if the user turn on the BASTS trace, he should watch this trace for debugging.

Okay, since we leave an error message above, we can return normal to caller. And now caller only print a simple error which doesn't have much meaning.

Quoted 8 lines

BTW, could we change it like:

ret = ocfs2_downconvert_lock();

if (ret == -EBUSY) {
	mlog(ML_BASTS, ...);
	/* Describe the case why we have to requeue */
	goto requeue;
}

According to the current code, it is easy to goto leave label with set ctl->requeue = 1 for requeue;

Thanks
Gang

Quoted 7 lines

...
requeue:

	ctl->requeue = 1;
	return 0;