Skip to contentSkip to conversation

ocfs2/dlm: remove unreasonable BUG_ON()

PATCH New Wengang Wang 2 participants 5 messages May 25, 2010 – Jun 11, 2010 OCFS2

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ocfs2/dlm: remove unreasonable BUG_ON()

A node "down" can happen at any time. When that happens, all locres' owned by the "down" node are move to recovery list. They also are marked as in recovery and the owner are set to "unknown" under dlm->spinlock and res->spinlock.

Any place shouldn't believe the owner is not changed to "unknown" after dropping the lockres and retaking them.

---
 fs/ocfs2/dlm/dlmthread.c |   17 +++++++++--------
 1 files changed, 9 insertions(+), 8 deletions(-)
fs/ocfs2/dlm/dlmthread.c +9 -8
index 11a6d1f..cf86b43 100644
@@ -675,15 +675,16 @@ static int dlm_thread(void *data)
 			 * dirty_list in this gap, but that is ok */
 
 			spin_lock(&res->spinlock);
-			if (res->owner != dlm->node_num) {
-				__dlm_print_one_lock_resource(res);
-				mlog(ML_ERROR, "inprog:%s, mig:%s, reco:%s, dirty:%s\n",
-				     res->state & DLM_LOCK_RES_IN_PROGRESS ? "yes" : "no",
-				     res->state & DLM_LOCK_RES_MIGRATING ? "yes" : "no",
-				     res->state & DLM_LOCK_RES_RECOVERING ? "yes" : "no",
-				     res->state & DLM_LOCK_RES_DIRTY ? "yes" : "no");
+			if (unlikely(res->owner != dlm->node_num)) {
+				/*
+				 * there is chance that the lockres is marked
+				 * as in recovery if a node down happened.
+				 */
+				if (!(res->state & DLM_LOCK_RES_RECOVERING)) {
+					__dlm_print_one_lock_resource(res);
+					BUG();
+				}
 			}
-			BUG_ON(res->owner != dlm->node_num);
 
 			/* it is now ok to move lockreses in these states
 			 * to the dirty list, assuming that they will only be
Signature · 1 line
1.6.6.1
#1 lore raw

PATCH 1/1 ocfs2/dlm: remove unreasonable BUG_ON()

NAK

How did this lockres get into the dirty list? The dlm only adds locks that it owns to that list. And such locks, by definition, can never be in the recovery list.

Quoted 43 lines · Wengang Wang

A node "down" can happen at any time. When that happens, all locres' owned by the "down" node are move to recovery list. They also are marked as in recovery and the owner are set to "unknown" under dlm->spinlock and res->spinlock.

Any place shouldn't believe the owner is not changed to "unknown" after dropping the lockres and retaking them.

---
  fs/ocfs2/dlm/dlmthread.c |   17 +++++++++--------
  1 files changed, 9 insertions(+), 8 deletions(-)
fs/ocfs2/dlm/dlmthread.c +9 -8
index 11a6d1f..cf86b43 100644
@@ -675,15 +675,16 @@ static int dlm_thread(void *data)
  			 * dirty_list in this gap, but that is ok */
 
  			spin_lock(&res->spinlock);
-			if (res->owner != dlm->node_num) {
-				__dlm_print_one_lock_resource(res);
-				mlog(ML_ERROR, "inprog:%s, mig:%s, reco:%s, dirty:%s\n",
-				     res->state&  DLM_LOCK_RES_IN_PROGRESS ? "yes" : "no",
-				     res->state&  DLM_LOCK_RES_MIGRATING ? "yes" : "no",
-				     res->state&  DLM_LOCK_RES_RECOVERING ? "yes" : "no",
-				     res->state&  DLM_LOCK_RES_DIRTY ? "yes" : "no");
+			if (unlikely(res->owner != dlm->node_num)) {
+				/*
+				 * there is chance that the lockres is marked
+				 * as in recovery if a node down happened.
+				 */
+				if (!(res->state&  DLM_LOCK_RES_RECOVERING)) {
+					__dlm_print_one_lock_resource(res);
+					BUG();
+				}
  			}
-			BUG_ON(res->owner != dlm->node_num);
 
  			/* it is now ok to move lockreses in these states
  			 * to the dirty list, assuming that they will only be
#2 lore raw

PATCH 1/1 ocfs2/dlm: remove unreasonable BUG_ON()

Quoted 5 lines · Sunil Mushran

NAK

How did this lockres get into the dirty list? The dlm only adds locks that it owns to that list. And such locks, by definition, can never be in the recovery list.

Yes that my description is not good.

Actually, I hit the BUG_ON(res->owner != dlm->node_num); during some tests.

When an recovery happened, the lockres' that is owned by the "dead" node is marked as in recovery and the owner is set as unknown. But note that a lockres owned by this node can also be marked as in recovery(and owner changed to unknown). That can happen when a migration for the lockres is in progress with the "dead" node. see dlm_clean_master_list().

So it's that the owner changed from dlm->node_num to unknown when the lockres is already on the list.

regards,
wengang.

Quoted 44 lines · Wengang Wang
Quoted 42 lines

A node "down" can happen at any time. When that happens, all locres' owned by the "down" node are move to recovery list. They also are marked as in recovery and the owner are set to "unknown" under dlm->spinlock and res->spinlock.

Any place shouldn't believe the owner is not changed to "unknown" after dropping the lockres and retaking them.

---
 fs/ocfs2/dlm/dlmthread.c |   17 +++++++++--------
 1 files changed, 9 insertions(+), 8 deletions(-)
fs/ocfs2/dlm/dlmthread.c +9 -8
index 11a6d1f..cf86b43 100644
@@ -675,15 +675,16 @@ static int dlm_thread(void *data)
 			 * dirty_list in this gap, but that is ok */
 
 			spin_lock(&res->spinlock);
-			if (res->owner != dlm->node_num) {
-				__dlm_print_one_lock_resource(res);
-				mlog(ML_ERROR, "inprog:%s, mig:%s, reco:%s, dirty:%s\n",
-				     res->state&  DLM_LOCK_RES_IN_PROGRESS ? "yes" : "no",
-				     res->state&  DLM_LOCK_RES_MIGRATING ? "yes" : "no",
-				     res->state&  DLM_LOCK_RES_RECOVERING ? "yes" : "no",
-				     res->state&  DLM_LOCK_RES_DIRTY ? "yes" : "no");
+			if (unlikely(res->owner != dlm->node_num)) {
+				/*
+				 * there is chance that the lockres is marked
+				 * as in recovery if a node down happened.
+				 */
+				if (!(res->state&  DLM_LOCK_RES_RECOVERING)) {
+					__dlm_print_one_lock_resource(res);
+					BUG();
+				}
 			}
-			BUG_ON(res->owner != dlm->node_num);
 
 			/* it is now ok to move lockreses in these states
 			 * to the dirty list, assuming that they will only be
#3 lore raw

PATCH 1/1 ocfs2/dlm: remove unreasonable BUG_ON()

Quoted 12 lines · Wengang Wang
Quoted 6 lines · Sunil Mushran

NAK

How did this lockres get into the dirty list? The dlm only adds locks that it owns to that list. And such locks, by definition, can never be in the recovery list.

Yes that my description is not good.

Actually, I hit the BUG_ON(res->owner != dlm->node_num); during some tests.

Quoted 9 lines

When an recovery happened, the lockres' that is owned by the "dead" node is marked as in recovery and the owner is set as unknown. But note that a lockres owned by this node can also be marked as in recovery(and owner changed to unknown). That can happen when a migration for the lockres is in progress with the "dead" node. see dlm_clean_master_list().

So it's that the owner changed from dlm->node_num to unknown when the lockres is already on the list.

Ok. That needs fixing. But it's a lot more involved than this. I had discussed this with Srini some time back.

#4 lore raw

PATCH 1/1 ocfs2/dlm: remove unreasonable BUG_ON()

Quoted 17 lines · Sunil Mushran
Quoted 16 lines · Wengang Wang
Quoted 5 lines · Sunil Mushran

NAK

How did this lockres get into the dirty list? The dlm only adds locks that it owns to that list. And such locks, by definition, can never be in the recovery list.

Yes that my description is not good.

Actually, I hit the BUG_ON(res->owner != dlm->node_num); during some tests.

When an recovery happened, the lockres' that is owned by the "dead" node is marked as in recovery and the owner is set as unknown. But note that a lockres owned by this node can also be marked as in recovery(and owner changed to unknown). That can happen when a migration for the lockres is in progress with the "dead" node. see dlm_clean_master_list().

Quoted 5 lines
Quoted 2 lines

So it's that the owner changed from dlm->node_num to unknown when the lockres is already on the list.

Ok. That needs fixing. But it's a lot more involved than this. I had discussed this with Srini some time back.

Any detail will follow?

regards,
wengang.