ocfs2/dlm: remove unreasonable BUG_ON()
PATCH 1/1 ocfs2/dlm: remove unreasonable BUG_ON()
A node "down" can happen at any time. When that happens, all locres' owned by the "down" node are move to recovery list. They also are marked as in recovery and the owner are set to "unknown" under dlm->spinlock and res->spinlock.
Any place shouldn't believe the owner is not changed to "unknown" after dropping the lockres and retaking them.
--- fs/ocfs2/dlm/dlmthread.c | 17 +++++++++-------- 1 files changed, 9 insertions(+), 8 deletions(-)
fs/ocfs2/dlm/dlmthread.c +9 -8
@@ -675,15 +675,16 @@ static int dlm_thread(void *data)* dirty_list in this gap, but that is ok */ spin_lock (&res->spinlock); -if (res->owner != dlm->node_num) { -__dlm_print_one_lock_resource (res); -mlog (ML_ERROR ,"inprog:%s, mig:%s, reco:%s, dirty:%s\n" , - res->state &DLM_LOCK_RES_IN_PROGRESS ?"yes" :"no" , - res->state &DLM_LOCK_RES_MIGRATING ?"yes" :"no" , - res->state &DLM_LOCK_RES_RECOVERING ?"yes" :"no" , - res->state &DLM_LOCK_RES_DIRTY ?"yes" :"no" ); +if (unlikely (res->owner != dlm->node_num)) { +/* +* there is chance that the lockres is marked +* as in recovery if a node down happened. +*/ +if (!(res->state &DLM_LOCK_RES_RECOVERING )) { +__dlm_print_one_lock_resource (res); +BUG (); + } } -BUG_ON (res->owner != dlm->node_num);/* it is now ok to move lockreses in these states * to the dirty list, assuming that they will only be
Signature · 1 line
1.6.6.1
Sunil Mushran
NAKPATCH 1/1 ocfs2/dlm: remove unreasonable BUG_ON()
NAK
How did this lockres get into the dirty list? The dlm only adds locks that it owns to that list. And such locks, by definition, can never be in the recovery list.
Quoted 43 lines · Wengang Wang
A node "down" can happen at any time. When that happens, all locres' owned by the "down" node are move to recovery list. They also are marked as in recovery and the owner are set to "unknown" under
dlm->spinlockandres->spinlock.Any place shouldn't believe the owner is not changed to "unknown" after dropping the lockres and retaking them.
Signed-off-by Wengang Wang<[email protected]>--- fs/ocfs2/dlm/dlmthread.c | 17 +++++++++-------- 1 files changed, 9 insertions(+), 8 deletions(-)fs/ocfs2/dlm/dlmthread.c +9
-8@@ -675,15 +675,16 @@ static int dlm_thread(void *data)* dirty_list in this gap, but that is ok */ spin_lock (&res->spinlock); -if (res->owner != dlm->node_num) { -__dlm_print_one_lock_resource (res); -mlog (ML_ERROR ,"inprog:%s, mig:%s, reco:%s, dirty:%s\n" , - res->state&DLM_LOCK_RES_IN_PROGRESS ?"yes" :"no" , - res->state&DLM_LOCK_RES_MIGRATING ?"yes" :"no" , - res->state&DLM_LOCK_RES_RECOVERING ?"yes" :"no" , - res->state&DLM_LOCK_RES_DIRTY ?"yes" :"no" ); +if (unlikely (res->owner != dlm->node_num)) { +/* +* there is chance that the lockres is marked +* as in recovery if a node down happened. +*/ +if (!(res->state&DLM_LOCK_RES_RECOVERING )) { +__dlm_print_one_lock_resource (res); +BUG (); + } } -BUG_ON (res->owner != dlm->node_num);/* it is now ok to move lockreses in these states * to the dirty list, assuming that they will only be
PATCH 1/1 ocfs2/dlm: remove unreasonable BUG_ON()
Quoted 5 lines · Sunil Mushran
NAK
How did this lockres get into the dirty list? The dlm only adds locks that it owns to that list. And such locks, by definition, can never be in the recovery list.
Yes that my description is not good.
Actually, I hit the BUG_ON(res->owner != dlm->node_num); during some tests.
When an recovery happened, the lockres' that is owned by the "dead" node is marked as in recovery and the owner is set as unknown. But note that a lockres owned by this node can also be marked as in recovery(and owner changed to unknown). That can happen when a migration for the lockres is in progress with the "dead" node. see dlm_clean_master_list().
So it's that the owner changed from dlm->node_num to unknown when the lockres is already on the list.
regards,
wengang.
Quoted 44 lines · Wengang Wang
Quoted 42 lines
A node "down" can happen at any time. When that happens, all locres' owned by the "down" node are move to recovery list. They also are marked as in recovery and the owner are set to "unknown" under
dlm->spinlockandres->spinlock.Any place shouldn't believe the owner is not changed to "unknown" after dropping the lockres and retaking them.
Signed-off-by Wengang Wang<[email protected]>--- fs/ocfs2/dlm/dlmthread.c | 17 +++++++++-------- 1 files changed, 9 insertions(+), 8 deletions(-)fs/ocfs2/dlm/dlmthread.c +9
-8@@ -675,15 +675,16 @@ static int dlm_thread(void *data)* dirty_list in this gap, but that is ok */ spin_lock (&res->spinlock); -if (res->owner != dlm->node_num) { -__dlm_print_one_lock_resource (res); -mlog (ML_ERROR ,"inprog:%s, mig:%s, reco:%s, dirty:%s\n" , - res->state&DLM_LOCK_RES_IN_PROGRESS ?"yes" :"no" , - res->state&DLM_LOCK_RES_MIGRATING ?"yes" :"no" , - res->state&DLM_LOCK_RES_RECOVERING ?"yes" :"no" , - res->state&DLM_LOCK_RES_DIRTY ?"yes" :"no" ); +if (unlikely (res->owner != dlm->node_num)) { +/* +* there is chance that the lockres is marked +* as in recovery if a node down happened. +*/ +if (!(res->state&DLM_LOCK_RES_RECOVERING )) { +__dlm_print_one_lock_resource (res); +BUG (); + } } -BUG_ON (res->owner != dlm->node_num);/* it is now ok to move lockreses in these states * to the dirty list, assuming that they will only be
PATCH 1/1 ocfs2/dlm: remove unreasonable BUG_ON()
Quoted 12 lines · Wengang Wang
Quoted 6 lines · Sunil Mushran
NAK
How did this lockres get into the dirty list? The dlm only adds locks that it owns to that list. And such locks, by definition, can never be in the recovery list.
Yes that my description is not good.
Actually, I hit the
BUG_ON(res->owner != dlm->node_num); during some tests.
Quoted 9 lines
When an recovery happened, the lockres' that is owned by the "dead" node is marked as in recovery and the owner is set as unknown. But note that a lockres owned by this node can also be marked as in recovery(and owner changed to unknown). That can happen when a migration for the lockres is in progress with the "dead" node. see
dlm_clean_master_list().So it's that the owner changed from
dlm->node_numto unknown when the lockres is already on the list.
Ok. That needs fixing. But it's a lot more involved than this. I had discussed this with Srini some time back.
PATCH 1/1 ocfs2/dlm: remove unreasonable BUG_ON()
Quoted 17 lines · Sunil Mushran
Quoted 16 lines · Wengang Wang
Quoted 5 lines · Sunil Mushran
NAK
How did this lockres get into the dirty list? The dlm only adds locks that it owns to that list. And such locks, by definition, can never be in the recovery list.
Yes that my description is not good.
Actually, I hit the
BUG_ON(res->owner != dlm->node_num); during some tests.When an recovery happened, the lockres' that is owned by the "dead" node is marked as in recovery and the owner is set as unknown. But note that a lockres owned by this node can also be marked as in recovery(and owner changed to unknown). That can happen when a migration for the lockres is in progress with the "dead" node. see
dlm_clean_master_list().
Quoted 5 lines
Quoted 2 lines
So it's that the owner changed from
dlm->node_numto unknown when the lockres is already on the list.Ok. That needs fixing. But it's a lot more involved than this. I had discussed this with Srini some time back.
Any detail will follow?
regards,
wengang.