md/raid5: release batch_last before waiting for another stripe_head
PATCH 1/1 md/raid5: release batch_last before waiting for another stripe_head
When raid5_get_active_stripe is called with a ctx containing a stripe_head in its batch_last pointer, it can cause a deadlock if the task sleeps waiting on another stripe_head to become available. The stripe_head held by batch_last can be blocking the advancement of other stripe_heads, leading to no stripe_heads being released so raid5_get_active_stripe waits forever.
Like with the quiesce state handling earlier in the function, batch_last needs to be released by raid5_get_active_stripe before it waits for another stripe_head.
--- drivers/md/raid5.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/md/raid5.c +7 -0
@@ -854,6 +854,13 @@ struct stripe_head *raid5_get_active_stripe(struct r5conf *conf,set_bit (R5_INACTIVE_BLOCKED , &conf->cache_state);r5l_wake_reclaim (conf->log,0 ); + +/* release batch_last before wait to avoid risk of deadlock */ +if (ctx && ctx->batch_last) { +raid5_release_stripe (ctx->batch_last); + ctx->batch_last =NULL ; + } +wait_event_lock_irq (conf->wait_for_stripe,is_inactive_blocked (conf, hash), *(conf->hash_locks + hash));
Signature · 1 line
2.41.0
Thanks a lot David. Song, as a note, David's patch was tested by a Red Hat customer and it indeed resolved their hit on the deadlock. cc. Laurence Oberman who assisted on that case.
Quoted 40 lines · David Jeffery
When
raid5_get_active_stripeis called with a ctx containing astripe_headin itsbatch_lastpointer, it can cause a deadlock if the task sleeps waiting on anotherstripe_headto become available. Thestripe_headheld bybatch_lastcan be blocking the advancement of otherstripe_heads, leading to nostripe_headsbeing released soraid5_get_active_stripewaits forever.Like with the quiesce state handling earlier in the function,
batch_lastneeds to be released byraid5_get_active_stripebefore it waits for anotherstripe_head.Fixes 3312e6c887fe ("md/raid5: Keep a reference to last stripe_head for batch")Signed-off-by David Jeffery <[email protected]>--- drivers/md/raid5.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/md/raid5.c +7
-1@@ -854,6 +854,13 @@ struct stripe_head *raid5_get_active_stripe(struct r5conf *conf,set_bit (R5_INACTIVE_BLOCKED , &conf->cache_state);r5l_wake_reclaim (conf->log,0 ); + +/* release batch_last before wait to avoid risk of deadlock */ +if (ctx && ctx->batch_last) { +raid5_release_stripe (ctx->batch_last); + ctx->batch_last =NULL ; + } +wait_event_lock_irq (conf->wait_for_stripe,is_inactive_blocked (conf, hash), *(conf->hash_locks + hash)); --2.41.0
Song Liu
maintainerCC Logan.
Quoted 21 lines · John Pittman
Thanks a lot David. Song, as a note, David's patch was tested by a Red Hat customer and it indeed resolved their hit on the deadlock. cc. Laurence Oberman who assisted on that case.
Quoted 14 lines · David Jeffery
When
raid5_get_active_stripeis called with a ctx containing astripe_headin itsbatch_lastpointer, it can cause a deadlock if the task sleeps waiting on anotherstripe_headto become available. Thestripe_headheld bybatch_lastcan be blocking the advancement of otherstripe_heads, leading to nostripe_headsbeing released soraid5_get_active_stripewaits forever.Like with the quiesce state handling earlier in the function,
batch_lastneeds to be released byraid5_get_active_stripebefore it waits for anotherstripe_head.Fixes 3312e6c887fe ("md/raid5: Keep a reference to last stripe_head for batch")Signed-off-by David Jeffery <[email protected]>
Thanks for the fix! I applied it to md-fixes.
Question: How easy/difficult is it to reproduce this issue?
Song
Quoted 26 lines
Quoted 25 lines
--- drivers/md/raid5.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/md/raid5.c +7
-1@@ -854,6 +854,13 @@ struct stripe_head *raid5_get_active_stripe(struct r5conf *conf,set_bit (R5_INACTIVE_BLOCKED , &conf->cache_state);r5l_wake_reclaim (conf->log,0 ); + +/* release batch_last before wait to avoid risk of deadlock */ +if (ctx && ctx->batch_last) { +raid5_release_stripe (ctx->batch_last); + ctx->batch_last =NULL ; + } +wait_event_lock_irq (conf->wait_for_stripe,is_inactive_blocked (conf, hash), *(conf->hash_locks + hash)); --2.41.0
Logan Gunthorpe
Reviewed-byQuoted 24 lines · Song Liu
CC Logan.
Quoted 21 lines · John Pittman
Thanks a lot David. Song, as a note, David's patch was tested by a Red Hat customer and it indeed resolved their hit on the deadlock. cc. Laurence Oberman who assisted on that case.
Quoted 14 lines · David Jeffery
When
raid5_get_active_stripeis called with a ctx containing astripe_headin itsbatch_lastpointer, it can cause a deadlock if the task sleeps waiting on anotherstripe_headto become available. Thestripe_headheld bybatch_lastcan be blocking the advancement of otherstripe_heads, leading to nostripe_headsbeing released soraid5_get_active_stripewaits forever.Like with the quiesce state handling earlier in the function,
batch_lastneeds to be released byraid5_get_active_stripebefore it waits for anotherstripe_head.Fixes 3312e6c887fe ("md/raid5: Keep a reference to last stripe_head for batch")Signed-off-by David Jeffery <[email protected]>
This makes sense to me. Nice catch on the difficult bug.
Logan
Quoted 5 lines · Song Liu
Thanks for the fix! I applied it to md-fixes.
Question: How easy/difficult is it to reproduce this issue?
Hello,
One customer system could trigger it reliably and confirmed the fix, but I haven't had any success recreating it with synthetic workloads on a test system.
David Jeffery
Song Liu
maintainerHi David,
Quoted 5 lines · David Jeffery
Quoted 3 lines · Song Liu
Thanks for the fix! I applied it to md-fixes.
Quoted 8 lines
Quoted 2 lines
Question: How easy/difficult is it to reproduce this issue?
Hello,
One customer system could trigger it reliably and confirmed the fix, but I haven't had any success recreating it with synthetic workloads on a test system.
Thanks for the information!
Song