Skip to contentSkip to conversation

md/raid5: release batch_last before waiting for another stripe_head

PATCH In review David Jeffery 4 participants 6 messages Oct 2, 2023 – Oct 3, 2023 MD RAID

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 md/raid5: release batch_last before waiting for another stripe_head

When raid5_get_active_stripe is called with a ctx containing a stripe_head in its batch_last pointer, it can cause a deadlock if the task sleeps waiting on another stripe_head to become available. The stripe_head held by batch_last can be blocking the advancement of other stripe_heads, leading to no stripe_heads being released so raid5_get_active_stripe waits forever.

Like with the quiesce state handling earlier in the function, batch_last needs to be released by raid5_get_active_stripe before it waits for another stripe_head.

Fixes 3312e6c887fe ("md/raid5: Keep a reference to last stripe_head for batch")
---
 drivers/md/raid5.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/md/raid5.c +7 -0
index 6383723468e5..0644b83fd3f4 100644
@@ -854,6 +854,13 @@ struct stripe_head *raid5_get_active_stripe(struct r5conf *conf,
 
 		set_bit(R5_INACTIVE_BLOCKED, &conf->cache_state);
 		r5l_wake_reclaim(conf->log, 0);
+
+		/* release batch_last before wait to avoid risk of deadlock */
+		if (ctx && ctx->batch_last) {
+			raid5_release_stripe(ctx->batch_last);
+			ctx->batch_last = NULL;
+		}
+
 		wait_event_lock_irq(conf->wait_for_stripe,
 				    is_inactive_blocked(conf, hash),
 				    *(conf->hash_locks + hash));
Signature · 1 line
2.41.0
#1 lore raw

Thanks a lot David. Song, as a note, David's patch was tested by a Red Hat customer and it indeed resolved their hit on the deadlock. cc. Laurence Oberman who assisted on that case.

Quoted 40 lines · David Jeffery

When raid5_get_active_stripe is called with a ctx containing a stripe_head in its batch_last pointer, it can cause a deadlock if the task sleeps waiting on another stripe_head to become available. The stripe_head held by batch_last can be blocking the advancement of other stripe_heads, leading to no stripe_heads being released so raid5_get_active_stripe waits forever.

Like with the quiesce state handling earlier in the function, batch_last needs to be released by raid5_get_active_stripe before it waits for another stripe_head.

Fixes 3312e6c887fe ("md/raid5: Keep a reference to last stripe_head for batch")
---
 drivers/md/raid5.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/md/raid5.c +7 -1
index 6383723468e5..0644b83fd3f4 100644
@@ -854,6 +854,13 @@ struct stripe_head *raid5_get_active_stripe(struct r5conf *conf,
 
                set_bit(R5_INACTIVE_BLOCKED, &conf->cache_state);
                r5l_wake_reclaim(conf->log, 0);
+
+               /* release batch_last before wait to avoid risk of deadlock */
+               if (ctx && ctx->batch_last) {
+                       raid5_release_stripe(ctx->batch_last);
+                       ctx->batch_last = NULL;
+               }
+
                wait_event_lock_irq(conf->wait_for_stripe,
                                    is_inactive_blocked(conf, hash),
                                    *(conf->hash_locks + hash));
--

2.41.0

maintainer
#2 lore raw

CC Logan.

Quoted 21 lines · John Pittman

Thanks a lot David. Song, as a note, David's patch was tested by a Red Hat customer and it indeed resolved their hit on the deadlock. cc. Laurence Oberman who assisted on that case.

Quoted 14 lines · David Jeffery

When raid5_get_active_stripe is called with a ctx containing a stripe_head in its batch_last pointer, it can cause a deadlock if the task sleeps waiting on another stripe_head to become available. The stripe_head held by batch_last can be blocking the advancement of other stripe_heads, leading to no stripe_heads being released so raid5_get_active_stripe waits forever.

Like with the quiesce state handling earlier in the function, batch_last needs to be released by raid5_get_active_stripe before it waits for another stripe_head.

Fixes 3312e6c887fe ("md/raid5: Keep a reference to last stripe_head for batch")

Thanks for the fix! I applied it to md-fixes.

Question: How easy/difficult is it to reproduce this issue?

Song

Quoted 26 lines
Quoted 25 lines
---
 drivers/md/raid5.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/md/raid5.c +7 -1
index 6383723468e5..0644b83fd3f4 100644
@@ -854,6 +854,13 @@ struct stripe_head *raid5_get_active_stripe(struct r5conf *conf,
 
                set_bit(R5_INACTIVE_BLOCKED, &conf->cache_state);
                r5l_wake_reclaim(conf->log, 0);
+
+               /* release batch_last before wait to avoid risk of deadlock */
+               if (ctx && ctx->batch_last) {
+                       raid5_release_stripe(ctx->batch_last);
+                       ctx->batch_last = NULL;
+               }
+
                wait_event_lock_irq(conf->wait_for_stripe,
                                    is_inactive_blocked(conf, hash),
                                    *(conf->hash_locks + hash));
--

2.41.0

Reviewed-by
#3 lore raw
Quoted 24 lines · Song Liu

CC Logan.

Quoted 21 lines · John Pittman

Thanks a lot David. Song, as a note, David's patch was tested by a Red Hat customer and it indeed resolved their hit on the deadlock. cc. Laurence Oberman who assisted on that case.

Quoted 14 lines · David Jeffery

When raid5_get_active_stripe is called with a ctx containing a stripe_head in its batch_last pointer, it can cause a deadlock if the task sleeps waiting on another stripe_head to become available. The stripe_head held by batch_last can be blocking the advancement of other stripe_heads, leading to no stripe_heads being released so raid5_get_active_stripe waits forever.

Like with the quiesce state handling earlier in the function, batch_last needs to be released by raid5_get_active_stripe before it waits for another stripe_head.

Fixes 3312e6c887fe ("md/raid5: Keep a reference to last stripe_head for batch")

This makes sense to me. Nice catch on the difficult bug.

Logan

#4 lore raw
In reply to Song Liu
Quoted 5 lines · Song Liu

Thanks for the fix! I applied it to md-fixes.

Question: How easy/difficult is it to reproduce this issue?

Hello,

One customer system could trigger it reliably and confirmed the fix, but I haven't had any success recreating it with synthetic workloads on a test system.

David Jeffery

maintainer
#5 lore raw

Hi David,

Quoted 5 lines · David Jeffery
Quoted 3 lines · Song Liu

Thanks for the fix! I applied it to md-fixes.

Quoted 8 lines
Quoted 2 lines

Question: How easy/difficult is it to reproduce this issue?

Hello,

One customer system could trigger it reliably and confirmed the fix, but I haven't had any success recreating it with synthetic workloads on a test system.

Thanks for the information!

Song