md/raid5: protect batch_head->bm_seq updates
PATCH 1/1 md/raid5: protect batch_head->bm_seq updates
From: Chen Cheng <[email protected]>
bm_seq means "stripe delay to flush until bm_seq <= seq_write".
do_release_stripe() keeps STRIPE_BIT_DELAY stripes on bitmap_list when bm_seq >= seq_write.
after raid5d() flushes bitmap update and ++seq_write, and active_bit_delay() retry to release delayed stripes.
the stripe batch head must carry the newest bm_seq among all member stripes, because the whole batch later released according to the batch head state and bm_seq.
race scenario:
===================
1. cpu0 - sh0->bm_seq=101; cpu1 - sh1->bm_seq=102;
2. both cpu0 and cpu1 read batch_head->bm_seq = 100;
3. cpu1 write 102, and cpu0 overwrite with 101;
the point is, if the head has a lower bm_seq than one of its members, the whole batch could be released before that member's bitmap is flushed.
and the on-disk bitmap not record sh1's changes.
--- drivers/md/raid5.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
drivers/md/raid5.c +6 -7
@@ -980,32 +980,31 @@ static void stripe_add_to_batch_list(struct r5conf *conf,/* * at this point, head's BATCH_READY could be cleared, but we * can still add the stripe to batch list */ list_add (&sh->batch_list, &head->batch_list); -spin_unlock (&head->batch_head->batch_lock); }else { head->batch_head = head; sh->batch_head = head->batch_head;spin_lock (&head->batch_lock);list_add_tail (&sh->batch_list, &head->batch_list); -spin_unlock (&head->batch_lock); } -if (test_and_clear_bit (STRIPE_PREREAD_ACTIVE , &sh->state)) -if (atomic_dec_return (&conf->preread_active_stripes) - <IO_THRESHOLD ) -md_wakeup_thread (conf->mddev->thread); -if (test_and_clear_bit (STRIPE_BIT_DELAY , &sh->state)) {int seq = sh->bm_seq;if (test_bit (STRIPE_BIT_DELAY , &sh->batch_head->state) && sh->batch_head->bm_seq - seq >0 ) seq = sh->batch_head->bm_seq;set_bit (STRIPE_BIT_DELAY , &sh->batch_head->state); sh->batch_head->bm_seq = seq; } +spin_unlock (&head->batch_head->batch_lock); + +if (test_and_clear_bit (STRIPE_PREREAD_ACTIVE , &sh->state)) +if (atomic_dec_return (&conf->preread_active_stripes) + <IO_THRESHOLD ) +md_wakeup_thread (conf->mddev->thread);atomic_inc (&sh->count); unlock_out:unlock_two_stripes (head, sh); out:
Signature · 1 line
2.54.0
Dear Chen,
Thank you for very much.
Am 18.06.26 um 08:55 schrieb Chen Cheng:
Quoted 20 lines
From: Chen Cheng <[email protected]>
bm_seqmeans "stripe delay to flush untilbm_seq<=seq_write".
do_release_stripe()keepsSTRIPE_BIT_DELAYstripes onbitmap_listwhenbm_seq>=seq_write.after
raid5d()flushes bitmap update and ++seq_write, andactive_bit_delay()retry to release delayed stripes.the stripe batch head must carry the newest
bm_seqamong all member stripes, because the whole batch later released according to the batch head state andbm_seq.race scenario:
===================
1. cpu0 - sh0->bm_seq=101; cpu1 - sh1->bm_seq=102;
2. both cpu0 and cpu1 readbatch_head->bm_seq= 100;
3. cpu1 write 102, and cpu0 overwrite with 101;
Quoted 4 lines
the point is, if the head has a lower
bm_seqthan one of its members, the whole batch could be released before that member's bitmap is flushed.
and the on-disk bitmap not record sh1's changes.
It’s a little hard to read. Could you please improve the wording of the last paragraph, and maybe also start each sentence with a capital letter. Maybe also use 75 characters per line.
Do you have a reproducer by any chance?
Quoted 1 line
Signed-off-by Chen Cheng <[email protected]>
Also add a Fixes: tag?
Quoted 37 lines
--- drivers/md/raid5.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)drivers/md/raid5.c +1
-7@@ -980,32 +980,31 @@ static void stripe_add_to_batch_list(struct r5conf *conf,/* * at this point, head's BATCH_READY could be cleared, but we * can still add the stripe to batch list */ list_add (&sh->batch_list, &head->batch_list); -spin_unlock (&head->batch_head->batch_lock); }else { head->batch_head = head; sh->batch_head = head->batch_head;spin_lock (&head->batch_lock);list_add_tail (&sh->batch_list, &head->batch_list); -spin_unlock (&head->batch_lock); } -if (test_and_clear_bit (STRIPE_PREREAD_ACTIVE , &sh->state)) -if (atomic_dec_return (&conf->preread_active_stripes) - <IO_THRESHOLD ) -md_wakeup_thread (conf->mddev->thread); -if (test_and_clear_bit (STRIPE_BIT_DELAY , &sh->state)) {int seq = sh->bm_seq;if (test_bit (STRIPE_BIT_DELAY , &sh->batch_head->state) && sh->batch_head->bm_seq - seq >0 ) seq = sh->batch_head->bm_seq;set_bit (STRIPE_BIT_DELAY , &sh->batch_head->state); sh->batch_head->bm_seq = seq; } +spin_unlock (&head->batch_head->batch_lock);
drivers/md/raid5.c static void stripe_add_to_batch_list(struct r5conf *conf, · line 1010
Quoted 10 lines
drivers/md/raid5.c +5
-0+ +if (test_and_clear_bit (STRIPE_PREREAD_ACTIVE , &sh->state)) +if (atomic_dec_return (&conf->preread_active_stripes) + <IO_THRESHOLD ) +md_wakeup_thread (conf->mddev->thread);atomic_inc (&sh->count); unlock_out:unlock_two_stripes (head, sh); out:
Kind regards,
Paul
Quoted 31 lines · Paul Menzel
Dear Chen,
Thank you for very much.
Am 18.06.26 um 08:55 schrieb Chen Cheng:
Quoted 20 lines
From: Chen Cheng <[email protected]>
bm_seqmeans "stripe delay to flush untilbm_seq<=seq_write".
do_release_stripe()keepsSTRIPE_BIT_DELAYstripes onbitmap_listwhenbm_seq>=seq_write.after
raid5d()flushes bitmap update and ++seq_write, andactive_bit_delay()retry to release delayed stripes.the stripe batch head must carry the newest
bm_seqamong all member stripes, because the whole batch later released according to the batch head state andbm_seq.race scenario:
===================
1. cpu0 - sh0->bm_seq=101; cpu1 - sh1->bm_seq=102;
2. both cpu0 and cpu1 readbatch_head->bm_seq= 100;
3. cpu1 write 102, and cpu0 overwrite with 101;Quoted 4 lines
the point is, if the head has a lower
bm_seqthan one of its members, the whole batch could be released before that member's bitmap is flushed.
and the on-disk bitmap not record sh1's changes.
Quoted 5 lines
It’s a little hard to read. Could you please improve the wording of the last paragraph, and maybe also start each sentence with a capital letter. Maybe also use 75 characters per line.
Do you have a reproducer by any chance?
Hi Paul,
Thanks to review, and , I will follow your advise.
Actually, I have some reproducer to hit KCSAN reports in RAID-5, but not for this one. Because it's reported by sashiko-review bot, and , I think it's a true risk.
I will try to make a reproducer for this case later , after I figure-out the other KCSAN reports.
Quoted 56 lines
Quoted 2 lines
Signed-off-by Chen Cheng <[email protected]>Also add a Fixes: tag?
Quoted 38 lines
---
drivers/md/raid5.c| 13 ++++++-------1 file changed, 6 insertions(+), 7 deletions(-)drivers/md/raid5.c +0
-7@@ -980,32 +980,31 @@ static void stripe_add_to_batch_list(structr5conf *conf, /* * at this point, head's BATCH_READY could be cleared, but we * can still add the stripe to batch list */ list_add(&sh->batch_list, &head->batch_list); - spin_unlock(&head->batch_head->batch_lock); } else { head->batch_head = head; sh->batch_head = head->batch_head; spin_lock(&head->batch_lock); list_add_tail(&sh->batch_list, &head->batch_list); - spin_unlock(&head->batch_lock); } - if (test_and_clear_bit(STRIPE_PREREAD_ACTIVE, &sh->state)) - if (atomic_dec_return(&conf->preread_active_stripes) - < IO_THRESHOLD) - md_wakeup_thread(conf->mddev->thread); - if (test_and_clear_bit(STRIPE_BIT_DELAY, &sh->state)) { int seq = sh->bm_seq;if (
test_bit(STRIPE_BIT_DELAY, &sh->batch_head->state)&&sh->batch_head->bm_seq- seq > 0)
seq =sh->batch_head->bm_seq;set_bit(STRIPE_BIT_DELAY, &sh->batch_head->state);sh->batch_head->bm_seq= seq;
}
+spin_unlock(&head->batch_head->batch_lock);
+Quoted 8 lines
+ if (
test_and_clear_bit(STRIPE_PREREAD_ACTIVE, &sh->state)) + if (atomic_dec_return(&conf->preread_active_stripes)+ <IO_THRESHOLD)
+md_wakeup_thread(conf->mddev->thread);atomic_inc(&sh->count);
unlock_out:
unlock_two_stripes(head, sh);
out:Kind regards,
Paul
Dear Chen,
Am 18.06.26 um 13:26 schrieb Chen Cheng:
Quoted 28 lines · Paul Menzel
Quoted 27 lines
Am 18.06.26 um 08:55 schrieb Chen Cheng:
Quoted 24 lines
From: Chen Cheng <[email protected]>
bm_seqmeans "stripe delay to flush untilbm_seq<=seq_write".
do_release_stripe()keepsSTRIPE_BIT_DELAYstripes onbitmap_listwhenbm_seq>=seq_write.after
raid5d()flushes bitmap update and ++seq_write, andactive_bit_delay()retry to release delayed stripes.the stripe batch head must carry the newest
bm_seqamong all member stripes, because the whole batch later released according to the batch head state andbm_seq.race scenario:
===================
1. cpu0 - sh0->bm_seq=101; cpu1 - sh1->bm_seq=102;
2. both cpu0 and cpu1 readbatch_head->bm_seq= 100;
3. cpu1 write 102, and cpu0 overwrite with 101;the point is, if the head has a lower
bm_seqthan one of its members, the whole batch could be released before that member's bitmap is flushed.
and the on-disk bitmap not record sh1's changes.
Quoted 7 lines
Quoted 5 lines
It’s a little hard to read. Could you please improve the wording of the last paragraph, and maybe also start each sentence with a capital letter. Maybe also use 75 characters per line.
Do you have a reproducer by any chance?
Thanks to review, and , I will follow your advise.
Thank you for reading my comments.
Quoted 3 lines
Actually, I have some reproducer to hit KCSAN reports in RAID-5, but not for this one. Because it's reported by sashiko-review bot, and, I think it's a true risk.
Maybe also mention sashiko-review.
Quoted 2 lines
I will try to make a reproducer for this case later, after I figure-out the other KCSAN reports.
A reproducer is not required, I was just interested, how the issue was found. So don’t spend too much on it or at all.
[…]
Kind regards,
Paul
Quoted 46 lines · Paul Menzel
Dear Chen,
Am 18.06.26 um 13:26 schrieb Chen Cheng:
Quoted 28 lines · Paul Menzel
Quoted 27 lines
Am 18.06.26 um 08:55 schrieb Chen Cheng:
Quoted 24 lines
From: Chen Cheng <[email protected]>
bm_seqmeans "stripe delay to flush untilbm_seq<=seq_write".
do_release_stripe()keepsSTRIPE_BIT_DELAYstripes onbitmap_listwhenbm_seq>=seq_write.after
raid5d()flushes bitmap update and ++seq_write, andactive_bit_delay()retry to release delayed stripes.the stripe batch head must carry the newest
bm_seqamong all member stripes, because the whole batch later released according to the batch head state andbm_seq.race scenario:
===================
1. cpu0 - sh0->bm_seq=101; cpu1 - sh1->bm_seq=102;
2. both cpu0 and cpu1 readbatch_head->bm_seq= 100;
3. cpu1 write 102, and cpu0 overwrite with 101;the point is, if the head has a lower
bm_seqthan one of its members, the whole batch could be released before that member's bitmap is flushed.
and the on-disk bitmap not record sh1's changes.Quoted 7 lines
Quoted 5 lines
It’s a little hard to read. Could you please improve the wording of the last paragraph, and maybe also start each sentence with a capital letter. Maybe also use 75 characters per line.
Do you have a reproducer by any chance?
Thanks to review, and , I will follow your advise.
Thank you for reading my comments.
Quoted 3 lines
Actually, I have some reproducer to hit KCSAN reports in RAID-5, but not for this one. Because it's reported by sashiko-review bot, and, I think it's a true risk.
Quoted 6 lines
Maybe also mention sashiko-review.
Quoted 2 lines
I will try to make a reproducer for this case later, after I figure-out the other KCSAN reports.
A reproducer is not required, I was just interested, how the issue was found. So don’t spend too much on it or at all.
Well , I think a good reproducer has to be :
- Concurrency as much path as possible ,
e.g. reshape concurrency with normal I/O, etc...
Optionally ,
- Run some Sanitizer,
- Inject some fault, e.g. bad block, write error, disk drop...
- Provide some background I/O pressure,
- Do some combinations describes above
Quoted 7 lines
[…]
Kind regards,
Paul