Don't set MD_BROKEN on failfast bio failure
v5 cover Don't set MD_BROKEN on failfast bio failure
Changes from V4:
- Use device_lock to serialize md_error() instead of adding a new spinlock_t.
- Rename new function md_bio_failure_error() to md_cond_error().
- Add helper function pers->should_error() to determine whether to fail rdev in failfast bio failure, instead of using the LastDev flag.
- Avoid changing the behavior of the LastDev flag.
- Drop fix for R{1,10}BIO_Uptodate not being set despite successful retry; this will be sent separately after Nan's refactor.
- Drop fix for the message 'Operation continuing on 0 devices'; as it is outside the scope of this patch, it will be sent separately.
- Improve logging when metadata writing fails.
- Rename LastDev to RetryingSBWrite.
Changes from V3:
- The error handling in md_error() is now serialized, and a new helper function, md_bio_failure_error, has been introduced.
- MD_FAILFAST bio failures are now processed by md_bio_failure_error instead of signaling via FailfastIOFailure.
- RAID10: Fix missing reschedule of failfast read bio failure
- Regardless of failfast, in narrow_write_error, writes that succeed in retry are returned to the higher layer as success
Changes from V2:
- Fix to prevent the array from being marked broken for all Failfast IOs, not just metadata.
- Reflecting the review, update raid{1,10}_error to clear FailfastIOFailure so that devices are properly marked Faulty.
Changes from V1:
- Avoid setting MD_BROKEN instead of clearing it
- Add pr_crit() when setting MD_BROKEN
- Fix the message may shown after all rdevs failure:
"Operation continuing on 0 devices"
v4: https://lore.kernel.org/linux-raid/[email protected]/
v3: https://lore.kernel.org/linux-raid/[email protected]/
v2: https://lore.kernel.org/linux-raid/[email protected]/
v1: https://lore.kernel.org/linux-raid/[email protected]/
When multiple MD_FAILFAST bios fail simultaneously on Failfast-enabled rdevs in RAID1/RAID10, the following issues can occur:
* MD_BROKEN is set and the array halts, even though this should not occur under the intended Failfast design.
* Writes retried through narrow_write_error succeed, but the I/O is still reported as BLK_STS_IOERR
* NOTE: a fix for this was removed in v5, will be send separetely
https://lore.kernel.org/linux-raid/[email protected]/
* RAID10 only: If a Failfast read I/O fails, it is not retried on any remaining rdev, and as a result, the upper layer receives an I/O error.
Simultaneous bio failures across multiple rdevs are uncommon; however, rdevs serviced via nvme-tcp can still experience them due to something as simple as an Ethernet fault. The issue can be reproduced using the following steps.
# prepare nvmet/nvme-tcp and md array #
sh-5.2# cat << 'EOF' > loopback-nvme.sh
set -eu
nqn="nqn.2025-08.io.example:nvmet-test-$1"
back=$2
cd /sys/kernel/config/nvmet/
mkdir subsystems/$nqn
echo 1 > subsystems/${nqn}/attr_allow_any_host
mkdir subsystems/${nqn}/namespaces/1
echo -n ${back} > subsystems/${nqn}/namespaces/1/device_path
echo 1 > subsystems/${nqn}/namespaces/1/enable
ports="ports/1"
if [ ! -d $ports ]; then
mkdir $ports
cd $ports
echo 127.0.0.1 > addr_traddr
echo tcp > addr_trtype
echo 4420 > addr_trsvcid
echo ipv4 > addr_adrfam
cd ../../
fi
ln -s /sys/kernel/config/nvmet/subsystems/${nqn} ${ports}/subsystems/
nvme connect -t tcp -n $nqn -a 127.0.0.1 -s 4420
EOFsh-5.2# chmod +x loopback-nvme.sh
sh-5.2# modprobe -a nvme-tcp nvmet-tcp
sh-5.2# truncate -s 1g a.img b.img
sh-5.2# losetup --show -f a.img/dev/loop0
sh-5.2# losetup --show -f b.img/dev/loop1
sh-5.2# ./loopback-nvme.sh 0 /dev/loop0
connecting to device: nvme0
sh-5.2# ./loopback-nvme.sh 1 /dev/loop1
connecting to device: nvme1
sh-5.2# mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 \
--failfast /dev/nvme0n1 --failfast /dev/nvme1n1
...
mdadm: array /dev/md0 started.
# run fio # sh-5.2# fio --name=test --filename=/dev/md0 --rw=randrw --rwmixread=50 \ --bs=4k --numjobs=9 --time_based --runtime=300s --group_reporting --direct=1
# It can reproduce the issue by block nvme traffic during fio # sh-5.2# iptables -A INPUT -m tcp -p tcp --dport 4420 -j DROP; sh-5.2# sleep 10; # twice the default KATO value sh-5.2# iptables -D INPUT -m tcp -p tcp --dport 4420 -j DROP
Patch 1-2 serialize the md_error() with device_lock
Patch 3-6 introduce md_cond_error() and dependent helpers Patch 7-8 preparation refactor for patch 11-12
Patch 9 adds the missing retry path for Failfast read errors in RAID10. Patch 10-12 prevents MD_FAILFAST bio failure causing the array to fail; this is what I want to achieve.
Patch 13-14 add the error log when the array stops functioning. Patch 15 simply rename the LastDev flag.
Patch 16 add the mddev and rdev name to the error log when metadata writing fails.
Kenta Akagi (16):
md: move device_lock from conf to mddev
md: serialize md_error()
md: add pers->should_error() callback
md: introduce md_cond_error()
md/raid1: implement pers->should_error()
md/raid10: implement pers->should_error()
md/raid1: refactor handle_read_error()
md/raid10: refactor handle_read_error()
md/raid10: fix failfast read error not rescheduled
md: prevent set MD_BROKEN on super_write failure with failfast md/raid1: Prevent set MD_BROKEN on failfast bio failure md/raid10: Prevent set MD_BROKEN on failfast bio failure md/raid1: add error message when setting MD_BROKEN md/raid10: Add error message when setting MD_BROKEN
md: rename 'LastDev' rdev flag to 'RetryingSBWrite'
md: Improve super_written() error logging
drivers/md/md-linear.c | 1 + drivers/md/md.c | 58 ++++++++++++++-- drivers/md/md.h | 12 +++- drivers/md/raid0.c | 1 + drivers/md/raid1.c | 111 ++++++++++++++++++++----------- drivers/md/raid1.h | 2 - drivers/md/raid10.c | 116 ++++++++++++++++++++------------ drivers/md/raid10.h | 1 - drivers/md/raid5-cache.c | 16 ++--- drivers/md/raid5.c | 139 +++++++++++++++++++-------------------- drivers/md/raid5.h | 1 - 11 files changed, 283 insertions(+), 175 deletions(-)
Signature · 1 line
2.50.1
PATCH v5 1/16 md: move device_lock from conf to mddev
Move device_lock from mddev->private (r1conf, r10conf, r5conf) to the mddev structure in preparation for serializing md_error() and introducing new function that conditional md_error() calls to improvement failfast bio error handling.
This commit only moves the device_lock location with no functional changes. Subsequent commits will serialize md_error() and introduce a new function that calls md_error() conditionally.
--- drivers/md/md.c | 1 + drivers/md/md.h | 2 + drivers/md/raid1.c | 51 +++++++------- drivers/md/raid1.h | 2 - drivers/md/raid10.c | 61 +++++++++-------- drivers/md/raid10.h | 1 - drivers/md/raid5-cache.c | 16 ++--- drivers/md/raid5.c | 139 +++++++++++++++++++-------------------- drivers/md/raid5.h | 1 - 9 files changed, 135 insertions(+), 139 deletions(-)
drivers/md/md.c +1 -0
@@ -760,6 +760,7 @@ int mddev_init(struct mddev *mddev)atomic_set (&mddev->openers,0 );atomic_set (&mddev->sync_seq,0 );spin_lock_init (&mddev->lock); +spin_lock_init (&mddev->device_lock);init_waitqueue_head (&mddev->sb_wait);init_waitqueue_head (&mddev->recovery_wait); mddev->reshape_position = MaxSector;
drivers/md/md.h +2 -0
@@ -543,6 +543,8 @@ struct mddev {/* used for register new sync thread */ struct work_struct sync_work; +spinlock_t device_lock; +/* "lock" protects: * flush_bio transition from NULL to !NULL * rdev superblocks, events
drivers/md/raid1.c +25 -26
174 lines not on this page · raw message
drivers/md/raid1.h +0 -2
@@ -57,8 +57,6 @@ struct r1conf {int raid_disks;int nonrot_disks; -spinlock_t device_lock; -/* list of 'struct r1bio' that need to be processed by raid1d, * whether to retry a read, writeout a resync or recovery * block, or anything else.
drivers/md/raid10.c +30 -31
211 lines not on this page · raw message
drivers/md/raid10.h +0 -1
@@ -29,7 +29,6 @@ struct r10conf {struct mddev *mddev;struct raid10_info *mirrors;struct raid10_info *mirrors_new, *mirrors_old; -spinlock_t device_lock;/* geometry */ struct geom {
drivers/md/raid5-cache.c +8 -8
@@ -1356,7 +1356,7 @@ static void r5l_write_super_and_discard_space(struct r5l_log *log,* r5c_flush_stripe moves stripe from cached list to handle_list. When called, * the stripe must be on r5c_cached_full_stripes or r5c_cached_partial_stripes. * -* must hold conf->device_lock +* must hold conf->mddev->device_lock */ static void r5c_flush_stripe (struct r5conf *conf,struct stripe_head *sh) {
@@ -1366,10 +1366,10 @@ static void r5c_flush_stripe(struct r5conf *conf, struct stripe_head *sh)/* * The stripe is not ON_RELEASE_LIST, so it is safe to call -* raid5_release_stripe() while holding conf->device_lock +* raid5_release_stripe() while holding conf->mddev->device_lock */ BUG_ON (test_bit (STRIPE_ON_RELEASE_LIST , &sh->state)); -lockdep_assert_held (&conf->device_lock); +lockdep_assert_held (&conf->mddev->device_lock);list_del_init (&sh->lru);atomic_inc (&sh->count);
@@ -1396,7 +1396,7 @@ void r5c_flush_cache(struct r5conf *conf, int num)int count;struct stripe_head *sh, *next; -lockdep_assert_held (&conf->device_lock); +lockdep_assert_held (&conf->mddev->device_lock);if (!READ_ONCE (conf->log))return ;
@@ -1455,15 +1455,15 @@ static void r5c_do_reclaim(struct r5conf *conf) stripes_to_flush =-1 ;if (stripes_to_flush >=0 ) { -spin_lock_irqsave (&conf->device_lock, flags); +spin_lock_irqsave (&conf->mddev->device_lock, flags);r5c_flush_cache (conf, stripes_to_flush); -spin_unlock_irqrestore (&conf->device_lock, flags); +spin_unlock_irqrestore (&conf->mddev->device_lock, flags); }/* if log space is tight, flush stripes on stripe_in_journal_list */ if (test_bit (R5C_LOG_TIGHT , &conf->cache_state)) {spin_lock_irqsave (&log->stripe_in_journal_lock, flags); -spin_lock (&conf->device_lock); +spin_lock (&conf->mddev->device_lock);list_for_each_entry (sh, &log->stripe_in_journal_list, r5c) {/* * stripes on stripe_in_journal_list could be in any
@@ -1481,7 +1481,7 @@ static void r5c_do_reclaim(struct r5conf *conf)break ; } } -spin_unlock (&conf->device_lock); +spin_unlock (&conf->mddev->device_lock);spin_unlock_irqrestore (&log->stripe_in_journal_lock, flags); }
drivers/md/raid5.c +69 -70
456 lines not on this page · raw message
drivers/md/raid5.h +0 -1
@@ -668,7 +668,6 @@ struct r5conf {unsigned long cache_state;struct shrinker *shrinker;int pool_size;/* number of disks in stripeheads in pool */ -spinlock_t device_lock;struct disk_info *disks;struct bio_set bio_split;
Signature · 1 line
2.50.1
Re: [PATCH v5 01/16] md: move device_lock from conf to mddev
Quoted 10 lines · Kenta Akagi
Move
device_lockfrommddev->private(r1conf, r10conf, r5conf) to the mddev structure in preparation for serializingmd_error()and introducing new function that conditionalmd_error()calls to improvement failfast bio error handling.This commit only moves the
device_locklocation with no functional changes. Subsequent commits will serializemd_error()and introduce a new function that callsmd_error()conditionally.Signed-off-by Kenta Akagi<[email protected]>
Quoted 11 lines
--- drivers/md/md.c | 1 + drivers/md/md.h | 2 + drivers/md/raid1.c | 51 +++++++------- drivers/md/raid1.h | 2 - drivers/md/raid10.c | 61 +++++++++-------- drivers/md/raid10.h | 1 - drivers/md/raid5-cache.c | 16 ++--- drivers/md/raid5.c | 139 +++++++++++++++++++-------------------- drivers/md/raid5.h | 1 - 9 files changed, 135 insertions(+), 139 deletions(-)
PATCH v5 2/16 md: serialize md_error()
In reply to Kenta AkagiSerialize the md_error() function in preparation for the introduction of a conditional md_error() in a subsequent commit. The conditional md_error() is intended to prevent unintentional setting of MD_BROKEN during RAID1/10 failfast handling.
To enhance the failfast bio error handler, it must verify that the affected rdev is not the last working device before marking it as faulty. Without serialization, a race condition can occur if multiple failfast bios attempt to call the error handler concurrently:
failfast bio1 failfast bio2
--- ---
md_cond_error(md,rdev1,bio) md_cond_error(md,rdev2,bio)
if(!is_degraded(md)) if(!is_degraded(md))
raid1_error(md,rdev1) raid1_error(md,rdev2)
spin_lock(md)
set_faulty(rdev1)
spin_unlock(md)
spin_lock(md)
set_faulty(rdev2)
set_broken(md)
spin_unlock(md)This can unintentionally cause the array to stop in situations where the 'Last' rdev should not be marked as Faulty.
This commit serializes the md_error() function for all RAID personalities to avoid this race condition. Future commits will introduce a conditional md_error() specifically for failfast bio handling.
Serialization is applied to both the standard and conditional md_error() for the following reasons:
- Both use the same error-handling mechanism, so it's clearer to serialize them consistently.
- The md_error() path is cold, meaning serialization has no performance impact.
--- drivers/md/md-linear.c | 1 + drivers/md/md.c | 10 +++++++++- drivers/md/md.h | 1 + drivers/md/raid0.c | 1 + drivers/md/raid1.c | 6 +----- drivers/md/raid10.c | 9 ++------- drivers/md/raid5.c | 4 +--- 7 files changed, 16 insertions(+), 16 deletions(-)
drivers/md/md-linear.c +1 -0
@@ -298,6 +298,7 @@ static void linear_status(struct seq_file *seq, struct mddev *mddev) }static void linear_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {if (!test_and_set_bit (MD_BROKEN , &mddev->flags)) {char *md_name =mdname (mddev);
drivers/md/md.c +9 -1
@@ -8444,7 +8444,8 @@ void md_unregister_thread(struct mddev *mddev, struct md_thread __rcu **threadp) }EXPORT_SYMBOL (md_unregister_thread); -void md_error (struct mddev *mddev,struct md_rdev *rdev) +void _md_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {if (!rdev ||test_bit (Faulty, &rdev->flags))return ;
@@ -8469,6 +8470,13 @@ void md_error(struct mddev *mddev, struct md_rdev *rdev)queue_work (md_misc_wq, &mddev->event_work);md_new_event (); } + +void md_error (struct mddev *mddev,struct md_rdev *rdev) +{ +spin_lock (&mddev->device_lock); +_md_error (mddev, rdev); +spin_unlock (&mddev->device_lock); +}EXPORT_SYMBOL (md_error);/* seq_file implementation /proc/mdstat */
drivers/md/md.h +1 -0
@@ -913,6 +913,7 @@ extern void md_write_start(struct mddev *mddev, struct bio *bi);extern void md_write_inc (struct mddev *mddev,struct bio *bi);extern void md_write_end (struct mddev *mddev);extern void md_done_sync (struct mddev *mddev,int blocks,int ok); +void _md_error (struct mddev *mddev,struct md_rdev *rdev);extern void md_error (struct mddev *mddev,struct md_rdev *rdev);extern void md_finish_reshape (struct mddev *mddev); void md_submit_discard_bio(struct mddev *mddev, struct md_rdev *rdev,
drivers/md/raid0.c +1 -0
@@ -625,6 +625,7 @@ static void raid0_status(struct seq_file *seq, struct mddev *mddev) }static void raid0_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {if (!test_and_set_bit (MD_BROKEN , &mddev->flags)) {char *md_name =mdname (mddev);
drivers/md/raid1.c +1 -5
@@ -1749,11 +1749,9 @@ static void raid1_status(struct seq_file *seq, struct mddev *mddev)* &mddev->fail_last_dev is off. */ static void raid1_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {struct r1conf *conf = mddev->private; -unsigned long flags; - -spin_lock_irqsave (&conf->mddev->device_lock, flags);if (test_bit (In_sync, &rdev->flags) && (conf->raid_disks - mddev->degraded) ==1 ) {
@@ -1761,7 +1759,6 @@ static void raid1_error(struct mddev *mddev, struct md_rdev *rdev)if (!mddev->fail_last_dev) { conf->recovery_disabled = mddev->recovery_disabled; -spin_unlock_irqrestore (&conf->mddev->device_lock, flags);return ; } }
@@ -1769,7 +1766,6 @@ static void raid1_error(struct mddev *mddev, struct md_rdev *rdev)if (test_and_clear_bit (In_sync, &rdev->flags)) mddev->degraded++;set_bit (Faulty, &rdev->flags); -spin_unlock_irqrestore (&conf->mddev->device_lock, flags);/* * if recovery is running, make sure it aborts. */
drivers/md/raid10.c +2 -7
@@ -1993,19 +1993,15 @@ static int enough(struct r10conf *conf, int ignore)* &mddev->fail_last_dev is off. */ static void raid10_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {struct r10conf *conf = mddev->private; -unsigned long flags; - -spin_lock_irqsave (&conf->mddev->device_lock, flags);if (test_bit (In_sync, &rdev->flags) && !enough (conf, rdev->raid_disk)) {set_bit (MD_BROKEN , &mddev->flags); -if (!mddev->fail_last_dev) { -spin_unlock_irqrestore (&conf->mddev->device_lock, flags); +if (!mddev->fail_last_dev)return ; - } }if (test_and_clear_bit (In_sync, &rdev->flags)) mddev->degraded++;
@@ -2015,7 +2011,6 @@ static void raid10_error(struct mddev *mddev, struct md_rdev *rdev)set_bit (Faulty, &rdev->flags);set_mask_bits (&mddev->sb_flags,0 ,BIT (MD_SB_CHANGE_DEVS ) |BIT (MD_SB_CHANGE_PENDING )); -spin_unlock_irqrestore (&conf->mddev->device_lock, flags); pr_crit("md/raid10:%s: Disk failure on %pg, disabling device.\n" "md/raid10:%s: Operation continuing on %d devices.\n", mdname(mddev), rdev->bdev,
drivers/md/raid5.c +1 -3
@@ -2905,15 +2905,14 @@ static void raid5_end_write_request(struct bio *bi) }static void raid5_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {struct r5conf *conf = mddev->private; -unsigned long flags;pr_debug ("raid456: error called\n" );pr_crit ("md/raid:%s: Disk failure on %pg, disabling device.\n" ,mdname (mddev), rdev->bdev); -spin_lock_irqsave (&conf->mddev->device_lock, flags);set_bit (Faulty, &rdev->flags);clear_bit (In_sync, &rdev->flags); mddev->degraded =raid5_calc_degraded (conf);
@@ -2929,7 +2928,6 @@ static void raid5_error(struct mddev *mddev, struct md_rdev *rdev) mdname(mddev), conf->raid_disks - mddev->degraded); } -spin_unlock_irqrestore (&conf->mddev->device_lock, flags);set_bit (MD_RECOVERY_INTR , &mddev->recovery);set_bit (Blocked, &rdev->flags);
Signature · 1 line
2.50.1
Re: [PATCH v5 02/16] md: serialize md_error()
Hi,
Quoted 55 lines · Kenta Akagi
Serialize the
md_error()function in preparation for the introduction of a conditionalmd_error()in a subsequent commit. The conditionalmd_error()is intended to prevent unintentional setting ofMD_BROKENduring RAID1/10 failfast handling.To enhance the failfast bio error handler, it must verify that the affected rdev is not the last working device before marking it as faulty. Without serialization, a race condition can occur if multiple failfast bios attempt to call the error handler concurrently:
failfast bio1 failfast bio2 --- --- md_cond_error(md,rdev1,bio) md_cond_error(md,rdev2,bio) if(!is_degraded(md)) if(!is_degraded(md)) raid1_error(md,rdev1) raid1_error(md,rdev2) spin_lock(md) set_faulty(rdev1) spin_unlock(md) spin_lock(md) set_faulty(rdev2) set_broken(md) spin_unlock(md)This can unintentionally cause the array to stop in situations where the 'Last' rdev should not be marked as Faulty.
This commit serializes the
md_error()function for all RAID personalities to avoid this race condition. Future commits will introduce a conditionalmd_error()specifically for failfast bio handling.Serialization is applied to both the standard and conditional
md_error()for the following reasons:- Both use the same error-handling mechanism, so it's clearer to serialize them consistently.
- Themd_error()path is cold, meaning serialization has no performance impact.Signed-off-by Kenta Akagi <[email protected]>--- drivers/md/md-linear.c | 1 + drivers/md/md.c | 10 +++++++++- drivers/md/md.h | 1 + drivers/md/raid0.c | 1 + drivers/md/raid1.c | 6 +----- drivers/md/raid10.c | 9 ++------- drivers/md/raid5.c | 4 +--- 7 files changed, 16 insertions(+), 16 deletions(-)drivers/md/md-linear.c +0
-0
drivers/md/md-linear.c static void linear_status(struct seq_file *seq, struct mddev *mddev) · line 301
Quoted 4 lines
drivers/md/md-linear.c +1
-0@@ -298,6 +298,7 @@ static void linear_status(struct seq_file *seq, struct mddev *mddev) } static void linear_error(struct mddev *mddev, struct md_rdev *rdev) + __must_hold(&mddev->device_lock)
This is fine, however, I personally prefer lockdep_assert_held(). :)
Otherwise LGTM.
Thanks,
Kuai
Quoted 151 lines
drivers/md/md-linear.c +0
-0{if (!test_and_set_bit (MD_BROKEN , &mddev->flags)) {char *md_name =mdname (mddev);drivers/md/md.c +9
-1@@ -8444,7 +8444,8 @@ void md_unregister_thread(struct mddev *mddev, struct md_thread __rcu **threadp) }EXPORT_SYMBOL (md_unregister_thread); -void md_error (struct mddev *mddev,struct md_rdev *rdev) +void _md_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {if (!rdev ||test_bit (Faulty, &rdev->flags))return ;@@ -8469,6 +8470,13 @@ void md_error(struct mddev *mddev, struct md_rdev *rdev)queue_work (md_misc_wq, &mddev->event_work);md_new_event (); } + +void md_error (struct mddev *mddev,struct md_rdev *rdev) +{ +spin_lock (&mddev->device_lock); +_md_error (mddev, rdev); +spin_unlock (&mddev->device_lock); +}EXPORT_SYMBOL (md_error);/* seq_file implementation /proc/mdstat */ drivers/md/md.h +1
-0@@ -913,6 +913,7 @@ extern void md_write_start(struct mddev *mddev, struct bio *bi);extern void md_write_inc (struct mddev *mddev,struct bio *bi);extern void md_write_end (struct mddev *mddev);extern void md_done_sync (struct mddev *mddev,int blocks,int ok); +void _md_error (struct mddev *mddev,struct md_rdev *rdev);extern void md_error (struct mddev *mddev,struct md_rdev *rdev);extern void md_finish_reshape (struct mddev *mddev); void md_submit_discard_bio(struct mddev *mddev, struct md_rdev *rdev,drivers/md/raid0.c +1
-0@@ -625,6 +625,7 @@ static void raid0_status(struct seq_file *seq, struct mddev *mddev) }static void raid0_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {if (!test_and_set_bit (MD_BROKEN , &mddev->flags)) {char *md_name =mdname (mddev);drivers/md/raid1.c +1
-5@@ -1749,11 +1749,9 @@ static void raid1_status(struct seq_file *seq, struct mddev *mddev)* &mddev->fail_last_dev is off. */ static void raid1_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {struct r1conf *conf = mddev->private; -unsigned long flags; - -spin_lock_irqsave (&conf->mddev->device_lock, flags);if (test_bit (In_sync, &rdev->flags) && (conf->raid_disks - mddev->degraded) ==1 ) {@@ -1761,7 +1759,6 @@ static void raid1_error(struct mddev *mddev, struct md_rdev *rdev)if (!mddev->fail_last_dev) { conf->recovery_disabled = mddev->recovery_disabled; -spin_unlock_irqrestore (&conf->mddev->device_lock, flags);return ; } }@@ -1769,7 +1766,6 @@ static void raid1_error(struct mddev *mddev, struct md_rdev *rdev)if (test_and_clear_bit (In_sync, &rdev->flags)) mddev->degraded++;set_bit (Faulty, &rdev->flags); -spin_unlock_irqrestore (&conf->mddev->device_lock, flags);/* * if recovery is running, make sure it aborts. */ drivers/md/raid10.c +2
-7@@ -1993,19 +1993,15 @@ static int enough(struct r10conf *conf, int ignore)* &mddev->fail_last_dev is off. */ static void raid10_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {struct r10conf *conf = mddev->private; -unsigned long flags; - -spin_lock_irqsave (&conf->mddev->device_lock, flags);if (test_bit (In_sync, &rdev->flags) && !enough (conf, rdev->raid_disk)) {set_bit (MD_BROKEN , &mddev->flags); -if (!mddev->fail_last_dev) { -spin_unlock_irqrestore (&conf->mddev->device_lock, flags); +if (!mddev->fail_last_dev)return ; - } }if (test_and_clear_bit (In_sync, &rdev->flags)) mddev->degraded++;@@ -2015,7 +2011,6 @@ static void raid10_error(struct mddev *mddev, struct md_rdev *rdev)set_bit (Faulty, &rdev->flags);set_mask_bits (&mddev->sb_flags,0 ,BIT (MD_SB_CHANGE_DEVS ) |BIT (MD_SB_CHANGE_PENDING )); -spin_unlock_irqrestore (&conf->mddev->device_lock, flags); pr_crit("md/raid10:%s: Disk failure on %pg, disabling device.\n" "md/raid10:%s: Operation continuing on %d devices.\n", mdname(mddev), rdev->bdev,drivers/md/raid5.c +1
-3@@ -2905,15 +2905,14 @@ static void raid5_end_write_request(struct bio *bi) }static void raid5_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {struct r5conf *conf = mddev->private; -unsigned long flags;pr_debug ("raid456: error called\n" );pr_crit ("md/raid:%s: Disk failure on %pg, disabling device.\n" ,mdname (mddev), rdev->bdev); -spin_lock_irqsave (&conf->mddev->device_lock, flags);set_bit (Faulty, &rdev->flags);clear_bit (In_sync, &rdev->flags); mddev->degraded =raid5_calc_degraded (conf);@@ -2929,7 +2928,6 @@ static void raid5_error(struct mddev *mddev, struct md_rdev *rdev) mdname(mddev), conf->raid_disks - mddev->degraded); } -spin_unlock_irqrestore (&conf->mddev->device_lock, flags);set_bit (MD_RECOVERY_INTR , &mddev->recovery);set_bit (Blocked, &rdev->flags);
Re: [PATCH v5 02/16] md: serialize md_error()
Hi, thank you for reviewing!
Quoted 60 lines · Yu Kuai
Hi,
Quoted 57 lines · Kenta Akagi
Serialize the
md_error()function in preparation for the introduction of a conditionalmd_error()in a subsequent commit. The conditionalmd_error()is intended to prevent unintentional setting ofMD_BROKENduring RAID1/10 failfast handling.To enhance the failfast bio error handler, it must verify that the affected rdev is not the last working device before marking it as faulty. Without serialization, a race condition can occur if multiple failfast bios attempt to call the error handler concurrently:
failfast bio1 failfast bio2 --- --- md_cond_error(md,rdev1,bio) md_cond_error(md,rdev2,bio) if(!is_degraded(md)) if(!is_degraded(md)) raid1_error(md,rdev1) raid1_error(md,rdev2) spin_lock(md) set_faulty(rdev1) spin_unlock(md) spin_lock(md) set_faulty(rdev2) set_broken(md) spin_unlock(md)This can unintentionally cause the array to stop in situations where the 'Last' rdev should not be marked as Faulty.
This commit serializes the
md_error()function for all RAID personalities to avoid this race condition. Future commits will introduce a conditionalmd_error()specifically for failfast bio handling.Serialization is applied to both the standard and conditional
md_error()for the following reasons:- Both use the same error-handling mechanism, so it's clearer to serialize them consistently.
- Themd_error()path is cold, meaning serialization has no performance impact.Signed-off-by Kenta Akagi <[email protected]>--- drivers/md/md-linear.c | 1 + drivers/md/md.c | 10 +++++++++- drivers/md/md.h | 1 + drivers/md/raid0.c | 1 + drivers/md/raid1.c | 6 +----- drivers/md/raid10.c | 9 ++------- drivers/md/raid5.c | 4 +--- 7 files changed, 16 insertions(+), 16 deletions(-)drivers/md/md-linear.c +0
-0@@ -298,6 +298,7 @@ static void linear_status(struct seq_file *seq, struct mddev *mddev) }
Quoted 4 lines
drivers/md/md-linear.c static void linear_status(struct seq_file *seq, struct mddev *mddev) · line 301
Quoted 2 lines
drivers/md/md-linear.c +1
-0static void linear_error(struct mddev *mddev, struct md_rdev *rdev) + __must_hold(&mddev->device_lock)This is fine, however, I personally prefer
lockdep_assert_held(). :)
Certainly seems like lockdep is better. I'll fix it.
By the way, I realized this PATCH can cause a deadlock in the following cases:
1. md_error acquires device_lock. It's interruptible because it's spin_lock.
2. a something - I can't find the specific function or scenario - does call spin_lock_irqsave(&mddev->device_lock);. Since spin_lock_irqsave callspreempt_disable first and then spin_acquire, device_lock will never be released in a non-SMP environment because md_error thread are never scheduled.
It seems like one of the following changes is necessary, what do you think?
- Add a dedicated spinlock_t member for md_error serialize
- Use spin_lock_irqsave instead of spin_lock in md_error
It's not cool, but I think it would be better to add new members.
Thakns,
Akagi
Quoted 158 lines
Otherwise LGTM.
Thanks,
KuaiQuoted 151 lines
drivers/md/md-linear.c +0
-0{if (!test_and_set_bit (MD_BROKEN , &mddev->flags)) {char *md_name =mdname (mddev);drivers/md/md.c +9
-1@@ -8444,7 +8444,8 @@ void md_unregister_thread(struct mddev *mddev, struct md_thread __rcu **threadp) }EXPORT_SYMBOL (md_unregister_thread); -void md_error (struct mddev *mddev,struct md_rdev *rdev) +void _md_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {if (!rdev ||test_bit (Faulty, &rdev->flags))return ;@@ -8469,6 +8470,13 @@ void md_error(struct mddev *mddev, struct md_rdev *rdev)queue_work (md_misc_wq, &mddev->event_work);md_new_event (); } + +void md_error (struct mddev *mddev,struct md_rdev *rdev) +{ +spin_lock (&mddev->device_lock); +_md_error (mddev, rdev); +spin_unlock (&mddev->device_lock); +}EXPORT_SYMBOL (md_error);/* seq_file implementation /proc/mdstat */ drivers/md/md.h +1
-0@@ -913,6 +913,7 @@ extern void md_write_start(struct mddev *mddev, struct bio *bi);extern void md_write_inc (struct mddev *mddev,struct bio *bi);extern void md_write_end (struct mddev *mddev);extern void md_done_sync (struct mddev *mddev,int blocks,int ok); +void _md_error (struct mddev *mddev,struct md_rdev *rdev);extern void md_error (struct mddev *mddev,struct md_rdev *rdev);extern void md_finish_reshape (struct mddev *mddev); void md_submit_discard_bio(struct mddev *mddev, struct md_rdev *rdev,drivers/md/raid0.c +1
-0@@ -625,6 +625,7 @@ static void raid0_status(struct seq_file *seq, struct mddev *mddev) }static void raid0_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {if (!test_and_set_bit (MD_BROKEN , &mddev->flags)) {char *md_name =mdname (mddev);drivers/md/raid1.c +1
-5@@ -1749,11 +1749,9 @@ static void raid1_status(struct seq_file *seq, struct mddev *mddev)* &mddev->fail_last_dev is off. */ static void raid1_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {struct r1conf *conf = mddev->private; -unsigned long flags; - -spin_lock_irqsave (&conf->mddev->device_lock, flags);if (test_bit (In_sync, &rdev->flags) && (conf->raid_disks - mddev->degraded) ==1 ) {@@ -1761,7 +1759,6 @@ static void raid1_error(struct mddev *mddev, struct md_rdev *rdev)if (!mddev->fail_last_dev) { conf->recovery_disabled = mddev->recovery_disabled; -spin_unlock_irqrestore (&conf->mddev->device_lock, flags);return ; } }@@ -1769,7 +1766,6 @@ static void raid1_error(struct mddev *mddev, struct md_rdev *rdev)if (test_and_clear_bit (In_sync, &rdev->flags)) mddev->degraded++;set_bit (Faulty, &rdev->flags); -spin_unlock_irqrestore (&conf->mddev->device_lock, flags);/* * if recovery is running, make sure it aborts. */ drivers/md/raid10.c +2
-7@@ -1993,19 +1993,15 @@ static int enough(struct r10conf *conf, int ignore)* &mddev->fail_last_dev is off. */ static void raid10_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {struct r10conf *conf = mddev->private; -unsigned long flags; - -spin_lock_irqsave (&conf->mddev->device_lock, flags);if (test_bit (In_sync, &rdev->flags) && !enough (conf, rdev->raid_disk)) {set_bit (MD_BROKEN , &mddev->flags); -if (!mddev->fail_last_dev) { -spin_unlock_irqrestore (&conf->mddev->device_lock, flags); +if (!mddev->fail_last_dev)return ; - } }if (test_and_clear_bit (In_sync, &rdev->flags)) mddev->degraded++;@@ -2015,7 +2011,6 @@ static void raid10_error(struct mddev *mddev, struct md_rdev *rdev)set_bit (Faulty, &rdev->flags);set_mask_bits (&mddev->sb_flags,0 ,BIT (MD_SB_CHANGE_DEVS ) |BIT (MD_SB_CHANGE_PENDING )); -spin_unlock_irqrestore (&conf->mddev->device_lock, flags); pr_crit("md/raid10:%s: Disk failure on %pg, disabling device.\n" "md/raid10:%s: Operation continuing on %d devices.\n", mdname(mddev), rdev->bdev,drivers/md/raid5.c +1
-3@@ -2905,15 +2905,14 @@ static void raid5_end_write_request(struct bio *bi) }static void raid5_error (struct mddev *mddev,struct md_rdev *rdev) + __must_hold(&mddev->device_lock) {struct r5conf *conf = mddev->private; -unsigned long flags;pr_debug ("raid456: error called\n" );pr_crit ("md/raid:%s: Disk failure on %pg, disabling device.\n" ,mdname (mddev), rdev->bdev); -spin_lock_irqsave (&conf->mddev->device_lock, flags);set_bit (Faulty, &rdev->flags);clear_bit (In_sync, &rdev->flags); mddev->degraded =raid5_calc_degraded (conf);@@ -2929,7 +2928,6 @@ static void raid5_error(struct mddev *mddev, struct md_rdev *rdev) mdname(mddev), conf->raid_disks - mddev->degraded); } -spin_unlock_irqrestore (&conf->mddev->device_lock, flags);set_bit (MD_RECOVERY_INTR , &mddev->recovery);set_bit (Blocked, &rdev->flags);
PATCH v5 3/16 md: add pers->should_error() callback
In reply to Kenta AkagiThe failfast feature in RAID1 and RAID10 assumes that when md_error() is called, the array remains functional because the last rdev neither fails nor sets MD_BROKEN.
However, the current implementation can cause the array to lose its last in-sync device or be marked as MD_BROKEN, which breaks the assumption and can lead to array failure.
To address this issue, a new handler md_cond_error() will be introduced to ensure that failfast I/O does not mark the array as broken.
As preparation, this commit adds a helper pers->should_error() to determine from outside the personality whether an rdev can fail safely, which is needed by md_cond_error().
--- drivers/md/md.h | 1 + 1 file changed, 1 insertion(+)
drivers/md/md.h +1 -0
@@ -763,6 +763,7 @@ struct md_personality* if appropriate, and should abort recovery if needed */ void (*error_handler)(struct mddev *mddev,struct md_rdev *rdev); +bool (*should_error)(struct mddev *mddev,struct md_rdev *rdev,struct bio *bio);int (*hot_add_disk) (struct mddev *mddev,struct md_rdev *rdev);int (*hot_remove_disk) (struct mddev *mddev,struct md_rdev *rdev);int (*spare_active) (struct mddev *mddev);
Signature · 1 line
2.50.1
Re: [PATCH v5 03/16] md: add pers->should_error() callback
Hi,
Quoted 25 lines · Kenta Akagi
The failfast feature in RAID1 and RAID10 assumes that when
md_error()is called, the array remains functional because the last rdev neither fails nor setsMD_BROKEN.However, the current implementation can cause the array to lose its last in-sync device or be marked as
MD_BROKEN, which breaks the assumption and can lead to array failure.To address this issue, a new handler
md_cond_error()will be introduced to ensure that failfast I/O does not mark the array as broken.As preparation, this commit adds a helper
pers->should_error()to determine from outside the personality whether an rdev can fail safely, which is needed bymd_cond_error().Signed-off-by Kenta Akagi <[email protected]>--- drivers/md/md.h | 1 + 1 file changed, 1 insertion(+)drivers/md/md.h +0
-0
drivers/md/md.h struct md_personality · line 766
Quoted 4 lines
drivers/md/md.h +1
-0@@ -763,6 +763,7 @@ struct md_personality* if appropriate, and should abort recovery if needed */ void (*error_handler)(struct mddev *mddev,struct md_rdev *rdev); +bool (*should_error)(struct mddev *mddev,struct md_rdev *rdev,struct bio *bio);
I think the name is not quite accurate, perhaps error_handler_check()?
Thanks,
Kuai
Quoted 3 lines
drivers/md/md.h +0
-0int (*hot_add_disk) (struct mddev *mddev,struct md_rdev *rdev);int (*hot_remove_disk) (struct mddev *mddev,struct md_rdev *rdev);int (*spare_active) (struct mddev *mddev);
PATCH v5 4/16 md: introduce md_cond_error()
In reply to Kenta AkagiThe failfast feature in RAID1 and RAID10 assumes that when md_error() is called, the array remains functional because the last rdev neither fails nor sets MD_BROKEN.
However, the current implementation can cause the array to lose its last in-sync device or be marked as MD_BROKEN, which breaks the assumption and can lead to array failure.
To address this issue, introduce md_cond_error() that handles failfast bio errors without stopped the array. This function checks whether the array will become inoperable if a rdev fails, and if so, it skips error handling to ensure the array remains in an operational state.
Callers of md_error() will be updated to use this new function in subsequent commits to properly handle failfast scenarios.
--- drivers/md/md.c | 33 +++++++++++++++++++++++++++++++++ drivers/md/md.h | 1 + 2 files changed, 34 insertions(+)
drivers/md/md.c +33 -0
@@ -8479,6 +8479,39 @@ void md_error(struct mddev *mddev, struct md_rdev *rdev) }EXPORT_SYMBOL (md_error); +/** md_cond_error() - conditionally md_error() +* @mddev: affected md device +* @rdev: member device to fail +* @bio: bio whose triggered device failure +* +* Check if the personality wants to fail this rdev for this bio, +* and if so, call _md_error(). +* This function has no different behavior from md_error except +* for the raid1/10 with failfast enabled rdevs. +* +* Returns: %true if rdev already or become Faulty, %false if not. +*/ +bool md_cond_error (struct mddev *mddev,struct md_rdev *rdev,struct bio *bio) +{ +if (WARN_ON_ONCE (!mddev->pers)) +/* return true because we don't want caller to retry */ +return true; + +spin_lock (&mddev->device_lock); + +if (mddev->pers->should_error && + !mddev->pers->should_error (mddev, rdev, bio)) { +spin_unlock (&mddev->device_lock); +return test_bit (Faulty, &rdev->flags); + } + +_md_error (mddev, rdev); +spin_unlock (&mddev->device_lock); + +return !WARN_ON_ONCE (!test_bit (Faulty, &rdev->flags)); +} +EXPORT_SYMBOL (md_cond_error); +/* seq_file implementation /proc/mdstat */ static void status_unused (struct seq_file *seq)
drivers/md/md.h +1 -0
@@ -916,6 +916,7 @@ extern void md_write_end(struct mddev *mddev);extern void md_done_sync (struct mddev *mddev,int blocks,int ok);void _md_error (struct mddev *mddev,struct md_rdev *rdev);extern void md_error (struct mddev *mddev,struct md_rdev *rdev); +extern bool md_cond_error (struct mddev *mddev,struct md_rdev *rdev,struct bio *bio);extern void md_finish_reshape (struct mddev *mddev);void md_submit_discard_bio (struct mddev *mddev,struct md_rdev *rdev,struct bio *bio,sector_t start,sector_t size);
Signature · 1 line
2.50.1
PATCH v5 5/16 md/raid1: implement pers->should_error()
In reply to Kenta AkagiThe failfast feature in RAID1 and RAID10 assumes that when md_error() is called, the array remains functional because the last rdev neither fails nor sets MD_BROKEN.
However, the current implementation can cause the array to lose its last in-sync device or be marked as MD_BROKEN, which breaks the assumption and can lead to array failure.
To address this issue, introduce a new handler, md_cond_error(), to ensure that failfast I/O does not mark the array as broken.
md_cond_error() checks whether a device should be faulted based on pers->should_error(). This commit implements should_error() callback for raid1 personality, which returns true if faulting the specified rdev would cause the mddev to become non-functional.
--- drivers/md/raid1.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
drivers/md/raid1.c +35 -0
@@ -1732,6 +1732,40 @@ static void raid1_status(struct seq_file *seq, struct mddev *mddev)seq_printf (seq,"]" ); } +/** +* raid1_should_error() - Determine if this rdev should be failed +* @mddev: affected md device +* @rdev: member device to check +* @bio: the bio that caused the failure +* +* When failfast bios failure, rdev can fail, but the mddev must not fail. +* This function tells md_cond_error() not to fail rdev if bio is failfast +* and last rdev. +* +* Returns: %false if bio is failfast and rdev is the last in-sync device. +* Otherwise %true - should fail this rdev. +*/ +static bool raid1_should_error (struct mddev *mddev,struct md_rdev *rdev,struct bio *bio) +{ +int i; +struct r1conf *conf = mddev->private; + +if (!(bio->bi_opf &MD_FAILFAST ) || + !test_bit (FailFast, &rdev->flags) || +test_bit (Faulty, &rdev->flags)) +return true; + +for (i =0 ; i < conf->raid_disks; i++) { +struct md_rdev *rdev2 = conf->mirrors[i].rdev; + +if (rdev2 && rdev2 != rdev && +test_bit (In_sync, &rdev2->flags) && + !test_bit (Faulty, &rdev2->flags)) +return true; + } +return false; +} +/** * raid1_error() - RAID1 error handler. * @mddev: affected md device.
@@ -3486,6 +3520,7 @@ static struct md_personality raid1_personality = .free = raid1_free, .status = raid1_status, .error_handler = raid1_error, + .should_error = raid1_should_error, .hot_add_disk = raid1_add_disk, .hot_remove_disk= raid1_remove_disk, .spare_active = raid1_spare_active,
Signature · 1 line
2.50.1
Re: [PATCH v5 05/16] md/raid1: implement pers->should_error()
Hi,
Quoted 41 lines · Kenta Akagi
The failfast feature in RAID1 and RAID10 assumes that when
md_error()is called, the array remains functional because the last rdev neither fails nor setsMD_BROKEN.However, the current implementation can cause the array to lose its last in-sync device or be marked as
MD_BROKEN, which breaks the assumption and can lead to array failure.To address this issue, introduce a new handler,
md_cond_error(), to ensure that failfast I/O does not mark the array as broken.
md_cond_error()checks whether a device should be faulted based onpers->should_error(). This commit implementsshould_error()callback for raid1 personality, which returns true if faulting the specified rdev would cause the mddev to become non-functional.Signed-off-by Kenta Akagi <[email protected]>--- drivers/md/raid1.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)drivers/md/raid1.c +12
-0@@ -1732,6 +1732,40 @@ static void raid1_status(struct seq_file *seq, struct mddev *mddev)seq_printf (seq,"]" ); } +/** +* raid1_should_error() - Determine if this rdev should be failed +* @mddev: affected md device +* @rdev: member device to check +* @bio: the bio that caused the failure +* +* When failfast bios failure, rdev can fail, but the mddev must not fail. +* This function tells md_cond_error() not to fail rdev if bio is failfast +* and last rdev. +* +* Returns: %false if bio is failfast and rdev is the last in-sync device. +* Otherwise %true - should fail this rdev.
drivers/md/raid1.c static void raid1_status(struct seq_file *seq, struct mddev *mddev) · line 1756
Quoted 10 lines
drivers/md/raid1.c +10
-0+*/ +static bool raid1_should_error (struct mddev *mddev,struct md_rdev *rdev,struct bio *bio) +{ +int i; +struct r1conf *conf = mddev->private; + +if (!(bio->bi_opf &MD_FAILFAST ) || + !test_bit (FailFast, &rdev->flags) || +test_bit (Faulty, &rdev->flags)) +return true;
The above checking is the same for raid1 and raid10, so I think it should go to the caller, and you don't need to pass in bio.
drivers/md/raid1.c static void raid1_status(struct seq_file *seq, struct mddev *mddev) · line 1765
Quoted 9 lines
drivers/md/raid1.c +9
-0+ +for (i =0 ; i < conf->raid_disks; i++) { +struct md_rdev *rdev2 = conf->mirrors[i].rdev; + +if (rdev2 && rdev2 != rdev && +test_bit (In_sync, &rdev2->flags) && + !test_bit (Faulty, &rdev2->flags)) +return true; + }
Why not use the same checking from raid1_error()? You can factor out a helper for this. And BTW, now I feel it's better to name the new method like rdev_last_in_sync().
Thanks,
Kuai
Quoted 14 lines
drivers/md/raid1.c +4
-0+return false; +} +/** * raid1_error() - RAID1 error handler. * @mddev: affected md device. @@ -3486,6 +3520,7 @@ static struct md_personality raid1_personality = .free = raid1_free, .status = raid1_status, .error_handler = raid1_error, + .should_error = raid1_should_error, .hot_add_disk = raid1_add_disk, .hot_remove_disk= raid1_remove_disk, .spare_active = raid1_spare_active,
PATCH v5 6/16 md/raid10: implement pers->should_error()
In reply to Kenta AkagiThe failfast feature in RAID1 and RAID10 assumes that when md_error() is called, the array remains functional because the last rdev neither fails nor sets MD_BROKEN.
However, the current implementation can cause the array to lose its last in-sync device or be marked as MD_BROKEN, which breaks the assumption and can lead to array failure.
To address this issue, introduce a new handler, md_cond_error(), to ensure that failfast I/O does not mark the array as broken.
md_cond_error() checks whether a device should be faulted based on pers->should_error(). This commit implements should_error() callback for raid10 personality, which returns true if faulting the specified rdev would cause the mddev to become non-functional.
--- drivers/md/raid10.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
drivers/md/raid10.c +26 -0
@@ -1977,6 +1977,31 @@ static int enough(struct r10conf *conf, int ignore)_enough (conf,1 , ignore); } +/** +* raid10_should_error() - Determine if this rdev should be failed +* @mddev: affected md device +* @rdev: member device to check +* @bio: the bio that caused the failure +* +* When failfast bios failure, rdev can fail, but the mddev must not fail. +* This function tells md_cond_error() not to fail rdev if bio is failfast +* and last rdev. +* +* Returns: %false if bio is failfast and rdev is the last in-sync device. +* Otherwise %true - should fail this rdev. +*/ +static bool raid10_should_error (struct mddev *mddev,struct md_rdev *rdev,struct bio *bio) +{ +struct r10conf *conf = mddev->private; + +if (!(bio->bi_opf &MD_FAILFAST ) || + !test_bit (FailFast, &rdev->flags) || +test_bit (Faulty, &rdev->flags)) +return true; + +return enough (conf, rdev->raid_disk); +} +/** * raid10_error() - RAID10 error handler. * @mddev: affected md device.
@@ -5116,6 +5141,7 @@ static struct md_personality raid10_personality = .free = raid10_free, .status = raid10_status, .error_handler = raid10_error, + .should_error = raid10_should_error, .hot_add_disk = raid10_add_disk, .hot_remove_disk= raid10_remove_disk, .spare_active = raid10_spare_active,
Signature · 1 line
2.50.1
PATCH v5 7/16 md/raid1: refactor handle_read_error()
In reply to Kenta AkagiFor the failfast bio feature, the behavior of handle_read_error() will be changed in a subsequent commit, but refactor it first.
This commit only refactors the code without functional changes. A subsequent commit will replace md_error() with md_cond_error() to implement proper failfast error handling.
--- drivers/md/raid1.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
drivers/md/raid1.c +6 -8
@@ -2675,24 +2675,22 @@ static void handle_read_error(struct r1conf *conf, struct r1bio *r1_bio) r1_bio->bios[r1_bio->read_disk] =NULL ; rdev = conf->mirrors[r1_bio->read_disk].rdev; -if (mddev->ro ==0 - && !test_bit (FailFast, &rdev->flags)) { +if (mddev->ro) { + r1_bio->bios[r1_bio->read_disk] =IO_BLOCKED ; + }else if (test_bit (FailFast, &rdev->flags)) { +md_error (mddev, rdev); + }else {freeze_array (conf,1 );fix_read_error (conf, r1_bio);unfreeze_array (conf); - }else if (mddev->ro ==0 &&test_bit (FailFast, &rdev->flags)) { -md_error (mddev, rdev); - }else { - r1_bio->bios[r1_bio->read_disk] =IO_BLOCKED ; }rdev_dec_pending (rdev, conf->mddev); sector = r1_bio->sector; - bio = r1_bio->master_bio;/* Reuse the old r1_bio so that the IO_BLOCKED settings are preserved */ r1_bio->state =0 ; -raid1_read_request (mddev, bio, r1_bio->sectors, r1_bio); +raid1_read_request (mddev, r1_bio->master_bio, r1_bio->sectors, r1_bio);allow_barrier (conf, sector); }
Signature · 1 line
2.50.1
Re: [PATCH v5 07/16] md/raid1: refactor handle_read_error()
Quoted 11 lines · Kenta Akagi
For the failfast bio feature, the behavior of
handle_read_error()will be changed in a subsequent commit, but refactor it first.This commit only refactors the code without functional changes. A subsequent commit will replace
md_error()withmd_cond_error()to implement proper failfast error handling.Signed-off-by Kenta Akagi<[email protected]>--- drivers/md/raid1.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
PATCH v5 8/16 md/raid10: refactor handle_read_error()
In reply to Kenta AkagiFor the failfast bio feature, the behavior of handle_read_error() will be changed in a subsequent commit, but refactor it first.
This commit only refactors the code without functional changes. A subsequent commit will replace md_error() with md_cond_error() to implement proper failfast error handling.
--- drivers/md/raid10.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/md/raid10.c +5 -4
@@ -2873,14 +2873,15 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio)bio_put (bio); r10_bio->devs[slot].bio =NULL ; -if (mddev->ro) +if (mddev->ro) { r10_bio->devs[slot].bio =IO_BLOCKED ; -else if (!test_bit (FailFast, &rdev->flags)) { + }else if (test_bit (FailFast, &rdev->flags)) { +md_error (mddev, rdev); + }else {freeze_array (conf,1 );fix_read_error (conf, mddev, r10_bio);unfreeze_array (conf); - }else -md_error (mddev, rdev); + }rdev_dec_pending (rdev, mddev); r10_bio->state =0 ;
Signature · 1 line
2.50.1
Re: [PATCH v5 08/16] md/raid10: refactor handle_read_error()
Quoted 11 lines · Kenta Akagi
For the failfast bio feature, the behavior of
handle_read_error()will be changed in a subsequent commit, but refactor it first.This commit only refactors the code without functional changes. A subsequent commit will replace
md_error()withmd_cond_error()to implement proper failfast error handling.Signed-off-by Kenta Akagi<[email protected]>--- drivers/md/raid10.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
PATCH v5 9/16 md/raid10: fix failfast read error not rescheduled
In reply to Kenta Akagiraid10_end_read_request lacks a path to retry when a FailFast IO fails. As a result, when Failfast Read IOs fail on all rdevs, the upper layer receives EIO, without read rescheduled.
Looking at the two commits below, it seems only raid10_end_read_request lacks the failfast read retry handling, while raid1_end_read_request has it. In RAID1, the retry works as expected.
* commit 8d3ca83dcf9c ("md/raid10: add failfast handling for reads.")
* commit 2e52d449bcec ("md/raid1: add failfast handling for reads.")
This commit will make the failfast read bio for the last rdev in raid10 retry if it fails.
--- drivers/md/raid10.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/md/raid10.c +7 -0
@@ -401,6 +401,13 @@ static void raid10_end_read_request(struct bio *bio)* wait for the 'master' bio. */ set_bit (R10BIO_Uptodate, &r10_bio->state); + }else if (test_bit (FailFast, &rdev->flags) && +test_bit (R10BIO_FailFast, &r10_bio->state)) { +/* +* This was a fail-fast read so we definitely +* want to retry +*/ + ; }else if (!raid1_should_handle_error (bio)) { uptodate =1 ; }else {
Signature · 1 line
2.50.1
Re: [PATCH v5 09/16] md/raid10: fix failfast read error not rescheduled
Quoted 13 lines · Kenta Akagi
raid10_end_read_requestlacks a path to retry when a FailFast IO fails. As a result, when Failfast Read IOs fail on all rdevs, the upper layer receives EIO, without read rescheduled.Looking at the two commits below, it seems only
raid10_end_read_requestlacks the failfast read retry handling, whileraid1_end_read_requesthas it. In RAID1, the retry works as expected.
* commit8d3ca83dcf9c("md/raid10: add failfast handling for reads.")
* commit2e52d449bcec("md/raid1: add failfast handling for reads.")This commit will make the failfast read bio for the last rdev in raid10 retry if it fails.
Quoted 6 lines
Fixes 8d3ca83dcf9c ("md/raid10: add failfast handling for reads.")Signed-off-by Kenta Akagi<[email protected]>Reviewed-by Li Nan<[email protected]>--- drivers/md/raid10.c | 7 +++++++ 1 file changed, 7 insertions(+)
PATCH v5 10/16 md: prevent set MD_BROKEN on super_write failure with failfast
In reply to Kenta AkagiFailfast is a feature implemented only for RAID1 and RAID10. It instructs the block device providing the rdev to immediately return a bio error without retrying if any issue occurs. This allows quickly detaching a problematic rdev and minimizes IO latency.
Due to its nature, failfast bios can fail easily, and md must not mark an essential rdev as Faulty or set MD_BROKEN on the array just because a failfast bio failed.
When failfast was introduced, RAID1 and RAID10 were designed to continue operating normally even if md_error was called for the last rdev. However, with the introduction of MD_BROKEN in RAID1/RAID10 in commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), calling md_error for the last rdev now prevents further writes to the array. Despite this, the current failfast error handler still assumes that calling md_error will not break the array.
Normally, this is not an issue because MD_FAILFAST is not set when a bio is issued to the last rdev. However, if the array is not degraded and a bio with MD_FAILFAST has been issued, simultaneous failures could potentially break the array. This is unusual but can happen; for example, this can occur when using NVMe over TCP if all rdevs depend on a single Ethernet link.
In other words, this becomes a problem under the following conditions:
Preconditions:
* Failfast is enabled on all rdevs.
* All rdevs are In_sync - This is a requirement for bio to be submit with MD_FAILFAST.
* At least one bio has been submitted but has not yet completed.
Trigger condition:
* All underlying devices of the rdevs return an error for their failfast bios.
Whether the bio is read or write, eventually both rdevs will be lost. In the write case, md_error is invoked on each rdev through its bi_end_io handler. In the read case, if bio has been issued to multiple rdevs via read_balance, it will be the same as write. Even in the read case where only a single rdev has bio issued, both rdevs will be lost in the following sequence:
1. losing the first rdev triggers a metadata update
2. md_super_write issues the bio with MD_FAILFAST, causing the bio to fail immediately. md_super_write always issues MD_FAILFAST bio if
rdev has FailFast, regardless of whether there are other rdevs or not.
3. md_super_write issued bio failed, so super_written calls md_error on the remaining rdev.
This commit fixes a second of read cases. Ensure that a failfast metadata write does not mark the last rdev as faulty or set MD_BROKEN on the array.
--- drivers/md/md.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/md/md.c +1 -2
@@ -1060,8 +1060,7 @@ static void super_written(struct bio *bio)if (bio->bi_status) {pr_err ("md: %s gets error=%d\n" , __func__,blk_status_to_errno (bio->bi_status)); -md_error (mddev, rdev); -if (!test_bit (Faulty, &rdev->flags) +if (!md_cond_error (mddev, rdev, bio) && (bio->bi_opf &MD_FAILFAST )) {set_bit (MD_SB_NEED_REWRITE , &mddev->sb_flags);set_bit (LastDev, &rdev->flags);
Signature · 1 line
2.50.1
PATCH v5 11/16 md/raid1: Prevent set MD_BROKEN on failfast bio failure
In reply to Kenta AkagiFailfast is a feature implemented only for RAID1 and RAID10. It instructs the block device providing the rdev to immediately return a bio error without retrying if any issue occurs. This allows quickly detaching a problematic rdev and minimizes IO latency.
Due to its nature, failfast bios can fail easily, and md must not mark an essential rdev as Faulty or set MD_BROKEN on the array just because a failfast bio failed.
When failfast was introduced, RAID1 and RAID10 were designed to continue operating normally even if md_error was called for the last rdev. However, with the introduction of MD_BROKEN in RAID1/RAID10
in commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), calling md_error for the last rdev now prevents further writes to the array. Despite this, the current failfast error handler still assumes that calling md_error will not break the array.
Normally, this is not an issue because MD_FAILFAST is not set when a bio is issued to the last rdev. However, if the array is not degraded and a bio with MD_FAILFAST has been issued, simultaneous failures could potentially break the array. This is unusual but can happen; for example, this can occur when using NVMe over TCP if all rdevs depend on a single Ethernet link.
In other words, this becomes a problem under the following conditions:
Preconditions:
* Failfast is enabled on all rdevs.
* All rdevs are In_sync - This is a requirement for bio to be submit with MD_FAILFAST.
* At least one bio has been submitted but has not yet completed.
Trigger condition:
* All underlying devices of the rdevs return an error for their failfast bios.
Whether the bio is read or write, eventually both rdevs will be lost. In the write case, md_error is invoked on each rdev through its bi_end_io handler. In the read case, if bio has been issued to multiple rdevs via read_balance, it will be the same as write. Even in the read case where only a single rdev has bio issued, both rdevs will be lost in the following sequence:
1. losing the first rdev triggers a metadata update
2. md_super_write issues the bio with MD_FAILFAST, causing the bio to fail immediately. md_super_write always issues MD_FAILFAST bio if
rdev has FailFast, regardless of whether there are other rdevs or not.
3. md_super_write issued bio failed, so super_written calls md_error on the remaining rdev.
This commit fixes the write case and first of read cases. Ensure that a failfast bio failure will not cause the last rdev to become faulty or the array to be marked MD_BROKEN.
The second of read, i.e., failure of metadata update, has already been fixed in the previous commit.
--- drivers/md/raid1.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
drivers/md/raid1.c +4 -5
@@ -470,7 +470,7 @@ static void raid1_end_write_request(struct bio *bio) (bio->bi_opf & MD_FAILFAST) && /* We never try FailFast to WriteMostly devices */ !test_bit(WriteMostly, &rdev->flags)) { -md_error (r1_bio->mddev, rdev); +md_cond_error (r1_bio->mddev, rdev, bio); }/*
@@ -2177,8 +2177,7 @@ static int fix_sync_read_error(struct r1bio *r1_bio)if (test_bit (FailFast, &rdev->flags)) {/* Don't try recovering from here - just fail it * ... unless it is the last working device of course */ -md_error (mddev, rdev); -if (test_bit (Faulty, &rdev->flags)) +if (md_cond_error (mddev, rdev, bio))/* Don't try to read from here, but make sure * put_buf does it's thing */
@@ -2671,20 +2670,20 @@ static void handle_read_error(struct r1conf *conf, struct r1bio *r1_bio)*/ bio = r1_bio->bios[r1_bio->read_disk]; -bio_put (bio); r1_bio->bios[r1_bio->read_disk] =NULL ; rdev = conf->mirrors[r1_bio->read_disk].rdev;if (mddev->ro) { r1_bio->bios[r1_bio->read_disk] =IO_BLOCKED ; }else if (test_bit (FailFast, &rdev->flags)) { -md_error (mddev, rdev); +md_cond_error (mddev, rdev, bio); }else {freeze_array (conf,1 );fix_read_error (conf, r1_bio);unfreeze_array (conf); } +bio_put (bio);rdev_dec_pending (rdev, conf->mddev); sector = r1_bio->sector;
Signature · 1 line
2.50.1
PATCH v5 12/16 md/raid10: Prevent set MD_BROKEN on failfast bio failure
In reply to Kenta AkagiFailfast is a feature implemented only for RAID1 and RAID10. It instructs the block device providing the rdev to immediately return a bio error without retrying if any issue occurs. This allows quickly detaching a problematic rdev and minimizes IO latency.
Due to its nature, failfast bios can fail easily, and md must not mark an essential rdev as Faulty or set MD_BROKEN on the array just because a failfast bio failed.
When failfast was introduced, RAID1 and RAID10 were designed to continue operating normally even if md_error was called for the last rdev. However, with the introduction of MD_BROKEN in RAID1/RAID10
in commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), calling md_error for the last rdev now prevents further writes to the array. Despite this, the current failfast error handler still assumes that calling md_error will not break the array.
Normally, this is not an issue because MD_FAILFAST is not set when a bio is issued to the last rdev. However, if the array is not degraded and a bio with MD_FAILFAST has been issued, simultaneous failures could potentially break the array. This is unusual but can happen; for example, this can occur when using NVMe over TCP if all rdevs depend on a single Ethernet link.
In other words, this becomes a problem under the following conditions:
Preconditions:
* Failfast is enabled on all rdevs.
* All rdevs are In_sync - This is a requirement for bio to be submit with MD_FAILFAST.
* At least one bio has been submitted but has not yet completed.
Trigger condition:
* All underlying devices of the rdevs return an error for their failfast bios.
Whether the bio is read or write, eventually both rdevs will be lost. In the write case, md_error is invoked on each rdev through its bi_end_io handler. In the read case, if bio has been issued to multiple rdevs via read_balance, it will be the same as write. Even in the read case where only a single rdev has bio issued, both rdevs will be lost in the following sequence:
1. losing the first rdev triggers a metadata update
2. md_super_write issues the bio with MD_FAILFAST, causing the bio to fail immediately. md_super_write always issues MD_FAILFAST bio if
rdev has FailFast, regardless of whether there are other rdevs or not.
3. md_super_write issued bio failed, so super_written calls md_error on the remaining rdev.
This commit fixes the write case and first of read cases. Ensure that a failfast bio failure will not cause the last rdev to become faulty or the array to be marked MD_BROKEN.
The second of read, i.e., failure of metadata update, has already been fixed in the previous commit.
--- drivers/md/raid10.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/md/raid10.c +4 -4
@@ -497,7 +497,7 @@ static void raid10_end_write_request(struct bio *bio) dec_rdev =0 ;if (test_bit (FailFast, &rdev->flags) && (bio->bi_opf &MD_FAILFAST )) { -md_error (rdev->mddev, rdev); +md_cond_error (rdev->mddev, rdev, bio); }/*
@@ -2434,7 +2434,7 @@ static void sync_request_write(struct mddev *mddev, struct r10bio *r10_bio)continue ; }else if (test_bit (FailFast, &rdev->flags)) {/* Just give up on this device */ -md_error (rdev->mddev, rdev); +md_cond_error (rdev->mddev, rdev, r10_bio->devs[i].bio);continue ; }/* Ok, we need to write this bio, either to correct an
@@ -2877,19 +2877,19 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio)* frozen. */ bio = r10_bio->devs[slot].bio; -bio_put (bio); r10_bio->devs[slot].bio =NULL ;if (mddev->ro) { r10_bio->devs[slot].bio =IO_BLOCKED ; }else if (test_bit (FailFast, &rdev->flags)) { -md_error (mddev, rdev); +md_cond_error (mddev, rdev, bio); }else {freeze_array (conf,1 );fix_read_error (conf, mddev, r10_bio);unfreeze_array (conf); } +bio_put (bio);rdev_dec_pending (rdev, mddev); r10_bio->state =0 ;raid10_read_request (mddev, r10_bio->master_bio, r10_bio, false);
Signature · 1 line
2.50.1
PATCH v5 13/16 md/raid1: add error message when setting MD_BROKEN
In reply to Kenta AkagiOnce MD_BROKEN is set on an array, no further writes can be performed. The user must be informed that the array cannot continue operation.
Add error logging when MD_BROKEN flag is set on raid1 arrays to improve debugging and system administration visibility.
--- drivers/md/raid1.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/md/raid1.c +4 -0
@@ -1790,6 +1790,10 @@ static void raid1_error(struct mddev *mddev, struct md_rdev *rdev)if (test_bit (In_sync, &rdev->flags) && (conf->raid_disks - mddev->degraded) ==1 ) {set_bit (MD_BROKEN , &mddev->flags); +pr_crit ("md/raid1:%s: Disk failure on %pg, this is the last device.\n" +"md/raid1:%s: Cannot continue operation (%d/%d failed).\n" , +mdname (mddev), rdev->bdev, +mdname (mddev), mddev->degraded +1 , conf->raid_disks);if (!mddev->fail_last_dev) { conf->recovery_disabled = mddev->recovery_disabled;
Signature · 1 line
2.50.1
PATCH v5 14/16 md/raid10: Add error message when setting MD_BROKEN
In reply to Kenta AkagiOnce MD_BROKEN is set on an array, no further writes can be performed. The user must be informed that the array cannot continue operation.
Add error logging when MD_BROKEN flag is set on raid10 arrays to improve debugging and system administration visibility.
--- drivers/md/raid10.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/md/raid10.c +4 -0
@@ -2031,6 +2031,10 @@ static void raid10_error(struct mddev *mddev, struct md_rdev *rdev)if (test_bit (In_sync, &rdev->flags) && !enough (conf, rdev->raid_disk)) {set_bit (MD_BROKEN , &mddev->flags); +pr_crit ("md/raid10:%s: Disk failure on %pg, this is the last device.\n" +"md/raid10:%s: Cannot continue operation (%d/%d failed).\n" , +mdname (mddev), rdev->bdev, +mdname (mddev), mddev->degraded +1 , conf->geo.raid_disks);if (!mddev->fail_last_dev)return ;
Signature · 1 line
2.50.1
PATCH v5 15/16 md: rename 'LastDev' rdev flag to 'RetryingSBWrite'
In reply to Kenta AkagiThe rdev LastDev flag is assigned to devices
that have had metadata written with MD_FAILFAST in a Failfast rdev, but whose bio failed but did not become faulty after an md_error. After that, the metadata write is retried by sb_flags MD_SB_NEED_REWRITE, but if rdev LastDev is set at this time, FailFast is not used and LastDev is cleared if the write is successful.
Although it's called LastDev, this rdev flag actually means "metadata write with FailFast failed and a retry is required. Do not use FailFast when retrying for this rdev."
This is different from what we might expect from the name LastDev, and can be confusing when reading the code.
This commit renames the LastDev flag to better reflect its actual behavior for improving readability. The implementation remains unchanged.
--- drivers/md/md.c | 6 +++--- drivers/md/md.h | 7 ++++--- 2 files changed, 7 insertions(+), 6 deletions(-)
drivers/md/md.c +3 -3
@@ -1063,10 +1063,10 @@ static void super_written(struct bio *bio)if (!md_cond_error (mddev, rdev, bio) && (bio->bi_opf &MD_FAILFAST )) {set_bit (MD_SB_NEED_REWRITE , &mddev->sb_flags); -set_bit (LastDev, &rdev->flags); +set_bit (RetryingSBWrite, &rdev->flags); } }else -clear_bit (LastDev, &rdev->flags); +clear_bit (RetryingSBWrite, &rdev->flags);bio_put (bio);
@@ -1119,7 +1119,7 @@ void md_write_metadata(struct mddev *mddev, struct md_rdev *rdev,if (test_bit (MD_FAILFAST_SUPPORTED , &mddev->flags) &&test_bit (FailFast, &rdev->flags) && - !test_bit (LastDev, &rdev->flags)) + !test_bit (RetryingSBWrite, &rdev->flags)) bio->bi_opf |=MD_FAILFAST ;atomic_inc (&mddev->pending_writes);
drivers/md/md.h +4 -3
@@ -282,9 +282,10 @@ enum flag_bits {* It is expects that no bad block log * is present. */ - LastDev,/* Seems to be the last working dev as -* it didn't fail, so don't use FailFast -* any more for metadata + RetryingSBWrite,/* +* metadata write with MD_FAILFAST failed, +* so it is being retried. Failfast +* will not be used during the retry. */ CollisionCheck, /** check if there is collision between raid1
Signature · 1 line
2.50.1
PATCH v5 16/16 md: Improve super_written() error logging
In reply to Kenta AkagiIn the current implementation, when super_write fails, the output log will be like this:
md: super_written gets error=-5
It is unless combined with other logs - e.g. I/O error message from blk, it's impossible to determine which md and rdev are causing the problem, and if the problem occurs on multiple devices, it becomes completely impossible to determine from the logs where the super_write failed.
Also, currently super_written does not output logs when retrying metadata write. If the metadata write fails, the array may be corrupted, but if it is retried and successful, then it is not. The user should be informed if a retry was attempted.
This commit adds output to see which array and which device had the problem, and adds a message to indicate when a metadata write retry is scheduled.
--- drivers/md/md.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/md/md.c +4 -1
@@ -1058,10 +1058,13 @@ static void super_written(struct bio *bio)struct mddev *mddev = rdev->mddev;if (bio->bi_status) { -pr_err ("md: %s gets error=%d\n" , __func__, +pr_err ("md: %s: %pg: %s gets error=%d\n" , +mdname (mddev), rdev->bdev, __func__,blk_status_to_errno (bio->bi_status));if (!md_cond_error (mddev, rdev, bio) && (bio->bi_opf &MD_FAILFAST )) { +pr_warn ("md: %s: %pg: retrying metadata write\n" , +mdname (mddev), rdev->bdev);set_bit (MD_SB_NEED_REWRITE , &mddev->sb_flags);set_bit (RetryingSBWrite, &rdev->flags); }
Signature · 1 line
2.50.1