Skip to contentSkip to conversation

Don't set MD_BROKEN on failfast bio failure

PATCH In review Kenta Akagi 4 participants 18 messages Jan 5 – Sep 24 MD RAID

lore
Share & tools
lore
#0 lore raw

v6 cover Don't set MD_BROKEN on failfast bio failure

Changes from V5:
- Prevent md being broken when rdev has FailFast, regardless of the bios flag. Thanks to Xiao for the advice:
https://lore.kernel.org/linux-raid/CALTww2_nJqyA99cG9YNarXEB4wimFK=pKy=qrxdkfB60PaUa1w@mail.gmail.com/#t
- Dropping preparation refactor, flag rename, error logging improvement commits
Changes from V4:
- Use device_lock to serialize md_error() instead of adding a new spinlock_t.
- Rename new function md_bio_failure_error() to md_cond_error().
- Add helper function pers->should_error() to determine whether to fail rdev in failfast bio failure, instead of using the LastDev flag.
- Avoid changing the behavior of the LastDev flag.
- Drop fix for R{1,10}BIO_Uptodate not being set despite successful retry; this will be sent separately after Nan's refactor.
- Drop fix for the message 'Operation continuing on 0 devices'; as it is outside the scope of this patch, it will be sent separately.
- Improve logging when metadata writing fails.
- Rename LastDev to RetryingSBWrite.
Changes from V3:
- The error handling in md_error() is now serialized, and a new helper function, md_bio_failure_error, has been introduced.
- MD_FAILFAST bio failures are now processed by md_bio_failure_error instead of signaling via FailfastIOFailure.
- RAID10: Fix missing reschedule of failfast read bio failure
- Regardless of failfast, in narrow_write_error, writes that succeed in retry are returned to the higher layer as success
Changes from V2:
- Fix to prevent the array from being marked broken for all Failfast IOs, not just metadata.
- Reflecting the review, update raid{1,10}_error to clear FailfastIOFailure so that devices are properly marked Faulty.
Changes from V1:
- Avoid setting MD_BROKEN instead of clearing it
- Add pr_crit() when setting MD_BROKEN
- Fix the message may shown after all rdevs failure:
"Operation continuing on 0 devices"

v5: https://lore.kernel.org/linux-raid/[email protected]/
v4: https://lore.kernel.org/linux-raid/[email protected]/
v3: https://lore.kernel.org/linux-raid/[email protected]/
v2: https://lore.kernel.org/linux-raid/[email protected]/
v1: https://lore.kernel.org/linux-raid/[email protected]/

When multiple MD_FAILFAST bios fail simultaneously on Failfast-enabled rdevs in RAID1/RAID10, the following issues can occur:
* MD_BROKEN is set and the array halts, even though this should not occur under the intended Failfast design.
* Writes retried through narrow_write_error succeed, but the I/O is still reported as BLK_STS_IOERR
* NOTE: a fix for this was removed in v5, will be send separetely

    https://lore.kernel.org/linux-raid/[email protected]/

* RAID10 only: If a Failfast read I/O fails, it is not retried on any remaining rdev, and as a result, the upper layer receives an I/O error.

Simultaneous bio failures across multiple rdevs are uncommon; however, rdevs serviced via nvme-tcp can still experience them due to something as simple as an Ethernet fault. The issue can be reproduced using the following steps.

# prepare nvmet/nvme-tcp and md array 
sh-5.2# cat << 'EOF' > loopback-nvme.sh
set -eu
nqn="nqn.2025-08.io.example:nvmet-test-$1"
back=$2
cd /sys/kernel/config/nvmet/
mkdir subsystems/$nqn
echo 1 > subsystems/${nqn}/attr_allow_any_host
mkdir subsystems/${nqn}/namespaces/1
echo -n ${back} > subsystems/${nqn}/namespaces/1/device_path
echo 1 > subsystems/${nqn}/namespaces/1/enable
ports="ports/1"
if [ ! -d $ports ]; then
        mkdir $ports
        cd $ports
        echo 127.0.0.1 > addr_traddr
        echo tcp       > addr_trtype
        echo 4420      > addr_trsvcid
        echo ipv4      > addr_adrfam
        cd ../../
fi
ln -s /sys/kernel/config/nvmet/subsystems/${nqn} ${ports}/subsystems/
nvme connect -t tcp -n $nqn -a 127.0.0.1 -s 4420
EOF

sh-5.2# chmod +x loopback-nvme.sh
sh-5.2# modprobe -a nvme-tcp nvmet-tcp
sh-5.2# truncate -s 1g a.img b.img
sh-5.2# losetup --show -f a.img
/dev/loop0
sh-5.2# losetup --show -f b.img
/dev/loop1
sh-5.2# ./loopback-nvme.sh 0 /dev/loop0
connecting to device: nvme0
sh-5.2# ./loopback-nvme.sh 1 /dev/loop1
connecting to device: nvme1
sh-5.2# mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 \
--failfast /dev/nvme0n1 --failfast /dev/nvme1n1
...
mdadm: array /dev/md0 started.

# run fio 
sh-5.2# fio --name=test --filename=/dev/md0 --rw=randrw --rwmixread=50 \
--bs=4k --numjobs=9 --time_based --runtime=300s --group_reporting --direct=1 &

# It can reproduce the issue by block nvme traffic during fio sh-5.2# iptables -A INPUT -m tcp -p tcp --dport 4420 -j DROP; sh-5.2# sleep 10; # twice the default KATO value sh-5.2# iptables -D INPUT -m tcp -p tcp --dport 4420 -j DROP

Patch 1 prevent array broken when FailFast is set on rdev Patch 2 adds the missing retry path for Failfast read errors in RAID10.

Kenta Akagi (2):
md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast md/raid10: fix failfast read error not rescheduled

 drivers/md/md.c     |  6 ++++--
 drivers/md/raid1.c  |  8 +++++++-
 drivers/md/raid10.c | 15 ++++++++++++++-
 3 files changed, 25 insertions(+), 4 deletions(-)
Signature · 1 line
2.50.1
#1 lore raw

PATCH v6 1/2 md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

After commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), if the error handler is called on the last rdev in RAID1 or RAID10, the MD_BROKEN flag will be set on that mddev.
When MD_BROKEN is set, write bios to the md will result in an I/O error.

This causes a problem when using FailFast.
The current implementation of FailFast expects the array to continue functioning without issues even after calling md_error for the last rdev. Furthermore, due to the nature of its functionality, FailFast may call md_error on all rdevs of the md. Even if retrying I/O on an rdev would succeed, it first calls md_error before retrying.

To fix this issue, this commit ensures that for RAID1 and RAID10, if the last In_sync rdev has the FailFast flag set and the mddev's fail_last_dev is off, the MD_BROKEN flag will not be set on that mddev.

This change impacts userspace. After this commit, If the rdev has the FailFast flag, the mddev never broken even if the failing bio is not FailFast. However, it's unlikely that any setup using FailFast expects the array to halt when md_error is called on the last rdev.

Since FailFast is only implemented for RAID1 and RAID10, no changes are needed for other personalities.

Fixes 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10")
---
 drivers/md/md.c     | 6 ++++--
 drivers/md/raid1.c  | 8 +++++++-
 drivers/md/raid10.c | 8 +++++++-
 3 files changed, 18 insertions(+), 4 deletions(-)
drivers/md/md.c +4 -2
index 6062e0deb616..f1745f8921fc 100644
@@ -3050,7 +3050,8 @@ state_store(struct md_rdev *rdev, const char *buf, size_t len)
 	if (cmd_match(buf, "faulty") && rdev->mddev->pers) {
 		md_error(rdev->mddev, rdev);
 
-		if (test_bit(MD_BROKEN, &rdev->mddev->flags))
+		if (test_bit(MD_BROKEN, &rdev->mddev->flags) ||
+		    !test_bit(Faulty, &rdev->flags))
 			err = -EBUSY;
 		else
 			err = 0;
@@ -7915,7 +7916,8 @@ static int set_disk_faulty(struct mddev *mddev, dev_t dev)
 		err =  -ENODEV;
 	else {
 		md_error(mddev, rdev);
-		if (test_bit(MD_BROKEN, &mddev->flags))
+		if (test_bit(MD_BROKEN, &mddev->flags) ||
+		    !test_bit(Faulty, &rdev->flags))
 			err = -EBUSY;
 	}
 	rcu_read_unlock();
drivers/md/raid1.c +7 -1
index 592a40233004..459b34cd358b 100644
@@ -1745,6 +1745,10 @@ static void raid1_status(struct seq_file *seq, struct mddev *mddev)
  *	- recovery is interrupted.
  *	- &mddev->degraded is bumped.
  *
+ * If the following conditions are met, @mddev never fails:
+ *	- The last In_sync @rdev has the &FailFast flag set.
+ *	- &mddev->fail_last_dev is off.
+ *
  * @rdev is marked as &Faulty excluding case when array is failed and
  * &mddev->fail_last_dev is off.
  */
@@ -1757,7 +1761,9 @@ static void raid1_error(struct mddev *mddev, struct md_rdev *rdev)
 
 	if (test_bit(In_sync, &rdev->flags) &&
 	    (conf->raid_disks - mddev->degraded) == 1) {
-		set_bit(MD_BROKEN, &mddev->flags);
+		if (!test_bit(FailFast, &rdev->flags) ||
+		    mddev->fail_last_dev)
+			set_bit(MD_BROKEN, &mddev->flags);
 
 		if (!mddev->fail_last_dev) {
 			conf->recovery_disabled = mddev->recovery_disabled;
drivers/md/raid10.c +7 -1
index 14dcd5142eb4..b33149aa5b29 100644
@@ -1989,6 +1989,10 @@ static int enough(struct r10conf *conf, int ignore)
  *	- recovery is interrupted.
  *	- &mddev->degraded is bumped.
  *
+ * If the following conditions are met, @mddev never fails:
+ *	- The last In_sync @rdev has the &FailFast flag set.
+ *	- &mddev->fail_last_dev is off.
+ *
  * @rdev is marked as &Faulty excluding case when array is failed and
  * &mddev->fail_last_dev is off.
  */
@@ -2000,7 +2004,9 @@ static void raid10_error(struct mddev *mddev, struct md_rdev *rdev)
 	spin_lock_irqsave(&conf->device_lock, flags);
 
 	if (test_bit(In_sync, &rdev->flags) && !enough(conf, rdev->raid_disk)) {
-		set_bit(MD_BROKEN, &mddev->flags);
+		if (!test_bit(FailFast, &rdev->flags) ||
+		    mddev->fail_last_dev)
+			set_bit(MD_BROKEN, &mddev->flags);
 
 		if (!mddev->fail_last_dev) {
 			spin_unlock_irqrestore(&conf->device_lock, flags);
Signature · 1 line
2.50.1
#2 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

Quoted 12 lines · Kenta Akagi

After commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), if the error handler is called on the last rdev in RAID1 or RAID10, the MD_BROKEN flag will be set on that mddev.
When MD_BROKEN is set, write bios to the md will result in an I/O error.

This causes a problem when using FailFast.
The current implementation of FailFast expects the array to continue functioning without issues even after calling md_error for the last rdev. Furthermore, due to the nature of its functionality, FailFast may call md_error on all rdevs of the md. Even if retrying I/O on an rdev would succeed, it first calls md_error before retrying.

Quoted 9 lines

To fix this issue, this commit ensures that for RAID1 and RAID10, if the last In_sync rdev has the FailFast flag set and the mddev's fail_last_dev is off, the MD_BROKEN flag will not be set on that mddev.

This change impacts userspace. After this commit, If the rdev has the FailFast flag, the mddev never broken even if the failing bio is not FailFast. However, it's unlikely that any setup using FailFast expects the array to halt when md_error is called on the last rdev.

In the current RAID design, when an IO error occurs, RAID ensures faulty data is not read via the following actions:
1. Mark the badblocks (no FailFast flag); if this fails,
2. Mark the disk as Faulty.

If neither action is taken, and BROKEN is not set to prevent continued RAID use, errors on the last remaining disk will be ignored. Subsequent reads may return incorrect data. This seems like a more serious issue in my opinion.

In scenarios with a large number of transient IO errors, is FailFast not a suitable configuration? As you mentioned: "retrying I/O on an rdev would succeed".

Signature · 2 lines
Thanks,
Nan
#3 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

Quoted 31 lines · Li Nan
Quoted 12 lines · Kenta Akagi

After commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), if the error handler is called on the last rdev in RAID1 or RAID10, the MD_BROKEN flag will be set on that mddev.
When MD_BROKEN is set, write bios to the md will result in an I/O error.

This causes a problem when using FailFast.
The current implementation of FailFast expects the array to continue functioning without issues even after calling md_error for the last rdev. Furthermore, due to the nature of its functionality, FailFast may call md_error on all rdevs of the md. Even if retrying I/O on an rdev would succeed, it first calls md_error before retrying.

Quoted 9 lines

To fix this issue, this commit ensures that for RAID1 and RAID10, if the last In_sync rdev has the FailFast flag set and the mddev's fail_last_dev is off, the MD_BROKEN flag will not be set on that mddev.

This change impacts userspace. After this commit, If the rdev has the FailFast flag, the mddev never broken even if the failing bio is not FailFast. However, it's unlikely that any setup using FailFast expects the array to halt when md_error is called on the last rdev.

In the current RAID design, when an IO error occurs, RAID ensures faulty data is not read via the following actions:
1. Mark the badblocks (no FailFast flag); if this fails,
2. Mark the disk as Faulty.

Quoted 7 lines

If neither action is taken, and BROKEN is not set to prevent continued RAID use, errors on the last remaining disk will be ignored. Subsequent reads may return incorrect data. This seems like a more serious issue in my opinion.

In scenarios with a large number of transient IO errors, is FailFast not a suitable configuration? As you mentioned: "retrying I/O on an rdev would succeed".

Hi Nan

According to my understanding, the policy here is to try to keep raid work if io error happens on the last device. It doesn't set faulty on the last in_sync device. It only sets MD_BROKEN to forbid write requests. But it still can read data from the last device.

static void raid1_error(struct mddev *mddev, struct md_rdev *rdev) {

    if (test_bit(In_sync, &rdev->flags) &&
        (conf->raid_disks - mddev->degraded) == 1) {
        set_bit(MD_BROKEN, &mddev->flags);

        if (!mddev->fail_last_dev) {
            return;  // return directly here
        }

static void md_submit_bio(struct bio *bio)

{
    if (unlikely(test_bit(MD_BROKEN, &mddev->flags)) && (rw == WRITE)) {
        bio_io_error(bio);
        return;
    }

Read requests can submit to the last working device. Right?

Best Regards
Xiao

Quoted 5 lines

--
Thanks,
Nan

#4 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

Quoted 67 lines · Xiao Ni
Quoted 31 lines · Li Nan
Quoted 12 lines · Kenta Akagi

After commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), if the error handler is called on the last rdev in RAID1 or RAID10, the MD_BROKEN flag will be set on that mddev.
When MD_BROKEN is set, write bios to the md will result in an I/O error.

This causes a problem when using FailFast.
The current implementation of FailFast expects the array to continue functioning without issues even after calling md_error for the last rdev. Furthermore, due to the nature of its functionality, FailFast may call md_error on all rdevs of the md. Even if retrying I/O on an rdev would succeed, it first calls md_error before retrying.

Quoted 9 lines

To fix this issue, this commit ensures that for RAID1 and RAID10, if the last In_sync rdev has the FailFast flag set and the mddev's fail_last_dev is off, the MD_BROKEN flag will not be set on that mddev.

This change impacts userspace. After this commit, If the rdev has the FailFast flag, the mddev never broken even if the failing bio is not FailFast. However, it's unlikely that any setup using FailFast expects the array to halt when md_error is called on the last rdev.

In the current RAID design, when an IO error occurs, RAID ensures faulty data is not read via the following actions:
1. Mark the badblocks (no FailFast flag); if this fails,
2. Mark the disk as Faulty.

Quoted 7 lines

If neither action is taken, and BROKEN is not set to prevent continued RAID use, errors on the last remaining disk will be ignored. Subsequent reads may return incorrect data. This seems like a more serious issue in my opinion.

In scenarios with a large number of transient IO errors, is FailFast not a suitable configuration? As you mentioned: "retrying I/O on an rdev would succeed".

Hi Nan

According to my understanding, the policy here is to try to keep raid work if io error happens on the last device. It doesn't set faulty on the last in_sync device. It only sets MD_BROKEN to forbid write requests. But it still can read data from the last device.

static void raid1_error(struct mddev *mddev, struct md_rdev *rdev) {

     if (test_bit(In_sync, &rdev->flags) &&
         (conf->raid_disks - mddev->degraded) == 1) {
         set_bit(MD_BROKEN, &mddev->flags);

         if (!mddev->fail_last_dev) {
             return;  // return directly here
         }

static void md_submit_bio(struct bio *bio)

{
     if (unlikely(test_bit(MD_BROKEN, &mddev->flags)) && (rw == WRITE)) {
         bio_io_error(bio);
         return;
     }
Quoted 5 lines

Read requests can submit to the last working device. Right?

Best Regards
Xiao

Yeah, after MD_BROKEN is set, read are forbidden but writes remain allowed. IMO we preserve the RAID array in this state to enable users to retrieve stored data, not to continue using it. However, continued writes to the array will cause subsequent errors to fail to be logged, either due to failfast or the badblocks being full. Read errors have no impact as they do not damage the original data.

Signature · 2 lines
Thanks,
Nan
#5 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

Quoted 71 lines · Li Nan
Quoted 67 lines · Xiao Ni
Quoted 31 lines · Li Nan
Quoted 12 lines · Kenta Akagi

After commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), if the error handler is called on the last rdev in RAID1 or RAID10, the MD_BROKEN flag will be set on that mddev.
When MD_BROKEN is set, write bios to the md will result in an I/O error.

This causes a problem when using FailFast.
The current implementation of FailFast expects the array to continue functioning without issues even after calling md_error for the last rdev. Furthermore, due to the nature of its functionality, FailFast may call md_error on all rdevs of the md. Even if retrying I/O on an rdev would succeed, it first calls md_error before retrying.

Quoted 9 lines

To fix this issue, this commit ensures that for RAID1 and RAID10, if the last In_sync rdev has the FailFast flag set and the mddev's fail_last_dev is off, the MD_BROKEN flag will not be set on that mddev.

This change impacts userspace. After this commit, If the rdev has the FailFast flag, the mddev never broken even if the failing bio is not FailFast. However, it's unlikely that any setup using FailFast expects the array to halt when md_error is called on the last rdev.

In the current RAID design, when an IO error occurs, RAID ensures faulty data is not read via the following actions:
1. Mark the badblocks (no FailFast flag); if this fails,
2. Mark the disk as Faulty.

Quoted 7 lines

If neither action is taken, and BROKEN is not set to prevent continued RAID use, errors on the last remaining disk will be ignored. Subsequent reads may return incorrect data. This seems like a more serious issue in my opinion.

In scenarios with a large number of transient IO errors, is FailFast not a suitable configuration? As you mentioned: "retrying I/O on an rdev would succeed".

Hi Nan

According to my understanding, the policy here is to try to keep raid work if io error happens on the last device. It doesn't set faulty on the last in_sync device. It only sets MD_BROKEN to forbid write requests. But it still can read data from the last device.

static void raid1_error(struct mddev *mddev, struct md_rdev *rdev) {

     if (test_bit(In_sync, &rdev->flags) &&
         (conf->raid_disks - mddev->degraded) == 1) {
         set_bit(MD_BROKEN, &mddev->flags);

         if (!mddev->fail_last_dev) {
             return;  // return directly here
         }

static void md_submit_bio(struct bio *bio)

{
     if (unlikely(test_bit(MD_BROKEN, &mddev->flags)) && (rw == WRITE)) {
         bio_io_error(bio);
         return;
     }
Quoted 7 lines
Quoted 5 lines

Read requests can submit to the last working device. Right?

Best Regards
Xiao

Yeah, after MD_BROKEN is set, read are forbidden but writes remain allowed.

Hmm, reverse way? Write requests are forbidden and read requests are allowed now. If MD_BROKEN is set, write requests return directly after bio_io_error.

Regards
Xiao

Quoted 10 lines

IMO we preserve the RAID array in this state to enable users to retrieve stored data, not to continue using it. However, continued writes to the array will cause subsequent errors to fail to be logged, either due to failfast or the badblocks being full. Read errors have no impact as they do not damage the original data.

--
Thanks,
Nan

#6 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

Quoted 78 lines · Xiao Ni
Quoted 77 lines · Li Nan
Quoted 72 lines · Xiao Ni
Quoted 31 lines · Li Nan
Quoted 12 lines · Kenta Akagi

After commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), if the error handler is called on the last rdev in RAID1 or RAID10, the MD_BROKEN flag will be set on that mddev.
When MD_BROKEN is set, write bios to the md will result in an I/O error.

This causes a problem when using FailFast.
The current implementation of FailFast expects the array to continue functioning without issues even after calling md_error for the last rdev. Furthermore, due to the nature of its functionality, FailFast may call md_error on all rdevs of the md. Even if retrying I/O on an rdev would succeed, it first calls md_error before retrying.

Quoted 9 lines

To fix this issue, this commit ensures that for RAID1 and RAID10, if the last In_sync rdev has the FailFast flag set and the mddev's fail_last_dev is off, the MD_BROKEN flag will not be set on that mddev.

This change impacts userspace. After this commit, If the rdev has the FailFast flag, the mddev never broken even if the failing bio is not FailFast. However, it's unlikely that any setup using FailFast expects the array to halt when md_error is called on the last rdev.

In the current RAID design, when an IO error occurs, RAID ensures faulty data is not read via the following actions:
1. Mark the badblocks (no FailFast flag); if this fails,
2. Mark the disk as Faulty.

Quoted 7 lines

If neither action is taken, and BROKEN is not set to prevent continued RAID use, errors on the last remaining disk will be ignored. Subsequent reads may return incorrect data. This seems like a more serious issue in my opinion.

In scenarios with a large number of transient IO errors, is FailFast not a suitable configuration? As you mentioned: "retrying I/O on an rdev would succeed".

Hi Nan

According to my understanding, the policy here is to try to keep raid work if io error happens on the last device. It doesn't set faulty on the last in_sync device. It only sets MD_BROKEN to forbid write requests. But it still can read data from the last device.

static void raid1_error(struct mddev *mddev, struct md_rdev *rdev) {

      if (test_bit(In_sync, &rdev->flags) &&
          (conf->raid_disks - mddev->degraded) == 1) {
          set_bit(MD_BROKEN, &mddev->flags);

          if (!mddev->fail_last_dev) {
              return;  // return directly here
          }

static void md_submit_bio(struct bio *bio)

{
      if (unlikely(test_bit(MD_BROKEN, &mddev->flags)) && (rw == WRITE)) {
          bio_io_error(bio);
          return;
      }

Read requests can submit to the last working device. Right?

Best Regards
Xiao

Quoted 9 lines
Quoted 1 line

Yeah, after MD_BROKEN is set, read are forbidden but writes remain allowed.

Hmm, reverse way? Write requests are forbidden and read requests are allowed now. If MD_BROKEN is set, write requests return directly after bio_io_error.

Regards
Xiao

Apologies for the typo... The rest of the content was written with this exact meaning in mind.

Quoted 6 lines
Quoted 6 lines

IMO we preserve the RAID array in this state to enable users to retrieve stored data, not to continue using it. However, continued writes to the array will cause subsequent errors to fail to be logged, either due to failfast or the badblocks being full. Read errors have no impact as they do not damage the original data.

Signature · 2 lines
Thanks,
Nan
#7 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

In reply to Li Nan

Hi,
Thank you for reviewing.

Quoted 25 lines · Li Nan
Quoted 12 lines · Kenta Akagi

After commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), if the error handler is called on the last rdev in RAID1 or RAID10, the MD_BROKEN flag will be set on that mddev.
When MD_BROKEN is set, write bios to the md will result in an I/O error.

This causes a problem when using FailFast.
The current implementation of FailFast expects the array to continue functioning without issues even after calling md_error for the last rdev.  Furthermore, due to the nature of its functionality, FailFast may call md_error on all rdevs of the md. Even if retrying I/O on an rdev would succeed, it first calls md_error before retrying.

Quoted 9 lines

To fix this issue, this commit ensures that for RAID1 and RAID10, if the last In_sync rdev has the FailFast flag set and the mddev's fail_last_dev is off, the MD_BROKEN flag will not be set on that mddev.

This change impacts userspace. After this commit, If the rdev has the FailFast flag, the mddev never broken even if the failing bio is not FailFast. However, it's unlikely that any setup using FailFast expects the array to halt when md_error is called on the last rdev.

Quoted 8 lines

In the current RAID design, when an IO error occurs, RAID ensures faulty data is not read via the following actions:
1. Mark the badblocks (no FailFast flag); if this fails,
2. Mark the disk as Faulty.

If neither action is taken, and BROKEN is not set to prevent continued RAID use, errors on the last remaining disk will be ignored. Subsequent reads may return incorrect data. This seems like a more serious issue in my opinion.

I agree that data inconsistency can certainly occur in this scenario.

However, a RAID1 with only one remaining rdev can considered the same as a plain disk. From that perspective, I do not believe it is the mandatory responsibility of md raid to block subsequent writes nor prevent data inconsistency in this situation.

The commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10") that introduced BROKEN for RAID1/10 also does not seem to have done so for that responsibility.

Quoted 3 lines

In scenarios with a large number of transient IO errors, is FailFast not a suitable configuration? As you mentioned: "retrying I/O on an rdev would

It seems be right about that. Using FailFast with unstable underlayer is not good. However, as md raid, which is issuer of FailFast bios,
I believe it is incorrect to shutdown the array due to the failure of a FailFast bio.

Thanks,
Akagi

Quoted 7 lines

succeed".

--
Thanks,
Nan

#8 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

Quoted 51 lines · Kenta Akagi

Hi,
Thank you for reviewing.

Quoted 25 lines · Li Nan
Quoted 12 lines · Kenta Akagi

After commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), if the error handler is called on the last rdev in RAID1 or RAID10, the MD_BROKEN flag will be set on that mddev.
When MD_BROKEN is set, write bios to the md will result in an I/O error.

This causes a problem when using FailFast.
The current implementation of FailFast expects the array to continue functioning without issues even after calling md_error for the last rdev.  Furthermore, due to the nature of its functionality, FailFast may call md_error on all rdevs of the md. Even if retrying I/O on an rdev would succeed, it first calls md_error before retrying.

Quoted 9 lines

To fix this issue, this commit ensures that for RAID1 and RAID10, if the last In_sync rdev has the FailFast flag set and the mddev's fail_last_dev is off, the MD_BROKEN flag will not be set on that mddev.

This change impacts userspace. After this commit, If the rdev has the FailFast flag, the mddev never broken even if the failing bio is not FailFast. However, it's unlikely that any setup using FailFast expects the array to halt when md_error is called on the last rdev.

Quoted 8 lines

In the current RAID design, when an IO error occurs, RAID ensures faulty data is not read via the following actions:
1. Mark the badblocks (no FailFast flag); if this fails,
2. Mark the disk as Faulty.

If neither action is taken, and BROKEN is not set to prevent continued RAID use, errors on the last remaining disk will be ignored. Subsequent reads may return incorrect data. This seems like a more serious issue in my opinion.

I agree that data inconsistency can certainly occur in this scenario.

However, a RAID1 with only one remaining rdev can considered the same as a plain disk. From that perspective, I do not believe it is the mandatory responsibility of md raid to block subsequent writes nor prevent data inconsistency in this situation.

The commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10") that introduced BROKEN for RAID1/10 also does not seem to have done so for that responsibility.

Quoted 3 lines

In scenarios with a large number of transient IO errors, is FailFast not a suitable configuration? As you mentioned: "retrying I/O on an rdev would

Quoted 7 lines

It seems be right about that. Using FailFast with unstable underlayer is not good. However, as md raid, which is issuer of FailFast bios,
I believe it is incorrect to shutdown the array due to the failure of a FailFast bio.

Thanks,
Akagi

I get your point, Kuai, what's your take on this?

Signature · 2 lines
Thanks,
Nan
#9 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

In reply to Kenta Akagi
Quoted 49 lines · Kenta Akagi

Hi,
Thank you for reviewing.

Quoted 25 lines · Li Nan
Quoted 12 lines · Kenta Akagi

After commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), if the error handler is called on the last rdev in RAID1 or RAID10, the MD_BROKEN flag will be set on that mddev.
When MD_BROKEN is set, write bios to the md will result in an I/O error.

This causes a problem when using FailFast.
The current implementation of FailFast expects the array to continue functioning without issues even after calling md_error for the last rdev. Furthermore, due to the nature of its functionality, FailFast may call md_error on all rdevs of the md. Even if retrying I/O on an rdev would succeed, it first calls md_error before retrying.

Quoted 9 lines

To fix this issue, this commit ensures that for RAID1 and RAID10, if the last In_sync rdev has the FailFast flag set and the mddev's fail_last_dev is off, the MD_BROKEN flag will not be set on that mddev.

This change impacts userspace. After this commit, If the rdev has the FailFast flag, the mddev never broken even if the failing bio is not FailFast. However, it's unlikely that any setup using FailFast expects the array to halt when md_error is called on the last rdev.

Quoted 8 lines

In the current RAID design, when an IO error occurs, RAID ensures faulty data is not read via the following actions:
1. Mark the badblocks (no FailFast flag); if this fails,
2. Mark the disk as Faulty.

If neither action is taken, and BROKEN is not set to prevent continued RAID use, errors on the last remaining disk will be ignored. Subsequent reads may return incorrect data. This seems like a more serious issue in my opinion.

I agree that data inconsistency can certainly occur in this scenario.

However, a RAID1 with only one remaining rdev can considered the same as a plain disk. From that perspective, I do not believe it is the mandatory responsibility of md raid to block subsequent writes nor prevent data inconsistency in this situation.

The commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10") that introduced BROKEN for RAID1/10 also does not seem to have done so for that responsibility.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

In scenarios with a large number of transient IO errors, is FailFast not a suitable configuration? As you mentioned: "retrying I/O on an rdev would

It seems be right about that. Using FailFast with unstable underlayer is not good. However, as md raid, which is issuer of FailFast bios,
I believe it is incorrect to shutdown the array due to the failure of a FailFast bio.

Hi all

I understand @Li Nan 's point now. The badblock can't be recorded in this situation and the last working device is not set to faulty. To be frank, I think consistency of data is more important. Users don't think it's a single disk, they must think raid1 should guarantee the consistency. But the write request should return an error when calling raid1_error for the last working device, right? So there is no consistency problem?

hi, Kenta. I have a question too. What will you do in your environment after the network connection works again? Add those disks one by one to do recovery?

Best Regards
Xiao

Quoted 12 lines

Thanks,
Akagi

Quoted 7 lines

succeed".

--
Thanks,
Nan

#10 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

Hi,

Quoted 57 lines · Xiao Ni
Quoted 49 lines · Kenta Akagi

Hi,
Thank you for reviewing.

Quoted 25 lines · Li Nan
Quoted 12 lines · Kenta Akagi

After commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), if the error handler is called on the last rdev in RAID1 or RAID10, the MD_BROKEN flag will be set on that mddev.
When MD_BROKEN is set, write bios to the md will result in an I/O error.

This causes a problem when using FailFast.
The current implementation of FailFast expects the array to continue functioning without issues even after calling md_error for the last rdev. Furthermore, due to the nature of its functionality, FailFast may call md_error on all rdevs of the md. Even if retrying I/O on an rdev would succeed, it first calls md_error before retrying.

Quoted 9 lines

To fix this issue, this commit ensures that for RAID1 and RAID10, if the last In_sync rdev has the FailFast flag set and the mddev's fail_last_dev is off, the MD_BROKEN flag will not be set on that mddev.

This change impacts userspace. After this commit, If the rdev has the FailFast flag, the mddev never broken even if the failing bio is not FailFast. However, it's unlikely that any setup using FailFast expects the array to halt when md_error is called on the last rdev.

Quoted 8 lines

In the current RAID design, when an IO error occurs, RAID ensures faulty data is not read via the following actions:
1. Mark the badblocks (no FailFast flag); if this fails,
2. Mark the disk as Faulty.

If neither action is taken, and BROKEN is not set to prevent continued RAID use, errors on the last remaining disk will be ignored. Subsequent reads may return incorrect data. This seems like a more serious issue in my opinion.

I agree that data inconsistency can certainly occur in this scenario.

However, a RAID1 with only one remaining rdev can considered the same as a plain disk. From that perspective, I do not believe it is the mandatory responsibility of md raid to block subsequent writes nor prevent data inconsistency in this situation.

The commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10") that introduced BROKEN for RAID1/10 also does not seem to have done so for that responsibility.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

In scenarios with a large number of transient IO errors, is FailFast not a suitable configuration? As you mentioned: "retrying I/O on an rdev would

It seems be right about that. Using FailFast with unstable underlayer is not good. However, as md raid, which is issuer of FailFast bios,
I believe it is incorrect to shutdown the array due to the failure of a FailFast bio.

Quoted 6 lines

Hi all

I understand @Li Nan 's point now. The badblock can't be recorded in this situation and the last working device is not set to faulty. To be frank, I think consistency of data is more important. Users don't think it's a single disk, they must think raid1 should guarantee the

Hmm, I see...

Quoted 7 lines

consistency. But the write request should return an error when calling raid1_error for the last working device, right? So there is no consistency problem?

hi, Kenta. I have a question too. What will you do in your environment after the network connection works again? Add those disks one by one to do recovery?

Yes. We will have to add a new disk or remove and add the rdev marked as faulty. Currently, the array is being recreated because it is mark as broken.

Thanks,
Akagi

Quoted 18 lines

Best Regards
Xiao

Quoted 12 lines

Thanks,
Akagi

Quoted 7 lines

succeed".

--
Thanks,
Nan

#11 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

In reply to Xiao Ni
Quoted 59 lines · Xiao Ni
Quoted 49 lines · Kenta Akagi

Hi,
Thank you for reviewing.

Quoted 25 lines · Li Nan
Quoted 12 lines · Kenta Akagi

After commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"), if the error handler is called on the last rdev in RAID1 or RAID10, the MD_BROKEN flag will be set on that mddev.
When MD_BROKEN is set, write bios to the md will result in an I/O error.

This causes a problem when using FailFast.
The current implementation of FailFast expects the array to continue functioning without issues even after calling md_error for the last rdev. Furthermore, due to the nature of its functionality, FailFast may call md_error on all rdevs of the md. Even if retrying I/O on an rdev would succeed, it first calls md_error before retrying.

Quoted 9 lines

To fix this issue, this commit ensures that for RAID1 and RAID10, if the last In_sync rdev has the FailFast flag set and the mddev's fail_last_dev is off, the MD_BROKEN flag will not be set on that mddev.

This change impacts userspace. After this commit, If the rdev has the FailFast flag, the mddev never broken even if the failing bio is not FailFast. However, it's unlikely that any setup using FailFast expects the array to halt when md_error is called on the last rdev.

Quoted 8 lines

In the current RAID design, when an IO error occurs, RAID ensures faulty data is not read via the following actions:
1. Mark the badblocks (no FailFast flag); if this fails,
2. Mark the disk as Faulty.

If neither action is taken, and BROKEN is not set to prevent continued RAID use, errors on the last remaining disk will be ignored. Subsequent reads may return incorrect data. This seems like a more serious issue in my opinion.

I agree that data inconsistency can certainly occur in this scenario.

However, a RAID1 with only one remaining rdev can considered the same as a plain disk. From that perspective, I do not believe it is the mandatory responsibility of md raid to block subsequent writes nor prevent data inconsistency in this situation.

The commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10") that introduced BROKEN for RAID1/10 also does not seem to have done so for that responsibility.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

In scenarios with a large number of transient IO errors, is FailFast not a suitable configuration? As you mentioned: "retrying I/O on an rdev would

It seems be right about that. Using FailFast with unstable underlayer is not good. However, as md raid, which is issuer of FailFast bios,
I believe it is incorrect to shutdown the array due to the failure of a FailFast bio.

Hi all

Quoted 7 lines

I understand @Li Nan 's point now. The badblock can't be recorded in this situation and the last working device is not set to faulty. To be frank, I think consistency of data is more important. Users don't think it's a single disk, they must think raid1 should guarantee the consistency. But the write request should return an error when calling raid1_error for the last working device, right? So there is no consistency problem?

Hi all,

I understand that when md_error is issued for the last remaining rdev, the array should be stopped except in the failfast case, also, it is no longer appropriate to treat an RAID1 array that has lost redundancy as "just a normal single drive" [1].

I will post an PATCH v7 based on v5.

[1] commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10")

Thanks.

Quoted 22 lines

hi, Kenta. I have a question too. What will you do in your environment after the network connection works again? Add those disks one by one to do recovery?

Best Regards
Xiao

Quoted 12 lines

Thanks,
Akagi

Quoted 7 lines

succeed".

--
Thanks,
Nan

#12 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

Hello Kenta, all,

Quoted 102 lines · Kenta Akagi
Quoted 85 lines · Xiao Ni
Quoted 76 lines · Kenta Akagi

Hi,
Thank you for reviewing.

Quoted 43 lines · Li Nan
Quoted 25 lines · Kenta Akagi

After commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"),
if the error handler is called on the last rdev in RAID1 or RAID10,
the MD_BROKEN flag will be set on that mddev.
When MD_BROKEN is set, write bios to the md will result in an I/O error.

This causes a problem when using FailFast.
The current implementation of FailFast expects the array to continue
functioning without issues even after calling md_error for the last
rdev.  Furthermore, due to the nature of its functionality, FailFast may
call md_error on all rdevs of the md. Even if retrying I/O on an rdev
would succeed, it first calls md_error before retrying.

To fix this issue, this commit ensures that for RAID1 and RAID10, if the
last In_sync rdev has the FailFast flag set and the mddev's fail_last_dev
is off, the MD_BROKEN flag will not be set on that mddev.

Quoted 8 lines

This change impacts userspace. After this commit, If the rdev has the
FailFast flag, the mddev never broken even if the failing bio is not
FailFast. However, it's unlikely that any setup using FailFast expects
the array to halt when md_error is called on the last rdev.

In the current RAID design, when an IO error occurs, RAID ensures faulty
data is not read via the following actions:
1. Mark the badblocks (no FailFast flag); if this fails,
2. Mark the disk as Faulty.

Quoted 6 lines

If neither action is taken, and BROKEN is not set to prevent continued RAID
use, errors on the last remaining disk will be ignored. Subsequent reads
may return incorrect data. This seems like a more serious issue in my opinion.

I agree that data inconsistency can certainly occur in this scenario.

However, a RAID1 with only one remaining rdev can considered the same as a plain
disk. From that perspective, I do not believe it is the mandatory responsibility
of md raid to block subsequent writes nor prevent data inconsistency in this situation.

The commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10") that introduced
BROKEN for RAID1/10 also does not seem to have done so for that responsibility.

Quoted 5 lines

In scenarios with a large number of transient IO errors, is FailFast not a
suitable configuration? As you mentioned: "retrying I/O on an rdev would

Quoted 5 lines

It seems be right about that. Using FailFast with unstable underlayer is not good.
However, as md raid, which is issuer of FailFast bios, I believe it is incorrect to shutdown the array due to the failure of a FailFast bio.

Hi all

Quoted 11 lines

I understand @Li Nan 's point now. The badblock can't be recorded in
this situation and the last working device is not set to faulty. To be
frank, I think consistency of data is more important. Users don't think it's a single disk, they must think raid1 should guarantee the
consistency. But the write request should return an error when calling
raid1_error for the last working device, right? So there is no consistency problem?

Hi all,

Quoted 7 lines

I understand that when md_error is issued for the last remaining rdev,
the array should be stopped except in the failfast case, also, it is no longer appropriate to treat an RAID1 array that has lost redundancy as "just a normal single drive" [1].

I will post an PATCH v7 based on v5.

I wonder what became of this v7 series. Have you given up on this? If yes, what is the bottom line - simply not using failfast in setups like the one you described?

Martin

Signature · 5 lines
Dr. Martin Wilck <[email protected]>
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg,
Germany
Geschäftsführer: Stefan Gaiser, Jochen Jaser, Abhinav Puri (HRB
36809,AG Nürnberg)
#13 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

Quoted 111 lines · Martin Wilck

Hello Kenta, all,

Quoted 108 lines · Kenta Akagi
Quoted 85 lines · Xiao Ni
Quoted 76 lines · Kenta Akagi

Hi,
Thank you for reviewing.

Quoted 43 lines · Li Nan
Quoted 25 lines · Kenta Akagi

After commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10"),
if the error handler is called on the last rdev in RAID1 or RAID10,
the MD_BROKEN flag will be set on that mddev.
When MD_BROKEN is set, write bios to the md will result in an I/O error.

This causes a problem when using FailFast.
The current implementation of FailFast expects the array to continue
functioning without issues even after calling md_error for the last
rdev.  Furthermore, due to the nature of its functionality, FailFast may
call md_error on all rdevs of the md. Even if retrying I/O on an rdev
would succeed, it first calls md_error before retrying.

To fix this issue, this commit ensures that for RAID1 and RAID10, if the
last In_sync rdev has the FailFast flag set and the mddev's fail_last_dev
is off, the MD_BROKEN flag will not be set on that mddev.

Quoted 8 lines

This change impacts userspace. After this commit, If the rdev has the
FailFast flag, the mddev never broken even if the failing bio is not
FailFast. However, it's unlikely that any setup using FailFast expects
the array to halt when md_error is called on the last rdev.

In the current RAID design, when an IO error occurs, RAID ensures faulty
data is not read via the following actions:
1. Mark the badblocks (no FailFast flag); if this fails,
2. Mark the disk as Faulty.

Quoted 6 lines

If neither action is taken, and BROKEN is not set to prevent continued RAID
use, errors on the last remaining disk will be ignored. Subsequent reads
may return incorrect data. This seems like a more serious issue in my opinion.

I agree that data inconsistency can certainly occur in this scenario.

However, a RAID1 with only one remaining rdev can considered the same as a plain
disk. From that perspective, I do not believe it is the mandatory responsibility
of md raid to block subsequent writes nor prevent data inconsistency in this situation.

The commit 9631abdbf406 ("md: Set MD_BROKEN for RAID1 and RAID10") that introduced
BROKEN for RAID1/10 also does not seem to have done so for that responsibility.

Quoted 5 lines

In scenarios with a large number of transient IO errors, is FailFast not a
suitable configuration? As you mentioned: "retrying I/O on an rdev would

Quoted 5 lines

It seems be right about that. Using FailFast with unstable underlayer is not good.
However, as md raid, which is issuer of FailFast bios, I believe it is incorrect to shutdown the array due to the failure of a FailFast bio.

Hi all

Quoted 11 lines

I understand @Li Nan 's point now. The badblock can't be recorded in
this situation and the last working device is not set to faulty. To be
frank, I think consistency of data is more important. Users don't think it's a single disk, they must think raid1 should guarantee the
consistency. But the write request should return an error when calling
raid1_error for the last working device, right? So there is no consistency problem?

Hi all,

I understand that when md_error is issued for the last remaining rdev,
the array should be stopped except in the failfast case, also, it is no longer appropriate to treat an RAID1 array that has lost redundancy as "just a normal single drive" [1].

Quoted 5 lines
Quoted 1 line

I will post an PATCH v7 based on v5.

I wonder what became of this v7 series. Have you given up on this? If yes, what is the bottom line - simply not using failfast in setups like the one you described?

Hi Martin,

I have not given up on it, but I have not managed to post v7 yet. I still think failfast should be usable even in setups like that.

It has been a while, but I intend to resume work on it.

Thanks,
Akagi

Quoted 10 lines

Martin

Signature · 5 lines
Dr. Martin Wilck <[email protected]>
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg,
Germany
Geschäftsführer: Stefan Gaiser, Jochen Jaser, Abhinav Puri (HRB
36809,AG Nürnberg)
#14 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

Hi Kenta,

Quoted 8 lines · Kenta Akagi

Hi Martin,

I have not given up on it, but I have not managed to post v7 yet. I still think failfast should be usable even in setups like that.

It has been a while, but I intend to resume work on it.

My thinking is that, in the fastfail case, code to prevent total failure could be placed in the end-IO code code path, e.g. by attempting a retry directly from the md layer when the last rdev fails, instead of setting failing the device. But I haven't thought it through.

Thanks
Martin

Signature · 5 lines
Dr. Martin Wilck <[email protected]>
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg,
Germany
Geschäftsführer: Stefan Gaiser, Jochen Jaser, Abhinav Puri (HRB
36809,AG Nürnberg)
#15 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

Quoted 10 lines · Martin Wilck

Hi Kenta,

Quoted 7 lines · Kenta Akagi

Hi Martin,

I have not given up on it, but I have not managed to post v7 yet. I still think failfast should be usable even in setups like that.

Quoted 7 lines
Quoted 1 line

It has been a while, but I intend to resume work on it.

My thinking is that, in the fastfail case, code to prevent total failure could be placed in the end-IO code code path, e.g. by attempting a retry directly from the md layer when the last rdev fails, instead of setting failing the device. But I haven't thought it through.

Hi Martin,

I may be misunderstanding your suggestion, but Neil's original failfast implementation already retries a failed failfast I/O to the last rdev. But a later change introduced a regression, which I intend to fix.

So the sequence should be:

1. The failfast bios to all mirrored rdevs fail.
2. The first rdev is marked faulty because its bio failed.
3. The other rdev is now the last, so it is not marked faulty.
4. Since the last rdev remains usable, its bio error handler retries the I/O without failfast.

Thanks,
Akagi

#16 lore raw

Re: [PATCH v6 1/2] md: Don't set MD_BROKEN for RAID1 and RAID10 when using FailFast

Quoted 22 lines · Kenta Akagi
Quoted 10 lines · Martin Wilck

Hi Kenta,

Quoted 7 lines · Kenta Akagi

Hi Martin,

I have not given up on it, but I have not managed to post v7 yet. I still think failfast should be usable even in setups like that.

Quoted 8 lines
Quoted 1 line

It has been a while, but I intend to resume work on it.

My thinking is that, in the fastfail case, code to prevent total failure could be placed in the end-IO code code path, e.g. by attempting a retry directly from the md layer when the last rdev fails,
instead of setting failing the device. But I haven't thought it through.

Quoted 7 lines

Hi Martin,

I may be misunderstanding your suggestion, but Neil's original failfast
implementation already retries a failed failfast I/O to the last rdev.
But a later change introduced a regression, which I intend to fix.

Ah OK, I wasn't aware of that.

Quoted 7 lines

So the sequence should be:

1. The failfast bios to all mirrored rdevs fail.
2. The first rdev is marked faulty because its bio failed.
3. The other rdev is now the last, so it is not marked faulty.
4. Since the last rdev remains usable, its bio error handler retries the I/O without failfast.

Yes, that makes sense to me.

Martin

Signature · 5 lines
Dr. Martin Wilck <[email protected]>
SUSE Software Solutions Germany GmbH, Frankenstr. 146, 90461 Nürnberg,
Germany
Geschäftsführer: Stefan Gaiser, Jochen Jaser, Abhinav Puri (HRB
36809,AG Nürnberg)
#17 lore raw

PATCH v6 2/2 md/raid10: fix failfast read error not rescheduled

In reply to Kenta Akagi

raid10_end_read_request lacks a path to retry when a FailFast IO fails. As a result, when Failfast Read IOs fail on all rdevs, the upper layer receives EIO, without read rescheduled.

Looking at the two commits below, it seems only raid10_end_read_request lacks the failfast read retry handling, while raid1_end_read_request has it. In RAID1, the retry works as expected.
* commit 8d3ca83dcf9c ("md/raid10: add failfast handling for reads.")
* commit 2e52d449bcec ("md/raid1: add failfast handling for reads.")

This commit will make the failfast read bio for the last rdev in raid10 retry if it fails.

Fixes 8d3ca83dcf9c ("md/raid10: add failfast handling for reads.")
---
 drivers/md/raid10.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/md/raid10.c +7 -0
index b33149aa5b29..8a254bab52e8 100644
@@ -401,6 +401,13 @@ static void raid10_end_read_request(struct bio *bio)
 		 * wait for the 'master' bio.
 		 */
 		set_bit(R10BIO_Uptodate, &r10_bio->state);
+	} else if (test_bit(FailFast, &rdev->flags) &&
+		 test_bit(R10BIO_FailFast, &r10_bio->state)) {
+		/*
+		 * This was a fail-fast read so we definitely
+		 * want to retry
+		 */
+		;
 	} else if (!raid1_should_handle_error(bio)) {
 		uptodate = 1;
 	} else {
Signature · 1 line
2.50.1