Last working drive in RAID1

Hi,

It is interesting to notice that RAID1 won't mark the last working drive as Faulty no matter what. The responsible code seems here:

static void error(struct mddev *mddev, struct md_rdev *rdev)

{
         ...
         /*
          * If it is not operational, then we have already marked it as dead
          * else if it is the last working disks, ignore the error, let the
          * next level up know.
          * else mark the drive as failed
          */
         if (test_bit(In_sync, &rdev->flags)
             && (conf->raid_disks - mddev->degraded) == 1) {
                 /*
                  * Don't fail the drive, act as though we were just a
                  * normal single drive.
                  * However don't try a recovery from this drive as
                  * it is very likely to fail.
                  */
                 conf->recovery_disabled = mddev->recovery_disabled;
                 return;
         }
         ...
}

The end result is that even if all the drives are physically gone, there still one drive remains in array forever, and mdadm continues to report the array is degraded instead of failed. RAID10 also has similar behavior.

Is there any reason we absolutely don't want to fail the last drive of RAID1?

Quoted 28 lines · On Wed, 04 Mar 2015 12:55:43 -0700 Eric Mei <[email protected]> wrote:

Hi,

It is interesting to notice that RAID1 won't mark the last working drive as Faulty no matter what. The responsible code seems here:

static void error(struct mddev *mddev, struct md_rdev *rdev)

{
          ...
          /*
           * If it is not operational, then we have already marked it as dead
           * else if it is the last working disks, ignore the error, let the
           * next level up know.
           * else mark the drive as failed
           */
          if (test_bit(In_sync, &rdev->flags)
              && (conf->raid_disks - mddev->degraded) == 1) {
                  /*
                   * Don't fail the drive, act as though we were just a
                   * normal single drive.
                   * However don't try a recovery from this drive as
                   * it is very likely to fail.
                   */
                  conf->recovery_disabled = mddev->recovery_disabled;
                  return;
          }
          ...
}

Quoted 7 lines

The end result is that even if all the drives are physically gone, there still one drive remains in array forever, and mdadm continues to report the array is degraded instead of failed. RAID10 also has similar behavior.

Is there any reason we absolutely don't want to fail the last drive of RAID1?

When a RAID1 only has one drive remaining, then it should act as much as possible like a single plain ordinary drive.

How does /dev/sda behave when you physically remove the device? md0 (as a raid1 with one drive) should do the same.

NeilBrown

Hi Neil,

I see, that does make sense. Thank you.

But it impose a problem for HA. We have 2 nodes as active-standby pair, if HW on node 1 have problem (e.g. SAS cable get pulled, thus all access to physical drives are gone), we hope the array failover to node 2. But with lingering drive reference, mdadm will report array is still alive thus failover won't happen.

I guess it depends on what kind of error on the drive. If it's just a media error we should keep it online as much as possible. But if the drive is really bad or physically gone, keeping the stale reference won't help anything. Back to your comparison with single drive /dev/sda, I think MD as an array should do the same as /dev/sda, not the individual drive inside MD, for them we should just let it go. How do you think?

Quoted 10 lines · On Wed, 04 Mar 2015 15:48:57 -0700 Eric Mei <[email protected]> wrote:

Hi Neil,

I see, that does make sense. Thank you.

But it impose a problem for HA. We have 2 nodes as active-standby pair, if HW on node 1 have problem (e.g. SAS cable get pulled, thus all access to physical drives are gone), we hope the array failover to node 2. But with lingering drive reference, mdadm will report array is still alive thus failover won't happen.

Quoted 7 lines

I guess it depends on what kind of error on the drive. If it's just a media error we should keep it online as much as possible. But if the drive is really bad or physically gone, keeping the stale reference won't help anything. Back to your comparison with single drive /dev/sda, I think MD as an array should do the same as /dev/sda, not the individual drive inside MD, for them we should just let it go. How do you think?

If there were some what that md could be told that the device really was gone and just just returning errors, then I would be OK with it being marked as faulty and being removed from the array.

I don't think there is any mechanism in the kernel to allow that. It would be easiest to capture a "REMOVE" event via udev, and have udev run "mdadm" to tell the md array that the device was gone.

Currently there is no way to do that ... I guess we could change raid1 so that a 'fail' event that came from user-space would always cause the device to be marked failed, even when an IO error would not... To preserve current behaviour, it should require something like "faulty-force" to be written to the "state" file. We would need to check that raid1 copes with having zero working drives - currently it might always assume there is at least one device.

Quoted 28 lines · NeilBrown

Quoted 10 lines · On Wed, 04 Mar 2015 15:48:57 -0700 Eric Mei <[email protected]> wrote:

Hi Neil,

I see, that does make sense. Thank you.

But it impose a problem for HA. We have 2 nodes as active-standby pair, if HW on node 1 have problem (e.g. SAS cable get pulled, thus all access to physical drives are gone), we hope the array failover to node 2. But with lingering drive reference, mdadm will report array is still alive thus failover won't happen.

Quoted 7 lines

I guess it depends on what kind of error on the drive. If it's just a media error we should keep it online as much as possible. But if the drive is really bad or physically gone, keeping the stale reference won't help anything. Back to your comparison with single drive /dev/sda, I think MD as an array should do the same as /dev/sda, not the individual drive inside MD, for them we should just let it go. How do you think?

If there were some what that md could be told that the device really was gone and just just returning errors, then I would be OK with it being marked as faulty and being removed from the array.

I don't think there is any mechanism in the kernel to allow that. It would be easiest to capture a "REMOVE" event via udev, and have udev run "mdadm" to tell the md array that the device was gone.

Quoted 9 lines

Currently there is no way to do that ... I guess we could change raid1 so that a 'fail' event that came from user-space would always cause the device to be marked failed, even when an IO error would not... To preserve current behaviour, it should require something like "faulty-force" to be written to the "state" file. We would need to check that raid1 copes with having zero working drives - currently it might always assume there is at least one device.

Sorry to butt in, but I'm finding this conversation a bit surreal ... take everything I say with a pinch of salt. But the really weird bit was "what does linux do if /dev/sda disappears?"

In the old days, with /dev/hd*, the * had a hard mapping to the hardware. hda was the ide0 primary, hdd was the ide1 secondary, etc etc. I think I ran several systems with just hdb and hdd. Not a good idea, but.

Nowadays, with sd*, the letter is assigned in order of finding the drive. So if sda is removed, linux moves all the other drives and what was sdb becomes sda. Which is why you're advised now to always refer to drives by their BLKDEV or whatever, as linux provides no guarantees whatsoever about sd*. The blockdev may only be a symlink to whatever the sd*n code of the disk is, but it makes sure you get the disk you want when the sd*n changes under you.

Equally surreal is the comment about "what does raid1 do with no working devices?". Surely it will do nothing, if there's no spinning rust or whatever underneath it? You can't corrupt it if there's nothing there to corrupt?

Sorry again if this is inappropriate, but you're coming over as so buried in the trees that you can't see the wood.

Cheers,
Wol

Sorry to butt in, but I'm finding this conversation a bit surreal ... take everything I say with a pinch of salt. But the really weird bit was "what does linux do if /dev/sda disappears?"

In the old days, with /dev/hd*, the * had a hard mapping to the hardware. hda was the ide0 primary, hdd was the ide1 secondary, etc etc. I think I ran several systems with just hdb and hdd. Not a good idea, but.

Nowadays, with sd*, the letter is assigned in order of finding the drive. So if sda is removed, linux moves all the other drives and what was sdb becomes sda.

Which is why you're advised now to always refer
to drives by their BLKDEV or whatever, as linux provides no guarantees whatsoever about sd*. The blockdev may only be a symlink to whatever the sd*n code of the disk is, but it makes sure you get the disk you want when the sd*n changes under you.

Equally surreal is the comment about "what does raid1 do with no working devices?". Surely it will do nothing, if there's no spinning rust or whatever underneath it? You can't corrupt it if there's nothing there to corrupt?

Sorry again if this is inappropriate, but you're coming over as so buried in the trees that you can't see the wood.

Quoted 4 lines · Wols Lists

Sorry to butt in, but I'm finding this conversation a bit surreal ... take everything I say with a pinch of salt. But the really weird bit was "what does linux do if /dev/sda disappears?"

Quoted 8 lines

In the old days, with /dev/hd*, the * had a hard mapping to the hardware. hda was the ide0 primary, hdd was the ide1 secondary, etc etc. I think I ran several systems with just hdb and hdd. Not a good idea, but.

Nowadays, with sd*, the letter is assigned in order of finding the drive. So if sda is removed, linux moves all the other drives and what was sdb becomes sda.

On reboot, yes. Not live. If /dev/sda has anything using it when unplugged, it stays there and gives errors. Existing devices retain their names, and new devices are added to the end. Only if all users are completely unhooked will a kernel name get re-used live.

Quoted 5 lines

Which is why you're advised now to always refer
to drives by their BLKDEV or whatever, as linux provides no guarantees whatsoever about sd*. The blockdev may only be a symlink to whatever the sd*n code of the disk is, but it makes sure you get the disk you want when the sd*n changes under you.

Correct, but once assembled or mounted, the kernel names are locked.

Quoted 4 lines

Equally surreal is the comment about "what does raid1 do with no working devices?". Surely it will do nothing, if there's no spinning rust or whatever underneath it? You can't corrupt it if there's nothing there to corrupt?

It has to stay there to give errors to the upper layers that are still hooked to it. When they are administratively "unhooked", aka unmounted or disassociated with mdadm --remove.

Or, quite possibly, the device is plugged back in, at which point the device name is there for it (as long as you use the same port, of course). In which case the filesystem may very well resume successfully.

Quoted 3 lines

Sorry again if this is inappropriate, but you're coming over as so buried in the trees that you can't see the wood.

Sometimes the expanded view is appropriate :-)

Regards,

Phil Turmel

Quoted 4 lines

Or, quite possibly, the device is plugged back in, at which point the device name is there for it (as long as you use the same port, of course). In which case the filesystem may very well resume successfully.

I was with you right up to this last point.
When a device is unplugged and then plugged back in, it will always get a new name.

It has to stay there to give errors to the upper layers that are still hooked to it. When they are administratively "unhooked", aka unmounted or disassociated with mdadm --remove.

Or, quite possibly, the device is plugged back in, at which point the device name is there for it (as long as you use the same port, of course). In which case the filesystem may very well resume successfully.

Quoted 10 lines · On Wed, 04 Mar 2015 15:48:57 -0700 Eric Mei <[email protected]> wrote:

Hi Neil,

I see, that does make sense. Thank you.

But it impose a problem for HA. We have 2 nodes as active-standby pair, if HW on node 1 have problem (e.g. SAS cable get pulled, thus all access to physical drives are gone), we hope the array failover to node 2. But with lingering drive reference, mdadm will report array is still alive thus failover won't happen.

Quoted 7 lines

I guess it depends on what kind of error on the drive. If it's just a media error we should keep it online as much as possible. But if the drive is really bad or physically gone, keeping the stale reference won't help anything. Back to your comparison with single drive /dev/sda, I think MD as an array should do the same as /dev/sda, not the individual drive inside MD, for them we should just let it go. How do you think?

If there were some what that md could be told that the device really was gone and just just returning errors, then I would be OK with it being marked as faulty and being removed from the array.

I don't think there is any mechanism in the kernel to allow that. It would be easiest to capture a "REMOVE" event via udev, and have udev run "mdadm" to tell the md array that the device was gone.

Currently there is no way to do that ... I guess we could change raid1 so that a 'fail' event that came from user-space would always cause the device to be marked failed, even when an IO error would not...
To preserve current behaviour, it should require something like "faulty-force"
to be written to the "state" file. We would need to check that raid1 copes
with having zero working drives - currently it might always assume there is at least one device.