Raid not shutting down when disks are lost?
Today one RAID6 array I manage decided to lose four out of eight disks. Oddly enough, the array did not shut down but instead I got intermittent read and writer errors from the filesystem.
It's been some time since I had a failure of this magnitude, but I seem to recall that once the array lost too many disks, it would shut down and refuse to write a single byte. The nice effect of this was that if it was a temporary error, you could just reboot and the array would start nicely (albeit in degraded mode).
Has something changed? Is this perhaps an effect of using RAID6 (I used to run RAID5 arrays)? Or was I simply lucky the previous instances I've had?
Related, it would be nice if you could control how it handles lost disks. E.g. I'd like it to go read-only when it goes in to fully degraded mode. In case the last disk lost was only a temporary glitch, the array could be made to recover without a lengthy resync.
Rgds
Signature · 6 lines
-- Pierre Ossman WARNING: This correspondence is being monitored by the Swedish government. Make sure your server uses encryption for SMTP traffic and consider using PGP for end-to-end encryption.
Neil?
Quoted 20 lines · On Thu, 8 Oct 2009 16:39:52 +0200 Pierre Ossman <[email protected]> wrote:
Today one RAID6 array I manage decided to lose four out of eight disks. Oddly enough, the array did not shut down but instead I got intermittent read and writer errors from the filesystem.
It's been some time since I had a failure of this magnitude, but I seem to recall that once the array lost too many disks, it would shut down and refuse to write a single byte. The nice effect of this was that if it was a temporary error, you could just reboot and the array would start nicely (albeit in degraded mode).
Has something changed? Is this perhaps an effect of using RAID6 (I used to run RAID5 arrays)? Or was I simply lucky the previous instances I've had?
Related, it would be nice if you could control how it handles lost disks. E.g. I'd like it to go read-only when it goes in to fully degraded mode. In case the last disk lost was only a temporary glitch, the array could be made to recover without a lengthy resync.
Rgds
Signature · 6 lines
-- Pierre Ossman WARNING: This correspondence is being monitored by FRA, a Swedish intelligence agency. Make sure your server uses encryption for SMTP traffic and consider using PGP for end-to-end encryption.
Quoted 8 lines · Pierre Ossman
Neil?
Quoted 3 lines · On Thu, 8 Oct 2009 16:39:52 +0200 Pierre Ossman <[email protected]> wrote:
Today one RAID6 array I manage decided to lose four out of eight disks. Oddly enough, the array did not shut down but instead I got intermittent read and writer errors from the filesystem.
This is expected.
The array can't shutdown when there is a mounted filesystem. Reads may still be serviced from the survivors, all writes should be aborted with an error.
Quoted 10 lines
Quoted 10 lines
It's been some time since I had a failure of this magnitude, but I seem to recall that once the array lost too many disks, it would shut down and refuse to write a single byte. The nice effect of this was that if it was a temporary error, you could just reboot and the array would start nicely (albeit in degraded mode).
Has something changed? Is this perhaps an effect of using RAID6 (I used to run RAID5 arrays)? Or was I simply lucky the previous instances I've had?
It should not come back up nicely in this scenario. You need "--force" to attempt to reassemble a failed array.
Quoted 6 lines
Quoted 6 lines
Related, it would be nice if you could control how it handles lost disks. E.g. I'd like it to go read-only when it goes in to fully degraded mode. In case the last disk lost was only a temporary glitch, the array could be made to recover without a lengthy resync.
When you say "fully-degraded" do you mean "failed"? In general the bitmap mechanism provides fast resync after temporary disk outages.
--
Dan
Quoted 10 lines · On Sat, 21 Nov 2009 12:21:58 -0700 Dan Williams <[email protected]>…
Quoted 8 lines · Pierre Ossman
Neil?
Quoted 3 lines · On Thu, 8 Oct 2009 16:39:52 +0200 Pierre Ossman <[email protected]> wrote:
Today one RAID6 array I manage decided to lose four out of eight disks. Oddly enough, the array did not shut down but instead I got intermittent read and writer errors from the filesystem.
Quoted 6 lines
This is expected.
The array can't shutdown when there is a mounted filesystem. Reads may still be serviced from the survivors, all writes should be aborted with an error.
It could "shut down" in the sense that it refuses to touch the underlying hardware and just report errors to upper layers. I.e. don't update the md superblock marking more disks as failed.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
It's been some time since I had a failure of this magnitude, but I seem to recall that once the array lost too many disks, it would shut down and refuse to write a single byte. The nice effect of this was that if it was a temporary error, you could just reboot and the array would start nicely (albeit in degraded mode).
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Has something changed? Is this perhaps an effect of using RAID6 (I used to run RAID5 arrays)? Or was I simply lucky the previous instances I've had?
It should not come back up nicely in this scenario. You need "--force" to attempt to reassemble a failed array.
If the last disk is thrown out either because of a read error, or because of the first write of a stripe (i.e. what's on the platters is still in sync) then a force would not be needed. This requires the md code to not mark that last disk as failed in the superblocks of the remaining disks though.
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Related, it would be nice if you could control how it handles lost disks. E.g. I'd like it to go read-only when it goes in to fully degraded mode. In case the last disk lost was only a temporary glitch, the array could be made to recover without a lengthy resync.
When you say "fully-degraded" do you mean "failed"? In general the bitmap mechanism provides fast resync after temporary disk outages.
Fully degraded means still working but without any redundancy. I.e. one lost disk with RAID 5 or two with RAID 6. And the bitmap mechanism seems to be broken in that case since I always experience full, multi-hour resyncs whenever a disk is lost.
Or is there perhaps some magic mdadm command to add a lost disk and get it up to speed without a complete sync?
Rgds
Signature · 6 lines
-- Pierre Ossman WARNING: This correspondence is being monitored by FRA, a Swedish intelligence agency. Make sure your server uses encryption for SMTP traffic and consider using PGP for end-to-end encryption.
Quoted 2 lines · Pierre Ossman
Or is there perhaps some magic mdadm command to add a lost disk and get it up to speed without a complete sync?
If you care about resync speed then look into adding a write intent bitmap to your configuration; otherwise, look into the --assume-clean option to recreate the array.
Quoting from the man page:
--assume-clean
Tell mdadm that the array pre-existed and is known to be clean.
It can be useful when trying to recover from a major failure as you can be sure that no data will be affected unless you actually write to the array. It can also be used when creating a RAID1 or RAID10 if you want to avoid the initial resync, however this practice, while normally safe, is not recommended. Use this only if you really know what you are doing.
--
Dan