Removing a failing drive from multiple arrays

I have a failing drive, and partitions are in multiple arrays. I'm looking for the least painful and most reliable way to replace it. It's internal, I have a twin in an external box, and can create all the parts now and then swap the drive physically. The layout is complex, here's what blkdevtra tells me about this device, the full trace is attached.

Block device sdd, logical device 8:48

Model Family:     Seagate Barracuda 7200.10
Device Model:     ST3750640AS
Serial Number:    5QD330ZW
     Device size   732.575 GB
            sdd1     0.201 GB
            sdd2     3.912 GB
            sdd3    24.419 GB
            sdd4     0.000 GB
            sdd5    48.838 GB [md123] /mnt/workspace
            sdd6     0.498 GB
            sdd7    19.543 GB [md125]
            sdd8    29.303 GB [md126]
            sdd9   605.859 GB [md127] /exports/common

Unpartitioned 0.003 GB

I think what I want to do is to partition the new drive, then one array at a time fail and remove the partition on the bad drive, and add a partition on the new good drive. Then repeat for each array until all are complete and on a new drive. Then I should be able to power off, remove the failed drive, put the good drive in the case, and the arrays should reassemble by UUID.

Does that sound right? Is there an easier way?

Quoted 22 lines · On Thu, 19 Apr 2012 14:54:30 -0400 Bill Davidsen<[email protected]> wrote:

I have a failing drive, and partitions are in multiple arrays. I'm looking for the least painful and most reliable way to replace it. It's internal, I have a twin in an external box, and can create all the parts now and then swap the drive physically. The layout is complex, here's what blkdevtra tells me about this device, the full trace is attached.

Block device sdd, logical device 8:48

Model Family:     Seagate Barracuda 7200.10
Device Model:     ST3750640AS
Serial Number:    5QD330ZW
      Device size   732.575 GB
             sdd1     0.201 GB
             sdd2     3.912 GB
             sdd3    24.419 GB
             sdd4     0.000 GB
             sdd5    48.838 GB [md123] /mnt/workspace
             sdd6     0.498 GB
             sdd7    19.543 GB [md125]
             sdd8    29.303 GB [md126]
             sdd9   605.859 GB [md127] /exports/common
    Unpartitioned     0.003 GB

Quoted 9 lines

I think what I want to do is to partition the new drive, then one array at a time fail and remove the partition on the bad drive, and add a partition on the new good drive. Then repeat for each array until all are complete and on a new drive. Then I should be able to power off, remove the failed drive, put the good drive in the case, and the arrays should reassemble by UUID.

Does that sound right? Is there an easier way?

I would add the new partition before failing the old but that isn't a big issues.

If you were running a really new kernel, used 1.x metadata, and were happy to try out code that that hasn't had a lot of real-life testing you could (after adding the new partition) do
echo want_replacement> /sys/block/md123/md/dev-sdd5/state
(for example).

Then it would build the spare before failing the original. You need linux 3.3 for this to have any chance of working.

Bill> I have a failing drive, and partitions are in multiple Bill> arrays.

Ugh! Why? This is why I love LVM on top of MD. I just mirror drives, then carve them up as needed. Yes, you need to have two (or more) drives of the same approximate size, but that's easy.

Mirroring partitions just seems to be asking for trouble to me.

On Fri, Apr 20, 2012 at 9:31 AM, John Robinson
<[email protected]> wrote:

Quoted 3 lines

On small machines (3-6 drives) I will regularly have a RAID-1 /boot, RAID-10 swap and RAID-5 or 6 everything else, done with partitions. I do use LVM too though - that "everything else" will be LVM over the big RAID.

Thanks for the info John. Can you tell me what are the requirements for the RAID-1 /boot? grub2? initrd? BIOS-based RAID? Something else? I'm still just mirroring my boot drives - booting from sda1 but copying everything to /sdb1, sdc1, etc. I think I'd like to go full RAID on /boot if the requirements are too high.

Quoted 22 lines · On Thu, 19 Apr 2012 14:54:30 -0400 Bill Davidsen<[email protected]> wrote:

I have a failing drive, and partitions are in multiple arrays. I'm looking for the least painful and most reliable way to replace it. It's internal, I have a twin in an external box, and can create all the parts now and then swap the drive physically. The layout is complex, here's what blkdevtra tells me about this device, the full trace is attached.

Block device sdd, logical device 8:48

Model Family:     Seagate Barracuda 7200.10
Device Model:     ST3750640AS
Serial Number:    5QD330ZW
      Device size   732.575 GB
             sdd1     0.201 GB
             sdd2     3.912 GB
             sdd3    24.419 GB
             sdd4     0.000 GB
             sdd5    48.838 GB [md123] /mnt/workspace
             sdd6     0.498 GB
             sdd7    19.543 GB [md125]
             sdd8    29.303 GB [md126]
             sdd9   605.859 GB [md127] /exports/common
    Unpartitioned     0.003 GB

Quoted 9 lines

I think what I want to do is to partition the new drive, then one array at a time fail and remove the partition on the bad drive, and add a partition on the new good drive. Then repeat for each array until all are complete and on a new drive. Then I should be able to power off, remove the failed drive, put the good drive in the case, and the arrays should reassemble by UUID.

Does that sound right? Is there an easier way?

I would add the new partition before failing the old but that isn't a big issues.

If you were running a really new kernel, used 1.x metadata, and were happy to try out code that that hasn't had a lot of real-life testing you could (after adding the new partition) do
echo want_replacement> /sys/block/md123/md/dev-sdd5/state
(for example).

Then it would build the spare before failing the original. You need linux 3.3 for this to have any chance of working.

NeilBrown

Quoted 45 lines · NeilBrown

Quoted 22 lines · On Thu, 19 Apr 2012 14:54:30 -0400 Bill Davidsen<[email protected]> wrote:

I have a failing drive, and partitions are in multiple arrays. I'm looking for the least painful and most reliable way to replace it. It's internal, I have a twin in an external box, and can create all the parts now and then swap the drive physically. The layout is complex, here's what blkdevtra tells me about this device, the full trace is attached.

Block device sdd, logical device 8:48

Model Family:     Seagate Barracuda 7200.10
Device Model:     ST3750640AS
Serial Number:    5QD330ZW
      Device size   732.575 GB
             sdd1     0.201 GB
             sdd2     3.912 GB
             sdd3    24.419 GB
             sdd4     0.000 GB
             sdd5    48.838 GB [md123] /mnt/workspace
             sdd6     0.498 GB
             sdd7    19.543 GB [md125]
             sdd8    29.303 GB [md126]
             sdd9   605.859 GB [md127] /exports/common
    Unpartitioned     0.003 GB

Quoted 9 lines

I think what I want to do is to partition the new drive, then one array at a time fail and remove the partition on the bad drive, and add a partition on the new good drive. Then repeat for each array until all are complete and on a new drive. Then I should be able to power off, remove the failed drive, put the good drive in the case, and the arrays should reassemble by UUID.

Does that sound right? Is there an easier way?

I would add the new partition before failing the old but that isn't a big issues.

If you were running a really new kernel, used 1.x metadata, and were happy to try out code that that hasn't had a lot of real-life testing you could (after adding the new partition) do
echo want_replacement> /sys/block/md123/md/dev-sdd5/state
(for example).

Then it would build the spare before failing the original. You need linux 3.3 for this to have any chance of working.

Quoted 1 line

NeilBrown

I expect to try this in a real world case tomorrow. Am I so lucky that when rebuilding the failing drive will be copies in a way which uses a recovered value for the chunk if there's a bad block? And only if there's a bad block, so that possible evil on the other drives would not be a problem unless they were at the same chunk?

As soon as the pack of replacements arrives I'll let you know how well this worked, if at all.

Quoted 22 lines · On Thu, 19 Apr 2012 14:54:30 -0400 Bill Davidsen<[email protected]> wrote:

I have a failing drive, and partitions are in multiple arrays. I'm looking for the least painful and most reliable way to replace it. It's internal, I have a twin in an external box, and can create all the parts now and then swap the drive physically. The layout is complex, here's what blkdevtra tells me about this device, the full trace is attached.

Block device sdd, logical device 8:48

Model Family:     Seagate Barracuda 7200.10
Device Model:     ST3750640AS
Serial Number:    5QD330ZW
      Device size   732.575 GB
             sdd1     0.201 GB
             sdd2     3.912 GB
             sdd3    24.419 GB
             sdd4     0.000 GB
             sdd5    48.838 GB [md123] /mnt/workspace
             sdd6     0.498 GB
             sdd7    19.543 GB [md125]
             sdd8    29.303 GB [md126]
             sdd9   605.859 GB [md127] /exports/common
    Unpartitioned     0.003 GB

Quoted 9 lines

I think what I want to do is to partition the new drive, then one array at a time fail and remove the partition on the bad drive, and add a partition on the new good drive. Then repeat for each array until all are complete and on a new drive. Then I should be able to power off, remove the failed drive, put the good drive in the case, and the arrays should reassemble by UUID.

Does that sound right? Is there an easier way?

I would add the new partition before failing the old but that isn't a big issues.

If you were running a really new kernel, used 1.x metadata, and were happy to try out code that that hasn't had a lot of real-life testing you could (after adding the new partition) do
echo want_replacement> /sys/block/md123/md/dev-sdd5/state
(for example).

Then it would build the spare before failing the original. You need linux 3.3 for this to have any chance of working.

Quoted 2 lines · Mark Knecht

On Fri, Apr 20, 2012 at 9:31 AM, John Robinson
<[email protected]> wrote:

[...]

Quoted 9 lines

Quoted 3 lines

On small machines (3-6 drives) I will regularly have a RAID-1 /boot, RAID-10 swap and RAID-5 or 6 everything else, done with partitions. I do use LVM too though - that "everything else" will be LVM over the big RAID.

Thanks for the info John. Can you tell me what are the requirements for the RAID-1 /boot? grub2? initrd? BIOS-based RAID? Something else? I'm still just mirroring my boot drives - booting from sda1 but copying everything to /sdb1, sdc1, etc. I think I'd like to go full RAID on /boot if the requirements are too high.

Pretty much any modern distro's installer will do the right thing with whatever boot loader it uses. Also, there are so many boot methods - BIOS/MBR, GPT, UEFI
- that in one short email it's tricky to give advice in one short email. It's all out there via Google.

But no, you can do a RAID-1 /boot with LILO or grub, and without BIOS RAID. The BIOS will boot off the first available hard drive and doesn't understand md RAID. grub doesn't understand md RAID either. You have to make your md RAID-1 /boot with metadata 1.0 (or 0.90) because they have the data at the beginning so when you create a filesystem on the array, each individual component (partition) looks like it has a filesystem on it. You install grub (or LILO) onto the MBR (or GPT boot partition, which isn't the same as your /boot partition), pointing to the partition (not the md array).

grub2 does understand md RAID, but has to be loaded by the BIOS, so there are still restrictions.

Without either hardware or BIOS RAID, you can still end up being unable to boot, e.g. the BIOS will try to boot from the first hard drive present, but if it has bad sectors in the MBR or /boot partition, booting may fail even though there's a perfectly good mirror on the second drive, because the BIOS doesn't understand RAID. This has happened to me :-(

John Robinson wrote:

Quoted 10 lines

Without either hardware or BIOS RAID, you can still end up being unable to boot,
e.g. the BIOS will try to boot from the first hard drive present, but if it has
bad sectors in the MBR or /boot partition, booting may fail even though there's
a perfectly good mirror on the second drive, because the BIOS doesn't understand
RAID. This has happened to me :-(

Doesn't need to understand RAID, just be willing to try the next item in the boot list on failure. My experience has been that almost every BIOS will try the 2nd item if the 1st fails totally (ie. drive isn't there). A _good_ BIOS will try the next item on sector error in the MBR.

After that the BIOS needs to understand a lot more to do anything smart after the MBR runs.

I have put MBR and boot partition on a USB thumb drive because the failure rate of a R/O flash is lower than rotating devices (in my experience). Use ext2 for boot, no journal so the drive works really read-only. Hopefully grub2 mounts the boot noatime.

Quoted 4 lines · Bill Davidsen

I have put MBR and boot partition on a USB thumb drive because the failure rate of a R/O flash is lower than rotating devices (in my experience). Use ext2 for boot, no journal so the drive works really read-only. Hopefully grub2 mounts the boot noatime.

Another option, although I've not done this for a long time, is PXE boot. You need a DHCP server giving out the correct parameters and a TFTP server for the kernel (and ramdisk?)

Quoted 4 lines

Another option, although I've not done this for a long time, is PXE boot. You need a DHCP server giving out the correct parameters and a TFTP server for the kernel (and ramdisk?)

I think that's addressing one point of failure while adding more.

Quoted 5 lines · Bill Davidsen

Quoted 4 lines

Another option, although I've not done this for a long time, is PXE boot. You need a DHCP server giving out the correct parameters and a TFTP server for the kernel (and ramdisk?)

I think that's addressing one point of failure while adding more.

Alternate point of view: you need reliable DNS anyway, and it's no harder to make reliable DHCP than reliable DNS (you just have two of them). Furthermore, this only needs to be available at the time a server boots up.

I worked in one place where they made all the servers (which were VMs) pick up their IPs via DHCP. This allowed them to dump the images across to the disaster recovery site, boot them up there, and bring them all up on new IPs without touching any configs. It worked well. They ran DHCP service on the Cisco L3 switches that the VM hosts were plugged into, on the basis that this was the most reliable kit that they had.

Like you say, it's a balancing act of cost, complexity, and reliability, and is also coloured by experience. I've had not-so-good experiences with USB thumb drives.