Skip to contentSkip to conversation

Replacing a RAID1 drive that has not failed.

DISCUSSION Doug Herr 4 participants 6 messages Oct 26, 2015 – Oct 28, 2015 1 message not in the archive MD RAID

lore
Share & tools
lore

Doug Herr

author
#0 lore raw

I think it might be time to replace one or both of my RAID 1 drives. They have been giving good service for six years. Both still have good smartctl reports but I am ready to move one out to be used as a member of my backup solution using old drives to hold rsync backups. I would thus give my RAID a fresh disk before there is an actual failure.

I recently experimented with adding a third partition to my swap raid partition and that went very well, so I was thinking that this would provide a safer method to replace one drive compared to failing it and then replacing it.

My test makes me think it might be as easy as:

1. Partition new drive (plugged in via external SATA dock)
Use: fdisk /dev/sdc

2. Add 3rd partition to each RAID.
Use: mdadm /dev/md4 --grow --raid-devices=3 --add /dev/sdc2

3. Wait for all the resyncs.
Use: watch cat /proc/mdstat

4. Fail out sda for each RAID.
Use: mdadm /dev/md4 --fail /dev/sda2

5. Drop back down to two disk.
Use: mdadm /dev/md4 --grow --raid-devices=2

6. Make the old partitions "safe".
Use: mdadm --zero-superblock /dev/sdc2

7. Power down and swap sdc into sda slot.

I only tested that with version 1.2 metadata, so I would be interested if there is anything above that would not work with version 0.90. Also note that I did my test using my swap partition but it seems like the procedure should not care what sort of partitioning is used.

Below is info about my setup and how I tested to come up with the above steps...

I am:
15:40-doug@wombat-~>uname -a
Linux wombat.wombatz.com 4.2.3-200.fc22.x86_64 #1 SMP Thu Oct 8 03:23:55 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

15:40-doug@wombat-~>cat /etc/fedora-release
Fedora release 22 (Twenty Two)

15:40-doug@wombat-~>mdadm -V
mdadm - v3.3.2 - 21st August 2014

My RAID setup:

(I am not using RAID for /boot, instead I rsync /boot to /bootalt each night. Otherwise everything is on RAID 1, including swap.)

15:23-doug@wombat-~/wombat-4/raid/current-2015-10-26>df -hT |grep -v ^tmpfs

Filesystem     Type      Size  Used Avail Use% Mounted on
devtmpfs       devtmpfs  4.4G     0  4.4G   0% /dev
/dev/md126     ext4       30G   11G   18G  39% /
/dev/sdb1      ext4      579M  126M  411M  24% /bootalt
/dev/md1       ext4       40G   18G   23G  44% /home
/dev/md123     ext4      422G  209G  192G  53% /data2
/dev/md125     ext4      422G   74G  327G  19% /data1
/dev/sda1      ext4      579M  126M  411M  24% /boot

15:09-doug@wombat-~>swapon -s
Filename Type Size Used
Priority
/dev/md4 partition 4946980 0 -1

(I am using UUID for /etc/mdadm.conf and in /etc/fstab so I did not bother to "fix" the names when they got changed to md12x during Fedora upgrades.)

15:19-doug@wombat-~>cat /proc/mdstat
Personalities : [raid1]
md123 : active raid1 sdb7[1] sda7[0]

      448888128 blocks [2/2] [UU]
      bitmap: 0/4 pages [0KB], 65536KB chunk

md125 : active raid1 sdb6[1] sda6[0]

      448888128 blocks [2/2] [UU]
      bitmap: 1/4 pages [4KB], 65536KB chunk

md1 : active raid1 sdb5[0] sda5[2]

      41952620 blocks super 1.2 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

md126 : active raid1 sdb3[0] sda3[1]

      31463232 blocks [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

md4 : active raid1 sdb2[2] sda2[3]

      4946984 blocks super 1.2 [2/2] [UU]

unused devices: <none>

Notes from the test I did using a partion on a older and smaller drive...

[root@wombat Documents]# mdadm --detail /dev/md4
/dev/md4:

        Version : 1.2

Creation Time : Sat Mar 24 16:53:28 2012
Raid Level : raid1
Array Size : 4946984 (4.72 GiB 5.07 GB)
Used Dev Size : 4946984 (4.72 GiB 5.07 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Fri Oct 23 10:55:06 2015
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Name : wombat.wombatz.com:4 (local to host wombat.wombatz.com) UUID : 2382da1b:c77c04c7:9e4f955c:9450c809
Events : 541

Number Major Minor RaidDevice State
3 8 2 0 active sync /dev/sda2
2 8 18 1 active sync /dev/sdb2

[root@wombat Documents]# fdisk /dev/sdc
[snip]

[root@wombat doug]# mdadm /dev/md4 --grow --raid-devices=3 --add /dev/sdc2
mdadm: added /dev/sdc2
unfreeze

[root@wombat Documents]# cat /proc/mdstat
[snip]
md4 : active raid1 sdc2[4] sdb2[2] sda2[3]

      4946984 blocks super 1.2 [3/2] [UU_]
      [=======>.............]  recovery = 39.2% (1941056/4946984) 

finish=0.6min speed=71890K/sec

Soon it showed:

md4 : active raid1 sdc2[4] sdb2[2] sda2[3]

      4946984 blocks super 1.2 [3/3] [UUU]

And now:

[root@wombat Documents]# mdadm --detail /dev/md4
/dev/md4:

        Version : 1.2

Creation Time : Sat Mar 24 16:53:28 2012
Raid Level : raid1
Array Size : 4946984 (4.72 GiB 5.07 GB)
Used Dev Size : 4946984 (4.72 GiB 5.07 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent

Update Time : Sat Oct 24 12:57:39 2015
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0

Name : wombat.wombatz.com:4 (local to host wombat.wombatz.com) UUID : 2382da1b:c77c04c7:9e4f955c:9450c809
Events : 562

Number Major Minor RaidDevice State
3 8 2 0 active sync /dev/sda2
2 8 18 1 active sync /dev/sdb2
4 8 33 2 active sync /dev/sdc2

Now try to remove it:
[root@wombat doug]# mdadm /dev/md4 --fail /dev/sdc2
mdadm: set /dev/sdc2 faulty in /dev/md4

[root@wombat doug]# mdadm /dev/md4 --grow --raid-devices=2 raid_disks for /dev/md4 set to 2
unfreeze

Seems good:
md4 : active raid1 sdc2[4](F) sdb2[2] sda2[3]

      4946984 blocks super 1.2 [2/2] [UU]

Just to be safe:
mdadm --zero-superblock /dev/sdc2

Signature · 1 line
Doug Herr 

A message that is not in the archive (sent off-list or not yet received): CACsGCyTM_J2+_oJm0rUsZvuF0AXXu6xX572F4HP+OGsjP_AvfQ@mail.gmail.com

#2 lore raw
Quoted 6 lines · Doug Herr

I think it might be time to replace one or both of my RAID 1 drives. [snip] I recently experimented with adding a third partition to my swap raid partition and that went very well, so I was thinking that this would provide a safer method to replace one drive compared to failing it and then replacing it.
[snip]

I tried basically the procedure you described, based on Internet research, to replace a working but suspect drive in a RAID. I ran into problems. FWIW, my metadata format is 1.2. You can see the thread at http://marc.info/?l=linux-raid&m=143668094907891&w=2
although the most pertinent EMail is the first response
http://marc.info/?l=linux-raid&m=143670871711935&w=2

I suppose it's possible that you might get different advice since you're doing RAID 1 and I was doing RAID 5. I'm not an expert. I just wanted to warn you that I ran into trouble basically doing what you propose. The advice I got was that I shouldn't grow and shrink to replace a disk. There's a --replace option for that. That'll remove a few of your steps, simplify and speed up the process, and remove risk. Hope this helps. Hopefully someone who knows a lot more than I do will also speak up. Good luck.

Eddie

author
#3 lore raw
In reply to Doug Herr

Thanks to both Wol and Eddie for pointing me to the --replace option and for not simply telling me to RTFM. Part of my confusion was based on my last attempt (years back) to find out if three drive RAID 1 was even an option. Just as before I found stuff that laughed at the mere idea of a three drive mirror. Seems people were getting too tied up on the word "mirror", which is not really a proper name for it in my view. The mirror copy of myself when looking into a mirror is *not* the same in me in so many ways. Oh, and now I think I understand that hardware RAID rarely or never allows it but Linux md RAID has it now and maybe had it the whole time.

That said, I really should have slogged thru the man page better. It does seem that --replace includes the key feature that I want:

"the device remains in service during the recovery process to increase resilience against multiple failures."

So, just in case somebody wants to see if they can find anything I missed, here is my current plan...

1. Partition new drive (plugged in via external SATA dock)
#fdisk /dev/sdc
(make it an exact match of sda/sdb unless it turns out to be smaller,
in which case I can shrink /boot to make room.)

2. Replace sda partition with sdc partition for each RAID.
(Could do all at once but feels safer to do one at a time.) #mdadm /dev/md123 --replace /dev/sda7 --with /dev/sdc7 #watch /proc/mdstat
#mdadm /dev/md125 --replace /dev/sda6 --with /dev/sdc6 #watch /proc/mdstat
#mdadm /dev/md126 --replace /dev/sda3 --with /dev/sdc3 #watch /proc/mdstat
#mdadm /dev/md1 --replace /dev/sda5 --with /dev/sdc5 #watch /proc/mdstat
#mdadm /dev/md4 --replace /dev/sda2 --with /dev/sdc2 #watch /proc/mdstat

3. All RAIDs should be "active" and in "[UU]" status, with no sda partitions.
#cat /proc/mdstat

(Consider if this should only be done later/never so that sda can
be a form of backup in case of issues during the post replace reboot.)
4. Prevent the old partitions from ever being pulled into the RAIDs.
#mdadm --zero-superblock /dev/sda7
#mdadm --zero-superblock /dev/sda6
#mdadm --zero-superblock /dev/sda3
#mdadm --zero-superblock /dev/sda5
#mdadm --zero-superblock /dev/sda2

5. Make a new /boot on sdc.
(using rsync since the language is taken from my daily cron and this is a
tad quicker if I end up running it more than once.)
(fstab line needing update: UUID=52b14d98-b284-41a0-a36f-459ae3ae12a7 /boot ext4 defaults 1 2) #mkfs.ext4 /dev/sdc1
#mkdir /bootnew
#mount /dev/sdc1 /bootnew
#rsync -a --delete /boot/ /bootnew
#grub-install /dev/sdc
#blkid /dev/sdc1
#vim /etc/fstab
#umount /bootnew ; rmdir /bootnew

6. Power down and swap sdc into sda "slot".

7. Make sure that all RAIDs are "active" and in "[UU]" status:
#cat /proc/mdstat

Signature · 1 line
Doug Herr 
#4 lore raw
Quoted 4 lines · Doug Herr

1. Partition new drive (plugged in via external SATA dock)
#fdisk /dev/sdc
(make it an exact match of sda/sdb unless it turns out to be smaller,
in which case I can shrink /boot to make room.)

man gdisk.

Look at the "u" option in the "expert" menu - it'll probably save you some work :-)

Cheers,
Wol

#5 lore raw
Quoted 10 lines · On Tue, 27 Oct 2015 21:31:01 +0000 Wols Lists <[email protected]> wrote:
Quoted 4 lines · Doug Herr

1. Partition new drive (plugged in via external SATA dock)
#fdisk /dev/sdc
(make it an exact match of sda/sdb unless it turns out to be smaller,
in which case I can shrink /boot to make room.)

man gdisk.

Look at the "u" option in the "expert" menu - it'll probably save you some work :-)

Unfamiliar with 'gdisk', but when I need the exact same set of partitions on another drive, I use sfdisk:

# sfdisk -d /dev/sda > sda.sf
# sfdisk /dev/sdb < sda.sf

You could even do it in one step:

# sfdisk -d /dev/sda | sfdisk /dev/sdb
Signature · 2 lines
With respect,
Roman
author
#6 lore raw
Quoted 9 lines · Roman Mamedov
Quoted 6 lines · On Tue, 27 Oct 2015 21:31:01 +0000 Wols Lists <[email protected]> wrote:
Quoted 4 lines · Doug Herr

1. Partition new drive (plugged in via external SATA dock)
#fdisk /dev/sdc
(make it an exact match of sda/sdb unless it turns out to be smaller,
in which case I can shrink /boot to make room.)

Quoted 6 lines
Quoted 4 lines

man gdisk.

Look at the "u" option in the "expert" menu - it'll probably save you some work :-)

Unfamiliar with 'gdisk', but when I need the exact same set of

partitions on

Quoted 8 lines

another drive, I use sfdisk:

# sfdisk -d /dev/sda > sda.sf
# sfdisk /dev/sdb < sda.sf

You could even do it in one step:

# sfdisk -d /dev/sda | sfdisk /dev/sdb

Thanks much to all who have replied and all who post here.

Yup, the sfdisk copy is one that I am ready to use if needed but your replies have triggered better googling and I am now planning to use fdisk (I am in Fedora 22) to manually partition the new disk. I don't think I have any need for GPT yet but what I realized is that my current set of disks was created when fdisk was still starting at sector 63 where now it uses 2048. I don't think it matters much for my current set of Hitachi HDT721010SLA360 drives. It might not even matter for the slightly newer "E7K1000" that is on the way to me, but I do want to stick with the current standards as best I can. I have room to tweak it since I have a non-RAID /boot partition that is easily large enough to lend space as needed.

I do plan to play with GPT to learn about it and to be more ready for the next computer if it is the default at that point.

Oh, I also added a step in my plan. Adding my new partitions as spares first will alert me of size errors (if smaller) *before* any of the replacement jobs kick off:

For each:

# mdadm /dev/md4 --add-spare /dev/sdc2

If the above does not complain then move forward for each:

# mdadm /dev/md3 --replace /dev/sda2 --with /dev/sdc2

And fear not, I will be updating my backups before moving forward with this.

Signature · 1 line
Doug Herr