Skip to contentSkip to conversation

Converting 4 disk RAID10 to RAID5

DISCUSSION Adam Goryachev 3 participants 9 messages Oct 26, 2015 – Oct 28, 2015 MD RAID

lore
Share & tools
lore
#0 lore raw

Hi all,

I'm trying to convert a 4 disk RAID10 to a RAID5. Currently I have:
cat /proc/mdstat
Personalities : [raid10]
md0 : active raid10 sdd1[2] sdc1[1] sdb1[0] sde1[3]

      7813772288 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
      bitmap: 0/59 pages [0KB], 65536KB chunk

Disks are:

Model Family:     Western Digital Red (AF)
Device Model:     WDC WD40EFRX-68WT0N0

My plan was to see if mdadm can do this directly, but it seems that it can't:
mdadm --grow --level=5 /dev/md0
mdadm: RAID10 can only be changed to RAID0
unfreeze

(Please let me know if a newer version of kernel/mdadm can do this):
mdadm - v3.3.2 - 21st August 2014
Linux dr 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u2 (2015-07-17) x86_64 GNU/Linux

So, my other idea is:
1) fail two drives from the array:
mdadm --manage /dev/md0 --fail /dev/sdb1 mdadm --manage /dev/md0 --remove /dev/sdb1 mdadm --manage /dev/md0 --fail /dev/sdd1 mdadm --manage /dev/md0 --remove /dev/sdd1 mdadm --misc --zero-superblock /dev/sdb1 mdadm --misc --zero-superblock /dev/sdd1

It seems that RAID10 device number is in order:
sdb1 device0
sdc1 device1
sdd1 device2
sde1 device3
Therefore, I can fail device 0 (sdb1) and device 2 (sdd1) without losing any data.

2) create a 3 disk RAID5 with one disk missing.
mdadm --create /dev/md1 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdd1 missing

3) Then copy all of the existing data across unmount partitions, stop LVM/etc
dd bs=16M if=/dev/md0 of=/dev/md1

4) Finally, stop the md0, and add the two devices to the new raid5, and then grow the array to use the space on the 4th drive. mdadm --manage --stop /dev/md0
mdadm --misc --zero-superblock /dev/sdc1
mdadm --manage /dev/md1 --add /dev/sdc1
mdadm --misc --zero-superblock /dev/sde1
mdadm --manage /dev/md1 --add /dev/sde1
mdadm --grow /dev/md1 --raid-devices=4

5) Add the space to my LVM
pvresize /dev/md1

6) Start up LVM, mount LV's, etc

Does the above sound reasonable? Any other suggestions which would be better/less dangerous? Some more detailed info on my existing array:
mdadm --misc --examine /dev/sdb1
/dev/sdb1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 40a09d68:b217d8ec:c90a61a7:ab35f26e
           Name : backuppc:0

Creation Time : Sat Mar 21 01:19:22 2015
Raid Level : raid10
Raid Devices : 4

Avail Dev Size : 7813772943 (3725.90 GiB 4000.65 GB)

     Array Size : 7813772288 (7451.79 GiB 8001.30 GB)

Used Dev Size : 7813772288 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 4b9d99c9:2a930721:e8052eb2:65121805

Internal Bitmap : 8 sectors from superblock

    Update Time : Mon Oct 26 12:00:12 2015

Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d435138c - correct
Events : 27019

Layout : near=2
Chunk Size : 512K

Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdc1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 40a09d68:b217d8ec:c90a61a7:ab35f26e
           Name : backuppc:0

Creation Time : Sat Mar 21 01:19:22 2015
Raid Level : raid10
Raid Devices : 4

Avail Dev Size : 7813772943 (3725.90 GiB 4000.65 GB)

     Array Size : 7813772288 (7451.79 GiB 8001.30 GB)

Used Dev Size : 7813772288 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : a8486bf8:b0e7c4d7:8e09bdc6:1a5f409b

Internal Bitmap : 8 sectors from superblock

    Update Time : Mon Oct 26 12:00:12 2015

Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 647b63cd - correct
Events : 27019

Layout : near=2
Chunk Size : 512K

Device Role : Active device 1
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdd1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 40a09d68:b217d8ec:c90a61a7:ab35f26e
           Name : backuppc:0

Creation Time : Sat Mar 21 01:19:22 2015
Raid Level : raid10
Raid Devices : 4

Avail Dev Size : 7813772943 (3725.90 GiB 4000.65 GB)

     Array Size : 7813772288 (7451.79 GiB 8001.30 GB)

Used Dev Size : 7813772288 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : c46cdf6f:19f0ea49:1f5cc79a:1df744d7

Internal Bitmap : 8 sectors from superblock

    Update Time : Mon Oct 26 12:00:12 2015

Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 5e247ae6 - correct
Events : 27019

Layout : near=2
Chunk Size : 512K

Device Role : Active device 2
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sde1:

          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 40a09d68:b217d8ec:c90a61a7:ab35f26e
           Name : backuppc:0

Creation Time : Sat Mar 21 01:19:22 2015
Raid Level : raid10
Raid Devices : 4

Avail Dev Size : 7813772943 (3725.90 GiB 4000.65 GB)

     Array Size : 7813772288 (7451.79 GiB 8001.30 GB)

Used Dev Size : 7813772288 (3725.90 GiB 4000.65 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : b9639e06:b48b15f4:8403c056:ea9bdcd3

Internal Bitmap : 8 sectors from superblock

    Update Time : Mon Oct 26 12:00:12 2015

Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 579e59a9 - correct
Events : 27019

Layout : near=2
Chunk Size : 512K

Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

Signature · 1 line
Adam Goryachev Website Managers www.websitemanagers.com.au
#1 lore raw

Dear Adam

Quoted 16 lines · Adam Goryachev

1) fail two drives from the array:
mdadm --manage /dev/md0 --fail /dev/sdb1
mdadm --manage /dev/md0 --remove /dev/sdb1
mdadm --manage /dev/md0 --fail /dev/sdd1
mdadm --manage /dev/md0 --remove /dev/sdd1
mdadm --misc --zero-superblock /dev/sdb1
mdadm --misc --zero-superblock /dev/sdd1

It seems that RAID10 device number is in order:
sdb1 device0
sdc1 device1
sdd1 device2
sde1 device3
Therefore, I can fail device 0 (sdb1) and device 2 (sdd1) without losing any data.

Quoted 7 lines

2) create a 3 disk RAID5 with one disk missing.
mdadm --create /dev/md1 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdd1 missing

3) Then copy all of the existing data across
unmount partitions, stop LVM/etc
dd bs=16M if=/dev/md0 of=/dev/md1

I dont know why this would be necessary.

Quoted 10 lines

4) Finally, stop the md0, and add the two devices to the new raid5, and then grow the array to use the space on the 4th drive.
mdadm --manage --stop /dev/md0
mdadm --misc --zero-superblock /dev/sdc1
mdadm --manage /dev/md1 --add /dev/sdc1
mdadm --misc --zero-superblock /dev/sde1
mdadm --manage /dev/md1 --add /dev/sde1
mdadm --grow /dev/md1 --raid-devices=4

Quoted 4 lines

5) Add the space to my LVM
pvresize /dev/md1

6) Start up LVM, mount LV's, etc

I tried your method on a local system, and it seemed to worked.

The reason why I said that dd would'nt be necessary, as it is raid syncing would work. You just need to be careful that you should not start doing remove/add and other changing aspects of raid till the time syncing is going on. Finally when you grow your raid in RAID5 also syncing would happen.

Regards
Anugraha Sinha

#2 lore raw
In reply to Adam Goryachev

Good morning Adam,

Quoted 8 lines · Adam Goryachev

Hi all,

I'm trying to convert a 4 disk RAID10 to a RAID5. Currently I have:
cat /proc/mdstat
Personalities : [raid10]
md0 : active raid10 sdd1[2] sdc1[1] sdb1[0] sde1[3]

      7813772288 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
      bitmap: 0/59 pages [0KB], 65536KB chunk

[trim /]

Your plan looks reasonable, if a bit long. I've done similar before. Usually I add the new array to the LVM volume group of the original, then pvmove the contents to the new physical volume. Let's me stay online through the entire operation.

Anyways, it seems you are missing an opportunity to convert in place, though. Consider setting up four small loop devices and test conversions from r10,n2 -n4 => r0 -n3 => r4 -n4 => r5 -n4 . I suspect you can get to your new raid5 without moving a bunch of data around.

Phil

#3 lore raw
Quoted 18 lines · Phil Turmel

Good morning Adam,

Quoted 8 lines · Adam Goryachev

Hi all,

I'm trying to convert a 4 disk RAID10 to a RAID5. Currently I have:
cat /proc/mdstat
Personalities : [raid10]
md0 : active raid10 sdd1[2] sdc1[1] sdb1[0] sde1[3]

       7813772288 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
       bitmap: 0/59 pages [0KB], 65536KB chunk

[trim /]

Your plan looks reasonable, if a bit long. I've done similar before. Usually I add the new array to the LVM volume group of the original, then pvmove the contents to the new physical volume. Let's me stay online through the entire operation.

Quoted 4 lines

Anyways, it seems you are missing an opportunity to convert in place, though. Consider setting up four small loop devices and test conversions from r10,n2 -n4 => r0 -n3 => r4 -n4 => r5 -n4 . I suspect you can get to your new raid5 without moving a bunch of data around.

This sounds interesting, I'm just messing around with a VM at the moment, but I'm not sure I understand your shorthand?

Currently, I have raid10 with 4 devices
I can convert to raid0 with 2 devices
mdadm --grow --level=0 --raid-devices=2 /dev/md0
md0 : active raid0 vde1[2] vdc1[0]

      2093056 blocks super 1.2 512k chunks
      bitmap: 1/1 pages [4KB], 65536KB chunk

So then I tried to add the other two drives back:
mdadm --manage /dev/md0 --add /dev/vdd1
mdadm: add new device failed for /dev/vdd1 as 3: Invalid argument
That failed, raid0 doesn't allow spares, because there is nothing for them to "do".

So I tried jumping to raid5:
mdadm --grow --level=5 --raid-devices=4 /dev/md0 /dev/vdd1 /dev/vdf1
mdadm: Must give -a/--add for devices to add: /dev/vdd1
OK, I was just guessing...
mdadm --grow --level=5 --raid-devices=4 /dev/md0 -a /dev/vdd1 -a /dev/vdf1
mdadm: level of /dev/md0 changed to raid5
mdadm: added /dev/vdd1
mdadm: added /dev/vdf1
mdadm: Need to backup 3072K of critical section..
mdadm: Cannot set device shape for /dev/md0
Bitmap must be removed before shape can be changed
unfreeze

Oh, alrighty... but wait, what did it actually do? cat /proc/mdstat
Personalities : [raid10] [raid0] [raid6] [raid5] [raid4] md0 : active raid5 vdf1[4] vdd1[3](S) vde1[2] vdc1[0]

      2093056 blocks super 1.2 level 5, 512k chunk, algorithm 5 [3/3] [UUU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

Interesting, it has changed to a three disk RAID5 without removing the bitmap, only adding the 4th disk seems to need that.

mdadm --grow --bitmap=none /dev/md0

root@testraid:~# cat /proc/mdstat
Personalities : [raid10] [raid0] [raid6] [raid5] [raid4]
md0 : active raid5 vdf1[4] vdd1[3](S) vde1[2] vdc1[0]
      2093056 blocks super 1.2 level 5, 512k chunk, algorithm 5 [3/3] [UUU]

unused devices: <none>

So, still 3 disk raid5 with one spare, but seems to be insync, so either it was really quick (possible since they are small drives) or it didn't need to do a sync??

mdadm --grow --level=5 --raid-devices=4 /dev/md0
mdadm: Need to backup 3072K of critical section..

cat /proc/mdstat
Personalities : [raid10] [raid0] [raid6] [raid5] [raid4] md0 : active raid5 vdf1[4] vdd1[3] vde1[2] vdc1[0]

      2093056 blocks super 1.2 level 5, 512k chunk, algorithm 5 [4/4] 

[UUUU]
resync=DELAYED

unused devices: <none>

OK, so now how to make it resync?

Here I'm stuck...
I've tried:
mdadm --misc /dev/md0 --action=check
mdadm --misc /dev/md0 --action=repair

Nothing seems to be happening.

BTW, I had the array mounted during my testing, as ideally that is what I will do with the live machine. Worst case scenario (on the live machine) I can afford to lose all the data, as it is only an extra backup of the other backup machine, but it would mean a few TB's of data across a slow WAN....

Any suggestions on getting this to progress? Did I do something wrong?

Thanks for the suggestion, it certainly looks promising so far.

PS test machine is running:
mdadm - v3.3.2 - 21st August 2014
Linux testraid 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux

Relevant sections from dmesg:

[  582.456118] md/raid10:md0: Disk failure on vdd1, disabling device.
[  582.456119] md/raid10:md0: Operation continuing on 3 devices.
[  582.459135] RAID10 conf printout:
[  582.459137]  --- wd:3 rd:4
[  582.459138]  disk 0, wo:0, o:1, dev:vdc1
[  582.459140]  disk 1, wo:1, o:0, dev:vdd1
[  582.459141]  disk 2, wo:0, o:1, dev:vde1
[  582.459142]  disk 3, wo:0, o:1, dev:vdf1
[  582.459144] RAID10 conf printout:
[  582.459145]  --- wd:3 rd:4
[  582.459146]  disk 0, wo:0, o:1, dev:vdc1
[  582.459147]  disk 2, wo:0, o:1, dev:vde1
[  582.459148]  disk 3, wo:0, o:1, dev:vdf1
[  582.459174] md: unbind<vdd1>
[  582.459178] md: export_rdev(vdd1)
[  582.460851] md/raid10:md0: Disk failure on vdf1, disabling device.
[  582.460852] md/raid10:md0: Operation continuing on 2 devices.
[  582.463685] RAID10 conf printout:
[  582.463687]  --- wd:2 rd:4
[  582.463689]  disk 0, wo:0, o:1, dev:vdc1
[  582.463700]  disk 2, wo:0, o:1, dev:vde1
[  582.463702]  disk 3, wo:1, o:0, dev:vdf1
[  582.463703] RAID10 conf printout:
[  582.463704]  --- wd:2 rd:4
[  582.463705]  disk 0, wo:0, o:1, dev:vdc1
[  582.463706]  disk 2, wo:0, o:1, dev:vde1
[  582.463731] md: unbind<vdf1>
[  582.463734] md: export_rdev(vdf1)
[  661.433432] md: raid0 personality registered for level 0
[  661.433775] md0: bitmaps are not supported for raid0
[ 1999.023095] md0: personality does not support diskops!
[ 1999.123016] md0: personality does not support diskops!
[ 1999.275905] md0: personality does not support diskops!
[ 1999.312123] md0: personality does not support diskops!
[ 2164.827683] async_tx: api initialized (async)
[ 2164.828731] xor: automatically using best checksumming function: generic_sse
[ 2164.848017]    generic_sse:  3623.000 MB/sec
[ 2164.848019] xor: using function: generic_sse (3623.000 MB/sec)
[ 2164.916020] raid6: int64x1   1743 MB/s
[ 2164.984015] raid6: int64x2   2248 MB/s
[ 2165.052028] raid6: int64x4   2537 MB/s
[ 2165.120021] raid6: int64x8   2143 MB/s
[ 2165.188010] raid6: sse2x1    6819 MB/s
[ 2165.256011] raid6: sse2x2    8521 MB/s
[ 2165.324014] raid6: sse2x4    9659 MB/s
[ 2165.324015] raid6: using algorithm sse2x4 (9659 MB/s)
[ 2165.328986] md: raid6 personality registered for level 6
[ 2165.328991] md: raid5 personality registered for level 5
[ 2165.328993] md: raid4 personality registered for level 4
[ 2165.329357] md/raid:md0: device vde1 operational as raid disk 1
[ 2165.329361] md/raid:md0: device vdc1 operational as raid disk 0
[ 2165.330125] md/raid:md0: allocated 3228kB
[ 2165.331574] md/raid:md0: raid level 5 active with 2 out of 3 devices, algorithm 5
[ 2165.332646] RAID conf printout:
[ 2165.332648]  --- level:5 rd:3 wd:2
[ 2165.332651]  disk 0, o:1, dev:vdc1
[ 2165.332653]  disk 1, o:1, dev:vde1
[ 2165.355069] md: bind<vdd1>
[ 2165.364753] md: bind<vdf1>
[ 2165.389506] md/raid0:md0: bad disk number -1 - aborting!
[ 2165.389849] md: md0: raid0 would not accept array
[ 2165.390051] RAID conf printout:
[ 2165.390053]  --- level:5 rd:3 wd:2
[ 2165.390054]  disk 0, o:1, dev:vdc1
[ 2165.390055]  disk 1, o:1, dev:vde1
[ 2165.390056]  disk 2, o:1, dev:vdf1
[ 2165.390061] RAID conf printout:
[ 2165.390062]  --- level:5 rd:3 wd:2
[ 2165.390063]  disk 0, o:1, dev:vdc1
[ 2165.390064]  disk 1, o:1, dev:vde1
[ 2165.390065]  disk 2, o:1, dev:vdf1
[ 2165.390960] md: recovery of RAID array md0
[ 2165.390962] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 2165.390963] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[ 2165.390973] md: using 128k window, over a total of 1046528k.
[ 2176.293108] md: md0: recovery done.
[ 2176.306088] RAID conf printout:
[ 2176.306091]  --- level:5 rd:3 wd:3
[ 2176.306092]  disk 0, o:1, dev:vdc1
[ 2176.306094]  disk 1, o:1, dev:vde1
[ 2176.306095]  disk 2, o:1, dev:vdf1
[ 2214.230914] RAID conf printout:
[ 2214.230920]  --- level:5 rd:4 wd:4
[ 2214.230923]  disk 0, o:1, dev:vdc1
[ 2214.230926]  disk 1, o:1, dev:vde1
[ 2214.230929]  disk 2, o:1, dev:vdf1
[ 2214.230931]  disk 3, o:1, dev:vdd1
[ 2214.235161] md: reshape of RAID array md0
[ 2214.235164] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 2214.235167] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
[ 2214.235186] md: using 128k window, over a total of 1046528k.

Regards,
Adam

Signature · 1 line
Adam Goryachev Website Managers www.websitemanagers.com.au
#4 lore raw

Dear Adam,

Quoted 41 lines · Adam Goryachev

mdadm --grow --bitmap=none /dev/md0

root@testraid:~# cat /proc/mdstat
Personalities : [raid10] [raid0] [raid6] [raid5] [raid4]
md0 : active raid5 vdf1[4] vdd1[3](S) vde1[2] vdc1[0]
       2093056 blocks super 1.2 level 5, 512k chunk, algorithm 5 [3/3]
[UUU]

unused devices: <none>

So, still 3 disk raid5 with one spare, but seems to be insync, so either it was really quick (possible since they are small drives) or it didn't need to do a sync??

mdadm --grow --level=5 --raid-devices=4 /dev/md0
mdadm: Need to backup 3072K of critical section..

cat /proc/mdstat
Personalities : [raid10] [raid0] [raid6] [raid5] [raid4] md0 : active raid5 vdf1[4] vdd1[3] vde1[2] vdc1[0]

       2093056 blocks super 1.2 level 5, 512k chunk, algorithm 5 [4/4]

[UUUU]
resync=DELAYED

unused devices: <none>

OK, so now how to make it resync?

Here I'm stuck...
I've tried:
mdadm --misc /dev/md0 --action=check
mdadm --misc /dev/md0 --action=repair

Nothing seems to be happening.

BTW, I had the array mounted during my testing, as ideally that is what I will do with the live machine. Worst case scenario (on the live machine) I can afford to lose all the data, as it is only an extra backup of the other backup machine, but it would mean a few TB's of data across a slow WAN....

Quoted 3 lines

Any suggestions on getting this to progress? Did I do something wrong?

Thanks for the suggestion, it certainly looks promising so far.

Why dont you stop your array once and do something like this?
mdadm --stop /dev/md0
mdadm --assemble /dev/md0 --run --force --update=resync /dev/vdf1 /dev/vdd1 /dev/vde1 vdc1

This will restart your array with the required raid-level and also start the resyncing process.

Regards
Anugraha

#5 lore raw
Quoted 46 lines · Anugraha Sinha

Dear Adam,

Quoted 43 lines · Adam Goryachev

mdadm --grow --bitmap=none /dev/md0

root@testraid:~# cat /proc/mdstat
Personalities : [raid10] [raid0] [raid6] [raid5] [raid4]
md0 : active raid5 vdf1[4] vdd1[3](S) vde1[2] vdc1[0]
       2093056 blocks super 1.2 level 5, 512k chunk, algorithm 5 [3/3]
[UUU]

unused devices: <none>

So, still 3 disk raid5 with one spare, but seems to be insync, so either it was really quick (possible since they are small drives) or it didn't need to do a sync??

mdadm --grow --level=5 --raid-devices=4 /dev/md0
mdadm: Need to backup 3072K of critical section..

cat /proc/mdstat
Personalities : [raid10] [raid0] [raid6] [raid5] [raid4] md0 : active raid5 vdf1[4] vdd1[3] vde1[2] vdc1[0]

       2093056 blocks super 1.2 level 5, 512k chunk, algorithm 5 [4/4]

[UUUU]
resync=DELAYED

unused devices: <none>

OK, so now how to make it resync?

Here I'm stuck...
I've tried:
mdadm --misc /dev/md0 --action=check
mdadm --misc /dev/md0 --action=repair

Nothing seems to be happening.

BTW, I had the array mounted during my testing, as ideally that is what I will do with the live machine. Worst case scenario (on the live machine) I can afford to lose all the data, as it is only an extra backup of the other backup machine, but it would mean a few TB's of data across a slow WAN....

Any suggestions on getting this to progress? Did I do something wrong?

Quoted 7 lines
Quoted 1 line

Thanks for the suggestion, it certainly looks promising so far.

Why dont you stop your array once and do something like this?
mdadm --stop /dev/md0
mdadm --assemble /dev/md0 --run --force --update=resync /dev/vdf1 /dev/vdd1 /dev/vde1 vdc1

This will restart your array with the required raid-level and also start the resyncing process.

I got:
mdadm: Failed to restore critical section for reshape, sorry. Possibly you needed to specify the --backup-file

Personalities : [raid10] [raid0] [raid6] [raid5] [raid4] md0 : inactive vdd1[3](S) vdf1[4](S) vde1[2](S) vdc1[0](S)

      4186112 blocks super 1.2

unused devices: <none>

Related dmesg output:

[27217.316713] md: md0 stopped.
[27217.316727] md: unbind<vdc1>
[27217.316732] md: export_rdev(vdc1)
[27217.316769] md: unbind<vdf1>
[27217.316772] md: export_rdev(vdf1)
[27217.316789] md: unbind<vdd1>
[27217.316791] md: export_rdev(vdd1)
[27217.316806] md: unbind<vde1>
[27217.316809] md: export_rdev(vde1)
[27248.819955] md: md0 stopped.
[27248.855348] md: bind<vdc1>
[27248.868655] md: bind<vde1>
[27248.872681] md: bind<vdf1>
[27248.876477] md: bind<vdd1>

Any further suggestions?

I have no problem with doing the whole process offline, but I assume it will take a lot longer on the live machine (4TB drives), and didn't want to leave it unmounted for a long time.

Regards,
Adam

Signature · 1 line
Adam Goryachev Website Managers www.websitemanagers.com.au
#6 lore raw
Quoted 1 line · Adam Goryachev

On 27/10/15 17:19, Anugraha Sinha wrote:

\

Quoted 2 lines
Quoted 2 lines

This will restart your array with the required raid-level and also start the resyncing process.

You shouldn't have needed to stop it.

Quoted 3 lines

I got:
mdadm: Failed to restore critical section for reshape, sorry. Possibly you needed to specify the --backup-file

Probably just need "mdadm /dev/md0 --grow --continue --backup-file=...."

where .... is a file location outside the array. You may also need --invalid-backup

Phil

#7 lore raw
Quoted 12 lines · Phil Turmel
Quoted 1 line · Adam Goryachev

On 27/10/15 17:19, Anugraha Sinha wrote:

\

Quoted 2 lines
Quoted 2 lines

This will restart your array with the required raid-level and also start the resyncing process.

You shouldn't have needed to stop it.

Quoted 3 lines

I got:
mdadm: Failed to restore critical section for reshape, sorry. Possibly you needed to specify the --backup-file

Probably just need "mdadm /dev/md0 --grow --continue --backup-file=...."

Quoted 8 lines

where .... is a file location outside the array. You may also need --invalid-backup

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Thank you, that did it (if I don't do the stop):
mdadm --grow --continue --backup-file=/tmp/nofile /dev/md0
mdadm: Need to backup 3072K of critical section..
mdadm: Reshape is delayed, but cannot wait carefully with this kernel.
You might experience problems until other reshapes complete.

Just in case someone else is looking later, this is how to convert from 4 disk RAID10 to 4 disk RAID5.

Starting again from scratch, I followed this process:
mdadm --create --level=10 --raid-devices=4 /dev/md0 /dev/vd[cdef]1 mdadm --grow --bitmap=internal /dev/md0

#This gets me to my current live config....

# Remove the bitmap since it causes problems later mdadm --grow --bitmap=none /dev/md0

# Grow to RAID0 which basically just drops two devices, it finishes immediately. mdadm --grow --level=0 --raid-devices=2 /dev/md0

# Grow to RAID5, only add one device
mdadm --grow --level=5 --raid-devices=3 /dev/md0 -a /dev/vdd1

# A resync is completed, looks good/normal

# Grow to add the 4th device
mdadm --grow --raid-devices=4 /dev/md0 -a /dev/vdf1

# Problems, stuck with resync=DELAYED, we were not asked to supply a backup file for this... # Just ask mdadm to continue, and now supply a backup file
mdadm --grow --continue --backup-file=/tmp/nofile /dev/md0

# All good. Add the bitmap again
mdadm --grow /dev/md0 --bitmap=internal

It would seem to me that this is generally a relatively "dangerous" exercise, you are very vulnerable to failed disks due to using RAID0, and then migrating to RAID5, any read/write failure during that process could cause significant data loss. Finally, the grow to add the 4th disk should be fine, unless you have a problem with the same block on two devices.

Any final comments/suggestions before I start the process "for real"?

Thanks,
Adam

Signature · 1 line
Adam Goryachev Website Managers www.websitemanagers.com.au
#8 lore raw
Quoted 2 lines · Adam Goryachev

Any final comments/suggestions before I start the process "for real"?

Do a 'check' scrub before the raid10 to raid0 conversion.

Great report!

Phil