4-disk RAID6 (non-standard layout) normalise hung, now all disks spare
I started with a 4x4TB disk RAID5 array and, over a few years changed all the drives to 8TB (WD Red - I hadn't seen the warnings before now, but it looks like these ones are OK). I then successfully migrated it to RAID6, but it then had a non-standard layout, so I ran:
sudo mdadm --grow /dev/md0 --raid-devices=4
--backup-file=/root/raid5backup --layout=normalize
After a few days it reached 99% complete, but then the "hours remaining" counter started counting up. After a few days I had to power the system down before I could get a backup of the non-critical data (Couldn't get hold of enough storage quickly enough, but it wouldn't be catastrophic to lose it), and now the four drives are in standby, with the array thinking it is RAID0. Running:
sudo mdadm --assemble /dev/md0 /dev/sd[bcde]
responds with:
mdadm: /dev/md0 assembled from 4 drives - not enough to start the
array while not clean - consider --force.
It appears to be similar to https://marc.info/?t=155492912100004&r=1&w=2,
but before trying --force I was considering using overlay files as I'm not sure of the risk of damage. The set-up process that is documented in the " Recovering a damaged RAID" Wiki article is excellent, however the latter part of the process isn't clear to me. If successful, are the overlay files written to the disk like a virtual machine snapshot, or is the process stopped, the overlays removed and the process repeated, knowing that it now has a low risk of damage?
System details follow. Thanks for any help.
============================================================================
=====
user@host:~$ uname -a Linux conan 5.4.0-74-generic #83-Ubuntu SMP Sat May 8 02:35:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
============================================================================
=====
user@host:~$ mdadm --version mdadm - v4.1 - 2018-10-01
============================================================================
=====
user@host:~$ sudo smartctl -H -i -l scterc /dev/sda smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-74-generic] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs Device Model: Samsung SSD 860 EVO M.2 250GB Serial Number: S413NX0K707647T LU WWN Device Id: 5 002538 e40528ae8 Firmware Version: RVT21B6Q User Capacity: 250,059,350,016 bytes [250 GB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Form Factor: M.2 Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5 SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Jun 20 10:44:10 2021 AEST SMART support is: Available - device has SMART capability. SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: Disabled
Write: Disabled
============================================================================
=====
user@host:~$ sudo smartctl -H -i -l scterc /dev/sdb smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-74-generic] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red Device Model: WDC WD80EFAX-68KNBN0 Serial Number: VGJM3NXK LU WWN Device Id: 5 000cca 0bee4dfda Firmware Version: 81.00A81 User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Jun 20 10:44:10 2021 AEST SMART support is: Available - device has SMART capability. SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)============================================================================
=====
user@host:~$ sudo smartctl -H -i -l scterc /dev/sdc smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-74-generic] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD80EFBX-68AZZN0 Serial Number: VRG5YT4K LU WWN Device Id: 5 000cca 0c2c2b5a4 Firmware Version: 85.00A85 User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Jun 20 10:44:11 2021 AEST SMART support is: Available - device has SMART capability. SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)============================================================================
=====
user@host:~$ sudo smartctl -H -i -l scterc /dev/sdd smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-74-generic] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red Device Model: WDC WD80EFAX-68KNBN0 Serial Number: VAGV1WLL LU WWN Device Id: 5 000cca 099cbd8be Firmware Version: 81.00A81 User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Jun 20 10:44:12 2021 AEST SMART support is: Available - device has SMART capability. SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)============================================================================
=====
user@host:~$ sudo smartctl -H -i -l scterc /dev/sde smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-74-generic] (local build) Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red Device Model: WDC WD80EFAX-68LHPN0 Serial Number: 7SJ5W2KW LU WWN Device Id: 5 000cca 252deda87 Firmware Version: 83.H0A83 User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sun Jun 20 10:44:12 2021 AEST SMART support is: Available - device has SMART capability. SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)============================================================================
=====
user@host:~$ sudo mdadm --examine /dev/sdb
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0xd
Array UUID : 3eee8746:8a3bf425:afb9b538:daa61b29
Name : Universe:0
Creation Time : Thu Jul 13 01:11:22 2017
Raid Level : raid6
Raid Devices : 4Avail Dev Size : 15627794096 (7451.91 GiB 8001.43 GB)
Array Size : 15627793408 (14903.83 GiB 16002.86 GB)
Used Dev Size : 15627793408 (7451.91 GiB 8001.43 GB)
Data Offset : 259072 sectors
Super Offset : 8 sectors
Unused Space : before=258992 sectors, after=688 sectors
State : active
Device UUID : eee9201e:d9769906:b6dccda1:b1f35abe
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 39006208 (37.20 GiB 39.94 GB)
Delta Devices : -1 (5->4)
New Layout : left-symmetric
Update Time : Fri Jun 18 08:56:43 2021
Bad Block Log : 512 entries available at offset 24 sectors - bad blocks
present.
Checksum : de1db60e - correct
Events : 184251
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing)
============================================================================
=====
user@host:~$ sudo mdadm --examine /dev/sdc
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0xd
Array UUID : 3eee8746:8a3bf425:afb9b538:daa61b29
Name : Universe:0
Creation Time : Thu Jul 13 01:11:22 2017
Raid Level : raid6
Raid Devices : 4Avail Dev Size : 15627794096 (7451.91 GiB 8001.43 GB)
Array Size : 15627793408 (14903.83 GiB 16002.86 GB)
Used Dev Size : 15627793408 (7451.91 GiB 8001.43 GB)
Data Offset : 259072 sectors
Super Offset : 8 sectors
Unused Space : before=258992 sectors, after=688 sectors
State : active
Device UUID : 7ed45d83:84db8f79:e3aadf4b:a88212d1
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 39006208 (37.20 GiB 39.94 GB)
Delta Devices : -1 (5->4)
New Layout : left-symmetric
Update Time : Fri Jun 18 08:56:43 2021
Bad Block Log : 512 entries available at offset 24 sectors - bad blocks
present.
Checksum : 731a6e9f - correct
Events : 184251
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing)
============================================================================
=====
user@host:~$ sudo mdadm --examine /dev/sdd
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0xd
Array UUID : 3eee8746:8a3bf425:afb9b538:daa61b29
Name : Universe:0
Creation Time : Thu Jul 13 01:11:22 2017
Raid Level : raid6
Raid Devices : 4Avail Dev Size : 15627794096 (7451.91 GiB 8001.43 GB)
Array Size : 15627793408 (14903.83 GiB 16002.86 GB)
Used Dev Size : 15627793408 (7451.91 GiB 8001.43 GB)
Data Offset : 259072 sectors
Super Offset : 8 sectors
Unused Space : before=258992 sectors, after=688 sectors
State : active
Device UUID : 015b3ea0:9b3a38d2:a860f58a:34c19985
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 39006208 (37.20 GiB 39.94 GB)
Delta Devices : -1 (5->4)
New Layout : left-symmetric
Update Time : Fri Jun 18 08:56:43 2021
Bad Block Log : 512 entries available at offset 24 sectors - bad blocks
present.
Checksum : dc4048b8 - correct
Events : 184251
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing)
============================================================================
=====
user@host:~$ sudo mdadm --examine /dev/sde
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x5
Array UUID : 3eee8746:8a3bf425:afb9b538:daa61b29
Name : Universe:0
Creation Time : Thu Jul 13 01:11:22 2017
Raid Level : raid6
Raid Devices : 4Avail Dev Size : 15627794096 (7451.91 GiB 8001.43 GB)
Array Size : 15627793408 (14903.83 GiB 16002.86 GB)
Used Dev Size : 15627793408 (7451.91 GiB 8001.43 GB)
Data Offset : 259072 sectors
Super Offset : 8 sectors
Unused Space : before=258992 sectors, after=688 sectors
State : active
Device UUID : bf9e316b:5910c7ca:1fd799e3:41a349b3
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 39006208 (37.20 GiB 39.94 GB)
Delta Devices : -1 (5->4)
New Layout : left-symmetric
Update Time : Fri Jun 18 08:56:43 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 2616ba80 - correct
Events : 184251
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing)
============================================================================
=====
user@host:~$ sudo mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Raid Level : raid0
Total Devices : 4
Persistence : Superblock is persistent State : inactive
Working Devices : 4
Delta Devices : -1, (1->0)
New Level : raid6
New Layout : left-symmetric
New Chunksize : 512K
Name : Universe:0
UUID : 3eee8746:8a3bf425:afb9b538:daa61b29
Events : 184251
Number Major Minor RaidDevice
- 8 64 - /dev/sde
- 8 32 - /dev/sdc
- 8 48 - /dev/sdd
- 8 16 - /dev/sdb
============================================================================
=====
user@host:~$ sudo lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 9.1M 1 loop /snap/canonical-livepatch/98 loop1 7:1 0 9.1M 1 loop /snap/canonical-livepatch/99 loop2 7:2 0 99.4M 1 loop /snap/core/11187 loop3 7:3 0 99.2M 1 loop /snap/core/11167 loop4 7:4 0 55.4M 1 loop /snap/core18/2066 loop5 7:5 0 70.4M 1 loop /snap/lxd/19647 loop7 7:7 0 217.5M 1 loop /snap/nextcloud/28088 loop8 7:8 0 67.6M 1 loop /snap/lxd/20326 loop9 7:9 0 217.5M 1 loop /snap/nextcloud/27920 loop10 7:10 0 55.5M 1 loop /snap/core18/2074 sda 8:0 0 232.9G 0 disk +-sda1 8:1 0 512M 0 part /boot/efi L-sda2 8:2 0 232.4G 0 part / sdb 8:16 0 7.3T 0 disk sdc 8:32 0 7.3T 0 disk sdd 8:48 0 7.3T 0 disk sde 8:64 0 7.3T 0 disk
============================================================================
=====
user@host:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4]
[raid10]
md0 : inactive sdc[7](S) sdb[6](S) sde[4](S) sdd[5](S)
31255588192 blocks super 1.2============================================================================
=====
user@host:~$ sudo mdadm --stop /dev/md0 mdadm: stopped /dev/md0
user@host:~$ sudo mdadm --assemble /dev/md0 /dev/sd[bcde] mdadm: /dev/md0 assembled from 4 drives - not enough to start the array while not clean - consider --force.
Good morning Jason,
Good report. Comments inline.
Quoted 6 lines · Jason Flood
I started with a 4x4TB disk RAID5 array and, over a few years changed all the drives to 8TB (WD Red - I hadn't seen the warnings before now, but it looks like these ones are OK). I then successfully migrated it to RAID6, but it then had a non-standard layout, so I ran:
sudo mdadm --grow /dev/md0 --raid-devices=4--backup-file=/root/raid5backup --layout=normalize
Ugh. You don't have to use a backup file unless mdadm tells you too. Now you are stuck with it.
Quoted 10 lines
After a few days it reached 99% complete, but then the "hours remaining" counter started counting up. After a few days I had to power the system down before I could get a backup of the non-critical data (Couldn't get hold of enough storage quickly enough, but it wouldn't be catastrophic to lose it), and now the four drives are in standby, with the array thinking it is RAID0. Running:
sudo mdadm --assemble /dev/md0 /dev/sd[bcde]responds with:
mdadm:/dev/md0assembled from 4 drives - not enough to start the
array while not clean - consider --force.
You have to specify the backup file on assembly if a reshape using one was interrupted.
Quoted 8 lines
It appears to be similar to https://marc.info/?t=155492912100004&r=1&w=2,
but before trying --force I was considering using overlay files as I'm not sure of the risk of damage. The set-up process that is documented in the " Recovering a damaged RAID" Wiki article is excellent, however the latter part of the process isn't clear to me. If successful, are the overlay files written to the disk like a virtual machine snapshot, or is the process stopped, the overlays removed and the process repeated, knowing that it now has a low risk of damage?
Using --force is very low risk on assembly. I would try it (without overlays, and with backup file specified) before you do anything else. Odds of success are high.
Also try the flags to treat the backup file as garbage if its contents don't match what mdadm expects.
Report back here after the above.
Quoted 1 line
System details follow. Thanks for any help.
[details trimmed]
Your report of the details was excellent. Thanks for helping us help you.
Phil
Thanks for that, Phil - I think I'm starting to piece it all together now. I was going from a 4-disk RAID5 to 4-disk RAID6, so from my reading the backup file was recommended. The non-standard layout meant that the array had over 20TB usable, but standardising the layout reduced that to 16TB. In that case the reshape starts at the end so the critical section (and so the backup file) may have been in progress at the 99% complete point when it failed, hence the need to specify the backup file for the assemble command.
I ran "sudo mdadm --assemble --verbose --force /dev/md0 /dev/sd[bcde] --backup-file=/root/raid5backup":
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 3.
mdadm: Marking array /dev/md0 as 'clean'
mdadm: /dev/md0 has an active reshape - checking if critical section needs to be restored
mdadm: No backup metadata on /root/raid5backup
mdadm: added /dev/sdc to /dev/md0 as 1
mdadm: added /dev/sdd to /dev/md0 as 2
mdadm: added /dev/sde to /dev/md0 as 3
mdadm: no uptodate device for slot 4 of /dev/md0
mdadm: added /dev/sdb to /dev/md0 as 0
mdadm: Need to backup 3072K of critical section..
mdadm: /dev/md0 has been started with 4 drives (out of 5).
=============================================================
sudo mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Jul 13 01:11:22 2017
Raid Level : raid6
Array Size : 15627793408 (14903.83 GiB 16002.86 GB)
Used Dev Size : 7813896704 (7451.91 GiB 8001.43 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sat Jun 26 19:40:16 2021
State : clean, reshaping
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric-6
Chunk Size : 512K
Consistency Policy : bitmap
Reshape Status : 99% complete
Delta Devices : -1, (5->4)
New Layout : left-symmetric
Name : Universe:0
UUID : 3eee8746:8a3bf425:afb9b538:daa61b29
Events : 184255
Number Major Minor RaidDevice State
6 8 16 0 active sync /dev/sdb
7 8 32 1 active sync /dev/sdc
5 8 48 2 active sync /dev/sdd
4 8 64 3 active sync /dev/sde
=============================================================
cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid6 sdb[6] sde[4] sdd[5] sdc[7]
15627793408 blocks super 1.2 level 6, 512k chunk, algorithm 18 [4/3] [UUUU]
[===================>.] reshape = 99.7% (7794393600/7813896704) finish=52211434.6min speed=0K/sec
bitmap: 14/30 pages [56KB], 131072KB chunk=============================================================
The drive mounts and the files are all intact, but still sitting on 99% complete with 52 million minutes to finish and counting up. The "No backup metadata" made me suspicious that it is stuck because it can't write to /root/raid5backup (and looking at it now I should have put it somewhere more sensible as I'm using sudo, but I used it in the RAID5 to RAID6 process and it was happy). It does seem to have modified the file, though:
stat raid5backup
File: raid5backup
Size: 3149824 Blocks: 6152 IO Block: 4096 regular file Device: 802h/2050d Inode: 1572897 Links: 1 Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-06-26 19:39:16.739983712 +1000
Modify: 2021-06-26 19:40:16.778498938 +1000
Change: 2021-06-26 19:40:16.778498938 +1000
Birth: -
=============================================================
But I believe those times are from when I first ran the assemble command - it's 20:30 now. I couldn't find a flag to conditionally treat the backup file as garbage - just the --invalid-backup "I know it's garbage" option. Given that the assemble isn't complaining about needing to restore the critical section, is my next step something like:
sudo mdadm --assemble --verbose --force /dev/md0 /dev/sd[bcde] --backup-file=raidbackup --invalid-backup
Thanks again, Phil. I haven't been using Linux seriously for very long, so this has been a steep learning curve for me.
Jason
=======================================================================================================================================
-----Original Message-----
From: Phil Turmel <[email protected]>
Sent: Saturday, 26 June 2021 00:00
To: Jason Flood <[email protected]>; [email protected]
Subject: Re: 4-disk RAID6 (non-standard layout) normalise hung, now all disks spare
Good morning Jason,
Good report. Comments inline.
Quoted 6 lines · Jason Flood
I started with a 4x4TB disk RAID5 array and, over a few years changed all the drives to 8TB (WD Red - I hadn't seen the warnings before now, but it looks like these ones are OK). I then successfully migrated it to RAID6, but it then had a non-standard layout, so I ran:
sudo mdadm --grow /dev/md0 --raid-devices=4--backup-file=/root/raid5backup --layout=normalize
Ugh. You don't have to use a backup file unless mdadm tells you too. Now you are stuck with it.
Quoted 9 lines
After a few days it reached 99% complete, but then the "hours remaining"
counter started counting up. After a few days I had to power the
system down before I could get a backup of the non-critical data
(Couldn't get hold of enough storage quickly enough, but it wouldn't
be catastrophic to lose it), and now the four drives are in standby, with the array thinking it is RAID0. Running:
sudo mdadm --assemble/dev/md0/dev/sd[bcde] responds with:
mdadm:/dev/md0assembled from 4 drives - not enough to start the
array while not clean - consider --force.
You have to specify the backup file on assembly if a reshape using one was interrupted.
Quoted 9 lines
It appears to be similar to
https://marc.info/?t=155492912100004&r=1&w=2,
but before trying --force I was considering using overlay files as I'm not sure of the risk of damage. The set-up process that is documented in the " Recovering a damaged RAID" Wiki article is excellent, however the latter part of the process isn't clear to me. If successful, are the overlay files written to the disk like a virtual machine snapshot, or is the process stopped, the overlays removed and the process repeated, knowing that it now has a low risk of damage?
Using --force is very low risk on assembly. I would try it (without overlays, and with backup file specified) before you do anything else. Odds of success are high.
Also try the flags to treat the backup file as garbage if its contents don't match what mdadm expects.
Report back here after the above.
Quoted 1 line
System details follow. Thanks for any help.
[details trimmed]
Your report of the details was excellent. Thanks for helping us help you.
Phil
Quoted 8 lines · Jason Flood
Reshape Status : 99% complete
Delta Devices : -1, (5->4)
New Layout : left-symmetricName : Universe:0
UUID : 3eee8746:8a3bf425:afb9b538:daa61b29
Events : 184255
Quoted 5 lines
Number Major Minor RaidDevice State
6 8 16 0 active sync/dev/sdb
7 8 32 1 active sync/dev/sdc
5 8 48 2 active sync/dev/sdd
4 8 64 3 active sync/dev/sde
Phil will know much more about this than me, but I did notice that the system thinks there should be FIVE raid drives. Is that an mdadm bug?
That would explain the failure to assemble - it thinks there's a drive missing. And while I don't think we've had data-eating trouble, reshaping a parity raid has caused quite a lot of grief for people over the years ...
However, you're running a recent Ubuntu and mdadm - that should all have been fixed by now.
Cheers,
Wol
Good morning Jason, Wol,
Quoted 9 lines · antlists
Quoted 8 lines · Jason Flood
Reshape Status : 99% complete
Delta Devices : -1, (5->4)
New Layout : left-symmetricName : Universe:0
UUID : 3eee8746:8a3bf425:afb9b538:daa61b29 Events : 184255
Quoted 7 lines
Quoted 5 lines
Number Major Minor RaidDevice State
6 8 16 0 active sync/dev/sdb7 8 32 1 active sync/dev/sdc5 8 48 2 active sync/dev/sdd4 8 64 3 active sync/dev/sdePhil will know much more about this than me, but I did notice that the system thinks there should be FIVE raid drives. Is that an mdadm bug?
Not a bug, but a reshape from a degraded array with a reduction in space.
Quoted 1 line
That would explain the failure to assemble - it thinks there's a drive missing. And while I don't think we've had data-eating trouble, reshaping a parity raid has caused quite a lot of grief for people over the years ...
I've never tried it starting from a degraded array. Might be a corner case bug not yet exposed.
Quoted 1 line
However, you're running a recent Ubuntu and mdadm - that should all have been fixed by now.
Indeed.
Quoted 2 lines
Cheers,
Wol
Quoted 4 lines · Jason Flood
Thanks for that, Phil - I think I'm starting to piece it all together now. I was going from a 4-disk RAID5 to 4-disk RAID6, so from my reading the backup file was recommended. The non-standard layout meant that the array had over 20TB usable, but standardising the layout reduced that to 16TB. In that case the reshape starts at the end so the critical section (and so the backup file) may have been in progress at the 99% complete point when it failed, hence the need to specify the backup file for the assemble command.
I ran "sudo mdadm --assemble --verbose --force
/dev/md0/dev/sd[bcde] --backup-file=/root/raid5backup":
Quoted 16 lines
mdadm: looking for devices for
/dev/md0
mdadm:/dev/sdbis identified as a member of/dev/md0, slot 0.
mdadm:/dev/sdcis identified as a member of/dev/md0, slot 1.
mdadm:/dev/sddis identified as a member of/dev/md0, slot 2.
mdadm:/dev/sdeis identified as a member of/dev/md0, slot 3.
mdadm: Marking array/dev/md0as 'clean'
mdadm:/dev/md0has an active reshape - checking if critical section needs to be restored
mdadm: No backup metadata on/root/raid5backup
mdadm: added/dev/sdcto/dev/md0as 1
mdadm: added/dev/sddto/dev/md0as 2
mdadm: added/dev/sdeto/dev/md0as 3
mdadm: no uptodate device for slot 4 of/dev/md0
mdadm: added/dev/sdbto/dev/md0as 0
mdadm: Need to backup 3072K of critical section..
mdadm:/dev/md0has been started with 4 drives (out of 5).
So force was sufficient to assemble. But you are still stuck at 99%.
Look at the output of ps to see if mdmon is still running (that is the background process that actually reshapes stripe by stripe). If not, look in your logs for clues as to why it died.
If you can't find anything significant, the next step would be to backup the currently functioning array to another system/drive collection and start from scratch. I wouldn't trust anything else with the information available.
Phil
ps. Convention on kernel.org mailing lists is to NOT top-post, and to trim unnecessary context.
Good morning Phil, Wol,
Quoted 7 lines
So force was sufficient to assemble. But you are still stuck at 99%.
Look at the output of ps to see if mdmon is still running (that is the background process that actually reshapes stripe by stripe). If not, look in your logs for clues as to why it died.
If you can't find anything significant, the next step would be to backup the currently functioning array to another system/drive collection and start from scratch. I wouldn't trust anything else with the information available.
Phil
Will do, thanks. I have a few assignments due next weekend so I may not be able to report back for a week.
Quoted 1 line
ps. Convention on kernel.org mailing lists is to NOT top-post, and to trim unnecessary context.
Sorry. First time on a mailing list since well before Outlook was invented!
Thanks again, Phil and Wol.
Good morning Jason,
Quoted 11 lines · Jason Flood
Good morning Phil, Wol,
Quoted 7 lines
So force was sufficient to assemble. But you are still stuck at 99%.
Look at the output of ps to see if mdmon is still running (that is the background process that actually reshapes stripe by stripe). If not, look in your logs for clues as to why it died.
If you can't find anything significant, the next step would be to backup the currently functioning array to another system/drive collection and start from scratch. I wouldn't trust anything else with the information available.
Phil
Will do, thanks. I have a few assignments due next weekend so I may not be able to report back for a week.
No worries.
Quoted 3 lines
Quoted 1 line
ps. Convention on kernel.org mailing lists is to NOT top-post, and to trim unnecessary context.
Sorry. First time on a mailing list since well before Outlook was invented!
No worries. Also note that many mailing lists disagree with this. And with kernel.org's convention to CC: all participants. You almost can't avoid getting flamed one way or the other. (:
Quoted 1 line
Thanks again, Phil and Wol.
You're welcome.
Phil