Raid 5: all devices marked spare, cannot assemble
Hi folks,
I have a rather curious issue with one of our storage machines. The machine has 36x 4TB disks (SuperMicro 847 chassis) which are divided over 4 dual SAS-HBAs and the on-board SAS. These disks are in RAID5 configurations, 6 raids of 6 disks each. Recently the machine ran out of memory (it has 32GB, and no swapspace as it boots from SATA-DOM) and the last entries in the syslog are from the OOM-killer. The machine is running Ubuntu 14.04.02 LTS, mdadm 3.2.5-5ubuntu4.1.
After doing a hard reset, the machine booted fine but one of the raids needed to resync. Worse, another of the raid5s will not assemble at all. All the drives are marked SPARE. Relevant output from /proc/mdstat (one working and the broken array):
md14 : active raid5 sdc1[2] sdag1[6] sde1[4] sdi1[3] sdz1[0] sdu1[1]
19534425600 blocks super 1.2 level 5, 512k chunk, algorithm 2
[6/6] [UUUUUU]
md15 : inactive sdd1[6](S) sdad1[0](S) sdy1[3](S) sdv1[4](S) sdm1[2](S) sdq1[1](S)
23441313792 blocks super 1.2
Using 'mdadm --examine' on each of the drives from the broken md15, I get:
sdd1: Spare, Events: 0
sdad1: Active device 0, Events 194
sdy1: Active device 3, Events 194
sdv1: Active device 4, Events 194
sdm1: Active device 2, Events 194
sdq1: Active device 1, Events 194
This numbering corresponds to how the raid5 was created when I installed the machine:
mdadm --create /dev/md15 -l 5 -n 6 /dev/sdad1 /dev/sdq1 /dev/sdm1 /dev/sdy1 /dev/sdv1 /dev/sdd1
Possible clues from /var/log/syslog:
md/raid:md13: not clean -- starting background reconstruction (at 14 seconds uptime).
md15 isn't even mentioned in the boot-time syslog, only once I manually try to assemble it did I get these errors:
md: kicking non-fresh sdd1 from array!
md: unbind<sdd1>
md: export_rdev(sdd1)
md/raid:md15: not clean -- starting background reconstruction md/raid:md15: device sdy1 operational as raid disk 3 md/raid:md15: device sdv1 operational as raid disk 4 md/raid:md15: device sdad1 operational as raid disk 0 md/raid:md15: device sdq1 operational as raid disk 1 md/raid:md15: device sdm1 operational as raid disk 2 md/raid:md15: allocated 0kB
md/raid:md15: cannot start dirty degraded array. RAID conf printout:
--- level:5 rd:6 wd:5
disk 0, o:1, dev:sdad1
disk 1, o:1, dev:sdq1
disk 2, o:1, dev:sdm1
disk 3, o:1, dev:sdy1
disk 4, o:1, dev:sdv1
md/raid:md15: failed to run raid set.
md: pers->run() failed ...
So the questions I'd like to pose are:
* Why does this raid5 not assemble? Only one drive (sdd) seems to be missing (marked spare), although I see no real issues with it and can read from it fine. There should still be enough drives to start the array.
# mdadm --assemble /dev/md15 --run
Returns without any error message, but leaves /proc/mdstat unchanged.
* How can the data be recovered, and the machine brought into production again
And of course
* What went wrong, and how can we guard against this?
Any insights and help are much appreciated.
Regards, Paul Boven.
Signature · 4 lines
Paul Boven <[email protected]> +31 (0)521-596547 Unix/Linux/Networking specialist Joint Institute for VLBI in Europe - www.jive.nl VLBI - It's a fringe science
Good morning Paul,
Quoted 9 lines · Paul Boven
Hi folks,
I have a rather curious issue with one of our storage machines. The machine has 36x 4TB disks (SuperMicro 847 chassis) which are divided over 4 dual SAS-HBAs and the on-board SAS. These disks are in RAID5 configurations, 6 raids of 6 disks each. Recently the machine ran out of memory (it has 32GB, and no swapspace as it boots from SATA-DOM) and the last entries in the syslog are from the OOM-killer. The machine is running Ubuntu 14.04.02 LTS, mdadm 3.2.5-5ubuntu4.1.
{BTW, I think raid5 is *insane* for this size array.}
Quoted 5 lines
After doing a hard reset, the machine booted fine but one of the raids needed to resync. Worse, another of the raid5s will not assemble at all. All the drives are marked SPARE. Relevant output from
/proc/mdstat(one working and the broken array):
Quoted 7 lines
md14 : active raid5 sdc1[2] sdag1[6] sde1[4] sdi1[3] sdz1[0] sdu1[1]
19534425600 blocks super 1.2 level 5, 512k chunk, algorithm 2[6/6] [UUUUUU]
md15 : inactive sdd1[6](S) sdad1[0](S) sdy1[3](S) sdv1[4](S) sdm1[2](S) sdq1[1](S)
23441313792 blocks super 1.2
Although (S) implies spare, that's only true if the array is active. md15 is assembled but not assembled.
Quoted 8 lines
Using 'mdadm --examine' on each of the drives from the broken md15, I get:
sdd1: Spare, Events: 0
sdad1: Active device 0, Events 194
sdy1: Active device 3, Events 194
sdv1: Active device 4, Events 194
sdm1: Active device 2, Events 194
sdq1: Active device 1, Events 194
Please don't trim the reports. This implies that your array simply didn't --run as it is unexpected degraded.
[trim /]
Quoted 11 lines
md: kicking non-fresh sdd1 from array!
md: unbind<sdd1>
md:export_rdev(sdd1)
md/raid:md15: not clean -- starting background reconstruction md/raid:md15: device sdy1 operational as raid disk 3 md/raid:md15: device sdv1 operational as raid disk 4 md/raid:md15: device sdad1 operational as raid disk 0 md/raid:md15: device sdq1 operational as raid disk 1 md/raid:md15: device sdm1 operational as raid disk 2 md/raid:md15: allocated 0kB
md/raid:md15: cannot start dirty degraded array.
Exactly.
Quoted 5 lines
* Why does this raid5 not assemble? Only one drive (sdd) seems to be missing (marked spare), although I see no real issues with it and can read from it fine. There should still be enough drives to start the array.
# mdadm --assemble /dev/md15 --run
Wrong syntax. It's already assembled. Just try "mdadm --run /dev/md15"
Quoted 2 lines
* How can the data be recovered, and the machine brought into production again
If the simple --run doesn't work, stop the array and force assemble the good drives:
mdadm --stop /dev/md15
mdadm --assemble --force --verbose /dev/md15 /dev/sd{ad,q,m,y,v}1
If that doesn't work, show the complete output of the --assemble.
Quoted 1 line
* What went wrong, and how can we guard against this?
The crash prevented mdadm from writing the current state of the array to the individual drives' metadata. You didn't provide complete --examine output, so I'm speculating, but the drives must disagree on the last known state of the array ==> "dirty". See "start_dirty_degraded" in man md(4), and the --run and --no-degraded options to assemble.
In other words, unclean shutdowns should have manual intervention, unless the array in question contains the root filesystem, in which case the risky "start_dirty_degraded" may be appropriate. In that case, you probably would want your initramfs to have a special mdadm.conf, deferring assembly of bulk arrays to normal userspace.
Phil
Hi Phil,
Good morning and thanks for your quick reply.
Quoted 9 lines · Phil Turmel
Quoted 7 lines
I have a rather curious issue with one of our storage machines. The machine has 36x 4TB disks (SuperMicro 847 chassis) which are divided over 4 dual SAS-HBAs and the on-board SAS. These disks are in RAID5 configurations, 6 raids of 6 disks each. Recently the machine ran out of memory (it has 32GB, and no swapspace as it boots from SATA-DOM) and the last entries in the syslog are from the OOM-killer. The machine is running Ubuntu 14.04.02 LTS, mdadm 3.2.5-5ubuntu4.1.
{BTW, I think raid5 is *insane* for this size array.}
It's 6 raid5s, not a single big one. This is only a temporary holding space for data to be processed. In its original incarnation the machine had 36 distinct file-systems that we would read from in a software stripe, just to get enough IO performance. So this is a trade-off between IO-speed and lost capacity versus convenience in case a drive inevitably fails.
I guess you would recommend raid6? I would have liked a global hot spare, maybe 7 arrays of 5 disks, but then we lose 8 disks in total instead of the current 6.
Quoted 1 line
Wrong syntax. It's already assembled. Just try "mdadm --run
/dev/md15"
Trying to 'run' md15 gives me the same errors as before:
md/raid:md15: not clean -- starting background reconstruction md/raid:md15: device sdad1 operational as raid disk 0 md/raid:md15: device sdy1 operational as raid disk 3 md/raid:md15: device sdv1 operational as raid disk 4 md/raid:md15: device sdm1 operational as raid disk 2 md/raid:md15: device sdq1 operational as raid disk 1 md/raid:md15: allocated 0kB
md/raid:md15: cannot start dirty degraded array. RAID conf printout:
--- level:5 rd:6 wd:5
disk 0, o:1, dev:sdad1
disk 1, o:1, dev:sdq1
disk 2, o:1, dev:sdm1
disk 3, o:1, dev:sdy1
disk 4, o:1, dev:sdv1
md/raid:md15: failed to run raid set.
md: pers->run() failed ...
Quoted 5 lines
If the simple --run doesn't work, stop the array and force assemble the good drives:
mdadm --stop
/dev/md15
mdadm --assemble --force --verbose/dev/md15/dev/sd{ad,q,m,y,v}1
That worked!
mdadm: looking for devices for /dev/md15
mdadm: /dev/sdad1 is identified as a member of /dev/md15, slot 0.
mdadm: /dev/sdq1 is identified as a member of /dev/md15, slot 1.
mdadm: /dev/sdm1 is identified as a member of /dev/md15, slot 2.
mdadm: /dev/sdy1 is identified as a member of /dev/md15, slot 3.
mdadm: /dev/sdv1 is identified as a member of /dev/md15, slot 4.
mdadm: Marking array /dev/md15 as 'clean'
mdadm: added /dev/sdq1 to /dev/md15 as 1
mdadm: added /dev/sdm1 to /dev/md15 as 2
mdadm: added /dev/sdy1 to /dev/md15 as 3
mdadm: added /dev/sdv1 to /dev/md15 as 4
mdadm: no uptodate device for slot 5 of /dev/md15
mdadm: added /dev/sdad1 to /dev/md15 as 0
mdadm: /dev/md15 has been started with 5 drives (out of 6).
I've checked that the filesystem is in good shape, and added /dev/sdd1 back in, the array is now resyncing. 680 minutes to go, but there's a few tricks I can do to speed that up a bit.
Quoted 5 lines
In other words, unclean shutdowns should have manual intervention, unless the array in question contains the root filesystem, in which case the risky "
start_dirty_degraded" may be appropriate. In that case, you probably would want your initramfs to have a special mdadm.conf, deferring assembly of bulk arrays to normal userspace.
I'm perfectly happy with doing the recovery in userspace, these drives are not critical for booting. Except that Ubuntu, Plymouth and a few other things conspire against booting a machine with any disk problems, but that's a different rant for a different place.
Thank you very much for your very helpful reply, things look a lot better now.
Regards, Paul Boven.
Signature · 4 lines
Paul Boven <[email protected]> +31 (0)521-596547 Unix/Linux/Networking specialist Joint Institute for VLBI in Europe - www.jive.nl VLBI - It's a fringe science
Bad block management in raid1
In reply to Phil TurmelHi There,
Can you conform the below scenario in which blocks are consider to be
a "bad" block.
1. A read error on a degraded array ( a state of raid when
array experiences the failure of one or more disks)for which the data cannot be found from other legs is a "bad" block and gets recorded.
2. When recovering, from source to target leg, for any reason
if unable to read from source, the target leg's block gets recorded as “bad” (thought the target block is writable and can be used in future).
3. Write to a block fails (Though it leads to degraded mode).
Are they all implemented and is there any other scenario?
When exactly the raid1 decides to make the device "Faulty"? Does that
depends on the number of bad blocks in the list ie: 512?
What is the size in the metadata for storing the bad block info.
Thanks,
Ankur
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html