data recovery on raid5
Having raid fail on friday evening is pretty bad timing - not there is perhaps any good time for such a thing. I'm the sys-admin for the machine in question (apologies for starting a new thread rather than replying - I just subscribed to the list)
From my reading, it seems like maybe:
mdadm --assemble /dev/md0 --uuid=8fe1fe85:eeb90460:c525faab:cdaab792 /dev/etherd/e0.[01234]
would be a thing to try?
Frankly, I'm terrified that I'll screw this up - I'm not too savy with raid.
following is a record of the only thing that I've done so far:
Please note that /dev/md1 is composed of 5 attitional drive which share the same hardware as the failed /dev/md0, but are in no other way related.
We're seriously considering sending the drives to a data recovery place and spending a bazillion bucks to recover the data. if anyone reading this feels confident that they can help us rebuild this array and get us to a place where we can copy the data off of it. Please send mail to [email protected]. We'll be happy to pay you for your services. - I'll post a summary of what we did when all is done.
help, please.
comparing the superblocks below with those posted yesterday, you can see that things have changed. I'm pulling my hair out - I hope I didn't bork our data.
-- Jonathan
hazel /tmp # df -H
Filesystem Size Used Avail Use% Mounted on /dev/hda4 67G 5.8G 58G 10% / udev 526M 177k 526M 1% /dev /dev/hda3 8.1G 34M 7.7G 1% /tmp none 526M 0 526M 0% /dev/shm /dev/md1 591G 34M 561G 1% /md1
hazel /tmp # mdadm -C /dev/md0 -n 4 -l 5 missing /dev/etherd/e0.[023]
mdadm: /dev/etherd/e0.0 appears to be part of a raid array:
level=5 devices=4 ctime=Mon Jan 3 03:16:48 2005
mdadm: /dev/etherd/e0.2 appears to be part of a raid array:
level=5 devices=4 ctime=Mon Jan 3 03:16:48 2005
mdadm: /dev/etherd/e0.3 appears to contain an ext2fs file system
size=720300416K mtime=Wed Oct 5 16:39:28 2005
mdadm: /dev/etherd/e0.3 appears to be part of a raid array:
level=5 devices=4 ctime=Mon Jan 3 03:16:48 2005
Continue creating array? y
mdadm: array /dev/md0 started.
hazel /tmp # aoe-stat
e0.0 eth1 up
e0.1 eth1 up
e0.2 eth1 up
e0.3 eth1 up
e0.4 eth1 up
e0.5 eth1 up
e0.6 eth1 up
e0.7 eth1 up
e0.8 eth1 up
e0.9 eth1 up
hazel /tmp # cat /proc/mdstat
Personalities : [raid5]
md1 : active raid5 etherd/e0.9[4] etherd/e0.8[3] etherd/e0.7[2] etherd/e0.6[1] etherd/e0.5[0]
586082688 blocks level 5, 32k chunk, algorithm 0 [4/4] [UUUU]md0 : active raid5 etherd/e0.3[3] etherd/e0.2[2] etherd/e0.0[1]
586082688 blocks level 5, 64k chunk, algorithm 2 [4/3] [_UUU]
unused devices: <none>
hazel /tmp # mkdir /md0
hazel /tmp # mount -r /dev/md0 /md0
mount: wrong fs type, bad option, bad superblock on /dev/md0,
or too many mounted file systems
hazel /tmp # mount -t ext2 -r /dev/md0 /md0
mount: wrong fs type, bad option, bad superblock on /dev/md0,
or too many mounted file systems
hazel /tmp # mdadm -S /dev/md0
hazel /tmp # aoe-stat
e0.0 eth1 up
e0.1 eth1 up
e0.2 eth1 up
e0.3 eth1 up
e0.4 eth1 up
e0.5 eth1 up
e0.6 eth1 up
e0.7 eth1 up
e0.8 eth1 up
e0.9 eth1 up
hazel /tmp # cat /proc/mdstat
Personalities : [raid5]
md1 : active raid5 etherd/e0.9[4] etherd/e0.8[3] etherd/e0.7[2] etherd/e0.6[1] etherd/e0.5[0]
586082688 blocks level 5, 32k chunk, algorithm 0 [4/4] [UUUU]unused devices: <none>
hazel /tmp # mdadm -E /dev/etherd/e0.[01234]/dev/etherd/e0.0:
Magic : a92b4efc
Version : 00.90.02
UUID : ec0bdbb3:f625880f:dbf65130:057d069c Creation Time : Fri Apr 21 22:56:18 2006
Raid Level : raid5
Device Size : 195360896 (186.31 GiB 200.05 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Fri Apr 21 22:56:18 2006
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Checksum : 1742f65 - correct
Events : 0.3493634
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 152 0 1 active sync /dev/etherd/e0.0
0 0 0 0 0 removed
1 1 152 0 1 active sync /dev/etherd/e0.0
2 2 152 32 2 active sync /dev/etherd/e0.2
3 3 152 48 3 active sync /dev/etherd/e0.3
/dev/etherd/e0.1:
Magic : a92b4efc
Version : 00.90.00
UUID : 8fe1fe85:eeb90460:c525faab:cdaab792 Creation Time : Mon Jan 3 03:16:48 2005
Raid Level : raid5
Device Size : 195360896 (186.31 GiB 200.05 GB)
Raid Devices : 4
Total Devices : 5
Preferred Minor : 0
Update Time : Fri Apr 21 14:03:12 2006
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 3
Spare Devices : 1
Checksum : 4cc991d7 - correct
Events : 0.3493633
Layout : left-asymmetric
Chunk Size : 32K
Number Major Minor RaidDevice State
this 4 152 16 4 spare /dev/etherd/e0.1
0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 152 32 2 active sync /dev/etherd/e0.2
3 3 152 48 3 active sync /dev/etherd/e0.3
4 4 152 16 4 spare /dev/etherd/e0.1
/dev/etherd/e0.2:
Magic : a92b4efc
Version : 00.90.02
UUID : ec0bdbb3:f625880f:dbf65130:057d069c Creation Time : Fri Apr 21 22:56:18 2006
Raid Level : raid5
Device Size : 195360896 (186.31 GiB 200.05 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Fri Apr 21 22:56:18 2006
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Checksum : 1742f87 - correct
Events : 0.3493634
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 152 32 2 active sync /dev/etherd/e0.2
0 0 0 0 0 removed
1 1 152 0 1 active sync /dev/etherd/e0.0
2 2 152 32 2 active sync /dev/etherd/e0.2
3 3 152 48 3 active sync /dev/etherd/e0.3
/dev/etherd/e0.3:
Magic : a92b4efc
Version : 00.90.02
UUID : ec0bdbb3:f625880f:dbf65130:057d069c Creation Time : Fri Apr 21 22:56:18 2006
Raid Level : raid5
Device Size : 195360896 (186.31 GiB 200.05 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Fri Apr 21 22:56:18 2006
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Checksum : 1742f99 - correct
Events : 0.3493634
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 152 48 3 active sync /dev/etherd/e0.3
0 0 0 0 0 removed
1 1 152 0 1 active sync /dev/etherd/e0.0
2 2 152 32 2 active sync /dev/etherd/e0.2
3 3 152 48 3 active sync /dev/etherd/e0.3
/dev/etherd/e0.4:
Magic : a92b4efc
Version : 00.90.00
UUID : 8fe1fe85:eeb90460:c525faab:cdaab792 Creation Time : Mon Jan 3 03:16:48 2005
Raid Level : raid5
Device Size : 195360896 (186.31 GiB 200.05 GB)
Raid Devices : 4
Total Devices : 5
Preferred Minor : 0
Update Time : Thu Apr 20 21:07:50 2006
State : clean
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1
Checksum : 4cc84d59 - correct
Events : 0.3482550
Layout : left-asymmetric
Chunk Size : 32K
Number Major Minor RaidDevice State
this 0 152 64 0 active sync /dev/etherd/e0.4
0 0 152 64 0 active sync /dev/etherd/e0.4 1 1 152 0 1 active sync /dev/etherd/e0.0 2 2 152 32 2 active sync /dev/etherd/e0.2 3 3 152 48 3 active sync /dev/etherd/e0.3 4 4 152 16 4 spare /dev/etherd/e0.1
Quoted 1 line · Jonathan
# mdadm -C /dev/md0 -n 4 -l 5 missing /dev/etherd/e0.[023]
I think you should have tried "mdadm --assemble --force" first, as I proposed earlier.
By doing the above, you have effectively replaced your version 0.9.0 superblocks with version 0.9.2. I don't know if version 0.9.2 superblocks are larger than 0.9.0, Neil hasn't responded to that yet. Potentially hazardous, who knows.
Anyway.
This is from your old superblock as described by Sam Hopkins:
Quoted 2 lines
/dev/etherd/<blah>:
Chunk Size : 32K
This is from what you've just posted:
Quoted 2 lines
/dev/etherd/<blah>:
Chunk Size : 64K
If I were you, I'd recreate your superblocks now, but with the correct chunk size (use -c).
Quoted 1 line
We'll be happy to pay you for your services.
I'll be modest and charge you a penny per byte of data recovered, ho hum.
I was already terrified of screwing things up -- now I'm afraid of making things worse
based on what was posted before is this a sensible thing to try?
mdadm -C /dev/md0 -c 32 -n 4 -l 5 missing /dev/etherd/e0.[023]
Is what I've done to the superblock size recoverable?
I don't understand how mdadm --assemble would know what to do, which is why I didn't try it initially. That said, obviouly my lack of understanding isn't helping one bit.
I don't think I can afford a penny per byte, but I'd happy part with hundreds of dollars to get the data back. I would really like someone with more knowledge than me to hold my hand before I continue to make things worse.
help please - [email protected]
-- Jonathan
Quoted 44 lines · Molle Bestefich
Quoted 3 lines · Jonathan
# mdadm -C /dev/md0 -n 4 -l 5 missing /dev/etherd/e0.[023]I think you should have tried "mdadm --assemble --force" first, as I proposed earlier.
By doing the above, you have effectively replaced your version 0.9.0
superblocks with version 0.9.2. I don't know if version 0.9.2
superblocks are larger than 0.9.0, Neil hasn't responded to that yet. Potentially hazardous, who knows.Anyway.
This is from your old superblock as described by Sam Hopkins:Quoted 4 lines
/dev/etherd/<blah>:
Chunk Size : 32KThis is from what you've just posted:
Quoted 4 lines
/dev/etherd/<blah>:
Chunk Size : 64KIf I were you, I'd recreate your superblocks now, but with the correct chunk size (use -c).
Quoted 3 lines
We'll be happy to pay you for your services.
I'll be modest and charge you a penny per byte of data recovered, ho hum.
Quoted 2 lines · Jonathan
I was already terrified of screwing things up
now I'm afraid of making things worse
Adrenalin... makes life worth living there for a sec, doesn't it ;o)
Quoted 2 lines
based on what was posted before is this a sensible thing to try? mdadm -C
/dev/md0-c 32 -n 4 -l 5 missing /dev/etherd/e0.[023]
Yes, looks exactly right.
Quoted 1 line
Is what I've done to the superblock size recoverable?
I don't think you've done anything at all.
I just *don't know* if you have, that's all.
Was just trying to say that it wasn't super-cautious of you to begin with, that's all :-).
Quoted 2 lines
I don't understand how mdadm --assemble would know what to do, which is why I didn't try it initially.
By giving it --force, you tell it to forcefully mount the array even though it might be damaged.
That means including some disks (the freshest ones) that are out of sync.
That help?
Well, the block sizes are back to 32k now, but I still had no luck mounting /dev/md0 once I created the array. below is a record of what I just tried:
how safe should the following be?
mdadm --assemble /dev/md0 --uuid=8fe1fe85:eeb90460:c525faab:cdaab792 /dev/etherd/e0.[01234]
I am *really* not interested in making my situation worse.
-- Jonathan
hazel /virtual # mdadm -C /dev/md0 -c 32 -n 4 -l 5 missing /dev/etherd/e0.[023]
mdadm: /dev/etherd/e0.0 appears to be part of a raid array:
level=5 devices=4 ctime=Fri Apr 21 22:56:18 2006
mdadm: /dev/etherd/e0.2 appears to be part of a raid array:
level=5 devices=4 ctime=Fri Apr 21 22:56:18 2006
mdadm: /dev/etherd/e0.3 appears to contain an ext2fs file system
size=720300416K mtime=Wed Oct 5 16:39:28 2005
mdadm: /dev/etherd/e0.3 appears to be part of a raid array:
level=5 devices=4 ctime=Fri Apr 21 22:56:18 2006
Continue creating array? y
mdadm: array /dev/md0 started.
hazel /virtual # cat /proc/mdstat
Personalities : [raid5]
md1 : active raid5 etherd/e0.9[4] etherd/e0.8[3] etherd/e0.7[2] etherd/e0.6[1] etherd/e0.5[0]
586082688 blocks level 5, 32k chunk, algorithm 0 [4/4] [UUUU]
md0 : active raid5 etherd/e0.3[3] etherd/e0.2[2] etherd/e0.0[1]
586082688 blocks level 5, 32k chunk, algorithm 2 [4/3] [_UUU]
unused devices: <none>
hazel /virtual # mount -t ext2 -r /dev/md0 /md0
mount: wrong fs type, bad option, bad superblock on /dev/md0,
or too many mounted file systems
hazel /virtual # mdadm -S /dev/md0
hazel /virtual # cat /proc/mdstat
Personalities : [raid5]
md1 : active raid5 etherd/e0.9[4] etherd/e0.8[3] etherd/e0.7[2] etherd/e0.6[1] etherd/e0.5[0]
586082688 blocks level 5, 32k chunk, algorithm 0 [4/4] [UUUU]
unused devices: <none>
hazel /virtual # mdadm -E /dev/etherd/e0.[01234]/dev/etherd/e0.0:
Magic : a92b4efc
Version : 00.90.02
UUID : 518b5d59:44292ca3:6c358813:c6f00804 Creation Time : Sat Apr 22 13:25:40 2006
Raid Level : raid5
Device Size : 195360896 (186.31 GiB 200.05 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Sat Apr 22 13:25:40 2006
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Checksum : 6aaa56f - correct
Events : 0.3493635
Layout : left-symmetric
Chunk Size : 32K
Number Major Minor RaidDevice State
this 1 152 0 1 active sync /dev/etherd/e0.0
0 0 0 0 0 removed
1 1 152 0 1 active sync /dev/etherd/e0.0
2 2 152 32 2 active sync /dev/etherd/e0.2
3 3 152 48 3 active sync /dev/etherd/e0.3
/dev/etherd/e0.1:
Magic : a92b4efc
Version : 00.90.00
UUID : 8fe1fe85:eeb90460:c525faab:cdaab792 Creation Time : Mon Jan 3 03:16:48 2005
Raid Level : raid5
Device Size : 195360896 (186.31 GiB 200.05 GB)
Raid Devices : 4
Total Devices : 5
Preferred Minor : 0
Update Time : Fri Apr 21 14:03:12 2006
State : clean
Active Devices : 2
Working Devices : 3
Failed Devices : 3
Spare Devices : 1
Checksum : 4cc991d7 - correct
Events : 0.3493633
Layout : left-asymmetric
Chunk Size : 32K
Number Major Minor RaidDevice State
this 4 152 16 4 spare /dev/etherd/e0.1
0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 152 32 2 active sync /dev/etherd/e0.2
3 3 152 48 3 active sync /dev/etherd/e0.3
4 4 152 16 4 spare /dev/etherd/e0.1
/dev/etherd/e0.2:
Magic : a92b4efc
Version : 00.90.02
UUID : 518b5d59:44292ca3:6c358813:c6f00804 Creation Time : Sat Apr 22 13:25:40 2006
Raid Level : raid5
Device Size : 195360896 (186.31 GiB 200.05 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Sat Apr 22 13:25:40 2006
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Checksum : 6aaa591 - correct
Events : 0.3493635
Layout : left-symmetric
Chunk Size : 32K
Number Major Minor RaidDevice State
this 2 152 32 2 active sync /dev/etherd/e0.2
0 0 0 0 0 removed
1 1 152 0 1 active sync /dev/etherd/e0.0
2 2 152 32 2 active sync /dev/etherd/e0.2
3 3 152 48 3 active sync /dev/etherd/e0.3
/dev/etherd/e0.3:
Magic : a92b4efc
Version : 00.90.02
UUID : 518b5d59:44292ca3:6c358813:c6f00804 Creation Time : Sat Apr 22 13:25:40 2006
Raid Level : raid5
Device Size : 195360896 (186.31 GiB 200.05 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Sat Apr 22 13:25:40 2006
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Checksum : 6aaa5a3 - correct
Events : 0.3493635
Layout : left-symmetric
Chunk Size : 32K
Number Major Minor RaidDevice State
this 3 152 48 3 active sync /dev/etherd/e0.3
0 0 0 0 0 removed
1 1 152 0 1 active sync /dev/etherd/e0.0
2 2 152 32 2 active sync /dev/etherd/e0.2
3 3 152 48 3 active sync /dev/etherd/e0.3
/dev/etherd/e0.4:
Magic : a92b4efc
Version : 00.90.00
UUID : 8fe1fe85:eeb90460:c525faab:cdaab792 Creation Time : Mon Jan 3 03:16:48 2005
Raid Level : raid5
Device Size : 195360896 (186.31 GiB 200.05 GB)
Raid Devices : 4
Total Devices : 5
Preferred Minor : 0
Update Time : Thu Apr 20 21:07:50 2006
State : clean
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1
Checksum : 4cc84d59 - correct
Events : 0.3482550
Layout : left-asymmetric
Chunk Size : 32K
Number Major Minor RaidDevice State
this 0 152 64 0 active sync /dev/etherd/e0.4
0 0 152 64 0 active sync /dev/etherd/e0.4 1 1 152 0 1 active sync /dev/etherd/e0.0 2 2 152 32 2 active sync /dev/etherd/e0.2 3 3 152 48 3 active sync /dev/etherd/e0.3 4 4 152 16 4 spare /dev/etherd/e0.1
Quoted 2 lines · Jonathan
Well, the block sizes are back to 32k now, but I still had no luck mounting
/dev/md0once I created the array.
Ahem, I missed something.
Sorry, the 'a' was hard to spot.
Your array used layout : left-asymmetric, while the superblock you've just created has layout: left-symmetric.
Try again, but add the option "--parity=left-asymmetric"
hazel /virtual # mdadm -C /dev/md0 -c 32 -n 4 -l 5 --parity=left-asymmetric missing /dev/etherd/e0.[023]
mdadm: /dev/etherd/e0.0 appears to be part of a raid array:
level=5 devices=4 ctime=Sat Apr 22 13:25:40 2006
mdadm: /dev/etherd/e0.2 appears to be part of a raid array:
level=5 devices=4 ctime=Sat Apr 22 13:25:40 2006
mdadm: /dev/etherd/e0.3 appears to contain an ext2fs file system
size=720300416K mtime=Wed Oct 5 16:39:28 2005
mdadm: /dev/etherd/e0.3 appears to be part of a raid array:
level=5 devices=4 ctime=Sat Apr 22 13:25:40 2006
Continue creating array? y
mdadm: array /dev/md0 started.
hazel /virtual # mount -t ext2 -r /dev/md0 /md0
hazel /virtual # df -H
Filesystem Size Used Avail Use% Mounted on /dev/hda4 67G 5.8G 58G 10% / udev 526M 177k 526M 1% /dev /dev/hda3 8.1G 34M 7.7G 1% /tmp none 526M 0 526M 0% /dev/shm /dev/md1 591G 11G 551G 2% /virtual /dev/md0 591G 54G 507G 10% /md0
now I'm doing a:
(cd /md0 && tar cf - . ) | (cd /virtual/recover/ && tar xvfp -)
thank you thank you thank you thank you thank you thank you
Quoted 16 lines · Molle Bestefich
Quoted 4 lines · Jonathan
Well, the block sizes are back to 32k now, but I still had no luck mounting
/dev/md0once I created the array.Ahem, I missed something.
Sorry, the 'a' was hard to spot.Your array used layout : left-asymmetric, while the superblock you've just created has layout: left-symmetric.
Try again, but add the option "--parity=left-asymmetric"
Quoted 1 line · Jonathan
/dev/md0591G 54G 507G 10% /md0
Hrm. Have you been hassling us for a mere 54G worth of data? (Just kidding :-D.)
Quoted 2 lines
now I'm doing a:
(cd /md0 && tar cf - . ) | (cd /virtual/recover/ && tar xvfp -)
I wonder - if you have the disk space, why not take David Greaves' advice and create a backup of the individual disks before fiddling with md superblocks?
Anyway, a quick cheat sheet might come in handy:
* Hot-add a disk to an array (outdated raid data on that disk won't be used):
# mdadm /dev/md1 --add /dev/etherd/e0.xyz
* Check that parity is OK for an entire array:
# echo check > /sys/block/md1/md/sync_action
* Manually start a resync:
# echo repair > /sys/block/md1/md/sync_action
* Experiment with MDADM by using sparse files:
# dd if=/dev/zero of=sparse0 bs=1M seek=200000 count=1
# dd if=/dev/zero of=sparse1 bs=1M seek=200000 count=1
# dd if=/dev/zero of=sparse2 bs=1M seek=200000 count=1
# dd if=/dev/zero of=sparse3 bs=1M seek=200000 count=1
# losetup /dev/loop0 sparse0
# losetup /dev/loop1 sparse1
# losetup /dev/loop2 sparse2
# losetup /dev/loop3 sparse3
# mdadm -C /dev/md5 -c 32 -p left-a -n 4 -l 5 missing /dev/loop{1,2,3}Quoted 1 line
thank you thank you thank you thank you thank you thank you
np.
(wait, does that mean I won't get my money? ;-))
Molle Bestefich ([email protected]) wrote on 22 April 2006 23:17:
Quoted 2 lines
Quoted 1 line · Jonathan
thank you thank you thank you thank you thank you thank you
Nice job Molle!
Quoted 2 lines
np.
(wait, does that mean I won't get my money? ;-))
At least you showed someone a new meaning of "support"...
Is it my turn now? Just look at this:
md1 : active raid5 sdp2[15](F) sdo2[16](F) sdn2[13] sdm2[12] sdl2[11] sdk2[10] sdj2[9] sdi2[8] sdh2[17](F) sdg2[18](F) sdf2[5] sde2[4] sdd2[3] sdc2[2] sdb2[1] sda2[0]
4198656 blocks level 5, 128k chunk, algorithm 2 [15/12] [UUUUUU__UUUUUU_]
md3 : active raid5 sdp5[15](F) sdo5[16](F) sdn5[13] sdm5[12] sdl5[11] sdk5[10] sdj5[9] sdi5[8] sdh5[17](F) sdg5[18](F) sdf5[5] sde5[4] sdd5[3] sdc5[2] sdb5[1] sda5[0]
588487424 blocks level 5, 128k chunk, algorithm 2 [15/12] [UUUUUU__UUUUUU_]
md4 : active raid5 sdp6[15](S) sdo6[16](F) sdn6[13] sdm6[12] sdl6[11] sdk6[10] sdj6[9] sdi6[8] sdh6[17](F) sdg6[18](F) sdf6[5] sde6[4] sdd6[3] sdc6[2] sdb6[1] sda6[0]
2141065472 blocks level 5, 128k chunk, algorithm 2 [15/12] [UUUUUU__UUUUUU_]
This is a 15-array + spare where 3 or 4 disks dropped, *very* likely because of these miserable crappy sata power connectors :-( :-( It's unbelievable that the industry changes from a working standard to an unreliable trash in such common and widespread devices...
I'm trying to get enough stamina to put the damn thing back on line... Or at least try to :-(
Quoted 1 line · Carlos Carvalho
Is it my turn now?
Hah, nah, based on your postings you seem to know more about MD than I do ;-)
Quoted 4 lines
[
UUUUUU__UUUUUU_]This is a 15-array + spare where 3 or 4 disks dropped, *very* likely because of these miserable crappy sata power connectors :-( :-(
They really suck, don't they.
I had a SATA data connector that was slightly skewed the other day, no more than 1.5mm, and it caused random read errors. Gah.
Quoted 2 lines
It's unbelievable that the industry changes from a working standard to an unreliable trash in such common and widespread devices...
Yes. Sigh..
Quoted 1 line
I'm trying to get enough stamina to put the damn thing back on line...
Guess it's just --assemble --force and then cross your fingers...
Hmm, your mdstat does look a bit weird.
Why wasn't the spare taken into use on md4?
Perhaps you should check the event counters.
Just in case the spares got added late in the game, got half-rebuilt and the MD crapped out?
Wouldn't want to include them in the assemble in that case... (Dunno if it could even happen. Hopefully not =))
Quoted 2 lines · Molle Bestefich
Anyway, a quick cheat sheet might come in handy:
Which is why I posted about a wiki a few days back :)
I'm progressing it and I'll see if we can't get something up.
There's a lot of info on the list and it would be nice to get it a little more focused...
David
nice to hear you got your data back.
now it's perhaps a good time to donate some money to some ppls/oss-projects for saving your ass ;) ;)
greets, chris
Quoted 54 lines · Jonathan
hazel /virtual # mdadm -C
/dev/md0-c 32 -n 4 -l 5 --parity=left-asymmetric missing /dev/etherd/e0.[023]
mdadm:/dev/etherd/e0.0appears to be part of a raid array:level=5 devices=4 ctime=Sat Apr 22 13:25:40 2006 mdadm: /dev/etherd/e0.2 appears to be part of a raid array: level=5 devices=4 ctime=Sat Apr 22 13:25:40 2006 mdadm: /dev/etherd/e0.3 appears to contain an ext2fs file system size=720300416K mtime=Wed Oct 5 16:39:28 2005 mdadm: /dev/etherd/e0.3 appears to be part of a raid array: level=5 devices=4 ctime=Sat Apr 22 13:25:40 2006Continue creating array? y
mdadm: array/dev/md0started.
hazel /virtual # mount -t ext2 -r/dev/md0/md0
hazel /virtual # df -HFilesystem Size Used Avail Use% Mounted on /dev/hda4 67G 5.8G 58G 10% / udev 526M 177k 526M 1% /dev /dev/hda3 8.1G 34M 7.7G 1% /tmp none 526M 0 526M 0% /dev/shm /dev/md1 591G 11G 551G 2% /virtual /dev/md0 591G 54G 507G 10% /md0now I'm doing a:
(cd /md0 && tar cf - . ) | (cd /virtual/recover/ && tar xvfp -)
thank you thank you thank you thank you thank you thank you
Quoted 8 lines · Molle Bestefich
Quoted 3 lines · Jonathan
Well, the block sizes are back to 32k now, but I still had no luck mounting
/dev/md0once I created the array.Quoted 8 lines
Ahem, I missed something.
Sorry, the 'a' was hard to spot.Your array used layout : left-asymmetric, while the superblock you've just created has layout: left-symmetric.
Try again, but add the option "--parity=left-asymmetric"
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Jonathan
how safe should the following be?
mdadm --assemble
/dev/md0--uuid=8fe1fe85:eeb90460:c525faab:cdaab792 /dev/etherd/e0.[01234]
You can hardly do --assemble anymore.
After you have recreated superblocks on some of the devices, those are conceptually part of a different raid array. At least as seen by MD.
Quoted 1 line
I am *really* not interested in making my situation worse.
We'll keep going till you got your data back.. Recreating superblocks again on e0.{0,2,3} can't hurt, since you've already done this and thereby nuked the old superblocks.
You can shake your own hand and thank yourself now (oh, and Sam too) for posting all the debug output you have. Otherwise we would probably never have spotted nor known about the parity/chunk size differences :o).
Jonathan ([email protected]) wrote on 22 April 2006 13:07:
Quoted 8 lines
I was already terrified of screwing things up -- now I'm afraid of making things worse
based on what was posted before is this a sensible thing to try?
mdadm -C
/dev/md0-c 32 -n 4 -l 5 missing /dev/etherd/e0.[023]Is what I've done to the superblock size recoverable?
Raid metadata are stored at the end of the partition, in a small area. Perhaps some overwrite of data happened, I don't know, but it'd be small. If you re-make the superblock with the right chunk size you can read the data back.
I'd suggest following the help you're getting on the list, and if you don't understand something ask before running any commands. Also don't do anything that causes writes to the disks (such as fsck) before getting the raid back...
Quoted 10 lines · On Saturday April 22, [email protected] wrote:
Quoted 1 line · Jonathan
# mdadm -C /dev/md0 -n 4 -l 5 missing /dev/etherd/e0.[023]I think you should have tried "mdadm --assemble --force" first, as I proposed earlier.
By doing the above, you have effectively replaced your version 0.9.0 superblocks with version 0.9.2. I don't know if version 0.9.2 superblocks are larger than 0.9.0, Neil hasn't responded to that yet. Potentially hazardous, who knows.
There is no difference in the superblock between 0.90.0 and 0.90.2.
md has always used version numbers, but always in a confusing way. There should be two completely separate version numbers: the version for the format of the superblock, and the version for the software implementation. md confuses these two.
To try to sort it out, I have decided that:
- The 'major' version number is the overall choice of superblock This is currently 0 or 1
- The 'minor' version encodes minor variation in the superblock. For version 1, this is different locations (there are other bits in the superblock to allow new fields to be added)
For version 0, it is currently only used to make sure old software
doesn't try to assemble an array which is undergoing a shape, as that would confuse it totals.
- The 'patchlevel' is used to indicate feature availability in the implementation. It really should be stored in the superblock, but it is for historical reasons. It is not checked when validating a superblock.
To quote from md.h
/*
* MD_PATCHLEVEL_VERSION indicates kernel functionality.
* >=1 means different superblock formats are selectable using SET_ARRAY_INFO
* and major_version/minor_version accordingly
* >=2 means that Internal bitmaps are supported by setting MD_SB_BITMAP_PRESENT
* in the super status byte
* >=3 means that bitmap superblock version 4 is supported, which uses
* little-ending representation rather than host-endian
*/
Hope that helps.
NeilBrown