Skip to contentSkip to conversation

Big mistake while changing a RAID5 disk (was Corrupted ext4 filesystem ...)

DISCUSSION L.M.J 2 participants 2 messages May 4, 2014 – May 5, 2014 MD RAID

lore
Share & tools
lore

L.M.J

author
#0 lore raw

Hello,

Sorry to ask for help again, I want to start from a fresh email to be sure I explain my problem correctly.

Here is my story 10 days again :

1) My 2 years old 3 disks RAID5 got a faulty drive : sdb, sdc and sdd remains good.

2) I maked sdb1 as faulty : mdadm --fail /dev/md0 /dev/sdb1

3) Stopped my computer, removed sdb and put a new hard drive

4) Restarted the computer and then, the RAID5 didn't want to start by itself (should be in degraded mode). Maybe I've mixed up SATA cables ?

5) I'm not a mdadm expert, after a couple of minutes on Google, I've found a command who looked quite good to me. I made a new partition on sdb (sdb1) and :
~# mdadm -Cv /dev/md0 --assume-clean --level=5 --raid-devices=3 /dev/sdc1 /dev/sdd1 /dev/sdb1 -> several mistakes here : "-C" was not a good option, mix drive order...

6) I had LVM on this md0, so I ran pvdisplay, pvscan, vgdisplay but they returned empty information...

7) At this time, I didn't know I've did a big mistake... Then, I try to rebuild the array :
~# mdadm --create /dev/md0 --level=5 --raid-devices=3 missing /dev/sdd1 /dev/sdc1 Here am I again : "--create" was not a good option, but I put the right number of drives... I guess I've lost all my chance to recover my data here :-(

8) Then, I try :
~# mdadm --assemble --force /dev/md0 /dev/sdc1 /dev/sdd1 ~# mdadm --add /dev/md0 /dev/sdb1
pvdisplay still returned empty information.

9) To be sure if I've wiped out my data or not, I did this ~# dd if=/dev/md0 bs=512 count=255 skip=1 of=/tmp/md0.txt md0 first bytes still contains valid informations !

	[..]
	physical_volumes {
		pv0 {
			id = "5DZit9-6o5V-a1vu-1D1q-fnc0-syEj-kVwAnW"
			device = "/dev/md0"
			status = ["ALLOCATABLE"]
			flags = []
			dev_size = 7814047360
			pe_start = 384
			pe_count = 953863
		}
	}
	logical_volumes {
		lvdata {
			id = "JiwAjc-qkvI-58Ru-RO8n-r63Z-ll3E-SJazO7"
			status = ["READ", "WRITE", "VISIBLE"]
			flags = []
			segment_count = 1
	[..]

That's the reason I still want to try to recover my data... So, I've done a
~# pvcreate --uuid "5DZit9-6o5V-a1vu-1D1q-fnc0-syEj-kVwAnW"

	--restorefile /etc/lvm/archive/lvm-raid_00302.vg /dev/md0 
   ~# vgcfgrestore lvm-raid
   ~# lvs -a -o +devices
        LV     VG       Attr   LSize   Origin Snap%  Move Log Copy%  Convert  Devices
        lvmp   lvm-raid -wi-a-  80,00g                                        /dev/md0(263680)
   ~# lvchange -ay /dev/lvm-raid/lv*
   ~# mount /home/foo/RAID_mp/  (ext4 partition)
   ~# df -h /home/foo/RAID_mp
      Filesystem                            Size  Used Avail Use%   Mounted on
      /dev/mapper/lvm--raid-lvmp   79G   61G   19G  77%   /home/foo/RAID_mp
   ~# ls -la /home/foo/RAID_mp
      total 0

    -> that's the big problem, my filesystem seems now corrupted...

8) Sad of my presumed mistakes, I asked help (too late) to linux-lvm and linux-raid I tried to recreate the array again without sdb and add it again : same problem.

9) I did a fsck on a /dev/mapper/lvm--raid-lvmp snapshot (avoid modification on the original filesystem). It recovered around 50% of the files only, all located in lost+found/ directory with names starting with #xxxxx.

10) Last news : yesterday, I rebooted my computer after an upgrade and now, the RAID is not available again. ~# cat /proc/mdstat

     Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
     md0 : inactive sdb1[2](S)
           1953511936 blocks
      
     md_d0 : inactive sdc1[1](S)
           1953511936 blocks

What is md_d0 ? Where is my RAID5 md0 with sdb1/sdc1/sdd1 ? Maybe that's a problem from /etc/mdadm/mdadm.conf ? ~# cat /etc/mdadm/mdadm.conf
[...]

   # definitions of existing MD arrays
   ARRAY /dev/md0 level=raid5 num-devices=3 UUID=eb75a31a:35312029:5e3c6b8a:6edaa46b
   ARRAY /dev/md0 level=raid5 num-devices=3 UUID=71b4b533:64c36783:5e3c6b8a:6edaa46b
   [...]

My mdadm config seems really fuck up, no ? Any chance to recover something ?

Thanks

#1 lore raw
Quoted 2 lines · L.M.J

5) I'm not a mdadm expert, after a couple of minutes on Google, I've found a command who looked quite good to me. I made a new partition on sdb (sdb1) and :

Where did you find this? https://raid.wiki.kernel.org/index.php/RAID_Recovery explicitly states not to do that unless as a last last resort.

It also has some hints about what you can do next (the permutate_array.pl script).

Signature · 1 line
Mikael Abrahamsson    email: [email protected]