RAID 5 Recovery Help Needed
Hi,
I'm looking for some assistance in recovering a filesystem (if possible) on a recently created RAID 5 array.
To summarize: I created the array from 3x 1.5TB drives, each having a GPT partition table and one non-fs data partition (0xDA), following the instructions located here:
http://linux-raid.osdl.org/index.php/RAID_setup#RAID-5 After creating the array, I formatted md0 with an ext4 (perhaps my first mistake) partition and used it without issue for a few days. After I rebooted, I found the array did not assemble at boot, and I was unable to manually assemble it getting errors of no md superblocks on the partitions. After a lot of reading online, I tried recreating the array using the original parameters and the addition of '--assume-clean' to prevent a resync or rebuild (under the assumption this will keep the data intact). The array was recreated, and as far as I could tell no data was touched (no noticable hard drive activity). Upon trying to mount md0 with the newly created array I get missing/bad superblock errors.
Now, the more verbose details:
I am running a vanilla 2.6.28 kernel (on Debian).
e2fslibs and e2fsprogs are both version 1.41.3
mdadm version 2.6.7.1
The steps I followed are as follows (exact commands pulled from bash history where possible):
Created GPT partition tables on sdb,sdc,sdd
Created the partitions:
# parted /dev/sdb mkpart non-fs 0% 100% # parted /dev/sdc mkpart non-fs 0% 100% # parted /dev/sdd mkpart non-fs 0% 100%
Created the array:
# mdadm --create --verbose /dev/md0 --level=5 --chunk=128 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1
/dev/md0:
Version : 00.90
Creation Time : Sun Jan 11 19:00:43 2009
Raid Level : raid5
Array Size : 2930276864 (2794.53 GiB 3000.60 GB)
Used Dev Size : 1465138432 (1397.26 GiB 1500.30 GB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Sun Jan 11 19:18:13 2009
State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 128K
Rebuild Status : 2% complete
UUID : 3128da32:c5e4ff31:b43fc0e6:226924cf (local to host 4400x2)
Events : 0.4
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
3 8 49 2 spare rebuilding /dev/sdd1
At this point, this could be seen in /var/log/messages:
kernel: md: bind<sdb1> kernel: md: bind<sdc1> kernel: md: bind<sdd1> kernel: xor: automatically using best checksumming function: pIII_sse kernel: pIII_sse : 11536.000 MB/sec kernel: xor: using function: pIII_sse (11536.000 MB/sec) kernel: async_tx: api initialized (sync-only) kernel: raid6: int32x1 1210 MB/s kernel: raid6: int32x2 1195 MB/s kernel: raid6: int32x4 898 MB/s kernel: raid6: int32x8 816 MB/s kernel: raid6: mmxx1 3835 MB/s kernel: raid6: mmxx2 4207 MB/s kernel: raid6: sse1x1 2640 MB/s kernel: raid6: sse1x2 3277 MB/s kernel: raid6: sse2x1 4988 MB/s kernel: raid6: sse2x2 5394 MB/s kernel: raid6: using algorithm sse2x2 (5394 MB/s) kernel: md: raid6 personality registered for level 6 kernel: md: raid5 personality registered for level 5 kernel: md: raid4 personality registered for level 4 kernel: raid5: device sdc1 operational as raid disk 1 kernel: raid5: device sdb1 operational as raid disk 0 kernel: raid5: allocated 3172kB for md0 kernel: RAID5 conf printout: kernel: --- rd:3 wd:2 kernel: disk 0, o:1, dev:sdb1 kernel: disk 1, o:1, dev:sdc1 kernel: md0:RAID5 conf printout: kernel: --- rd:3 wd:2 kernel: disk 0, o:1, dev:sdb1 kernel: disk 1, o:1, dev:sdc1 kernel: disk 2, o:1, dev:sdd1 kernel: md: recovery of RAID array md0
kernel: md: md0: recovery done. kernel: RAID5 conf printout: kernel: --- rd:3 wd:3 kernel: disk 0, o:1, dev:sdb1 kernel: disk 1, o:1, dev:sdc1 kernel: disk 2, o:1, dev:sdd1
Once the recovery was done I created the ext4 filesystem on md0:
# mkfs.ext4 -b 4096 -m 0 -O extents,uninit_bg -E stride=32,stripe-width=64 /dev/md0
mke2fs 1.41.3 (12-Oct-2008)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
183148544 inodes, 732569216 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
22357 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
At this point I used the newly created array for a few days without any issues at all.
Then, I edited /etc/fstab to contain mount entry for /dev/md0 and rebooted. I should note that before brining the system back up I removed a drive (formerly sde) that was no longer being used).
After the reboot, md0 was not assembled and I see this in messages (note that I have the raid modules compiled in statically).
kernel: pIII_sse : 11536.000 MB/sec
kernel: xor: using function: pIII_sse (11536.000 MB/sec)
kernel: async_tx: api initialized (sync-only)
kernel: raid6: int32x1 1218 MB/s
kernel: raid6: int32x2 1199 MB/s
kernel: raid6: int32x4 898 MB/s
kernel: raid6: int32x8 816 MB/s
kernel: raid6: mmxx1 3863 MB/s
kernel: raid6: mmxx2 4273 MB/s
kernel: raid6: sse1x1 2640 MB/s
kernel: raid6: sse1x2 3285 MB/s
kernel: raid6: sse2x1 5007 MB/s
kernel: raid6: sse2x2 5402 MB/s
kernel: raid6: using algorithm sse2x2 (5402 MB/s)
kernel: md: raid6 personality registered for level 6
kernel: md: raid5 personality registered for level 5
kernel: md: raid4 personality registered for level 4
kernel: device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised:
[email protected]
kernel: md: md0 stopped.
Now I tried to assemble the array manually using the following commands, all of which failed (note that I had never edited mdadm.conf).
# mdadm --assemble --scan # mdadm --assemble --scan --uuid=3128da32:c5e4ff31:b43fc0e6:226924cf # mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 # mdadm --assemble -f /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 # mdadm --assemble --uuid=3128da32:c5e4ff31:b43fc0e6:226924cf /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1
The contents of mdadm.conf while executing the above were:
DEVICE partitions
CREATE owner=root group=disk mode=0660 auto=yes
HOMEHOST <system>
MAILADDR root
While attempting to assemble using the above, I received the output:
no devices found for /dev/md0
or
no recogniseable superblock on sdb1
After some googling, I read that recreating the array using the same parameters and --assume-clean should keep the data intact. So, I did the following (this may be my biggest mistake):
# mdadm --create --verbose /dev/md0 --level=5 --chunk=128 --assume-clean --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1
The recreation happened immediately with no rebuilding or other noticable data movement. I got the following output:
/dev/md0:
Version : 00.90
Creation Time : Thu Jan 15 21:45:26 2009
Raid Level : raid5
Array Size : 2930276864 (2794.53 GiB 3000.60 GB)
Used Dev Size : 1465138432 (1397.26 GiB 1500.30 GB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Thu Jan 15 21:45:26 2009
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 128K
UUID : 78902c67:a59cf188:b43fc0e6:226924cf (local to host 4400x2)
Events : 0.1
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1
And the following was seen in messages:
kernel: md: bind<sdb1>
kernel: md: bind<sdc1>
kernel: md: bind<sdd1>
kernel: raid5: device sdd1 operational as raid disk 2
kernel: raid5: device sdc1 operational as raid disk 1
kernel: raid5: device sdb1 operational as raid disk 0
kernel: raid5: allocated 3172kB for md0
kernel: raid5: raid level 5 set md0 active with 3 out of 3 devices, algorithm 2
kernel: RAID5 conf printout: kernel: --- rd:3 wd:3 kernel: disk 0, o:1, dev:sdb1 kernel: disk 1, o:1, dev:sdc1 kernel: disk 2, o:1, dev:sdd1 kernel: md0: unknown partition table
Upon trying to with:
# mount -t ext4 /dev/md0 /media/archive
I get the following:
mount: wrong fs type, bad option, bad superblock on /dev/md0
When I try to run fsck with
# fsck.ext4 -n /dev/md0
I get:
fsck.ext4: Superblock invalid, trying backup blocks... fsck.ext4: Bad magic number in super-block while trying to open /dev/md0
I've tried specifying the blocksize and specifying the superblock manually using the backup superblocks from when I ran mkfs.ext4, but get the same result. I haven't dared to run fsck without -n until I hear from someone more knowledged.
So, if anyone has any suggestions on how I can get md0 mounted or recover my data it would be much appreciated.
Thanks,
Mike Berger
Quoted 3 lines · Mike Berger
Created the array:
# mdadm --create --verbose /dev/md0 --level=5 --chunk=128 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1
eek ... a RAID5 on 3 drives?
Did you do an
mdadm --detail --scan > /etc/mdadm.conf
after this?
[...]
Quoted 2 lines
At this point I used the newly created array for a few days without any issues at all.
... but did you update mdadm.conf as above?
[...]
Quoted 4 lines
Now I tried to assemble the array manually using the following commands, all of which failed (note that I had never edited mdadm.conf).
# mdadm --assemble --scan
This generally requires an /etc/mdadm.conf (or similar /etc/mdadm/mdadm.conf)
Quoted 5 lines
# mdadm --assemble --scan --uuid=3128da32:c5e4ff31:b43fc0e6:226924cf # mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 # mdadm --assemble -f /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 # mdadm --assemble --uuid=3128da32:c5e4ff31:b43fc0e6:226924cf /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1
You can often (re)construct this file if you forget to create it with a little detective work ...
mdadm --examine /dev/sdb1
could be your friend.
[...]
Quoted 7 lines
# fsck.ext4 -n /dev/md0I get:
fsck.ext4: Superblock invalid, trying backup blocks... fsck.ext4: Bad magic number in super-block while trying to open
/dev/md0
Quoted 7 lines
I've tried specifying the blocksize and specifying the superblock manually using the backup superblocks from when I ran mkfs.ext4, but get the same result. I haven't dared to run fsck without -n until I hear from someone more knowledged.
So, if anyone has any suggestions on how I can get md0 mounted or recover my data it would be much appreciated.
I am not completely sure, but I would bet that with the changes you have made, that this data may not be recoverable at this point.
Before you do anything else, I would definitely suggest creating the mdadm.conf file properly as noted above.
Joe
Signature · 4 lines
Joe Landman Scalable Informatics LLC, email: [email protected] web : http://www.scalableinformatics.com
Quoted 1 line · Joe Landman
eek ... a RAID5 on 3 drives?
I was planning on growing it as I needed more space and could afford more drives (scary, I know).
Quoted 5 lines
Did you do an
mdadm --detail --scan > /etc/mdadm.confafter this?
Unfortunately no. The guide I followed neglected to mention that step.
Quoted 9 lines
... but did you update mdadm.conf as above?
This generally requires an/etc/mdadm.conf(or similar/etc/mdadm/mdadm.conf)
You can often (re)construct this file if you forget to create it with a little detective work ...
mdadm --examine/dev/sdb1
could be your friend.[...]
I did try putting the following into mdadm.conf manually after the reboot (and before doing the create again):
DEVICES partitions
ARRAY /dev/md0 UUID=3128da32:c5e4ff31:b43fc0e6:226924cf
Then I tried to assemble it again (with scan), and still no luck. Unfortunately since the guide I used never mentioned the config file at all, and seeing "DEVICE partitions" in it, I assumed it was fine. I never gave it more thought until the second create failed to improve things.
Quoted 16 lines
Quoted 8 lines
# fsck.ext4 -n /dev/md0I get:
fsck.ext4: Superblock invalid, trying backup blocks... fsck.ext4: Bad magic number in super-block while trying to open
/dev/md0Quoted 7 lines
I've tried specifying the blocksize and specifying the superblock manually using the backup superblocks from when I ran mkfs.ext4, but get the same result. I haven't dared to run fsck without -n until I hear from someone more knowledged.
So, if anyone has any suggestions on how I can get md0 mounted or recover my data it would be much appreciated.
Quoted 8 lines
I am not completely sure, but I would bet that with the changes you have made, that this data may not be recoverable at this point.
Before you do anything else, I would definitely suggest creating the mdadm.conf file properly as noted above.
Joe
I've created an mdadm.conf file via 'mdadm --detail --scan', so assuming that was the original problem with the raid not being assembled properly before, that should be solved. The problem now lies in getting my data back if possible. I don't plan to assemble the array again or try anything else until I've given enough time for others wiser than I to respond.
Thanks,
Mike
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 14 lines · Bill Davidsen
Depending on your distribution, I would not be surprised to see your array assemble just fine with or without an ARRAY line, it seems some try harder than others and I just noticed that the system I use most has only "super minor" lines for the first three arrays and nothing for the rest, yet arrays 0..6 all come up at boot.
However, the bad news is that many distributions don't do ext4 very well, and may mount an ext4 file system as ext3, resulting in damage which gets worse if you write to it. I would suggest starting the array, unmounted, running fsck.ext4 on the array, and seeing if it offers any hope of recovery. If so, after the fsck mount the array *read-only* and see what's there.
Quoted 10 lines
Some distributions seem to need "ext4" on the boot command line to detect the file system type for mounting. Of course if it's in fstab it should work correctly, but I wouldn't bet the farm on it.
Signature · 3 lines
Bill Davidsen <[email protected]> "Woe unto the statesman who makes war without a reason that will still be valid when the war is over..." Otto von Bismark
I have an ext4 partition on another drive working flawlessly, so I don't think it is or was being mounted incorrectly as ext3. I have a feeling the ext4 boot option is needed for the root partition in the cases you mention.
Thanks for the suggestions. I've tried a dry run (-n) of fsck without success, but put off doing a full fsck for fear of corrupting the data were it to be recoverable. I will try the full fsck before abandoning all hope.
Mike