Recovering from default FC6 install

I tried something new on a test system, using the install partitioning tools to partition the disk. I had three drives and went with RAID-1 for boot, and RAID-5+LVM for the rest. After the install was complete I noted that it was solid busy on the drives, and found that the base RAID appears to have been created (a) with no superblock and (b) with no bitmap. That last is an issue, as a test system it WILL be getting hung and rebooted, and recovering the 1.5TB took hours.

Is there an easy way to recover this? The LVM dropped on it has a lot of partitions, and there is a lot of data in them asfter several hours of feeding with GigE, so I can't readily back up and recreate by hand.

Suggestions?

bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979

First, the Fedora installer *always* creates persistent arrays, so I'm not sure what is making you say it didn't, but they should be persistent.

So, assuming that they are persistent, just recreate the arrays in place as version 1.0 superblocks with internal bitmap. I did that exact thing on my FC6 machine I was testing with (raid1, not raid5, but no biggie there) and it worked fine. The detailed list of instructions:

Reboot with a rescue CD, skip the finding of the installation, when you are at a prompt, use mdadm to examine the raid superblocks so you get all the pertinent data such as chunk size on the raid5 and ordering of constituent drives in the raid5 right. Then recreate the arrays as version 1.0 superblocks with internal write intent bitmaps. Then mount the partitions manually, bind mount things like /dev and /proc into wherever you mounted the root filesystem, edit the mdadm.conf on the root filesystem and remove the ARRAY lines (the GUIDs will be wrong now), use mdadm -Db or mdadm -Eb to get new ARRAY lines and append them to the mdadm.conf (possibly altering the device names for the arrays, and if you use -E remember to correct the printout of the GUID in the array line, it's 10:8:8:6 instead of 8:8:8:8), patch mkinitrd with something like the attached patch, patch /etc/rc.d/rc.sysinit with something like the other attached patch (or leave this patch out but manually add the correct auto= parameter to your ARRAY lines in the mdadm.conf), chroot into the root filesystem, remake your initrd image, fdisk the drives and switch the linux partition types from raid autodetect to plain linux, reboot, and you are done.

Doug Ledford <[email protected]>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

@@ -1096,6 +1096,13 @@
     mknod $MNTIMAGE/dev/efirtc c 10 136
 fi
 
+if [ -n "$raiddevices" ]; then
+    inst /sbin/mdadm.static "$MNTIMAGE/bin/mdadm"
+    if [ -f /etc/mdadm.conf ]; then
+        cp $verbose /etc/mdadm.conf "$MNTIMAGE/etc/mdadm.conf"
+    fi
+fi
+
 # FIXME -- this can really go poorly with clvm or duplicate vg names.
 # nash should do lvm probing for us and write its own configs.
 if [ -n "$vg_list" ]; then

@@ -1234,8 +1241,7 @@
 
 if [ -n "$raiddevices" ]; then
     for dev in $raiddevices; do
-        cp -a /dev/${dev} $MNTIMAGE/dev
-        emit "raidautorun /dev/${dev}"
+        emit "mdadm -As --auto=yes /dev/${dev}"
     done
 fi
 

@@ -403,7 +403,7 @@
 update_boot_stage RCraid
 [ -x /sbin/nash ] && echo "raidautorun /dev/md0" | nash --quiet
 if [ -f /etc/mdadm.conf ]; then
-    /sbin/mdadm -A -s
+    /sbin/mdadm -A -s --auto=yes
 fi
 
 # Device mapper & related initialization

I got the detail on the md device, then -E on the components, and got a "no super block found" message, which made me think it wasn't there. Given that, I didn't have much hope for the part which starts "assuming that they are persistent" but I do thank you for the information, I'm sure it will be useful.

I did try recreating, from the running FC6 rather than the rescue, since the large data was on it's own RAID and I could umount the f/s and stop the array. Alas, I think a "grow" is needed somewhere, after configuration, start, and mount of the f/s on RAID-5, e2fsck told me my data was toast. Shortest time to solution was to recreate the f/s and reload the data.

The RAID-1 stuff is small, a total rebuild is acceptable in the case of a failure.

FC install suggestion: more optional control over the RAID features during creation. Maybe there's an "advanced features" button in the install and I just missed it, but there should be, since the non-average user might be able to do useful things with the chunk size, and specify a bitmap. I would think that a bitmap would be the default on large arrays, assuming that >1TB is still large for the moment.

Instructions and attachments save for future use, trimmed here.

bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979

You can add a bitmap to an array while it is active.
mdadm --grow --bitmap=internal /dev/mdX

(Providing the kernel is new enough - I'm not sure when that went in, but it was a while ago).

NeilBrown

Thank you!!! Works like a charm!

The latest FC6 kernel is 2.6.18 something, should be okay.

bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979