Using mdadm instead of dmraid for BIOS-RAID root volume

I have a number of systems with Ubuntu 12.04 and Intel BIOS-RAID mirrored pairs for the boot disk. These come up as dmraid, with the root filesystem on /dev/mapper/isw_XXXXXXXXXX_Volume0p1.

I would like to convert them to use mdadm instead (so for example I can monitor them with /proc/mdstat)

The mdadm version is 3.2.5, and the manpage says that dff and imsm metadata is supported. mdadm --examine confirms this:

|# mdadm --examine /dev/sda
/dev/sda:
Magic : Intel Raid ISM Cfg Sig.
Version : 1.1.00
Orig Family : 8bc6b015
Family : 8bc6b015
Generation : 00000010
Attributes : All supported
UUID : 2ff8e106:XXXXXXXX:XXXXXXXX:XXXXXXXX
Checksum : b92d117e correct
MPB Sectors : 1
Disks : 2
RAID Devices : 1

Disk00 Serial : S14CNEAXXXXXXXX

          State : active
             Id : 00000000
    Usable Size : 234435342 (111.79 GiB 120.03 GB)

[Volume0]:
UUID : a7fb0d20:XXXXXXXX:XXXXXXXX:XXXXXXXX
RAID Level : 1
Members : 2
Slots : [UU]
Failed disk : none
This Slot : 0
Array Size : 222715904 (106.20 GiB 114.03 GB)
Per Dev Size : 222716168 (106.20 GiB 114.03 GB)
Sector Offset : 0
Num Stripes : 869984
Chunk Size : 64 KiB
Reserved : 0
Migrate State : idle
Map State : uninitialized
Dirty State : clean

Disk01 Serial : S14CNEAXXXXXXXX

          State : active
             Id : 00000001
    Usable Size : 234435342 (111.79 GiB 120.03 GB)

|(ditto for /dev/sdb)

Since this machine is going to need a reinstall for an unrelated reason anyway, I thought as an experiment I'd first try to convert it to use mdadm at boot.

What I did was:

1. apt-get remove dmraid; apt-get autoremove

This gets rid of:

/lib/udev/rules.d/97-dmraid.rules /usr/share/initramfs-tools/scripts/local-top/dmraid /usr/share/initramfs-tools/hooks/dmraid

but we still have:

/lib/udev/rules.d/64-md-raid.rules /usr/share/initramfs-tools/hooks/mdadm /usr/share/initramfs-tools/scripts/mdadm-functions /usr/share/initramfs-tools/scripts/local-premount/mdadm /usr/share/initramfs-tools/scripts/init-premount/mdadm

2. /usr/share/mdadm/mkconf >/etc/mdadm/mdadm.conf

ARRAY metadata=imsm UUID=2ff8...
ARRAY /dev/md/Volume0 container=2ff8... member=0 UUID=...

However, after a reboot, the machine came up with /dev/sdb1 as its root. I was hoping for /dev/md/Volume0

3. Reboot with "raid=autodetect md=1"

No difference, although it did come up with /dev/sda1 as its root (possibly coincidental change)

4. After a few more failed experiments with kernel command line options, I tried to put it back using "apt-get install dmraid". However after that, further boots failed - the kernel panicked because it couldn't find the root partition with the given UUID. I guess this means that neither dmraid nor mdadm was able to assemble the volume.

The screen shows:

md: Scanned 0 and added 0 devices.
md: autorun ..
md: ... autorun DONE.
VFS: Cannot open root device "UUID=..." or unknown-block(0,0) Please append a correct "root=" boot option; here are the available partitions (then shows sda, sda1, sdb, sdb1 and a backtrace)

Unfortunately I cannot scroll back to see previous lines. I *can* boot the system in a crippled way with root=/dev/sda1 though.

Back at the shell, it looks like the dmraid volume is still present:

# dmraid -r
/dev/sdb: isw, "isw_XXXXXXXXXX", GROUP, ok, 234441646 sectors, data@ 0
/dev/sda: isw, "isw_XXXXXXXXXX", GROUP, ok, 234441646 sectors, data@ 0
# dmraid -s
*** Group superset isw_XXXXXXXXXX
--> Subset
name   : isw_XXXXXXXXXX_Volume0
size   : 222716160
stride : 128
type   : mirror
status : ok
subsets: 0
devs   : 2
spares : 0
# ls /dev/mapper
control
#

The machine is probably toast anyway, as it has written to /dev/sdb1 and /dev/sda1 independently.

Anyway, I'm not so worried about having broken this machine, as it needed a reinstall anyway, but I do wonder what would have been the correct way to get mdraid instead of dmraid at boot time for this root volume?

Thanks,

Brian.

After some more searching, it looks like the udev rules were nobbled to disable this in
https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1030292

A possible way to re-enable is here:
https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1054948/comments/9

I'm a bit concerned about the issues around clean shutdown, and hence whether is really production-ready yet.

Regards,

Brian.

You may need

AUTO +imsm +1.x -all

or similar in your /etc/mdadm.conf

Afterwards you may also need to recreate your initramfs or whatever it is that Ubuntu uses. Boot devices are assembled in the initramfs (at least in Fedora), so if you still have the old initramfs sitting around, it will not see your changes.

Cheers,
Jes

In general, this works. I have seen it work with CentOS, Fedora, and various SUSE distributions. It may require some work on Ubuntu's side.

1 The udev rules for incremental mdadm autoassembly need to be in
place. The upstream rules should be fine. They can normally coexist with the rules for dmraid.
2 For shutdown, the distro must take care not to kill mdmon before file
systems are unmounted, and to run mdadm --wait-clean after any write access to file systems is finished.

Martin

Details of what I tried for ubuntu, and the results, are at
https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1054948/comments/16

Although it kind-of works, there are a lot of rough edges: most seriously, locking up during a shutdown. This isn't ready for production unfortunately.

Also, I tried booting Ubuntu 13.10beta2, and it is still using dmraid:
https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1054948/comments/17

Regards,

Brian.

FWIW, I found problems on stock CentOS 6.4.

I had created two RAID volumes within the BIOS:
"BOOT" (4GB)
"LVM" (rest of disk)

These are correctly detected and come up as md0 (container), md125 (BOOT), md126 (LVM). In a previous install using Debian I had set these up as ext4 and LVM respectively.

In the CentOS graphical installer, they are shown as:

V Hard Drives

    md125    4096    ext4
    md126    902246 vg   Physical volume (LVM)

(md0 is not shown)

Problem: if I double-click on md125, or select md125 and click Edit..., nothing happens. Therefore I cannot mark it as being used for the /boot filesystem.

However, if I double-click on md126, it correctly pops up "You cannot edit this drive: This device is part of the LVM volume group 'vg'."

Then if I delete all the logical volumes, and the volume group, I expected to be able to edit md126 - but I can't. Again, just nothing happens when I double-click on it. I can neither partition it, nor change it to ext4 and mount it.

So it looks like there's work remaining to make this usable in CentOS too.

Regards,

Brian.

Maybe, but your setup is pretty unusual. These MD arrays are "disks" for the installer, and thus would need to be partitioned. I believe it would work better that way. I have never tried an LVM PV on a whole disk.

Martin

It wouldn't let me partition md125 either. It just did nothing when I clicked "Edit".

I assume that that has something to do with the pre-exisiting PV. But that's just a guess. I have successfully installed CentOS on BIOS RAID on several occasions. In any case, what you describe is "just" an anaconda problem. Annoying of course, but you should get in touch with anaconda people to solve it.

Martin

There were two md volumes: one was an existing PV; one had been used an existing ext4 filesystem natively, and was neither partitioned nor a PV.

The one which was ext4 did nothing when I tried to partition it (by clicking the "Edit..." button). The one which was a PV gave an error when I tried to edit it (fair enough). But then after I removed all the LVs and the VG, clicking "Edit..." on that one no longer gave an error, but it just did nothing.

True enough.