Odd booting problem

Good day all, and Happy Thanksgiving for those in Canada!

An odd problem, and hopefully someone reading this can tell me what to do differently:

I am converting a machine from a single disk to one with 2 disks and 4 md RAIDs.

I do not know how much detail people might like, so I will keep it as terse as possible to start:
Original install CentOS 6.0, with all updates.
XC86_64 kernel

mdadm v3.2.2

I have added the second physical disk, and duplicated the partition table from the existing system disk created 4 md RAID1s, to be used for /boot, /, /var, and /home
All were created with the second part missing:
mdadm --create /dev/md0 --metadata=0.90 --level=1 --raid-disks=2 /dev/sda1 missing mdadm --create /dev/md1 --level=1 --raid-disks=2 /dev/sda2 missing
mdadm --create /dev/md2 --level=1 --raid-disks=2 /dev/sda5 missing
mdadm --create /dev/md3 --level=1 --raid-disks=2 /dev/sda6 missing

System is booting on GRUB 0.97.
Therefore I made the md0 ( /boot) with 0.90
And then made md1, md2 and md3 with 1.2

Everything looks good in /proc/mdstat
I made file systems ( ext for boot and ext for others) , then copied over all data from original disk.

Created the mdadm.conf file:
mdadm --detail --scan > /etc/mdadm.conf

I used dracut to rebuild the initramfs with the new mdadm.conf:

dracut --mdadmconf --force /boot/initramfs-$(uname -r).img $(uname -r) This completed uneventfully.

Edited /etc/fstab and rebooted.
Now I have all my filesystems on the md devices with no complaints.

Ran grub and installed on the new disk :
root (hd0,0)
setup (hd0)
Completed successfully

Next, rebooted again, and this time manually edited the grub command line to change from root=/dev/sdb2 to root=/dev/md1

And here it gets funny:

Boots a bit, then I see:

dracut Warning: No root device "block: /dev/md1" found.

And a few seconds later we get a kernel panic.

Cheers,
Maurice Hilarius
eMail: /[email protected]/

Good morning,

I am top and bottom posting this reply to myself, so as to satisfy the tastes of each camp:

I just tried using only one of the disks, and NOT as md RAID devices.
This works perfectly.
Which, at least to my eyes, implies there is some interaction with mdadm that is causing the failure of dracut and of booting.

Is there anyone here interested in helping me with this?

Good morning,

I am top and bottom posting this reply to myself, so as to satisfy the tastes of each camp:

I just tried using only one of the disks, and NOT as md RAID devices.
This works perfectly.
Which, at least to my eyes, implies there is some interaction with mdadm that is causing the failure of dracut and of booting.

Is there anyone here interested in helping me with this?

Cheers,
Maurice Hilarius
eMail: /[email protected]/

Maurice-

I am not an expert, but I have observed that initrd/initramfs/dracut is fairly stupid, and can't handle what would seem like benign changes to the operating evironment. For example, if you install a linux kernel on a system where the / filesystem is on an LVM volume, then you move the / filesystem to a raw partition, change the grub boot commands and /etc/fstab appropriately, initrd/initramfs still fails with an "/bin/lvm exited abnormally with error 5" message. It does not seem to me that there is any reason that initrd/initramfs should care what return code lvm returns as long as it can mount the / filesystem. It also messes about with raids, generally unnecessarily.

Again, initrd /initramfsdoes not allow for many changes in the system configurated from the state when the initrd/initramfs was created. My guess is that you are going to have to create a new initrd/initramfs. Here is the procedure at a high level, as best I can recall from the last time I had to do it

1) Boot the computer with some recent linux kernel. For example, use Knoppix or Fedora live.

2) Mount your target root filesystem and mount the target boot filesystem under it, if you use a separate filesystem for boot.

3) chroot to the target root filesystem.

4) Run mkinitrd to create a new initrd/initramfs. mkinitrd has a zillion options that I do not pretend to understand. I generally find that it works well enough with default options.

5) If needed, update your grub.conf to point to the new initrd/initramfs.

Good luck!

Jim

Thanks Jim,

I will try some of your suggestions.

Cheers,
Maurice Hilarius
eMail: /[email protected]/

Well, dracut (currently, I have some big changes staged for submission, but for a bit later) will do forced assembly (if no root can be found after a few udev loops), so technically degraded arrays you created are not likely the reason.

If I'm to guess the cause, it's probably naming - modern mdadm /will not/ export device names to the file, and leaves that for udev - and by default udev will use decreasing minor numbers, starting with 127. Don't expect the same (minor number ~ device) relations during next bootings either.

Try this - get the UUID of actual filesystem, and change the boot commandline to something with root=UUID=<uuid>

You can get the uuid with e.g.

blkid -o udev <device>

and use the ID_FS_UUID_ENC value. Or peek into /dev/disk/by-uuid and get the value from there. You can also use LABEL, and quite a few other alternatives. dracut is pretty felxible.

Another possibility, is to create (or update) the array's name, and use /dev/md/<name> for root= which should work just fine as well (with sane udev rules and modern mdadm, /dev/md/ should contain array specific name symlinked to proper /dev entry ).

If the above fails, then there's some other reason lurking.

Btw, what version of dracut are you using ? Anything relatively current should drop you into emergency shell, not end with kernel panic.

Btw2: stock md rules (3.2.x) will try to assemble anything as udev goes. If you want to limit the assembly to something specific, use mdadm.conf with:

AUTO -all
ARRAY ....

see mdadm.conf(5) for details.

Problem Solved!

Thanks VERY much to Michal Soltys who offered me lots of useful suggestions.

Also thanks to Jim Schatzman who chipped in with some good ideas as well.

In the end, while I tried those suggestions, it did not solve the problem. I did a lot of reading and some experiments and here is what turned out to be the problem:

Dracut does the following:

+ [ -d /sysroot/proc ]
+ . /mount/99mount-root.sh
...
+ [ -n block:/dev/md1 -a -z ]
+ mount -t auto -o ro /dev/md1 /sysroot
+ ROOTFS_MOUNTED=yes

So far so good.

BUT: as I found, and for unclear reasons, there was no 'proc' on the file system present on /dev/md1 (which was intended as our new /).

So dracut gets into a loop despite of ROOTFS_MOUNTED being already set:

+ [ -d /sysroot/proc ]
+ . /mount/99mount-root.sh
...
+ [ -n block:/dev/md1 -a -z ]
+ mount -t auto -o ro /dev/md1 /sysroot
mount: /dev/md1 already mounted or /sysroot busy
mount: according to mtab, /dev/md1 is already mounted on /sysroot

That is absolutely true but not particularly illuminating. Eventually this ends, quite late,

+ i=21
+ [ 21 -gt 20 ]
+ flock -s 9
+ emergency_shell Can't mount root filesystem

which is not particularly informative.

Bailing out quite a bit earlier, before everything scrolled off of the screen with something like:
"Cannot find /proc on a canditate for a root filesystem; that cannot be right."
would be quite a bit more helpful.

Alternatively if ROOTFS_MOUNTED is "yes" and that filesystem looks like a reasonable candidate for /
(say 'etc/issue' and 'sbin' present)
Then why not offer to run 'mkdir -p /sysroot/{dev,proc,sys}'? Do this after making sysroot writeable and then continue?

That is possibly going too far but a notification with some useful
information
pointing to an error would be much better than "Can't mount".

Cheers,
Maurice Hilarius
eMail: /[email protected]/

For the record, the loop you're talking about sources mount hooks of each module that installs one. And those hooks are internally responsible for mounting. The "success" check is, if /proc directory showed up at /sysroot point. So yea, for certain unusual situations it could be more precise/verbose. I'll prep a patch for that.

Btw, the version you reported offlist, is relatively old (current is 013 with quite a few commits ahead). FYI

IMHO the init script itself should keep its hands off the filesystem.

It's prefectly fine though to create a module that would do that through e.g. a pre-pivot hook - but that's distro's/user's policy, or module's specific necessity.

PS. If you run into other dracut-related issues, initramfs@vger will likely get you answer faster.