2.6 upgrade - LVM version ? & root partition access problems...

I wanted to upgrade to a 2.6 kernel, but I seem to be running into problems getting the new kernel to recognize my LVM root '/' partition during boot. I also think I have a mixed LVM1 and LVM2 toolset as part of the upgrade and that this is why I'm failing. Now, I'm even more concerned with the stability of my system than with the original 2.6 upgrade task.

Sorry for the length of the post, but I think my main problem is that I don't know what it is that I don't know... Makes it damn hard to Google or RTFM otherwise. I'll start with the most obvious and move into the more obscure. Stop if you get bored or see my problem! :-D

I'm running on RH9 with what used to be a pretty standard install except for a EXT3 LVM root partition and a ReiserFS Software Raid1 /home partition. (OK so a LVM '/' partition is a bad idea. I didn't know that then. I do now!)

I can't boot with my newly compiled 2.6.5 kernel, lvmcreate_initrd doesn't work for either 2.4 or 2.6 kernels with my current setup and tools, and "lvm version" gives an error (see below)

In one of the many guides about 2.6 upgrading I was suggested to upgrade to the *.i386.rpms in http://people.redhat.com/arjanv/2.6/RPMS.kernel/
(The URL seemed to indicate a reputable source for RH9 rpms) Among others, I installed
lvm-1.0.3-17.i386.rpm
lvm2-2.00.08-2.i386.rpm
device-mapper-1.00.07-3.1.i386.rpm

Now I'm not even sure what LVM version I'm running:
[root@valde root]# lvm version
LVM version: 2.00.08 (2003-11-14)
Library version: 1.00.07-ioctl (2003-11-21) /dev/mapper/control: open failed: No such file or directory Is device-mapper driver missing from kernel?

[root@valde root]# lvm lvs
/proc/lvm/VGs/Volume00 exists: Is the original LVM driver using this
volume group?
Can't lock Volume00: skipping

[root@valde new]# lvm pvscan

  PV /dev/hda3   VG Volume00   lvm1 [56.22 GB / 0    free]
  PV /dev/hdc3   VG Volume00   lvm1 [55.83 GB / 4.00 MB free]

Total: 2 [4.00 MB] / in use: 2 [4.00 MB] / in no VG: 0 [0 ]

man lvm mentions /etc/lvm/lvm.conf. I have a /etc/lvm directory but no /etc/lvm/lvm.conf file.

In the output, I'm asked:
"Is device-mapper driver missing from kernel?".
I don't know. Is it? I'm running a stock 2.4.20-20.9 kernel from redhat with the non-stock bridge-nf patch. If device-mapper driver is missing from kernel what do I do about that?

FAQ entry: "I get errors about /dev/mapper/control when I try to use the LVM 2 tools" from http://www.tldp.org/HOWTO/LVM-HOWTO/lvm2faq.html#AEN231
says "The primary cause of this is not having run the devmap_mknod.sh script after rebooting into a dm capable kernel."
True. I haven't run devmap_mknod.sh. In fact I don't have it on my system. It isn't part of any of my lvm* rpms. But am I running "a dm capable kernel" yet with my RedHat 2.4.20-20.9 kernel?

It seems I'm now in a chicken-and-egg situation. I can't get lvmcreate_initrd to work until I update the kernel to a dm kernel. I can't boot into one until I have lvmcreate_initrd (or mkinitrd) working.
Hmm...

So: Should I
*) Get, compile, install and run "a dm capable kernel." Which one? *) Accept that installing the lvm2 rpm was a bad idea and install lvm2 from tarball to get and run devmap_mknod.sh? Anything to watch out for here? *) Upgrade my /dev/Volume00/LogVol00 to LVM2 somehow?
*) Turn my '/' into a non-LVM volume? How?
*) Backup all my 90 GB somewhere, reinstall with a fresh 2.6 kernel distro and get all my apps & setup working again from scratch? (YIKES!) *) Some other option I'm missing?

***************************
Boot disk creation?
***************************
I also don't have a good boot disk around with the LVM access. How do I create one, now that my tools are messed up but before I loose my system forever? Once I have it, how do I test that its working and not accessing the lvm tools from the / partition in some sneaky fashion? Are all the LVM access tools already in /boot/initrd-2.4.20-20.9.img (It does contain e.g. /bin/vgchange)? Am I fine as long as I leave an entry for this 2.4.20-20.9 kernel in grub.conf?

***************************
lvmcreate_initrd
seems to not work for me
***************************
OK, so then I thought I'd just put MD, RAID1, EXT3 and REISERFS into the kernel instead of as modules and follow the
http://www.tldp.org/HOWTO/LVM-HOWTO/
on how to run "lvmcreate_initrd 2.6.5" to create the initrd-lvm file. I have not yet succeeded in running lvmcreate_initrd successfully. I always get approx 2000 "cpio: No space left on device" errors that finish with
lvmcreate_initrd -- ERROR cpio to ram disk
(Because of messed up lvm tools?)

Even with a fresh 2.4.20 kernel build with the above modules directly in the kernel (that works fine with all the above as modules using mkinitrd) I can't get lvmcreate_initrd to work. When running lvmcreate_initrd with a parameter indicating a 2.4 kernel (other than the one currently running), I get the same many may "No space left on device" errors in addition to "depmod: *** Unresolved symbols in /lib/modules/2.4.<snip>" errors. It also fails with "ERROR cpio to ram disk".

Going to another machine, running Fedora Core 1 and 2.4.22 with a gazillion patches (for "myth"), I get the same "depmod: *** Unresolved symbols in /lib/modules/2.4.<snip>" errors, but it does finish by creating /boot/initrd-lvm-2.4.22-1.2115.nptl.gz. Are the depmod errors normal and to be expected as part of normal behavior?

***************************
mkinitrd woes with
device-mapper-1.00.07-3.1.i386.rpm
***************************
Among others, I installed device-mapper-1.00.07-3.1.i386.rpm from the above location. With it, mkinitrd (also "new" & from above) doesn't work. Removing it and reinstating the original /lib/libdevmapper.so.1.00 makes mkinitrd work, but "lvm version" and "lvm lvs" doesn't work with either of them. (Even after rebooting after changing it).

***************************
Running my 2.6.5 kernel
anyway
***************************
If I ignore all of this, and create a 2.6.5 kernel with RAID1, EXT3, LVM and REISERFS as modules, and mkinitrd, then when I try to boot it I get:

RAMDISK: Compressed image found at block 0
RAMDISK: incomplete write (-28 != 32768) 4194304
VFS: Cannot open root device "Volume00/LogVol00" or unknown-block(0,0) Please append a correct "root=" boot option

Kernel panic: VFS: Unable to mount root fs on unknown-block(0,0

Help! I don't know what to do now!

Peter

Peter Valdemar M�rch
http://www.morch.com

It's not necessarily a bad idea, it just makes it a lot more tricky in cases like this.

Someone made a new initrd creation script and posted it here:
http://www.poochiereds.net/svn/lvm2/
I can't guarentee it'll work, but i've heard good things about it. Use it against the 2.6.5 kernel you compiled.

You're using lvm version 2 - however, can you run 'pvscan -V' without the 'lvm' prefix? If so, you have both tools available to you, and running 'vgchange -ay' again, without the 'lvm' prefix should work just fine for you in your 2.4 kernel. If you're booted into your 2.4 kernel, it sounds like you still have the lvm1 tools and are just fine (lvm1 did *not* have the 'lvm' command)

Nope, device-mapper is *not* in that kernel. You either need to upgrade to a 2.6 kernel, or use the patches that are found in the lvm2 tarball to patch a 2.4 kernel to contain device-mapper.

If that initrd creation script works for you, you may not need to run the devmap_mknod.sh script. However, what the script does is looks in /proc/misc for a '$some-number device-mapper' entry, and creates /dev/mapper/control with 'mknod /dev/mapper/control c 10 $some-number', so you can do that by hand if you like.

2.6.5 will be fine as long as you don't need snapshots or pvmove. If you do need those capabilities right now, grab the lvm2 tarball and a 2.4.26 kernel, and apply the device-mapper kernel patches from the tarball. You don't need to use the tarball to install the tools

The lvm2 rpm should be fine, you just need to make sure /dev/mapper/control is created.

Don't need to do this

Scary. It's possible, but wait until the above options don't work before trying it.

Shouldn't need to

Forget this for the moment, let's get the system stabilized first.

This does not work for lvm2.

Yep, need that lvm2-capable initrd.

Well, I've heard good things about the initrd creation script i mentioned at the beginning, so maybe you could give that a shot first with the 2.6.5 kernel?

Regards,

AJ Lewis                                   Voice:  612-638-0500
Red Hat Inc.                               E-Mail: [email protected]
720 Washington Ave. SE, Suite 200
Minneapolis, MN 55414

Current GPG fingerprint = FE77 4B43 6A9B F982 A731  02FA 2BF5 7574 294A AA5A
Grab the key at: http://people.redhat.com/alewis/gpg.html or one of the
many keyservers out there...
-----Begin Obligatory Humorous Quote----------------------------------------
Mountain Dew... because breakfast is the most important meal of the day.
-----End Obligatory Humorous Quote------------------------------------------

OK, so now I've tried to use that. After dealing with a missing /lib/lvm-200/lvm (mkdir /lib/lvm-200; ln -s /sbin/lvm /lib/lvm-200/lvm)
and a missing /bin/busybox (cp /sbin/busybox /bin) I now have a
/boot/initrd-lvm2-2.6.5.gz!!!! This is progress!!!!

During boot of my freshly compiled 2.6.5 kernel+initrd, I get this message on the console:

RAMDISK: Compressed image found at block 0
VFS: Mounted root (ext2 filesystem) readonly. Freeing unused kernel memory: 236k freed Warning unable to open an initial console.

Kernel panic: Attempted to kill init!

At this point all I can do is hit the reset button and reboot in good ol' 2.4.20. What do I need to do now?

It prints out a lot more than this, but I don't know how to capture it. (do you?) Also, it prints more than one screen, but I don't know how to scroll up, and the previous screens scroll by very fast... (E.g. it detects my RAID1 volumes apparently - that fills most of the screen I can see)

[root@valde linux-2.6.5]# grep -i console .config # CONFIG_NETCONSOLE is not set CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_LP_CONSOLE=y
# Console display driver support CONFIG_VGA_CONSOLE=y
CONFIG_MDA_CONSOLE=m CONFIG_DUMMY_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE is not set

Uhm, what is the console that it can't open, and how do I fix that? Is this still LVM related? Or is it more of a standard 2.6.5 thing?

***********************************************************************
grub.conf entry
***********************************************************************
lvm2create_initrd.sh suggests that images should be relative to /boot/ but I could only get this far with images relative to / as in:

title 2.6.5

        kernel /vmlinuz-lvm2-2.6.5 root=/dev/ram0 
lvm2root=/dev/Volume00/LogVol00
        initrd /initrd-lvm2-2.6.5.gz

Does the missing '/boot/' prefix surprise anybody?

***********************************************************************
Maybe the following is relevant (I think not), but it seems to have to do with consoles...
***********************************************************************

During running lvm2create_initrd.sh, it prints out this:

lvm2create_initrd.sh -- adding required /dev files don't know how to make device "consoleonly"

Well, neither do I! A little snippet from my console:

[root@valde root]# grep BASICDEV /sbin/lvm2create_initrd.sh BASICDEVICES="std consoleonly fd"
verbose "BASICDEVICES: echo $BASICDEVICES"
(cd $TMPMNT/dev; /dev/MAKEDEV $OPT_Q $BASICDEVICES $BLOCKDEVICES)

So basically /sbin/lvm2create_initrd.sh has hardcoded the fact that it should:
/dev/MAKEDEV std consoleonly
among other params. But *my* MAKEDEV doesn't know how to make a consoleonly:

[root@valde root]# /dev/MAKEDEV std [root@valde root]# /dev/MAKEDEV consoleonly don't know how to make device "consoleonly" [root@valde root]# rpm -qf /dev/MAKEDEV MAKEDEV-3.3.2-5

Peter Valdemar M�rch
http://www.XXXXXXXXX

Well, I did finally did get it to work. Quite a strenuous road, though. For others that may google later, here are some of the things I ran into, and their solutions:

1) There was no console. The script created /dev/consoleonly, but I had & needed /dev/console.

2) "No space left on device" The initrd ran out of inodes during creation. Presumably because MKDEV on my system creates very, very many items for /dev (2601 /dev/tty* and 1728 /dev/sd* for instance)

3) MKDEV takes a -d option on my RH9 system telling it where to put the files. Instead, the script cd-ed to the directory and expected them to be put in ./

4) fsck.ext3 failed during boot. I discovered that adding " fastboot" as a parameter to the kernel allowed me to get past that point and still end up in a 2.6.5 kernel, so I could run "lvm vgmknodes" that apparently created what I needed, so that I could get past the fsck hurdle, even without " fastboot". (Took me a while to figure *that* one out! :-D)

I've created a new version (incl. svn diff) of the script here:
http://wwww.morch.com/postings/lvm2/

Sincerely,

Peter

Peter Valdemar M�rch
http://www.morch.com

Bah - I guess I should have pointed you at the fedora mkinitrd - it might have saved you the hassle. :(
http://distro.ibiblio.org/pub/linux/distributions/fedora/linux/core/development/i386/Fedora/RPMS/mkinitrd-3.5.21-1.i386.rpm
is the current version AFAIK. I think this will work better with RH9 systems than the other one (which was originally built around debian)

K - so if the fedora mkinitrd doesn't work for people using RH9, they can try that one. :) Thanks!

AJ Lewis                                   Voice:  612-638-0500
Red Hat Inc.                               E-Mail: [email protected]
720 Washington Ave. SE, Suite 200
Minneapolis, MN 55414

Current GPG fingerprint = FE77 4B43 6A9B F982 A731  02FA 2BF5 7574 294A AA5A
Grab the key at: http://people.redhat.com/alewis/gpg.html or one of the
many keyservers out there...
-----Begin Obligatory Humorous Quote----------------------------------------
APATHY ERROR: Don't bother striking any key.
-----End Obligatory Humorous Quote------------------------------------------

Possibly. I can't get it to work with either 2.6.5 or my 2.4 image, though:

valde@peter:~> rpm -q mkinitrd mkinitrd-3.5.21-1
valde@peter:~> sudo mkinitrd ost.img 2.6.5
No module raid1 found for kernel 2.6.5, aborting. valde@peter:~> sudo mkinitrd ost.img 2.4.20-20.9custom pvscan -- LVM driver/module not loaded?

(While I'm running in a 2.6 kernel. module raid1 is not not found for 2.6.5 because I've built it into the kernel now. I can't be bothered to test it as a module at this point)

Wups, what happened with the extra 'w' (4 in all)? It does work, though...

To make as few changes as possible, RH9 users will have to edit it and uncomment these lines

INITRDSIZE=4096
INITRDINODES=5000

and run it with the -d option, such as in

# lvm2create_initrd.sh -d 2.6.5

Also, I forgot to mention something weird. If I attempt to boot my old 2.4 kernel, fsck complains. Booting with kernel option " fastboot" gets me past the fsck point, all the way to the final login screen. However, in the 2.4 kernel, I no longer have keyboard or mouse support. But I can ssh into it from outside. Rebooting again into 2.6 also stops at fsck, so the " fastboot" is required *again*. When running in 2.6, I then run "lvm vgmknodes" and then it can boot in 2.6 fsck and all - no problemo. But it seems I've passed the point of no-return-to-2.4.

Later,

Peter

Peter Valdemar M�rch
http://www.XXXXXXXXX

Sorry for the late post. I've hit this too when setting up a new box. The problem here is that the kernel or init (?) requires a /dev/console device very early in the boot process, before udev gets mounted.

This means that even if you're using udev (and maybe devfs) then you still need some static entries in /dev that will be overlayed by the udev ramdisk. Obviously, this is a difficult problem to correct in the mkinitrd script, since you'd have to unmount /dev to make sure those entries were present.

I'll prob. add something about this to the README sometime, when I get a chance to work on this script...

-- Jeff