Export initial ramdisk compression config
PATCH 2/2 Export initial ramdisk compression config
Hello,
While building the 3.11 kernel recently, I bumped into this issue.
Menuconfig allows one to choose compression format of an initial ramdisk image. But this selection does not result in creation of a duly compressed initramfs image. Because - $ make install - does not pass on the selected compression option to dracut(8) tool, which generates the initramfs file. This leads to a situation wherein the initramfs file is gzip(1) compressed, but the kernel knows only to decompress the user's chosen compression format.
The attached patch herein, attempts to fix this issue by exporting the compression configuration option to an environment variable of a sub-shell spawned by Make(1) with the invocation of installkernel(8) during the installation process.
The dracut(8) & new-kernel-pkg(8) tools need to be patched to recognise and read such an environment variable and use the selected compression algorithm while creating the initial ramdisk image files.
Could someone please review this one too?
Thank you.
--
Prasad J Pandit / Red Hat Security Response Team DB7A 84C5 D3F9 7CD1 B5EB C939 D048 7860 3655 602B
From 88878c7dfb04e69b75f1518fe809b917095a33b8 Mon Sep 17 00:00:00 2001
From: P J P <[email protected]>
Date: Sun, 15 Sep 2013 13:37:43 +0530
Subject: [PATCH 2/2] Export initial ramdisk compression config option
Make menuconfig allows one to choose compression format of an initial ramdisk image. But this choice does not result in duly compressed ramdisk image. Because - $ make install - does not pass on the selected compression choice to the dracut(8) tool, which creates the initramfs file. dracut(8) generates the image with the default compression, ie. gzip(1).
This patch exports the selected compression option to a sub-shell environment, so that it could be used by dracut(8) tool to generate appropriately compressed initramfs images.
There isn't a straight forward way to pass on options to dracut(8) via positional parameters. Because it is indirectly invoked at the end of a $ make install sequence.
# make install
-> arch/$arch/boot/Makefile
-> arch/$arch/boot/install.sh
-> /sbing/installkernel ...
-> /sbin/new-kernel-pkg ...
-> /sbin/dracut ...arch/x86/Makefile +17 -0
@@ -11,6 +11,23 @@ else KBUILD_DEFCONFIG := $(ARCH)_defconfig endif +# Select initial ramdisk compression format, default is gzip(1). +# This shall be used by the dracut(8) tool while creating an initramfs image. +# +INITRD_COMPRESS=gzip +ifeq ($(CONFIG_RD_BZIP2), y) + INITRD_COMPRESS=bzip2 +else ifeq ($(CONFIG_RD_LZMA), y) + INITRD_COMPRESS=lzma +else ifeq ($(CONFIG_RD_XZ), y) + INITRD_COMPRESS=xz +else ifeq ($(CONFIG_RD_LZO), y) + INITRD_COMPRESS=lzo +else ifeq ($(CONFIG_RD_LZ4), y) + INITRD_COMPRESS=lz4 +endif +export INITRD_COMPRESS + # BITS is used as extension for files which are available in a 32 bit # and a 64 bit version to simplify shared Makefiles. # e.g.: obj-y += foo_$(BITS).o
init/do_mounts_rd.c +5 -0
@@ -57,6 +57,11 @@ static int __init crd_load(int in_fd, int out_fd, decompress_fn deco);* cramfs * squashfs * gzip +* bzip2 +* lzma +* xz +* lzo +* lz4 */ static int __init identify_ramdisk_image (int fd,int start_block,decompress_fn *decompressor)
Signature · 1 line
1.8.3.1
Quoted 15 lines · On Sun, 15 Sep 2013 14:49:05 +0530 (IST) P J P <[email protected]> wrote:
Make menuconfig allows one to choose compression format of an initial ramdisk image. But this choice does not result in duly compressed ramdisk image. Because - $ make install - does not pass on the selected compression choice to the dracut(8) tool, which creates the initramfs file. dracut(8) generates the image with the default compression, ie. gzip(1).
This patch exports the selected compression option to a sub-shell environment, so that it could be used by dracut(8) tool to generate appropriately compressed initramfs images.
There isn't a straight forward way to pass on options to dracut(8) via positional parameters. Because it is indirectly invoked at the end of a $ make install sequence.
Quoted 7 lines
# make install
-> arch/$arch/boot/Makefile
-> arch/$arch/boot/install.sh
-> /sbing/installkernel ...
-> /sbin/new-kernel-pkg ...
-> /sbin/dracut ...It's a bit confusing whether all this appiles to initrd, to initramfs or to both. Can you please clarify all this and be sure that it's all consistent?
Quoted 12 lines
arch/x86/Makefile +6
-0@@ -11,6 +11,23 @@ else KBUILD_DEFCONFIG := $(ARCH)_defconfig endif +# Select initial ramdisk compression format, default is gzip(1). +# This shall be used by the dracut(8) tool while creating an initramfs image. +# +INITRD_COMPRESS=gzip +ifeq ($(CONFIG_RD_BZIP2), y) + INITRD_COMPRESS=bzip2
arch/x86/Makefile else · line 29
Quoted 10 lines
arch/x86/Makefile +10
-0+else ifeq ($(CONFIG_RD_LZMA), y) + INITRD_COMPRESS=lzma +else ifeq ($(CONFIG_RD_XZ), y) + INITRD_COMPRESS=xz +else ifeq ($(CONFIG_RD_LZO), y) + INITRD_COMPRESS=lzo +else ifeq ($(CONFIG_RD_LZ4), y) + INITRD_COMPRESS=lz4 +endif +export INITRD_COMPRESS
A few things...
- Why is it specific to x86? Other architcetures use initramfs?
- People add new compression schemes fairly regularly. We should add a code comment somewhere in a place where such people are sure to find it. That comment should explain what's going on, remind them to update this environment varlaible and should explain to them the process by which they get dracut(8) updated if needed.
- Can we avoid having to update dracut each time a new compression scheme is added? I assume your dracut changes will just exec "$INITRD_COMPRESS -d", so as long as any new decompression application uses the expected -d argument in the expected way, it should work seamlessly?
Hello Andrew,
Thank you so much for reviewing these patches.
+-- On Mon, 23 Sep 2013, Andrew Morton wrote --+ | It's a bit confusing whether all this appiles to initrd, to initramfs | or to both. Can you please clarify all this and be sure that it's all | consistent?
IIUC, we no longer use initrd block device images. Instead we use initramfs
which is a cpio(1) archive image. Both mkinitrd(8) & dracut(8) are invoked from /sbin/new-kernel-pkg tool. And mkinitrd(8) is nothing but a wrapper around dracut(8) these days.
===
$ man mkinitrd
mkinitrd - is a compat wrapper, which calls dracut to generate an
initramfs $ less /sbin/new-kernel-pkg # my local patched version
...
doMkinitrd() {
if [ -n "$dracut" ]; then
+ if [ -n "$INITRD_COMPRESS" ]; then
+ dracutcompress=--$INITRD_COMPRESS
+ fi
+ tool="dracut $dracuthostonly -f $dracutcompress $initrdfile $version"
else
tool="mkinitrd --allow-missing -f $initrdfile $version"
fi
[ -n "$verbose" ] && echo "creating initrd: $tool"
$tool
rc=$?
if [ $rc != 0 ]; then
echo "mkinitrd failed" >&2
exit 1
fi
}
===Because mkinitrd(8) does not support passing of compression argument to dracut(8), it must use the default compression, ie. gzip(1).
| A few things...
| - Why is it specific to x86? Other architcetures use initramfs?
No, it is not specific to x86, most likely all architecturess' Makefile
would need similar patch. But I haven't looked into those yet.
| - People add new compression schemes fairly regularly. We should add | a code comment somewhere in a place where such people are sure to | find it. That comment should explain what's going on, remind them to | update this environment varlaible and should explain to them the | process by which they get dracut(8) updated if needed.
True. Maybe user/Kconfig could have a note about updating INITRD_COMPRESS
environment variable through a relevant $arch/Makefile ?
Looking at the dracut(8) tool, it seems it can easily support any number of compression tools, for it already has options --gzip, --bzip2, --xz & --lzma. Adding --lzo, --lz4 etc won't be much difficult.
| - Can we avoid having to update dracut each time a new compression | scheme is added? I assume your dracut changes will just exec | "$INITRD_COMPRESS -d", so as long as any new decompression | application uses the expected -d argument in the expected way, it | should work seamlessly?
Yes, dracut(8) has | --compress=$INITRD_COMPRESS | option which could be
used to pass compression program to dracut(8). Not sure if dracut(8) expects an absolute path or just the program name. But to use this option, we'll need to patch /sbin/new-kernel-pkg tool.
Alternatively, dracut(8) could be patched to recognise and read $INITRD_COMPRESS variable and internally it can define precedence order between environment variable, command-line options and default fall-back option.
Thank you.
--
Prasad J Pandit / Red Hat Security Response Team DB7A 84C5 D3F9 7CD1 B5EB C939 D048 7860 3655 602B
Quoted 9 lines · P J P
Hello Andrew,
Thank you so much for reviewing these patches.
+-- On Mon, 23 Sep 2013, Andrew Morton wrote --+ | It's a bit confusing whether all this appiles to initrd, to initramfs | or to both. Can you please clarify all this and be sure that it's all | consistent?
Quoted 5 lines
IIUC, we no longer use initrd block device images. Instead we use
initramfs
which is a cpio(1) archive image. Both mkinitrd(8) & dracut(8) are invoked from/sbin/new-kernel-pkgtool. And mkinitrd(8) is nothing but a wrapper around dracut(8) these days.
$ find . -name "Makefile*" | xargs grep new-kernel-pkg $ grep -r new-kernel-pkg scripts $
Ah, so it's an out of tree bespoke Red Hat tool. No wonder I couldn't find it.
Quoted 4 lines
Looking at the dracut(8) tool, it seems it can easily support any number of compression tools, for it already has options --gzip, --bzip2, --xz & --lzma.
Adding --lzo, --lz4 etc won't be much difficult.
You're reimplemented the posix "pax" command?
Quoted 9 lines
| - Can we avoid having to update dracut each time a new compression | scheme is added? I assume your dracut changes will just exec | "$INITRD_COMPRESS -d", so as long as any new decompression | application uses the expected -d argument in the expected way, it | should work seamlessly?Yes, dracut(8) has | --compress=$INITRD_COMPRESS | option which
could be
used to pass compression program to dracut(8).
$ grep -r INITRD_COMPRESS linux $
Is this what you're currently doing, or the change you're requesting in the kernel?
Quoted 8 lines
Not sure if dracut(8) expects
an absolute path or just the program name. But to use this option, we'll need to patch/sbin/new-kernel-pkgtool.Alternatively, dracut(8) could be patched to recognise and read $INITRD_COMPRESS variable and internally it can define precedence order between environment variable, command-line options and default fall-back option.
How does your code get called?
Rob
+-- On Wed, 25 Sep 2013, Rob Landley wrote --+ | Ah, so it's an out of tree bespoke Red Hat tool. No wonder I couldn't find it.
It is not Red Hat tool.
| You're reimplemented the posix "pax" command?
Ummn, not sure. Didn't see anything about 'pax'.
| Is this what you're currently doing, or the change you're requesting in the | kernel?
Yes, I'm proposing to export $INITRD_COMPRESS variable so that it can be
used by dracut(8).
| How does your code get called?
The patch's commit log explains it.
Thank you.
--
Prasad J Pandit / Red Hat Security Response Team DB7A 84C5 D3F9 7CD1 B5EB C939 D048 7860 3655 602B
Quoted 5 lines · On Wed, 25 Sep 2013 01:11:54 +0530 (IST) P J P <[email protected]> wrote:
| A few things...
| - Why is it specific to x86? Other architcetures use initramfs?No, it is not specific to x86, most likely all architecturess'
Makefile
would need similar patch. But I haven't looked into those yet.
It would be better to make the change in one place, rather than for each architecture. That would appear to involve moving a hunk from arch/x86/Makefile into init/Makefile, or perhaps ./Makefile.
That way it can still be tested on x86, and it is very reasonable to assume that it will work as intended on other architectures.
Hello Andrew,
+-- On Wed, 9 Oct 2013, Andrew Morton wrote --+ | It would be better to make the change in one place, rather than for each | architecture. That would appear to involve moving a hunk from | arch/x86/Makefile into init/Makefile, or perhaps ./Makefile.
Right, I was trying to figure the same. I'll do the changes and send an
updated patch.
Thank you so much for reviewing it and the update.
Thank you.
--
Prasad J Pandit / Red Hat Security Response Team
Hi,
+-- On Wed, 9 Oct 2013, Andrew Morton wrote --+ | It would be better to make the change in one place, rather than for | each architecture. That would appear to involve moving a hunk from | arch/x86/Makefile into init/Makefile, or perhaps ./Makefile.
Please find attached herein, an updated patch which adds the export snippet
to the top level ./Makefile. I tried init/Makefile too, but it does not seem to work.
Thank you!
--
Prasad J Pandit / Red Hat Security Response Team
From 2bb603f666371289ee660be5cf7a6f87740edd8e Mon Sep 17 00:00:00 2001
From: P J P <[email protected]>
Date: Thu, 10 Oct 2013 19:53:24 +0530
Subject: Export initial ramdisk compression config option
Make menuconfig allows one to choose compression format of an initial ramdisk image. But this choice does not result in duly compressed ramdisk image. Because - $ make install - does not pass on the selected compression choice to the dracut(8) tool, which creates the initramfs file. dracut(8) generates the image with the default compression, ie. gzip(1).
This patch exports the selected compression option to a sub-shell environment, so that it could be used by dracut(8) tool to generate appropriately compressed initramfs images.
There isn't a straight forward way to pass on options to dracut(8) via positional parameters. Because it is indirectly invoked at the end of a $ make install sequence.
# make install
-> arch/$arch/boot/Makefile
-> arch/$arch/boot/install.sh
-> /sbing/installkernel ...
-> /sbin/new-kernel-pkg ...
-> /sbin/dracut ...Makefile +16 -0
@@ -720,6 +720,22 @@ mod_strip_cmd = true endif # INSTALL_MOD_STRIP export mod_strip_cmd +# Select initial ramdisk compression format, default is gzip(1). +# This shall be used by the dracut(8) tool while creating an initramfs image. +# +INITRD_COMPRESS=gzip +ifeq ($(CONFIG_RD_BZIP2), y) + INITRD_COMPRESS=bzip2 +else ifeq ($(CONFIG_RD_LZMA), y) + INITRD_COMPRESS=lzma +else ifeq ($(CONFIG_RD_XZ), y) + INITRD_COMPRESS=xz +else ifeq ($(CONFIG_RD_LZO), y) + INITRD_COMPRESS=lzo +else ifeq ($(CONFIG_RD_LZ4), y) + INITRD_COMPRESS=lz4 +endif +export INITRD_COMPRESS ifdef CONFIG_MODULE_SIG_ALL MODSECKEY = ./signing_key.priv
Signature · 1 line
1.8.3.1
Hello Andrew,
+-- On Wed, 9 Oct 2013, Andrew Morton wrote --+ | It would be better to make the change in one place, rather than for | each architecture. That would appear to involve moving a hunk from | arch/x86/Makefile into init/Makefile, or perhaps ./Makefile.
Please see this revised version, I missed to include init/do_mounts_rd.c
changes in the previous one.
Thank you.
--
Prasad J Pandit / Red Hat Security Response Team
From 14dccee98f12f2dbca224510bcfedd1a397ed177 Mon Sep 17 00:00:00 2001
From: P J P <[email protected]>
Date: Thu, 10 Oct 2013 20:37:48 +0530
Subject: Export initial ramdisk compression config option
Make menuconfig allows one to choose compression format of an initial ramdisk image. But this choice does not result in duly compressed ramdisk image. Because - $ make install - does not pass on the selected compression choice to the dracut(8) tool, which creates the initramfs file. dracut(8) generates the image with the default compression, ie. gzip(1).
This patch exports the selected compression option to a sub-shell environment, so that it could be used by dracut(8) tool to generate appropriately compressed initramfs images.
There isn't a straight forward way to pass on options to dracut(8) via positional parameters. Because it is indirectly invoked at the end of a $ make install sequence.
# make install
-> arch/$arch/boot/Makefile
-> arch/$arch/boot/install.sh
-> /sbing/installkernel ...
-> /sbin/new-kernel-pkg ...
-> /sbin/dracut ...Makefile +16 -0
@@ -720,6 +720,22 @@ mod_strip_cmd = true endif # INSTALL_MOD_STRIP export mod_strip_cmd +# Select initial ramdisk compression format, default is gzip(1). +# This shall be used by the dracut(8) tool while creating an initramfs image. +# +INITRD_COMPRESS=gzip +ifeq ($(CONFIG_RD_BZIP2), y) + INITRD_COMPRESS=bzip2 +else ifeq ($(CONFIG_RD_LZMA), y) + INITRD_COMPRESS=lzma +else ifeq ($(CONFIG_RD_XZ), y) + INITRD_COMPRESS=xz +else ifeq ($(CONFIG_RD_LZO), y) + INITRD_COMPRESS=lzo +else ifeq ($(CONFIG_RD_LZ4), y) + INITRD_COMPRESS=lz4 +endif +export INITRD_COMPRESS ifdef CONFIG_MODULE_SIG_ALL MODSECKEY = ./signing_key.priv
init/do_mounts_rd.c +5 -0
@@ -57,6 +57,11 @@ static int __init crd_load(int in_fd, int out_fd, decompress_fn deco);* cramfs * squashfs * gzip +* bzip2 +* lzma +* xz +* lzo +* lz4 */ static int __init identify_ramdisk_image (int fd,int start_block,decompress_fn *decompressor)
Signature · 1 line
1.8.3.1
Hello Andrew,
I was wondering if you had a chance to review this patch further? Should I send similar patches for other architectures too? As in you aren't waiting on me for that, are you?
Thank you.
--
Prasad J Pandit / Red Hat Security Response Team DB7A 84C5 D3F9 7CD1 B5EB C939 D048 7860 3655 602B
Hello Andrew,
Just to check, did you have a chance to review it further? Are you waiting on me for something?
--
Prasad J Pandit / Red Hat Security Response Team
DB7A 84C5 D3F9 7CD1 B5EB C939 D048 7860 3655 602B