fs/cpio: add option to manually specify file list
cover fs/cpio: add option to manually specify file list
Hello All,
Again, in my use case I require an initramfs. However my system is memory restricted so I cannot use my full rootfs image. This series contains the modifications that I made to support this. I am not sure if this is generally useful to the larger buildroot community, but I figured since Arnout and Peter were kind enough to give me pointers I should share the changes I made in case someone else can benefit from them.
-Seamus
PATCH 1/1 fs/cpio: add option to manually specify file list
For some systems it may be desireable to have an initramfs that is smaller than the full root filesystem. This commit adds support for creating an initramfs based on a manually specified set of files. It is up to the user to define a sensible set of files. This file list could either be statically created, or it could be created as the result of a pre-fs hook script.
--- fs/cpio/Config.in | 30 ++++++++++++++++++++++++++++++ fs/cpio/cpio.mk | 10 ++++++++++ 2 files changed, 40 insertions(+)
fs/cpio/Config.in +30 -0
@@ -7,6 +7,36 @@ config BR2_TARGET_ROOTFS_CPIO if BR2_TARGET_ROOTFS_CPIO +choice + prompt "Seleted Fileset" + default BR2_TARGET_ROOTFS_CPIO_INCLUDE_ALL + help + Select the set of files to include in the cpio of the root filesystem. + This may either be all files installed to the target, or it may be a + subset defined in the custom file list. + +config BR2_TARGET_ROOTFS_CPIO_INCLUDE_ALL + bool "Include all files from target" + help + Include all files from the target directory in the root filesystem. + +config BR2_TARGET_ROOTFS_CPIO_INCLUDE_CUSTOM + bool "Include a subset of files from target" + help + Include a subset of the files from the target directory in the root filesystem. + This requires a custom file list to be defined. The actual file list may be + generated by a user script defined as part of the pre-fs hook. + +endchoice + +config BR2_TARGET_ROOTFS_CPIO_CUSTOM_FILELIST + string "path to a custom file list" + depends on BR2_TARGET_ROOTFS_CPIO && BR2_TARGET_ROOTFS_CPIO_INCLUDE_CUSTOM + help + This file defines the contents of the rootfilesystem. All files and folders + created in the rootfilesystem must be defined here. This file can be statically + defined, or created as part of the pre-fs hook. + choice prompt "Compression method" default BR2_TARGET_ROOTFS_CPIO_NONE
fs/cpio/cpio.mk +10 -0
@@ -27,10 +27,20 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT +ifeq ($(BR2_TARGET_ROOTFS_CPIO_INCLUDE_ALL),y) + define ROOTFS_CPIO_CMD cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@ endef +else + +define ROOTFS_CPIO_CMD + cd $(TARGET_DIR) && cpio --quiet -o -H newc < $(BR2_TARGET_ROOTFS_CPIO_CUSTOM_FILELIST) > $@ +endef + +endif + $(BINARIES_DIR)/rootfs.cpio.uboot: $(BINARIES_DIR)/rootfs.cpio host-uboot-tools $(MKIMAGE) -A $(MKIMAGE_ARCH) -T ramdisk \ -C none -d $<$(ROOTFS_CPIO_COMPRESS_EXT) $@
Signature · 1 line
2.14.2
PATCH 1/1 fs/cpio: add option to manually specify file list
I was thinking that a good improvement to this patch would be to specify the file list in a gitignore format.? That way you could just chop off the portions of the rootfs you dont need, or manually include the portions that you do need.
Quoted 79 lines · Seamus Connor
For some systems it may be desireable to have an initramfs that is smaller than the full root filesystem. This commit adds support for creating an initramfs based on a manually specified set of files. It is up to the user to define a sensible set of files. This file list could either be statically created, or it could be created as the result of a pre-fs hook script.
Signed-off-by Seamus Connor <[email protected]>--- fs/cpio/Config.in | 30 ++++++++++++++++++++++++++++++ fs/cpio/cpio.mk | 10 ++++++++++ 2 files changed, 40 insertions(+)fs/cpio/Config.in +30
-0@@ -7,6 +7,36 @@ config BR2_TARGET_ROOTFS_CPIO if BR2_TARGET_ROOTFS_CPIO +choice + prompt "Seleted Fileset" + default BR2_TARGET_ROOTFS_CPIO_INCLUDE_ALL + help + Select the set of files to include in the cpio of the root filesystem. + This may either be all files installed to the target, or it may be a + subset defined in the custom file list. + +config BR2_TARGET_ROOTFS_CPIO_INCLUDE_ALL + bool "Include all files from target" + help + Include all files from the target directory in the root filesystem. + +config BR2_TARGET_ROOTFS_CPIO_INCLUDE_CUSTOM + bool "Include a subset of files from target" + help + Include a subset of the files from the target directory in the root filesystem. + This requires a custom file list to be defined. The actual file list may be + generated by a user script defined as part of the pre-fs hook. + +endchoice + +config BR2_TARGET_ROOTFS_CPIO_CUSTOM_FILELIST + string "path to a custom file list" + depends on BR2_TARGET_ROOTFS_CPIO && BR2_TARGET_ROOTFS_CPIO_INCLUDE_CUSTOM + help + This file defines the contents of the rootfilesystem. All files and folders + created in the rootfilesystem must be defined here. This file can be statically + defined, or created as part of the pre-fs hook. + choice prompt "Compression method" default BR2_TARGET_ROOTFS_CPIO_NONEfs/cpio/cpio.mk +10
-0@@ -27,10 +27,20 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT +ifeq ($(BR2_TARGET_ROOTFS_CPIO_INCLUDE_ALL),y) + define ROOTFS_CPIO_CMD cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@ endef +else + +define ROOTFS_CPIO_CMD + cd $(TARGET_DIR) && cpio --quiet -o -H newc < $(BR2_TARGET_ROOTFS_CPIO_CUSTOM_FILELIST) > $@ +endef + +endif + $(BINARIES_DIR)/rootfs.cpio.uboot: $(BINARIES_DIR)/rootfs.cpio host-uboot-tools $(MKIMAGE) -A $(MKIMAGE_ARCH) -T ramdisk \ -C none -d $<$(ROOTFS_CPIO_COMPRESS_EXT) $@
PATCH 1/1 fs/cpio: add option to manually specify file list
In reply to Seamus ConnorPeter, Arnout, Yann, all,
This patch has been pending since ~2 months with no feedback. Do we want something like this?
One thing that bothers me a bit is that it would make the initramfs/cpio image contain a filesystem that is different from other root filesystem images. But perhaps this is OK, because initramfs is indeed special.
So I'm inclined to just apply this, as it may be useful for some users. If you don't voice a concern within the next days, I'll apply.
Thanks!
Thmas
Quoted 79 lines · Seamus Connor
For some systems it may be desireable to have an initramfs that is smaller than the full root filesystem. This commit adds support for creating an initramfs based on a manually specified set of files. It is up to the user to define a sensible set of files. This file list could either be statically created, or it could be created as the result of a pre-fs hook script.
Signed-off-by Seamus Connor <[email protected]>--- fs/cpio/Config.in | 30 ++++++++++++++++++++++++++++++ fs/cpio/cpio.mk | 10 ++++++++++ 2 files changed, 40 insertions(+)fs/cpio/Config.in +30
-0@@ -7,6 +7,36 @@ config BR2_TARGET_ROOTFS_CPIO if BR2_TARGET_ROOTFS_CPIO +choice + prompt "Seleted Fileset" + default BR2_TARGET_ROOTFS_CPIO_INCLUDE_ALL + help + Select the set of files to include in the cpio of the root filesystem. + This may either be all files installed to the target, or it may be a + subset defined in the custom file list. + +config BR2_TARGET_ROOTFS_CPIO_INCLUDE_ALL + bool "Include all files from target" + help + Include all files from the target directory in the root filesystem. + +config BR2_TARGET_ROOTFS_CPIO_INCLUDE_CUSTOM + bool "Include a subset of files from target" + help + Include a subset of the files from the target directory in the root filesystem. + This requires a custom file list to be defined. The actual file list may be + generated by a user script defined as part of the pre-fs hook. + +endchoice + +config BR2_TARGET_ROOTFS_CPIO_CUSTOM_FILELIST + string "path to a custom file list" + depends on BR2_TARGET_ROOTFS_CPIO && BR2_TARGET_ROOTFS_CPIO_INCLUDE_CUSTOM + help + This file defines the contents of the rootfilesystem. All files and folders + created in the rootfilesystem must be defined here. This file can be statically + defined, or created as part of the pre-fs hook. + choice prompt "Compression method" default BR2_TARGET_ROOTFS_CPIO_NONEfs/cpio/cpio.mk +10
-0@@ -27,10 +27,20 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT +ifeq ($(BR2_TARGET_ROOTFS_CPIO_INCLUDE_ALL),y) + define ROOTFS_CPIO_CMD cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@ endef +else + +define ROOTFS_CPIO_CMD + cd $(TARGET_DIR) && cpio --quiet -o -H newc < $(BR2_TARGET_ROOTFS_CPIO_CUSTOM_FILELIST) > $@ +endef + +endif + $(BINARIES_DIR)/rootfs.cpio.uboot: $(BINARIES_DIR)/rootfs.cpio host-uboot-tools $(MKIMAGE) -A $(MKIMAGE_ARCH) -T ramdisk \ -C none -d $<$(ROOTFS_CPIO_COMPRESS_EXT) $@
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH 1/1 fs/cpio: add option to manually specify file list
Thomas, Seanus, All,
On 2018-01-01 12:54 +0100, Thomas Petazzoni spake thusly:
Quoted 5 lines
Peter, Arnout, Yann, all,
This patch has been pending since ~2 months with no feedback. Do we want something like this?
Quoted 7 lines
One thing that bothers me a bit is that it would make the initramfs/cpio image contain a filesystem that is different from other root filesystem images. But perhaps this is OK, because initramfs is indeed special.
So I'm inclined to just apply this, as it may be useful for some users. If you don't voice a concern within the next days, I'll apply.
I don't like this much, in fact.
In such a case, I would say that the initramfs filesystem is a different system, and thus should be built with a different .config.
Basically, what I do in this situation:
- have a first .config for kernel+modules + initramfs, and with a post-build script that:
1. copy all the kernel modules "somewhere",
2. trim the kernel modules that end up in the initramfs, to keep just what is needed to boot;
- have a second .config with just userland, plus a post-build script that vampirises the kernel modules from "somwehere" (as saved above)
Of course, the "somewhere" is agreed on because I know it.
Besides, this is much more flexible than what this patch proposes.
Really, the core of the issue is to believe that the two systems are one. They are not: they are two different systems.
So, I am not in favour of this patch.
Regards,
Yann E. MORIN.
Quoted 91 lines
Thanks!
Thmas
Quoted 79 lines · Seamus Connor
For some systems it may be desireable to have an initramfs that is smaller than the full root filesystem. This commit adds support for creating an initramfs based on a manually specified set of files. It is up to the user to define a sensible set of files. This file list could either be statically created, or it could be created as the result of a pre-fs hook script.
Signed-off-by Seamus Connor <[email protected]>--- fs/cpio/Config.in | 30 ++++++++++++++++++++++++++++++ fs/cpio/cpio.mk | 10 ++++++++++ 2 files changed, 40 insertions(+)fs/cpio/Config.in +30
-0@@ -7,6 +7,36 @@ config BR2_TARGET_ROOTFS_CPIO if BR2_TARGET_ROOTFS_CPIO +choice + prompt "Seleted Fileset" + default BR2_TARGET_ROOTFS_CPIO_INCLUDE_ALL + help + Select the set of files to include in the cpio of the root filesystem. + This may either be all files installed to the target, or it may be a + subset defined in the custom file list. + +config BR2_TARGET_ROOTFS_CPIO_INCLUDE_ALL + bool "Include all files from target" + help + Include all files from the target directory in the root filesystem. + +config BR2_TARGET_ROOTFS_CPIO_INCLUDE_CUSTOM + bool "Include a subset of files from target" + help + Include a subset of the files from the target directory in the root filesystem. + This requires a custom file list to be defined. The actual file list may be + generated by a user script defined as part of the pre-fs hook. + +endchoice + +config BR2_TARGET_ROOTFS_CPIO_CUSTOM_FILELIST + string "path to a custom file list" + depends on BR2_TARGET_ROOTFS_CPIO && BR2_TARGET_ROOTFS_CPIO_INCLUDE_CUSTOM + help + This file defines the contents of the rootfilesystem. All files and folders + created in the rootfilesystem must be defined here. This file can be statically + defined, or created as part of the pre-fs hook. + choice prompt "Compression method" default BR2_TARGET_ROOTFS_CPIO_NONEfs/cpio/cpio.mk +10
-0@@ -27,10 +27,20 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT +ifeq ($(BR2_TARGET_ROOTFS_CPIO_INCLUDE_ALL),y) + define ROOTFS_CPIO_CMD cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@ endef +else + +define ROOTFS_CPIO_CMD + cd $(TARGET_DIR) && cpio --quiet -o -H newc < $(BR2_TARGET_ROOTFS_CPIO_CUSTOM_FILELIST) > $@ +endef + +endif + $(BINARIES_DIR)/rootfs.cpio.uboot: $(BINARIES_DIR)/rootfs.cpio host-uboot-tools $(MKIMAGE) -A $(MKIMAGE_ARCH) -T ramdisk \ -C none -d $<$(ROOTFS_CPIO_COMPRESS_EXT) $@Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/1 fs/cpio: add option to manually specify file list
Hello,
Quoted 19 lines · Yann E. MORIN
Quoted 2 lines
So I'm inclined to just apply this, as it may be useful for some users. If you don't voice a concern within the next days, I'll apply.
I don't like this much, in fact.
In such a case, I would say that the initramfs filesystem is a different system, and thus should be built with a different .config.
Basically, what I do in this situation:
- have a first .config for kernel+modules + initramfs, and with a post-build script that:
1. copy all the kernel modules "somewhere",
2. trim the kernel modules that end up in the initramfs, to keep just what is needed to boot;- have a second .config with just userland, plus a post-build script that vampirises the kernel modules from "somwehere" (as saved above)
Quoted 6 lines
Of course, the "somewhere" is agreed on because I know it.
Besides, this is much more flexible than what this patch proposes.
Really, the core of the issue is to believe that the two systems are one. They are not: they are two different systems.
On the other hand, the initramfs is very often a subset of the complete root filesystem, and being able to generate an image of this subset directly from the same configuration as the full-blown system is definitely a nice thing.
I agree that a separate configuration to build the initramfs is more flexible, but it also has its drawbacks (to handle kernel modules, as you mentioned, but also in terms of rebuilding the toolchain if you're using an internal toolchain). One example where it might be annoying to have a separate configuration is if you have a single package that builds both a kernel module and a user-space counterpart.
So yeah, Seamus proposal is not ideal in terms of flexibility, but it does solve one common use-case with little complexity.
Again, I'm not 100% convinced either, so it's good to have this discussion.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
PATCH 1/1 fs/cpio: add option to manually specify file list
All,
Quoted 24 lines · Thomas Petazzoni
Hello,
Quoted 20 lines · Yann E. MORIN
Quoted 2 lines
So I'm inclined to just apply this, as it may be useful for some users. If you don't voice a concern within the next days, I'll apply.
I don't like this much, in fact.
In such a case, I would say that the initramfs filesystem is a different system, and thus should be built with a different .config.
Basically, what I do in this situation:
- have a first .config for kernel+modules + initramfs, and with a post-build script that:
1. copy all the kernel modules "somewhere",
2. trim the kernel modules that end up in the initramfs, to keep just what is needed to boot;- have a second .config with just userland, plus a post-build script that vampirises the kernel modules from "somwehere" (as saved above)
Of course, the "somewhere" is agreed on because I know it.
I'm not a fan of this method, because it introduces more complexity than I think is worth for such a simple concept.
Quoted 11 lines
Quoted 5 lines
Besides, this is much more flexible than what this patch proposes.
Really, the core of the issue is to believe that the two systems are one. They are not: they are two different systems.
On the other hand, the initramfs is very often a subset of the complete root filesystem, and being able to generate an image of this subset directly from the same configuration as the full-blown system is definitely a nice thing.
Quoted 4 lines
I agree that a separate configuration to build the initramfs is more flexible, but it also has its drawbacks (to handle kernel modules, as you mentioned, but also in terms of rebuilding the toolchain if you're using an internal toolchain).
We often use internal toolchain, and rebuilding it for a pure subset seems excessive.
Quoted 9 lines
One example where it might be annoying to
have a separate configuration is if you have a single package that builds both a kernel module and a user-space counterpart.So yeah, Seamus proposal is not ideal in terms of flexibility, but it does solve one common use-case with little complexity.
Again, I'm not 100% convinced either, so it's good to have this discussion.
Overall, I think this is a reasonable solution, however maybe not the "best" solution.
Sam
PATCH 1/1 fs/cpio: add option to manually specify file list
In reply to Thomas PetazzoniOn 01-01-18 15:47, Thomas Petazzoni wrote:
[snip]
Quoted 2 lines
So yeah, Seamus proposal is not ideal in terms of flexibility, but it does solve one common use-case with little complexity.
I don't think it's a very common usecase. In fact, usually there will be some
additional files in the initramfs, e.g. to do a pivotroot.
More importantly, though: the same result can be relatively easily achieved in
a POST_FAKEROOT_SCRIPT, and completely flexible.
Regards,
Arnout
Quoted 6 lines
Again, I'm not 100% convinced either, so it's good to have this discussion.
Thomas
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH 1/1 fs/cpio: add option to manually specify file list
In reply to Seamus ConnorHello,
Quoted 8 lines · Seamus Connor
For some systems it may be desireable to have an initramfs that is smaller than the full root filesystem. This commit adds support for creating an initramfs based on a manually specified set of files. It is up to the user to define a sensible set of files. This file list could either be statically created, or it could be created as the result of a pre-fs hook script.
Signed-off-by Seamus Connor <[email protected]>
Considering the replies from both Yann and Arnout who were both not very convinced by the approach, and Arnout's idea of using a fakeroot script to achieve the same thing in a more flexible way, I've marked this patch as Rejected in our patch tracking system.
Best regards,
Thomas Petazzoni
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering http://bootlin.com