fs: support parallel filesystems build
v4 cover fs: support parallel filesystems build
Hello All!
TL;DR: filsystems are not parallel-safe, so we generate an intermediate tarball from target/, which then serves as input for the various filesystems; we also run pre-gen hooks under fakeroot, and get rid of post-fs package-provided hooks.
Full explanations follows:
Currently, the filesystems are all built sequentially. This is unfortunate, because each filesystem commands may modify the content of target/, e.g. when systemd is enabled on a read-only filesystem, or cpio to add its /init stub. cpio even changes the packages-supplied list of devices, to shoe-horn /dev/console.
This has two issues:
- the content of the generated rootfs depends on whether other
filesystems are enabled or not;
- it is not posible to build filesystems in parallel, becasue of concurency on target/.
A further (less critical) issue is that the fakeroot stuff is repeated again and again for each filesystem: generating the user, groups and device tables...
The only sane solution is to have each filesystem operate on its own copy of target/, which is assembled once uner fakeroot and then re-used by each filesystem, so that they can each do their own last-ditch tweaks to the content of the filesystem before generating the image.
So we introduce an intermediate tarball that is assembled under fakeroot, to run all the common fakeroot-needing setup. Each filesytem then run a very simple fakeroot script that extracts that intermediate tarball in a transient per-filesystem location and use that as input to their image generation commands. This also allows to run the fakerooted, common actions only once.
Today, the filesystems pre-/post-gen hooks are called very early, outside of the fakeroot script, which means they can not do modifications that require root access, like creating device nodes.
And as a last straw, the recently added (in August 2017) support for package-provided pre-/post-fs hooks, and used solely by systemd on a read-only filesystem, means that the target directory may be left in an inconsistent state should the filesystem image generator fails, because those pre-/post-fs hooks really need an atomic seciont which we can not provide.
For this reason, the only solution is to act on a copy of target/.
So, now that all the issues have been exposed and their solutions explained, the series is split as such:
- first we move the filesystemns pre-gen hooks under fakeroot, and we fix the cpio filesystem to correctly create its device node; the post-gen hooks need not run under fakeroot, because they are only supposed to act on the generated image, which does not require root rights (and modifying target/ after the iamge has been generated is pointless anyway;
- then we introduce per-rootfs variables (per rootfs name and directory, and per rootfs target directory) and fakeroot scripts, and off-load the common dependencies to a common rule;
- then we eventually split the filesystem generation in two, introducing the intermediate tarball, which filesystems re-use as input to generate their images. We also introduce a copy of target/ from which to generate the intermediate tarball, which allows us to get rid of the post-fs hooks provided by packages.
- finally, the last patchs adds the removal of the transient copies of the target/ directory, as well as of the intermediate tarball, since they all can be rather large and are not meant to be user-visible.
As an aside, the pre-gen and post-gen hooks are not totally symetric
anymore: the pre-gen hooks run under fakeroot while the post-gen hooks do not. A brief discussion on IRC suggested a change of name, since they are no longer symetric, but in the end, I left this renaming as a further exercise.
The series still runs the test suite with a few failures, that are already present on master it based on:
- master: https://gitlab.com/buildroot.org/buildroot/pipelines/17079175
- series: https://gitlab.com/ymorin/buildroot-ci/pipelines/17080587
Note: the entire series was tested by Matthew, but I carry his tested-by tag only on the last commit.
Changes v2 -> v3:
- none; rebased on master
Changes v1 -> v2:
- drop applied patches
- write a complete commit log for now-first patch (Thomas)
Regards,
Yann E. MORIN.
The following changes since commit 5966e2dc54dfb19c5fde3a09d72f3abc6125c202
package/openocd: fix fallout after no-C++ fixups (2018-03-31 09:26:19 +0200)
are available in the git repository at:
git://git.buildroot.org/~ymorin/git/buildroot.git
for you to fetch changes up to aa32fec260607f3c9e3a436873b8ef72802078e7
fs: remove intermediate artefacts (2018-03-31 10:50:25 +0200)
----------------------------------------------------------------
Yann E. MORIN (15):
fs: run filesystem hooks under fakeroot
fs/cpio: don't extend packages' permissions table fs/iso9660: rename internal variable
core: introduce intermediate BASE_TARGET_DIR variable
fs: set per-rootfs variable name
fs: define per-rootfs directory
fs: use a per-rootfs fakeroot script
fs: split per-rootfs dependency line
fs: introduce per-rootfs TARGET_DIR variable
fs: new intermediate rootfs-common to gather common dependencies
fs: introduce a macro for reproducible command
fs: use a common tarball as base for the other filesystems
fs: run packages' filesystem hooks in a copy of target/
fs: get rid of package-provided post-fs hooks
fs: remove intermediate artefacts
Makefile | 17 ++- fs/common.mk | 114 +++++++++++++++------ fs/cpio/cpio.mk | 5 +- fs/iso9660/iso9660.mk | 30 +++--- package/pkg-generic.mk | 6 +- .../skeleton-init-systemd/skeleton-init-systemd.mk | 6 -- support/testing/tests/core/test_post_scripts.py | 23 +++-- 7 files changed, 128 insertions(+), 73 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v4 1/15 fs: run filesystem hooks under fakeroot
Currently, the pre-gen hooks of the various filesystems are run before we enter the fakeroot.
However, this precludes those hooks from doing actions that require root, like creating a pseudo-device or the likes.
So, move those pre-gen hooks under fakeroot.
This has currently no side-effect, as they are still called before everything else in the fakeroot script, even the system-wide chown call.
--- fs/common.mk | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/common.mk +2 -1
@@ -76,9 +76,10 @@ $$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES) @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") rm -rf $(FS_DIR) mkdir -p $(FS_DIR) - $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep)) echo '#!/bin/sh' > $$(FAKEROOT_SCRIPT) echo "set -e" >> $$(FAKEROOT_SCRIPT) + $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),\ + $$(call PRINTF,$$($$(hook))) >> $$(FAKEROOT_SCRIPT)$$(sep)) echo "chown -h -R 0:0 $$(TARGET_DIR)" >> $$(FAKEROOT_SCRIPT) ifneq ($$(ROOTFS_USERS_TABLES),) cat $$(ROOTFS_USERS_TABLES) >> $$(USERS_TABLE)
Signature · 1 line
2.14.1
PATCH v4 2/15 fs/cpio: don't extend packages' permissions table
In reply to Yann E. MORINCurrently, when we build a cpio filesystem without static devices, we shoehorn the /dev/console node as if it were specified by a package.
This means that this device is added for all filesystems as well, not just the cpio. But if we disable cpio, that device is not created for other filesystems.
This is not very clean, and may break expectations.
Instead, use an explicit mknod as part of the _CMD, as we know it's going to run under fakeroot.
This is still visible to all filesystems built after cpio, and not to those built before it, though.
--- fs/cpio/cpio.mk | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
fs/cpio/cpio.mk +3 -2
@@ -15,14 +15,15 @@ endef else # devtmpfs does not get automounted when initramfs is used. # Add a pre-init script to mount it before running init +# We must have /dev/console very early, even before /init runs, so +# that it can at least open its stdin/stdout/stderr filedescriptors. define ROOTFS_CPIO_ADD_INIT if [ ! -e $(TARGET_DIR)/init ]; then \ $(INSTALL) -m 0755 fs/cpio/init $(TARGET_DIR)/init; \ fi + mknod -m 0622 $(TARGET_DIR)/dev/console c 5 1 endef -PACKAGES_PERMISSIONS_TABLE += /dev/console c 622 0 0 5 1 - - -$(sep) - endif # BR2_ROOTFS_DEVICE_CREATION_STATIC ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT
Signature · 1 line
2.14.1
PATCH v4 2/15 fs/cpio: don't extend packages' permissions table
Quoted 29 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Currently, when we build a cpio filesystem without static devices, we shoehorn the
/dev/consolenode as if it were specified by a package.This means that this device is added for all filesystems as well, not just the cpio. But if we disable cpio, that device is not created for other filesystems.
This is not very clean, and may break expectations.
Instead, use an explicit mknod as part of the
_CMD, as we know it's going to run under fakeroot.This is still visible to all filesystems built after cpio, and not to those built before it, though.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>--- fs/cpio/cpio.mk | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)fs/cpio/cpio.mk +0
-0
fs/cpio/cpio.mk endef · line 24
Quoted 10 lines
fs/cpio/cpio.mk +3
-0@@ -15,14 +15,15 @@ endef else # devtmpfs does not get automounted when initramfs is used. # Add a pre-init script to mount it before running init +# We must have /dev/console very early, even before /init runs, so +# that it can at least open its stdin/stdout/stderr filedescriptors. define ROOTFS_CPIO_ADD_INIT if [ ! -e $(TARGET_DIR)/init ]; then \ $(INSTALL) -m 0755 fs/cpio/init $(TARGET_DIR)/init; \ fi + mknod -m 0622 $(TARGET_DIR)/dev/console c 5 1
This is executed before makedevs, so we cannot be sure /dev exists (our default skeleton has it, but a custom one might not).
I've fixed that by adding a mkdir -p $(TARGET_DIR)/dev before mknod.
Signature · 1 line
Bye, Peter Korsgaard
PATCH v4 3/15 fs/iso9660: rename internal variable
In reply to Yann E. MORINWe're soon to introduce a generic variable with that name, i.e. $(FS_NAME)_TARGET_DIR, so our internal, temporary variable would clash with it.
Just rename it.
--- fs/iso9660/iso9660.mk | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
fs/iso9660/iso9660.mk +15 -15
@@ -35,47 +35,47 @@ ROOTFS_ISO9660_USE_INITRD = YES endif ifeq ($(ROOTFS_ISO9660_USE_INITRD),YES) -ROOTFS_ISO9660_TARGET_DIR = $(FS_DIR)/rootfs.iso9660.tmp +ROOTFS_ISO9660_TMP_TARGET_DIR = $(FS_DIR)/rootfs.iso9660.tmp define ROOTFS_ISO9660_CREATE_TEMPDIR - $(RM) -rf $(ROOTFS_ISO9660_TARGET_DIR) - mkdir -p $(ROOTFS_ISO9660_TARGET_DIR) + $(RM) -rf $(ROOTFS_ISO9660_TMP_TARGET_DIR) + mkdir -p $(ROOTFS_ISO9660_TMP_TARGET_DIR) endef ROOTFS_ISO9660_PRE_GEN_HOOKS += ROOTFS_ISO9660_CREATE_TEMPDIR else ifeq ($(BR2_TARGET_ROOTFS_ISO9660_TRANSPARENT_COMPRESSION),y) ROOTFS_ISO9660_DEPENDENCIES += host-zisofs-tools -ROOTFS_ISO9660_TARGET_DIR = $(FS_DIR)/rootfs.iso9660.tmp +ROOTFS_ISO9660_TMP_TARGET_DIR = $(FS_DIR)/rootfs.iso9660.tmp # This must be early, before we copy the bootloader files. define ROOTFS_ISO9660_MKZFTREE - $(RM) -rf $(ROOTFS_ISO9660_TARGET_DIR) + $(RM) -rf $(ROOTFS_ISO9660_TMP_TARGET_DIR) $(HOST_DIR)/bin/mkzftree -X -z 9 -p $(PARALLEL_JOBS) \ $(TARGET_DIR) \ - $(ROOTFS_ISO9660_TARGET_DIR) + $(ROOTFS_ISO9660_TMP_TARGET_DIR) endef ROOTFS_ISO9660_PRE_GEN_HOOKS += ROOTFS_ISO9660_MKZFTREE ROOTFS_ISO9660_GENISOIMAGE_OPTS += -z else -ROOTFS_ISO9660_TARGET_DIR = $(TARGET_DIR) +ROOTFS_ISO9660_TMP_TARGET_DIR = $(TARGET_DIR) endif ifeq ($(BR2_TARGET_ROOTFS_ISO9660_GRUB2),y) ROOTFS_ISO9660_DEPENDENCIES += grub2 ROOTFS_ISO9660_BOOTLOADER_CONFIG_PATH = \ - $(ROOTFS_ISO9660_TARGET_DIR)/boot/grub/grub.cfg + $(ROOTFS_ISO9660_TMP_TARGET_DIR)/boot/grub/grub.cfg ROOTFS_ISO9660_BOOT_IMAGE = boot/grub/grub-eltorito.img define ROOTFS_ISO9660_INSTALL_BOOTLOADER $(INSTALL) -D -m 0644 $(BINARIES_DIR)/grub-eltorito.img \ - $(ROOTFS_ISO9660_TARGET_DIR)/boot/grub/grub-eltorito.img + $(ROOTFS_ISO9660_TMP_TARGET_DIR)/boot/grub/grub-eltorito.img endef else ifeq ($(BR2_TARGET_ROOTFS_ISO9660_ISOLINUX),y) ROOTFS_ISO9660_DEPENDENCIES += syslinux ROOTFS_ISO9660_BOOTLOADER_CONFIG_PATH = \ - $(ROOTFS_ISO9660_TARGET_DIR)/isolinux/isolinux.cfg + $(ROOTFS_ISO9660_TMP_TARGET_DIR)/isolinux/isolinux.cfg ROOTFS_ISO9660_BOOT_IMAGE = isolinux/isolinux.bin define ROOTFS_ISO9660_INSTALL_BOOTLOADER $(INSTALL) -D -m 0644 $(BINARIES_DIR)/syslinux/* \ - $(ROOTFS_ISO9660_TARGET_DIR)/isolinux/ + $(ROOTFS_ISO9660_TMP_TARGET_DIR)/isolinux/ $(INSTALL) -D -m 0644 $(HOST_DIR)/share/syslinux/ldlinux.c32 \ - $(ROOTFS_ISO9660_TARGET_DIR)/isolinux/ldlinux.c32 + $(ROOTFS_ISO9660_TMP_TARGET_DIR)/isolinux/ldlinux.c32 endef endif
@@ -96,7 +96,7 @@ endef # Copy the kernel to temporary filesystem define ROOTFS_ISO9660_COPY_KERNEL $(INSTALL) -D -m 0644 $(LINUX_IMAGE_PATH) \ - $(ROOTFS_ISO9660_TARGET_DIR)/boot/$(LINUX_IMAGE_NAME) + $(ROOTFS_ISO9660_TMP_TARGET_DIR)/boot/$(LINUX_IMAGE_NAME) endef ifeq ($(ROOTFS_ISO9660_USE_INITRD),YES)
@@ -111,7 +111,7 @@ else ROOTFS_ISO9660_DEPENDENCIES += rootfs-cpio define ROOTFS_ISO9660_COPY_INITRD $(INSTALL) -D -m 0644 $(BINARIES_DIR)/rootfs.cpio$(ROOTFS_CPIO_COMPRESS_EXT) \ - $(ROOTFS_ISO9660_TARGET_DIR)/boot/initrd + $(ROOTFS_ISO9660_TMP_TARGET_DIR)/boot/initrd $(SED) "s%__INITRD_PATH__%/boot/initrd%" \ $(ROOTFS_ISO9660_BOOTLOADER_CONFIG_PATH) endef
@@ -133,7 +133,7 @@ define ROOTFS_ISO9660_CMD $(HOST_DIR)/bin/genisoimage -J -R -b $(ROOTFS_ISO9660_BOOT_IMAGE) \ -no-emul-boot -boot-load-size 4 -boot-info-table \ $(ROOTFS_ISO9660_GENISOIMAGE_OPTS) \ - -o $@ $(ROOTFS_ISO9660_TARGET_DIR) + -o $@ $(ROOTFS_ISO9660_TMP_TARGET_DIR) endef ifeq ($(BR2_TARGET_ROOTFS_ISO9660_HYBRID),y)
Signature · 1 line
2.14.1
PATCH v4 4/15 core: introduce intermediate BASE_TARGET_DIR variable
In reply to Yann E. MORINThis new BASE_TARGET_DIR variable is set in stone to point to the real location where packages will be installed. Its name is modelled after its definition: it is located in $(BASE_DIR), and it is named 'target/', hence BASE_TARGET_DIR.
The already-existing TARGET_DIR variable now simply points to the same location, except that it is recursively expanded, so that we can later change it depending on the context.
All locations that really need to reference the existing target/ directory, are changed to use BASE_TARGET_DIR; surprinsigly enough, they all seem to be located in the main Makefile. :-) The rest is left with using good-old TARGET_DIR.
--- Makefile | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
Makefile +6 -5
@@ -215,7 +215,8 @@ BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf) BUILD_DIR := $(BASE_DIR)/build BINARIES_DIR := $(BASE_DIR)/images -TARGET_DIR := $(BASE_DIR)/target +BASE_TARGET_DIR := $(BASE_DIR)/target +TARGET_DIR = $(BASE_TARGET_DIR) # initial definition so that 'make clean' works for most users, even without # .config. HOST_DIR will be overwritten later when .config is included. HOST_DIR := $(BASE_DIR)/host
@@ -237,7 +238,7 @@ LEGAL_REPORT = $(LEGAL_INFO_DIR)/README # dependencies anywhere else # ################################################################################ -$(BUILD_DIR) $(TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST): +$(BUILD_DIR) $(BASE_TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST): @mkdir -p $@ BR2_CONFIG = $(CONFIG_DIR)/.config
@@ -459,7 +460,7 @@ BR_PATH = "$(HOST_DIR)/bin:$(HOST_DIR)/sbin:$(PATH)" # Location of a file giving a big fat warning that output/target # should not be used as the root filesystem. -TARGET_DIR_WARNING_FILE = $(TARGET_DIR)/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM +TARGET_DIR_WARNING_FILE = $(BASE_TARGET_DIR)/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM ifeq ($(BR2_CCACHE),y) CCACHE := $(HOST_DIR)/bin/ccache
@@ -556,7 +557,7 @@ $(foreach pkg,$(call UPPERCASE,$(PACKAGES)),\ endif .PHONY: dirs -dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \ +dirs: $(BUILD_DIR) $(STAGING_DIR) $(BASE_TARGET_DIR) \ $(HOST_DIR) $(BINARIES_DIR) $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
@@ -972,7 +973,7 @@ printvars: .PHONY: clean clean: - rm -rf $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \ + rm -rf $(BASE_TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \ $(BUILD_DIR) $(BASE_DIR)/staging \ $(LEGAL_INFO_DIR) $(GRAPHS_DIR)
Signature · 1 line
2.14.1
PATCH v4 5/15 fs: set per-rootfs variable name
In reply to Yann E. MORINLike we do for packages with the PKG variable, set ROOTFS to contain the upper-case name of the rootfs currently being generated.
This will be usefull in later patches, when we need more per-rootfs variables, like a per-rootfs TARGET_DIR for example.
In Makefiles, per-rule variables trickle down the dependency chain, to all dependencies of that rule, so we have to stop ROOTFS as soone as we're not in a rootfs. This means we have to stop it at target-finalize (which is a dependency of all filesystems), and for each package individually, since some packages (host or target) can be direct dependencies of filesystems as well.
--- Makefile | 3 +++ fs/common.mk | 3 +++ package/pkg-generic.mk | 4 ++++ 3 files changed, 10 insertions(+)
Makefile +3 -0
@@ -680,6 +680,9 @@ endif $(TARGETS_ROOTFS): target-finalize +# Avoid the rootfs name leaking down the dependency chain +target-finalize: ROOTFS= + .PHONY: target-finalize target-finalize: $(PACKAGES) @$(call MESSAGE,"Finalizing target directory")
fs/common.mk +3 -0
@@ -39,6 +39,8 @@ ROOTFS_USERS_TABLES = $(call qstrip,$(BR2_ROOTFS_USERS_TABLES)) # all variable references except the arguments must be $$-quoted. define inner-rootfs +ROOTFS_$(2)_NAME = $(1) + # extra deps ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs \ $$(if $$(PACKAGES_USERS)$$(ROOTFS_USERS_TABLES),host-mkpasswd)
@@ -72,6 +74,7 @@ ROOTFS_$(2)_COMPRESS_EXT = .xz ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c endif +$$(BINARIES_DIR)/rootfs.$(1): ROOTFS=$(2) $$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES) @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") rm -rf $(FS_DIR)
package/pkg-generic.mk +4 -0
@@ -384,6 +384,10 @@ endef define inner-generic-package +# When doing a package, we're definitely not doing a rootfs, but we +# may inherit it via the dependency chain, so we reset it. +$(1): ROOTFS= + # Ensure the package is only declared once, i.e. do not accept that a # package be re-defined by a br2-external tree ifneq ($(call strip,$(filter $(1),$(PACKAGES_ALL))),)
Signature · 1 line
2.14.1
PATCH v4 5/15 fs: set per-rootfs variable name
Quoted 41 lines · Yann E. MORIN
Like we do for packages with the PKG variable, set ROOTFS to contain the upper-case name of the rootfs currently being generated.
This will be usefull in later patches, when we need more per-rootfs variables, like a per-rootfs
TARGET_DIRfor example.In Makefiles, per-rule variables trickle down the dependency chain, to all dependencies of that rule, so we have to stop ROOTFS as soone as we're not in a rootfs. This means we have to stop it at target-finalize (which is a dependency of all filesystems), and for each package individually, since some packages (host or target) can be direct dependencies of filesystems as well.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>--- Makefile | 3 +++ fs/common.mk | 3 +++ package/pkg-generic.mk | 4 ++++ 3 files changed, 10 insertions(+)Makefile +3
-0@@ -680,6 +680,9 @@ endif $(TARGETS_ROOTFS): target-finalize +# Avoid the rootfs name leaking down the dependency chain +target-finalize: ROOTFS= + .PHONY: target-finalize target-finalize: $(PACKAGES) @$(call MESSAGE,"Finalizing target directory")fs/common.mk +0
-0
fs/common.mk ROOTFS_USERS_TABLES = $(call qstrip,$(BR2_ROOTFS_USERS_TABLES)) · line 42
Quoted 4 lines
fs/common.mk +1
-0@@ -39,6 +39,8 @@ ROOTFS_USERS_TABLES = $(call qstrip,$(BR2_ROOTFS_USERS_TABLES)) # all variable references except the arguments must be $$-quoted. define inner-rootfs +ROOTFS_$(2)_NAME = $(1)
This variable is not used anywhere...
Regards,
Arnout
Quoted 28 lines
fs/common.mk +2
-0+ # extra deps ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs \ $$(if $$(PACKAGES_USERS)$$(ROOTFS_USERS_TABLES),host-mkpasswd)@@ -72,6 +74,7 @@ ROOTFS_$(2)_COMPRESS_EXT = .xz ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c endif +$$(BINARIES_DIR)/rootfs.$(1): ROOTFS=$(2) $$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES) @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") rm -rf $(FS_DIR)package/pkg-generic.mk +4
-0@@ -384,6 +384,10 @@ endef define inner-generic-package +# When doing a package, we're definitely not doing a rootfs, but we +# may inherit it via the dependency chain, so we reset it. +$(1): ROOTFS= + # Ensure the package is only declared once, i.e. do not accept that a # package be re-defined by a br2-external tree ifneq ($(call strip,$(filter $(1),$(PACKAGES_ALL))),)
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v4 6/15 fs: define per-rootfs directory
In reply to Yann E. MORINThis will serve in future commits to store pre-rootfs files, like fakeroot script...
--- fs/common.mk | 1 + 1 file changed, 1 insertion(+)
fs/common.mk +1 -0
@@ -40,6 +40,7 @@ ROOTFS_USERS_TABLES = $(call qstrip,$(BR2_ROOTFS_USERS_TABLES)) define inner-rootfs ROOTFS_$(2)_NAME = $(1) +ROOTFS_$(2)_DIR = $$(FS_DIR)/$(1) # extra deps ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs \
Signature · 1 line
2.14.1
PATCH v4 7/15 fs: use a per-rootfs fakeroot script
In reply to Yann E. MORIN... and locate that script in a per-rootfs directory.
--- fs/common.mk | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
fs/common.mk +3 -3
@@ -28,7 +28,6 @@ # macro will automatically generate a compressed filesystem image. FS_DIR = $(BUILD_DIR)/buildroot-fs -FAKEROOT_SCRIPT = $(FS_DIR)/fakeroot.fs FULL_DEVICE_TABLE = $(FS_DIR)/device_table.txt ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE) \ $(BR2_ROOTFS_STATIC_DEVICE_TABLE))
@@ -76,10 +75,11 @@ ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c endif $$(BINARIES_DIR)/rootfs.$(1): ROOTFS=$(2) +$$(BINARIES_DIR)/rootfs.$(1): FAKEROOT_SCRIPT=$$(ROOTFS_$(2)_DIR)/fakeroot $$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES) @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") - rm -rf $(FS_DIR) - mkdir -p $(FS_DIR) + rm -rf $(FS_DIR) $$(ROOTFS_$(2)_DIR) + mkdir -p $(FS_DIR) $$(ROOTFS_$(2)_DIR) echo '#!/bin/sh' > $$(FAKEROOT_SCRIPT) echo "set -e" >> $$(FAKEROOT_SCRIPT) $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),\
Signature · 1 line
2.14.1
PATCH v4 7/15 fs: use a per-rootfs fakeroot script
Quoted 23 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
... and locate that script in a per-rootfs directory.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>--- fs/common.mk | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)fs/common.mk +0
-1@@ -28,7 +28,6 @@ # macro will automatically generate a compressed filesystem image. FS_DIR = $(BUILD_DIR)/buildroot-fs -FAKEROOT_SCRIPT = $(FS_DIR)/fakeroot.fs FULL_DEVICE_TABLE = $(FS_DIR)/device_table.txt ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE) \ $(BR2_ROOTFS_STATIC_DEVICE_TABLE))
fs/common.mk ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c · line 78
Quoted 4 lines
fs/common.mk +1
-0@@ -76,10 +75,11 @@ ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c endif $$(BINARIES_DIR)/rootfs.$(1): ROOTFS=$(2) +$$(BINARIES_DIR)/rootfs.$(1): FAKEROOT_SCRIPT=$$(ROOTFS_$(2)_DIR)/fakeroot
It wasn't quite clear to me why we had to go through the hoops to ensure ROOTFS didn't leak but we don't do it for FAKEROOT_SCRIPT, so I've extended the commit message a bit.
Signature · 1 line
Bye, Peter Korsgaard
PATCH v4 7/15 fs: use a per-rootfs fakeroot script
In reply to Yann E. MORINQuoted 22 lines · Yann E. MORIN
... and locate that script in a per-rootfs directory.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>--- fs/common.mk | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)fs/common.mk +0
-1@@ -28,7 +28,6 @@ # macro will automatically generate a compressed filesystem image. FS_DIR = $(BUILD_DIR)/buildroot-fs -FAKEROOT_SCRIPT = $(FS_DIR)/fakeroot.fs FULL_DEVICE_TABLE = $(FS_DIR)/device_table.txt ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE) \ $(BR2_ROOTFS_STATIC_DEVICE_TABLE))
fs/common.mk ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c · line 78
Quoted 4 lines
fs/common.mk +1
-0@@ -76,10 +75,11 @@ ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c endif $$(BINARIES_DIR)/rootfs.$(1): ROOTFS=$(2) +$$(BINARIES_DIR)/rootfs.$(1): FAKEROOT_SCRIPT=$$(ROOTFS_$(2)_DIR)/fakeroot
I don't agree with this bit. We want to avoid this kind of per-target override
whenever possible.
So IMO we should define the global FAKEROOT_SCRIPT as
-FAKEROOT_SCRIPT = $(FS_DIR)/fakeroot.fs +FAKEROOT_SCRIPT = $(ROOTFS_$(ROOTFS)_DIR)/fakeroot.fs
Regards,
Arnout
Quoted 10 lines
fs/common.mk +2
-2$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES) @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") - rm -rf $(FS_DIR) - mkdir -p $(FS_DIR) + rm -rf $(FS_DIR) $$(ROOTFS_$(2)_DIR) + mkdir -p $(FS_DIR) $$(ROOTFS_$(2)_DIR) echo '#!/bin/sh' > $$(FAKEROOT_SCRIPT) echo "set -e" >> $$(FAKEROOT_SCRIPT) $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),\
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v4 7/15 fs: use a per-rootfs fakeroot script
Quoted 35 lines · Arnout Vandecappelle
Quoted 22 lines · Yann E. MORIN
... and locate that script in a per-rootfs directory.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>--- fs/common.mk | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)fs/common.mk +0
-1@@ -28,7 +28,6 @@ # macro will automatically generate a compressed filesystem image. FS_DIR = $(BUILD_DIR)/buildroot-fs -FAKEROOT_SCRIPT = $(FS_DIR)/fakeroot.fs FULL_DEVICE_TABLE = $(FS_DIR)/device_table.txt ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE) \ $(BR2_ROOTFS_STATIC_DEVICE_TABLE))fs/common.mk ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c · line 78
Quoted 4 lines
fs/common.mk +1
-0@@ -76,10 +75,11 @@ ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c endif $$(BINARIES_DIR)/rootfs.$(1): ROOTFS=$(2) +$$(BINARIES_DIR)/rootfs.$(1): FAKEROOT_SCRIPT=$$(ROOTFS_$(2)_DIR)/fakerootI don't agree with this bit. We want to avoid this kind of per-target override
whenever possible.So IMO we should define the global
FAKEROOT_SCRIPTas
Review
Quoted 2 lines
(quoted diff) +1
-1-FAKEROOT_SCRIPT = $(FS_DIR)/fakeroot.fs +FAKEROOT_SCRIPT = $(ROOTFS_$(ROOTFS)_DIR)/fakeroot.fs
As Yann pointed out to me, if we do this, then patch 12/15 should also add a
definition of ROOTFS_COMMON_DIR = $(FS_DIR). And then probably FS_DIR should be renamed to ROOTFS_COMMON_DIR (which it is, in the end...).
Given that this is getting a little complicated, I'm OK to do this in a
follow-up patch instead.
Regards,
Arnout
Quoted 15 lines
Regards,
ArnoutQuoted 10 lines
fs/common.mk +2
-2$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES) @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") - rm -rf $(FS_DIR) - mkdir -p $(FS_DIR) + rm -rf $(FS_DIR) $$(ROOTFS_$(2)_DIR) + mkdir -p $(FS_DIR) $$(ROOTFS_$(2)_DIR) echo '#!/bin/sh' > $$(FAKEROOT_SCRIPT) echo "set -e" >> $$(FAKEROOT_SCRIPT) $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),\
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v4 8/15 fs: split per-rootfs dependency line
In reply to Yann E. MORIN---
Note: in its own patch for easier review of a later, bigger patch.
--- fs/common.mk | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
fs/common.mk +2 -1
@@ -76,7 +76,8 @@ endif $$(BINARIES_DIR)/rootfs.$(1): ROOTFS=$(2) $$(BINARIES_DIR)/rootfs.$(1): FAKEROOT_SCRIPT=$$(ROOTFS_$(2)_DIR)/fakeroot -$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES) +$$(BINARIES_DIR)/rootfs.$(1): target-finalize +$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES) @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") rm -rf $(FS_DIR) $$(ROOTFS_$(2)_DIR) mkdir -p $(FS_DIR) $$(ROOTFS_$(2)_DIR)
Signature · 1 line
2.14.1
PATCH v4 9/15 fs: introduce per-rootfs TARGET_DIR variable
In reply to Yann E. MORIN... which for now still points to the base target directory, but this is a step forward.
--- Makefile | 4 +++- fs/common.mk | 1 + 2 files changed, 4 insertions(+), 1 deletion(-)
Makefile +3 -1
@@ -215,8 +215,10 @@ BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf) BUILD_DIR := $(BASE_DIR)/build BINARIES_DIR := $(BASE_DIR)/images +# The target directory is common to all packages, +# but there is one that is specific to each filesystem. BASE_TARGET_DIR := $(BASE_DIR)/target -TARGET_DIR = $(BASE_TARGET_DIR) +TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR)) # initial definition so that 'make clean' works for most users, even without # .config. HOST_DIR will be overwritten later when .config is included. HOST_DIR := $(BASE_DIR)/host
fs/common.mk +1 -0
@@ -40,6 +40,7 @@ define inner-rootfs ROOTFS_$(2)_NAME = $(1) ROOTFS_$(2)_DIR = $$(FS_DIR)/$(1) +ROOTFS_$(2)_TARGET_DIR = $$(BASE_TARGET_DIR) # extra deps ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs \
Signature · 1 line
2.14.1
PATCH v4 9/15 fs: introduce per-rootfs TARGET_DIR variable
Quoted 16 lines · Yann E. MORIN
... which for now still points to the base target directory, but this is a step forward.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>--- Makefile | 4 +++- fs/common.mk | 1 + 2 files changed, 4 insertions(+), 1 deletion(-)Makefile +0
-0
Makefile BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf) · line 221
Quoted 8 lines
Makefile +3
-1@@ -215,8 +215,10 @@ BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf) BUILD_DIR := $(BASE_DIR)/build BINARIES_DIR := $(BASE_DIR)/images +# The target directory is common to all packages, +# but there is one that is specific to each filesystem. BASE_TARGET_DIR := $(BASE_DIR)/target -TARGET_DIR = $(BASE_TARGET_DIR) +TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR))
OK for now, but I think when PPS happens we should s/ROOTFS/PKG/ and simply use
$($(PKG)_TARGET_DIR) here.
Regards,
Arnout
Quoted 16 lines
Makefile +0
-0# initial definition so that 'make clean' works for most users, even without # .config. HOST_DIR will be overwritten later when .config is included. HOST_DIR := $(BASE_DIR)/hostfs/common.mk +1
-0@@ -40,6 +40,7 @@ define inner-rootfs ROOTFS_$(2)_NAME = $(1) ROOTFS_$(2)_DIR = $$(FS_DIR)/$(1) +ROOTFS_$(2)_TARGET_DIR = $$(BASE_TARGET_DIR) # extra deps ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs \
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v4 9/15 fs: introduce per-rootfs TARGET_DIR variable
Arnout, All,
On 2018-03-31 20:30 +0200, Arnout Vandecappelle spake thusly:
Quoted 18 lines · Yann E. MORIN
Quoted 17 lines
... which for now still points to the base target directory, but this is a step forward.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>--- Makefile | 4 +++- fs/common.mk | 1 + 2 files changed, 4 insertions(+), 1 deletion(-)Makefile +0
-0@@ -215,8 +215,10 @@ BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf)
Quoted 10 lines
Makefile BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf) · line 221
Quoted 7 lines
Makefile +3
-1BUILD_DIR := $(BASE_DIR)/build BINARIES_DIR := $(BASE_DIR)/images +# The target directory is common to all packages, +# but there is one that is specific to each filesystem. BASE_TARGET_DIR := $(BASE_DIR)/target -TARGET_DIR = $(BASE_TARGET_DIR) +TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR))OK for now, but I think when PPS happens we should s/ROOTFS/PKG/ and simply use
$($(PKG)_TARGET_DIR) here.
Why so? After PPS, each rootfs will still get their own private copy of target, which is what we will want them to use...
So, with PPS, this would evetually be something like:
TARGET_DIR = \
$(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),\
$(if $(PKG),$($(PKG)_TARGET_DIR),\
$(BASE_TARGET_DIR)))Regards,
Yann E. MORIN.
Quoted 27 lines
Regards,
ArnoutQuoted 16 lines
Makefile +0
-0# initial definition so that 'make clean' works for most users, even without # .config. HOST_DIR will be overwritten later when .config is included. HOST_DIR := $(BASE_DIR)/hostfs/common.mk +1
-0@@ -40,6 +40,7 @@ define inner-rootfs ROOTFS_$(2)_NAME = $(1) ROOTFS_$(2)_DIR = $$(FS_DIR)/$(1) +ROOTFS_$(2)_TARGET_DIR = $$(BASE_TARGET_DIR) # extra deps ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs \Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v4 9/15 fs: introduce per-rootfs TARGET_DIR variable
Quoted 35 lines · Yann E. MORIN
Arnout, All,
On 2018-03-31 20:30 +0200, Arnout Vandecappelle spake thusly:
Quoted 18 lines · Yann E. MORIN
Quoted 17 lines
... which for now still points to the base target directory, but this is a step forward.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>--- Makefile | 4 +++- fs/common.mk | 1 + 2 files changed, 4 insertions(+), 1 deletion(-)Makefile +0
-0@@ -215,8 +215,10 @@ BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf)Quoted 10 lines
Makefile BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf) · line 221
Quoted 7 lines
Makefile +3
-1BUILD_DIR := $(BASE_DIR)/build BINARIES_DIR := $(BASE_DIR)/images +# The target directory is common to all packages, +# but there is one that is specific to each filesystem. BASE_TARGET_DIR := $(BASE_DIR)/target -TARGET_DIR = $(BASE_TARGET_DIR) +TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR))OK for now, but I think when PPS happens we should s/ROOTFS/PKG/ and simply use
$($(PKG)_TARGET_DIR) here.Why so? After PPS, each rootfs will still get their own private copy of target, which is what we will want them to use...
Quoted 6 lines
So, with PPS, this would evetually be something like:
TARGET_DIR = \ $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),\ $(if $(PKG),$($(PKG)_TARGET_DIR),\ $(BASE_TARGET_DIR)))
Which can be very simply
TARGET_DIR = $(if $(PKG),$($(PKG)_TARGET_DIR),$(BASE_TARGET_DIR))
if we rename ROOTFS to PKG.
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v4 9/15 fs: introduce per-rootfs TARGET_DIR variable
Arnout, All,
On 2018-03-31 20:57 +0200, Arnout Vandecappelle spake thusly:
Quoted 43 lines · Yann E. MORIN
Quoted 35 lines
Arnout, All,
On 2018-03-31 20:30 +0200, Arnout Vandecappelle spake thusly:
Quoted 18 lines · Yann E. MORIN
Quoted 17 lines
... which for now still points to the base target directory, but this is a step forward.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>--- Makefile | 4 +++- fs/common.mk | 1 + 2 files changed, 4 insertions(+), 1 deletion(-)Makefile +0
-0@@ -215,8 +215,10 @@ BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf)Quoted 10 lines
Makefile BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf) · line 221
Quoted 7 lines
Makefile +3
-1BUILD_DIR := $(BASE_DIR)/build BINARIES_DIR := $(BASE_DIR)/images +# The target directory is common to all packages, +# but there is one that is specific to each filesystem. BASE_TARGET_DIR := $(BASE_DIR)/target -TARGET_DIR = $(BASE_TARGET_DIR) +TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR))OK for now, but I think when PPS happens we should s/ROOTFS/PKG/ and simply use
$($(PKG)_TARGET_DIR) here.Why so? After PPS, each rootfs will still get their own private copy of target, which is what we will want them to use...
Quoted 6 lines
So, with PPS, this would evetually be something like:
TARGET_DIR = \ $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),\ $(if $(PKG),$($(PKG)_TARGET_DIR),\ $(BASE_TARGET_DIR)))
Quoted 5 lines
Which can be very simply
TARGET_DIR= $(if $(PKG),$($(PKG)_TARGET_DIR),$(BASE_TARGET_DIR))if we rename ROOTFS to PKG.
Ah, right. I already thought of that, but since this is not a package but rootfs, I did not see the point in naming this PKG, just for the sake of simplifying a code that will come later and is still far from ready for review yet...
But yes...
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v4 10/15 fs: new intermediate rootfs-common to gather common dependencies
In reply to Yann E. MORINBefore we can create an intermediate tarball for all filesystems, we nedd to move the common dependencies needed to generate that intermediate tarball, rather than levae those dependencies to each filesystem.
So, we introduce rootfs-common, which gathers all those common dependencies.
--- fs/common.mk | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-)
fs/common.mk +19 -4
@@ -34,6 +34,24 @@ ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE) \ USERS_TABLE = $(FS_DIR)/users_table.txt ROOTFS_USERS_TABLES = $(call qstrip,$(BR2_ROOTFS_USERS_TABLES)) +ROOTFS_COMMON_TAR = $(FS_DIR)/rootfs.common.tar + +.PHONY: rootfs-common +rootfs-common: $(ROOTFS_COMMON_TAR) + +ROOTFS_COMMON_DEPENDENCIES = \ + host-fakeroot host-makedevs \ + $(if $(PACKAGES_USERS)$(ROOTFS_USERS_TABLES),host-mkpasswd) + +.PHONY: $(ROOTFS_COMMON_TAR) +# When doing the common tarball, we're not really doing a rootfs. +$(ROOTFS_COMMON_TAR): ROOTFS= +$(ROOTFS_COMMON_TAR): $(ROOTFS_COMMON_DEPENDENCIES) target-finalize + @: + +rootfs-common-show-depends: + @echo $(ROOTFS_COMMON_DEPENDENCIES) + # Since this function will be called from within an $(eval ...) # all variable references except the arguments must be $$-quoted. define inner-rootfs
@@ -42,9 +60,7 @@ ROOTFS_$(2)_NAME = $(1) ROOTFS_$(2)_DIR = $$(FS_DIR)/$(1) ROOTFS_$(2)_TARGET_DIR = $$(BASE_TARGET_DIR) -# extra deps -ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs \ - $$(if $$(PACKAGES_USERS)$$(ROOTFS_USERS_TABLES),host-mkpasswd) +ROOTFS_$(2)_DEPENDENCIES += rootfs-common ifeq ($$(BR2_TARGET_ROOTFS_$(2)_GZIP),y) ROOTFS_$(2)_COMPRESS_EXT = .gz
@@ -77,7 +93,6 @@ endif $$(BINARIES_DIR)/rootfs.$(1): ROOTFS=$(2) $$(BINARIES_DIR)/rootfs.$(1): FAKEROOT_SCRIPT=$$(ROOTFS_$(2)_DIR)/fakeroot -$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES) @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") rm -rf $(FS_DIR) $$(ROOTFS_$(2)_DIR)
Signature · 1 line
2.14.1
PATCH v4 11/15 fs: introduce a macro for reproducible command
In reply to Yann E. MORIN--- fs/common.mk | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
fs/common.mk +7 -3
@@ -34,6 +34,12 @@ ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE) \ USERS_TABLE = $(FS_DIR)/users_table.txt ROOTFS_USERS_TABLES = $(call qstrip,$(BR2_ROOTFS_USERS_TABLES)) +ifeq ($(BR2_REPRODUCIBLE),y) +define ROOTFS_REPRODUCIBLE + find $(TARGET_DIR) -print0 | xargs -0 -r touch -hd @$(SOURCE_DATE_EPOCH) +endef +endif + ROOTFS_COMMON_TAR = $(FS_DIR)/rootfs.common.tar .PHONY: rootfs-common
@@ -120,9 +126,7 @@ endif echo $$(EXTRA_ENV) $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep)) $$(foreach hook,$$(ROOTFS_PRE_CMD_HOOKS),\ $$(call PRINTF,$$($$(hook))) >> $$(FAKEROOT_SCRIPT)$$(sep)) -ifeq ($$(BR2_REPRODUCIBLE),y) - echo "find $$(TARGET_DIR) -print0 | xargs -0 -r touch -hd @$$(SOURCE_DATE_EPOCH)" >> $$(FAKEROOT_SCRIPT) -endif + $$(call PRINTF,$$(ROOTFS_REPRODUCIBLE)) >> $$(FAKEROOT_SCRIPT) $$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT) $$(foreach hook,$$(ROOTFS_POST_CMD_HOOKS),\ $$(call PRINTF,$$($$(hook))) >> $$(FAKEROOT_SCRIPT)$$(sep))
Signature · 1 line
2.14.1
PATCH v4 12/15 fs: use a common tarball as base for the other filesystems
In reply to Yann E. MORINCurrently, some filesystems may want to tweak the content of the target directory, create special device nodes etc... This all means that:
- the content of the target directory for a specific filesystems may depend on whether another filesystem is enabled or not; for example, cpio will create a /init script or symlink and a /dev/console node;
- the filesysems can not be built in parallel, because they may change the content of the target directory wil another is being assembled.
Furthermore, the same fakeroot script is executed over-and-over-again for each filesystem, to create the device nodes, the users and their homes and files, and setting permissions...
We introduce an intermediate tarball, for which we do the full fakeroot shebang.
That tarball then serves as the base for the other filesystems, with a very simple fakeroot script that untars the common tarball, and calls the actual filesystem image generator on that.
Note that we use a very simple tar command to generate the intermediate tarball, because we are not concerned with reproducibility of the archive itself (only of the archived files).
--- fs/common.mk | 75 +++++++++++++++++++++++++++++++++++++----------------------- 1 file changed, 47 insertions(+), 28 deletions(-)
fs/common.mk +47 -28
@@ -42,6 +42,19 @@ endif ROOTFS_COMMON_TAR = $(FS_DIR)/rootfs.common.tar +# Command to create the common tarball from the base target directory. +define ROOTFS_COMMON_TAR_CMD + tar cf $(ROOTFS_COMMON_TAR) --numeric-owner \ + --exclude=$(notdir $(TARGET_DIR_WARNING_FILE)) \ + -C $(TARGET_DIR) . +endef + +# Command to extract the common tarball into the per-rootfs target directory +define ROOTFS_COMMON_UNTAR_CMD + mkdir -p $(TARGET_DIR) + tar xf $(ROOTFS_COMMON_TAR) -C $(TARGET_DIR) +endef + .PHONY: rootfs-common rootfs-common: $(ROOTFS_COMMON_TAR)
@@ -49,11 +62,39 @@ ROOTFS_COMMON_DEPENDENCIES = \ host-fakeroot host-makedevs \ $(if $(PACKAGES_USERS)$(ROOTFS_USERS_TABLES),host-mkpasswd) -.PHONY: $(ROOTFS_COMMON_TAR) # When doing the common tarball, we're not really doing a rootfs. $(ROOTFS_COMMON_TAR): ROOTFS= +$(ROOTFS_COMMON_TAR): FAKEROOT_SCRIPT=$(FS_DIR)/fakeroot.fs $(ROOTFS_COMMON_TAR): $(ROOTFS_COMMON_DEPENDENCIES) target-finalize - @: + @$(call MESSAGE,"Generating common rootfs tarball") + rm -rf $(FS_DIR) + mkdir -p $(FS_DIR) + echo '#!/bin/sh' > $(FAKEROOT_SCRIPT) + echo "set -e" >> $(FAKEROOT_SCRIPT) + echo "chown -h -R 0:0 $(TARGET_DIR)" >> $(FAKEROOT_SCRIPT) +ifneq ($(ROOTFS_USERS_TABLES),) + cat $(ROOTFS_USERS_TABLES) >> $(USERS_TABLE) +endif + $(call PRINTF,$(PACKAGES_USERS)) >> $(USERS_TABLE) + PATH=$(BR_PATH) $(TOPDIR)/support/scripts/mkusers $(USERS_TABLE) $(TARGET_DIR) >> $(FAKEROOT_SCRIPT) +ifneq ($(ROOTFS_DEVICE_TABLES),) + cat $(ROOTFS_DEVICE_TABLES) > $(FULL_DEVICE_TABLE) +ifeq ($(BR2_ROOTFS_DEVICE_CREATION_STATIC),y) + $(call PRINTF,$(PACKAGES_DEVICES_TABLE)) >> $(FULL_DEVICE_TABLE) +endif +endif + $(call PRINTF,$(PACKAGES_PERMISSIONS_TABLE)) >> $(FULL_DEVICE_TABLE) + echo "$(HOST_DIR)/bin/makedevs -d $(FULL_DEVICE_TABLE) $(TARGET_DIR)" >> $(FAKEROOT_SCRIPT) + $(foreach s,$(call qstrip,$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\ + echo "echo '$(TERM_BOLD)>>> Executing fakeroot script $(s)$(TERM_RESET)'" >> $(FAKEROOT_SCRIPT); \ + echo $(EXTRA_ENV) $(s) $(TARGET_DIR) $(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $(FAKEROOT_SCRIPT)$(sep)) + $(foreach hook,$(ROOTFS_PRE_CMD_HOOKS),\ + $(call PRINTF,$($(hook))) >> $(FAKEROOT_SCRIPT)$(sep)) + $(call PRINTF,$(ROOTFS_COMMON_TAR_CMD)) >> $(FAKEROOT_SCRIPT) + $(foreach hook,$(ROOTFS_POST_CMD_HOOKS),\ + $(call PRINTF,$($(hook))) >> $(FAKEROOT_SCRIPT)$(sep)) + chmod a+x $(FAKEROOT_SCRIPT) + PATH=$(BR_PATH) $(HOST_DIR)/bin/fakeroot -- $(FAKEROOT_SCRIPT) rootfs-common-show-depends: @echo $(ROOTFS_COMMON_DEPENDENCIES)
@@ -64,7 +105,7 @@ define inner-rootfs ROOTFS_$(2)_NAME = $(1) ROOTFS_$(2)_DIR = $$(FS_DIR)/$(1) -ROOTFS_$(2)_TARGET_DIR = $$(BASE_TARGET_DIR) +ROOTFS_$(2)_TARGET_DIR = $$(ROOTFS_$(2)_DIR)/target ROOTFS_$(2)_DEPENDENCIES += rootfs-common
@@ -101,39 +142,17 @@ $$(BINARIES_DIR)/rootfs.$(1): ROOTFS=$(2) $$(BINARIES_DIR)/rootfs.$(1): FAKEROOT_SCRIPT=$$(ROOTFS_$(2)_DIR)/fakeroot $$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES) @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)") - rm -rf $(FS_DIR) $$(ROOTFS_$(2)_DIR) - mkdir -p $(FS_DIR) $$(ROOTFS_$(2)_DIR) + rm -rf $$(ROOTFS_$(2)_DIR) + mkdir -p $$(ROOTFS_$(2)_DIR) echo '#!/bin/sh' > $$(FAKEROOT_SCRIPT) echo "set -e" >> $$(FAKEROOT_SCRIPT) + $$(call PRINTF,$$(ROOTFS_COMMON_UNTAR_CMD)) >> $$(FAKEROOT_SCRIPT) $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),\ $$(call PRINTF,$$($$(hook))) >> $$(FAKEROOT_SCRIPT)$$(sep)) - echo "chown -h -R 0:0 $$(TARGET_DIR)" >> $$(FAKEROOT_SCRIPT) -ifneq ($$(ROOTFS_USERS_TABLES),) - cat $$(ROOTFS_USERS_TABLES) >> $$(USERS_TABLE) -endif - $$(call PRINTF,$$(PACKAGES_USERS)) >> $$(USERS_TABLE) - PATH=$$(BR_PATH) $$(TOPDIR)/support/scripts/mkusers $$(USERS_TABLE) $$(TARGET_DIR) >> $$(FAKEROOT_SCRIPT) -ifneq ($$(ROOTFS_DEVICE_TABLES),) - cat $$(ROOTFS_DEVICE_TABLES) > $$(FULL_DEVICE_TABLE) -ifeq ($$(BR2_ROOTFS_DEVICE_CREATION_STATIC),y) - $$(call PRINTF,$$(PACKAGES_DEVICES_TABLE)) >> $$(FULL_DEVICE_TABLE) -endif -endif - $$(call PRINTF,$$(PACKAGES_PERMISSIONS_TABLE)) >> $$(FULL_DEVICE_TABLE) - echo "$$(HOST_DIR)/bin/makedevs -d $$(FULL_DEVICE_TABLE) $$(TARGET_DIR)" >> $$(FAKEROOT_SCRIPT) - $$(foreach s,$$(call qstrip,$$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\ - echo "echo '$$(TERM_BOLD)>>> Executing fakeroot script $$(s)$$(TERM_RESET)'" >> $$(FAKEROOT_SCRIPT); \ - echo $$(EXTRA_ENV) $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep)) - $$(foreach hook,$$(ROOTFS_PRE_CMD_HOOKS),\ - $$(call PRINTF,$$($$(hook))) >> $$(FAKEROOT_SCRIPT)$$(sep)) $$(call PRINTF,$$(ROOTFS_REPRODUCIBLE)) >> $$(FAKEROOT_SCRIPT) $$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT) - $$(foreach hook,$$(ROOTFS_POST_CMD_HOOKS),\ - $$(call PRINTF,$$($$(hook))) >> $$(FAKEROOT_SCRIPT)$$(sep)) chmod a+x $$(FAKEROOT_SCRIPT) - rm -f $$(TARGET_DIR_WARNING_FILE) PATH=$$(BR_PATH) $$(HOST_DIR)/bin/fakeroot -- $$(FAKEROOT_SCRIPT) - $$(INSTALL) -m 0644 support/misc/target-dir-warning.txt $$(TARGET_DIR_WARNING_FILE) ifneq ($$(ROOTFS_$(2)_COMPRESS_CMD),) PATH=$$(BR_PATH) $$(ROOTFS_$(2)_COMPRESS_CMD) $$@ > $$@$$(ROOTFS_$(2)_COMPRESS_EXT) endif
Signature · 1 line
2.14.1
PATCH v4 12/15 fs: use a common tarball as base for the other filesystems
Quoted 17 lines · Yann E. MORIN
Currently, some filesystems may want to tweak the content of the target directory, create special device nodes etc... This all means that:
- the content of the target directory for a specific filesystems may depend on whether another filesystem is enabled or not; for example, cpio will create a /init script or symlink and a
/dev/consolenode;- the filesysems can not be built in parallel, because they may change the content of the target directory wil another is being assembled.
Furthermore, the same fakeroot script is executed over-and-over-again for each filesystem, to create the device nodes, the users and their homes and files, and setting permissions...
We introduce an intermediate tarball, for which we do the full fakeroot shebang.
Quoted 7 lines
That tarball then serves as the base for the other filesystems, with a very simple fakeroot script that untars the common tarball, and calls the actual filesystem image generator on that.
Note that we use a very simple tar command to generate the intermediate tarball, because we are not concerned with reproducibility of the archive itself (only of the archived files).
Could be good to mention explicitly in the commit log that now theTHIS_IS_NOT_YOUR_ROOT_FILESYSTEM is "removed" by excluding it from the tarball.
Otherwise LGTM.
Regards,
Arnout
Quoted 5 lines
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>---[snip]
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v4 13/15 fs: run packages' filesystem hooks in a copy of target/
In reply to Yann E. MORINCurrently, some packages may regigter hooks to be run just before and judt after the generic tarball image is generated, because they need to prepare the filesystem for read-only or read-write operation.
However, this means that, if any of the hooks or the image generation fails, the target directory is left in a dangling, inconsistent state.
We fix that by doing a copy of target/, run the hooks on that copy, generate the generic tarball image out of that, and get rid of the copy.
This way, we can guarantee consistency of the target directory, and we can even ditch support for post-fs hooks (those that restore target/).
--- fs/common.mk | 7 +++++-- support/testing/tests/core/test_post_scripts.py | 23 +++++++++++++---------- 2 files changed, 18 insertions(+), 12 deletions(-)
fs/common.mk +5 -2
@@ -58,17 +58,20 @@ endef .PHONY: rootfs-common rootfs-common: $(ROOTFS_COMMON_TAR) +# Emulate being in a filesystem, so that we can have our own TARGET_DIR. +ROOTFS_COMMON_TARGET_DIR = $(FS_DIR)/target + ROOTFS_COMMON_DEPENDENCIES = \ host-fakeroot host-makedevs \ $(if $(PACKAGES_USERS)$(ROOTFS_USERS_TABLES),host-mkpasswd) -# When doing the common tarball, we're not really doing a rootfs. -$(ROOTFS_COMMON_TAR): ROOTFS= +$(ROOTFS_COMMON_TAR): ROOTFS=COMMON $(ROOTFS_COMMON_TAR): FAKEROOT_SCRIPT=$(FS_DIR)/fakeroot.fs $(ROOTFS_COMMON_TAR): $(ROOTFS_COMMON_DEPENDENCIES) target-finalize @$(call MESSAGE,"Generating common rootfs tarball") rm -rf $(FS_DIR) mkdir -p $(FS_DIR) + rsync -au $(BASE_TARGET_DIR)/ $(TARGET_DIR) echo '#!/bin/sh' > $(FAKEROOT_SCRIPT) echo "set -e" >> $(FAKEROOT_SCRIPT) echo "chown -h -R 0:0 $(TARGET_DIR)" >> $(FAKEROOT_SCRIPT)
support/testing/tests/core/test_post_scripts.py +13 -10
@@ -18,17 +18,17 @@ class TestPostScripts(infra.basetest.BRTest): infra.filepath("tests/core/post-fakeroot.sh"), infra.filepath("tests/core/post-image.sh")) - def check_post_log_file(self, path, what): + def check_post_log_file(self, f, what, target_dir): lines = {} - with open(path, 'rb') as csvfile: + with open(os.path.join(self.builddir, "build", f), 'rb') as csvfile: r = csv.reader(csvfile, delimiter=',') for row in r: lines[row[0]] = row[1] - self.assertEqual(lines["arg1"], os.path.join(self.builddir, what)) + self.assertEqual(lines["arg1"], what) self.assertEqual(lines["arg2"], "foobar") self.assertEqual(lines["arg3"], "baz") - self.assertEqual(lines["TARGET_DIR"], os.path.join(self.builddir, "target")) + self.assertEqual(lines["TARGET_DIR"], target_dir) self.assertEqual(lines["BUILD_DIR"], os.path.join(self.builddir, "build")) self.assertEqual(lines["HOST_DIR"], os.path.join(self.builddir, "host")) staging = os.readlink(os.path.join(self.builddir, "staging"))
@@ -37,9 +37,12 @@ class TestPostScripts(infra.basetest.BRTest): self.assertEqual(lines["BR2_CONFIG"], os.path.join(self.builddir, ".config")) def test_run(self): - f = os.path.join(self.builddir, "build", "post-build.log") - self.check_post_log_file(f, "target") - f = os.path.join(self.builddir, "build", "post-fakeroot.log") - self.check_post_log_file(f, "target") - f = os.path.join(self.builddir, "build", "post-image.log") - self.check_post_log_file(f, "images") + self.check_post_log_file("post-build.log", + os.path.join(self.builddir, "target"), + os.path.join(self.builddir, "target")) + self.check_post_log_file("post-fakeroot.log", + os.path.join(self.builddir, "build/buildroot-fs/target"), + os.path.join(self.builddir, "build/buildroot-fs/target")) + self.check_post_log_file("post-image.log", + os.path.join(self.builddir, "images"), + os.path.join(self.builddir, "target"))
Signature · 1 line
2.14.1
PATCH v4 13/15 fs: run packages' filesystem hooks in a copy of target/
Quoted 35 lines · Yann E. MORIN
Currently, some packages may regigter hooks to be run just before and judt after the generic tarball image is generated, because they need to prepare the filesystem for read-only or read-write operation.
However, this means that, if any of the hooks or the image generation fails, the target directory is left in a dangling, inconsistent state.
We fix that by doing a copy of target/, run the hooks on that copy, generate the generic tarball image out of that, and get rid of the copy.
This way, we can guarantee consistency of the target directory, and we can even ditch support for post-fs hooks (those that restore target/).
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>--- fs/common.mk | 7 +++++-- support/testing/tests/core/test_post_scripts.py | 23 +++++++++++++---------- 2 files changed, 18 insertions(+), 12 deletions(-)fs/common.mk +3
-0@@ -58,17 +58,20 @@ endef .PHONY: rootfs-common rootfs-common: $(ROOTFS_COMMON_TAR) +# Emulate being in a filesystem, so that we can have our own TARGET_DIR. +ROOTFS_COMMON_TARGET_DIR = $(FS_DIR)/target + ROOTFS_COMMON_DEPENDENCIES = \ host-fakeroot host-makedevs \ $(if $(PACKAGES_USERS)$(ROOTFS_USERS_TABLES),host-mkpasswd)
fs/common.mk endef · line 74
Quoted 10 lines
fs/common.mk +2
-2-# When doing the common tarball, we're not really doing a rootfs. -$(ROOTFS_COMMON_TAR): ROOTFS= +$(ROOTFS_COMMON_TAR): ROOTFS=COMMON $(ROOTFS_COMMON_TAR): FAKEROOT_SCRIPT=$(FS_DIR)/fakeroot.fs $(ROOTFS_COMMON_TAR): $(ROOTFS_COMMON_DEPENDENCIES) target-finalize @$(call MESSAGE,"Generating common rootfs tarball") rm -rf $(FS_DIR) mkdir -p $(FS_DIR) + rsync -au $(BASE_TARGET_DIR)/ $(TARGET_DIR)
This would benefit a lot from using the --link-dest= that we will also use in
PPS, in my opinion.
Regards,
Arnout
Quoted 3 lines
fs/common.mk +0
-0echo '#!/bin/sh' > $(FAKEROOT_SCRIPT) echo "set -e" >> $(FAKEROOT_SCRIPT) echo "chown -h -R 0:0 $(TARGET_DIR)" >> $(FAKEROOT_SCRIPT)
[snip]
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v4 13/15 fs: run packages' filesystem hooks in a copy of target/
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:
Hi,
Quoted 5 lines
fs/common.mk endef · line 74
Quoted 2 lines
mkdir -p $(FS_DIR)
fs/common.mk +1
-0+ rsync -au $(BASE_TARGET_DIR)/ $(TARGET_DIR)This would benefit a lot from using the --link-dest= that we will also use in
PPS, in my opinion.
Wouldn't hard links be potentially dangerous - E.G. in case a script appends to a file.
As I mentioned in my comments to patch 15, we could as well use cp instead of rsync as we throw away the copy after each build.
Signature · 1 line
Bye, Peter Korsgaard
PATCH v4 13/15 fs: run packages' filesystem hooks in a copy of target/
Quoted 4 lines · Peter Korsgaard
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:
Hi,
Quoted 8 lines
Quoted 5 lines
fs/common.mk endef · line 74
Quoted 2 lines
mkdir -p $(FS_DIR)
fs/common.mk +1
-0+ rsync -au $(BASE_TARGET_DIR)/ $(TARGET_DIR)This would benefit a lot from using the --link-dest= that we will also use in
PPS, in my opinion.Wouldn't hard links be potentially dangerous - E.G. in case a script appends to a file.
If this is a script that runs in the context of building a package, then the
hard link does exactly what we want.
If it is a script that runs in the context of rootfs generation, you have a point.
If your target is a bit large and you have several filesystems to support, I
expect that making all these copies is really going to hit you in build time for a simple package rebuild. Of course, we can fix that problem when it happens.
Regards,
Arnout
Quoted 2 lines
As I mentioned in my comments to patch 15, we could as well use cp instead of rsync as we throw away the copy after each build.
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
PATCH v4 13/15 fs: run packages' filesystem hooks in a copy of target/
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle <[email protected]> writes:
Wouldn't hard links be potentially dangerous - E.G. in case a script appends to a file.
If this is a script that runs in the context of building a package, then the
hard link does exactly what we want.If it is a script that runs in the context of rootfs generation, you have a point.
Ok.
Quoted 3 lines
If your target is a bit large and you have several filesystems to support, I
expect that making all these copies is really going to hit you in build time for a simple package rebuild. Of course, we can fix that problem when it happens.
Yes, I'm also a bit scared of these extra steps, but I understand the reason for it.
One of the most common extra rootfs options is probably .tar, and for that an optimization could be to directly create the final .tar and use that for the other filesystem types.
Signature · 1 line
Bye, Peter Korsgaard
PATCH v4 14/15 fs: get rid of package-provided post-fs hooks
In reply to Yann E. MORINNow that the pre-fs ones are run on a transient copy of target/, the post-fs hooks are no longer needed because we no longer need to restore the target/ directory as it is only a internal copy.
Remove support for the post-fs hooks, and update the sole package using them.
Unlike for the pre-/post-gen hooks, we do not add a legacy check because this was mostly a purely-internal detail that was never really exposed nor documented.
--- fs/common.mk | 2 -- package/pkg-generic.mk | 2 -- package/skeleton-init-systemd/skeleton-init-systemd.mk | 6 ------ 3 files changed, 10 deletions(-)
fs/common.mk +0 -2
@@ -94,8 +94,6 @@ endif $(foreach hook,$(ROOTFS_PRE_CMD_HOOKS),\ $(call PRINTF,$($(hook))) >> $(FAKEROOT_SCRIPT)$(sep)) $(call PRINTF,$(ROOTFS_COMMON_TAR_CMD)) >> $(FAKEROOT_SCRIPT) - $(foreach hook,$(ROOTFS_POST_CMD_HOOKS),\ - $(call PRINTF,$($(hook))) >> $(FAKEROOT_SCRIPT)$(sep)) chmod a+x $(FAKEROOT_SCRIPT) PATH=$(BR_PATH) $(HOST_DIR)/bin/fakeroot -- $(FAKEROOT_SCRIPT)
package/pkg-generic.mk +0 -2
@@ -644,7 +644,6 @@ $(2)_PRE_LEGAL_INFO_HOOKS ?= $(2)_POST_LEGAL_INFO_HOOKS ?= $(2)_TARGET_FINALIZE_HOOKS ?= $(2)_ROOTFS_PRE_CMD_HOOKS ?= -$(2)_ROOTFS_POST_CMD_HOOKS ?= # human-friendly targets and target sequencing $(1): $(1)-install
@@ -954,7 +953,6 @@ PACKAGES_USERS += $$($(2)_USERS)$$(sep) endif TARGET_FINALIZE_HOOKS += $$($(2)_TARGET_FINALIZE_HOOKS) ROOTFS_PRE_CMD_HOOKS += $$($(2)_ROOTFS_PRE_CMD_HOOKS) -ROOTFS_POST_CMD_HOOKS += $$($(2)_ROOTFS_POST_CMD_HOOKS) ifeq ($$($(2)_SITE_METHOD),svn) DL_TOOLS_DEPENDENCIES += svn
package/skeleton-init-systemd/skeleton-init-systemd.mk +0 -6
@@ -55,12 +55,6 @@ define SKELETON_INIT_SYSTEMD_PRE_ROOTFS_VAR endef SKELETON_INIT_SYSTEMD_ROOTFS_PRE_CMD_HOOKS += SKELETON_INIT_SYSTEMD_PRE_ROOTFS_VAR -define SKELETON_INIT_SYSTEMD_POST_ROOTFS_VAR - rm -rf $(TARGET_DIR)/var - mv $(TARGET_DIR)/usr/share/factory/var $(TARGET_DIR)/var -endef -SKELETON_INIT_SYSTEMD_ROOTFS_POST_CMD_HOOKS += SKELETON_INIT_SYSTEMD_POST_ROOTFS_VAR - endif define SKELETON_INIT_SYSTEMD_INSTALL_TARGET_CMDS
Signature · 1 line
2.14.1
PATCH v4 15/15 fs: remove intermediate artefacts
In reply to Yann E. MORINEach of the intermediate, per-rootfs target directories, as well as the intermediate tarball, can take quite some place, and is mostly a duplication of what's already in target/. The only delta, if any, would be the tweaks made by the filesystem image generations, but those tweaks are most probably only meaningfull when seen as root.
We normally do not remove intermediate files, but those can be quite large, and are not directly usable by, nor accessible to the user. So, get rid of them once the filesystem has been generated.
This does not need to be done in fakeroot.
--- Makefile | 1 + fs/common.mk | 2 ++ 2 files changed, 3 insertions(+)
Makefile +1 -0
@@ -756,6 +756,7 @@ endif .PHONY: target-post-image target-post-image: $(TARGETS_ROOTFS) target-finalize + @rm -f $(ROOTFS_COMMON_TAR) @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \ $(call MESSAGE,"Executing post-image script $(s)"); \ $(EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
fs/common.mk +2 -0
@@ -96,6 +96,7 @@ endif $(call PRINTF,$(ROOTFS_COMMON_TAR_CMD)) >> $(FAKEROOT_SCRIPT) chmod a+x $(FAKEROOT_SCRIPT) PATH=$(BR_PATH) $(HOST_DIR)/bin/fakeroot -- $(FAKEROOT_SCRIPT) + $(Q)rm -rf $(TARGET_DIR) rootfs-common-show-depends: @echo $(ROOTFS_COMMON_DEPENDENCIES)
@@ -154,6 +155,7 @@ $$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES) $$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT) chmod a+x $$(FAKEROOT_SCRIPT) PATH=$$(BR_PATH) $$(HOST_DIR)/bin/fakeroot -- $$(FAKEROOT_SCRIPT) + $(Q)rm -rf $$(TARGET_DIR) ifneq ($$(ROOTFS_$(2)_COMPRESS_CMD),) PATH=$$(BR_PATH) $$(ROOTFS_$(2)_COMPRESS_CMD) $$@ > $$@$$(ROOTFS_$(2)_COMPRESS_EXT) endif
Signature · 1 line
2.14.1
PATCH v4 15/15 fs: remove intermediate artefacts
Quoted 40 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Each of the intermediate, per-rootfs target directories, as well as the intermediate tarball, can take quite some place, and is mostly a duplication of what's already in target/. The only delta, if any, would be the tweaks made by the filesystem image generations, but those tweaks are most probably only meaningfull when seen as root.
We normally do not remove intermediate files, but those can be quite large, and are not directly usable by, nor accessible to the user. So, get rid of them once the filesystem has been generated.
This does not need to be done in fakeroot.
Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>Tested-by Matt Weber <[email protected]>--- Makefile | 1 + fs/common.mk | 2 ++ 2 files changed, 3 insertions(+)Makefile +1
-0@@ -756,6 +756,7 @@ endif .PHONY: target-post-image target-post-image: $(TARGETS_ROOTFS) target-finalize + @rm -f $(ROOTFS_COMMON_TAR) @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \ $(call MESSAGE,"Executing post-image script $(s)"); \ $(EXTRA_ENV) $(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))fs/common.mk +0
-0
fs/common.mk endif · line 99
Quoted 4 lines
fs/common.mk +1
-0@@ -96,6 +96,7 @@ endif $(call PRINTF,$(ROOTFS_COMMON_TAR_CMD)) >> $(FAKEROOT_SCRIPT) chmod a+x $(FAKEROOT_SCRIPT) PATH=$(BR_PATH) $(HOST_DIR)/bin/fakeroot -- $(FAKEROOT_SCRIPT) + $(Q)rm -rf $(TARGET_DIR)
This throws away the output/target copy on each invocation, so there is not much sense in using rsync to create it and we could just use cp -a, but OK, that can be done later.
Signature · 1 line
Bye, Peter Korsgaard
v4 cover fs: support parallel filesystems build
In reply to Yann E. MORINQuoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Hello All!
TL;DR: filsystems are not parallel-safe, so we generate an intermediate tarball from target/, which then serves as input for the various filesystems; we also run pre-gen hooks under fakeroot, and get rid of post-fs package-provided hooks.
Committed series with the minor fixups pointed out by Arnout and me, thanks.
Signature · 1 line
Bye, Peter Korsgaard