Use the kernel from sstate when building fitImages
v4 cover Use the kernel from sstate when building fitImages
Compared to v3, only one space changes. Unfortunately it is an important white space. The diff between v3 and v4 is:
meta/classes-recipe/kernel-fitimage.bbclass +1 -1
@@ -65,7 +65,7 @@ python __anonymous () { fit_assemble_task = 'do_deploy_fitimage_unbundled' bb.build.addtask(fit_assemble_task, 'do_build', 'do_deploy', d) bb.build.addtask(fit_assemble_task + '_setscene', '', '', d) - d.appendVar('SSTATETASKS', fit_assemble_task) + d.appendVar('SSTATETASKS', ' ' + fit_assemble_task)
Summary:
If the KERNEL_IMAGETYPES(S) contains fitImage, the kernel is always rebuilt when something changes in the initramfs, which happens frequently.
This is even worse if the build runs from an empty TMPDIR. The kernel re-build starts by fetching the large kernel git repository and gets recompiled from scratch.
This cannot be improved if INITRAMFS_IMAGE_BUNDLE = "1". If the kernel Makefile is needed to generate the initramfs bundle the kernel build folder is required.
But for a build configuration with INITRAMFS_IMAGE_BUNDLE = "" the build folder is not needed. Creating the initramfs bundle requires:
linux.bin, DTBs and the initramfs which are available in the deploy directory. That means creating the fitImage can be done with artifacts which are already cached by the sstate.
Testing:
- oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg \
reproducible -T yocto-mirrors -T machine -T toolchain-user \
-T toolchain-system
- Build linux-yocto with empty TMPDIR and empty sstate, followed by a build
with populated sstate.
- Clean build config:
KERNEL_IMAGETYPE = "Image"
KERNEL_IMAGETYPES += " fitImage "
KERNEL_CLASSES = " kernel-fitimage "
IMAGE_FSTYPES += "cpio.gz"
INITRAMFS_IMAGE = "core-image-minimal"
IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""
UBOOT_RD_LOADADDRESS = "0x88000000"
UBOOT_RD_ENTRYPOINT = "0x88000000"
UBOOT_LOADADDRESS = "0x80080000"
UBOOT_ENTRYPOINT = "0x80080000"
FIT_DESC = "A model description"
FOO_VAR = "1"
INHERIT += "image-buildinfo"
IMAGE_BUILDINFO_VARS:append = " FOO_VAR"
- bitbake linux-yocto
- Append the following and rebuild with sstate:
FOO_VAR = "2"
TMPDIR = "${TOPDIR}/tmp-2" - bitbake linux-yocto
- Check the log files:
- tmp/log/cooker/qemux86-64 contains:
linux-yocto-6.6.43+git-r0: task do_deploy: Succeeded linux-yocto-6.6.43+git-r0: task do_deploy_fitimage_unbundled: Succeeded
- tmp-2/log/cooker/qemux86-64 contains:
linux-yocto-6.6.43+git-r0: task do_deploy_setscene: Succeeded linux-yocto-6.6.43+git-r0: task do_deploy_fitimage_unbundled: Succeeded
- To re-view the changes in the task dependencies, the bash script which was sent as a separate (html formated) e-mail with v3 has been used. It confirms:
- For builds with fitImage and unbundled initramfs:
- the do_deploy_fitimage_unbundled task runs after do_deploy
- do_assemble_fitimage_initramfs is not executed
- do_bundle_initramfs is not executed
- It works for fitImage in KERNEL_IMAGETYPE as well as for fitImage in KERNEL_IMAGETYPES
- For builds with fitImage and bundled initramfs: No changes
- For builds with fitImage, without initramfs:
- do_assemble_fitimage_initramfs is not executed
- do_bundle_initramfs is not executed
Adrian Freihofer (4):
kernel: refactor linux compression
kernel-fitimage: refactor fitimage_assemble
kernel: refactor fitimage
kernel-fitimage: run unbundled fitimage after deploy
meta/classes-recipe/image.bbclass | 12 +- meta/classes-recipe/kernel-fitimage.bbclass | 152 +++++++++++++------- meta/classes-recipe/kernel-uboot.bbclass | 1 + meta/classes-recipe/kernel.bbclass | 32 +++-- 4 files changed, 130 insertions(+), 67 deletions(-)
Signature · 1 line
2.46.0
PATCH v4 1/4 kernel: refactor linux compression
From: Adrian Freihofer <[email protected]>
This is a step to avoid unecessary kernel rebuilds when the initramfs changes but not the kernel itself. In case of an unbundled fitImage the kernel should be staged in the sstate instead of re-compiled.
The uboot_prep_kimage function generates the kernel binary which gets included into the fitImage. It also defines the variable linux_comp which is used by the fitimage_assemble function later on.
To make the fitimage_assemble function working with kernel binaries from sstate it must be decoupled from calling uboot_prep_kimage in ${B} by:
- refactoring uboot_prep_kimage to write the result into files (linux.bin, linux.comp).
- calling the uboot_prep_kimage before the artifacts are used or cached.
- refactoring the fitimage_assemble function to use files rather than calling the uboot_prep_kimage function which depends on ${B}.
--- meta/classes-recipe/kernel-fitimage.bbclass | 5 ++--- meta/classes-recipe/kernel-uboot.bbclass | 1 + 2 files changed, 3 insertions(+), 3 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +2 -3
@@ -579,9 +579,7 @@ fitimage_assemble() { # Step 1: Prepare a kernel image section. # fitimage_emit_section_maint $1 imagestart - - uboot_prep_kimage - fitimage_emit_section_kernel $1 $kernelcount linux.bin "$linux_comp" + fitimage_emit_section_kernel $1 $kernelcount linux.bin "$(cat linux.comp)" # # Step 2: Prepare a DTB image section
@@ -764,6 +762,7 @@ fitimage_assemble() { do_assemble_fitimage() { if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then cd ${B} + uboot_prep_kimage fitimage_assemble fit-image.its fitImage-none "" if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then ln -sf fitImage-none ${B}/${KERNEL_OUTPUT_DIR}/fitImage
meta/classes-recipe/kernel-uboot.bbclass +1 -0
@@ -45,5 +45,6 @@ uboot_prep_kimage() { mv -f "linux.bin${linux_suffix}" linux.bin fi + [ -n "${vmlinux_path}" ] && echo "${linux_comp}" > linux.comp echo "${linux_comp}" }
Signature · 1 line
2.46.0
PATCH v4 2/4 kernel-fitimage: refactor fitimage_assemble
In reply to Adrian FreihoferFrom: Adrian Freihofer <[email protected]>
Make the fitimage_assemble function usable with absolute paths for the generated its and the fitImage file.
Later on this will allow to take the linux.bin and the DTB files from the sstated deploy derectory and write the generated files to a separate folder with independent sstate-cache handling.
--- meta/classes-recipe/kernel-fitimage.bbclass | 32 +++++++++++++++------ 1 file changed, 24 insertions(+), 8 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +24 -8
@@ -169,8 +169,9 @@ fitimage_emit_section_kernel() { ENTRYPOINT="${UBOOT_ENTRYPOINT}" if [ -n "${UBOOT_ENTRYSYMBOL}" ]; then + kernel_base=$(basename $3) ENTRYPOINT=`${HOST_PREFIX}nm vmlinux | \ - awk '$3=="${UBOOT_ENTRYSYMBOL}" {print "0x"$1;exit}'` + awk '$kernel_base=="${UBOOT_ENTRYSYMBOL}" {print "0x"$1;exit}'` fi cat << EOF >> $1
@@ -567,7 +568,7 @@ fitimage_assemble() { setupcount="" bootscr_id="" default_dtb_image="" - rm -f $1 arch/${ARCH}/boot/$2 + rm -f "$1" "arch/${ARCH}/boot/$(basename $2)" if [ -n "${UBOOT_SIGN_IMG_KEYNAME}" -a "${UBOOT_SIGN_KEYNAME}" = "${UBOOT_SIGN_IMG_KEYNAME}" ]; then bbfatal "Keys used to sign images and configuration nodes must be different."
@@ -608,6 +609,17 @@ fitimage_assemble() { DTB=`basename $DTB` fi + # Find DTBs without sub-folders when running in deploy folder + if [ ! -e "$DTB_PATH" ]; then + DTB=$(basename $DTB) + DTB_PATH=$(readlink -f $DTB) + fi + + # Fail as early as possible if there is still no DTB file found + if [ ! -e "$DTB_PATH" ]; then + bberror "Cannot find the DTB file at $DTB_PATH" + fi + # Set the default dtb image if it exists in the devicetree. if [ "${FIT_CONF_DEFAULT_DTB}" = "$DTB" ];then default_dtb_image=$(echo "$DTB" | tr '/' '_')
@@ -658,9 +670,8 @@ fitimage_assemble() { if [ -n "${UBOOT_ENV}" ] && [ -d "${STAGING_DIR_HOST}/boot" ]; then if [ -e "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY}" ]; then - cp ${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY} ${B} bootscr_id="${UBOOT_ENV_BINARY}" - fitimage_emit_section_boot_script $1 "$bootscr_id" ${UBOOT_ENV_BINARY} + fitimage_emit_section_boot_script $1 "$bootscr_id" "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY}" else bbwarn "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY} not found." fi
@@ -669,9 +680,14 @@ fitimage_assemble() { # # Step 4: Prepare a setup section. (For x86) # + # Run from kernel build folder (bundled mode) if [ -e ${KERNEL_OUTPUT_DIR}/setup.bin ]; then setupcount=1 fitimage_emit_section_setup $1 $setupcount ${KERNEL_OUTPUT_DIR}/setup.bin + # Run from deploy folder (unbundled mode) + elif [ -e setup.bin ]; then + setupcount=1 + fitimage_emit_section_setup $1 $setupcount "$(readlink -f setup.bin)" fi #
@@ -744,8 +760,8 @@ fitimage_assemble() { # ${UBOOT_MKIMAGE} \ ${@'-D "${UBOOT_MKIMAGE_DTCOPTS}"' if len('${UBOOT_MKIMAGE_DTCOPTS}') else ''} \ - -f $1 \ - ${KERNEL_OUTPUT_DIR}/$2 + -f "$1" \ + "$2" # # Step 8: Sign the image
@@ -754,7 +770,7 @@ fitimage_assemble() { ${UBOOT_MKIMAGE_SIGN} \ ${@'-D "${UBOOT_MKIMAGE_DTCOPTS}"' if len('${UBOOT_MKIMAGE_DTCOPTS}') else ''} \ -F -k "${UBOOT_SIGN_KEYDIR}" \ - -r ${KERNEL_OUTPUT_DIR}/$2 \ + -r "$2" \ ${UBOOT_MKIMAGE_SIGN_ARGS} fi }
@@ -763,7 +779,7 @@ do_assemble_fitimage() { if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then cd ${B} uboot_prep_kimage - fitimage_assemble fit-image.its fitImage-none "" + fitimage_assemble fit-image.its ${KERNEL_OUTPUT_DIR}/fitImage-none "" if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then ln -sf fitImage-none ${B}/${KERNEL_OUTPUT_DIR}/fitImage fi
Signature · 1 line
2.46.0
PATCH v4 3/4 kernel: refactor fitimage
In reply to Adrian FreihoferFrom: Adrian Freihofer <[email protected]>
* do_bundle_initramfs only when needed
With INITRAMFS_IMAGE_BUNDLE = "1" the kernel Makefile is used to bundle the kernel binary and the initramfs. This works only when the build folder of the kernel is available. Building with sstate but with an empty TMPDIR requires to rebuild the kernel from scratch whenever something in the initramfs changes.
With INITRAMFS_IMAGE_BUNDLE = "" the fitImage generation is basically independent from the kernel Makefile and could therefore work with kernel binaries and tools provided by the sstate. But the dependency on the do_bundle_initramfs tasks does not allow this. However, if the INTIRAMFS_IMAGE is set but the INITRAMFS_IMAGE_BUNDLE is set to 0 the do_bundle_initramfs does nothing. There is no real argument for running this tasks.
As a first step towards getting the kernel from sstate when building an unbundled fitImage, the task dependencies need to be simplified. The do_bundle_initramfs task is now scheduled only when needed.
Removing the do_bundle_initramfs also removes these task dependencies do_deploy --> do_bundle_initramfs --> do_install. But the dependency do_deploy --> do_install must be kept because the do_deploy task needs the build folder when deploying the kernel artifacts. Therefore the do_install depedency is added.
There is also a dependendency like do_deploy --> do_populate_sysroot --> do_install. However since the do_populate_sysroot task is sstate cached, the build folder is not available for the do_deploy task if the do_populate_sysroot setscene tasks runs.
* If KERNEL_IMAGETYPES does not contain fitImage and INITRAMFS_IMAGE is not set, the do_assemble_fitimage_initramfs ends up as an empty task. Add the do_assemble_fitimage_initramfs only if a fitImage with initramfs needs to be built.
* cd in bitbake task is easy to break e.g. by bbappend. Replace the cd by defining the task's dirs variable flag.
* Add 3 comments which are helpful for the next commits.
* Since the tasks and task dependencies are anyway created by an anonymous function, align and make the DEPENDS code more specific too.
This refactoring is not very valuable on its own. But it simplifies the task dependencies which is helpful for the next commit.
--- meta/classes-recipe/kernel-fitimage.bbclass | 57 ++++++++++++--------- meta/classes-recipe/kernel.bbclass | 29 ++++++----- 2 files changed, 50 insertions(+), 36 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +33 -24
@@ -26,20 +26,8 @@ def get_fit_replacement_type(d): return replacementtype KERNEL_IMAGETYPE_REPLACEMENT ?= "${@get_fit_replacement_type(d)}" -DEPENDS:append = " ${@'u-boot-tools-native dtc-native' if 'fitImage' in (d.getVar('KERNEL_IMAGETYPES') or '').split() else ''}" python __anonymous () { - # Override KERNEL_IMAGETYPE_FOR_MAKE variable, which is internal - # to kernel.bbclass . We have to override it, since we pack zImage - # (at least for now) into the fitImage . - typeformake = d.getVar("KERNEL_IMAGETYPE_FOR_MAKE") or "" - if 'fitImage' in typeformake.split(): - d.setVar('KERNEL_IMAGETYPE_FOR_MAKE', typeformake.replace('fitImage', d.getVar('KERNEL_IMAGETYPE_REPLACEMENT'))) - - image = d.getVar('INITRAMFS_IMAGE') - if image: - d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete') - ubootenv = d.getVar('UBOOT_ENV') if ubootenv: d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/bootloader:do_populate_sysroot')
@@ -48,8 +36,31 @@ python __anonymous () { providerdtb = d.getVar("PREFERRED_PROVIDER_virtual/dtb") if providerdtb: d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/dtb:do_populate_sysroot') - d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' virtual/dtb:do_populate_sysroot') d.setVar('EXTERNAL_KERNEL_DEVICETREE', "${RECIPE_SYSROOT}/boot/devicetree") + + typeformake = d.getVar("KERNEL_IMAGETYPE_FOR_MAKE") or "" + if 'fitImage' in typeformake.split(): + # Override KERNEL_IMAGETYPE_FOR_MAKE variable, which is internal + # to kernel.bbclass . We have to override it, since we pack zImage + # (at least for now) into the fitImage . + d.setVar('KERNEL_IMAGETYPE_FOR_MAKE', typeformake.replace('fitImage', d.getVar('KERNEL_IMAGETYPE_REPLACEMENT'))) + + # mkimage and dtc are required by the fitimage_assemble function + d.appendVarFlag('do_assemble_fitimage', 'depends', + ' u-boot-tools-native:do_populate_sysroot dtc-native:do_populate_sysroot') + + initramfs_image = d.getVar('INITRAMFS_IMAGE') + bundled = bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')) + if initramfs_image: + if bundled: + bb.build.addtask('do_assemble_fitimage_initramfs', 'do_deploy', 'do_bundle_initramfs', d) + else: + bb.build.addtask('do_assemble_fitimage_initramfs', 'do_deploy', 'do_install', d) + + d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', + ' u-boot-tools-native:do_populate_sysroot dtc-native:do_populate_sysroot ${INITRAMFS_IMAGE}:do_image_complete') + if providerdtb: + d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' virtual/dtb:do_populate_sysroot') }
@@ -797,19 +808,14 @@ do_install:append() { } do_assemble_fitimage_initramfs() { - if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage" && \ - test -n "${INITRAMFS_IMAGE}" ; then - cd ${B} - if [ "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then - fitimage_assemble fit-image-${INITRAMFS_IMAGE}.its fitImage-bundle "" - ln -sf fitImage-bundle ${B}/${KERNEL_OUTPUT_DIR}/fitImage - else - fitimage_assemble fit-image-${INITRAMFS_IMAGE}.its fitImage-${INITRAMFS_IMAGE} 1 - fi + if [ "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then + fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage-bundle" "" + ln -sf fitImage-bundle "${B}/${KERNEL_OUTPUT_DIR}/fitImage" + else + fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage-${INITRAMFS_IMAGE}" 1 fi } - -addtask assemble_fitimage_initramfs before do_deploy after do_bundle_initramfs +do_assemble_fitimage_initramfs[dirs] = "${B}" do_kernel_generate_rsa_keys() { if [ "${UBOOT_SIGN_ENABLE}" = "0" ] && [ "${FIT_GENERATE_KEYS}" = "1" ]; then
@@ -864,6 +870,7 @@ kernel_do_deploy:append() { if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then + # deploy the artifacts of do_assemble_fitimage bbnote "Copying fit-image.its source file..." install -m 0644 ${B}/fit-image.its "$deployDir/fitImage-its-${KERNEL_FIT_NAME}.its" if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then
@@ -878,12 +885,14 @@ kernel_do_deploy:append() { fi if [ -n "${INITRAMFS_IMAGE}" ]; then + # deploy the artifacts of do_assemble_fitimage_initramfs for bundled as well as un-bundled mode bbnote "Copying fit-image-${INITRAMFS_IMAGE}.its source file..." install -m 0644 ${B}/fit-image-${INITRAMFS_IMAGE}.its "$deployDir/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its" if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then ln -snf fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its "$deployDir/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_LINK_NAME}" fi + # deploy the artifacts of do_assemble_fitimage_initramfs for bundled mode only if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then bbnote "Copying fitImage-${INITRAMFS_IMAGE} file..." install -m 0644 ${B}/${KERNEL_OUTPUT_DIR}/fitImage-${INITRAMFS_IMAGE} "$deployDir/fitImage-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}${KERNEL_FIT_BIN_EXT}"
meta/classes-recipe/kernel.bbclass +17 -12
@@ -139,18 +139,23 @@ fi set -e """ % (type, type, type)) - image = d.getVar('INITRAMFS_IMAGE') - # If the INTIRAMFS_IMAGE is set but the INITRAMFS_IMAGE_BUNDLE is set to 0, - # the do_bundle_initramfs does nothing, but the INITRAMFS_IMAGE is built - # standalone for use by wic and other tools. if image: - if d.getVar('INITRAMFS_MULTICONFIG'): - d.appendVarFlag('do_bundle_initramfs', 'mcdepends', ' mc::${INITRAMFS_MULTICONFIG}:${INITRAMFS_IMAGE}:do_image_complete') + # If the INTIRAMFS_IMAGE is set but the INITRAMFS_IMAGE_BUNDLE is set to 0, + # the do_bundle_initramfs is not needed, but the INITRAMFS_IMAGE is built + # standalone for use by wic and other tools. + def add_initramfs_dep_task(initramfs_dep_task, d): + if d.getVar('INITRAMFS_MULTICONFIG'): + d.appendVarFlag(initramfs_dep_task, 'mcdepends', ' mc::${INITRAMFS_MULTICONFIG}:${INITRAMFS_IMAGE}:do_image_complete') + else: + d.appendVarFlag(initramfs_dep_task, 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete') + + if bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')): + bb.build.addtask('do_bundle_initramfs', 'do_deploy', 'do_install', d) + add_initramfs_dep_task('do_bundle_initramfs', d) + bb.build.addtask('do_transform_bundled_initramfs', 'do_deploy', 'do_bundle_initramfs', d) else: - d.appendVarFlag('do_bundle_initramfs', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete') - if image and bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')): - bb.build.addtask('do_transform_bundled_initramfs', 'do_deploy', 'do_bundle_initramfs', d) + add_initramfs_dep_task('do_deploy', d) # NOTE: setting INITRAMFS_TASK is for backward compatibility # The preferred method is to set INITRAMFS_IMAGE, because
@@ -330,6 +335,8 @@ do_bundle_initramfs () { mv -f ${KERNEL_OUTPUT_DIR}/$imageType.bak ${KERNEL_OUTPUT_DIR}/$imageType fi done + else + bbwarn "Calling do_bundle_initramfs with INITRAMFS_IMAGE_BUNDLE=1 is deprecated" fi } do_bundle_initramfs[dirs] = "${B}"
@@ -350,8 +357,6 @@ python do_devshell:prepend () { os.environ["LDFLAGS"] = '' } -addtask bundle_initramfs after do_install before do_deploy - KERNEL_DEBUG_TIMESTAMPS ??= "0" kernel_do_compile() {
@@ -873,7 +878,7 @@ kernel_do_deploy() { # ensure we get the right values for both do_deploy[prefuncs] += "read_subpackage_metadata" -addtask deploy after do_populate_sysroot do_packagedata +addtask deploy after do_install do_populate_sysroot do_packagedata EXPORT_FUNCTIONS do_deploy
Signature · 1 line
2.46.0
Re: [PATCH 3/4] kernel: refactor fitimage
Quoted 264 lines · Adrian Freihofer
From: Adrian Freihofer <[email protected]>
*
do_bundle_initramfsonly when needed
WithINITRAMFS_IMAGE_BUNDLE= "1" the kernelMakefileis used to bundle the kernel binary and the initramfs. This works only when the
build folder of the kernel is available. Building with sstate but with
an empty TMPDIR requires to rebuild the kernel from scratch whenever
something in the initramfs changes.With
INITRAMFS_IMAGE_BUNDLE= "" the fitImage generation is basically
independent from the kernelMakefileand could therefore work with kernel binaries and tools provided by the sstate. But the dependency
on thedo_bundle_initramfstasks does not allow this. However, if the
INTIRAMFS_IMAGEis set but theINITRAMFS_IMAGE_BUNDLEis set to 0 the
do_bundle_initramfsdoes nothing. There is no real argument for running this tasks.As a first step towards getting the kernel from sstate when building
an unbundled fitImage, the task dependencies need to be simplified. Thedo_bundle_initramfstask is now scheduled only when needed.Removing the
do_bundle_initramfsalso removes these task dependencies
do_deploy-->do_bundle_initramfs-->do_install. But the dependency
do_deploy-->do_installmust be kept because thedo_deploytask needs
the build folder when deploying the kernel artifacts. Therefore thedo_installdepedency is added.
There is also a dependendency likedo_deploy-->do_populate_sysroot
-->do_install. However since thedo_populate_sysroottask is sstate
cached, the build folder is not available for thedo_deploytask if thedo_populate_sysrootsetscene tasks runs.* If
KERNEL_IMAGETYPESdoes not contain fitImage andINITRAMFS_IMAGEis
not set, thedo_assemble_fitimage_initramfsends up as an empty task.
Add thedo_assemble_fitimage_initramfsonly if a fitImage with initramfs needs to be built.* cd in bitbake task is easy to break e.g. by bbappend. Replace the cd
by defining the task's dirs variable flag.* Add 3 comments which are helpful for the next commits.
* Since the tasks and task dependencies are anyway created by an anonymous function, align and make the DEPENDS code more specific too.
This refactoring is not very valuable on its own. But it simplifies the
task dependencies which is helpful for the next commit.Signed-off-by Adrian Freihofer <[email protected]>---
meta/classes-recipe/kernel-fitimage.bbclass | 57 ++++++++++++------- --
meta/classes-recipe/kernel.bbclass | 29 ++++++-----2 files changed, 50 insertions(+), 36 deletions(-)meta/classes-recipe/kernel-fitimage.bbclass +0
-0@@ -26,20 +26,8 @@ def get_fit_replacement_type(d):return replacementtype
KERNEL_IMAGETYPE_REPLACEMENT ?= "${@get_fit_replacement_type(d)}" -DEPENDS:append = " ${@'u-boot-tools-native dtc-native' if 'fitImage' in (d.getVar('KERNEL_IMAGETYPES')or '').split() else ''}"
python__anonymous() {
- # OverrideKERNEL_IMAGETYPE_FOR_MAKEvariable, which is internal - # to kernel.bbclass . We have to override it, since we pack zImage
- # (at least for now) into the fitImage .
- typeformake =d.getVar("KERNEL_IMAGETYPE_FOR_MAKE")or "" - if 'fitImage' intypeformake.split():
- d.setVar('KERNEL_IMAGETYPE_FOR_MAKE',typeformake.replace('fitImage', d.getVar('KERNEL_IMAGETYPE_REPLACEMENT')))
-
- image =d.getVar('INITRAMFS_IMAGE')
- if image:
-d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')
-
ubootenv =d.getVar('UBOOT_ENV')
if ubootenv:
d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/bootloader:do_populate_sysroot')meta/classes-recipe/kernel-fitimage.bbclass +0
-0@@ -48,8 +36,31 @@ python __anonymous () { providerdtb = d.getVar("PREFERRED_PROVIDER_virtual/dtb")if providerdtb:
d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/dtb:do_populate_sysroot')
-d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' virtual/dtb:do_populate_sysroot')d.setVar('EXTERNAL_KERNEL_DEVICETREE', "${RECIPE_SYSROOT}/boot/devicetree")
+
+ typeformake =d.getVar("KERNEL_IMAGETYPE_FOR_MAKE")or "" + if 'fitImage' intypeformake.split():
+ # OverrideKERNEL_IMAGETYPE_FOR_MAKEvariable, which is internal
+ # to kernel.bbclass . We have to override it, since we pack zImage
+ # (at least for now) into the fitImage .
+ d.setVar('KERNEL_IMAGETYPE_FOR_MAKE',typeformake.replace('fitImage', d.getVar('KERNEL_IMAGETYPE_REPLACEMENT')))
+
+ # mkimage and dtc are required by thefitimage_assemblefunction
+ d.appendVarFlag('do_assemble_fitimage', 'depends', + ' u-boot-tools-native:do_populate_sysroot dtc-native:do_populate_sysroot')
+
+initramfs_image=d.getVar('INITRAMFS_IMAGE')
+ bundled = bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE'))
+ ifinitramfs_image:
+ if bundled:
+bb.build.addtask('do_assemble_fitimage_initramfs', 'do_deploy', 'do_bundle_initramfs', d)
+ else:
+bb.build.addtask('do_assemble_fitimage_initramfs', 'do_deploy', 'do_install', d)
+
+ d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends',
+ ' u-boot-tools- native:do_populate_sysroot dtc-native:do_populate_sysroot ${INITRAMFS_IMAGE}:do_image_complete')
+ if providerdtb:
+d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' virtual/dtb:do_populate_sysroot')
}meta/classes-recipe/kernel-fitimage.bbclass +0
-0@@ -797,19 +808,14 @@ do_install:append() { }
do_assemble_fitimage_initramfs() {meta/classes-recipe/kernel-fitimage.bbclass +0
-5- if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage" && \ - test -n "${INITRAMFS_IMAGE}" ; then - cd ${B} - if [ "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then - fitimage_assemble fit-image-${INITRAMFS_IMAGE}.its fitImage-bundle ""
meta/classes-recipe/kernel-fitimage.bbclass +0
-1- ln -sf fitImage-bundle${B}/${KERNEL_OUTPUT_DIR}/fitImage
meta/classes-recipe/kernel-fitimage.bbclass +0
-2- else - fitimage_assemble fit-image-${INITRAMFS_IMAGE}.its fitImage-${INITRAMFS_IMAGE} 1
meta/classes-recipe/kernel-fitimage.bbclass +2
-1- fi + if [ "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then + fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its""${KERNEL_OUTPUT_DIR}/fitImage-bundle" ""
meta/classes-recipe/kernel-fitimage.bbclass +1
-0+ ln -sf fitImage-bundle"${B}/${KERNEL_OUTPUT_DIR}/fitImage"
meta/classes-recipe/kernel-fitimage.bbclass +2
-0+ else + fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its""${KERNEL_OUTPUT_DIR}/fitImage-${INITRAMFS_IMAGE}" 1 fi
}meta/classes-recipe/kernel-fitimage.bbclass +0
-2- -addtask assemble_fitimage_initramfs before do_deploy after
do_bundle_initramfs
+do_assemble_fitimage_initramfs[dirs] = "${B}"
do_kernel_generate_rsa_keys() {
if [ "${UBOOT_SIGN_ENABLE}" = "0" ] && [
"${FIT_GENERATE_KEYS}" = "1" ]; thenmeta/classes-recipe/kernel-fitimage.bbclass +0
-0@@ -864,6 +870,7 @@ kernel_do_deploy:append() {if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then
if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; thenmeta/classes-recipe/kernel-fitimage.bbclass +1
-0+ # deploy the artifacts of
do_assemble_fitimage
bbnote "Copying fit-image.its source file..."
install -m 0644 ${B}/fit-image.its "$deployDir/fitImage-its-${KERNEL_FIT_NAME}.its"
if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; thenmeta/classes-recipe/kernel-fitimage.bbclass +0
-0@@ -878,12 +885,14 @@ kernel_do_deploy:append() {fi
if [ -n "${INITRAMFS_IMAGE}" ]; thenmeta/classes-recipe/kernel-fitimage.bbclass +1
-0+ # deploy the artifacts of
do_assemble_fitimage_initramfsfor bundled as well as un-bundled mode bbnote "Copying fit-image- ${INITRAMFS_IMAGE}.its source file..."
install -m 0644 ${B}/fit-image- ${INITRAMFS_IMAGE}.its "$deployDir/fitImage-its- ${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its"
if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then ln -snf fitImage-its- ${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its "$deployDir/fitImage- its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_LINK_NAME}"
fimeta/classes-recipe/kernel-fitimage.bbclass +1
-0+ # deploy the artifacts of
do_assemble_fitimage_initramfsfor bundled mode only
if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
bbnote "Copying fitImage- ${INITRAMFS_IMAGE} file..."
install -m 0644 ${B}/${KERNEL_OUTPUT_DIR}/fitImage-${INITRAMFS_IMAGE} "$deployDir/fitImage-${INITRAMFS_IMAGE_NAME}- ${KERNEL_FIT_NAME}${KERNEL_FIT_BIN_EXT}"meta/classes- renamed from meta/classes-recipe/kernel.bbclass +0
-0recipe/kernel.bbclass
index4464155b730..217f42b4f4f100644meta/classes-recipe/kernel.bbclass +0
-0@@ -139,18 +139,23 @@ fiset -e
""" % (type, type, type))
-
image =d.getVar('INITRAMFS_IMAGE')
- # If theINTIRAMFS_IMAGEis set but theINITRAMFS_IMAGE_BUNDLEis set to 0,
- # thedo_bundle_initramfsdoes nothing, but theINITRAMFS_IMAGEis built
- # standalone for use by wic and other tools.
if image:
- ifd.getVar('INITRAMFS_MULTICONFIG'):
- d.appendVarFlag('do_bundle_initramfs', 'mcdepends', ' mc::${INITRAMFS_MULTICONFIG}:${INITRAMFS_IMAGE}:do_image_complete') + # If theINTIRAMFS_IMAGEis set but theINITRAMFS_IMAGE_BUNDLEis set to 0,
+ # thedo_bundle_initramfsis not needed, but theINITRAMFS_IMAGEis built
+ # standalone for use by wic and other tools.
+ defadd_initramfs_dep_task(initramfs_dep_task, d):
+ ifd.getVar('INITRAMFS_MULTICONFIG'):
+ d.appendVarFlag(initramfs_dep_task, 'mcdepends', ' mc::${INITRAMFS_MULTICONFIG}:${INITRAMFS_IMAGE}:do_image_complete') + else:
+d.appendVarFlag(initramfs_dep_task, 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')
+
+ if bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')):
+bb.build.addtask('do_bundle_initramfs', 'do_deploy', 'do_install', d)
+add_initramfs_dep_task('do_bundle_initramfs', d)+bb.build.addtask('do_transform_bundled_initramfs', 'do_deploy', 'do_bundle_initramfs', d)
Quoted 8 lines
else:
-d.appendVarFlag('do_bundle_initramfs', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')
- if image and bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')):
-bb.build.addtask('do_transform_bundled_initramfs', 'do_deploy', 'do_bundle_initramfs', d)
+add_initramfs_dep_task('do_deploy', d)
This line is still not ideal. I'm working on a v5 without it. Maybe also with less changes in the kernel.bbclass.
Regards,
Adrian
Quoted 32 lines
# NOTE: settingINITRAMFS_TASKis for backward compatibility # The preferred method is to setINITRAMFS_IMAGE, becausemeta/classes-recipe/kernel.bbclass +0
-0@@ -330,6 +335,8 @@ do_bundle_initramfs () {mv -f ${KERNEL_OUTPUT_DIR}/$imageType.bak ${KERNEL_OUTPUT_DIR}/$imageType fi
donemeta/classes-recipe/kernel.bbclass +2
-0+ else + bbwarn "Calling do_bundle_initramfs withINITRAMFS_IMAGE_BUNDLE=1 is deprecated"
fi
}
do_bundle_initramfs[dirs] = "${B}"meta/classes-recipe/kernel.bbclass +0
-0@@ -350,8 +357,6 @@ python do_devshell:prepend () {os.environ["LDFLAGS"] = ''
}meta/classes-recipe/kernel.bbclass +0
-2-addtask bundle_initramfs after do_install before do_deploy -KERNEL_DEBUG_TIMESTAMPS ??= "0"
kernel_do_compile() {meta/classes-recipe/kernel.bbclass +0
-0@@ -873,7 +878,7 @@ kernel_do_deploy() { # ensure we get the right values for bothdo_deploy[prefuncs] += "
read_subpackage_metadata"meta/classes-recipe/kernel.bbclass +1
-1-addtask deploy after do_populate_sysroot do_packagedata +addtask deploy after do_install do_populate_sysroot do_packagedata
EXPORT_FUNCTIONSdo_deploy
Signature · 4 lines
Adrian Freihofer Siemens AG Mobile: +41 796819596 www.siemens.com
PATCH v4 4/4 kernel-fitimage: run unbundled fitimage after deploy
In reply to Adrian FreihoferFrom: Adrian Freihofer <[email protected]>
Without this change, the kernel is always rebuilt if something changes in the initramfs. If the kernel is rebuilt with an empty TMPDIR, the process starts by fetching the large kernel git repository and recompiling it from scratch. This is very inefficient.
This cannot be improved if INITRAMFS_IMAGE_BUNDLE = "1". If the kernel Makefile is needed to generate the initramfs bundle the kernel build folder is required. For this cases nothing changes.
But for a build configuration with INITRAMFS_IMAGE_BUNDLE = "" the build folder is not needed. Creating the initramfs bundle requires:
linux.bin, DTBs and the initramfs. These artifacts are all available in the deploy folder which is also sstate cached.
Therefore do_assemble_fitimage_initramfs gets split into two tasks:
- do_assemble_fitimage_initramfs creates the bundled fitImage which runs before do_deploy. This is without sstate and therefore the do_deploy task still deploys the artifacts generated by this task.
- do_deploy_fitimage_unbundled creates the fitImage in unbundled mode. This task runs after do_deploy. It uses the linux.bin artifact from do_deploy to create the fitImage with initramfs.
This task is done with SSTATE_SKIP_CREATION, because this also works with the eSDK installer (oe-selftest -r esdk is successful). It is also more effective to quickly create the fit image instead of handling it via sstate.
Unfortunately, this change is not 100% backward compatible. Previously, a dependency on do_deploy deployed all types of fit images. With this change, deploying the unbundled fit image requires a dependency on do_deploy_fitimage_unbundled. To address this the image.bbclass gets improved to handle both cases with the KERNEL_DEPLOY_DEPEND variable.
--- meta/classes-recipe/image.bbclass | 12 ++- meta/classes-recipe/kernel-fitimage.bbclass | 86 +++++++++++++-------- meta/classes-recipe/kernel.bbclass | 3 +- 3 files changed, 67 insertions(+), 34 deletions(-)
meta/classes-recipe/image.bbclass +11 -1
@@ -147,7 +147,17 @@ do_rootfs[vardeps] += "${@rootfs_variables(d)}" # This follows many common usecases and user expectations. # But if you are building an image which doesn't need the kernel image at all, # you can unset this variable manually. -KERNEL_DEPLOY_DEPEND ?= "virtual/kernel:do_deploy" +def get_kernel_do_deploy(d): + kerneltypes = d.getVar('KERNEL_IMAGETYPES') or "" + kerneltype = d.getVar('KERNEL_IMAGETYPE') or "" + fitimage = kerneltype == 'fitImage' or 'fitImage' in kerneltypes.split() + initramfs_image = d.getVar('INITRAMFS_IMAGE') or "" + bundled = d.getVar('INITRAMFS_IMAGE_BUNDLE') == '1' + if fitimage and initramfs_image and not bundled: + return "virtual/kernel:do_deploy_fitimage_unbundled" + return "virtual/kernel:do_deploy" + +KERNEL_DEPLOY_DEPEND ?= "${@get_kernel_do_deploy(d)}" do_build[depends] += "${KERNEL_DEPLOY_DEPEND}"
meta/classes-recipe/kernel-fitimage.bbclass +54 -32
126 lines not on this page · raw message
meta/classes-recipe/kernel.bbclass +2 -1
@@ -490,7 +490,8 @@ kernel_do_install() { # So, at the level of the install task we should not try to install the fitImage. fitImage is still not # generated yet. # After the generation of the fitImage, the deploy task copies the fitImage from the build directory to - # the deploy folder. + # the deploy folder. If INITRAMFS_IMAGE_BUNDLE != 1 the fitImage with initramfs is deployed after the + # deploy task. # for imageType in ${KERNEL_IMAGETYPES} ; do
Signature · 1 line
2.46.0
Re: [OE-core] [PATCH 4/4] kernel-fitimage: run unbundled fitimage after deploy
I have been struggling to find time to sit and think through the implications of this. I have finally done that, at least a little and I have some questions below. Sorry about the delay in getting to this, I keep hoping someone else will help with this kind of review but it is very time consuming and I suspect nobody has quite the right set of knowledge to make it easier.
Quoted 10 lines · Adrian Freihofer via lists.openembedded.org
From: Adrian Freihofer <[email protected]>
Without this change, the kernel is always rebuilt if something changes in the initramfs. If the kernel is rebuilt with an empty TMPDIR, the process starts by fetching the large kernel git repository and recompiling it from scratch. This is very inefficient.
This cannot be improved if
INITRAMFS_IMAGE_BUNDLE= "1". If the kernelMakefileis needed to generate the initramfs bundle the kernel build folder is required. For this cases nothing changes.
When you say "kernel Makefile", I'm assuming to rebuild the kernel image, you need much more than just the Makefile but a complete kernel build tree? The Makefile is a solvable problem, the tree isn't!
Quoted 5 lines
But for a build configuration with
INITRAMFS_IMAGE_BUNDLE= "" the build folder is not needed. Creating the initramfs bundle requires:
linux.bin, DTBs and the initramfs. These artifacts are all available in the deploy folder which is also sstate cached.
Quoted 7 lines
Therefore
do_assemble_fitimage_initramfsgets split into two tasks:
-do_assemble_fitimage_initramfscreates the bundled fitImage which runs beforedo_deploy. This is without sstate and therefore thedo_deploytask still deploys the artifacts generated by this task.
-do_deploy_fitimage_unbundledcreates the fitImage in unbundled mode. This task runs afterdo_deploy. It uses the linux.bin artifact fromdo_deployto create the fitImage with initramfs.
I'm a bit worried about having a second do_deploy_xxx task after do_deploy. This creates a recipe with two deploy tasks and whilst I can understand why you're doing it, I worry this is going to create confusion down the road.
The reason I say this is that there are rules for "deploy" tasks, e.g. in meta-classes/sstate.bbclass:setscene_depvalid() has:
# do_package/packagedata/package_qa/deploy don't need do_populate_sysroot
if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package', 'do_packagedata', 'do_package_qa', 'do_deploy']:
continuedo_deploy_fitimage_unbundled isn't listed in SSTATETASKS. Whilst that is simple to fix, I'm not sure what side effects that is then going to cause.
Also, there is this to consider:
meta/classes-global/base.bbclass:do_build[recrdeptask] += "do_deploy"
which is basically making sure "build" targets (the default) trigger all artefacts to be deployed. I think this new task will be missed by global code as it isn't covered there. The intent was to have one name which covered all cases, not having recipe specific entries in there.
I'm starting to wonder if that unbundled case should be handled by a separate recipe, which can then have its own do_deploy task and depend on all the correct pieces.
Quoted 4 lines
This task is done with
SSTATE_SKIP_CREATION, because this also works with the eSDK installer (oe-selftest -r esdk is successful). It is also more effective to quickly create the fit image instead of handling it via sstate.
Skipping sstate creation is a huge red warning sign in my mind. So far we've only had images and eSDKs using this option and I never wanted to add the option generically in the first place since it was going to lead to more exceptions like this.
The implications of not creating the sstate can be problematic, for example the sstate tests then need to learn that some new subset of sstate artefacts will never be created. This also complicated the testing of eSDKs which do a similar verification which will also need exceptions. I can understand why you've done it but I am worried the side effects will be left for others (such as myself) to resolve as fit images get used in new codepaths :/.
Quoted 5 lines
Unfortunately, this change is not 100% backward compatible. Previously, a dependency on
do_deploydeployed all types of fit images. With this change, deploying the unbundled fit image requires a dependency ondo_deploy_fitimage_unbundled. To address this the image.bbclass gets improved to handle both cases with theKERNEL_DEPLOY_DEPENDvariable.
I think this proves my point about why adding a new deploy task is potentially problematic. Have you considered using an intermediate recipe for the unbundled case instead?
I wish we could determine how much use the bundled and unbundled cases have. I'm also starting to wonder if the two cases should be separated apart into two clear workflows rather than the current entwined situation which is very hard to visualise and understand.
Cheers,
Richard
Re: [OE-core] [PATCH 4/4] kernel-fitimage: run unbundled fitimage after deploy
Hi Richard
Thank you for the review.
Quoted 26 lines · Richard Purdie
I have been struggling to find time to sit and think through the implications of this. I have finally done that, at least a little and I
have some questions below. Sorry about the delay in getting to this, I
keep hoping someone else will help with this kind of review but it is very time consuming and I suspect nobody has quite the right set of knowledge to make it easier.Quoted 9 lines · Adrian Freihofer via lists.openembedded.org
From: Adrian Freihofer <[email protected]>
Without this change, the kernel is always rebuilt if something changes
in the initramfs. If the kernel is rebuilt with an empty TMPDIR, the
process starts by fetching the large kernel git repository and recompiling it from scratch. This is very inefficient.Quoted 5 lines
This cannot be improved if
INITRAMFS_IMAGE_BUNDLE= "1". If the kernelMakefileis needed to generate the initramfs bundle the kernel build
folder is required. For this cases nothing changes.
Quoted 4 lines
When you say "kernel
Makefile", I'm assuming to rebuild the kernel image, you need much more than just theMakefilebut a complete kernel
build tree? TheMakefileis a solvable problem, the tree isn't!
Yes, I should have written kernel build tree.
Quoted 29 lines
Quoted 7 lines
But for a build configuration with
INITRAMFS_IMAGE_BUNDLE= "" the build folder is not needed. Creating the initramfs bundle requires:
linux.bin, DTBs and the initramfs. These artifacts are all available in
the deploy folder which is also sstate cached.Quoted 11 lines
Therefore
do_assemble_fitimage_initramfsgets split into two tasks:
-do_assemble_fitimage_initramfscreates the bundled fitImage which runs
beforedo_deploy. This is without sstate and therefore thedo_deploy
task still deploys the artifacts generated by this task.
-do_deploy_fitimage_unbundledcreates the fitImage in unbundled mode.
This task runs afterdo_deploy. It uses the linux.bin artifact from
do_deployto create the fitImage with initramfs.I'm a bit worried about having a second
do_deploy_xxxtask afterdo_deploy. This creates a recipe with two deploy tasks and whilst I can
understand why you're doing it, I worry this is going to create confusion down the road.The reason I say this is that there are rules for "deploy" tasks, e.g.
in meta-classes/sstate.bbclass:setscene_depvalid() has:
Quoted 8 lines
# do_package/packagedata/package_qa/deploy don't need do_populate_sysroot if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package', 'do_packagedata', 'do_package_qa', 'do_deploy']: continue
do_deploy_fitimage_unbundledisn't listed in SSTATETASKS. Whilst that
Thank you for the hint. I don't know how this could have been overlooked until now.
Quoted 53 lines
is simple to fix, I'm not sure what side effects that is then going to
cause.Also, there is this to consider:
meta/classes-global/base.bbclass:do_build[recrdeptask] += "
do_deploy"which is basically making sure "build" targets (the default) trigger all artefacts to be deployed. I think this new task will be missed by global code as it isn't covered there. The intent was to have one name
which covered all cases, not having recipe specific entries in there.I'm starting to wonder if that unbundled case should be handled by a separate recipe, which can then have its own
do_deploytask and depend
on all the correct pieces.Quoted 6 lines
This task is done with
SSTATE_SKIP_CREATION, because this also works
with the eSDK installer (oe-selftest -r esdk is successful). It is
also more effective to quickly create the fit image instead of handling it via sstate.Skipping sstate creation is a huge red warning sign in my mind. So far
we've only had images and eSDKs using this option and I never wanted to
add the option generically in the first place since it was going to lead to more exceptions like this.The implications of not creating the sstate can be problematic, for example the sstate tests then need to learn that some new subset of sstate artefacts will never be created. This also complicated the testing of eSDKs which do a similar verification which will also need exceptions. I can understand why you've done it but I am worried the side effects will be left for others (such as myself) to resolve as fit
images get used in new codepaths :/.Quoted 9 lines
Unfortunately, this change is not 100% backward compatible. Previously,
a dependency ondo_deploydeployed all types of fit images. With this
change, deploying the unbundled fit image requires a dependency ondo_deploy_fitimage_unbundled. To address this the image.bbclass gets
improved to handle both cases with theKERNEL_DEPLOY_DEPENDvariable.
Quoted 3 lines
I think this proves my point about why adding a new deploy task is potentially problematic. Have you considered using an intermediate recipe for the unbundled case instead?
Yes, I have thought about and tried 3 different implementations:
- This one here which is supposed to be backward compatible.
- Renaming the sstated kernel do_deploy task to e.g. do_deploy_stage1. Then bring the do_deploy task back as an empty task depending on the do_deploy_stage1. This looks equal to the current implementation but allows to inject the do_deploy_fitimage_unbundled task from the kernel- fitimage.bbclass. However, this has similar side effects. For example the eSDK assumes that the do_deploy task is the last sstated task in the chain which provides the relevant sstate artifact.
- Add a separate recipe for the unbundled fitImage. This could simplify the implementation, but is probably somewhat cumbersome from the user's point of view. Bundled fitImage remains in the kernel but unbundled needs dealing with different recipes.
Quoted 7 lines
I wish we could determine how much use the bundled and unbundled cases
have. I'm also starting to wonder if the two cases should be separated
apart into two clear workflows rather than the current entwined situation which is very hard to visualise and understand.
Yes, that's probably the conclusion so far. The kernel recipes have reached a very complex state. Especially the implementation of the unbundled fitImage support is a bit strange and adds complexity that could possibly be simplified by splitting it into a separate recipe. I will try to come up with a different implementation where the undbundled fitImage is created by a dedicated recipe. This recipe will basically have the do_deploy_fitimage_unbundled implementation as its do_deploy task which depends on the kernel do_deploy.
Not sure if such a recipe could also provide a package for the fitImage kernel. Currently the fitImage is deployed only. Other kernel image types are packaged and deployed.
The challenge will then probably be to hide the extra complexity of 2 recipes as good as possible and to refactor the tests accordingly.
Kind regards,
Adrian
Quoted 4 lines
Cheers,
Richard