Use the kernel from sstate when building fitImages
v2 cover Use the kernel from sstate when building fitImages
Changes in comparison to v1:
- Add the missing dependency from do_image_wic to initramfs.do_image_complete which caused the build failure on the AB.
- Support symbolic links for $dtb_path again, as suggested by Mark.
- Simplify the refactoring of the uboot_prep_kimage function.
- Evaluate staging via sysroot instead of taking the kernel artifacts from the deploy folder. But there is an issue with this approach: It does not allow to remove the dependency from do_deploy on do_install.
- Rebasing to latest master
If the KERNEL_IMAGETYPES(S) contains fitImage, the kernel is always rebuilt when something changes in the initramfs frequently. This is even worse if the build runs from an empty TMPDIR. The kernel re-build starts by fetching the large kernel git repository and recompiling it from scratch.
This cannot be improved if INITRAMFS_IMAGE_BUNDLE = "1". If the kernel Makefile is needed to generate the initramfs bundle the kernel build folder is required.
But for a build configuration with INITRAMFS_IMAGE_BUNDLE = "" the build folder is not needed. Creating the initramfs bundle requires:
linux.bin, DTBs and the initramfs which are available in the deploy directory. That means creating the fitImage can be done with artifacts which are already cached by the sstate.
There is an extra commit providing a html file. This html file provides some graphics which show the changes in the kernel's task dependencies. I hope this simplifies the re-view of this patch series.
Testing:
- oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg \
reproducible -T yocto-mirrors -T machine -T toolchain-user \
-T toolchain-system- Build for core-image-minimal for MACHINE = "genericarm64" (failure from AB was reproducible and is fixed now)
- Build clean followed by build from sstate
Note: Adding this test to oe-selftest failed because sstate is read only
- Clean build config:
KERNEL_IMAGETYPE = "Image"
KERNEL_IMAGETYPES += " fitImage "
KERNEL_CLASSES = " kernel-fitimage "
IMAGE_FSTYPES += "cpio.gz"
INITRAMFS_IMAGE = "core-image-minimal"
IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""
UBOOT_RD_LOADADDRESS = "0x88000000"
UBOOT_RD_ENTRYPOINT = "0x88000000"
UBOOT_LOADADDRESS = "0x80080000"
UBOOT_ENTRYPOINT = "0x80080000"
FIT_DESC = "A model description"
FOO_VAR = "1"
INHERIT += "image-buildinfo"
IMAGE_BUILDINFO_VARS:append = " FOO_VAR"
- Append the following and rebuild with sstate:
FOO_VAR = "2"
TMPDIR = "${TOPDIR}/tmp-2" - Check the log files:
- tmp/log/cooker/qemux86-64 contains:
linux-yocto-6.6.35+git-r0: task do_deploy: Succeeded linux-yocto-6.6.35+git-r0: task do_deploy_fitimage_unbundled: Succeeded
- tmp-2/log/cooker/qemux86-64 contains:
linux-yocto-6.6.35+git-r0: task do_deploy_setscene: Succeeded linux-yocto-6.6.35+git-r0: task do_deploy_fitimage_unbundled: Succeeded
- To re-view the chagnes in the task dependencies, the script bellow has been used. It confirms:
- For builds with fitImage and unbundled initramfs:
- the do_deploy_fitimage_unbundled task runs after do_deploy
- do_assemble_fitimage_initramfs is not executed
- do_bundle_initramfs is not executed
- It works for fitImage in KERNEL_IMAGETYPE as well as for fitImage in KERNEL_IMAGETYPES
- For builds with fitImage and bundled initramfs: No changes
- For builds with fitImage, without initramfs:
- do_assemble_fitimage_initramfs is not executed
- do_bundle_initramfs is not executed
OUTPUT_FILE=task-depends.md
GIT_BRANCH=master
GIT_BRANCH_NEXT=adrianf/kernel-fitimage-sstate
echo "# Task dependeny changes" > "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"
run_bitbake(){
echo "$1" >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"
bitbake virtual/kernel -g
grep -E '(digraph depends|linux-yocto|\})' task-depends.dot \
| grep -v -E '(spdx|do_kernel_configcheck|do_prepare_recipe_sysroot|do_populate_lic|native.do_populate_sysroot)' \
| grep -v -E '(depmodwrapper-cross.do_populate_sysroot|binutils-cross-x86_64.do_populate_sysroot|do_package)' \
| grep -v -E '(do_validate_branches|do_unpack|do_recipe_qa|do_patch|do_kernel_checkout|do_kernel_configme|do_kernel_metadata)' \
| grep -v -E '(do_sizecheck|do_strip|do_compile_kernelmodules|do_shared_workdir|do_transform_kernel|do_kernel_link_images)' \
| grep -v -E '(do_kernel_version_sanity_check|do_symlink_kernsrc|do_deploy_source_date_epoch|do_configure|do_fetch)' \
| sed -e 's;\\n\:.*.bb;;g' -e 's;linux-yocto[. ];;g' > task-depends-filtered.dot
echo '```plantuml' >> "$OUTPUT_FILE"
cat task-depends-filtered.dot >> "$OUTPUT_FILE"
echo '```' >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"
}run_bitbake_per_branch(){
echo "## Configuration: $1" >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"
echo '```raw' >> "$OUTPUT_FILE"
cat build/conf/auto.conf >> "$OUTPUT_FILE"
echo '```' >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"
git checkout $GIT_BRANCH
run_bitbake "### branch: $GIT_BRANCH"
mv -f task-depends-filtered.dot task-depends-filtered-old.dot
git checkout $GIT_BRANCH_NEXT
run_bitbake "### branch: $GIT_BRANCH_NEXT"
echo "## Diff" >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"
echo '```patch' >> "$OUTPUT_FILE"
diff task-depends-filtered-old.dot task-depends-filtered.dot >> "$OUTPUT_FILE"
echo '```' >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"
}cat << EOF > build/conf/auto.confKERNEL_IMAGETYPE = "Image"KERNEL_IMAGETYPES += " fitImage "KERNEL_CLASSES = " kernel-fitimage "IMAGE_FSTYPES += "cpio.gz"INITRAMFS_IMAGE = "core-image-minimal-initramfs"
EOFrun_bitbake_per_branch "image, fitimage, initramfs, unbundled"
cat << EOF > build/conf/auto.conf KERNEL_IMAGETYPE:forcevariable = "fitImage" KERNEL_CLASSES = " kernel-fitimage " IMAGE_FSTYPES += "cpio.gz"INITRAMFS_IMAGE = "core-image-minimal-initramfs" EOFrun_bitbake_per_branch "fitimage, initramfs, unbundled"
cat << EOF > build/conf/auto.confKERNEL_IMAGETYPE = "Image"KERNEL_IMAGETYPES += " fitImage "KERNEL_CLASSES = " kernel-fitimage "IMAGE_FSTYPES += "cpio.gz"INITRAMFS_IMAGE = "core-image-minimal-initramfs" INITRAMFS_IMAGE_BUNDLE = "1"
EOFrun_bitbake_per_branch "image, fitimage, initramfs, bundled"
cat << EOF > build/conf/auto.confKERNEL_IMAGETYPE = "Image"KERNEL_IMAGETYPES += " fitImage "KERNEL_CLASSES = " kernel-fitimage "
EOFrun_bitbake_per_branch "image, fitimage"
rm -f build/conf/auto.conf task-depends-filtered-old.dot task-depends-filtered.dot task-depends.dot
Adrian Freihofer (6):
kernel-fitimage: fix intentation
kernel-fitimage: fix external dtb check
kernel: refactor linux compression
kernel-fitimage: refactor fitimage_assemble
kernel: refactor fitimage
kernel-fitimage: run unbundled fitimage after deploy
meta/classes-recipe/image.bbclass | 12 +- meta/classes-recipe/kernel-fitimage.bbclass | 188 ++++++++++++-------- meta/classes-recipe/kernel-uboot.bbclass | 1 + meta/classes-recipe/kernel.bbclass | 31 ++-- 4 files changed, 148 insertions(+), 84 deletions(-)
Signature · 1 line
2.45.2
PATCH v2 1/6 kernel-fitimage: fix intentation
From: Adrian Freihofer <[email protected]>
white space changes only.
- python part should be 4 spaces, not 8.
- use tabs for shell
--- meta/classes-recipe/kernel-fitimage.bbclass | 64 ++++++++++----------- 1 file changed, 32 insertions(+), 32 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +32 -32
@@ -29,27 +29,27 @@ KERNEL_IMAGETYPE_REPLACEMENT ?= "${@get_fit_replacement_type(d)}" DEPENDS:append = " ${@'u-boot-tools-native dtc-native' if 'fitImage' in (d.getVar('KERNEL_IMAGETYPES') or '').split() else ''}" python __anonymous () { - # Override KERNEL_IMAGETYPE_FOR_MAKE variable, which is internal - # to kernel.bbclass . We have to override it, since we pack zImage - # (at least for now) into the fitImage . - typeformake = d.getVar("KERNEL_IMAGETYPE_FOR_MAKE") or "" - if 'fitImage' in typeformake.split(): - d.setVar('KERNEL_IMAGETYPE_FOR_MAKE', typeformake.replace('fitImage', d.getVar('KERNEL_IMAGETYPE_REPLACEMENT'))) - - image = d.getVar('INITRAMFS_IMAGE') - if image: - d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete') - - ubootenv = d.getVar('UBOOT_ENV') - if ubootenv: - d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/bootloader:do_populate_sysroot') - - #check if there are any dtb providers - providerdtb = d.getVar("PREFERRED_PROVIDER_virtual/dtb") - if providerdtb: - d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/dtb:do_populate_sysroot') - d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' virtual/dtb:do_populate_sysroot') - d.setVar('EXTERNAL_KERNEL_DEVICETREE', "${RECIPE_SYSROOT}/boot/devicetree") + # Override KERNEL_IMAGETYPE_FOR_MAKE variable, which is internal + # to kernel.bbclass . We have to override it, since we pack zImage + # (at least for now) into the fitImage . + typeformake = d.getVar("KERNEL_IMAGETYPE_FOR_MAKE") or "" + if 'fitImage' in typeformake.split(): + d.setVar('KERNEL_IMAGETYPE_FOR_MAKE', typeformake.replace('fitImage', d.getVar('KERNEL_IMAGETYPE_REPLACEMENT'))) + + image = d.getVar('INITRAMFS_IMAGE') + if image: + d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete') + + ubootenv = d.getVar('UBOOT_ENV') + if ubootenv: + d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/bootloader:do_populate_sysroot') + + #check if there are any dtb providers + providerdtb = d.getVar("PREFERRED_PROVIDER_virtual/dtb") + if providerdtb: + d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/dtb:do_populate_sysroot') + d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' virtual/dtb:do_populate_sysroot') + d.setVar('EXTERNAL_KERNEL_DEVICETREE', "${RECIPE_SYSROOT}/boot/devicetree") }
@@ -480,13 +480,13 @@ fitimage_emit_section_config() { # default node is selected based on dtb ID if it is present, # otherwise its selected based on kernel ID if [ -n "$dtb_image" ]; then - # Select default node as user specified dtb when - # multiple dtb exists. - if [ -n "$default_dtb_image" ]; then - default_line="default = \"${FIT_CONF_PREFIX}$default_dtb_image\";" - else - default_line="default = \"${FIT_CONF_PREFIX}$dtb_image\";" - fi + # Select default node as user specified dtb when + # multiple dtb exists. + if [ -n "$default_dtb_image" ]; then + default_line="default = \"${FIT_CONF_PREFIX}$default_dtb_image\";" + else + default_line="default = \"${FIT_CONF_PREFIX}$dtb_image\";" + fi else default_line="default = \"${FIT_CONF_PREFIX}$kernel_id\";" fi
@@ -605,9 +605,9 @@ fitimage_assemble() { DTB_PATH="${KERNEL_OUTPUT_DIR}/$DTB" fi - # Strip off the path component from the filename + # Strip off the path component from the filename if "${@'false' if oe.types.boolean(d.getVar('KERNEL_DTBVENDORED')) else 'true'}"; then - DTB=`basename $DTB` + DTB=`basename $DTB` fi # Set the default dtb image if it exists in the devicetree.
@@ -715,8 +715,8 @@ fitimage_assemble() { # kernel-fitimage.bbclass currently only supports a single kernel (no less or # more) to be added to the FIT image along with 0 or more device trees and # 0 or 1 ramdisk. - # It is also possible to include an initramfs bundle (kernel and rootfs in one binary) - # When the initramfs bundle is used ramdisk is disabled. + # It is also possible to include an initramfs bundle (kernel and rootfs in one binary) + # When the initramfs bundle is used ramdisk is disabled. # If a device tree is to be part of the FIT image, then select # the default configuration to be used is based on the dtbcount. If there is # no dtb present than select the default configuation to be based on
Signature · 1 line
2.45.2
PATCH v2 2/6 kernel-fitimage: fix external dtb check
In reply to Adrian FreihoferFrom: Adrian Freihofer <[email protected]>
If EXTERNAL_KERNEL_DEVICETREE and dtb_image_sect are empty variables dtb_path ends up as "/" which is available on most Unix systems but probably not the dtb_path which is needed here. Checking for a file makes more sense and also solves the issue with the "/".
--- meta/classes-recipe/kernel-fitimage.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-recipe/kernel-fitimage.bbclass +1 -1
@@ -429,7 +429,7 @@ fitimage_emit_section_config() { fi dtb_path="${EXTERNAL_KERNEL_DEVICETREE}/${dtb_image_sect}" - if [ -e "$dtb_path" ]; then + if [ -f "$dtb_path" ] || [ -L "$dtb_path" ]; then compat=$(fdtget -t s "$dtb_path" / compatible | sed 's/ /", "/g') if [ -n "$compat" ]; then compatible_line="compatible = \"$compat\";"
Signature · 1 line
2.45.2
PATCH v2 3/6 kernel: refactor linux compression
In reply to Adrian FreihoferFrom: Adrian Freihofer <[email protected]>
This is a step to avoid unecessary kernel rebuilds when the initramfs changes but not the kernel itself. In case of an unbundled fitImage the kernel should be staged in the sstate instead of re-compiled.
The uboot_prep_kimage function generates the kernel binary which gets included into the fitImage. It also defines the variable linux_comp which is used by the fitimage_assemble function later on.
To make the fitimage_assemble function working with kernel binaries from sstate it must be decoupled from calling uboot_prep_kimage in ${B} by:
- refactoring uboot_prep_kimage to write the result into files (linux.bin, linux.comp).
- calling the uboot_prep_kimage before the artifacts are used or cached.
- refactoring the fitimage_assemble function to use files rather than calling the uboot_prep_kimage function which depends on ${B}.
--- meta/classes-recipe/kernel-fitimage.bbclass | 5 ++--- meta/classes-recipe/kernel-uboot.bbclass | 1 + 2 files changed, 3 insertions(+), 3 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +2 -3
@@ -579,9 +579,7 @@ fitimage_assemble() { # Step 1: Prepare a kernel image section. # fitimage_emit_section_maint $1 imagestart - - uboot_prep_kimage - fitimage_emit_section_kernel $1 $kernelcount linux.bin "$linux_comp" + fitimage_emit_section_kernel $1 $kernelcount linux.bin "$(cat linux.comp)" # # Step 2: Prepare a DTB image section
@@ -764,6 +762,7 @@ fitimage_assemble() { do_assemble_fitimage() { if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then cd ${B} + uboot_prep_kimage fitimage_assemble fit-image.its fitImage-none "" if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then ln -sf fitImage-none ${B}/${KERNEL_OUTPUT_DIR}/fitImage
meta/classes-recipe/kernel-uboot.bbclass +1 -0
@@ -45,5 +45,6 @@ uboot_prep_kimage() { mv -f "linux.bin${linux_suffix}" linux.bin fi + [ -n "${vmlinux_path}" ] && echo "${linux_comp}" > linux.comp echo "${linux_comp}" }
Signature · 1 line
2.45.2
PATCH v2 4/6 kernel-fitimage: refactor fitimage_assemble
In reply to Adrian FreihoferFrom: Adrian Freihofer <[email protected]>
Make the fitimage_assemble function usable with absolute paths for the generated its and the fitImage file.
Later on this will allow to take the linux.bin and the DTB files from the sstated deploy derectory and write the generated files to a separate folder with independent sstate-cache handling.
--- meta/classes-recipe/kernel-fitimage.bbclass | 32 +++++++++++++++------ 1 file changed, 24 insertions(+), 8 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +24 -8
@@ -169,8 +169,9 @@ fitimage_emit_section_kernel() { ENTRYPOINT="${UBOOT_ENTRYPOINT}" if [ -n "${UBOOT_ENTRYSYMBOL}" ]; then + kernel_base=$(basename $3) ENTRYPOINT=`${HOST_PREFIX}nm vmlinux | \ - awk '$3=="${UBOOT_ENTRYSYMBOL}" {print "0x"$1;exit}'` + awk '$kernel_base=="${UBOOT_ENTRYSYMBOL}" {print "0x"$1;exit}'` fi cat << EOF >> $1
@@ -567,7 +568,7 @@ fitimage_assemble() { setupcount="" bootscr_id="" default_dtb_image="" - rm -f $1 arch/${ARCH}/boot/$2 + rm -f "$1" "arch/${ARCH}/boot/$(basename $2)" if [ -n "${UBOOT_SIGN_IMG_KEYNAME}" -a "${UBOOT_SIGN_KEYNAME}" = "${UBOOT_SIGN_IMG_KEYNAME}" ]; then bbfatal "Keys used to sign images and configuration nodes must be different."
@@ -608,6 +609,17 @@ fitimage_assemble() { DTB=`basename $DTB` fi + # Find DTBs without sub-folders when running in deploy folder + if [ ! -e "$DTB_PATH" ]; then + DTB=$(basename $DTB) + DTB_PATH=$(readlink -f $DTB) + fi + + # Fail as early as possible if there is still no DTB file found + if [ ! -e "$DTB_PATH" ]; then + bberror "Cannot find the DTB file at $DTB_PATH" + fi + # Set the default dtb image if it exists in the devicetree. if [ "${FIT_CONF_DEFAULT_DTB}" = "$DTB" ];then default_dtb_image=$(echo "$DTB" | tr '/' '_')
@@ -658,9 +670,8 @@ fitimage_assemble() { if [ -n "${UBOOT_ENV}" ] && [ -d "${STAGING_DIR_HOST}/boot" ]; then if [ -e "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY}" ]; then - cp ${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY} ${B} bootscr_id="${UBOOT_ENV_BINARY}" - fitimage_emit_section_boot_script $1 "$bootscr_id" ${UBOOT_ENV_BINARY} + fitimage_emit_section_boot_script $1 "$bootscr_id" "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY}" else bbwarn "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY} not found." fi
@@ -669,9 +680,14 @@ fitimage_assemble() { # # Step 4: Prepare a setup section. (For x86) # + # Run from kernel build folder (bundled mode) if [ -e ${KERNEL_OUTPUT_DIR}/setup.bin ]; then setupcount=1 fitimage_emit_section_setup $1 $setupcount ${KERNEL_OUTPUT_DIR}/setup.bin + # Run from deploy folder (unbundled mode) + elif [ -e setup.bin ]; then + setupcount=1 + fitimage_emit_section_setup $1 $setupcount "$(readlink -f setup.bin)" fi #
@@ -744,8 +760,8 @@ fitimage_assemble() { # ${UBOOT_MKIMAGE} \ ${@'-D "${UBOOT_MKIMAGE_DTCOPTS}"' if len('${UBOOT_MKIMAGE_DTCOPTS}') else ''} \ - -f $1 \ - ${KERNEL_OUTPUT_DIR}/$2 + -f "$1" \ + "$2" # # Step 8: Sign the image
@@ -754,7 +770,7 @@ fitimage_assemble() { ${UBOOT_MKIMAGE_SIGN} \ ${@'-D "${UBOOT_MKIMAGE_DTCOPTS}"' if len('${UBOOT_MKIMAGE_DTCOPTS}') else ''} \ -F -k "${UBOOT_SIGN_KEYDIR}" \ - -r ${KERNEL_OUTPUT_DIR}/$2 \ + -r "$2" \ ${UBOOT_MKIMAGE_SIGN_ARGS} fi }
@@ -763,7 +779,7 @@ do_assemble_fitimage() { if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then cd ${B} uboot_prep_kimage - fitimage_assemble fit-image.its fitImage-none "" + fitimage_assemble fit-image.its ${KERNEL_OUTPUT_DIR}/fitImage-none "" if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then ln -sf fitImage-none ${B}/${KERNEL_OUTPUT_DIR}/fitImage fi
Signature · 1 line
2.45.2
PATCH v2 5/6 kernel: refactor fitimage
In reply to Adrian FreihoferFrom: Adrian Freihofer <[email protected]>
* do_bundle_initramfs only when needed
With INITRAMFS_IMAGE_BUNDLE = "1" the kernel Makefile is used to bundle the kernel binary and the initramfs. This works only when the build folder of the kernel is available. Building with sstate but with an empty TMPDIR requires to rebuild the kernel from scratch whenever something in the initramfs changes.
With INITRAMFS_IMAGE_BUNDLE = "" the fitImage generation is basically independent from the kernel Makefile and could therefore work with kernel binaries and tools provided by the sstate. But the dependency on the do_bundle_initramfs tasks does not allow this. However, if the INTIRAMFS_IMAGE is set but the INITRAMFS_IMAGE_BUNDLE is set to 0 the do_bundle_initramfs does nothing. There is no real argument for running this tasks.
As a first step towards getting the kernel from sstate when building an unbundled fitImage, the task dependencies need to be simplified. The do_bundle_initramfs task is now scheduled only when needed.
* If KERNEL_IMAGETYPES does not contain fitImage and INITRAMFS_IMAGE is not set, the do_assemble_fitimage_initramfs ends up as an empty task. Add the do_assemble_fitimage_initramfs only if a fitImage with initramfs needs to be built.
* cd in bitbake task is easy to break e.g. by bbappend. Replace the cd by defining the task's dirs variable flag.
* Add 3 comments which are helpful for the next commits.
* Since the tasks and task dependencies are anyway created by an anonymous function, align and make the DEPENDS code more specific too.
This refactoring is not very valuable on its own. But it simplifies the task dependencies which is helpful for the next commit.
--- meta/classes-recipe/kernel-fitimage.bbclass | 57 ++++++++++++--------- meta/classes-recipe/kernel.bbclass | 28 ++++++---- 2 files changed, 50 insertions(+), 35 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +33 -24
@@ -26,20 +26,8 @@ def get_fit_replacement_type(d): return replacementtype KERNEL_IMAGETYPE_REPLACEMENT ?= "${@get_fit_replacement_type(d)}" -DEPENDS:append = " ${@'u-boot-tools-native dtc-native' if 'fitImage' in (d.getVar('KERNEL_IMAGETYPES') or '').split() else ''}" python __anonymous () { - # Override KERNEL_IMAGETYPE_FOR_MAKE variable, which is internal - # to kernel.bbclass . We have to override it, since we pack zImage - # (at least for now) into the fitImage . - typeformake = d.getVar("KERNEL_IMAGETYPE_FOR_MAKE") or "" - if 'fitImage' in typeformake.split(): - d.setVar('KERNEL_IMAGETYPE_FOR_MAKE', typeformake.replace('fitImage', d.getVar('KERNEL_IMAGETYPE_REPLACEMENT'))) - - image = d.getVar('INITRAMFS_IMAGE') - if image: - d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete') - ubootenv = d.getVar('UBOOT_ENV') if ubootenv: d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/bootloader:do_populate_sysroot')
@@ -48,8 +36,31 @@ python __anonymous () { providerdtb = d.getVar("PREFERRED_PROVIDER_virtual/dtb") if providerdtb: d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/dtb:do_populate_sysroot') - d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' virtual/dtb:do_populate_sysroot') d.setVar('EXTERNAL_KERNEL_DEVICETREE', "${RECIPE_SYSROOT}/boot/devicetree") + + typeformake = d.getVar("KERNEL_IMAGETYPE_FOR_MAKE") or "" + if 'fitImage' in typeformake.split(): + # Override KERNEL_IMAGETYPE_FOR_MAKE variable, which is internal + # to kernel.bbclass . We have to override it, since we pack zImage + # (at least for now) into the fitImage . + d.setVar('KERNEL_IMAGETYPE_FOR_MAKE', typeformake.replace('fitImage', d.getVar('KERNEL_IMAGETYPE_REPLACEMENT'))) + + # mkimage and dtc are required by the fitimage_assemble function + d.appendVarFlag('do_assemble_fitimage', 'depends', + ' u-boot-tools-native:do_populate_sysroot dtc-native:do_populate_sysroot') + + initramfs_image = d.getVar('INITRAMFS_IMAGE') + bundled = bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')) + if initramfs_image: + if bundled: + bb.build.addtask('do_assemble_fitimage_initramfs', 'do_deploy', 'do_bundle_initramfs', d) + else: + bb.build.addtask('do_assemble_fitimage_initramfs', 'do_deploy', 'do_install', d) + + d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', + ' u-boot-tools-native:do_populate_sysroot dtc-native:do_populate_sysroot ${INITRAMFS_IMAGE}:do_image_complete') + if providerdtb: + d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' virtual/dtb:do_populate_sysroot') }
@@ -797,19 +808,14 @@ do_install:append() { } do_assemble_fitimage_initramfs() { - if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage" && \ - test -n "${INITRAMFS_IMAGE}" ; then - cd ${B} - if [ "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then - fitimage_assemble fit-image-${INITRAMFS_IMAGE}.its fitImage-bundle "" - ln -sf fitImage-bundle ${B}/${KERNEL_OUTPUT_DIR}/fitImage - else - fitimage_assemble fit-image-${INITRAMFS_IMAGE}.its fitImage-${INITRAMFS_IMAGE} 1 - fi + if [ "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then + fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage-bundle" "" + ln -sf fitImage-bundle "${B}/${KERNEL_OUTPUT_DIR}/fitImage" + else + fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage-${INITRAMFS_IMAGE}" 1 fi } - -addtask assemble_fitimage_initramfs before do_deploy after do_bundle_initramfs +do_assemble_fitimage_initramfs[dirs] = "${B}" do_kernel_generate_rsa_keys() { if [ "${UBOOT_SIGN_ENABLE}" = "0" ] && [ "${FIT_GENERATE_KEYS}" = "1" ]; then
@@ -864,6 +870,7 @@ kernel_do_deploy:append() { if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then + # deploy the artifacts of do_assemble_fitimage bbnote "Copying fit-image.its source file..." install -m 0644 ${B}/fit-image.its "$deployDir/fitImage-its-${KERNEL_FIT_NAME}.its" if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then
@@ -878,12 +885,14 @@ kernel_do_deploy:append() { fi if [ -n "${INITRAMFS_IMAGE}" ]; then + # deploy the artifacts of do_assemble_fitimage_initramfs for bundled as well as un-bundled mode bbnote "Copying fit-image-${INITRAMFS_IMAGE}.its source file..." install -m 0644 ${B}/fit-image-${INITRAMFS_IMAGE}.its "$deployDir/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its" if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then ln -snf fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its "$deployDir/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_LINK_NAME}" fi + # deploy the artifacts of do_assemble_fitimage_initramfs for bundled mode only if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then bbnote "Copying fitImage-${INITRAMFS_IMAGE} file..." install -m 0644 ${B}/${KERNEL_OUTPUT_DIR}/fitImage-${INITRAMFS_IMAGE} "$deployDir/fitImage-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}${KERNEL_FIT_BIN_EXT}"
meta/classes-recipe/kernel.bbclass +17 -11
@@ -136,18 +136,24 @@ fi set -e """ % (type, type, type)) - image = d.getVar('INITRAMFS_IMAGE') - # If the INTIRAMFS_IMAGE is set but the INITRAMFS_IMAGE_BUNDLE is set to 0, - # the do_bundle_initramfs does nothing, but the INITRAMFS_IMAGE is built - # standalone for use by wic and other tools. if image: - if d.getVar('INITRAMFS_MULTICONFIG'): - d.appendVarFlag('do_bundle_initramfs', 'mcdepends', ' mc::${INITRAMFS_MULTICONFIG}:${INITRAMFS_IMAGE}:do_image_complete') + # If the INTIRAMFS_IMAGE is set but the INITRAMFS_IMAGE_BUNDLE is set to 0, + # the do_bundle_initramfs is not needed, but the INITRAMFS_IMAGE is built + # standalone for use by wic and other tools. + def add_initramfs_dep_task(initramfs_dep_task, d): + if d.getVar('INITRAMFS_MULTICONFIG'): + d.appendVarFlag(initramfs_dep_task, 'mcdepends', ' mc::${INITRAMFS_MULTICONFIG}:${INITRAMFS_IMAGE}:do_image_complete') + else: + d.appendVarFlag(initramfs_dep_task, 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete') + + if bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')): + bb.build.addtask('do_bundle_initramfs', 'do_deploy', 'do_install', d) + add_initramfs_dep_task('do_bundle_initramfs', d) + bb.build.addtask('do_transform_bundled_initramfs', 'do_deploy', 'do_bundle_initramfs', d) else: - d.appendVarFlag('do_bundle_initramfs', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete') - if image and bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')): - bb.build.addtask('do_transform_bundled_initramfs', 'do_deploy', 'do_bundle_initramfs', d) + add_initramfs_dep_task('do_deploy', d) + d.appendVarFlag('do_deploy', 'depends', ' ${PN}:do_install') # NOTE: setting INITRAMFS_TASK is for backward compatibility # The preferred method is to set INITRAMFS_IMAGE, because
@@ -327,6 +333,8 @@ do_bundle_initramfs () { mv -f ${KERNEL_OUTPUT_DIR}/$imageType.bak ${KERNEL_OUTPUT_DIR}/$imageType fi done + else + bbwarn "Calling do_bundle_initramfs with INITRAMFS_IMAGE_BUNDLE=1 is deprecated" fi } do_bundle_initramfs[dirs] = "${B}"
@@ -347,8 +355,6 @@ python do_devshell:prepend () { os.environ["LDFLAGS"] = '' } -addtask bundle_initramfs after do_install before do_deploy - KERNEL_DEBUG_TIMESTAMPS ??= "0" kernel_do_compile() {
Signature · 1 line
2.45.2
PATCH v2 6/6 kernel-fitimage: run unbundled fitimage after deploy
In reply to Adrian FreihoferFrom: Adrian Freihofer <[email protected]>
Without this change, the kernel is always rebuilt if something changes in the initramfs. If the kernel is rebuilt with an empty TMPDIR, the process starts by fetching the large kernel git repository and recompiling it from scratch. This is very inefficient.
This cannot be improved if INITRAMFS_IMAGE_BUNDLE = "1". If the kernel Makefile is needed to generate the initramfs bundle the kernel build folder is required. For this cases nothing changes.
But for a build configuration with INITRAMFS_IMAGE_BUNDLE = "" the build folder is not needed. Creating the initramfs bundle requires:
linux.bin, DTBs and the initramfs. These artifacts are all available in the deploy folder which is also sstate cached.
Therefore do_assemble_fitimage_initramfs gets split into two tasks:
- do_assemble_fitimage_initramfs creates the bundled fitImage which runs before do_deploy_image. This is without sstate and therefore the do_deploy_image task still deploys the artifacts generated by this task.
- do_deploy_fitimage_unbundled creates the fitImage in unbundled mode. This task runs after do_deploy_image. It uses the linux.bin artifact from do_deploy_image to create the fitImage with initramfs. This task is done with SSTATE_SKIP_CREATION, because this also works with the eSDK installer (oe-selftest -r esdk is successful). It is also more effective to quickly create the fit image instead of handling it via sstate.
Unfortunately, this change is not 100% backward compatible. Previously, a dependency on do_deploy deployed all types of fit images. With this change, deploying the unbundled fit image requires a dependency on do_deploy_fitimage_unbundled. The assignment of the KERNEL_DEPLOY_DEPEND is improved to handle this appropriately.
--- meta/classes-recipe/image.bbclass | 12 ++- meta/classes-recipe/kernel-fitimage.bbclass | 84 +++++++++++++-------- meta/classes-recipe/kernel.bbclass | 3 +- 3 files changed, 66 insertions(+), 33 deletions(-)
meta/classes-recipe/image.bbclass +11 -1
@@ -147,7 +147,17 @@ do_rootfs[vardeps] += "${@rootfs_variables(d)}" # This follows many common usecases and user expectations. # But if you are building an image which doesn't need the kernel image at all, # you can unset this variable manually. -KERNEL_DEPLOY_DEPEND ?= "virtual/kernel:do_deploy" +def get_kernel_do_deploy(d): + kerneltypes = d.getVar('KERNEL_IMAGETYPES') or "" + kerneltype = d.getVar('KERNEL_IMAGETYPE') or "" + fitimage = kerneltype == 'fitImage' or 'fitImage' in kerneltypes.split() + initramfs_image = d.getVar('INITRAMFS_IMAGE') or "" + bundled = d.getVar('INITRAMFS_IMAGE_BUNDLE') == '1' + if fitimage and initramfs_image and not bundled: + return "virtual/kernel:do_deploy_fitimage_unbundled" + return "virtual/kernel:do_deploy" + +KERNEL_DEPLOY_DEPEND ?= "${@get_kernel_do_deploy(d)}" do_build[depends] += "${KERNEL_DEPLOY_DEPEND}"
meta/classes-recipe/kernel-fitimage.bbclass +53 -31
118 lines not on this page · raw message
meta/classes-recipe/kernel.bbclass +2 -1
@@ -488,7 +488,8 @@ kernel_do_install() { # So, at the level of the install task we should not try to install the fitImage. fitImage is still not # generated yet. # After the generation of the fitImage, the deploy task copies the fitImage from the build directory to - # the deploy folder. + # the deploy folder. If INITRAMFS_IMAGE_BUNDLE != 1 the fitImage with initramfs is deployed after the + # deploy task. # for imageType in ${KERNEL_IMAGETYPES} ; do
Signature · 1 line
2.45.2
task-depends Task dependeny changes
Configuration: image, fitimage, initramfs, unbundled
KERNEL_IMAGETYPE = "Image" KERNEL_IMAGETYPES += " fitImage " KERNEL_CLASSES = " kernel-fitimage " IMAGE_FSTYPES += "cpio.gz" INITRAMFS_IMAGE = "core-image-minimal-initramfs"
branch: master
depends
undefined
core-image-minimal.do_build undefined core-image-minimal.do_build
do_deploy
undefineddo_deploy
core-image-minimal.do_build->do_deploy undefined
undefined
do_assemble_fitimage_initramfs undefined do_assemble_fitimage_initramfs
do_deploy->do_assemble_fitimage_initramfs undefined
undefined
do_bundle_initramfs
undefineddo_bundle_initramfs
do_deploy->do_bundle_initramfs undefined
undefined
do_populate_sysroot
undefineddo_populate_sysroot
do_deploy->do_populate_sysroot undefined
undefined
do_assemble_fitimage
undefineddo_assemble_fitimage
do_compile
undefineddo_compile
do_assemble_fitimage->do_compile undefined
undefined
do_kernel_generate_rsa_keys undefined do_kernel_generate_rsa_keys
do_assemble_fitimage->do_kernel_generate_rsa_keys undefined
undefined
do_kernel_generate_rsa_keys->do_compile undefined
undefined
core-image-minimal.do_image_complete undefined core-image-minimal.do_image_complete
do_assemble_fitimage_initramfs->core-image-minimal.do_image_complete undefined
undefined
do_assemble_fitimage_initramfs->do_bundle_initramfs undefined
undefined
do_bundle_initramfs->core-image-minimal.do_image_complete undefined
undefined
do_install
undefineddo_install
do_bundle_initramfs->do_install undefined
undefined
do_install->do_assemble_fitimage undefined
undefined
do_install->do_compile undefined
undefined
do_populate_sysroot->do_install undefined
undefined
branch: adrianf/kernel-fitimage-sstate
depends
undefined
core-image-minimal.do_build undefined core-image-minimal.do_build
do_deploy
undefineddo_deploy
core-image-minimal.do_build->do_deploy undefined
undefined
do_deploy_fitimage_unbundled undefined do_deploy_fitimage_unbundled
core-image-minimal.do_build->do_deploy_fitimage_unbundled undefined
undefined
core-image-minimal.do_image_complete undefined core-image-minimal.do_image_complete
do_deploy->core-image-minimal.do_image_complete undefined
undefined
do_install
undefineddo_install
do_deploy->do_install undefined
undefined
do_populate_sysroot
undefineddo_populate_sysroot
do_deploy->do_populate_sysroot undefined
undefined
do_deploy_fitimage_unbundled->do_deploy undefined
undefined
do_deploy_fitimage_unbundled->core-image-minimal.do_image_complete undefined
undefined
do_assemble_fitimage
undefineddo_assemble_fitimage
do_compile
undefineddo_compile
do_assemble_fitimage->do_compile undefined
undefined
do_kernel_generate_rsa_keys undefined do_kernel_generate_rsa_keys
do_assemble_fitimage->do_kernel_generate_rsa_keys undefined
undefined
do_kernel_generate_rsa_keys->do_compile undefined
undefined
do_install->do_assemble_fitimage undefined
undefined
do_install->do_compile undefined
undefined
do_populate_sysroot->do_install undefined
undefined
Diff
2a3
Quoted 1 line
"core-image-minimal.do_build" -> "
do_deploy_fitimage_unbundled"
6,11d6
< "do_assemble_fitimage_initramfs" [label="do_assemble_fitimage_initramfs"]
< "do_assemble_fitimage_initramfs" -> "core-image- minimal.do_image_complete"
< "do_assemble_fitimage_initramfs" -> "do_bundle_initramfs" < "do_bundle_initramfs" [label="do_bundle_initramfs"]
< "do_bundle_initramfs" -> "core-image-minimal.do_image_complete" < "do_bundle_initramfs" -> "do_install"
14,15c9,10
< "do_deploy" -> "do_assemble_fitimage_initramfs"
< "do_deploy" -> "do_bundle_initramfs"
---
Quoted 2 lines
"
do_deploy" -> "core-image-minimal.do_image_complete" "do_deploy" -> "do_install"
16a12,14
Quoted 2 lines
"
do_deploy_fitimage_unbundled" [label="do_deploy_fitimage_unbundled"] "do_deploy_fitimage_unbundled" -> "core-image-
minimal.do_image_complete"
Quoted 1 line
"
do_deploy_fitimage_unbundled" -> "do_deploy"
Configuration: fitimage, initramfs, unbundled
KERNEL_IMAGETYPE:forcevariable = "fitImage" KERNEL_CLASSES = " kernel-fitimage " IMAGE_FSTYPES += "cpio.gz" INITRAMFS_IMAGE = "core-image-minimal-initramfs"
branch: master
depends
undefined
core-image-minimal.do_build undefined core-image-minimal.do_build
do_deploy
undefineddo_deploy
core-image-minimal.do_build->do_deploy undefined
undefined
do_assemble_fitimage_initramfs undefined do_assemble_fitimage_initramfs
do_deploy->do_assemble_fitimage_initramfs undefined
undefined
do_bundle_initramfs
undefineddo_bundle_initramfs
do_deploy->do_bundle_initramfs undefined
undefined
do_populate_sysroot
undefineddo_populate_sysroot
do_deploy->do_populate_sysroot undefined
undefined
do_assemble_fitimage
undefineddo_assemble_fitimage
do_compile
undefineddo_compile
do_assemble_fitimage->do_compile undefined
undefined
do_kernel_generate_rsa_keys undefined do_kernel_generate_rsa_keys
do_assemble_fitimage->do_kernel_generate_rsa_keys undefined
undefined
do_kernel_generate_rsa_keys->do_compile undefined
undefined
core-image-minimal.do_image_complete undefined core-image-minimal.do_image_complete
do_assemble_fitimage_initramfs->core-image-minimal.do_image_complete undefined
undefined
do_assemble_fitimage_initramfs->do_bundle_initramfs undefined
undefined
do_bundle_initramfs->core-image-minimal.do_image_complete undefined
undefined
do_install
undefineddo_install
do_bundle_initramfs->do_install undefined
undefined
do_install->do_assemble_fitimage undefined
undefined
do_install->do_compile undefined
undefined
do_populate_sysroot->do_install undefined
undefined
branch: adrianf/kernel-fitimage-sstate
depends
undefined
core-image-minimal.do_build undefined core-image-minimal.do_build
do_deploy
undefineddo_deploy
core-image-minimal.do_build->do_deploy undefined
undefined
do_deploy_fitimage_unbundled undefined do_deploy_fitimage_unbundled
core-image-minimal.do_build->do_deploy_fitimage_unbundled undefined
undefined
core-image-minimal.do_image_complete undefined core-image-minimal.do_image_complete
do_deploy->core-image-minimal.do_image_complete undefined
undefined
do_install
undefineddo_install
do_deploy->do_install undefined
undefined
do_populate_sysroot
undefineddo_populate_sysroot
do_deploy->do_populate_sysroot undefined
undefined
do_deploy_fitimage_unbundled->do_deploy undefined
undefined
do_deploy_fitimage_unbundled->core-image-minimal.do_image_complete undefined
undefined
do_assemble_fitimage
undefineddo_assemble_fitimage
do_compile
undefineddo_compile
do_assemble_fitimage->do_compile undefined
undefined
do_kernel_generate_rsa_keys undefined do_kernel_generate_rsa_keys
do_assemble_fitimage->do_kernel_generate_rsa_keys undefined
undefined
do_kernel_generate_rsa_keys->do_compile undefined
undefined
do_install->do_assemble_fitimage undefined
undefined
do_install->do_compile undefined
undefined
do_populate_sysroot->do_install undefined
undefined
Diff
2a3
Quoted 1 line
"core-image-minimal.do_build" -> "
do_deploy_fitimage_unbundled"
6,11d6
< "do_assemble_fitimage_initramfs" [label="do_assemble_fitimage_initramfs"]
< "do_assemble_fitimage_initramfs" -> "core-image- minimal.do_image_complete"
< "do_assemble_fitimage_initramfs" -> "do_bundle_initramfs" < "do_bundle_initramfs" [label="do_bundle_initramfs"]
< "do_bundle_initramfs" -> "core-image-minimal.do_image_complete" < "do_bundle_initramfs" -> "do_install"
14,15c9,10
< "do_deploy" -> "do_assemble_fitimage_initramfs"
< "do_deploy" -> "do_bundle_initramfs"
---
Quoted 2 lines
"
do_deploy" -> "core-image-minimal.do_image_complete" "do_deploy" -> "do_install"
16a12,14
Quoted 2 lines
"
do_deploy_fitimage_unbundled" [label="do_deploy_fitimage_unbundled"] "do_deploy_fitimage_unbundled" -> "core-image-
minimal.do_image_complete"
Quoted 1 line
"
do_deploy_fitimage_unbundled" -> "do_deploy"
Configuration: image, fitimage, initramfs, bundled
KERNEL_IMAGETYPE = "Image" KERNEL_IMAGETYPES += " fitImage " KERNEL_CLASSES = " kernel-fitimage " IMAGE_FSTYPES += "cpio.gz" INITRAMFS_IMAGE = "core-image-minimal-initramfs" INITRAMFS_IMAGE_BUNDLE = "1"
branch: master
depends
undefined
core-image-minimal.do_build undefined core-image-minimal.do_build
do_deploy
undefineddo_deploy
core-image-minimal.do_build->do_deploy undefined
undefined
do_assemble_fitimage_initramfs undefined do_assemble_fitimage_initramfs
do_deploy->do_assemble_fitimage_initramfs undefined
undefined
do_bundle_initramfs
undefineddo_bundle_initramfs
do_deploy->do_bundle_initramfs undefined
undefined
do_populate_sysroot
undefineddo_populate_sysroot
do_deploy->do_populate_sysroot undefined
undefined
do_transform_bundled_initramfs undefined do_transform_bundled_initramfs
do_deploy->do_transform_bundled_initramfs undefined
undefined
do_assemble_fitimage
undefineddo_assemble_fitimage
do_compile
undefineddo_compile
do_assemble_fitimage->do_compile undefined
undefined
do_kernel_generate_rsa_keys undefined do_kernel_generate_rsa_keys
do_assemble_fitimage->do_kernel_generate_rsa_keys undefined
undefined
do_kernel_generate_rsa_keys->do_compile undefined
undefined
core-image-minimal.do_image_complete undefined core-image-minimal.do_image_complete
do_assemble_fitimage_initramfs->core-image-minimal.do_image_complete undefined
undefined
do_assemble_fitimage_initramfs->do_bundle_initramfs undefined
undefined
do_bundle_initramfs->core-image-minimal.do_image_complete undefined
undefined
do_install
undefineddo_install
do_bundle_initramfs->do_install undefined
undefined
do_install->do_assemble_fitimage undefined
undefined
do_install->do_compile undefined
undefined
do_populate_sysroot->do_install undefined
undefined
do_transform_bundled_initramfs->do_bundle_initramfs undefined
undefined
branch: adrianf/kernel-fitimage-sstate
depends
undefined
core-image-minimal.do_build undefined core-image-minimal.do_build
do_deploy
undefineddo_deploy
core-image-minimal.do_build->do_deploy undefined
undefined
do_assemble_fitimage_initramfs undefined do_assemble_fitimage_initramfs
do_deploy->do_assemble_fitimage_initramfs undefined
undefined
do_bundle_initramfs
undefineddo_bundle_initramfs
do_deploy->do_bundle_initramfs undefined
undefined
do_populate_sysroot
undefineddo_populate_sysroot
do_deploy->do_populate_sysroot undefined
undefined
do_transform_bundled_initramfs undefined do_transform_bundled_initramfs
do_deploy->do_transform_bundled_initramfs undefined
undefined
do_assemble_fitimage
undefineddo_assemble_fitimage
do_compile
undefineddo_compile
do_assemble_fitimage->do_compile undefined
undefined
do_kernel_generate_rsa_keys undefined do_kernel_generate_rsa_keys
do_assemble_fitimage->do_kernel_generate_rsa_keys undefined
undefined
do_kernel_generate_rsa_keys->do_compile undefined
undefined
core-image-minimal.do_image_complete undefined core-image-minimal.do_image_complete
do_assemble_fitimage_initramfs->core-image-minimal.do_image_complete undefined
undefined
do_assemble_fitimage_initramfs->do_bundle_initramfs undefined
undefined
do_bundle_initramfs->core-image-minimal.do_image_complete undefined
undefined
do_install
undefineddo_install
do_bundle_initramfs->do_install undefined
undefined
do_install->do_assemble_fitimage undefined
undefined
do_install->do_compile undefined
undefined
do_populate_sysroot->do_install undefined
undefined
do_transform_bundled_initramfs->do_bundle_initramfs undefined
undefined
Diff
Configuration: image, fitimage
KERNEL_IMAGETYPE = "Image" KERNEL_IMAGETYPES += " fitImage " KERNEL_CLASSES = " kernel-fitimage "
branch: master
depends
undefined
core-image-minimal.do_build undefined core-image-minimal.do_build
do_deploy
undefineddo_deploy
core-image-minimal.do_build->do_deploy undefined
undefined
do_assemble_fitimage_initramfs undefined do_assemble_fitimage_initramfs
do_deploy->do_assemble_fitimage_initramfs undefined
undefined
do_bundle_initramfs
undefineddo_bundle_initramfs
do_deploy->do_bundle_initramfs undefined
undefined
do_populate_sysroot
undefineddo_populate_sysroot
do_deploy->do_populate_sysroot undefined
undefined
do_assemble_fitimage
undefineddo_assemble_fitimage
do_compile
undefineddo_compile
do_assemble_fitimage->do_compile undefined
undefined
do_kernel_generate_rsa_keys undefined do_kernel_generate_rsa_keys
do_assemble_fitimage->do_kernel_generate_rsa_keys undefined
undefined
do_kernel_generate_rsa_keys->do_compile undefined
undefined
core-image-minimal.do_image_complete undefined core-image-minimal.do_image_complete
do_assemble_fitimage_initramfs->core-image-minimal.do_image_complete undefined
undefined
do_assemble_fitimage_initramfs->do_bundle_initramfs undefined
undefined
do_bundle_initramfs->core-image-minimal.do_image_complete undefined
undefined
do_install
undefineddo_install
do_bundle_initramfs->do_install undefined
undefined
do_install->do_assemble_fitimage undefined
undefined
do_install->do_compile undefined
undefined
do_populate_sysroot->do_install undefined
undefined
branch: adrianf/kernel-fitimage-sstate
depends
undefined
core-image-minimal.do_build undefined core-image-minimal.do_build
do_deploy
undefineddo_deploy
core-image-minimal.do_build->do_deploy undefined
undefined
do_deploy_fitimage_unbundled undefined do_deploy_fitimage_unbundled
core-image-minimal.do_build->do_deploy_fitimage_unbundled undefined
undefined
core-image-minimal.do_image_complete undefined core-image-minimal.do_image_complete
do_deploy->core-image-minimal.do_image_complete undefined
undefined
do_install
undefineddo_install
do_deploy->do_install undefined
undefined
do_populate_sysroot
undefineddo_populate_sysroot
do_deploy->do_populate_sysroot undefined
undefined
do_deploy_fitimage_unbundled->do_deploy undefined
undefined
do_deploy_fitimage_unbundled->core-image-minimal.do_image_complete undefined
undefined
do_assemble_fitimage
undefineddo_assemble_fitimage
do_compile
undefineddo_compile
do_assemble_fitimage->do_compile undefined
undefined
do_kernel_generate_rsa_keys undefined do_kernel_generate_rsa_keys
do_assemble_fitimage->do_kernel_generate_rsa_keys undefined
undefined
do_kernel_generate_rsa_keys->do_compile undefined
undefined
do_install->do_assemble_fitimage undefined
undefined
do_install->do_compile undefined
undefined
do_populate_sysroot->do_install undefined
undefined
Diff
2a3
Quoted 1 line
"core-image-minimal.do_build" -> "
do_deploy_fitimage_unbundled"
6,11d6
< "do_assemble_fitimage_initramfs" [label="do_assemble_fitimage_initramfs"]
< "do_assemble_fitimage_initramfs" -> "core-image- minimal.do_image_complete"
< "do_assemble_fitimage_initramfs" -> "do_bundle_initramfs" < "do_bundle_initramfs" [label="do_bundle_initramfs"]
< "do_bundle_initramfs" -> "core-image-minimal.do_image_complete" < "do_bundle_initramfs" -> "do_install"
14,15c9,10
< "do_deploy" -> "do_assemble_fitimage_initramfs"
< "do_deploy" -> "do_bundle_initramfs"
---
Quoted 2 lines
"
do_deploy" -> "core-image-minimal.do_image_complete" "do_deploy" -> "do_install"
16a12,14
Quoted 2 lines
"
do_deploy_fitimage_unbundled" [label="do_deploy_fitimage_unbundled"] "do_deploy_fitimage_unbundled" -> "core-image-
minimal.do_image_complete"
Quoted 1 line
"
do_deploy_fitimage_unbundled" -> "do_deploy"
Doesn't it still rebuild from scratch when IMAGE_VERSION_SUFFIX changes?
To be able to re-use the kernel from sstate even when IMAGE_VERSION_SUFFIX changes I've implemented:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12937
which was unfortunately rejected due to lack of interest from other people outside LGE, but maybe you could take advantage of that as well.
Quoted 142 lines · Adrian Freihofer via lists.openembedded.org
Task dependeny changes
Configuration: image, fitimage, initramfs, unbundled
KERNEL_IMAGETYPE= "Image"KERNEL_IMAGETYPES+= " fitImage "KERNEL_CLASSES= " kernel-fitimage "IMAGE_FSTYPES+= "cpio.gz"INITRAMFS_IMAGE= "core-image-minimal-initramfs"branch: master
core-image-minimal.do_build
do_deploydo_assemble_fitimage_initramfsdo_bundle_initramfsdo_populate_sysrootdo_assemble_fitimagedo_compiledo_kernel_generate_rsa_keyscore-image-minimal.do_image_completedo_installbranch: adrianf/kernel-fitimage-sstate
core-image-minimal.do_build
do_deploydo_deploy_fitimage_unbundledcore-image-minimal.do_image_completedo_installdo_populate_sysrootdo_assemble_fitimagedo_compiledo_kernel_generate_rsa_keysDiff
2a3
Quoted 1 line
"core-image-minimal.do_build" -> "
do_deploy_fitimage_unbundled"6,11d6
< "do_assemble_fitimage_initramfs" [label="do_assemble_fitimage_initramfs"] < "do_assemble_fitimage_initramfs" -> "core-image-minimal.do_image_complete" < "do_assemble_fitimage_initramfs" -> "do_bundle_initramfs"
< "do_bundle_initramfs" [label="do_bundle_initramfs"]
< "do_bundle_initramfs" -> "core-image-minimal.do_image_complete" < "do_bundle_initramfs" -> "do_install"
14,15c9,10
< "do_deploy" -> "do_assemble_fitimage_initramfs"
< "do_deploy" -> "do_bundle_initramfs"
---Quoted 2 lines
"
do_deploy" -> "core-image-minimal.do_image_complete" "do_deploy" -> "do_install"16a12,14
Quoted 3 lines
"
do_deploy_fitimage_unbundled" [label="do_deploy_fitimage_unbundled"] "do_deploy_fitimage_unbundled" -> "core-image-minimal.do_image_complete" "do_deploy_fitimage_unbundled" -> "do_deploy"Configuration: fitimage, initramfs, unbundled
KERNEL_IMAGETYPE:forcevariable = "fitImage"
KERNEL_CLASSES= " kernel-fitimage "IMAGE_FSTYPES+= "cpio.gz"INITRAMFS_IMAGE= "core-image-minimal-initramfs"branch: master
core-image-minimal.do_build
do_deploydo_assemble_fitimage_initramfsdo_bundle_initramfsdo_populate_sysrootdo_assemble_fitimagedo_compiledo_kernel_generate_rsa_keyscore-image-minimal.do_image_completedo_installbranch: adrianf/kernel-fitimage-sstate
core-image-minimal.do_build
do_deploydo_deploy_fitimage_unbundledcore-image-minimal.do_image_completedo_installdo_populate_sysrootdo_assemble_fitimagedo_compiledo_kernel_generate_rsa_keysDiff
2a3
Quoted 1 line
"core-image-minimal.do_build" -> "
do_deploy_fitimage_unbundled"6,11d6
< "do_assemble_fitimage_initramfs" [label="do_assemble_fitimage_initramfs"] < "do_assemble_fitimage_initramfs" -> "core-image-minimal.do_image_complete" < "do_assemble_fitimage_initramfs" -> "do_bundle_initramfs"
< "do_bundle_initramfs" [label="do_bundle_initramfs"]
< "do_bundle_initramfs" -> "core-image-minimal.do_image_complete" < "do_bundle_initramfs" -> "do_install"
14,15c9,10
< "do_deploy" -> "do_assemble_fitimage_initramfs"
< "do_deploy" -> "do_bundle_initramfs"
---Quoted 2 lines
"
do_deploy" -> "core-image-minimal.do_image_complete" "do_deploy" -> "do_install"16a12,14
Quoted 3 lines
"
do_deploy_fitimage_unbundled" [label="do_deploy_fitimage_unbundled"] "do_deploy_fitimage_unbundled" -> "core-image-minimal.do_image_complete" "do_deploy_fitimage_unbundled" -> "do_deploy"Configuration: image, fitimage, initramfs, bundled
KERNEL_IMAGETYPE= "Image"KERNEL_IMAGETYPES+= " fitImage "KERNEL_CLASSES= " kernel-fitimage "IMAGE_FSTYPES+= "cpio.gz"INITRAMFS_IMAGE= "core-image-minimal-initramfs"INITRAMFS_IMAGE_BUNDLE= "1"branch: master
core-image-minimal.do_build
do_deploydo_assemble_fitimage_initramfsdo_bundle_initramfsdo_populate_sysrootdo_transform_bundled_initramfsdo_assemble_fitimagedo_compiledo_kernel_generate_rsa_keyscore-image-minimal.do_image_completedo_installbranch: adrianf/kernel-fitimage-sstate
core-image-minimal.do_build
do_deploydo_assemble_fitimage_initramfsdo_bundle_initramfsdo_populate_sysrootdo_transform_bundled_initramfsdo_assemble_fitimagedo_compiledo_kernel_generate_rsa_keyscore-image-minimal.do_image_completedo_installDiff
Configuration: image, fitimage
KERNEL_IMAGETYPE= "Image"KERNEL_IMAGETYPES+= " fitImage "KERNEL_CLASSES= " kernel-fitimage "branch: master
core-image-minimal.do_build
do_deploydo_assemble_fitimage_initramfsdo_bundle_initramfsdo_populate_sysrootdo_assemble_fitimagedo_compiledo_kernel_generate_rsa_keyscore-image-minimal.do_image_completedo_installbranch: adrianf/kernel-fitimage-sstate
core-image-minimal.do_build
do_deploydo_deploy_fitimage_unbundledcore-image-minimal.do_image_completedo_installdo_populate_sysrootdo_assemble_fitimagedo_compiledo_kernel_generate_rsa_keysDiff
2a3
Quoted 1 line
"core-image-minimal.do_build" -> "
do_deploy_fitimage_unbundled"6,11d6
< "do_assemble_fitimage_initramfs" [label="do_assemble_fitimage_initramfs"] < "do_assemble_fitimage_initramfs" -> "core-image-minimal.do_image_complete" < "do_assemble_fitimage_initramfs" -> "do_bundle_initramfs"
< "do_bundle_initramfs" [label="do_bundle_initramfs"]
< "do_bundle_initramfs" -> "core-image-minimal.do_image_complete" < "do_bundle_initramfs" -> "do_install"
14,15c9,10
< "do_deploy" -> "do_assemble_fitimage_initramfs"
< "do_deploy" -> "do_bundle_initramfs"
---Quoted 2 lines
"
do_deploy" -> "core-image-minimal.do_image_complete" "do_deploy" -> "do_install"16a12,14
Quoted 3 lines
"
do_deploy_fitimage_unbundled" [label="do_deploy_fitimage_unbundled"] "do_deploy_fitimage_unbundled" -> "core-image-minimal.do_image_complete" "do_deploy_fitimage_unbundled" -> "do_deploy"-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201933): https://lists.openembedded.org/g/openembedded-core/message/201933
Mute This Topic: https://lists.openembedded.org/mt/107231736/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Hi Martin
Thank you for looking at my patches.
Quoted 2 lines · Martin Jansa
Doesn't it still rebuild from scratch when
IMAGE_VERSION_SUFFIXchanges?
I wonder why this happens in WebOS. I think that this line kernel_do_deploy[vardepsexclude] = "DATETIME" prevents such unnecessary rebuilds.
So far we haven't had any problems with rebuilding kernels. The trouble started when we started using fitImages with initramfs for some devices. It is also worth noting that we build from sstate, but with an empty TMPDIR.
In such a scenario bitbake does:
* The initramfs gets assembled. This is expected because images are not sstate cached. This runs quickly because all the packages which are required to build the initramfs come from sstate-cache.
* Since the initramfs changes, the fitImage which includes it must be re-assembled as well.
* Now the hassle starts: The do_assemble_fitimage_initramfs needs the kernel binaries (linux.bin, DTBs...) which are not available from sttate. Bitbake has to start with kernel do_fetch in case of an empty TMPDIR just to get the fitImage assembled.
With my patches the kernel binaries used for the fitImage come from sstate (if the initramfs is not bundled).
I tested this like that:
cat build/conf/local.conf
KERNEL_IMAGETYPE = "Image"KERNEL_IMAGETYPES += " fitImage "KERNEL_CLASSES = " kernel-fitimage "and RAM diskIMAGE_FSTYPES += "cpio.gz"INITRAMFS_IMAGE = "core-image-minimal"
IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""UBOOT_RD_LOADADDRESS = "0x88000000"UBOOT_RD_ENTRYPOINT = "0x88000000"UBOOT_LOADADDRESS = "0x80080000"UBOOT_ENTRYPOINT = "0x80080000"FIT_DESC = "A model description"
# Allow to change the initramfsFOO_VAR = "1"
INHERIT += "image-buildinfo"
IMAGE_BUILDINFO_VARS:append = " FOO_VAR"
bitbake virtual/kernel
mv build/tmp build/tmp-old-1
bitbake virtual/kernel
-> bitbake does:
- setscene tasks
- core-image-minimal do_rootfs ...
- kernel do_deploy (from sstate)
- kernel do_deploy_fitimage_unbundled
bitbake virtual/kernel
-> bitbake does: nothing
# Enforce a re-build of the initramfs
mv build/tmp build/tmp-old-2
echo 'FOO_VAR = "1"' >> build/conf/local.conf
bitbake virtual/kernel
-> bitbake does:
- setscene tasks
- core-image-minimal do_rootfs ...
- kernel do_deploy (from sstate)
- kernel do_deploy_fitimage_unbundled
bitbake virtual/kernel
-> bitbake does: nothing
diff \
<(ls build/tmp-old-2/deploy/images/qemux86-64 -1) \
<(ls build/tmp/deploy/images/qemux86-64/ -1) 4,10c4,10
< core-image-minimal-qemux86-64-20240718065105.cpio.gz < core-image-minimal-qemux86-64-20240718065105.ext4 < core-image-minimal-qemux86-64-20240718065105.manifest < core-image-minimal-qemux86-64-20240718065105.qemuboot.conf < core-image-minimal-qemux86-64-20240718065105.spdx.tar.zst < core-image-minimal-qemux86-64-20240718065105.tar.bz2 < core-image-minimal-qemux86-64-20240718065105.testdata.json
---
Quoted 7 lines
core-image-minimal-qemux86-64-20240718073341.cpio.gz core-image-minimal-qemux86-64-20240718073341.ext4 core-image-minimal-qemux86-64-20240718073341.manifest core-image-minimal-qemux86-64-20240718073341.qemuboot.conf core-image-minimal-qemux86-64-20240718073341.spdx.tar.zst core-image-minimal-qemux86-64-20240718073341.tar.bz2 core-image-minimal-qemux86-64-20240718073341.testdata.json
20c20
< fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64- 20240718065105.bin
---
Quoted 1 line
fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
20240718073341.bin
22c22
< fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64- 20240718065105.its
---
Quoted 1 line
fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
20240718073341.its
The answer is then: No it does not re-build. At least not in the scenario which I would like to fix it, it does not rebuild from scratch.
Does that make sense?
Regards,
Adrian
Quoted 13 lines
Quoted 13 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#201933):
https://lists.openembedded.org/g/openembedded-core/message/201933
Mute This Topic:
https://lists.openembedded.org/mt/107231736/3617156
Group Owner: [email protected] Unsubscribe:
https://lists.openembedded.org/g/openembedded-core/unsub [
[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 5 lines · Adrian Freihofer
Hi Martin
Thank you for looking at my patches.
Quoted 7 lines
Quoted 2 lines · Martin Jansa
Doesn't it still rebuild from scratch when
IMAGE_VERSION_SUFFIXchanges?I wonder why this happens in WebOS. I think that this line kernel_do_deploy[vardepsexclude] = "DATETIME" prevents such unnecessary rebuilds.
Hi Adrian,
yes DATETIME is excluded by default, but it's useful to use different suffix in IMAGE_VERSION_SUFFIX (e.g. BUILD_NUMBER from CI on jenkins or release version when building release) and in such cases you don't want to vardepexclude it, because it's useful to produce matching version in images as well as kernel (and other artifacts you might have), so your 1.2.1 release images doesn't end with 1.2.0 kernel artifacts just because kernel sstate signature didn't change in 1.2.1 bugfix release.
Does this make sense? I bet it would happen in your builds as well.
Whole point of:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12937
is to allow "renaming" (adding more hardlinks) artifacts with matching IMAGE_VERSION_SUFFIX without the need of rebuilding anything (e.g. version-less kernel artifacts are re-used from sstate and then only quick deploy-links task is executed to add right versioned hardlinks in deploy directory.)
Cheers,
Quoted 131 lines
So far we haven't had any problems with rebuilding kernels. The trouble started when we started using fitImages with initramfs for some devices. It is also worth noting that we build from sstate, but with an empty TMPDIR.
In such a scenario bitbake does:
* The initramfs gets assembled. This is expected because images are not sstate cached. This runs quickly because all the packages which are required to build the initramfs come from sstate-cache.
* Since the initramfs changes, the fitImage which includes it must be re-assembled as well.
* Now the hassle starts: Thedo_assemble_fitimage_initramfsneeds the kernel binaries (linux.bin, DTBs...) which are not available from sttate. Bitbake has to start with kerneldo_fetchin case of an empty TMPDIR just to get the fitImage assembled.With my patches the kernel binaries used for the fitImage come from sstate (if the initramfs is not bundled).
I tested this like that:
cat
build/conf/local.conf
KERNEL_IMAGETYPE= "Image"KERNEL_IMAGETYPES+= " fitImage "KERNEL_CLASSES= " kernel-fitimage "and RAM diskIMAGE_FSTYPES+= "cpio.gz"INITRAMFS_IMAGE= "core-image-minimal"
IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""UBOOT_RD_LOADADDRESS= "0x88000000"UBOOT_RD_ENTRYPOINT= "0x88000000"UBOOT_LOADADDRESS= "0x80080000"UBOOT_ENTRYPOINT= "0x80080000"FIT_DESC= "A model description"# Allow to change the initramfs
FOO_VAR= "1"
INHERIT += "image-buildinfo"
IMAGE_BUILDINFO_VARS:append = "FOO_VAR"bitbake virtual/kernel
mv build/tmp build/tmp-old-1
bitbake virtual/kernel
-> bitbake does:
- setscene tasks
- core-image-minimaldo_rootfs...
- kerneldo_deploy(from sstate)
- kerneldo_deploy_fitimage_unbundledbitbake virtual/kernel
-> bitbake does: nothing# Enforce a re-build of the initramfs
mv build/tmp build/tmp-old-2
echo 'FOO_VAR= "1"' >>build/conf/local.confbitbake virtual/kernel
-> bitbake does:
- setscene tasks
- core-image-minimaldo_rootfs...
- kerneldo_deploy(from sstate)
- kerneldo_deploy_fitimage_unbundledbitbake virtual/kernel
-> bitbake does: nothingdiff \
<(ls build/tmp-old-2/deploy/images/qemux86-64 -1) \
<(ls build/tmp/deploy/images/qemux86-64/ -1) 4,10c4,10
< core-image-minimal-qemux86-64-20240718065105.cpio.gz < core-image-minimal-qemux86-64-20240718065105.ext4 < core-image-minimal-qemux86-64-20240718065105.manifest < core-image-minimal-qemux86-64-20240718065105.qemuboot.conf < core-image-minimal-qemux86-64-20240718065105.spdx.tar.zst < core-image-minimal-qemux86-64-20240718065105.tar.bz2 < core-image-minimal-qemux86-64-20240718065105.testdata.json
---Quoted 7 lines
core-image-minimal-qemux86-64-20240718073341.cpio.gz core-image-minimal-qemux86-64-20240718073341.ext4 core-image-minimal-qemux86-64-20240718073341.manifest core-image-minimal-qemux86-64-20240718073341.qemuboot.conf core-image-minimal-qemux86-64-20240718073341.spdx.tar.zst core-image-minimal-qemux86-64-20240718073341.tar.bz2 core-image-minimal-qemux86-64-20240718073341.testdata.json
20c20
< fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64- 20240718065105.bin
---Quoted 1 line
fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
20240718073341.bin
22c22
< fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64- 20240718065105.its
---Quoted 1 line
fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
20240718073341.its
The answer is then: No it does not re-build. At least not in the scenario which I would like to fix it, it does not rebuild from scratch.
Does that make sense?
Regards,
AdrianQuoted 13 lines
Quoted 13 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#201933):
https://lists.openembedded.org/g/openembedded-core/message/201933
Mute This Topic:
https://lists.openembedded.org/mt/107231736/3617156
Group Owner: [email protected] Unsubscribe:
https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 26 lines · Martin Jansa
Quoted 5 lines · Adrian Freihofer
Hi Martin
Thank you for looking at my patches.
Quoted 7 lines
Quoted 2 lines · Martin Jansa
Doesn't it still rebuild from scratch when
IMAGE_VERSION_SUFFIXchanges?I wonder why this happens in WebOS. I think that this line kernel_do_deploy[vardepsexclude] = "DATETIME" prevents such unnecessary rebuilds.
Hi Adrian,
yes DATETIME is excluded by default, but it's useful to use different suffix in
IMAGE_VERSION_SUFFIX(e.g.BUILD_NUMBERfrom CI on jenkins or release version when building release) and in such cases you don't want to vardepexclude it, because it's useful to produce matching version in images as well as kernel (and other artifacts you might have), so your 1.2.1 release images doesn't end with 1.2.0 kernel artifacts just because kernel sstate signature didn't change in 1.2.1 bugfix release.
Quoted 1 line
Does this make sense? I bet it would happen in your builds as well.
I try to optimize everything towards binary-reproducible builds. This means that we remove all kinds of variables that change because the time changes or because the CI build ID changes or something like that. I see no benefit in compiling such information into the firmware. But the problem I see is that such ideas clash with efficient and binary-reproducible builds.
That means that I don't see a reason for changing the IMAGE_VERSION_SUFFIX variable in such a way.
Quoted 10 lines
Whole point of:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12937
is to allow "renaming" (adding more hardlinks) artifacts with matchingIMAGE_VERSION_SUFFIXwithout the need of rebuilding anything (e.g. version-less kernel artifacts are re-used from sstate and then only quick deploy-links task is executed to add right versioned hardlinks in deploy directory.)
Well, yes, it would. But, I am confident that it is a better strategy to simply not do this than to support it.
If such non-reproducible information is needed anyway, it can be archived independently of Bitbake.
Adrian
Quoted 142 lines
Cheers,
Quoted 140 lines
So far we haven't had any problems with rebuilding kernels. The trouble
started when we started using fitImages with initramfs for some devices. It is also worth noting that we build from sstate, but with an
empty TMPDIR.In such a scenario bitbake does:
* The initramfs gets assembled. This is expected because images are
not sstate cached. This runs quickly because all the packages which
are required to build the initramfs come from sstate-cache.
* Since the initramfs changes, the fitImage which includes it must be
re-assembled as well.
* Now the hassle starts: Thedo_assemble_fitimage_initramfsneeds the
kernel binaries (linux.bin, DTBs...) which are not available from
sttate. Bitbake has to start with kerneldo_fetchin case of an empty TMPDIR just to get the fitImage assembled.With my patches the kernel binaries used for the fitImage come from sstate (if the initramfs is not bundled).
I tested this like that:
cat
build/conf/local.conf
KERNEL_IMAGETYPE= "Image"KERNEL_IMAGETYPES+= " fitImage "KERNEL_CLASSES= " kernel-fitimage "and RAM diskIMAGE_FSTYPES+= "cpio.gz"INITRAMFS_IMAGE= "core-image-minimal"
IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""UBOOT_RD_LOADADDRESS= "0x88000000"UBOOT_RD_ENTRYPOINT= "0x88000000"UBOOT_LOADADDRESS= "0x80080000"UBOOT_ENTRYPOINT= "0x80080000"FIT_DESC= "A model description"# Allow to change the initramfs
FOO_VAR= "1"
INHERIT += "image-buildinfo"
IMAGE_BUILDINFO_VARS:append = "FOO_VAR"bitbake virtual/kernel
mv build/tmp build/tmp-old-1
bitbake virtual/kernel
-> bitbake does:
- setscene tasks
- core-image-minimaldo_rootfs...
- kerneldo_deploy(from sstate)
- kerneldo_deploy_fitimage_unbundledbitbake virtual/kernel
-> bitbake does: nothing# Enforce a re-build of the initramfs
mv build/tmp build/tmp-old-2
echo 'FOO_VAR= "1"' >>build/conf/local.confbitbake virtual/kernel
-> bitbake does:
- setscene tasks
- core-image-minimaldo_rootfs...
- kerneldo_deploy(from sstate)
- kerneldo_deploy_fitimage_unbundledbitbake virtual/kernel
-> bitbake does: nothingdiff \
<(ls build/tmp-old-2/deploy/images/qemux86-64 -1) \
<(ls build/tmp/deploy/images/qemux86-64/ -1) 4,10c4,10
< core-image-minimal-qemux86-64-20240718065105.cpio.gz < core-image-minimal-qemux86-64-20240718065105.ext4 < core-image-minimal-qemux86-64-20240718065105.manifest < core-image-minimal-qemux86-64-20240718065105.qemuboot.conf < core-image-minimal-qemux86-64-20240718065105.spdx.tar.zst < core-image-minimal-qemux86-64-20240718065105.tar.bz2 < core-image-minimal-qemux86-64-20240718065105.testdata.json
---Quoted 7 lines
core-image-minimal-qemux86-64-20240718073341.cpio.gz core-image-minimal-qemux86-64-20240718073341.ext4 core-image-minimal-qemux86-64-20240718073341.manifest core-image-minimal-qemux86-64-20240718073341.qemuboot.conf core-image-minimal-qemux86-64-20240718073341.spdx.tar.zst core-image-minimal-qemux86-64-20240718073341.tar.bz2 core-image-minimal-qemux86-64-20240718073341.testdata.json
20c20
< fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64- 20240718065105.bin
---Quoted 1 line
fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
20240718073341.bin
22c22
< fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0- qemux86-64-
20240718065105.its
---Quoted 2 lines
fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0- qemux86-64-
20240718073341.its
The answer is then: No it does not re-build. At least not in the scenario which I would like to fix it, it does not rebuild from scratch.
Does that make sense?
Regards,
AdrianQuoted 13 lines
Quoted 13 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#201933):
https://lists.openembedded.org/g/openembedded-core/message/201933
Mute This Topic:
https://lists.openembedded.org/mt/107231736/3617156
Group Owner: [email protected] Unsubscribe:
https://lists.openembedded.org/g/openembedded-core/unsub [
[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 51 lines · Adrian Freihofer
Quoted 26 lines · Martin Jansa
Quoted 5 lines · Adrian Freihofer
Hi Martin
Thank you for looking at my patches.
Quoted 7 lines
Quoted 2 lines · Martin Jansa
Doesn't it still rebuild from scratch when
IMAGE_VERSION_SUFFIXchanges?I wonder why this happens in WebOS. I think that this line kernel_do_deploy[vardepsexclude] = "DATETIME" prevents such unnecessary rebuilds.
Hi Adrian,
yes DATETIME is excluded by default, but it's useful to use different suffix in
IMAGE_VERSION_SUFFIX(e.g.BUILD_NUMBERfrom CI on jenkins or release version when building release) and in such cases you don't want to vardepexclude it, because it's useful to produce matching version in images as well as kernel (and other artifacts you might have), so your 1.2.1 release images doesn't end with 1.2.0 kernel artifacts just because kernel sstate signature didn't change in 1.2.1 bugfix release.Quoted 1 line
Does this make sense? I bet it would happen in your builds as well.
I try to optimize everything towards binary-reproducible builds. This means that we remove all kinds of variables that change because the time changes or because the CI build ID changes or something like that. I see no benefit in compiling such information into the firmware. But the problem I see is that such ideas clash with efficient and binary-reproducible builds.
That means that I don't see a reason for changing the
IMAGE_VERSION_SUFFIXvariable in such a way.Quoted 10 lines
Whole point of:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12937
is to allow "renaming" (adding more hardlinks) artifacts with matchingIMAGE_VERSION_SUFFIXwithout the need of rebuilding anything (e.g. version-less kernel artifacts are re-used from sstate and then only quick deploy-links task is executed to add right versioned hardlinks in deploy directory.)
Quoted 5 lines
Well, yes, it would. But, I am confident that it is a better strategy to simply not do this than to support it.
If such non-reproducible information is needed anyway, it can be archived independently of Bitbake.
Well the changes from:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12937
make the builds _more_ reproducible, because they move the DATETIME or whatever IMAGE_VERSION_SUFFIX out of the artifacts in sstate and move creating versioned names to separate task which can be disabled (by setting IMAGE_VERSION_SUFFIX to empty if you prefer not to create them).
Adrian Freihofer via lists.openembedded.org <adrian.freihofer= [email protected]> escreveu (quinta, 18/07/2024 à(s) 09:00):
Quoted 17 lines
Hi Martin
Thank you for looking at my patches.
Quoted 2 lines · Martin Jansa
Doesn't it still rebuild from scratch when
IMAGE_VERSION_SUFFIXchanges?I wonder why this happens in WebOS. I think that this line kernel_do_deploy[vardepsexclude] = "DATETIME" prevents such unnecessary rebuilds.
So far we haven't had any problems with rebuilding kernels. The trouble started when we started using fitImages with initramfs for some devices. It is also worth noting that we build from sstate, but with an empty TMPDIR.
Quoted 8 lines
In such a scenario bitbake does:
* The initramfs gets assembled. This is expected because images are not sstate cached. This runs quickly because all the packages which are required to build the initramfs come from sstate-cache.
* Since the initramfs changes, the fitImage which includes it must be re-assembled as well.
I have faced this problem and sent a patch [1] to support storing the full images on the sstate.
As it was not a tested and widely used case, it was rejected. We continue using this in our distro but without the patch we need to change more parts on our side.
[1] https://lists.openembedded.org/g/openembedded-core/message/184243
Quoted 8 lines
* Now the hassle starts: The
do_assemble_fitimage_initramfsneeds the kernel binaries (linux.bin, DTBs...) which are not available from sttate. Bitbake has to start with kerneldo_fetchin case of an empty TMPDIR just to get the fitImage assembled.With my patches the kernel binaries used for the fitImage come from sstate (if the initramfs is not bundled).
Anyway your overwall patch set is still a good improvement to not rebuilding the kernel.
That was my concern also and the main reason for my patch.
Jose
Quoted 114 lines
I tested this like that:
cat
build/conf/local.conf
KERNEL_IMAGETYPE= "Image"KERNEL_IMAGETYPES+= " fitImage "KERNEL_CLASSES= " kernel-fitimage "and RAM diskIMAGE_FSTYPES+= "cpio.gz"INITRAMFS_IMAGE= "core-image-minimal"
IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""UBOOT_RD_LOADADDRESS= "0x88000000"UBOOT_RD_ENTRYPOINT= "0x88000000"UBOOT_LOADADDRESS= "0x80080000"UBOOT_ENTRYPOINT= "0x80080000"FIT_DESC= "A model description"# Allow to change the initramfs
FOO_VAR= "1"
INHERIT += "image-buildinfo"
IMAGE_BUILDINFO_VARS:append = "FOO_VAR"bitbake virtual/kernel
mv build/tmp build/tmp-old-1
bitbake virtual/kernel
-> bitbake does:
- setscene tasks
- core-image-minimaldo_rootfs...
- kerneldo_deploy(from sstate)
- kerneldo_deploy_fitimage_unbundledbitbake virtual/kernel
-> bitbake does: nothing# Enforce a re-build of the initramfs
mv build/tmp build/tmp-old-2
echo 'FOO_VAR= "1"' >>build/conf/local.confbitbake virtual/kernel
-> bitbake does:
- setscene tasks
- core-image-minimaldo_rootfs...
- kerneldo_deploy(from sstate)
- kerneldo_deploy_fitimage_unbundledbitbake virtual/kernel
-> bitbake does: nothingdiff \
<(ls build/tmp-old-2/deploy/images/qemux86-64 -1) \
<(ls build/tmp/deploy/images/qemux86-64/ -1) 4,10c4,10
< core-image-minimal-qemux86-64-20240718065105.cpio.gz < core-image-minimal-qemux86-64-20240718065105.ext4 < core-image-minimal-qemux86-64-20240718065105.manifest < core-image-minimal-qemux86-64-20240718065105.qemuboot.conf < core-image-minimal-qemux86-64-20240718065105.spdx.tar.zst < core-image-minimal-qemux86-64-20240718065105.tar.bz2 < core-image-minimal-qemux86-64-20240718065105.testdata.json
---Quoted 7 lines
core-image-minimal-qemux86-64-20240718073341.cpio.gz core-image-minimal-qemux86-64-20240718073341.ext4 core-image-minimal-qemux86-64-20240718073341.manifest core-image-minimal-qemux86-64-20240718073341.qemuboot.conf core-image-minimal-qemux86-64-20240718073341.spdx.tar.zst core-image-minimal-qemux86-64-20240718073341.tar.bz2 core-image-minimal-qemux86-64-20240718073341.testdata.json
20c20
< fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64- 20240718065105.bin
---Quoted 1 line
fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
20240718073341.bin
22c22
< fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64- 20240718065105.its
---Quoted 1 line
fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
20240718073341.its
The answer is then: No it does not re-build. At least not in the scenario which I would like to fix it, it does not rebuild from scratch.
Does that make sense?
Regards,
AdrianQuoted 3 lines
Quoted 3 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#202191):
https://lists.openembedded.org/g/openembedded-core/message/202191
Mute This Topic: https://lists.openembedded.org/mt/107231736/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Best regards, José Quaresma
Quoted 37 lines · Jose Quaresma
Adrian Freihofer via lists.openembedded.org <[email protected]> escreveu (quinta, 18/07/2024 à(s) 09:00):
Quoted 21 lines
Hi Martin
Thank you for looking at my patches.
Quoted 2 lines · Martin Jansa
Doesn't it still rebuild from scratch when
IMAGE_VERSION_SUFFIXchanges?I wonder why this happens in WebOS. I think that this line kernel_do_deploy[vardepsexclude] = "DATETIME" prevents such unnecessary rebuilds.
So far we haven't had any problems with rebuilding kernels. The trouble
started when we started using fitImages with initramfs for some devices. It is also worth noting that we build from sstate, but with an
empty TMPDIR.In such a scenario bitbake does:
Quoted 9 lines
* The initramfs gets assembled. This is expected because images are
not sstate cached. This runs quickly because all the packages which
are required to build the initramfs come from sstate-cache.
* Since the initramfs changes, the fitImage which includes it must be
re-assembled as well.
Quoted 5 lines
I have faced this problem and sent a patch [1] to support storing the full images on the sstate.
As it was not a tested and widely used case, it was rejected. We continue using this in our distro but without the patch we need to change more parts on our side.
I don't think adding images to the sstate cache can help much. But it comes at a high cost. It definitely wouldn't help in the case that should be optimized with my patches here:
* If the initramfs changes it needs to be re-built and cannot be taken from the sstate-cache anyway.
* The issue is: If the initramfs changes, the kernel gets re-built. What we need is an sstate for the kernel artifacts but not for the initramfs artifacts.
Adrian
Quoted 143 lines
[1] https://lists.openembedded.org/g/openembedded-core/message/184243
Quoted 10 lines
* Now the hassle starts: The
do_assemble_fitimage_initramfsneeds the
kernel binaries (linux.bin, DTBs...) which are not available from
sttate. Bitbake has to start with kerneldo_fetchin case of an empty TMPDIR just to get the fitImage assembled.With my patches the kernel binaries used for the fitImage come from sstate (if the initramfs is not bundled).
Anyway your overwall patch set is still a good improvement to not rebuilding the kernel.
That was my concern also and the main reason for my patch.
JoseQuoted 116 lines
I tested this like that:
cat
build/conf/local.conf
KERNEL_IMAGETYPE= "Image"KERNEL_IMAGETYPES+= " fitImage "KERNEL_CLASSES= " kernel-fitimage "and RAM diskIMAGE_FSTYPES+= "cpio.gz"INITRAMFS_IMAGE= "core-image-minimal"
IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""UBOOT_RD_LOADADDRESS= "0x88000000"UBOOT_RD_ENTRYPOINT= "0x88000000"UBOOT_LOADADDRESS= "0x80080000"UBOOT_ENTRYPOINT= "0x80080000"FIT_DESC= "A model description"# Allow to change the initramfs
FOO_VAR= "1"
INHERIT += "image-buildinfo"
IMAGE_BUILDINFO_VARS:append = "FOO_VAR"bitbake virtual/kernel
mv build/tmp build/tmp-old-1
bitbake virtual/kernel
-> bitbake does:
- setscene tasks
- core-image-minimaldo_rootfs...
- kerneldo_deploy(from sstate)
- kerneldo_deploy_fitimage_unbundledbitbake virtual/kernel
-> bitbake does: nothing# Enforce a re-build of the initramfs
mv build/tmp build/tmp-old-2
echo 'FOO_VAR= "1"' >>build/conf/local.confbitbake virtual/kernel
-> bitbake does:
- setscene tasks
- core-image-minimaldo_rootfs...
- kerneldo_deploy(from sstate)
- kerneldo_deploy_fitimage_unbundledbitbake virtual/kernel
-> bitbake does: nothingdiff \
<(ls build/tmp-old-2/deploy/images/qemux86-64 -1) \
<(ls build/tmp/deploy/images/qemux86-64/ -1) 4,10c4,10
< core-image-minimal-qemux86-64-20240718065105.cpio.gz < core-image-minimal-qemux86-64-20240718065105.ext4 < core-image-minimal-qemux86-64-20240718065105.manifest < core-image-minimal-qemux86-64-20240718065105.qemuboot.conf < core-image-minimal-qemux86-64-20240718065105.spdx.tar.zst < core-image-minimal-qemux86-64-20240718065105.tar.bz2 < core-image-minimal-qemux86-64-20240718065105.testdata.json
---Quoted 7 lines
core-image-minimal-qemux86-64-20240718073341.cpio.gz core-image-minimal-qemux86-64-20240718073341.ext4 core-image-minimal-qemux86-64-20240718073341.manifest core-image-minimal-qemux86-64-20240718073341.qemuboot.conf core-image-minimal-qemux86-64-20240718073341.spdx.tar.zst core-image-minimal-qemux86-64-20240718073341.tar.bz2 core-image-minimal-qemux86-64-20240718073341.testdata.json
20c20
< fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64- 20240718065105.bin
---Quoted 1 line
fitImage-core-image-minimal-qemux86-64--6.6.35+git-r0-qemux86-64-
20240718073341.bin
22c22
< fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0- qemux86-64-
20240718065105.its
---Quoted 2 lines
fitImage-its-core-image-minimal-qemux86-64--6.6.35+git-r0- qemux86-64-
20240718073341.its
The answer is then: No it does not re-build. At least not in the scenario which I would like to fix it, it does not rebuild from scratch.
Does that make sense?
Regards,
AdrianQuoted 3 lines
Quoted 3 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#202191):
https://lists.openembedded.org/g/openembedded-core/message/202191
Mute This Topic:
https://lists.openembedded.org/mt/107231736/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-
core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-Signature · 3 lines
Best regards, José Quaresma
Hello,
I'm pretty sure this causes this failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/151/builds/2005/steps/12/logs/stdio
And also those oe-selftest failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6945/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6987/steps/15/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/7000/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/7010/steps/14/logs/stdio
Quoted 187 lines · Adrian Freihofer
Changes in comparison to v1:
- Add the missing dependency fromdo_image_wicto initramfs.do_image_complete which caused the build failure on the AB.
- Support symbolic links for $dtb_path again, as suggested by Mark.
- Simplify the refactoring of theuboot_prep_kimagefunction.
- Evaluate staging via sysroot instead of taking the kernel artifacts from the deploy folder. But there is an issue with this approach: It does not allow to remove the dependency fromdo_deployondo_install.
- Rebasing to latest masterIf the
KERNEL_IMAGETYPES(S)contains fitImage, the kernel is always rebuilt when something changes in the initramfs frequently. This is even worse if the build runs from an empty TMPDIR. The kernel re-build starts by fetching the large kernel git repository and recompiling it from scratch.This cannot be improved if
INITRAMFS_IMAGE_BUNDLE= "1". If the kernelMakefileis needed to generate the initramfs bundle the kernel build folder is required.
But for a build configuration withINITRAMFS_IMAGE_BUNDLE= "" the build folder is not needed. Creating the initramfs bundle requires:
linux.bin, DTBs and the initramfs which are available in the deploy directory. That means creating the fitImage can be done with artifacts which are already cached by the sstate.There is an extra commit providing a html file. This html file provides some graphics which show the changes in the kernel's task dependencies. I hope this simplifies the re-view of this patch series.
Testing:
- oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg \ reproducible -T yocto-mirrors -T machine -T toolchain-user \ -T toolchain-system- Build for core-image-minimal for MACHINE = "genericarm64" (failure from AB was reproducible and is fixed now)
- Build clean followed by build from sstate
Note: Adding this test to oe-selftest failed because sstate is read only
- Clean build config:KERNEL_IMAGETYPE = "Image" KERNEL_IMAGETYPES += " fitImage " KERNEL_CLASSES = " kernel-fitimage " IMAGE_FSTYPES += "cpio.gz" INITRAMFS_IMAGE = "core-image-minimal" IMAGE_NAME_SUFFIX:pn-core-image-minimal = "" UBOOT_RD_LOADADDRESS = "0x88000000" UBOOT_RD_ENTRYPOINT = "0x88000000" UBOOT_LOADADDRESS = "0x80080000" UBOOT_ENTRYPOINT = "0x80080000" FIT_DESC = "A model description" FOO_VAR = "1" INHERIT += "image-buildinfo" IMAGE_BUILDINFO_VARS:append = " FOO_VAR" - Append the following and rebuild with sstate: FOO_VAR = "2" TMPDIR = "${TOPDIR}/tmp-2"- Check the log files:
- tmp/log/cooker/qemux86-64 contains:
linux-yocto-6.6.35+git-r0: taskdo_deploy: Succeeded linux-yocto-6.6.35+git-r0: taskdo_deploy_fitimage_unbundled: Succeeded
- tmp-2/log/cooker/qemux86-64 contains:
linux-yocto-6.6.35+git-r0: taskdo_deploy_setscene: Succeeded linux-yocto-6.6.35+git-r0: taskdo_deploy_fitimage_unbundled: Succeeded
- To re-view the chagnes in the task dependencies, the script bellow has been used. It confirms:
- For builds with fitImage and unbundled initramfs:
- thedo_deploy_fitimage_unbundledtask runs afterdo_deploy
-do_assemble_fitimage_initramfsis not executed
-do_bundle_initramfsis not executed
- It works for fitImage inKERNEL_IMAGETYPEas well as for fitImage inKERNEL_IMAGETYPES
- For builds with fitImage and bundled initramfs: No changes
- For builds with fitImage, without initramfs:
-do_assemble_fitimage_initramfsis not executed
-do_bundle_initramfsis not executedOUTPUT_FILE=task-depends.md
GIT_BRANCH=master
GIT_BRANCH_NEXT=adrianf/kernel-fitimage-sstateecho "# Task dependeny changes" > "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"run_bitbake(){ echo "$1" >> "$OUTPUT_FILE" echo "" >> "$OUTPUT_FILE" bitbake virtual/kernel -g grep -E '(digraph depends|linux-yocto|\})' task-depends.dot \ | grep -v -E '(spdx|do_kernel_configcheck|do_prepare_recipe_sysroot|do_populate_lic|native.do_populate_sysroot)' \ | grep -v -E '(depmodwrapper-cross.do_populate_sysroot|binutils-cross-x86_64.do_populate_sysroot|do_package)' \ | grep -v -E '(do_validate_branches|do_unpack|do_recipe_qa|do_patch|do_kernel_checkout|do_kernel_configme|do_kernel_metadata)' \ | grep -v -E '(do_sizecheck|do_strip|do_compile_kernelmodules|do_shared_workdir|do_transform_kernel|do_kernel_link_images)' \ | grep -v -E '(do_kernel_version_sanity_check|do_symlink_kernsrc|do_deploy_source_date_epoch|do_configure|do_fetch)' \ | sed -e 's;\\n\:.*.bb;;g' -e 's;linux-yocto[. ];;g' > task-depends-filtered.dot echo '```plantuml' >> "$OUTPUT_FILE" cat task-depends-filtered.dot >> "$OUTPUT_FILE" echo '```' >> "$OUTPUT_FILE" echo "" >> "$OUTPUT_FILE" }run_bitbake_per_branch(){ echo "## Configuration: $1" >> "$OUTPUT_FILE" echo "" >> "$OUTPUT_FILE" echo '```raw' >> "$OUTPUT_FILE" cat build/conf/auto.conf >> "$OUTPUT_FILE" echo '```' >> "$OUTPUT_FILE" echo "" >> "$OUTPUT_FILE" git checkout $GIT_BRANCH run_bitbake "### branch: $GIT_BRANCH" mv -f task-depends-filtered.dot task-depends-filtered-old.dot git checkout $GIT_BRANCH_NEXT run_bitbake "### branch: $GIT_BRANCH_NEXT" echo "## Diff" >> "$OUTPUT_FILE" echo "" >> "$OUTPUT_FILE" echo '```patch' >> "$OUTPUT_FILE" diff task-depends-filtered-old.dot task-depends-filtered.dot >> "$OUTPUT_FILE" echo '```' >> "$OUTPUT_FILE" echo "" >> "$OUTPUT_FILE" }cat << EOF >
build/conf/auto.confKERNEL_IMAGETYPE= "Image"KERNEL_IMAGETYPES+= " fitImage "KERNEL_CLASSES= " kernel-fitimage "IMAGE_FSTYPES+= "cpio.gz"INITRAMFS_IMAGE= "core-image-minimal-initramfs"
EOFrun_bitbake_per_branch"image, fitimage, initramfs, unbundled"cat << EOF >
build/conf/auto.confKERNEL_IMAGETYPE:forcevariable = "fitImage"KERNEL_CLASSES= " kernel-fitimage "IMAGE_FSTYPES+= "cpio.gz"INITRAMFS_IMAGE= "core-image-minimal-initramfs" EOFrun_bitbake_per_branch"fitimage, initramfs, unbundled"cat << EOF >
build/conf/auto.confKERNEL_IMAGETYPE= "Image"KERNEL_IMAGETYPES+= " fitImage "KERNEL_CLASSES= " kernel-fitimage "IMAGE_FSTYPES+= "cpio.gz"INITRAMFS_IMAGE= "core-image-minimal-initramfs"INITRAMFS_IMAGE_BUNDLE= "1"
EOFrun_bitbake_per_branch"image, fitimage, initramfs, bundled"cat << EOF >
build/conf/auto.confKERNEL_IMAGETYPE= "Image"KERNEL_IMAGETYPES+= " fitImage "KERNEL_CLASSES= " kernel-fitimage "
EOFrun_bitbake_per_branch"image, fitimage"rm -f
build/conf/auto.conftask-depends-filtered-old.dot task-depends-filtered.dot task-depends.dotAdrian Freihofer (6):
kernel-fitimage: fix intentation
kernel-fitimage: fix external dtb check
kernel: refactor linux compression
kernel-fitimage: refactorfitimage_assemble
kernel: refactor fitimage
kernel-fitimage: run unbundled fitimage after deploymeta/classes-recipe/image.bbclass | 12 +- meta/classes-recipe/kernel-fitimage.bbclass | 188 ++++++++++++-------- meta/classes-recipe/kernel-uboot.bbclass | 1 + meta/classes-recipe/kernel.bbclass | 31 ++-- 4 files changed, 148 insertions(+), 84 deletions(-)Signature · 11 lines
2.45.2 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#201926): https://lists.openembedded.org/g/openembedded-core/message/201926 Mute This Topic: https://lists.openembedded.org/mt/107231736/3617179 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com