Skip to contentSkip to conversation

Use the kernel from sstate when building fitImages

PATCH Superseded Adrian Freihofer 3 participants 21 messages Jul 4, 2024 – Jul 10, 2024

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover Use the kernel from sstate when building fitImages

If the KERNEL_IMAGETYPES(S) contains fitImage, the kernel is always rebuilt when something changes in the initramfs, which happens frequently.
This is even worse if the build runs from an empty TMPDIR. The kernel re-build starts by fetching the large kernel git repository and recompiling it from scratch.

This cannot be improved if INITRAMFS_IMAGE_BUNDLE = "1". If the kernel Makefile is needed to generate the initramfs bundle the kernel build folder is required.
But for a build configuration with INITRAMFS_IMAGE_BUNDLE = "" the build folder is not needed. Creating the initramfs bundle requires:
linux.bin, DTBs and the initramfs which are available in the deploy directory. That means creating the fitImage can be done with artifacts which are already cached by the sstate.

There is an extra commit providing a html file. This html file provides some graphics which show the changes in the kernel's task dependencies. I hope this simplifies the re-view of this patch series.

Adrian Freihofer (14):
oe-selftest: wic fix deps of test_efi_plugin_unified_kernel_image_qemu
oe-selftest: fitimage refactor u-boot-tools-native
oe-selftest: fitimage drop test-mkimage-wrapper
oe-selftest: fitimage cleanup asserts
oe-selftest: fitimage fix test_initramfs_bundle
kernel-fitimage: fix handling of empty default dtb
kernel-fitimage: fix external dtb check
kernel: execute do_bundle_initramfs only when needed
kernel-fitimage: refactor do_assemble_fitimage_initramfs
kernel-fitimage: refactor creation of linux.bin
kernel-fitimage: fix intentation
kernel-fitimage: run unbundled fitimage after deploy
image: depends do_deploy_fitimage_unbundled
DO NOT MERGE: explanation and testing

 kernel-tasks.sh                               |   76 ++
 .../classes/test-mkimage-wrapper.bbclass      |   19 -
 meta/classes-recipe/image.bbclass             |   12 +-
 meta/classes-recipe/kernel-fitimage.bbclass   |  179 ++-
 meta/classes-recipe/kernel-uboot.bbclass      |   17 +
 meta/classes-recipe/kernel.bbclass            |   17 +-
 meta/lib/oeqa/selftest/cases/fitimage.py      |  264 ++--
 meta/lib/oeqa/selftest/cases/wic.py           |    6 +-
 task-depends.html                             | 1122 +++++++++++++++++
 9 files changed, 1490 insertions(+), 222 deletions(-)
 create mode 100755 kernel-tasks.sh
 delete mode 100644 meta-selftest/classes/test-mkimage-wrapper.bbclass
 create mode 100644 task-depends.html
Signature · 1 line
2.45.2
#1 lore raw

PATCH 1/14 oe-selftest: wic fix deps of test_efi_plugin_unified_kernel_image_qemu

From: Adrian Freihofer <[email protected]>

oe-selftest wic.Wic2.test_efi_plugin_unified_kernel_image_qemu failed with:
AssertionError: Command
'bitbake core-image-minimal core-image-minimal-initramfs ovmf' returned non-zero exit status 1:

bitbake reported:
FileNotFoundError: [Errno 2] No such file or directory:
'.../build-st-242167/tmp/deploy/images/qemux86-64/core-image-minimal-initramfs-qemux86-64.cpio.gz'
WARNING: .../build-st-242167/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0/temp/run.do_image_wic.986859:162

    exit 1 from 'BUILDDIR="/home/adrian/projets/oss/poky/build-st-242167" PSEUDO_UNLOAD=1
    wic create "$wks" --vars ".../build-st-242167/tmp/sysroots/qemux86-64/imgdata/"
       -e "core-image-minimal" -o "$build_wic/" -w "$tmp_wic"'

Creating a configuration as the test does and calling
bitbake -g core-image-minimal
does not show a dependency from core-image-minimal:do_image_wic on core-image-minimal-initramfs:do_image_complete.
Adding the dependency directly to the test case resolves the issue.

Since the dependency is basically created by the wks file solving this in a more generic way, e.g. by handling the dependency in image_types_wic.bbclass does not look like simple.

---
 meta/lib/oeqa/selftest/cases/wic.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/wic.py +5 -1
index b616759209a..b1315c5b432 100644
@@ -1294,7 +1294,11 @@ class Wic2(WicTestCase):
                  'WKS_FILE = "test_efi_plugin.wks"\n'\
                  'MACHINE_FEATURES:append = " efi"\n'
         self.append_config(config)
-        bitbake('core-image-minimal core-image-minimal-initramfs ovmf')
+        # Rather than creating a special image which adds the dependency which is needed by the wks file
+        # core-image-minimal:do_image_wic[depends] += "core-image-minimal-initramfs:do_image_complete"
+        # core-image-minimal-initramf is built before it is used by core-image-minimal.
+        bitbake('core-image-minimal-initramfs')
+        bitbake('core-image-minimal ovmf')
         self.remove_config(config)
 
         with runqemu('core-image-minimal', ssh=False,
Signature · 1 line
2.45.2
#2 lore raw

PATCH 2/14 oe-selftest: fitimage refactor u-boot-tools-native

In reply to Adrian Freihofer

From: Adrian Freihofer <[email protected]>

Eliminate code duplication with a new function.
Use get_bb_var instead of bitbake -e | grep.

---
 meta/lib/oeqa/selftest/cases/fitimage.py | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
meta/lib/oeqa/selftest/cases/fitimage.py +11 -12
index 347c0653776..15baf3b2392 100644
@@ -11,6 +11,11 @@ import re
 
 class FitImageTests(OESelftestTestCase):
 
+    def _setup_uboot_tools_native(self):
+        """build u-boot-tools-native and return RECIPE_SYSROOT_NATIVE"""
+        bitbake("u-boot-tools-native -c addto_recipe_sysroot")
+        return get_bb_var('RECIPE_SYSROOT_NATIVE', 'u-boot-tools-native')
+
     def test_fit_image(self):
         """
         Summary:     Check if FIT image and Image Tree Source (its) are built
@@ -195,10 +200,8 @@ UBOOT_MKIMAGE_SIGN_ARGS = "-c 'a smart comment'"
                 self.assertEqual(value, reqvalue)
 
         # Dump the image to see if it really got signed
-        bitbake("u-boot-tools-native -c addto_recipe_sysroot")
-        result = runCmd('bitbake -e u-boot-tools-native | grep ^RECIPE_SYSROOT_NATIVE=')
-        recipe_sysroot_native = result.output.split('=')[1].strip('"')
-        dumpimage_path = os.path.join(recipe_sysroot_native, 'usr', 'bin', 'dumpimage')
+        uboot_tools_sysroot_native = self._setup_uboot_tools_native()
+        dumpimage_path = os.path.join(uboot_tools_sysroot_native, 'usr', 'bin', 'dumpimage')
         result = runCmd('%s -l %s' % (dumpimage_path, fitimage_path))
         in_signed = None
         signed_sections = {}
@@ -516,10 +519,8 @@ UBOOT_FIT_HASH_ALG = "sha256"
                 self.assertEqual(value, reqvalue)
 
         # Dump the image to see if it really got signed
-        bitbake("u-boot-tools-native -c addto_recipe_sysroot")
-        result = runCmd('bitbake -e u-boot-tools-native | grep ^RECIPE_SYSROOT_NATIVE=')
-        recipe_sysroot_native = result.output.split('=')[1].strip('"')
-        dumpimage_path = os.path.join(recipe_sysroot_native, 'usr', 'bin', 'dumpimage')
+        uboot_tools_sysroot_native = self._setup_uboot_tools_native()
+        dumpimage_path = os.path.join(uboot_tools_sysroot_native, 'usr', 'bin', 'dumpimage')
         result = runCmd('%s -l %s' % (dumpimage_path, fitimage_path))
         in_signed = None
         signed_sections = {}
@@ -671,10 +672,8 @@ FIT_SIGN_INDIVIDUAL = "1"
                 self.assertEqual(value, reqvalue)
 
         # Dump the image to see if it really got signed
-        bitbake("u-boot-tools-native -c addto_recipe_sysroot")
-        result = runCmd('bitbake -e u-boot-tools-native | grep ^RECIPE_SYSROOT_NATIVE=')
-        recipe_sysroot_native = result.output.split('=')[1].strip('"')
-        dumpimage_path = os.path.join(recipe_sysroot_native, 'usr', 'bin', 'dumpimage')
+        uboot_tools_sysroot_native = self._setup_uboot_tools_native()
+        dumpimage_path = os.path.join(uboot_tools_sysroot_native, 'usr', 'bin', 'dumpimage')
         result = runCmd('%s -l %s' % (dumpimage_path, fitimage_path))
         in_signed = None
         signed_sections = {}
Signature · 1 line
2.45.2
#3 lore raw

PATCH 3/14 oe-selftest: fitimage drop test-mkimage-wrapper

In reply to Adrian Freihofer

From: Adrian Freihofer <[email protected]>

Rather than writing hints into log files and verify the hints can be found, the tests should verify that the artifacts in the deploy folder are correctly signed. This is a much better test. u-boot-tools provide a utility fit_check_sign which can verify the signatures in fit images. Lets use it.

grepping in temp/run. or temp/log. files also does not work if the tasks runs from sstate and the corresponding run file is not even generated.

---
 .../classes/test-mkimage-wrapper.bbclass      |  19 ---
 meta/lib/oeqa/selftest/cases/fitimage.py      | 118 ++++++++++++------
 2 files changed, 77 insertions(+), 60 deletions(-)
 delete mode 100644 meta-selftest/classes/test-mkimage-wrapper.bbclass
meta-selftest/classes/test-mkimage-wrapper.bbclass deleted +0 -19
deleted file mode 100644
index 7c98d7b71e4..00000000000
@@ -1,19 +0,0 @@
-# Class to test UBOOT_MKIMAGE and UBOOT_MKIMAGE_SIGN
-# (in conjunction with kernel-fitimage.bbclass)
-#
-# SPDX-License-Identifier: MIT
-#
-
-UBOOT_MKIMAGE = "test_mkimage_wrapper"
-UBOOT_MKIMAGE_SIGN = "test_mkimage_signing_wrapper"
-
-test_mkimage_wrapper() {
-    echo "### uboot-mkimage wrapper message"
-    uboot-mkimage "$@"
-}
-
-test_mkimage_signing_wrapper() {
-    echo "### uboot-mkimage signing wrapper message"
-    uboot-mkimage "$@"
-}
-
meta/lib/oeqa/selftest/cases/fitimage.py +77 -41

225 lines not on this page · raw message

Signature · 1 line
2.45.2
#4 lore raw

PATCH 4/14 oe-selftest: fitimage cleanup asserts

In reply to Adrian Freihofer

From: Adrian Freihofer <[email protected]>

Use more specific assert statements to make the code more readable and the error messages more verbose.

Cleanup the asserts of the test case test_initramfs_bundle.

---
 meta/lib/oeqa/selftest/cases/fitimage.py | 76 +++++++-----------------
 1 file changed, 21 insertions(+), 55 deletions(-)
meta/lib/oeqa/selftest/cases/fitimage.py +21 -55
index 4891ac8010b..ee688c79099 100644
@@ -98,10 +98,8 @@ FIT_DESC = "A model description"
         fitimage_path = os.path.join(bb_vars['DEPLOY_DIR_IMAGE'],
             "fitImage-%s-%s" % (bb_vars['INITRAMFS_IMAGE_NAME'], bb_vars['KERNEL_FIT_LINK_NAME']))
 
-        self.assertTrue(os.path.exists(fitimage_its_path),
-            "%s image tree source doesn't exist" % (fitimage_its_path))
-        self.assertTrue(os.path.exists(fitimage_path),
-            "%s FIT image doesn't exist" % (fitimage_path))
+        self.assertExists(fitimage_its_path, "%s image tree source doesn't exist" % (fitimage_its_path))
+        self.assertExists(fitimage_path, "%s FIT image doesn't exist" % (fitimage_path))
 
         # Check that the type, load address, entrypoint address and default
         # values for kernel and ramdisk in Image Tree Source are as expected.
@@ -180,10 +178,8 @@ UBOOT_MKIMAGE_SIGN_ARGS = "-c '%s'"
         fitimage_path = os.path.join(bb_vars['DEPLOY_DIR_IMAGE'],
             "fitImage-%s.bin" % (bb_vars['KERNEL_FIT_LINK_NAME']))
 
-        self.assertTrue(os.path.exists(fitimage_its_path),
-            "%s image tree source doesn't exist" % (fitimage_its_path))
-        self.assertTrue(os.path.exists(fitimage_path),
-            "%s FIT image doesn't exist" % (fitimage_path))
+        self.assertExists(fitimage_its_path, "%s image tree source doesn't exist" % (fitimage_its_path))
+        self.assertExists(fitimage_path, "%s FIT image doesn't exist" % (fitimage_path))
 
         req_itspaths = [
             ['/', 'images', 'kernel-1'],
@@ -330,10 +326,8 @@ FIT_SIGN_INDIVIDUAL = "1"
         fitimage_path = os.path.join(deploy_dir_image,
             "u-boot-fitImage-%s" % (machine,))
 
-        self.assertTrue(os.path.exists(fitimage_its_path),
-            "%s image tree source doesn't exist" % (fitimage_its_path))
-        self.assertTrue(os.path.exists(fitimage_path),
-            "%s FIT image doesn't exist" % (fitimage_path))
+        self.assertExists(fitimage_its_path, "%s image tree source doesn't exist" % (fitimage_its_path))
+        self.assertExists(fitimage_path, "%s FIT image doesn't exist" % (fitimage_path))
 
         # Check that the type, load address, entrypoint address and default
         # values for kernel and ramdisk in Image Tree Source are as expected.
@@ -414,10 +408,8 @@ UBOOT_MKIMAGE_SIGN_ARGS = "-c 'a smart U-Boot comment'"
         fitimage_path = os.path.join(deploy_dir_image,
             "u-boot-fitImage-%s" % (machine,))
 
-        self.assertTrue(os.path.exists(fitimage_its_path),
-            "%s image tree source doesn't exist" % (fitimage_its_path))
-        self.assertTrue(os.path.exists(fitimage_path),
-            "%s FIT image doesn't exist" % (fitimage_path))
+        self.assertExists(fitimage_its_path, "%s image tree source doesn't exist" % (fitimage_its_path))
+        self.assertExists(fitimage_path, "%s FIT image doesn't exist" % (fitimage_path))
 
         # Check that the type, load address, entrypoint address and default
         # values for kernel and ramdisk in Image Tree Source are as expected.
@@ -506,10 +498,8 @@ UBOOT_FIT_HASH_ALG = "sha256"
         fitimage_path = os.path.join(deploy_dir_image,
             "u-boot-fitImage-%s" % (machine,))
 
-        self.assertTrue(os.path.exists(fitimage_its_path),
-            "%s image tree source doesn't exist" % (fitimage_its_path))
-        self.assertTrue(os.path.exists(fitimage_path),
-            "%s FIT image doesn't exist" % (fitimage_path))
+        self.assertExists(fitimage_its_path, "%s image tree source doesn't exist" % (fitimage_its_path))
+        self.assertExists(fitimage_path, "%s FIT image doesn't exist" % (fitimage_path))
 
         req_itspaths = [
             ['/', 'images', 'uboot'],
@@ -656,10 +646,8 @@ FIT_SIGN_INDIVIDUAL = "1"
         fitimage_path = os.path.join(deploy_dir_image,
             "u-boot-fitImage-%s" % (machine,))
 
-        self.assertTrue(os.path.exists(fitimage_its_path),
-            "%s image tree source doesn't exist" % (fitimage_its_path))
-        self.assertTrue(os.path.exists(fitimage_path),
-            "%s FIT image doesn't exist" % (fitimage_path))
+        self.assertExists(fitimage_its_path, "%s image tree source doesn't exist" % (fitimage_its_path))
+        self.assertExists(fitimage_path, "%s FIT image doesn't exist" % (fitimage_path))
 
         req_itspaths = [
             ['/', 'images', 'uboot'],
@@ -793,10 +781,8 @@ FIT_HASH_ALG = "sha256"
                     "fitImage-its-%s-%s-%s" % (image_type, machine, machine))
         fitimage_path = os.path.join(deploy_dir_image,"fitImage")
 
-        self.assertTrue(os.path.exists(fitimage_its_path),
-            "%s image tree source doesn't exist" % (fitimage_its_path))
-        self.assertTrue(os.path.exists(fitimage_path),
-            "%s FIT image doesn't exist" % (fitimage_path))
+        self.assertExists(fitimage_its_path, "%s image tree source doesn't exist" % (fitimage_its_path))
+        self.assertExists(fitimage_path, "%s FIT image doesn't exist" % (fitimage_path))
 
         kernel_load = str(get_bb_var('UBOOT_LOADADDRESS'))
         kernel_entry = str(get_bb_var('UBOOT_ENTRYPOINT'))
@@ -827,17 +813,12 @@ FIT_HASH_ALG = "sha256"
 
         node_str = exp_node_lines[0]
 
-        test_passed = False
-
         print ("checking kernel node\n")
+        self.assertIn(node_str, its_lines)
 
-        if node_str in its_lines:
-            node_start_idx = its_lines.index(node_str)
-            node = its_lines[node_start_idx:(node_start_idx + len(exp_node_lines))]
-            if node == exp_node_lines:
-                print("kernel node verified")
-            else:
-                self.assertTrue(test_passed == True,"kernel node does not match expectation")
+        node_start_idx = its_lines.index(node_str)
+        node = its_lines[node_start_idx:(node_start_idx + len(exp_node_lines))]
+        self.assertEqual(node, exp_node_lines, "kernel node does not match expectation")
 
         rx_configs = re.compile("^conf-.*")
         its_configs = list(filter(rx_configs.match, its_lines))
@@ -854,27 +835,12 @@ FIT_HASH_ALG = "sha256"
             node = its_lines[cfg_start_idx:line_idx]
             print("checking configuration " + cfg_str.rstrip(" {"))
             rx_desc_line = re.compile("^description.*1 Linux kernel.*")
-            if len(list(filter(rx_desc_line.match, node))) != 1:
-                self.assertTrue(test_passed == True,"kernel keyword not found in the description line")
-                break
-            else:
-                print("kernel keyword found in the description line")
+            self.assertNotEqual(len(list(filter(rx_desc_line.match, node))), 1, "kernel keyword not found in the description line")
 
-            if 'kernel = "kernel-1";' not in node:
-                self.assertTrue(test_passed == True,"kernel line not found")
-                break
-            else:
-                print("kernel line found")
+            self.assertNotIn('kernel = "kernel-1";', node)
 
             rx_sign_line = re.compile("^sign-images.*kernel.*")
-            if len(list(filter(rx_sign_line.match, node))) != 1:
-                self.assertTrue(test_passed == True,"kernel hash not signed")
-                break
-            else:
-                print("kernel hash signed")
-
-            test_passed = True
-            self.assertTrue(test_passed == True,"Initramfs bundle test success")
+            self.assertNotEqual(len(list(filter(rx_sign_line.match, node))), 1, "kernel hash not signed")
 
         # Verify the signature
         uboot_tools_sysroot_native = self._setup_uboot_tools_native()
Signature · 1 line
2.45.2
#5 lore raw

PATCH 5/14 oe-selftest: fitimage fix test_initramfs_bundle

In reply to Adrian Freihofer

From: Adrian Freihofer <[email protected]>

It looks like most of the assertions of this tests were by-passed because of combinations of invalid regexes and inverted assert statement. Try to fix.

Use the variables from the kernel recipe instead of potentially invalid values from get_bb_vars with undefined recipe. Use one get_bb_vars statement instead of many seaparate calls mainly to improve the duration of this test case.

Make the test working for an its file with an absolute path to the kernel binary. This will be needed with one of the following commits and does not harm in general.

---
 meta/lib/oeqa/selftest/cases/fitimage.py | 57 +++++++++++++-----------
 1 file changed, 32 insertions(+), 25 deletions(-)
meta/lib/oeqa/selftest/cases/fitimage.py +32 -25
index ee688c79099..0b5f4602fbc 100644
@@ -774,23 +774,25 @@ FIT_HASH_ALG = "sha256"
         # fitImage is created as part of linux recipe
         bitbake("virtual/kernel")
 
-        image_type = get_bb_var('INITRAMFS_IMAGE')
-        deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE')
-        machine = get_bb_var('MACHINE')
-        fitimage_its_path = os.path.join(deploy_dir_image,
-                    "fitImage-its-%s-%s-%s" % (image_type, machine, machine))
-        fitimage_path = os.path.join(deploy_dir_image,"fitImage")
+        bb_vars = get_bb_vars([
+            'DEPLOY_DIR_IMAGE',
+            'FIT_HASH_ALG',
+            'FIT_KERNEL_COMP_ALG',
+            'INITRAMFS_IMAGE',
+            'MACHINE',
+            'UBOOT_ARCH',
+            'UBOOT_ENTRYPOINT',
+            'UBOOT_LOADADDRESS',
+            'UBOOT_MKIMAGE_KERNEL_TYPE'
+            ],
+            'virtual/kernel')
+        fitimage_its_path = os.path.join(bb_vars['DEPLOY_DIR_IMAGE'],
+                    "fitImage-its-%s-%s-%s" % (bb_vars['INITRAMFS_IMAGE'], bb_vars['MACHINE'], bb_vars['MACHINE']))
+        fitimage_path = os.path.join(bb_vars['DEPLOY_DIR_IMAGE'],"fitImage")
 
         self.assertExists(fitimage_its_path, "%s image tree source doesn't exist" % (fitimage_its_path))
         self.assertExists(fitimage_path, "%s FIT image doesn't exist" % (fitimage_path))
 
-        kernel_load = str(get_bb_var('UBOOT_LOADADDRESS'))
-        kernel_entry = str(get_bb_var('UBOOT_ENTRYPOINT'))
-        kernel_type = str(get_bb_var('UBOOT_MKIMAGE_KERNEL_TYPE'))
-        kernel_compression = str(get_bb_var('FIT_KERNEL_COMP_ALG'))
-        uboot_arch = str(get_bb_var('UBOOT_ARCH'))
-        fit_hash_alg = str(get_bb_var('FIT_HASH_ALG'))
-
         its_file = open(fitimage_its_path)
 
         its_lines = [line.strip() for line in its_file.readlines()]
@@ -799,14 +801,14 @@ FIT_HASH_ALG = "sha256"
             'kernel-1 {',
             'description = "Linux kernel";',
             'data = /incbin/("linux.bin");',
-            'type = "' + kernel_type + '";',
-            'arch = "' + uboot_arch + '";',
+            'type = "' + str(bb_vars['UBOOT_MKIMAGE_KERNEL_TYPE']) + '";',
+            'arch = "' + str(bb_vars['UBOOT_ARCH']) + '";',
             'os = "linux";',
-            'compression = "' + kernel_compression + '";',
-            'load = <' + kernel_load + '>;',
-            'entry = <' + kernel_entry + '>;',
+            'compression = "' + str(bb_vars['FIT_KERNEL_COMP_ALG']) + '";',
+            'load = <' + str(bb_vars['UBOOT_LOADADDRESS']) + '>;',
+            'entry = <' + str(bb_vars['UBOOT_ENTRYPOINT']) + '>;',
             'hash-1 {',
-            'algo = "' + fit_hash_alg +'";',
+            'algo = "' + str(bb_vars['FIT_HASH_ALG']) +'";',
             '};',
             '};'
         ]
@@ -818,6 +820,11 @@ FIT_HASH_ALG = "sha256"
 
         node_start_idx = its_lines.index(node_str)
         node = its_lines[node_start_idx:(node_start_idx + len(exp_node_lines))]
+
+        # Remove the absolute path. This refers to WORKDIR which is not always predictable.
+        re_data = re.compile(r'^data = /incbin/\(.*/linux\.bin"\);$')
+        node = [re.sub(re_data, 'data = /incbin/("linux.bin");', cfg_str) for cfg_str in node]
+
         self.assertEqual(node, exp_node_lines, "kernel node does not match expectation")
 
         rx_configs = re.compile("^conf-.*")
@@ -834,14 +841,14 @@ FIT_HASH_ALG = "sha256"
 
             node = its_lines[cfg_start_idx:line_idx]
             print("checking configuration " + cfg_str.rstrip(" {"))
-            rx_desc_line = re.compile("^description.*1 Linux kernel.*")
-            self.assertNotEqual(len(list(filter(rx_desc_line.match, node))), 1, "kernel keyword not found in the description line")
+            rx_desc_line = re.compile(r'^description = ".*Linux kernel.*')
+            self.assertEqual(len(list(filter(rx_desc_line.match, node))), 1, "kernel keyword not found in the description line")
 
-            self.assertNotIn('kernel = "kernel-1";', node)
+            self.assertIn('kernel = "kernel-1";', node)
 
-            rx_sign_line = re.compile("^sign-images.*kernel.*")
-            self.assertNotEqual(len(list(filter(rx_sign_line.match, node))), 1, "kernel hash not signed")
+            rx_sign_line = re.compile(r'^sign-images = .*kernel.*')
+            self.assertEqual(len(list(filter(rx_sign_line.match, node))), 1, "kernel hash not signed")
 
         # Verify the signature
         uboot_tools_sysroot_native = self._setup_uboot_tools_native()
-        self._verify_fit_image_signature(uboot_tools_sysroot_native, fitimage_path, os.path.join(deploy_dir_image, 'am335x-bone.dtb'))
+        self._verify_fit_image_signature(uboot_tools_sysroot_native, fitimage_path, os.path.join(bb_vars['DEPLOY_DIR_IMAGE'], 'am335x-bone.dtb'))
Signature · 1 line
2.45.2
#6 lore raw

PATCH 6/14 kernel-fitimage: fix handling of empty default dtb

In reply to Adrian Freihofer

From: Adrian Freihofer <[email protected]>

Fix error in run.do_assemble_fitimage_initramfs.2779256:
line 238: [: =: unary operator expected
if FIT_CONF_DEFAULT_DTB is empty.

---
 meta/classes-recipe/kernel-fitimage.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-recipe/kernel-fitimage.bbclass +1 -1
index 4b74ddc2015..6e6898e5f6f 100644
@@ -611,7 +611,7 @@ fitimage_assemble() {
 			fi
 
 			# Set the default dtb image if it exists in the devicetree.
-			if [ ${FIT_CONF_DEFAULT_DTB} = $DTB ];then
+			if [ "${FIT_CONF_DEFAULT_DTB}" = "$DTB" ];then
 				default_dtb_image=$(echo "$DTB" | tr '/' '_')
 			fi
 
Signature · 1 line
2.45.2
#7 lore raw

PATCH 7/14 kernel-fitimage: fix external dtb check

In reply to Adrian Freihofer

From: Adrian Freihofer <[email protected]>

If EXTERNAL_KERNEL_DEVICETREE and dtb_image_sect are empty variables dtb_path ends up as "/" which is available on most Unix systems but probably not the dtb_path which is needed here. Checking for a file makes more sense and also solves the issue with the "/".

---
 meta/classes-recipe/kernel-fitimage.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-recipe/kernel-fitimage.bbclass +1 -1
index 6e6898e5f6f..664ebc65d7c 100644
@@ -429,7 +429,7 @@ fitimage_emit_section_config() {
 	fi
 
 	dtb_path="${EXTERNAL_KERNEL_DEVICETREE}/${dtb_image_sect}"
-	if [ -e "$dtb_path" ]; then
+	if [ -f "$dtb_path" ]; then
 		compat=$(fdtget -t s "$dtb_path" / compatible | sed 's/ /", "/g')
 		if [ -n "$compat" ]; then
 			compatible_line="compatible = \"$compat\";"
Signature · 1 line
2.45.2
#8 lore raw

Re: [OE-core] [PATCH 07/14] kernel-fitimage: fix external dtb check

Quoted 17 lines · Adrian Freihofer

From: Adrian Freihofer <[email protected]>

If EXTERNAL_KERNEL_DEVICETREE and dtb_image_sect are empty variables dtb_path ends up as "/" which is available on most Unix systems but probably not the dtb_path which is needed here. Checking for a file makes more sense and also solves the issue with the "/".

---
  meta/classes-recipe/kernel-fitimage.bbclass | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-recipe/kernel-fitimage.bbclass +0 -0
index 6e6898e5f6f..664ebc65d7c 100644

meta/classes-recipe/kernel-fitimage.bbclass fitimage_emit_section_config() { · line 435

Quoted 8 lines
meta/classes-recipe/kernel-fitimage.bbclass +1 -1
@@ -429,7 +429,7 @@ fitimage_emit_section_config() {
  	fi
 
  	dtb_path="${EXTERNAL_KERNEL_DEVICETREE}/${dtb_image_sect}"
-	if [ -e "$dtb_path" ]; then
+	if [ -f "$dtb_path" ]; then
  		compat=$(fdtget -t s "$dtb_path" / compatible | sed 's/ /", "/g')
  		if [ -n "$compat" ]; then
  			compatible_line="compatible = \"$compat\";"

I can understand avoiding directories with this, but I've definitely seen cases where the dtb_path is pointing to a symbolic link.

Would [ -f "$dtb_path" -o -L "$dtb_path" ]; (or similar) solve this?

--Mark

Quoted 11 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#201511): https://lists.openembedded.org/g/openembedded-core/message/201511
Mute This Topic: https://lists.openembedded.org/mt/107033892/3616948
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#9 lore raw

Re: [OE-core] [PATCH 07/14] kernel-fitimage: fix external dtb check

Quoted 37 lines · Mark Hatle
Quoted 24 lines · Adrian Freihofer

From: Adrian Freihofer <[email protected]>

If EXTERNAL_KERNEL_DEVICETREE and dtb_image_sect are empty variables
dtb_path ends up as "/" which is available on most Unix systems but probably not the dtb_path which is needed here. Checking for a file makes more sense and also solves the issue with the "/".

---
  meta/classes-recipe/kernel-fitimage.bbclass | 2 +-

  1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-recipe/kernel-fitimage.bbclass +0 -0
index 6e6898e5f6f..664ebc65d7c 100644
@@ -429,7 +429,7 @@ fitimage_emit_section_config() {

   fi
dtb_path="${EXTERNAL_KERNEL_DEVICETREE}/${dtb_image_sect}"

meta/classes-recipe/kernel-fitimage.bbclass +1 -1
-	if [ -e "$dtb_path" ]; then
+	if [ -f "$dtb_path" ]; then
Quoted 5 lines

   compat=$(fdtget -t s "$dtb_path" / compatible | sed 's/ /", "/g')
   if [ -n "$compat" ]; then compatible_line="compatible = \"$compat\";"

I can understand avoiding directories with this, but I've definitely seen cases
where the dtb_path is pointing to a symbolic link.

Quoted 1 line

Would [ -f "$dtb_path" -o -L "$dtb_path" ]; (or similar) solve this?

Thank you for the re-view. I will fix this with v2. Adrian

Quoted 18 lines

--Mark

Quoted 15 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#201511):
https://lists.openembedded.org/g/openembedded-core/message/201511
Mute This Topic:
https://lists.openembedded.org/mt/107033892/3616948
Group Owner: [email protected] Unsubscribe:
https://lists.openembedded.org/g/openembedded-core/unsub [
[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#10 lore raw

PATCH 8/14 kernel: execute do_bundle_initramfs only when needed

In reply to Adrian Freihofer

From: Adrian Freihofer <[email protected]>

With INITRAMFS_IMAGE_BUNDLE = "1" the kernel Makefile is used to bundle the kernel binary and the initramfs. This works only when the build folder of the kernel is available. Building with sstate but with an empty TMPDIR requires to rebuild the kernel from scratch whenever something in the initramfs changes.

With INITRAMFS_IMAGE_BUNDLE = "" the fitImage generation is basically independent from the kernel Makefile and could therefore work with kernel binaries and tools provided by the sstate. But the dependency on the do_bundle_initramfs tasks does not allow this. However, if the INTIRAMFS_IMAGE is set but the INITRAMFS_IMAGE_BUNDLE is set to 0 the do_bundle_initramfs does nothing. There is no real argument for running this tasks.
As a first step towards getting the kernel from sstate when building an unbundled fitImage, the task dependencies need to be simplified. The do_bundle_initramfs task is now scheduled only when needed.

---
 meta/classes-recipe/kernel-fitimage.bbclass |  7 +++++--
 meta/classes-recipe/kernel.bbclass          | 14 ++++++++------
 2 files changed, 13 insertions(+), 8 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +5 -2
index 664ebc65d7c..fbeb20596ea 100644
@@ -36,6 +36,11 @@ python __anonymous () {
         if 'fitImage' in typeformake.split():
             d.setVar('KERNEL_IMAGETYPE_FOR_MAKE', typeformake.replace('fitImage', d.getVar('KERNEL_IMAGETYPE_REPLACEMENT')))
 
+        if bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')):
+            bb.build.addtask('assemble_fitimage_initramfs', 'do_deploy', 'do_bundle_initramfs', d)
+        else:
+            bb.build.addtask('assemble_fitimage_initramfs', 'do_deploy', 'do_install', d)
+
         image = d.getVar('INITRAMFS_IMAGE')
         if image:
             d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')
@@ -794,8 +799,6 @@ do_assemble_fitimage_initramfs() {
 	fi
 }
 
-addtask assemble_fitimage_initramfs before do_deploy after do_bundle_initramfs
-
 do_kernel_generate_rsa_keys() {
 	if [ "${UBOOT_SIGN_ENABLE}" = "0" ] && [ "${FIT_GENERATE_KEYS}" = "1" ]; then
 		bbwarn "FIT_GENERATE_KEYS is set to 1 even though UBOOT_SIGN_ENABLE is set to 0. The keys will not be generated as they won't be used."
meta/classes-recipe/kernel.bbclass +8 -6
index 89badd90f18..6d242114852 100644
@@ -139,15 +139,17 @@ set -e
 
     image = d.getVar('INITRAMFS_IMAGE')
     # If the INTIRAMFS_IMAGE is set but the INITRAMFS_IMAGE_BUNDLE is set to 0,
-    # the do_bundle_initramfs does nothing, but the INITRAMFS_IMAGE is built
-    # standalone for use by wic and other tools.
-    if image:
+    # the do_bundle_initramfs does nothing. Add the dependency only when needed.
+    if image and bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')):
+        bb.build.addtask('bundle_initramfs', 'do_deploy', 'do_install', d)
         if d.getVar('INITRAMFS_MULTICONFIG'):
             d.appendVarFlag('do_bundle_initramfs', 'mcdepends', ' mc::${INITRAMFS_MULTICONFIG}:${INITRAMFS_IMAGE}:do_image_complete')
         else:
             d.appendVarFlag('do_bundle_initramfs', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')
-    if image and bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')):
         bb.build.addtask('do_transform_bundled_initramfs', 'do_deploy', 'do_bundle_initramfs', d)
+    else:
+        d.appendVarFlag('do_deploy', 'depends', ' ${PN}:do_install')
+
 
     # NOTE: setting INITRAMFS_TASK is for backward compatibility
     #       The preferred method is to set INITRAMFS_IMAGE, because
@@ -327,6 +329,8 @@ do_bundle_initramfs () {
 				mv -f ${KERNEL_OUTPUT_DIR}/$imageType.bak ${KERNEL_OUTPUT_DIR}/$imageType
 			fi
 		done
+	else
+		bbwarn "Calling do_bundle_initramfs with INITRAMFS_IMAGE_BUNDLE=1 is deprecated"
 	fi
 }
 do_bundle_initramfs[dirs] = "${B}"
@@ -347,8 +351,6 @@ python do_devshell:prepend () {
     os.environ["LDFLAGS"] = ''
 }
 
-addtask bundle_initramfs after do_install before do_deploy
-
 KERNEL_DEBUG_TIMESTAMPS ??= "0"
 
 kernel_do_compile() {
Signature · 1 line
2.45.2
#11 lore raw

PATCH 9/14 kernel-fitimage: refactor do_assemble_fitimage_initramfs

In reply to Adrian Freihofer

From: Adrian Freihofer <[email protected]>

Make the fitimage_assemble function usable with absolute paths for the generated its and the fitImage file.

Later on this will allow to take the linux.bin and the DTB files from the sstated deploy derectory and write the generated files to a separate folder with independent sstate-cache handling.

Add 3 comments which are helpful for the next commit.

---
 meta/classes-recipe/kernel-fitimage.bbclass | 49 ++++++++++++++-------
 1 file changed, 34 insertions(+), 15 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +34 -15
index fbeb20596ea..e084dc57573 100644
@@ -174,8 +174,9 @@ fitimage_emit_section_kernel() {
 
 	ENTRYPOINT="${UBOOT_ENTRYPOINT}"
 	if [ -n "${UBOOT_ENTRYSYMBOL}" ]; then
+		kernel_base=$(basename $3)
 		ENTRYPOINT=`${HOST_PREFIX}nm vmlinux | \
-			awk '$3=="${UBOOT_ENTRYSYMBOL}" {print "0x"$1;exit}'`
+			awk '$kernel_base=="${UBOOT_ENTRYSYMBOL}" {print "0x"$1;exit}'`
 	fi
 
 	cat << EOF >> $1
@@ -572,7 +573,7 @@ fitimage_assemble() {
 	setupcount=""
 	bootscr_id=""
 	default_dtb_image=""
-	rm -f $1 arch/${ARCH}/boot/$2
+	rm -f "$1" "arch/${ARCH}/boot/$(basename $2)"
 
 	if [ -n "${UBOOT_SIGN_IMG_KEYNAME}" -a "${UBOOT_SIGN_KEYNAME}" = "${UBOOT_SIGN_IMG_KEYNAME}" ]; then
 		bbfatal "Keys used to sign images and configuration nodes must be different."
@@ -586,7 +587,7 @@ fitimage_assemble() {
 	fitimage_emit_section_maint $1 imagestart
 
 	uboot_prep_kimage
-	fitimage_emit_section_kernel $1 $kernelcount linux.bin "$linux_comp"
+	fitimage_emit_section_kernel $1 $kernelcount "$(readlink -f linux.bin)" "$linux_comp"
 
 	#
 	# Step 2: Prepare a DTB image section
@@ -610,9 +611,20 @@ fitimage_assemble() {
 				DTB_PATH="${KERNEL_OUTPUT_DIR}/$DTB"
 			fi
 
-		        # Strip off the path component from the filename
+			# Strip off the path component from the filename
 			if "${@'false' if oe.types.boolean(d.getVar('KERNEL_DTBVENDORED')) else 'true'}"; then
-			    DTB=`basename $DTB`
+				DTB=`basename $DTB`
+			fi
+
+			# Find DTBs without sub-folders when running in deploy folder
+			if [ ! -e "$DTB_PATH" ]; then
+				DTB=$(basename $DTB)
+				DTB_PATH=$(readlink -f $DTB)
+			fi
+
+			# Fail as early as possible if there is still no DTB file found
+			if [ ! -e "$DTB_PATH" ]; then
+				bberror "Cannot find the DTB file at $DTB_PATH"
 			fi
 
 			# Set the default dtb image if it exists in the devicetree.
@@ -665,9 +677,8 @@ fitimage_assemble() {
 
 	if [ -n "${UBOOT_ENV}" ] && [ -d "${STAGING_DIR_HOST}/boot" ]; then
 		if [ -e "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY}" ]; then
-			cp ${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY} ${B}
 			bootscr_id="${UBOOT_ENV_BINARY}"
-			fitimage_emit_section_boot_script $1 "$bootscr_id" ${UBOOT_ENV_BINARY}
+			fitimage_emit_section_boot_script $1 "$bootscr_id" "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY}"
 		else
 			bbwarn "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY} not found."
 		fi
@@ -676,9 +687,14 @@ fitimage_assemble() {
 	#
 	# Step 4: Prepare a setup section. (For x86)
 	#
+	# Run from kernel build folder (bundled mode)
 	if [ -e ${KERNEL_OUTPUT_DIR}/setup.bin ]; then
 		setupcount=1
 		fitimage_emit_section_setup $1 $setupcount ${KERNEL_OUTPUT_DIR}/setup.bin
+	# Run from deploy folder (unbundled mode)
+	elif [ -e setup.bin ]; then
+		setupcount=1
+		fitimage_emit_section_setup $1 $setupcount "$(readlink -f setup.bin)"
 	fi
 
 	#
@@ -751,8 +767,8 @@ fitimage_assemble() {
 	#
 	${UBOOT_MKIMAGE} \
 		${@'-D "${UBOOT_MKIMAGE_DTCOPTS}"' if len('${UBOOT_MKIMAGE_DTCOPTS}') else ''} \
-		-f $1 \
-		${KERNEL_OUTPUT_DIR}/$2
+		-f "$1" \
+		"$2"
 
 	#
 	# Step 8: Sign the image
@@ -761,7 +777,7 @@ fitimage_assemble() {
 		${UBOOT_MKIMAGE_SIGN} \
 			${@'-D "${UBOOT_MKIMAGE_DTCOPTS}"' if len('${UBOOT_MKIMAGE_DTCOPTS}') else ''} \
 			-F -k "${UBOOT_SIGN_KEYDIR}" \
-			-r ${KERNEL_OUTPUT_DIR}/$2 \
+			-r "$2" \
 			${UBOOT_MKIMAGE_SIGN_ARGS}
 	fi
 }
@@ -769,9 +785,9 @@ fitimage_assemble() {
 do_assemble_fitimage() {
 	if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then
 		cd ${B}
-		fitimage_assemble fit-image.its fitImage-none ""
+		fitimage_assemble fit-image.its "${KERNEL_OUTPUT_DIR}/fitImage-none" ""
 		if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
-			ln -sf fitImage-none ${B}/${KERNEL_OUTPUT_DIR}/fitImage
+			ln -sf fitImage-none "${B}/${KERNEL_OUTPUT_DIR}/fitImage"
 		fi
 	fi
 }
@@ -791,10 +807,10 @@ do_assemble_fitimage_initramfs() {
 		test -n "${INITRAMFS_IMAGE}" ; then
 		cd ${B}
 		if [ "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then
-			fitimage_assemble fit-image-${INITRAMFS_IMAGE}.its fitImage-bundle ""
-			ln -sf fitImage-bundle ${B}/${KERNEL_OUTPUT_DIR}/fitImage
+			fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage-bundle" ""
+			ln -sf fitImage-bundle "${B}/${KERNEL_OUTPUT_DIR}/fitImage"
 		else
-			fitimage_assemble fit-image-${INITRAMFS_IMAGE}.its fitImage-${INITRAMFS_IMAGE} 1
+			fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage-${INITRAMFS_IMAGE}" 1
 		fi
 	fi
 }
@@ -852,6 +868,7 @@ kernel_do_deploy:append() {
 	if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then
 
 		if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
+			# deploy the artifacts of do_assemble_fitimage
 			bbnote "Copying fit-image.its source file..."
 			install -m 0644 ${B}/fit-image.its "$deployDir/fitImage-its-${KERNEL_FIT_NAME}.its"
 			if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then
@@ -866,12 +883,14 @@ kernel_do_deploy:append() {
 		fi
 
 		if [ -n "${INITRAMFS_IMAGE}" ]; then
+			# deploy the artifacts of do_assemble_fitimage_initramfs for bundled as well as un-bundled mode
 			bbnote "Copying fit-image-${INITRAMFS_IMAGE}.its source file..."
 			install -m 0644 ${B}/fit-image-${INITRAMFS_IMAGE}.its "$deployDir/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its"
 			if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then
 				ln -snf fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its "$deployDir/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_LINK_NAME}"
 			fi
 
+			# deploy the artifacts of do_assemble_fitimage_initramfs for bundled mode only
 			if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
 				bbnote "Copying fitImage-${INITRAMFS_IMAGE} file..."
 				install -m 0644 ${B}/${KERNEL_OUTPUT_DIR}/fitImage-${INITRAMFS_IMAGE} "$deployDir/fitImage-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}${KERNEL_FIT_BIN_EXT}"
Signature · 1 line
2.45.2
#12 lore raw

Re: [OE-core] [PATCH 09/14] kernel-fitimage: refactor do_assemble_fitimage_initramfs

Quoted 50 lines · Adrian Freihofer

From: Adrian Freihofer <[email protected]>

Make the fitimage_assemble function usable with absolute paths for the generated its and the fitImage file.

Later on this will allow to take the linux.bin and the DTB files from the sstated deploy derectory and write the generated files to a separate folder with independent sstate-cache handling.

Add 3 comments which are helpful for the next commit.

---
  meta/classes-recipe/kernel-fitimage.bbclass | 49 ++++++++++++++-------
  1 file changed, 34 insertions(+), 15 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +4 -3
index fbeb20596ea..e084dc57573 100644
@@ -174,8 +174,9 @@ fitimage_emit_section_kernel() {
 
  	ENTRYPOINT="${UBOOT_ENTRYPOINT}"
  	if [ -n "${UBOOT_ENTRYSYMBOL}" ]; then
+		kernel_base=$(basename $3)
  		ENTRYPOINT=`${HOST_PREFIX}nm vmlinux | \
-			awk '$3=="${UBOOT_ENTRYSYMBOL}" {print "0x"$1;exit}'`
+			awk '$kernel_base=="${UBOOT_ENTRYSYMBOL}" {print "0x"$1;exit}'`
  	fi
 
  	cat << EOF >> $1
@@ -572,7 +573,7 @@ fitimage_assemble() {
  	setupcount=""
  	bootscr_id=""
  	default_dtb_image=""
-	rm -f $1 arch/${ARCH}/boot/$2
+	rm -f "$1" "arch/${ARCH}/boot/$(basename $2)"
 
  	if [ -n "${UBOOT_SIGN_IMG_KEYNAME}" -a "${UBOOT_SIGN_KEYNAME}" = "${UBOOT_SIGN_IMG_KEYNAME}" ]; then
  		bbfatal "Keys used to sign images and configuration nodes must be different."
@@ -586,7 +587,7 @@ fitimage_assemble() {
  	fitimage_emit_section_maint $1 imagestart
 
  	uboot_prep_kimage
-	fitimage_emit_section_kernel $1 $kernelcount linux.bin "$linux_comp"
+	fitimage_emit_section_kernel $1 $kernelcount "$(readlink -f linux.bin)" "$linux_comp"
 
  	#
  	# Step 2: Prepare a DTB image section

meta/classes-recipe/kernel-fitimage.bbclass fitimage_assemble() { · line 618

Quoted 10 lines
meta/classes-recipe/kernel-fitimage.bbclass +4 -2
@@ -610,9 +611,20 @@ fitimage_assemble() {
  				DTB_PATH="${KERNEL_OUTPUT_DIR}/$DTB"
  			fi
 
-		        # Strip off the path component from the filename
+			# Strip off the path component from the filename
  			if "${@'false' if oe.types.boolean(d.getVar('KERNEL_DTBVENDORED')) else 'true'}"; then
-			    DTB=`basename $DTB`
+				DTB=`basename $DTB`
+			fi
+

I'm purely reading the comment here (below). It talks about using the 'deploy' folder. I'm not sure what current behavior is, but with our (amd fpga) code, we've tried to completely move away from using ANYTHING out of the deploy directory. We found it horrible fragile [during the build] because the do_deploy can be delayed or not executed in some cases. (To be clear that isn't a bug!)

Instead everything we've done has moved to using the workdir/recipe-sysroot to access files and components.

Quoted 24 lines
meta/classes-recipe/kernel-fitimage.bbclass +10 -2
+			# Find DTBs without sub-folders when running in deploy folder
+			if [ ! -e "$DTB_PATH" ]; then
+				DTB=$(basename $DTB)
+				DTB_PATH=$(readlink -f $DTB)
+			fi
+
+			# Fail as early as possible if there is still no DTB file found
+			if [ ! -e "$DTB_PATH" ]; then
+				bberror "Cannot find the DTB file at $DTB_PATH"
  			fi
 
  			# Set the default dtb image if it exists in the devicetree.
@@ -665,9 +677,8 @@ fitimage_assemble() {
 
  	if [ -n "${UBOOT_ENV}" ] && [ -d "${STAGING_DIR_HOST}/boot" ]; then
  		if [ -e "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY}" ]; then
-			cp ${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY} ${B}
  			bootscr_id="${UBOOT_ENV_BINARY}"
-			fitimage_emit_section_boot_script $1 "$bootscr_id" ${UBOOT_ENV_BINARY}
+			fitimage_emit_section_boot_script $1 "$bootscr_id" "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY}"
  		else
  			bbwarn "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY} not found."
  		fi

meta/classes-recipe/kernel-fitimage.bbclass fitimage_assemble() { · line 693

Quoted 7 lines
meta/classes-recipe/kernel-fitimage.bbclass +1 -0
@@ -676,9 +687,14 @@ fitimage_assemble() {
  	#
  	# Step 4: Prepare a setup section. (For x86)
  	#
+	# Run from kernel build folder (bundled mode)
  	if [ -e ${KERNEL_OUTPUT_DIR}/setup.bin ]; then
  		setupcount=1
  		fitimage_emit_section_setup $1 $setupcount ${KERNEL_OUTPUT_DIR}/setup.bin

Same comment here about the 'deploy' vs recipe-sysroot. It looks to me like it's doing all of this in the CWD, but I'm unclear what it has been set to. So this could be an issue? I think that's why it was KERNEL_OUTPUT_DIR before.

Quoted 87 lines
meta/classes-recipe/kernel-fitimage.bbclass +15 -9
+	# Run from deploy folder (unbundled mode)
+	elif [ -e setup.bin ]; then
+		setupcount=1
+		fitimage_emit_section_setup $1 $setupcount "$(readlink -f setup.bin)"
  	fi
 
  	#
@@ -751,8 +767,8 @@ fitimage_assemble() {
  	#
  	${UBOOT_MKIMAGE} \
  		${@'-D "${UBOOT_MKIMAGE_DTCOPTS}"' if len('${UBOOT_MKIMAGE_DTCOPTS}') else ''} \
-		-f $1 \
-		${KERNEL_OUTPUT_DIR}/$2
+		-f "$1" \
+		"$2"
 
  	#
  	# Step 8: Sign the image
@@ -761,7 +777,7 @@ fitimage_assemble() {
  		${UBOOT_MKIMAGE_SIGN} \
  			${@'-D "${UBOOT_MKIMAGE_DTCOPTS}"' if len('${UBOOT_MKIMAGE_DTCOPTS}') else ''} \
  			-F -k "${UBOOT_SIGN_KEYDIR}" \
-			-r ${KERNEL_OUTPUT_DIR}/$2 \
+			-r "$2" \
  			${UBOOT_MKIMAGE_SIGN_ARGS}
  	fi
  }
@@ -769,9 +785,9 @@ fitimage_assemble() {
  do_assemble_fitimage() {
  	if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then
  		cd ${B}
-		fitimage_assemble fit-image.its fitImage-none ""
+		fitimage_assemble fit-image.its "${KERNEL_OUTPUT_DIR}/fitImage-none" ""
  		if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
-			ln -sf fitImage-none ${B}/${KERNEL_OUTPUT_DIR}/fitImage
+			ln -sf fitImage-none "${B}/${KERNEL_OUTPUT_DIR}/fitImage"
  		fi
  	fi
  }
@@ -791,10 +807,10 @@ do_assemble_fitimage_initramfs() {
  		test -n "${INITRAMFS_IMAGE}" ; then
  		cd ${B}
  		if [ "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then
-			fitimage_assemble fit-image-${INITRAMFS_IMAGE}.its fitImage-bundle ""
-			ln -sf fitImage-bundle ${B}/${KERNEL_OUTPUT_DIR}/fitImage
+			fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage-bundle" ""
+			ln -sf fitImage-bundle "${B}/${KERNEL_OUTPUT_DIR}/fitImage"
  		else
-			fitimage_assemble fit-image-${INITRAMFS_IMAGE}.its fitImage-${INITRAMFS_IMAGE} 1
+			fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage-${INITRAMFS_IMAGE}" 1
  		fi
  	fi
  }
@@ -852,6 +868,7 @@ kernel_do_deploy:append() {
  	if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then
 
  		if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
+			# deploy the artifacts of do_assemble_fitimage
  			bbnote "Copying fit-image.its source file..."
  			install -m 0644 ${B}/fit-image.its "$deployDir/fitImage-its-${KERNEL_FIT_NAME}.its"
  			if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then
@@ -866,12 +883,14 @@ kernel_do_deploy:append() {
  		fi
 
  		if [ -n "${INITRAMFS_IMAGE}" ]; then
+			# deploy the artifacts of do_assemble_fitimage_initramfs for bundled as well as un-bundled mode
  			bbnote "Copying fit-image-${INITRAMFS_IMAGE}.its source file..."
  			install -m 0644 ${B}/fit-image-${INITRAMFS_IMAGE}.its "$deployDir/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its"
  			if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then
  				ln -snf fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its "$deployDir/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_LINK_NAME}"
  			fi
 
+			# deploy the artifacts of do_assemble_fitimage_initramfs for bundled mode only
  			if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
  				bbnote "Copying fitImage-${INITRAMFS_IMAGE} file..."
  				install -m 0644 ${B}/${KERNEL_OUTPUT_DIR}/fitImage-${INITRAMFS_IMAGE} "$deployDir/fitImage-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}${KERNEL_FIT_BIN_EXT}"
 
 
 
-=-=-=-=-=-=-=-=-=-=-=-

Links: You receive all messages sent to this group.
View/Reply Online (#201513): https://lists.openembedded.org/g/openembedded-core/message/201513
Mute This Topic: https://lists.openembedded.org/mt/107033894/3616948
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#13 lore raw

Re: [OE-core] [PATCH 09/14] kernel-fitimage: refactor do_assemble_fitimage_initramfs

Quoted 92 lines · Mark Hatle
Quoted 72 lines · Adrian Freihofer

From: Adrian Freihofer <[email protected]>

Make the fitimage_assemble function usable with absolute paths for the
generated its and the fitImage file.

Later on this will allow to take the linux.bin and the DTB files from
the sstated deploy derectory and write the generated files to a separate
folder with independent sstate-cache handling.

Add 3 comments which are helpful for the next commit.

---
  meta/classes-recipe/kernel-fitimage.bbclass | 49 ++++++++++++++--
-----

  1 file changed, 34 insertions(+), 15 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +0 -0
index fbeb20596ea..e084dc57573 100644
@@ -174,8 +174,9 @@ fitimage_emit_section_kernel() {


   ENTRYPOINT="${UBOOT_ENTRYPOINT}"
   if [ -n "${UBOOT_ENTRYSYMBOL}" ]; then

meta/classes-recipe/kernel-fitimage.bbclass +1 -0
+		kernel_base=$(basename $3)

   ENTRYPOINT=`${HOST_PREFIX}nm vmlinux | \

meta/classes-recipe/kernel-fitimage.bbclass +0 -1
-			awk '$3=="${UBOOT_ENTRYSYMBOL}" {print

"0x"$1;exit}'`

meta/classes-recipe/kernel-fitimage.bbclass +1 -0
+			awk '$kernel_base=="${UBOOT_ENTRYSYMBOL}"

{print "0x"$1;exit}'`
   fi

   cat << EOF >> $1

meta/classes-recipe/kernel-fitimage.bbclass +0 -0
@@ -572,7 +573,7 @@ fitimage_assemble() {

   setupcount=""
   bootscr_id=""
   default_dtb_image=""

meta/classes-recipe/kernel-fitimage.bbclass +1 -1
-	rm -f $1 arch/${ARCH}/boot/$2
+	rm -f "$1" "arch/${ARCH}/boot/$(basename $2)"


   if [ -n "${UBOOT_SIGN_IMG_KEYNAME}" -a "${UBOOT_SIGN_KEYNAME}" = "${UBOOT_SIGN_IMG_KEYNAME}" ]; then bbfatal "Keys used to sign images and configuration nodes must be different."

meta/classes-recipe/kernel-fitimage.bbclass +0 -0
@@ -586,7 +587,7 @@ fitimage_assemble() {

   fitimage_emit_section_maint $1 imagestart

   uboot_prep_kimage

meta/classes-recipe/kernel-fitimage.bbclass +0 -1
-	fitimage_emit_section_kernel $1 $kernelcount linux.bin

"$linux_comp"

meta/classes-recipe/kernel-fitimage.bbclass +1 -0
+	fitimage_emit_section_kernel $1 $kernelcount "$(readlink -

f linux.bin)" "$linux_comp"

   #
   # Step 2: Prepare a DTB image section

meta/classes-recipe/kernel-fitimage.bbclass +0 -0
@@ -610,9 +611,20 @@ fitimage_assemble() {

   DTB_PATH="${KERNEL_OUTPUT_DIR}/$DT B"
   fi

meta/classes-recipe/kernel-fitimage.bbclass +0 -1
-		        # Strip off the path component from the

filename

meta/classes-recipe/kernel-fitimage.bbclass +1 -0
+			# Strip off the path component from the

filename
   if "${@'false' if oe.types.boolean(d.getVar('KERNEL_DTBVENDORED')) else 'true'}"; then

meta/classes-recipe/kernel-fitimage.bbclass fitimage_assemble() { · line 618

Quoted 4 lines
meta/classes-recipe/kernel-fitimage.bbclass +3 -1
-			    DTB=`basename $DTB`
+				DTB=`basename $DTB`
+			fi
+

I'm purely reading the comment here (below).  It talks about using the 'deploy'
folder.  I'm not sure what current behavior is, but with our (amd fpga) code,
we've tried to completely move away from using ANYTHING out of the deploy
directory.  We found it horrible fragile [during the build] because the
do_deploy can be delayed or not executed in some cases.  (To be clear that isn't
a bug!)

Quoted 3 lines

Instead everything we've done has moved to using the workdir/recipe- sysroot to
access files and components.

In general, I see no difference between do_deploy and do_recipe_sysroot. Both are sstated tasks, and if the task dependencies are set correctly, the files should be available in one folder or the other. Or am I missing something?

But anyway, I'm working on a different implementation now, following your advice. Let's see if it helps to fix the build error that occurred on the AB. It looks like moving a task to after do_deploy causes problems that are almost unsolvable. Staging via do_recipe_sysroot would run before do_deploy.

Quoted 42 lines
Quoted 42 lines
meta/classes-recipe/kernel-fitimage.bbclass +1 -0
+			# Find DTBs without sub-folders when

running in deploy folder

meta/classes-recipe/kernel-fitimage.bbclass +6 -0
+			if [ ! -e "$DTB_PATH" ]; then
+				DTB=$(basename $DTB)
+				DTB_PATH=$(readlink -f $DTB)
+			fi
+
+			# Fail as early as possible if there is

still no DTB file found

meta/classes-recipe/kernel-fitimage.bbclass +2 -0
+			if [ ! -e "$DTB_PATH" ]; then
+				bberror "Cannot find the DTB file

at $DTB_PATH"
   fi

   # Set the default dtb image if it exists in the devicetree.

meta/classes-recipe/kernel-fitimage.bbclass +0 -0
@@ -665,9 +677,8 @@ fitimage_assemble() {


   if [ -n "${UBOOT_ENV}" ] && [ -d "${STAGING_DIR_HOST}/boot" ]; then
   if [ -e "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY}" ]; then

meta/classes-recipe/kernel-fitimage.bbclass +0 -1
-			cp

${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY} ${B} bootscr_id="${UBOOT_ENV_BINARY}"

meta/classes-recipe/kernel-fitimage.bbclass +0 -1
-			fitimage_emit_section_boot_script $1

"$bootscr_id" ${UBOOT_ENV_BINARY}

meta/classes-recipe/kernel-fitimage.bbclass +1 -0
+			fitimage_emit_section_boot_script $1

"$bootscr_id" "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY}" else
   bbwarn "${STAGING_DIR_HOST}/boot/${UBOOT_ENV_BINARY} not found." fi

meta/classes-recipe/kernel-fitimage.bbclass +1 -0
@@ -676,9 +687,14 @@ fitimage_assemble() {
   	#
   	# Step 4: Prepare a setup section. (For x86)
   	#
+	# Run from kernel build folder (bundled mode)

   if [ -e ${KERNEL_OUTPUT_DIR}/setup.bin ]; then setupcount=1 fitimage_emit_section_setup $1 $setupcount

Quoted 8 lines
Quoted 1 line

${KERNEL_OUTPUT_DIR}/setup.bin

Same comment here about the 'deploy' vs recipe-sysroot.  It looks to me like
it's doing all of this in the CWD, but I'm unclear what it has been set to.  So
this could be an issue?  I think that's why it was KERNEL_OUTPUT_DIR before.

Using CWD or KERNEL_OUTPUT_DIR should be the same in that sense since KERNEL_OUTPUT_DIR is relative to CWD:
bitbake-getvar -r virtual/kernel KERNEL_OUTPUT_DIR KERNEL_OUTPUT_DIR="arch/x86/boot"

CWD is defined by the [dirs] variable flag of the task (acc. to
https://docs.yoctoproject.org/bitbake/singleindex.html). I think this is more reliable than using cd in tasks. cd is likely to break with bbappends.

But also this will probably be simpler with v2.

Thank you for the re-view
Adrian

Quoted 122 lines
Quoted 122 lines
meta/classes-recipe/kernel-fitimage.bbclass +4 -0
+	# Run from deploy folder (unbundled mode)
+	elif [ -e setup.bin ]; then
+		setupcount=1
+		fitimage_emit_section_setup $1 $setupcount

"$(readlink -f setup.bin)"
   fi

   #

meta/classes-recipe/kernel-fitimage.bbclass +0 -0
@@ -751,8 +767,8 @@ fitimage_assemble() {
   	#
   	${UBOOT_MKIMAGE} \

   ${@'-D "${UBOOT_MKIMAGE_DTCOPTS}"' if len('${UBOOT_MKIMAGE_DTCOPTS}') else ''} \

meta/classes-recipe/kernel-fitimage.bbclass +2 -2
-		-f $1 \
-		${KERNEL_OUTPUT_DIR}/$2
+		-f "$1" \
+		"$2"


   #
   # Step 8: Sign the image

meta/classes-recipe/kernel-fitimage.bbclass +0 -0
@@ -761,7 +777,7 @@ fitimage_assemble() {
   		${UBOOT_MKIMAGE_SIGN} \

   ${@'-D "${UBOOT_MKIMAGE_DTCOPTS}"' if len('${UBOOT_MKIMAGE_DTCOPTS}') else ''} \
   -F -k "${UBOOT_SIGN_KEYDIR}" \

meta/classes-recipe/kernel-fitimage.bbclass +1 -1
-			-r ${KERNEL_OUTPUT_DIR}/$2 \
+			-r "$2" \

   ${UBOOT_MKIMAGE_SIGN_ARGS} fi
  }

meta/classes-recipe/kernel-fitimage.bbclass +0 -0
@@ -769,9 +785,9 @@ fitimage_assemble() {
   do_assemble_fitimage() {

   if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then cd ${B}

meta/classes-recipe/kernel-fitimage.bbclass +1 -1
-		fitimage_assemble fit-image.its fitImage-none ""
+		fitimage_assemble fit-image.its

"${KERNEL_OUTPUT_DIR}/fitImage-none" ""
   if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then

meta/classes-recipe/kernel-fitimage.bbclass +0 -1
-			ln -sf fitImage-none

${B}/${KERNEL_OUTPUT_DIR}/fitImage

meta/classes-recipe/kernel-fitimage.bbclass +1 -0
+			ln -sf fitImage-none

"${B}/${KERNEL_OUTPUT_DIR}/fitImage"
   fi
   fi
  }

meta/classes-recipe/kernel-fitimage.bbclass +0 -0
@@ -791,10 +807,10 @@ do_assemble_fitimage_initramfs() {

   test -n "${INITRAMFS_IMAGE}" ; then cd ${B}
   if [ "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then

meta/classes-recipe/kernel-fitimage.bbclass +0 -1
-			fitimage_assemble fit-image-

${INITRAMFS_IMAGE}.its fitImage-bundle ""

meta/classes-recipe/kernel-fitimage.bbclass +0 -1
-			ln -sf fitImage-bundle

${B}/${KERNEL_OUTPUT_DIR}/fitImage

meta/classes-recipe/kernel-fitimage.bbclass +1 -0
+			fitimage_assemble "fit-image-

${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage-bundle" ""

meta/classes-recipe/kernel-fitimage.bbclass +1 -0
+			ln -sf fitImage-bundle

"${B}/${KERNEL_OUTPUT_DIR}/fitImage"
   else

meta/classes-recipe/kernel-fitimage.bbclass +0 -1
-			fitimage_assemble fit-image-

${INITRAMFS_IMAGE}.its fitImage-${INITRAMFS_IMAGE} 1

meta/classes-recipe/kernel-fitimage.bbclass +1 -0
+			fitimage_assemble "fit-image-

${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage- ${INITRAMFS_IMAGE}" 1
   fi
   fi
  }

meta/classes-recipe/kernel-fitimage.bbclass +0 -0
@@ -852,6 +868,7 @@ kernel_do_deploy:append() {

   if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then
   if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then

meta/classes-recipe/kernel-fitimage.bbclass +1 -0
+			# deploy the artifacts of

do_assemble_fitimage
   bbnote "Copying fit-image.its source file..."
   install -m 0644 ${B}/fit-image.its "$deployDir/fitImage-its-${KERNEL_FIT_NAME}.its"
   if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then

meta/classes-recipe/kernel-fitimage.bbclass +0 -0
@@ -866,12 +883,14 @@ kernel_do_deploy:append() {

   fi

   if [ -n "${INITRAMFS_IMAGE}" ]; then

meta/classes-recipe/kernel-fitimage.bbclass +1 -0
+			# deploy the artifacts of

do_assemble_fitimage_initramfs for bundled as well as un-bundled mode
   bbnote "Copying fit-image- ${INITRAMFS_IMAGE}.its source file..."
   install -m 0644 ${B}/fit-image- ${INITRAMFS_IMAGE}.its "$deployDir/fitImage-its- ${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its"
   if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then ln -snf fitImage-its- ${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its "$deployDir/fitImage-its-${INITRAMFS_IMAGE_NAME}- ${KERNEL_FIT_LINK_NAME}"
   fi

meta/classes-recipe/kernel-fitimage.bbclass +1 -0
+			# deploy the artifacts of

do_assemble_fitimage_initramfs for bundled mode only
   if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
   bbnote "Copying fitImage- ${INITRAMFS_IMAGE} file..."
   install -m 0644 ${B}/${KERNEL_OUTPUT_DIR}/fitImage-${INITRAMFS_IMAGE} "$deployDir/fitImage-${INITRAMFS_IMAGE_NAME}- ${KERNEL_FIT_NAME}${KERNEL_FIT_BIN_EXT}"

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#201513):
https://lists.openembedded.org/g/openembedded-core/message/201513
Mute This Topic:
https://lists.openembedded.org/mt/107033894/3616948
Group Owner: [email protected] Unsubscribe:
https://lists.openembedded.org/g/openembedded-core/unsub [
[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#14 lore raw

PATCH 10/14 kernel-fitimage: refactor creation of linux.bin

In reply to Adrian Freihofer

From: Adrian Freihofer <[email protected]>

Assembling the fitImage with initramfs requires the linux.bin and the corresponding DTB files. To avoid that the kernel gets re-built from scratch whenever the initramfs changes, these files must be sstate cached after the kernel is built, before the fitImage gets assembled.

Therefore the uboot_prep_kimage needs to be split:
- a function creating the linux.bin artifact
- a function which can find the linux.bin file from an independent task later on.

This will allow to run the do_assemble_fitimage_initramfs task from the deploy folder where the linux.bin and the DTBs are provided via sstate-cache.

---
 meta/classes-recipe/kernel-fitimage.bbclass |  8 ++++----
 meta/classes-recipe/kernel-uboot.bbclass    | 17 +++++++++++++++++
 2 files changed, 21 insertions(+), 4 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +4 -4
index e084dc57573..64836f6bef0 100644
@@ -165,12 +165,12 @@ EOF
 # $1 ... .its filename
 # $2 ... Image counter
 # $3 ... Path to kernel image
-# $4 ... Compression type
 fitimage_emit_section_kernel() {
 
 	kernel_csum="${FIT_HASH_ALG}"
 	kernel_sign_algo="${FIT_SIGN_ALG}"
 	kernel_sign_keyname="${UBOOT_SIGN_IMG_KEYNAME}"
+	get_linux_comp
 
 	ENTRYPOINT="${UBOOT_ENTRYPOINT}"
 	if [ -n "${UBOOT_ENTRYSYMBOL}" ]; then
@@ -186,7 +186,7 @@ fitimage_emit_section_kernel() {
                         type = "${UBOOT_MKIMAGE_KERNEL_TYPE}";
                         arch = "${UBOOT_ARCH}";
                         os = "linux";
-                        compression = "$4";
+                        compression = "$linux_comp";
                         load = <${UBOOT_LOADADDRESS}>;
                         entry = <$ENTRYPOINT>;
                         hash-1 {
@@ -585,8 +585,6 @@ fitimage_assemble() {
 	# Step 1: Prepare a kernel image section.
 	#
 	fitimage_emit_section_maint $1 imagestart
-
-	uboot_prep_kimage
 	fitimage_emit_section_kernel $1 $kernelcount "$(readlink -f linux.bin)" "$linux_comp"
 
 	#
@@ -785,6 +783,7 @@ fitimage_assemble() {
 do_assemble_fitimage() {
 	if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then
 		cd ${B}
+		uboot_prep_kimage
 		fitimage_assemble fit-image.its "${KERNEL_OUTPUT_DIR}/fitImage-none" ""
 		if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
 			ln -sf fitImage-none "${B}/${KERNEL_OUTPUT_DIR}/fitImage"
@@ -806,6 +805,7 @@ do_assemble_fitimage_initramfs() {
 	if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage" && \
 		test -n "${INITRAMFS_IMAGE}" ; then
 		cd ${B}
+		uboot_prep_kimage
 		if [ "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then
 			fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage-bundle" ""
 			ln -sf fitImage-bundle "${B}/${KERNEL_OUTPUT_DIR}/fitImage"
meta/classes-recipe/kernel-uboot.bbclass +17 -0
index 30a85ccc287..e692ca27cdd 100644
@@ -47,3 +47,20 @@ uboot_prep_kimage() {
 
 	echo "${linux_comp}"
 }
+
+# Get the linux_comp variable back from the linux.bin file
+# which was created by uboot_prep_kimage
+get_linux_comp() {
+	if [ ! -e linux.bin ]; then
+	    bbfatal "linux.bin does not exist $(pwd)"
+	fi
+
+	linux_comp=none
+	if gzip -l linux.bin> /dev/null 2>&1; then
+		linux_comp=gzip
+	elif lzop -l linux.bin> /dev/null 2>&1; then
+		linux_comp=lzo
+	fi
+
+	echo "${linux_comp}"
+}
Signature · 1 line
2.45.2
#15 lore raw

PATCH 11/14 kernel-fitimage: fix intentation

In reply to Adrian Freihofer

From: Adrian Freihofer <[email protected]>

python part should be 4 spaces, not 8.

---
 meta/classes-recipe/kernel-fitimage.bbclass | 52 ++++++++++-----------
 1 file changed, 26 insertions(+), 26 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +26 -26
index 64836f6bef0..88b8edb9b54 100644
@@ -29,32 +29,32 @@ KERNEL_IMAGETYPE_REPLACEMENT ?= "${@get_fit_replacement_type(d)}"
 DEPENDS:append = " ${@'u-boot-tools-native dtc-native' if 'fitImage' in (d.getVar('KERNEL_IMAGETYPES') or '').split() else ''}"
 
 python __anonymous () {
-        # Override KERNEL_IMAGETYPE_FOR_MAKE variable, which is internal
-        # to kernel.bbclass . We have to override it, since we pack zImage
-        # (at least for now) into the fitImage .
-        typeformake = d.getVar("KERNEL_IMAGETYPE_FOR_MAKE") or ""
-        if 'fitImage' in typeformake.split():
-            d.setVar('KERNEL_IMAGETYPE_FOR_MAKE', typeformake.replace('fitImage', d.getVar('KERNEL_IMAGETYPE_REPLACEMENT')))
-
-        if bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')):
-            bb.build.addtask('assemble_fitimage_initramfs', 'do_deploy', 'do_bundle_initramfs', d)
-        else:
-            bb.build.addtask('assemble_fitimage_initramfs', 'do_deploy', 'do_install', d)
-
-        image = d.getVar('INITRAMFS_IMAGE')
-        if image:
-            d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')
-
-        ubootenv = d.getVar('UBOOT_ENV')
-        if ubootenv:
-            d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/bootloader:do_populate_sysroot')
-
-        #check if there are any dtb providers
-        providerdtb = d.getVar("PREFERRED_PROVIDER_virtual/dtb")
-        if providerdtb:
-            d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/dtb:do_populate_sysroot')
-            d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' virtual/dtb:do_populate_sysroot')
-            d.setVar('EXTERNAL_KERNEL_DEVICETREE', "${RECIPE_SYSROOT}/boot/devicetree")
+    # Override KERNEL_IMAGETYPE_FOR_MAKE variable, which is internal
+    # to kernel.bbclass . We have to override it, since we pack zImage
+    # (at least for now) into the fitImage .
+    typeformake = d.getVar("KERNEL_IMAGETYPE_FOR_MAKE") or ""
+    if 'fitImage' in typeformake.split():
+        d.setVar('KERNEL_IMAGETYPE_FOR_MAKE', typeformake.replace('fitImage', d.getVar('KERNEL_IMAGETYPE_REPLACEMENT')))
+
+    if bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')):
+        bb.build.addtask('assemble_fitimage_initramfs', 'do_deploy', 'do_bundle_initramfs', d)
+    else:
+        bb.build.addtask('assemble_fitimage_initramfs', 'do_deploy', 'do_install', d)
+
+    image = d.getVar('INITRAMFS_IMAGE')
+    if image:
+        d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')
+
+    ubootenv = d.getVar('UBOOT_ENV')
+    if ubootenv:
+        d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/bootloader:do_populate_sysroot')
+
+    #check if there are any dtb providers
+    providerdtb = d.getVar("PREFERRED_PROVIDER_virtual/dtb")
+    if providerdtb:
+        d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/dtb:do_populate_sysroot')
+        d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' virtual/dtb:do_populate_sysroot')
+        d.setVar('EXTERNAL_KERNEL_DEVICETREE', "${RECIPE_SYSROOT}/boot/devicetree")
 }
 
 
Signature · 1 line
2.45.2
#16 lore raw

PATCH 12/14 kernel-fitimage: run unbundled fitimage after deploy

In reply to Adrian Freihofer

From: Adrian Freihofer <[email protected]>

Generate the fitImage with unbundled initramfs after do_deploy. This allows to use the kernel image from the sstate of the do_deploy task.

Without this change, the kernel is always rebuilt if something changes in the initramfs, which happens frequently. If the kernel is rebuilt with an empty TMPDIR, the process starts by fetching the large kernel git repository and recompiling it from scratch. This is very inefficient.
This cannot be improved if INITRAMFS_IMAGE_BUNDLE = "1". If the kernel Makefile is needed to generate the initramfs bundle the kernel build folder is required.
But for a build configuration with INITRAMFS_IMAGE_BUNDLE = "" the build folder is not needed. Creating the initramfs bundle requires:
linux.bin, DTBs and the initramfs which are availabe in the deploy directory. That means creating the fitImage can be done with artifacts which are already cached by the sstate.

The do_assemble_fitimage_initramfs gets split into two tasks:
- do_assemble_fitimage_initramfs creates the bundled fitImage which runs before do_deploy. This is without sstate and therefore the do_deploy task still deploys the artifacts generated by this task.
- do_deploy_fitimage_unbundled creates the fitImage in unbundled mode. This task runs after do_deploy. It uses the linux.bin artifact from do_deploy (and hence from sstate if available) to create the fitImage with initramfs.
This task is done with SSTATE_SKIP_CREATION, because calling mkimage is probably quicker than handling this via sstate. Further this also works with the eSDK installer (oe-selftest -r esdk is successful).

Unfortunately, this change is not 100% backwards compatible. Previously, a dependency on do_deploy deployed all types of fit images. With this change, deploying the unbundled fit image requires a dependency on do_deploy_fitimage_unbundled. This will be mitigated with a next commit.

---
 meta/classes-recipe/kernel-fitimage.bbclass | 131 ++++++++++++--------
 meta/classes-recipe/kernel.bbclass          |   3 +-
 2 files changed, 80 insertions(+), 54 deletions(-)
meta/classes-recipe/kernel-fitimage.bbclass +78 -53
index 88b8edb9b54..c4d474b8c1f 100644
@@ -26,35 +26,52 @@ def get_fit_replacement_type(d):
     return replacementtype
 
 KERNEL_IMAGETYPE_REPLACEMENT ?= "${@get_fit_replacement_type(d)}"
-DEPENDS:append = " ${@'u-boot-tools-native dtc-native' if 'fitImage' in (d.getVar('KERNEL_IMAGETYPES') or '').split() else ''}"
 
 python __anonymous () {
-    # Override KERNEL_IMAGETYPE_FOR_MAKE variable, which is internal
-    # to kernel.bbclass . We have to override it, since we pack zImage
-    # (at least for now) into the fitImage .
-    typeformake = d.getVar("KERNEL_IMAGETYPE_FOR_MAKE") or ""
-    if 'fitImage' in typeformake.split():
-        d.setVar('KERNEL_IMAGETYPE_FOR_MAKE', typeformake.replace('fitImage', d.getVar('KERNEL_IMAGETYPE_REPLACEMENT')))
-
-    if bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE')):
-        bb.build.addtask('assemble_fitimage_initramfs', 'do_deploy', 'do_bundle_initramfs', d)
-    else:
-        bb.build.addtask('assemble_fitimage_initramfs', 'do_deploy', 'do_install', d)
-
-    image = d.getVar('INITRAMFS_IMAGE')
-    if image:
-        d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' ${INITRAMFS_IMAGE}:do_image_complete')
-
-    ubootenv = d.getVar('UBOOT_ENV')
-    if ubootenv:
-        d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/bootloader:do_populate_sysroot')
-
     #check if there are any dtb providers
     providerdtb = d.getVar("PREFERRED_PROVIDER_virtual/dtb")
     if providerdtb:
         d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/dtb:do_populate_sysroot')
-        d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' virtual/dtb:do_populate_sysroot')
         d.setVar('EXTERNAL_KERNEL_DEVICETREE', "${RECIPE_SYSROOT}/boot/devicetree")
+
+    d.appendVarFlag('do_assemble_fitimage', 'depends',
+                    ' u-boot-tools-native:do_populate_sysroot dtc-native:do_populate_sysroot')
+    ubootenv = d.getVar('UBOOT_ENV')
+    if ubootenv:
+        d.appendVarFlag('do_assemble_fitimage', 'depends', ' virtual/bootloader:do_populate_sysroot')
+
+    typeformake = d.getVar("KERNEL_IMAGETYPE_FOR_MAKE") or ""
+    if 'fitImage' in typeformake.split():
+        # Override KERNEL_IMAGETYPE_FOR_MAKE variable, which is internal
+        # to kernel.bbclass . We have to override it, since we pack zImage
+        # (at least for now) into the fitImage .
+        d.setVar('KERNEL_IMAGETYPE_FOR_MAKE', typeformake.replace('fitImage', d.getVar('KERNEL_IMAGETYPE_REPLACEMENT')))
+
+        # A fitImage with the initramfs bundled in the kernel Image is deployed by do_deploy.
+        # In case of a build from an empty TMPDIR a clean kernel re-build is required just because creating the bundled
+        # kernel image requires the kernel Makefile and therefore the kernel's populated build directory.
+        # In case of an unbundled fitImage assembling the fitImage works independenly from the kernel build framework.
+        # This allow to take the kernel binary from the sstated deploy directory. But it requires to assemble the
+        # fitImage in a separate task (do_deploy_fitimage_unbundled) running after do_deploy.
+        initramfs_image = d.getVar('INITRAMFS_IMAGE')
+        bundled = bb.utils.to_boolean(d.getVar('INITRAMFS_IMAGE_BUNDLE'))
+        if initramfs_image:
+            if bundled:
+                bb.build.addtask('do_assemble_fitimage_initramfs', 'do_deploy', 'do_bundle_initramfs', d)
+                d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' ' + initramfs_image + ':do_image_complete')
+                d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends',
+                                ' u-boot-tools-native:do_populate_sysroot dtc-native:do_populate_sysroot')
+                if providerdtb:
+                    d.appendVarFlag('do_assemble_fitimage_initramfs', 'depends', ' virtual/dtb:do_populate_sysroot')
+            else:
+                bb.build.addtask('do_deploy_fitimage_unbundled', 'do_build', 'do_deploy', d)
+                bb.build.addtask('do_deploy_fitimage_unbundled_setscene', '', '', d)
+                d.appendVar('SSTATETASKS', ' do_deploy_fitimage_unbundled')
+                d.appendVarFlag('do_deploy_fitimage_unbundled', 'depends', ' ' + initramfs_image + ':do_image_complete')
+                d.appendVarFlag('do_deploy_fitimage_unbundled', 'depends',
+                                ' u-boot-tools-native:do_populate_sysroot dtc-native:do_populate_sysroot')
+                if providerdtb:
+                    d.appendVarFlag('do_deploy_fitimage_unbundled', 'depends', ' virtual/dtb:do_populate_sysroot')
 }
 
 
@@ -802,18 +819,30 @@ do_install:append() {
 }
 
 do_assemble_fitimage_initramfs() {
-	if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage" && \
-		test -n "${INITRAMFS_IMAGE}" ; then
-		cd ${B}
-		uboot_prep_kimage
-		if [ "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then
-			fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage-bundle" ""
-			ln -sf fitImage-bundle "${B}/${KERNEL_OUTPUT_DIR}/fitImage"
-		else
-			fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage-${INITRAMFS_IMAGE}" 1
-		fi
+	uboot_prep_kimage
+	fitimage_assemble "fit-image-${INITRAMFS_IMAGE}.its" "${KERNEL_OUTPUT_DIR}/fitImage-bundle" ""
+	ln -sf fitImage-bundle "${B}/${KERNEL_OUTPUT_DIR}/fitImage"
+}
+do_assemble_fitimage_initramfs[dirs] = "${B}"
+
+do_deploy_fitimage_unbundled() {
+	fitimage_assemble "${DEPLOY_DIR_IMAGE}/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its"\
+	  "${DEPLOY_DIR_IMAGE}/fitImage-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}${KERNEL_FIT_BIN_EXT}" 1
+	if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then
+		ln -snf "fitImage-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}${KERNEL_FIT_BIN_EXT}" "${DEPLOY_DIR_IMAGE}/fitImage-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_LINK_NAME}"
+		ln -snf fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its "${DEPLOY_DIR_IMAGE}/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_LINK_NAME}"
 	fi
 }
+DEPLOYDIR_FITIMAGE_UNBUNDLED = "${WORKDIR}/deploy-fitimage-unbundled-${PN}"
+SSTATE_SKIP_CREATION:task-deploy-fitimage-unbundled = '1'
+do_deploy_fitimage_unbundled[sstate-inputdirs] = "${DEPLOYDIR_FITIMAGE_UNBUNDLED}"
+do_deploy_fitimage_unbundled[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
+python do_deploy_fitimage_unbundled_setscene () {
+    sstate_setscene(d)
+}
+do_deploy_fitimage_unbundled[dirs] = "${DEPLOY_DIR_IMAGE}"
+do_deploy_fitimage_unbundled[cleandirs] = "${DEPLOYDIR_FITIMAGE_UNBUNDLED}"
+do_deploy_fitimage_unbundled[stamp-extra-info] = "${MACHINE_ARCH}"
 
 do_kernel_generate_rsa_keys() {
 	if [ "${UBOOT_SIGN_ENABLE}" = "0" ] && [ "${FIT_GENERATE_KEYS}" = "1" ]; then
@@ -866,9 +895,23 @@ kernel_do_deploy[vardepsexclude] = "DATETIME"
 kernel_do_deploy:append() {
 	# Update deploy directory
 	if echo ${KERNEL_IMAGETYPES} | grep -wq "fitImage"; then
-
-		if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
-			# deploy the artifacts of do_assemble_fitimage
+		if  [ -n "${INITRAMFS_IMAGE}" ] && [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
+			# do_deploy_fitimage_unbundled needs the linux.bin file for the unbundled fitImage deployment
+			bbnote "Deploying linux.bin file for do_deploy_fitimage_unbundled..."
+			uboot_prep_kimage
+			install -m 0644 ${B}/linux.bin $deployDir/linux.bin
+			if [ -e "${B}/${KERNEL_OUTPUT_DIR}/setup.bin" ]; then
+				install -m 0644 "${B}/${KERNEL_OUTPUT_DIR}/setup.bin" "$deployDir/setup.bin"
+			fi
+		elif [ -n "${INITRAMFS_IMAGE}" ]; then
+			# deploy the artifacts created by do_assemble_fitimage_initramfs for bundled mode
+			bbnote "Copying fit-image-${INITRAMFS_IMAGE}.its source file..."
+			install -m 0644 ${B}/fit-image-${INITRAMFS_IMAGE}.its "$deployDir/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its"
+			if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then
+				ln -snf fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its "$deployDir/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_LINK_NAME}"
+			fi
+		else
+			# deploy the artifacts of do_assemble_fitimage (fitImage without initramfs)
 			bbnote "Copying fit-image.its source file..."
 			install -m 0644 ${B}/fit-image.its "$deployDir/fitImage-its-${KERNEL_FIT_NAME}.its"
 			if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then
@@ -881,23 +924,5 @@ kernel_do_deploy:append() {
 				ln -snf fitImage-linux.bin-${KERNEL_FIT_NAME}${KERNEL_FIT_BIN_EXT} "$deployDir/fitImage-linux.bin-${KERNEL_FIT_LINK_NAME}"
 			fi
 		fi
-
-		if [ -n "${INITRAMFS_IMAGE}" ]; then
-			# deploy the artifacts of do_assemble_fitimage_initramfs for bundled as well as un-bundled mode
-			bbnote "Copying fit-image-${INITRAMFS_IMAGE}.its source file..."
-			install -m 0644 ${B}/fit-image-${INITRAMFS_IMAGE}.its "$deployDir/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its"
-			if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then
-				ln -snf fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}.its "$deployDir/fitImage-its-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_LINK_NAME}"
-			fi
-
-			# deploy the artifacts of do_assemble_fitimage_initramfs for bundled mode only
-			if [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
-				bbnote "Copying fitImage-${INITRAMFS_IMAGE} file..."
-				install -m 0644 ${B}/${KERNEL_OUTPUT_DIR}/fitImage-${INITRAMFS_IMAGE} "$deployDir/fitImage-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}${KERNEL_FIT_BIN_EXT}"
-				if [ -n "${KERNEL_FIT_LINK_NAME}" ] ; then
-					ln -snf fitImage-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_NAME}${KERNEL_FIT_BIN_EXT} "$deployDir/fitImage-${INITRAMFS_IMAGE_NAME}-${KERNEL_FIT_LINK_NAME}"
-				fi
-			fi
-		fi
 	fi
 }
meta/classes-recipe/kernel.bbclass +2 -1
index 6d242114852..3a3b2d40b2c 100644
@@ -484,7 +484,8 @@ kernel_do_install() {
 	# So, at the level of the install task we should not try to install the fitImage. fitImage is still not
 	# generated yet.
 	# After the generation of the fitImage, the deploy task copies the fitImage from the build directory to
-	# the deploy folder.
+	# the deploy folder. If INITRAMFS_IMAGE_BUNDLE != 1 the fitImage with initramfs is deployed after the
+	# deploy task.
 	#
 
 	for imageType in ${KERNEL_IMAGETYPES} ; do
Signature · 1 line
2.45.2
#17 lore raw

PATCH 13/14 image: depends do_deploy_fitimage_unbundled

In reply to Adrian Freihofer

From: Adrian Freihofer <[email protected]>

If a fitImage kernel with unbundled initramfs is built the final deploy task is now do_deploy_fitimage_unbundled.

---
 meta/classes-recipe/image.bbclass | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
meta/classes-recipe/image.bbclass +11 -1
index 28be6c63623..02a7952bcf9 100644
@@ -142,7 +142,17 @@ do_rootfs[vardeps] += "${@rootfs_variables(d)}"
 # This follows many common usecases and user expectations.
 # But if you are building an image which doesn't need the kernel image at all,
 # you can unset this variable manually.
-KERNEL_DEPLOY_DEPEND ?= "virtual/kernel:do_deploy"
+def get_kernel_do_deploy(d):
+    kerneltypes = d.getVar('KERNEL_IMAGETYPES') or ""
+    kerneltype = d.getVar('KERNEL_IMAGETYPE') or ""
+    fitimage = kerneltype == 'fitImage' or 'fitImage' in kerneltypes.split()
+    initramfs_image = d.getVar('INITRAMFS_IMAGE') or ""
+    bundled = d.getVar('INITRAMFS_IMAGE_BUNDLE') == '1'
+    if fitimage and initramfs_image and not bundled:
+        return "virtual/kernel:do_deploy_fitimage_unbundled"
+    return "virtual/kernel:do_deploy"
+
+KERNEL_DEPLOY_DEPEND ?= "${@get_kernel_do_deploy(d)}"
 do_build[depends] += "${KERNEL_DEPLOY_DEPEND}"
 
 
Signature · 1 line
2.45.2
#18 lore raw

PATCH 14/14 DO NOT MERGE: explanation and testing

In reply to Adrian Freihofer

From: Adrian Freihofer <[email protected]>

These patches are a bit complicated to explain.
This extra commit provides some additional documentation with some task graph diagrams in html format.

The html file was generated by the shell script which is also added to this commit. (Notes are added manually.)

---
 kernel-tasks.sh   |   76 +++
 task-depends.html | 1122 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 1198 insertions(+)
 create mode 100755 kernel-tasks.sh
 create mode 100644 task-depends.html
kernel-tasks.sh new file +76 -0
new file mode 100755
index 00000000000..b8b096f7d57
@@ -0,0 +1,76 @@
+#!/bin/sh
+
+OUTPUT_FILE=task-depends.md
+
+echo "# Task dependeny changes" > "$OUTPUT_FILE"
+echo "" >> "$OUTPUT_FILE"
+
+run_bitbake(){
+    echo "$1" >> "$OUTPUT_FILE"
+    echo "" >> "$OUTPUT_FILE"
+    echo '```plantuml' >> "$OUTPUT_FILE"
+
+    bitbake virtual/kernel -g
+    grep -E '(digraph depends|linux-yocto|\})' task-depends.dot \
+    | grep -v -E '(spdx|do_kernel_configcheck|do_prepare_recipe_sysroot|do_populate_lic|do_populate_sysroot|do_package|do_validate_branches|do_unpack|do_recipe_qa|do_patch|do_kernel_checkout|do_kernel_configme|do_kernel_metadata|do_sizecheck|do_strip|do_compile_kernelmodules|do_shared_workdir|do_transform_kernel|do_kernel_link_images|do_kernel_version_sanity_check|do_symlink_kernsrc|do_deploy_source_date_epoch|do_configure|do_fetch)' \
+    | sed 's;\\n\:.*.bb;;g' >> "$OUTPUT_FILE"
+    
+    #  > task-depends-$1.dot
+    # dot -T svg task-depends-$1.dot > task-depends-$1.svg
+
+    echo '```' >> "$OUTPUT_FILE"
+    echo "" >> "$OUTPUT_FILE"
+}
+
+run_bitbake_per_branch(){
+    echo "## Configuration: $1" >> "$OUTPUT_FILE"
+    echo "" >> "$OUTPUT_FILE"
+    echo '```raw' >> "$OUTPUT_FILE"
+    cat build/conf/auto.conf >> "$OUTPUT_FILE"
+    echo '```' >> "$OUTPUT_FILE"
+    echo "" >> "$OUTPUT_FILE"
+
+    for branch in master adrianf/kernel-fitimage-sstate; do
+        git checkout $branch
+        run_bitbake "### branch: $branch"
+    done
+}
+
+cat << EOF > build/conf/auto.conf
+KERNEL_IMAGETYPE = "Image"
+KERNEL_IMAGETYPES += " fitImage "
+KERNEL_CLASSES = " kernel-fitimage "
+IMAGE_FSTYPES += "cpio.gz"
+INITRAMFS_IMAGE = "core-image-minimal"
+EOF
+run_bitbake_per_branch "image, fitimage, initramfs, unbundled"
+
+
+cat << EOF > build/conf/auto.conf
+KERNEL_IMAGETYPE:forcevariable = "fitImage"
+KERNEL_CLASSES = " kernel-fitimage "
+IMAGE_FSTYPES += "cpio.gz"
+INITRAMFS_IMAGE = "core-image-minimal"
+EOF
+run_bitbake_per_branch "fitimage, initramfs, unbundled"
+
+
+cat << EOF > build/conf/auto.conf
+KERNEL_IMAGETYPE = "Image"
+KERNEL_IMAGETYPES += " fitImage "
+KERNEL_CLASSES = " kernel-fitimage "
+IMAGE_FSTYPES += "cpio.gz"
+INITRAMFS_IMAGE = "core-image-minimal"
+INITRAMFS_IMAGE_BUNDLE = "1"
+EOF
+run_bitbake_per_branch "image, fitimage, initramfs, bundled"
+
+
+cat << EOF > build/conf/auto.conf
+KERNEL_IMAGETYPE = "Image"
+KERNEL_IMAGETYPES += " fitImage "
+KERNEL_CLASSES = " kernel-fitimage "
+EOF
+run_bitbake_per_branch "image, fitimage"
+
+rm build/conf/auto.conf
task-depends.html new file +1122 -0

1,123 lines not on this page · raw message

Signature · 1 line
2.45.2
#19 lore raw
In reply to Adrian Freihofer
Quoted 20 lines · Adrian Freihofer via lists.openembedded.org

If the KERNEL_IMAGETYPES(S) contains fitImage, the kernel is always rebuilt when something changes in the initramfs, which happens frequently.
This is even worse if the build runs from an empty TMPDIR. The kernel re-build starts by fetching the large kernel git repository and recompiling it from scratch.

This cannot be improved if INITRAMFS_IMAGE_BUNDLE = "1". If the kernel Makefile is needed to generate the initramfs bundle the kernel build folder is required.
But for a build configuration with INITRAMFS_IMAGE_BUNDLE = "" the build folder is not needed. Creating the initramfs bundle requires:
linux.bin, DTBs and the initramfs which are available in the deploy directory. That means creating the fitImage can be done with artifacts which are already cached by the sstate.

There is an extra commit providing a html file. This html file provides some graphics which show the changes in the kernel's task dependencies. I hope this simplifies the re-view of this patch series.

Quoted 15 lines

Adrian Freihofer (14):
  oe-selftest: wic fix deps of test_efi_plugin_unified_kernel_image_qemu
  oe-selftest: fitimage refactor u-boot-tools-native
  oe-selftest: fitimage drop test-mkimage-wrapper
  oe-selftest: fitimage cleanup asserts
  oe-selftest: fitimage fix test_initramfs_bundle
  kernel-fitimage: fix handling of empty default dtb
  kernel-fitimage: fix external dtb check
  kernel: execute do_bundle_initramfs only when needed
  kernel-fitimage: refactor do_assemble_fitimage_initramfs
  kernel-fitimage: refactor creation of linux.bin
  kernel-fitimage: fix intentation
  kernel-fitimage: run unbundled fitimage after deploy
  image: depends do_deploy_fitimage_unbundled
  DO NOT MERGE: explanation and testing

I think there was a failure in automated testing from this series on genericarm64:

https://autobuilder.yoctoproject.org/typhoon/#/builders/165/builds/583
https://autobuilder.yoctoproject.org/typhoon/#/builders/166/builds/587

There was also:

https://autobuilder.yoctoproject.org/typhoon/#/builders/113/builds/5730

but I think this was systemd upgrade related. I'm not 100% sure though.

Cheers,

Richard

#20 lore raw
Quoted 45 lines · Richard Purdie
Quoted 26 lines · Adrian Freihofer via lists.openembedded.org

If the KERNEL_IMAGETYPES(S) contains fitImage, the kernel is always rebuilt when something changes in the initramfs, which happens frequently.
This is even worse if the build runs from an empty TMPDIR. The kernel
re-build starts by fetching the large kernel git repository and recompiling it from scratch.

This cannot be improved if INITRAMFS_IMAGE_BUNDLE = "1". If the kernel
Makefile is needed to generate the initramfs bundle the kernel build
folder is required.
But for a build configuration with INITRAMFS_IMAGE_BUNDLE = "" the build folder is not needed. Creating the initramfs bundle requires:
linux.bin, DTBs and the initramfs which are available in the deploy directory. That means creating the fitImage can be done with artifacts
which are already cached by the sstate.

There is an extra commit providing a html file. This html file provides
some graphics which show the changes in the kernel's task dependencies.
I hope this simplifies the re-view of this patch series.

Quoted 16 lines

Adrian Freihofer (14):
  oe-selftest: wic fix deps of test_efi_plugin_unified_kernel_image_qemu
  oe-selftest: fitimage refactor u-boot-tools-native
  oe-selftest: fitimage drop test-mkimage-wrapper
  oe-selftest: fitimage cleanup asserts
  oe-selftest: fitimage fix test_initramfs_bundle
  kernel-fitimage: fix handling of empty default dtb
  kernel-fitimage: fix external dtb check
  kernel: execute do_bundle_initramfs only when needed
  kernel-fitimage: refactor do_assemble_fitimage_initramfs
  kernel-fitimage: refactor creation of linux.bin
  kernel-fitimage: fix intentation
  kernel-fitimage: run unbundled fitimage after deploy
  image: depends do_deploy_fitimage_unbundled
  DO NOT MERGE: explanation and testing

Quoted 5 lines

I think there was a failure in automated testing from this series on genericarm64:

https://autobuilder.yoctoproject.org/typhoon/#/builders/165/builds/583
https://autobuilder.yoctoproject.org/typhoon/#/builders/166/builds/587

This is reproducible on my side. It works without my last patch but it breaks with my last patch.

Thank you for the feedback, I will investigate.
Adrian

Quoted 11 lines

There was also:

https://autobuilder.yoctoproject.org/typhoon/#/builders/113/builds/5730

but I think this was systemd upgrade related. I'm not 100% sure though.

Cheers,

Richard