Skip to contentSkip to conversation

Fix: allow specification of dtb directory

PATCH Superseded [email protected] 2 participants 7 messages May 12, 2023 – May 22, 2023

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Fix: allow specification of dtb directory

From: Randolph Sapp <[email protected]>

Fix a few of the oddities introduced by the previous patch that allowed specification of the DTB install directory and specification of vendor directories in that directory.

Add a recursive search for dtb and dtbo files to make sure upstream dtb directory standards don't break our packages. This uses a seperate function to avoid metadata changes as reported with the previous revision [1].

Makes the kernel-devicetree class posix compliant again by using BB variables more efficiently.

[1] https://lists.openembedded.org/g/openembedded-core/message/181157

Martin Jansa (1):
kernel-devicetree: make shell scripts posix compliant

Randolph Sapp (1):
kernel-devicetree: recursively search for dtbs

 meta/classes-recipe/kernel-devicetree.bbclass | 25 +++++++++++++------
 meta/classes-recipe/kernel.bbclass            |  2 +-
 2 files changed, 18 insertions(+), 9 deletions(-)
Signature · 1 line
2.40.1
#1 lore raw

PATCH v2 1/2 kernel-devicetree: make shell scripts posix compliant

From: Martin Jansa <[email protected]>

Use the notation suggested by Martin Jansa to avoid a bashism. Also switch KERNEL_DTBVENDORED to the more common 0/1 notation instead of true/false.

Fixes 04ab57d200 (kernel-devicetree: allow specification of dtb

directory, 2023-05-02)

---
 meta/classes-recipe/kernel-devicetree.bbclass | 4 ++--
 meta/classes-recipe/kernel.bbclass            | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)
meta/classes-recipe/kernel-devicetree.bbclass +2 -2
index a6c6c5f227..831fdd1527 100644
@@ -82,7 +82,7 @@ do_install:append() {
 	for dtbf in ${KERNEL_DEVICETREE}; do
 		dtb=`normalize_dtb "$dtbf"`
 		dtb_path=`get_real_dtb_path_in_kernel "$dtb"`
-		if [ ${KERNEL_DTBVENDORED} == "false" ]; then
+		if "${@'false' if oe.types.boolean(d.getVar('KERNEL_DTBVENDORED')) else 'true'}"; then
 			dtb_ext=${dtb##*.}
 			dtb_base_name=`basename $dtb .$dtb_ext`
 			dtb=$dtb_base_name.$dtb_ext
@@ -97,7 +97,7 @@ do_deploy:append() {
 		dtb_ext=${dtb##*.}
 		dtb_base_name=`basename $dtb .$dtb_ext`
 		install -d $deployDir
-		if [ ${KERNEL_DTBVENDORED} == "false" ]; then
+		if "${@'false' if oe.types.boolean(d.getVar('KERNEL_DTBVENDORED')) else 'true'}"; then
 			dtb=$dtb_base_name.$dtb_ext
 		fi
 		install -m 0644 ${D}/${KERNEL_DTBDEST}/$dtb $deployDir/$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext
meta/classes-recipe/kernel.bbclass +1 -1
index 8f022b234d..4bce64cf6a 100644
@@ -216,7 +216,7 @@ KERNEL_RELEASE ?= "${KERNEL_VERSION}"
 KERNEL_OUTPUT_DIR ?= "arch/${ARCH}/boot"
 KERNEL_IMAGEDEST ?= "boot"
 KERNEL_DTBDEST ?= "${KERNEL_IMAGEDEST}"
-KERNEL_DTBVENDORED ?= "false"
+KERNEL_DTBVENDORED ?= "0"
 
 #
 # configuration
Signature · 1 line
2.40.1
#2 lore raw

PATCH v2 2/2 kernel-devicetree: recursively search for dtbs

In reply to [email protected]

From: Randolph Sapp <[email protected]>

Upstream's dtb directory structure has no real standard. They just tend to idle around the 2/3 directory depth. Recursively search for the dtb/dtbo files instead of assuming anything.

Fixes 04ab57d200 (kernel-devicetree: allow specification of dtb

directory, 2023-05-02)

---
 meta/classes-recipe/kernel-devicetree.bbclass | 21 +++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)
meta/classes-recipe/kernel-devicetree.bbclass +15 -6
index 831fdd1527..fa6de7a140 100644
@@ -12,12 +12,21 @@ python () {
             d.appendVar("PACKAGES", " ${KERNEL_PACKAGE_NAME}-image-zimage-bundle")
 }
 
-FILES:${KERNEL_PACKAGE_NAME}-devicetree = " \
-    /${KERNEL_DTBDEST}/*.dtb \
-    /${KERNEL_DTBDEST}/*.dtbo \
-    /${KERNEL_DTBDEST}/*/*.dtb \
-    /${KERNEL_DTBDEST}/*/*.dtbo \
-"
+# recursivly search for devicetree files
+def find_device_trees(d):
+    import glob
+    import os
+
+    file_paths = []
+    dest_dir = d.getVar('D')
+    full_dtb_dir = dest_dir + '/' + d.getVar('KERNEL_DTBDEST')
+
+    for file in glob.glob(full_dtb_dir + '/**/*.dtb*', recursive=True):
+        file_paths.append('/' + os.path.relpath(file, dest_dir))
+
+    return ' '.join(file_paths)
+
+FILES:${KERNEL_PACKAGE_NAME}-devicetree = "${@find_device_trees(d)}"
 FILES:${KERNEL_PACKAGE_NAME}-image-zimage-bundle = "/${KERNEL_IMAGEDEST}/zImage-*.dtb.bin"
 
 # Generate kernel+devicetree bundle
Signature · 1 line
2.40.1
#3 lore raw

Re: [OE-core][PATCHv2 2/2] kernel-devicetree: recursively search for dtbs

Quoted 34 lines · Randolph Sapp via lists.openembedded.org

From: Randolph Sapp <[email protected]>

Upstream's dtb directory structure has no real standard. They just tend to idle around the 2/3 directory depth. Recursively search for the dtb/dtbo files instead of assuming anything.

Fixes 04ab57d200 (kernel-devicetree: allow specification of dtb

directory, 2023-05-02)

---
 meta/classes-recipe/kernel-devicetree.bbclass | 21 +++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)
meta/classes-recipe/kernel-devicetree.bbclass +6 -6
index 831fdd1527..fa6de7a140 100644
@@ -12,12 +12,21 @@ python () {
             d.appendVar("PACKAGES", " ${KERNEL_PACKAGE_NAME}-image-zimage-bundle")
 }
 
-FILES:${KERNEL_PACKAGE_NAME}-devicetree = " \
-    /${KERNEL_DTBDEST}/*.dtb \
-    /${KERNEL_DTBDEST}/*.dtbo \
-    /${KERNEL_DTBDEST}/*/*.dtb \
-    /${KERNEL_DTBDEST}/*/*.dtbo \
-"
+# recursivly search for devicetree files
+def find_device_trees(d):
+    import glob
+    import os
+
+    file_paths = []

meta/classes-recipe/kernel-devicetree.bbclass python () { · line 30

Quoted 10 lines
meta/classes-recipe/kernel-devicetree.bbclass +9 -0
+    dest_dir = d.getVar('D')
+    full_dtb_dir = dest_dir + '/' + d.getVar('KERNEL_DTBDEST')
+
+    for file in glob.glob(full_dtb_dir + '/**/*.dtb*', recursive=True):
+        file_paths.append('/' + os.path.relpath(file, dest_dir))
+
+    return ' '.join(file_paths)
+
+FILES:${KERNEL_PACKAGE_NAME}-devicetree = "${@find_device_trees(d)}"
 FILES:${KERNEL_PACKAGE_NAME}-image-zimage-bundle = "/${KERNEL_IMAGEDEST}/zImage-*.dtb.bin"

I suspect this still suffers from a determinism problem as the previous patch did but now it depends on whether the build had occurred or not.

I also really don't like resorting to python functions like this, they're horrible for performance overhead. I did wonder if:

FILES:${KERNEL_PACKAGE_NAME}-devicetree = "/${KERNEL_DTBDEST}/**/*.dtb*"

would work?

Also, are there automated tests we should be adding to catch bugs like this?

Cheers,

Richard

#4 lore raw

Re: [EXTERNAL] Re: [OE-core][PATCHv2 2/2] kernel-devicetree: recursively search for dtbs

Quoted 50 lines · Richard Purdie
Quoted 34 lines · Randolph Sapp via lists.openembedded.org

From: Randolph Sapp <[email protected]>

Upstream's dtb directory structure has no real standard. They just tend to idle around the 2/3 directory depth. Recursively search for the dtb/dtbo files instead of assuming anything.

Fixes 04ab57d200 (kernel-devicetree: allow specification of dtb

directory, 2023-05-02)

---
  meta/classes-recipe/kernel-devicetree.bbclass | 21 +++++++++++++------
  1 file changed, 15 insertions(+), 6 deletions(-)
meta/classes-recipe/kernel-devicetree.bbclass +6 -6
index 831fdd1527..fa6de7a140 100644
@@ -12,12 +12,21 @@ python () {
              d.appendVar("PACKAGES", " ${KERNEL_PACKAGE_NAME}-image-zimage-bundle")
  }
 
-FILES:${KERNEL_PACKAGE_NAME}-devicetree = " \
-    /${KERNEL_DTBDEST}/*.dtb \
-    /${KERNEL_DTBDEST}/*.dtbo \
-    /${KERNEL_DTBDEST}/*/*.dtb \
-    /${KERNEL_DTBDEST}/*/*.dtbo \
-"
+# recursivly search for devicetree files
+def find_device_trees(d):
+    import glob
+    import os
+
+    file_paths = []

meta/classes-recipe/kernel-devicetree.bbclass python () { · line 30

Quoted 10 lines
meta/classes-recipe/kernel-devicetree.bbclass +9 -0
+    dest_dir = d.getVar('D')
+    full_dtb_dir = dest_dir + '/' + d.getVar('KERNEL_DTBDEST')
+
+    for file in glob.glob(full_dtb_dir + '/**/*.dtb*', recursive=True):
+        file_paths.append('/' + os.path.relpath(file, dest_dir))
+
+    return ' '.join(file_paths)
+
+FILES:${KERNEL_PACKAGE_NAME}-devicetree = "${@find_device_trees(d)}"
  FILES:${KERNEL_PACKAGE_NAME}-image-zimage-bundle = "/${KERNEL_IMAGEDEST}/zImage-*.dtb.bin"

I suspect this still suffers from a determinism problem as the previous patch did but now it depends on whether the build had occurred or not.

Quoted 6 lines

I also really don't like resorting to python functions like this, they're horrible for performance overhead. I did wonder if:

FILES:${KERNEL_PACKAGE_NAME}-devicetree = "/${KERNEL_DTBDEST}/**/*.dtb*"

would work?

Tried that first. Unfortunately bitbake doesn't enable the recursive option in it's path globs. That would be too easy...

Quoted 3 lines

Also, are there automated tests we should be adding to catch bugs like this?

Unsure. I haven't looked into automated tests for yocto recipes yet. If you know of some docs for whatever OE-core is doing, feel free to shoot them my way.

Quoted 5 lines

Cheers,

Richard

#5 lore raw

Re: [EXTERNAL] Re: [OE-core][PATCHv2 2/2] kernel-devicetree: recursively search for dtbs

Quoted 54 lines · Randolph Sapp
Quoted 53 lines · Richard Purdie
Quoted 34 lines · Randolph Sapp via lists.openembedded.org

From: Randolph Sapp <[email protected]>

Upstream's dtb directory structure has no real standard. They just tend to idle around the 2/3 directory depth. Recursively search for the dtb/dtbo files instead of assuming anything.

Fixes 04ab57d200 (kernel-devicetree: allow specification of dtb

directory, 2023-05-02)

---
  meta/classes-recipe/kernel-devicetree.bbclass | 21 +++++++++++++------
  1 file changed, 15 insertions(+), 6 deletions(-)
meta/classes-recipe/kernel-devicetree.bbclass +6 -6
index 831fdd1527..fa6de7a140 100644
@@ -12,12 +12,21 @@ python () {
              d.appendVar("PACKAGES", " ${KERNEL_PACKAGE_NAME}-image-zimage-bundle")
  }
 
-FILES:${KERNEL_PACKAGE_NAME}-devicetree = " \
-    /${KERNEL_DTBDEST}/*.dtb \
-    /${KERNEL_DTBDEST}/*.dtbo \
-    /${KERNEL_DTBDEST}/*/*.dtb \
-    /${KERNEL_DTBDEST}/*/*.dtbo \
-"
+# recursivly search for devicetree files
+def find_device_trees(d):
+    import glob
+    import os
+
+    file_paths = []

meta/classes-recipe/kernel-devicetree.bbclass python () { · line 30

Quoted 10 lines
meta/classes-recipe/kernel-devicetree.bbclass +9 -0
+    dest_dir = d.getVar('D')
+    full_dtb_dir = dest_dir + '/' + d.getVar('KERNEL_DTBDEST')
+
+    for file in glob.glob(full_dtb_dir + '/**/*.dtb*', recursive=True):
+        file_paths.append('/' + os.path.relpath(file, dest_dir))
+
+    return ' '.join(file_paths)
+
+FILES:${KERNEL_PACKAGE_NAME}-devicetree = "${@find_device_trees(d)}"
  FILES:${KERNEL_PACKAGE_NAME}-image-zimage-bundle = "/${KERNEL_IMAGEDEST}/zImage-*.dtb.bin"

I suspect this still suffers from a determinism problem as the previous patch did but now it depends on whether the build had occurred or not.

I also really don't like resorting to python functions like this, they're horrible for performance overhead. I did wonder if:

Quoted 6 lines
Quoted 3 lines

FILES:${KERNEL_PACKAGE_NAME}-devicetree = "/${KERNEL_DTBDEST}/**/*.dtb*"

would work?

Tried that first. Unfortunately bitbake doesn't enable the recursive option in it's path globs. That would be too easy...

I wonder if we should do something like:

meta/lib/oe/package.py +1 -1
index edb70daaf17..70040f09e7c 100644
@@ -550,7 +550,7 @@ def files_from_filevars(filevars):
             f = '.' + f
         if not f.startswith("./"):
             f = './' + f
-        globbed = glob.glob(f)
+        globbed = glob.glob(f, recursive=True)
         if globbed:
             if [ f ] != globbed:
                 files += globbed

we now have the right minimum python version for that. My main concern would be potential any performance implications.

Quoted 6 lines
Quoted 2 lines

Also, are there automated tests we should be adding to catch bugs like this?

Unsure. I haven't looked into automated tests for yocto recipes yet. If you know of some docs for whatever OE-core is doing, feel free to shoot them my way.

oe-selftest -r <test-section-name>

where you can see the test cases in lib/oeqa/selftest/cases/

but I'm not seeing much for devicetree specifically. We should have some!

Cheers,

Richard

#6 lore raw

Re: [EXTERNAL] Re: [OE-core][PATCHv2 2/2] kernel-devicetree: recursively search for dtbs

Quoted 78 lines · Richard Purdie
Quoted 54 lines · Randolph Sapp
Quoted 53 lines · Richard Purdie
Quoted 34 lines · Randolph Sapp via lists.openembedded.org

From: Randolph Sapp <[email protected]>

Upstream's dtb directory structure has no real standard. They just tend to idle around the 2/3 directory depth. Recursively search for the dtb/dtbo files instead of assuming anything.

Fixes 04ab57d200 (kernel-devicetree: allow specification of dtb

directory, 2023-05-02)

---
   meta/classes-recipe/kernel-devicetree.bbclass | 21 +++++++++++++------
   1 file changed, 15 insertions(+), 6 deletions(-)
meta/classes-recipe/kernel-devicetree.bbclass +6 -6
index 831fdd1527..fa6de7a140 100644
@@ -12,12 +12,21 @@ python () {
               d.appendVar("PACKAGES", " ${KERNEL_PACKAGE_NAME}-image-zimage-bundle")
   }
 
-FILES:${KERNEL_PACKAGE_NAME}-devicetree = " \
-    /${KERNEL_DTBDEST}/*.dtb \
-    /${KERNEL_DTBDEST}/*.dtbo \
-    /${KERNEL_DTBDEST}/*/*.dtb \
-    /${KERNEL_DTBDEST}/*/*.dtbo \
-"
+# recursivly search for devicetree files
+def find_device_trees(d):
+    import glob
+    import os
+
+    file_paths = []

meta/classes-recipe/kernel-devicetree.bbclass python () { · line 30

Quoted 10 lines
meta/classes-recipe/kernel-devicetree.bbclass +9 -0
+    dest_dir = d.getVar('D')
+    full_dtb_dir = dest_dir + '/' + d.getVar('KERNEL_DTBDEST')
+
+    for file in glob.glob(full_dtb_dir + '/**/*.dtb*', recursive=True):
+        file_paths.append('/' + os.path.relpath(file, dest_dir))
+
+    return ' '.join(file_paths)
+
+FILES:${KERNEL_PACKAGE_NAME}-devicetree = "${@find_device_trees(d)}"
   FILES:${KERNEL_PACKAGE_NAME}-image-zimage-bundle = "/${KERNEL_IMAGEDEST}/zImage-*.dtb.bin"

I suspect this still suffers from a determinism problem as the previous patch did but now it depends on whether the build had occurred or not.

I also really don't like resorting to python functions like this, they're horrible for performance overhead. I did wonder if:

Quoted 6 lines
Quoted 3 lines

FILES:${KERNEL_PACKAGE_NAME}-devicetree = "/${KERNEL_DTBDEST}/**/*.dtb*"

would work?

Tried that first. Unfortunately bitbake doesn't enable the recursive option in it's path globs. That would be too easy...

I wonder if we should do something like:

meta/lib/oe/package.py +1 -1
index edb70daaf17..70040f09e7c 100644
@@ -550,7 +550,7 @@ def files_from_filevars(filevars):
              f = '.' + f
          if not f.startswith("./"):
              f = './' + f
-        globbed = glob.glob(f)
+        globbed = glob.glob(f, recursive=True)
          if globbed:
              if [ f ] != globbed:
                  files += globbed
Quoted 2 lines

we now have the right minimum python version for that. My main concern would be potential any performance implications.

This should not be an issue. Recursive=true just enables the interpretation of **, it *shouldn't* add extra recursion to any of the existing recipes... That being said, I've been wrong before.

Quoted 21 lines
Quoted 7 lines
Quoted 3 lines

Also, are there automated tests we should be adding to catch bugs like this?

Unsure. I haven't looked into automated tests for yocto recipes yet. If you know of some docs for whatever OE-core is doing, feel free to shoot them my way.

oe-selftest -r <test-section-name>

where you can see the test cases in lib/oeqa/selftest/cases/

but I'm not seeing much for devicetree specifically. We should have some!

Cheers,

Richard