Fix: allow specification of dtb directory
v2 cover Fix: allow specification of dtb directory
From: Randolph Sapp <[email protected]>
Fix a few of the oddities introduced by the previous patch that allowed specification of the DTB install directory and specification of vendor directories in that directory.
Add a recursive search for dtb and dtbo files to make sure upstream dtb directory standards don't break our packages. This uses a seperate function to avoid metadata changes as reported with the previous revision [1].
Makes the kernel-devicetree class posix compliant again by using BB variables more efficiently.
[1] https://lists.openembedded.org/g/openembedded-core/message/181157
Martin Jansa (1):
kernel-devicetree: make shell scripts posix compliant
Randolph Sapp (1):
kernel-devicetree: recursively search for dtbs
meta/classes-recipe/kernel-devicetree.bbclass | 25 +++++++++++++------ meta/classes-recipe/kernel.bbclass | 2 +- 2 files changed, 18 insertions(+), 9 deletions(-)
Signature · 1 line
2.40.1
PATCH v2 1/2 kernel-devicetree: make shell scripts posix compliant
From: Martin Jansa <[email protected]>
Use the notation suggested by Martin Jansa to avoid a bashism. Also switch KERNEL_DTBVENDORED to the more common 0/1 notation instead of true/false.
directory, 2023-05-02)
--- meta/classes-recipe/kernel-devicetree.bbclass | 4 ++-- meta/classes-recipe/kernel.bbclass | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
meta/classes-recipe/kernel-devicetree.bbclass +2 -2
@@ -82,7 +82,7 @@ do_install:append() { for dtbf in ${KERNEL_DEVICETREE}; do dtb=`normalize_dtb "$dtbf"` dtb_path=`get_real_dtb_path_in_kernel "$dtb"` - if [ ${KERNEL_DTBVENDORED} == "false" ]; then + if "${@'false' if oe.types.boolean(d.getVar('KERNEL_DTBVENDORED')) else 'true'}"; then dtb_ext=${dtb##*.} dtb_base_name=`basename $dtb .$dtb_ext` dtb=$dtb_base_name.$dtb_ext
@@ -97,7 +97,7 @@ do_deploy:append() { dtb_ext=${dtb##*.} dtb_base_name=`basename $dtb .$dtb_ext` install -d $deployDir - if [ ${KERNEL_DTBVENDORED} == "false" ]; then + if "${@'false' if oe.types.boolean(d.getVar('KERNEL_DTBVENDORED')) else 'true'}"; then dtb=$dtb_base_name.$dtb_ext fi install -m 0644 ${D}/${KERNEL_DTBDEST}/$dtb $deployDir/$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext
meta/classes-recipe/kernel.bbclass +1 -1
@@ -216,7 +216,7 @@ KERNEL_RELEASE ?= "${KERNEL_VERSION}" KERNEL_OUTPUT_DIR ?= "arch/${ARCH}/boot" KERNEL_IMAGEDEST ?= "boot" KERNEL_DTBDEST ?= "${KERNEL_IMAGEDEST}" -KERNEL_DTBVENDORED ?= "false" +KERNEL_DTBVENDORED ?= "0" # # configuration
Signature · 1 line
2.40.1
PATCH v2 2/2 kernel-devicetree: recursively search for dtbs
In reply to [email protected]From: Randolph Sapp <[email protected]>
Upstream's dtb directory structure has no real standard. They just tend to idle around the 2/3 directory depth. Recursively search for the dtb/dtbo files instead of assuming anything.
directory, 2023-05-02)
--- meta/classes-recipe/kernel-devicetree.bbclass | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)
meta/classes-recipe/kernel-devicetree.bbclass +15 -6
@@ -12,12 +12,21 @@ python () { d.appendVar("PACKAGES", " ${KERNEL_PACKAGE_NAME}-image-zimage-bundle") } -FILES:${KERNEL_PACKAGE_NAME}-devicetree = " \ - /${KERNEL_DTBDEST}/*.dtb \ - /${KERNEL_DTBDEST}/*.dtbo \ - /${KERNEL_DTBDEST}/*/*.dtb \ - /${KERNEL_DTBDEST}/*/*.dtbo \ -" +# recursivly search for devicetree files +def find_device_trees(d): + import glob + import os + + file_paths = [] + dest_dir = d.getVar('D') + full_dtb_dir = dest_dir + '/' + d.getVar('KERNEL_DTBDEST') + + for file in glob.glob(full_dtb_dir + '/**/*.dtb*', recursive=True): + file_paths.append('/' + os.path.relpath(file, dest_dir)) + + return ' '.join(file_paths) + +FILES:${KERNEL_PACKAGE_NAME}-devicetree = "${@find_device_trees(d)}" FILES:${KERNEL_PACKAGE_NAME}-image-zimage-bundle = "/${KERNEL_IMAGEDEST}/zImage-*.dtb.bin" # Generate kernel+devicetree bundle
Signature · 1 line
2.40.1
Re: [OE-core][PATCHv2 2/2] kernel-devicetree: recursively search for dtbs
Quoted 34 lines · Randolph Sapp via lists.openembedded.org
From: Randolph Sapp <[email protected]>
Upstream's dtb directory structure has no real standard. They just tend to idle around the 2/3 directory depth. Recursively search for the dtb/dtbo files instead of assuming anything.
Fixes 04ab57d200 (kernel-devicetree: allow specification of dtbdirectory, 2023-05-02)
Signed-off-by Randolph Sapp <[email protected]>--- meta/classes-recipe/kernel-devicetree.bbclass | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)meta/classes-recipe/kernel-devicetree.bbclass +6
-6@@ -12,12 +12,21 @@ python () { d.appendVar("PACKAGES", " ${KERNEL_PACKAGE_NAME}-image-zimage-bundle") } -FILES:${KERNEL_PACKAGE_NAME}-devicetree = " \ - /${KERNEL_DTBDEST}/*.dtb \ - /${KERNEL_DTBDEST}/*.dtbo \ - /${KERNEL_DTBDEST}/*/*.dtb \ - /${KERNEL_DTBDEST}/*/*.dtbo \ -" +# recursivly search for devicetree files +def find_device_trees(d): + import glob + import os + + file_paths = []
meta/classes-recipe/kernel-devicetree.bbclass python () { · line 30
Quoted 10 lines
meta/classes-recipe/kernel-devicetree.bbclass +9
-0+ dest_dir = d.getVar('D') + full_dtb_dir = dest_dir + '/' + d.getVar('KERNEL_DTBDEST') + + for file in glob.glob(full_dtb_dir + '/**/*.dtb*', recursive=True): + file_paths.append('/' + os.path.relpath(file, dest_dir)) + + return ' '.join(file_paths) + +FILES:${KERNEL_PACKAGE_NAME}-devicetree = "${@find_device_trees(d)}" FILES:${KERNEL_PACKAGE_NAME}-image-zimage-bundle = "/${KERNEL_IMAGEDEST}/zImage-*.dtb.bin"
I suspect this still suffers from a determinism problem as the previous patch did but now it depends on whether the build had occurred or not.
I also really don't like resorting to python functions like this, they're horrible for performance overhead. I did wonder if:
FILES:${KERNEL_PACKAGE_NAME}-devicetree = "/${KERNEL_DTBDEST}/**/*.dtb*"
would work?
Also, are there automated tests we should be adding to catch bugs like this?
Cheers,
Richard
Re: [EXTERNAL] Re: [OE-core][PATCHv2 2/2] kernel-devicetree: recursively search for dtbs
Quoted 50 lines · Richard Purdie
Quoted 34 lines · Randolph Sapp via lists.openembedded.org
From: Randolph Sapp <[email protected]>
Upstream's dtb directory structure has no real standard. They just tend to idle around the 2/3 directory depth. Recursively search for the dtb/dtbo files instead of assuming anything.
Fixes 04ab57d200 (kernel-devicetree: allow specification of dtbdirectory, 2023-05-02)
Signed-off-by Randolph Sapp <[email protected]>--- meta/classes-recipe/kernel-devicetree.bbclass | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)meta/classes-recipe/kernel-devicetree.bbclass +6
-6@@ -12,12 +12,21 @@ python () { d.appendVar("PACKAGES", " ${KERNEL_PACKAGE_NAME}-image-zimage-bundle") } -FILES:${KERNEL_PACKAGE_NAME}-devicetree = " \ - /${KERNEL_DTBDEST}/*.dtb \ - /${KERNEL_DTBDEST}/*.dtbo \ - /${KERNEL_DTBDEST}/*/*.dtb \ - /${KERNEL_DTBDEST}/*/*.dtbo \ -" +# recursivly search for devicetree files +def find_device_trees(d): + import glob + import os + + file_paths = []meta/classes-recipe/kernel-devicetree.bbclass python () { · line 30
Quoted 10 lines
meta/classes-recipe/kernel-devicetree.bbclass +9
-0+ dest_dir = d.getVar('D') + full_dtb_dir = dest_dir + '/' + d.getVar('KERNEL_DTBDEST') + + for file in glob.glob(full_dtb_dir + '/**/*.dtb*', recursive=True): + file_paths.append('/' + os.path.relpath(file, dest_dir)) + + return ' '.join(file_paths) + +FILES:${KERNEL_PACKAGE_NAME}-devicetree = "${@find_device_trees(d)}" FILES:${KERNEL_PACKAGE_NAME}-image-zimage-bundle = "/${KERNEL_IMAGEDEST}/zImage-*.dtb.bin"I suspect this still suffers from a determinism problem as the previous patch did but now it depends on whether the build had occurred or not.
Quoted 6 lines
I also really don't like resorting to python functions like this, they're horrible for performance overhead. I did wonder if:
FILES:${KERNEL_PACKAGE_NAME}-devicetree = "/${KERNEL_DTBDEST}/**/*.dtb*"
would work?
Tried that first. Unfortunately bitbake doesn't enable the recursive option in it's path globs. That would be too easy...
Quoted 3 lines
Also, are there automated tests we should be adding to catch bugs like this?
Unsure. I haven't looked into automated tests for yocto recipes yet. If you know of some docs for whatever OE-core is doing, feel free to shoot them my way.
Quoted 5 lines
Cheers,
Richard
Re: [EXTERNAL] Re: [OE-core][PATCHv2 2/2] kernel-devicetree: recursively search for dtbs
Quoted 54 lines · Randolph Sapp
Quoted 53 lines · Richard Purdie
Quoted 34 lines · Randolph Sapp via lists.openembedded.org
From: Randolph Sapp <[email protected]>
Upstream's dtb directory structure has no real standard. They just tend to idle around the 2/3 directory depth. Recursively search for the dtb/dtbo files instead of assuming anything.
Fixes 04ab57d200 (kernel-devicetree: allow specification of dtbdirectory, 2023-05-02)
Signed-off-by Randolph Sapp <[email protected]>--- meta/classes-recipe/kernel-devicetree.bbclass | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)meta/classes-recipe/kernel-devicetree.bbclass +6
-6@@ -12,12 +12,21 @@ python () { d.appendVar("PACKAGES", " ${KERNEL_PACKAGE_NAME}-image-zimage-bundle") } -FILES:${KERNEL_PACKAGE_NAME}-devicetree = " \ - /${KERNEL_DTBDEST}/*.dtb \ - /${KERNEL_DTBDEST}/*.dtbo \ - /${KERNEL_DTBDEST}/*/*.dtb \ - /${KERNEL_DTBDEST}/*/*.dtbo \ -" +# recursivly search for devicetree files +def find_device_trees(d): + import glob + import os + + file_paths = []meta/classes-recipe/kernel-devicetree.bbclass python () { · line 30
Quoted 10 lines
meta/classes-recipe/kernel-devicetree.bbclass +9
-0+ dest_dir = d.getVar('D') + full_dtb_dir = dest_dir + '/' + d.getVar('KERNEL_DTBDEST') + + for file in glob.glob(full_dtb_dir + '/**/*.dtb*', recursive=True): + file_paths.append('/' + os.path.relpath(file, dest_dir)) + + return ' '.join(file_paths) + +FILES:${KERNEL_PACKAGE_NAME}-devicetree = "${@find_device_trees(d)}" FILES:${KERNEL_PACKAGE_NAME}-image-zimage-bundle = "/${KERNEL_IMAGEDEST}/zImage-*.dtb.bin"I suspect this still suffers from a determinism problem as the previous patch did but now it depends on whether the build had occurred or not.
I also really don't like resorting to python functions like this, they're horrible for performance overhead. I did wonder if:
Quoted 6 lines
Quoted 3 lines
FILES:${KERNEL_PACKAGE_NAME}-devicetree = "/${KERNEL_DTBDEST}/**/*.dtb*"
would work?
Tried that first. Unfortunately bitbake doesn't enable the recursive option in it's path globs. That would be too easy...
I wonder if we should do something like:
meta/lib/oe/package.py +1 -1
@@ -550,7 +550,7 @@ def files_from_filevars(filevars): f = '.' + f if not f.startswith("./"): f = './' + f - globbed = glob.glob(f) + globbed = glob.glob(f, recursive=True) if globbed: if [ f ] != globbed: files += globbed
we now have the right minimum python version for that. My main concern would be potential any performance implications.
Quoted 6 lines
Quoted 2 lines
Also, are there automated tests we should be adding to catch bugs like this?
Unsure. I haven't looked into automated tests for yocto recipes yet. If you know of some docs for whatever OE-core is doing, feel free to shoot them my way.
oe-selftest -r <test-section-name>
where you can see the test cases in lib/oeqa/selftest/cases/
but I'm not seeing much for devicetree specifically. We should have some!
Cheers,
Richard
Re: [EXTERNAL] Re: [OE-core][PATCHv2 2/2] kernel-devicetree: recursively search for dtbs
Quoted 78 lines · Richard Purdie
Quoted 54 lines · Randolph Sapp
Quoted 53 lines · Richard Purdie
Quoted 34 lines · Randolph Sapp via lists.openembedded.org
From: Randolph Sapp <[email protected]>
Upstream's dtb directory structure has no real standard. They just tend to idle around the 2/3 directory depth. Recursively search for the dtb/dtbo files instead of assuming anything.
Fixes 04ab57d200 (kernel-devicetree: allow specification of dtbdirectory, 2023-05-02)
Signed-off-by Randolph Sapp <[email protected]>--- meta/classes-recipe/kernel-devicetree.bbclass | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)meta/classes-recipe/kernel-devicetree.bbclass +6
-6@@ -12,12 +12,21 @@ python () { d.appendVar("PACKAGES", " ${KERNEL_PACKAGE_NAME}-image-zimage-bundle") } -FILES:${KERNEL_PACKAGE_NAME}-devicetree = " \ - /${KERNEL_DTBDEST}/*.dtb \ - /${KERNEL_DTBDEST}/*.dtbo \ - /${KERNEL_DTBDEST}/*/*.dtb \ - /${KERNEL_DTBDEST}/*/*.dtbo \ -" +# recursivly search for devicetree files +def find_device_trees(d): + import glob + import os + + file_paths = []meta/classes-recipe/kernel-devicetree.bbclass python () { · line 30
Quoted 10 lines
meta/classes-recipe/kernel-devicetree.bbclass +9
-0+ dest_dir = d.getVar('D') + full_dtb_dir = dest_dir + '/' + d.getVar('KERNEL_DTBDEST') + + for file in glob.glob(full_dtb_dir + '/**/*.dtb*', recursive=True): + file_paths.append('/' + os.path.relpath(file, dest_dir)) + + return ' '.join(file_paths) + +FILES:${KERNEL_PACKAGE_NAME}-devicetree = "${@find_device_trees(d)}" FILES:${KERNEL_PACKAGE_NAME}-image-zimage-bundle = "/${KERNEL_IMAGEDEST}/zImage-*.dtb.bin"I suspect this still suffers from a determinism problem as the previous patch did but now it depends on whether the build had occurred or not.
I also really don't like resorting to python functions like this, they're horrible for performance overhead. I did wonder if:
Quoted 6 lines
Quoted 3 lines
FILES:${KERNEL_PACKAGE_NAME}-devicetree = "/${KERNEL_DTBDEST}/**/*.dtb*"
would work?
Tried that first. Unfortunately bitbake doesn't enable the recursive option in it's path globs. That would be too easy...
I wonder if we should do something like:
meta/lib/oe/package.py +1
-1@@ -550,7 +550,7 @@ def files_from_filevars(filevars): f = '.' + f if not f.startswith("./"): f = './' + f - globbed = glob.glob(f) + globbed = glob.glob(f, recursive=True) if globbed: if [ f ] != globbed: files += globbed
Quoted 2 lines
we now have the right minimum python version for that. My main concern would be potential any performance implications.
This should not be an issue. Recursive=true just enables the interpretation of **, it *shouldn't* add extra recursion to any of the existing recipes... That being said, I've been wrong before.
Quoted 21 lines
Quoted 7 lines
Quoted 3 lines
Also, are there automated tests we should be adding to catch bugs like this?
Unsure. I haven't looked into automated tests for yocto recipes yet. If you know of some docs for whatever OE-core is doing, feel free to shoot them my way.
oe-selftest -r <test-section-name>
where you can see the test cases in
lib/oeqa/selftest/cases/but I'm not seeing much for devicetree specifically. We should have some!
Cheers,
Richard