cargo-update-recipe-crates: Make do_update_crates() a Python function
PATCH v2 1/5 cargo-update-recipe-crates: Make do_update_crates() a Python function
Port the do_update_crates() from Shell to a real Python function. This improves readability by directly executing Python code, instead of redirecting it through a string.
In the process, two new variables "target_file" and "cargo_lock_src_dir" are introduced, which just contain the previously, similarly named Bitbake variables in the Python function.
No functional changes are made.
---
Changes in v2:
- Split changes into multiple patches, for improved readability
--- .../cargo-update-recipe-crates.bbclass | 93 +++++++++---------- 1 file changed, 46 insertions(+), 47 deletions(-)
meta/classes-recipe/cargo-update-recipe-crates.bbclass +46 -47
@@ -23,59 +23,58 @@ RECIPE_UPGRADE_EXTRA_TASKS += "do_update_crates" # The directory where to search for Cargo.lock files CARGO_LOCK_SRC_DIR ??= "${S}" -do_update_crates() { - TARGET_FILE="${THISDIR}/${BPN}-crates.inc" +python do_update_crates() { + import tomllib - nativepython3 - <<EOF + target_file = d.expand('${THISDIR}/${BPN}-crates.inc') + cargo_lock_src_dir = d.getVar('CARGO_LOCK_SRC_DIR') -def get_crates(f): - import tomllib - c_list = '# from %s' % os.path.relpath(f, '${CARGO_LOCK_SRC_DIR}') - c_list += '\nSRC_URI += " \\\' - crates = tomllib.load(open(f, 'rb')) + def get_crates(f): + c_list = '# from %s' % os.path.relpath(f, cargo_lock_src_dir) + c_list += '\nSRC_URI += " \\' + crates = tomllib.load(open(f, 'rb')) - # Build a list with crates info that have crates.io in the source - crates_candidates = list(filter(lambda c: 'crates.io' in c.get('source', ''), crates['package'])) + # Build a list with crates info that have crates.io in the source + crates_candidates = list(filter(lambda c: 'crates.io' in c.get('source', ''), crates['package'])) - if not crates_candidates: - raise ValueError("Unable to find any candidate crates that use crates.io") + if not crates_candidates: + raise ValueError("Unable to find any candidate crates that use crates.io") - # Update crates uri and their checksum, to avoid name clashing on the checksum - # we need to rename crates with name and version to have a unique key - cksum_list = '' - for c in crates_candidates: - rename = "%s-%s" % (c['name'], c['version']) - c_list += '\n crate://crates.io/%s/%s \\\' % (c['name'], c['version']) - if 'checksum' in c: - cksum_list += '\nSRC_URI[%s.sha256sum] = "%s"' % (rename, c['checksum']) + # Update crates uri and their checksum, to avoid name clashing on the checksum + # we need to rename crates with name and version to have a unique key + cksum_list = '' + for c in crates_candidates: + rename = "%s-%s" % (c['name'], c['version']) + c_list += '\n crate://crates.io/%s/%s \\' % (c['name'], c['version']) + if 'checksum' in c: + cksum_list += '\nSRC_URI[%s.sha256sum] = "%s"' % (rename, c['checksum']) - c_list += '\n"\n' - c_list += cksum_list - c_list += '\n' - return c_list + c_list += '\n"\n' + c_list += cksum_list + c_list += '\n' + return c_list -import os -crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n" -found = False -for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'): - # ignore git and patches directories - if root.startswith(os.path.join('${CARGO_LOCK_SRC_DIR}', '.pc')): - continue - if root.startswith(os.path.join('${CARGO_LOCK_SRC_DIR}', '.git')): - continue - for file in files: - if file == 'Cargo.lock': - try: - cargo_lock_path = os.path.join(root, file) - crates += get_crates(os.path.join(root, file)) - except Exception as e: - raise ValueError("Cannot parse '%s'" % cargo_lock_path) from e - else: - found = True -if not found: - raise ValueError("Unable to find any Cargo.lock in ${CARGO_LOCK_SRC_DIR}") -open("${TARGET_FILE}", 'w').write(crates) -EOF + import os + crates = d.expand("# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n") + found = False + for root, dirs, files in os.walk(cargo_lock_src_dir): + # ignore git and patches directories + if root.startswith(os.path.join(cargo_lock_src_dir, '.pc')): + continue + if root.startswith(os.path.join(cargo_lock_src_dir, '.git')): + continue + for file in files: + if file == 'Cargo.lock': + try: + cargo_lock_path = os.path.join(root, file) + crates += get_crates(os.path.join(root, file)) + except Exception as e: + raise ValueError("Cannot parse '%s'" % cargo_lock_path) from e + else: + found = True + if not found: + raise ValueError(f"Unable to find any Cargo.lock in {cargo_lock_src_dir}") + open(target_file, 'w').write(crates) - bbnote "Successfully update crates inside '${TARGET_FILE}'" + bb.note(f"Successfully update crates inside '{target_file}'") }
Signature · 1 line
2.43.0
PATCH v2 2/5 cargo-update-recipe-crates: Don't fail for partially empty Cargo.lock
Do not fail if only some Cargo.lock files are empty. Only fail, if there are absolutely no dependencies found in any Cargo.lock.
This fixes the following error message, which would occur with "bitbake -c update_crates python3-orjson":
ERROR: python3-orjson-3.10.17-r0 do_update_crates: Execution of '.../python3-orjson/3.10.17/temp/run.do_update_crates.70693' failed with exit code 1
ERROR: Logfile of failure stored in: .../python3-orjson/3.10.17/temp/log.do_update_crates.70693
Log data follows:
| DEBUG: Executing python function extend_recipe_sysroot
| NOTE: Direct dependencies are ['.../sources/oe-core/../oe-core/meta/recipes-devtools/quilt/quilt-native_0.69.bb:do_populate_sysroot', 'virtual:native:.../sources/oe-core/../oe-core/meta/recipes-devtools/patch/patch_2.8.bb:do_populate_sysroot', 'virtual:native:.../sources/oe-core/../oe-core/meta/recipes-devtools/python/python3_3.13.9.bb:do_populate_sysroot'] | NOTE: Installed into sysroot: []
| NOTE: Skipping as already exists in sysroot: ['gettext-minimal-native', 'cmake-native', 'libtool-native', 'quilt-native', 'texinfo-dummy-native', 'openssl-native', 'expat-native', 'ncurses-native', 'util-linux-libuuid-native', 'zlib-native', 'libedit-native', 'make-native', 'patch-native', 'perl-native', 'python3-native', 'bzip2-native', 'xz-native', 'zstd-native', 'attr-native', 'gdbm-native', 'libffi-native', 'sqlite3-native'] | DEBUG: Python function extend_recipe_sysroot finished
| DEBUG: Executing shell function do_update_crates
| Traceback (most recent call last):
| File "<stdin>", line 41, in <module>
| File "<stdin>", line 12, in get_crates
| ValueError: Unable to find any candidate crates that use crates.io
|
| The above exception was the direct cause of the following exception:
|
| Traceback (most recent call last):
| File "<stdin>", line 43, in <module>
| ValueError: Cannot parse '.../python3-orjson/3.10.17/sources/orjson-3.10.17/include/cargo/simdutf8-0.1.5/Cargo.lock'
| WARNING: exit code 1 from a shell command.
ERROR: Task (.../sources/oe-core/../meta-openembedded/meta-python/recipes-devtools/python/python3-orjson_3.10.17.bb:do_update_crates) failed with exit code '1'
After this change, a working crates include file is produced and warning messages are output instead to inform about potential issues in the corresponding project:
WARNING: python3-orjson-3.10.17-r0 do_update_crates: Unable to find any candidate crates that use crates.io
WARNING: python3-orjson-3.10.17-r0 do_update_crates: Cannot parse '.../python3-orjson/3.10.17/sources/orjson-3.10.17/include/cargo/simdutf8-0.1.5/Cargo.lock'
---
Changes in v2:
- Split changes into multiple patches, for improved readability
--- .../cargo-update-recipe-crates.bbclass | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
meta/classes-recipe/cargo-update-recipe-crates.bbclass +9 -9
@@ -38,7 +38,8 @@ python do_update_crates() { crates_candidates = list(filter(lambda c: 'crates.io' in c.get('source', ''), crates['package'])) if not crates_candidates: - raise ValueError("Unable to find any candidate crates that use crates.io") + bb.warn('Unable to find any candidate crates that use crates.io') + return None # Update crates uri and their checksum, to avoid name clashing on the checksum # we need to rename crates with name and version to have a unique key
@@ -65,15 +66,14 @@ python do_update_crates() { continue for file in files: if file == 'Cargo.lock': - try: - cargo_lock_path = os.path.join(root, file) - crates += get_crates(os.path.join(root, file)) - except Exception as e: - raise ValueError("Cannot parse '%s'" % cargo_lock_path) from e + cargo_lock_path = os.path.join(root, file) + c = get_crates(cargo_lock_path) + if c is None: + bb.warn('Cannot parse \'%s\'' % cargo_lock_path) else: - found = True - if not found: - raise ValueError(f"Unable to find any Cargo.lock in {cargo_lock_src_dir}") + crates += c + if crates is None: + bb.error(f'Unable to find any Cargo.lock in {cargo_lock_src_dir}') open(target_file, 'w').write(crates) bb.note(f"Successfully update crates inside '{target_file}'")
Signature · 1 line
2.43.0
PATCH v2 3/5 cargo-update-recipe-crates: Consistently use single quotes in Python
In reply to Martin SchwanFor the Python function do_update_crates(), consistently use single quotes for Python strings. No functional changes.
---
Changes in v2:
- Split changes into multiple patches, for improved readability
--- meta/classes-recipe/cargo-update-recipe-crates.bbclass | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
meta/classes-recipe/cargo-update-recipe-crates.bbclass +3 -3
@@ -45,7 +45,7 @@ python do_update_crates() { # we need to rename crates with name and version to have a unique key cksum_list = '' for c in crates_candidates: - rename = "%s-%s" % (c['name'], c['version']) + rename = '%s-%s' % (c['name'], c['version']) c_list += '\n crate://crates.io/%s/%s \\' % (c['name'], c['version']) if 'checksum' in c: cksum_list += '\nSRC_URI[%s.sha256sum] = "%s"' % (rename, c['checksum'])
@@ -56,7 +56,7 @@ python do_update_crates() { return c_list import os - crates = d.expand("# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n") + crates = d.expand('# Autogenerated with \'bitbake -c update_crates ${PN}\'\n\n') found = False for root, dirs, files in os.walk(cargo_lock_src_dir): # ignore git and patches directories
@@ -76,5 +76,5 @@ python do_update_crates() { bb.error(f'Unable to find any Cargo.lock in {cargo_lock_src_dir}') open(target_file, 'w').write(crates) - bb.note(f"Successfully update crates inside '{target_file}'") + bb.note(f'Successfully update crates inside \'{target_file}\'') }
Signature · 1 line
2.43.0
RE: [OE-core] [PATCH v2 3/5] cargo-update-recipe-crates: Consistently use single quotes in Python
Quoted 31 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Martin Schwan via lists.openembedded.org
Sent: den 12 december 2025 10:19
To: [email protected]
Subject: [OE-core] [PATCH v2 3/5] cargo-update-recipe-crates: Consistently use single quotes in PythonFor the Python function
do_update_crates(), consistently use single quotes for Python strings. No functional changes.Signed-off-by Martin Schwan <[email protected]>---
Changes in v2:
- Split changes into multiple patches, for improved readability--- meta/classes-recipe/cargo-update-recipe-crates.bbclass | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)meta/classes-recipe/cargo-update-recipe-crates.bbclass +1
-1@@ -45,7 +45,7 @@ python do_update_crates() { # we need to rename crates with name and version to have a unique key cksum_list = '' for c in crates_candidates: - rename = "%s-%s" % (c['name'], c['version']) + rename = '%s-%s' % (c['name'], c['version']) c_list += '\n crate://crates.io/%s/%s \\' % (c['name'], c['version']) if 'checksum' in c: cksum_list += '\nSRC_URI[%s.sha256sum] = "%s"' % (rename, c['checksum'])
meta/classes-recipe/cargo-update-recipe-crates.bbclass python do_update_crates() { · line 59
Quoted 5 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1
-1@@ -56,7 +56,7 @@ python do_update_crates() { return c_list import os - crates = d.expand("# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n") + crates = d.expand('# Autogenerated with \'bitbake -c update_crates ${PN}\'\n\n')
IMHO, avoiding unnecessary backslashes is preferable to only using apostrophes to quote strings. So I would skip this and the one below.
Quoted 11 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1
-2found = False for root, dirs, files in os.walk(cargo_lock_src_dir): # ignore git and patches directories@@ -76,5 +76,5 @@ python do_update_crates() { bb.error(f'Unable to find any Cargo.lock in {cargo_lock_src_dir}') open(target_file, 'w').write(crates) - bb.note(f"Successfully update crates inside '{target_file}'") + bb.note(f'Successfully update crates inside \'{target_file}\'') } --
Quoted 1 line
2.43.0
//Peter
PATCH v2 4/5 cargo-update-recipe-crates: Properly close target file after writing
In reply to Martin SchwanIn do_update_crates(), properly close the target_file after writing to it using the "with" statement.
---
Changes in v2:
- Split changes into multiple patches, for improved readability
--- meta/classes-recipe/cargo-update-recipe-crates.bbclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/classes-recipe/cargo-update-recipe-crates.bbclass +2 -1
@@ -74,7 +74,8 @@ python do_update_crates() { crates += c if crates is None: bb.error(f'Unable to find any Cargo.lock in {cargo_lock_src_dir}') - open(target_file, 'w').write(crates) + with open(target_file, 'w') as f: + f.write(crates) bb.note(f'Successfully update crates inside \'{target_file}\'') }
Signature · 1 line
2.43.0
PATCH v2 5/5 cargo-update-recipe-crates: Do not import os
In reply to Martin SchwanThere is no need to import os in do_update_crates(), as this is already done before.
---
Changes in v2:
- Split changes into multiple patches, for improved readability
--- meta/classes-recipe/cargo-update-recipe-crates.bbclass | 1 - 1 file changed, 1 deletion(-)
meta/classes-recipe/cargo-update-recipe-crates.bbclass +0 -1
@@ -55,7 +55,6 @@ python do_update_crates() { c_list += '\n' return c_list - import os crates = d.expand('# Autogenerated with \'bitbake -c update_crates ${PN}\'\n\n') found = False for root, dirs, files in os.walk(cargo_lock_src_dir):
Signature · 1 line
2.43.0
meta/classes-recipe/cargo-update-recipe-crates.bbclass RECIPE_UPGRADE_EXTRA_TASKS += "do_update_crates" · line 27
Quoted 2 lines · Martin Schwan via lists.openembedded.org
meta/classes-recipe/cargo-update-recipe-crates.bbclass +2
-0+python do_update_crates() { + import tomllib
Sadly this is a problem right here.
tomllib is available since python 3.11, and the minimum python requirement for oe-core is 3.9, as seen in sanity.bbclass. Yocto CI builds are executed on older distibutions to check that dependencies on newer python features do not creep in. I suspect this is why this was and still should be piped into python3-native, since that is provided from a recipe in oe-core.
Alex
Quoted 10 lines · Alexander Kanavin
Quoted 2 lines · Martin Schwan via lists.openembedded.org
+python
do_update_crates(){
+ import tomllibSadly this is a problem right here.
tomllib is available since python 3.11, and the minimum python requirement for oe-core is 3.9, as seen in sanity.bbclass.
Okay, understood.
Are there any plans to move to Python 3.11? Python 3.9 is now at its EOL for some months. I don't want to open a can of worms, but just like to understand the future/intentions in this regard.
Quoted 5 lines
Yocto CI
builds are executed on older distibutions to check that dependencies on newer python features do not creep in. I suspect this is why this was and still should be piped into python3-native, since that is provided from a recipe in oe-core.
Alright, I'll revert that thing back then and pipe it to python3-native again.
Martin
Quoted 2 lines
Alex
Quoted 3 lines · Martin Schwan
Are there any plans to move to Python 3.11? Python 3.9 is now at its EOL for some months. I don't want to open a can of worms, but just like to understand the future/intentions in this regard.
There is no specific process or criteria for bumping the minimum python version, it happens every sometimes - you can probably dig through git history and read the commit messages for any such version increases. Doing that should not clash too much with what is in the set of supported host distributions, e.g. maybe one or two soon-obsolete distros may not have it, and so installing buildtools would be required for them going forward, but not more.
Note that sanity.bbclass is not the only place where python version is checked. There's also a check in meta/scripts/oe-buildenv-internal, and bitbake/lib/bb/__init__.py
Alex
Quoted 3 lines · Martin Schwan via lists.openembedded.org
Are there any plans to move to Python 3.11? Python 3.9 is now at its EOL for some months. I don't want to open a can of worms, but just like to understand the future/intentions in this regard.
I’ll happily endorse a patch to upgrade the minimal python version for master, but the following needs to happen:
For every supported distribution [1] where buildtools is not already required (which will provide a new python), what is the Python version?
There’s a script in yocto-autobuilder-helper to assist here, and it’s easier for me to do:
[1] 12:23:16 [SUCCESS] alma9-vk-2.yocto.io
Python 3.9.23
[2] 12:23:16 [SUCCESS] fedora41-vk-1.yocto.io
Python 3.13.3
[3] 12:23:16 [SUCCESS] fedora39-vk-1.yocto.io
Python 3.12.7
[4] 12:23:17 [SUCCESS] rocky8-vk-1.yocto.io
Python 3.6.8
[5] 12:23:18 [SUCCESS] fedora42-vk-1.yocto.io
Python 3.13.6
[6] 12:23:18 [SUCCESS] fedora40-vk-1.yocto.io
Python 3.12.10
[7] 12:23:18 [SUCCESS] opensuse155-vk-1.yocto.io
Python 3.6.15
[8] 12:23:18 [SUCCESS] ubuntu2504-vk-1.yocto.io
Python 3.13.3
[9] 12:23:19 [SUCCESS] ubuntu2404-vk-1.yocto.io
Python 3.12.3
[10] 12:23:19 [SUCCESS] rocky9-vk-1.yocto.io
Python 3.9.23
[11] 12:23:19 [SUCCESS] ubuntu2204-vk-1.yocto.io
Python 3.10.12
[12] 12:23:19 [SUCCESS] stream9-vk-1.yocto.io
Python 3.9.25
[13] 12:23:20 [SUCCESS] debian11-vk-1.yocto.io
Python 3.9.2
[14] 12:23:20 [SUCCESS] alma8-vk-1.yocto.io
Python 3.6.8
[15] 12:23:20 [SUCCESS] opensuse156-vk-1.yocto.io
Python 3.6.15
[16] 12:23:21 [SUCCESS] debian13-vk-1.yocto.io
Python 3.13.5
[17] 12:23:22 [SUCCESS] debian12-vk-1.yocto.io
Python 3.11.2
[18] 12:23:23 [SUCCESS] fedora43-vk-2.yocto.io
Python 3.14.0
[19] 12:23:23 [SUCCESS] ubuntu2510-vk-1.yocto.io
Python 3.13.7
The distros which have <3.9 need buildtools already, but that leaves Alma 9, Debian 11, Rocky 9 and Stream 9 which are all long-term support and won’t be EOLing shortly. Maybe something will happen that means we need to use buildtools on these, but tomllib is probably not going to be the impetus here.
Ross
[1] https://docs.yoctoproject.org/ref-manual/system-requirements.html#supported-linux-distributions
Thanks for the answer!
Martin
Quoted 67 lines · Ross Burton
Quoted 5 lines · Martin Schwan via lists.openembedded.org
Are there any plans to move to Python 3.11? Python 3.9 is now at its
EOL for some months. I don't want to open a can of worms, but just like
to understand the future/intentions in this regard.I’ll happily endorse a patch to upgrade the minimal python version for master, but the following needs to happen:
For every supported distribution [1] where buildtools is not already required (which will provide a new python), what is the Python version?
There’s a script in yocto-autobuilder-helper to assist here, and it’s easier for me to do:
[1] 12:23:16 [SUCCESS] alma9-vk-2.yocto.io
Python 3.9.23
[2] 12:23:16 [SUCCESS] fedora41-vk-1.yocto.io
Python 3.13.3
[3] 12:23:16 [SUCCESS] fedora39-vk-1.yocto.io
Python 3.12.7
[4] 12:23:17 [SUCCESS] rocky8-vk-1.yocto.io
Python 3.6.8
[5] 12:23:18 [SUCCESS] fedora42-vk-1.yocto.io
Python 3.13.6
[6] 12:23:18 [SUCCESS] fedora40-vk-1.yocto.io
Python 3.12.10
[7] 12:23:18 [SUCCESS] opensuse155-vk-1.yocto.io
Python 3.6.15
[8] 12:23:18 [SUCCESS] ubuntu2504-vk-1.yocto.io
Python 3.13.3
[9] 12:23:19 [SUCCESS] ubuntu2404-vk-1.yocto.io
Python 3.12.3
[10] 12:23:19 [SUCCESS] rocky9-vk-1.yocto.io
Python 3.9.23
[11] 12:23:19 [SUCCESS] ubuntu2204-vk-1.yocto.io
Python 3.10.12
[12] 12:23:19 [SUCCESS] stream9-vk-1.yocto.io
Python 3.9.25
[13] 12:23:20 [SUCCESS] debian11-vk-1.yocto.io
Python 3.9.2
[14] 12:23:20 [SUCCESS] alma8-vk-1.yocto.io
Python 3.6.8
[15] 12:23:20 [SUCCESS] opensuse156-vk-1.yocto.io
Python 3.6.15
[16] 12:23:21 [SUCCESS] debian13-vk-1.yocto.io
Python 3.13.5
[17] 12:23:22 [SUCCESS] debian12-vk-1.yocto.io
Python 3.11.2
[18] 12:23:23 [SUCCESS] fedora43-vk-2.yocto.io
Python 3.14.0
[19] 12:23:23 [SUCCESS] ubuntu2510-vk-1.yocto.io
Python 3.13.7The distros which have <3.9 need buildtools already, but that leaves Alma 9, Debian 11, Rocky 9 and Stream 9 which are all long-term support and won’t be EOLing shortly. Maybe something will happen that means we need to use buildtools on these, but tomllib is probably not going to be the impetus here.
Ross
[1]
https://docs.yoctoproject.org/ref-manual/system-requirements.html#supported-linux-distributions