cargo-update-recipe-crates: Don't fail for partially empty Cargo.lock
PATCH 1/1 cargo-update-recipe-crates: Don't fail for partially empty Cargo.lock
Do not fail if only some Cargo.lock files are empty. Only fail, if there are absolutely no dependencies found in any Cargo.lock.
This fixes the following error message, which would occur with "bitbake -c update_crates python3-orjson":
ERROR: python3-orjson-3.10.17-r0 do_update_crates: Execution of '.../python3-orjson/3.10.17/temp/run.do_update_crates.70693' failed with exit code 1
ERROR: Logfile of failure stored in: .../python3-orjson/3.10.17/temp/log.do_update_crates.70693
Log data follows:
| DEBUG: Executing python function extend_recipe_sysroot
| NOTE: Direct dependencies are ['.../sources/oe-core/../oe-core/meta/recipes-devtools/quilt/quilt-native_0.69.bb:do_populate_sysroot', 'virtual:native:.../sources/oe-core/../oe-core/meta/recipes-devtools/patch/patch_2.8.bb:do_populate_sysroot', 'virtual:native:.../sources/oe-core/../oe-core/meta/recipes-devtools/python/python3_3.13.9.bb:do_populate_sysroot'] | NOTE: Installed into sysroot: []
| NOTE: Skipping as already exists in sysroot: ['gettext-minimal-native', 'cmake-native', 'libtool-native', 'quilt-native', 'texinfo-dummy-native', 'openssl-native', 'expat-native', 'ncurses-native', 'util-linux-libuuid-native', 'zlib-native', 'libedit-native', 'make-native', 'patch-native', 'perl-native', 'python3-native', 'bzip2-native', 'xz-native', 'zstd-native', 'attr-native', 'gdbm-native', 'libffi-native', 'sqlite3-native'] | DEBUG: Python function extend_recipe_sysroot finished
| DEBUG: Executing shell function do_update_crates
| Traceback (most recent call last):
| File "<stdin>", line 41, in <module>
| File "<stdin>", line 12, in get_crates
| ValueError: Unable to find any candidate crates that use crates.io
|
| The above exception was the direct cause of the following exception:
|
| Traceback (most recent call last):
| File "<stdin>", line 43, in <module>
| ValueError: Cannot parse '.../python3-orjson/3.10.17/sources/orjson-3.10.17/include/cargo/simdutf8-0.1.5/Cargo.lock'
| WARNING: exit code 1 from a shell command.
ERROR: Task (.../sources/oe-core/../meta-openembedded/meta-python/recipes-devtools/python/python3-orjson_3.10.17.bb:do_update_crates) failed with exit code '1'
After this change, a working crates include file is produced and warning messages are output instead to inform about potential issues in the corresponding project:
WARNING: python3-orjson-3.10.17-r0 do_update_crates: Unable to find any candidate crates that use crates.io
WARNING: python3-orjson-3.10.17-r0 do_update_crates: Cannot parse '.../python3-orjson/3.10.17/sources/orjson-3.10.17/include/cargo/simdutf8-0.1.5/Cargo.lock'
While at it, restructure do_update_crates() to be a real Python function and use the corresponding code directly inline.
--- .../cargo-update-recipe-crates.bbclass | 91 +++++++++---------- 1 file changed, 45 insertions(+), 46 deletions(-)
meta/classes-recipe/cargo-update-recipe-crates.bbclass +45 -46
@@ -23,59 +23,58 @@ RECIPE_UPGRADE_EXTRA_TASKS += "do_update_crates" # The directory where to search for Cargo.lock files CARGO_LOCK_SRC_DIR ??= "${S}" -do_update_crates() { - TARGET_FILE="${THISDIR}/${BPN}-crates.inc" +python do_update_crates() { + import tomllib - nativepython3 - <<EOF + target_file = d.expand('${THISDIR}/${BPN}-crates.inc') + cargo_lock_src_dir = d.getVar('CARGO_LOCK_SRC_DIR') -def get_crates(f): - import tomllib - c_list = '# from %s' % os.path.relpath(f, '${CARGO_LOCK_SRC_DIR}') - c_list += '\nSRC_URI += " \\\' - crates = tomllib.load(open(f, 'rb')) + def get_crates(f): + c_list = '# from %s' % os.path.relpath(f, cargo_lock_src_dir) + c_list += '\nSRC_URI += " \\' + crates = tomllib.load(open(f, 'rb')) - # Build a list with crates info that have crates.io in the source - crates_candidates = list(filter(lambda c: 'crates.io' in c.get('source', ''), crates['package'])) + # Build a list with crates info that have crates.io in the source + crates_candidates = list(filter(lambda c: 'crates.io' in c.get('source', ''), crates['package'])) - if not crates_candidates: - raise ValueError("Unable to find any candidate crates that use crates.io") + if not crates_candidates: + bb.warn('Unable to find any candidate crates that use crates.io') + return None - # Update crates uri and their checksum, to avoid name clashing on the checksum - # we need to rename crates with name and version to have a unique key - cksum_list = '' - for c in crates_candidates: - rename = "%s-%s" % (c['name'], c['version']) - c_list += '\n crate://crates.io/%s/%s \\\' % (c['name'], c['version']) - if 'checksum' in c: - cksum_list += '\nSRC_URI[%s.sha256sum] = "%s"' % (rename, c['checksum']) + # Update crates uri and their checksum, to avoid name clashing on the checksum + # we need to rename crates with name and version to have a unique key + cksum_list = '' + for c in crates_candidates: + rename = "%s-%s" % (c['name'], c['version']) + c_list += '\n crate://crates.io/%s/%s \\' % (c['name'], c['version']) + if 'checksum' in c: + cksum_list += '\nSRC_URI[%s.sha256sum] = "%s"' % (rename, c['checksum']) - c_list += '\n"\n' - c_list += cksum_list - c_list += '\n' - return c_list + c_list += '\n"\n' + c_list += cksum_list + c_list += '\n' + return c_list -import os -crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n" -found = False -for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'): - # ignore git and patches directories - if root.startswith(os.path.join('${CARGO_LOCK_SRC_DIR}', '.pc')): - continue - if root.startswith(os.path.join('${CARGO_LOCK_SRC_DIR}', '.git')): - continue - for file in files: - if file == 'Cargo.lock': - try: + crates = d.expand('# Autogenerated with \'bitbake -c update_crates ${PN}\'\n\n') + for root, dirs, files in os.walk(cargo_lock_src_dir): + # ignore git and patches directories + if root.startswith(os.path.join(cargo_lock_src_dir, '.pc')): + continue + if root.startswith(os.path.join(cargo_lock_src_dir, '.git')): + continue + for file in files: + if file == 'Cargo.lock': cargo_lock_path = os.path.join(root, file) - crates += get_crates(os.path.join(root, file)) - except Exception as e: - raise ValueError("Cannot parse '%s'" % cargo_lock_path) from e - else: - found = True -if not found: - raise ValueError("Unable to find any Cargo.lock in ${CARGO_LOCK_SRC_DIR}") -open("${TARGET_FILE}", 'w').write(crates) -EOF + c = get_crates(cargo_lock_path) + if c is None: + bb.warn('Cannot parse \'%s\'' % cargo_lock_path) + else: + crates += c + if crates is None: + bb.error(f'Unable to find any Cargo.lock in {cargo_lock_src_dir}') + + with open(target_file, 'w') as f: + f.write(crates) - bbnote "Successfully update crates inside '${TARGET_FILE}'" + bb.note(f'Successfully updated crates inside \'{target_file}\'') }
Signature · 1 line
2.43.0
Quoted 3 lines · Martin Schwan via lists.openembedded.org
Do not fail if only some Cargo.lock files are empty. Only fail, if there are absolutely no dependencies found in any Cargo.lock.
...
Quoted 2 lines
While at it, restructure
do_update_crates()to be a real Python function and use the corresponding code directly inline.
This, as a diff on the mailing list, is incredibly hard to review, and needs to be split up. Refactoring in one commit (explaining the benefits), functional changes in another please. Even better if you can split things further, and explain what the particular modifications are (rather than use terms like 'restructure').
Aelx
Quoted 14 lines · Alexander Kanavin
Quoted 4 lines · Martin Schwan via lists.openembedded.org
Do not fail if only some Cargo.lock files are empty. Only fail, if there
are absolutely no dependencies found in any Cargo.lock....
Quoted 3 lines
While at it, restructure
do_update_crates()to be a real Python function
and use the corresponding code directly inline.
Quoted 6 lines
This, as a diff on the mailing list, is incredibly hard to review, and
needs to be split up. Refactoring in one commit (explaining the benefits), functional changes in another please. Even better if you can split things further, and explain what the particular modifications are (rather than use terms like 'restructure').
Thanks for the feedback. Admittedly, I was a bit lazy with this one, as the changes were quite interwoven. I'll send a v2 and split things up, as best as I can.
Regards,
Martin
Quoted 2 lines
Aelx
Quoted 2 lines
Do not fail if only some Cargo.lock files are empty. Only fail, if there are absolutely no dependencies found in any Cargo.lock.
meta-security's suricata also has this problem.
I currently use a local patch to change the exception to a warning.
See https://lists.yoctoproject.org/g/yocto-patches/message/2577
--
Clayton Casciato