cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock
RFC PATCH 1/2 cargo-update-recipe-crates.bbclass: add a class to generate SRC_URI crate lists from Cargo.lock
For better or worse, more and more rust components are appearing that do not include their dependencies in tarballs (or git trees), and rely on cargo to fetch them. On the other hand, bitbake does not use cargo (and quite possible won't ever be able to), and relies on having each item explicitly listed in SRC_URI with a crate:// prefix. This however creates a problem of both making such lists in the first place and updating them when a recipe is updated to a newer version.
So this class can be used to perform such updates by implementing a task that does it; the next commit shows the outcome for python3-bcrypt (which has been tested to work and produce a successful build).
Note: the python script relies on tomllib library, which appears in Python 3.11 and does not exist in earlier versions - I've tested this by first updating python to 3.11-rc2 in oe-core.
--- .../cargo-update-recipe-crates.bbclass | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 meta/classes-recipe/cargo-update-recipe-crates.bbclass
meta/classes-recipe/cargo-update-recipe-crates.bbclass new file +41 -0
@@ -0,0 +1,41 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +## +## Purpose: +## This class is used to update the list of crates in SRC_URI +## by reading Cargo.lock in the source tree. +## +## See meta/recipes-devtools/python/python3-bcrypt_*.bb for an example +## +## To perform the update: bitbake -c update_crates recipe-name + +addtask do_update_crates after do_patch +do_update_crates[depends] = "python3-native:do_populate_sysroot" + +do_update_crates() { + nativepython3 - <<EOF + +def get_crates(f): + import tomllib + c_list = 'SRC_URI += " \\ \n' + crates = tomllib.load(open(f, 'rb')) + for c in crates['package']: + if 'source' in c and 'crates.io' in c['source']: + c_list += " crate://crates.io/{}/{} \\ \n".format(c['name'], c['version']) + c_list += '"\n' + return c_list + +import os +crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n" +for root, dirs, files in os.walk('${S}'): + for file in files: + if file == 'Cargo.lock': + crates += get_crates(os.path.join(root, file)) +open(os.path.join('${THISDIR}', '${PN}'+"-crates.inc"), 'w').write(crates) + +EOF +}
Signature · 1 line
2.30.2
RFC PATCH 2/2 python3-bcrypt: update 3.2.2 -> 4.0.0
The component has been reimplemented in rust, and comes with a large list of dependencies in Cargo.toml/Cargo.lock.
Rather than list them by hand, use a file generated with cargo-update-recipe-crates class.
---
.../python/python3-bcrypt-crates.inc | 53 +++++++++++++++++++
...crypt_3.2.2.bb => python3-bcrypt_4.0.0.bb} | 6 ++-
2 files changed, 57 insertions(+), 2 deletions(-)
create mode 100644 meta/recipes-devtools/python/python3-bcrypt-crates.inc
rename meta/recipes-devtools/python/{python3-bcrypt_3.2.2.bb => python3-bcrypt_4.0.0.bb} (76%)meta/recipes-devtools/python/python3-bcrypt-crates.inc new file +53 -0
@@ -0,0 +1,53 @@ +# Autogenerated with 'bitbake -c update_crates python3-bcrypt' + +SRC_URI += " \ + crate://crates.io/autocfg/1.1.0 \ + crate://crates.io/base64/0.13.0 \ + crate://crates.io/bcrypt/0.13.0 \ + crate://crates.io/bcrypt-pbkdf/0.8.1 \ + crate://crates.io/bitflags/1.3.2 \ + crate://crates.io/block-buffer/0.10.2 \ + crate://crates.io/blowfish/0.9.1 \ + crate://crates.io/byteorder/1.4.3 \ + crate://crates.io/cfg-if/1.0.0 \ + crate://crates.io/cipher/0.4.3 \ + crate://crates.io/cpufeatures/0.2.4 \ + crate://crates.io/crypto-common/0.1.6 \ + crate://crates.io/digest/0.10.3 \ + crate://crates.io/generic-array/0.14.6 \ + crate://crates.io/getrandom/0.2.7 \ + crate://crates.io/indoc/0.3.6 \ + crate://crates.io/indoc-impl/0.3.6 \ + crate://crates.io/inout/0.1.3 \ + crate://crates.io/instant/0.1.12 \ + crate://crates.io/libc/0.2.132 \ + crate://crates.io/lock_api/0.4.7 \ + crate://crates.io/once_cell/1.13.1 \ + crate://crates.io/parking_lot/0.11.2 \ + crate://crates.io/parking_lot_core/0.8.5 \ + crate://crates.io/paste/0.1.18 \ + crate://crates.io/paste-impl/0.1.18 \ + crate://crates.io/pbkdf2/0.10.1 \ + crate://crates.io/proc-macro-hack/0.5.19 \ + crate://crates.io/proc-macro2/1.0.43 \ + crate://crates.io/pyo3/0.15.2 \ + crate://crates.io/pyo3-build-config/0.15.2 \ + crate://crates.io/pyo3-macros/0.15.2 \ + crate://crates.io/pyo3-macros-backend/0.15.2 \ + crate://crates.io/quote/1.0.21 \ + crate://crates.io/redox_syscall/0.2.16 \ + crate://crates.io/scopeguard/1.1.0 \ + crate://crates.io/sha2/0.10.2 \ + crate://crates.io/smallvec/1.9.0 \ + crate://crates.io/subtle/2.4.1 \ + crate://crates.io/syn/1.0.99 \ + crate://crates.io/typenum/1.15.0 \ + crate://crates.io/unicode-ident/1.0.3 \ + crate://crates.io/unindent/0.1.10 \ + crate://crates.io/version_check/0.9.4 \ + crate://crates.io/wasi/0.11.0+wasi-snapshot-preview1 \ + crate://crates.io/winapi/0.3.9 \ + crate://crates.io/winapi-i686-pc-windows-gnu/0.4.0 \ + crate://crates.io/winapi-x86_64-pc-windows-gnu/0.4.0 \ + crate://crates.io/zeroize/1.5.7 \ +"
meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb renamed from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb +4 -2
@@ -5,14 +5,16 @@ HOMEPAGE = "https://pypi.org/project/bcrypt/" DEPENDS += "${PYTHON_PN}-cffi-native" -SRC_URI[sha256sum] = "433c410c2177057705da2a9f2cd01dd157493b2a7ac14c8593a16b3dab6b6bfb" +SRC_URI[sha256sum] = "c59c170fc9225faad04dde1ba61d85b413946e8ce2e5f5f5ff30dfd67283f319" -inherit pypi python_setuptools_build_meta ptest +inherit pypi python_setuptools3_rust ptest cargo-update-recipe-crates SRC_URI += " \ file://run-ptest \ " +require ${BPN}-crates.inc + RDEPENDS:${PN}-ptest += " \ ${PYTHON_PN}-pytest \ "
Signature · 1 line
2.30.2
RE: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0
Quoted 92 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Alexander Kanavin
Sent: den 30 september 2022 19:55
To: [email protected]
Cc: Alexander Kanavin <[email protected]>
Subject: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0The component has been reimplemented in rust, and comes with a large list of dependencies in Cargo.toml/Cargo.lock.
Rather than list them by hand, use a file generated with cargo-update-recipe-crates class.
Signed-off-by Alexander Kanavin <[email protected]>--- .../python/python3-bcrypt-crates.inc | 53 +++++++++++++++++++ ...crypt_3.2.2.bb => python3-bcrypt_4.0.0.bb} | 6 ++- 2 files changed, 57 insertions(+), 2 deletions(-) create mode 100644 meta/recipes-devtools/python/python3-bcrypt-crates.inc rename meta/recipes-devtools/python/{python3-bcrypt_3.2.2.bb => python3-bcrypt_4.0.0.bb} (76%)meta/recipes-devtools/python/python3-bcrypt-crates.inc new file +53
-0@@ -0,0 +1,53 @@ +# Autogenerated with 'bitbake -c update_crates python3-bcrypt' + +SRC_URI += " \ + crate://crates.io/autocfg/1.1.0 \ + crate://crates.io/base64/0.13.0 \ + crate://crates.io/bcrypt/0.13.0 \ + crate://crates.io/bcrypt-pbkdf/0.8.1 \ + crate://crates.io/bitflags/1.3.2 \ + crate://crates.io/block-buffer/0.10.2 \ + crate://crates.io/blowfish/0.9.1 \ + crate://crates.io/byteorder/1.4.3 \ + crate://crates.io/cfg-if/1.0.0 \ + crate://crates.io/cipher/0.4.3 \ + crate://crates.io/cpufeatures/0.2.4 \ + crate://crates.io/crypto-common/0.1.6 \ + crate://crates.io/digest/0.10.3 \ + crate://crates.io/generic-array/0.14.6 \ + crate://crates.io/getrandom/0.2.7 \ + crate://crates.io/indoc/0.3.6 \ + crate://crates.io/indoc-impl/0.3.6 \ + crate://crates.io/inout/0.1.3 \ + crate://crates.io/instant/0.1.12 \ + crate://crates.io/libc/0.2.132 \ + crate://crates.io/lock_api/0.4.7 \ + crate://crates.io/once_cell/1.13.1 \ + crate://crates.io/parking_lot/0.11.2 \ + crate://crates.io/parking_lot_core/0.8.5 \ + crate://crates.io/paste/0.1.18 \ + crate://crates.io/paste-impl/0.1.18 \ + crate://crates.io/pbkdf2/0.10.1 \ + crate://crates.io/proc-macro-hack/0.5.19 \ + crate://crates.io/proc-macro2/1.0.43 \ + crate://crates.io/pyo3/0.15.2 \ + crate://crates.io/pyo3-build-config/0.15.2 \ + crate://crates.io/pyo3-macros/0.15.2 \ + crate://crates.io/pyo3-macros-backend/0.15.2 \ + crate://crates.io/quote/1.0.21 \ + crate://crates.io/redox_syscall/0.2.16 \ + crate://crates.io/scopeguard/1.1.0 \ + crate://crates.io/sha2/0.10.2 \ + crate://crates.io/smallvec/1.9.0 \ + crate://crates.io/subtle/2.4.1 \ + crate://crates.io/syn/1.0.99 \ + crate://crates.io/typenum/1.15.0 \ + crate://crates.io/unicode-ident/1.0.3 \ + crate://crates.io/unindent/0.1.10 \ + crate://crates.io/version_check/0.9.4 \ + crate://crates.io/wasi/0.11.0+wasi-snapshot-preview1 \ + crate://crates.io/winapi/0.3.9 \ + crate://crates.io/winapi-i686-pc-windows-gnu/0.4.0 \ + crate://crates.io/winapi-x86_64-pc-windows-gnu/0.4.0 \ + crate://crates.io/zeroize/1.5.7 \ +"meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb renamed from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb +0
-1@@ -5,14 +5,16 @@ HOMEPAGE = "https://pypi.org/project/bcrypt/" DEPENDS += "${PYTHON_PN}-cffi-native" -SRC_URI[sha256sum] = "433c410c2177057705da2a9f2cd01dd157493b2a7ac14c8593a16b3dab6b6bfb"
meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb HOMEPAGE = "https://pypi.org/project/bcrypt/" · line 16
Quoted 10 lines
meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb renamed from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb +3
-1+SRC_URI[sha256sum] = "c59c170fc9225faad04dde1ba61d85b413946e8ce2e5f5f5ff30dfd67283f319" -inherit pypi python_setuptools_build_meta ptest +inherit pypi python_setuptools3_rust ptest cargo-update-recipe-crates SRC_URI += " \ file://run-ptest \ " +require ${BPN}-crates.inc
Would it make sense to instead use:
require ${BP}-crates.inc
i.e., make the crates.inc file versioned? That would help to make sure one does not forget to update the versions of the crates when the version of the recipe is updated. (My assumption is that with the huge lists of crates for a typical rust application, there will most likely always be updates to them when the application is updated.)
Quoted 6 lines
meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb renamed from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb +1
-1+ RDEPENDS:${PN}-ptest += " \ ${PYTHON_PN}-pytest \ " --2.30.2
//Peter
Re: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0
Any breakage would quickly be exposed in do_compile as cargo will fail against mismatching lock file. I usually do the opposite and remove version numbers everywhere except .bb, as it tends to cause trouble with automated updates.
Alex
Quoted 132 lines · Peter Kjellerstedt
Quoted 2 lines
-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Alexander Kanavin
Quoted 11 lines
Sent: den 30 september 2022 19:55
To: [email protected]
Cc: Alexander Kanavin <[email protected]>
Subject: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0The component has been reimplemented in rust, and comes with a large list of dependencies in Cargo.toml/Cargo.lock.
Rather than list them by hand, use a file generated with cargo-update-recipe-crates class.
Quoted 6 lines
Signed-off-by Alexander Kanavin <[email protected]>--- .../python/python3-bcrypt-crates.inc | 53 +++++++++++++++++++ ...crypt_3.2.2.bb => python3-bcrypt_4.0.0.bb} | 6 ++- 2 files changed, 57 insertions(+), 2 deletions(-) create mode 100644meta/recipes-devtools/python/python3-bcrypt-crates.inc
Quoted 1 line
rename meta/recipes-devtools/python/{python3-bcrypt_3.2.2.bb =>python3-bcrypt_4.0.0.bb} (76%)
Review
Quoted 2 lines
(quoted diff) +0
-0b/meta/recipes-devtools/python/python3-bcrypt-crates.inc
Review
Quoted 59 lines
new file mode 100644
index 0000000000..78c5d5aa22meta/recipes-devtools/python/python3-bcrypt-crates.inc new file +53
-0@@ -0,0 +1,53 @@ +# Autogenerated with 'bitbake -c update_crates python3-bcrypt' + +SRC_URI += " \ + crate://crates.io/autocfg/1.1.0 \ + crate://crates.io/base64/0.13.0 \ + crate://crates.io/bcrypt/0.13.0 \ + crate://crates.io/bcrypt-pbkdf/0.8.1 \ + crate://crates.io/bitflags/1.3.2 \ + crate://crates.io/block-buffer/0.10.2 \ + crate://crates.io/blowfish/0.9.1 \ + crate://crates.io/byteorder/1.4.3 \ + crate://crates.io/cfg-if/1.0.0 \ + crate://crates.io/cipher/0.4.3 \ + crate://crates.io/cpufeatures/0.2.4 \ + crate://crates.io/crypto-common/0.1.6 \ + crate://crates.io/digest/0.10.3 \ + crate://crates.io/generic-array/0.14.6 \ + crate://crates.io/getrandom/0.2.7 \ + crate://crates.io/indoc/0.3.6 \ + crate://crates.io/indoc-impl/0.3.6 \ + crate://crates.io/inout/0.1.3 \ + crate://crates.io/instant/0.1.12 \ + crate://crates.io/libc/0.2.132 \ + crate://crates.io/lock_api/0.4.7 \ + crate://crates.io/once_cell/1.13.1 \ + crate://crates.io/parking_lot/0.11.2 \ + crate://crates.io/parking_lot_core/0.8.5 \ + crate://crates.io/paste/0.1.18 \ + crate://crates.io/paste-impl/0.1.18 \ + crate://crates.io/pbkdf2/0.10.1 \ + crate://crates.io/proc-macro-hack/0.5.19 \ + crate://crates.io/proc-macro2/1.0.43 \ + crate://crates.io/pyo3/0.15.2 \ + crate://crates.io/pyo3-build-config/0.15.2 \ + crate://crates.io/pyo3-macros/0.15.2 \ + crate://crates.io/pyo3-macros-backend/0.15.2 \ + crate://crates.io/quote/1.0.21 \ + crate://crates.io/redox_syscall/0.2.16 \ + crate://crates.io/scopeguard/1.1.0 \ + crate://crates.io/sha2/0.10.2 \ + crate://crates.io/smallvec/1.9.0 \ + crate://crates.io/subtle/2.4.1 \ + crate://crates.io/syn/1.0.99 \ + crate://crates.io/typenum/1.15.0 \ + crate://crates.io/unicode-ident/1.0.3 \ + crate://crates.io/unindent/0.1.10 \ + crate://crates.io/version_check/0.9.4 \ + crate://crates.io/wasi/0.11.0+wasi-snapshot-preview1 \ + crate://crates.io/winapi/0.3.9 \ + crate://crates.io/winapi-i686-pc-windows-gnu/0.4.0 \ + crate://crates.io/winapi-x86_64-pc-windows-gnu/0.4.0 \ + crate://crates.io/zeroize/1.5.7 \ +"(quoted diff) +0
-0b/meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb
meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb HOMEPAGE = "https://pypi.org/project/bcrypt/" · line 8
Quoted 11 lines
similarity index 76%
rename from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb rename to meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb index54070e4b9c..857b38df2c100644meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb renamed from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb +0
-1@@ -5,14 +5,16 @@ HOMEPAGE = "https://pypi.org/project/bcrypt/" DEPENDS += "${PYTHON_PN}-cffi-native" -SRC_URI[sha256sum] ="
433c410c2177057705da2a9f2cd01dd157493b2a7ac14c8593a16b3dab6b6bfb"meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb HOMEPAGE = "https://pypi.org/project/bcrypt/" · line 8
Quoted 1 line
meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb renamed from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb +1
-0+SRC_URI[sha256sum] ="
c59c170fc9225faad04dde1ba61d85b413946e8ce2e5f5f5ff30dfd67283f319"meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb HOMEPAGE = "https://pypi.org/project/bcrypt/" · line 16
Quoted 9 lines
meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb renamed from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb +2
-1-inherit pypi python_setuptools_build_meta ptest +inherit pypi python_setuptools3_rust ptest cargo-update-recipe-crates SRC_URI += " \ file://run-ptest \ " +require ${BPN}-crates.incWould it make sense to instead use:
require ${BP}-crates.inc
i.e., make the crates.inc file versioned? That would help to make sure one does not forget to update the versions of the crates when the version of the recipe is updated. (My assumption is that with the huge lists of crates
for a typical rust application, there will most likely always be updates to
them when the application is updated.)Quoted 6 lines
meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb renamed from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb +1
-1+ RDEPENDS:${PN}-ptest += " \ ${PYTHON_PN}-pytest \ " --2.30.2
//Peter
RE: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0
Ok, as long as any problems are automatically detected, your proposed solution seems fine then.
//Peter
From: Alexander Kanavin <[email protected]>
Sent: den 1 oktober 2022 14:42
To: Peter Kjellerstedt <[email protected]>
Cc: Alexander Kanavin <[email protected]>; [email protected]
Subject: Re: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0
Any breakage would quickly be exposed in do_compile as cargo will fail against mismatching lock file. I usually do the opposite and remove version numbers everywhere except .bb, as it tends to cause trouble with automated updates.
Alex
Quoted 92 lines · On Sat 1. Oct 2022 at 14.05, Peter Kjellerstedt…
-----Original Message-----
From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> On Behalf Of Alexander Kanavin
Sent: den 30 september 2022 19:55
To: [email protected]<mailto:[email protected]>
Cc: Alexander Kanavin <[email protected]<mailto:[email protected]>>
Subject: [OE-core] [RFC PATCH 2/2] python3-bcrypt: update 3.2.2 -> 4.0.0The component has been reimplemented in rust, and comes with a large list of dependencies in Cargo.toml/Cargo.lock.
Rather than list them by hand, use a file generated with cargo-update-recipe-crates class.
--- .../python/python3-bcrypt-crates.inc | 53 +++++++++++++++++++ ...crypt_3.2.2.bb<http://crypt_3.2.2.bb> => python3-bcrypt_4.0.0.bb<http://python3-bcrypt_4.0.0.bb>} | 6 ++- 2 files changed, 57 insertions(+), 2 deletions(-) create mode 100644 meta/recipes-devtools/python/python3-bcrypt-crates.inc rename meta/recipes-devtools/python/{python3-bcrypt_3.2.2.bb<http://python3-bcrypt_3.2.2.bb> => python3-bcrypt_4.0.0.bb<http://python3-bcrypt_4.0.0.bb>} (76%)meta/recipes-devtools/python/python3-bcrypt-crates.inc new file +53
-0@@ -0,0 +1,53 @@ +# Autogenerated with 'bitbake -c update_crates python3-bcrypt' + +SRC_URI += " \ + crate://crates.io/autocfg/1.1.0<http://crates.io/autocfg/1.1.0> \ + crate://crates.io/base64/0.13.0<http://crates.io/base64/0.13.0> \ + crate://crates.io/bcrypt/0.13.0<http://crates.io/bcrypt/0.13.0> \ + crate://crates.io/bcrypt-pbkdf/0.8.1<http://crates.io/bcrypt-pbkdf/0.8.1> \ + crate://crates.io/bitflags/1.3.2<http://crates.io/bitflags/1.3.2> \ + crate://crates.io/block-buffer/0.10.2<http://crates.io/block-buffer/0.10.2> \ + crate://crates.io/blowfish/0.9.1<http://crates.io/blowfish/0.9.1> \ + crate://crates.io/byteorder/1.4.3<http://crates.io/byteorder/1.4.3> \ + crate://crates.io/cfg-if/1.0.0<http://crates.io/cfg-if/1.0.0> \ + crate://crates.io/cipher/0.4.3<http://crates.io/cipher/0.4.3> \ + crate://crates.io/cpufeatures/0.2.4<http://crates.io/cpufeatures/0.2.4> \ + crate://crates.io/crypto-common/0.1.6<http://crates.io/crypto-common/0.1.6> \ + crate://crates.io/digest/0.10.3<http://crates.io/digest/0.10.3> \ + crate://crates.io/generic-array/0.14.6<http://crates.io/generic-array/0.14.6> \ + crate://crates.io/getrandom/0.2.7<http://crates.io/getrandom/0.2.7> \ + crate://crates.io/indoc/0.3.6<http://crates.io/indoc/0.3.6> \ + crate://crates.io/indoc-impl/0.3.6<http://crates.io/indoc-impl/0.3.6> \ + crate://crates.io/inout/0.1.3<http://crates.io/inout/0.1.3> \ + crate://crates.io/instant/0.1.12<http://crates.io/instant/0.1.12> \ + crate://crates.io/libc/0.2.132<http://crates.io/libc/0.2.132> \ + crate://crates.io/lock_api/0.4.7<http://crates.io/lock_api/0.4.7> \ + crate://crates.io/once_cell/1.13.1<http://crates.io/once_cell/1.13.1> \ + crate://crates.io/parking_lot/0.11.2<http://crates.io/parking_lot/0.11.2> \ + crate://crates.io/parking_lot_core/0.8.5<http://crates.io/parking_lot_core/0.8.5> \ + crate://crates.io/paste/0.1.18<http://crates.io/paste/0.1.18> \ + crate://crates.io/paste-impl/0.1.18<http://crates.io/paste-impl/0.1.18> \ + crate://crates.io/pbkdf2/0.10.1<http://crates.io/pbkdf2/0.10.1> \ + crate://crates.io/proc-macro-hack/0.5.19<http://crates.io/proc-macro-hack/0.5.19> \ + crate://crates.io/proc-macro2/1.0.43<http://crates.io/proc-macro2/1.0.43> \ + crate://crates.io/pyo3/0.15.2<http://crates.io/pyo3/0.15.2> \ + crate://crates.io/pyo3-build-config/0.15.2<http://crates.io/pyo3-build-config/0.15.2> \ + crate://crates.io/pyo3-macros/0.15.2<http://crates.io/pyo3-macros/0.15.2> \ + crate://crates.io/pyo3-macros-backend/0.15.2<http://crates.io/pyo3-macros-backend/0.15.2> \ + crate://crates.io/quote/1.0.21<http://crates.io/quote/1.0.21> \ + crate://crates.io/redox_syscall/0.2.16<http://crates.io/redox_syscall/0.2.16> \ + crate://crates.io/scopeguard/1.1.0<http://crates.io/scopeguard/1.1.0> \ + crate://crates.io/sha2/0.10.2<http://crates.io/sha2/0.10.2> \ + crate://crates.io/smallvec/1.9.0<http://crates.io/smallvec/1.9.0> \ + crate://crates.io/subtle/2.4.1<http://crates.io/subtle/2.4.1> \ + crate://crates.io/syn/1.0.99<http://crates.io/syn/1.0.99> \ + crate://crates.io/typenum/1.15.0<http://crates.io/typenum/1.15.0> \ + crate://crates.io/unicode-ident/1.0.3<http://crates.io/unicode-ident/1.0.3> \ + crate://crates.io/unindent/0.1.10<http://crates.io/unindent/0.1.10> \ + crate://crates.io/version_check/0.9.4<http://crates.io/version_check/0.9.4> \ + crate://crates.io/wasi/0.11.0+wasi-snapshot-preview1<http://crates.io/wasi/0.11.0+wasi-snapshot-preview1> \ + crate://crates.io/winapi/0.3.9<http://crates.io/winapi/0.3.9> \ + crate://crates.io/winapi-i686-pc-windows-gnu/0.4.0<http://crates.io/winapi-i686-pc-windows-gnu/0.4.0> \ + crate://crates.io/winapi-x86_64-pc-windows-gnu/0.4.0<http://crates.io/winapi-x86_64-pc-windows-gnu/0.4.0> \ + crate://crates.io/zeroize/1.5.7<http://crates.io/zeroize/1.5.7> \ +"meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb<http://python3-bcrypt_4.0.0.bb> renamed from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb<http://python3-bcrypt_3.2.2.bb> +0
-1@@ -5,14 +5,16 @@ HOMEPAGE = "https://pypi.org/project/bcrypt/" DEPENDS += "${PYTHON_PN}-cffi-native" -SRC_URI[sha256sum] = "433c410c2177057705da2a9f2cd01dd157493b2a7ac14c8593a16b3dab6b6bfb"
meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb<http://python3-bcrypt_4.0.0.bb> HOMEPAGE = "https://pypi.org/project/bcrypt/" · line 16
Quoted 10 lines
meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb<http://python3-bcrypt_4.0.0.bb> renamed from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb<http://python3-bcrypt_3.2.2.bb> +3
-1+SRC_URI[sha256sum] = "c59c170fc9225faad04dde1ba61d85b413946e8ce2e5f5f5ff30dfd67283f319" -inherit pypi python_setuptools_build_meta ptest +inherit pypi python_setuptools3_rust ptest cargo-update-recipe-crates SRC_URI += " \ file://run-ptest \ " +require ${BPN}-crates.inc
Would it make sense to instead use:
require ${BP}-crates.inc
i.e., make the crates.inc file versioned? That would help to make sure one does not forget to update the versions of the crates when the version of the recipe is updated. (My assumption is that with the huge lists of crates for a typical rust application, there will most likely always be updates to them when the application is updated.)
Quoted 6 lines
meta/recipes-devtools/python/python3-bcrypt_4.0.0.bb<http://python3-bcrypt_4.0.0.bb> renamed from meta/recipes-devtools/python/python3-bcrypt_3.2.2.bb<http://python3-bcrypt_3.2.2.bb> +1
-1+ RDEPENDS:${PN}-ptest += " \ ${PYTHON_PN}-pytest \ " --2.30.2
//Peter
Hi Alex,
Quoted 42 lines · Alexander Kanavin
For better or worse, more and more rust components are appearing that do not include their dependencies in tarballs (or git trees), and rely on cargo to fetch them. On the other hand, bitbake does not use cargo (and quite possible won't ever be able to), and relies on having each item explicitly listed in
SRC_URIwith a crate:// prefix. This however creates a problem of both making such lists in the first place and updating them when a recipe is updated to a newer version.So this class can be used to perform such updates by implementing a task that does it; the next commit shows the outcome for python3-bcrypt (which has been tested to work and produce a successful build).
Note: the python script relies on tomllib library, which appears in Python 3.11 and does not exist in earlier versions - I've tested this by first updating python to 3.11-rc2 in oe-core.
Signed-off-by Alexander Kanavin <[email protected]>--- .../cargo-update-recipe-crates.bbclass | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+)create mode 100644 meta/classes-recipe/cargo-update-recipe-crates.bbclass
meta/classes-recipe/cargo-update-recipe-crates.bbclass new file +15
-0@@ -0,0 +1,41 @@ +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: MIT +# + +## +## Purpose: +## This class is used to update the list of crates in SRC_URI +## by reading Cargo.lock in the source tree. +## +## See meta/recipes-devtools/python/python3-bcrypt_*.bb for an example +## +## To perform the update: bitbake -c update_crates recipe-name +
meta/classes-recipe/cargo-update-recipe-crates.bbclass line 25
Quoted 10 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass new file +10
-0+addtask do_update_crates after do_patch +do_update_crates[depends] = "python3-native:do_populate_sysroot" + +do_update_crates() { + nativepython3 - <<EOF + +def get_crates(f): + import tomllib + c_list = 'SRC_URI += " \\ \n' + crates = tomllib.load(open(f, 'rb'))
Aren't we supposed to use context for this or close the file descriptor after opening it manually?
e.g.
with open(f, 'rb') as file:
crates = tommlib.load(file)
?
meta/classes-recipe/cargo-update-recipe-crates.bbclass line 27
Quoted 2 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass new file +2
-0+ for c in crates['package']: + if 'source' in c and 'crates.io' in c['source']:
if 'crates.io' in c.get('source', '')
meta/classes-recipe/cargo-update-recipe-crates.bbclass line 37
Quoted 10 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass new file +10
-0+ c_list += " crate://crates.io/{}/{} \\ \n".format(c['name'], c['version']) + c_list += '"\n' + return c_list + +import os +crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n" +for root, dirs, files in os.walk('${S}'): + for file in files: + if file == 'Cargo.lock': + crates += get_crates(os.path.join(root, file))
from pathlib import Path
for file in Path('${S}').glob('**/Cargo.lock'):
crates += get_crates(str(file))
meta/classes-recipe/cargo-update-recipe-crates.bbclass line 39
Quoted 2 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass new file +2
-0+open(os.path.join('${THISDIR}', '${PN}'+"-crates.inc"), 'w').write(crates) +
with open... here again?
Cheers,
Quentin
Quoted 5 lines · Quentin Schulz
meta/classes-recipe/cargo-update-recipe-crates.bbclass line 25
Quoted 4 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass new file +4
-0+def get_crates(f): + import tomllib + c_list = 'SRC_URI += " \\ \n' + crates = tomllib.load(open(f, 'rb'))
Quoted 8 lines
Aren't we supposed to use context for this or close the file descriptor after opening it manually?
e.g.
withopen(f, 'rb')as file:
crates =tommlib.load(file)?
python3.11 no longer prints warnings about this, so I assume it is now ok.
For the other two suggested changes please explain why.
Alex
On Fri, Sep 30, 2022 at 10:55 AM Alexander Kanavin <[email protected]> wrote:
%> Snip %<
Quoted 3 lines
On the other hand, bitbake does not use cargo (and quite possible won't ever be able to),
Can you elaborate on this?
I only ask because we are using this[1] and this[2] to build this[3] recipe for hardknott.
1. https://github.com/meta-rust/meta-rust/blob/master/lib/crate.py
2. https://crates.io/crates/cargo-bitbake
3.
https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3-cryptography-vectors_36.0.2.bb?h=kirkstone
..Ch:W..
Signature · 2 lines
*"Perfection must be reached by degrees; she requires the slow hand of time." - Voltaire*
Quoted 7 lines · Chuck Wolber
Can you elaborate on this?
I only ask because we are using this[1] and this[2] to build this[3] recipe for hardknott.
1. https://github.com/meta-rust/meta-rust/blob/master/lib/crate.py
2. https://crates.io/crates/cargo-bitbake
3. https://git.yoctoproject.org/poky/tree/meta/recipes-devtools/python/python3-cryptography-vectors_36.0.2.bb?h=kirkstone
As you can see in [1], crate:// fetcher is not using cargo, it's using wget - so the question of resolving and listing dependencies from Cargo.lock falls on the tooling that writes and updates the recipe itself. cargo-bitbake failed at the installation step for me (both with --locked and without it), and in general I do not think we should rely on external tools with unknown maintenance status. The lists of open issues and pull requests are not at all reassuring:
https://github.com/meta-rust/cargo-bitbake
Alex
Quoted 11 lines · Alexander Kanavin
Quoted 3 lines · Chuck Wolber
Can you elaborate on this?
I only ask because we are using this[1] and this[2] to build this[3]
recipe for hardknott.
Quoted 10 lines
As you can see in [1], crate:// fetcher is not using cargo, it's using wget - so the question of resolving and listing dependencies from Cargo.lock falls on the tooling that writes and updates the recipe itself. cargo-bitbake failed at the installation step for me (both with --locked and without it), and in general I do not think we should rely on external tools with unknown maintenance status. The lists of open issues and pull requests are not at all reassuring:
https://github.com/meta-rust/cargo-bitbake
I have cargo bitbake running locally, but in recent python3-cryptography and python3-bcrypt upgrades
it produced crate://* SRC_URI that actually failed to build. I had to manually loop through builds until
I sussed out all the versions that would work. This proves Alex's point.
Having looked at the implementation, I am very much in favor of this proposal. In fact, I wonder if
a similar approach could help with golang and maybe even nodejs/npm. But that is for another day.
Alex
Quoted 12 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#171453):
https://lists.openembedded.org/g/openembedded-core/message/171453
Mute This Topic: https://lists.openembedded.org/mt/94022674/924729
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
PATCH 3/3 cargo-update-recipe-crates: small improvements
In reply to Alexander Kanavin* use 4 spaces
* avoid trailing space
* add CARGO_LOCK_SRC_DIR to allow searching outside ${S}
* use BPN in output filename
* First I've used CARGO_LOCK_SRC_DIR as relative to ${S}, because that's what CARGO_SRC_DIR in cargo.bbclass is using:
meta/classes-recipe/cargo.bbclass:CARGO_SRC_DIR ??= "" meta/classes-recipe/cargo.bbclass:MANIFEST_PATH ??= "${S}/${CARGO_SRC_DIR}/Cargo.toml"
but change to absolute path (defaulting to ${S}) as requested by Alex:
11:44 < kanavin> JaMa, would prefer CARGO_LOCK_SRC_DIR ??= "${S}"
11:46 < kanavin> otherwise looks good* I've resolved my usecase for CARGO_LOCK_SRC_DIR by changing S back to ${WORKDIR}/git and using CARGO_SRC_DIR to select the right subdirectory to be built, because the Cargo.toml in this subdirectory was also referencing other subdirectories with relative path:
https://github.com/solana-labs/solana-program-library/blob/88b147506d5b9515f3a4762421a0b8c309188dc9/token/cli/Cargo.toml#L30
so including all Cargo.lock files in whole ${WORKDIR}/git seems like reasonable approach
--- .../cargo-update-recipe-crates.bbclass | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
meta/classes-recipe/cargo-update-recipe-crates.bbclass +9 -5
@@ -16,26 +16,30 @@ addtask do_update_crates after do_patch do_update_crates[depends] = "python3-native:do_populate_sysroot" +# The directory where to search for Cargo.lock files +CARGO_LOCK_SRC_DIR ??= "${S}" + do_update_crates() { nativepython3 - <<EOF def get_crates(f): import tomllib - c_list = 'SRC_URI += " \\ \n' + c_list = '# from %s' % os.path.relpath(f, '${CARGO_LOCK_SRC_DIR}') + c_list += '\nSRC_URI += " \\\' crates = tomllib.load(open(f, 'rb')) for c in crates['package']: if 'source' in c and 'crates.io' in c['source']: - c_list += " crate://crates.io/{}/{} \\ \n".format(c['name'], c['version']) - c_list += '"\n' + c_list += '\n crate://crates.io/%s/%s \\\' % (c['name'], c['version']) + c_list += '\n"\n' return c_list import os crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n" -for root, dirs, files in os.walk('${S}'): +for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'): for file in files: if file == 'Cargo.lock': crates += get_crates(os.path.join(root, file)) -open(os.path.join('${THISDIR}', '${PN}'+"-crates.inc"), 'w').write(crates) +open(os.path.join('${THISDIR}', '${BPN}'+"-crates.inc"), 'w').write(crates) EOF }
Signature · 1 line
2.38.1