Skip to contentSkip to conversation

Extend cargo based recipe support

PATCH Superseded [email protected] 4 participants 21 messages Mar 30, 2023 – Apr 3, 2023 3 messages not in the archive

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v5 cover Extend cargo based recipe support

From: Frederic Martinsons <[email protected]>

This series brings the support of local git repository inside a cargo based recipe.

It also enables devtool capacity to such a recipe along with an example of recipe and a new selftest case.

The following changes since commit c3fe173d5196506d89aa464ba56aabcf581a60db:

oeqa/selftest/cases/package.py: adding unittest for package rename conflicts (2023-03-30 12:32:26 +0100)

are available in the Git repository at:

https://gitlab.com/fmartinsons/openembedded-core cargo-extend-support-git-and-devtool

Alex Kiernan (1):
cargo_common.bbclass: Support local github repos

Frederic Martinsons (5):
cargo_common.bbclass: add support of user in url for patch
devtool: add support for multiple git url inside a cargo based recipe
patch: support of git patches when the source uri contained subpath
parameter
meta-selftest: provide a recipe for zvariant cargo-update-recipe-crates: don't walk on the whole dir

 .../zvariant/zvariant-crates.inc              |  268 ++++
 .../0001-Tweak-zvariant-crate-config.patch    | 1292 +++++++++++++++++
 .../zvariant/zvariant_3.12.0.bb               |   26 +
 .../cargo-update-recipe-crates.bbclass        |   12 +-
 meta/classes-recipe/cargo_common.bbclass      |   33 +
 meta/classes/externalsrc.bbclass              |    4 +-
 meta/lib/oe/patch.py                          |   57 +-
 meta/lib/oeqa/selftest/cases/devtool.py       |   93 ++
 8 files changed, 1765 insertions(+), 20 deletions(-)
 create mode 100644 meta-selftest/recipes-extended/zvariant/zvariant-crates.inc
 create mode 100644 meta-selftest/recipes-extended/zvariant/zvariant/0001-Tweak-zvariant-crate-config.patch
 create mode 100644 meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb
Signature · 1 line
2.34.1
#1 lore raw

PATCH v5 1/6 cargo_common.bbclass: Support local github repos

From: Alex Kiernan <[email protected]>

Since disable network was added cargo configurations which reference git repos fail as they attempt to fetch across the network as part of do_compile, even if EXTRA_OECARGO_PATHS to add them as part of paths is used, as this is documented as only working for packages which exist in crates.io.

Add parsing of the SRC_URIs for git repos and include [patch] sections to redirect to the checked out source repos which the bitbake fetcher has already populated.

There are still cases which don't work - if you have multiple copies of the same repo with different revisions, there's currently no way to represent that and anything using a repo which has a virtual manifest will fail to build (see https://github.com/rust-lang/cargo/issues/4934).

---
 meta/classes-recipe/cargo_common.bbclass | 30 ++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
meta/classes-recipe/cargo_common.bbclass +30 -0
index f503a001dd..63b1382908 100644
@@ -116,6 +116,36 @@ cargo_common_do_configure () {
 	EOF
 }
 
+python cargo_common_do_patch_paths() {
+    cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config")
+    if not os.path.exists(cargo_config):
+        return
+
+    src_uri = (d.getVar('SRC_URI') or "").split()
+    if len(src_uri) == 0:
+        return
+
+    patches = dict()
+    workdir = d.getVar('WORKDIR')
+    fetcher = bb.fetch2.Fetch(src_uri, d)
+    for url in fetcher.urls:
+        ud = fetcher.ud[url]
+        if ud.type == 'git':
+            name = ud.parm.get('name')
+            destsuffix = ud.parm.get('destsuffix')
+            if name is not None and destsuffix is not None:
+                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
+                path = '%s = { path = "%s" }' % (name, os.path.join(workdir, destsuffix))
+                patches.setdefault(repo, []).append(path)
+
+    with open(cargo_config, "a+") as config:
+        for k, v in patches.items():
+            print('\n[patch."%s"]' % k, file=config)
+            for name in v:
+                print(name, file=config)
+}
+do_configure[postfuncs] += "cargo_common_do_patch_paths"
+
 oe_cargo_fix_env () {
 	export CC="${RUST_TARGET_CC}"
 	export CXX="${RUST_TARGET_CXX}"
Signature · 1 line
2.34.1
#2 lore raw

PATCH v5 2/6 cargo_common.bbclass: add support of user in url for patch

In reply to [email protected]

From: Frederic Martinsons <[email protected]>

To handle url like git://git@repo/project

---
 meta/classes-recipe/cargo_common.bbclass | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
meta/classes-recipe/cargo_common.bbclass +4 -1
index 63b1382908..82ab25b59c 100644
@@ -134,7 +134,10 @@ python cargo_common_do_patch_paths() {
             name = ud.parm.get('name')
             destsuffix = ud.parm.get('destsuffix')
             if name is not None and destsuffix is not None:
-                repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
+                if ud.user:
+                    repo = '%s://%s@%s%s' % (ud.proto, ud.user, ud.host, ud.path)
+                else:
+                    repo = '%s://%s%s' % (ud.proto, ud.host, ud.path)
                 path = '%s = { path = "%s" }' % (name, os.path.join(workdir, destsuffix))
                 patches.setdefault(repo, []).append(path)
 
Signature · 1 line
2.34.1
#3 lore raw

PATCH v5 3/6 devtool: add support for multiple git url inside a cargo based recipe

In reply to [email protected]

From: Frederic Martinsons <[email protected]>

Without that, the possible git urls that are in SRC_URI of a recipe are removed from SRC_URI during devtool process and so the cargo_common_do_patch_paths in cargo_common.bbclass cannot patch these packages to fetch them locally.

I use a generic type name because I foresee this change will be useful for recipe that used a package manager (cargo but also npm) see https://bugzilla.yoctoproject.org/show_bug.cgi?id=11015

---
 meta/classes/externalsrc.bbclass | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
meta/classes/externalsrc.bbclass +1 -3
index 26c5803ee6..b00fdba8e9 100644
@@ -68,9 +68,7 @@ python () {
         for url in fetch.urls:
             url_data = fetch.ud[url]
             parm = url_data.parm
-            if (url_data.type == 'file' or
-                    url_data.type == 'npmsw' or url_data.type == 'crate' or
-                    'type' in parm and parm['type'] == 'kmeta'):
+            if url_data.type in ['file', 'npmsw', 'crate'] or parm.get('type') in ['kmeta', 'git-dependency']:
                 local_srcuri.append(url)
 
         d.setVar('SRC_URI', ' '.join(local_srcuri))
Signature · 1 line
2.34.1
#4 lore raw

PATCH v5 4/6 patch: support of git patches when the source uri contained subpath parameter

In reply to [email protected]

From: Frederic Martinsons <[email protected]>

This is for a specific case where:
- A recipe use a subpath on a git repo (e.g. git://repo.git/projects;subpath=subproject)
- The recipe contains a patch to apply
- a devtool modify is used on this recipe

With these conditions, the patch cannot be applied at all. GitApplyTree class is used for handling patch under devtool, but when subpath is present in SRC_URI, the resulting git tree
is dirty (every files and directories which was not in subpath are suppressed) and so "git am" refuse to apply patches.

That would not be an issue since the GitApplyTree have a fallback to PatchTree in case of error, but during this error management, there is a "git reset --hard HEAD" call which suppress the subpath operation and finally prevents the patch to be applied even with PatchTree.

When devtool is not involved, only PatchTree class is used and the above problem is irrelevant.

To support git patching during devtool, the presence of subpath and the dirtyness of the repo are checked. If both conditions are met, we directly call PatchTree like it was already done in case of error during git apply.

---
 meta/lib/oe/patch.py | 57 ++++++++++++++++++++++++++++++++++----------
 1 file changed, 44 insertions(+), 13 deletions(-)
meta/lib/oe/patch.py +44 -13
index b2dc8d0a90..d047b3b947 100644
@@ -499,6 +499,36 @@ class GitApplyTree(PatchTree):
         finally:
             shutil.rmtree(tempdir)
 
+    def _need_dirty_check(self):
+        fetch = bb.fetch2.Fetch([], self.d)
+        check_dirtyness = False
+        for url in fetch.urls:
+            url_data = fetch.ud[url]
+            parm = url_data.parm
+            # a git url with subpath param will surely be dirty
+            # since the git tree from which we clone will be emptied
+            # from all files that are not in the subpath
+            if url_data.type == 'git' and parm.get('subpath'):
+                check_dirtyness = True
+        return check_dirtyness
+
+    def _commitpatch(self, patch, patchfilevar):
+        output = ""
+        # Add all files
+        shellcmd = ["git", "add", "-f", "-A", "."]
+        output += runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
+        # Exclude the patches directory
+        shellcmd = ["git", "reset", "HEAD", self.patchdir]
+        output += runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
+        # Commit the result
+        (tmpfile, shellcmd) = self.prepareCommit(patch['file'], self.commituser, self.commitemail)
+        try:
+            shellcmd.insert(0, patchfilevar)
+            output += runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
+        finally:
+            os.remove(tmpfile)
+        return output
+
     def _applypatch(self, patch, force = False, reverse = False, run = True):
         import shutil
 
@@ -534,6 +564,19 @@ class GitApplyTree(PatchTree):
         shutil.copy2(commithook, applyhook)
         try:
             patchfilevar = 'PATCHFILE="%s"' % os.path.basename(patch['file'])
+            if self._need_dirty_check():
+                # Check dirtyness of the tree
+                try:
+                    output = runcmd(["git", "--work-tree=%s" % reporoot, "status", "--short"])
+                except CmdError:
+                    pass
+                else:
+                    if output:
+                        # The tree is dirty, not need to try to apply patches with git anymore
+                        # since they fail, fallback directly to patch
+                        output = PatchTree._applypatch(self, patch, force, reverse, run)
+                        output += self._commitpatch(patch, patchfilevar)
+                        return output
             try:
                 shellcmd = [patchfilevar, "git", "--work-tree=%s" % reporoot]
                 self.gitCommandUserOptions(shellcmd, self.commituser, self.commitemail)
@@ -560,19 +603,7 @@ class GitApplyTree(PatchTree):
                 except CmdError:
                     # Fall back to patch
                     output = PatchTree._applypatch(self, patch, force, reverse, run)
-                # Add all files
-                shellcmd = ["git", "add", "-f", "-A", "."]
-                output += runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
-                # Exclude the patches directory
-                shellcmd = ["git", "reset", "HEAD", self.patchdir]
-                output += runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
-                # Commit the result
-                (tmpfile, shellcmd) = self.prepareCommit(patch['file'], self.commituser, self.commitemail)
-                try:
-                    shellcmd.insert(0, patchfilevar)
-                    output += runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
-                finally:
-                    os.remove(tmpfile)
+                output += self._commitpatch(patch, patchfilevar)
                 return output
         finally:
             shutil.rmtree(hooks_dir)
Signature · 1 line
2.34.1
#5 lore raw

PATCH v5 5/6 meta-selftest: provide a recipe for zvariant

In reply to [email protected]

From: Frederic Martinsons <[email protected]>

This recipe is for showing a "real world" example of a crate that depends on some git repositories.

Usually, this kind of crate is built within a global workspace (here it is the zbus project) and so doesn't need a Cargo.lock on its own.

For the sake of the demonstration, I had to tweak things a little to be able to compile zvariant in standalone (no relative path in dependency, no symlink to LICENSE provide a Cargo.lock)

The use case where the crate had some git repository in dependency is very common for "private" crate that are not aimed to be published on crates.io. When the project grow bigger, it is common to have a bin and multiple lib developped in parallel, and these libs are surely on a git repostitory.

A test case have been also added to check for:
- the previous patch about git subpath parameter and devtool
- the correctness of overriding dependencies (first patch of the
series)

---
 .../zvariant/zvariant-crates.inc              |  268 ++++
 .../0001-Tweak-zvariant-crate-config.patch    | 1292 +++++++++++++++++
 .../zvariant/zvariant_3.12.0.bb               |   26 +
 meta/lib/oeqa/selftest/cases/devtool.py       |   93 ++
 4 files changed, 1679 insertions(+)
 create mode 100644 meta-selftest/recipes-extended/zvariant/zvariant-crates.inc
 create mode 100644 meta-selftest/recipes-extended/zvariant/zvariant/0001-Tweak-zvariant-crate-config.patch
 create mode 100644 meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb
meta-selftest/recipes-extended/zvariant/zvariant-crates.inc new file +268 -0
new file mode 100644
index 0000000000..63129ba990
@@ -0,0 +1,268 @@
+SRC_URI += " \
+    crate://crates.io/anes/0.1.6 \
+    crate://crates.io/anyhow/1.0.70 \
+    crate://crates.io/arrayvec/0.7.2 \
+    crate://crates.io/atty/0.2.14 \
+    crate://crates.io/autocfg/1.1.0 \
+    crate://crates.io/bitflags/1.3.2 \
+    crate://crates.io/bumpalo/3.12.0 \
+    crate://crates.io/byteorder/1.4.3 \
+    crate://crates.io/cast/0.3.0 \
+    crate://crates.io/cfg-if/1.0.0 \
+    crate://crates.io/chrono/0.4.24 \
+    crate://crates.io/ciborium-io/0.2.0 \
+    crate://crates.io/ciborium-ll/0.2.0 \
+    crate://crates.io/ciborium/0.2.0 \
+    crate://crates.io/clap/3.2.23 \
+    crate://crates.io/clap_lex/0.2.4 \
+    crate://crates.io/criterion-plot/0.5.0 \
+    crate://crates.io/criterion/0.4.0 \
+    crate://crates.io/crossbeam-channel/0.5.7 \
+    crate://crates.io/crossbeam-deque/0.8.3 \
+    crate://crates.io/crossbeam-epoch/0.9.14 \
+    crate://crates.io/crossbeam-utils/0.8.15 \
+    crate://crates.io/either/1.8.1 \
+    crate://crates.io/enumflags2/0.7.5 \
+    crate://crates.io/enumflags2_derive/0.7.4 \
+    crate://crates.io/form_urlencoded/1.1.0 \
+    crate://crates.io/futures-channel/0.3.27 \
+    crate://crates.io/futures-core/0.3.27 \
+    crate://crates.io/futures-executor/0.3.27 \
+    crate://crates.io/futures-macro/0.3.27 \
+    crate://crates.io/futures-task/0.3.27 \
+    crate://crates.io/futures-util/0.3.27 \
+    crate://crates.io/getrandom/0.2.8 \
+    crate://crates.io/half/1.8.2 \
+    crate://crates.io/hashbrown/0.12.3 \
+    crate://crates.io/heck/0.3.3 \
+    crate://crates.io/hermit-abi/0.1.19;name=hermit-abi-0.1.19 \
+    crate://crates.io/hermit-abi/0.2.6;name=hermit-abi-0.2.6 \
+    crate://crates.io/idna/0.3.0 \
+    crate://crates.io/indexmap/1.9.2 \
+    crate://crates.io/itertools/0.9.0;name=itertools-0.9.0 \
+    crate://crates.io/itertools/0.10.5;name=itertools-0.10.5 \
+    crate://crates.io/itoa/1.0.6 \
+    crate://crates.io/js-sys/0.3.61 \
+    crate://crates.io/lazy_static/1.4.0 \
+    crate://crates.io/libc/0.2.140 \
+    crate://crates.io/log/0.4.17 \
+    crate://crates.io/memchr/2.5.0 \
+    crate://crates.io/memoffset/0.8.0 \
+    crate://crates.io/num-integer/0.1.45 \
+    crate://crates.io/num-traits/0.2.15 \
+    crate://crates.io/num_cpus/1.15.0 \
+    crate://crates.io/once_cell/1.17.1 \
+    crate://crates.io/oorandom/11.1.3 \
+    crate://crates.io/os_str_bytes/6.5.0 \
+    crate://crates.io/percent-encoding/2.2.0 \
+    crate://crates.io/pin-project-lite/0.2.9 \
+    crate://crates.io/pin-utils/0.1.0 \
+    crate://crates.io/pkg-config/0.3.26 \
+    crate://crates.io/plotters-backend/0.3.4 \
+    crate://crates.io/plotters-svg/0.3.3 \
+    crate://crates.io/plotters/0.3.4 \
+    crate://crates.io/ppv-lite86/0.2.17 \
+    crate://crates.io/proc-macro-crate/0.1.5;name=proc-macro-crate-0.1.5 \
+    crate://crates.io/proc-macro-crate/1.3.1;name=proc-macro-crate-1.3.1 \
+    crate://crates.io/proc-macro-error/1.0.4 \
+    crate://crates.io/proc-macro-error-attr/1.0.4 \
+    crate://crates.io/proc-macro-error/1.0.4 \
+    crate://crates.io/proc-macro2/1.0.53 \
+    crate://crates.io/quote/1.0.26 \
+    crate://crates.io/rand/0.8.5 \
+    crate://crates.io/rand_chacha/0.3.1 \
+    crate://crates.io/rand_core/0.6.4 \
+    crate://crates.io/rayon-core/1.11.0 \
+    crate://crates.io/rayon/1.7.0 \
+    crate://crates.io/regex-syntax/0.6.29 \
+    crate://crates.io/regex/1.7.2 \
+    crate://crates.io/ryu/1.0.13 \
+    crate://crates.io/same-file/1.0.6 \
+    crate://crates.io/scopeguard/1.1.0 \
+    crate://crates.io/serde/1.0.158 \
+    crate://crates.io/serde_bytes/0.11.9 \
+    crate://crates.io/serde_derive/1.0.158 \
+    crate://crates.io/serde_json/1.0.94 \
+    crate://crates.io/serde_repr/0.1.12 \
+    crate://crates.io/slab/0.4.8 \
+    crate://crates.io/static_assertions/1.1.0 \
+    crate://crates.io/strum/0.18.0 \
+    crate://crates.io/strum_macros/0.18.0 \
+    crate://crates.io/syn/1.0.109;name=syn-1.0.109 \
+    crate://crates.io/syn/2.0.8;name=syn-2.0.8 \
+    crate://crates.io/system-deps/1.3.2 \
+    crate://crates.io/textwrap/0.16.0 \
+    crate://crates.io/thiserror-impl/1.0.40 \
+    crate://crates.io/thiserror/1.0.40 \
+    crate://crates.io/time-core/0.1.0 \
+    crate://crates.io/time-macros/0.2.8 \
+    crate://crates.io/time/0.3.20 \
+    crate://crates.io/tinytemplate/1.2.1 \
+    crate://crates.io/tinyvec/1.6.0 \
+    crate://crates.io/tinyvec_macros/0.1.1 \
+    crate://crates.io/toml/0.5.11 \
+    crate://crates.io/toml_datetime/0.6.1 \
+    crate://crates.io/toml_edit/0.19.8 \
+    crate://crates.io/unicode-bidi/0.3.13 \
+    crate://crates.io/unicode-ident/1.0.8 \
+    crate://crates.io/unicode-normalization/0.1.22 \
+    crate://crates.io/unicode-segmentation/1.10.1 \
+    crate://crates.io/url/2.3.1 \
+    crate://crates.io/uuid/1.3.0 \
+    crate://crates.io/version-compare/0.0.10 \
+    crate://crates.io/version_check/0.9.4 \
+    crate://crates.io/walkdir/2.3.3 \
+    crate://crates.io/wasi/0.11.0+wasi-snapshot-preview1 \
+    crate://crates.io/wasm-bindgen-backend/0.2.84 \
+    crate://crates.io/wasm-bindgen-macro-support/0.2.84 \
+    crate://crates.io/wasm-bindgen-macro/0.2.84 \
+    crate://crates.io/wasm-bindgen-shared/0.2.84 \
+    crate://crates.io/wasm-bindgen/0.2.84 \
+    crate://crates.io/web-sys/0.3.61 \
+    crate://crates.io/winapi-i686-pc-windows-gnu/0.4.0 \
+    crate://crates.io/winapi-util/0.1.5 \
+    crate://crates.io/winapi-x86_64-pc-windows-gnu/0.4.0 \
+    crate://crates.io/winapi/0.3.9 \
+    crate://crates.io/winnow/0.4.0 \
+    crate://crates.io/zvariant_derive/3.12.0 \
+    crate://crates.io/zvariant_utils/1.0.0 \
+    git://github.com/gtk-rs/glib;protocol=https;nobranch=1;name=glib;destsuffix=glib;type=git-dependency \
+    git://github.com/gtk-rs/sys;protocol=https;nobranch=1;name=glib-sys;destsuffix=glib-sys;subpath=glib-sys;type=git-dependency \
+    git://github.com/gtk-rs/sys;protocol=https;nobranch=1;name=gobject-sys;destsuffix=gobject-sys;subpath=gobject-sys;type=git-dependency \
+"
+
+SRCREV_FORMAT .= "_glib"
+SRCREV_glib = "c9ee583cea07830c099cdcccd33eda9ef705ea93"
+
+SRCREV_FORMAT .= "_glib-sys"
+SRCREV_glib-sys = "5f35e26c65d24f8f018f643218de0f5807ba5f01"
+
+SRCREV_FORMAT .= "_gobject-sys"
+SRCREV_gobject-sys = "5f35e26c65d24f8f018f643218de0f5807ba5f01"
+
+SRC_URI[anes.sha256sum] = "4b46cbb362ab8752921c97e041f5e366ee6297bd428a31275b9fcf1e380f7299"
+SRC_URI[anyhow.sha256sum] = "7de8ce5e0f9f8d88245311066a578d72b7af3e7088f32783804676302df237e4"
+SRC_URI[arrayvec.sha256sum] = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"
+SRC_URI[atty.sha256sum] = "d9b39be18770d11421cdb1b9947a45dd3f37e93092cbf377614828a319d5fee8"
+SRC_URI[autocfg.sha256sum] = "d468802bab17cbc0cc575e9b053f41e72aa36bfa6b7f55e3529ffa43161b97fa"
+SRC_URI[bitflags.sha256sum] = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
+SRC_URI[bumpalo.sha256sum] = "0d261e256854913907f67ed06efbc3338dfe6179796deefc1ff763fc1aee5535"
+SRC_URI[byteorder.sha256sum] = "14c189c53d098945499cdfa7ecc63567cf3886b3332b312a5b4585d8d3a6a610"
+SRC_URI[cast.sha256sum] = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5"
+SRC_URI[cfg-if.sha256sum] = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
+SRC_URI[chrono.sha256sum] = "4e3c5919066adf22df73762e50cffcde3a758f2a848b113b586d1f86728b673b"
+SRC_URI[ciborium.sha256sum] = "b0c137568cc60b904a7724001b35ce2630fd00d5d84805fbb608ab89509d788f"
+SRC_URI[ciborium-io.sha256sum] = "346de753af073cc87b52b2083a506b38ac176a44cfb05497b622e27be899b369"
+SRC_URI[ciborium-ll.sha256sum] = "213030a2b5a4e0c0892b6652260cf6ccac84827b83a85a534e178e3906c4cf1b"
+SRC_URI[clap.sha256sum] = "71655c45cb9845d3270c9d6df84ebe72b4dad3c2ba3f7023ad47c144e4e473a5"
+SRC_URI[clap_lex.sha256sum] = "2850f2f5a82cbf437dd5af4d49848fbdfc27c157c3d010345776f952765261c5"
+SRC_URI[criterion.sha256sum] = "e7c76e09c1aae2bc52b3d2f29e13c6572553b30c4aa1b8a49fd70de6412654cb"
+SRC_URI[criterion-plot.sha256sum] = "6b50826342786a51a89e2da3a28f1c32b06e387201bc2d19791f622c673706b1"
+SRC_URI[crossbeam-channel.sha256sum] = "cf2b3e8478797446514c91ef04bafcb59faba183e621ad488df88983cc14128c"
+SRC_URI[crossbeam-deque.sha256sum] = "ce6fd6f855243022dcecf8702fef0c297d4338e226845fe067f6341ad9fa0cef"
+SRC_URI[crossbeam-epoch.sha256sum] = "46bd5f3f85273295a9d14aedfb86f6aadbff6d8f5295c4a9edb08e819dcf5695"
+SRC_URI[crossbeam-utils.sha256sum] = "3c063cd8cc95f5c377ed0d4b49a4b21f632396ff690e8470c29b3359b346984b"
+SRC_URI[either.sha256sum] = "7fcaabb2fef8c910e7f4c7ce9f67a1283a1715879a7c230ca9d6d1ae31f16d91"
+SRC_URI[enumflags2.sha256sum] = "e75d4cd21b95383444831539909fbb14b9dc3fdceb2a6f5d36577329a1f55ccb"
+SRC_URI[enumflags2_derive.sha256sum] = "f58dc3c5e468259f19f2d46304a6b28f1c3d034442e14b322d2b850e36f6d5ae"
+SRC_URI[form_urlencoded.sha256sum] = "a9c384f161156f5260c24a097c56119f9be8c798586aecc13afbcbe7b7e26bf8"
+SRC_URI[futures-channel.sha256sum] = "164713a5a0dcc3e7b4b1ed7d3b433cabc18025386f9339346e8daf15963cf7ac"
+SRC_URI[futures-core.sha256sum] = "86d7a0c1aa76363dac491de0ee99faf6941128376f1cf96f07db7603b7de69dd"
+SRC_URI[futures-executor.sha256sum] = "1997dd9df74cdac935c76252744c1ed5794fac083242ea4fe77ef3ed60ba0f83"
+SRC_URI[futures-macro.sha256sum] = "3eb14ed937631bd8b8b8977f2c198443447a8355b6e3ca599f38c975e5a963b6"
+SRC_URI[futures-task.sha256sum] = "fd65540d33b37b16542a0438c12e6aeead10d4ac5d05bd3f805b8f35ab592879"
+SRC_URI[futures-util.sha256sum] = "3ef6b17e481503ec85211fed8f39d1970f128935ca1f814cd32ac4a6842e84ab"
+SRC_URI[getrandom.sha256sum] = "c05aeb6a22b8f62540c194aac980f2115af067bfe15a0734d7277a768d396b31"
+SRC_URI[half.sha256sum] = "eabb4a44450da02c90444cf74558da904edde8fb4e9035a9a6a4e15445af0bd7"
+SRC_URI[hashbrown.sha256sum] = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888"
+SRC_URI[heck.sha256sum] = "6d621efb26863f0e9924c6ac577e8275e5e6b77455db64ffa6c65c904e9e132c"
+SRC_URI[hermit-abi-0.1.19.sha256sum] = "62b467343b94ba476dcb2500d242dadbb39557df889310ac77c5d99100aaac33"
+SRC_URI[hermit-abi-0.2.6.sha256sum] = "ee512640fe35acbfb4bb779db6f0d80704c2cacfa2e39b601ef3e3f47d1ae4c7"
+SRC_URI[idna.sha256sum] = "e14ddfc70884202db2244c223200c204c2bda1bc6e0998d11b5e024d657209e6"
+SRC_URI[indexmap.sha256sum] = "1885e79c1fc4b10f0e172c475f458b7f7b93061064d98c3293e98c5ba0c8b399"
+SRC_URI[itertools-0.9.0.sha256sum] = "284f18f85651fe11e8a991b2adb42cb078325c996ed026d994719efcfca1d54b"
+SRC_URI[itertools-0.10.5.sha256sum] = "b0fd2260e829bddf4cb6ea802289de2f86d6a7a690192fbe91b3f46e0f2c8473"
+SRC_URI[itoa.sha256sum] = "453ad9f582a441959e5f0d088b02ce04cfe8d51a8eaf077f12ac6d3e94164ca6"
+SRC_URI[js-sys.sha256sum] = "445dde2150c55e483f3d8416706b97ec8e8237c307e5b7b4b8dd15e6af2a0730"
+SRC_URI[lazy_static.sha256sum] = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"
+SRC_URI[libc.sha256sum] = "99227334921fae1a979cf0bfdfcc6b3e5ce376ef57e16fb6fb3ea2ed6095f80c"
+SRC_URI[log.sha256sum] = "abb12e687cfb44aa40f41fc3978ef76448f9b6038cad6aef4259d3c095a2382e"
+SRC_URI[memchr.sha256sum] = "2dffe52ecf27772e601905b7522cb4ef790d2cc203488bbd0e2fe85fcb74566d"
+SRC_URI[memoffset.sha256sum] = "d61c719bcfbcf5d62b3a09efa6088de8c54bc0bfcd3ea7ae39fcc186108b8de1"
+SRC_URI[num-integer.sha256sum] = "225d3389fb3509a24c93f5c29eb6bde2586b98d9f016636dff58d7c6f7569cd9"
+SRC_URI[num-traits.sha256sum] = "578ede34cf02f8924ab9447f50c28075b4d3e5b269972345e7e0372b38c6cdcd"
+SRC_URI[num_cpus.sha256sum] = "0fac9e2da13b5eb447a6ce3d392f23a29d8694bff781bf03a16cd9ac8697593b"
+SRC_URI[once_cell.sha256sum] = "b7e5500299e16ebb147ae15a00a942af264cf3688f47923b8fc2cd5858f23ad3"
+SRC_URI[oorandom.sha256sum] = "0ab1bc2a289d34bd04a330323ac98a1b4bc82c9d9fcb1e66b63caa84da26b575"
+SRC_URI[os_str_bytes.sha256sum] = "ceedf44fb00f2d1984b0bc98102627ce622e083e49a5bacdb3e514fa4238e267"
+SRC_URI[percent-encoding.sha256sum] = "478c572c3d73181ff3c2539045f6eb99e5491218eae919370993b890cdbdd98e"
+SRC_URI[pin-project-lite.sha256sum] = "e0a7ae3ac2f1173085d398531c705756c94a4c56843785df85a60c1a0afac116"
+SRC_URI[pin-utils.sha256sum] = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184"
+SRC_URI[pkg-config.sha256sum] = "6ac9a59f73473f1b8d852421e59e64809f025994837ef743615c6d0c5b305160"
+SRC_URI[plotters.sha256sum] = "2538b639e642295546c50fcd545198c9d64ee2a38620a628724a3b266d5fbf97"
+SRC_URI[plotters-backend.sha256sum] = "193228616381fecdc1224c62e96946dfbc73ff4384fba576e052ff8c1bea8142"
+SRC_URI[plotters-svg.sha256sum] = "f9a81d2759aae1dae668f783c308bc5c8ebd191ff4184aaa1b37f65a6ae5a56f"
+SRC_URI[ppv-lite86.sha256sum] = "5b40af805b3121feab8a3c29f04d8ad262fa8e0561883e7653e024ae4479e6de"
+SRC_URI[proc-macro-crate-0.1.5.sha256sum] = "1d6ea3c4595b96363c13943497db34af4460fb474a95c43f4446ad341b8c9785"
+SRC_URI[proc-macro-crate-1.3.1.sha256sum] = "7f4c021e1093a56626774e81216a4ce732a735e5bad4868a03f3ed65ca0c3919"
+SRC_URI[proc-macro-error.sha256sum] = "da25490ff9892aab3fcf7c36f08cfb902dd3e71ca0f9f9517bea02a73a5ce38c"
+SRC_URI[proc-macro-error-attr.sha256sum] = "a1be40180e52ecc98ad80b184934baf3d0d29f979574e439af5a55274b35f869"
+SRC_URI[proc-macro2.sha256sum] = "ba466839c78239c09faf015484e5cc04860f88242cff4d03eb038f04b4699b73"
+SRC_URI[quote.sha256sum] = "4424af4bf778aae2051a77b60283332f386554255d722233d09fbfc7e30da2fc"
+SRC_URI[rand.sha256sum] = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404"
+SRC_URI[rand_chacha.sha256sum] = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
+SRC_URI[rand_core.sha256sum] = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
+SRC_URI[rayon.sha256sum] = "1d2df5196e37bcc87abebc0053e20787d73847bb33134a69841207dd0a47f03b"
+SRC_URI[rayon-core.sha256sum] = "4b8f95bd6966f5c87776639160a66bd8ab9895d9d4ab01ddba9fc60661aebe8d"
+SRC_URI[regex.sha256sum] = "cce168fea28d3e05f158bda4576cf0c844d5045bc2cc3620fa0292ed5bb5814c"
+SRC_URI[regex-syntax.sha256sum] = "f162c6dd7b008981e4d40210aca20b4bd0f9b60ca9271061b07f78537722f2e1"
+SRC_URI[ryu.sha256sum] = "f91339c0467de62360649f8d3e185ca8de4224ff281f66000de5eb2a77a79041"
+SRC_URI[same-file.sha256sum] = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502"
+SRC_URI[scopeguard.sha256sum] = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd"
+SRC_URI[serde.sha256sum] = "771d4d9c4163ee138805e12c710dd365e4f44be8be0503cb1bb9eb989425d9c9"
+SRC_URI[serde_bytes.sha256sum] = "416bda436f9aab92e02c8e10d49a15ddd339cea90b6e340fe51ed97abb548294"
+SRC_URI[serde_derive.sha256sum] = "e801c1712f48475582b7696ac71e0ca34ebb30e09338425384269d9717c62cad"
+SRC_URI[serde_json.sha256sum] = "1c533a59c9d8a93a09c6ab31f0fd5e5f4dd1b8fc9434804029839884765d04ea"
+SRC_URI[serde_repr.sha256sum] = "bcec881020c684085e55a25f7fd888954d56609ef363479dc5a1305eb0d40cab"
+SRC_URI[slab.sha256sum] = "6528351c9bc8ab22353f9d776db39a20288e8d6c37ef8cfe3317cf875eecfc2d"
+SRC_URI[static_assertions.sha256sum] = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"
+SRC_URI[strum.sha256sum] = "57bd81eb48f4c437cadc685403cad539345bf703d78e63707418431cecd4522b"
+SRC_URI[strum_macros.sha256sum] = "87c85aa3f8ea653bfd3ddf25f7ee357ee4d204731f6aa9ad04002306f6e2774c"
+SRC_URI[syn-1.0.109.sha256sum] = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237"
+SRC_URI[syn-2.0.8.sha256sum] = "bcc02725fd69ab9f26eab07fad303e2497fad6fb9eba4f96c4d1687bdf704ad9"
+SRC_URI[system-deps.sha256sum] = "0f3ecc17269a19353b3558b313bba738b25d82993e30d62a18406a24aba4649b"
+SRC_URI[textwrap.sha256sum] = "222a222a5bfe1bba4a77b45ec488a741b3cb8872e5e499451fd7d0129c9c7c3d"
+SRC_URI[thiserror.sha256sum] = "978c9a314bd8dc99be594bc3c175faaa9794be04a5a5e153caba6915336cebac"
+SRC_URI[thiserror-impl.sha256sum] = "f9456a42c5b0d803c8cd86e73dd7cc9edd429499f37a3550d286d5e86720569f"
+SRC_URI[time.sha256sum] = "cd0cbfecb4d19b5ea75bb31ad904eb5b9fa13f21079c3b92017ebdf4999a5890"
+SRC_URI[time-core.sha256sum] = "2e153e1f1acaef8acc537e68b44906d2db6436e2b35ac2c6b42640fff91f00fd"
+SRC_URI[time-macros.sha256sum] = "fd80a657e71da814b8e5d60d3374fc6d35045062245d80224748ae522dd76f36"
+SRC_URI[tinytemplate.sha256sum] = "be4d6b5f19ff7664e8c98d03e2139cb510db9b0a60b55f8e8709b689d939b6bc"
+SRC_URI[tinyvec.sha256sum] = "87cc5ceb3875bb20c2890005a4e226a4651264a5c75edb2421b52861a0a0cb50"
+SRC_URI[tinyvec_macros.sha256sum] = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
+SRC_URI[toml.sha256sum] = "f4f7f0dd8d50a853a531c426359045b1998f04219d88799810762cd4ad314234"
+SRC_URI[toml_datetime.sha256sum] = "3ab8ed2edee10b50132aed5f331333428b011c99402b5a534154ed15746f9622"
+SRC_URI[toml_edit.sha256sum] = "239410c8609e8125456927e6707163a3b1fdb40561e4b803bc041f466ccfdc13"
+SRC_URI[unicode-bidi.sha256sum] = "92888ba5573ff080736b3648696b70cafad7d250551175acbaa4e0385b3e1460"
+SRC_URI[unicode-ident.sha256sum] = "e5464a87b239f13a63a501f2701565754bae92d243d4bb7eb12f6d57d2269bf4"
+SRC_URI[unicode-normalization.sha256sum] = "5c5713f0fc4b5db668a2ac63cdb7bb4469d8c9fed047b1d0292cc7b0ce2ba921"
+SRC_URI[unicode-segmentation.sha256sum] = "1dd624098567895118886609431a7c3b8f516e41d30e0643f03d94592a147e36"
+SRC_URI[url.sha256sum] = "0d68c799ae75762b8c3fe375feb6600ef5602c883c5d21eb51c09f22b83c4643"
+SRC_URI[uuid.sha256sum] = "1674845326ee10d37ca60470760d4288a6f80f304007d92e5c53bab78c9cfd79"
+SRC_URI[version-compare.sha256sum] = "d63556a25bae6ea31b52e640d7c41d1ab27faba4ccb600013837a3d0b3994ca1"
+SRC_URI[version_check.sha256sum] = "49874b5167b65d7193b8aba1567f5c7d93d001cafc34600cee003eda787e483f"
+SRC_URI[walkdir.sha256sum] = "36df944cda56c7d8d8b7496af378e6b16de9284591917d307c9b4d313c44e698"
+SRC_URI[wasi.sha256sum] = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423"
+SRC_URI[wasm-bindgen.sha256sum] = "31f8dcbc21f30d9b8f2ea926ecb58f6b91192c17e9d33594b3df58b2007ca53b"
+SRC_URI[wasm-bindgen-backend.sha256sum] = "95ce90fd5bcc06af55a641a86428ee4229e44e07033963a2290a8e241607ccb9"
+SRC_URI[wasm-bindgen-macro.sha256sum] = "4c21f77c0bedc37fd5dc21f897894a5ca01e7bb159884559461862ae90c0b4c5"
+SRC_URI[wasm-bindgen-macro-support.sha256sum] = "2aff81306fcac3c7515ad4e177f521b5c9a15f2b08f4e32d823066102f35a5f6"
+SRC_URI[wasm-bindgen-shared.sha256sum] = "0046fef7e28c3804e5e38bfa31ea2a0f73905319b677e57ebe37e49358989b5d"
+SRC_URI[web-sys.sha256sum] = "e33b99f4b23ba3eec1a53ac264e35a755f00e966e0065077d6027c0f575b0b97"
+SRC_URI[winapi.sha256sum] = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
+SRC_URI[winapi-i686-pc-windows-gnu.sha256sum] = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
+SRC_URI[winapi-util.sha256sum] = "70ec6ce85bb158151cae5e5c87f95a8e97d2c0c4b001223f33a334e3ce5de178"
+SRC_URI[winapi-x86_64-pc-windows-gnu.sha256sum] = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
+SRC_URI[winnow.sha256sum] = "deac0939bd6e4f24ab5919fbf751c97a8cfc8543bb083a305ed5c0c10bb241d1"
+SRC_URI[zvariant_derive.sha256sum] = "34c20260af4b28b3275d6676c7e2a6be0d4332e8e0aba4616d34007fd84e462a"
+SRC_URI[zvariant_utils.sha256sum] = "53b22993dbc4d128a17a3b6c92f1c63872dd67198537ee728d8b5d7c40640a8b"
meta-selftest/recipes-extended/zvariant/zvariant/0001-Tweak-zvariant-crate-config.patch new file +1292 -0

1,292 lines not on this page · raw message

meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb new file +26 -0
new file mode 100644
index 0000000000..4fe327ae07
@@ -0,0 +1,26 @@
+SUMMARY = "Provides API for encoding/decoding of data to/from D-Bus wire format"
+DESCRIPTION = "This crate provides API for encoding/decoding of data to/from D-Bus wire format.\
+This binary wire format is simple and very efficient and hence useful outside of D-Bus context as well.\
+A modified form of this format, GVariant is very commonly used for efficient storage of arbitrary \
+data and is also supported by this crate."
+HOMEPAGE = "https://gitlab.freedesktop.org/dbus/zbus/"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=b377b220f43d747efdec40d69fcaa69d"
+
+SRC_URI = " \
+    git://gitlab.freedesktop.org/dbus/zbus;protocol=https;branch=main;subpath=zvariant \
+    file://0001-Tweak-zvariant-crate-config.patch;striplevel=2 \
+"
+
+SRCREV = "07506776fab5f58e029760bb4b288f670c7eecd6"
+S = "${WORKDIR}/zvariant"
+
+python do_clean_lic_file_symlink() {
+    bb.utils.remove("LICENCE")
+}
+
+addtask clean_lic_file_symlink after do_unpack before do_patch
+
+inherit cargo cargo-update-recipe-crates
+
+require ${BPN}-crates.inc
meta/lib/oeqa/selftest/cases/devtool.py +93 -0

99 lines not on this page · raw message

Signature · 1 line
2.34.1
#6 lore raw

PATCH v5 6/6 cargo-update-recipe-crates: don't walk on the whole dir

In reply to [email protected]

From: Frederic Martinsons <[email protected]>

There is no need to do such things, Cargo.lock file has to be at the root of CARGO_LOCK_SRC_DIR.
This avoid finding other possible Cargo.lock that would be in subdir (for example if a patch is applied on the recipe, we can have .pc subdir in S and a Cargo.lock can be there)

---
 .../cargo-update-recipe-crates.bbclass               | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
meta/classes-recipe/cargo-update-recipe-crates.bbclass +8 -4
index daa363b0dd..549cfe627e 100644
@@ -68,10 +68,14 @@ def get_crates(f):
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
 found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
+for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):
+    if file == 'Cargo.lock':
+        try:
+            cargo_lock_path = os.path.join('${CARGO_LOCK_SRC_DIR}', file)
+            crates += get_crates(cargo_lock_path)
+        except Exception as e:
+            raise ValueError("Cannot parse '%s'" % cargo_lock_path) from e
+        else:
             found = True
 if not found:
     raise ValueError("Unable to find Cargo.lock in ${CARGO_LOCK_SRC_DIR}")
Signature · 1 line
2.34.1
#7 lore raw

Re: [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

Quoted 28 lines · On Thu, Mar 30, 2023 at 4:45 PM <[email protected]> wrote:

From: Frederic Martinsons <[email protected]>

There is no need to do such things, Cargo.lock file has to be at the root of CARGO_LOCK_SRC_DIR.
This avoid finding other possible Cargo.lock that would be in subdir (for example if a patch is applied on the recipe, we can have .pc subdir in S and a Cargo.lock can be there)

---
 .../cargo-update-recipe-crates.bbclass               | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -4
index daa363b0dd..549cfe627e 100644
@@ -68,10 +68,14 @@ def get_crates(f):
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
 found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
+for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 81

Quoted 10 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +7 -0
+    if file == 'Cargo.lock':
+        try:
+            cargo_lock_path = os.path.join('${CARGO_LOCK_SRC_DIR}', file)
+            crates += get_crates(cargo_lock_path)
+        except Exception as e:
+            raise ValueError("Cannot parse '%s'" % cargo_lock_path) from e
+        else:
             found = True
 if not found:
     raise ValueError("Unable to find Cargo.lock in ${CARGO_LOCK_SRC_DIR}")

Isn't this just a long-winded version of something like this (completely untested):

(quoted diff) +0 -0

b/meta/classes-recipe/cargo-update-recipe-crates.bbclass index daa363b0dd65..22ddcfa5c1e3 100644

meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -8
@@ -67,14 +67,7 @@ def get_crates(f):
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
-            found = True
-if not found:
-    raise ValueError("Unable to find Cargo.lock in ${CARGO_LOCK_SRC_DIR}")
+crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}", "Cargo.lock"))
 open("${TARGET_FILE}", 'w').write(crates)
 EOF
 
Signature · 1 line
Alex Kiernan
#8 lore raw

Re: [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

Pretty much indeed, except for the exception catching part (which allows a better error message)
Do I need to submit another whole series (PATCH V6) or can I submit just this change ?

Quoted 71 lines · Alex Kiernan
Quoted 10 lines · On Thu, Mar 30, 2023 at 4:45 PM <[email protected]> wrote:

From: Frederic Martinsons <[email protected]>

There is no need to do such things, Cargo.lock file has to be at the root of CARGO_LOCK_SRC_DIR.
This avoid finding other possible Cargo.lock that would be in subdir (for example if a patch is applied on the recipe, we can have .pc subdir in S and a Cargo.lock can be there)

Review

Quoted 6 lines
---
 .../cargo-update-recipe-crates.bbclass               | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
(quoted diff) +0 -0

b/meta/classes-recipe/cargo-update-recipe-crates.bbclass

Quoted 5 lines

index daa363b0dd..549cfe627e 100644

meta/classes-recipe/cargo-update-recipe-crates.bbclass +0 -0
@@ -68,10 +68,14 @@ def get_crates(f):
 import os

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 74

Quoted 10 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +4 -4
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
 found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
+for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):
+    if file == 'Cargo.lock':
+        try:
+            cargo_lock_path = os.path.join('${CARGO_LOCK_SRC_DIR}',

file)

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 77

Quoted 3 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +3 -0
+            crates += get_crates(cargo_lock_path)
+        except Exception as e:
+            raise ValueError("Cannot parse '%s'" % cargo_lock_path)

from e

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 81

Quoted 4 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -0
+        else:
             found = True
 if not found:
     raise ValueError("Unable to find Cargo.lock in

${CARGO_LOCK_SRC_DIR}")

Isn't this just a long-winded version of something like this (completely untested):

meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -9
index daa363b0dd65..22ddcfa5c1e3 100644
@@ -67,14 +67,7 @@ def get_crates(f):
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
-            found = True
-if not found:
-    raise ValueError("Unable to find Cargo.lock in ${CARGO_LOCK_SRC_DIR}")
+crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}", "Cargo.lock"))
 open("${TARGET_FILE}", 'w').write(crates)
 EOF
 
--

Alex Kiernan

#10 lore raw

Re: [OE-core] [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

In reply to a message not in the archive

Nevermind, I'll submit a new v6 series tomorrow.

Your comment is totally accurate and follow the PEP20 which I should follow too ("readability counts" and "explicit is better than implicit" at most )

Quoted 91 lines · Frederic Martinsons via lists.openembedded.org

Pretty much indeed, except for the exception catching part (which allows a better error message)
Do I need to submit another whole series (PATCH V6) or can I submit just this change ?

Quoted 70 lines · Alex Kiernan
Quoted 10 lines · On Thu, Mar 30, 2023 at 4:45 PM <[email protected]> wrote:

From: Frederic Martinsons <[email protected]>

There is no need to do such things, Cargo.lock file has to be at the root of CARGO_LOCK_SRC_DIR.
This avoid finding other possible Cargo.lock that would be in subdir (for example if a patch is applied on the recipe, we can have .pc subdir in S and a Cargo.lock can be there)

Review

Quoted 6 lines
---
 .../cargo-update-recipe-crates.bbclass               | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
(quoted diff) +0 -0

b/meta/classes-recipe/cargo-update-recipe-crates.bbclass

Quoted 5 lines

index daa363b0dd..549cfe627e 100644

meta/classes-recipe/cargo-update-recipe-crates.bbclass +0 -0
@@ -68,10 +68,14 @@ def get_crates(f):
 import os

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 74

Quoted 10 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +4 -4
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
 found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
+for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):
+    if file == 'Cargo.lock':
+        try:
+            cargo_lock_path = os.path.join('${CARGO_LOCK_SRC_DIR}',

file)

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 77

Quoted 3 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +3 -0
+            crates += get_crates(cargo_lock_path)
+        except Exception as e:
+            raise ValueError("Cannot parse '%s'" % cargo_lock_path)

from e

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 81

Quoted 4 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -0
+        else:
             found = True
 if not found:
     raise ValueError("Unable to find Cargo.lock in

${CARGO_LOCK_SRC_DIR}")

Isn't this just a long-winded version of something like this (completely untested):

meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -9
index daa363b0dd65..22ddcfa5c1e3 100644
@@ -67,14 +67,7 @@ def get_crates(f):
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
-            found = True
-if not found:
-    raise ValueError("Unable to find Cargo.lock in
 ${CARGO_LOCK_SRC_DIR}")
+crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}", "Cargo.lock"))
 open("${TARGET_FILE}", 'w').write(crates)
 EOF
 
--
Quoted 2 lines

Alex Kiernan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#179346):
https://lists.openembedded.org/g/openembedded-core/message/179346
Mute This Topic: https://lists.openembedded.org/mt/97953740/6213388
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#11 lore raw

Re: [OE-core] [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

In reply to Alex Kiernan

I don't remember the exact details now, but when I was working on updating solana recipes to use this
https://github.com/webosose/meta-webosose/commit/9bdfae7988077d0211eeee79cc339d0770cd86b4

the S was pointing to just some subdirectory and multiple Cargo.locks files were parsed in other directories as well, that's why I was adding
https://git.openembedded.org/openembedded-core/commit/meta/classes-recipe/cargo-update-recipe-crates.bbclass?id=7636a2b8080521ed2ad54b0edce47a8742a12d58

in the end I've changed to use the root directory in S and just set CARGO_SRC_DIR to right subdirectory to build.

I don't have much experience with crates other than these solana recipes, but isn't there some valid use-case for multiple Cargo.locks? I assume Alex in original implementation didn't use the os.walk just to make it more complicated :).

And FWIW when trying to regenerate these .inc files with current master and with:

$ bitbake -c update_crates solana-keygen
it fails with:
ERROR: solana-keygen-1.14.5-r0 do_fetch: No checksum specified for
'/OE/lge/build/webos/mickledore/downloads/Inflector-0.11.4.crate', please
add at least one to the recipe:
SRC_URI[Inflector.sha256sum] =
"fe438c63458706e03479442743baae6c88256498e6431708f6dfc520a26515d3"
ERROR: solana-keygen-1.14.5-r0 do_fetch: Bitbake Fetcher Error:
NoChecksumError('Missing SRC_URI checksum', '
https://crates.io/api/v1/crates/Inflector/0.11.4/download')

Isn't it an chicken&egg issue now when do_fetch enforces checksums to be used? Or am I just missing some of the pending changes or just using it incorrectly?

Quoted 82 lines · Alex Kiernan
Quoted 10 lines · On Thu, Mar 30, 2023 at 4:45 PM <[email protected]> wrote:

From: Frederic Martinsons <[email protected]>

There is no need to do such things, Cargo.lock file has to be at the root of CARGO_LOCK_SRC_DIR.
This avoid finding other possible Cargo.lock that would be in subdir (for example if a patch is applied on the recipe, we can have .pc subdir in S and a Cargo.lock can be there)

Review

Quoted 6 lines
---
 .../cargo-update-recipe-crates.bbclass               | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
(quoted diff) +0 -0

b/meta/classes-recipe/cargo-update-recipe-crates.bbclass

Quoted 5 lines

index daa363b0dd..549cfe627e 100644

meta/classes-recipe/cargo-update-recipe-crates.bbclass +0 -0
@@ -68,10 +68,14 @@ def get_crates(f):
 import os

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 74

Quoted 10 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +4 -4
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
 found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
+for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):
+    if file == 'Cargo.lock':
+        try:
+            cargo_lock_path = os.path.join('${CARGO_LOCK_SRC_DIR}',

file)

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 77

Quoted 3 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +3 -0
+            crates += get_crates(cargo_lock_path)
+        except Exception as e:
+            raise ValueError("Cannot parse '%s'" % cargo_lock_path)

from e

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 81

Quoted 4 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -0
+        else:
             found = True
 if not found:
     raise ValueError("Unable to find Cargo.lock in

${CARGO_LOCK_SRC_DIR}")

Isn't this just a long-winded version of something like this (completely untested):

meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -9
index daa363b0dd65..22ddcfa5c1e3 100644
@@ -67,14 +67,7 @@ def get_crates(f):
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
-            found = True
-if not found:
-    raise ValueError("Unable to find Cargo.lock in ${CARGO_LOCK_SRC_DIR}")
+crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}", "Cargo.lock"))
 open("${TARGET_FILE}", 'w').write(crates)
 EOF
 
--

Alex Kiernan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#179344):
https://lists.openembedded.org/g/openembedded-core/message/179344
Mute This Topic: https://lists.openembedded.org/mt/97953740/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#12 lore raw

Re: [OE-core] [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

Well I see what you mean, I'll take a look at your example to try to find out if multiple Cargo.lock could be expected.

And for your second remark, yes, there is a chicken and eggs issue for updating crates checksum from scratch.
You didn't miss anything, I came across this when updating crates checksum that was "pristine"
and I only manage to execute update_crates by locally patching
bitbake/lib/bb/fetch2/crate.py for
recommends_checksum method to return False .
I'm aware that is not ideal but I don't know how to make this better (maybe make do_update_crates
after do_fetch instead of after do_patch ? )

Quoted 122 lines · Martin Jansa

I don't remember the exact details now, but when I was working on updating solana recipes to use this

https://github.com/webosose/meta-webosose/commit/9bdfae7988077d0211eeee79cc339d0770cd86b4

the S was pointing to just some subdirectory and multiple Cargo.locks files were parsed in other directories as well, that's why I was adding

https://git.openembedded.org/openembedded-core/commit/meta/classes-recipe/cargo-update-recipe-crates.bbclass?id=7636a2b8080521ed2ad54b0edce47a8742a12d58

in the end I've changed to use the root directory in S and just set CARGO_SRC_DIR to right subdirectory to build.

I don't have much experience with crates other than these solana recipes, but isn't there some valid use-case for multiple Cargo.locks? I assume Alex in original implementation didn't use the os.walk just to make it more complicated :).

And FWIW when trying to regenerate these .inc files with current master and with:

$ bitbake -c update_crates solana-keygen
it fails with:
ERROR: solana-keygen-1.14.5-r0 do_fetch: No checksum specified for
'/OE/lge/build/webos/mickledore/downloads/Inflector-0.11.4.crate', please
add at least one to the recipe:
SRC_URI[Inflector.sha256sum] =
"fe438c63458706e03479442743baae6c88256498e6431708f6dfc520a26515d3"
ERROR: solana-keygen-1.14.5-r0 do_fetch: Bitbake Fetcher Error:
NoChecksumError('Missing SRC_URI checksum', '
https://crates.io/api/v1/crates/Inflector/0.11.4/download')

Isn't it an chicken&egg issue now when do_fetch enforces checksums to be used? Or am I just missing some of the pending changes or just using it incorrectly?

Quoted 83 lines · Alex Kiernan
Quoted 10 lines · On Thu, Mar 30, 2023 at 4:45 PM <[email protected]> wrote:

From: Frederic Martinsons <[email protected]>

There is no need to do such things, Cargo.lock file has to be at the root of CARGO_LOCK_SRC_DIR.
This avoid finding other possible Cargo.lock that would be in subdir (for example if a patch is applied on the recipe, we can have .pc subdir in S and a Cargo.lock can be there)

Review

Quoted 6 lines
---
 .../cargo-update-recipe-crates.bbclass               | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
(quoted diff) +0 -0

b/meta/classes-recipe/cargo-update-recipe-crates.bbclass

Quoted 5 lines

index daa363b0dd..549cfe627e 100644

meta/classes-recipe/cargo-update-recipe-crates.bbclass +0 -0
@@ -68,10 +68,14 @@ def get_crates(f):
 import os

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 74

Quoted 10 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +4 -4
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
 found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
+for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):
+    if file == 'Cargo.lock':
+        try:
+            cargo_lock_path = os.path.join('${CARGO_LOCK_SRC_DIR}',

file)

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 77

Quoted 3 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +3 -0
+            crates += get_crates(cargo_lock_path)
+        except Exception as e:
+            raise ValueError("Cannot parse '%s'" % cargo_lock_path)

from e

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 81

Quoted 4 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -0
+        else:
             found = True
 if not found:
     raise ValueError("Unable to find Cargo.lock in

${CARGO_LOCK_SRC_DIR}")

Isn't this just a long-winded version of something like this (completely untested):

meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -9
index daa363b0dd65..22ddcfa5c1e3 100644
@@ -67,14 +67,7 @@ def get_crates(f):
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
-            found = True
-if not found:
-    raise ValueError("Unable to find Cargo.lock in
 ${CARGO_LOCK_SRC_DIR}")
+crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}", "Cargo.lock"))
 open("${TARGET_FILE}", 'w').write(crates)
 EOF
 
--

Alex Kiernan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#179344):
https://lists.openembedded.org/g/openembedded-core/message/179344
Mute This Topic: https://lists.openembedded.org/mt/97953740/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#13 lore raw

Re: [OE-core] [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

By the way, about this "chickens and eggs" problem, isn't it the same for a regular recipe you just upgraded?

Personnaly, when I update a recipe, I let bitbake tell me what is the new checksum expected and put it the recipe (as the error message says).

But I'm aware this is not exactly the same, since a cargo based recipe could contain a ton of crate:// uri and if you apply this method, you have to copy the new checksum one by one, bitbake error after another...

Don't know how to make this better and I plead guilty for not mentioning that in a dedicated commit message.

Of course, if someone come up with a smoothier way of doing this, I'll make a new patch.

Quoted 141 lines · Frédéric Martinsons

Well I see what you mean, I'll take a look at your example to try to find out if multiple Cargo.lock could be expected.

And for your second remark, yes, there is a chicken and eggs issue for updating crates checksum from scratch.
You didn't miss anything, I came across this when updating crates checksum that was "pristine"
and I only manage to execute update_crates by locally patching
bitbake/lib/bb/fetch2/crate.py for
recommends_checksum method to return False .
I'm aware that is not ideal but I don't know how to make this better (maybe make do_update_crates
after do_fetch instead of after do_patch ? )

Le jeu. 30 mars 2023, 21:23, Martin Jansa <[email protected]> a écrit :

Quoted 123 lines

I don't remember the exact details now, but when I was working on updating solana recipes to use this

https://github.com/webosose/meta-webosose/commit/9bdfae7988077d0211eeee79cc339d0770cd86b4

the S was pointing to just some subdirectory and multiple Cargo.locks files were parsed in other directories as well, that's why I was adding

https://git.openembedded.org/openembedded-core/commit/meta/classes-recipe/cargo-update-recipe-crates.bbclass?id=7636a2b8080521ed2ad54b0edce47a8742a12d58

in the end I've changed to use the root directory in S and just set CARGO_SRC_DIR to right subdirectory to build.

I don't have much experience with crates other than these solana recipes, but isn't there some valid use-case for multiple Cargo.locks? I assume Alex in original implementation didn't use the os.walk just to make it more complicated :).

And FWIW when trying to regenerate these .inc files with current master and with:

$ bitbake -c update_crates solana-keygen
it fails with:
ERROR: solana-keygen-1.14.5-r0 do_fetch: No checksum specified for
'/OE/lge/build/webos/mickledore/downloads/Inflector-0.11.4.crate', please
add at least one to the recipe:
SRC_URI[Inflector.sha256sum] =
"fe438c63458706e03479442743baae6c88256498e6431708f6dfc520a26515d3"
ERROR: solana-keygen-1.14.5-r0 do_fetch: Bitbake Fetcher Error:
NoChecksumError('Missing SRC_URI checksum', '
https://crates.io/api/v1/crates/Inflector/0.11.4/download')

Isn't it an chicken&egg issue now when do_fetch enforces checksums to be used? Or am I just missing some of the pending changes or just using it incorrectly?

Quoted 84 lines · Alex Kiernan
Quoted 10 lines · On Thu, Mar 30, 2023 at 4:45 PM <[email protected]> wrote:

From: Frederic Martinsons <[email protected]>

There is no need to do such things, Cargo.lock file has to be at the root of CARGO_LOCK_SRC_DIR.
This avoid finding other possible Cargo.lock that would be in subdir (for example if a patch is applied on the recipe, we can have .pc subdir in S and a Cargo.lock can be there)

Review

Quoted 6 lines
---
 .../cargo-update-recipe-crates.bbclass               | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
(quoted diff) +0 -0

b/meta/classes-recipe/cargo-update-recipe-crates.bbclass

Quoted 5 lines

index daa363b0dd..549cfe627e 100644

meta/classes-recipe/cargo-update-recipe-crates.bbclass +0 -0
@@ -68,10 +68,14 @@ def get_crates(f):
 import os

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 74

Quoted 10 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +4 -4
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
 found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
+for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):
+    if file == 'Cargo.lock':
+        try:
+            cargo_lock_path = os.path.join('${CARGO_LOCK_SRC_DIR}',

file)

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 77

Quoted 3 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +3 -0
+            crates += get_crates(cargo_lock_path)
+        except Exception as e:
+            raise ValueError("Cannot parse '%s'" % cargo_lock_path)

from e

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 81

Quoted 4 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -0
+        else:
             found = True
 if not found:
     raise ValueError("Unable to find Cargo.lock in

${CARGO_LOCK_SRC_DIR}")

Isn't this just a long-winded version of something like this (completely untested):

meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -8
index daa363b0dd65..22ddcfa5c1e3 100644
@@ -67,14 +67,7 @@ def get_crates(f):
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
-            found = True
-if not found:
-    raise ValueError("Unable to find Cargo.lock in
 ${CARGO_LOCK_SRC_DIR}")
+crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}",
 "Cargo.lock"))
 open("${TARGET_FILE}", 'w').write(crates)

EOF

--
Alex Kiernan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#179344):
https://lists.openembedded.org/g/openembedded-core/message/179344
Mute This Topic: https://lists.openembedded.org/mt/97953740/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#22 lore raw

Re: [OE-core] [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

In reply to Frédéric Martinsons

echo > recipes-upstreamable/sugar/sugar-crates.inc

before running "bitbake -c update_crates"

seems like easiest solution to let it regenerate the .inc file (or remove "require" and then add it back after refresh).

But there is still something strange in this solana, after refreshing the .inc files in sugar as well as solana-*, it's failing with:

NOTE: recipe sugar-1.1.0-r0: task do_unpack: Started
ERROR: sugar-1.1.0-r0 do_unpack: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:base_do_unpack(d)
     0003:
File:
'/OE/lge/build/webos/mickledore/oe-core/meta/classes-global/base.bbclass',
lineno: 160, function: base_do_unpack
     0156:        return
     0157:
     0158:    try:
     0159:        fetcher = bb.fetch2.Fetch(src_uri, d)

*** 0160: fetcher.unpack(d.getVar('WORKDIR'))
0161: except bb.fetch2.BBFetchException as e:
0162: bb.fatal("Bitbake Fetcher Error: " + repr(e))
0163:}
0164:
File: '/OE/lge/build/webos/mickledore/bitbake/lib/bb/fetch2/__init__.py',
lineno: 1882, function: unpack
1878:
1879: if ud.lockfile:
1880: lf = bb.utils.lockfile(ud.lockfile)
1881:
*** 1882: ud.method.unpack(ud, root, self.d)

     1883:
     1884:            if ud.lockfile:
     1885:                bb.utils.unlockfile(lf)
     1886:

File: '/OE/lge/build/webos/mickledore/bitbake/lib/bb/fetch2/crate.py',
lineno: 84, function: unpack
0080: """
0081: Uses the crate to build the necessary paths for cargo to
utilize it

     0082:        """
     0083:        if ud.type == 'crate':
 *** 0084:            return self._crate_unpack(ud, rootdir, d)
     0085:        else:
     0086:            super(Crate, self).unpack(ud, rootdir, d)
     0087:
     0088:    def _crate_unpack(self, ud, rootdir, d):

File: '/OE/lge/build/webos/mickledore/bitbake/lib/bb/fetch2/crate.py',
lineno: 113, function: _crate_unpack
0109: # ensure we've got these paths made
0110: bb.utils.mkdirhier(cargo_bitbake)
0111:
0112: # generate metadata necessary
*** 0113: with open(thefile, 'rb') as f:

     0114:                # get the SHA256 of the original tarball
     0115:                tarhash = hashlib.sha256(f.read()).hexdigest()
     0116:
     0117:            metadata['files'] = {}

Exception: FileNotFoundError: [Errno 2] No such file or directory:
'/OE/lge/build/webos/mickledore/downloads/arrayvec-0.7.2.crate'

ERROR: Logfile of failure stored in:
/OE/lge/build/webos/mickledore/BUILD/work/qemux86_64-webos-linux/sugar/1.1.0-r0/temp/log.do_unpack.2600442
NOTE: recipe sugar-1.1.0-r0: task do_unpack: Failed

...

NOTE: recipe solana-cli-1.14.5-r0: task do_fetch: Started
WARNING: solana-cli-1.14.5-r0 do_fetch: Checksum failure encountered with download of crate://crates.io/arrayvec/0.7.2 - will attempt other sources if available
ERROR: solana-cli-1.14.5-r0 do_fetch: Fetcher failure for URL: '
https://crates.io/api/v1/crates/arrayvec/0.7.2/download'. Checksum mismatch!
File: '/OE/lge/build/webos/mickledore/downloads/arrayvec-0.7.2.crate.tmp' has sha256 checksum '8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6' when 'be4dc07131ffa69b8072d35f5007352af944213cde02545e2103680baed38fcd' was expected
If this change is expected (e.g. you have upgraded to a new version without updating the checksums) then you can use these lines within the recipe:
SRC_URI[arrayvec.sha256sum] = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6" Otherwise you should retry the download and/or check with upstream to determine if the file has become corrupted or otherwise unexpectedly modified.

This is just FYI in case you can see what's going on right away, I'll continue to debug it from my end as well.

Quoted 141 lines · Frédéric Martinsons

Well I see what you mean, I'll take a look at your example to try to find out if multiple Cargo.lock could be expected.

And for your second remark, yes, there is a chicken and eggs issue for updating crates checksum from scratch.
You didn't miss anything, I came across this when updating crates checksum that was "pristine"
and I only manage to execute update_crates by locally patching
bitbake/lib/bb/fetch2/crate.py for
recommends_checksum method to return False .
I'm aware that is not ideal but I don't know how to make this better (maybe make do_update_crates
after do_fetch instead of after do_patch ? )

Le jeu. 30 mars 2023, 21:23, Martin Jansa <[email protected]> a écrit :

Quoted 123 lines

I don't remember the exact details now, but when I was working on updating solana recipes to use this

https://github.com/webosose/meta-webosose/commit/9bdfae7988077d0211eeee79cc339d0770cd86b4

the S was pointing to just some subdirectory and multiple Cargo.locks files were parsed in other directories as well, that's why I was adding

https://git.openembedded.org/openembedded-core/commit/meta/classes-recipe/cargo-update-recipe-crates.bbclass?id=7636a2b8080521ed2ad54b0edce47a8742a12d58

in the end I've changed to use the root directory in S and just set CARGO_SRC_DIR to right subdirectory to build.

I don't have much experience with crates other than these solana recipes, but isn't there some valid use-case for multiple Cargo.locks? I assume Alex in original implementation didn't use the os.walk just to make it more complicated :).

And FWIW when trying to regenerate these .inc files with current master and with:

$ bitbake -c update_crates solana-keygen
it fails with:
ERROR: solana-keygen-1.14.5-r0 do_fetch: No checksum specified for
'/OE/lge/build/webos/mickledore/downloads/Inflector-0.11.4.crate', please
add at least one to the recipe:
SRC_URI[Inflector.sha256sum] =
"fe438c63458706e03479442743baae6c88256498e6431708f6dfc520a26515d3"
ERROR: solana-keygen-1.14.5-r0 do_fetch: Bitbake Fetcher Error:
NoChecksumError('Missing SRC_URI checksum', '
https://crates.io/api/v1/crates/Inflector/0.11.4/download')

Isn't it an chicken&egg issue now when do_fetch enforces checksums to be used? Or am I just missing some of the pending changes or just using it incorrectly?

Quoted 84 lines · Alex Kiernan
Quoted 10 lines · On Thu, Mar 30, 2023 at 4:45 PM <[email protected]> wrote:

From: Frederic Martinsons <[email protected]>

There is no need to do such things, Cargo.lock file has to be at the root of CARGO_LOCK_SRC_DIR.
This avoid finding other possible Cargo.lock that would be in subdir (for example if a patch is applied on the recipe, we can have .pc subdir in S and a Cargo.lock can be there)

Review

Quoted 6 lines
---
 .../cargo-update-recipe-crates.bbclass               | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
(quoted diff) +0 -0

b/meta/classes-recipe/cargo-update-recipe-crates.bbclass

Quoted 5 lines

index daa363b0dd..549cfe627e 100644

meta/classes-recipe/cargo-update-recipe-crates.bbclass +0 -0
@@ -68,10 +68,14 @@ def get_crates(f):
 import os

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 74

Quoted 10 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +4 -4
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
 found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
+for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):
+    if file == 'Cargo.lock':
+        try:
+            cargo_lock_path = os.path.join('${CARGO_LOCK_SRC_DIR}',

file)

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 77

Quoted 3 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +3 -0
+            crates += get_crates(cargo_lock_path)
+        except Exception as e:
+            raise ValueError("Cannot parse '%s'" % cargo_lock_path)

from e

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 81

Quoted 4 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -0
+        else:
             found = True
 if not found:
     raise ValueError("Unable to find Cargo.lock in

${CARGO_LOCK_SRC_DIR}")

Isn't this just a long-winded version of something like this (completely untested):

meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -8
index daa363b0dd65..22ddcfa5c1e3 100644
@@ -67,14 +67,7 @@ def get_crates(f):
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
-            found = True
-if not found:
-    raise ValueError("Unable to find Cargo.lock in
 ${CARGO_LOCK_SRC_DIR}")
+crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}",
 "Cargo.lock"))
 open("${TARGET_FILE}", 'w').write(crates)

EOF

--
Alex Kiernan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#179344):
https://lists.openembedded.org/g/openembedded-core/message/179344
Mute This Topic: https://lists.openembedded.org/mt/97953740/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#23 lore raw

Re: [OE-core] [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

In reply to Martin Jansa

Oh my word... I wonder if we need to introspect what cargo does (a la cargo bitbake) to get this kind of thing right?

Quoted 99 lines · Martin Jansa

I don't remember the exact details now, but when I was working on updating solana recipes to use this
https://github.com/webosose/meta-webosose/commit/9bdfae7988077d0211eeee79cc339d0770cd86b4

the S was pointing to just some subdirectory and multiple Cargo.locks files were parsed in other directories as well, that's why I was adding
https://git.openembedded.org/openembedded-core/commit/meta/classes-recipe/cargo-update-recipe-crates.bbclass?id=7636a2b8080521ed2ad54b0edce47a8742a12d58

in the end I've changed to use the root directory in S and just set CARGO_SRC_DIR to right subdirectory to build.

I don't have much experience with crates other than these solana recipes, but isn't there some valid use-case for multiple Cargo.locks? I assume Alex in original implementation didn't use the os.walk just to make it more complicated :).

And FWIW when trying to regenerate these .inc files with current master and with:

$ bitbake -c update_crates solana-keygen
it fails with:
ERROR: solana-keygen-1.14.5-r0 do_fetch: No checksum specified for '/OE/lge/build/webos/mickledore/downloads/Inflector-0.11.4.crate', please add at least one to the recipe:
SRC_URI[Inflector.sha256sum] = "fe438c63458706e03479442743baae6c88256498e6431708f6dfc520a26515d3"
ERROR: solana-keygen-1.14.5-r0 do_fetch: Bitbake Fetcher Error: NoChecksumError('Missing SRC_URI checksum', 'https://crates.io/api/v1/crates/Inflector/0.11.4/download')

Isn't it an chicken&egg issue now when do_fetch enforces checksums to be used? Or am I just missing some of the pending changes or just using it incorrectly?

Quoted 76 lines · Alex Kiernan
Quoted 28 lines · On Thu, Mar 30, 2023 at 4:45 PM <[email protected]> wrote:

From: Frederic Martinsons <[email protected]>

There is no need to do such things, Cargo.lock file has to be at the root of CARGO_LOCK_SRC_DIR.
This avoid finding other possible Cargo.lock that would be in subdir (for example if a patch is applied on the recipe, we can have .pc subdir in S and a Cargo.lock can be there)

---
 .../cargo-update-recipe-crates.bbclass               | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -4
index daa363b0dd..549cfe627e 100644
@@ -68,10 +68,14 @@ def get_crates(f):
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
 found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
+for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 81

Quoted 10 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +7 -0
+    if file == 'Cargo.lock':
+        try:
+            cargo_lock_path = os.path.join('${CARGO_LOCK_SRC_DIR}', file)
+            crates += get_crates(cargo_lock_path)
+        except Exception as e:
+            raise ValueError("Cannot parse '%s'" % cargo_lock_path) from e
+        else:
             found = True
 if not found:
     raise ValueError("Unable to find Cargo.lock in ${CARGO_LOCK_SRC_DIR}")

Isn't this just a long-winded version of something like this (completely untested):

meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -9
index daa363b0dd65..22ddcfa5c1e3 100644
@@ -67,14 +67,7 @@ def get_crates(f):
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
-            found = True
-if not found:
-    raise ValueError("Unable to find Cargo.lock in ${CARGO_LOCK_SRC_DIR}")
+crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}", "Cargo.lock"))
 open("${TARGET_FILE}", 'w').write(crates)
 EOF
 
--

Alex Kiernan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#179344): https://lists.openembedded.org/g/openembedded-core/message/179344
Mute This Topic: https://lists.openembedded.org/mt/97953740/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 1 line
Alex Kiernan
#14 lore raw

Re: [OE-core] [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

In reply to Frédéric Martinsons
Quoted 3 lines · Frédéric Martinsons

By the way, about this "chickens and eggs" problem, isn't it the same for a regular recipe you just upgraded?

I don't know, I haven't upgraded a recipe with crates yet :)

It's just this time I was aware that the .inc file will need to be refreshed and it wasn't obvious at first that easiest way to avoid do_fetch issue blocking .inc refresh is to remove all crates first by emptying .inc file.

Personnaly, when I update a recipe, I let bitbake tell me what is the new

Quoted 2 lines

checksum expected and put it the recipe (as the error message says).

For regular recipes with a few checksum I do the same. But it doesn't scale here as you mention bellow.
as it reported only the first wrong checksum from MANY.

But I'm aware this is not exactly the same, since a cargo based recipe

Quoted 6 lines

could contain a ton of crate:// uri and if you apply this method, you have to copy the new checksum one by one, bitbake error after another...

Don't know how to make this better and I plead guilty for not mentioning that in a dedicated commit message.

Better late than never, you can include it in v6 of this patch (I have read the git log of the .bbclass before going to refresh the .inc files, so others might notice it there as well).

Of course, if someone come up with a smoothier way of doing this, I'll make

Quoted 2 lines

a new patch.

I think I found what the other issue with arrayvec create was:

meta-webos/recipes-upstreamable/solana/solana-crates.inc: crate://
crates.io/arrayvec/0.7.2 \
meta-webos/recipes-upstreamable/solana/solana-crates.inc:SRC_URI[arrayvec.sha256sum] = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"
meta-webos/recipes-upstreamable/solana/solana-crates.inc: crate://
crates.io/arrayvec/0.7.1 \
meta-webos/recipes-upstreamable/solana/solana-crates.inc:SRC_URI[arrayvec.sha256sum] = "be4dc07131ffa69b8072d35f5007352af944213cde02545e2103680baed38fcd"
meta-webos/recipes-upstreamable/sugar/sugar-crates.inc: crate://
crates.io/arrayvec/0.7.2 \
meta-webos/recipes-upstreamable/sugar/sugar-crates.inc:SRC_URI[arrayvec.sha256sum] = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"

So different Cargo.lock files included in the same ${S} have different version, but in this case it didn't set different name parameter, probably because different Cargo.lock files are processed separately?

Thanks

Le jeu. 30 mars 2023, 21:50, Frédéric Martinsons <

Quoted 144 lines
Quoted 142 lines · [email protected]> a écrit :

Well I see what you mean, I'll take a look at your example to try to find out if multiple Cargo.lock could be expected.

And for your second remark, yes, there is a chicken and eggs issue for updating crates checksum from scratch.
You didn't miss anything, I came across this when updating crates checksum that was "pristine"
and I only manage to execute update_crates by locally patching
bitbake/lib/bb/fetch2/crate.py for
recommends_checksum method to return False .
I'm aware that is not ideal but I don't know how to make this better (maybe make do_update_crates
after do_fetch instead of after do_patch ? )

Le jeu. 30 mars 2023, 21:23, Martin Jansa <[email protected]> a écrit :

Quoted 124 lines

I don't remember the exact details now, but when I was working on updating solana recipes to use this

https://github.com/webosose/meta-webosose/commit/9bdfae7988077d0211eeee79cc339d0770cd86b4

the S was pointing to just some subdirectory and multiple Cargo.locks files were parsed in other directories as well, that's why I was adding

https://git.openembedded.org/openembedded-core/commit/meta/classes-recipe/cargo-update-recipe-crates.bbclass?id=7636a2b8080521ed2ad54b0edce47a8742a12d58

in the end I've changed to use the root directory in S and just set CARGO_SRC_DIR to right subdirectory to build.

I don't have much experience with crates other than these solana recipes, but isn't there some valid use-case for multiple Cargo.locks? I assume Alex in original implementation didn't use the os.walk just to make it more complicated :).

And FWIW when trying to regenerate these .inc files with current master and with:

$ bitbake -c update_crates solana-keygen
it fails with:
ERROR: solana-keygen-1.14.5-r0 do_fetch: No checksum specified for
'/OE/lge/build/webos/mickledore/downloads/Inflector-0.11.4.crate', please
add at least one to the recipe:
SRC_URI[Inflector.sha256sum] =
"fe438c63458706e03479442743baae6c88256498e6431708f6dfc520a26515d3"
ERROR: solana-keygen-1.14.5-r0 do_fetch: Bitbake Fetcher Error:
NoChecksumError('Missing SRC_URI checksum', '
https://crates.io/api/v1/crates/Inflector/0.11.4/download')

Isn't it an chicken&egg issue now when do_fetch enforces checksums to be used? Or am I just missing some of the pending changes or just using it incorrectly?

Quoted 85 lines · Alex Kiernan
Quoted 10 lines · On Thu, Mar 30, 2023 at 4:45 PM <[email protected]> wrote:

From: Frederic Martinsons <[email protected]>

There is no need to do such things, Cargo.lock file has to be at the root of CARGO_LOCK_SRC_DIR.
This avoid finding other possible Cargo.lock that would be in subdir (for example if a patch is applied on the recipe, we can have .pc subdir in S and a Cargo.lock can be there)

Quoted 3 lines
---
 .../cargo-update-recipe-crates.bbclass               | 12

++++++++----

Review

Quoted 3 lines
 1 file changed, 8 insertions(+), 4 deletions(-)
(quoted diff) +0 -0

b/meta/classes-recipe/cargo-update-recipe-crates.bbclass

Quoted 5 lines

index daa363b0dd..549cfe627e 100644

meta/classes-recipe/cargo-update-recipe-crates.bbclass +0 -0
@@ -68,10 +68,14 @@ def get_crates(f):
 import os

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 74

Quoted 10 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +4 -4
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
 found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
+for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):
+    if file == 'Cargo.lock':
+        try:
+            cargo_lock_path = os.path.join('${CARGO_LOCK_SRC_DIR}',

file)

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 77

Quoted 3 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +3 -0
+            crates += get_crates(cargo_lock_path)
+        except Exception as e:
+            raise ValueError("Cannot parse '%s'" % cargo_lock_path)

from e

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 81

Quoted 4 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -0
+        else:
             found = True
 if not found:
     raise ValueError("Unable to find Cargo.lock in

${CARGO_LOCK_SRC_DIR}")

Isn't this just a long-winded version of something like this (completely untested):

meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -8
index daa363b0dd65..22ddcfa5c1e3 100644
@@ -67,14 +67,7 @@ def get_crates(f):
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
-            found = True
-if not found:
-    raise ValueError("Unable to find Cargo.lock in
 ${CARGO_LOCK_SRC_DIR}")
+crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}",
 "Cargo.lock"))
 open("${TARGET_FILE}", 'w').write(crates)

EOF

--
Alex Kiernan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#179344):
https://lists.openembedded.org/g/openembedded-core/message/179344
Mute This Topic: https://lists.openembedded.org/mt/97953740/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#16 lore raw

Re: [OE-core] [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

In reply to a message not in the archive

I've added the updated .inc files to meta-webosose fork if it's easier to see there:

https://github.com/shr-project/meta-webosose/commits/master-2023-03-30 (top 2 commits)

this checksum mismatch is also the reason for that ugly exception as explained in top commit, it's race-condition between sugar.do_unpack and solana-cli.do_fetch where sugar has right arrayvec 0.7.2 with right checksum in do_unpack, while solana-cli renames it to _bad-checksum_ suffix, because solana-cli looks for checksum of arrayvec 0.7.1.

I've tried to manually add name= param to verify this, but there are too many :), so I'll update the bbclass to always add name with version even if it doesn't see duplicated crates (and then this can be improved if multiple Cargo.lock files really need to be supported).

I honestly don't know, I've was only working on these solana recipes for short time and wanted to offer them more as strange example.

Cheers,

Quoted 207 lines · Martin Jansa via lists.openembedded.org
Quoted 3 lines · Frédéric Martinsons

By the way, about this "chickens and eggs" problem, isn't it the same for a regular recipe you just upgraded?

I don't know, I haven't upgraded a recipe with crates yet :)

It's just this time I was aware that the .inc file will need to be refreshed and it wasn't obvious at first that easiest way to avoid do_fetch issue blocking .inc refresh is to remove all crates first by emptying .inc file.

Personnaly, when I update a recipe, I let bitbake tell me what is the new

Quoted 2 lines

checksum expected and put it the recipe (as the error message says).

For regular recipes with a few checksum I do the same. But it doesn't scale here as you mention bellow.
as it reported only the first wrong checksum from MANY.

But I'm aware this is not exactly the same, since a cargo based recipe

Quoted 6 lines

could contain a ton of crate:// uri and if you apply this method, you have to copy the new checksum one by one, bitbake error after another...

Don't know how to make this better and I plead guilty for not mentioning that in a dedicated commit message.

Better late than never, you can include it in v6 of this patch (I have read the git log of the .bbclass before going to refresh the .inc files, so others might notice it there as well).

Of course, if someone come up with a smoothier way of doing this, I'll

Quoted 2 lines

make a new patch.

I think I found what the other issue with arrayvec create was:

meta-webos/recipes-upstreamable/solana/solana-crates.inc: crate://
crates.io/arrayvec/0.7.2 \
meta-webos/recipes-upstreamable/solana/solana-crates.inc:SRC_URI[arrayvec.sha256sum] = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"
meta-webos/recipes-upstreamable/solana/solana-crates.inc: crate://
crates.io/arrayvec/0.7.1 \
meta-webos/recipes-upstreamable/solana/solana-crates.inc:SRC_URI[arrayvec.sha256sum] = "be4dc07131ffa69b8072d35f5007352af944213cde02545e2103680baed38fcd"
meta-webos/recipes-upstreamable/sugar/sugar-crates.inc: crate://
crates.io/arrayvec/0.7.2 \
meta-webos/recipes-upstreamable/sugar/sugar-crates.inc:SRC_URI[arrayvec.sha256sum] = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"

So different Cargo.lock files included in the same ${S} have different version, but in this case it didn't set different name parameter, probably because different Cargo.lock files are processed separately?

Thanks

Le jeu. 30 mars 2023, 21:50, Frédéric Martinsons <

Quoted 130 lines
Quoted 128 lines · [email protected]> a écrit :

Well I see what you mean, I'll take a look at your example to try to find out if multiple Cargo.lock could be expected.

And for your second remark, yes, there is a chicken and eggs issue for updating crates checksum from scratch.
You didn't miss anything, I came across this when updating crates checksum that was "pristine"
and I only manage to execute update_crates by locally patching
bitbake/lib/bb/fetch2/crate.py for
recommends_checksum method to return False .
I'm aware that is not ideal but I don't know how to make this better (maybe make do_update_crates
after do_fetch instead of after do_patch ? )

Le jeu. 30 mars 2023, 21:23, Martin Jansa <[email protected]> a écrit :

Quoted 110 lines

I don't remember the exact details now, but when I was working on updating solana recipes to use this

https://github.com/webosose/meta-webosose/commit/9bdfae7988077d0211eeee79cc339d0770cd86b4

the S was pointing to just some subdirectory and multiple Cargo.locks files were parsed in other directories as well, that's why I was adding

https://git.openembedded.org/openembedded-core/commit/meta/classes-recipe/cargo-update-recipe-crates.bbclass?id=7636a2b8080521ed2ad54b0edce47a8742a12d58

in the end I've changed to use the root directory in S and just set CARGO_SRC_DIR to right subdirectory to build.

I don't have much experience with crates other than these solana recipes, but isn't there some valid use-case for multiple Cargo.locks? I assume Alex in original implementation didn't use the os.walk just to make it more complicated :).

And FWIW when trying to regenerate these .inc files with current master and with:

$ bitbake -c update_crates solana-keygen
it fails with:
ERROR: solana-keygen-1.14.5-r0 do_fetch: No checksum specified for
'/OE/lge/build/webos/mickledore/downloads/Inflector-0.11.4.crate', please
add at least one to the recipe:
SRC_URI[Inflector.sha256sum] =
"fe438c63458706e03479442743baae6c88256498e6431708f6dfc520a26515d3"
ERROR: solana-keygen-1.14.5-r0 do_fetch: Bitbake Fetcher Error:
NoChecksumError('Missing SRC_URI checksum', '
https://crates.io/api/v1/crates/Inflector/0.11.4/download')

Isn't it an chicken&egg issue now when do_fetch enforces checksums to be used? Or am I just missing some of the pending changes or just using it incorrectly?

Quoted 71 lines · Alex Kiernan
Quoted 10 lines · On Thu, Mar 30, 2023 at 4:45 PM <[email protected]> wrote:

From: Frederic Martinsons <[email protected]>

There is no need to do such things, Cargo.lock file has to be at the root of CARGO_LOCK_SRC_DIR.
This avoid finding other possible Cargo.lock that would be in subdir (for example if a patch is applied on the recipe, we can have .pc subdir in S and a Cargo.lock can be there)

Quoted 3 lines
---
 .../cargo-update-recipe-crates.bbclass               | 12

++++++++----

Review

Quoted 3 lines
 1 file changed, 8 insertions(+), 4 deletions(-)
(quoted diff) +0 -0

b/meta/classes-recipe/cargo-update-recipe-crates.bbclass

Quoted 5 lines

index daa363b0dd..549cfe627e 100644

meta/classes-recipe/cargo-update-recipe-crates.bbclass +0 -0
@@ -68,10 +68,14 @@ def get_crates(f):
 import os

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 74

Quoted 10 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +4 -4
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
 found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
+for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):
+    if file == 'Cargo.lock':
+        try:
+            cargo_lock_path = os.path.join('${CARGO_LOCK_SRC_DIR}',

file)

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 77

Quoted 3 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +3 -0
+            crates += get_crates(cargo_lock_path)
+        except Exception as e:
+            raise ValueError("Cannot parse '%s'" % cargo_lock_path)

from e

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 81

Quoted 4 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -0
+        else:
             found = True
 if not found:
     raise ValueError("Unable to find Cargo.lock in

${CARGO_LOCK_SRC_DIR}")

Isn't this just a long-winded version of something like this (completely untested):

meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -8
index daa363b0dd65..22ddcfa5c1e3 100644
@@ -67,14 +67,7 @@ def get_crates(f):
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
-            found = True
-if not found:
-    raise ValueError("Unable to find Cargo.lock in
 ${CARGO_LOCK_SRC_DIR}")
+crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}",
 "Cargo.lock"))
 open("${TARGET_FILE}", 'w').write(crates)

EOF

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

--
Alex Kiernan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#179358):
https://lists.openembedded.org/g/openembedded-core/message/179358
Mute This Topic: https://lists.openembedded.org/mt/97953740/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#17 lore raw

Re: [OE-core] [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

Hello,

The multiple Cargo.lock seems totally valid, a project can have multiple binaries provided (which is the case for Solana cli suite) so I'll submit a new v6 of my series to reflect that, I just have to ignored specify dirs like .git (no use to walk inside it) or .pc (can contains a patched Cargo.lock that would be redundant and syntaxically invalid )

I'll also override the crate names with their version for all the crates found.

For the chicken and eggs issue, I need to think more of it (maybe make bitbake print the list of all missing checksum would be enough) and, in order to not delay this series, I'll submit a separate bitbake patch if I manage to find something suitable.

Many thanks for the comments Martin.

Quoted 234 lines · Martin Jansa

I've added the updated .inc files to meta-webosose fork if it's easier to see there:

https://github.com/shr-project/meta-webosose/commits/master-2023-03-30
(top 2 commits)

this checksum mismatch is also the reason for that ugly exception as explained in top commit, it's race-condition between sugar.do_unpack and solana-cli.do_fetch where sugar has right arrayvec 0.7.2 with right checksum in do_unpack, while solana-cli renames it to _bad-checksum_ suffix, because solana-cli looks for checksum of arrayvec 0.7.1.

I've tried to manually add name= param to verify this, but there are too many :), so I'll update the bbclass to always add name with version even if it doesn't see duplicated crates (and then this can be improved if multiple Cargo.lock files really need to be supported).

I honestly don't know, I've was only working on these solana recipes for short time and wanted to offer them more as strange example.

Cheers,

Quoted 209 lines · Martin Jansa via lists.openembedded.org
Quoted 3 lines · Frédéric Martinsons

By the way, about this "chickens and eggs" problem, isn't it the same for a regular recipe you just upgraded?

I don't know, I haven't upgraded a recipe with crates yet :)

It's just this time I was aware that the .inc file will need to be refreshed and it wasn't obvious at first that easiest way to avoid do_fetch issue blocking .inc refresh is to remove all crates first by emptying .inc file.

Personnaly, when I update a recipe, I let bitbake tell me what is the new

Quoted 2 lines

checksum expected and put it the recipe (as the error message says).

For regular recipes with a few checksum I do the same. But it doesn't scale here as you mention bellow.
as it reported only the first wrong checksum from MANY.

But I'm aware this is not exactly the same, since a cargo based recipe

Quoted 6 lines

could contain a ton of crate:// uri and if you apply this method, you have to copy the new checksum one by one, bitbake error after another...

Don't know how to make this better and I plead guilty for not mentioning that in a dedicated commit message.

Better late than never, you can include it in v6 of this patch (I have read the git log of the .bbclass before going to refresh the .inc files, so others might notice it there as well).

Of course, if someone come up with a smoothier way of doing this, I'll

Quoted 2 lines

make a new patch.

I think I found what the other issue with arrayvec create was:

meta-webos/recipes-upstreamable/solana/solana-crates.inc: crate://
crates.io/arrayvec/0.7.2 \
meta-webos/recipes-upstreamable/solana/solana-crates.inc:SRC_URI[arrayvec.sha256sum] = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"
meta-webos/recipes-upstreamable/solana/solana-crates.inc: crate://
crates.io/arrayvec/0.7.1 \
meta-webos/recipes-upstreamable/solana/solana-crates.inc:SRC_URI[arrayvec.sha256sum] = "be4dc07131ffa69b8072d35f5007352af944213cde02545e2103680baed38fcd"
meta-webos/recipes-upstreamable/sugar/sugar-crates.inc: crate://
crates.io/arrayvec/0.7.2 \
meta-webos/recipes-upstreamable/sugar/sugar-crates.inc:SRC_URI[arrayvec.sha256sum] = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"

So different Cargo.lock files included in the same ${S} have different version, but in this case it didn't set different name parameter, probably because different Cargo.lock files are processed separately?

Thanks

Le jeu. 30 mars 2023, 21:50, Frédéric Martinsons <

Quoted 132 lines
Quoted 130 lines · [email protected]> a écrit :

Well I see what you mean, I'll take a look at your example to try to find out if multiple Cargo.lock could be expected.

And for your second remark, yes, there is a chicken and eggs issue for updating crates checksum from scratch.
You didn't miss anything, I came across this when updating crates checksum that was "pristine"
and I only manage to execute update_crates by locally patching
bitbake/lib/bb/fetch2/crate.py for
recommends_checksum method to return False .
I'm aware that is not ideal but I don't know how to make this better (maybe make do_update_crates
after do_fetch instead of after do_patch ? )

Le jeu. 30 mars 2023, 21:23, Martin Jansa <[email protected]> a écrit :

Quoted 112 lines

I don't remember the exact details now, but when I was working on updating solana recipes to use this

https://github.com/webosose/meta-webosose/commit/9bdfae7988077d0211eeee79cc339d0770cd86b4

the S was pointing to just some subdirectory and multiple Cargo.locks files were parsed in other directories as well, that's why I was adding

https://git.openembedded.org/openembedded-core/commit/meta/classes-recipe/cargo-update-recipe-crates.bbclass?id=7636a2b8080521ed2ad54b0edce47a8742a12d58

in the end I've changed to use the root directory in S and just set CARGO_SRC_DIR to right subdirectory to build.

I don't have much experience with crates other than these solana recipes, but isn't there some valid use-case for multiple Cargo.locks? I assume Alex in original implementation didn't use the os.walk just to make it more complicated :).

And FWIW when trying to regenerate these .inc files with current master and with:

$ bitbake -c update_crates solana-keygen
it fails with:
ERROR: solana-keygen-1.14.5-r0 do_fetch: No checksum specified for
'/OE/lge/build/webos/mickledore/downloads/Inflector-0.11.4.crate', please
add at least one to the recipe:
SRC_URI[Inflector.sha256sum] =
"fe438c63458706e03479442743baae6c88256498e6431708f6dfc520a26515d3"
ERROR: solana-keygen-1.14.5-r0 do_fetch: Bitbake Fetcher Error:
NoChecksumError('Missing SRC_URI checksum', '
https://crates.io/api/v1/crates/Inflector/0.11.4/download')

Isn't it an chicken&egg issue now when do_fetch enforces checksums to be used? Or am I just missing some of the pending changes or just using it incorrectly?

Quoted 73 lines · Alex Kiernan
Quoted 10 lines · On Thu, Mar 30, 2023 at 4:45 PM <[email protected]> wrote:

From: Frederic Martinsons <[email protected]>

There is no need to do such things, Cargo.lock file has to be at the root of CARGO_LOCK_SRC_DIR.
This avoid finding other possible Cargo.lock that would be in subdir (for example if a patch is applied on the recipe, we can have .pc subdir in S and a Cargo.lock can be there)

Quoted 3 lines
---
 .../cargo-update-recipe-crates.bbclass               | 12

++++++++----

Review

Quoted 3 lines
 1 file changed, 8 insertions(+), 4 deletions(-)
(quoted diff) +0 -0

b/meta/classes-recipe/cargo-update-recipe-crates.bbclass

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 69

Quoted 6 lines

index daa363b0dd..549cfe627e 100644

meta/classes-recipe/cargo-update-recipe-crates.bbclass +0 -0
@@ -68,10 +68,14 @@ def get_crates(f):
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates

${PN}'\n\n"

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 74

Quoted 9 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +4 -4
 found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
+for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):
+    if file == 'Cargo.lock':
+        try:
+            cargo_lock_path =

os.path.join('${CARGO_LOCK_SRC_DIR}', file)

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 77

Quoted 3 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +3 -0
+            crates += get_crates(cargo_lock_path)
+        except Exception as e:
+            raise ValueError("Cannot parse '%s'" %

cargo_lock_path) from e

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 81

Quoted 4 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -0
+        else:
             found = True
 if not found:
     raise ValueError("Unable to find Cargo.lock in

${CARGO_LOCK_SRC_DIR}")

Isn't this just a long-winded version of something like this (completely untested):

meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -8
index daa363b0dd65..22ddcfa5c1e3 100644
@@ -67,14 +67,7 @@ def get_crates(f):
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
-            found = True
-if not found:
-    raise ValueError("Unable to find Cargo.lock in
 ${CARGO_LOCK_SRC_DIR}")
+crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}",
 "Cargo.lock"))
 open("${TARGET_FILE}", 'w').write(crates)

EOF

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

--
Alex Kiernan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#179358):
https://lists.openembedded.org/g/openembedded-core/message/179358
Mute This Topic: https://lists.openembedded.org/mt/97953740/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#19 lore raw

Re: [OE-core] [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

In reply to a message not in the archive

Hello, I just submitted a patch inside the bitbake fetcher to display all the missing
checksum at once instead of only one.

For the moment, I have no better solution, let's see what bitbake folks tell us about that.

Quoted 259 lines · Frederic Martinsons via lists.openembedded.org

Hello,

The multiple Cargo.lock seems totally valid, a project can have multiple binaries provided (which is the case for Solana cli suite) so I'll submit a new v6 of my series to reflect that, I just have to ignored specify dirs like .git (no use to walk inside it) or .pc (can contains a patched Cargo.lock that would be redundant and syntaxically invalid )

I'll also override the crate names with their version for all the crates found.

For the chicken and eggs issue, I need to think more of it (maybe make bitbake print the list of all missing checksum would be enough) and, in order to not delay this series, I'll submit a separate bitbake patch if I manage to find something suitable.

Many thanks for the comments Martin.

Le jeu. 30 mars 2023, 23:15, Martin Jansa <[email protected]> a écrit :

Quoted 17 lines

I've added the updated .inc files to meta-webosose fork if it's easier to see there:

https://github.com/shr-project/meta-webosose/commits/master-2023-03-30
(top 2 commits)

this checksum mismatch is also the reason for that ugly exception as explained in top commit, it's race-condition between sugar.do_unpack and solana-cli.do_fetch where sugar has right arrayvec 0.7.2 with right checksum in do_unpack, while solana-cli renames it to _bad-checksum_ suffix, because solana-cli looks for checksum of arrayvec 0.7.1.

I've tried to manually add name= param to verify this, but there are too many :), so I'll update the bbclass to always add name with version even if it doesn't see duplicated crates (and then this can be improved if multiple Cargo.lock files really need to be supported).

Quoted 210 lines

I honestly don't know, I've was only working on these solana recipes for short time and wanted to offer them more as strange example.

Cheers,

Quoted 202 lines · Martin Jansa via lists.openembedded.org
Quoted 3 lines · Frédéric Martinsons

By the way, about this "chickens and eggs" problem, isn't it the same for a regular recipe you just upgraded?

I don't know, I haven't upgraded a recipe with crates yet :)

It's just this time I was aware that the .inc file will need to be refreshed and it wasn't obvious at first that easiest way to avoid do_fetch issue blocking .inc refresh is to remove all crates first by emptying .inc file.

Personnaly, when I update a recipe, I let bitbake tell me what is the

Quoted 2 lines

new checksum expected and put it the recipe (as the error message says).

For regular recipes with a few checksum I do the same. But it doesn't scale here as you mention bellow.
as it reported only the first wrong checksum from MANY.

But I'm aware this is not exactly the same, since a cargo based recipe

Quoted 6 lines

could contain a ton of crate:// uri and if you apply this method, you have to copy the new checksum one by one, bitbake error after another...

Don't know how to make this better and I plead guilty for not mentioning that in a dedicated commit message.

Better late than never, you can include it in v6 of this patch (I have read the git log of the .bbclass before going to refresh the .inc files, so others might notice it there as well).

Of course, if someone come up with a smoothier way of doing this, I'll

Quoted 2 lines

make a new patch.

I think I found what the other issue with arrayvec create was:

meta-webos/recipes-upstreamable/solana/solana-crates.inc: crate://
crates.io/arrayvec/0.7.2 \
meta-webos/recipes-upstreamable/solana/solana-crates.inc:SRC_URI[arrayvec.sha256sum] = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"
meta-webos/recipes-upstreamable/solana/solana-crates.inc: crate://
crates.io/arrayvec/0.7.1 \
meta-webos/recipes-upstreamable/solana/solana-crates.inc:SRC_URI[arrayvec.sha256sum] = "be4dc07131ffa69b8072d35f5007352af944213cde02545e2103680baed38fcd"
meta-webos/recipes-upstreamable/sugar/sugar-crates.inc: crate://
crates.io/arrayvec/0.7.2 \
meta-webos/recipes-upstreamable/sugar/sugar-crates.inc:SRC_URI[arrayvec.sha256sum] = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"

So different Cargo.lock files included in the same ${S} have different version, but in this case it didn't set different name parameter, probably because different Cargo.lock files are processed separately?

Thanks

Le jeu. 30 mars 2023, 21:50, Frédéric Martinsons <

Quoted 139 lines
Quoted 137 lines · [email protected]> a écrit :

Well I see what you mean, I'll take a look at your example to try to find out if multiple Cargo.lock could be expected.

And for your second remark, yes, there is a chicken and eggs issue for updating crates checksum from scratch.
You didn't miss anything, I came across this when updating crates checksum that was "pristine"
and I only manage to execute update_crates by locally patching
bitbake/lib/bb/fetch2/crate.py for
recommends_checksum method to return False .
I'm aware that is not ideal but I don't know how to make this better (maybe make do_update_crates
after do_fetch instead of after do_patch ? )

Le jeu. 30 mars 2023, 21:23, Martin Jansa <[email protected]> a écrit :

Quoted 119 lines

I don't remember the exact details now, but when I was working on updating solana recipes to use this

https://github.com/webosose/meta-webosose/commit/9bdfae7988077d0211eeee79cc339d0770cd86b4

the S was pointing to just some subdirectory and multiple Cargo.locks files were parsed in other directories as well, that's why I was adding

https://git.openembedded.org/openembedded-core/commit/meta/classes-recipe/cargo-update-recipe-crates.bbclass?id=7636a2b8080521ed2ad54b0edce47a8742a12d58

in the end I've changed to use the root directory in S and just set CARGO_SRC_DIR to right subdirectory to build.

I don't have much experience with crates other than these solana recipes, but isn't there some valid use-case for multiple Cargo.locks? I assume Alex in original implementation didn't use the os.walk just to make it more complicated :).

And FWIW when trying to regenerate these .inc files with current master and with:

$ bitbake -c update_crates solana-keygen
it fails with:
ERROR: solana-keygen-1.14.5-r0 do_fetch: No checksum specified for
'/OE/lge/build/webos/mickledore/downloads/Inflector-0.11.4.crate', please
add at least one to the recipe:
SRC_URI[Inflector.sha256sum] =
"fe438c63458706e03479442743baae6c88256498e6431708f6dfc520a26515d3"
ERROR: solana-keygen-1.14.5-r0 do_fetch: Bitbake Fetcher Error:
NoChecksumError('Missing SRC_URI checksum', '
https://crates.io/api/v1/crates/Inflector/0.11.4/download')

Isn't it an chicken&egg issue now when do_fetch enforces checksums to be used? Or am I just missing some of the pending changes or just using it incorrectly?

Quoted 80 lines · Alex Kiernan
Quoted 10 lines · On Thu, Mar 30, 2023 at 4:45 PM <[email protected]> wrote:

From: Frederic Martinsons <[email protected]>

There is no need to do such things, Cargo.lock file has to be at the root of CARGO_LOCK_SRC_DIR.
This avoid finding other possible Cargo.lock that would be in subdir (for example if a patch is applied on the recipe, we can have .pc subdir in S and a Cargo.lock can be there)

Quoted 3 lines
---
 .../cargo-update-recipe-crates.bbclass               | 12

++++++++----

Quoted 3 lines
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git

a/meta/classes-recipe/cargo-update-recipe-crates.bbclass b/meta/classes-recipe/cargo-update-recipe-crates.bbclass

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 69

Quoted 6 lines

index daa363b0dd..549cfe627e 100644

meta/classes-recipe/cargo-update-recipe-crates.bbclass +0 -0
@@ -68,10 +68,14 @@ def get_crates(f):
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates

${PN}'\n\n"

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 74

Quoted 9 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +4 -4
 found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
+for file in os.listdir('${CARGO_LOCK_SRC_DIR}'):
+    if file == 'Cargo.lock':
+        try:
+            cargo_lock_path =

os.path.join('${CARGO_LOCK_SRC_DIR}', file)

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 77

Quoted 3 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +3 -0
+            crates += get_crates(cargo_lock_path)
+        except Exception as e:
+            raise ValueError("Cannot parse '%s'" %

cargo_lock_path) from e

meta/classes-recipe/cargo-update-recipe-crates.bbclass def get_crates(f): · line 81

Quoted 4 lines
meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -0
+        else:
             found = True
 if not found:
     raise ValueError("Unable to find Cargo.lock in

${CARGO_LOCK_SRC_DIR}")

Isn't this just a long-winded version of something like this (completely untested):

meta/classes-recipe/cargo-update-recipe-crates.bbclass +1 -8
index daa363b0dd65..22ddcfa5c1e3 100644
@@ -67,14 +67,7 @@ def get_crates(f):
 
 import os
 crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    for file in files:
-        if file == 'Cargo.lock':
-            crates += get_crates(os.path.join(root, file))
-            found = True
-if not found:
-    raise ValueError("Unable to find Cargo.lock in
 ${CARGO_LOCK_SRC_DIR}")
+crates += get_crates(os.path.join("${CARGO_LOCK_SRC_DIR}",
 "Cargo.lock"))
 open("${TARGET_FILE}", 'w').write(crates)

EOF

--
Alex Kiernan

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#179391):
https://lists.openembedded.org/g/openembedded-core/message/179391
Mute This Topic: https://lists.openembedded.org/mt/97953740/6213388
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#20 lore raw

Re: [OE-core] [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

Quoted 5 lines · Frederic Martinsons

Hello, I just submitted a patch inside the bitbake fetcher to display all the missing checksum at once instead of only one.

For the moment, I have no better solution, let's see what bitbake folks tell us about that.

I'm fine with that FWIW...

Cheers,

Richard

#21 lore raw

Re: [OE-core] [PATCH V5 6/6] cargo-update-recipe-crates: don't walk on the whole dir

Nice, glad to hear that.

Quoted 13 lines · Richard Purdie
Quoted 5 lines · Frederic Martinsons

Hello, I just submitted a patch inside the bitbake fetcher to display all the missing checksum at once instead of only one.

For the moment, I have no better solution, let's see what bitbake folks tell us about that.

I'm fine with that FWIW...

Cheers,

Richard