Skip to contentSkip to conversation

Use frozen flag for cargo build

PATCH In review [email protected] 1 participant 5 messages Aug 2, 2023

lore
Share & tools
lore
#0 lore raw

cover Use frozen flag for cargo build

From: Frederic Martinsons <[email protected]>

This patch series force the usage of --frozen instead of --offline during cargo build. This has the advantage to be sure that Cargo.lock file will not be modified. Moreover, raise a clear error when Cargo.lock is not present.
For this to work, we must modify ourself the Cargo.lock before building in case there are git repo that have been patched.

The last commit was for removing rust-hello-world, patch to the documentation project will follow.

The following changes since commit 692e414aed5313ff275b69e93179aa7c559700f3:

ffmpeg: Fix wrong code found with gas/2.41 (2023-08-01 09:49:14 +0100)

are available in the Git repository at:

https://gitlab.com/fmartinsons/openembedded-core 15104-check-cargo-lock

Frederic Martinsons (3):
cargo.bbclass: Use --frozen flag for cargo operations cargo_common.bbclass: Handle Cargo.lock modifications for git
dependencies
drop rust-hello-world recipe

 meta/classes-recipe/cargo.bbclass             |  7 +++-
 meta/classes-recipe/cargo_common.bbclass      | 41 +++++++++++++++++++
 meta/conf/distro/include/maintainers.inc      |  1 -
 meta/lib/oeqa/runtime/cases/rust.py           | 12 ------
 .../packagegroup-core-tools-testapps.bb       |  3 --
 .../rust-hello-world/0001-enable-LTO.patch    | 24 -----------
 .../rust-example/rust-hello-world_git.bb      | 19 ---------
 7 files changed, 47 insertions(+), 60 deletions(-)
 delete mode 100644 meta/recipes-extended/rust-example/rust-hello-world/0001-enable-LTO.patch
 delete mode 100644 meta/recipes-extended/rust-example/rust-hello-world_git.bb
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/3 cargo.bbclass: Use --frozen flag for cargo operations

From: Frederic Martinsons <[email protected]>

It supersed the --offline flag and guarantee that Cargo.lock file will not be modified during the build.

---
 meta/classes-recipe/cargo.bbclass | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
meta/classes-recipe/cargo.bbclass +6 -1
index 3ef0bbbb44..8c0b92df8d 100644
@@ -39,7 +39,12 @@ MANIFEST_PATH ??= "${S}/${CARGO_SRC_DIR}/Cargo.toml"
 
 RUSTFLAGS ??= ""
 BUILD_MODE = "${@['--release', ''][d.getVar('DEBUG_BUILD') == '1']}"
-CARGO_BUILD_FLAGS = "-v --offline --target ${RUST_HOST_SYS} ${BUILD_MODE} --manifest-path=${MANIFEST_PATH}"
+# --frozen flag will prevent network access (which is required since only
+# the do_fetch step is authorized to access network)
+# and will require an up to date Cargo.lock file.
+# This force the package being built to already ship a Cargo.lock, in the end
+# this is what we want, at least, for reproducibility of the build.
+CARGO_BUILD_FLAGS = "-v --frozen --target ${RUST_HOST_SYS} ${BUILD_MODE} --manifest-path=${MANIFEST_PATH}"
 
 # This is based on the content of CARGO_BUILD_FLAGS and generally will need to
 # change if CARGO_BUILD_FLAGS changes.
Signature · 1 line
2.34.1
#2 lore raw

PATCH 2/3 cargo_common.bbclass: Handle Cargo.lock modifications for git dependencies

In reply to [email protected]

From: Frederic Martinsons <[email protected]>

Now we use --frozen, Cargo.lock cannot be modified by cargo build. These patched git dependencies requires that the git url is removed from Cargo.lock.

Fixes #15104

---
 meta/classes-recipe/cargo_common.bbclass | 41 ++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
meta/classes-recipe/cargo_common.bbclass +41 -0
index db54826ddb..b732a1bd95 100644
@@ -117,6 +117,8 @@ cargo_common_do_configure () {
 }
 
 python cargo_common_do_patch_paths() {
+    import shutil
+
     cargo_config = os.path.join(d.getVar("CARGO_HOME"), "config")
     if not os.path.exists(cargo_config):
         return
@@ -146,6 +148,45 @@ python cargo_common_do_patch_paths() {
             print('\n[patch."%s"]' % k, file=config)
             for name in v:
                 print(name, file=config)
+
+    if not patches:
+        return
+
+    # Cargo.lock file is needed for to be sure that artifacts
+    # downloaded by the fetch steps are those expected by the
+    # project and that the possible patches are correctly applied.
+    # Moreover since we do not want any modification
+    # of this file (for reproducibility purpose), we prevent it by
+    # using --frozen flag (in CARGO_BUILD_FLAGS) and raise a clear error
+    # here is better than letting cargo tell (in case the file is missing)
+    # "Cargo.lock should be modified but --frozen was given"
+
+    manifest_path = d.getVar("MANIFEST_PATH", True)
+    lockfile = os.path.join(os.path.dirname(manifest_path), "Cargo.lock")
+    if not os.path.exists(lockfile):
+        bb.fatal(f"{lockfile} file doesn't exist")
+
+    # There are patched files and so Cargo.lock should be modified but we use
+    # --frozen so let's handle that modifications here.
+    #
+    # Note that a "better" (more elegant ?) would have been to use cargo update for
+    # patched packages:
+    #  cargo update --offline -p package_1 -p package_2
+    # But this is not possible since it requires that cargo local git db
+    # to be populated and this is not the case as we fetch git repo ourself.
+
+    lockfile_orig = lockfile + ".orig"
+    if not os.path.exists(lockfile_orig):
+        shutil.copy(lockfile, lockfile_orig)
+
+    newlines = []
+    with open(lockfile_orig, "r") as f:
+        for line in f.readlines():
+            if not line.startswith("source = \"git"):
+                newlines.append(line)
+
+    with open(lockfile, "w") as f:
+        f.writelines(newlines)
 }
 do_configure[postfuncs] += "cargo_common_do_patch_paths"
 
Signature · 1 line
2.34.1
#3 lore raw

PATCH 3/3 drop rust-hello-world recipe

In reply to [email protected]

From: Frederic Martinsons <[email protected]>

we now have more rust/cargo recipes and test that covers various use cases.

---
 meta/conf/distro/include/maintainers.inc      |  1 -
 meta/lib/oeqa/runtime/cases/rust.py           | 12 ----------
 .../packagegroup-core-tools-testapps.bb       |  3 ---
 .../rust-hello-world/0001-enable-LTO.patch    | 24 -------------------
 .../rust-example/rust-hello-world_git.bb      | 19 ---------------
 5 files changed, 59 deletions(-)
 delete mode 100644 meta/recipes-extended/rust-example/rust-hello-world/0001-enable-LTO.patch
 delete mode 100644 meta/recipes-extended/rust-example/rust-hello-world_git.bb
meta/conf/distro/include/maintainers.inc +0 -1
index 1c9c5616d8..202e5739a0 100644
@@ -734,7 +734,6 @@ RECIPE_MAINTAINER:pn-ruby = "Ross Burton <[email protected]>"
 RECIPE_MAINTAINER:pn-run-postinsts = "Ross Burton <[email protected]>"
 RECIPE_MAINTAINER:pn-rust = "Randy MacLeod <[email protected]>"
 RECIPE_MAINTAINER:pn-rust-cross-canadian-${TRANSLATED_TARGET_ARCH} = "Randy MacLeod <[email protected]>"
-RECIPE_MAINTAINER:pn-rust-hello-world = "Randy MacLeod <[email protected]>"
 RECIPE_MAINTAINER:pn-rust-llvm = "Randy MacLeod <[email protected]>"
 RECIPE_MAINTAINER:pn-rxvt-unicode = "Unassigned <[email protected]>"
 RECIPE_MAINTAINER:pn-sato-screenshot = "Ross Burton <[email protected]>"
meta/lib/oeqa/runtime/cases/rust.py +0 -12
index c9c60e16fd..9bf0312669 100644
@@ -48,15 +48,3 @@ class RustCompileTest(OERuntimeTestCase):
         status, output = self.target.run('cargo run --manifest-path=/tmp/hello/Cargo.toml')
         msg = 'running compiled file failed, output: %s' % output
         self.assertEqual(status, 0, msg=msg)
-
-class RustHelloworldTest(OERuntimeTestCase):
-    @OETestDepends(['ssh.SSHTest.test_ssh'])
-    @OEHasPackage(['rust-hello-world'])
-    def test_rusthelloworld(self):
-        cmd = "rust-hello-world"
-        status, output = self.target.run(cmd)
-        msg = 'Exit status was not 0. Output: %s' % output
-        self.assertEqual(status, 0, msg=msg)
-
-        msg = 'Incorrect output: %s' % output
-        self.assertEqual(output, "Hello, world!", msg=msg)
meta/recipes-core/packagegroups/packagegroup-core-tools-testapps.bb +0 -3
index e05e329020..1fee1c925d 100644
@@ -25,8 +25,6 @@ GOTOOLS ?= "go-helloworld"
 GOTOOLS:powerpc ?= ""
 GOTOOLS:riscv32 ?= ""
 
-RUSTTOOLS ?= "rust-hello-world"
-
 GSTEXAMPLES ?= "gst-examples"
 GSTEXAMPLES:riscv64 = ""
 
@@ -59,5 +57,4 @@ RDEPENDS:${PN} = "\
     ${@bb.utils.contains('DISTRO_FEATURES', 'x11 opengl', "${X11GLTOOLS}", "", d)} \
     ${@bb.utils.contains('DISTRO_FEATURES', '3g', "${3GTOOLS}", "", d)} \
     ${GOTOOLS} \
-    ${RUSTTOOLS} \
     "
meta/recipes-extended/rust-example/rust-hello-world/0001-enable-LTO.patch deleted +0 -24
deleted file mode 100644
index f319545ee1..0000000000
@@ -1,24 +0,0 @@
-From fa40b874f6470ec11a8fd7b0c9909d0cdd2d6feb Mon Sep 17 00:00:00 2001
-From: Dan Callaghan <[email protected]>
-Date: Fri, 5 Feb 2021 08:56:34 +1000
-Subject: [PATCH] enable LTO
-
-Upstream-Status: Pending
----
- Cargo.toml | 3 +++
- 1 file changed, 3 insertions(+)
-
-diff --git a/Cargo.toml b/Cargo.toml
-index 7a2f6c8..cdb6b5d 100644
---- a/Cargo.toml
-+++ b/Cargo.toml
-@@ -3,3 +3,6 @@
- name = "rust-hello-world"
- version = "0.0.1"
- authors = ["Cody P Schafer <[email protected]>"]
-+
-+[profile.release]
-+lto = true
--- 
-2.28.0
-
meta/recipes-extended/rust-example/rust-hello-world_git.bb deleted +0 -19
deleted file mode 100644
index 1d91109b51..0000000000
@@ -1,19 +0,0 @@
-inherit cargo
-
-SRC_URI = "git://github.com/meta-rust/rust-hello-world.git;protocol=https;branch=master"
-SRCREV="e0fa23f1a3cb1eb1407165bd2fc36d2f6e6ad728"
-LIC_FILES_CHKSUM="file://COPYRIGHT;md5=e6b2207ac3740d2d01141c49208c2147"
-
-SRC_URI += "\
-    file://0001-enable-LTO.patch \
-    "
-
-UPSTREAM_CHECK_COMMITS = "1"
-
-SUMMARY = "Hello World by Cargo for Rust"
-HOMEPAGE = "https://github.com/meta-rust/rust-hello-world"
-LICENSE = "MIT | Apache-2.0"
-
-S = "${WORKDIR}/git"
-
-BBCLASSEXTEND = "native"
Signature · 1 line
2.34.1
#4 lore raw
In reply to [email protected]
Quoted 39 lines · Le mer. 2 août 2023, 15:16, <[email protected]> a écrit :

From: Frederic Martinsons <[email protected]>

This patch series force the usage of --frozen instead of --offline during cargo build.
This has the advantage to be sure that Cargo.lock file will not be modified.
Moreover, raise a clear error when Cargo.lock is not present. For this to work, we must modify ourself the Cargo.lock before building in case there are git repo that have been patched.

The last commit was for removing rust-hello-world, patch to the documentation project will follow.

The following changes since commit
692e414aed5313ff275b69e93179aa7c559700f3:

ffmpeg: Fix wrong code found with gas/2.41 (2023-08-01 09:49:14 +0100)

are available in the Git repository at:

https://gitlab.com/fmartinsons/openembedded-core 15104-check-cargo-lock

Frederic Martinsons (3):
cargo.bbclass: Use --frozen flag for cargo operations cargo_common.bbclass: Handle Cargo.lock modifications for git
dependencies
drop rust-hello-world recipe

 meta/classes-recipe/cargo.bbclass             |  7 +++-
 meta/classes-recipe/cargo_common.bbclass      | 41 +++++++++++++++++++
 meta/conf/distro/include/maintainers.inc      |  1 -
 meta/lib/oeqa/runtime/cases/rust.py           | 12 ------
 .../packagegroup-core-tools-testapps.bb       |  3 --
 .../rust-hello-world/0001-enable-LTO.patch    | 24 -----------
 .../rust-example/rust-hello-world_git.bb      | 19 ---------
 7 files changed, 47 insertions(+), 60 deletions(-)
 delete mode 100644
meta/recipes-extended/rust-example/rust-hello-world/0001-enable-LTO.patch
 delete mode 100644 meta/recipes-extended/rust-example/
Quoted 5 lines

rust-hello-world_git.bb

--
2.34.1

Sorry, I missed the v2 part in header. These are the follow up after remarks made yesterday (about removal of rust-hello-world and robustness when modifying Cargo.lock file in place)

Quoted 1 line