Skip to contentSkip to conversation

rust: Fix reproducibility with shared source trees

PATCH Superseded [email protected] 3 participants 6 messages Jul 25 – Jul 29

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/3 rust: Fix reproducibility with shared source trees

From: Deepesh Varatharajan <[email protected]>

Backport an upstream Rust commit to fix a reproducibility issue when Rust family recipes share the same source tree.

---
 ...ild-script-path-leakage-in-artifacts.patch | 28 +++++++++++++++++++
 meta/recipes-devtools/rust/rust-source.inc    |  1 +
 2 files changed, 29 insertions(+)
 create mode 100644 meta/recipes-devtools/rust/files/0005-remap-OUT_DIR-paths-to-fix-build-script-path-leakage-in-artifacts.patch
meta/recipes-devtools/rust/files/0005-remap-OUT_DIR-paths-to-fix-build-script-path-leakage-in-artifacts.patch new file +28 -0
new file mode 100644
index 0000000000..8c93624df7
@@ -0,0 +1,28 @@
+Backport a commit that remap OUT_DIR paths to fix build script path leakage in artifacts
+
+Upstream-Status: Backport [https://github.com/rust-lang/rust/commit/e743a4db62336027dcc1f1a755f4ca35f56b67df]
+
+Signed-off-by: Deepesh Varatharajan <[email protected]>
+
+diff --git a/src/bootstrap/src/core/builder/cargo.rs b/src/bootstrap/src/core/builder/cargo.rs
+index f785254..0264667 100644
+--- a/src/bootstrap/src/core/builder/cargo.rs
++++ b/src/bootstrap/src/core/builder/cargo.rs
+@@ -1077,6 +1077,8 @@ impl Builder<'_> {
+                         // rustc creates absolute paths (in part bc of the `rust-src` unremap
+                         // and for working directory) so let's remap the build directory as well.
+                         format!("{}={map_to}", self.build.src.display()),
++                        // remap OUT_DIR so they don't leak into artifacts.
++                        format!("{}={map_to}/out", self.build.out.display()),
+                     ]
+                     .join("\t");
+                     cargo.env("RUSTC_DEBUGINFO_MAP", map);
+@@ -1097,6 +1099,8 @@ impl Builder<'_> {
+                         // rustc creates absolute paths (in part bc of the `rust-src` unremap
+                         // and for working directory) so let's remap the build directory as well.
+                         format!("{}={map_to}", self.build.src.display()),
++                        // remap OUT_DIR so they don't leak into artifacts.
++                        format!("{}={map_to}/out", self.build.out.display()),
+                     ]
+                     .join("\t");
+                     cargo.env("RUSTC_DEBUGINFO_MAP", map);
meta/recipes-devtools/rust/rust-source.inc +1 -0
index a8d8de7165..5678e00671 100644
@@ -9,6 +9,7 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n
             file://0002-Fix-rust-build-failure-with-unstable-options.patch;patchdir=${RUSTSRC} \
             file://0003-explicit-tail-calls-disable-two-tests.patch;patchdir=${RUSTSRC} \
             file://0004-explicit-tail-calls-ignore-tests-on-unsupported-llvm.patch;patchdir=${RUSTSRC} \
+            file://0005-remap-OUT_DIR-paths-to-fix-build-script-path-leakage-in-artifacts.patch;patchdir=${RUSTSRC} \
 "
 SRC_URI[rust.sha256sum] = "77a6ff3003a4ad0cb00697b043c879e3e1a15d945b1a1f63818903bfc3fa8b98"
 
Signature · 1 line
2.49.0
#1 lore raw

PATCH v3 2/3 rust: refactor source handling into a shared work-shared recipe

From: Deepesh Varatharajan <[email protected]>

[YOCTO #15808]

Consolidate Rust source management across all Rust recipes (rust, cargo, libstd-rs and their native/nativesdk variants) into a single shared source tree using the work-shared pattern.

Disk savings: a world build with multilibs enabled, together with nativesdk-rust and nativesdk-cargo, now shares a single ~4.2 GB rustc-src tree instead of maintaining up to 11 independent copies (~46.2 GB total).

Performance (clean core-image-sato build, qemux86-64):

Before: 68m53.507s
After: 65m46.159s
Improvement: 3m07.348s (~4.5%)

New files:

- rust-source.bb: A shared recipe that fetches, unpacks, and patches the rustc source tree into ${TMPDIR}/work-shared/rust-source-${PV}-${PR}. All build/install/packaging tasks are disabled; this recipe exists solely to provide a single copy of the source. Inherits allarch/nopackages, excludes itself from rm_work, and is hidden from world builds. Also defines RUST_BUILD_ARCH locally since this recipe does not inherit rust-common.bbclass but still needs the mapping for snapshot fetching via rust-snapshot.inc.

- common-source.inc: Included by consumer recipes (rust, cargo, libstd-rs) to disable their local do_fetch/do_unpack/do_patch tasks, clear SRC_URI, and add task dependencies on rust-source-${PV} for configure, licensing, and SPDX generation. Also provides is_work_shared_spdx() override for correct SPDX shared-workdir detection.

Modified files:

- rust-source.inc:
- Replace RUST_VERSION with explicit MAJOR_VER/MINOR_VER/PATCH_VER and PV.
- Set UNPACKDIR to the work-shared location so all consumers share one source tree via RUSTSRC and S.

- rust_git.bb (renamed from rust_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- Set B = ${WORKDIR}/build to separate build artifacts from shared source.
- Set CARGO_HOME = ${WORKDIR}/cargo_home to isolate cargo caching.
- Set build-dir = "rust-build" in config.toml to distinguish Yocto's build directory (B), which holds config.toml, from the bootstrap's own build output directory. Without this, bootstrap defaults to "build/" inside B, creating an ambiguous build/build/ nesting.
- Use absolute ${S}/src/bootstrap/bootstrap.py path since cwd is now B.
- Update install functions to reference rust-build/ instead of build/.
- Update do_test_compile to use absolute ${S}/src/tools/ path.
- Change do_rust_setup_snapshot from 'after do_unpack' to depend on

    rust-source-${PV}:do_unpack (since local do_unpack is removed).

- cargo_git.bb (renamed from cargo_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- Change do_cargo_setup_snapshot from 'after do_unpack' to depend on

    rust-source-${PV}:do_unpack.

- libstd-rs_git.bb (renamed from libstd-rs_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.

- rust-cross-canadian_git.bb (renamed from rust-cross-canadian_1.96.1.bb):
- Rename only (version now comes from rust-source.inc via PV).

- rust-common.bbclass:
- Add a second --remap-path-prefix for ${TMPDIR}/work-shared so debug info from the shared source tree is correctly remapped.
- Retain RUST_BUILD_ARCH definition so that consumer recipes (rust, cargo, libstd-rs) which inherit this class can resolve snapshot directory names used by do_rust_setup_snapshot and do_cargo_setup_snapshot.

- lib/oeqa/selftest/cases/rust.py:
- Adapt the selftest to the build-dir/source-dir split: use B for build artifacts and RUSTSRC for source paths.
- Update remote-test-server copy path from build/ to rust-build/.
- Pass --build-dir and --config explicitly to bootstrap.py.

- conf/distro/include/maintainers.inc:
- Add maintainer entry for the new rust-source recipe.

License-Update: recipe file renames only, no license change

---
 meta/classes-recipe/rust-common.bbclass       |  4 +-
 meta/conf/distro/include/maintainers.inc      |  1 +
 meta/lib/oeqa/selftest/cases/rust.py          |  9 ++-
 .../rust/{cargo_1.96.1.bb => cargo_git.bb}    |  8 ++-
 meta/recipes-devtools/rust/common-source.inc  | 28 ++++++++++
 .../{libstd-rs_1.96.1.bb => libstd-rs_git.bb} |  4 ++
 ...n_1.96.1.bb => rust-cross-canadian_git.bb} |  0
 meta/recipes-devtools/rust/rust-source.bb     | 56 +++++++++++++++++++
 meta/recipes-devtools/rust/rust-source.inc    | 16 +++++-
 .../rust/{rust_1.96.1.bb => rust_git.bb}      | 31 +++++++---
 10 files changed, 140 insertions(+), 17 deletions(-)
 rename meta/recipes-devtools/rust/{cargo_1.96.1.bb => cargo_git.bb} (91%)
 create mode 100644 meta/recipes-devtools/rust/common-source.inc
 rename meta/recipes-devtools/rust/{libstd-rs_1.96.1.bb => libstd-rs_git.bb} (92%)
 rename meta/recipes-devtools/rust/{rust-cross-canadian_1.96.1.bb => rust-cross-canadian_git.bb} (100%)
 create mode 100644 meta/recipes-devtools/rust/rust-source.bb
 rename meta/recipes-devtools/rust/{rust_1.96.1.bb => rust_git.bb} (93%)
meta/classes-recipe/rust-common.bbclass +3 -1
index 057aeda67a..6bc42016d1 100644
@@ -14,7 +14,9 @@ FILES:${PN}-dev += "${rustlibdir}/*.rlib ${rustlibdir}/*.rmeta"
 FILES:${PN}-dbg += "${rustlibdir}/.debug"
 
 RUSTLIB ?= "-L ${STAGING_DIR_HOST}${rustlibdir}"
-RUST_DEBUG_REMAP = "--remap-path-prefix=${WORKDIR}=${TARGET_DBGSRC_DIR}"
+RUST_DEBUG_REMAP = "--remap-path-prefix=${WORKDIR}=${TARGET_DBGSRC_DIR} \
+                    --remap-path-prefix=${TMPDIR}/work-shared=${TARGET_DBGSRC_DIR} \
+"
 RUSTFLAGS += "${RUSTLIB} ${RUST_DEBUG_REMAP}"
 RUSTLIB_DEP ??= "libstd-rs"
 RUST_PANIC_STRATEGY ??= "unwind"
meta/conf/distro/include/maintainers.inc +1 -0
index b49386a27d..e18f17fd81 100644
@@ -783,6 +783,7 @@ RECIPE_MAINTAINER:pn-ruby = "Ross Burton <[email protected]>"
 RECIPE_MAINTAINER:pn-run-postinsts = "Ross Burton <[email protected]>"
 RECIPE_MAINTAINER:pn-rust = "Randy MacLeod <[email protected]>"
 RECIPE_MAINTAINER:pn-rust-cross-canadian-${TRANSLATED_TARGET_ARCH} = "Randy MacLeod <[email protected]>"
+RECIPE_MAINTAINER:pn-rust-source-1.96.1 = "Deepesh Varatharajan <[email protected]>"
 RECIPE_MAINTAINER:pn-rxvt-unicode = "Unassigned <[email protected]>"
 RECIPE_MAINTAINER:pn-sassc = "Simone Weiß <[email protected]>"
 RECIPE_MAINTAINER:pn-sato-icon-theme = "Richard Purdie <[email protected]>"
meta/lib/oeqa/selftest/cases/rust.py +6 -3
index c961a8f652..4111d72044 100644
@@ -46,7 +46,8 @@ class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase):
         recipe = "rust"
         start_time = time.time()
         bitbake("{} -c test_compile".format(recipe))
-        builddir = get_bb_var("RUSTSRC", "rust")
+        builddir = get_bb_var("B", "rust")
+        sourcedir = get_bb_var("RUSTSRC", "rust")
         # build core-image-minimal with required packages
         default_installed_packages = ["libgcc", "libstdc++", "libatomic", "libgomp", "libzstd", "llvm", "openssl"]
         features = []
@@ -111,7 +112,7 @@ class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase):
             # Copy remote-test-server to image through scp
             host_sys = get_bb_var("RUST_BUILD_SYS", "rust")
             ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user="root")
-            ssh.copy_to(builddir + "/build/" + host_sys + "/stage2-tools-bin/remote-test-server","~/")
+            ssh.copy_to(builddir + "/rust-build/" + host_sys + "/stage2-tools-bin/remote-test-server","~/")
             # Execute remote-test-server on image through background ssh
             command = '~/remote-test-server --bind 0.0.0.0:12345 -v'
             sshrun=subprocess.Popen(("ssh", '-o',  'UserKnownHostsFile=/dev/null', '-o',  'StrictHostKeyChecking=no', '-f', "root@%s" % qemu.ip, command), shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
@@ -131,8 +132,10 @@ class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase):
             # PowerPC mac99 QEMU has 768MB RAM limit, so we need to minimize test binary sizes
             cmd = cmd + " export RUSTFLAGS='-C strip=debuginfo -Clink-arg=-lz -Clink-arg=-lzstd';"
             # Trigger testing.
+            # Run bootstrap from sourcedir with explicit --build-dir and --config
+            # pointing to the separate build directory.
             cmd = cmd + " export TEST_DEVICE_ADDR=\"%s:12345\";" % qemu.ip
-            cmd = cmd + " cd %s; python3 src/bootstrap/bootstrap.py test %s --target %s" % (builddir, testargs, targetsys)
+            cmd = cmd + " cd %s; python3 src/bootstrap/bootstrap.py --build-dir %s/rust-build --config %s/config.toml test %s --target %s" % (sourcedir, builddir, builddir, testargs, targetsys)
             retval = runCmd(cmd)
             end_time = time.time()
 
meta/recipes-devtools/rust/cargo_git.bb renamed from meta/recipes-devtools/rust/cargo_1.96.1.bb +7 -1
similarity index 91%
rename from meta/recipes-devtools/rust/cargo_1.96.1.bb
rename to meta/recipes-devtools/rust/cargo_git.bb
index adc1f0d331..16f8a76310 100644
@@ -13,6 +13,8 @@ LIC_FILES_CHKSUM = " \
 
 require rust-source.inc
 require rust-snapshot.inc
+# Use shared source tree from rust-source.bb; disables local fetch/unpack/patch
+require common-source.inc
 
 S = "${RUSTSRC}/src/tools/cargo"
 CARGO_VENDORING_DIRECTORY = "${RUSTSRC}/vendor"
@@ -21,6 +23,9 @@ CVE_PRODUCT = "rust-lang:cargo"
 
 inherit cargo pkgconfig
 
+# Explicit CARGO_HOME avoids cargo caching in the shared source tree
+CARGO_HOME = "${WORKDIR}/cargo_home"
+
 DEBUG_PREFIX_MAP += "-ffile-prefix-map=${RUSTSRC}/vendor=${TARGET_DBGSRC_DIR}"
 
 do_cargo_setup_snapshot () {
@@ -32,9 +37,10 @@ do_cargo_setup_snapshot () {
 	fi
 }
 
-addtask cargo_setup_snapshot after do_unpack before do_configure
+addtask cargo_setup_snapshot before do_configure
 do_cargo_setup_snapshot[dirs] += "${WORKDIR}/${CARGO_SNAPSHOT}"
 do_cargo_setup_snapshot[vardepsexclude] += "UNINATIVE_LOADER"
+do_cargo_setup_snapshot[depends] += "rust-source-${PV}:do_unpack"
 
 do_compile:prepend () {
 	export RUSTC_BOOTSTRAP="1"
meta/recipes-devtools/rust/common-source.inc new file +28 -0
new file mode 100644
index 0000000000..a0d4dc542c
@@ -0,0 +1,28 @@
+# common-source.inc - Disable source tasks for recipes using shared rust-source
+#
+# Recipes that include this file rely on rust-source-${PV} for fetching,
+# unpacking, and patching the Rust source tree. This avoids duplicating the
+# ~4.2 GB rustc-src tree per recipe variant (rust, cargo, libstd-rs and their
+# native/nativesdk counterparts).
+
+# Disable local fetch/unpack/patch since rust-source-${PV} handles them
+do_fetch() {
+        :
+}
+do_fetch[noexec] = "1"
+deltask do_unpack
+deltask do_patch
+
+# Clear SRC_URI to prevent fetcher warnings from stale URIs inherited via .inc
+SRC_URI = ""
+
+# Ensure the shared source is fully ready before dependent tasks run
+do_configure[depends] += "rust-source-${PV}:do_patch"
+do_populate_lic[depends] += "rust-source-${PV}:do_unpack"
+do_create_spdx[depends] += "rust-source-${PV}:do_patch"
+
+# The SPDX class uses is_work_shared_spdx() to detect shared workdirs.
+# Since we redirect S rather than WORKDIR, the default detection fails.
+# Override to always return True so SPDX processing works correctly.
+def is_work_shared_spdx(d):
+    return True
meta/recipes-devtools/rust/libstd-rs_git.bb renamed from meta/recipes-devtools/rust/libstd-rs_1.96.1.bb +4 -0
similarity index 92%
rename from meta/recipes-devtools/rust/libstd-rs_1.96.1.bb
rename to meta/recipes-devtools/rust/libstd-rs_git.bb
index 1ecd6bff3f..939e5824b3 100644
@@ -5,6 +5,7 @@ LICENSE = "Unicode-3.0 AND (Apache-2.0 OR MIT)"
 LIC_FILES_CHKSUM = "file://../../COPYRIGHT;md5=11a3899825f4376896e438c8c753f8dc"
 
 require rust-source.inc
+require common-source.inc
 
 # The dummy crate named `sysroot` represents the standard library target.
 #
@@ -30,6 +31,9 @@ CARGO_FEATURES ?= "panic-unwind backtrace"
 CARGO_BUILD_FLAGS += "--features '${CARGO_FEATURES}'"
 CARGO_VENDORING_DIRECTORY = "${RUSTSRC}/vendor"
 
+# Explicit CARGO_HOME avoids cargo caching in the shared source tree
+CARGO_HOME = "${WORKDIR}/cargo_home"
+
 do_compile:prepend () {
     # For Rust 1.13.0 and newer
     export RUSTC_BOOTSTRAP="1"
meta/recipes-devtools/rust/rust-cross-canadian_git.bb renamed from meta/recipes-devtools/rust/rust-cross-canadian_1.96.1.bb +0 -0
similarity index 100%
rename from meta/recipes-devtools/rust/rust-cross-canadian_1.96.1.bb
rename to meta/recipes-devtools/rust/rust-cross-canadian_git.bb
meta/recipes-devtools/rust/rust-source.bb new file +56 -0

56 lines not on this page · raw message

meta/recipes-devtools/rust/rust-source.inc +13 -3
index 5678e00671..49207a3947 100644
@@ -1,6 +1,12 @@
-RUST_VERSION ?= "${@d.getVar('PV').split('-')[0]}"
+# Version is split into MAJOR_VER and MINOR_VER for easy bumping and
+# to allow consistent use of PV across all consuming recipes.
+MAJOR_VER = "1"
+MINOR_VER = "96"
+PATCH_VER = "1"
 
-SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;name=rust \
+PV = "${MAJOR_VER}.${MINOR_VER}.${PATCH_VER}"
+
+SRC_URI += "https://static.rust-lang.org/dist/rustc-${PV}-src.tar.xz;name=rust \
             file://0001-Define-more-ioctl-codes-on-riscv32gc-unknown-linux-g.patch;patchdir=${RUSTSRC} \
             file://rust-oe-selftest.patch;patchdir=${RUSTSRC} \
             file://repro-issue-fix-with-cc-crate-hashmap.patch;patchdir=${RUSTSRC} \
@@ -13,7 +19,11 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n
 "
 SRC_URI[rust.sha256sum] = "77a6ff3003a4ad0cb00697b043c879e3e1a15d945b1a1f63818903bfc3fa8b98"
 
-RUSTSRC = "${UNPACKDIR}/rustc-${RUST_VERSION}-src"
+# Point UNPACKDIR into the work-shared location so rust-source.bb and all
+# consumers (via S/RUSTSRC) reference a single shared source tree.
+UNPACKDIR = "${TMPDIR}/work-shared/rust-source-${PV}-${PR}/sources"
+RUSTSRC = "${UNPACKDIR}/rustc-${PV}-src"
+S = "${RUSTSRC}"
 
 UPSTREAM_CHECK_URI = "https://forge.rust-lang.org/infra/other-installation-methods.html"
 UPSTREAM_CHECK_REGEX = "rustc-(?P<pver>\d+(\.\d+)+)-src"
meta/recipes-devtools/rust/rust_git.bb renamed from meta/recipes-devtools/rust/rust_1.96.1.bb +22 -9
similarity index 93%
rename from meta/recipes-devtools/rust/rust_1.96.1.bb
rename to meta/recipes-devtools/rust/rust_git.bb
index fa8777826f..4bbe4ed0bd 100644
@@ -4,6 +4,10 @@ SECTION = "devel"
 LICENSE = "Unicode-3.0 AND (Apache-2.0 OR MIT)"
 LIC_FILES_CHKSUM = "file://COPYRIGHT;md5=11a3899825f4376896e438c8c753f8dc"
 
+require rust-source.inc
+require rust-snapshot.inc
+require common-source.inc
+
 inherit rust
 inherit cargo_common
 
@@ -24,6 +28,9 @@ INHIBIT_DEFAULT_RUST_DEPS:class-native = "1"
 PROVIDES:class-native = "virtual/${TARGET_PREFIX}rust"
 
 S = "${RUSTSRC}"
+# Separate build directory from the shared source tree so that multiple
+# variants (target, native, nativesdk) don't conflict in the source dir.
+B = "${WORKDIR}/build"
 
 # Use at your own risk, accepted values are stable, beta and nightly
 RUST_CHANNEL ?= "stable"
@@ -38,6 +45,9 @@ RUST_ALTERNATE_EXE_PATH_NATIVE = "${STAGING_BINDIR_NATIVE}/llvm-config"
 # own vendoring.
 CARGO_DISABLE_BITBAKE_VENDORING = "1"
 
+# Explicit CARGO_HOME avoids cargo caching in the shared source tree
+CARGO_HOME = "${WORKDIR}/cargo_home"
+
 setup_cargo_environment () {
     # The first step is to build bootstrap and some early stage tools,
     # these are build for the same target as the snapshot, e.g.
@@ -66,11 +76,11 @@ do_rust_setup_snapshot () {
         done
     fi
 }
-addtask rust_setup_snapshot after do_unpack before do_configure
+addtask rust_setup_snapshot before do_configure
 addtask do_test_compile after do_configure do_rust_gen_targets
 do_rust_setup_snapshot[dirs] += "${WORKDIR}/rust-snapshot"
 do_rust_setup_snapshot[vardepsexclude] += "UNINATIVE_LOADER"
-do_rust_setup_snapshot[depends] += "patchelf-native:do_populate_sysroot"
+do_rust_setup_snapshot[depends] += "patchelf-native:do_populate_sysroot rust-source-${PV}:do_unpack"
 
 RUSTC_BOOTSTRAP = "${STAGING_BINDIR_NATIVE}/rustc"
 CARGO_BOOTSTRAP = "${STAGING_BINDIR_NATIVE}/cargo"
@@ -173,6 +183,12 @@ python do_configure() {
     # nothing about when trying to build some stage0 tools (like fabricate)
     config.set("build", "build", e(d.getVar("RUST_BUILD_SYS")))
 
+    # Yocto's B (${WORKDIR}/build) is where config.toml lives. Rust's
+    # bootstrap creates its own build output directory inside B; name it
+    # "rust-build" to avoid ambiguity with Yocto's build directory and
+    # prevent a confusing build/build/ nesting.
+    config.set("build", "build-dir", e("rust-build"))
+
     # [install]
     config.add_section("install")
     # ./x.py install doesn't have any notion of "destdir"
@@ -223,13 +239,10 @@ rust_runx () {
 
     oe_cargo_fix_env
 
-    python3 src/bootstrap/bootstrap.py ${@oe.utils.parallel_make_argument(d, '-j %d')} "$@" --verbose
+    python3 ${S}/src/bootstrap/bootstrap.py ${@oe.utils.parallel_make_argument(d, '-j %d')} "$@" --verbose
 }
 rust_runx[vardepsexclude] += "PARALLEL_MAKE"
 
-require rust-source.inc
-require rust-snapshot.inc
-
 INSANE_SKIP:${PN}:class-native = "already-stripped"
 FILES:${PN} += "${libdir}/rustlib"
 FILES:${PN} += "${libdir}/*.so"
@@ -241,7 +254,7 @@ do_compile () {
 do_test_compile[dirs] = "${B}"
 do_test_compile () {
     replace_llvm_config_path
-    rust_runx build src/tools/remote-test-server --target "${RUST_TARGET_SYS}"
+    rust_runx build ${S}/src/tools/remote-test-server --target "${RUST_TARGET_SYS}"
 }
 
 ALLOW_EMPTY:${PN} = "1"
@@ -282,7 +295,7 @@ rust_do_install:class-nativesdk() {
 
     install -d ${D}${bindir}
     for i in cargo-clippy clippy-driver rustfmt; do
-        cp build/${RUST_BUILD_SYS}/stage2-tools/${RUST_HOST_SYS}/release/$i ${D}${bindir}
+        cp rust-build/${RUST_BUILD_SYS}/stage2-tools/${RUST_HOST_SYS}/release/$i ${D}${bindir}
         patchelf --set-rpath "\$ORIGIN/../lib" ${D}${bindir}/$i
     done
 
@@ -322,7 +335,7 @@ rust_do_install:class-target() {
 
     install -d ${D}${bindir}
     for i in ${EXTRA_TOOLS}; do
-        cp build/${RUST_BUILD_SYS}/stage2-tools/${RUST_HOST_SYS}/release/$i ${D}${bindir}
+        cp rust-build/${RUST_BUILD_SYS}/stage2-tools/${RUST_HOST_SYS}/release/$i ${D}${bindir}
         patchelf --set-rpath "\$ORIGIN/../lib" ${D}${bindir}/$i
     done
 
Signature · 1 line
2.49.0
#2 lore raw

Re: [OE-core] [PATCH v3 2/3] rust: refactor source handling into a shared work-shared recipe

Quoted 2 lines · Deepesh via lists.openembedded.org Varatharajan

From: Deepesh Varatharajan <[email protected]>

Hi Deepesh,

Quoted 20 lines

[YOCTO #15808]

Consolidate Rust source management across all Rust recipes (rust, cargo, libstd-rs and their native/nativesdk variants) into a single shared source tree using the work-shared pattern.

Disk savings: a world build with multilibs enabled, together with nativesdk-rust and nativesdk-cargo, now shares a single ~4.2 GB rustc-src tree instead of maintaining up to 11 independent copies (~46.2 GB total).

Performance (clean core-image-sato build, qemux86-64):

meta/recipes-devtools/rust/rust-source.bb new file +5 -0
@@ -0,0 +1,56 @@
+# rust-source.bb - Shared source recipe for the Rust toolchain
+#
+# This recipe fetches, unpacks, and patches the rustc source tree into a
+# work-shared location so that rust, cargo, libstd-rs, and their native/nativesdk
+# variants all share a single copy of the source. This eliminates duplication

meta/recipes-devtools/rust/rust-source.bb line 15

Quoted 10 lines
meta/recipes-devtools/rust/rust-source.bb new file +10 -0
+# (up to 11 copies of the ~4.2 GB source tree in multilib + nativesdk builds)
+# and significantly reduces disk usage (~46.2 GB -> ~4.2 GB for rustc-src).
+#
+# Only do_fetch, do_unpack, and do_patch are meaningful here. All other
+# tasks are disabled since this recipe exists solely to provide sources.
+
+SUMMARY = "Rust sources"
+HOMEPAGE = "http://www.rust-lang.org"
+SECTION = "devel"
+LICENSE = "(Unicode-3.0 AND (Apache-2.0 OR MIT)"

We are missing a closing parentheses here. I believe you can remove the opening one.

Thanks,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw

Re: [OE-core] [PATCH v3 2/3] rust: refactor source handling into a shared work-shared recipe

In reply to [email protected]
Quoted 97 lines · Deepesh via lists.openembedded.org

From: Deepesh Varatharajan <[email protected]>

[YOCTO #15808]

Consolidate Rust source management across all Rust recipes (rust, cargo, libstd-rs and their native/nativesdk variants) into a single shared source tree using the work-shared pattern.

Disk savings: a world build with multilibs enabled, together with nativesdk-rust and nativesdk-cargo, now shares a single ~4.2 GB rustc-src tree instead of maintaining up to 11 independent copies (~46.2 GB total).

Performance (clean core-image-sato build, qemux86-64):

Before: 68m53.507s
After:  65m46.159s
Improvement: 3m07.348s (~4.5%)

New files:

- rust-source.bb: A shared recipe that fetches, unpacks, and patches the rustc source tree into ${TMPDIR}/work-shared/rust-source-${PV}-${PR}. All build/install/packaging tasks are disabled; this recipe exists solely to provide a single copy of the source. Inherits allarch/nopackages, excludes itself from rm_work, and is hidden from world builds. Also defines RUST_BUILD_ARCH locally since this recipe does not inherit rust-common.bbclass but still needs the mapping for snapshot fetching via rust-snapshot.inc.

- common-source.inc: Included by consumer recipes (rust, cargo, libstd-rs) to disable their local do_fetch/do_unpack/do_patch tasks, clear SRC_URI, and add task dependencies on rust-source-${PV} for configure, licensing, and SPDX generation. Also provides is_work_shared_spdx() override for correct SPDX shared-workdir detection.

Modified files:

- rust-source.inc:
  - Replace RUST_VERSION with explicit MAJOR_VER/MINOR_VER/PATCH_VER and PV.
  - Set UNPACKDIR to the work-shared location so all consumers share one source tree via RUSTSRC and S.

- rust_git.bb (renamed from rust_1.96.1.bb):
  - Add 'require common-source.inc' to use the shared source tree.
  - Set B = ${WORKDIR}/build to separate build artifacts from shared source.
  - Set CARGO_HOME = ${WORKDIR}/cargo_home to isolate cargo caching.
  - Set build-dir = "rust-build" in config.toml to distinguish Yocto's build directory (B), which holds config.toml, from the bootstrap's own build output directory. Without this, bootstrap defaults to "build/" inside B, creating an ambiguous build/build/ nesting.
  - Use absolute ${S}/src/bootstrap/bootstrap.py path since cwd is now B.
  - Update install functions to reference rust-build/ instead of build/.
  - Update do_test_compile to use absolute ${S}/src/tools/ path.
  - Change do_rust_setup_snapshot from 'after do_unpack' to depend on rust-source-${PV}:do_unpack (since local do_unpack is removed).

- cargo_git.bb (renamed from cargo_1.96.1.bb):
  - Add 'require common-source.inc' to use the shared source tree.
  - Change do_cargo_setup_snapshot from 'after do_unpack' to depend on rust-source-${PV}:do_unpack.

- libstd-rs_git.bb (renamed from libstd-rs_1.96.1.bb):
  - Add 'require common-source.inc' to use the shared source tree.

- rust-cross-canadian_git.bb (renamed from rust-cross-canadian_1.96.1.bb):
  - Rename only (version now comes from rust-source.inc via PV).

- rust-common.bbclass:
  - Add a second --remap-path-prefix for ${TMPDIR}/work-shared so debug info from the shared source tree is correctly remapped.
  - Retain RUST_BUILD_ARCH definition so that consumer recipes (rust, cargo, libstd-rs) which inherit this class can resolve snapshot directory names used by do_rust_setup_snapshot and do_cargo_setup_snapshot.

- lib/oeqa/selftest/cases/rust.py:
  - Adapt the selftest to the build-dir/source-dir split: use B for build artifacts and RUSTSRC for source paths.
  - Update remote-test-server copy path from build/ to rust-build/.
  - Pass --build-dir and --config explicitly to bootstrap.py.

- conf/distro/include/maintainers.inc:
  - Add maintainer entry for the new rust-source recipe.

License-Update: recipe file renames only, no license change

---
 meta/classes-recipe/rust-common.bbclass       |  4 +- meta/conf/distro/include/maintainers.inc      |  1 + meta/lib/oeqa/selftest/cases/rust.py          |  9 ++- .../rust/{cargo_1.96.1.bb => cargo_git.bb}    |  8 ++- meta/recipes-devtools/rust/common-source.inc  | 28 ++++++++++ .../{libstd-rs_1.96.1.bb => libstd-rs_git.bb} |  4 ++ ...n_1.96.1.bb => rust-cross-canadian_git.bb} |  0 meta/recipes-devtools/rust/rust-source.bb     | 56 +++++++++++++++++++ meta/recipes-devtools/rust/rust-source.inc    | 16 +++++- .../rust/{rust_1.96.1.bb => rust_git.bb}      | 31 +++++++---

Quoted 7 lines
 10 files changed, 140 insertions(+), 17 deletions(-)

 rename meta/recipes-devtools/rust/{cargo_1.96.1.bb => cargo_git.bb} (91%)
 create mode 100644 meta/recipes-devtools/rust/common-source.inc
 rename meta/recipes-devtools/rust/{libstd-rs_1.96.1.bb => libstd-rs_git.bb} (92%) rename meta/recipes-devtools/rust/{rust-cross-canadian_1.96.1.bb => rust-cross-canadian_git.bb} (100%) create mode 100644 meta/recipes-devtools/rust/rust-source.bb
 rename meta/recipes-devtools/rust/{rust_1.96.1.bb => rust_git.bb} (93%)

In the patch review call we noted that "_git" isn't accurate as these are version'd recipes, not git based. There was a suggestion to just remove that piece but without the versions, it would make it harder to have multiple rust versions in parallel should that be needed. This is why the gcc recipes retain a version in the filenames.

Our suggestion would be to have rust match what gcc does for those two reasons.

Otherwise with the LICENSE bracket fix, we think is is probably ready to merge. The space savings are nice to have!

Cheers,

Richard

#4 lore raw

Re: [OE-core] [PATCH v3 2/3] rust: refactor source handling into a shared work-shared recipe

Quoted 117 lines · Richard Purdie

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Quoted 98 lines · Deepesh via lists.openembedded.org

From: Deepesh Varatharajan <[email protected]>

[YOCTO #15808]

Consolidate Rust source management across all Rust recipes (rust, cargo, libstd-rs and their native/nativesdk variants) into a single shared source tree using the work-shared pattern.

Disk savings: a world build with multilibs enabled, together with nativesdk-rust and nativesdk-cargo, now shares a single ~4.2 GB rustc-src tree instead of maintaining up to 11 independent copies (~46.2 GB total).

Performance (clean core-image-sato build, qemux86-64):

Before: 68m53.507s
After: 65m46.159s
Improvement: 3m07.348s (~4.5%)

New files:

- rust-source.bb: A shared recipe that fetches, unpacks, and patches the rustc source tree into ${TMPDIR}/work-shared/rust-source-${PV}-${PR}. All build/install/packaging tasks are disabled; this recipe exists solely to provide a single copy of the source. Inherits allarch/nopackages, excludes itself from rm_work, and is hidden from world builds. Also defines RUST_BUILD_ARCH locally since this recipe does not inherit rust-common.bbclass but still needs the mapping for snapshot fetching via rust-snapshot.inc.

- common-source.inc: Included by consumer recipes (rust, cargo, libstd-rs) to disable their local do_fetch/do_unpack/do_patch tasks, clear SRC_URI, and add task dependencies on rust-source-${PV} for configure, licensing, and SPDX generation. Also provides is_work_shared_spdx() override for correct SPDX shared-workdir detection.

Modified files:

- rust-source.inc:
- Replace RUST_VERSION with explicit MAJOR_VER/MINOR_VER/PATCH_VER and PV.
- Set UNPACKDIR to the work-shared location so all consumers share one source tree via RUSTSRC and S.

- rust_git.bb (renamed from rust_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- Set B = ${WORKDIR}/build to separate build artifacts from shared source.
- Set CARGO_HOME = ${WORKDIR}/cargo_home to isolate cargo caching.
- Set build-dir = "rust-build" in config.toml to distinguish Yocto's build directory (B), which holds config.toml, from the bootstrap's own build output directory. Without this, bootstrap defaults to "build/" inside B, creating an ambiguous build/build/ nesting.
- Use absolute ${S}/src/bootstrap/bootstrap.py path since cwd is now B.
- Update install functions to reference rust-build/ instead of build/.
- Update do_test_compile to use absolute ${S}/src/tools/ path.
- Change do_rust_setup_snapshot from 'after do_unpack' to depend on

     rust-source-${PV}:do_unpack (since local do_unpack is removed).

- cargo_git.bb (renamed from cargo_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- Change do_cargo_setup_snapshot from 'after do_unpack' to depend on

     rust-source-${PV}:do_unpack.

- libstd-rs_git.bb (renamed from libstd-rs_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.

- rust-cross-canadian_git.bb (renamed from rust-cross-canadian_1.96.1.bb):
- Rename only (version now comes from rust-source.inc via PV).

- rust-common.bbclass:
- Add a second --remap-path-prefix for ${TMPDIR}/work-shared so debug info from the shared source tree is correctly remapped.
- Retain RUST_BUILD_ARCH definition so that consumer recipes (rust, cargo, libstd-rs) which inherit this class can resolve snapshot directory names used by do_rust_setup_snapshot and do_cargo_setup_snapshot.

- lib/oeqa/selftest/cases/rust.py:
- Adapt the selftest to the build-dir/source-dir split: use B for build artifacts and RUSTSRC for source paths.
- Update remote-test-server copy path from build/ to rust-build/.
- Pass --build-dir and --config explicitly to bootstrap.py.

- conf/distro/include/maintainers.inc:
- Add maintainer entry for the new rust-source recipe.

License-Update: recipe file renames only, no license change

---
  meta/classes-recipe/rust-common.bbclass       |  4 +-
  meta/conf/distro/include/maintainers.inc      |  1 +
  meta/lib/oeqa/selftest/cases/rust.py          |  9 ++-
  .../rust/{cargo_1.96.1.bb => cargo_git.bb}    |  8 ++-
  meta/recipes-devtools/rust/common-source.inc  | 28 ++++++++++
  .../{libstd-rs_1.96.1.bb => libstd-rs_git.bb} |  4 ++
  ...n_1.96.1.bb => rust-cross-canadian_git.bb} |  0
  meta/recipes-devtools/rust/rust-source.bb     | 56 +++++++++++++++++++
  meta/recipes-devtools/rust/rust-source.inc    | 16 +++++-
  .../rust/{rust_1.96.1.bb => rust_git.bb}      | 31 +++++++---
  10 files changed, 140 insertions(+), 17 deletions(-)
Quoted 6 lines

rename meta/recipes-devtools/rust/{cargo_1.96.1.bb => cargo_git.bb} (91%)
create mode 100644 meta/recipes-devtools/rust/common-source.inc
rename meta/recipes-devtools/rust/{libstd-rs_1.96.1.bb => libstd-rs_git.bb} (92%) rename meta/recipes-devtools/rust/{rust-cross-canadian_1.96.1.bb => rust-cross-canadian_git.bb} (100%) create mode 100644 meta/recipes-devtools/rust/rust-source.bb
rename meta/recipes-devtools/rust/{rust_1.96.1.bb => rust_git.bb} (93%)

In the patch review call we noted that "_git" isn't accurate as these are version'd recipes, not git based. There was a suggestion to just remove that piece but without the versions, it would make it harder to have multiple rust versions in parallel should that be needed. This is why the gcc recipes retain a version in the filenames.

Our suggestion would be to have rust match what gcc does for those two reasons.

Quoted 6 lines

Otherwise with the LICENSE bracket fix, we think is is probably ready to merge. The space savings are nice to have!

Cheers,

Richard

Hi Richard,

Thanks for the review feedback!

Makes sense I'll rename the recipes to include the version in the filename, matching the gcc convention. Also get the LICENSE bracket fix in. Will send a new version shortly.

Regards,
Deepesh

#5 lore raw

PATCH v3 3/3 rust: Move do_update_snapshot to rust-source

In reply to [email protected]

From: Deepesh Varatharajan <[email protected]>

Move do_update_snapshot from rust to rust-source, since rust-source now owns do_fetch/do_unpack/do_patch tasks after the shared source refactoring.

---
 meta/recipes-devtools/rust/rust-source.bb | 72 ++++++++++++++++++++++
 meta/recipes-devtools/rust/rust_git.bb    | 73 -----------------------
 2 files changed, 72 insertions(+), 73 deletions(-)
meta/recipes-devtools/rust/rust-source.bb +72 -0
index 0afef845fc..4b3c3a6d52 100644
@@ -54,3 +54,75 @@ B = "${WORKDIR}/build"
 RUST_BUILD_ARCH = "${@{'ppc': 'powerpc', 'ppc64': 'powerpc64', 'ppc64le': 'powerpc64le', 'riscv64': 'riscv64gc'}.get(d.getVar('BUILD_ARCH'), d.getVar('BUILD_ARCH'))}"
 
 EXCLUDE_FROM_WORLD = "1"
+
+addtask do_update_snapshot after do_patch
+do_update_snapshot[nostamp] = "1"
+
+# Run with `bitbake rust-source-${PV} -c update_snapshot` to update `rust-snapshot.inc`
+# with the checksums for the rust snapshot associated with this rustc-src tarball.
+python do_update_snapshot() {
+    import json
+    import re
+    import sys
+
+    from collections import defaultdict
+
+    key_value_pairs = {}
+    with open(os.path.join(d.getVar("S"), "src", "stage0")) as f:
+        for line in f:
+            # Skip empty lines or comments
+            if not line.strip() or line.startswith("#"):
+                continue
+            # Split the line into key and value using '=' as separator
+            match = re.match(r'(\S+)\s*=\s*(\S+)', line.strip())
+            if match:
+                key = match.group(1)
+                value = match.group(2)
+                key_value_pairs[key] = value
+    # Extract the required values from key_value_pairs
+    config_dist_server = key_value_pairs.get('dist_server', '')
+    compiler_date = key_value_pairs.get('compiler_date', '')
+    compiler_version = key_value_pairs.get('compiler_version', '')
+
+    src_uri = defaultdict(list)
+    # Assuming checksums_sha256 is now a key-value pair like: checksum_key = checksum_value
+    for k, v in key_value_pairs.items():
+        # Match the pattern for checksums
+        if "dist" in k and "tar.xz" in k:
+            m = re.search(f"dist/{compiler_date}/(?P<component>.*)-{compiler_version}-(?P<arch>.*)-unknown-linux-gnu\\.tar\\.xz", k)
+            if m:
+                component = m.group('component')
+                arch = m.group('arch')
+                src_uri[arch].append(f"SRC_URI[{component}-snapshot-{arch}.sha256sum] = \"{v}\"")
+    # Create the snapshot string with the extracted values
+    snapshot = """\
+## This is information on the rust-snapshot (binary) used to build our current release.
+## snapshot info is taken from rust/src/stage0
+## Rust is self-hosting and bootstraps itself with a pre-built previous version of itself.
+## The exact (previous) version that has been used is specified in the source tarball.
+## The version is replicated here.
+
+SNAPSHOT_VERSION = "%s"
+
+""" % compiler_version
+    # Add the checksum components to the snapshot
+    for arch, components in src_uri.items():
+        snapshot += "\n".join(components) + "\n\n"
+    # Add the additional snapshot URIs
+    snapshot += """\
+SRC_URI += " \\
+    ${RUST_DIST_SERVER}/dist/${RUST_STD_SNAPSHOT}.tar.xz;name=rust-std-snapshot-${RUST_BUILD_ARCH};subdir=rust-snapshot-components \\
+    ${RUST_DIST_SERVER}/dist/${RUSTC_SNAPSHOT}.tar.xz;name=rustc-snapshot-${RUST_BUILD_ARCH};subdir=rust-snapshot-components \\
+    ${RUST_DIST_SERVER}/dist/${CARGO_SNAPSHOT}.tar.xz;name=cargo-snapshot-${RUST_BUILD_ARCH};subdir=rust-snapshot-components \\
+"
+
+RUST_DIST_SERVER = "%s"
+
+RUST_STD_SNAPSHOT = "rust-std-${SNAPSHOT_VERSION}-${RUST_BUILD_ARCH}-unknown-linux-gnu"
+RUSTC_SNAPSHOT = "rustc-${SNAPSHOT_VERSION}-${RUST_BUILD_ARCH}-unknown-linux-gnu"
+CARGO_SNAPSHOT = "cargo-${SNAPSHOT_VERSION}-${RUST_BUILD_ARCH}-unknown-linux-gnu"
+""" % config_dist_server
+    # Write the updated snapshot information to the rust-snapshot.inc file
+    with open(os.path.join(d.getVar("THISDIR"), "rust-snapshot.inc"), "w") as f:
+        f.write(snapshot)
+}
meta/recipes-devtools/rust/rust_git.bb +0 -73

79 lines not on this page · raw message

Signature · 1 line
2.49.0