rust: Fix reproducibility with shared source trees
PATCH v2 1/3 rust: Fix reproducibility with shared source trees
From: Deepesh Varatharajan <[email protected]>
Backport an upstream Rust commit to fix a reproducibility issue when Rust family recipes share the same source tree.
--- ...ild-script-path-leakage-in-artifacts.patch | 28 +++++++++++++++++++ meta/recipes-devtools/rust/rust-source.inc | 1 + 2 files changed, 29 insertions(+) create mode 100644 meta/recipes-devtools/rust/files/0005-remap-OUT_DIR-paths-to-fix-build-script-path-leakage-in-artifacts.patch
meta/recipes-devtools/rust/files/0005-remap-OUT_DIR-paths-to-fix-build-script-path-leakage-in-artifacts.patch new file +28 -0
@@ -0,0 +1,28 @@ +Backport a commit that remap OUT_DIR paths to fix build script path leakage in artifacts + +Upstream-Status: Backport [https://github.com/rust-lang/rust/commit/e743a4db62336027dcc1f1a755f4ca35f56b67df] + +Signed-off-by: Deepesh Varatharajan <[email protected]> + +diff --git a/src/bootstrap/src/core/builder/cargo.rs b/src/bootstrap/src/core/builder/cargo.rs +index f785254..0264667 100644 +--- a/src/bootstrap/src/core/builder/cargo.rs ++++ b/src/bootstrap/src/core/builder/cargo.rs +@@ -1077,6 +1077,8 @@ impl Builder<'_> { + // rustc creates absolute paths (in part bc of the `rust-src` unremap + // and for working directory) so let's remap the build directory as well. + format!("{}={map_to}", self.build.src.display()), ++ // remap OUT_DIR so they don't leak into artifacts. ++ format!("{}={map_to}/out", self.build.out.display()), + ] + .join("\t"); + cargo.env("RUSTC_DEBUGINFO_MAP", map); +@@ -1097,6 +1099,8 @@ impl Builder<'_> { + // rustc creates absolute paths (in part bc of the `rust-src` unremap + // and for working directory) so let's remap the build directory as well. + format!("{}={map_to}", self.build.src.display()), ++ // remap OUT_DIR so they don't leak into artifacts. ++ format!("{}={map_to}/out", self.build.out.display()), + ] + .join("\t"); + cargo.env("RUSTC_DEBUGINFO_MAP", map);
meta/recipes-devtools/rust/rust-source.inc +1 -0
@@ -9,6 +9,7 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n file://0002-Fix-rust-build-failure-with-unstable-options.patch;patchdir=${RUSTSRC} \ file://0003-explicit-tail-calls-disable-two-tests.patch;patchdir=${RUSTSRC} \ file://0004-explicit-tail-calls-ignore-tests-on-unsupported-llvm.patch;patchdir=${RUSTSRC} \ + file://0005-remap-OUT_DIR-paths-to-fix-build-script-path-leakage-in-artifacts.patch;patchdir=${RUSTSRC} \ " SRC_URI[rust.sha256sum] = "77a6ff3003a4ad0cb00697b043c879e3e1a15d945b1a1f63818903bfc3fa8b98"
Signature · 1 line
2.49.0
PATCH v2 2/3 rust: refactor source handling into a shared work-shared recipe
From: Deepesh Varatharajan <[email protected]>
[YOCTO #15808]
Consolidate Rust source management across all Rust recipes (rust, cargo, libstd-rs and their native/nativesdk variants) into a single shared source tree using the work-shared pattern.
Disk savings: a world build with multilibs enabled, together with nativesdk-rust and nativesdk-cargo, now shares a single ~4.2 GB rustc-src tree instead of maintaining up to 11 independent copies (~46.2 GB total).
Performance (clean core-image-sato build, qemux86-64):
Before: 68m53.507s
After: 65m46.159s
Improvement: 3m07.348s (~4.5%)
New files:
- rust-source.bb: A shared recipe that fetches, unpacks, and patches the rustc source tree into ${TMPDIR}/work-shared/rust-source-${PV}-${PR}. All build/install/packaging tasks are disabled; this recipe exists solely to provide a single copy of the source. Inherits allarch/nopackages, excludes itself from rm_work, and is hidden from world builds. Also defines RUST_BUILD_ARCH locally since this recipe does not inherit rust-common.bbclass but still needs the mapping for snapshot fetching via rust-snapshot.inc.
- common-source.inc: Included by consumer recipes (rust, cargo, libstd-rs) to disable their local do_fetch/do_unpack/do_patch tasks, clear SRC_URI, and add task dependencies on rust-source-${PV} for configure, licensing, and SPDX generation. Also provides is_work_shared_spdx() override for correct SPDX shared-workdir detection.
Modified files:
- rust-source.inc:
- Replace RUST_VERSION with explicit MAJOR_VER/MINOR_VER/PATCH_VER and PV.
- Set UNPACKDIR to the work-shared location so all consumers share one source tree via RUSTSRC and S.
- rust_git.bb (renamed from rust_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- Set B = ${WORKDIR}/build to separate build artifacts from shared source.
- Set CARGO_HOME = ${WORKDIR}/cargo_home to isolate cargo caching.
- Set build-dir = "rust-build" in config.toml to distinguish Yocto's build directory (B), which holds config.toml, from the bootstrap's own build output directory. Without this, bootstrap defaults to "build/" inside B, creating an ambiguous build/build/ nesting.
- Use absolute ${S}/src/bootstrap/bootstrap.py path since cwd is now B.
- Update install functions to reference rust-build/ instead of build/.
- Update do_test_compile to use absolute ${S}/src/tools/ path.
- Change do_rust_setup_snapshot from 'after do_unpack' to depend on
rust-source-${PV}:do_unpack (since local do_unpack is removed).- cargo_git.bb (renamed from cargo_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- Change do_cargo_setup_snapshot from 'after do_unpack' to depend on
rust-source-${PV}:do_unpack.- libstd-rs_git.bb (renamed from libstd-rs_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- rust-cross-canadian_git.bb (renamed from rust-cross-canadian_1.96.1.bb):
- Rename only (version now comes from rust-source.inc via PV).
- rust-common.bbclass:
- Add a second --remap-path-prefix for ${TMPDIR}/work-shared so debug info from the shared source tree is correctly remapped.
- Retain RUST_BUILD_ARCH definition so that consumer recipes (rust, cargo, libstd-rs) which inherit this class can resolve snapshot directory names used by do_rust_setup_snapshot and do_cargo_setup_snapshot.
- lib/oeqa/selftest/cases/rust.py:
- Adapt the selftest to the build-dir/source-dir split: use B for build artifacts and RUSTSRC for source paths.
- Update remote-test-server copy path from build/ to rust-build/.
- Pass --build-dir and --config explicitly to bootstrap.py.
- conf/distro/include/maintainers.inc:
- Add maintainer entry for the new rust-source recipe.
License-Update: recipe file renames only, no license change
---
meta/classes-recipe/rust-common.bbclass | 4 +-
meta/conf/distro/include/maintainers.inc | 1 +
meta/lib/oeqa/selftest/cases/rust.py | 9 ++-
.../rust/{cargo_1.96.1.bb => cargo_git.bb} | 8 ++-
meta/recipes-devtools/rust/common-source.inc | 28 ++++++++++
.../{libstd-rs_1.96.1.bb => libstd-rs_git.bb} | 4 ++
...n_1.96.1.bb => rust-cross-canadian_git.bb} | 0
meta/recipes-devtools/rust/rust-source.bb | 56 +++++++++++++++++++
meta/recipes-devtools/rust/rust-source.inc | 16 +++++-
.../rust/{rust_1.96.1.bb => rust_git.bb} | 31 +++++++---
10 files changed, 140 insertions(+), 17 deletions(-)
rename meta/recipes-devtools/rust/{cargo_1.96.1.bb => cargo_git.bb} (91%)
create mode 100644 meta/recipes-devtools/rust/common-source.inc
rename meta/recipes-devtools/rust/{libstd-rs_1.96.1.bb => libstd-rs_git.bb} (92%)
rename meta/recipes-devtools/rust/{rust-cross-canadian_1.96.1.bb => rust-cross-canadian_git.bb} (100%)
create mode 100644 meta/recipes-devtools/rust/rust-source.bb
rename meta/recipes-devtools/rust/{rust_1.96.1.bb => rust_git.bb} (93%)meta/classes-recipe/rust-common.bbclass +3 -1
@@ -14,7 +14,9 @@ FILES:${PN}-dev += "${rustlibdir}/*.rlib ${rustlibdir}/*.rmeta" FILES:${PN}-dbg += "${rustlibdir}/.debug" RUSTLIB ?= "-L ${STAGING_DIR_HOST}${rustlibdir}" -RUST_DEBUG_REMAP = "--remap-path-prefix=${WORKDIR}=${TARGET_DBGSRC_DIR}" +RUST_DEBUG_REMAP = "--remap-path-prefix=${WORKDIR}=${TARGET_DBGSRC_DIR} \ + --remap-path-prefix=${TMPDIR}/work-shared=${TARGET_DBGSRC_DIR} \ +" RUSTFLAGS += "${RUSTLIB} ${RUST_DEBUG_REMAP}" RUSTLIB_DEP ??= "libstd-rs" RUST_PANIC_STRATEGY ??= "unwind"
meta/conf/distro/include/maintainers.inc +1 -0
@@ -784,6 +784,7 @@ RECIPE_MAINTAINER:pn-ruby = "Ross Burton <[email protected]>" RECIPE_MAINTAINER:pn-run-postinsts = "Ross Burton <[email protected]>" RECIPE_MAINTAINER:pn-rust = "Randy MacLeod <[email protected]>" RECIPE_MAINTAINER:pn-rust-cross-canadian-${TRANSLATED_TARGET_ARCH} = "Randy MacLeod <[email protected]>" +RECIPE_MAINTAINER:pn-rust-source-1.96.1 = "Deepesh Varatharajan <[email protected]>" RECIPE_MAINTAINER:pn-rxvt-unicode = "Unassigned <[email protected]>" RECIPE_MAINTAINER:pn-sassc = "Simone Weiß <[email protected]>" RECIPE_MAINTAINER:pn-sato-icon-theme = "Richard Purdie <[email protected]>"
meta/lib/oeqa/selftest/cases/rust.py +6 -3
@@ -46,7 +46,8 @@ class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase): recipe = "rust" start_time = time.time() bitbake("{} -c test_compile".format(recipe)) - builddir = get_bb_var("RUSTSRC", "rust") + builddir = get_bb_var("B", "rust") + sourcedir = get_bb_var("RUSTSRC", "rust") # build core-image-minimal with required packages default_installed_packages = ["libgcc", "libstdc++", "libatomic", "libgomp", "libzstd", "llvm", "openssl"] features = []
@@ -111,7 +112,7 @@ class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase): # Copy remote-test-server to image through scp host_sys = get_bb_var("RUST_BUILD_SYS", "rust") ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user="root") - ssh.copy_to(builddir + "/build/" + host_sys + "/stage2-tools-bin/remote-test-server","~/") + ssh.copy_to(builddir + "/rust-build/" + host_sys + "/stage2-tools-bin/remote-test-server","~/") # Execute remote-test-server on image through background ssh command = '~/remote-test-server --bind 0.0.0.0:12345 -v' sshrun=subprocess.Popen(("ssh", '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-f', "root@%s" % qemu.ip, command), shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
@@ -131,8 +132,10 @@ class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase): # PowerPC mac99 QEMU has 768MB RAM limit, so we need to minimize test binary sizes cmd = cmd + " export RUSTFLAGS='-C strip=debuginfo -Clink-arg=-lz -Clink-arg=-lzstd';" # Trigger testing. + # Run bootstrap from sourcedir with explicit --build-dir and --config + # pointing to the separate build directory. cmd = cmd + " export TEST_DEVICE_ADDR=\"%s:12345\";" % qemu.ip - cmd = cmd + " cd %s; python3 src/bootstrap/bootstrap.py test %s --target %s" % (builddir, testargs, targetsys) + cmd = cmd + " cd %s; python3 src/bootstrap/bootstrap.py --build-dir %s/rust-build --config %s/config.toml test %s --target %s" % (sourcedir, builddir, builddir, testargs, targetsys) retval = runCmd(cmd) end_time = time.time()
meta/recipes-devtools/rust/cargo_git.bb renamed from meta/recipes-devtools/rust/cargo_1.96.1.bb +7 -1
@@ -13,6 +13,8 @@ LIC_FILES_CHKSUM = " \ require rust-source.inc require rust-snapshot.inc +# Use shared source tree from rust-source.bb; disables local fetch/unpack/patch +require common-source.inc S = "${RUSTSRC}/src/tools/cargo" CARGO_VENDORING_DIRECTORY = "${RUSTSRC}/vendor"
@@ -21,6 +23,9 @@ CVE_PRODUCT = "rust-lang:cargo" inherit cargo pkgconfig +# Explicit CARGO_HOME avoids cargo caching in the shared source tree +CARGO_HOME = "${WORKDIR}/cargo_home" + DEBUG_PREFIX_MAP += "-ffile-prefix-map=${RUSTSRC}/vendor=${TARGET_DBGSRC_DIR}" do_cargo_setup_snapshot () {
@@ -32,9 +37,10 @@ do_cargo_setup_snapshot () { fi } -addtask cargo_setup_snapshot after do_unpack before do_configure +addtask cargo_setup_snapshot before do_configure do_cargo_setup_snapshot[dirs] += "${WORKDIR}/${CARGO_SNAPSHOT}" do_cargo_setup_snapshot[vardepsexclude] += "UNINATIVE_LOADER" +do_cargo_setup_snapshot[depends] += "rust-source-${PV}:do_unpack" do_compile:prepend () { export RUSTC_BOOTSTRAP="1"
meta/recipes-devtools/rust/common-source.inc new file +28 -0
@@ -0,0 +1,28 @@ +# common-source.inc - Disable source tasks for recipes using shared rust-source +# +# Recipes that include this file rely on rust-source-${PV} for fetching, +# unpacking, and patching the Rust source tree. This avoids duplicating the +# ~4.2 GB rustc-src tree per recipe variant (rust, cargo, libstd-rs and their +# native/nativesdk counterparts). + +# Disable local fetch/unpack/patch since rust-source-${PV} handles them +do_fetch() { + : +} +do_fetch[noexec] = "1" +deltask do_unpack +deltask do_patch + +# Clear SRC_URI to prevent fetcher warnings from stale URIs inherited via .inc +SRC_URI = "" + +# Ensure the shared source is fully ready before dependent tasks run +do_configure[depends] += "rust-source-${PV}:do_patch" +do_populate_lic[depends] += "rust-source-${PV}:do_unpack" +do_create_spdx[depends] += "rust-source-${PV}:do_patch" + +# The SPDX class uses is_work_shared_spdx() to detect shared workdirs. +# Since we redirect S rather than WORKDIR, the default detection fails. +# Override to always return True so SPDX processing works correctly. +def is_work_shared_spdx(d): + return True
meta/recipes-devtools/rust/libstd-rs_git.bb renamed from meta/recipes-devtools/rust/libstd-rs_1.96.1.bb +4 -0
@@ -5,6 +5,7 @@ LICENSE = "(MIT | Apache-2.0) & Unicode-3.0" LIC_FILES_CHKSUM = "file://../../COPYRIGHT;md5=11a3899825f4376896e438c8c753f8dc" require rust-source.inc +require common-source.inc # The dummy crate named `sysroot` represents the standard library target. #
@@ -30,6 +31,9 @@ CARGO_FEATURES ?= "panic-unwind backtrace" CARGO_BUILD_FLAGS += "--features '${CARGO_FEATURES}'" CARGO_VENDORING_DIRECTORY = "${RUSTSRC}/vendor" +# Explicit CARGO_HOME avoids cargo caching in the shared source tree +CARGO_HOME = "${WORKDIR}/cargo_home" + do_compile:prepend () { # For Rust 1.13.0 and newer export RUSTC_BOOTSTRAP="1"
meta/recipes-devtools/rust/rust-cross-canadian_git.bb renamed from meta/recipes-devtools/rust/rust-cross-canadian_1.96.1.bb +0 -0
meta/recipes-devtools/rust/rust-source.bb new file +56 -0
56 lines not on this page · raw message
meta/recipes-devtools/rust/rust-source.inc +13 -3
@@ -1,6 +1,12 @@ -RUST_VERSION ?= "${@d.getVar('PV').split('-')[0]}" +# Version is split into MAJOR_VER and MINOR_VER for easy bumping and +# to allow consistent use of PV across all consuming recipes. +MAJOR_VER = "1" +MINOR_VER = "96" +PATCH_VER = "1" -SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;name=rust \ +PV = "${MAJOR_VER}.${MINOR_VER}.${PATCH_VER}" + +SRC_URI += "https://static.rust-lang.org/dist/rustc-${PV}-src.tar.xz;name=rust \ file://0001-Define-more-ioctl-codes-on-riscv32gc-unknown-linux-g.patch;patchdir=${RUSTSRC} \ file://rust-oe-selftest.patch;patchdir=${RUSTSRC} \ file://repro-issue-fix-with-cc-crate-hashmap.patch;patchdir=${RUSTSRC} \
@@ -13,7 +19,11 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n " SRC_URI[rust.sha256sum] = "77a6ff3003a4ad0cb00697b043c879e3e1a15d945b1a1f63818903bfc3fa8b98" -RUSTSRC = "${UNPACKDIR}/rustc-${RUST_VERSION}-src" +# Point UNPACKDIR into the work-shared location so rust-source.bb and all +# consumers (via S/RUSTSRC) reference a single shared source tree. +UNPACKDIR = "${TMPDIR}/work-shared/rust-source-${PV}-${PR}/sources" +RUSTSRC = "${UNPACKDIR}/rustc-${PV}-src" +S = "${RUSTSRC}" UPSTREAM_CHECK_URI = "https://forge.rust-lang.org/infra/other-installation-methods.html" UPSTREAM_CHECK_REGEX = "rustc-(?P<pver>\d+(\.\d+)+)-src"
meta/recipes-devtools/rust/rust_git.bb renamed from meta/recipes-devtools/rust/rust_1.96.1.bb +22 -9
@@ -4,6 +4,10 @@ SECTION = "devel" LICENSE = "(MIT | Apache-2.0) & Unicode-3.0" LIC_FILES_CHKSUM = "file://COPYRIGHT;md5=11a3899825f4376896e438c8c753f8dc" +require rust-source.inc +require rust-snapshot.inc +require common-source.inc + inherit rust inherit cargo_common
@@ -24,6 +28,9 @@ INHIBIT_DEFAULT_RUST_DEPS:class-native = "1" PROVIDES:class-native = "virtual/${TARGET_PREFIX}rust" S = "${RUSTSRC}" +# Separate build directory from the shared source tree so that multiple +# variants (target, native, nativesdk) don't conflict in the source dir. +B = "${WORKDIR}/build" # Use at your own risk, accepted values are stable, beta and nightly RUST_CHANNEL ?= "stable"
@@ -38,6 +45,9 @@ RUST_ALTERNATE_EXE_PATH_NATIVE = "${STAGING_BINDIR_NATIVE}/llvm-config" # own vendoring. CARGO_DISABLE_BITBAKE_VENDORING = "1" +# Explicit CARGO_HOME avoids cargo caching in the shared source tree +CARGO_HOME = "${WORKDIR}/cargo_home" + setup_cargo_environment () { # The first step is to build bootstrap and some early stage tools, # these are build for the same target as the snapshot, e.g.
@@ -66,11 +76,11 @@ do_rust_setup_snapshot () { done fi } -addtask rust_setup_snapshot after do_unpack before do_configure +addtask rust_setup_snapshot before do_configure addtask do_test_compile after do_configure do_rust_gen_targets do_rust_setup_snapshot[dirs] += "${WORKDIR}/rust-snapshot" do_rust_setup_snapshot[vardepsexclude] += "UNINATIVE_LOADER" -do_rust_setup_snapshot[depends] += "patchelf-native:do_populate_sysroot" +do_rust_setup_snapshot[depends] += "patchelf-native:do_populate_sysroot rust-source-${PV}:do_unpack" RUSTC_BOOTSTRAP = "${STAGING_BINDIR_NATIVE}/rustc" CARGO_BOOTSTRAP = "${STAGING_BINDIR_NATIVE}/cargo"
@@ -173,6 +183,12 @@ python do_configure() { # nothing about when trying to build some stage0 tools (like fabricate) config.set("build", "build", e(d.getVar("RUST_BUILD_SYS"))) + # Yocto's B (${WORKDIR}/build) is where config.toml lives. Rust's + # bootstrap creates its own build output directory inside B; name it + # "rust-build" to avoid ambiguity with Yocto's build directory and + # prevent a confusing build/build/ nesting. + config.set("build", "build-dir", e("rust-build")) + # [install] config.add_section("install") # ./x.py install doesn't have any notion of "destdir"
@@ -223,13 +239,10 @@ rust_runx () { oe_cargo_fix_env - python3 src/bootstrap/bootstrap.py ${@oe.utils.parallel_make_argument(d, '-j %d')} "$@" --verbose + python3 ${S}/src/bootstrap/bootstrap.py ${@oe.utils.parallel_make_argument(d, '-j %d')} "$@" --verbose } rust_runx[vardepsexclude] += "PARALLEL_MAKE" -require rust-source.inc -require rust-snapshot.inc - INSANE_SKIP:${PN}:class-native = "already-stripped" FILES:${PN} += "${libdir}/rustlib" FILES:${PN} += "${libdir}/*.so"
@@ -241,7 +254,7 @@ do_compile () { do_test_compile[dirs] = "${B}" do_test_compile () { replace_llvm_config_path - rust_runx build src/tools/remote-test-server --target "${RUST_TARGET_SYS}" + rust_runx build ${S}/src/tools/remote-test-server --target "${RUST_TARGET_SYS}" } ALLOW_EMPTY:${PN} = "1"
@@ -282,7 +295,7 @@ rust_do_install:class-nativesdk() { install -d ${D}${bindir} for i in cargo-clippy clippy-driver rustfmt; do - cp build/${RUST_BUILD_SYS}/stage2-tools/${RUST_HOST_SYS}/release/$i ${D}${bindir} + cp rust-build/${RUST_BUILD_SYS}/stage2-tools/${RUST_HOST_SYS}/release/$i ${D}${bindir} patchelf --set-rpath "\$ORIGIN/../lib" ${D}${bindir}/$i done
@@ -322,7 +335,7 @@ rust_do_install:class-target() { install -d ${D}${bindir} for i in ${EXTRA_TOOLS}; do - cp build/${RUST_BUILD_SYS}/stage2-tools/${RUST_HOST_SYS}/release/$i ${D}${bindir} + cp rust-build/${RUST_BUILD_SYS}/stage2-tools/${RUST_HOST_SYS}/release/$i ${D}${bindir} patchelf --set-rpath "\$ORIGIN/../lib" ${D}${bindir}/$i done
Signature · 1 line
2.49.0
Re: [OE-core] [PATCH v2 2/3] rust: refactor source handling into a shared work-shared recipe
Quoted 4 lines · Deepesh via lists.openembedded.org Varatharajan
From: Deepesh Varatharajan <[email protected]>
[YOCTO #15808]
Quoted 8 lines
Consolidate Rust source management across all Rust recipes (rust, cargo, libstd-rs and their native/nativesdk variants) into a single shared source tree using the work-shared pattern.
Disk savings: a world build with multilibs enabled, together with nativesdk-rust and nativesdk-cargo, now shares a single ~4.2 GB rustc-src tree instead of maintaining up to 11 independent copies (~46.2 GB total).
Hi Deepesh,
Quoted 8 lines
meta/recipes-devtools/rust/rust-source.bb new file +5
-0@@ -0,0 +1,56 @@ +# rust-source.bb - Shared source recipe for the Rust toolchain +# +# This recipe fetches, unpacks, and patches the rustc source tree into a +# work-shared location so that rust, cargo, libstd-rs, and their native/nativesdk +# variants all share a single copy of the source. This eliminates duplication
meta/recipes-devtools/rust/rust-source.bb line 15
Quoted 10 lines
meta/recipes-devtools/rust/rust-source.bb new file +10
-0+# (up to 11 copies of the ~4.2 GB source tree in multilib + nativesdk builds) +# and significantly reduces disk usage (~46.2 GB -> ~4.2 GB for rustc-src). +# +# Only do_fetch, do_unpack, and do_patch are meaningful here. All other +# tasks are disabled since this recipe exists solely to provide sources. + +SUMMARY = "Rust compiler and runtime libaries" +HOMEPAGE = "http://www.rust-lang.org" +SECTION = "devel" +LICENSE = "(MIT | Apache-2.0) & Unicode-3.0"
Please use the new LICENSE syntax here...
meta/recipes-devtools/rust/rust_git.bb SECTION = "devel" · line 4
Quoted 4 lines
meta/recipes-devtools/rust/rust_git.bb renamed from meta/recipes-devtools/rust/rust_1.96.1.bb +0
-0@@ -4,6 +4,10 @@ SECTION = "devel" LICENSE = "(MIT | Apache-2.0) & Unicode-3.0"
... and here.
Parsing recipes...WARNING: /srv/pokybuild/yocto-worker/beaglebone/build/layers/openembedded-core/meta/recipes-devtools/rust/rust-source.bb: rust-source-1.96.1: LICENSE is using an old syntax and should be upgraded to: "Unicode-3.0 AND (Apache-2.0 OR MIT)"
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Re: [OE-core] [PATCH v2 2/3] rust: refactor source handling into a shared work-shared recipe
Quoted 45 lines · Mathieu Dubois-Briand
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.Quoted 4 lines · Deepesh via lists.openembedded.org Varatharajan
From: Deepesh Varatharajan <[email protected]>
[YOCTO #15808]
Quoted 8 lines
Consolidate Rust source management across all Rust recipes (rust, cargo, libstd-rs and their native/nativesdk variants) into a single shared source tree using the work-shared pattern.
Disk savings: a world build with multilibs enabled, together with nativesdk-rust and nativesdk-cargo, now shares a single ~4.2 GB rustc-src tree instead of maintaining up to 11 independent copies (~46.2 GB total).
Hi Deepesh,
Quoted 8 lines
meta/recipes-devtools/rust/rust-source.bb new file +5
-0@@ -0,0 +1,56 @@ +# rust-source.bb - Shared source recipe for the Rust toolchain +# +# This recipe fetches, unpacks, and patches the rustc source tree into a +# work-shared location so that rust, cargo, libstd-rs, and their native/nativesdk +# variants all share a single copy of the source. This eliminates duplicationmeta/recipes-devtools/rust/rust-source.bb line 15
Quoted 10 lines
meta/recipes-devtools/rust/rust-source.bb new file +10
-0+# (up to 11 copies of the ~4.2 GB source tree in multilib + nativesdk builds) +# and significantly reduces disk usage (~46.2 GB -> ~4.2 GB for rustc-src). +# +# Only do_fetch, do_unpack, and do_patch are meaningful here. All other +# tasks are disabled since this recipe exists solely to provide sources. + +SUMMARY = "Rust compiler and runtime libaries" +HOMEPAGE = "http://www.rust-lang.org" +SECTION = "devel" +LICENSE = "(MIT | Apache-2.0) & Unicode-3.0"Please use the new LICENSE syntax here...
meta/recipes-devtools/rust/rust_git.bb SECTION = "devel" · line 4
Quoted 4 lines
meta/recipes-devtools/rust/rust_git.bb renamed from meta/recipes-devtools/rust/rust_1.96.1.bb +0
-0@@ -4,6 +4,10 @@ SECTION = "devel" LICENSE = "(MIT | Apache-2.0) & Unicode-3.0"... and here.
Quoted 4 lines
Parsing recipes...WARNING:
/srv/pokybuild/yocto-worker/beaglebone/build/layers/openembedded-core/meta/recipes-devtools/rust/rust-source.bb: rust-source-1.96.1: LICENSE is using an old syntax and should be upgraded to: "Unicode-3.0 AND (Apache-2.0 OR MIT)"Thanks,
Mathieu
Hi Mathieu,
Fixed in v3.
Regards,
Deepesh
Quoted 6 lines
--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
PATCH v2 3/3 rust: Move do_update_snapshot to rust-source
In reply to [email protected]From: Deepesh Varatharajan <[email protected]>
Move do_update_snapshot from rust to rust-source, since rust-source now owns do_fetch/do_unpack/do_patch tasks after the shared source refactoring.
--- meta/recipes-devtools/rust/rust-source.bb | 72 ++++++++++++++++++++++ meta/recipes-devtools/rust/rust_git.bb | 73 ----------------------- 2 files changed, 72 insertions(+), 73 deletions(-)
meta/recipes-devtools/rust/rust-source.bb +72 -0
@@ -54,3 +54,75 @@ B = "${WORKDIR}/build" RUST_BUILD_ARCH = "${@{'ppc': 'powerpc', 'ppc64': 'powerpc64', 'ppc64le': 'powerpc64le', 'riscv64': 'riscv64gc'}.get(d.getVar('BUILD_ARCH'), d.getVar('BUILD_ARCH'))}" EXCLUDE_FROM_WORLD = "1" + +addtask do_update_snapshot after do_patch +do_update_snapshot[nostamp] = "1" + +# Run with `bitbake rust-source-${PV} -c update_snapshot` to update `rust-snapshot.inc` +# with the checksums for the rust snapshot associated with this rustc-src tarball. +python do_update_snapshot() { + import json + import re + import sys + + from collections import defaultdict + + key_value_pairs = {} + with open(os.path.join(d.getVar("S"), "src", "stage0")) as f: + for line in f: + # Skip empty lines or comments + if not line.strip() or line.startswith("#"): + continue + # Split the line into key and value using '=' as separator + match = re.match(r'(\S+)\s*=\s*(\S+)', line.strip()) + if match: + key = match.group(1) + value = match.group(2) + key_value_pairs[key] = value + # Extract the required values from key_value_pairs + config_dist_server = key_value_pairs.get('dist_server', '') + compiler_date = key_value_pairs.get('compiler_date', '') + compiler_version = key_value_pairs.get('compiler_version', '') + + src_uri = defaultdict(list) + # Assuming checksums_sha256 is now a key-value pair like: checksum_key = checksum_value + for k, v in key_value_pairs.items(): + # Match the pattern for checksums + if "dist" in k and "tar.xz" in k: + m = re.search(f"dist/{compiler_date}/(?P<component>.*)-{compiler_version}-(?P<arch>.*)-unknown-linux-gnu\\.tar\\.xz", k) + if m: + component = m.group('component') + arch = m.group('arch') + src_uri[arch].append(f"SRC_URI[{component}-snapshot-{arch}.sha256sum] = \"{v}\"") + # Create the snapshot string with the extracted values + snapshot = """\ +## This is information on the rust-snapshot (binary) used to build our current release. +## snapshot info is taken from rust/src/stage0 +## Rust is self-hosting and bootstraps itself with a pre-built previous version of itself. +## The exact (previous) version that has been used is specified in the source tarball. +## The version is replicated here. + +SNAPSHOT_VERSION = "%s" + +""" % compiler_version + # Add the checksum components to the snapshot + for arch, components in src_uri.items(): + snapshot += "\n".join(components) + "\n\n" + # Add the additional snapshot URIs + snapshot += """\ +SRC_URI += " \\ + ${RUST_DIST_SERVER}/dist/${RUST_STD_SNAPSHOT}.tar.xz;name=rust-std-snapshot-${RUST_BUILD_ARCH};subdir=rust-snapshot-components \\ + ${RUST_DIST_SERVER}/dist/${RUSTC_SNAPSHOT}.tar.xz;name=rustc-snapshot-${RUST_BUILD_ARCH};subdir=rust-snapshot-components \\ + ${RUST_DIST_SERVER}/dist/${CARGO_SNAPSHOT}.tar.xz;name=cargo-snapshot-${RUST_BUILD_ARCH};subdir=rust-snapshot-components \\ +" + +RUST_DIST_SERVER = "%s" + +RUST_STD_SNAPSHOT = "rust-std-${SNAPSHOT_VERSION}-${RUST_BUILD_ARCH}-unknown-linux-gnu" +RUSTC_SNAPSHOT = "rustc-${SNAPSHOT_VERSION}-${RUST_BUILD_ARCH}-unknown-linux-gnu" +CARGO_SNAPSHOT = "cargo-${SNAPSHOT_VERSION}-${RUST_BUILD_ARCH}-unknown-linux-gnu" +""" % config_dist_server + # Write the updated snapshot information to the rust-snapshot.inc file + with open(os.path.join(d.getVar("THISDIR"), "rust-snapshot.inc"), "w") as f: + f.write(snapshot) +}
meta/recipes-devtools/rust/rust_git.bb +0 -73
79 lines not on this page · raw message
Signature · 1 line
2.49.0
Quoted 7 lines · Deepesh via lists.openembedded.org Varatharajan
From: Deepesh Varatharajan <[email protected]>
Backport an upstream Rust commit to fix a reproducibility issue when Rust family recipes share the same source tree.
Signed-off-by Deepesh Varatharajan <[email protected]>---
Hi Deepesh,
Thanks for your patch.
Do you expect this to fix the reproducibility issue after the rust upgrade to 1.96.1, as sent by Sundeep [1]? Or is this for another issue?
[1]: https://lore.kernel.org/openembedded-core/[email protected]/
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Hi Mathieu,
This is different issue. We are still trying to reproduce the intermittent reproducible issue seen with rust 1.97 upgrade.
Regards,
Deepesh
Sent from Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Mathieu Dubois-Briand <[email protected]>
Sent: Wednesday, 22 July 2026 19:03:10
To: Varatharajan, Deepesh <[email protected]>; [email protected] <[email protected]>
Cc: Varatharajan, Deepesh <[email protected]>
Subject: Re: [OE-core] [PATCH v2 1/3] rust: Fix reproducibility with shared source trees
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.
Quoted 7 lines · Deepesh via lists.openembedded.org Varatharajan
From: Deepesh Varatharajan <[email protected]>
Backport an upstream Rust commit to fix a reproducibility issue when Rust family recipes share the same source tree.
Signed-off-by Deepesh Varatharajan <[email protected]>---
Hi Deepesh,
Thanks for your patch.
Do you expect this to fix the reproducibility issue after the rust upgrade to 1.96.1, as sent by Sundeep [1]? Or is this for another issue?
[1]: https://lore.kernel.org/openembedded-core/[email protected]/
--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com