rust: Fix reproducibility with shared source trees
PATCH v3 1/3 rust: Fix reproducibility with shared source trees
From: Deepesh Varatharajan <[email protected]>
Backport an upstream Rust commit to fix a reproducibility issue when Rust family recipes share the same source tree.
--- ...ild-script-path-leakage-in-artifacts.patch | 28 +++++++++++++++++++ meta/recipes-devtools/rust/rust-source.inc | 1 + 2 files changed, 29 insertions(+) create mode 100644 meta/recipes-devtools/rust/files/0005-remap-OUT_DIR-paths-to-fix-build-script-path-leakage-in-artifacts.patch
meta/recipes-devtools/rust/files/0005-remap-OUT_DIR-paths-to-fix-build-script-path-leakage-in-artifacts.patch new file +28 -0
@@ -0,0 +1,28 @@ +Backport a commit that remap OUT_DIR paths to fix build script path leakage in artifacts + +Upstream-Status: Backport [https://github.com/rust-lang/rust/commit/e743a4db62336027dcc1f1a755f4ca35f56b67df] + +Signed-off-by: Deepesh Varatharajan <[email protected]> + +diff --git a/src/bootstrap/src/core/builder/cargo.rs b/src/bootstrap/src/core/builder/cargo.rs +index f785254..0264667 100644 +--- a/src/bootstrap/src/core/builder/cargo.rs ++++ b/src/bootstrap/src/core/builder/cargo.rs +@@ -1077,6 +1077,8 @@ impl Builder<'_> { + // rustc creates absolute paths (in part bc of the `rust-src` unremap + // and for working directory) so let's remap the build directory as well. + format!("{}={map_to}", self.build.src.display()), ++ // remap OUT_DIR so they don't leak into artifacts. ++ format!("{}={map_to}/out", self.build.out.display()), + ] + .join("\t"); + cargo.env("RUSTC_DEBUGINFO_MAP", map); +@@ -1097,6 +1099,8 @@ impl Builder<'_> { + // rustc creates absolute paths (in part bc of the `rust-src` unremap + // and for working directory) so let's remap the build directory as well. + format!("{}={map_to}", self.build.src.display()), ++ // remap OUT_DIR so they don't leak into artifacts. ++ format!("{}={map_to}/out", self.build.out.display()), + ] + .join("\t"); + cargo.env("RUSTC_DEBUGINFO_MAP", map);
meta/recipes-devtools/rust/rust-source.inc +1 -0
@@ -9,6 +9,7 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n file://0002-Fix-rust-build-failure-with-unstable-options.patch;patchdir=${RUSTSRC} \ file://0003-explicit-tail-calls-disable-two-tests.patch;patchdir=${RUSTSRC} \ file://0004-explicit-tail-calls-ignore-tests-on-unsupported-llvm.patch;patchdir=${RUSTSRC} \ + file://0005-remap-OUT_DIR-paths-to-fix-build-script-path-leakage-in-artifacts.patch;patchdir=${RUSTSRC} \ " SRC_URI[rust.sha256sum] = "77a6ff3003a4ad0cb00697b043c879e3e1a15d945b1a1f63818903bfc3fa8b98"
Signature · 1 line
2.49.0
PATCH v3 2/3 rust: refactor source handling into a shared work-shared recipe
From: Deepesh Varatharajan <[email protected]>
[YOCTO #15808]
Consolidate Rust source management across all Rust recipes (rust, cargo, libstd-rs and their native/nativesdk variants) into a single shared source tree using the work-shared pattern.
Disk savings: a world build with multilibs enabled, together with nativesdk-rust and nativesdk-cargo, now shares a single ~4.2 GB rustc-src tree instead of maintaining up to 11 independent copies (~46.2 GB total).
Performance (clean core-image-sato build, qemux86-64):
Before: 68m53.507s
After: 65m46.159s
Improvement: 3m07.348s (~4.5%)
New files:
- rust-source.bb: A shared recipe that fetches, unpacks, and patches the rustc source tree into ${TMPDIR}/work-shared/rust-source-${PV}-${PR}. All build/install/packaging tasks are disabled; this recipe exists solely to provide a single copy of the source. Inherits allarch/nopackages, excludes itself from rm_work, and is hidden from world builds. Also defines RUST_BUILD_ARCH locally since this recipe does not inherit rust-common.bbclass but still needs the mapping for snapshot fetching via rust-snapshot.inc.
- common-source.inc: Included by consumer recipes (rust, cargo, libstd-rs) to disable their local do_fetch/do_unpack/do_patch tasks, clear SRC_URI, and add task dependencies on rust-source-${PV} for configure, licensing, and SPDX generation. Also provides is_work_shared_spdx() override for correct SPDX shared-workdir detection.
Modified files:
- rust-source.inc:
- Replace RUST_VERSION with explicit MAJOR_VER/MINOR_VER/PATCH_VER and PV.
- Set UNPACKDIR to the work-shared location so all consumers share one source tree via RUSTSRC and S.
- rust_git.bb (renamed from rust_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- Set B = ${WORKDIR}/build to separate build artifacts from shared source.
- Set CARGO_HOME = ${WORKDIR}/cargo_home to isolate cargo caching.
- Set build-dir = "rust-build" in config.toml to distinguish Yocto's build directory (B), which holds config.toml, from the bootstrap's own build output directory. Without this, bootstrap defaults to "build/" inside B, creating an ambiguous build/build/ nesting.
- Use absolute ${S}/src/bootstrap/bootstrap.py path since cwd is now B.
- Update install functions to reference rust-build/ instead of build/.
- Update do_test_compile to use absolute ${S}/src/tools/ path.
- Change do_rust_setup_snapshot from 'after do_unpack' to depend on
rust-source-${PV}:do_unpack (since local do_unpack is removed).- cargo_git.bb (renamed from cargo_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- Change do_cargo_setup_snapshot from 'after do_unpack' to depend on
rust-source-${PV}:do_unpack.- libstd-rs_git.bb (renamed from libstd-rs_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- rust-cross-canadian_git.bb (renamed from rust-cross-canadian_1.96.1.bb):
- Rename only (version now comes from rust-source.inc via PV).
- rust-common.bbclass:
- Add a second --remap-path-prefix for ${TMPDIR}/work-shared so debug info from the shared source tree is correctly remapped.
- Retain RUST_BUILD_ARCH definition so that consumer recipes (rust, cargo, libstd-rs) which inherit this class can resolve snapshot directory names used by do_rust_setup_snapshot and do_cargo_setup_snapshot.
- lib/oeqa/selftest/cases/rust.py:
- Adapt the selftest to the build-dir/source-dir split: use B for build artifacts and RUSTSRC for source paths.
- Update remote-test-server copy path from build/ to rust-build/.
- Pass --build-dir and --config explicitly to bootstrap.py.
- conf/distro/include/maintainers.inc:
- Add maintainer entry for the new rust-source recipe.
License-Update: recipe file renames only, no license change
---
meta/classes-recipe/rust-common.bbclass | 4 +-
meta/conf/distro/include/maintainers.inc | 1 +
meta/lib/oeqa/selftest/cases/rust.py | 9 ++-
.../rust/{cargo_1.96.1.bb => cargo_git.bb} | 8 ++-
meta/recipes-devtools/rust/common-source.inc | 28 ++++++++++
.../{libstd-rs_1.96.1.bb => libstd-rs_git.bb} | 4 ++
...n_1.96.1.bb => rust-cross-canadian_git.bb} | 0
meta/recipes-devtools/rust/rust-source.bb | 56 +++++++++++++++++++
meta/recipes-devtools/rust/rust-source.inc | 16 +++++-
.../rust/{rust_1.96.1.bb => rust_git.bb} | 31 +++++++---
10 files changed, 140 insertions(+), 17 deletions(-)
rename meta/recipes-devtools/rust/{cargo_1.96.1.bb => cargo_git.bb} (91%)
create mode 100644 meta/recipes-devtools/rust/common-source.inc
rename meta/recipes-devtools/rust/{libstd-rs_1.96.1.bb => libstd-rs_git.bb} (92%)
rename meta/recipes-devtools/rust/{rust-cross-canadian_1.96.1.bb => rust-cross-canadian_git.bb} (100%)
create mode 100644 meta/recipes-devtools/rust/rust-source.bb
rename meta/recipes-devtools/rust/{rust_1.96.1.bb => rust_git.bb} (93%)meta/classes-recipe/rust-common.bbclass +3 -1
@@ -14,7 +14,9 @@ FILES:${PN}-dev += "${rustlibdir}/*.rlib ${rustlibdir}/*.rmeta" FILES:${PN}-dbg += "${rustlibdir}/.debug" RUSTLIB ?= "-L ${STAGING_DIR_HOST}${rustlibdir}" -RUST_DEBUG_REMAP = "--remap-path-prefix=${WORKDIR}=${TARGET_DBGSRC_DIR}" +RUST_DEBUG_REMAP = "--remap-path-prefix=${WORKDIR}=${TARGET_DBGSRC_DIR} \ + --remap-path-prefix=${TMPDIR}/work-shared=${TARGET_DBGSRC_DIR} \ +" RUSTFLAGS += "${RUSTLIB} ${RUST_DEBUG_REMAP}" RUSTLIB_DEP ??= "libstd-rs" RUST_PANIC_STRATEGY ??= "unwind"
meta/conf/distro/include/maintainers.inc +1 -0
@@ -783,6 +783,7 @@ RECIPE_MAINTAINER:pn-ruby = "Ross Burton <[email protected]>" RECIPE_MAINTAINER:pn-run-postinsts = "Ross Burton <[email protected]>" RECIPE_MAINTAINER:pn-rust = "Randy MacLeod <[email protected]>" RECIPE_MAINTAINER:pn-rust-cross-canadian-${TRANSLATED_TARGET_ARCH} = "Randy MacLeod <[email protected]>" +RECIPE_MAINTAINER:pn-rust-source-1.96.1 = "Deepesh Varatharajan <[email protected]>" RECIPE_MAINTAINER:pn-rxvt-unicode = "Unassigned <[email protected]>" RECIPE_MAINTAINER:pn-sassc = "Simone Weiß <[email protected]>" RECIPE_MAINTAINER:pn-sato-icon-theme = "Richard Purdie <[email protected]>"
meta/lib/oeqa/selftest/cases/rust.py +6 -3
@@ -46,7 +46,8 @@ class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase): recipe = "rust" start_time = time.time() bitbake("{} -c test_compile".format(recipe)) - builddir = get_bb_var("RUSTSRC", "rust") + builddir = get_bb_var("B", "rust") + sourcedir = get_bb_var("RUSTSRC", "rust") # build core-image-minimal with required packages default_installed_packages = ["libgcc", "libstdc++", "libatomic", "libgomp", "libzstd", "llvm", "openssl"] features = []
@@ -111,7 +112,7 @@ class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase): # Copy remote-test-server to image through scp host_sys = get_bb_var("RUST_BUILD_SYS", "rust") ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user="root") - ssh.copy_to(builddir + "/build/" + host_sys + "/stage2-tools-bin/remote-test-server","~/") + ssh.copy_to(builddir + "/rust-build/" + host_sys + "/stage2-tools-bin/remote-test-server","~/") # Execute remote-test-server on image through background ssh command = '~/remote-test-server --bind 0.0.0.0:12345 -v' sshrun=subprocess.Popen(("ssh", '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-f', "root@%s" % qemu.ip, command), shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
@@ -131,8 +132,10 @@ class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase): # PowerPC mac99 QEMU has 768MB RAM limit, so we need to minimize test binary sizes cmd = cmd + " export RUSTFLAGS='-C strip=debuginfo -Clink-arg=-lz -Clink-arg=-lzstd';" # Trigger testing. + # Run bootstrap from sourcedir with explicit --build-dir and --config + # pointing to the separate build directory. cmd = cmd + " export TEST_DEVICE_ADDR=\"%s:12345\";" % qemu.ip - cmd = cmd + " cd %s; python3 src/bootstrap/bootstrap.py test %s --target %s" % (builddir, testargs, targetsys) + cmd = cmd + " cd %s; python3 src/bootstrap/bootstrap.py --build-dir %s/rust-build --config %s/config.toml test %s --target %s" % (sourcedir, builddir, builddir, testargs, targetsys) retval = runCmd(cmd) end_time = time.time()
meta/recipes-devtools/rust/cargo_git.bb renamed from meta/recipes-devtools/rust/cargo_1.96.1.bb +7 -1
@@ -13,6 +13,8 @@ LIC_FILES_CHKSUM = " \ require rust-source.inc require rust-snapshot.inc +# Use shared source tree from rust-source.bb; disables local fetch/unpack/patch +require common-source.inc S = "${RUSTSRC}/src/tools/cargo" CARGO_VENDORING_DIRECTORY = "${RUSTSRC}/vendor"
@@ -21,6 +23,9 @@ CVE_PRODUCT = "rust-lang:cargo" inherit cargo pkgconfig +# Explicit CARGO_HOME avoids cargo caching in the shared source tree +CARGO_HOME = "${WORKDIR}/cargo_home" + DEBUG_PREFIX_MAP += "-ffile-prefix-map=${RUSTSRC}/vendor=${TARGET_DBGSRC_DIR}" do_cargo_setup_snapshot () {
@@ -32,9 +37,10 @@ do_cargo_setup_snapshot () { fi } -addtask cargo_setup_snapshot after do_unpack before do_configure +addtask cargo_setup_snapshot before do_configure do_cargo_setup_snapshot[dirs] += "${WORKDIR}/${CARGO_SNAPSHOT}" do_cargo_setup_snapshot[vardepsexclude] += "UNINATIVE_LOADER" +do_cargo_setup_snapshot[depends] += "rust-source-${PV}:do_unpack" do_compile:prepend () { export RUSTC_BOOTSTRAP="1"
meta/recipes-devtools/rust/common-source.inc new file +28 -0
@@ -0,0 +1,28 @@ +# common-source.inc - Disable source tasks for recipes using shared rust-source +# +# Recipes that include this file rely on rust-source-${PV} for fetching, +# unpacking, and patching the Rust source tree. This avoids duplicating the +# ~4.2 GB rustc-src tree per recipe variant (rust, cargo, libstd-rs and their +# native/nativesdk counterparts). + +# Disable local fetch/unpack/patch since rust-source-${PV} handles them +do_fetch() { + : +} +do_fetch[noexec] = "1" +deltask do_unpack +deltask do_patch + +# Clear SRC_URI to prevent fetcher warnings from stale URIs inherited via .inc +SRC_URI = "" + +# Ensure the shared source is fully ready before dependent tasks run +do_configure[depends] += "rust-source-${PV}:do_patch" +do_populate_lic[depends] += "rust-source-${PV}:do_unpack" +do_create_spdx[depends] += "rust-source-${PV}:do_patch" + +# The SPDX class uses is_work_shared_spdx() to detect shared workdirs. +# Since we redirect S rather than WORKDIR, the default detection fails. +# Override to always return True so SPDX processing works correctly. +def is_work_shared_spdx(d): + return True
meta/recipes-devtools/rust/libstd-rs_git.bb renamed from meta/recipes-devtools/rust/libstd-rs_1.96.1.bb +4 -0
@@ -5,6 +5,7 @@ LICENSE = "Unicode-3.0 AND (Apache-2.0 OR MIT)" LIC_FILES_CHKSUM = "file://../../COPYRIGHT;md5=11a3899825f4376896e438c8c753f8dc" require rust-source.inc +require common-source.inc # The dummy crate named `sysroot` represents the standard library target. #
@@ -30,6 +31,9 @@ CARGO_FEATURES ?= "panic-unwind backtrace" CARGO_BUILD_FLAGS += "--features '${CARGO_FEATURES}'" CARGO_VENDORING_DIRECTORY = "${RUSTSRC}/vendor" +# Explicit CARGO_HOME avoids cargo caching in the shared source tree +CARGO_HOME = "${WORKDIR}/cargo_home" + do_compile:prepend () { # For Rust 1.13.0 and newer export RUSTC_BOOTSTRAP="1"
meta/recipes-devtools/rust/rust-cross-canadian_git.bb renamed from meta/recipes-devtools/rust/rust-cross-canadian_1.96.1.bb +0 -0
meta/recipes-devtools/rust/rust-source.bb new file +56 -0
56 lines not on this page · raw message
meta/recipes-devtools/rust/rust-source.inc +13 -3
@@ -1,6 +1,12 @@ -RUST_VERSION ?= "${@d.getVar('PV').split('-')[0]}" +# Version is split into MAJOR_VER and MINOR_VER for easy bumping and +# to allow consistent use of PV across all consuming recipes. +MAJOR_VER = "1" +MINOR_VER = "96" +PATCH_VER = "1" -SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;name=rust \ +PV = "${MAJOR_VER}.${MINOR_VER}.${PATCH_VER}" + +SRC_URI += "https://static.rust-lang.org/dist/rustc-${PV}-src.tar.xz;name=rust \ file://0001-Define-more-ioctl-codes-on-riscv32gc-unknown-linux-g.patch;patchdir=${RUSTSRC} \ file://rust-oe-selftest.patch;patchdir=${RUSTSRC} \ file://repro-issue-fix-with-cc-crate-hashmap.patch;patchdir=${RUSTSRC} \
@@ -13,7 +19,11 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n " SRC_URI[rust.sha256sum] = "77a6ff3003a4ad0cb00697b043c879e3e1a15d945b1a1f63818903bfc3fa8b98" -RUSTSRC = "${UNPACKDIR}/rustc-${RUST_VERSION}-src" +# Point UNPACKDIR into the work-shared location so rust-source.bb and all +# consumers (via S/RUSTSRC) reference a single shared source tree. +UNPACKDIR = "${TMPDIR}/work-shared/rust-source-${PV}-${PR}/sources" +RUSTSRC = "${UNPACKDIR}/rustc-${PV}-src" +S = "${RUSTSRC}" UPSTREAM_CHECK_URI = "https://forge.rust-lang.org/infra/other-installation-methods.html" UPSTREAM_CHECK_REGEX = "rustc-(?P<pver>\d+(\.\d+)+)-src"
meta/recipes-devtools/rust/rust_git.bb renamed from meta/recipes-devtools/rust/rust_1.96.1.bb +22 -9
@@ -4,6 +4,10 @@ SECTION = "devel" LICENSE = "Unicode-3.0 AND (Apache-2.0 OR MIT)" LIC_FILES_CHKSUM = "file://COPYRIGHT;md5=11a3899825f4376896e438c8c753f8dc" +require rust-source.inc +require rust-snapshot.inc +require common-source.inc + inherit rust inherit cargo_common
@@ -24,6 +28,9 @@ INHIBIT_DEFAULT_RUST_DEPS:class-native = "1" PROVIDES:class-native = "virtual/${TARGET_PREFIX}rust" S = "${RUSTSRC}" +# Separate build directory from the shared source tree so that multiple +# variants (target, native, nativesdk) don't conflict in the source dir. +B = "${WORKDIR}/build" # Use at your own risk, accepted values are stable, beta and nightly RUST_CHANNEL ?= "stable"
@@ -38,6 +45,9 @@ RUST_ALTERNATE_EXE_PATH_NATIVE = "${STAGING_BINDIR_NATIVE}/llvm-config" # own vendoring. CARGO_DISABLE_BITBAKE_VENDORING = "1" +# Explicit CARGO_HOME avoids cargo caching in the shared source tree +CARGO_HOME = "${WORKDIR}/cargo_home" + setup_cargo_environment () { # The first step is to build bootstrap and some early stage tools, # these are build for the same target as the snapshot, e.g.
@@ -66,11 +76,11 @@ do_rust_setup_snapshot () { done fi } -addtask rust_setup_snapshot after do_unpack before do_configure +addtask rust_setup_snapshot before do_configure addtask do_test_compile after do_configure do_rust_gen_targets do_rust_setup_snapshot[dirs] += "${WORKDIR}/rust-snapshot" do_rust_setup_snapshot[vardepsexclude] += "UNINATIVE_LOADER" -do_rust_setup_snapshot[depends] += "patchelf-native:do_populate_sysroot" +do_rust_setup_snapshot[depends] += "patchelf-native:do_populate_sysroot rust-source-${PV}:do_unpack" RUSTC_BOOTSTRAP = "${STAGING_BINDIR_NATIVE}/rustc" CARGO_BOOTSTRAP = "${STAGING_BINDIR_NATIVE}/cargo"
@@ -173,6 +183,12 @@ python do_configure() { # nothing about when trying to build some stage0 tools (like fabricate) config.set("build", "build", e(d.getVar("RUST_BUILD_SYS"))) + # Yocto's B (${WORKDIR}/build) is where config.toml lives. Rust's + # bootstrap creates its own build output directory inside B; name it + # "rust-build" to avoid ambiguity with Yocto's build directory and + # prevent a confusing build/build/ nesting. + config.set("build", "build-dir", e("rust-build")) + # [install] config.add_section("install") # ./x.py install doesn't have any notion of "destdir"
@@ -223,13 +239,10 @@ rust_runx () { oe_cargo_fix_env - python3 src/bootstrap/bootstrap.py ${@oe.utils.parallel_make_argument(d, '-j %d')} "$@" --verbose + python3 ${S}/src/bootstrap/bootstrap.py ${@oe.utils.parallel_make_argument(d, '-j %d')} "$@" --verbose } rust_runx[vardepsexclude] += "PARALLEL_MAKE" -require rust-source.inc -require rust-snapshot.inc - INSANE_SKIP:${PN}:class-native = "already-stripped" FILES:${PN} += "${libdir}/rustlib" FILES:${PN} += "${libdir}/*.so"
@@ -241,7 +254,7 @@ do_compile () { do_test_compile[dirs] = "${B}" do_test_compile () { replace_llvm_config_path - rust_runx build src/tools/remote-test-server --target "${RUST_TARGET_SYS}" + rust_runx build ${S}/src/tools/remote-test-server --target "${RUST_TARGET_SYS}" } ALLOW_EMPTY:${PN} = "1"
@@ -282,7 +295,7 @@ rust_do_install:class-nativesdk() { install -d ${D}${bindir} for i in cargo-clippy clippy-driver rustfmt; do - cp build/${RUST_BUILD_SYS}/stage2-tools/${RUST_HOST_SYS}/release/$i ${D}${bindir} + cp rust-build/${RUST_BUILD_SYS}/stage2-tools/${RUST_HOST_SYS}/release/$i ${D}${bindir} patchelf --set-rpath "\$ORIGIN/../lib" ${D}${bindir}/$i done
@@ -322,7 +335,7 @@ rust_do_install:class-target() { install -d ${D}${bindir} for i in ${EXTRA_TOOLS}; do - cp build/${RUST_BUILD_SYS}/stage2-tools/${RUST_HOST_SYS}/release/$i ${D}${bindir} + cp rust-build/${RUST_BUILD_SYS}/stage2-tools/${RUST_HOST_SYS}/release/$i ${D}${bindir} patchelf --set-rpath "\$ORIGIN/../lib" ${D}${bindir}/$i done
Signature · 1 line
2.49.0
Re: [OE-core] [PATCH v3 2/3] rust: refactor source handling into a shared work-shared recipe
Quoted 2 lines · Deepesh via lists.openembedded.org Varatharajan
From: Deepesh Varatharajan <[email protected]>
Hi Deepesh,
Quoted 20 lines
[YOCTO #15808]
Consolidate Rust source management across all Rust recipes (rust, cargo, libstd-rs and their native/nativesdk variants) into a single shared source tree using the work-shared pattern.
Disk savings: a world build with multilibs enabled, together with nativesdk-rust and nativesdk-cargo, now shares a single ~4.2 GB rustc-src tree instead of maintaining up to 11 independent copies (~46.2 GB total).
Performance (clean core-image-sato build, qemux86-64):
meta/recipes-devtools/rust/rust-source.bb new file +5
-0@@ -0,0 +1,56 @@ +# rust-source.bb - Shared source recipe for the Rust toolchain +# +# This recipe fetches, unpacks, and patches the rustc source tree into a +# work-shared location so that rust, cargo, libstd-rs, and their native/nativesdk +# variants all share a single copy of the source. This eliminates duplication
meta/recipes-devtools/rust/rust-source.bb line 15
Quoted 10 lines
meta/recipes-devtools/rust/rust-source.bb new file +10
-0+# (up to 11 copies of the ~4.2 GB source tree in multilib + nativesdk builds) +# and significantly reduces disk usage (~46.2 GB -> ~4.2 GB for rustc-src). +# +# Only do_fetch, do_unpack, and do_patch are meaningful here. All other +# tasks are disabled since this recipe exists solely to provide sources. + +SUMMARY = "Rust sources" +HOMEPAGE = "http://www.rust-lang.org" +SECTION = "devel" +LICENSE = "(Unicode-3.0 AND (Apache-2.0 OR MIT)"
We are missing a closing parentheses here. I believe you can remove the opening one.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Re: [OE-core] [PATCH v3 2/3] rust: refactor source handling into a shared work-shared recipe
In reply to [email protected]Quoted 97 lines · Deepesh via lists.openembedded.org
From: Deepesh Varatharajan <[email protected]>
[YOCTO #15808]
Consolidate Rust source management across all Rust recipes (rust, cargo, libstd-rs and their native/nativesdk variants) into a single shared source tree using the work-shared pattern.
Disk savings: a world build with multilibs enabled, together with nativesdk-rust and nativesdk-cargo, now shares a single ~4.2 GB rustc-src tree instead of maintaining up to 11 independent copies (~46.2 GB total).
Performance (clean core-image-sato build, qemux86-64):
Before: 68m53.507s
After: 65m46.159s
Improvement: 3m07.348s (~4.5%)New files:
- rust-source.bb: A shared recipe that fetches, unpacks, and patches the rustc source tree into ${TMPDIR}/work-shared/rust-source-${PV}-${PR}. All build/install/packaging tasks are disabled; this recipe exists solely to provide a single copy of the source. Inherits allarch/nopackages, excludes itself from
rm_work, and is hidden from world builds. Also definesRUST_BUILD_ARCHlocally since this recipe does not inherit rust-common.bbclass but still needs the mapping for snapshot fetching via rust-snapshot.inc.- common-source.inc: Included by consumer recipes (rust, cargo, libstd-rs) to disable their local do_fetch/do_unpack/do_patch tasks, clear
SRC_URI, and add task dependencies on rust-source-${PV} for configure, licensing, and SPDX generation. Also providesis_work_shared_spdx()override for correct SPDX shared-workdir detection.Modified files:
- rust-source.inc:
- ReplaceRUST_VERSIONwith explicit MAJOR_VER/MINOR_VER/PATCH_VER and PV.
- Set UNPACKDIR to the work-shared location so all consumers share one source tree via RUSTSRC and S.- rust_git.bb (renamed from rust_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- Set B = ${WORKDIR}/build to separate build artifacts from shared source.
- SetCARGO_HOME= ${WORKDIR}/cargo_home to isolate cargo caching.
- Set build-dir = "rust-build" in config.toml to distinguish Yocto's build directory (B), which holds config.toml, from the bootstrap's own build output directory. Without this, bootstrap defaults to "build/" inside B, creating an ambiguous build/build/ nesting.
- Use absolute${S}/src/bootstrap/bootstrap.pypath since cwd is now B.
- Update install functions to reference rust-build/ instead of build/.
- Updatedo_test_compileto use absolute ${S}/src/tools/ path.
- Changedo_rust_setup_snapshotfrom 'afterdo_unpack' to depend on rust-source-${PV}:do_unpack (since localdo_unpackis removed).- cargo_git.bb (renamed from cargo_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- Changedo_cargo_setup_snapshotfrom 'afterdo_unpack' to depend on rust-source-${PV}:do_unpack.- libstd-rs_git.bb (renamed from libstd-rs_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.- rust-cross-canadian_git.bb (renamed from rust-cross-canadian_1.96.1.bb):
- Rename only (version now comes from rust-source.inc via PV).- rust-common.bbclass:
- Add a second --remap-path-prefix for ${TMPDIR}/work-shared so debug info from the shared source tree is correctly remapped.
- RetainRUST_BUILD_ARCHdefinition so that consumer recipes (rust, cargo, libstd-rs) which inherit this class can resolve snapshot directory names used bydo_rust_setup_snapshotanddo_cargo_setup_snapshot.-
lib/oeqa/selftest/cases/rust.py:
- Adapt the selftest to the build-dir/source-dir split: use B for build artifacts and RUSTSRC for source paths.
- Update remote-test-server copy path from build/ to rust-build/.
- Pass --build-dir and --config explicitly tobootstrap.py.- conf/distro/include/maintainers.inc:
- Add maintainer entry for the new rust-source recipe.License-Update: recipe file renames only, no license change
Signed-off-by Deepesh Varatharajan <[email protected]>---
meta/classes-recipe/rust-common.bbclass | 4 +- meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oeqa/selftest/cases/rust.py | 9 ++- .../rust/{cargo_1.96.1.bb => cargo_git.bb} | 8 ++- meta/recipes-devtools/rust/common-source.inc | 28 ++++++++++ .../{libstd-rs_1.96.1.bb => libstd-rs_git.bb} | 4 ++ ...n_1.96.1.bb => rust-cross-canadian_git.bb} | 0 meta/recipes-devtools/rust/rust-source.bb | 56 +++++++++++++++++++ meta/recipes-devtools/rust/rust-source.inc | 16 +++++- .../rust/{rust_1.96.1.bb => rust_git.bb} | 31 +++++++---
Quoted 7 lines
10 files changed, 140 insertions(+), 17 deletions(-)rename meta/recipes-devtools/rust/{cargo_1.96.1.bb => cargo_git.bb} (91%)
create mode 100644 meta/recipes-devtools/rust/common-source.inc
rename meta/recipes-devtools/rust/{libstd-rs_1.96.1.bb => libstd-rs_git.bb} (92%) rename meta/recipes-devtools/rust/{rust-cross-canadian_1.96.1.bb => rust-cross-canadian_git.bb} (100%) create mode 100644 meta/recipes-devtools/rust/rust-source.bb
rename meta/recipes-devtools/rust/{rust_1.96.1.bb => rust_git.bb} (93%)
In the patch review call we noted that "_git" isn't accurate as these are version'd recipes, not git based. There was a suggestion to just remove that piece but without the versions, it would make it harder to have multiple rust versions in parallel should that be needed. This is why the gcc recipes retain a version in the filenames.
Our suggestion would be to have rust match what gcc does for those two reasons.
Otherwise with the LICENSE bracket fix, we think is is probably ready to merge. The space savings are nice to have!
Cheers,
Richard
Re: [OE-core] [PATCH v3 2/3] rust: refactor source handling into a shared work-shared recipe
Quoted 117 lines · Richard Purdie
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.Quoted 98 lines · Deepesh via lists.openembedded.org
From: Deepesh Varatharajan <[email protected]>
[YOCTO #15808]
Consolidate Rust source management across all Rust recipes (rust, cargo, libstd-rs and their native/nativesdk variants) into a single shared source tree using the work-shared pattern.
Disk savings: a world build with multilibs enabled, together with nativesdk-rust and nativesdk-cargo, now shares a single ~4.2 GB rustc-src tree instead of maintaining up to 11 independent copies (~46.2 GB total).
Performance (clean core-image-sato build, qemux86-64):
Before: 68m53.507s
After: 65m46.159s
Improvement: 3m07.348s (~4.5%)New files:
- rust-source.bb: A shared recipe that fetches, unpacks, and patches the rustc source tree into ${TMPDIR}/work-shared/rust-source-${PV}-${PR}. All build/install/packaging tasks are disabled; this recipe exists solely to provide a single copy of the source. Inherits allarch/nopackages, excludes itself from
rm_work, and is hidden from world builds. Also definesRUST_BUILD_ARCHlocally since this recipe does not inherit rust-common.bbclass but still needs the mapping for snapshot fetching via rust-snapshot.inc.- common-source.inc: Included by consumer recipes (rust, cargo, libstd-rs) to disable their local do_fetch/do_unpack/do_patch tasks, clear
SRC_URI, and add task dependencies on rust-source-${PV} for configure, licensing, and SPDX generation. Also providesis_work_shared_spdx()override for correct SPDX shared-workdir detection.Modified files:
- rust-source.inc:
- ReplaceRUST_VERSIONwith explicit MAJOR_VER/MINOR_VER/PATCH_VER and PV.
- Set UNPACKDIR to the work-shared location so all consumers share one source tree via RUSTSRC and S.- rust_git.bb (renamed from rust_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- Set B = ${WORKDIR}/build to separate build artifacts from shared source.
- SetCARGO_HOME= ${WORKDIR}/cargo_home to isolate cargo caching.
- Set build-dir = "rust-build" in config.toml to distinguish Yocto's build directory (B), which holds config.toml, from the bootstrap's own build output directory. Without this, bootstrap defaults to "build/" inside B, creating an ambiguous build/build/ nesting.
- Use absolute${S}/src/bootstrap/bootstrap.pypath since cwd is now B.
- Update install functions to reference rust-build/ instead of build/.
- Updatedo_test_compileto use absolute ${S}/src/tools/ path.
- Changedo_rust_setup_snapshotfrom 'afterdo_unpack' to depend onrust-source-${PV}:do_unpack (since local do_unpack is removed).- cargo_git.bb (renamed from cargo_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.
- Changedo_cargo_setup_snapshotfrom 'afterdo_unpack' to depend onrust-source-${PV}:do_unpack.- libstd-rs_git.bb (renamed from libstd-rs_1.96.1.bb):
- Add 'require common-source.inc' to use the shared source tree.- rust-cross-canadian_git.bb (renamed from rust-cross-canadian_1.96.1.bb):
- Rename only (version now comes from rust-source.inc via PV).- rust-common.bbclass:
- Add a second --remap-path-prefix for ${TMPDIR}/work-shared so debug info from the shared source tree is correctly remapped.
- RetainRUST_BUILD_ARCHdefinition so that consumer recipes (rust, cargo, libstd-rs) which inherit this class can resolve snapshot directory names used bydo_rust_setup_snapshotanddo_cargo_setup_snapshot.-
lib/oeqa/selftest/cases/rust.py:
- Adapt the selftest to the build-dir/source-dir split: use B for build artifacts and RUSTSRC for source paths.
- Update remote-test-server copy path from build/ to rust-build/.
- Pass --build-dir and --config explicitly tobootstrap.py.- conf/distro/include/maintainers.inc:
- Add maintainer entry for the new rust-source recipe.License-Update: recipe file renames only, no license change
Signed-off-by Deepesh Varatharajan <[email protected]>--- meta/classes-recipe/rust-common.bbclass | 4 +- meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oeqa/selftest/cases/rust.py | 9 ++- .../rust/{cargo_1.96.1.bb => cargo_git.bb} | 8 ++- meta/recipes-devtools/rust/common-source.inc | 28 ++++++++++ .../{libstd-rs_1.96.1.bb => libstd-rs_git.bb} | 4 ++ ...n_1.96.1.bb => rust-cross-canadian_git.bb} | 0 meta/recipes-devtools/rust/rust-source.bb | 56 +++++++++++++++++++ meta/recipes-devtools/rust/rust-source.inc | 16 +++++- .../rust/{rust_1.96.1.bb => rust_git.bb} | 31 +++++++--- 10 files changed, 140 insertions(+), 17 deletions(-)Quoted 6 lines
rename meta/recipes-devtools/rust/{cargo_1.96.1.bb => cargo_git.bb} (91%)
create mode 100644 meta/recipes-devtools/rust/common-source.inc
rename meta/recipes-devtools/rust/{libstd-rs_1.96.1.bb => libstd-rs_git.bb} (92%) rename meta/recipes-devtools/rust/{rust-cross-canadian_1.96.1.bb => rust-cross-canadian_git.bb} (100%) create mode 100644 meta/recipes-devtools/rust/rust-source.bb
rename meta/recipes-devtools/rust/{rust_1.96.1.bb => rust_git.bb} (93%)In the patch review call we noted that "
_git" isn't accurate as these are version'd recipes, not git based. There was a suggestion to just remove that piece but without the versions, it would make it harder to have multiple rust versions in parallel should that be needed. This is why the gcc recipes retain a version in the filenames.Our suggestion would be to have rust match what gcc does for those two reasons.
Quoted 6 lines
Otherwise with the LICENSE bracket fix, we think is is probably ready to merge. The space savings are nice to have!
Cheers,
Richard
Hi Richard,
Thanks for the review feedback!
Makes sense I'll rename the recipes to include the version in the filename, matching the gcc convention. Also get the LICENSE bracket fix in. Will send a new version shortly.
Regards,
Deepesh
PATCH v3 3/3 rust: Move do_update_snapshot to rust-source
In reply to [email protected]From: Deepesh Varatharajan <[email protected]>
Move do_update_snapshot from rust to rust-source, since rust-source now owns do_fetch/do_unpack/do_patch tasks after the shared source refactoring.
--- meta/recipes-devtools/rust/rust-source.bb | 72 ++++++++++++++++++++++ meta/recipes-devtools/rust/rust_git.bb | 73 ----------------------- 2 files changed, 72 insertions(+), 73 deletions(-)
meta/recipes-devtools/rust/rust-source.bb +72 -0
@@ -54,3 +54,75 @@ B = "${WORKDIR}/build" RUST_BUILD_ARCH = "${@{'ppc': 'powerpc', 'ppc64': 'powerpc64', 'ppc64le': 'powerpc64le', 'riscv64': 'riscv64gc'}.get(d.getVar('BUILD_ARCH'), d.getVar('BUILD_ARCH'))}" EXCLUDE_FROM_WORLD = "1" + +addtask do_update_snapshot after do_patch +do_update_snapshot[nostamp] = "1" + +# Run with `bitbake rust-source-${PV} -c update_snapshot` to update `rust-snapshot.inc` +# with the checksums for the rust snapshot associated with this rustc-src tarball. +python do_update_snapshot() { + import json + import re + import sys + + from collections import defaultdict + + key_value_pairs = {} + with open(os.path.join(d.getVar("S"), "src", "stage0")) as f: + for line in f: + # Skip empty lines or comments + if not line.strip() or line.startswith("#"): + continue + # Split the line into key and value using '=' as separator + match = re.match(r'(\S+)\s*=\s*(\S+)', line.strip()) + if match: + key = match.group(1) + value = match.group(2) + key_value_pairs[key] = value + # Extract the required values from key_value_pairs + config_dist_server = key_value_pairs.get('dist_server', '') + compiler_date = key_value_pairs.get('compiler_date', '') + compiler_version = key_value_pairs.get('compiler_version', '') + + src_uri = defaultdict(list) + # Assuming checksums_sha256 is now a key-value pair like: checksum_key = checksum_value + for k, v in key_value_pairs.items(): + # Match the pattern for checksums + if "dist" in k and "tar.xz" in k: + m = re.search(f"dist/{compiler_date}/(?P<component>.*)-{compiler_version}-(?P<arch>.*)-unknown-linux-gnu\\.tar\\.xz", k) + if m: + component = m.group('component') + arch = m.group('arch') + src_uri[arch].append(f"SRC_URI[{component}-snapshot-{arch}.sha256sum] = \"{v}\"") + # Create the snapshot string with the extracted values + snapshot = """\ +## This is information on the rust-snapshot (binary) used to build our current release. +## snapshot info is taken from rust/src/stage0 +## Rust is self-hosting and bootstraps itself with a pre-built previous version of itself. +## The exact (previous) version that has been used is specified in the source tarball. +## The version is replicated here. + +SNAPSHOT_VERSION = "%s" + +""" % compiler_version + # Add the checksum components to the snapshot + for arch, components in src_uri.items(): + snapshot += "\n".join(components) + "\n\n" + # Add the additional snapshot URIs + snapshot += """\ +SRC_URI += " \\ + ${RUST_DIST_SERVER}/dist/${RUST_STD_SNAPSHOT}.tar.xz;name=rust-std-snapshot-${RUST_BUILD_ARCH};subdir=rust-snapshot-components \\ + ${RUST_DIST_SERVER}/dist/${RUSTC_SNAPSHOT}.tar.xz;name=rustc-snapshot-${RUST_BUILD_ARCH};subdir=rust-snapshot-components \\ + ${RUST_DIST_SERVER}/dist/${CARGO_SNAPSHOT}.tar.xz;name=cargo-snapshot-${RUST_BUILD_ARCH};subdir=rust-snapshot-components \\ +" + +RUST_DIST_SERVER = "%s" + +RUST_STD_SNAPSHOT = "rust-std-${SNAPSHOT_VERSION}-${RUST_BUILD_ARCH}-unknown-linux-gnu" +RUSTC_SNAPSHOT = "rustc-${SNAPSHOT_VERSION}-${RUST_BUILD_ARCH}-unknown-linux-gnu" +CARGO_SNAPSHOT = "cargo-${SNAPSHOT_VERSION}-${RUST_BUILD_ARCH}-unknown-linux-gnu" +""" % config_dist_server + # Write the updated snapshot information to the rust-snapshot.inc file + with open(os.path.join(d.getVar("THISDIR"), "rust-snapshot.inc"), "w") as f: + f.write(snapshot) +}
meta/recipes-devtools/rust/rust_git.bb +0 -73
79 lines not on this page · raw message
Signature · 1 line
2.49.0