Skip to contentSkip to conversation

rust: Avoid passing host-specific information to crates

PATCH New Alejandro Hernandez 1 participant 1 message Sep 8

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/1 rust: Avoid passing host-specific information to crates

There were several build contamination issues found on our rust builds:

SVH - rustc computes each crate's Strict Version Hash (SVH) using inputs that
include the *stage0/stage1 bootstrap compiler* fingerprint, which in turn depends on the build host arch.

This eventually may cause sstate matches across architectures for artifacts that are actually different, causing autobuilder intermitent reproducibility issues.

To avoid this, pass a fixed value instead of host-specific bits to the specified hash.

Cmetadata - Cargo hashes the host: line of rustc -vV into the metadata of
host units (build scripts and proc-macros), so target libraries that depend on a crate carrying a build script inherited the build host triple through -Cmetadata and, from there, each crate's StableCrateId.

Unordered data - DocLinkResMap is an UnordMap wrapping an FxHashMap, so the
doc-link table was serialised into crate metadata in hash iteration order, which is not stable across hosts. Switch it to FxIndexMap, which is already Encodable, Decodable and HashStable.

HygieneEncodeContext::encode consumed latest_ctxts and latest_expns directly from an FxHashSet without sorting. Payload blobs are serialized into the crate metadata buffer during this loop in hash iteration order. Sort both before encoding so payloads are written into metadata in deterministic order.

With all three addressed, Rust metadata and compiled libraries are reproducible across mixed-architecture build hosts (x86_64 vs aarch64).

[YOCTO #16376]

Assisted-by AI - OpenAI
---
 ...ide-cfg-version-from-stable-crate-id.patch | 36 +++++++++++
 ...-host-triple-from-unit-metadata-hash.patch | 55 +++++++++++++++++
 ...oc-link-metadata-order-deterministic.patch | 36 +++++++++++
 ...hygiene-encoding-order-deterministic.patch | 59 +++++++++++++++++++
 meta/recipes-devtools/rust/rust-source.inc    |  4 ++
 5 files changed, 190 insertions(+)
 create mode 100644 meta/recipes-devtools/rust/files/0006-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch
 create mode 100644 meta/recipes-devtools/rust/files/0007-cargo-omit-host-triple-from-unit-metadata-hash.patch
 create mode 100644 meta/recipes-devtools/rust/files/0008-rustc-hir-make-doc-link-metadata-order-deterministic.patch
 create mode 100644 meta/recipes-devtools/rust/files/0009-rustc-span-make-hygiene-encoding-order-deterministic.patch
meta/recipes-devtools/rust/files/0006-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch new file +36 -0
new file mode 100644
index 00000000000..a7d03d9c363
@@ -0,0 +1,36 @@
+rust: Avoid passing host-dependent fingerprint to build artifacts
+
+To fix reproducibility issues, pass a fixed value instead of passing a
+host-dependent fingerprint to the rust build artifacts via the Strict
+Version Hash (SVH)
+
+Upstream-Status: Inappropriate [OE-specific]
+Assisted-by: AI - OpenAI
+Signed-off-by: Alejandro Hernandez <[email protected]>
+---
+--- a/compiler/rustc_span/src/def_id.rs
++++ b/compiler/rustc_span/src/def_id.rs
+@@ -163,7 +163,7 @@
+         crate_name: Symbol,
+         is_exe: bool,
+         mut metadata: Vec<String>,
+-        cfg_version: &'static str,
++        _cfg_version: &'static str,
+     ) -> StableCrateId {
+         let mut hasher = StableHasher::new();
+         // We must hash the string text of the crate name, not the id, as the id is not stable
+@@ -195,11 +195,9 @@
+         //
+         // RUSTC_FORCE_RUSTC_VERSION is used to inject rustc version information
+         // during testing.
+-        if let Some(val) = std::env::var_os("RUSTC_FORCE_RUSTC_VERSION") {
+-            hasher.write(val.to_string_lossy().into_owned().as_bytes())
+-        } else {
+-            hasher.write(cfg_version.as_bytes())
+-        }
++        // OE reproducible builds use a fixed value so host-varying bootstrap
++        // fingerprints do not perturb StableCrateId.
++        hasher.write(b"oe-stable-crate-id-no-cfg-version");
+ 
+         StableCrateId(hasher.finish())
+     }
meta/recipes-devtools/rust/files/0007-cargo-omit-host-triple-from-unit-metadata-hash.patch new file +55 -0

55 lines not on this page · raw message

meta/recipes-devtools/rust/files/0008-rustc-hir-make-doc-link-metadata-order-deterministic.patch new file +36 -0
new file mode 100644
index 00000000000..46fb2ed9712
@@ -0,0 +1,36 @@
+rustc_hir: serialise doc-link resolutions in a deterministic order
+
+DocLinkResMap is an UnordMap, which wraps an FxHashMap and derives its
+Encodable implementation, so crate metadata records the doc-link table in hash
+iteration order. That order is not stable across build hosts, leaving a few
+hundred bytes of the .rustc section, and the DefIndex values that follow it,
+different for otherwise identical builds.
+
+Use an insertion-ordered FxIndexMap instead. Entries are added while walking
+the AST, so insertion order is deterministic, and the consumers of this map
+only ever look entries up by key.
+
+Upstream-Status: Inappropriate [OE-specific]
+Assisted-by: AI - OpenAI
+Signed-off-by: Alejandro Hernandez <[email protected]>
+---
+--- a/compiler/rustc_hir/src/def.rs
++++ b/compiler/rustc_hir/src/def.rs
+@@ -4,7 +4,7 @@
+ 
+ use rustc_ast as ast;
+ use rustc_ast::NodeId;
+-use rustc_data_structures::unord::UnordMap;
++use rustc_data_structures::fx::FxIndexMap;
+ use rustc_error_messages::{DiagArgValue, IntoDiagArg};
+ use rustc_macros::{Decodable, Encodable, StableHash};
+ use rustc_span::Symbol;
+@@ -969,4 +969,7 @@
+     ElidedAnchor { start: NodeId, end: NodeId },
+ }
+ 
+-pub type DocLinkResMap = UnordMap<(Symbol, Namespace), Option<Res<NodeId>>>;
++// Serialise doc-link resolutions in insertion order: UnordMap wraps an
++// FxHashMap, whose iteration order varies with the build host and leaves
++// crate metadata unreproducible across builders.
++pub type DocLinkResMap = FxIndexMap<(Symbol, Namespace), Option<Res<NodeId>>>;
meta/recipes-devtools/rust/files/0009-rustc-span-make-hygiene-encoding-order-deterministic.patch new file +59 -0

59 lines not on this page · raw message

meta/recipes-devtools/rust/rust-source.inc +4 -0
index 019e2585e28..1d192b1c716 100644
@@ -9,6 +9,10 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n
             file://0003-bootstrap-skip-StdarchVerify-when-remote-testing.patch;patchdir=${RUSTSRC} \
             file://0004-Backport-commits-from-rust-Fix-selftest-llvm23.patch;patchdir=${RUSTSRC} \
             file://0005-rustc_codegen_llvm-Do-not-pass-amx-tf32-to-LLVM-23.patch;patchdir=${RUSTSRC} \
+            file://0006-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch;patchdir=${RUSTSRC} \
+            file://0007-cargo-omit-host-triple-from-unit-metadata-hash.patch;patchdir=${RUSTSRC} \
+            file://0008-rustc-hir-make-doc-link-metadata-order-deterministic.patch;patchdir=${RUSTSRC} \
+            file://0009-rustc-span-make-hygiene-encoding-order-deterministic.patch;patchdir=${RUSTSRC} \
 "
 SRC_URI[rust.sha256sum] = "be1816e7f6c40abb90245ad6e024bed2a7e88d7dda4561e4d5470207df616b9f"
 
Signature · 1 line
2.43.0