rust: Avoid passing host-specific information to crates
PATCH v4 1/1 rust: Avoid passing host-specific information to crates
There were several build contamination issues found on our rust builds:
SVH - rustc computes each crate's Strict Version Hash (SVH) using inputs that
include the *stage0/stage1 bootstrap compiler* fingerprint, which in turn depends on the build host arch.
This eventually may cause sstate matches across architectures for artifacts that are actually different, causing autobuilder intermitent reproducibility issues.
To avoid this, pass a fixed value instead of host-specific bits to the specified hash.
Cmetadata - Cargo hashes the host: line of rustc -vV into the metadata of
host units (build scripts and proc-macros), so target libraries that depend on a crate carrying a build script inherited the build host triple through -Cmetadata and, from there, each crate's StableCrateId.
Unordered data - DocLinkResMap is an UnordMap wrapping an FxHashMap, so the
doc-link table was serialised into crate metadata in hash iteration order, which is not stable across hosts. Switch it to FxIndexMap, which is already Encodable, Decodable and HashStable.
HygieneEncodeContext::encode consumed latest_ctxts and latest_expns directly from an FxHashSet without sorting. Payload blobs are serialized into the crate metadata buffer during this loop in hash iteration order. Sort both before encoding so payloads are written into metadata in deterministic order.
With all three addressed, Rust metadata and compiled libraries are reproducible across mixed-architecture build hosts (x86_64 vs aarch64).
[YOCTO #16376]
--- ...ide-cfg-version-from-stable-crate-id.patch | 36 +++++++++++ ...-host-triple-from-unit-metadata-hash.patch | 55 +++++++++++++++++ ...oc-link-metadata-order-deterministic.patch | 36 +++++++++++ ...hygiene-encoding-order-deterministic.patch | 60 +++++++++++++++++++ meta/recipes-devtools/rust/rust-source.inc | 4 ++ 5 files changed, 191 insertions(+) create mode 100644 meta/recipes-devtools/rust/files/0006-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch create mode 100644 meta/recipes-devtools/rust/files/0007-cargo-omit-host-triple-from-unit-metadata-hash.patch create mode 100644 meta/recipes-devtools/rust/files/0008-rustc-hir-make-doc-link-metadata-order-deterministic.patch create mode 100644 meta/recipes-devtools/rust/files/0009-rustc-span-make-hygiene-encoding-order-deterministic.patch
meta/recipes-devtools/rust/files/0006-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch new file +36 -0
@@ -0,0 +1,36 @@ +rust: Avoid passing host-dependent fingerprint to build artifacts + +To fix reproducibility issues, pass a fixed value instead of passing a +host-dependent fingerprint to the rust build artifacts via the Strict +Version Hash (SVH) + +Upstream-Status: Inappropriate [OE-specific] +Assisted-by: AI - OpenAI +Signed-off-by: Alejandro Hernandez <[email protected]> +--- +--- a/compiler/rustc_span/src/def_id.rs ++++ b/compiler/rustc_span/src/def_id.rs +@@ -163,7 +163,7 @@ + crate_name: Symbol, + is_exe: bool, + mut metadata: Vec<String>, +- cfg_version: &'static str, ++ _cfg_version: &'static str, + ) -> StableCrateId { + let mut hasher = StableHasher::new(); + // We must hash the string text of the crate name, not the id, as the id is not stable +@@ -195,11 +195,9 @@ + // + // RUSTC_FORCE_RUSTC_VERSION is used to inject rustc version information + // during testing. +- if let Some(val) = std::env::var_os("RUSTC_FORCE_RUSTC_VERSION") { +- hasher.write(val.to_string_lossy().into_owned().as_bytes()) +- } else { +- hasher.write(cfg_version.as_bytes()) +- } ++ // OE reproducible builds use a fixed value so host-varying bootstrap ++ // fingerprints do not perturb StableCrateId. ++ hasher.write(b"oe-stable-crate-id-no-cfg-version"); + + StableCrateId(hasher.finish()) + }
meta/recipes-devtools/rust/files/0007-cargo-omit-host-triple-from-unit-metadata-hash.patch new file +55 -0
55 lines not on this page · raw message
meta/recipes-devtools/rust/files/0008-rustc-hir-make-doc-link-metadata-order-deterministic.patch new file +36 -0
@@ -0,0 +1,36 @@ +rustc_hir: serialise doc-link resolutions in a deterministic order + +DocLinkResMap is an UnordMap, which wraps an FxHashMap and derives its +Encodable implementation, so crate metadata records the doc-link table in hash +iteration order. That order is not stable across build hosts, leaving a few +hundred bytes of the .rustc section, and the DefIndex values that follow it, +different for otherwise identical builds. + +Use an insertion-ordered FxIndexMap instead. Entries are added while walking +the AST, so insertion order is deterministic, and the consumers of this map +only ever look entries up by key. + +Upstream-Status: Inappropriate [OE-specific] +Assisted-by: AI - OpenAI +Signed-off-by: Alejandro Hernandez <[email protected]> +--- +--- a/compiler/rustc_hir/src/def.rs ++++ b/compiler/rustc_hir/src/def.rs +@@ -4,7 +4,7 @@ + + use rustc_ast as ast; + use rustc_ast::NodeId; +-use rustc_data_structures::unord::UnordMap; ++use rustc_data_structures::fx::FxIndexMap; + use rustc_error_messages::{DiagArgValue, IntoDiagArg}; + use rustc_macros::{Decodable, Encodable, StableHash}; + use rustc_span::Symbol; +@@ -969,4 +969,7 @@ + ElidedAnchor { start: NodeId, end: NodeId }, + } + +-pub type DocLinkResMap = UnordMap<(Symbol, Namespace), Option<Res<NodeId>>>; ++// Serialise doc-link resolutions in insertion order: UnordMap wraps an ++// FxHashMap, whose iteration order varies with the build host and leaves ++// crate metadata unreproducible across builders. ++pub type DocLinkResMap = FxIndexMap<(Symbol, Namespace), Option<Res<NodeId>>>;
meta/recipes-devtools/rust/files/0009-rustc-span-make-hygiene-encoding-order-deterministic.patch new file +60 -0
60 lines not on this page · raw message
meta/recipes-devtools/rust/rust-source.inc +4 -0
@@ -9,6 +9,10 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n file://0003-bootstrap-skip-StdarchVerify-when-remote-testing.patch;patchdir=${RUSTSRC} \ file://0004-Backport-commits-from-rust-Fix-selftest-llvm23.patch;patchdir=${RUSTSRC} \ file://0005-rustc_codegen_llvm-Do-not-pass-amx-tf32-to-LLVM-23.patch;patchdir=${RUSTSRC} \ + file://0006-rustc-span-add-oe-knob-to-elide-cfg-version-from-stable-crate-id.patch;patchdir=${RUSTSRC} \ + file://0007-cargo-omit-host-triple-from-unit-metadata-hash.patch;patchdir=${RUSTSRC} \ + file://0008-rustc-hir-make-doc-link-metadata-order-deterministic.patch;patchdir=${RUSTSRC} \ + file://0009-rustc-span-make-hygiene-encoding-order-deterministic.patch;patchdir=${RUSTSRC} \ " SRC_URI[rust.sha256sum] = "be1816e7f6c40abb90245ad6e024bed2a7e88d7dda4561e4d5470207df616b9f"
Signature · 1 line
2.43.0
Patchtest results for [PATCH v4] rust: Avoid passing host-specific information to crates
Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:
---
Testing patch /home/patchtest/share/mboxes/v4-rust-Avoid-passing-host-specific-information-to-crates.patch
FAIL: test Signed-off-by presence: A patch file has been added without a Signed-off-by tag: '0009-rustc-span-make-hygiene-encoding-order-deterministic.patch' (test_patch.TestPatch.test_signed_off_by_presence)
PASS: test CVE tag format (test_patch.TestPatch.test_cve_tag_format)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test Upstream-Status presence (test_patch.TestPatch.test_upstream_status_presence_format)
PASS: test auh changelog truncation notice (test_mbox.TestMbox.test_auh_changelog_truncation_notice)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test bugzilla entry format (test_mbox.TestMbox.test_bugzilla_entry_format)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test commit message user tags (test_mbox.TestMbox.test_commit_message_user_tags)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)
SKIP: pretest pylint: No python related patches, skipping test (test_python_pylint.PyLint.pretest_pylint)
SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test pylint: No python related patches, skipping test (test_python_pylint.PyLint.test_pylint)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping test (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
---
Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!