Skip to contentSkip to conversation

rust: rustdoc reproducibility issue fix

PATCH In review [email protected] 2 participants 6 messages Aug 19, 2024

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 rust: rustdoc reproducibility issue fix

From: Deepesh Varatharajan <[email protected]>

The 'codegen-units' option split the crate into multiple compilation units for parallel compilation. Currently, this split is causing the rustdoc to generate differnt binary between the builds. To fix this the codegen-units & the lto options are disabled.

More info about options:
https://doc.rust-lang.org/cargo/reference/profiles.html#codegen-units
https://doc.rust-lang.org/rustc/codegen-options/index.html#lto

---
 ...ue-fix-cargo-config-for-codegenunits.patch | 29 +++++++++++++++++++
 meta/recipes-devtools/rust/rust-source.inc    |  1 +
 meta/recipes-devtools/rust/rust_1.75.0.bb     |  1 +
 3 files changed, 31 insertions(+)
 create mode 100644 meta/recipes-devtools/rust/files/rustdoc-repro-issue-fix-cargo-config-for-codegenunits.patch
meta/recipes-devtools/rust/files/rustdoc-repro-issue-fix-cargo-config-for-codegenunits.patch new file +29 -0
new file mode 100644
index 0000000000..fdb2a4a498
@@ -0,0 +1,29 @@
+commit e2e7017350d0b5324811fef3b841f98b00273887
+Author: Sundeep KOKKONDA <[email protected]>
+Date:   Fri Jul 26 06:35:59 2024 -0700
+
+    rust: rustdoc reproducibility issue fix
+
+    The 'codegen-units' option split the crate into multiple compilation units for parallel compilation. 
+    Currently, this split is causing the rustdoc to generate differnt binary between the builds.
+    To fix this the codegen-units & the lto options are disabled.
+
+    More info about options:
+    https://doc.rust-lang.org/cargo/reference/profiles.html#codegen-units
+    https://doc.rust-lang.org/rustc/codegen-options/index.html#lto
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Deepesh Varatharajan <[email protected]>
+---
+--- a/.cargo/config.toml
++++ b/.cargo/config.toml
+@@ -3,3 +3,7 @@
+
+ [source.vendored-sources]
+ directory = "vendor"
++
++[profile.release]
++codegen-units = 1
++
+
meta/recipes-devtools/rust/rust-source.inc +1 -0
index b14221b6cb..4ffc9faf18 100644
@@ -13,6 +13,7 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n
             file://target-build-value.patch;patchdir=${RUSTSRC} \
             file://0001-Handle-vendored-sources-when-remapping-paths.patch;patchdir=${RUSTSRC} \
             file://repro-issue-fix-with-v175.patch;patchdir=${RUSTSRC} \
+	    file://rustdoc-repro-issue-fix-cargo-config-for-codegenunits.patch;patchdir=${RUSTSRC} \
 "
 SRC_URI[rust.sha256sum] = "4526f786d673e4859ff2afa0bab2ba13c918b796519a25c1acce06dba9542340"
 
meta/recipes-devtools/rust/rust_1.75.0.bb +1 -0
index c33f31d261..7cd34a61c3 100644
@@ -132,6 +132,7 @@ python do_configure() {
     config.add_section("rust")
     config.set("rust", "rpath", e(True))
     config.set("rust", "remap-debuginfo", e(True))
+    config.set("rust", "lto", "\"off\"")
     config.set("rust", "channel", e(d.expand("${RUST_CHANNEL}")))
 
     # Whether or not to optimize the compiler and standard library
Signature · 1 line
2.43.0
#1 lore raw

This is the same patch as the previous one, and doesn't address my comment.

Alex

Quoted 91 lines · deepesh.varatharajan via lists.openembedded.org

From: Deepesh Varatharajan <[email protected]>

The 'codegen-units' option split the crate into multiple compilation units for parallel compilation. Currently, this split is causing the rustdoc to generate differnt binary between the builds. To fix this the codegen-units & the lto options are disabled.

More info about options:
https://doc.rust-lang.org/cargo/reference/profiles.html#codegen-units
https://doc.rust-lang.org/rustc/codegen-options/index.html#lto

---
 ...ue-fix-cargo-config-for-codegenunits.patch | 29 +++++++++++++++++++
 meta/recipes-devtools/rust/rust-source.inc    |  1 +
 meta/recipes-devtools/rust/rust_1.75.0.bb     |  1 +
 3 files changed, 31 insertions(+)
 create mode 100644 meta/recipes-devtools/rust/files/rustdoc-repro-issue-fix-cargo-config-for-codegenunits.patch
meta/recipes-devtools/rust/files/rustdoc-repro-issue-fix-cargo-config-for-codegenunits.patch new file +29 -0
new file mode 100644
index 0000000000..fdb2a4a498
@@ -0,0 +1,29 @@
+commit e2e7017350d0b5324811fef3b841f98b00273887
+Author: Sundeep KOKKONDA <[email protected]>
+Date:   Fri Jul 26 06:35:59 2024 -0700
+
+    rust: rustdoc reproducibility issue fix
+
+    The 'codegen-units' option split the crate into multiple compilation units for parallel compilation.
+    Currently, this split is causing the rustdoc to generate differnt binary between the builds.
+    To fix this the codegen-units & the lto options are disabled.
+
+    More info about options:
+    https://doc.rust-lang.org/cargo/reference/profiles.html#codegen-units
+    https://doc.rust-lang.org/rustc/codegen-options/index.html#lto
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Deepesh Varatharajan <[email protected]>
+---
+--- a/.cargo/config.toml
++++ b/.cargo/config.toml
+@@ -3,3 +3,7 @@
+
+ [source.vendored-sources]
+ directory = "vendor"
++
++[profile.release]
++codegen-units = 1
++
+
meta/recipes-devtools/rust/rust-source.inc +1 -0
index b14221b6cb..4ffc9faf18 100644
@@ -13,6 +13,7 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n
             file://target-build-value.patch;patchdir=${RUSTSRC} \
             file://0001-Handle-vendored-sources-when-remapping-paths.patch;patchdir=${RUSTSRC} \
             file://repro-issue-fix-with-v175.patch;patchdir=${RUSTSRC} \
+           file://rustdoc-repro-issue-fix-cargo-config-for-codegenunits.patch;patchdir=${RUSTSRC} \
 "
 SRC_URI[rust.sha256sum] = "4526f786d673e4859ff2afa0bab2ba13c918b796519a25c1acce06dba9542340"
 
meta/recipes-devtools/rust/rust_1.75.0.bb +1 -1
index c33f31d261..7cd34a61c3 100644
@@ -132,6 +132,7 @@ python do_configure() {
     config.add_section("rust")
     config.set("rust", "rpath", e(True))
     config.set("rust", "remap-debuginfo", e(True))
+    config.set("rust", "lto", "\"off\"")
     config.set("rust", "channel", e(d.expand("${RUST_CHANNEL}")))
 
     # Whether or not to optimize the compiler and standard library
--

2.43.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#203501): https://lists.openembedded.org/g/openembedded-core/message/203501
Mute This Topic: https://lists.openembedded.org/mt/107977579/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw

Hi Alex,

The exception list 'exclude_packages' in meta/lib/oeqa/selftest/cases/reproducible.py is already empty.

Thanks,
Deepesh

Get Outlook for Android<https://aka.ms/AAb9ysg>

________________________________
From: Alexander Kanavin <[email protected]>
Sent: Monday, August 19, 2024 4:24:31 PM
To: Varatharajan, Deepesh <[email protected]>
Cc: [email protected] <[email protected]>; MacLeod, Randy <[email protected]>; Gowda, Naveen <[email protected]>; Moodalappa, Shivaprasad <[email protected]>; Kokkonda, Sundeep <[email protected]>
Subject: Re: [OE-core] [scarthgap][PATCH] rust: rustdoc reproducibility issue fix

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

This is the same patch as the previous one, and doesn't address my comment.

Alex

Quoted 91 lines · deepesh.varatharajan via lists.openembedded.org

From: Deepesh Varatharajan <[email protected]>

The 'codegen-units' option split the crate into multiple compilation units for parallel compilation. Currently, this split is causing the rustdoc to generate differnt binary between the builds. To fix this the codegen-units & the lto options are disabled.

More info about options:
https://doc.rust-lang.org/cargo/reference/profiles.html#codegen-units
https://doc.rust-lang.org/rustc/codegen-options/index.html#lto

---
 ...ue-fix-cargo-config-for-codegenunits.patch | 29 +++++++++++++++++++
 meta/recipes-devtools/rust/rust-source.inc    |  1 +
 meta/recipes-devtools/rust/rust_1.75.0.bb     |  1 +
 3 files changed, 31 insertions(+)
 create mode 100644 meta/recipes-devtools/rust/files/rustdoc-repro-issue-fix-cargo-config-for-codegenunits.patch
meta/recipes-devtools/rust/files/rustdoc-repro-issue-fix-cargo-config-for-codegenunits.patch new file +29 -0
new file mode 100644
index 0000000000..fdb2a4a498
@@ -0,0 +1,29 @@
+commit e2e7017350d0b5324811fef3b841f98b00273887
+Author: Sundeep KOKKONDA <[email protected]>
+Date:   Fri Jul 26 06:35:59 2024 -0700
+
+    rust: rustdoc reproducibility issue fix
+
+    The 'codegen-units' option split the crate into multiple compilation units for parallel compilation.
+    Currently, this split is causing the rustdoc to generate differnt binary between the builds.
+    To fix this the codegen-units & the lto options are disabled.
+
+    More info about options:
+    https://doc.rust-lang.org/cargo/reference/profiles.html#codegen-units
+    https://doc.rust-lang.org/rustc/codegen-options/index.html#lto
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Deepesh Varatharajan <[email protected]>
+---
+--- a/.cargo/config.toml
++++ b/.cargo/config.toml
+@@ -3,3 +3,7 @@
+
+ [source.vendored-sources]
+ directory = "vendor"
++
++[profile.release]
++codegen-units = 1
++
+
meta/recipes-devtools/rust/rust-source.inc +1 -0
index b14221b6cb..4ffc9faf18 100644
@@ -13,6 +13,7 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n
             file://target-build-value.patch;patchdir=${RUSTSRC} \
             file://0001-Handle-vendored-sources-when-remapping-paths.patch;patchdir=${RUSTSRC} \
             file://repro-issue-fix-with-v175.patch;patchdir=${RUSTSRC} \
+           file://rustdoc-repro-issue-fix-cargo-config-for-codegenunits.patch;patchdir=${RUSTSRC} \
 "
 SRC_URI[rust.sha256sum] = "4526f786d673e4859ff2afa0bab2ba13c918b796519a25c1acce06dba9542340"
 
meta/recipes-devtools/rust/rust_1.75.0.bb +1 -1
index c33f31d261..7cd34a61c3 100644
@@ -132,6 +132,7 @@ python do_configure() {
     config.add_section("rust")
     config.set("rust", "rpath", e(True))
     config.set("rust", "remap-debuginfo", e(True))
+    config.set("rust", "lto", "\"off\"")
     config.set("rust", "channel", e(d.expand("${RUST_CHANNEL}")))
 
     # Whether or not to optimize the compiler and standard library
--

2.43.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#203501): https://lists.openembedded.org/g/openembedded-core/message/203501
Mute This Topic: https://lists.openembedded.org/mt/107977579/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw
Quoted 1 line · Deepesh

The exception list 'exclude_packages' in meta/lib/oeqa/selftest/cases/reproducible.py is already empty.

If it's already empty, then what reproducibility problem is the patch fixing? Something does not compute here.

Alex

#4 lore raw

Hi Alex,

Yes, but the rustdoc reproducibility changes from this patch are not there in the Scarthgap branch. So, the missed changes are backported from master branch.

Thanks,
Deepesh

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Alexander Kanavin <[email protected]>
Sent: Monday, August 19, 2024 4:37:26 PM
To: Varatharajan, Deepesh <[email protected]>
Cc: [email protected] <[email protected]>; MacLeod, Randy <[email protected]>; Gowda, Naveen <[email protected]>; Moodalappa, Shivaprasad <[email protected]>; Kokkonda, Sundeep <[email protected]>
Subject: Re: [OE-core] [scarthgap][PATCH] rust: rustdoc reproducibility issue fix

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Quoted 1 line · Deepesh

The exception list 'exclude_packages' in meta/lib/oeqa/selftest/cases/reproducible.py is already empty.

If it's already empty, then what reproducibility problem is the patch fixing? Something does not compute here.

Alex

#5 lore raw
Quoted 1 line · Deepesh

Yes, but the rustdoc reproducibility changes from this patch are not there in the Scarthgap branch. So, the missed changes are backported from master branch.

Why do we need to backport them, if reproducibility test is already passing and rust is not excluded from the test? This does not make sense, and needs to be looked into.

I would suggest looking at the point where scarhgap and master diverges, and checking what changes to meta/recipes-devtools/rust happened since in master. I just did this, and I'm certain you need to backport additional commits from master, but I'm reluctant to tell you which ones exactly - you'd learn more by doing this exercise.

Alex