Skip to contentSkip to conversation

kbuild: mark `rustc` (and others) invocations as recursive

PATCH In review Miguel Ojeda 5 participants 6 messages Feb 17, 2024 – Feb 29, 2024

Subsystems (4)
lore
Share & tools
lore

Miguel Ojeda

author maintainer
#0 lore raw

PATCH 1/2 kbuild: mark `rustc` (and others) invocations as recursive

rustc (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call rustc with -Ccodegen-units=1 (and -Zthreads is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by rustc when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and --print sysroot do, but --version does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), rustc will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style).

One solution is to remove the jobserver from MAKEFLAGS. However, we can mark the lines with calls to rustc (and Cargo) as recursive, which looks simpler. This is what is now documented and recommended by rustc and allows us to be ready for the time we may use parallelism inside rustc (potentially now, if a user passes -Zthreads). Thus do so.

Similarly, do the same for rustdoc and cargo calls.

Finally, there is one case that the solution does not cover, which is the $(shell ...) call we have. Thus, for that one, set an empty MAKEFLAGS environment variable.

---
 Makefile               |  4 ++--
 rust/Makefile          | 48 +++++++++++++++++++++---------------------
 scripts/Makefile.build |  8 +++----
 scripts/Makefile.host  |  2 +-
 4 files changed, 31 insertions(+), 31 deletions(-)
Makefile +2 -2
index 9869f57c3fb3..cbcdd8d9d0e3 100644
@@ -1197,7 +1197,7 @@ prepare0: archprepare
 # All the preparing..
 prepare: prepare0
 ifdef CONFIG_RUST
-	$(Q)$(CONFIG_SHELL) $(srctree)/scripts/rust_is_available.sh
+	+$(Q)$(CONFIG_SHELL) $(srctree)/scripts/rust_is_available.sh
 	$(Q)$(MAKE) $(build)=rust
 endif
 
@@ -1707,7 +1707,7 @@ $(DOC_TARGETS):
 # "Is Rust available?" target
 PHONY += rustavailable
 rustavailable:
-	$(Q)$(CONFIG_SHELL) $(srctree)/scripts/rust_is_available.sh && echo "Rust is available!"
+	+$(Q)$(CONFIG_SHELL) $(srctree)/scripts/rust_is_available.sh && echo "Rust is available!"
 
 # Documentation target
 #
rust/Makefile +24 -24

148 lines not on this page · raw message

scripts/Makefile.build +4 -4
index dae447a1ad30..0fb7a785594c 100644
@@ -290,7 +290,7 @@ quiet_cmd_rustc_o_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
       cmd_rustc_o_rs = $(rust_common_cmd) --emit=obj=$@ $<
 
 $(obj)/%.o: $(src)/%.rs FORCE
-	$(call if_changed_dep,rustc_o_rs)
+	+$(call if_changed_dep,rustc_o_rs)
 
 quiet_cmd_rustc_rsi_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
       cmd_rustc_rsi_rs = \
@@ -298,19 +298,19 @@ quiet_cmd_rustc_rsi_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
 	command -v $(RUSTFMT) >/dev/null && $(RUSTFMT) $@
 
 $(obj)/%.rsi: $(src)/%.rs FORCE
-	$(call if_changed_dep,rustc_rsi_rs)
+	+$(call if_changed_dep,rustc_rsi_rs)
 
 quiet_cmd_rustc_s_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
       cmd_rustc_s_rs = $(rust_common_cmd) --emit=asm=$@ $<
 
 $(obj)/%.s: $(src)/%.rs FORCE
-	$(call if_changed_dep,rustc_s_rs)
+	+$(call if_changed_dep,rustc_s_rs)
 
 quiet_cmd_rustc_ll_rs = $(RUSTC_OR_CLIPPY_QUIET) $(quiet_modtag) $@
       cmd_rustc_ll_rs = $(rust_common_cmd) --emit=llvm-ir=$@ $<
 
 $(obj)/%.ll: $(src)/%.rs FORCE
-	$(call if_changed_dep,rustc_ll_rs)
+	+$(call if_changed_dep,rustc_ll_rs)
 
 # Compile assembler sources (.S)
 # ---------------------------------------------------------------------------
scripts/Makefile.host +1 -1
index 08d83d9db31a..3c17e6ba421c 100644
@@ -156,7 +156,7 @@ quiet_cmd_host-rust	= HOSTRUSTC $@
       cmd_host-rust	= \
 	$(HOSTRUSTC) $(hostrust_flags) --emit=link=$@ $<
 $(host-rust): $(obj)/%: $(src)/%.rs FORCE
-	$(call if_changed_dep,host-rust)
+	+$(call if_changed_dep,host-rust)
 
 targets += $(host-csingle) $(host-cmulti) $(host-cobjs) \
 	   $(host-cxxmulti) $(host-cxxobjs) $(host-rust)

base-commit: f090f0d0eea9666a96702b29bc9a64cbabee85c5 --
2.43.0

author maintainer
#1 lore raw

PATCH 2/2 rust: upgrade to Rust 1.76.0

This is the next upgrade to the Rust toolchain, from 1.75.0 to 1.76.0 (i.e. the latest) [1].

See the upgrade policy [2] and the comments on the first upgrade in commit 3ed03f4da06e ("rust: upgrade to Rust 1.68.2").

# Unstable features

No unstable features that we use were stabilized in Rust 1.76.0.

The only unstable features allowed to be used outside the kernel crate are still new_uninit,offset_of, though other code to be upstreamed may increase the list.

Please see [3] for details.

# Required changes

rustc (and others) now warns when it cannot connect to the Make jobserver, thus mark those invocations as recursive as needed. Please see the previous commit for details.

# Other changes

Rust 1.76.0 does not emit the .debug_pub{names,types} sections anymore for DWARFv4 [4][5]. For instance, in the uncompressed debug info case, this debug information took:

samples/rust/rust_minimal.o ~64 KiB (~18% of total object size)
rust/kernel.o ~92 KiB (~15%)
rust/core.o ~114 KiB ( ~5%)

In the compressed debug info (zlib) case:

    samples/rust/rust_minimal.o   ~11 KiB (~6%)
    rust/kernel.o                 ~17 KiB (~5%)
    rust/core.o                   ~21 KiB (~1.5%)

In addition, the rustc_codegen_gcc backend now does not emit the .eh_frame section when compiling under -Cpanic=abort [6], thus removing the need for the patch in the CI to compile the kernel [7]. Moreover, it also now emits the .comment section too [6].

# alloc upgrade and reviewing

The vast majority of changes are due to our alloc fork being upgraded at once.

There are two kinds of changes to be aware of: the ones coming from upstream, which we should follow as closely as possible, and the updates needed in our added fallible APIs to keep them matching the newer infallible APIs coming from upstream.

Instead of taking a look at the diff of this patch, an alternative approach is reviewing a diff of the changes between upstream alloc and the kernel's. This allows to easily inspect the kernel additions only, especially to check if the fallible methods we already have still match the infallible ones in the new version coming from upstream.

Another approach is reviewing the changes introduced in the additions in the kernel fork between the two versions. This is useful to spot potentially unintended changes to our additions.

To apply these approaches, one may follow steps similar to the following to generate a pair of patches that show the differences between upstream Rust and the kernel (for the subset of alloc we use) before and after applying this patch:

# Get the difference with respect to the old version.
git -C rust checkout $(linux/scripts/min-tool-version.sh rustc) git -C linux ls-tree -r --name-only HEAD -- rust/alloc |
cut -d/ -f3- |
grep -Fv README.md |
xargs -IPATH cp rust/library/alloc/src/PATH linux/rust/alloc/PATH
git -C linux diff --patch-with-stat --summary -R > old.patch git -C linux restore rust/alloc

# Apply this patch.
git -C linux am rust-upgrade.patch

# Get the difference with respect to the new version.
git -C rust checkout $(linux/scripts/min-tool-version.sh rustc) git -C linux ls-tree -r --name-only HEAD -- rust/alloc |
cut -d/ -f3- |
grep -Fv README.md |
xargs -IPATH cp rust/library/alloc/src/PATH linux/rust/alloc/PATH
git -C linux diff --patch-with-stat --summary -R > new.patch git -C linux restore rust/alloc

Now one may check the new.patch to take a look at the additions (first approach) or at the difference between those two patches (second approach). For the latter, a side-by-side tool is recommended.

---
 Documentation/process/changes.rst |  2 +-
 rust/alloc/alloc.rs               |  3 ++
 rust/alloc/boxed.rs               | 14 +++++--
 rust/alloc/collections/mod.rs     |  1 +
 rust/alloc/lib.rs                 |  8 ++--
 rust/alloc/raw_vec.rs             | 58 +++++++++++++++++++--------
 rust/alloc/vec/into_iter.rs       | 16 +++++---
 rust/alloc/vec/mod.rs             | 65 ++++++++++++++++++++++++-------
 scripts/min-tool-version.sh       |  2 +-
 9 files changed, 125 insertions(+), 44 deletions(-)
Documentation/process/changes.rst +1 -1
index eab7e2f8c196..c78ecc1e176f 100644
@@ -31,7 +31,7 @@ you probably needn't concern yourself with pcmciautils.
 ====================== ===============  ========================================
 GNU C                  5.1              gcc --version
 Clang/LLVM (optional)  11.0.0           clang --version
-Rust (optional)        1.75.0           rustc --version
+Rust (optional)        1.76.0           rustc --version
 bindgen (optional)     0.65.1           bindgen --version
 GNU make               3.82             make --version
 bash                   4.2              bash --version
rust/alloc/alloc.rs +3 -0
index 8a6be8c98173..abb791cc2371 100644
@@ -425,12 +425,14 @@ pub unsafe fn __rdl_oom(size: usize, _align: usize) -> ! {
     }
 }
 
+#[cfg(not(no_global_oom_handling))]
 /// Specialize clones into pre-allocated, uninitialized memory.
 /// Used by `Box::clone` and `Rc`/`Arc::make_mut`.
 pub(crate) trait WriteCloneIntoRaw: Sized {
     unsafe fn write_clone_into_raw(&self, target: *mut Self);
 }
 
+#[cfg(not(no_global_oom_handling))]
 impl<T: Clone> WriteCloneIntoRaw for T {
     #[inline]
     default unsafe fn write_clone_into_raw(&self, target: *mut Self) {
@@ -440,6 +442,7 @@ impl<T: Clone> WriteCloneIntoRaw for T {
     }
 }
 
+#[cfg(not(no_global_oom_handling))]
 impl<T: Copy> WriteCloneIntoRaw for T {
     #[inline]
     unsafe fn write_clone_into_raw(&self, target: *mut Self) {
rust/alloc/boxed.rs +11 -3
index f5f40778a193..c93a22a5c97f 100644
@@ -1042,10 +1042,18 @@ impl<T: ?Sized, A: Allocator> Box<T, A> {
     /// use std::ptr;
     ///
     /// let x = Box::new(String::from("Hello"));
-    /// let p = Box::into_raw(x);
+    /// let ptr = Box::into_raw(x);
     /// unsafe {
-    ///     ptr::drop_in_place(p);
-    ///     dealloc(p as *mut u8, Layout::new::<String>());
+    ///     ptr::drop_in_place(ptr);
+    ///     dealloc(ptr as *mut u8, Layout::new::<String>());
+    /// }
+    /// ```
+    /// Note: This is equivalent to the following:
+    /// ```
+    /// let x = Box::new(String::from("Hello"));
+    /// let ptr = Box::into_raw(x);
+    /// unsafe {
+    ///     drop(Box::from_raw(ptr));
     /// }
     /// ```
     ///
rust/alloc/collections/mod.rs +1 -0
index 2506065d158a..00ffb3b97365 100644
@@ -150,6 +150,7 @@ fn fmt(
 
 /// An intermediate trait for specialization of `Extend`.
 #[doc(hidden)]
+#[cfg(not(no_global_oom_handling))]
 trait SpecExtend<I: IntoIterator> {
     /// Extends `self` with the contents of the given iterator.
     fn spec_extend(&mut self, iter: I);
rust/alloc/lib.rs +4 -4
index 345cf5c9cf92..36f79c075593 100644
@@ -80,8 +80,8 @@
     not(no_sync),
     target_has_atomic = "ptr"
 ))]
-#![cfg_attr(not(bootstrap), doc(rust_logo))]
-#![cfg_attr(not(bootstrap), feature(rustdoc_internals))]
+#![doc(rust_logo)]
+#![feature(rustdoc_internals)]
 #![no_std]
 #![needs_allocator]
 // Lints:
@@ -142,7 +142,6 @@
 #![feature(maybe_uninit_uninit_array)]
 #![feature(maybe_uninit_uninit_array_transpose)]
 #![feature(pattern)]
-#![feature(ptr_addr_eq)]
 #![feature(ptr_internals)]
 #![feature(ptr_metadata)]
 #![feature(ptr_sub_ptr)]
@@ -157,6 +156,7 @@
 #![feature(std_internals)]
 #![feature(str_internals)]
 #![feature(strict_provenance)]
+#![feature(trusted_fused)]
 #![feature(trusted_len)]
 #![feature(trusted_random_access)]
 #![feature(try_trait_v2)]
@@ -277,7 +277,7 @@ pub(crate) mod test_helpers {
     /// seed not being the same for every RNG invocation too.
     pub(crate) fn test_rng() -> rand_xorshift::XorShiftRng {
         use std::hash::{BuildHasher, Hash, Hasher};
-        let mut hasher = std::collections::hash_map::RandomState::new().build_hasher();
+        let mut hasher = std::hash::RandomState::new().build_hasher();
         std::panic::Location::caller().hash(&mut hasher);
         let hc64 = hasher.finish();
         let seed_vec =
rust/alloc/raw_vec.rs +42 -16

156 lines not on this page · raw message

rust/alloc/vec/into_iter.rs +11 -5
index aac0ec16aef1..136bfe94af6c 100644
@@ -9,7 +9,8 @@
 use core::array;
 use core::fmt;
 use core::iter::{
-    FusedIterator, InPlaceIterable, SourceIter, TrustedLen, TrustedRandomAccessNoCoerce,
+    FusedIterator, InPlaceIterable, SourceIter, TrustedFused, TrustedLen,
+    TrustedRandomAccessNoCoerce,
 };
 use core::marker::PhantomData;
 use core::mem::{self, ManuallyDrop, MaybeUninit, SizedTypeProperties};
@@ -287,9 +288,7 @@ unsafe fn __iterator_get_unchecked(&mut self, i: usize) -> Self::Item
         // Also note the implementation of `Self: TrustedRandomAccess` requires
         // that `T: Copy` so reading elements from the buffer doesn't invalidate
         // them for `Drop`.
-        unsafe {
-            if T::IS_ZST { mem::zeroed() } else { ptr::read(self.ptr.add(i)) }
-        }
+        unsafe { if T::IS_ZST { mem::zeroed() } else { ptr::read(self.ptr.add(i)) } }
     }
 }
 
@@ -341,6 +340,10 @@ fn is_empty(&self) -> bool {
 #[stable(feature = "fused", since = "1.26.0")]
 impl<T, A: Allocator> FusedIterator for IntoIter<T, A> {}
 
+#[doc(hidden)]
+#[unstable(issue = "none", feature = "trusted_fused")]
+unsafe impl<T, A: Allocator> TrustedFused for IntoIter<T, A> {}
+
 #[unstable(feature = "trusted_len", issue = "37572")]
 unsafe impl<T, A: Allocator> TrustedLen for IntoIter<T, A> {}
 
@@ -425,7 +428,10 @@ fn drop(&mut self) {
 // also refer to the vec::in_place_collect module documentation to get an overview
 #[unstable(issue = "none", feature = "inplace_iteration")]
 #[doc(hidden)]
-unsafe impl<T, A: Allocator> InPlaceIterable for IntoIter<T, A> {}
+unsafe impl<T, A: Allocator> InPlaceIterable for IntoIter<T, A> {
+    const EXPAND_BY: Option<NonZeroUsize> = NonZeroUsize::new(1);
+    const MERGE_BY: Option<NonZeroUsize> = NonZeroUsize::new(1);
+}
 
 #[unstable(issue = "none", feature = "inplace_iteration")]
 #[doc(hidden)]
rust/alloc/vec/mod.rs +51 -14

127 lines not on this page · raw message

scripts/min-tool-version.sh +1 -1
index ef6f286a4d47..e217683b10d6 100755
@@ -33,7 +33,7 @@ llvm)
 	fi
 	;;
 rustc)
-	echo 1.75.0
+	echo 1.76.0
 	;;
 bindgen)
 	echo 0.65.1

--
2.43.0

Tested-by on v1 2/2
#2 lore raw

Re: [PATCH 2/2] rust: upgrade to Rust 1.76.0

Quoted 3 lines · Miguel Ojeda

This is the next upgrade to the Rust toolchain, from 1.75.0 to 1.76.0 (i.e. the latest) [1].

I've applied this on the merge point of for-next/rust-for-arm64 in arm64 tree and today's rust-next (patch #1 has a small conflict with the target.json changes in arm64 tree), tested with kunit for both x86 and arm64.

Regards,
Boqun

Quoted 3 lines

See the upgrade policy [2] and the comments on the first upgrade in commit 3ed03f4da06e ("rust: upgrade to Rust 1.68.2").

[...]

reviewer Reviewed-by on v1 2/2
#3 lore raw

Re: [PATCH 2/2] rust: upgrade to Rust 1.76.0

In reply to Miguel Ojeda
Quoted 45 lines · Miguel Ojeda

This is the next upgrade to the Rust toolchain, from 1.75.0 to 1.76.0 (i.e. the latest) [1].

See the upgrade policy [2] and the comments on the first upgrade in commit 3ed03f4da06e ("rust: upgrade to Rust 1.68.2").

# Unstable features

No unstable features that we use were stabilized in Rust 1.76.0.

The only unstable features allowed to be used outside the kernel crate are still new_uninit,offset_of, though other code to be upstreamed may increase the list.

Please see [3] for details.

# Required changes

rustc (and others) now warns when it cannot connect to the Make jobserver, thus mark those invocations as recursive as needed. Please see the previous commit for details.

# Other changes

Rust 1.76.0 does not emit the .debug_pub{names,types} sections anymore for DWARFv4 [4][5]. For instance, in the uncompressed debug info case, this debug information took:

samples/rust/rust_minimal.o ~64 KiB (~18% of total object size)
rust/kernel.o ~92 KiB (~15%)
rust/core.o ~114 KiB ( ~5%)

In the compressed debug info (zlib) case:

    samples/rust/rust_minimal.o   ~11 KiB (~6%)
    rust/kernel.o                 ~17 KiB (~5%)
    rust/core.o                   ~21 KiB (~1.5%)

In addition, the rustc_codegen_gcc backend now does not emit the .eh_frame section when compiling under -Cpanic=abort [6], thus removing the need for the patch in the CI to compile the kernel [7]. Moreover, it also now emits the .comment section too [6].

# alloc upgrade and reviewing

[...]

Quoted 8 lines
Acked-by on v1 1/2
#4 lore raw
In reply to Miguel Ojeda
Quoted 18 lines · Miguel Ojeda

rustc (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call rustc with -Ccodegen-units=1 (and -Zthreads is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by rustc when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and --print sysroot do, but --version does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), rustc will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style).

One solution is to remove the jobserver from MAKEFLAGS. However, we can mark the lines with calls to rustc (and Cargo) as recursive, which looks simpler. This is what is now documented and recommended by rustc and allows us to be ready for the time we may use parallelism inside rustc (potentially now, if a user passes -Zthreads). Thus do so.

Quoted 8 lines

Similarly, do the same for rustdoc and cargo calls.

Finally, there is one case that the solution does not cover, which is the $(shell ...) call we have. Thus, for that one, set an empty MAKEFLAGS environment variable.

Signature · 2 lines
Best Regards
Masahiro Yamada
#5 lore raw
In reply to Miguel Ojeda
Quoted 18 lines · Miguel Ojeda

rustc (like Cargo) may take advantage of the jobserver at any time (e.g. for backend parallelism, or eventually frontend too). In the kernel, we call rustc with -Ccodegen-units=1 (and -Zthreads is 1 so far), so we do not expect parallelism. However, in the upcoming Rust 1.76.0, a warning is emitted by rustc when it cannot connect to the jobserver it was passed (in many cases, but not all: compiling and --print sysroot do, but --version does not). And given GNU Make always passes the jobserver in the environment variable (even when a line is deemed non-recursive), rustc will end up complaining about it (in particular in Make 4.3 where there is only the simple pipe jobserver style).

One solution is to remove the jobserver from MAKEFLAGS. However, we can mark the lines with calls to rustc (and Cargo) as recursive, which looks simpler. This is what is now documented and recommended by rustc and allows us to be ready for the time we may use parallelism inside rustc (potentially now, if a user passes -Zthreads). Thus do so.

Quoted 8 lines

Similarly, do the same for rustdoc and cargo calls.

Finally, there is one case that the solution does not cover, which is the $(shell ...) call we have. Thus, for that one, set an empty MAKEFLAGS environment variable.

Applied (i.e. including the upgrade to 1.76.0) to rust-next -- thanks everyone!

[ Reworded to add link to PR documenting the recommendation. ]

Cheers,
Miguel