Skip to contentSkip to conversation

Allocation APIs

PATCH Applied Wedson Almeida Filho 6 participants 31 messages Mar 28, 2024 – May 1, 2024

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

v3 cover Allocation APIs

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

---

Changes in v3:
- Rebased on top of the latest rust-next branch.
- Updated krealloc_aligned to use Flags instead of bindings::gfp_t.
- Added __GFP_ZERO to flags, as part of the previous change.
- Avoiding temporary stack value in Box::new_uninit.
- Implement Box::new using Box::new_uninit (so only one of them actually allocates).
- Added examples/tests to VecExt methods.
- Fixed bug in length in extend_from_slice
- Link to v2: https://lore.kernel.org/rust-for-linux/[email protected]/T/#t

Changes in v2:
- Updated description of alloc crate.
- Renamed vecext and boxext modules to vec_ext and box_ext.
- Added derive directive to AllocError.
- Updated safety comment in BoxExt::new.
- Updated VecExt::push and VecExt::extend_from_slice to use
spare_capacity_mut
- Added directive to not compile destructure and rebuild when test or testlib are configured. Otherwise we have a warning because push and extend_from_slice don't use them anymore.
- Updated indentation in Arc::new_uninit
- Moved the removal of TryReserveError convesion to Error to patch 7, where usage of TryReserveError is actually removed.
- Link to v1: https://lore.kernel.org/rust-for-linux/[email protected]/T/#t

Wedson Almeida Filho (10):
rust: kernel: move allocator module under alloc
rust: alloc: introduce the VecExt trait
kbuild: use the upstream alloc crate
rust: alloc: remove our fork of the alloc crate
rust: alloc: introduce allocation flags
rust: alloc: introduce the BoxExt trait
rust: alloc: update VecExt to take allocation flags
rust: sync: update Arc and UniqueArc to take allocation flags
rust: init: update init module to take allocation flags
rust: kernel: remove usage of allocator_api unstable feature

 rust/Makefile                        |   16 +-
 rust/alloc/README.md                 |   36 -
 rust/alloc/alloc.rs                  |  452 ----
 rust/alloc/boxed.rs                  | 2463 -----------------
 rust/alloc/collections/mod.rs        |  160 --
 rust/alloc/lib.rs                    |  288 --
 rust/alloc/raw_vec.rs                |  611 -----
 rust/alloc/slice.rs                  |  890 -------
 rust/alloc/vec/drain.rs              |  255 --
 rust/alloc/vec/extract_if.rs         |  115 -
 rust/alloc/vec/into_iter.rs          |  454 ----
 rust/alloc/vec/is_zero.rs            |  204 --
 rust/alloc/vec/mod.rs                | 3683 --------------------------
 rust/alloc/vec/partial_eq.rs         |   49 -
 rust/alloc/vec/set_len_on_drop.rs    |   35 -
 rust/alloc/vec/spec_extend.rs        |  119 -
 rust/bindings/bindings_helper.h      |    3 +
 rust/kernel/alloc.rs                 |   74 +
 rust/kernel/{ => alloc}/allocator.rs |   17 +-
 rust/kernel/alloc/box_ext.rs         |   59 +
 rust/kernel/alloc/vec_ext.rs         |  176 ++
 rust/kernel/error.rs                 |   13 +-
 rust/kernel/init.rs                  |   57 +-
 rust/kernel/lib.rs                   |    5 +-
 rust/kernel/prelude.rs               |    2 +
 rust/kernel/str.rs                   |    6 +-
 rust/kernel/sync/arc.rs              |   50 +-
 rust/kernel/sync/condvar.rs          |    2 +-
 rust/kernel/sync/lock/mutex.rs       |    2 +-
 rust/kernel/sync/lock/spinlock.rs    |    2 +-
 rust/kernel/types.rs                 |    4 +-
 rust/kernel/workqueue.rs             |   14 +-
 samples/rust/rust_minimal.rs         |    6 +-
 samples/rust/rust_print.rs           |    4 +-
 scripts/generate_rust_analyzer.py    |    2 +-
 35 files changed, 405 insertions(+), 9923 deletions(-)
 delete mode 100644 rust/alloc/README.md
 delete mode 100644 rust/alloc/alloc.rs
 delete mode 100644 rust/alloc/boxed.rs
 delete mode 100644 rust/alloc/collections/mod.rs
 delete mode 100644 rust/alloc/lib.rs
 delete mode 100644 rust/alloc/raw_vec.rs
 delete mode 100644 rust/alloc/slice.rs
 delete mode 100644 rust/alloc/vec/drain.rs
 delete mode 100644 rust/alloc/vec/extract_if.rs
 delete mode 100644 rust/alloc/vec/into_iter.rs
 delete mode 100644 rust/alloc/vec/is_zero.rs
 delete mode 100644 rust/alloc/vec/mod.rs
 delete mode 100644 rust/alloc/vec/partial_eq.rs
 delete mode 100644 rust/alloc/vec/set_len_on_drop.rs
 delete mode 100644 rust/alloc/vec/spec_extend.rs
 create mode 100644 rust/kernel/alloc.rs
 rename rust/kernel/{ => alloc}/allocator.rs (86%)
 create mode 100644 rust/kernel/alloc/box_ext.rs
 create mode 100644 rust/kernel/alloc/vec_ext.rs

base-commit: 768409cff6cc89fe1194da880537a09857b6e4db

Signature · 1 line
2.34.1
#1 lore raw

PATCH v3 1/10 rust: kernel: move `allocator` module under `alloc`

From: Wedson Almeida Filho <[email protected]>

We will add more to the alloc module in subsequent patches (e.g., allocation flags and extension traits).

---
 rust/kernel/alloc.rs                 | 7 +++++++
 rust/kernel/{ => alloc}/allocator.rs | 0
 rust/kernel/lib.rs                   | 4 +---
 3 files changed, 8 insertions(+), 3 deletions(-)
 create mode 100644 rust/kernel/alloc.rs
 rename rust/kernel/{ => alloc}/allocator.rs (100%)
rust/kernel/alloc.rs new file +7 -0
new file mode 100644
index 000000000000..9aa5077d4a4c
@@ -0,0 +1,7 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Extensions to the [`alloc`] crate.
+
+#[cfg(not(test))]
+#[cfg(not(testlib))]
+mod allocator;
rust/kernel/alloc/allocator.rs renamed from rust/kernel/allocator.rs +0 -0
similarity index 100%
rename from rust/kernel/allocator.rs
rename to rust/kernel/alloc/allocator.rs
rust/kernel/lib.rs +1 -3
index be68d5e567b1..51f30e55bd00 100644
@@ -28,9 +28,7 @@
 // Allow proc-macros to refer to `::kernel` inside the `kernel` crate (this crate).
 extern crate self as kernel;
 
-#[cfg(not(test))]
-#[cfg(not(testlib))]
-mod allocator;
+pub mod alloc;
 mod build_assert;
 pub mod error;
 pub mod init;
Signature · 1 line
2.34.1
#2 lore raw

PATCH v3 2/10 rust: alloc: introduce the `VecExt` trait

In reply to Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Make try_with_capacity, try_push, and try_extend_from_slice methods available in Vec even though it doesn't implement them. It is implemented with try_reserve and push_within_capacity.

This is in preparation for switching to the upstream alloc crate.

---
 rust/kernel/alloc.rs         |  1 +
 rust/kernel/alloc/vec_ext.rs | 48 ++++++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs           |  1 +
 rust/kernel/prelude.rs       |  2 ++
 4 files changed, 52 insertions(+)
 create mode 100644 rust/kernel/alloc/vec_ext.rs
rust/kernel/alloc.rs +1 -0
index 9aa5077d4a4c..2c99635f9fd3 100644
@@ -5,3 +5,4 @@
 #[cfg(not(test))]
 #[cfg(not(testlib))]
 mod allocator;
+pub mod vec_ext;
rust/kernel/alloc/vec_ext.rs new file +48 -0

48 lines not on this page · raw message

rust/kernel/lib.rs +1 -0
index 51f30e55bd00..7f2841a18d05 100644
@@ -19,6 +19,7 @@
 #![feature(offset_of)]
 #![feature(receiver_trait)]
 #![feature(unsize)]
+#![feature(vec_push_within_capacity)]
 
 // Ensure conditional compilation based on the kernel configuration works;
 // otherwise we may silently break things like initcall handling.
rust/kernel/prelude.rs +2 -0
index ae21600970b3..c85b5972c0d3 100644
@@ -14,6 +14,8 @@
 #[doc(no_inline)]
 pub use core::pin::Pin;
 
+pub use crate::alloc::vec_ext::VecExt;
+
 #[doc(no_inline)]
 pub use alloc::{boxed::Box, vec::Vec};
 
Signature · 1 line
2.34.1
#3 lore raw

PATCH v3 3/10 kbuild: use the upstream `alloc` crate

In reply to Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Switch away from our fork of the alloc crate. We remove it altogether in the next commit.

---
 rust/Makefile                     | 16 +++-------------
 rust/kernel/str.rs                |  1 +
 scripts/generate_rust_analyzer.py |  2 +-
 3 files changed, 5 insertions(+), 14 deletions(-)
rust/Makefile +3 -13
index 846e6ab9d5a9..ca86abac4ed0 100644
@@ -61,15 +61,9 @@ core-cfgs = \
     --cfg no_fp_fmt_parse
 
 alloc-cfgs = \
-    --cfg no_borrow \
-    --cfg no_fmt \
     --cfg no_global_oom_handling \
-    --cfg no_macros \
     --cfg no_rc \
-    --cfg no_str \
-    --cfg no_string \
-    --cfg no_sync \
-    --cfg no_thin
+    --cfg no_sync
 
 quiet_cmd_rustdoc = RUSTDOC $(if $(rustdoc_host),H, ) $<
       cmd_rustdoc = \
@@ -123,7 +117,7 @@ rustdoc-compiler_builtins: $(src)/compiler_builtins.rs rustdoc-core FORCE
 # due to things that are "configured out" vs. entirely non-existing ones.
 rustdoc-alloc: private rustc_target_flags = $(alloc-cfgs) \
     -Arustdoc::broken_intra_doc_links
-rustdoc-alloc: $(src)/alloc/lib.rs rustdoc-core rustdoc-compiler_builtins FORCE
+rustdoc-alloc: $(RUST_LIB_SRC)/alloc/src/lib.rs rustdoc-core rustdoc-compiler_builtins FORCE
 	+$(call if_changed,rustdoc)
 
 rustdoc-kernel: private rustc_target_flags = --extern alloc \
@@ -219,8 +213,6 @@ rusttest: rusttest-macros rusttest-kernel
 #   - `cargo` only considers the use case of building the standard library
 #     to use it in a given package. Thus we need to create a dummy package
 #     and pick the generated libraries from there.
-#   - Since we only keep a subset of upstream `alloc` in-tree, we need
-#     to recreate it on the fly by putting our sources on top.
 #   - The usual ways of modifying the dependency graph in `cargo` do not seem
 #     to apply for the `-Zbuild-std` steps, thus we have to mislead it
 #     by modifying the sources in the sysroot.
@@ -239,8 +231,6 @@ quiet_cmd_rustsysroot = RUSTSYSROOT
 	rm -rf $(objtree)/$(obj)/test; \
 	mkdir -p $(objtree)/$(obj)/test; \
 	cp -a $(rustc_sysroot) $(objtree)/$(obj)/test/sysroot; \
-	cp -r $(srctree)/$(src)/alloc/* \
-		$(objtree)/$(obj)/test/sysroot/lib/rustlib/src/rust/library/alloc/src; \
 	echo '\#!/bin/sh' > $(objtree)/$(obj)/test/rustc_sysroot; \
 	echo "$(RUSTC) --sysroot=$(abspath $(objtree)/$(obj)/test/sysroot) \"\$$@\"" \
 		>> $(objtree)/$(obj)/test/rustc_sysroot; \
@@ -448,7 +438,7 @@ $(obj)/compiler_builtins.o: $(src)/compiler_builtins.rs $(obj)/core.o FORCE
 $(obj)/alloc.o: private skip_clippy = 1
 $(obj)/alloc.o: private skip_flags = -Dunreachable_pub
 $(obj)/alloc.o: private rustc_target_flags = $(alloc-cfgs)
-$(obj)/alloc.o: $(src)/alloc/lib.rs $(obj)/compiler_builtins.o FORCE
+$(obj)/alloc.o: $(RUST_LIB_SRC)/alloc/src/lib.rs $(obj)/compiler_builtins.o FORCE
 	+$(call if_changed_dep,rustc_library)
 
 $(obj)/build_error.o: $(src)/build_error.rs $(obj)/compiler_builtins.o FORCE
rust/kernel/str.rs +1 -0
index 925ced8fdc61..e1be6c5e72a9 100644
@@ -2,6 +2,7 @@
 
 //! String representations.
 
+use crate::alloc::vec_ext::VecExt;
 use alloc::alloc::AllocError;
 use alloc::vec::Vec;
 use core::fmt::{self, Write};
scripts/generate_rust_analyzer.py +1 -1
index fc52bc41d3e7..f270c7b0cf34 100755
@@ -66,7 +66,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
 
     append_crate(
         "alloc",
-        srctree / "rust" / "alloc" / "lib.rs",
+        sysroot_src / "alloc" / "src" / "lib.rs",
         ["core", "compiler_builtins"],
         cfg=crates_cfgs.get("alloc", []),
     )
Signature · 1 line
2.34.1
#4 lore raw

PATCH v3 4/10 rust: alloc: remove our fork of the `alloc` crate

In reply to Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

It is not used anymore as VecExt now provides the functionality we depend on.

---
 rust/alloc/README.md              |   36 -
 rust/alloc/alloc.rs               |  452 ----
 rust/alloc/boxed.rs               | 2463 -------------------
 rust/alloc/collections/mod.rs     |  160 --
 rust/alloc/lib.rs                 |  288 ---
 rust/alloc/raw_vec.rs             |  611 -----
 rust/alloc/slice.rs               |  890 -------
 rust/alloc/vec/drain.rs           |  255 --
 rust/alloc/vec/extract_if.rs      |  115 -
 rust/alloc/vec/into_iter.rs       |  454 ----
 rust/alloc/vec/is_zero.rs         |  204 --
 rust/alloc/vec/mod.rs             | 3683 -----------------------------
 rust/alloc/vec/partial_eq.rs      |   49 -
 rust/alloc/vec/set_len_on_drop.rs |   35 -
 rust/alloc/vec/spec_extend.rs     |  119 -
 15 files changed, 9814 deletions(-)
 delete mode 100644 rust/alloc/README.md
 delete mode 100644 rust/alloc/alloc.rs
 delete mode 100644 rust/alloc/boxed.rs
 delete mode 100644 rust/alloc/collections/mod.rs
 delete mode 100644 rust/alloc/lib.rs
 delete mode 100644 rust/alloc/raw_vec.rs
 delete mode 100644 rust/alloc/slice.rs
 delete mode 100644 rust/alloc/vec/drain.rs
 delete mode 100644 rust/alloc/vec/extract_if.rs
 delete mode 100644 rust/alloc/vec/into_iter.rs
 delete mode 100644 rust/alloc/vec/is_zero.rs
 delete mode 100644 rust/alloc/vec/mod.rs
 delete mode 100644 rust/alloc/vec/partial_eq.rs
 delete mode 100644 rust/alloc/vec/set_len_on_drop.rs
 delete mode 100644 rust/alloc/vec/spec_extend.rs
rust/alloc/README.md deleted +0 -36
deleted file mode 100644
index eb6f22e94ebf..000000000000
@@ -1,36 +0,0 @@
-# `alloc`
-
-These source files come from the Rust standard library, hosted in
-the <https://github.com/rust-lang/rust> repository, licensed under
-"Apache-2.0 OR MIT" and adapted for kernel use. For copyright details,
-see <https://github.com/rust-lang/rust/blob/master/COPYRIGHT>.
-
-Please note that these files should be kept as close as possible to
-upstream. In general, only additions should be performed (e.g. new
-methods). Eventually, changes should make it into upstream so that,
-at some point, this fork can be dropped from the kernel tree.
-
-The Rust upstream version on top of which these files are based matches
-the output of `scripts/min-tool-version.sh rustc`.
-
-
-## Rationale
-
-On one hand, kernel folks wanted to keep `alloc` in-tree to have more
-freedom in both workflow and actual features if actually needed
-(e.g. receiver types if we ended up using them), which is reasonable.
-
-On the other hand, Rust folks wanted to keep `alloc` as close as
-upstream as possible and avoid as much divergence as possible, which
-is also reasonable.
-
-We agreed on a middle-ground: we would keep a subset of `alloc`
-in-tree that would be as small and as close as possible to upstream.
-Then, upstream can start adding the functions that we add to `alloc`
-etc., until we reach a point where the kernel already knows exactly
-what it needs in `alloc` and all the new methods are merged into
-upstream, so that we can drop `alloc` from the kernel tree and go back
-to using the upstream one.
-
-By doing this, the kernel can go a bit faster now, and Rust can
-slowly incorporate and discuss the changes as needed.
rust/alloc/alloc.rs deleted +0 -452

452 lines not on this page · raw message

rust/alloc/boxed.rs deleted +0 -2463

2,463 lines not on this page · raw message

rust/alloc/collections/mod.rs deleted +0 -160

49 lines not on this page · raw message

rust/alloc/lib.rs deleted +0 -288

288 lines not on this page · raw message

rust/alloc/raw_vec.rs deleted +0 -611

611 lines not on this page · raw message

rust/alloc/slice.rs deleted +0 -890

890 lines not on this page · raw message

rust/alloc/vec/drain.rs deleted +0 -255

255 lines not on this page · raw message

rust/alloc/vec/extract_if.rs deleted +0 -115

115 lines not on this page · raw message

rust/alloc/vec/into_iter.rs deleted +0 -454

454 lines not on this page · raw message

rust/alloc/vec/is_zero.rs deleted +0 -204

204 lines not on this page · raw message

rust/alloc/vec/mod.rs deleted +0 -3683

3,683 lines not on this page · raw message

rust/alloc/vec/partial_eq.rs deleted +0 -49

49 lines not on this page · raw message

rust/alloc/vec/set_len_on_drop.rs deleted +0 -35
deleted file mode 100644
index d3c7297b80ec..000000000000

Lines of this file are not shown. View the raw message.

rust/alloc/vec/spec_extend.rs deleted +0 -119

119 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.34.1

Part of the diff is not shown here. View the raw message.

#5 lore raw

PATCH v3 5/10 rust: alloc: introduce allocation flags

In reply to Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

We'll use them when allocating Box, Arc, and UniqueArc instances, as well as when allocating memory for Vec elements. These changes will come in subsequent patches.

---
 rust/bindings/bindings_helper.h |  3 ++
 rust/kernel/alloc.rs            | 61 +++++++++++++++++++++++++++++++++
 rust/kernel/alloc/allocator.rs  | 17 ++++-----
 rust/kernel/prelude.rs          |  2 +-
 4 files changed, 71 insertions(+), 12 deletions(-)
rust/bindings/bindings_helper.h +3 -0
index 65b98831b975..ddb5644d4fd9 100644
@@ -20,5 +20,8 @@
 
 /* `bindgen` gets confused at certain things. */
 const size_t RUST_CONST_HELPER_ARCH_SLAB_MINALIGN = ARCH_SLAB_MINALIGN;
+const gfp_t RUST_CONST_HELPER_GFP_ATOMIC = GFP_ATOMIC;
 const gfp_t RUST_CONST_HELPER_GFP_KERNEL = GFP_KERNEL;
+const gfp_t RUST_CONST_HELPER_GFP_KERNEL_ACCOUNT = GFP_KERNEL_ACCOUNT;
+const gfp_t RUST_CONST_HELPER_GFP_NOWAIT = GFP_NOWAIT;
 const gfp_t RUST_CONST_HELPER___GFP_ZERO = __GFP_ZERO;
rust/kernel/alloc.rs +61 -0

64 lines not on this page · raw message

rust/kernel/alloc/allocator.rs +6 -11
index 01ad139e19bc..8cc7099d6ae1 100644
@@ -2,6 +2,7 @@
 
 //! Allocator support.
 
+use super::{flags::*, Flags};
 use core::alloc::{GlobalAlloc, Layout};
 use core::ptr;
 
@@ -15,7 +16,7 @@
 ///
 /// - `ptr` can be either null or a pointer which has been allocated by this allocator.
 /// - `new_layout` must have a non-zero size.
-unsafe fn krealloc_aligned(ptr: *mut u8, new_layout: Layout, flags: bindings::gfp_t) -> *mut u8 {
+unsafe fn krealloc_aligned(ptr: *mut u8, new_layout: Layout, flags: Flags) -> *mut u8 {
     // Customized layouts from `Layout::from_size_align()` can have size < align, so pad first.
     let layout = new_layout.pad_to_align();
 
@@ -36,14 +37,14 @@ unsafe fn krealloc_aligned(ptr: *mut u8, new_layout: Layout, flags: bindings::gf
     //   function safety requirement.
     // - `size` is greater than 0 since it's either a `layout.size()` (which cannot be zero
     //   according to the function safety requirement) or a result from `next_power_of_two()`.
-    unsafe { bindings::krealloc(ptr as *const core::ffi::c_void, size, flags) as *mut u8 }
+    unsafe { bindings::krealloc(ptr as *const core::ffi::c_void, size, flags.0) as *mut u8 }
 }
 
 unsafe impl GlobalAlloc for KernelAllocator {
     unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
         // SAFETY: `ptr::null_mut()` is null and `layout` has a non-zero size by the function safety
         // requirement.
-        unsafe { krealloc_aligned(ptr::null_mut(), layout, bindings::GFP_KERNEL) }
+        unsafe { krealloc_aligned(ptr::null_mut(), layout, GFP_KERNEL) }
     }
 
     unsafe fn dealloc(&self, ptr: *mut u8, _layout: Layout) {
@@ -64,19 +65,13 @@ unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut
         //   requirement.
         // - the size of `layout` is not zero because `new_size` is not zero by the function safety
         //   requirement.
-        unsafe { krealloc_aligned(ptr, layout, bindings::GFP_KERNEL) }
+        unsafe { krealloc_aligned(ptr, layout, GFP_KERNEL) }
     }
 
     unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
         // SAFETY: `ptr::null_mut()` is null and `layout` has a non-zero size by the function safety
         // requirement.
-        unsafe {
-            krealloc_aligned(
-                ptr::null_mut(),
-                layout,
-                bindings::GFP_KERNEL | bindings::__GFP_ZERO,
-            )
-        }
+        unsafe { krealloc_aligned(ptr::null_mut(), layout, GFP_KERNEL | __GFP_ZERO) }
     }
 }
 
rust/kernel/prelude.rs +1 -1
index c85b5972c0d3..827e4dfd77df 100644
@@ -14,7 +14,7 @@
 #[doc(no_inline)]
 pub use core::pin::Pin;
 
-pub use crate::alloc::vec_ext::VecExt;
+pub use crate::alloc::{flags::*, vec_ext::VecExt};
 
 #[doc(no_inline)]
 pub use alloc::{boxed::Box, vec::Vec};
Signature · 1 line
2.34.1
#6 lore raw

PATCH v3 6/10 rust: alloc: introduce the `BoxExt` trait

In reply to Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Make fallible versions of new and new_uninit methods available in Box even though it doesn't implement them because we build alloc with the no_global_oom_handling config.

They also have an extra flags parameter that allows callers to pass flags to the allocator.

---
 rust/kernel/alloc.rs           |  1 +
 rust/kernel/alloc/allocator.rs |  2 +-
 rust/kernel/alloc/box_ext.rs   | 60 ++++++++++++++++++++++++++++++++++
 rust/kernel/init.rs            | 13 ++++----
 rust/kernel/prelude.rs         |  2 +-
 rust/kernel/sync/arc.rs        |  3 +-
 6 files changed, 72 insertions(+), 9 deletions(-)
 create mode 100644 rust/kernel/alloc/box_ext.rs
rust/kernel/alloc.rs +1 -0
index fd5541991a22..9bc1b48b5641 100644
@@ -5,6 +5,7 @@
 #[cfg(not(test))]
 #[cfg(not(testlib))]
 mod allocator;
+pub mod box_ext;
 pub mod vec_ext;
 
 /// Flags to be used when allocating memory.
rust/kernel/alloc/allocator.rs +1 -1
index 8cc7099d6ae1..ff88bce04fd4 100644
@@ -16,7 +16,7 @@
 ///
 /// - `ptr` can be either null or a pointer which has been allocated by this allocator.
 /// - `new_layout` must have a non-zero size.
-unsafe fn krealloc_aligned(ptr: *mut u8, new_layout: Layout, flags: Flags) -> *mut u8 {
+pub(crate) unsafe fn krealloc_aligned(ptr: *mut u8, new_layout: Layout, flags: Flags) -> *mut u8 {
     // Customized layouts from `Layout::from_size_align()` can have size < align, so pad first.
     let layout = new_layout.pad_to_align();
 
rust/kernel/alloc/box_ext.rs new file +60 -0

60 lines not on this page · raw message

rust/kernel/init.rs +7 -6
index 424257284d16..9c798cffc8e4 100644
@@ -210,6 +210,7 @@
 //! [`pin_init!`]: crate::pin_init!
 
 use crate::{
+    alloc::{box_ext::BoxExt, flags::*},
     error::{self, Error},
     sync::UniqueArc,
     types::{Opaque, ScopeGuard},
@@ -305,9 +306,9 @@ macro_rules! stack_pin_init {
 ///
 /// stack_try_pin_init!(let foo: Result<Pin<&mut Foo>, AllocError> = pin_init!(Foo {
 ///     a <- new_mutex!(42),
-///     b: Box::try_new(Bar {
+///     b: Box::new(Bar {
 ///         x: 64,
-///     })?,
+///     }, GFP_KERNEL)?,
 /// }));
 /// let foo = foo.unwrap();
 /// pr_info!("a: {}", &*foo.a.lock());
@@ -331,9 +332,9 @@ macro_rules! stack_pin_init {
 ///
 /// stack_try_pin_init!(let foo: Pin<&mut Foo> =? pin_init!(Foo {
 ///     a <- new_mutex!(42),
-///     b: Box::try_new(Bar {
+///     b: Box::new(Bar {
 ///         x: 64,
-///     })?,
+///     }, GFP_KERNEL)?,
 /// }));
 /// pr_info!("a: {}", &*foo.a.lock());
 /// # Ok::<_, AllocError>(())
@@ -1158,7 +1159,7 @@ fn try_pin_init<E>(init: impl PinInit<T, E>) -> Result<Pin<Self>, E>
     where
         E: From<AllocError>,
     {
-        let mut this = Box::try_new_uninit()?;
+        let mut this = <Box<_> as BoxExt<_>>::new_uninit(GFP_KERNEL)?;
         let slot = this.as_mut_ptr();
         // SAFETY: When init errors/panics, slot will get deallocated but not dropped,
         // slot is valid and will not be moved, because we pin it later.
@@ -1172,7 +1173,7 @@ fn try_init<E>(init: impl Init<T, E>) -> Result<Self, E>
     where
         E: From<AllocError>,
     {
-        let mut this = Box::try_new_uninit()?;
+        let mut this = <Box<_> as BoxExt<_>>::new_uninit(GFP_KERNEL)?;
         let slot = this.as_mut_ptr();
         // SAFETY: When init errors/panics, slot will get deallocated but not dropped,
         // slot is valid.
rust/kernel/prelude.rs +1 -1
index 827e4dfd77df..b37a0b3180fb 100644
@@ -14,7 +14,7 @@
 #[doc(no_inline)]
 pub use core::pin::Pin;
 
-pub use crate::alloc::{flags::*, vec_ext::VecExt};
+pub use crate::alloc::{box_ext::BoxExt, flags::*, vec_ext::VecExt};
 
 #[doc(no_inline)]
 pub use alloc::{boxed::Box, vec::Vec};
rust/kernel/sync/arc.rs +2 -1
index 7d4c4bf58388..1252a1b630ed 100644
@@ -16,6 +16,7 @@
 //! [`Arc`]: https://doc.rust-lang.org/std/sync/struct.Arc.html
 
 use crate::{
+    alloc::{box_ext::BoxExt, flags::*},
     bindings,
     error::{self, Error},
     init::{self, InPlaceInit, Init, PinInit},
@@ -170,7 +171,7 @@ pub fn try_new(contents: T) -> Result<Self, AllocError> {
             data: contents,
         };
 
-        let inner = Box::try_new(value)?;
+        let inner = <Box<_> as BoxExt<_>>::new(value, GFP_KERNEL)?;
 
         // SAFETY: We just created `inner` with a reference count of 1, which is owned by the new
         // `Arc` object.
Signature · 1 line
2.34.1
Reviewed-by on v3 6/10
#7 lore raw

Re: [PATCH v3 06/10] rust: alloc: introduce the `BoxExt` trait

Quoted 10 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Make fallible versions of new and new_uninit methods available in Box even though it doesn't implement them because we build alloc with the no_global_oom_handling config.

They also have an extra flags parameter that allows callers to pass flags to the allocator.

A small nit below..

Quoted 60 lines
---
 rust/kernel/alloc.rs           |  1 +
 rust/kernel/alloc/allocator.rs |  2 +-
 rust/kernel/alloc/box_ext.rs   | 60 ++++++++++++++++++++++++++++++++++
 rust/kernel/init.rs            | 13 ++++----
 rust/kernel/prelude.rs         |  2 +-
 rust/kernel/sync/arc.rs        |  3 +-
 6 files changed, 72 insertions(+), 9 deletions(-)
 create mode 100644 rust/kernel/alloc/box_ext.rs
rust/kernel/alloc.rs +1 -0
index fd5541991a22..9bc1b48b5641 100644
@@ -5,6 +5,7 @@
 #[cfg(not(test))]
 #[cfg(not(testlib))]
 mod allocator;
+pub mod box_ext;
 pub mod vec_ext;
 
 /// Flags to be used when allocating memory.
rust/kernel/alloc/allocator.rs +1 -1
index 8cc7099d6ae1..ff88bce04fd4 100644
@@ -16,7 +16,7 @@
 ///
 /// - `ptr` can be either null or a pointer which has been allocated by this allocator.
 /// - `new_layout` must have a non-zero size.
-unsafe fn krealloc_aligned(ptr: *mut u8, new_layout: Layout, flags: Flags) -> *mut u8 {
+pub(crate) unsafe fn krealloc_aligned(ptr: *mut u8, new_layout: Layout, flags: Flags) -> *mut u8 {
     // Customized layouts from `Layout::from_size_align()` can have size < align, so pad first.
     let layout = new_layout.pad_to_align();
 
rust/kernel/alloc/box_ext.rs new file +19 -0
new file mode 100644
index 000000000000..c62db0aa4dd1
@@ -0,0 +1,60 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Extensions to [`Box`] for fallible allocations.
+
+use super::Flags;
+use alloc::boxed::Box;
+use core::alloc::AllocError;
+use core::mem::MaybeUninit;
+use core::result::Result;
+
+/// Extensions to [`Box`].
+pub trait BoxExt<T>: Sized {
+    /// Allocates a new box.
+    ///
+    /// The allocation may fail, in which case an error is returned.
+    fn new(x: T, flags: Flags) -> Result<Self, AllocError>;
+
+    /// Allocates a new uninitialised box.
+    ///

rust/kernel/alloc/box_ext.rs line 29

Quoted 10 lines
rust/kernel/alloc/box_ext.rs new file +10 -0
+    /// The allocation may fail, in which case an error is returned.
+    fn new_uninit(flags: Flags) -> Result<Box<MaybeUninit<T>>, AllocError>;
+}
+
+impl<T> BoxExt<T> for Box<T> {
+    fn new(x: T, flags: Flags) -> Result<Self, AllocError> {
+        let mut b = <Self as BoxExt<_>>::new_uninit(flags)?;
+        b.write(x);
+        // SAFETY: The contents were just initialised in the line above.
+        Ok(unsafe { b.assume_init() })

, since we have the new_uninit feature selected, we can actually do

	fn new(x: T, flags: Flags) -> Result<Self, AllocError> {
	    <Self as BoxExt<_>>::new_uninit(flags).map(|b| Box::write(b, x))
	}

here.

Regards,
Boqun

Quoted 2 lines
rust/kernel/alloc/box_ext.rs new file +2 -0
+    }
+

[...]

#8 lore raw

Re: [PATCH v3 06/10] rust: alloc: introduce the `BoxExt` trait

Quoted 90 lines · Boqun Feng
Quoted 10 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Make fallible versions of new and new_uninit methods available in Box even though it doesn't implement them because we build alloc with the no_global_oom_handling config.

They also have an extra flags parameter that allows callers to pass flags to the allocator.

A small nit below..

Quoted 60 lines
---
 rust/kernel/alloc.rs           |  1 +
 rust/kernel/alloc/allocator.rs |  2 +-
 rust/kernel/alloc/box_ext.rs   | 60 ++++++++++++++++++++++++++++++++++
 rust/kernel/init.rs            | 13 ++++----
 rust/kernel/prelude.rs         |  2 +-
 rust/kernel/sync/arc.rs        |  3 +-
 6 files changed, 72 insertions(+), 9 deletions(-)
 create mode 100644 rust/kernel/alloc/box_ext.rs
rust/kernel/alloc.rs +1 -0
index fd5541991a22..9bc1b48b5641 100644
@@ -5,6 +5,7 @@
 #[cfg(not(test))]
 #[cfg(not(testlib))]
 mod allocator;
+pub mod box_ext;
 pub mod vec_ext;
 
 /// Flags to be used when allocating memory.
rust/kernel/alloc/allocator.rs +1 -1
index 8cc7099d6ae1..ff88bce04fd4 100644
@@ -16,7 +16,7 @@
 ///
 /// - `ptr` can be either null or a pointer which has been allocated by this allocator.
 /// - `new_layout` must have a non-zero size.
-unsafe fn krealloc_aligned(ptr: *mut u8, new_layout: Layout, flags: Flags) -> *mut u8 {
+pub(crate) unsafe fn krealloc_aligned(ptr: *mut u8, new_layout: Layout, flags: Flags) -> *mut u8 {
     // Customized layouts from `Layout::from_size_align()` can have size < align, so pad first.
     let layout = new_layout.pad_to_align();
 
rust/kernel/alloc/box_ext.rs new file +19 -0
new file mode 100644
index 000000000000..c62db0aa4dd1
@@ -0,0 +1,60 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Extensions to [`Box`] for fallible allocations.
+
+use super::Flags;
+use alloc::boxed::Box;
+use core::alloc::AllocError;
+use core::mem::MaybeUninit;
+use core::result::Result;
+
+/// Extensions to [`Box`].
+pub trait BoxExt<T>: Sized {
+    /// Allocates a new box.
+    ///
+    /// The allocation may fail, in which case an error is returned.
+    fn new(x: T, flags: Flags) -> Result<Self, AllocError>;
+
+    /// Allocates a new uninitialised box.
+    ///

rust/kernel/alloc/box_ext.rs line 29

Quoted 10 lines
rust/kernel/alloc/box_ext.rs new file +10 -0
+    /// The allocation may fail, in which case an error is returned.
+    fn new_uninit(flags: Flags) -> Result<Box<MaybeUninit<T>>, AllocError>;
+}
+
+impl<T> BoxExt<T> for Box<T> {
+    fn new(x: T, flags: Flags) -> Result<Self, AllocError> {
+        let mut b = <Self as BoxExt<_>>::new_uninit(flags)?;
+        b.write(x);
+        // SAFETY: The contents were just initialised in the line above.
+        Ok(unsafe { b.assume_init() })

, since we have the new_uninit feature selected, we can actually do

Quoted 3 lines
        fn new(x: T, flags: Flags) -> Result<Self, AllocError> {
            <Self as BoxExt<_>>::new_uninit(flags).map(|b| Box::write(b, x))
        }

Oh, this avoids an unsafe block, I like it. Though I think I'm going to do a slightly different version:

    let b = <Self as BoxExt<_>>::new_uninit(flags)?;
    Ok(Box::write(b, x))
Quoted 8 lines

here.

Regards,
Boqun

Quoted 2 lines
rust/kernel/alloc/box_ext.rs new file +2 -0
+    }
+

[...]

Reviewed-by on v3 6/10
#9 lore raw

Re: [PATCH v3 06/10] rust: alloc: introduce the `BoxExt` trait

In reply to Wedson Almeida Filho
Quoted 18 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Make fallible versions of new and new_uninit methods available in Box even though it doesn't implement them because we build alloc with the no_global_oom_handling config.

They also have an extra flags parameter that allows callers to pass flags to the allocator.

---
  rust/kernel/alloc.rs           |  1 +
  rust/kernel/alloc/allocator.rs |  2 +-
  rust/kernel/alloc/box_ext.rs   | 60 ++++++++++++++++++++++++++++++++++
  rust/kernel/init.rs            | 13 ++++----
  rust/kernel/prelude.rs         |  2 +-
  rust/kernel/sync/arc.rs        |  3 +-
  6 files changed, 72 insertions(+), 9 deletions(-)
Quoted 1 line

create mode 100644 rust/kernel/alloc/box_ext.rs

With Boqun's suggestion (feel free to take your variant):

[...]

Quoted 15 lines
rust/kernel/alloc/box_ext.rs new file +15 -0
+impl<T> BoxExt<T> for Box<T> {
+    fn new(x: T, flags: Flags) -> Result<Self, AllocError> {
+        let mut b = <Self as BoxExt<_>>::new_uninit(flags)?;
+        b.write(x);
+        // SAFETY: The contents were just initialised in the line above.
+        Ok(unsafe { b.assume_init() })
+    }
+
+    #[cfg(any(test, testlib))]
+    fn new_uninit(_flags: Flags) -> Result<Box<MaybeUninit<T>>, AllocError> {
+        Ok(Box::new_uninit())
+    }
+
+    #[cfg(not(any(test, testlib)))]
+    fn new_uninit(flags: Flags) -> Result<Box<MaybeUninit<T>>, AllocError> {

rust/kernel/alloc/box_ext.rs line 48

Quoted 10 lines
rust/kernel/alloc/box_ext.rs new file +10 -0
+        let ptr = if core::mem::size_of::<MaybeUninit<T>>() == 0 {
+            core::ptr::NonNull::<_>::dangling().as_ptr()
+        } else {
+            let layout = core::alloc::Layout::new::<MaybeUninit<T>>();
+
+            // SAFETY: Memory is being allocated (first arg is null). The only other source of
+            // safety issues is sleeping on atomic context, which is addressed by klint. Lastly,
+            // the type is not a SZT (checked above).
+            let ptr =
+                unsafe { super::allocator::krealloc_aligned(core::ptr::null_mut(), layout, flags) };

Personally, I would rather import krealloc_aligned and not have this weird formatting here.

Signature · 2 lines
Cheers,
Benno
Quoted 12 lines
rust/kernel/alloc/box_ext.rs new file +12 -0
+            if ptr.is_null() {
+                return Err(AllocError);
+            }
+
+            ptr.cast::<MaybeUninit<T>>()
+        };
+
+        // SAFETY: For non-zero-sized types, we allocate above using the global allocator. For
+        // zero-sized types, we use `NonNull::dangling`.
+        Ok(unsafe { Box::from_raw(ptr) })
+    }
+}

[...]

#10 lore raw

PATCH v3 7/10 rust: alloc: update `VecExt` to take allocation flags

In reply to Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

We also rename the methods by removing the try_ prefix since the names are available due to our usage of the no_global_oom_handling config when building the alloc crate.

---
 rust/kernel/alloc/vec_ext.rs | 158 +++++++++++++++++++++++++++++++----
 rust/kernel/error.rs         |  11 +--
 rust/kernel/lib.rs           |   1 -
 rust/kernel/str.rs           |   6 +-
 rust/kernel/types.rs         |   4 +-
 samples/rust/rust_minimal.rs |   6 +-
 6 files changed, 152 insertions(+), 34 deletions(-)
rust/kernel/alloc/vec_ext.rs +143 -15
index 311e62cc5784..e24d7c7675ca 100644
@@ -2,47 +2,175 @@
 
 //! Extensions to [`Vec`] for fallible allocations.
 
-use alloc::{collections::TryReserveError, vec::Vec};
+use super::Flags;
+use alloc::{alloc::AllocError, vec::Vec};
 use core::result::Result;
 
 /// Extensions to [`Vec`].
 pub trait VecExt<T>: Sized {
     /// Creates a new [`Vec`] instance with at least the given capacity.
-    fn try_with_capacity(capacity: usize) -> Result<Self, TryReserveError>;
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// let v = Vec::<u32>::with_capacity(20, GFP_KERNEL)?;
+    ///
+    /// assert!(v.capacity() >= 20);
+    /// # Ok::<(), Error>(())
+    /// ```
+    fn with_capacity(capacity: usize, flags: Flags) -> Result<Self, AllocError>;
 
     /// Appends an element to the back of the [`Vec`] instance.
-    fn try_push(&mut self, v: T) -> Result<(), TryReserveError>;
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// let mut v = Vec::new();
+    /// v.push(1, GFP_KERNEL)?;
+    /// assert_eq!(&v, &[1]);
+    ///
+    /// v.push(2, GFP_KERNEL)?;
+    /// assert_eq!(&v, &[1, 2]);
+    /// # Ok::<(), Error>(())
+    /// ```
+    fn push(&mut self, v: T, flags: Flags) -> Result<(), AllocError>;
 
     /// Pushes clones of the elements of slice into the [`Vec`] instance.
-    fn try_extend_from_slice(&mut self, other: &[T]) -> Result<(), TryReserveError>
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// let mut v = Vec::new();
+    /// v.push(1, GFP_KERNEL)?;
+    ///
+    /// v.extend_from_slice(&[20, 30, 40], GFP_KERNEL)?;
+    /// assert_eq!(&v, &[1, 20, 30, 40]);
+    ///
+    /// v.extend_from_slice(&[50, 60], GFP_KERNEL)?;
+    /// assert_eq!(&v, &[1, 20, 30, 40, 50, 60]);
+    /// # Ok::<(), Error>(())
+    /// ```
+    fn extend_from_slice(&mut self, other: &[T], flags: Flags) -> Result<(), AllocError>
     where
         T: Clone;
+
+    /// Ensures that the capacity exceeds the length by at least `additional` elements.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// let mut v = Vec::new();
+    /// v.push(1, GFP_KERNEL)?;
+    ///
+    /// v.reserve(10, GFP_KERNEL)?;
+    /// let cap = v.capacity();
+    /// assert!(cap >= 10);
+    ///
+    /// v.reserve(10, GFP_KERNEL)?;
+    /// let new_cap = v.capacity();
+    /// assert_eq!(new_cap, cap);
+    ///
+    /// # Ok::<(), Error>(())
+    /// ```
+    fn reserve(&mut self, additional: usize, flags: Flags) -> Result<(), AllocError>;
 }
 
 impl<T> VecExt<T> for Vec<T> {
-    fn try_with_capacity(capacity: usize) -> Result<Self, TryReserveError> {
+    fn with_capacity(capacity: usize, flags: Flags) -> Result<Self, AllocError> {
         let mut v = Vec::new();
-        v.try_reserve(capacity)?;
+        <Self as VecExt<_>>::reserve(&mut v, capacity, flags)?;
         Ok(v)
     }
 
-    fn try_push(&mut self, v: T) -> Result<(), TryReserveError> {
-        if let Err(retry) = self.push_within_capacity(v) {
-            self.try_reserve(1)?;
-            let _ = self.push_within_capacity(retry);
-        }
+    fn push(&mut self, v: T, flags: Flags) -> Result<(), AllocError> {
+        <Self as VecExt<_>>::reserve(self, 1, flags)?;
+        let s = self.spare_capacity_mut();
+        s[0].write(v);
+
+        // SAFETY: We just initialised the first spare entry, so it is safe to increase the length
+        // by 1. We also know that the new length is <= capacity because of the previous call to
+        // `reserve` above.
+        unsafe { self.set_len(self.len() + 1) };
         Ok(())
     }
 
-    fn try_extend_from_slice(&mut self, other: &[T]) -> Result<(), TryReserveError>
+    fn extend_from_slice(&mut self, other: &[T], flags: Flags) -> Result<(), AllocError>
     where
         T: Clone,
     {
-        self.try_reserve(other.len())?;
-        for item in other {
-            self.try_push(item.clone())?;
+        <Self as VecExt<_>>::reserve(self, other.len(), flags)?;
+        for (slot, item) in core::iter::zip(self.spare_capacity_mut(), other) {
+            slot.write(item.clone());
         }
 
+        // SAFETY: We just initialised the `other.len()` spare entries, so it is safe to increase
+        // the length by the same amount. We also know that the new length is <= capacity because
+        // of the previous call to `reserve` above.
+        unsafe { self.set_len(self.len() + other.len()) };
+        Ok(())
+    }
+
+    #[cfg(any(test, testlib))]
+    fn reserve(&mut self, additional: usize, _flags: Flags) -> Result<(), AllocError> {
+        Vec::reserve(self, additional);
         Ok(())
     }
+
+    #[cfg(not(any(test, testlib)))]
+    fn reserve(&mut self, additional: usize, flags: Flags) -> Result<(), AllocError> {
+        let len = self.len();
+        let cap = self.capacity();
+
+        if cap - len >= additional {
+            return Ok(());
+        }
+
+        if core::mem::size_of::<T>() == 0 {
+            // The capacity is already `usize::MAX` for SZTs, we can't go higher.
+            return Err(AllocError);
+        }
+
+        // We know cap is <= `isize::MAX` because `Layout::array` fails if the resulting byte size
+        // is greater than `isize::MAX`. So the multiplication by two won't overflow.
+        let new_cap = core::cmp::max(cap * 2, len.checked_add(additional).ok_or(AllocError)?);
+        let layout = core::alloc::Layout::array::<T>(new_cap).map_err(|_| AllocError)?;
+
+        let (ptr, len, cap) = destructure(self);
+
+        // SAFETY: `ptr` is valid because it's either NULL or comes from a previous call to
+        // `krealloc_aligned`. We also verified that the type is not a ZST.
+        let new_ptr = unsafe { super::allocator::krealloc_aligned(ptr.cast(), layout, flags) };
+        if new_ptr.is_null() {
+            // SAFETY: We are just rebuilding the existing `Vec` with no changes.
+            unsafe { rebuild(self, ptr, len, cap) };
+            Err(AllocError)
+        } else {
+            // SAFETY: `ptr` has been reallocated with the layout for `new_cap` elements. New cap
+            // is greater than `cap`, so it continues to be >= `len`.
+            unsafe { rebuild(self, new_ptr.cast::<T>(), len, new_cap) };
+            Ok(())
+        }
+    }
+}
+
+#[cfg(not(any(test, testlib)))]
+fn destructure<T>(v: &mut Vec<T>) -> (*mut T, usize, usize) {
+    let mut tmp = Vec::new();
+    core::mem::swap(&mut tmp, v);
+    let mut tmp = core::mem::ManuallyDrop::new(tmp);
+    let len = tmp.len();
+    let cap = tmp.capacity();
+    (tmp.as_mut_ptr(), len, cap)
+}
+
+/// Rebuilds a `Vec` from a pointer, length, and capacity.
+///
+/// # Safety
+///
+/// The same as [`Vec::from_raw_parts`].
+#[cfg(not(any(test, testlib)))]
+unsafe fn rebuild<T>(v: &mut Vec<T>, ptr: *mut T, len: usize, cap: usize) {
+    // SAFETY: The safety requirements from this function satisfy those of `from_raw_parts`.
+    let mut tmp = unsafe { Vec::from_raw_parts(ptr, len, cap) };
+    core::mem::swap(&mut tmp, v);
 }
rust/kernel/error.rs +1 -10
index 4786d3ee1e92..e53466937796 100644
@@ -6,10 +6,7 @@
 
 use crate::str::CStr;
 
-use alloc::{
-    alloc::{AllocError, LayoutError},
-    collections::TryReserveError,
-};
+use alloc::alloc::{AllocError, LayoutError};
 
 use core::convert::From;
 use core::fmt;
@@ -192,12 +189,6 @@ fn from(_: Utf8Error) -> Error {
     }
 }
 
-impl From<TryReserveError> for Error {
-    fn from(_: TryReserveError) -> Error {
-        code::ENOMEM
-    }
-}
-
 impl From<LayoutError> for Error {
     fn from(_: LayoutError) -> Error {
         code::ENOMEM
rust/kernel/lib.rs +0 -1
index 7f2841a18d05..51f30e55bd00 100644
@@ -19,7 +19,6 @@
 #![feature(offset_of)]
 #![feature(receiver_trait)]
 #![feature(unsize)]
-#![feature(vec_push_within_capacity)]
 
 // Ensure conditional compilation based on the kernel configuration works;
 // otherwise we may silently break things like initcall handling.
rust/kernel/str.rs +3 -3
index e1be6c5e72a9..58f2ddafb326 100644
@@ -2,7 +2,7 @@
 
 //! String representations.
 
-use crate::alloc::vec_ext::VecExt;
+use crate::alloc::{flags::*, vec_ext::VecExt};
 use alloc::alloc::AllocError;
 use alloc::vec::Vec;
 use core::fmt::{self, Write};
@@ -730,7 +730,7 @@ pub fn try_from_fmt(args: fmt::Arguments<'_>) -> Result<Self, Error> {
         let size = f.bytes_written();
 
         // Allocate a vector with the required number of bytes, and write to it.
-        let mut buf = Vec::try_with_capacity(size)?;
+        let mut buf = <Vec<_> as VecExt<_>>::with_capacity(size, GFP_KERNEL)?;
         // SAFETY: The buffer stored in `buf` is at least of size `size` and is valid for writes.
         let mut f = unsafe { Formatter::from_buffer(buf.as_mut_ptr(), size) };
         f.write_fmt(args)?;
@@ -771,7 +771,7 @@ impl<'a> TryFrom<&'a CStr> for CString {
     fn try_from(cstr: &'a CStr) -> Result<CString, AllocError> {
         let mut buf = Vec::new();
 
-        buf.try_extend_from_slice(cstr.as_bytes_with_nul())
+        <Vec<_> as VecExt<_>>::extend_from_slice(&mut buf, cstr.as_bytes_with_nul(), GFP_KERNEL)
             .map_err(|_| AllocError)?;
 
         // INVARIANT: The `CStr` and `CString` types have the same invariants for
rust/kernel/types.rs +2 -2
index aa77bad9bce4..8fad61268465 100644
@@ -157,11 +157,11 @@ unsafe fn from_foreign(_: *const core::ffi::c_void) -> Self {}
 ///     let mut vec =
 ///         ScopeGuard::new_with_data(Vec::new(), |v| pr_info!("vec had {} elements\n", v.len()));
 ///
-///     vec.try_push(10u8)?;
+///     vec.push(10u8, GFP_KERNEL)?;
 ///     if arg {
 ///         return Ok(());
 ///     }
-///     vec.try_push(20u8)?;
+///     vec.push(20u8, GFP_KERNEL)?;
 ///     Ok(())
 /// }
 ///
samples/rust/rust_minimal.rs +3 -3
index dc05f4bbe27e..2a9eaab62d1c 100644
@@ -22,9 +22,9 @@ fn init(_module: &'static ThisModule) -> Result<Self> {
         pr_info!("Am I built-in? {}\n", !cfg!(MODULE));
 
         let mut numbers = Vec::new();
-        numbers.try_push(72)?;
-        numbers.try_push(108)?;
-        numbers.try_push(200)?;
+        numbers.push(72, GFP_KERNEL)?;
+        numbers.push(108, GFP_KERNEL)?;
+        numbers.push(200, GFP_KERNEL)?;
 
         Ok(RustMinimal { numbers })
     }
Signature · 1 line
2.34.1
Reviewed-by on v3 7/10
#11 lore raw

Re: [PATCH v3 07/10] rust: alloc: update `VecExt` to take allocation flags

Quoted 8 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

We also rename the methods by removing the try_ prefix since the names are available due to our usage of the no_global_oom_handling config when building the alloc crate.

Quoted 8 lines
---
  rust/kernel/alloc/vec_ext.rs | 158 +++++++++++++++++++++++++++++++----
  rust/kernel/error.rs         |  11 +--
  rust/kernel/lib.rs           |   1 -
  rust/kernel/str.rs           |   6 +-
  rust/kernel/types.rs         |   4 +-
  samples/rust/rust_minimal.rs |   6 +-
  6 files changed, 152 insertions(+), 34 deletions(-)

[...]

Quoted 8 lines
  impl<T> VecExt<T> for Vec<T> {
rust/kernel/alloc/vec_ext.rs +2 -2
-    fn try_with_capacity(capacity: usize) -> Result<Self, TryReserveError> {
+    fn with_capacity(capacity: usize, flags: Flags) -> Result<Self, AllocError> {
          let mut v = Vec::new();
-        v.try_reserve(capacity)?;
+        <Self as VecExt<_>>::reserve(&mut v, capacity, flags)?;
          Ok(v)
      }

rust/kernel/alloc/vec_ext.rs line 88

Quoted 10 lines
rust/kernel/alloc/vec_ext.rs +4 -5
 
-    fn try_push(&mut self, v: T) -> Result<(), TryReserveError> {
-        if let Err(retry) = self.push_within_capacity(v) {
-            self.try_reserve(1)?;
-            let _ = self.push_within_capacity(retry);
-        }
+    fn push(&mut self, v: T, flags: Flags) -> Result<(), AllocError> {
+        <Self as VecExt<_>>::reserve(self, 1, flags)?;
+        let s = self.spare_capacity_mut();
+        s[0].write(v);

I haven't checked the codegen, but this could result in an additional bound check. We don't need to care about this now, but at some point we should do a performance test. There are probably other things that I have missed.

Signature · 2 lines
Cheers,
Benno
Quoted 7 lines
rust/kernel/alloc/vec_ext.rs +5 -0
+
+        // SAFETY: We just initialised the first spare entry, so it is safe to increase the length
+        // by 1. We also know that the new length is <= capacity because of the previous call to
+        // `reserve` above.
+        unsafe { self.set_len(self.len() + 1) };
          Ok(())
      }

[...]

#12 lore raw

PATCH v3 8/10 rust: sync: update `Arc` and `UniqueArc` to take allocation flags

In reply to Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

We also remove the try_ prefix to align with how Box and Vec are providing methods now.

init is temporarily updated with uses of GFP_KERNEL. These will be updated in a subsequent patch to take flags as well.

---
 rust/kernel/init.rs        |  4 ++--
 rust/kernel/sync/arc.rs    | 28 ++++++++++++++--------------
 samples/rust/rust_print.rs |  4 ++--
 3 files changed, 18 insertions(+), 18 deletions(-)
rust/kernel/init.rs +2 -2
index 9c798cffc8e4..a5911fec428b 100644
@@ -1189,7 +1189,7 @@ fn try_pin_init<E>(init: impl PinInit<T, E>) -> Result<Pin<Self>, E>
     where
         E: From<AllocError>,
     {
-        let mut this = UniqueArc::try_new_uninit()?;
+        let mut this = UniqueArc::new_uninit(GFP_KERNEL)?;
         let slot = this.as_mut_ptr();
         // SAFETY: When init errors/panics, slot will get deallocated but not dropped,
         // slot is valid and will not be moved, because we pin it later.
@@ -1203,7 +1203,7 @@ fn try_init<E>(init: impl Init<T, E>) -> Result<Self, E>
     where
         E: From<AllocError>,
     {
-        let mut this = UniqueArc::try_new_uninit()?;
+        let mut this = UniqueArc::new_uninit(GFP_KERNEL)?;
         let slot = this.as_mut_ptr();
         // SAFETY: When init errors/panics, slot will get deallocated but not dropped,
         // slot is valid.
rust/kernel/sync/arc.rs +14 -14
index 1252a1b630ed..b67bb876ddf7 100644
@@ -16,7 +16,7 @@
 //! [`Arc`]: https://doc.rust-lang.org/std/sync/struct.Arc.html
 
 use crate::{
-    alloc::{box_ext::BoxExt, flags::*},
+    alloc::{box_ext::BoxExt, Flags},
     bindings,
     error::{self, Error},
     init::{self, InPlaceInit, Init, PinInit},
@@ -58,7 +58,7 @@
 /// }
 ///
 /// // Create a refcounted instance of `Example`.
-/// let obj = Arc::try_new(Example { a: 10, b: 20 })?;
+/// let obj = Arc::new(Example { a: 10, b: 20 }, GFP_KERNEL)?;
 ///
 /// // Get a new pointer to `obj` and increment the refcount.
 /// let cloned = obj.clone();
@@ -97,7 +97,7 @@
 ///     }
 /// }
 ///
-/// let obj = Arc::try_new(Example { a: 10, b: 20 })?;
+/// let obj = Arc::new(Example { a: 10, b: 20 }, GFP_KERNEL)?;
 /// obj.use_reference();
 /// obj.take_over();
 /// # Ok::<(), Error>(())
@@ -120,7 +120,7 @@
 /// impl MyTrait for Example {}
 ///
 /// // `obj` has type `Arc<Example>`.
-/// let obj: Arc<Example> = Arc::try_new(Example)?;
+/// let obj: Arc<Example> = Arc::new(Example, GFP_KERNEL)?;
 ///
 /// // `coerced` has type `Arc<dyn MyTrait>`.
 /// let coerced: Arc<dyn MyTrait> = obj;
@@ -163,7 +163,7 @@ unsafe impl<T: ?Sized + Sync + Send> Sync for Arc<T> {}
 
 impl<T> Arc<T> {
     /// Constructs a new reference counted instance of `T`.
-    pub fn try_new(contents: T) -> Result<Self, AllocError> {
+    pub fn new(contents: T, flags: Flags) -> Result<Self, AllocError> {
         // INVARIANT: The refcount is initialised to a non-zero value.
         let value = ArcInner {
             // SAFETY: There are no safety requirements for this FFI call.
@@ -171,7 +171,7 @@ pub fn try_new(contents: T) -> Result<Self, AllocError> {
             data: contents,
         };
 
-        let inner = <Box<_> as BoxExt<_>>::new(value, GFP_KERNEL)?;
+        let inner = <Box<_> as BoxExt<_>>::new(value, flags)?;
 
         // SAFETY: We just created `inner` with a reference count of 1, which is owned by the new
         // `Arc` object.
@@ -388,7 +388,7 @@ fn from(item: Pin<UniqueArc<T>>) -> Self {
 ///     e.into()
 /// }
 ///
-/// let obj = Arc::try_new(Example)?;
+/// let obj = Arc::new(Example, GFP_KERNEL)?;
 /// let cloned = do_something(obj.as_arc_borrow());
 ///
 /// // Assert that both `obj` and `cloned` point to the same underlying object.
@@ -412,7 +412,7 @@ fn from(item: Pin<UniqueArc<T>>) -> Self {
 ///     }
 /// }
 ///
-/// let obj = Arc::try_new(Example { a: 10, b: 20 })?;
+/// let obj = Arc::new(Example { a: 10, b: 20 }, GFP_KERNEL)?;
 /// obj.as_arc_borrow().use_reference();
 /// # Ok::<(), Error>(())
 /// ```
@@ -500,7 +500,7 @@ fn deref(&self) -> &Self::Target {
 /// }
 ///
 /// fn test() -> Result<Arc<Example>> {
-///     let mut x = UniqueArc::try_new(Example { a: 10, b: 20 })?;
+///     let mut x = UniqueArc::new(Example { a: 10, b: 20 }, GFP_KERNEL)?;
 ///     x.a += 1;
 ///     x.b += 1;
 ///     Ok(x.into())
@@ -523,7 +523,7 @@ fn deref(&self) -> &Self::Target {
 /// }
 ///
 /// fn test() -> Result<Arc<Example>> {
-///     let x = UniqueArc::try_new_uninit()?;
+///     let x = UniqueArc::new_uninit(GFP_KERNEL)?;
 ///     Ok(x.write(Example { a: 10, b: 20 }).into())
 /// }
 ///
@@ -543,7 +543,7 @@ fn deref(&self) -> &Self::Target {
 /// }
 ///
 /// fn test() -> Result<Arc<Example>> {
-///     let mut pinned = Pin::from(UniqueArc::try_new(Example { a: 10, b: 20 })?);
+///     let mut pinned = Pin::from(UniqueArc::new(Example { a: 10, b: 20 }, GFP_KERNEL)?);
 ///     // We can modify `pinned` because it is `Unpin`.
 ///     pinned.as_mut().a += 1;
 ///     Ok(pinned.into())
@@ -557,15 +557,15 @@ pub struct UniqueArc<T: ?Sized> {
 
 impl<T> UniqueArc<T> {
     /// Tries to allocate a new [`UniqueArc`] instance.
-    pub fn try_new(value: T) -> Result<Self, AllocError> {
+    pub fn new(value: T, flags: Flags) -> Result<Self, AllocError> {
         Ok(Self {
             // INVARIANT: The newly-created object has a refcount of 1.
-            inner: Arc::try_new(value)?,
+            inner: Arc::new(value, flags)?,
         })
     }
 
     /// Tries to allocate a new [`UniqueArc`] instance whose contents are not initialised yet.
-    pub fn try_new_uninit() -> Result<UniqueArc<MaybeUninit<T>>, AllocError> {
+    pub fn new_uninit(_flags: Flags) -> Result<UniqueArc<MaybeUninit<T>>, AllocError> {
         // INVARIANT: The refcount is initialised to a non-zero value.
         let inner = Box::try_init::<AllocError>(try_init!(ArcInner {
             // SAFETY: There are no safety requirements for this FFI call.
samples/rust/rust_print.rs +2 -2
index 67ed8ebf8e8e..6eabb0d79ea3 100644
@@ -18,8 +18,8 @@
 fn arc_print() -> Result {
     use kernel::sync::*;
 
-    let a = Arc::try_new(1)?;
-    let b = UniqueArc::try_new("hello, world")?;
+    let a = Arc::new(1, GFP_KERNEL)?;
+    let b = UniqueArc::new("hello, world", GFP_KERNEL)?;
 
     // Prints the value of data in `a`.
     pr_info!("{}", a);
Signature · 1 line
2.34.1
#13 lore raw

PATCH v3 9/10 rust: init: update `init` module to take allocation flags

In reply to Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

This is the last component in the conversion for allocators to take allocation flags as parameters.

---
 rust/kernel/init.rs               | 49 ++++++++++++++++---------------
 rust/kernel/sync/arc.rs           | 23 ++++++++-------
 rust/kernel/sync/condvar.rs       |  2 +-
 rust/kernel/sync/lock/mutex.rs    |  2 +-
 rust/kernel/sync/lock/spinlock.rs |  2 +-
 rust/kernel/workqueue.rs          | 13 +++++---
 6 files changed, 50 insertions(+), 41 deletions(-)
rust/kernel/init.rs +25 -24
index a5911fec428b..fec47b274ec3 100644
@@ -68,7 +68,7 @@
 //! #     a <- new_mutex!(42, "Foo::a"),
 //! #     b: 24,
 //! # });
-//! let foo: Result<Pin<Box<Foo>>> = Box::pin_init(foo);
+//! let foo: Result<Pin<Box<Foo>>> = Box::pin_init(foo, GFP_KERNEL);
 //! ```
 //!
 //! For more information see the [`pin_init!`] macro.
@@ -80,7 +80,8 @@
 //!
 //! ```rust
 //! # use kernel::sync::{new_mutex, Arc, Mutex};
-//! let mtx: Result<Arc<Mutex<usize>>> = Arc::pin_init(new_mutex!(42, "example::mtx"));
+//! let mtx: Result<Arc<Mutex<usize>>> =
+//!     Arc::pin_init(new_mutex!(42, "example::mtx"), GFP_KERNEL);
 //! ```
 //!
 //! To declare an init macro/function you just return an [`impl PinInit<T, E>`]:
@@ -99,7 +100,7 @@
 //!     fn new() -> impl PinInit<Self, Error> {
 //!         try_pin_init!(Self {
 //!             status <- new_mutex!(0, "DriverData::status"),
-//!             buffer: Box::init(kernel::init::zeroed())?,
+//!             buffer: Box::init(kernel::init::zeroed(), GFP_KERNEL)?,
 //!         })
 //!     }
 //! }
@@ -210,7 +211,7 @@
 //! [`pin_init!`]: crate::pin_init!
 
 use crate::{
-    alloc::{box_ext::BoxExt, flags::*},
+    alloc::{box_ext::BoxExt, Flags},
     error::{self, Error},
     sync::UniqueArc,
     types::{Opaque, ScopeGuard},
@@ -391,7 +392,7 @@ macro_rules! stack_try_pin_init {
 ///     },
 /// });
 /// # initializer }
-/// # Box::pin_init(demo()).unwrap();
+/// # Box::pin_init(demo(), GFP_KERNEL).unwrap();
 /// ```
 ///
 /// Arbitrary Rust expressions can be used to set the value of a variable.
@@ -461,7 +462,7 @@ macro_rules! stack_try_pin_init {
 /// #         })
 /// #     }
 /// # }
-/// let foo = Box::pin_init(Foo::new());
+/// let foo = Box::pin_init(Foo::new(), GFP_KERNEL);
 /// ```
 ///
 /// They can also easily embed it into their own `struct`s:
@@ -601,7 +602,7 @@ macro_rules! pin_init {
 /// impl BigBuf {
 ///     fn new() -> impl PinInit<Self, Error> {
 ///         try_pin_init!(Self {
-///             big: Box::init(init::zeroed())?,
+///             big: Box::init(init::zeroed(), GFP_KERNEL)?,
 ///             small: [0; 1024 * 1024],
 ///             ptr: core::ptr::null_mut(),
 ///         }? Error)
@@ -702,7 +703,7 @@ macro_rules! init {
 /// impl BigBuf {
 ///     fn new() -> impl Init<Self, Error> {
 ///         try_init!(Self {
-///             big: Box::init(zeroed())?,
+///             big: Box::init(zeroed(), GFP_KERNEL)?,
 ///             small: [0; 1024 * 1024],
 ///         }? Error)
 ///     }
@@ -1014,7 +1015,7 @@ pub fn uninit<T, E>() -> impl Init<MaybeUninit<T>, E> {
 ///
 /// ```rust
 /// use kernel::{error::Error, init::init_array_from_fn};
-/// let array: Box<[usize; 1_000]> = Box::init::<Error>(init_array_from_fn(|i| i)).unwrap();
+/// let array: Box<[usize; 1_000]> = Box::init::<Error>(init_array_from_fn(|i| i), GFP_KERNEL).unwrap();
 /// assert_eq!(array.len(), 1_000);
 /// ```
 pub fn init_array_from_fn<I, const N: usize, T, E>(
@@ -1058,7 +1059,7 @@ pub fn init_array_from_fn<I, const N: usize, T, E>(
 /// ```rust
 /// use kernel::{sync::{Arc, Mutex}, init::pin_init_array_from_fn, new_mutex};
 /// let array: Arc<[Mutex<usize>; 1_000]> =
-///     Arc::pin_init(pin_init_array_from_fn(|i| new_mutex!(i))).unwrap();
+///     Arc::pin_init(pin_init_array_from_fn(|i| new_mutex!(i)), GFP_KERNEL).unwrap();
 /// assert_eq!(array.len(), 1_000);
 /// ```
 pub fn pin_init_array_from_fn<I, const N: usize, T, E>(
@@ -1116,7 +1117,7 @@ pub trait InPlaceInit<T>: Sized {
     /// type.
     ///
     /// If `T: !Unpin` it will not be able to move afterwards.
-    fn try_pin_init<E>(init: impl PinInit<T, E>) -> Result<Pin<Self>, E>
+    fn try_pin_init<E>(init: impl PinInit<T, E>, flags: Flags) -> Result<Pin<Self>, E>
     where
         E: From<AllocError>;
 
@@ -1124,7 +1125,7 @@ fn try_pin_init<E>(init: impl PinInit<T, E>) -> Result<Pin<Self>, E>
     /// type.
     ///
     /// If `T: !Unpin` it will not be able to move afterwards.
-    fn pin_init<E>(init: impl PinInit<T, E>) -> error::Result<Pin<Self>>
+    fn pin_init<E>(init: impl PinInit<T, E>, flags: Flags) -> error::Result<Pin<Self>>
     where
         Error: From<E>,
     {
@@ -1132,16 +1133,16 @@ fn pin_init<E>(init: impl PinInit<T, E>) -> error::Result<Pin<Self>>
         let init = unsafe {
             pin_init_from_closure(|slot| init.__pinned_init(slot).map_err(|e| Error::from(e)))
         };
-        Self::try_pin_init(init)
+        Self::try_pin_init(init, flags)
     }
 
     /// Use the given initializer to in-place initialize a `T`.
-    fn try_init<E>(init: impl Init<T, E>) -> Result<Self, E>
+    fn try_init<E>(init: impl Init<T, E>, flags: Flags) -> Result<Self, E>
     where
         E: From<AllocError>;
 
     /// Use the given initializer to in-place initialize a `T`.
-    fn init<E>(init: impl Init<T, E>) -> error::Result<Self>
+    fn init<E>(init: impl Init<T, E>, flags: Flags) -> error::Result<Self>
     where
         Error: From<E>,
     {
@@ -1149,17 +1150,17 @@ fn init<E>(init: impl Init<T, E>) -> error::Result<Self>
         let init = unsafe {
             init_from_closure(|slot| init.__pinned_init(slot).map_err(|e| Error::from(e)))
         };
-        Self::try_init(init)
+        Self::try_init(init, flags)
     }
 }
 
 impl<T> InPlaceInit<T> for Box<T> {
     #[inline]
-    fn try_pin_init<E>(init: impl PinInit<T, E>) -> Result<Pin<Self>, E>
+    fn try_pin_init<E>(init: impl PinInit<T, E>, flags: Flags) -> Result<Pin<Self>, E>
     where
         E: From<AllocError>,
     {
-        let mut this = <Box<_> as BoxExt<_>>::new_uninit(GFP_KERNEL)?;
+        let mut this = <Box<_> as BoxExt<_>>::new_uninit(flags)?;
         let slot = this.as_mut_ptr();
         // SAFETY: When init errors/panics, slot will get deallocated but not dropped,
         // slot is valid and will not be moved, because we pin it later.
@@ -1169,11 +1170,11 @@ fn try_pin_init<E>(init: impl PinInit<T, E>) -> Result<Pin<Self>, E>
     }
 
     #[inline]
-    fn try_init<E>(init: impl Init<T, E>) -> Result<Self, E>
+    fn try_init<E>(init: impl Init<T, E>, flags: Flags) -> Result<Self, E>
     where
         E: From<AllocError>,
     {
-        let mut this = <Box<_> as BoxExt<_>>::new_uninit(GFP_KERNEL)?;
+        let mut this = <Box<_> as BoxExt<_>>::new_uninit(flags)?;
         let slot = this.as_mut_ptr();
         // SAFETY: When init errors/panics, slot will get deallocated but not dropped,
         // slot is valid.
@@ -1185,11 +1186,11 @@ fn try_init<E>(init: impl Init<T, E>) -> Result<Self, E>
 
 impl<T> InPlaceInit<T> for UniqueArc<T> {
     #[inline]
-    fn try_pin_init<E>(init: impl PinInit<T, E>) -> Result<Pin<Self>, E>
+    fn try_pin_init<E>(init: impl PinInit<T, E>, flags: Flags) -> Result<Pin<Self>, E>
     where
         E: From<AllocError>,
     {
-        let mut this = UniqueArc::new_uninit(GFP_KERNEL)?;
+        let mut this = UniqueArc::new_uninit(flags)?;
         let slot = this.as_mut_ptr();
         // SAFETY: When init errors/panics, slot will get deallocated but not dropped,
         // slot is valid and will not be moved, because we pin it later.
@@ -1199,11 +1200,11 @@ fn try_pin_init<E>(init: impl PinInit<T, E>) -> Result<Pin<Self>, E>
     }
 
     #[inline]
-    fn try_init<E>(init: impl Init<T, E>) -> Result<Self, E>
+    fn try_init<E>(init: impl Init<T, E>, flags: Flags) -> Result<Self, E>
     where
         E: From<AllocError>,
     {
-        let mut this = UniqueArc::new_uninit(GFP_KERNEL)?;
+        let mut this = UniqueArc::new_uninit(flags)?;
         let slot = this.as_mut_ptr();
         // SAFETY: When init errors/panics, slot will get deallocated but not dropped,
         // slot is valid.
rust/kernel/sync/arc.rs +13 -10
index b67bb876ddf7..0866378f1360 100644
@@ -182,22 +182,22 @@ pub fn new(contents: T, flags: Flags) -> Result<Self, AllocError> {
     ///
     /// If `T: !Unpin` it will not be able to move afterwards.
     #[inline]
-    pub fn pin_init<E>(init: impl PinInit<T, E>) -> error::Result<Self>
+    pub fn pin_init<E>(init: impl PinInit<T, E>, flags: Flags) -> error::Result<Self>
     where
         Error: From<E>,
     {
-        UniqueArc::pin_init(init).map(|u| u.into())
+        UniqueArc::pin_init(init, flags).map(|u| u.into())
     }
 
     /// Use the given initializer to in-place initialize a `T`.
     ///
     /// This is equivalent to [`Arc<T>::pin_init`], since an [`Arc`] is always pinned.
     #[inline]
-    pub fn init<E>(init: impl Init<T, E>) -> error::Result<Self>
+    pub fn init<E>(init: impl Init<T, E>, flags: Flags) -> error::Result<Self>
     where
         Error: From<E>,
     {
-        UniqueArc::init(init).map(|u| u.into())
+        UniqueArc::init(init, flags).map(|u| u.into())
     }
 }
 
@@ -565,13 +565,16 @@ pub fn new(value: T, flags: Flags) -> Result<Self, AllocError> {
     }
 
     /// Tries to allocate a new [`UniqueArc`] instance whose contents are not initialised yet.
-    pub fn new_uninit(_flags: Flags) -> Result<UniqueArc<MaybeUninit<T>>, AllocError> {
+    pub fn new_uninit(flags: Flags) -> Result<UniqueArc<MaybeUninit<T>>, AllocError> {
         // INVARIANT: The refcount is initialised to a non-zero value.
-        let inner = Box::try_init::<AllocError>(try_init!(ArcInner {
-            // SAFETY: There are no safety requirements for this FFI call.
-            refcount: Opaque::new(unsafe { bindings::REFCOUNT_INIT(1) }),
-            data <- init::uninit::<T, AllocError>(),
-        }? AllocError))?;
+        let inner = Box::try_init::<AllocError>(
+            try_init!(ArcInner {
+                // SAFETY: There are no safety requirements for this FFI call.
+                refcount: Opaque::new(unsafe { bindings::REFCOUNT_INIT(1) }),
+                data <- init::uninit::<T, AllocError>(),
+            }? AllocError),
+            flags,
+        )?;
         Ok(UniqueArc {
             // INVARIANT: The newly-created object has a refcount of 1.
             // SAFETY: The pointer from the `Box` is valid.
rust/kernel/sync/condvar.rs +1 -1
index 0c3671caffeb..ef6ffef0aa88 100644
@@ -75,7 +75,7 @@ macro_rules! new_condvar {
 ///     Box::pin_init(pin_init!(Example {
 ///         value <- new_mutex!(0),
 ///         value_changed <- new_condvar!(),
-///     }))
+///     }), GFP_KERNEL)
 /// }
 /// ```
 ///
rust/kernel/sync/lock/mutex.rs +1 -1
index ef4c4634d294..93e1c982facf 100644
@@ -60,7 +60,7 @@ macro_rules! new_mutex {
 /// }
 ///
 /// // Allocate a boxed `Example`.
-/// let e = Box::pin_init(Example::new())?;
+/// let e = Box::pin_init(Example::new(), GFP_KERNEL)?;
 /// assert_eq!(e.c, 10);
 /// assert_eq!(e.d.lock().a, 20);
 /// assert_eq!(e.d.lock().b, 30);
rust/kernel/sync/lock/spinlock.rs +1 -1
index 0b22c635634f..6e900807d3b7 100644
@@ -58,7 +58,7 @@ macro_rules! new_spinlock {
 /// }
 ///
 /// // Allocate a boxed `Example`.
-/// let e = Box::pin_init(Example::new())?;
+/// let e = Box::pin_init(Example::new(), GFP_KERNEL)?;
 /// assert_eq!(e.c, 10);
 /// assert_eq!(e.d.lock().a, 20);
 /// assert_eq!(e.d.lock().b, 30);
rust/kernel/workqueue.rs +9 -4
index 480cb292e7c2..4d1f5306497a 100644
@@ -53,7 +53,7 @@
 //!         Arc::pin_init(pin_init!(MyStruct {
 //!             value,
 //!             work <- new_work!("MyStruct::work"),
-//!         }))
+//!         }), GFP_KERNEL)
 //!     }
 //! }
 //!
@@ -101,7 +101,7 @@
 //!             value_2,
 //!             work_1 <- new_work!("MyStruct::work_1"),
 //!             work_2 <- new_work!("MyStruct::work_2"),
-//!         }))
+//!         }), GFP_KERNEL)
 //!     }
 //! }
 //!
@@ -132,6 +132,7 @@
 //!
 //! C header: [`include/linux/workqueue.h`](srctree/include/linux/workqueue.h)
 
+use crate::alloc::Flags;
 use crate::{bindings, prelude::*, sync::Arc, sync::LockClassKey, types::Opaque};
 use alloc::alloc::AllocError;
 use alloc::boxed::Box;
@@ -210,13 +211,17 @@ pub fn enqueue<W, const ID: u64>(&self, w: W) -> W::EnqueueOutput
     /// Tries to spawn the given function or closure as a work item.
     ///
     /// This method can fail because it allocates memory to store the work item.
-    pub fn try_spawn<T: 'static + Send + FnOnce()>(&self, func: T) -> Result<(), AllocError> {
+    pub fn try_spawn<T: 'static + Send + FnOnce()>(
+        &self,
+        flags: Flags,
+        func: T,
+    ) -> Result<(), AllocError> {
         let init = pin_init!(ClosureWork {
             work <- new_work!("Queue::try_spawn"),
             func: Some(func),
         });
 
-        self.enqueue(Box::pin_init(init).map_err(|_| AllocError)?);
+        self.enqueue(Box::pin_init(init, flags).map_err(|_| AllocError)?);
         Ok(())
     }
 }
Signature · 1 line
2.34.1
#14 lore raw

PATCH v3 10/10 rust: kernel: remove usage of `allocator_api` unstable feature

In reply to Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

With the adoption of BoxExt and VecExt, we don't need the functions provided by this feature (namely the methods prefixed with try_ and different allocator per collection instance).

We do need AllocError, but we define our own as it is a trivial empty struct.

---
 rust/kernel/alloc.rs         | 4 ++++
 rust/kernel/alloc/box_ext.rs | 3 +--
 rust/kernel/alloc/vec_ext.rs | 4 ++--
 rust/kernel/error.rs         | 4 ++--
 rust/kernel/init.rs          | 3 +--
 rust/kernel/lib.rs           | 1 -
 rust/kernel/str.rs           | 3 +--
 rust/kernel/sync/arc.rs      | 4 ++--
 rust/kernel/workqueue.rs     | 3 +--
 9 files changed, 14 insertions(+), 15 deletions(-)
rust/kernel/alloc.rs +4 -0
index 9bc1b48b5641..f1c2c4aa22d2 100644
@@ -8,6 +8,10 @@
 pub mod box_ext;
 pub mod vec_ext;
 
+/// Indicates an allocation error.
+#[derive(Copy, Clone, PartialEq, Eq, Debug)]
+pub struct AllocError;
+
 /// Flags to be used when allocating memory.
 ///
 /// They can be combined with the operators `|`, `&`, and `!`.
rust/kernel/alloc/box_ext.rs +1 -2
index c62db0aa4dd1..3b48cfc70deb 100644
@@ -2,9 +2,8 @@
 
 //! Extensions to [`Box`] for fallible allocations.
 
-use super::Flags;
+use super::{AllocError, Flags};
 use alloc::boxed::Box;
-use core::alloc::AllocError;
 use core::mem::MaybeUninit;
 use core::result::Result;
 
rust/kernel/alloc/vec_ext.rs +2 -2
index e24d7c7675ca..6a916fcf8bf1 100644
@@ -2,8 +2,8 @@
 
 //! Extensions to [`Vec`] for fallible allocations.
 
-use super::Flags;
-use alloc::{alloc::AllocError, vec::Vec};
+use super::{AllocError, Flags};
+use alloc::vec::Vec;
 use core::result::Result;
 
 /// Extensions to [`Vec`].
rust/kernel/error.rs +2 -2
index e53466937796..fc986bc24c6d 100644
@@ -4,9 +4,9 @@
 //!
 //! C header: [`include/uapi/asm-generic/errno-base.h`](srctree/include/uapi/asm-generic/errno-base.h)
 
-use crate::str::CStr;
+use crate::{alloc::AllocError, str::CStr};
 
-use alloc::alloc::{AllocError, LayoutError};
+use alloc::alloc::LayoutError;
 
 use core::convert::From;
 use core::fmt;
rust/kernel/init.rs +1 -2
index fec47b274ec3..9608f2bd2211 100644
@@ -211,14 +211,13 @@
 //! [`pin_init!`]: crate::pin_init!
 
 use crate::{
-    alloc::{box_ext::BoxExt, Flags},
+    alloc::{box_ext::BoxExt, AllocError, Flags},
     error::{self, Error},
     sync::UniqueArc,
     types::{Opaque, ScopeGuard},
 };
 use alloc::boxed::Box;
 use core::{
-    alloc::AllocError,
     cell::UnsafeCell,
     convert::Infallible,
     marker::PhantomData,
rust/kernel/lib.rs +0 -1
index 51f30e55bd00..5c641233e26d 100644
@@ -12,7 +12,6 @@
 //! do so first instead of bypassing this crate.
 
 #![no_std]
-#![feature(allocator_api)]
 #![feature(coerce_unsized)]
 #![feature(dispatch_from_dyn)]
 #![feature(new_uninit)]
rust/kernel/str.rs +1 -2
index 58f2ddafb326..dfd1c40d46a2 100644
@@ -2,8 +2,7 @@
 
 //! String representations.
 
-use crate::alloc::{flags::*, vec_ext::VecExt};
-use alloc::alloc::AllocError;
+use crate::alloc::{flags::*, vec_ext::VecExt, AllocError};
 use alloc::vec::Vec;
 use core::fmt::{self, Write};
 use core::ops::{self, Deref, Index};
rust/kernel/sync/arc.rs +2 -2
index 0866378f1360..c2a3a2c7cbc5 100644
@@ -16,7 +16,7 @@
 //! [`Arc`]: https://doc.rust-lang.org/std/sync/struct.Arc.html
 
 use crate::{
-    alloc::{box_ext::BoxExt, Flags},
+    alloc::{box_ext::BoxExt, AllocError, Flags},
     bindings,
     error::{self, Error},
     init::{self, InPlaceInit, Init, PinInit},
@@ -25,7 +25,7 @@
 };
 use alloc::boxed::Box;
 use core::{
-    alloc::{AllocError, Layout},
+    alloc::Layout,
     fmt,
     marker::{PhantomData, Unsize},
     mem::{ManuallyDrop, MaybeUninit},
rust/kernel/workqueue.rs +1 -2
index 4d1f5306497a..dfbf1080ccec 100644
@@ -132,9 +132,8 @@
 //!
 //! C header: [`include/linux/workqueue.h`](srctree/include/linux/workqueue.h)
 
-use crate::alloc::Flags;
+use crate::alloc::{AllocError, Flags};
 use crate::{bindings, prelude::*, sync::Arc, sync::LockClassKey, types::Opaque};
-use alloc::alloc::AllocError;
 use alloc::boxed::Box;
 use core::marker::PhantomData;
 use core::pin::Pin;
Signature · 1 line
2.34.1
#15 lore raw
In reply to Wedson Almeida Filho
Quoted 20 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

Quoted 7 lines

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

---

FWIW, I've put this into rust-dev:

https://github.com/Rust-for-Linux/linux rust-dev

a few adjustments are needed to work with other commits that have been queued in rust-dev, so I add an commit on the top for everyone's reference. (Besides this commit, I also removed all updates to our own alloc in Miguel's 1.77.0 compiler version bump patch)

Regards,
Boqun

---------------------------------------------------->8

rust/kernel/file.rs +10 -5
index 8902f490ccc8..a5930f0f2bc5 100644
@@ -6,13 +6,15 @@
 //! [`include/linux/file.h`](srctree/include/linux/file.h)
 
 use crate::{
+    alloc::AllocError,
     bindings,
     cred::Credential,
     error::{code::*, Error, Result},
+    prelude::*,
     types::{ARef, AlwaysRefCounted, NotThreadSafe, Opaque},
 };
 use alloc::boxed::Box;
-use core::{alloc::AllocError, mem, ptr};
+use core::{mem, ptr};
 
 /// Flags associated with a [`File`].
 pub mod flags {
@@ -348,10 +350,13 @@ impl DeferredFdCloser {
     pub fn new() -> Result<Self, AllocError> {
         Ok(Self {
             // INVARIANT: The `file` pointer is null, so the type invariant does not apply.
-            inner: Box::try_new(DeferredFdCloserInner {
-                twork: mem::MaybeUninit::uninit(),
-                file: core::ptr::null_mut(),
-            })?,
+            inner: Box::new(
+                DeferredFdCloserInner {
+                    twork: mem::MaybeUninit::uninit(),
+                    file: core::ptr::null_mut(),
+                },
+                GFP_KERNEL,
+            )?,
         })
     }
 
rust/kernel/sync/arc.rs +2 -2
index c4a5e175b574..13a2166c4f41 100644
@@ -302,7 +302,7 @@ pub fn ptr_eq(this: &Self, other: &Self) -> bool {
     /// ```
     /// use kernel::sync::{Arc, UniqueArc};
     ///
-    /// let arc = Arc::try_new(42)?;
+    /// let arc = Arc::new(42, GFP_KERNEL)?;
     /// let unique_arc = arc.into_unique_or_drop();
     ///
     /// // The above conversion should succeed since refcount of `arc` is 1.
@@ -316,7 +316,7 @@ pub fn ptr_eq(this: &Self, other: &Self) -> bool {
     /// ```
     /// use kernel::sync::{Arc, UniqueArc};
     ///
-    /// let arc = Arc::try_new(42)?;
+    /// let arc = Arc::new(42, GFP_KERNEL)?;
     /// let another = arc.clone();
     ///
     /// let unique_arc = arc.into_unique_or_drop();
#16 lore raw
Quoted 33 lines · Boqun Feng
Quoted 20 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

Quoted 7 lines

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

---

FWIW, I've put this into rust-dev:

https://github.com/Rust-for-Linux/linux rust-dev

Quoted 7 lines

a few adjustments are needed to work with other commits that have been queued in rust-dev, so I add an commit on the top for everyone's reference. (Besides this commit, I also removed all updates to our own alloc in Miguel's 1.77.0 compiler version bump patch)

Regards,
Boqun

As Miguel pointed out offline, the following is also needed:

rust/kernel/file.rs +1 -1
index a5930f0f2bc5..bf68a0ce9f14 100644
@@ -350,7 +350,7 @@ impl DeferredFdCloser {
     pub fn new() -> Result<Self, AllocError> {
         Ok(Self {
             // INVARIANT: The `file` pointer is null, so the type invariant does not apply.
-            inner: Box::new(
+            inner: <Box<_> as BoxExt<_>>::new(
                 DeferredFdCloserInner {
                     twork: mem::MaybeUninit::uninit(),
                     file: core::ptr::null_mut(),
#17 lore raw
In reply to Boqun Feng
Quoted 48 lines · Boqun Feng
Quoted 20 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

Quoted 7 lines

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

---

FWIW, I've put this into rust-dev:

https://github.com/Rust-for-Linux/linux rust-dev

a few adjustments are needed to work with other commits that have been queued in rust-dev, so I add an commit on the top for everyone's reference. (Besides this commit, I also removed all updates to our own alloc in Miguel's 1.77.0 compiler version bump patch)

Regards,
Boqun

---------------------------------------------------->8

rust/kernel/file.rs +0 -0
index 8902f490ccc8..a5930f0f2bc5 100644

rust/kernel/file.rs line 9

Quoted 4 lines
rust/kernel/file.rs +1 -0
@@ -6,13 +6,15 @@
 //! [`include/linux/file.h`](srctree/include/linux/file.h)
 
 use crate::{
+    alloc::AllocError,

I would do alloc::{flags::*, AlocError}, here and add not add prelude::* below.

Quoted 52 lines
rust/kernel/file.rs +9 -5
     bindings,
     cred::Credential,
     error::{code::*, Error, Result},
+    prelude::*,
     types::{ARef, AlwaysRefCounted, NotThreadSafe, Opaque},
 };
 use alloc::boxed::Box;
-use core::{alloc::AllocError, mem, ptr};
+use core::{mem, ptr};
 
 /// Flags associated with a [`File`].
 pub mod flags {
@@ -348,10 +350,13 @@ impl DeferredFdCloser {
     pub fn new() -> Result<Self, AllocError> {
         Ok(Self {
             // INVARIANT: The `file` pointer is null, so the type invariant does not apply.
-            inner: Box::try_new(DeferredFdCloserInner {
-                twork: mem::MaybeUninit::uninit(),
-                file: core::ptr::null_mut(),
-            })?,
+            inner: Box::new(
+                DeferredFdCloserInner {
+                    twork: mem::MaybeUninit::uninit(),
+                    file: core::ptr::null_mut(),
+                },
+                GFP_KERNEL,
+            )?,
         })
     }
 
rust/kernel/sync/arc.rs +2 -2
index c4a5e175b574..13a2166c4f41 100644
@@ -302,7 +302,7 @@ pub fn ptr_eq(this: &Self, other: &Self) -> bool {
     /// ```
     /// use kernel::sync::{Arc, UniqueArc};
     ///
-    /// let arc = Arc::try_new(42)?;
+    /// let arc = Arc::new(42, GFP_KERNEL)?;
     /// let unique_arc = arc.into_unique_or_drop();
     ///
     /// // The above conversion should succeed since refcount of `arc` is 1.
@@ -316,7 +316,7 @@ pub fn ptr_eq(this: &Self, other: &Self) -> bool {
     /// ```
     /// use kernel::sync::{Arc, UniqueArc};
     ///
-    /// let arc = Arc::try_new(42)?;
+    /// let arc = Arc::new(42, GFP_KERNEL)?;
     /// let another = arc.clone();
     ///
     /// let unique_arc = arc.into_unique_or_drop();
#18 lore raw
In reply to Wedson Almeida Filho
Quoted 9 lines · On Wed, 27 Mar 2024 22:35:53 -0300 Wedson Almeida Filho <[email protected]>…

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Quoted 5 lines

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

IMO, It seems the allocation flag here means GFP flags according to Patch 5 and I understand the benefit of introducing the flags.

I am interested in the future plan. With this change, will Vec and Box stick to kernel memory application APIs with GFP flags in the future? e.g. GUP, kmalloc, those mostly allocates continuous memory for small objects. Is that the future for Vec and Box in kernel?

Is there any plan for having vmalloc() in rust kernel? Currently, if I push a large object into a Vec, kernel MM will complain for sure. And literally Vec/Box themselves don't imply to the user that they allocate memory with limitations.

Kernel uses different MM alloc APIs for different usages. For rust, should we have a different kind of "Vec/Box" or having a Vec/Box with different set of allocation flags that covers both GFP MM APIs and vmalloc()? I am curious about the plan.

Quoted 113 lines

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

---

Changes in v3:
- Rebased on top of the latest rust-next branch.
- Updated krealloc_aligned to use Flags instead of bindings::gfp_t.
- Added __GFP_ZERO to flags, as part of the previous change.
- Avoiding temporary stack value in Box::new_uninit.
- Implement Box::new using Box::new_uninit (so only one of them actually allocates).
- Added examples/tests to VecExt methods.
- Fixed bug in length in extend_from_slice
- Link to v2:
https://lore.kernel.org/rust-for-linux/[email protected]/T/#t

Changes in v2:
- Updated description of alloc crate.
- Renamed vecext and boxext modules to vec_ext and box_ext.
- Added derive directive to AllocError.
- Updated safety comment in BoxExt::new.
- Updated VecExt::push and VecExt::extend_from_slice to use spare_capacity_mut
- Added directive to not compile destructure and rebuild when test or testlib are configured. Otherwise we have a warning because push and extend_from_slice don't use them anymore.
- Updated indentation in Arc::new_uninit
- Moved the removal of TryReserveError convesion to Error to patch 7, where usage of TryReserveError is actually removed.
- Link to v1:
https://lore.kernel.org/rust-for-linux/[email protected]/T/#t

Wedson Almeida Filho (10):
rust: kernel: move allocator module under alloc
rust: alloc: introduce the VecExt trait
kbuild: use the upstream alloc crate
rust: alloc: remove our fork of the alloc crate
rust: alloc: introduce allocation flags
rust: alloc: introduce the BoxExt trait
rust: alloc: update VecExt to take allocation flags
rust: sync: update Arc and UniqueArc to take allocation flags
rust: init: update init module to take allocation flags
rust: kernel: remove usage of allocator_api unstable feature

 rust/Makefile                        |   16 +-
 rust/alloc/README.md                 |   36 -
 rust/alloc/alloc.rs                  |  452 ----
 rust/alloc/boxed.rs                  | 2463 -----------------
 rust/alloc/collections/mod.rs        |  160 --
 rust/alloc/lib.rs                    |  288 --
 rust/alloc/raw_vec.rs                |  611 -----
 rust/alloc/slice.rs                  |  890 -------
 rust/alloc/vec/drain.rs              |  255 --
 rust/alloc/vec/extract_if.rs         |  115 -
 rust/alloc/vec/into_iter.rs          |  454 ----
 rust/alloc/vec/is_zero.rs            |  204 --
 rust/alloc/vec/mod.rs                | 3683
-------------------------- rust/alloc/vec/partial_eq.rs         |
49 - rust/alloc/vec/set_len_on_drop.rs    |   35 -
 rust/alloc/vec/spec_extend.rs        |  119 -
 rust/bindings/bindings_helper.h      |    3 +
 rust/kernel/alloc.rs                 |   74 +
 rust/kernel/{ => alloc}/allocator.rs |   17 +-
 rust/kernel/alloc/box_ext.rs         |   59 +
 rust/kernel/alloc/vec_ext.rs         |  176 ++
 rust/kernel/error.rs                 |   13 +-
 rust/kernel/init.rs                  |   57 +-
 rust/kernel/lib.rs                   |    5 +-
 rust/kernel/prelude.rs               |    2 +
 rust/kernel/str.rs                   |    6 +-
 rust/kernel/sync/arc.rs              |   50 +-
 rust/kernel/sync/condvar.rs          |    2 +-
 rust/kernel/sync/lock/mutex.rs       |    2 +-
 rust/kernel/sync/lock/spinlock.rs    |    2 +-
 rust/kernel/types.rs                 |    4 +-
 rust/kernel/workqueue.rs             |   14 +-
 samples/rust/rust_minimal.rs         |    6 +-
 samples/rust/rust_print.rs           |    4 +-
 scripts/generate_rust_analyzer.py    |    2 +-
 35 files changed, 405 insertions(+), 9923 deletions(-)
 delete mode 100644 rust/alloc/README.md
 delete mode 100644 rust/alloc/alloc.rs
 delete mode 100644 rust/alloc/boxed.rs
 delete mode 100644 rust/alloc/collections/mod.rs
 delete mode 100644 rust/alloc/lib.rs
 delete mode 100644 rust/alloc/raw_vec.rs
 delete mode 100644 rust/alloc/slice.rs
 delete mode 100644 rust/alloc/vec/drain.rs
 delete mode 100644 rust/alloc/vec/extract_if.rs
 delete mode 100644 rust/alloc/vec/into_iter.rs
 delete mode 100644 rust/alloc/vec/is_zero.rs
 delete mode 100644 rust/alloc/vec/mod.rs
 delete mode 100644 rust/alloc/vec/partial_eq.rs
 delete mode 100644 rust/alloc/vec/set_len_on_drop.rs
 delete mode 100644 rust/alloc/vec/spec_extend.rs
 create mode 100644 rust/kernel/alloc.rs
 rename rust/kernel/{ => alloc}/allocator.rs (86%)
 create mode 100644 rust/kernel/alloc/box_ext.rs
 create mode 100644 rust/kernel/alloc/vec_ext.rs

base-commit: 768409cff6cc89fe1194da880537a09857b6e4db

#19 lore raw
Quoted 14 lines · Zhi Wang

IMO, It seems the allocation flag here means GFP flags according to Patch 5 and I understand the benefit of introducing the flags.

I am interested in the future plan. With this change, will Vec and Box stick to kernel memory application APIs with GFP flags in the future? e.g. GUP, kmalloc, those mostly allocates continuous memory for small objects. Is that the future for Vec and Box in kernel?

Is there any plan for having vmalloc() in rust kernel? Currently, if I push a large object into a Vec, kernel MM will complain for sure. And literally Vec/Box themselves don't imply to the user that they allocate memory with limitations.

Quoted 4 lines

Kernel uses different MM alloc APIs for different usages. For rust, should we have a different kind of "Vec/Box" or having a Vec/Box with different set of allocation flags that covers both GFP MM APIs and vmalloc()? I am curious about the plan.

Sorry Zhi, Danilo brought this message to my attention today (thanks!).

We have been trying to find the best way forward to support fallible allocations, per-call-site flags, other kernel allocators and so on for a long time, including discussions with upstream Rust.

At this point, after these years, I think the ideal path is to try our best to avoid re-enabling allocator_api and to have perhaps our own custom APIs/types as needed (possibly several depending on the use case) -- please also see my reply at
https://lore.kernel.org/rust-for-linux/CANiq72=0BNw-KiURBjosLqfuUEPpjZPbRg1XMFZyobOzBt7aMA@mail.gmail.com/

Regarding this patch series, we mainly wanted to make progress on (finally) allowing at least to use the GFP flags and also, importantly, dropping alloc since we want to start supporting several compiler versions soon.

Thank you for your feedback on this. It would be nice to hear what you think about Danilo's approach in the other series and other possible approaches that may not rely on allocator_api.

Cheers,
Miguel

Applied
#20 lore raw
In reply to Wedson Almeida Filho
Quoted 17 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Quoted 8 lines

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

Applied to rust-next -- thanks everyone!

[ Used Box::write() to avoid one unsafe block as suggested by Boqun. ]

Cheers,
Miguel

#21 lore raw
In reply to Wedson Almeida Filho

Hi all,

Quoted 16 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Quoted 8 lines

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

With that series, how do we implement alternative allocators, such as (k)vmalloc or DMA coherent?

For instance, I recently sketched up some firmware bindings we want to use in Nova providing

fn copy<A: core::alloc::Allocator>(&self, alloc: A) -> Result<Vec<u8, A>> [1]

making use of Vec::try_with_capacity_in(). How would I implement something similar now?

- Danilo

[1] https://gitlab.freedesktop.org/drm/nova/-/blob/topic/firmware/rust/kernel/firmware.rs?ref_type=heads#L63

Quoted 99 lines

---

Changes in v3:
- Rebased on top of the latest rust-next branch.
- Updated krealloc_aligned to use Flags instead of bindings::gfp_t.
- Added __GFP_ZERO to flags, as part of the previous change.
- Avoiding temporary stack value in Box::new_uninit.
- Implement Box::new using Box::new_uninit (so only one of them actually allocates).
- Added examples/tests to VecExt methods.
- Fixed bug in length in extend_from_slice
- Link to v2: https://lore.kernel.org/rust-for-linux/[email protected]/T/#t

Changes in v2:
- Updated description of alloc crate.
- Renamed vecext and boxext modules to vec_ext and box_ext.
- Added derive directive to AllocError.
- Updated safety comment in BoxExt::new.
- Updated VecExt::push and VecExt::extend_from_slice to use
spare_capacity_mut
- Added directive to not compile destructure and rebuild when test or testlib are configured. Otherwise we have a warning because push and extend_from_slice don't use them anymore.
- Updated indentation in Arc::new_uninit
- Moved the removal of TryReserveError convesion to Error to patch 7, where usage of TryReserveError is actually removed.
- Link to v1: https://lore.kernel.org/rust-for-linux/[email protected]/T/#t

Wedson Almeida Filho (10):
rust: kernel: move allocator module under alloc
rust: alloc: introduce the VecExt trait
kbuild: use the upstream alloc crate
rust: alloc: remove our fork of the alloc crate
rust: alloc: introduce allocation flags
rust: alloc: introduce the BoxExt trait
rust: alloc: update VecExt to take allocation flags
rust: sync: update Arc and UniqueArc to take allocation flags
rust: init: update init module to take allocation flags
rust: kernel: remove usage of allocator_api unstable feature

  rust/Makefile                        |   16 +-
  rust/alloc/README.md                 |   36 -
  rust/alloc/alloc.rs                  |  452 ----
  rust/alloc/boxed.rs                  | 2463 -----------------
  rust/alloc/collections/mod.rs        |  160 --
  rust/alloc/lib.rs                    |  288 --
  rust/alloc/raw_vec.rs                |  611 -----
  rust/alloc/slice.rs                  |  890 -------
  rust/alloc/vec/drain.rs              |  255 --
  rust/alloc/vec/extract_if.rs         |  115 -
  rust/alloc/vec/into_iter.rs          |  454 ----
  rust/alloc/vec/is_zero.rs            |  204 --
  rust/alloc/vec/mod.rs                | 3683 --------------------------
  rust/alloc/vec/partial_eq.rs         |   49 -
  rust/alloc/vec/set_len_on_drop.rs    |   35 -
  rust/alloc/vec/spec_extend.rs        |  119 -
  rust/bindings/bindings_helper.h      |    3 +
  rust/kernel/alloc.rs                 |   74 +
  rust/kernel/{ => alloc}/allocator.rs |   17 +-
  rust/kernel/alloc/box_ext.rs         |   59 +
  rust/kernel/alloc/vec_ext.rs         |  176 ++
  rust/kernel/error.rs                 |   13 +-
  rust/kernel/init.rs                  |   57 +-
  rust/kernel/lib.rs                   |    5 +-
  rust/kernel/prelude.rs               |    2 +
  rust/kernel/str.rs                   |    6 +-
  rust/kernel/sync/arc.rs              |   50 +-
  rust/kernel/sync/condvar.rs          |    2 +-
  rust/kernel/sync/lock/mutex.rs       |    2 +-
  rust/kernel/sync/lock/spinlock.rs    |    2 +-
  rust/kernel/types.rs                 |    4 +-
  rust/kernel/workqueue.rs             |   14 +-
  samples/rust/rust_minimal.rs         |    6 +-
  samples/rust/rust_print.rs           |    4 +-
  scripts/generate_rust_analyzer.py    |    2 +-
  35 files changed, 405 insertions(+), 9923 deletions(-)

delete mode 100644 rust/alloc/README.md delete mode 100644 rust/alloc/alloc.rs delete mode 100644 rust/alloc/boxed.rs delete mode 100644 rust/alloc/collections/mod.rs delete mode 100644 rust/alloc/lib.rs
delete mode 100644 rust/alloc/raw_vec.rs delete mode 100644 rust/alloc/slice.rs delete mode 100644 rust/alloc/vec/drain.rs delete mode 100644 rust/alloc/vec/extract_if.rs delete mode 100644 rust/alloc/vec/into_iter.rs delete mode 100644 rust/alloc/vec/is_zero.rs delete mode 100644 rust/alloc/vec/mod.rs delete mode 100644 rust/alloc/vec/partial_eq.rs delete mode 100644 rust/alloc/vec/set_len_on_drop.rs delete mode 100644 rust/alloc/vec/spec_extend.rs create mode 100644 rust/kernel/alloc.rs rename rust/kernel/{ => alloc}/allocator.rs (86%) create mode 100644 rust/kernel/alloc/box_ext.rs create mode 100644 rust/kernel/alloc/vec_ext.rs

base-commit: 768409cff6cc89fe1194da880537a09857b6e4db

#22 lore raw

(adding folks from [1])

Quoted 31 lines · Danilo Krummrich

Hi all,

Quoted 16 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Quoted 8 lines

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

With that series, how do we implement alternative allocators, such as (k)vmalloc or DMA coherent?

Quoted 8 lines

For instance, I recently sketched up some firmware bindings we want to use in Nova providing

fn copy<A: core::alloc::Allocator>(&self, alloc: A) -> Result<Vec<u8, A>>
[1]

making use of Vec::try_with_capacity_in(). How would I implement something similar now?

I want to follow up on this topic after also bringing it up in yesterday's weekly Rust call.

In the call a few ideas were discussed, e.g. whether we could just re-enable the allocator_api feature and try getting it stabilized.

With the introduction of alloc::Flags (gfp_t abstraction) allocator_api might not be a viable choice anymore.

I think it would work for (k)vmalloc, where we could pass the page flags through const generics for instance.

But I don't see how it could work with kmem_cache, where we can't just create a new allocator instance when we want to change the page flags, but need to support allocations with different page flags on the same allocator (same kmem_cache) instance.

So, I think we have to create our own allocator trait / API.

Any other thoughts on that?

- Danilo

[1] https://lore.kernel.org/rust-for-linux/[email protected]/

Quoted 104 lines

- Danilo

[1] https://gitlab.freedesktop.org/drm/nova/-/blob/topic/firmware/rust/kernel/firmware.rs?ref_type=heads#L63

Quoted 99 lines

---

Changes in v3:
- Rebased on top of the latest rust-next branch.
- Updated krealloc_aligned to use Flags instead of bindings::gfp_t.
- Added __GFP_ZERO to flags, as part of the previous change.
- Avoiding temporary stack value in Box::new_uninit.
- Implement Box::new using Box::new_uninit (so only one of them actually allocates).
- Added examples/tests to VecExt methods.
- Fixed bug in length in extend_from_slice
- Link to v2: https://lore.kernel.org/rust-for-linux/[email protected]/T/#t

Changes in v2:
- Updated description of alloc crate.
- Renamed vecext and boxext modules to vec_ext and box_ext.
- Added derive directive to AllocError.
- Updated safety comment in BoxExt::new.
- Updated VecExt::push and VecExt::extend_from_slice to use
spare_capacity_mut
- Added directive to not compile destructure and rebuild when test or testlib are configured. Otherwise we have a warning because push and extend_from_slice don't use them anymore.
- Updated indentation in Arc::new_uninit
- Moved the removal of TryReserveError convesion to Error to patch 7, where usage of TryReserveError is actually removed.
- Link to v1: https://lore.kernel.org/rust-for-linux/[email protected]/T/#t

Wedson Almeida Filho (10):
rust: kernel: move allocator module under alloc
rust: alloc: introduce the VecExt trait
kbuild: use the upstream alloc crate
rust: alloc: remove our fork of the alloc crate
rust: alloc: introduce allocation flags
rust: alloc: introduce the BoxExt trait
rust: alloc: update VecExt to take allocation flags
rust: sync: update Arc and UniqueArc to take allocation flags
rust: init: update init module to take allocation flags
rust: kernel: remove usage of allocator_api unstable feature

  rust/Makefile                        |   16 +-
  rust/alloc/README.md                 |   36 -
  rust/alloc/alloc.rs                  |  452 ----
  rust/alloc/boxed.rs                  | 2463 -----------------
  rust/alloc/collections/mod.rs        |  160 --
  rust/alloc/lib.rs                    |  288 --
  rust/alloc/raw_vec.rs                |  611 -----
  rust/alloc/slice.rs                  |  890 -------
  rust/alloc/vec/drain.rs              |  255 --
  rust/alloc/vec/extract_if.rs         |  115 -
  rust/alloc/vec/into_iter.rs          |  454 ----
  rust/alloc/vec/is_zero.rs            |  204 --
  rust/alloc/vec/mod.rs                | 3683 --------------------------
  rust/alloc/vec/partial_eq.rs         |   49 -
  rust/alloc/vec/set_len_on_drop.rs    |   35 -
  rust/alloc/vec/spec_extend.rs        |  119 -
  rust/bindings/bindings_helper.h      |    3 +
  rust/kernel/alloc.rs                 |   74 +
  rust/kernel/{ => alloc}/allocator.rs |   17 +-
  rust/kernel/alloc/box_ext.rs         |   59 +
  rust/kernel/alloc/vec_ext.rs         |  176 ++
  rust/kernel/error.rs                 |   13 +-
  rust/kernel/init.rs                  |   57 +-
  rust/kernel/lib.rs                   |    5 +-
  rust/kernel/prelude.rs               |    2 +
  rust/kernel/str.rs                   |    6 +-
  rust/kernel/sync/arc.rs              |   50 +-
  rust/kernel/sync/condvar.rs          |    2 +-
  rust/kernel/sync/lock/mutex.rs       |    2 +-
  rust/kernel/sync/lock/spinlock.rs    |    2 +-
  rust/kernel/types.rs                 |    4 +-
  rust/kernel/workqueue.rs             |   14 +-
  samples/rust/rust_minimal.rs         |    6 +-
  samples/rust/rust_print.rs           |    4 +-
  scripts/generate_rust_analyzer.py    |    2 +-
  35 files changed, 405 insertions(+), 9923 deletions(-)

delete mode 100644 rust/alloc/README.md delete mode 100644 rust/alloc/alloc.rs delete mode 100644 rust/alloc/boxed.rs delete mode 100644 rust/alloc/collections/mod.rs delete mode 100644 rust/alloc/lib.rs
delete mode 100644 rust/alloc/raw_vec.rs delete mode 100644 rust/alloc/slice.rs delete mode 100644 rust/alloc/vec/drain.rs delete mode 100644 rust/alloc/vec/extract_if.rs delete mode 100644 rust/alloc/vec/into_iter.rs delete mode 100644 rust/alloc/vec/is_zero.rs delete mode 100644 rust/alloc/vec/mod.rs delete mode 100644 rust/alloc/vec/partial_eq.rs delete mode 100644 rust/alloc/vec/set_len_on_drop.rs delete mode 100644 rust/alloc/vec/spec_extend.rs create mode 100644 rust/kernel/alloc.rs rename rust/kernel/{ => alloc}/allocator.rs (86%) create mode 100644 rust/kernel/alloc/box_ext.rs create mode 100644 rust/kernel/alloc/vec_ext.rs

base-commit: 768409cff6cc89fe1194da880537a09857b6e4db

#23 lore raw
Quoted 43 lines · Danilo Krummrich

(adding folks from [1])

Quoted 31 lines · Danilo Krummrich

Hi all,

Quoted 16 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Quoted 8 lines

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

With that series, how do we implement alternative allocators, such as (k)vmalloc or DMA coherent?

Quoted 8 lines

For instance, I recently sketched up some firmware bindings we want to use in Nova providing

fn copy<A: core::alloc::Allocator>(&self, alloc: A) -> Result<Vec<u8, A>>
[1]

making use of Vec::try_with_capacity_in(). How would I implement something similar now?

Quoted 8 lines

I want to follow up on this topic after also bringing it up in yesterday's weekly Rust call.

In the call a few ideas were discussed, e.g. whether we could just re-enable the allocator_api feature and try getting it stabilized.

With the introduction of alloc::Flags (gfp_t abstraction) allocator_api might not be a viable choice anymore.

Bringing in some more context from the meeting: Gary suggested we create a custom trait for allocators that can also handle allocation flags:

     pub trait AllocatorWithFlags: Allocator {
         type Flags;
         
         fn allocate_with_flags(&self, layout: Layout, flags: Self::Flags) -> Result<NonNull<[u8]>, AllocError>;

         /* ... */
     }
     
     impl AllocatorWithFlags for Global { /* ... */ }
     
     impl<T, A> VecExt<T> for Vec<T, A> where A: AllocatorWithFlags {
         /* ... */
     }

I think that this would work, but we would have to ensure that users are only allowed to call allocating functions if they are functions that we control. For example Vec::try_reserve [1] would still use the normal Allocator trait that doesn't support our flags.
Gary noted that this could be solved by klint [2].

But we only need to extend the allocator API, if you want to use the std library types that allocate. If you would also be happy with a custom newtype wrapper, then we could also do that.
I think that we probably want a more general solution (ie Allocator enriched with flags), but we would have to design that before you can use it.

[1]: https://doc.rust-lang.org/alloc/vec/struct.Vec.html#method.try_reserve
[2]: https://github.com/Rust-for-Linux/klint

Quoted 8 lines

I think it would work for (k)vmalloc, where we could pass the page flags through const generics for instance.

But I don't see how it could work with kmem_cache, where we can't just create a new allocator instance when we want to change the page flags, but need to support allocations with different page flags on the same allocator (same kmem_cache) instance.

I think that you can write the kmem_cache abstraction without using the allocator api. You just give the function that allocates a flags argument like in C.

The Allocator API might make it more *convenient* to use it, because you don't have to explicitly pass the flags every time (since the flags are determined by the allocator). But I have also heard that it might be desirable to always be explicit.

Signature · 2 lines
Cheers,
Benno
Quoted 9 lines

So, I think we have to create our own allocator trait / API.

Any other thoughts on that?

- Danilo

[1] https://lore.kernel.org/rust-for-linux/[email protected]/

#24 lore raw
Quoted 81 lines · On Thu, 25 Apr 2024 16:09:46 +0000 Benno Lossin <[email protected]> wrote:
Quoted 45 lines · Danilo Krummrich

(adding folks from [1])

Quoted 33 lines · Danilo Krummrich

Hi all,

Quoted 18 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Quoted 8 lines

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

With that series, how do we implement alternative allocators, such as (k)vmalloc or DMA coherent?

Quoted 8 lines

For instance, I recently sketched up some firmware bindings we want to use in Nova providing

fn copy<A: core::alloc::Allocator>(&self, alloc: A) -> Result<Vec<u8, A>> [1]

making use of Vec::try_with_capacity_in(). How would I implement something similar now?

Quoted 8 lines

I want to follow up on this topic after also bringing it up in yesterday's weekly Rust call.

In the call a few ideas were discussed, e.g. whether we could just re-enable the allocator_api feature and try getting it stabilized.

With the introduction of alloc::Flags (gfp_t abstraction) allocator_api might not be a viable choice anymore.

Bringing in some more context from the meeting: Gary suggested we create a custom trait for allocators that can also handle allocation flags:

     pub trait AllocatorWithFlags: Allocator {
         type Flags;
         
         fn allocate_with_flags(&self, layout: Layout, flags:

Self::Flags) -> Result<NonNull<[u8]>, AllocError>;

         /* ... */
     }
     
     impl AllocatorWithFlags for Global { /* ... */ }
     
     impl<T, A> VecExt<T> for Vec<T, A> where A: AllocatorWithFlags {
         /* ... */
     }

I think that this would work, but we would have to ensure that users are only allowed to call allocating functions if they are functions that we control. For example Vec::try_reserve [1] would still use the normal Allocator trait that doesn't support our flags. Gary noted that this could be solved by klint [2].

Quoted 7 lines

But we only need to extend the allocator API, if you want to use the std library types that allocate. If you would also be happy with a custom newtype wrapper, then we could also do that.
I think that we probably want a more general solution (ie Allocator enriched with flags), but we would have to design that before you can use it.

I agree that we should have a trait for allocator API. I think the purpose of the trait is serving different upper-layer memory allocation APIs that wants to have Vec/Box kind-like methods.

Look at the rust DMA memory allocation nowadays, there is already a similar kind of of allocator with similar requirements. They might sit on the same allocator API and implement traits with their kernel memory allocation APIs if the trait is properly defined.

I think it is essential for a kernel driver to know the essence of the heap that a Vec/Box is using. Hiding it away from the driver doesn't look promising to me.

For wrapping the kernel memory allocation APIs, it can be done by either having one unique Vec/Box API with different flags or having different kind of xxVec/xxBox (Maybe it doesn't even need to be named as xxVec or xxBox). Personally, I prefer the later approach that is identical to the current kernel memory allocation APIs and more straight-forward.

Thanks,
Zhi.

Quoted 23 lines

[1]:
https://doc.rust-lang.org/alloc/vec/struct.Vec.html#method.try_reserve
[2]: https://github.com/Rust-for-Linux/klint

Quoted 8 lines

I think it would work for (k)vmalloc, where we could pass the page flags through const generics for instance.

But I don't see how it could work with kmem_cache, where we can't just create a new allocator instance when we want to change the page flags, but need to support allocations with different page flags on the same allocator (same kmem_cache) instance.

I think that you can write the kmem_cache abstraction without using the allocator api. You just give the function that allocates a flags argument like in C.

The Allocator API might make it more *convenient* to use it, because you don't have to explicitly pass the flags every time (since the flags are determined by the allocator). But I have also heard that it might be desirable to always be explicit.

#25 lore raw
In reply to Benno Lossin
Quoted 70 lines · Benno Lossin
Quoted 43 lines · Danilo Krummrich

(adding folks from [1])

Quoted 31 lines · Danilo Krummrich

Hi all,

Quoted 16 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Quoted 8 lines

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

With that series, how do we implement alternative allocators, such as (k)vmalloc or DMA coherent?

Quoted 8 lines

For instance, I recently sketched up some firmware bindings we want to use in Nova providing

fn copy<A: core::alloc::Allocator>(&self, alloc: A) -> Result<Vec<u8, A>>
[1]

making use of Vec::try_with_capacity_in(). How would I implement something similar now?

Quoted 8 lines

I want to follow up on this topic after also bringing it up in yesterday's weekly Rust call.

In the call a few ideas were discussed, e.g. whether we could just re-enable the allocator_api feature and try getting it stabilized.

With the introduction of alloc::Flags (gfp_t abstraction) allocator_api might not be a viable choice anymore.

Bringing in some more context from the meeting: Gary suggested we create a custom trait for allocators that can also handle allocation flags:

     pub trait AllocatorWithFlags: Allocator {
         type Flags;
         
         fn allocate_with_flags(&self, layout: Layout, flags: Self::Flags) -> Result<NonNull<[u8]>, AllocError>;

         /* ... */
     }
     
     impl AllocatorWithFlags for Global { /* ... */ }
     
     impl<T, A> VecExt<T> for Vec<T, A> where A: AllocatorWithFlags {
         /* ... */
     }
Quoted 5 lines

I think that this would work, but we would have to ensure that users are only allowed to call allocating functions if they are functions that we control. For example Vec::try_reserve [1] would still use the normal Allocator trait that doesn't support our flags.
Gary noted that this could be solved by klint [2].

I agree, extending the Allocator trait should work.

Regarding allocating functions we don't control, isn't that the case already? AFAICS, we're currently always falling back to GFP_KERNEL when calling Vec::try_reserve().

But yes, I also think it would be better to enforce being explicit.

Given that, is there any value extending the existing Allocator trait at all?

Quoted 5 lines

But we only need to extend the allocator API, if you want to use the std library types that allocate. If you would also be happy with a custom newtype wrapper, then we could also do that.

What do you mean with "custom newtype wrapper"?

Quoted 12 lines

I think that we probably want a more general solution (ie Allocator enriched with flags), but we would have to design that before you can use it.

[1]: https://doc.rust-lang.org/alloc/vec/struct.Vec.html#method.try_reserve
[2]: https://github.com/Rust-for-Linux/klint

Quoted 4 lines

I think it would work for (k)vmalloc, where we could pass the page flags through const generics for instance.

Quoted 8 lines
Quoted 4 lines

But I don't see how it could work with kmem_cache, where we can't just create a new allocator instance when we want to change the page flags, but need to support allocations with different page flags on the same allocator (same kmem_cache) instance.

I think that you can write the kmem_cache abstraction without using the allocator api. You just give the function that allocates a flags argument like in C.

Guess you mean letting the kmem_cache implementation construct the corresponding container? Something like:

KmemCache<T>::alloc_box(flags: alloc::Flags) -> Box<T>

I think that'd make a lot of sense, since the size of an allocation is fixed anyways.

Quoted 20 lines

The Allocator API might make it more *convenient* to use it, because you don't have to explicitly pass the flags every time (since the flags are determined by the allocator). But I have also heard that it might be desirable to always be explicit.

Signature · 2 lines
Cheers,
Benno
Quoted 9 lines

So, I think we have to create our own allocator trait / API.

Any other thoughts on that?

- Danilo

[1] https://lore.kernel.org/rust-for-linux/[email protected]/

#26 lore raw
Quoted 77 lines · Danilo Krummrich
Quoted 70 lines · Benno Lossin
Quoted 43 lines · Danilo Krummrich

(adding folks from [1])

Quoted 31 lines · Danilo Krummrich

Hi all,

Quoted 16 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Quoted 8 lines

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

With that series, how do we implement alternative allocators, such as (k)vmalloc or DMA coherent?

Quoted 8 lines

For instance, I recently sketched up some firmware bindings we want to use in Nova providing

fn copy<A: core::alloc::Allocator>(&self, alloc: A) -> Result<Vec<u8, A>>
[1]

making use of Vec::try_with_capacity_in(). How would I implement something similar now?

Quoted 8 lines

I want to follow up on this topic after also bringing it up in yesterday's weekly Rust call.

In the call a few ideas were discussed, e.g. whether we could just re-enable the allocator_api feature and try getting it stabilized.

With the introduction of alloc::Flags (gfp_t abstraction) allocator_api might not be a viable choice anymore.

Bringing in some more context from the meeting: Gary suggested we create a custom trait for allocators that can also handle allocation flags:

      pub trait AllocatorWithFlags: Allocator {
          type Flags;

          fn allocate_with_flags(&self, layout: Layout, flags: Self::Flags) -> Result<NonNull<[u8]>, AllocError>;

          /* ... */
      }

      impl AllocatorWithFlags for Global { /* ... */ }

      impl<T, A> VecExt<T> for Vec<T, A> where A: AllocatorWithFlags {
          /* ... */
      }
Quoted 5 lines

I think that this would work, but we would have to ensure that users are only allowed to call allocating functions if they are functions that we control. For example Vec::try_reserve [1] would still use the normal Allocator trait that doesn't support our flags.
Gary noted that this could be solved by klint [2].

Quoted 5 lines

I agree, extending the Allocator trait should work.

Regarding allocating functions we don't control, isn't that the case already? AFAICS, we're currently always falling back to GFP_KERNEL when calling Vec::try_reserve().

Yes we're falling back to that, but
1. there are currently no calls to try_reserve in tree,
2. if you use eg a vmalloc allocator, then I don't know if it would be fine to reallocate that pointer using krealloc. I assumed that that would not be OK (hence my extra care with functions outside of our control).

Quoted 3 lines

But yes, I also think it would be better to enforce being explicit.

Given that, is there any value extending the existing Allocator trait at all?

This is what I meant in the meeting by "do you really need the allocator trait?". What you lose is the ability to use Vec and Box, instead you have to use your own wrapper types (see below). But what you gain is freedom to experiment. In the end we should still try to upstream our findings to Rust or at least share our knowledge, but doing that from the get-go is not ideal for productivity.

Quoted 5 lines
Quoted 3 lines

But we only need to extend the allocator API, if you want to use the std library types that allocate. If you would also be happy with a custom newtype wrapper, then we could also do that.

What do you mean with "custom newtype wrapper"?

You create a newtype struct ("newtype" means that it wraps an inner type and adds/removes/changes features from the inner type):

     pub struct BigVec<T>(Vec<T>);

And then you implement the common operations on it:

     impl<T> BigVec<T> {
         pub fn push(&mut self, item: T) -> Result {
             self.reserve(1)?;

             self.0.spare_capacity_mut()[0].write(item);

             // SAFETY: <omitted for brevity>
             unsafe { self.0.set_len(self.0.len() + 1) };
             Ok(())
         }

         pub fn reserve(&mut self, additional: usize) -> Result {
             /*
              * implemented like `VecExt::reserve` from this patchset,
              * except that it uses `vmalloc` instead of `krealloc`.
              */
         }
     }

If we need several of these, we can also create a general API that makes it easier to create them and avoids the duplication.

Quoted 21 lines
Quoted 12 lines

I think that we probably want a more general solution (ie Allocator enriched with flags), but we would have to design that before you can use it.

[1]: https://doc.rust-lang.org/alloc/vec/struct.Vec.html#method.try_reserve
[2]: https://github.com/Rust-for-Linux/klint

Quoted 4 lines

I think it would work for (k)vmalloc, where we could pass the page flags through const generics for instance.

Quoted 8 lines
Quoted 4 lines

But I don't see how it could work with kmem_cache, where we can't just create a new allocator instance when we want to change the page flags, but need to support allocations with different page flags on the same allocator (same kmem_cache) instance.

I think that you can write the kmem_cache abstraction without using the allocator api. You just give the function that allocates a flags argument like in C.

Quoted 7 lines

Guess you mean letting the kmem_cache implementation construct the corresponding container? Something like:

KmemCache<T>::alloc_box(flags: alloc::Flags) -> Box<T>

I think that'd make a lot of sense, since the size of an allocation is fixed anyways.

Yes, but you would probably need a newtype return type, since you need to control how it is being freed. Also in the example above there is no kmem_cache object that stores the state.

I think that the API would look more like this:

     impl<T> KMemCache<T> {
         pub fn alloc(&self, value: T, flags: Flags) -> Result<KMemObj<'_, T>>;
     }

Here are a couple of interesting elements of this API:
- I don't know if kmem_cache uses the same flags as the alloc module, if not you will need a custom Flags type.
- I assume that an object allocated by a kmem_cache is only valid while the cache still exists (ie if you drop the cache, all allocated objects are also invalidated). That is why the return type contains a lifetime ('_ refers to the elided lifetime on the &self parameter. It means that the KMemObj is only valid while the KMemCache is also valid).
- The return type is its own kind of smart pointer that allows you to modify the inner value like Box, but it takes care of all the kmem_cache specifics (eg ensuring that the associated cache is still valid, freeing the object etc).r

Since I assumed several things, in the end the API might look different, but I think that this could be a more fruitful starting point.

Signature · 2 lines
Cheers,
Benno
#27 lore raw
Quoted 84 lines · Benno Lossin
Quoted 77 lines · Danilo Krummrich
Quoted 70 lines · Benno Lossin
Quoted 43 lines · Danilo Krummrich

(adding folks from [1])

Quoted 31 lines · Danilo Krummrich

Hi all,

Quoted 16 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Quoted 8 lines

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

With that series, how do we implement alternative allocators, such as (k)vmalloc or DMA coherent?

Quoted 8 lines

For instance, I recently sketched up some firmware bindings we want to use in Nova providing

fn copy<A: core::alloc::Allocator>(&self, alloc: A) -> Result<Vec<u8, A>>
[1]

making use of Vec::try_with_capacity_in(). How would I implement something similar now?

Quoted 8 lines

I want to follow up on this topic after also bringing it up in yesterday's weekly Rust call.

In the call a few ideas were discussed, e.g. whether we could just re-enable the allocator_api feature and try getting it stabilized.

With the introduction of alloc::Flags (gfp_t abstraction) allocator_api might not be a viable choice anymore.

Bringing in some more context from the meeting: Gary suggested we create a custom trait for allocators that can also handle allocation flags:

      pub trait AllocatorWithFlags: Allocator {
          type Flags;

          fn allocate_with_flags(&self, layout: Layout, flags: Self::Flags) -> Result<NonNull<[u8]>, AllocError>;

          /* ... */
      }

      impl AllocatorWithFlags for Global { /* ... */ }

      impl<T, A> VecExt<T> for Vec<T, A> where A: AllocatorWithFlags {
          /* ... */
      }
Quoted 5 lines

I think that this would work, but we would have to ensure that users are only allowed to call allocating functions if they are functions that we control. For example Vec::try_reserve [1] would still use the normal Allocator trait that doesn't support our flags.
Gary noted that this could be solved by klint [2].

Quoted 5 lines

I agree, extending the Allocator trait should work.

Regarding allocating functions we don't control, isn't that the case already? AFAICS, we're currently always falling back to GFP_KERNEL when calling Vec::try_reserve().

Quoted 6 lines

Yes we're falling back to that, but
1. there are currently no calls to try_reserve in tree,
2. if you use eg a vmalloc allocator, then I don't know if it would be fine to reallocate that pointer using krealloc. I assumed that that would not be OK (hence my extra care with functions outside of our control).

Well, this would indeed not be valid. However, a vmalloc allocater wouldn't implement realloc() this way.

Or are you saying that Vec always uses the global allocator in that case? Why would it do that?

Quoted 7 lines
Quoted 4 lines

But yes, I also think it would be better to enforce being explicit.

Given that, is there any value extending the existing Allocator trait at all?

This is what I meant in the meeting by "do you really need the allocator trait?". What you lose is the ability to use Vec and Box, instead

Oh, indeed. I forgot about that when I wrote that. In that case I feel like it's worth extending the existing allocator_api.

Quoted 37 lines

you have to use your own wrapper types (see below). But what you gain is freedom to experiment. In the end we should still try to upstream our findings to Rust or at least share our knowledge, but doing that from the get-go is not ideal for productivity.

Quoted 5 lines
Quoted 3 lines

But we only need to extend the allocator API, if you want to use the std library types that allocate. If you would also be happy with a custom newtype wrapper, then we could also do that.

What do you mean with "custom newtype wrapper"?

You create a newtype struct ("newtype" means that it wraps an inner type and adds/removes/changes features from the inner type):

     pub struct BigVec<T>(Vec<T>);

And then you implement the common operations on it:

     impl<T> BigVec<T> {
         pub fn push(&mut self, item: T) -> Result {
             self.reserve(1)?;

             self.0.spare_capacity_mut()[0].write(item);

             // SAFETY: <omitted for brevity>
             unsafe { self.0.set_len(self.0.len() + 1) };
             Ok(())
         }

         pub fn reserve(&mut self, additional: usize) -> Result {
             /*
              * implemented like `VecExt::reserve` from this patchset,
              * except that it uses `vmalloc` instead of `krealloc`.
              */
         }
     }
Quoted 2 lines

If we need several of these, we can also create a general API that makes it easier to create them and avoids the duplication.

Thanks for for explaining.

I'd probably tend to extending allocator_api then. Do you see any major advantages / disadvantages doing one or the other?

Quoted 25 lines
Quoted 25 lines
Quoted 13 lines

I think that we probably want a more general solution (ie Allocator enriched with flags), but we would have to design that before you can use it.

[1]: https://doc.rust-lang.org/alloc/vec/struct.Vec.html#method.try_reserve
[2]: https://github.com/Rust-for-Linux/klint

Quoted 4 lines

I think it would work for (k)vmalloc, where we could pass the page flags through const generics for instance.

Quoted 8 lines
Quoted 4 lines

But I don't see how it could work with kmem_cache, where we can't just create a new allocator instance when we want to change the page flags, but need to support allocations with different page flags on the same allocator (same kmem_cache) instance.

I think that you can write the kmem_cache abstraction without using the allocator api. You just give the function that allocates a flags argument like in C.

Guess you mean letting the kmem_cache implementation construct the corresponding container? Something like:

Quoted 8 lines
Quoted 4 lines

KmemCache<T>::alloc_box(flags: alloc::Flags) -> Box<T>

I think that'd make a lot of sense, since the size of an allocation is fixed anyways.

Yes, but you would probably need a newtype return type, since you need to control how it is being freed. Also in the example above there is no kmem_cache object that stores the state.

Sure, the above was just meant to see whether I understood you correctly.

Quoted 21 lines

I think that the API would look more like this:

     impl<T> KMemCache<T> {
         pub fn alloc(&self, value: T, flags: Flags) -> Result<KMemObj<'_, T>>;
     }

Here are a couple of interesting elements of this API:
- I don't know if kmem_cache uses the same flags as the alloc module, if not you will need a custom Flags type.
- I assume that an object allocated by a kmem_cache is only valid while the cache still exists (ie if you drop the cache, all allocated objects are also invalidated). That is why the return type contains a lifetime ('_ refers to the elided lifetime on the &self parameter. It means that the KMemObj is only valid while the KMemCache is also valid).
- The return type is its own kind of smart pointer that allows you to modify the inner value like Box, but it takes care of all the kmem_cache specifics (eg ensuring that the associated cache is still valid, freeing the object etc).r

Quoted 2 lines

Since I assumed several things, in the end the API might look different, but I think that this could be a more fruitful starting point.

Thanks for the hints - your assumptions are correct. The page flags are the same.

Quoted 5 lines
Signature · 3 lines
-- 
Cheers,
Benno
#28 lore raw
Quoted 92 lines · Danilo Krummrich
Quoted 84 lines · Benno Lossin
Quoted 77 lines · Danilo Krummrich
Quoted 70 lines · Benno Lossin
Quoted 43 lines · Danilo Krummrich

(adding folks from [1])

Quoted 31 lines · Danilo Krummrich

Hi all,

Quoted 16 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Quoted 8 lines

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

With that series, how do we implement alternative allocators, such as (k)vmalloc or DMA coherent?

Quoted 8 lines

For instance, I recently sketched up some firmware bindings we want to use in Nova providing

fn copy<A: core::alloc::Allocator>(&self, alloc: A) -> Result<Vec<u8, A>>
[1]

making use of Vec::try_with_capacity_in(). How would I implement something similar now?

Quoted 8 lines

I want to follow up on this topic after also bringing it up in yesterday's weekly Rust call.

In the call a few ideas were discussed, e.g. whether we could just re-enable the allocator_api feature and try getting it stabilized.

With the introduction of alloc::Flags (gfp_t abstraction) allocator_api might not be a viable choice anymore.

Bringing in some more context from the meeting: Gary suggested we create a custom trait for allocators that can also handle allocation flags:

       pub trait AllocatorWithFlags: Allocator {
           type Flags;

           fn allocate_with_flags(&self, layout: Layout, flags: Self::Flags) -> Result<NonNull<[u8]>, AllocError>;

           /* ... */
       }

       impl AllocatorWithFlags for Global { /* ... */ }

       impl<T, A> VecExt<T> for Vec<T, A> where A: AllocatorWithFlags {
           /* ... */
       }
Quoted 5 lines

I think that this would work, but we would have to ensure that users are only allowed to call allocating functions if they are functions that we control. For example Vec::try_reserve [1] would still use the normal Allocator trait that doesn't support our flags.
Gary noted that this could be solved by klint [2].

Quoted 5 lines

I agree, extending the Allocator trait should work.

Regarding allocating functions we don't control, isn't that the case already? AFAICS, we're currently always falling back to GFP_KERNEL when calling Vec::try_reserve().

Quoted 6 lines

Yes we're falling back to that, but
1. there are currently no calls to try_reserve in tree,
2. if you use eg a vmalloc allocator, then I don't know if it would be fine to reallocate that pointer using krealloc. I assumed that that would not be OK (hence my extra care with functions outside of our control).

Quoted 2 lines

Well, this would indeed not be valid. However, a vmalloc allocater wouldn't implement realloc() this way.

Oh yeah that is correct.

Quoted 2 lines

Or are you saying that Vec always uses the global allocator in that case? Why would it do that?

No, it would use the vmalloc allocator.

So I guess the issue isn't as bad as I at first thought. I still think we should lint for this though (but maybe a warning instead of an error).

Quoted 47 lines
Quoted 6 lines
Quoted 3 lines

But yes, I also think it would be better to enforce being explicit.

Given that, is there any value extending the existing Allocator trait at all?

This is what I meant in the meeting by "do you really need the allocator trait?". What you lose is the ability to use Vec and Box, instead

Oh, indeed. I forgot about that when I wrote that. In that case I feel like it's worth extending the existing allocator_api.

Quoted 37 lines

you have to use your own wrapper types (see below). But what you gain is freedom to experiment. In the end we should still try to upstream our findings to Rust or at least share our knowledge, but doing that from the get-go is not ideal for productivity.

Quoted 5 lines
Quoted 3 lines

But we only need to extend the allocator API, if you want to use the std library types that allocate. If you would also be happy with a custom newtype wrapper, then we could also do that.

What do you mean with "custom newtype wrapper"?

You create a newtype struct ("newtype" means that it wraps an inner type and adds/removes/changes features from the inner type):

      pub struct BigVec<T>(Vec<T>);

And then you implement the common operations on it:

      impl<T> BigVec<T> {
          pub fn push(&mut self, item: T) -> Result {
              self.reserve(1)?;

              self.0.spare_capacity_mut()[0].write(item);

              // SAFETY: <omitted for brevity>
              unsafe { self.0.set_len(self.0.len() + 1) };
              Ok(())
          }

          pub fn reserve(&mut self, additional: usize) -> Result {
              /*
               * implemented like `VecExt::reserve` from this patchset,
               * except that it uses `vmalloc` instead of `krealloc`.
               */
          }
      }
Quoted 7 lines
Quoted 2 lines

If we need several of these, we can also create a general API that makes it easier to create them and avoids the duplication.

Thanks for for explaining.

I'd probably tend to extending allocator_api then. Do you see any major advantages / disadvantages doing one or the other?

So aside from being able to use Vec and Box etc, I don't think there are any advantages to using allocator_api. The disadvantages are that it's another unstable feature that we need to get stabilized in some form. So it increases the amount of time it takes for us to be able to support multiple versions of Rust.

I think it's fine for you to experiment with the allocator_api and see where that leads you. But when we discuss merging patches that enable unstable features, we should be sure that the feature is truly needed. And that it cannot be replaced by custom code (it also depends on how complicated it is, but I think allocator_api would be simple enough).

Signature · 2 lines
Cheers,
Benno
#29 lore raw
Quoted 167 lines · Benno Lossin
Quoted 92 lines · Danilo Krummrich
Quoted 84 lines · Benno Lossin
Quoted 77 lines · Danilo Krummrich
Quoted 70 lines · Benno Lossin
Quoted 43 lines · Danilo Krummrich

(adding folks from [1])

Quoted 31 lines · Danilo Krummrich

Hi all,

Quoted 16 lines · Wedson Almeida Filho

From: Wedson Almeida Filho <[email protected]>

Revamp how we use the alloc crate.

We currently have a fork of the crate with changes to Vec; other changes have been upstreamed (to the Rust project). This series removes the fork and exposes all the functionality as extension traits.

Additionally, it also introduces allocation flag parameters to all functions that may result in allocations (e.g., Box::new, Arc::new, Vec::push, etc.) without the try_ prefix -- the names are available because we build alloc with no_global_oom_handling.

Lastly, the series also removes our reliance on the allocator_api unstable feature.

Quoted 8 lines

Long term, we still want to make such functionality available in upstream Rust, but this allows us to make progress now and reduces our maintainance burden.

In summary:
1. Removes alloc fork
2. Removes use of allocator_api unstable feature
3. Introduces flags (e.g., GFP_KERNEL, GFP_ATOMIC) when allocating

With that series, how do we implement alternative allocators, such as (k)vmalloc or DMA coherent?

Quoted 8 lines

For instance, I recently sketched up some firmware bindings we want to use in Nova providing

fn copy<A: core::alloc::Allocator>(&self, alloc: A) -> Result<Vec<u8, A>>
[1]

making use of Vec::try_with_capacity_in(). How would I implement something similar now?

Quoted 8 lines

I want to follow up on this topic after also bringing it up in yesterday's weekly Rust call.

In the call a few ideas were discussed, e.g. whether we could just re-enable the allocator_api feature and try getting it stabilized.

With the introduction of alloc::Flags (gfp_t abstraction) allocator_api might not be a viable choice anymore.

Bringing in some more context from the meeting: Gary suggested we create a custom trait for allocators that can also handle allocation flags:

       pub trait AllocatorWithFlags: Allocator {
           type Flags;

           fn allocate_with_flags(&self, layout: Layout, flags: Self::Flags) -> Result<NonNull<[u8]>, AllocError>;

           /* ... */
       }

       impl AllocatorWithFlags for Global { /* ... */ }

       impl<T, A> VecExt<T> for Vec<T, A> where A: AllocatorWithFlags {
           /* ... */
       }
Quoted 5 lines

I think that this would work, but we would have to ensure that users are only allowed to call allocating functions if they are functions that we control. For example Vec::try_reserve [1] would still use the normal Allocator trait that doesn't support our flags.
Gary noted that this could be solved by klint [2].

Quoted 5 lines

I agree, extending the Allocator trait should work.

Regarding allocating functions we don't control, isn't that the case already? AFAICS, we're currently always falling back to GFP_KERNEL when calling Vec::try_reserve().

Quoted 6 lines

Yes we're falling back to that, but
1. there are currently no calls to try_reserve in tree,
2. if you use eg a vmalloc allocator, then I don't know if it would be fine to reallocate that pointer using krealloc. I assumed that that would not be OK (hence my extra care with functions outside of our control).

Quoted 2 lines

Well, this would indeed not be valid. However, a vmalloc allocater wouldn't implement realloc() this way.

Oh yeah that is correct.

Quoted 2 lines

Or are you saying that Vec always uses the global allocator in that case? Why would it do that?

No, it would use the vmalloc allocator.

So I guess the issue isn't as bad as I at first thought. I still think we should lint for this though (but maybe a warning instead of an error).

Quoted 47 lines
Quoted 6 lines
Quoted 3 lines

But yes, I also think it would be better to enforce being explicit.

Given that, is there any value extending the existing Allocator trait at all?

This is what I meant in the meeting by "do you really need the allocator trait?". What you lose is the ability to use Vec and Box, instead

Oh, indeed. I forgot about that when I wrote that. In that case I feel like it's worth extending the existing allocator_api.

Quoted 37 lines

you have to use your own wrapper types (see below). But what you gain is freedom to experiment. In the end we should still try to upstream our findings to Rust or at least share our knowledge, but doing that from the get-go is not ideal for productivity.

Quoted 5 lines
Quoted 3 lines

But we only need to extend the allocator API, if you want to use the std library types that allocate. If you would also be happy with a custom newtype wrapper, then we could also do that.

What do you mean with "custom newtype wrapper"?

You create a newtype struct ("newtype" means that it wraps an inner type and adds/removes/changes features from the inner type):

      pub struct BigVec<T>(Vec<T>);

And then you implement the common operations on it:

      impl<T> BigVec<T> {
          pub fn push(&mut self, item: T) -> Result {
              self.reserve(1)?;

              self.0.spare_capacity_mut()[0].write(item);

              // SAFETY: <omitted for brevity>
              unsafe { self.0.set_len(self.0.len() + 1) };
              Ok(())
          }

          pub fn reserve(&mut self, additional: usize) -> Result {
              /*
               * implemented like `VecExt::reserve` from this patchset,
               * except that it uses `vmalloc` instead of `krealloc`.
               */
          }
      }
Quoted 7 lines
Quoted 2 lines

If we need several of these, we can also create a general API that makes it easier to create them and avoids the duplication.

Thanks for for explaining.

I'd probably tend to extending allocator_api then. Do you see any major advantages / disadvantages doing one or the other?

So aside from being able to use Vec and Box etc, I don't think there are any advantages to using allocator_api. The disadvantages are that it's another unstable feature that we need to get stabilized in some form. So it increases the amount of time it takes for us to be able to support multiple versions of Rust.

Quoted 5 lines

I think it's fine for you to experiment with the allocator_api and see where that leads you. But when we discuss merging patches that enable unstable features, we should be sure that the feature is truly needed. And that it cannot be replaced by custom code (it also depends on how complicated it is, but I think allocator_api would be simple enough).

I agree, though I'm not asking to re-enable allocator_api necessarily.

My original question regarding this series was what's the plan on how to implement alternative allocators given the changes of this series.

This series clearly is a huge improvement, however, before it was quite clear how to implement alternative allocators. Now it's rather unclear.

It's good that we discuss the options now and I'm also happy to contribute to the implementation, but I also think that within the context of this series we should give an answer to this question.

- Danilo

Quoted 5 lines
Signature · 3 lines
-- 
Cheers,
Benno
#30 lore raw
Quoted 1 line · Danilo Krummrich

On Fri, Apr 26, 2024 at 06:32:26AM +0000, Benno Lossin wrote:

<snip>

Quoted 17 lines
Quoted 11 lines

So aside from being able to use Vec and Box etc, I don't think there are any advantages to using allocator_api. The disadvantages are that it's another unstable feature that we need to get stabilized in some form. So it increases the amount of time it takes for us to be able to support multiple versions of Rust.

I think it's fine for you to experiment with the allocator_api and see where that leads you. But when we discuss merging patches that enable unstable features, we should be sure that the feature is truly needed. And that it cannot be replaced by custom code (it also depends on how complicated it is, but I think allocator_api would be simple enough).

I agree, though I'm not asking to re-enable allocator_api necessarily.

My original question regarding this series was what's the plan on how to implement alternative allocators given the changes of this series.

Quoted 6 lines

This series clearly is a huge improvement, however, before it was quite clear how to implement alternative allocators. Now it's rather unclear.

It's good that we discuss the options now and I'm also happy to contribute to the implementation, but I also think that within the context of this series we should give an answer to this question.

In order to get a little closer to an answer I sketched up a patch series [1] to support alternative allocators again and added, besides the KernelAllocator itself, VmAllocator as an example. The patches can also be found in this tree [2].

Please let me know what you think.

- Danilo

[1] https://lore.kernel.org/rust-for-linux/[email protected]/T/#u
[2] https://gitlab.freedesktop.org/drm/nova/-/tree/topic/allocator