SpinLockIrq for rust
Subsystems (3)
Lyude Paul
reviewerv19 cover SpinLockIrq for rust
This is the latest patch series for adding rust bindings for controlling local processor interrupts, adding support for spinlocks in rust that are acquired with local processor interrupts disabled, and implementing local interrupt controls through refcounting in the kernel.
The previous version of this patch series can be found here:
https://lore.kernel.org/all/[email protected]/
This patch series applies on top of boqun's rust-sync branch:
https://git.kernel.org/pub/scm/linux/kernel/git/boqun/linux.git/?h=rust-sync
Now that we've gotten the C-side of these changes in, this patch series is now exclusively concerned with the rust side of these changes. For more information on how we came to the design for the C side of the API, see the explanation in the previous version of the patch series.
Boqun Feng (1):
rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers
Lyude Paul (4):
rust: Introduce interrupt module
rust: sync: use super::* in spinlock.rs
rust: sync: Add SpinLockIrq
rust: sync: Introduce SpinLockIrq::lock_with() and friends
FULL CHANGELOG FOR EACH PATCH AVAILABLE IN THE PATCH UNDER THE CUT LINE
rust/helpers/helpers.c | 1 + rust/helpers/interrupt.c | 18 ++ rust/helpers/spinlock.c | 15 ++ rust/helpers/sync.c | 5 + rust/kernel/interrupt.rs | 89 ++++++++ rust/kernel/lib.rs | 1 + rust/kernel/sync.rs | 9 +- rust/kernel/sync/lock/global.rs | 3 + rust/kernel/sync/lock/spinlock.rs | 331 +++++++++++++++++++++++++++++- 9 files changed, 466 insertions(+), 6 deletions(-) create mode 100644 rust/helpers/interrupt.c create mode 100644 rust/kernel/interrupt.rs
base-commit: 4a653dbaf6f3a8c950a72e4b78e93576e1916381
Signature · 1 line
2.53.0
Lyude Paul
reviewerPATCH v19 1/5 rust: Introduce interrupt module
This introduces a module for dealing with interrupt-disabled contexts, including the ability to enable and disable interrupts along with the ability to annotate functions as expecting that IRQs are already disabled on the local CPU.
[Boqun: This is based on Lyude's work on interrupt disable abstraction, I port to the new local_interrupt_disable() mechanism to make it work as a guard type. I cannot even take the credit of this design, since Lyude also brought up the same idea in zulip. Anyway, this is only for POC purpose, and of course all bugs are mine]
---
V10:
* Fix documentation typos
V11:
* Get rid of unneeded use bindings;
* Move ASSUME_DISABLED into assume_disabled()
* Confirm using lockdep_assert_irqs_disabled() that local interrupts are in fact disabled when LocalInterruptDisabled::assume_disabled() is called.
V18:
* Add missing __rust_helper annotations
V19:
* Use crate:: instead of kernel::
* #[inline] all of the things
rust/helpers/helpers.c | 1 + rust/helpers/interrupt.c | 18 ++++++++ rust/helpers/sync.c | 5 +++ rust/kernel/interrupt.rs | 89 ++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 1 + 5 files changed, 114 insertions(+) create mode 100644 rust/helpers/interrupt.c create mode 100644 rust/kernel/interrupt.rs
rust/helpers/helpers.c +1 -0
@@ -32,6 +32,7 @@#include "err.c" #include "irq.c" #include "fs.c" +#include "interrupt.c" #include "io.c" #include "jump_label.c" #include "kunit.c"
rust/helpers/interrupt.c new file +18 -0
@@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/spinlock.h> + +__rust_helper void rust_helper_local_interrupt_disable(void) +{ +local_interrupt_disable (); +} + +__rust_helper void rust_helper_local_interrupt_enable(void) +{ +local_interrupt_enable (); +} + +__rust_helper bool rust_helper_irqs_disabled(void) +{ +return irqs_disabled (); +}
rust/helpers/sync.c +5 -0
@@ -11,3 +11,8 @@ __rust_helper void rust_helper_lockdep_unregister_key(struct lock_class_key *k) {lockdep_unregister_key (k); } + +__rust_helper void rust_helper_lockdep_assert_irqs_disabled(void) +{ +lockdep_assert_irqs_disabled (); +}
rust/kernel/interrupt.rs new file +89 -0
89 lines not on this page · raw message
rust/kernel/lib.rs +1 -0
@@ -103,6 +103,7 @@#[doc(hidden)] pub mod impl_flags;pub mod init; +pub mod interrupt;pub mod io;pub mod ioctl;pub mod iommu;
Signature · 1 line
2.53.0
Re: [PATCH v19 1/5] rust: Introduce interrupt module
Quoted 11 lines · Lyude Paul
This introduces a module for dealing with interrupt-disabled contexts, including the ability to enable and disable interrupts along with the ability to annotate functions as expecting that IRQs are already disabled on the local CPU.
[Boqun: This is based on Lyude's work on interrupt disable abstraction, I port to the new
local_interrupt_disable()mechanism to make it work as a guard type. I cannot even take the credit of this design, since Lyude also brought up the same idea in zulip. Anyway, this is only for POC purpose, and of course all bugs are mine]
Quoted 5 lines
Signed-off-by Lyude Paul <[email protected]>Co-developed-by Boqun Feng <[email protected]>Signed-off-by Boqun Feng <[email protected]>Reviewed-by Benno Lossin <[email protected]>Reviewed-by Andreas Hindborg <[email protected]>
Quoted 23 lines
---
V10:
* Fix documentation typos
V11:
* Get rid of unneededuse bindings;
* MoveASSUME_DISABLEDintoassume_disabled()
* Confirm usinglockdep_assert_irqs_disabled()that local interrupts are in fact disabled whenLocalInterruptDisabled::assume_disabled()is called.
V18:
* Add missing__rust_helperannotations
V19:
* Use crate:: instead of kernel::
* #[inline] all of the thingsrust/helpers/helpers.c | 1 + rust/helpers/interrupt.c | 18 ++++++++ rust/helpers/sync.c | 5 +++ rust/kernel/interrupt.rs | 89 ++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 1 + 5 files changed, 114 insertions(+) create mode 100644 rust/helpers/interrupt.c create mode 100644 rust/kernel/interrupt.rs
Lyude Paul
reviewerPATCH v19 2/5 rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers
In reply to Lyude PaulFrom: Boqun Feng <[email protected]>
spin_lock_irq_disable() and spin_unlock_irq_enable() are inline functions, to use them in Rust, helpers are introduced. This is for interrupt disabling lock abstraction in Rust.
---
V18:
* Add missing __rust_helper annotations
rust/helpers/spinlock.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
rust/helpers/spinlock.c +15 -0
@@ -36,3 +36,18 @@ __rust_helper void rust_helper_spin_assert_is_held(spinlock_t *lock) {lockdep_assert_held (lock); } + +__rust_helper void rust_helper_spin_lock_irq_disable(spinlock_t *lock) +{ +spin_lock_irq_disable (lock); +} + +__rust_helper void rust_helper_spin_unlock_irq_enable(spinlock_t *lock) +{ +spin_unlock_irq_enable (lock); +} + +__rust_helper int rust_helper_spin_trylock_irq_disable(spinlock_t *lock) +{ +return spin_trylock_irq_disable (lock); +}
Signature · 1 line
2.53.0
Lyude Paul
reviewerPATCH v19 3/5 rust: sync: use super::* in spinlock.rs
In reply to Lyude PaulNo functional changes.
--- rust/kernel/sync/lock/spinlock.rs | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
rust/kernel/sync/lock/spinlock.rs +4 -5
@@ -3,6 +3,7 @@//! A kernel spinlock. //! //! This module allows Rust code to use the kernel's `spinlock_t`. +use super ::*;/// Creates a [`SpinLock`] initialiser with the given name and a newly-created lock class. ///
@@ -82,7 +83,7 @@ macro_rules! new_spinlock {/// ``` /// /// [`spinlock_t`]: srctree/include/linux/spinlock.h -pub type SpinLock <T > =super ::Lock <T ,SpinLockBackend >; +pub type SpinLock <T > =Lock <T ,SpinLockBackend >;/// A kernel `spinlock_t` lock backend. pub struct SpinLockBackend ;
@@ -91,13 +92,11 @@ macro_rules! new_spinlock {/// /// This is simply a type alias for a [`Guard`] returned from locking a [`SpinLock`]. It will unlock /// the [`SpinLock`] upon being dropped. -/// -/// [`Guard`]: super::Guard -pub type SpinLockGuard <'a,T > =super ::Guard <'a,T ,SpinLockBackend >; +pub type SpinLockGuard <'a,T > =Guard <'a,T ,SpinLockBackend >;// SAFETY: The underlying kernel `spinlock_t` object ensures mutual exclusion. `relock` uses the // default implementation that always calls the same locking method. -unsafe impl super ::Backend for SpinLockBackend { +unsafe impl Backend for SpinLockBackend {type State = bindings::spinlock_t ;type GuardState = ();
Signature · 1 line
2.53.0
Lyude Paul
reviewerPATCH v19 4/5 rust: sync: Add SpinLockIrq
In reply to Lyude PaulA variant of SpinLock that ensures interrupts are disabled in the critical section. lock() will ensure that either interrupts are already disabled or disable them. unlock() will reverse the respective operation.
[Boqun: Port to use spin_lock_irq_disable() andspin_unlock_irq_enable()]
---
V10:
* Also add support to GlobalLock
* Documentation fixes from Dirk
V11:
* Add unit test requested by Daniel Almeida
V14:
* Improve rustdoc for SpinLockIrqBackend
V17:
* Update Git summary according to Benno's review
V18:
* Add missing comment change requested by Benno in V16
V19:
* Convert to the kernel import style
* #[inline] all the things
* Add an assertion to ensure that IRQs are actually disabled for the kunit tests.
* Use kernel import style
rust/kernel/sync.rs | 9 +- rust/kernel/sync/lock/global.rs | 3 + rust/kernel/sync/lock/spinlock.rs | 243 ++++++++++++++++++++++++++++++ 3 files changed, 254 insertions(+), 1 deletion(-)
rust/kernel/sync.rs +8 -1
@@ -27,7 +27,14 @@pub use condvar::{new_condvar,CondVar ,CondVarTimeoutResult };pub use lock::global::{global_lock,GlobalGuard ,GlobalLock ,GlobalLockBackend ,GlobalLockedBy };pub use lock::mutex::{new_mutex,Mutex ,MutexGuard }; -pub use lock::spinlock::{new_spinlock,SpinLock ,SpinLockGuard }; +pub use lock::spinlock::{ + new_spinlock, + new_spinlock_irq, +SpinLock , +SpinLockGuard , +SpinLockIrq , +SpinLockIrqGuard ,// +};pub use locked_by::LockedBy ;pub use refcount::Refcount ;pub use set_once::SetOnce ;
rust/kernel/sync/lock/global.rs +3 -0
@@ -304,4 +304,7 @@ macro_rules! global_lock_inner { (backend SpinLock) => { $crate::sync::lock::spinlock::SpinLockBackend }; + (backend SpinLockIrq) => { + $crate::sync::lock::spinlock::SpinLockIrqBackend + }; }
rust/kernel/sync/lock/spinlock.rs +243 -0
252 lines not on this page · raw message
Signature · 1 line
2.53.0
Lyude Paul
reviewerPATCH v19 5/5 rust: sync: Introduce SpinLockIrq::lock_with() and friends
In reply to Lyude PaulSpinLockIrq and SpinLock use the exact same underlying C structure, with the only real difference being that the former uses the irq_disable() and irq_enable() variants for locking/unlocking. These variants can introduce some minor overhead in contexts where we already know that local processor interrupts are disabled, and as such we want a way to be able to skip modifying processor interrupt state in said contexts in order to avoid some overhead - just like the current C API allows us to do.
In order to do this, we add some special functions for SpinLockIrq:lock_with() and try_lock_with(), which allow acquiring the lock without changing the interrupt state - as long as the caller can provide a LocalInterruptDisabled reference to prove that local processor interrupts have been disabled.
In some hacked-together benchmarks we ran, most of the time this did actually seem to lead to a noticeable difference in overhead:
From an aarch64 VM running on a MacBook M4:
lock() when irq is disabled, 100 times cost Delta { nanos: 500 }
lock_with() when irq is disabled, 100 times cost Delta { nanos: 292 }
lock() when irq is enabled, 100 times cost Delta { nanos: 834 }
lock() when irq is disabled, 100 times cost Delta { nanos: 459 }
lock_with() when irq is disabled, 100 times cost Delta { nanos: 291 }
lock() when irq is enabled, 100 times cost Delta { nanos: 709 } From an x86_64 VM (qemu/kvm) running on a i7-13700H
lock() when irq is disabled, 100 times cost Delta { nanos: 1002 }
lock_with() when irq is disabled, 100 times cost Delta { nanos: 729 }
lock() when irq is enabled, 100 times cost Delta { nanos: 1516 }
lock() when irq is disabled, 100 times cost Delta { nanos: 754 }
lock_with() when irq is disabled, 100 times cost Delta { nanos: 966 }
lock() when irq is enabled, 100 times cost Delta { nanos: 1227 }
(note that there were some runs on x86_64 where lock() on irq disabled
vs. lock_with() on irq disabled had equivalent benchmarks, but it very
much appeared to be a minority of test runs.While it's not clear how this affects real-world workloads yet, let's add this for the time being so we can find out.
This makes it so that a SpinLockIrq will work like a SpinLock if interrupts are disabled. So a function:
(&'a SpinLockIrq, &'a InterruptDisabled) -> Guard<'a, .., SpinLockBackend>
makes sense. Note that due to Guard and InterruptDisabled having the same lifetime, interrupts cannot be enabled while the Guard exists.
---
This was originally two patches, but keeping them split didn't make sense after going from BackendInContext to BackendWithContext.
V10:
* Fix typos - Dirk/Lyude
* Since we're adding support for context locks to GlobalLock as well, let's also make sure to cover try_lock while we're at it and add try_lock_with
* Add a private function as_lock_in_context() for handling casting from a Lock<T, B> to Lock<T, B::ContextualBackend> so we don't have to duplicate safety comments
V11:
* Fix clippy::ref_as_ptr error in Lock::as_lock_in_context() V14:
* Add benchmark results, rewrite commit message
V17:
* Introduce BackendWithContext, move context-related bits into there and out of Backend.
* Add missing #[must_use = …] for try_lock_with()
* Remove all unsafe code from lock_with() and try_lock_with():
Somehow I never noticed that literally none of the unsafe code in these two functions is needed with as_lock_in_context()...
V18:
* Get rid of BackendWithContext
* Just use transmute in as_lock_in_context()
* Now that we're only supporting IRQ spinlocks and not using traits, use the type aliases for SpinLock and SpinLockGuard
* Improve the docs now that we're not using traits.
V19:
* #[inline] #[inline] #[inline]
rust/kernel/sync/lock/spinlock.rs | 81 ++++++++++++++++++++++++++++++- 1 file changed, 80 insertions(+), 1 deletion(-)
rust/kernel/sync/lock/spinlock.rs +80 -1
@@ -4,7 +4,10 @@//! //! This module allows Rust code to use the kernel's `spinlock_t`. use super ::*; -use crate ::prelude::*; +use crate ::{ + interrupt::LocalInterruptDisabled , + prelude::*,// +};/// Creates a [`SpinLock`] initialiser with the given name and a newly-created lock class. ///
@@ -224,6 +227,45 @@ macro_rules! new_spinlock_irq {/// # Ok::<(), Error>(()) /// ``` /// +/// The next example demonstrates locking a [`SpinLockIrq`] using [`lock_with()`] in a function +/// which can only be called when local processor interrupts are already disabled. +/// +/// ``` +/// use kernel::sync::{new_spinlock_irq, SpinLockIrq}; +/// use kernel::interrupt::*; +/// +/// struct Inner { +/// a: u32, +/// } +/// +/// #[pin_data] +/// struct Example { +/// #[pin] +/// inner: SpinLockIrq<Inner>, +/// } +/// +/// impl Example { +/// fn new() -> impl PinInit<Self> { +/// pin_init!(Self { +/// inner <- new_spinlock_irq!(Inner { a: 20 }), +/// }) +/// } +/// } +/// +/// // Accessing an `Example` from a function that can only be called in no-interrupt contexts. +/// fn noirq_work(e: &Example, interrupt_disabled: &LocalInterruptDisabled) { +/// // Because we know interrupts are disabled from interrupt_disable, we can skip toggling +/// // interrupt state using lock_with() and the provided token +/// assert_eq!(e.inner.lock_with(interrupt_disabled).a, 20); +/// } +/// +/// # let e = KBox::pin_init(Example::new(), GFP_KERNEL)?; +/// # let interrupt_guard = local_interrupt_disable(); +/// # noirq_work(&e, &interrupt_guard); +/// # +/// # Ok::<(), Error>(()) +/// ``` +/// /// [`lock()`]: SpinLockIrq::lock /// [`lock_with()`]: SpinLockIrq::lock_with pub type SpinLockIrq <T > =super ::Lock <T ,SpinLockIrqBackend >;
@@ -291,6 +333,43 @@ unsafe fn assert_is_held(ptr: *mut Self::State) { } } +impl <T : ?Sized >Lock <T ,SpinLockIrqBackend > { +/// Casts the lock as a `Lock<T, SpinLockBackend>`. +#[inline] +fn as_lock_in_interrupt <'a>(&'a self, _context: &'aLocalInterruptDisabled ) -> &'aSpinLock <T > { +// SAFETY: +// - `Lock<T, SpinLockBackend>` and `Lock<T, SpinLockIrqBackend>` both have identical data +// layouts. +// - As long as local interrupts are disabled (which is proven to be true by _context), it +// is safe to treat a lock with SpinLockIrqBackend as a SpinLockBackend lock. +unsafe { core::mem::transmute (self) } + } + +/// Acquires the lock without modifying local interrupt state. +/// +/// This function should be used in place of the more expensive [`Lock::lock()`] function when +/// possible for [`SpinLockIrq`] locks. +#[inline] +pub fn lock_with <'a>(&'a self, context: &'aLocalInterruptDisabled ) ->SpinLockGuard <'a,T > { + self.as_lock_in_interrupt (context).lock () + } + +/// Tries to acquire the lock without modifying local interrupt state. +/// +/// This function should be used in place of the more expensive [`Lock::try_lock()`] function +/// when possible for [`SpinLockIrq`] locks. +/// +/// Returns a guard that can be used to access the data protected by the lock if successful. +#[must_use = "if unused, the lock will be immediately unlocked" ] +#[inline] +pub fn try_lock_with <'a>( + &'a self, + context: &'aLocalInterruptDisabled , + ) ->Option <SpinLockGuard <'a,T >> { + self.as_lock_in_interrupt (context).try_lock () + } +} +#[kunit_tests(rust_spinlock_irq_condvar)] mod tests {use super ::*;
Signature · 1 line
2.53.0