Skip to contentSkip to conversation

Refcounted interrupts, SpinLockIrq for rust

PATCH Superseded Lyude Paul 9 participants 47 messages Jan 21 – Feb 7

Subsystems (13)

v17 of 19 Superseded Latest is v19 →

lore
Share & tools
lore

Lyude Paul

author reviewer
#0 lore raw

v17 cover Refcounted interrupts, SpinLockIrq for rust

This is the latest patch series for adding rust bindings for controlling local processor interrupts, adding support for spinlocks in rust that are acquired with local processor interrupts disabled, and implementing local interrupt controls through refcounting in the kernel.

The previous version of this patch series can be found here:

https://lkml.org/lkml/2025/12/15/1190

This patch series applies on top of the rust-next branch.

There's a few big changes from the last time. Mainly that we've addressed all(?) of the open questions on this patch series:

* Thanks to Joel Fernandes, we now have a seperate per-CPU counter for tracking NMI nesting - which ensures that we don't have to sacrifice NMI nest level bits in order to store a counter for refcounted IRQs. These patches have been included at the start of the series.
* We've been able to prove that being able to convert the kernel over to this new interface is indeed possible, more on this below.
* Also thank to Joel, we also now have actual benchmarks for how this affects performance:
https://lore.kernel.org/rust-for-linux/[email protected]/
* Also some small changes to the kunit test I added, mainly just making sure I don't forget to include a MODULE_DESCRIPTION or MODULE_LICENSE.

Regarding the conversion plan: we've had some success at getting kernels to boot after attempting to convert the entire kernel from the non-refcounted API to the new refcounted API. It will definitely take quite a lot of work to get this right though, at least in the kernel core side of things. To give readers an idea of what I mean, here's a few of the issues that we ended up running into:

On my end, I tried running a number of coccinelle conversions for this. At first I did actually try simply rewiring local_irq_disable()/local_irq_enable() to local_interrupt_enable()/local_interrupt_disable(). This wasn't really workable though, as it causes the kernel to crash very early on in a number of ways that I haven't fully untangled. Doing this with coccinelle on the other hand allowed me to convert individual files at a time, along with specific usage patterns of the old API, and as a result this ended up giving me a pretty good idea of where our issues are coming from. This coccinelle script, while still leaving most of the kernel unconverted, was at least able to be run on almost all of kernel/ while still allowing us to boot on x86_64

@depends on patch && !report@
@@
- local_irq_disable();
+ local_interrupt_disable();
...
- local_irq_enable();
+ local_interrupt_enable();

There were two files in kernel/ that were exceptions to this:

* kernel/softirq.c
* kernel/main.c (I figured out at least one fix to an issue here)

The reason this worked is because it seems like the vast majority of the issues we're seeing come from "unbalanced"/"misordered" usages of the old irq API. And there seems to be a few reasons for this:

* The first simple reason: occasionally the enable/disable was split across a function, which this script didn't handle.
* The second more complicated reason: some portions of the kernel core end up calling processor instructions that modify the processor's local interrupt flags independently of the kernel. In x86_64's case, I believe we came to the conclusion the iret instruction (interrupt return) was modifying the interrupt flag state. There's possibly a few more instances like this elsewhere.

Boqun also took a stab at this on aarch64, and ended up having similar findings. In their case, they discovered one of the culprits being raw_spin_rq_unlock_irq(). Here the reason is that on aarch64 preempt_count is per-thread and not just per-cpu, and when context switching you generally disable interrupts from one task and restore it in the other task. So in order to fix it, we'll need to make some modifications to the aarch64 context-switching code.

So - with this being said, we decided that the best way of converting it is likely to just leave us with 3 APIs for the time being - and have new drivers and code use the new API while we go through and convert the rest of the kernel.

FULL CHANGELOG BELOW

Boqun Feng (5):
preempt: Introduce HARDIRQ_DISABLE_BITS
preempt: Introduce __preempt_count_{sub, add}_return()
irq & spin_lock: Add counted interrupt disabling/enabling
rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers
locking: Switch to _irq_{disable,enable}() variants in cleanup guards

Joel Fernandes (1):
preempt: Track NMI nesting to separate per-CPU counter

Lyude Paul (10):
openrisc: Include <linux/cpumask.h> in smp.h
irq: Add KUnit test for refcounted interrupt enable/disable
rust: Introduce interrupt module
rust: sync: Add SpinLockIrq
rust: sync: Introduce lock::Lock::lock_with() and friends
rust: sync: Expose lock::Backend
rust: sync: lock/global: Rename B to G in trait bounds
rust: sync: Add a lifetime parameter to lock::global::GlobalGuard
rust: sync: lock/global: Add Backend parameter to GlobalGuard
rust: sync: lock/global: Add ContextualBackend support to GlobalLock

--- Changelog ---

Series-wide:
V15:
* Added a patch for fixing build erors introduced on OpenRISC
V17:
* Squash:
- "rust: sync: Introduce lock::Backend::Context"
- "rust: sync: lock: Add Backend::BackendInContext" With BackendWithContext being a separate trait, it no longer makes sense for these patches to be separate.

* preempt: Introduce HARDIRQ_DISABLE_BITS
V14:
* Fix HARDIRQ_DISABLE_MASK definition
* preempt: Track NMI nesting to separate per-CPU counter
* preempt: Introduce __preempt_count_{sub, add}_return()
V10:
* Add commit message I forgot
* Rebase against latest pcpu_hot changes
V11:
* Remove CONFIG_PROFILE_ALL_BRANCHES workaround from __preempt_count_add_return()
* openrisc: Include <linux/cpumask.h> in smp.h
* irq & spin_lock: Add counted interrupt disabling/enabling
V10:
* Add missing __raw_spin_lock_irq_disable() definition in spinlock.c
V11:
* Move definition of spin_trylock_irq_disable() into this commit
* Get rid of leftover space
* Remove unneeded preempt_disable()/preempt_enable()
V12:
* Move local_interrupt_enable()/local_interrupt_disable() out of include/linux/spinlock.h, into include/linux/irqflags.h
V14:
* Move local_interrupt_enable()/disable() again, this time into its own header, interrupt_rc.h, in order to fix a hexagon-specific build issue caught by the CKI bot.
The reason this is needed is because on most architectures, irqflags.h ends up including <arch/smp.h>. This provides a definition for the raw_smp_processor_id() function which we depend on like so:

<linux/percpu-defs.h> <arch/smp.h>
local_interrupt_disable() → raw_cpu_write() → raw_smp_processor_id()

Unfortunately, hexagon appears to be one such architecture which does not pull in <arch/smp.h> by default here - causing kernel builds to fail and claim that raw_smp_processor_id() is undefined:

In file included from kernel/sched/rq-offsets.c:5:
In file included from kernel/sched/sched.h:8:
In file included from include/linux/sched/affinity.h:1:
In file included from include/linux/sched.h:37:
In file included from include/linux/spinlock.h:59:
>> include/linux/irqflags.h:277:3: error: call to undeclared function
'raw_smp_processor_id'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
277 | raw_cpu_write(local_interrupt_disable_state.flags, flags);
| ^
include/linux/percpu-defs.h:413:34: note: expanded from macro 'raw_cpu_write'

While including <arch/smp.h> in <linux/irqflags.h> does fix the build on hexagon, it ends up breaking the build on x86_64:

In file included from kernel/sched/rq-offsets.c:5:
In file included from kernel/sched/sched.h:8:
In file included from ./include/linux/sched/affinity.h:1:
In file included from ./include/linux/sched.h:13:
In file included from ./arch/x86/include/asm/processor.h:25:
In file included from ./arch/x86/include/asm/special_insns.h:10:
In file included from ./include/linux/irqflags.h:22:
In file included from ./arch/x86/include/asm/smp.h:6:
In file included from ./include/linux/thread_info.h:60:
In file included from ./arch/x86/include/asm/thread_info.h:59:
./arch/x86/include/asm/cpufeature.h:110:40: error: use of undeclared identifier 'boot_cpu_data'

[cap_byte] "i" (&((const char *)boot_cpu_data.x86_capability)[bit >> 3])
^
While boot_cpu_data is defined in <asm/processor.h>, it's not possible for us to include that header in irqflags.h because we're already inside of <asm/processor.h>.

As a result, I just concluded there's no reasonable way of having these functions in <linux/irqflags.h> because of how many low level ASM headers depend on it. So, we go with the solution of simply giving ourselves our own header file.
V15:
* Fix build error on CONFIG_SMP=n reported by Kernel CI

* irq: Add KUnit test for refcounted interrupt enable/disable
* rust: Introduce interrupt module
V10:
* Fix documentation typos
V11:
* Get rid of unneeded use bindings;
* Move ASSUME_DISABLED into assume_disabled()
* Confirm using lockdep_assert_irqs_disabled() that local interrupts are in fact disabled when LocalInterruptDisabled::assume_disabled() is called.
* rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers
* rust: sync: Add SpinLockIrq
V10:
* Also add support to GlobalLock
* Documentation fixes from Dirk
V11:
* Add unit test requested by Daniel Almeida
V14:
* Improve rustdoc for SpinLockIrqBackend
* rust: sync: Introduce lock::Lock::lock_with() and friends
V10:
* Fix typos - Dirk
V17:
* Introduce BackendWithContext, move context-related bits into there and out of Backend.
* Add missing #[must_use = …] for try_lock_with()
* Remove all unsafe code from lock_with() and try_lock_with():
Somehow I never noticed that literally none of the unsafe code in these two functions is needed with as_lock_in_context()...
* rust: sync: Expose lock::Backend
V10:
* Fix typos - Dirk/Lyude
* Since we're adding support for context locks to GlobalLock as well, let's also make sure to cover try_lock while we're at it and add try_lock_with
* Add a private function as_lock_in_context() for handling casting from a Lock<T, B> to Lock<T, B::BackendInContext> so we don't have to duplicate safety comments
V11:
* Fix clippy::ref_as_ptr error in Lock::as_lock_in_context()
V14:
* Add benchmark results, rewrite commit message
* rust: sync: lock/global: Rename B to G in trait bounds
* rust: sync: Add a lifetime parameter to lock::global::GlobalGuard
* rust: sync: lock/global: Add Backend parameter to GlobalGuard V17:
* Add default parameter for generic B to GlobalGuard
* rust: sync: lock/global: Add ContextualBackend support to GlobalLock
* locking: Switch to _irq_{disable,enable}() variants in cleanup guards V10:
* Add PREEMPT_RT build fix from Guangbo Cui

 arch/arm64/include/asm/preempt.h     |  18 ++
 arch/openrisc/include/asm/smp.h      |   2 +
 arch/s390/include/asm/preempt.h      |  10 +
 arch/x86/include/asm/preempt.h       |  10 +
 include/asm-generic/preempt.h        |  14 ++
 include/linux/hardirq.h              |  16 +-
 include/linux/interrupt_rc.h         |  63 ++++++
 include/linux/preempt.h              |  23 ++-
 include/linux/spinlock.h             |  51 +++--
 include/linux/spinlock_api_smp.h     |  27 +++
 include/linux/spinlock_api_up.h      |   8 +
 include/linux/spinlock_rt.h          |  15 ++
 kernel/irq/Makefile                  |   1 +
 kernel/irq/refcount_interrupt_test.c | 109 +++++++++++
 kernel/locking/spinlock.c            |  29 +++
 kernel/softirq.c                     |   5 +
 rust/helpers/helpers.c               |   1 +
 rust/helpers/interrupt.c             |  18 ++
 rust/helpers/spinlock.c              |  15 ++
 rust/helpers/sync.c                  |   5 +
 rust/kernel/interrupt.rs             |  86 +++++++++
 rust/kernel/lib.rs                   |   1 +
 rust/kernel/sync.rs                  |   5 +-
 rust/kernel/sync/lock.rs             |  71 ++++++-
 rust/kernel/sync/lock/global.rs      |  95 +++++----
 rust/kernel/sync/lock/spinlock.rs    | 275 +++++++++++++++++++++++++++
 26 files changed, 917 insertions(+), 56 deletions(-)
 create mode 100644 include/linux/interrupt_rc.h
 create mode 100644 kernel/irq/refcount_interrupt_test.c
 create mode 100644 rust/helpers/interrupt.c
 create mode 100644 rust/kernel/interrupt.rs

base-commit: 2ad6c5cdc89acfefb01b84afa5e55262c40d6fec

Signature · 1 line
2.52.0
author reviewer
#1 lore raw

PATCH v17 1/16 preempt: Introduce HARDIRQ_DISABLE_BITS

From: Boqun Feng <[email protected]>

In order to support preempt_disable()-like interrupt disabling, that is, using part of preempt_count() to track interrupt disabling nested level, change the preempt_count() layout to contain 8-bit HARDIRQ_DISABLE count.

Note that HARDIRQ_BITS and NMI_BITS are reduced by 1 because of this, and it changes the maximum of their (hardirq and nmi) nesting level.

---
V14:
* Fix HARDIRQ_DISABLE_MASK definition

 include/linux/preempt.h | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
include/linux/preempt.h +11 -5
index d964f965c8ffc..f07e7f37f3ca5 100644
@@ -17,6 +17,7 @@
  *
  * - bits 0-7 are the preemption count (max preemption depth: 256)
  * - bits 8-15 are the softirq count (max # of softirqs: 256)
+ * - bits 16-23 are the hardirq disable count (max # of hardirq disable: 256)
  *
  * The hardirq count could in theory be the same as the number of
  * interrupts in the system, but we run all interrupt handlers with
@@ -26,29 +27,34 @@
  *
  *         PREEMPT_MASK:	0x000000ff
  *         SOFTIRQ_MASK:	0x0000ff00
- *         HARDIRQ_MASK:	0x000f0000
- *             NMI_MASK:	0x00f00000
+ * HARDIRQ_DISABLE_MASK:	0x00ff0000
+ *         HARDIRQ_MASK:	0x07000000
+ *             NMI_MASK:	0x38000000
  * PREEMPT_NEED_RESCHED:	0x80000000
  */
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
-#define HARDIRQ_BITS	4
-#define NMI_BITS	4
+#define HARDIRQ_DISABLE_BITS	8
+#define HARDIRQ_BITS	3
+#define NMI_BITS	3
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
-#define HARDIRQ_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_DISABLE_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_SHIFT	(HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS)
 #define NMI_SHIFT	(HARDIRQ_SHIFT + HARDIRQ_BITS)
 
 #define __IRQ_MASK(x)	((1UL << (x))-1)
 
 #define PREEMPT_MASK	(__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)
 #define SOFTIRQ_MASK	(__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
+#define HARDIRQ_DISABLE_MASK	(__IRQ_MASK(HARDIRQ_DISABLE_BITS) << HARDIRQ_DISABLE_SHIFT)
 #define HARDIRQ_MASK	(__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
 #define NMI_MASK	(__IRQ_MASK(NMI_BITS)     << NMI_SHIFT)
 
 #define PREEMPT_OFFSET	(1UL << PREEMPT_SHIFT)
 #define SOFTIRQ_OFFSET	(1UL << SOFTIRQ_SHIFT)
+#define HARDIRQ_DISABLE_OFFSET	(1UL << HARDIRQ_DISABLE_SHIFT)
 #define HARDIRQ_OFFSET	(1UL << HARDIRQ_SHIFT)
 #define NMI_OFFSET	(1UL << NMI_SHIFT)
 
Signature · 1 line
2.52.0
author reviewer
#2 lore raw

PATCH v17 2/16 preempt: Track NMI nesting to separate per-CPU counter

In reply to Lyude Paul

From: Joel Fernandes <[email protected]>

Move NMI nesting tracking from the preempt_count bits to a separate per-CPU counter (nmi_nesting). This is to free up the NMI bits in the preempt_count, allowing those bits to be repurposed for other uses. This also has the benefit of tracking more than 16-levels deep if there is ever a need.

Reduce multiple bits in preempt_count for NMI tracking. Reduce NMI_BITS from 3 to 1, using it only to detect if we're in an NMI.

---
 include/linux/hardirq.h | 16 ++++++++++++----
 include/linux/preempt.h | 13 +++++++++----
 kernel/softirq.c        |  2 ++
 3 files changed, 23 insertions(+), 8 deletions(-)
include/linux/hardirq.h +12 -4
index d57cab4d4c06f..cc06bda52c3e5 100644
@@ -10,6 +10,8 @@
 #include <linux/vtime.h>
 #include <asm/hardirq.h>
 
+DECLARE_PER_CPU(unsigned int, nmi_nesting);
+
 extern void synchronize_irq(unsigned int irq);
 extern bool synchronize_hardirq(unsigned int irq);
 
@@ -102,14 +104,16 @@ void irq_exit_rcu(void);
  */
 
 /*
- * nmi_enter() can nest up to 15 times; see NMI_BITS.
+ * nmi_enter() can nest - nesting is tracked in a per-CPU counter.
  */
 #define __nmi_enter()						\
 	do {							\
 		lockdep_off();					\
 		arch_nmi_enter();				\
-		BUG_ON(in_nmi() == NMI_MASK);			\
-		__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		BUG_ON(__this_cpu_read(nmi_nesting) == UINT_MAX);	\
+		__this_cpu_inc(nmi_nesting);			\
+		__preempt_count_add(HARDIRQ_OFFSET);		\
+		preempt_count_set(preempt_count() | NMI_MASK);	\
 	} while (0)
 
 #define nmi_enter()						\
@@ -124,8 +128,12 @@ void irq_exit_rcu(void);
 
 #define __nmi_exit()						\
 	do {							\
+		unsigned int nesting;				\
 		BUG_ON(!in_nmi());				\
-		__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		__preempt_count_sub(HARDIRQ_OFFSET);		\
+		nesting = __this_cpu_dec_return(nmi_nesting);	\
+		if (!nesting)					\
+			__preempt_count_sub(NMI_OFFSET);	\
 		arch_nmi_exit();				\
 		lockdep_on();					\
 	} while (0)
include/linux/preempt.h +9 -4
index f07e7f37f3ca5..e2d3079d3f5f1 100644
@@ -18,6 +18,8 @@
  * - bits 0-7 are the preemption count (max preemption depth: 256)
  * - bits 8-15 are the softirq count (max # of softirqs: 256)
  * - bits 16-23 are the hardirq disable count (max # of hardirq disable: 256)
+ * - bits 24-27 are the hardirq count (max # of hardirqs: 16)
+ * - bit 28 is the NMI flag (no nesting count, tracked separately)
  *
  * The hardirq count could in theory be the same as the number of
  * interrupts in the system, but we run all interrupt handlers with
@@ -25,18 +27,21 @@
  * there are a few palaeontologic drivers which reenable interrupts in
  * the handler, so we need more than one bit here.
  *
+ * NMI nesting depth is tracked in a separate per-CPU variable
+ * (nmi_nesting) to save bits in preempt_count.
+ *
  *         PREEMPT_MASK:	0x000000ff
  *         SOFTIRQ_MASK:	0x0000ff00
  * HARDIRQ_DISABLE_MASK:	0x00ff0000
- *         HARDIRQ_MASK:	0x07000000
- *             NMI_MASK:	0x38000000
+ *         HARDIRQ_MASK:	0x0f000000
+ *             NMI_MASK:	0x10000000
  * PREEMPT_NEED_RESCHED:	0x80000000
  */
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
 #define HARDIRQ_DISABLE_BITS	8
-#define HARDIRQ_BITS	3
-#define NMI_BITS	3
+#define HARDIRQ_BITS	4
+#define NMI_BITS	1
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
kernel/softirq.c +2 -0
index 77198911b8dd4..af47ea23aba3b 100644
@@ -88,6 +88,8 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
+DEFINE_PER_CPU(unsigned int, nmi_nesting);
+
 /*
  * SOFTIRQ_OFFSET usage:
  *
Signature · 1 line
2.52.0
maintainer
#3 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

kernel/softirq.c EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled); · line 91

Quoted 9 lines · Lyude Paul
kernel/softirq.c +1 -0
index 77198911b8dd4..af47ea23aba3b 100644
@@ -88,6 +88,8 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
+DEFINE_PER_CPU(unsigned int, nmi_nesting);

What happened with putting this in the same line as preempt_count?

#4 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Lyude Paul
kernel/softirq.c +0 -0
index 77198911b8dd4..af47ea23aba3b 100644
@@ -88,6 +88,8 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
Quoted 3 lines

kernel/softirq.c EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled); · line 91

Quoted 1 line
kernel/softirq.c +1 -0
+DEFINE_PER_CPU(unsigned int, nmi_nesting);

What happened with putting this in the same line as preempt_count?

I can try to do that again if we still want to go with this patch. When I tried that last, I ran into issues I can't remember now (the space being limited?).

maintainer
#5 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

In reply to Lyude Paul
Quoted 17 lines · Lyude Paul

#define __nmi_enter() \

include/linux/hardirq.h +4 -2
 	do {							\
 		lockdep_off();					\
 		arch_nmi_enter();				\
-		BUG_ON(in_nmi() == NMI_MASK);			\
-		__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		BUG_ON(__this_cpu_read(nmi_nesting) == UINT_MAX);	\
+		__this_cpu_inc(nmi_nesting);			\
+		__preempt_count_add(HARDIRQ_OFFSET);		\
+		preempt_count_set(preempt_count() | NMI_MASK);	\
 	} while (0)
 
 #define nmi_enter()						\
@@ -124,8 +128,12 @@ void irq_exit_rcu(void);
 
 #define __nmi_exit()						\
 	do {							\

include/linux/hardirq.h void irq_exit_rcu(void); · line 139

Quoted 10 lines
include/linux/hardirq.h +5 -1
+		unsigned int nesting;				\
 		BUG_ON(!in_nmi());				\
-		__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		__preempt_count_sub(HARDIRQ_OFFSET);		\
+		nesting = __this_cpu_dec_return(nmi_nesting);	\
+		if (!nesting)					\
+			__preempt_count_sub(NMI_OFFSET);	\
 		arch_nmi_exit();				\
 		lockdep_on();					\
 	} while (0)

While not wrong like last time; it is pretty awful.

preempt_count_set() is a cmpxchg() loop.

Would not something like so be better?

#define __nmi_enter()                                           \
        do {                                                    \
+               unsigned int _o = NMI_MASK + HARDIRQ_OFFSET;    \
                lockdep_off();                                  \
                arch_nmi_enter();                               \
-               BUG_ON(in_nmi() == NMI_MASK);                   \
-               __preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);       \
+               BUG_ON(__this_cpu_read(nmi_nesting) == ~0U);    \
+               __this_cpu_inc(nmi_nesting);                    \
+               _o -= (preempt_count() & NMI_MASK);             \
+               __preempt_count_add(_o);                        \
        } while (0)
 #define __nmi_exit()                                           \
        do {                                                    \
+               unsigned int _o = HARDIRQ_OFFSET;               \
                BUG_ON(!in_nmi());                              \
-               __preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);       \
+               if (!__this_cpu_dec_return(nmi_nesting))        \
+                       _o += NMI_MASK;                         \
+               __preempt_count_sub(_o);                        \
                arch_nmi_exit();                                \
                lockdep_on();                                   \
        } while (0)

But I'm really somewhat sad that 64bit can't do better than this.

maintainer
#6 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

Quoted 1 line · Peter Zijlstra

But I'm really somewhat sad that 64bit can't do better than this.

Here, the below builds and boots (albeit with warnings because printf format crap sucks).

---
 arch/x86/Kconfig               |  1 +
 arch/x86/include/asm/preempt.h | 53 ++++++++++++++++++++++++++++++------------
 arch/x86/kernel/cpu/common.c   |  2 +-
 include/linux/hardirq.h        |  7 +++---
 include/linux/preempt.h        | 52 ++++++++++++++++++++++++++++++++++-------
 init/main.c                    |  2 +-
 kernel/Kconfig.preempt         |  4 ++++
 kernel/sched/core.c            |  8 +++----
 kernel/softirq.c               | 10 +++++++-
 kernel/time/timer.c            |  2 +-
 lib/locking-selftest.c         |  2 +-
 11 files changed, 106 insertions(+), 37 deletions(-)
arch/x86/Kconfig +1 -0
index 80527299f859..2bd1972fd4c7 100644
@@ -326,6 +326,7 @@ config X86
 	select USER_STACKTRACE_SUPPORT
 	select HAVE_ARCH_KCSAN			if X86_64
 	select PROC_PID_ARCH_STATUS		if PROC_FS
+	select PREEMPT_LONG			if X86_64
 	select HAVE_ARCH_NODE_DEV_GROUP		if X86_SGX
 	select FUNCTION_ALIGNMENT_16B		if X86_64 || X86_ALIGNMENT_16
 	select FUNCTION_ALIGNMENT_4B
arch/x86/include/asm/preempt.h +38 -15

110 lines not on this page · raw message

arch/x86/kernel/cpu/common.c +1 -1
index e7ab22fce3b5..9d3602f085c9 100644
@@ -2219,7 +2219,7 @@ DEFINE_PER_CPU_CACHE_HOT(struct task_struct *, current_task) = &init_task;
 EXPORT_PER_CPU_SYMBOL(current_task);
 EXPORT_PER_CPU_SYMBOL(const_current_task);
 
-DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) = INIT_PREEMPT_COUNT;
+DEFINE_PER_CPU_CACHE_HOT(unsigned long, __preempt_count) = INIT_PREEMPT_COUNT;
 EXPORT_PER_CPU_SYMBOL(__preempt_count);
 
 DEFINE_PER_CPU_CACHE_HOT(unsigned long, cpu_current_top_of_stack) = TOP_OF_INIT_STACK;
include/linux/hardirq.h +3 -4
index d57cab4d4c06..77defd9624bf 100644
@@ -108,15 +108,14 @@ void irq_exit_rcu(void);
 	do {							\
 		lockdep_off();					\
 		arch_nmi_enter();				\
-		BUG_ON(in_nmi() == NMI_MASK);			\
-		__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		__preempt_count_nmi_enter();			\
 	} while (0)
 
 #define nmi_enter()						\
 	do {							\
 		__nmi_enter();					\
 		lockdep_hardirq_enter();			\
-		ct_nmi_enter();				\
+		ct_nmi_enter();					\
 		instrumentation_begin();			\
 		ftrace_nmi_enter();				\
 		instrumentation_end();				\
@@ -125,7 +124,7 @@ void irq_exit_rcu(void);
 #define __nmi_exit()						\
 	do {							\
 		BUG_ON(!in_nmi());				\
-		__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		__preempt_count_nmi_exit();			\
 		arch_nmi_exit();				\
 		lockdep_on();					\
 	} while (0)
include/linux/preempt.h +43 -9

94 lines not on this page · raw message

init/main.c +1 -1
index b84818ad9685..f8f4b78b7a06 100644
@@ -1367,7 +1367,7 @@ static inline void do_trace_initcall_level(const char *level)
 
 int __init_or_module do_one_initcall(initcall_t fn)
 {
-	int count = preempt_count();
+	long count = preempt_count();
 	char msgbuf[64];
 	int ret;
 
kernel/Kconfig.preempt +4 -0
index 88c594c6d7fc..2ad9365915eb 100644
@@ -122,6 +122,10 @@ config PREEMPT_RT_NEEDS_BH_LOCK
 config PREEMPT_COUNT
        bool
 
+config PREEMPT_LONG
+	bool
+	depends on PREEMPT_COUNT && 64BIT
+
 config PREEMPTION
        bool
        select PREEMPT_COUNT
kernel/sched/core.c +4 -4
index b411e4feff7f..f54dd3cb66f2 100644
@@ -5709,7 +5709,7 @@ static inline void sched_tick_stop(int cpu) { }
  * If the value passed in is equal to the current preempt count
  * then we just disabled preemption. Start timing the latency.
  */
-static inline void preempt_latency_start(int val)
+static inline void preempt_latency_start(long val)
 {
 	if (preempt_count() == val) {
 		unsigned long ip = get_lock_parent_ip();
@@ -5746,7 +5746,7 @@ NOKPROBE_SYMBOL(preempt_count_add);
  * If the value passed in equals to the current preempt count
  * then we just enabled preemption. Stop timing the latency.
  */
-static inline void preempt_latency_stop(int val)
+static inline void preempt_latency_stop(long val)
 {
 	if (preempt_count() == val)
 		trace_preempt_on(CALLER_ADDR0, get_lock_parent_ip());
@@ -8774,7 +8774,7 @@ void __might_sleep(const char *file, int line)
 }
 EXPORT_SYMBOL(__might_sleep);
 
-static void print_preempt_disable_ip(int preempt_offset, unsigned long ip)
+static void print_preempt_disable_ip(long preempt_offset, unsigned long ip)
 {
 	if (!IS_ENABLED(CONFIG_DEBUG_PREEMPT))
 		return;
@@ -8846,7 +8846,7 @@ void __might_resched(const char *file, int line, unsigned int offsets)
 }
 EXPORT_SYMBOL(__might_resched);
 
-void __cant_sleep(const char *file, int line, int preempt_offset)
+void __cant_sleep(const char *file, int line, long preempt_offset)
 {
 	static unsigned long prev_jiffy;
 
kernel/softirq.c +9 -1
index 77198911b8dd..51a7f391edab 100644
@@ -88,6 +88,14 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
+#ifndef CONFIG_PREEMPT_LONG
+/*
+ * Any 32bit architecture that still cares about performance should
+ * probably ensure this is near preempt_count.
+ */
+DEFINE_PER_CPU(unsigned int, nmi_nesting);
+#endif
+
 /*
  * SOFTIRQ_OFFSET usage:
  *
@@ -609,7 +617,7 @@ static void handle_softirqs(bool ksirqd)
 
 	while ((softirq_bit = ffs(pending))) {
 		unsigned int vec_nr;
-		int prev_count;
+		long prev_count;
 
 		h += softirq_bit - 1;
 
kernel/time/timer.c +1 -1
index 1f2364126894..89c348139218 100644
@@ -1723,7 +1723,7 @@ static void call_timer_fn(struct timer_list *timer,
 			  void (*fn)(struct timer_list *),
 			  unsigned long baseclk)
 {
-	int count = preempt_count();
+	long count = preempt_count();
 
 #ifdef CONFIG_LOCKDEP
 	/*
lib/locking-selftest.c +1 -1
index d939403331b5..8fd216bd0be6 100644
@@ -1429,7 +1429,7 @@ static int unexpected_testcase_failures;
 
 static void dotest(void (*testcase_fn)(void), int expected, int lockclass_mask)
 {
-	int saved_preempt_count = preempt_count();
+	long saved_preempt_count = preempt_count();
 #ifdef CONFIG_PREEMPT_RT
 #ifdef CONFIG_SMP
 	int saved_mgd_count = current->migration_disabled;
maintainer
#7 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

Quoted 5 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra

But I'm really somewhat sad that 64bit can't do better than this.

Here, the below builds and boots (albeit with warnings because printf format crap sucks).

Hi Peter,

I am not sure if it's worth the complexity to do this for the NMI code path. I don't think NMI code path is hot enough that this is necessary?

Best,
Gary

Quoted 14 lines
---
 arch/x86/Kconfig               |  1 +
 arch/x86/include/asm/preempt.h | 53 ++++++++++++++++++++++++++++++------------
 arch/x86/kernel/cpu/common.c   |  2 +-
 include/linux/hardirq.h        |  7 +++---
 include/linux/preempt.h        | 52 ++++++++++++++++++++++++++++++++++-------
 init/main.c                    |  2 +-
 kernel/Kconfig.preempt         |  4 ++++
 kernel/sched/core.c            |  8 +++----
 kernel/softirq.c               | 10 +++++++-
 kernel/time/timer.c            |  2 +-
 lib/locking-selftest.c         |  2 +-
 11 files changed, 106 insertions(+), 37 deletions(-)
maintainer
#8 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

Quoted 11 lines · Gary Guo
Quoted 5 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra

But I'm really somewhat sad that 64bit can't do better than this.

Here, the below builds and boots (albeit with warnings because printf format crap sucks).

Hi Peter,

I am not sure if it's worth the complexity to do this for the NMI code path. I don't think NMI code path is hot enough that this is necessary?

Perf uses NMI. Also, the 64bit code is actually simpler.

maintainer
#9 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

In reply to Peter Zijlstra
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra

But I'm really somewhat sad that 64bit can't do better than this.

Here, the below builds and boots (albeit with warnings because printf format crap sucks).

Thanks! I will drop patch #1 and #2 and use this one (with a commit log and some more tests), given it's based on the work of Joel, Lyude and me, would the following tags make sense to all of you?

Regards,
Boqun

Quoted 413 lines
---
 arch/x86/Kconfig               |  1 +
 arch/x86/include/asm/preempt.h | 53 ++++++++++++++++++++++++++++++------------
 arch/x86/kernel/cpu/common.c   |  2 +-
 include/linux/hardirq.h        |  7 +++---
 include/linux/preempt.h        | 52 ++++++++++++++++++++++++++++++++++-------
 init/main.c                    |  2 +-
 kernel/Kconfig.preempt         |  4 ++++
 kernel/sched/core.c            |  8 +++----
 kernel/softirq.c               | 10 +++++++-
 kernel/time/timer.c            |  2 +-
 lib/locking-selftest.c         |  2 +-
 11 files changed, 106 insertions(+), 37 deletions(-)
arch/x86/Kconfig +1 -0
index 80527299f859..2bd1972fd4c7 100644
@@ -326,6 +326,7 @@ config X86
 	select USER_STACKTRACE_SUPPORT
 	select HAVE_ARCH_KCSAN			if X86_64
 	select PROC_PID_ARCH_STATUS		if PROC_FS
+	select PREEMPT_LONG			if X86_64
 	select HAVE_ARCH_NODE_DEV_GROUP		if X86_SGX
 	select FUNCTION_ALIGNMENT_16B		if X86_64 || X86_ALIGNMENT_16
 	select FUNCTION_ALIGNMENT_4B
arch/x86/include/asm/preempt.h +38 -15
index 578441db09f0..1b54d5555138 100644
@@ -7,10 +7,19 @@
 
 #include <linux/static_call_types.h>
 
-DECLARE_PER_CPU_CACHE_HOT(int, __preempt_count);
+DECLARE_PER_CPU_CACHE_HOT(unsigned long, __preempt_count);
 
-/* We use the MSB mostly because its available */
-#define PREEMPT_NEED_RESCHED	0x80000000
+/*
+ * We use the MSB for PREEMPT_NEED_RESCHED mostly because it is available.
+ */
+
+#ifdef CONFIG_64BIT
+#define PREEMPT_NEED_RESCHED	(~((-1L) >> 1))
+#define __pc_op(op, ...)	raw_cpu_##op##_8(__VA_ARGS__)
+#else
+#define PREEMPT_NEED_RESCHED	(~((-1) >> 1))
+#define __pc_op(op, ...)	raw_cpu_##op##_4(__VA_ARGS__)
+#endif
 
 /*
  * We use the PREEMPT_NEED_RESCHED bit as an inverted NEED_RESCHED such
@@ -24,18 +33,18 @@ DECLARE_PER_CPU_CACHE_HOT(int, __preempt_count);
  */
 static __always_inline int preempt_count(void)
 {
-	return raw_cpu_read_4(__preempt_count) & ~PREEMPT_NEED_RESCHED;
+	return __pc_op(read, __preempt_count) & ~PREEMPT_NEED_RESCHED;
 }
 
-static __always_inline void preempt_count_set(int pc)
+static __always_inline void preempt_count_set(long pc)
 {
 	int old, new;
 
-	old = raw_cpu_read_4(__preempt_count);
+	old = __pc_op(read, __preempt_count);
 	do {
 		new = (old & PREEMPT_NEED_RESCHED) |
 			(pc & ~PREEMPT_NEED_RESCHED);
-	} while (!raw_cpu_try_cmpxchg_4(__preempt_count, &old, new));
+	} while (!__pc_op(try_cmpxchg, __preempt_count, &old, new));
 }
 
 /*
@@ -58,33 +67,45 @@ static __always_inline void preempt_count_set(int pc)
 
 static __always_inline void set_preempt_need_resched(void)
 {
-	raw_cpu_and_4(__preempt_count, ~PREEMPT_NEED_RESCHED);
+	__pc_op(and, __preempt_count, ~PREEMPT_NEED_RESCHED);
 }
 
 static __always_inline void clear_preempt_need_resched(void)
 {
-	raw_cpu_or_4(__preempt_count, PREEMPT_NEED_RESCHED);
+	__pc_op(or, __preempt_count, PREEMPT_NEED_RESCHED);
 }
 
 static __always_inline bool test_preempt_need_resched(void)
 {
-	return !(raw_cpu_read_4(__preempt_count) & PREEMPT_NEED_RESCHED);
+	return !(__pc_op(read, __preempt_count) & PREEMPT_NEED_RESCHED);
 }
 
 /*
  * The various preempt_count add/sub methods
  */
 
-static __always_inline void __preempt_count_add(int val)
+static __always_inline void __preempt_count_add(long val)
 {
-	raw_cpu_add_4(__preempt_count, val);
+	__pc_op(add, __preempt_count, val);
 }
 
-static __always_inline void __preempt_count_sub(int val)
+static __always_inline void __preempt_count_sub(long val)
 {
-	raw_cpu_add_4(__preempt_count, -val);
+	__pc_op(add, __preempt_count, -val);
 }
 
+#ifdef CONFIG_64BIT
+static __always_inline void __preempt_count_nmi_enter(void)
+{
+	__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);
+}
+
+static __always_inline void __preempt_count_nmi_exit(void)
+{
+	__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);
+}
+#endif
+
 /*
  * Because we keep PREEMPT_NEED_RESCHED set when we do _not_ need to reschedule
  * a decrement which hits zero means we have no preempt_count and should
@@ -101,7 +122,7 @@ static __always_inline bool __preempt_count_dec_and_test(void)
  */
 static __always_inline bool should_resched(int preempt_offset)
 {
-	return unlikely(raw_cpu_read_4(__preempt_count) == preempt_offset);
+	return unlikely(__pc_op(read, __preempt_count) == preempt_offset);
 }
 
 #ifdef CONFIG_PREEMPTION
@@ -148,4 +169,6 @@ do { \
 
 #endif /* PREEMPTION */
 
+#undef __pc_op
+
 #endif /* __ASM_PREEMPT_H */
arch/x86/kernel/cpu/common.c +1 -1
index e7ab22fce3b5..9d3602f085c9 100644
@@ -2219,7 +2219,7 @@ DEFINE_PER_CPU_CACHE_HOT(struct task_struct *, current_task) = &init_task;
 EXPORT_PER_CPU_SYMBOL(current_task);
 EXPORT_PER_CPU_SYMBOL(const_current_task);
 
-DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) = INIT_PREEMPT_COUNT;
+DEFINE_PER_CPU_CACHE_HOT(unsigned long, __preempt_count) = INIT_PREEMPT_COUNT;
 EXPORT_PER_CPU_SYMBOL(__preempt_count);
 
 DEFINE_PER_CPU_CACHE_HOT(unsigned long, cpu_current_top_of_stack) = TOP_OF_INIT_STACK;
include/linux/hardirq.h +3 -4
index d57cab4d4c06..77defd9624bf 100644
@@ -108,15 +108,14 @@ void irq_exit_rcu(void);
 	do {							\
 		lockdep_off();					\
 		arch_nmi_enter();				\
-		BUG_ON(in_nmi() == NMI_MASK);			\
-		__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		__preempt_count_nmi_enter();			\
 	} while (0)
 
 #define nmi_enter()						\
 	do {							\
 		__nmi_enter();					\
 		lockdep_hardirq_enter();			\
-		ct_nmi_enter();				\
+		ct_nmi_enter();					\
 		instrumentation_begin();			\
 		ftrace_nmi_enter();				\
 		instrumentation_end();				\
@@ -125,7 +124,7 @@ void irq_exit_rcu(void);
 #define __nmi_exit()						\
 	do {							\
 		BUG_ON(!in_nmi());				\
-		__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		__preempt_count_nmi_exit();			\
 		arch_nmi_exit();				\
 		lockdep_on();					\
 	} while (0)
include/linux/preempt.h +43 -9
index d964f965c8ff..7617ca97f442 100644
@@ -17,6 +17,9 @@
  *
  * - bits 0-7 are the preemption count (max preemption depth: 256)
  * - bits 8-15 are the softirq count (max # of softirqs: 256)
+ * - bits 16-23 are the hardirq disable count (max # of hardirq disable: 256)
+ * - bits 24-27 are the hardirq count (max # of hardirqs: 16)
+ * - bit 28 is the NMI flag (no nesting count, tracked separately)
  *
  * The hardirq count could in theory be the same as the number of
  * interrupts in the system, but we run all interrupt handlers with
@@ -24,31 +27,41 @@
  * there are a few palaeontologic drivers which reenable interrupts in
  * the handler, so we need more than one bit here.
  *
- *         PREEMPT_MASK:	0x000000ff
- *         SOFTIRQ_MASK:	0x0000ff00
- *         HARDIRQ_MASK:	0x000f0000
- *             NMI_MASK:	0x00f00000
- * PREEMPT_NEED_RESCHED:	0x80000000
+ * NMI nesting depth is tracked in a separate per-CPU variable
+ * (nmi_nesting) to save bits in preempt_count.
+ *
+ *				32bit		64bit + PREEMPT_LONG
+ *
+ *         PREEMPT_MASK:	0x000000ff	0x00000000000000ff
+ *         SOFTIRQ_MASK:	0x0000ff00	0x000000000000ff00
+ * HARDIRQ_DISABLE_MASK:	0x00ff0000	0x0000000000ff0000
+ *         HARDIRQ_MASK:	0x0f000000	0x000000000f000000
+ *             NMI_MASK:	0x10000000	0x00000000f0000000
+ * PREEMPT_NEED_RESCHED:	0x80000000	0x8000000000000000
  */
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
+#define HARDIRQ_DISABLE_BITS	8
 #define HARDIRQ_BITS	4
-#define NMI_BITS	4
+#define NMI_BITS	(1 + 3*IS_ENABLED(CONFIG_PREEMPT_LONG))
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
-#define HARDIRQ_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_DISABLE_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_SHIFT	(HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS)
 #define NMI_SHIFT	(HARDIRQ_SHIFT + HARDIRQ_BITS)
 
 #define __IRQ_MASK(x)	((1UL << (x))-1)
 
 #define PREEMPT_MASK	(__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)
 #define SOFTIRQ_MASK	(__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
+#define HARDIRQ_DISABLE_MASK	(__IRQ_MASK(HARDIRQ_DISABLE_BITS) << HARDIRQ_DISABLE_SHIFT)
 #define HARDIRQ_MASK	(__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
 #define NMI_MASK	(__IRQ_MASK(NMI_BITS)     << NMI_SHIFT)
 
 #define PREEMPT_OFFSET	(1UL << PREEMPT_SHIFT)
 #define SOFTIRQ_OFFSET	(1UL << SOFTIRQ_SHIFT)
+#define HARDIRQ_DISABLE_OFFSET	(1UL << HARDIRQ_DISABLE_SHIFT)
 #define HARDIRQ_OFFSET	(1UL << HARDIRQ_SHIFT)
 #define NMI_OFFSET	(1UL << NMI_SHIFT)
 
@@ -105,8 +118,8 @@ static __always_inline unsigned char interrupt_context_level(void)
  * preempt_count() is commonly implemented with READ_ONCE().
  */
 
-#define nmi_count()	(preempt_count() & NMI_MASK)
-#define hardirq_count()	(preempt_count() & HARDIRQ_MASK)
+#define nmi_count()		(preempt_count() & NMI_MASK)
+#define hardirq_count()		(preempt_count() & HARDIRQ_MASK)
 #ifdef CONFIG_PREEMPT_RT
 # define softirq_count()	(current->softirq_disable_cnt & SOFTIRQ_MASK)
 # define irq_count()		((preempt_count() & (NMI_MASK | HARDIRQ_MASK)) | softirq_count())
@@ -132,6 +145,27 @@ static __always_inline unsigned char interrupt_context_level(void)
 # define in_task()		(!(preempt_count() & (NMI_MASK | HARDIRQ_MASK | SOFTIRQ_OFFSET)))
 #endif
 
+#ifndef CONFIG_PREEMPT_LONG
+DECLARE_PER_CPU(unsigned int, nmi_nesting);
+
+#define __preempt_count_nmi_enter()				\
+	do {							\
+		unsigned int _o = NMI_MASK + HARDIRQ_OFFSET;	\
+		__this_cpu_inc(nmi_nesting);			\
+		_o -= (preempt_count() & NMI_MASK);		\
+		__preempt_count_add(_o);			\
+	} while (0)
+
+#define __preempt_count_nmi_exit()				\
+	do {							\
+		unsigned int _o = HARDIRQ_OFFSET;		\
+		if (!__this_cpu_dec_return(nmi_nesting))	\
+			_o += NMI_MASK;				\
+		__preempt_count_sub(_o);			\
+	} while (0)
+
+#endif
+
 /*
  * The following macros are deprecated and should not be used in new code:
  * in_softirq()   - We have BH disabled, or are processing softirqs
init/main.c +1 -1
index b84818ad9685..f8f4b78b7a06 100644
@@ -1367,7 +1367,7 @@ static inline void do_trace_initcall_level(const char *level)
 
 int __init_or_module do_one_initcall(initcall_t fn)
 {
-	int count = preempt_count();
+	long count = preempt_count();
 	char msgbuf[64];
 	int ret;
 
kernel/Kconfig.preempt +4 -0
index 88c594c6d7fc..2ad9365915eb 100644
@@ -122,6 +122,10 @@ config PREEMPT_RT_NEEDS_BH_LOCK
 config PREEMPT_COUNT
        bool
 
+config PREEMPT_LONG
+	bool
+	depends on PREEMPT_COUNT && 64BIT
+
 config PREEMPTION
        bool
        select PREEMPT_COUNT
kernel/sched/core.c +4 -4
index b411e4feff7f..f54dd3cb66f2 100644
@@ -5709,7 +5709,7 @@ static inline void sched_tick_stop(int cpu) { }
  * If the value passed in is equal to the current preempt count
  * then we just disabled preemption. Start timing the latency.
  */
-static inline void preempt_latency_start(int val)
+static inline void preempt_latency_start(long val)
 {
 	if (preempt_count() == val) {
 		unsigned long ip = get_lock_parent_ip();
@@ -5746,7 +5746,7 @@ NOKPROBE_SYMBOL(preempt_count_add);
  * If the value passed in equals to the current preempt count
  * then we just enabled preemption. Stop timing the latency.
  */
-static inline void preempt_latency_stop(int val)
+static inline void preempt_latency_stop(long val)
 {
 	if (preempt_count() == val)
 		trace_preempt_on(CALLER_ADDR0, get_lock_parent_ip());
@@ -8774,7 +8774,7 @@ void __might_sleep(const char *file, int line)
 }
 EXPORT_SYMBOL(__might_sleep);
 
-static void print_preempt_disable_ip(int preempt_offset, unsigned long ip)
+static void print_preempt_disable_ip(long preempt_offset, unsigned long ip)
 {
 	if (!IS_ENABLED(CONFIG_DEBUG_PREEMPT))
 		return;
@@ -8846,7 +8846,7 @@ void __might_resched(const char *file, int line, unsigned int offsets)
 }
 EXPORT_SYMBOL(__might_resched);
 
-void __cant_sleep(const char *file, int line, int preempt_offset)
+void __cant_sleep(const char *file, int line, long preempt_offset)
 {
 	static unsigned long prev_jiffy;
 
kernel/softirq.c +9 -1
index 77198911b8dd..51a7f391edab 100644
@@ -88,6 +88,14 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
+#ifndef CONFIG_PREEMPT_LONG
+/*
+ * Any 32bit architecture that still cares about performance should
+ * probably ensure this is near preempt_count.
+ */
+DEFINE_PER_CPU(unsigned int, nmi_nesting);
+#endif
+
 /*
  * SOFTIRQ_OFFSET usage:
  *
@@ -609,7 +617,7 @@ static void handle_softirqs(bool ksirqd)
 
 	while ((softirq_bit = ffs(pending))) {
 		unsigned int vec_nr;
-		int prev_count;
+		long prev_count;
 
 		h += softirq_bit - 1;
 
kernel/time/timer.c +1 -1
index 1f2364126894..89c348139218 100644
@@ -1723,7 +1723,7 @@ static void call_timer_fn(struct timer_list *timer,
 			  void (*fn)(struct timer_list *),
 			  unsigned long baseclk)
 {
-	int count = preempt_count();
+	long count = preempt_count();
 
 #ifdef CONFIG_LOCKDEP
 	/*
lib/locking-selftest.c +1 -1
index d939403331b5..8fd216bd0be6 100644
@@ -1429,7 +1429,7 @@ static int unexpected_testcase_failures;
 
 static void dotest(void (*testcase_fn)(void), int expected, int lockclass_mask)
 {
-	int saved_preempt_count = preempt_count();
+	long saved_preempt_count = preempt_count();
 #ifdef CONFIG_PREEMPT_RT
 #ifdef CONFIG_SMP
 	int saved_mgd_count = current->migration_disabled;
#10 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

Quoted 4 lines · Boqun Feng
Quoted 3 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra

But I'm really somewhat sad that 64bit can't do better than this.

Quoted 8 lines
Quoted 3 lines

Here, the below builds and boots (albeit with warnings because printf format crap sucks).

Thanks! I will drop patch #1 and #2 and use this one (with a commit log and some more tests), given it's based on the work of Joel, Lyude and me, would the following tags make sense to all of you?

Quoted 1 line

I don't know, I am not a big fan of the alternative patch because it adds a per-cpu counter anyway if !CONFIG_PREEMPT_LONG [1]. And it is also a much bigger patch than the one I wrote. Purely from an objective perspective, I would still want to keep my original patch because it is simple. What is really the objection to it?

[1]
+#ifndef CONFIG_PREEMPT_LONG
+/*
+ * Any 32bit architecture that still cares about performance should
+ * probably ensure this is near preempt_count.
+ */
+DEFINE_PER_CPU(unsigned int, nmi_nesting);
+#endif

Thanks,
--
Joel Fernandes

#11 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

Quoted 24 lines · Joel Fernandes
Quoted 4 lines · Boqun Feng
Quoted 3 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra

But I'm really somewhat sad that 64bit can't do better than this.

Quoted 8 lines
Quoted 3 lines

Here, the below builds and boots (albeit with warnings because printf format crap sucks).

Thanks! I will drop patch #1 and #2 and use this one (with a commit log and some more tests), given it's based on the work of Joel, Lyude and me, would the following tags make sense to all of you?

Quoted 1 line

I don't know, I am not a big fan of the alternative patch because it adds a per-cpu counter anyway if !CONFIG_PREEMPT_LONG [1]. And it is also a much bigger patch than the one I wrote. Purely from an objective perspective, I would still want to keep my original patch because it is simple. What is really the objection to it?

[1]

kernel/softirq.c EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled); · line 97

Quoted 7 lines
kernel/softirq.c +7 -0
+#ifndef CONFIG_PREEMPT_LONG
+/*
+ * Any 32bit architecture that still cares about performance should
+ * probably ensure this is near preempt_count.
+ */
+DEFINE_PER_CPU(unsigned int, nmi_nesting);
+#endif

If the objection to my patch is modifying a per-cpu counter, isn't NMI a slow path? If we agree, then keeping things simple is better IMO unless we have data showing that it is an issue. This is code is already quite convoluted, let us not convolute it more with 32-bit specific things.

I had tried moving it to DEFINE_PER_CPU_CACHE_HOT, but ISTR that did not work out (I think something about a limit to how many things could be moved to cache hot).

Happy to revise patch again with any other suggestions,

--
Joel Fernandes

maintainer
#12 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

Quoted 17 lines · Joel Fernandes
Quoted 14 lines · Joel Fernandes
Quoted 11 lines · Boqun Feng
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra

But I'm really somewhat sad that 64bit can't do better than this.

Here, the below builds and boots (albeit with warnings because printf format crap sucks).

Thanks! I will drop patch #1 and #2 and use this one (with a commit log and some more tests), given it's based on the work of Joel, Lyude and me, would the following tags make sense to all of you?

Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line

I don't know, I am not a big fan of the alternative patch because it adds a per-cpu counter anyway if !CONFIG_PREEMPT_LONG [1]. And it is also a much bigger patch than the one I wrote. Purely from an objective perspective, I would still want to keep my original patch because it is simple. What is really the objection to it?

PREEMPT_LONG is an architecture-specific way to improve the performance IMO. Just to be clear, do you object it at all, or do you object combining it with your original patch? If it's the latter, I could make another patch as a follow to enable PREEMPT_LONG.

Quoted 11 lines

kernel/softirq.c EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled); · line 97

Quoted 9 lines

[1]

kernel/softirq.c +7 -0
+#ifndef CONFIG_PREEMPT_LONG
+/*
+ * Any 32bit architecture that still cares about performance should
+ * probably ensure this is near preempt_count.
+ */
+DEFINE_PER_CPU(unsigned int, nmi_nesting);
+#endif

If the objection to my patch is modifying a per-cpu counter, isn't NMI a slow path? If we agree, then keeping things simple is better IMO unless we have data

I guess Peter was trying to say it's not a slow path if you consider perf event interrupts on x86? [1]

Quoted 9 lines

showing that it is an issue. This is code is already quite convoluted, let us not convolute it more with 32-bit specific things.

I had tried moving it to DEFINE_PER_CPU_CACHE_HOT, but ISTR that did not work out (I think something about a limit to how many things could be moved to cache hot).

Happy to revise patch again with any other suggestions,

[1]: https://lore.kernel.org/rust-for-linux/[email protected]/

Regards,
Boqun

Quoted 3 lines

--
Joel Fernandes

#13 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

Quoted 27 lines · Boqun Feng
Quoted 17 lines · Joel Fernandes
Quoted 14 lines · Joel Fernandes
Quoted 11 lines · Boqun Feng
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra

But I'm really somewhat sad that 64bit can't do better than this.

Here, the below builds and boots (albeit with warnings because printf format crap sucks).

Thanks! I will drop patch #1 and #2 and use this one (with a commit log and some more tests), given it's based on the work of Joel, Lyude and me, would the following tags make sense to all of you?

Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line

I don't know, I am not a big fan of the alternative patch because it adds a per-cpu counter anyway if !CONFIG_PREEMPT_LONG [1]. And it is also a much bigger patch than the one I wrote. Purely from an objective perspective, I would still want to keep my original patch because it is simple. What is really the objection to it?

Quoted 4 lines

PREEMPT_LONG is an architecture-specific way to improve the performance IMO. Just to be clear, do you object it at all, or do you object combining it with your original patch? If it's the latter, I could make another patch as a follow to enable PREEMPT_LONG.

When I looked at the alternative patch, I did consider that it was overcomplicated and it should be justified. Otherwise, I don't object to it. It seems to be a matter of preference I think. I would prefer a simpler fix than an overcomplicated fix for a hypothetical issue (unless we have data showing issue). If it was a few lines of change, that'd be different story.

Quoted 13 lines
Quoted 12 lines

kernel/softirq.c EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled); · line 97

Quoted 10 lines

[1]

kernel/softirq.c +7 -0
+#ifndef CONFIG_PREEMPT_LONG
+/*
+ * Any 32bit architecture that still cares about performance should
+ * probably ensure this is near preempt_count.
+ */
+DEFINE_PER_CPU(unsigned int, nmi_nesting);
+#endif

If the objection to my patch is modifying a per-cpu counter, isn't NMI a slow path? If we agree, then keeping things simple is better IMO unless we have data

Quoted 2 lines

I guess Peter was trying to say it's not a slow path if you consider perf event interrupts on x86? [1]

How are we handling this performance issue then on 32-bit x86 architecture with perf? Or are we saying we don't care about performance on 32-bit?

Signature · 1 line
Joel Fernandes
maintainer
#14 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

Quoted 30 lines · Joel Fernandes
Quoted 27 lines · Boqun Feng
Quoted 25 lines · Joel Fernandes
Quoted 22 lines · Joel Fernandes
Quoted 4 lines · Boqun Feng
Quoted 3 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra

But I'm really somewhat sad that 64bit can't do better than this.

Quoted 8 lines
Quoted 3 lines

Here, the below builds and boots (albeit with warnings because printf format crap sucks).

Thanks! I will drop patch #1 and #2 and use this one (with a commit log and some more tests), given it's based on the work of Joel, Lyude and me, would the following tags make sense to all of you?

Quoted 1 line

I don't know, I am not a big fan of the alternative patch because it adds a per-cpu counter anyway if !CONFIG_PREEMPT_LONG [1]. And it is also a much bigger patch than the one I wrote. Purely from an objective perspective, I would still want to keep my original patch because it is simple. What is really the objection to it?

Quoted 7 lines
Quoted 4 lines

PREEMPT_LONG is an architecture-specific way to improve the performance IMO. Just to be clear, do you object it at all, or do you object combining it with your original patch? If it's the latter, I could make another patch as a follow to enable PREEMPT_LONG.

When I looked at the alternative patch, I did consider that it was overcomplicated and it should be justified. Otherwise, I don't object to it. It

I don't think that's overcomplicated. Note that people have different goals, for us (you, Lyude and me), we want to have a safer interrupt-disabling lock API, hence this patchset. I think Peter on the other hand while agreeing with us on the necessity, but wants to avoid potential performance lost (maybe in general also likes the idea of preempt_count being 64bit on 64bit machines ;-)) That patch looks "overcomplicated" because it contains both goals (it actually contains patch #1 and #2 along with the improvement). If you look them separately, it would be not that complicated (Peter's diff against patch 1 + 2 will be relatively small).

Quoted 17 lines

seems to be a matter of preference I think. I would prefer a simpler fix than an overcomplicated fix for a hypothetical issue (unless we have data showing issue). If it was a few lines of change, that'd be different story.

Quoted 13 lines
Quoted 12 lines

kernel/softirq.c EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled); · line 97

Quoted 10 lines

[1]

kernel/softirq.c +7 -0
+#ifndef CONFIG_PREEMPT_LONG
+/*
+ * Any 32bit architecture that still cares about performance should
+ * probably ensure this is near preempt_count.
+ */
+DEFINE_PER_CPU(unsigned int, nmi_nesting);
+#endif

If the objection to my patch is modifying a per-cpu counter, isn't NMI a slow path? If we agree, then keeping things simple is better IMO unless we have data

Quoted 6 lines
Quoted 2 lines

I guess Peter was trying to say it's not a slow path if you consider perf event interrupts on x86? [1]

How are we handling this performance issue then on 32-bit x86 architecture with perf? Or are we saying we don't care about performance on 32-bit?

I'm not in the position to answer this (mostly for the second question). Either we have data proving that the performance gap caused by your original patch is small enough (if there is any) or it's up to x86 maintainers.

Regards,
Boqun

Quoted 3 lines
Signature · 1 line
Joel Fernandes
#15 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

Quoted 38 lines · Boqun Feng
Quoted 35 lines · Joel Fernandes
Quoted 34 lines · Boqun Feng
Quoted 17 lines · Joel Fernandes
Quoted 14 lines · Joel Fernandes
Quoted 11 lines · Boqun Feng
Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Peter Zijlstra

But I'm really somewhat sad that 64bit can't do better than this.

Here, the below builds and boots (albeit with warnings because printf format crap sucks).

Thanks! I will drop patch #1 and #2 and use this one (with a commit log and some more tests), given it's based on the work of Joel, Lyude and me, would the following tags make sense to all of you?

Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line

I don't know, I am not a big fan of the alternative patch because it adds a per-cpu counter anyway if !CONFIG_PREEMPT_LONG [1]. And it is also a much bigger patch than the one I wrote. Purely from an objective perspective, I would still want to keep my original patch because it is simple. What is really the objection to it?

PREEMPT_LONG is an architecture-specific way to improve the performance IMO. Just to be clear, do you object it at all, or do you object combining it with your original patch? If it's the latter, I could make another patch as a follow to enable PREEMPT_LONG.

Quoted 6 lines
Quoted 2 lines

When I looked at the alternative patch, I did consider that it was overcomplicated and it should be justified. Otherwise, I don't object to it. It

I don't think that's overcomplicated. Note that people have different goals, for us (you, Lyude and me), we want to have a safer interrupt-disabling lock API, hence this patchset.

I was also coming from the goal of long term kernel code maintainability. If we decide to have additional preempt count flags in the future, does special casing 32 bit add even more complexity? (not rhetorical, really asking)

Quoted 8 lines

I think Peter on the
other hand while agreeing with us on the necessity, but wants to avoid potential performance lost (maybe in general also likes the idea of preempt_count being 64bit on 64bit machines ;-)) That patch looks "overcomplicated" because it contains both goals (it actually contains patch #1 and #2 along with the improvement). If you look them separately, it would be not that complicated (Peter's diff against patch 1 + 2 will be relatively small).

On further looking, I think my hesitation is mostly around the extra config option and special casing of 32 bit as mentioned above. But answering your other question, if it is decided to go with Peter's patch, you can use my codevelop tag.

Signature · 1 line
Joel Fernandes
maintainer
#16 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

On Fri, Feb 06, 2026 at 03:13:40AM -0500, Joel Fernandes wrote:
[..]

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

PREEMPT_LONG is an architecture-specific way to improve the performance IMO. Just to be clear, do you object it at all, or do you object combining it with your original patch? If it's the latter, I could make another patch as a follow to enable PREEMPT_LONG.

When I looked at the alternative patch, I did consider that it was overcomplicated and it should be justified. Otherwise, I don't object to it. It

Quoted 8 lines
Quoted 3 lines

I don't think that's overcomplicated. Note that people have different goals, for us (you, Lyude and me), we want to have a safer interrupt-disabling lock API, hence this patchset.

I was also coming from the goal of long term kernel code maintainability. If we decide to have additional preempt count flags in the future, does special casing 32 bit add even more complexity? (not rhetorical, really asking)

First, given what preempt count is, I don't think that'll happen frequently. Also I think the reality is that we care about 64bit performance more than 32bit, in that sense, if this "conditional 32 bit preempt count case" becomes an issue, the reasonable action to me is just making all preempt count 64bit (using an irq disabling critical section for 32bit or plus a special locking), and this would make things simpler. That's the long term view from me. (Now think about this, the NMI tracking we proposed in this patch is actually a special case of that ;-))

Quoted 9 lines
Quoted 8 lines

I think Peter on the
other hand while agreeing with us on the necessity, but wants to avoid potential performance lost (maybe in general also likes the idea of preempt_count being 64bit on 64bit machines ;-)) That patch looks "overcomplicated" because it contains both goals (it actually contains patch #1 and #2 along with the improvement). If you look them separately, it would be not that complicated (Peter's diff against patch 1 + 2 will be relatively small).

Quoted 5 lines

On further looking, I think my hesitation is mostly around the extra config option and special casing of 32 bit as mentioned above. But answering your other question, if it is decided to go with Peter's patch, you can use my codevelop tag.

Thank you! But I realized more things are needed, so we probably should add PREEMPT_LONG as a follow-up (for example, should_resched() should take a long instead of int, and the print format issues that Peter mentioned).

Regards,
Boqun

Quoted 3 lines
Signature · 1 line
Joel Fernandes
#17 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

Quoted 6 lines · Boqun Feng
Quoted 4 lines

I was also coming from the goal of long term kernel code maintainability. If we decide to have additional preempt count flags in the future, does special casing 32 bit add even more complexity? (not rhetorical, really asking)

First, given what preempt count is, I don't think that'll happen frequently.

Not sure I buy the argument of not happening frequently. I don't think any of us have a crystal ball. There are cases in the future that can come up IMO.

Quoted 1 line

Also I think the reality is that we care about 64bit> performance more than

32bit, in that sense, if this "conditional 32 bit

Quoted 2 lines

preempt count case" becomes an issue, the reasonable action to me is just making all preempt count 64bit

You might be missing something here. You can't make all of preempt count 64 bit, that's the point, it doesn't work. That's why Peter did what he did to special-case 32 bit. See:
https://lore.kernel.org/all/20251020204421.GA197647@joelbox2/

That said, I am ok with the approach now that Peter mentions 32-bit x86 is "deprecated". :-)

Signature · 1 line
Joel Fernandes
maintainer
#18 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

Quoted 7 lines · Joel Fernandes
Quoted 4 lines · Boqun Feng
Quoted 4 lines

I was also coming from the goal of long term kernel code maintainability. If we decide to have additional preempt count flags in the future, does special casing 32 bit add even more complexity? (not rhetorical, really asking)

Quoted 6 lines
Quoted 2 lines

First, given what preempt count is, I don't think that'll happen frequently.

Not sure I buy the argument of not happening frequently. I don't think any of us have a crystal ball. There are cases in the future that can come up IMO.

It's just being realistic, and we pretty much use all the bits there.

Quoted 5 lines
Quoted 1 line

Also I think the reality is that we care about 64bit> performance more than

32bit, in that sense, if this "conditional 32 bit

Quoted 2 lines

preempt count case" becomes an issue, the reasonable action to me is just making all preempt count 64bit

Quoted 8 lines

You might be missing something here. You can't make all of preempt count 64 bit, that's the point, it doesn't work. That's why Peter did what he did to special-case 32 bit. See:
https://lore.kernel.org/all/20251020204421.GA197647@joelbox2/

That said, I am ok with the approach now that Peter mentions 32-bit x86 is "deprecated". :-)

Yeah, "can't" is a strong word? ;-) I did say if we care more about performance on 64bit than 32bit and can afford slowing down 32bit preemption disabling in the case where "we decide to have additional preempt count flags", THEN we can make all preempt count 64bit.

Regards,
Boqun

Quoted 3 lines
Signature · 1 line
Joel Fernandes
#19 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

On Fri, Feb 06, 2026 at 08:16:39AM -0800, Boqun Feng wrote:
[...]

Quoted 4 lines

Yeah, "can't" is a strong word? ;-) I did say if we care more about performance on 64bit than 32bit and can afford slowing down 32bit preemption disabling in the case where "we decide to have additional preempt count flags", THEN we can make all preempt count 64bit.

Ok, hopefully x86 32-bit goes away before we have to maintain/improve these workarounds in it. :-)

Signature · 5 lines
Joel Fernandes

-- 
-- 
Joel Fernandes
maintainer
#20 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

In reply to Joel Fernandes
Quoted 5 lines · Joel Fernandes
Quoted 2 lines

I guess Peter was trying to say it's not a slow path if you consider perf event interrupts on x86? [1]

How are we handling this performance issue then on 32-bit x86 architecture with perf? Or are we saying we don't care about performance on 32-bit?

Yeah, in general I don't consider any 32bit architecture performance critical at this point. Its pure legacy code, to be removed at some point.

To x86_32 in particular, we make it limp along. It sorta builds and sorta boots but meh. It doesn't even have most of the speculation fixes. You really, as in *REALLY* should not be running a x86_32 kernel.

I mean, if you still want to run Linux on your museum grade Pentium-II processor, don't let me stop you. Just don't expect miracles.

maintainer
#21 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

In reply to Peter Zijlstra

On Wed, Feb 04, 2026 at 12:12:34PM +0100, Peter Zijlstra wrote:
[...]

Quoted 36 lines

DEFINE_PER_CPU_CACHE_HOT(unsigned long, cpu_current_top_of_stack) = TOP_OF_INIT_STACK;

include/linux/hardirq.h +3 -4
index d57cab4d4c06..77defd9624bf 100644
@@ -108,15 +108,14 @@ void irq_exit_rcu(void);
 	do {							\
 		lockdep_off();					\
 		arch_nmi_enter();				\
-		BUG_ON(in_nmi() == NMI_MASK);			\
-		__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		__preempt_count_nmi_enter();			\
 	} while (0)
 
 #define nmi_enter()						\
 	do {							\
 		__nmi_enter();					\
 		lockdep_hardirq_enter();			\
-		ct_nmi_enter();				\
+		ct_nmi_enter();					\
 		instrumentation_begin();			\
 		ftrace_nmi_enter();				\
 		instrumentation_end();				\
@@ -125,7 +124,7 @@ void irq_exit_rcu(void);
 #define __nmi_exit()						\
 	do {							\
 		BUG_ON(!in_nmi());				\
-		__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		__preempt_count_nmi_exit();			\
 		arch_nmi_exit();				\
 		lockdep_on();					\
 	} while (0)
include/linux/preempt.h +0 -0
index d964f965c8ff..7617ca97f442 100644

[...]

Quoted 15 lines
include/linux/preempt.h +11 -0
@@ -132,6 +145,27 @@ static __always_inline unsigned char interrupt_context_level(void)
 # define in_task()		(!(preempt_count() & (NMI_MASK | HARDIRQ_MASK | SOFTIRQ_OFFSET)))
 #endif
 
+#ifndef CONFIG_PREEMPT_LONG
+DECLARE_PER_CPU(unsigned int, nmi_nesting);
+
+#define __preempt_count_nmi_enter()				\
+	do {							\
+		unsigned int _o = NMI_MASK + HARDIRQ_OFFSET;	\
+		__this_cpu_inc(nmi_nesting);			\
+		_o -= (preempt_count() & NMI_MASK);		\
+		__preempt_count_add(_o);			\
+	} while (0)
+

include/linux/preempt.h static __always_inline unsigned char interrupt_context_level(void) · line 168

Quoted 10 lines
include/linux/preempt.h +10 -0
+#define __preempt_count_nmi_exit()				\
+	do {							\
+		unsigned int _o = HARDIRQ_OFFSET;		\
+		if (!__this_cpu_dec_return(nmi_nesting))	\
+			_o += NMI_MASK;				\
+		__preempt_count_sub(_o);			\
+	} while (0)
+
+#endif
+

We need to move it into include/linux/hardirq.h because percpu is not included in <linux/preempt.h>.

Regards,
Boqun

Quoted 3 lines
include/linux/preempt.h +0 -0
 /*
  * The following macros are deprecated and should not be used in new code:
  * in_softirq()   - We have BH disabled, or are processing softirqs
maintainer
#22 lore raw

Re: [PATCH v17 02/16] preempt: Track NMI nesting to separate per-CPU counter

Quoted 59 lines · Boqun Feng

On Wed, Feb 04, 2026 at 12:12:34PM +0100, Peter Zijlstra wrote:
[...]

Quoted 36 lines

DEFINE_PER_CPU_CACHE_HOT(unsigned long, cpu_current_top_of_stack) = TOP_OF_INIT_STACK;

include/linux/hardirq.h +3 -4
index d57cab4d4c06..77defd9624bf 100644
@@ -108,15 +108,14 @@ void irq_exit_rcu(void);
 	do {							\
 		lockdep_off();					\
 		arch_nmi_enter();				\
-		BUG_ON(in_nmi() == NMI_MASK);			\
-		__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		__preempt_count_nmi_enter();			\
 	} while (0)
 
 #define nmi_enter()						\
 	do {							\
 		__nmi_enter();					\
 		lockdep_hardirq_enter();			\
-		ct_nmi_enter();				\
+		ct_nmi_enter();					\
 		instrumentation_begin();			\
 		ftrace_nmi_enter();				\
 		instrumentation_end();				\
@@ -125,7 +124,7 @@ void irq_exit_rcu(void);
 #define __nmi_exit()						\
 	do {							\
 		BUG_ON(!in_nmi());				\
-		__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		__preempt_count_nmi_exit();			\
 		arch_nmi_exit();				\
 		lockdep_on();					\
 	} while (0)
include/linux/preempt.h +0 -0
index d964f965c8ff..7617ca97f442 100644

[...]

Quoted 20 lines
include/linux/preempt.h +16 -0
@@ -132,6 +145,27 @@ static __always_inline unsigned char interrupt_context_level(void)
 # define in_task()		(!(preempt_count() & (NMI_MASK | HARDIRQ_MASK | SOFTIRQ_OFFSET)))
 #endif
 
+#ifndef CONFIG_PREEMPT_LONG
+DECLARE_PER_CPU(unsigned int, nmi_nesting);
+
+#define __preempt_count_nmi_enter()				\
+	do {							\
+		unsigned int _o = NMI_MASK + HARDIRQ_OFFSET;	\
+		__this_cpu_inc(nmi_nesting);			\
+		_o -= (preempt_count() & NMI_MASK);		\
+		__preempt_count_add(_o);			\
+	} while (0)
+
+#define __preempt_count_nmi_exit()				\
+	do {							\
+		unsigned int _o = HARDIRQ_OFFSET;		\
+		if (!__this_cpu_dec_return(nmi_nesting))	\
+			_o += NMI_MASK;				\
Quoted 8 lines

include/linux/preempt.h static __always_inline unsigned char interrupt_context_level(void) · line 168

Quoted 5 lines
include/linux/preempt.h +5 -0
+		__preempt_count_sub(_o);			\
+	} while (0)
+
+#endif
+

We need to move it into include/linux/hardirq.h because percpu is not included in <linux/preempt.h>.

That is fine. I also realized you can move the variants from arch/x86/asm/preempt.h right next to it, it only depends on PREEMPT_LONG, not anything else, so there is nothing arch specific to it.

Avoids that getting duplicated on arm64,s390 etc.

author reviewer
#23 lore raw

PATCH v17 3/16 preempt: Introduce __preempt_count_{sub, add}_return()

In reply to Lyude Paul

From: Boqun Feng <[email protected]>

In order to use preempt_count() to tracking the interrupt disable nesting level, __preempt_count_{add,sub}_return() are introduced, as their name suggest, these primitives return the new value of the preempt_count() after changing it. The following example shows the usage of it in local_interrupt_disable():

	// increase the HARDIRQ_DISABLE bit
	new_count = __preempt_count_add_return(HARDIRQ_DISABLE_OFFSET);

	// if it's the first-time increment, then disable the interrupt
	// at hardware level.
	if (new_count & HARDIRQ_DISABLE_MASK == HARDIRQ_DISABLE_OFFSET) {
		local_irq_save(flags);
		raw_cpu_write(local_interrupt_disable_state.flags, flags);
	}

Having these primitives will avoid a read of preempt_count() after changing preempt_count() on certain architectures.

---
V10:
* Add commit message I forgot
* Rebase against latest pcpu_hot changes V11:
* Remove CONFIG_PROFILE_ALL_BRANCHES workaround from __preempt_count_add_return()

 arch/arm64/include/asm/preempt.h | 18 ++++++++++++++++++
 arch/s390/include/asm/preempt.h  | 10 ++++++++++
 arch/x86/include/asm/preempt.h   | 10 ++++++++++
 include/asm-generic/preempt.h    | 14 ++++++++++++++
 4 files changed, 52 insertions(+)
arch/arm64/include/asm/preempt.h +18 -0
index 932ea4b620428..0dd8221d1bef7 100644
@@ -55,6 +55,24 @@ static inline void __preempt_count_sub(int val)
 	WRITE_ONCE(current_thread_info()->preempt.count, pc);
 }
 
+static inline int __preempt_count_add_return(int val)
+{
+	u32 pc = READ_ONCE(current_thread_info()->preempt.count);
+	pc += val;
+	WRITE_ONCE(current_thread_info()->preempt.count, pc);
+
+	return pc;
+}
+
+static inline int __preempt_count_sub_return(int val)
+{
+	u32 pc = READ_ONCE(current_thread_info()->preempt.count);
+	pc -= val;
+	WRITE_ONCE(current_thread_info()->preempt.count, pc);
+
+	return pc;
+}
+
 static inline bool __preempt_count_dec_and_test(void)
 {
 	struct thread_info *ti = current_thread_info();
arch/s390/include/asm/preempt.h +10 -0
index 6ccd033acfe52..5ae366e26c57d 100644
@@ -98,6 +98,16 @@ static __always_inline bool should_resched(int preempt_offset)
 	return unlikely(READ_ONCE(get_lowcore()->preempt_count) == preempt_offset);
 }
 
+static __always_inline int __preempt_count_add_return(int val)
+{
+	return val + __atomic_add(val, &get_lowcore()->preempt_count);
+}
+
+static __always_inline int __preempt_count_sub_return(int val)
+{
+	return __preempt_count_add_return(-val);
+}
+
 #define init_task_preempt_count(p)	do { } while (0)
 /* Deferred to CPU bringup time */
 #define init_idle_preempt_count(p, cpu)	do { } while (0)
arch/x86/include/asm/preempt.h +10 -0
index 578441db09f0b..1220656f3370b 100644
@@ -85,6 +85,16 @@ static __always_inline void __preempt_count_sub(int val)
 	raw_cpu_add_4(__preempt_count, -val);
 }
 
+static __always_inline int __preempt_count_add_return(int val)
+{
+	return raw_cpu_add_return_4(__preempt_count, val);
+}
+
+static __always_inline int __preempt_count_sub_return(int val)
+{
+	return raw_cpu_add_return_4(__preempt_count, -val);
+}
+
 /*
  * Because we keep PREEMPT_NEED_RESCHED set when we do _not_ need to reschedule
  * a decrement which hits zero means we have no preempt_count and should
include/asm-generic/preempt.h +14 -0
index 51f8f3881523a..c8683c046615d 100644
@@ -59,6 +59,20 @@ static __always_inline void __preempt_count_sub(int val)
 	*preempt_count_ptr() -= val;
 }
 
+static __always_inline int __preempt_count_add_return(int val)
+{
+	*preempt_count_ptr() += val;
+
+	return *preempt_count_ptr();
+}
+
+static __always_inline int __preempt_count_sub_return(int val)
+{
+	*preempt_count_ptr() -= val;
+
+	return *preempt_count_ptr();
+}
+
 static __always_inline bool __preempt_count_dec_and_test(void)
 {
 	/*
Signature · 1 line
2.52.0
author reviewer
#24 lore raw

PATCH v17 4/16 openrisc: Include <linux/cpumask.h> in smp.h

In reply to Lyude Paul

While OpenRISC currently doesn't fail to build upstream, it appears that include <asm/smp.h> in the right headers is enough to break that - primarily because OpenRISC's asm/smp.h header doesn't actually provide any definition for struct cpumask. Which means the only reason we aren't failing to build kernel is because we've been lucky enough that every spot including asm/smp.h already has definitions for struct cpumask pulled in.

This became evident when trying to work on a patch series for adding ref-counted interrupt enable/disables to the kernel, where introducing a new interrupt_rc.h header suddenly introduced a build error on OpenRISC:

In file included from include/linux/interrupt_rc.h:17,
from include/linux/spinlock.h:60,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:7,
from include/linux/mm.h:7,
from arch/openrisc/include/asm/pgalloc.h:20,
from arch/openrisc/include/asm/io.h:18,
from include/linux/io.h:12,
from drivers/irqchip/irq-ompic.c:61:
arch/openrisc/include/asm/smp.h:21:59: warning: 'struct cpumask' declared inside parameter list will not be visible outside of this definition or declaration
21 | extern void arch_send_call_function_ipi_mask(const struct cpumask *mask);
| ^~~~~~~
arch/openrisc/include/asm/smp.h:23:54: warning: 'struct cpumask' declared inside parameter list will not be visible outside of this definition or declaration
23 | extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));
| ^~~~~~~
drivers/irqchip/irq-ompic.c: In function 'ompic_of_init':

Quoted 1 line
Quoted 1 line

drivers/irqchip/irq-ompic.c:191:28: error: passing argument 1 of

'set_smp_cross_call' from incompatible pointer type [-Werror=incompatible-pointer-types]
191 | set_smp_cross_call(ompic_raise_softirq);
| ^~~~~~~~~~~~~~~~~~~
| |
| void (*)(const struct cpumask *, unsigned int)
arch/openrisc/include/asm/smp.h:23:32: note: expected 'void (*)(const struct cpumask *, unsigned int)' but argument is of type 'void
(*)(const struct cpumask *, unsigned int)'
23 | extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));

To fix this, let's take an example from the smp.h headers of other architectures (x86, hexagon, arm64, probably more): just include linux/cpumask.h at the top.

---
 arch/openrisc/include/asm/smp.h | 2 ++
 1 file changed, 2 insertions(+)
arch/openrisc/include/asm/smp.h +2 -0
index e21d2f12b5b67..0327d8cdae2d0 100644
@@ -9,6 +9,8 @@
 #ifndef __ASM_OPENRISC_SMP_H
 #define __ASM_OPENRISC_SMP_H
 
+#include <linux/cpumask.h>
+
 #include <asm/spr.h>
 #include <asm/spr_defs.h>
 
Signature · 1 line
2.52.0
author reviewer
#25 lore raw

PATCH v17 5/16 irq & spin_lock: Add counted interrupt disabling/enabling

In reply to Lyude Paul

From: Boqun Feng <[email protected]>

Currently the nested interrupt disabling and enabling is present by _irqsave() and _irqrestore() APIs, which are relatively unsafe, for example:

	<interrupts are enabled as beginning>
	spin_lock_irqsave(l1, flag1);
	spin_lock_irqsave(l2, flag2);
	spin_unlock_irqrestore(l1, flags1);
	<l2 is still held but interrupts are enabled>
	// accesses to interrupt-disable protect data will cause races.

This is even easier to triggered with guard facilities:

	unsigned long flag2;

	scoped_guard(spin_lock_irqsave, l1) {
		spin_lock_irqsave(l2, flag2);
	}
	// l2 locked but interrupts are enabled.
	spin_unlock_irqrestore(l2, flag2);

(Hand-to-hand locking critical sections are not uncommon for a fine-grained lock design)

And because this unsafety, Rust cannot easily wrap the interrupt-disabling locks in a safe API, which complicates the design.

To resolve this, introduce a new set of interrupt disabling APIs:

* local_interrupt_disable();
* local_interrupt_enable();

They work like local_irq_save() and local_irq_restore() except that 1) the outermost local_interrupt_disable() call save the interrupt state into a percpu variable, so that the outermost local_interrupt_enable() can restore the state, and 2) a percpu counter is added to record the nest level of these calls, so that interrupts are not accidentally enabled inside the outermost critical section.

Also add the corresponding spin_lock primitives: spin_lock_irq_disable() and spin_unlock_irq_enable(), as a result, code as follow:

	spin_lock_irq_disable(l1);
	spin_lock_irq_disable(l2);
	spin_unlock_irq_enable(l1);
	// Interrupts are still disabled.
	spin_unlock_irq_enable(l2);

doesn't have the issue that interrupts are accidentally enabled.

This also makes the wrapper of interrupt-disabling locks on Rust easier to design.

---
V10:
* Add missing __raw_spin_lock_irq_disable() definition in spinlock.c
V11:
* Move definition of spin_trylock_irq_disable() into this commit
* Get rid of leftover space
* Remove unneeded preempt_disable()/preempt_enable()
V12:
* Move local_interrupt_enable()/local_interrupt_disable() out of include/linux/spinlock.h, into include/linux/irqflags.h
V14:
* Move local_interrupt_enable()/disable() again, this time into its own header, interrupt_rc.h, in order to fix a hexagon-specific build issue caught by the CKI bot.
The reason this is needed is because on most architectures, irqflags.h ends up including <arch/smp.h>. This provides a definition for the raw_smp_processor_id() function which we depend on like so:

<linux/percpu-defs.h> <arch/smp.h>
local_interrupt_disable() → raw_cpu_write() → raw_smp_processor_id()

Unfortunately, hexagon appears to be one such architecture which does not pull in <arch/smp.h> by default here - causing kernel builds to fail and claim that raw_smp_processor_id() is undefined:

In file included from kernel/sched/rq-offsets.c:5:
In file included from kernel/sched/sched.h:8:
In file included from include/linux/sched/affinity.h:1:
In file included from include/linux/sched.h:37:
In file included from include/linux/spinlock.h:59:
>> include/linux/irqflags.h:277:3: error: call to undeclared function
'raw_smp_processor_id'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
277 | raw_cpu_write(local_interrupt_disable_state.flags, flags);
| ^
include/linux/percpu-defs.h:413:34: note: expanded from macro 'raw_cpu_write'

While including <arch/smp.h> in <linux/irqflags.h> does fix the build on hexagon, it ends up breaking the build on x86_64:

In file included from kernel/sched/rq-offsets.c:5:
In file included from kernel/sched/sched.h:8:
In file included from ./include/linux/sched/affinity.h:1:
In file included from ./include/linux/sched.h:13:
In file included from ./arch/x86/include/asm/processor.h:25:
In file included from ./arch/x86/include/asm/special_insns.h:10:
In file included from ./include/linux/irqflags.h:22:
In file included from ./arch/x86/include/asm/smp.h:6:
In file included from ./include/linux/thread_info.h:60:
In file included from ./arch/x86/include/asm/thread_info.h:59:
./arch/x86/include/asm/cpufeature.h:110:40: error: use of undeclared identifier 'boot_cpu_data'

[cap_byte] "i" (&((const char *)boot_cpu_data.x86_capability)[bit >> 3])
^

While boot_cpu_data is defined in <asm/processor.h>, it's not possible for us to include that header in irqflags.h because we're already inside of <asm/processor.h>.

As a result, I just concluded there's no reasonable way of having these functions in <linux/irqflags.h> because of how many low level ASM headers depend on it. So, we go with the solution of simply giving ourselves our own header file.
V15:
* Fix build error on CONFIG_SMP=n reported by Kernel CI

 include/linux/interrupt_rc.h     | 63 ++++++++++++++++++++++++++++++++
 include/linux/preempt.h          |  4 ++
 include/linux/spinlock.h         | 25 +++++++++++++
 include/linux/spinlock_api_smp.h | 27 ++++++++++++++
 include/linux/spinlock_api_up.h  |  8 ++++
 include/linux/spinlock_rt.h      | 15 ++++++++
 kernel/locking/spinlock.c        | 29 +++++++++++++++
 kernel/softirq.c                 |  3 ++
 8 files changed, 174 insertions(+)
 create mode 100644 include/linux/interrupt_rc.h
include/linux/interrupt_rc.h new file +63 -0
new file mode 100644
index 0000000000000..d6d05498731b2
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * include/linux/interrupt_rc.h - refcounted local processor interrupt
+ * management.
+ *
+ * Since the implementation of this API currently depends on
+ * local_irq_save()/local_irq_restore(), we split this into it's own header to
+ * make it easier to include without hitting circular header dependencies.
+ */
+
+#ifndef __LINUX_INTERRUPT_RC_H
+#define __LINUX_INTERRUPT_RC_H
+
+#include <linux/irqflags.h>
+#include <asm/processor.h>
+#ifdef CONFIG_SMP
+#include <asm/smp.h>
+#endif
+
+/* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}() */
+struct interrupt_disable_state {
+	unsigned long flags;
+};
+
+DECLARE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state);
+
+static inline void local_interrupt_disable(void)
+{
+	unsigned long flags;
+	int new_count;
+
+	new_count = hardirq_disable_enter();
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET) {
+		local_irq_save(flags);
+		raw_cpu_write(local_interrupt_disable_state.flags, flags);
+	}
+}
+
+static inline void local_interrupt_enable(void)
+{
+	int new_count;
+
+	new_count = hardirq_disable_exit();
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == 0) {
+		unsigned long flags;
+
+		flags = raw_cpu_read(local_interrupt_disable_state.flags);
+		local_irq_restore(flags);
+		/*
+		 * TODO: re-read preempt count can be avoided, but it needs
+		 * should_resched() taking another parameter as the current
+		 * preempt count
+		 */
+#ifdef PREEMPTION
+		if (should_resched(0))
+			__preempt_schedule();
+#endif
+	}
+}
+
+#endif /* !__LINUX_INTERRUPT_RC_H */
include/linux/preempt.h +4 -0
index e2d3079d3f5f1..33fc4c814a9f0 100644
@@ -151,6 +151,10 @@ static __always_inline unsigned char interrupt_context_level(void)
 #define in_softirq()		(softirq_count())
 #define in_interrupt()		(irq_count())
 
+#define hardirq_disable_count()	((preempt_count() & HARDIRQ_DISABLE_MASK) >> HARDIRQ_DISABLE_SHIFT)
+#define hardirq_disable_enter()	__preempt_count_add_return(HARDIRQ_DISABLE_OFFSET)
+#define hardirq_disable_exit()	__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET)
+
 /*
  * The preempt_count offset after preempt_disable();
  */
include/linux/spinlock.h +25 -0
index d3561c4a080e2..bbbee61c6f5df 100644
@@ -57,6 +57,7 @@
 #include <linux/linkage.h>
 #include <linux/compiler.h>
 #include <linux/irqflags.h>
+#include <linux/interrupt_rc.h>
 #include <linux/thread_info.h>
 #include <linux/stringify.h>
 #include <linux/bottom_half.h>
@@ -272,9 +273,11 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
 #endif
 
 #define raw_spin_lock_irq(lock)		_raw_spin_lock_irq(lock)
+#define raw_spin_lock_irq_disable(lock)	_raw_spin_lock_irq_disable(lock)
 #define raw_spin_lock_bh(lock)		_raw_spin_lock_bh(lock)
 #define raw_spin_unlock(lock)		_raw_spin_unlock(lock)
 #define raw_spin_unlock_irq(lock)	_raw_spin_unlock_irq(lock)
+#define raw_spin_unlock_irq_enable(lock)	_raw_spin_unlock_irq_enable(lock)
 
 #define raw_spin_unlock_irqrestore(lock, flags)		\
 	do {							\
@@ -300,6 +303,13 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
 	1 : ({ local_irq_restore(flags); 0; }); \
 })
 
+#define raw_spin_trylock_irq_disable(lock) \
+({ \
+	local_interrupt_disable(); \
+	raw_spin_trylock(lock) ? \
+	1 : ({ local_interrupt_enable(); 0; }); \
+})
+
 #ifndef CONFIG_PREEMPT_RT
 /* Include rwlock functions for !RT */
 #include <linux/rwlock.h>
@@ -376,6 +386,11 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)
 	raw_spin_lock_irq(&lock->rlock);
 }
 
+static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
+{
+	raw_spin_lock_irq_disable(&lock->rlock);
+}
+
 #define spin_lock_irqsave(lock, flags)				\
 do {								\
 	raw_spin_lock_irqsave(spinlock_check(lock), flags);	\
@@ -401,6 +416,11 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)
 	raw_spin_unlock_irq(&lock->rlock);
 }
 
+static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
+{
+	raw_spin_unlock_irq_enable(&lock->rlock);
+}
+
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags)
 {
 	raw_spin_unlock_irqrestore(&lock->rlock, flags);
@@ -421,6 +441,11 @@ static __always_inline int spin_trylock_irq(spinlock_t *lock)
 	raw_spin_trylock_irqsave(spinlock_check(lock), flags); \
 })
 
+static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
+{
+	return raw_spin_trylock_irq_disable(&lock->rlock);
+}
+
 /**
  * spin_is_locked() - Check whether a spinlock is locked.
  * @lock: Pointer to the spinlock.
include/linux/spinlock_api_smp.h +27 -0
index 9ecb0ab504e32..92532103b9eaa 100644
@@ -28,6 +28,8 @@ _raw_spin_lock_nest_lock(raw_spinlock_t *lock, struct lockdep_map *map)
 void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock)		__acquires(lock);
 void __lockfunc _raw_spin_lock_irq(raw_spinlock_t *lock)
 								__acquires(lock);
+void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+								__acquires(lock);
 
 unsigned long __lockfunc _raw_spin_lock_irqsave(raw_spinlock_t *lock)
 								__acquires(lock);
@@ -39,6 +41,7 @@ int __lockfunc _raw_spin_trylock_bh(raw_spinlock_t *lock);
 void __lockfunc _raw_spin_unlock(raw_spinlock_t *lock)		__releases(lock);
 void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock)	__releases(lock);
 void __lockfunc _raw_spin_unlock_irq(raw_spinlock_t *lock)	__releases(lock);
+void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)	__releases(lock);
 void __lockfunc
 _raw_spin_unlock_irqrestore(raw_spinlock_t *lock, unsigned long flags)
 								__releases(lock);
@@ -55,6 +58,11 @@ _raw_spin_unlock_irqrestore(raw_spinlock_t *lock, unsigned long flags)
 #define _raw_spin_lock_irq(lock) __raw_spin_lock_irq(lock)
 #endif
 
+/* Use the same config as spin_lock_irq() temporarily. */
+#ifdef CONFIG_INLINE_SPIN_LOCK_IRQ
+#define _raw_spin_lock_irq_disable(lock) __raw_spin_lock_irq_disable(lock)
+#endif
+
 #ifdef CONFIG_INLINE_SPIN_LOCK_IRQSAVE
 #define _raw_spin_lock_irqsave(lock) __raw_spin_lock_irqsave(lock)
 #endif
@@ -79,6 +87,11 @@ _raw_spin_unlock_irqrestore(raw_spinlock_t *lock, unsigned long flags)
 #define _raw_spin_unlock_irq(lock) __raw_spin_unlock_irq(lock)
 #endif
 
+/* Use the same config as spin_unlock_irq() temporarily. */
+#ifdef CONFIG_INLINE_SPIN_UNLOCK_IRQ
+#define _raw_spin_unlock_irq_enable(lock) __raw_spin_unlock_irq_enable(lock)
+#endif
+
 #ifdef CONFIG_INLINE_SPIN_UNLOCK_IRQRESTORE
 #define _raw_spin_unlock_irqrestore(lock, flags) __raw_spin_unlock_irqrestore(lock, flags)
 #endif
@@ -120,6 +133,13 @@ static inline void __raw_spin_lock_irq(raw_spinlock_t *lock)
 	LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
 }
 
+static inline void __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	local_interrupt_disable();
+	spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
+	LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
+}
+
 static inline void __raw_spin_lock_bh(raw_spinlock_t *lock)
 {
 	__local_bh_disable_ip(_RET_IP_, SOFTIRQ_LOCK_OFFSET);
@@ -160,6 +180,13 @@ static inline void __raw_spin_unlock_irq(raw_spinlock_t *lock)
 	preempt_enable();
 }
 
+static inline void __raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
+{
+	spin_release(&lock->dep_map, _RET_IP_);
+	do_raw_spin_unlock(lock);
+	local_interrupt_enable();
+}
+
 static inline void __raw_spin_unlock_bh(raw_spinlock_t *lock)
 {
 	spin_release(&lock->dep_map, _RET_IP_);
include/linux/spinlock_api_up.h +8 -0
index 819aeba1c87e6..d02a73671713b 100644
@@ -36,6 +36,9 @@
 #define __LOCK_IRQ(lock) \
   do { local_irq_disable(); __LOCK(lock); } while (0)
 
+#define __LOCK_IRQ_DISABLE(lock) \
+  do { local_interrupt_disable(); __LOCK(lock); } while (0)
+
 #define __LOCK_IRQSAVE(lock, flags) \
   do { local_irq_save(flags); __LOCK(lock); } while (0)
 
@@ -52,6 +55,9 @@
 #define __UNLOCK_IRQ(lock) \
   do { local_irq_enable(); __UNLOCK(lock); } while (0)
 
+#define __UNLOCK_IRQ_ENABLE(lock) \
+  do { __UNLOCK(lock); local_interrupt_enable(); } while (0)
+
 #define __UNLOCK_IRQRESTORE(lock, flags) \
   do { local_irq_restore(flags); __UNLOCK(lock); } while (0)
 
@@ -64,6 +70,7 @@
 #define _raw_read_lock_bh(lock)			__LOCK_BH(lock)
 #define _raw_write_lock_bh(lock)		__LOCK_BH(lock)
 #define _raw_spin_lock_irq(lock)		__LOCK_IRQ(lock)
+#define _raw_spin_lock_irq_disable(lock)	__LOCK_IRQ_DISABLE(lock)
 #define _raw_read_lock_irq(lock)		__LOCK_IRQ(lock)
 #define _raw_write_lock_irq(lock)		__LOCK_IRQ(lock)
 #define _raw_spin_lock_irqsave(lock, flags)	__LOCK_IRQSAVE(lock, flags)
@@ -80,6 +87,7 @@
 #define _raw_write_unlock_bh(lock)		__UNLOCK_BH(lock)
 #define _raw_read_unlock_bh(lock)		__UNLOCK_BH(lock)
 #define _raw_spin_unlock_irq(lock)		__UNLOCK_IRQ(lock)
+#define _raw_spin_unlock_irq_enable(lock)	__UNLOCK_IRQ_ENABLE(lock)
 #define _raw_read_unlock_irq(lock)		__UNLOCK_IRQ(lock)
 #define _raw_write_unlock_irq(lock)		__UNLOCK_IRQ(lock)
 #define _raw_spin_unlock_irqrestore(lock, flags) \
include/linux/spinlock_rt.h +15 -0
index f6499c37157df..074182f7cfeea 100644
@@ -93,6 +93,11 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)
 	rt_spin_lock(lock);
 }
 
+static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
+{
+	rt_spin_lock(lock);
+}
+
 #define spin_lock_irqsave(lock, flags)			 \
 	do {						 \
 		typecheck(unsigned long, flags);	 \
@@ -116,12 +121,22 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)
 	rt_spin_unlock(lock);
 }
 
+static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
+{
+	rt_spin_unlock(lock);
+}
+
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,
 						   unsigned long flags)
 {
 	rt_spin_unlock(lock);
 }
 
+static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
+{
+	return rt_spin_trylock(lock);
+}
+
 #define spin_trylock(lock)				\
 	__cond_lock(lock, rt_spin_trylock(lock))
 
kernel/locking/spinlock.c +29 -0
index 7685defd7c526..da54b220b5a45 100644
@@ -125,6 +125,19 @@ static void __lockfunc __raw_##op##_lock_bh(locktype##_t *lock)		\
  */
 BUILD_LOCK_OPS(spin, raw_spinlock);
 
+/* No rwlock_t variants for now, so just build this function by hand */
+static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	for (;;) {
+		local_interrupt_disable();
+		if (likely(do_raw_spin_trylock(lock)))
+			break;
+		local_interrupt_enable();
+
+		arch_spin_relax(&lock->raw_lock);
+	}
+}
+
 #ifndef CONFIG_PREEMPT_RT
 BUILD_LOCK_OPS(read, rwlock);
 BUILD_LOCK_OPS(write, rwlock);
@@ -172,6 +185,14 @@ noinline void __lockfunc _raw_spin_lock_irq(raw_spinlock_t *lock)
 EXPORT_SYMBOL(_raw_spin_lock_irq);
 #endif
 
+#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ
+noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	__raw_spin_lock_irq_disable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable);
+#endif
+
 #ifndef CONFIG_INLINE_SPIN_LOCK_BH
 noinline void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock)
 {
@@ -204,6 +225,14 @@ noinline void __lockfunc _raw_spin_unlock_irq(raw_spinlock_t *lock)
 EXPORT_SYMBOL(_raw_spin_unlock_irq);
 #endif
 
+#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ
+noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
+{
+	__raw_spin_unlock_irq_enable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable);
+#endif
+
 #ifndef CONFIG_INLINE_SPIN_UNLOCK_BH
 noinline void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock)
 {
kernel/softirq.c +3 -0
index af47ea23aba3b..b681545eabbbe 100644
@@ -88,6 +88,9 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
+DEFINE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state);
+EXPORT_PER_CPU_SYMBOL_GPL(local_interrupt_disable_state);
+
 DEFINE_PER_CPU(unsigned int, nmi_nesting);
 
 /*
Signature · 1 line
2.52.0
author reviewer
#26 lore raw

PATCH v17 6/16 irq: Add KUnit test for refcounted interrupt enable/disable

In reply to Lyude Paul

While making changes to the refcounted interrupt patch series, at some point on my local branch I broke something and ended up writing some kunit tests for testing refcounted interrupts as a result. So, let's include these tests now that we have refcounted interrupts.

---
V13:
* Add missing MODULE_DESCRIPTION/MODULE_LICENSE lines
* Switch from kunit_test_suites(…) to kunit_test_suite(…)

 kernel/irq/Makefile                  |   1 +
 kernel/irq/refcount_interrupt_test.c | 109 +++++++++++++++++++++++++++
 2 files changed, 110 insertions(+)
 create mode 100644 kernel/irq/refcount_interrupt_test.c
kernel/irq/Makefile +1 -0
index 6ab3a40556670..7b5bb5510b110 100644
@@ -20,3 +20,4 @@ obj-$(CONFIG_SMP) += affinity.o
 obj-$(CONFIG_GENERIC_IRQ_DEBUGFS) += debugfs.o
 obj-$(CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR) += matrix.o
 obj-$(CONFIG_IRQ_KUNIT_TEST) += irq_test.o
+obj-$(CONFIG_KUNIT) += refcount_interrupt_test.o
kernel/irq/refcount_interrupt_test.c new file +109 -0

109 lines not on this page · raw message

Signature · 1 line
2.52.0
Reviewed-by on v17 6/16
#27 lore raw

Re: [PATCH v17 06/16] irq: Add KUnit test for refcounted interrupt enable/disable

Quoted 9 lines · Lyude Paul

While making changes to the refcounted interrupt patch series, at some point on my local branch I broke something and ended up writing some kunit tests for testing refcounted interrupts as a result. So, let's include these tests now that we have refcounted interrupts.

---

Looks good to me. Not sure if it'd make sense to move this into kernel/irq/irq_test.c alongside other IRQs, or if you like having it separate. I'm happy either way.

Cheers,
-- David

Quoted 137 lines

V13:
* Add missing MODULE_DESCRIPTION/MODULE_LICENSE lines
* Switch from kunit_test_suites(…) to kunit_test_suite(…)

 kernel/irq/Makefile                  |   1 +
 kernel/irq/refcount_interrupt_test.c | 109 +++++++++++++++++++++++++++
 2 files changed, 110 insertions(+)
 create mode 100644 kernel/irq/refcount_interrupt_test.c
kernel/irq/Makefile +1 -0
index 6ab3a40556670..7b5bb5510b110 100644
@@ -20,3 +20,4 @@ obj-$(CONFIG_SMP) += affinity.o
 obj-$(CONFIG_GENERIC_IRQ_DEBUGFS) += debugfs.o
 obj-$(CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR) += matrix.o
 obj-$(CONFIG_IRQ_KUNIT_TEST) += irq_test.o
+obj-$(CONFIG_KUNIT) += refcount_interrupt_test.o
kernel/irq/refcount_interrupt_test.c new file +109 -1
new file mode 100644
index 0000000000000..b4f224595f261
@@ -0,0 +1,109 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * KUnit test for refcounted interrupt enable/disables.
+ */
+
+#include <kunit/test.h>
+#include <linux/interrupt_rc.h>
+
+#define TEST_IRQ_ON() KUNIT_EXPECT_FALSE(test, irqs_disabled())
+#define TEST_IRQ_OFF() KUNIT_EXPECT_TRUE(test, irqs_disabled())
+
+/* ===== Test cases ===== */
+static void test_single_irq_change(struct kunit *test)
+{
+       local_interrupt_disable();
+       TEST_IRQ_OFF();
+       local_interrupt_enable();
+}
+
+static void test_nested_irq_change(struct kunit *test)
+{
+       local_interrupt_disable();
+       TEST_IRQ_OFF();
+       local_interrupt_disable();
+       TEST_IRQ_OFF();
+       local_interrupt_disable();
+       TEST_IRQ_OFF();
+
+       local_interrupt_enable();
+       TEST_IRQ_OFF();
+       local_interrupt_enable();
+       TEST_IRQ_OFF();
+       local_interrupt_enable();
+       TEST_IRQ_ON();
+}
+
+static void test_multiple_irq_change(struct kunit *test)
+{
+       local_interrupt_disable();
+       TEST_IRQ_OFF();
+       local_interrupt_disable();
+       TEST_IRQ_OFF();
+
+       local_interrupt_enable();
+       TEST_IRQ_OFF();
+       local_interrupt_enable();
+       TEST_IRQ_ON();
+
+       local_interrupt_disable();
+       TEST_IRQ_OFF();
+       local_interrupt_enable();
+       TEST_IRQ_ON();
+}
+
+static void test_irq_save(struct kunit *test)
+{
+       unsigned long flags;
+
+       local_irq_save(flags);
+       TEST_IRQ_OFF();
+       local_interrupt_disable();
+       TEST_IRQ_OFF();
+       local_interrupt_enable();
+       TEST_IRQ_OFF();
+       local_irq_restore(flags);
+       TEST_IRQ_ON();
+
+       local_interrupt_disable();
+       TEST_IRQ_OFF();
+       local_irq_save(flags);
+       TEST_IRQ_OFF();
+       local_irq_restore(flags);
+       TEST_IRQ_OFF();
+       local_interrupt_enable();
+       TEST_IRQ_ON();
+}
+
+static struct kunit_case test_cases[] = {
+       KUNIT_CASE(test_single_irq_change),
+       KUNIT_CASE(test_nested_irq_change),
+       KUNIT_CASE(test_multiple_irq_change),
+       KUNIT_CASE(test_irq_save),
+       {},
+};
+
+/* (init and exit are the same */
+static int test_init(struct kunit *test)
+{
+       TEST_IRQ_ON();
+
+       return 0;
+}
+
+static void test_exit(struct kunit *test)
+{
+       TEST_IRQ_ON();
+}
+
+static struct kunit_suite refcount_interrupt_test_suite = {
+       .name = "refcount_interrupt",
+       .test_cases = test_cases,
+       .init = test_init,
+       .exit = test_exit,
+};
+
+kunit_test_suite(refcount_interrupt_test_suite);
+MODULE_AUTHOR("Lyude Paul <[email protected]>");
+MODULE_DESCRIPTION("Refcounted interrupt unit test suite");
+MODULE_LICENSE("GPL");
--

2.52.0

author reviewer
#28 lore raw

PATCH v17 7/16 rust: Introduce interrupt module

In reply to Lyude Paul

This introduces a module for dealing with interrupt-disabled contexts, including the ability to enable and disable interrupts along with the ability to annotate functions as expecting that IRQs are already disabled on the local CPU.

[Boqun: This is based on Lyude's work on interrupt disable abstraction, I port to the new local_interrupt_disable() mechanism to make it work as a guard type. I cannot even take the credit of this design, since Lyude also brought up the same idea in zulip. Anyway, this is only for POC purpose, and of course all bugs are mine]

---

V10:
* Fix documentation typos
V11:
* Get rid of unneeded use bindings;
* Move ASSUME_DISABLED into assume_disabled()
* Confirm using lockdep_assert_irqs_disabled() that local interrupts are in fact disabled when LocalInterruptDisabled::assume_disabled() is called.

 rust/helpers/helpers.c   |  1 +
 rust/helpers/interrupt.c | 18 +++++++++
 rust/helpers/sync.c      |  5 +++
 rust/kernel/interrupt.rs | 86 ++++++++++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs       |  1 +
 5 files changed, 111 insertions(+)
 create mode 100644 rust/helpers/interrupt.c
 create mode 100644 rust/kernel/interrupt.rs
rust/helpers/helpers.c +1 -0
index 79c72762ad9c4..f97d1cb63e0cd 100644
@@ -29,6 +29,7 @@
 #include "err.c"
 #include "irq.c"
 #include "fs.c"
+#include "interrupt.c"
 #include "io.c"
 #include "jump_label.c"
 #include "kunit.c"
rust/helpers/interrupt.c new file +18 -0
new file mode 100644
index 0000000000000..f2380dd461ca5
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/spinlock.h>
+
+void rust_helper_local_interrupt_disable(void)
+{
+	local_interrupt_disable();
+}
+
+void rust_helper_local_interrupt_enable(void)
+{
+	local_interrupt_enable();
+}
+
+bool rust_helper_irqs_disabled(void)
+{
+	return irqs_disabled();
+}
rust/helpers/sync.c +5 -0
index ff7e68b488101..45b2f519f4e2e 100644
@@ -11,3 +11,8 @@ void rust_helper_lockdep_unregister_key(struct lock_class_key *k)
 {
 	lockdep_unregister_key(k);
 }
+
+void rust_helper_lockdep_assert_irqs_disabled(void)
+{
+	lockdep_assert_irqs_disabled();
+}
rust/kernel/interrupt.rs new file +86 -0

86 lines not on this page · raw message

rust/kernel/lib.rs +1 -0
index f812cf1200428..fbe830c4c54e4 100644
@@ -101,6 +101,7 @@
 pub mod i2c;
 pub mod id_pool;
 pub mod init;
+pub mod interrupt;
 pub mod io;
 pub mod ioctl;
 pub mod iov;
Signature · 1 line
2.52.0
author reviewer
#29 lore raw

PATCH v17 8/16 rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers

In reply to Lyude Paul

From: Boqun Feng <[email protected]>

spin_lock_irq_disable() and spin_unlock_irq_enable() are inline functions, to use them in Rust, helpers are introduced. This is for interrupt disabling lock abstraction in Rust.

---
 rust/helpers/spinlock.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
rust/helpers/spinlock.c +15 -0
index 42c4bf01a23e4..d4e61057c2a7a 100644
@@ -35,3 +35,18 @@ void rust_helper_spin_assert_is_held(spinlock_t *lock)
 {
 	lockdep_assert_held(lock);
 }
+
+void rust_helper_spin_lock_irq_disable(spinlock_t *lock)
+{
+	spin_lock_irq_disable(lock);
+}
+
+void rust_helper_spin_unlock_irq_enable(spinlock_t *lock)
+{
+	spin_unlock_irq_enable(lock);
+}
+
+int rust_helper_spin_trylock_irq_disable(spinlock_t *lock)
+{
+	return spin_trylock_irq_disable(lock);
+}
Signature · 1 line
2.52.0
maintainer
#30 lore raw

Re: [PATCH v17 08/16] rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers

Quoted 26 lines · Lyude Paul

From: Boqun Feng <[email protected]>

spin_lock_irq_disable() and spin_unlock_irq_enable() are inline functions, to use them in Rust, helpers are introduced. This is for interrupt disabling lock abstraction in Rust.

---
 rust/helpers/spinlock.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
rust/helpers/spinlock.c +5 -0
index 42c4bf01a23e4..d4e61057c2a7a 100644
@@ -35,3 +35,18 @@ void rust_helper_spin_assert_is_held(spinlock_t *lock)
 {
 	lockdep_assert_held(lock);
 }
+
+void rust_helper_spin_lock_irq_disable(spinlock_t *lock)
+{
+	spin_lock_irq_disable(lock);
+}

rust/helpers/spinlock.c void rust_helper_spin_assert_is_held(spinlock_t *lock) · line 52

Quoted 10 lines
rust/helpers/spinlock.c +10 -0
+
+void rust_helper_spin_unlock_irq_enable(spinlock_t *lock)
+{
+	spin_unlock_irq_enable(lock);
+}
+
+int rust_helper_spin_trylock_irq_disable(spinlock_t *lock)
+{
+	return spin_trylock_irq_disable(lock);
+}

My comments from v16 about adding __rust_helper is not addressed.

Best,
Gary

author reviewer
#31 lore raw

PATCH v17 9/16 rust: sync: Add SpinLockIrq

In reply to Lyude Paul

A variant of SpinLock that ensures interrupts are disabled in the critical section. lock() will ensure that either interrupts are already disabled or disable them. unlock() will reverse the respective operation.

[Boqun: Port to use spin_lock_irq_disable() and
spin_unlock_irq_enable()]

---
V10:
* Also add support to GlobalLock
* Documentation fixes from Dirk
V11:
* Add unit test requested by Daniel Almeida
V14:
* Improve rustdoc for SpinLockIrqBackend
V17:
* Update Git summary according to Benno's review

 rust/kernel/sync.rs               |   4 +-
 rust/kernel/sync/lock/global.rs   |   3 +
 rust/kernel/sync/lock/spinlock.rs | 229 ++++++++++++++++++++++++++++++
 3 files changed, 235 insertions(+), 1 deletion(-)
rust/kernel/sync.rs +3 -1
index 5df87e2bd212e..48a7cae86c50c 100644
@@ -27,7 +27,9 @@
 pub use condvar::{new_condvar, CondVar, CondVarTimeoutResult};
 pub use lock::global::{global_lock, GlobalGuard, GlobalLock, GlobalLockBackend, GlobalLockedBy};
 pub use lock::mutex::{new_mutex, Mutex, MutexGuard};
-pub use lock::spinlock::{new_spinlock, SpinLock, SpinLockGuard};
+pub use lock::spinlock::{
+    new_spinlock, new_spinlock_irq, SpinLock, SpinLockGuard, SpinLockIrq, SpinLockIrqGuard,
+};
 pub use locked_by::LockedBy;
 pub use refcount::Refcount;
 pub use set_once::SetOnce;
rust/kernel/sync/lock/global.rs +3 -0
index eab48108a4aeb..7030a47bc0ad1 100644
@@ -302,4 +302,7 @@ macro_rules! global_lock_inner {
     (backend SpinLock) => {
         $crate::sync::lock::spinlock::SpinLockBackend
     };
+    (backend SpinLockIrq) => {
+        $crate::sync::lock::spinlock::SpinLockIrqBackend
+    };
 }
rust/kernel/sync/lock/spinlock.rs +229 -0

238 lines not on this page · raw message

Signature · 1 line
2.52.0
reviewer Reviewed-by on v17 9/16
#32 lore raw

Re: [PATCH v17 09/16] rust: sync: Add SpinLockIrq

Quoted 21 lines · Lyude Paul
rust/kernel/sync/lock/spinlock.rs +6 -0
index d7be38ccbdc7d..3fdfb0a8a0ab1 100644
@@ -3,6 +3,7 @@
 //! A kernel spinlock.
 //!
 //! This module allows Rust code to use the kernel's `spinlock_t`.
+use crate::prelude::*;
 
 /// Creates a [`SpinLock`] initialiser with the given name and a newly-created lock class.
 ///
@@ -139,3 +140,231 @@ unsafe fn assert_is_held(ptr: *mut Self::State) {
         unsafe { bindings::spin_assert_is_held(ptr) }
     }
 }
+
+/// Creates a [`SpinLockIrq`] initialiser with the given name and a newly-created lock class.
+///
+/// It uses the name if one is given, otherwise it generates one based on the file name and line
+/// number.

rust/kernel/sync/lock/spinlock.rs unsafe fn assert_is_held(ptr: *mut Self::State) { · line 157

Quoted 10 lines
rust/kernel/sync/lock/spinlock.rs +10 -0
+#[macro_export]
+macro_rules! new_spinlock_irq {
+    ($inner:expr $(, $name:literal)? $(,)?) => {
+        $crate::sync::SpinLockIrq::new(
+            $inner, $crate::optional_name!($($name)?), $crate::static_lock_class!())
+    };
+}
+pub use new_spinlock_irq;
+
+/// A spinlock that may be acquired when local processor interrupts are disabled.

This hasn't been updated with my previous suggestion and I don't see a reply on the mailing list. Am I missing something?

With that fixed:

Cheers,
Benno

author reviewer
#33 lore raw

PATCH v17 10/16 rust: sync: Introduce lock::Lock::lock_with() and friends

In reply to Lyude Paul

SpinLockIrq and SpinLock use the exact same underlying C structure, with the only real difference being that the former uses the irq_disable() and irq_enable() variants for locking/unlocking. These variants can introduce some minor overhead in contexts where we already know that local processor interrupts are disabled, and as such we want a way to be able to skip modifying processor interrupt state in said contexts in order to avoid some overhead - just like the current C API allows us to do. So, ContextualBackend allows us to cast a lock into it's contextless version for situations where we already have whatever guarantees would be provided by BackendWithContext::ContextualBackend in place.

In some hacked-together benchmarks we ran, most of the time this did actually seem to lead to a noticeable difference in overhead:

From an aarch64 VM running on a MacBook M4:

    lock() when irq is disabled, 100 times cost Delta { nanos: 500 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 292 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 834 }

    lock() when irq is disabled, 100 times cost Delta { nanos: 459 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 291 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 709 }

From an x86_64 VM (qemu/kvm) running on a i7-13700H

    lock() when irq is disabled, 100 times cost Delta { nanos: 1002 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 729 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 1516 }

    lock() when irq is disabled, 100 times cost Delta { nanos: 754 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 966 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 1227 }

    (note that there were some runs on x86_64 where lock() on irq disabled
    vs. lock_with() on irq disabled had equivalent benchmarks, but it very
    much appeared to be a minority of test runs.

While it's not clear how this affects real-world workloads yet, let's add this for the time being so we can find out. Implement lock::Lock::lock_with() and lock::BackendWithContext::ContextualBackend. This makes it so that a SpinLockIrq will work like a SpinLock if interrupts are disabled. So a function:

        (&'a SpinLockIrq, &'a InterruptDisabled) -> Guard<'a, .., SpinLockBackend>

makes sense. Note that due to Guard and InterruptDisabled having the same lifetime, interrupts cannot be enabled while the Guard exists.

---
This was originally two patches, but keeping them split didn't make sense after going from BackendInContext to BackendWithContext.

V10:
* Fix typos - Dirk/Lyude
* Since we're adding support for context locks to GlobalLock as well, let's also make sure to cover try_lock while we're at it and add try_lock_with
* Add a private function as_lock_in_context() for handling casting from a Lock<T, B> to Lock<T, B::ContextualBackend> so we don't have to duplicate safety comments
V11:
* Fix clippy::ref_as_ptr error in Lock::as_lock_in_context() V14:
* Add benchmark results, rewrite commit message
V17:
* Introduce BackendWithContext, move context-related bits into there and out of Backend.
* Add missing #[must_use = …] for try_lock_with()
* Remove all unsafe code from lock_with() and try_lock_with():
Somehow I never noticed that literally none of the unsafe code in these two functions is needed with as_lock_in_context()...

 rust/kernel/sync/lock.rs          | 71 ++++++++++++++++++++++++++++++-
 rust/kernel/sync/lock/spinlock.rs | 48 ++++++++++++++++++++-
 2 files changed, 117 insertions(+), 2 deletions(-)
rust/kernel/sync/lock.rs +70 -1
index 46a57d1fc309d..9f6d7b381bd15 100644
@@ -30,10 +30,15 @@
 ///   is owned, that is, between calls to [`lock`] and [`unlock`].
 /// - Implementers must also ensure that [`relock`] uses the same locking method as the original
 ///   lock operation.
+/// - Implementers must ensure if [`BackendInContext`] is a [`Backend`], it's safe to acquire the
+///   lock under the [`Context`], the [`State`] of two backends must be the same.
 ///
 /// [`lock`]: Backend::lock
 /// [`unlock`]: Backend::unlock
 /// [`relock`]: Backend::relock
+/// [`BackendInContext`]: Backend::BackendInContext
+/// [`Context`]: Backend::Context
+/// [`State`]: Backend::State
 pub unsafe trait Backend {
     /// The state required by the lock.
     type State;
@@ -97,6 +102,34 @@ unsafe fn relock(ptr: *mut Self::State, guard_state: &mut Self::GuardState) {
     unsafe fn assert_is_held(ptr: *mut Self::State);
 }
 
+/// A lock [`Backend`] with a [`ContextualBackend`] that can make lock acquisition cheaper.
+///
+/// Some locks, such as [`SpinLockIrq`](super::SpinLockIrq), can only be acquired in specific
+/// hardware contexts (e.g. local processor interrupts disabled). Entering and exiting these
+/// contexts incurs additional overhead. But this overhead may be avoided if we know ahead of time
+/// that we are already within the correct context for a given lock as we can then skip any costly
+/// operations required for entering/exiting said context.
+///
+/// Any lock implementing this trait requires such a interrupt context, and can provide cheaper
+/// lock-acquisition functions through [`Lock::lock_with`] and [`Lock::try_lock_with`] as long as a
+/// context token of type [`Context`] is available.
+///
+/// # Safety
+///
+/// - Implementors must ensure that it is safe to acquire the lock under [`Context`].
+///
+/// [`ContextualBackend`]: BackendWithContext::ContextualBackend
+/// [`Context`]: BackendWithContext::Context
+pub unsafe trait BackendWithContext: Backend {
+    /// The context which must be provided in order to acquire the lock with the
+    /// [`ContextualBackend`](BackendWithContext::ContextualBackend).
+    type Context<'a>;
+
+    /// The alternative cheaper backend we can use if a [`Context`](BackendWithContext::Context) is
+    /// provided.
+    type ContextualBackend: Backend<State = Self::State>;
+}
+
 /// A mutual exclusion primitive.
 ///
 /// Exposes one of the kernel locking primitives. Which one is exposed depends on the lock
@@ -169,7 +202,8 @@ pub unsafe fn from_raw<'a>(ptr: *mut B::State) -> &'a Self {
 
 impl<T: ?Sized, B: Backend> Lock<T, B> {
     /// Acquires the lock and gives the caller access to the data protected by it.
-    pub fn lock(&self) -> Guard<'_, T, B> {
+    #[inline]
+    pub fn lock<'a>(&'a self) -> Guard<'a, T, B> {
         // SAFETY: The constructor of the type calls `init`, so the existence of the object proves
         // that `init` was called.
         let state = unsafe { B::lock(self.state.get()) };
@@ -189,6 +223,41 @@ pub fn try_lock(&self) -> Option<Guard<'_, T, B>> {
     }
 }
 
+impl<T: ?Sized, B: BackendWithContext> Lock<T, B> {
+    /// Casts the lock as a `Lock<T, B::ContextualBackend>`.
+    fn as_lock_in_context<'a>(
+        &'a self,
+        _context: B::Context<'a>,
+    ) -> &'a Lock<T, B::ContextualBackend>
+    where
+        B::ContextualBackend: Backend,
+    {
+        // SAFETY:
+        // - Per the safety guarantee of `Backend`, if `B::ContextualBackend` and `B` should
+        //   have the same state, the layout of the lock is the same so it's safe to convert one to
+        //   another.
+        // - The caller provided `B::Context<'a>`, so it is safe to recast and return this lock.
+        unsafe { &*(core::ptr::from_ref(self) as *const _) }
+    }
+
+    /// Acquires the lock with the given context and gives the caller access to the data protected
+    /// by it.
+    pub fn lock_with<'a>(&'a self, context: B::Context<'a>) -> Guard<'a, T, B::ContextualBackend> {
+        self.as_lock_in_context(context).lock()
+    }
+
+    /// Tries to acquire the lock with the given context.
+    ///
+    /// Returns a guard that can be used to access the data protected by the lock if successful.
+    #[must_use = "if unused, the lock will be immediately unlocked"]
+    pub fn try_lock_with<'a>(
+        &'a self,
+        context: B::Context<'a>,
+    ) -> Option<Guard<'a, T, B::ContextualBackend>> {
+        self.as_lock_in_context(context).try_lock()
+    }
+}
+
 /// A lock guard.
 ///
 /// Allows mutual exclusion primitives that implement the [`Backend`] trait to automatically unlock
rust/kernel/sync/lock/spinlock.rs +47 -1

66 lines not on this page · raw message

Signature · 1 line
2.52.0
#34 lore raw

Re: [PATCH v17 10/16] rust: sync: Introduce lock::Lock::lock_with() and friends

kernel test robot noticed the following build warnings:

Hi Lyude,

kernel test robot noticed the following build warnings:

[auto build test WARNING on 2ad6c5cdc89acfefb01b84afa5e55262c40d6fec]

url:    https://github.com/intel-lab-lkp/linux/commits/Lyude-Paul/preempt-Introduce-HARDIRQ_DISABLE_BITS/20260122-064928
base:   2ad6c5cdc89acfefb01b84afa5e55262c40d6fec
patch link:    https://lore.kernel.org/r/20260121223933.1568682-11-lyude%40redhat.com
patch subject: [PATCH v17 10/16] rust: sync: Introduce lock::Lock::lock_with() and friends
config: x86_64-rhel-9.4-rust (https://download.01.org/0day-ci/archive/20260122/[email protected]/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
rustc: rustc 1.88.0 (6b00bc388 2025-06-23)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260122/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> warning: unresolved link to `Backend::BackendInContext`

--> rust/kernel/sync/lock.rs:39:27
|
39 | /// [BackendInContext]: Backend::BackendInContext
| ^^^^^^^^^^^^^^^^^^^^^^^^^ the trait Backend has no associated item named BackendInContext
|
= note: #[warn(rustdoc::broken_intra_doc_links)] on by default
--

>> warning: unresolved link to `Backend::Context`

--> rust/kernel/sync/lock.rs:40:18
|
40 | /// [Context]: Backend::Context
| ^^^^^^^^^^^^^^^^ the trait Backend has no associated item named Context

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
reviewer
#35 lore raw

Re: [PATCH v17 10/16] rust: sync: Introduce lock::Lock::lock_with() and friends

In reply to Lyude Paul
Quoted 42 lines · Lyude Paul

SpinLockIrq and SpinLock use the exact same underlying C structure, with the only real difference being that the former uses the irq_disable() and irq_enable() variants for locking/unlocking. These variants can introduce some minor overhead in contexts where we already know that local processor interrupts are disabled, and as such we want a way to be able to skip modifying processor interrupt state in said contexts in order to avoid some overhead - just like the current C API allows us to do. So, ContextualBackend allows us to cast a lock into it's contextless version for situations where we already have whatever guarantees would be provided by BackendWithContext::ContextualBackend in place.

In some hacked-together benchmarks we ran, most of the time this did actually seem to lead to a noticeable difference in overhead:

From an aarch64 VM running on a MacBook M4:

    lock() when irq is disabled, 100 times cost Delta { nanos: 500 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 292 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 834 }

    lock() when irq is disabled, 100 times cost Delta { nanos: 459 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 291 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 709 }

From an x86_64 VM (qemu/kvm) running on a i7-13700H

    lock() when irq is disabled, 100 times cost Delta { nanos: 1002 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 729 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 1516 }

    lock() when irq is disabled, 100 times cost Delta { nanos: 754 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 966 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 1227 }

    (note that there were some runs on x86_64 where lock() on irq disabled
    vs. lock_with() on irq disabled had equivalent benchmarks, but it very
    much appeared to be a minority of test runs.

While it's not clear how this affects real-world workloads yet, let's add this for the time being so we can find out. Implement lock::Lock::lock_with() and lock::BackendWithContext::ContextualBackend. This makes it so that a SpinLockIrq will work like a SpinLock if interrupts are disabled. So a function:

Quoted 8 lines
        (&'a SpinLockIrq, &'a InterruptDisabled) -> Guard<'a, .., SpinLockBackend>

makes sense. Note that due to Guard and InterruptDisabled having the same lifetime, interrupts cannot be enabled while the Guard exists.

My overall opinion of the design is that we should no longer use the generic approach with the Backend traits. I think I mentioned this on Zulip already at multiple points. I'm okay with having this extension, but it would be ideal if we could move to not having a single Lock struct, but one for each locking primitive.

Quoted 33 lines

---
This was originally two patches, but keeping them split didn't make sense after going from BackendInContext to BackendWithContext.

V10:
* Fix typos - Dirk/Lyude
* Since we're adding support for context locks to GlobalLock as well, let's also make sure to cover try_lock while we're at it and add try_lock_with
* Add a private function as_lock_in_context() for handling casting from a Lock<T, B> to Lock<T, B::ContextualBackend> so we don't have to duplicate safety comments
V11:
* Fix clippy::ref_as_ptr error in Lock::as_lock_in_context() V14:
* Add benchmark results, rewrite commit message
V17:
* Introduce BackendWithContext, move context-related bits into there and out of Backend.
* Add missing #[must_use = …] for try_lock_with()
* Remove all unsafe code from lock_with() and try_lock_with():
Somehow I never noticed that literally none of the unsafe code in these two functions is needed with as_lock_in_context()...

 rust/kernel/sync/lock.rs          | 71 ++++++++++++++++++++++++++++++-
 rust/kernel/sync/lock/spinlock.rs | 48 ++++++++++++++++++++-
 2 files changed, 117 insertions(+), 2 deletions(-)
rust/kernel/sync/lock.rs +0 -0
index 46a57d1fc309d..9f6d7b381bd15 100644

rust/kernel/sync/lock.rs line 34

Quoted 5 lines
rust/kernel/sync/lock.rs +2 -0
@@ -30,10 +30,15 @@
 ///   is owned, that is, between calls to [`lock`] and [`unlock`].
 /// - Implementers must also ensure that [`relock`] uses the same locking method as the original
 ///   lock operation.
+/// - Implementers must ensure if [`BackendInContext`] is a [`Backend`], it's safe to acquire the
+///   lock under the [`Context`], the [`State`] of two backends must be the same.

This isn't needed, since we don't have Backend::Context any longer.

rust/kernel/sync/lock.rs line 41

Quoted 7 lines
rust/kernel/sync/lock.rs +3 -0
 ///
 /// [`lock`]: Backend::lock
 /// [`unlock`]: Backend::unlock
 /// [`relock`]: Backend::relock
+/// [`BackendInContext`]: Backend::BackendInContext
+/// [`Context`]: Backend::Context
+/// [`State`]: Backend::State

Same for these.

Quoted 12 lines
rust/kernel/sync/lock.rs +5 -0
 pub unsafe trait Backend {
     /// The state required by the lock.
     type State;
@@ -97,6 +102,34 @@ unsafe fn relock(ptr: *mut Self::State, guard_state: &mut Self::GuardState) {
     unsafe fn assert_is_held(ptr: *mut Self::State);
 }
 
+/// A lock [`Backend`] with a [`ContextualBackend`] that can make lock acquisition cheaper.
+///
+/// Some locks, such as [`SpinLockIrq`](super::SpinLockIrq), can only be acquired in specific
+/// hardware contexts (e.g. local processor interrupts disabled). Entering and exiting these
+/// contexts incurs additional overhead. But this overhead may be avoided if we know ahead of time

rust/kernel/sync/lock.rs unsafe fn relock(ptr: *mut Self::State, guard_state: &mut Self::GuardState) { · line 119

Quoted 10 lines
rust/kernel/sync/lock.rs +10 -0
+/// that we are already within the correct context for a given lock as we can then skip any costly
+/// operations required for entering/exiting said context.
+///
+/// Any lock implementing this trait requires such a interrupt context, and can provide cheaper
+/// lock-acquisition functions through [`Lock::lock_with`] and [`Lock::try_lock_with`] as long as a
+/// context token of type [`Context`] is available.
+///
+/// # Safety
+///
+/// - Implementors must ensure that it is safe to acquire the lock under [`Context`].

This safety comment needs some improvements. We probably should just put the entire cast into this.

Quoted 17 lines
rust/kernel/sync/lock.rs +13 -0
+///
+/// [`ContextualBackend`]: BackendWithContext::ContextualBackend
+/// [`Context`]: BackendWithContext::Context
+pub unsafe trait BackendWithContext: Backend {
+    /// The context which must be provided in order to acquire the lock with the
+    /// [`ContextualBackend`](BackendWithContext::ContextualBackend).
+    type Context<'a>;
+
+    /// The alternative cheaper backend we can use if a [`Context`](BackendWithContext::Context) is
+    /// provided.
+    type ContextualBackend: Backend<State = Self::State>;
+}
+
 /// A mutual exclusion primitive.
 ///
 /// Exposes one of the kernel locking primitives. Which one is exposed depends on the lock

rust/kernel/sync/lock.rs pub unsafe fn from_raw<'a>(ptr: *mut B::State) -> &'a Self { · line 206

Quoted 6 lines
rust/kernel/sync/lock.rs +2 -1
@@ -169,7 +202,8 @@ pub unsafe fn from_raw<'a>(ptr: *mut B::State) -> &'a Self {
 
 impl<T: ?Sized, B: Backend> Lock<T, B> {
     /// Acquires the lock and gives the caller access to the data protected by it.
-    pub fn lock(&self) -> Guard<'_, T, B> {
+    #[inline]
+    pub fn lock<'a>(&'a self) -> Guard<'a, T, B> {

Why this change?

Quoted 10 lines
rust/kernel/sync/lock.rs +3 -0
         // SAFETY: The constructor of the type calls `init`, so the existence of the object proves
         // that `init` was called.
         let state = unsafe { B::lock(self.state.get()) };
@@ -189,6 +223,41 @@ pub fn try_lock(&self) -> Option<Guard<'_, T, B>> {
     }
 }
 
+impl<T: ?Sized, B: BackendWithContext> Lock<T, B> {
+    /// Casts the lock as a `Lock<T, B::ContextualBackend>`.
+    fn as_lock_in_context<'a>(

rust/kernel/sync/lock.rs pub fn try_lock(&self) -> Option<Guard<'_, T, B>> { · line 238

Quoted 10 lines
rust/kernel/sync/lock.rs +10 -0
+        &'a self,
+        _context: B::Context<'a>,
+    ) -> &'a Lock<T, B::ContextualBackend>
+    where
+        B::ContextualBackend: Backend,
+    {
+        // SAFETY:
+        // - Per the safety guarantee of `Backend`, if `B::ContextualBackend` and `B` should
+        //   have the same state, the layout of the lock is the same so it's safe to convert one to
+        //   another.

This also relies on Lock being repr(C). repr(Rust) types are allowed to change layout in cases where their generics are substituted for others (even if those other ones have the same layout!).

Cheers,
Benno

maintainer
#36 lore raw

Re: [PATCH v17 10/16] rust: sync: Introduce lock::Lock::lock_with() and friends

In reply to Lyude Paul
Quoted 119 lines · Lyude Paul

SpinLockIrq and SpinLock use the exact same underlying C structure, with the only real difference being that the former uses the irq_disable() and irq_enable() variants for locking/unlocking. These variants can introduce some minor overhead in contexts where we already know that local processor interrupts are disabled, and as such we want a way to be able to skip modifying processor interrupt state in said contexts in order to avoid some overhead - just like the current C API allows us to do. So, ContextualBackend allows us to cast a lock into it's contextless version for situations where we already have whatever guarantees would be provided by BackendWithContext::ContextualBackend in place.

In some hacked-together benchmarks we ran, most of the time this did actually seem to lead to a noticeable difference in overhead:

From an aarch64 VM running on a MacBook M4:

    lock() when irq is disabled, 100 times cost Delta { nanos: 500 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 292 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 834 }

    lock() when irq is disabled, 100 times cost Delta { nanos: 459 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 291 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 709 }

From an x86_64 VM (qemu/kvm) running on a i7-13700H

    lock() when irq is disabled, 100 times cost Delta { nanos: 1002 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 729 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 1516 }

    lock() when irq is disabled, 100 times cost Delta { nanos: 754 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 966 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 1227 }

    (note that there were some runs on x86_64 where lock() on irq disabled
    vs. lock_with() on irq disabled had equivalent benchmarks, but it very
    much appeared to be a minority of test runs.

While it's not clear how this affects real-world workloads yet, let's add this for the time being so we can find out. Implement lock::Lock::lock_with() and lock::BackendWithContext::ContextualBackend. This makes it so that a SpinLockIrq will work like a SpinLock if interrupts are disabled. So a function:

        (&'a SpinLockIrq, &'a InterruptDisabled) -> Guard<'a, .., SpinLockBackend>

makes sense. Note that due to Guard and InterruptDisabled having the same lifetime, interrupts cannot be enabled while the Guard exists.

---
This was originally two patches, but keeping them split didn't make sense after going from BackendInContext to BackendWithContext.

V10:
* Fix typos - Dirk/Lyude
* Since we're adding support for context locks to GlobalLock as well, let's also make sure to cover try_lock while we're at it and add try_lock_with
* Add a private function as_lock_in_context() for handling casting from a Lock<T, B> to Lock<T, B::ContextualBackend> so we don't have to duplicate safety comments
V11:
* Fix clippy::ref_as_ptr error in Lock::as_lock_in_context() V14:
* Add benchmark results, rewrite commit message
V17:
* Introduce BackendWithContext, move context-related bits into there and out of Backend.
* Add missing #[must_use = …] for try_lock_with()
* Remove all unsafe code from lock_with() and try_lock_with():
Somehow I never noticed that literally none of the unsafe code in these two functions is needed with as_lock_in_context()...

 rust/kernel/sync/lock.rs          | 71 ++++++++++++++++++++++++++++++-
 rust/kernel/sync/lock/spinlock.rs | 48 ++++++++++++++++++++-
 2 files changed, 117 insertions(+), 2 deletions(-)
rust/kernel/sync/lock.rs +22 -0
index 46a57d1fc309d..9f6d7b381bd15 100644
@@ -30,10 +30,15 @@
 ///   is owned, that is, between calls to [`lock`] and [`unlock`].
 /// - Implementers must also ensure that [`relock`] uses the same locking method as the original
 ///   lock operation.
+/// - Implementers must ensure if [`BackendInContext`] is a [`Backend`], it's safe to acquire the
+///   lock under the [`Context`], the [`State`] of two backends must be the same.
 ///
 /// [`lock`]: Backend::lock
 /// [`unlock`]: Backend::unlock
 /// [`relock`]: Backend::relock
+/// [`BackendInContext`]: Backend::BackendInContext
+/// [`Context`]: Backend::Context
+/// [`State`]: Backend::State
 pub unsafe trait Backend {
     /// The state required by the lock.
     type State;
@@ -97,6 +102,34 @@ unsafe fn relock(ptr: *mut Self::State, guard_state: &mut Self::GuardState) {
     unsafe fn assert_is_held(ptr: *mut Self::State);
 }
 
+/// A lock [`Backend`] with a [`ContextualBackend`] that can make lock acquisition cheaper.
+///
+/// Some locks, such as [`SpinLockIrq`](super::SpinLockIrq), can only be acquired in specific
+/// hardware contexts (e.g. local processor interrupts disabled). Entering and exiting these
+/// contexts incurs additional overhead. But this overhead may be avoided if we know ahead of time
+/// that we are already within the correct context for a given lock as we can then skip any costly
+/// operations required for entering/exiting said context.
+///
+/// Any lock implementing this trait requires such a interrupt context, and can provide cheaper
+/// lock-acquisition functions through [`Lock::lock_with`] and [`Lock::try_lock_with`] as long as a
+/// context token of type [`Context`] is available.
+///
+/// # Safety
+///
+/// - Implementors must ensure that it is safe to acquire the lock under [`Context`].
+///
+/// [`ContextualBackend`]: BackendWithContext::ContextualBackend

rust/kernel/sync/lock.rs unsafe fn relock(ptr: *mut Self::State, guard_state: &mut Self::GuardState) { · line 131

Quoted 10 lines
rust/kernel/sync/lock.rs +10 -0
+/// [`Context`]: BackendWithContext::Context
+pub unsafe trait BackendWithContext: Backend {
+    /// The context which must be provided in order to acquire the lock with the
+    /// [`ContextualBackend`](BackendWithContext::ContextualBackend).
+    type Context<'a>;
+
+    /// The alternative cheaper backend we can use if a [`Context`](BackendWithContext::Context) is
+    /// provided.
+    type ContextualBackend: Backend<State = Self::State>;
+}

The dicsussion on Zulip seems to arrive in a consensus that we want to avoid generic approach at all and do an inherent implementation on Lock<T, SpinLockIqBackend> instead.

Best,
Gary

Quoted 129 lines
rust/kernel/sync/lock.rs +38 -1
+
 /// A mutual exclusion primitive.
 ///
 /// Exposes one of the kernel locking primitives. Which one is exposed depends on the lock
@@ -169,7 +202,8 @@ pub unsafe fn from_raw<'a>(ptr: *mut B::State) -> &'a Self {
 
 impl<T: ?Sized, B: Backend> Lock<T, B> {
     /// Acquires the lock and gives the caller access to the data protected by it.
-    pub fn lock(&self) -> Guard<'_, T, B> {
+    #[inline]
+    pub fn lock<'a>(&'a self) -> Guard<'a, T, B> {
         // SAFETY: The constructor of the type calls `init`, so the existence of the object proves
         // that `init` was called.
         let state = unsafe { B::lock(self.state.get()) };
@@ -189,6 +223,41 @@ pub fn try_lock(&self) -> Option<Guard<'_, T, B>> {
     }
 }
 
+impl<T: ?Sized, B: BackendWithContext> Lock<T, B> {
+    /// Casts the lock as a `Lock<T, B::ContextualBackend>`.
+    fn as_lock_in_context<'a>(
+        &'a self,
+        _context: B::Context<'a>,
+    ) -> &'a Lock<T, B::ContextualBackend>
+    where
+        B::ContextualBackend: Backend,
+    {
+        // SAFETY:
+        // - Per the safety guarantee of `Backend`, if `B::ContextualBackend` and `B` should
+        //   have the same state, the layout of the lock is the same so it's safe to convert one to
+        //   another.
+        // - The caller provided `B::Context<'a>`, so it is safe to recast and return this lock.
+        unsafe { &*(core::ptr::from_ref(self) as *const _) }
+    }
+
+    /// Acquires the lock with the given context and gives the caller access to the data protected
+    /// by it.
+    pub fn lock_with<'a>(&'a self, context: B::Context<'a>) -> Guard<'a, T, B::ContextualBackend> {
+        self.as_lock_in_context(context).lock()
+    }
+
+    /// Tries to acquire the lock with the given context.
+    ///
+    /// Returns a guard that can be used to access the data protected by the lock if successful.
+    #[must_use = "if unused, the lock will be immediately unlocked"]
+    pub fn try_lock_with<'a>(
+        &'a self,
+        context: B::Context<'a>,
+    ) -> Option<Guard<'a, T, B::ContextualBackend>> {
+        self.as_lock_in_context(context).try_lock()
+    }
+}
+
 /// A lock guard.
 ///
 /// Allows mutual exclusion primitives that implement the [`Backend`] trait to automatically unlock
rust/kernel/sync/lock/spinlock.rs +47 -1
index 3fdfb0a8a0ab1..e082791a0d23c 100644
@@ -3,7 +3,7 @@
 //! A kernel spinlock.
 //!
 //! This module allows Rust code to use the kernel's `spinlock_t`.
-use crate::prelude::*;
+use crate::{interrupt::LocalInterruptDisabled, prelude::*};
 
 /// Creates a [`SpinLock`] initialiser with the given name and a newly-created lock class.
 ///
@@ -220,6 +220,45 @@ macro_rules! new_spinlock_irq {
 /// # Ok::<(), Error>(())
 /// ```
 ///
+/// The next example demonstrates locking a [`SpinLockIrq`] using [`lock_with()`] in a function
+/// which can only be called when local processor interrupts are already disabled.
+///
+/// ```
+/// use kernel::sync::{new_spinlock_irq, SpinLockIrq};
+/// use kernel::interrupt::*;
+///
+/// struct Inner {
+///     a: u32,
+/// }
+///
+/// #[pin_data]
+/// struct Example {
+///     #[pin]
+///     inner: SpinLockIrq<Inner>,
+/// }
+///
+/// impl Example {
+///     fn new() -> impl PinInit<Self> {
+///         pin_init!(Self {
+///             inner <- new_spinlock_irq!(Inner { a: 20 }),
+///         })
+///     }
+/// }
+///
+/// // Accessing an `Example` from a function that can only be called in no-interrupt contexts.
+/// fn noirq_work(e: &Example, interrupt_disabled: &LocalInterruptDisabled) {
+///     // Because we know interrupts are disabled from interrupt_disable, we can skip toggling
+///     // interrupt state using lock_with() and the provided token
+///     assert_eq!(e.inner.lock_with(interrupt_disabled).a, 20);
+/// }
+///
+/// # let e = KBox::pin_init(Example::new(), GFP_KERNEL)?;
+/// # let interrupt_guard = local_interrupt_disable();
+/// # noirq_work(&e, &interrupt_guard);
+/// #
+/// # Ok::<(), Error>(())
+/// ```
+///
 /// [`lock()`]: SpinLockIrq::lock
 /// [`lock_with()`]: SpinLockIrq::lock_with
 pub type SpinLockIrq<T> = super::Lock<T, SpinLockIrqBackend>;
@@ -283,6 +322,13 @@ unsafe fn assert_is_held(ptr: *mut Self::State) {
     }
 }
 
+// SAFETY: When executing with local processor interrupts disabled, [`SpinLock`] and [`SpinLockIrq`]
+// are identical.
+unsafe impl super::BackendWithContext for SpinLockIrqBackend {
+    type Context<'a> = &'a LocalInterruptDisabled;
+    type ContextualBackend = SpinLockBackend;
+}
+
 #[kunit_tests(rust_spinlock_irq_condvar)]
 mod tests {
     use super::*;
author reviewer
#37 lore raw

PATCH v17 11/16 rust: sync: Expose lock::Backend

In reply to Lyude Paul

Due to the addition of sync::lock::Backend::Context, lock guards can be returned with a different Backend than their respective lock. Since we'll be adding a trait bound for Backend to GlobalGuard in order to support this, users will need to be able to directly refer to Backend so that they can use it in trait bounds.

So, let's make this easier for users and expose Backend in sync.

---
 rust/kernel/sync.rs | 1 +
 1 file changed, 1 insertion(+)
rust/kernel/sync.rs +1 -0
index 48a7cae86c50c..ce31154198cea 100644
@@ -30,6 +30,7 @@
 pub use lock::spinlock::{
     new_spinlock, new_spinlock_irq, SpinLock, SpinLockGuard, SpinLockIrq, SpinLockIrqGuard,
 };
+pub use lock::Backend;
 pub use locked_by::LockedBy;
 pub use refcount::Refcount;
 pub use set_once::SetOnce;
Signature · 1 line
2.52.0
reviewer Reviewed-by on v17 11/16
#38 lore raw

Re: [PATCH v17 11/16] rust: sync: Expose lock::Backend

Quoted 9 lines · Lyude Paul

Due to the addition of sync::lock::Backend::Context, lock guards can be returned with a different Backend than their respective lock. Since we'll be adding a trait bound for Backend to GlobalGuard in order to support this, users will need to be able to directly refer to Backend so that they can use it in trait bounds.

So, let's make this easier for users and expose Backend in sync.

Cheers,
Benno

Quoted 16 lines
---
 rust/kernel/sync.rs | 1 +
 1 file changed, 1 insertion(+)
rust/kernel/sync.rs +1 -0
index 48a7cae86c50c..ce31154198cea 100644
@@ -30,6 +30,7 @@
 pub use lock::spinlock::{
     new_spinlock, new_spinlock_irq, SpinLock, SpinLockGuard, SpinLockIrq, SpinLockIrqGuard,
 };
+pub use lock::Backend;
 pub use locked_by::LockedBy;
 pub use refcount::Refcount;
 pub use set_once::SetOnce;
author reviewer
#39 lore raw

PATCH v17 12/16 rust: sync: lock/global: Rename B to G in trait bounds

In reply to Lyude Paul

Due to the introduction of BackendWithContext::ContextualBackend, if we want to be able support Lock types with a Context we need to be able to handle the fact that the Backend for a returned Guard may not exactly match the Backend for the lock. Before we add this though, rename B to G in all of our trait bounds to make sure things don't become more difficult to understand once we add a Backend bound.

There should be no functional changes in this patch.

---
 rust/kernel/sync/lock/global.rs | 58 ++++++++++++++++-----------------
 1 file changed, 29 insertions(+), 29 deletions(-)
rust/kernel/sync/lock/global.rs +29 -29
index 7030a47bc0ad1..06d62ad02f90d 100644
@@ -33,18 +33,18 @@ pub trait GlobalLockBackend {
 /// Type used for global locks.
 ///
 /// See [`global_lock!`] for examples.
-pub struct GlobalLock<B: GlobalLockBackend> {
-    inner: Lock<B::Item, B::Backend>,
+pub struct GlobalLock<G: GlobalLockBackend> {
+    inner: Lock<G::Item, G::Backend>,
 }
 
-impl<B: GlobalLockBackend> GlobalLock<B> {
+impl<G: GlobalLockBackend> GlobalLock<G> {
     /// Creates a global lock.
     ///
     /// # Safety
     ///
     /// * Before any other method on this lock is called, [`Self::init`] must be called.
-    /// * The type `B` must not be used with any other lock.
-    pub const unsafe fn new(data: B::Item) -> Self {
+    /// * The type `G` must not be used with any other lock.
+    pub const unsafe fn new(data: G::Item) -> Self {
         Self {
             inner: Lock {
                 state: Opaque::uninit(),
@@ -68,23 +68,23 @@ pub unsafe fn init(&'static self) {
         // `init` before using any other methods. As `init` can only be called once, all other
         // uses of this lock must happen after this call.
         unsafe {
-            B::Backend::init(
+            G::Backend::init(
                 self.inner.state.get(),
-                B::NAME.as_char_ptr(),
-                B::get_lock_class().as_ptr(),
+                G::NAME.as_char_ptr(),
+                G::get_lock_class().as_ptr(),
             )
         }
     }
 
     /// Lock this global lock.
-    pub fn lock(&'static self) -> GlobalGuard<B> {
+    pub fn lock(&'static self) -> GlobalGuard<G> {
         GlobalGuard {
             inner: self.inner.lock(),
         }
     }
 
     /// Try to lock this global lock.
-    pub fn try_lock(&'static self) -> Option<GlobalGuard<B>> {
+    pub fn try_lock(&'static self) -> Option<GlobalGuard<G>> {
         Some(GlobalGuard {
             inner: self.inner.try_lock()?,
         })
@@ -94,21 +94,21 @@ pub fn try_lock(&'static self) -> Option<GlobalGuard<B>> {
 /// A guard for a [`GlobalLock`].
 ///
 /// See [`global_lock!`] for examples.
-pub struct GlobalGuard<B: GlobalLockBackend> {
-    inner: Guard<'static, B::Item, B::Backend>,
+pub struct GlobalGuard<G: GlobalLockBackend> {
+    inner: Guard<'static, G::Item, G::Backend>,
 }
 
-impl<B: GlobalLockBackend> core::ops::Deref for GlobalGuard<B> {
-    type Target = B::Item;
+impl<G: GlobalLockBackend> core::ops::Deref for GlobalGuard<G> {
+    type Target = G::Item;
 
     fn deref(&self) -> &Self::Target {
         &self.inner
     }
 }
 
-impl<B: GlobalLockBackend> core::ops::DerefMut for GlobalGuard<B>
+impl<G: GlobalLockBackend> core::ops::DerefMut for GlobalGuard<G>
 where
-    B::Item: Unpin,
+    G::Item: Unpin,
 {
     fn deref_mut(&mut self) -> &mut Self::Target {
         &mut self.inner
@@ -118,33 +118,33 @@ fn deref_mut(&mut self) -> &mut Self::Target {
 /// A version of [`LockedBy`] for a [`GlobalLock`].
 ///
 /// See [`global_lock!`] for examples.
-pub struct GlobalLockedBy<T: ?Sized, B: GlobalLockBackend> {
-    _backend: PhantomData<B>,
+pub struct GlobalLockedBy<T: ?Sized, G: GlobalLockBackend> {
+    _backend: PhantomData<G>,
     value: UnsafeCell<T>,
 }
 
 // SAFETY: The same thread-safety rules as `LockedBy` apply to `GlobalLockedBy`.
-unsafe impl<T, B> Send for GlobalLockedBy<T, B>
+unsafe impl<T, G> Send for GlobalLockedBy<T, G>
 where
     T: ?Sized,
-    B: GlobalLockBackend,
-    LockedBy<T, B::Item>: Send,
+    G: GlobalLockBackend,
+    LockedBy<T, G::Item>: Send,
 {
 }
 
 // SAFETY: The same thread-safety rules as `LockedBy` apply to `GlobalLockedBy`.
-unsafe impl<T, B> Sync for GlobalLockedBy<T, B>
+unsafe impl<T, G> Sync for GlobalLockedBy<T, G>
 where
     T: ?Sized,
-    B: GlobalLockBackend,
-    LockedBy<T, B::Item>: Sync,
+    G: GlobalLockBackend,
+    LockedBy<T, G::Item>: Sync,
 {
 }
 
-impl<T, B: GlobalLockBackend> GlobalLockedBy<T, B> {
+impl<T, G: GlobalLockBackend> GlobalLockedBy<T, G> {
     /// Create a new [`GlobalLockedBy`].
     ///
-    /// The provided value will be protected by the global lock indicated by `B`.
+    /// The provided value will be protected by the global lock indicated by `G`.
     pub fn new(val: T) -> Self {
         Self {
             value: UnsafeCell::new(val),
@@ -153,11 +153,11 @@ pub fn new(val: T) -> Self {
     }
 }
 
-impl<T: ?Sized, B: GlobalLockBackend> GlobalLockedBy<T, B> {
+impl<T: ?Sized, G: GlobalLockBackend> GlobalLockedBy<T, G> {
     /// Access the value immutably.
     ///
     /// The caller must prove shared access to the lock.
-    pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<B>) -> &'a T {
+    pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<G>) -> &'a T {
         // SAFETY: The lock is globally unique, so there can only be one guard.
         unsafe { &*self.value.get() }
     }
@@ -165,7 +165,7 @@ pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<B>) -> &'a T {
     /// Access the value mutably.
     ///
     /// The caller must prove shared exclusive to the lock.
-    pub fn as_mut<'a>(&'a self, _guard: &'a mut GlobalGuard<B>) -> &'a mut T {
+    pub fn as_mut<'a>(&'a self, _guard: &'a mut GlobalGuard<G>) -> &'a mut T {
         // SAFETY: The lock is globally unique, so there can only be one guard.
         unsafe { &mut *self.value.get() }
     }
Signature · 1 line
2.52.0
author reviewer
#40 lore raw

PATCH v17 13/16 rust: sync: Add a lifetime parameter to lock::global::GlobalGuard

In reply to Lyude Paul

While a GlobalLock is always going to be static, in the case of locks with explicit backend contexts the GlobalGuard will not be 'static and will instead share the lifetime of the context. So, add a lifetime parameter to GlobalGuard to allow for this so we can implement GlobalGuard support for SpinlockIrq.

---
 rust/kernel/sync/lock/global.rs | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
rust/kernel/sync/lock/global.rs +9 -9
index 06d62ad02f90d..be17a30c66bf8 100644
@@ -77,14 +77,14 @@ pub unsafe fn init(&'static self) {
     }
 
     /// Lock this global lock.
-    pub fn lock(&'static self) -> GlobalGuard<G> {
+    pub fn lock(&'static self) -> GlobalGuard<'static, G> {
         GlobalGuard {
             inner: self.inner.lock(),
         }
     }
 
     /// Try to lock this global lock.
-    pub fn try_lock(&'static self) -> Option<GlobalGuard<G>> {
+    pub fn try_lock(&'static self) -> Option<GlobalGuard<'static, G>> {
         Some(GlobalGuard {
             inner: self.inner.try_lock()?,
         })
@@ -94,11 +94,11 @@ pub fn try_lock(&'static self) -> Option<GlobalGuard<G>> {
 /// A guard for a [`GlobalLock`].
 ///
 /// See [`global_lock!`] for examples.
-pub struct GlobalGuard<G: GlobalLockBackend> {
-    inner: Guard<'static, G::Item, G::Backend>,
+pub struct GlobalGuard<'a, G: GlobalLockBackend> {
+    inner: Guard<'a, G::Item, G::Backend>,
 }
 
-impl<G: GlobalLockBackend> core::ops::Deref for GlobalGuard<G> {
+impl<'a, G: GlobalLockBackend> core::ops::Deref for GlobalGuard<'a, G> {
     type Target = G::Item;
 
     fn deref(&self) -> &Self::Target {
@@ -106,7 +106,7 @@ fn deref(&self) -> &Self::Target {
     }
 }
 
-impl<G: GlobalLockBackend> core::ops::DerefMut for GlobalGuard<G>
+impl<'a, G: GlobalLockBackend> core::ops::DerefMut for GlobalGuard<'a, G>
 where
     G::Item: Unpin,
 {
@@ -157,7 +157,7 @@ impl<T: ?Sized, G: GlobalLockBackend> GlobalLockedBy<T, G> {
     /// Access the value immutably.
     ///
     /// The caller must prove shared access to the lock.
-    pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<G>) -> &'a T {
+    pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<'_, G>) -> &'a T {
         // SAFETY: The lock is globally unique, so there can only be one guard.
         unsafe { &*self.value.get() }
     }
@@ -165,7 +165,7 @@ pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<G>) -> &'a T {
     /// Access the value mutably.
     ///
     /// The caller must prove shared exclusive to the lock.
-    pub fn as_mut<'a>(&'a self, _guard: &'a mut GlobalGuard<G>) -> &'a mut T {
+    pub fn as_mut<'a>(&'a self, _guard: &'a mut GlobalGuard<'_, G>) -> &'a mut T {
         // SAFETY: The lock is globally unique, so there can only be one guard.
         unsafe { &mut *self.value.get() }
     }
@@ -235,7 +235,7 @@ pub fn get_mut(&mut self) -> &mut T {
 ///     /// Increment the counter in this instance.
 ///     ///
 ///     /// The caller must hold the `MY_MUTEX` mutex.
-///     fn increment(&self, guard: &mut GlobalGuard<MY_MUTEX>) -> u32 {
+///     fn increment(&self, guard: &mut GlobalGuard<'_, MY_MUTEX>) -> u32 {
 ///         let my_counter = self.my_counter.as_mut(guard);
 ///         *my_counter += 1;
 ///         *my_counter
Signature · 1 line
2.52.0
author reviewer
#41 lore raw

PATCH v17 14/16 rust: sync: lock/global: Add Backend parameter to GlobalGuard

In reply to Lyude Paul

Due to the introduction of sync::lock::Backend::Context, it's now possible for normal locks to return a Guard with a different Backend than their respective lock (e.g. Backend::BackendInContext). We want to be able to support global locks with contexts as well, so add a trait bound to explicitly specify which Backend is in use for a GlobalGuard.

---
V17:
* Add default parameter for generic B to GlobalGuard

 rust/kernel/sync/lock/global.rs | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
rust/kernel/sync/lock/global.rs +10 -10
index be17a30c66bf8..94f6b3b21324f 100644
@@ -77,14 +77,14 @@ pub unsafe fn init(&'static self) {
     }
 
     /// Lock this global lock.
-    pub fn lock(&'static self) -> GlobalGuard<'static, G> {
+    pub fn lock(&'static self) -> GlobalGuard<'static, G, G::Backend> {
         GlobalGuard {
             inner: self.inner.lock(),
         }
     }
 
     /// Try to lock this global lock.
-    pub fn try_lock(&'static self) -> Option<GlobalGuard<'static, G>> {
+    pub fn try_lock(&'static self) -> Option<GlobalGuard<'static, G, G::Backend>> {
         Some(GlobalGuard {
             inner: self.inner.try_lock()?,
         })
@@ -94,11 +94,11 @@ pub fn try_lock(&'static self) -> Option<GlobalGuard<'static, G>> {
 /// A guard for a [`GlobalLock`].
 ///
 /// See [`global_lock!`] for examples.
-pub struct GlobalGuard<'a, G: GlobalLockBackend> {
-    inner: Guard<'a, G::Item, G::Backend>,
+pub struct GlobalGuard<'a, G: GlobalLockBackend, B: Backend = <G as GlobalLockBackend>::Backend> {
+    inner: Guard<'a, G::Item, B>,
 }
 
-impl<'a, G: GlobalLockBackend> core::ops::Deref for GlobalGuard<'a, G> {
+impl<'a, G: GlobalLockBackend, B: Backend> core::ops::Deref for GlobalGuard<'a, G, B> {
     type Target = G::Item;
 
     fn deref(&self) -> &Self::Target {
@@ -106,7 +106,7 @@ fn deref(&self) -> &Self::Target {
     }
 }
 
-impl<'a, G: GlobalLockBackend> core::ops::DerefMut for GlobalGuard<'a, G>
+impl<'a, G: GlobalLockBackend, B: Backend> core::ops::DerefMut for GlobalGuard<'a, G, B>
 where
     G::Item: Unpin,
 {
@@ -157,7 +157,7 @@ impl<T: ?Sized, G: GlobalLockBackend> GlobalLockedBy<T, G> {
     /// Access the value immutably.
     ///
     /// The caller must prove shared access to the lock.
-    pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<'_, G>) -> &'a T {
+    pub fn as_ref<'a, B: Backend>(&'a self, _guard: &'a GlobalGuard<'_, G, B>) -> &'a T {
         // SAFETY: The lock is globally unique, so there can only be one guard.
         unsafe { &*self.value.get() }
     }
@@ -165,7 +165,7 @@ pub fn as_ref<'a>(&'a self, _guard: &'a GlobalGuard<'_, G>) -> &'a T {
     /// Access the value mutably.
     ///
     /// The caller must prove shared exclusive to the lock.
-    pub fn as_mut<'a>(&'a self, _guard: &'a mut GlobalGuard<'_, G>) -> &'a mut T {
+    pub fn as_mut<'a, B: Backend>(&'a self, _guard: &'a mut GlobalGuard<'_, G, B>) -> &'a mut T {
         // SAFETY: The lock is globally unique, so there can only be one guard.
         unsafe { &mut *self.value.get() }
     }
@@ -219,7 +219,7 @@ pub fn get_mut(&mut self) -> &mut T {
 /// ```
 /// # mod ex {
 /// # use kernel::prelude::*;
-/// use kernel::sync::{GlobalGuard, GlobalLockedBy};
+/// use kernel::sync::{Backend, GlobalGuard, GlobalLockedBy};
 ///
 /// kernel::sync::global_lock! {
 ///     // SAFETY: Initialized in module initializer before first use.
@@ -235,7 +235,7 @@ pub fn get_mut(&mut self) -> &mut T {
 ///     /// Increment the counter in this instance.
 ///     ///
 ///     /// The caller must hold the `MY_MUTEX` mutex.
-///     fn increment(&self, guard: &mut GlobalGuard<'_, MY_MUTEX>) -> u32 {
+///     fn increment<B: Backend>(&self, guard: &mut GlobalGuard<'_, MY_MUTEX, B>) -> u32 {
 ///         let my_counter = self.my_counter.as_mut(guard);
 ///         *my_counter += 1;
 ///         *my_counter
Signature · 1 line
2.52.0
author reviewer
#42 lore raw

PATCH v17 15/16 rust: sync: lock/global: Add ContextualBackend support to GlobalLock

In reply to Lyude Paul

Now that we have the ability to provide an explicit lifetime for a GlobalGuard and an explicit Backend for a GlobalGuard, we can finally implement lock_with() and try_lock_with().

---
 rust/kernel/sync/lock/global.rs | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
rust/kernel/sync/lock/global.rs +29 -1
index 94f6b3b21324f..cfc71af7482ed 100644
@@ -6,7 +6,7 @@
 
 use crate::{
     str::{CStr, CStrExt as _},
-    sync::lock::{Backend, Guard, Lock},
+    sync::lock::{Backend, BackendWithContext, Guard, Lock},
     sync::{LockClassKey, LockedBy},
     types::Opaque,
 };
@@ -89,6 +89,34 @@ pub fn try_lock(&'static self) -> Option<GlobalGuard<'static, G, G::Backend>> {
             inner: self.inner.try_lock()?,
         })
     }
+
+    /// Lock this global lock with the provided `context`.
+    pub fn lock_with<'a, B>(
+        &'static self,
+        context: <G::Backend as BackendWithContext>::Context<'a>,
+    ) -> GlobalGuard<'a, G, B>
+    where
+        G::Backend: BackendWithContext<ContextualBackend = B>,
+        B: Backend,
+    {
+        GlobalGuard {
+            inner: self.inner.lock_with(context),
+        }
+    }
+
+    /// Try to lock this global lock with the provided `context`.
+    pub fn try_lock_with<'a, B>(
+        &'static self,
+        context: <G::Backend as BackendWithContext>::Context<'a>,
+    ) -> Option<GlobalGuard<'a, G, B>>
+    where
+        G::Backend: BackendWithContext<ContextualBackend = B>,
+        B: Backend,
+    {
+        Some(GlobalGuard {
+            inner: self.inner.try_lock_with(context)?,
+        })
+    }
 }
 
 /// A guard for a [`GlobalLock`].
Signature · 1 line
2.52.0
author reviewer
#43 lore raw

PATCH v17 16/16 locking: Switch to _irq_{disable,enable}() variants in cleanup guards

In reply to Lyude Paul

From: Boqun Feng <[email protected]>

The semantics of various irq disabling guards match what *_irq_{disable,enable}() provide, i.e. the interrupt disabling is properly nested, therefore it's OK to switch to use *_irq_{disable,enable}() primitives.

---
V10:
* Add PREEMPT_RT build fix from Guangbo Cui

 include/linux/spinlock.h | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)
include/linux/spinlock.h +12 -14
index bbbee61c6f5df..72bb6ae5319c7 100644
@@ -568,10 +568,10 @@ DEFINE_LOCK_GUARD_1(raw_spinlock_nested, raw_spinlock_t,
 		    raw_spin_unlock(_T->lock))
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irq, raw_spinlock_t,
-		    raw_spin_lock_irq(_T->lock),
-		    raw_spin_unlock_irq(_T->lock))
+		    raw_spin_lock_irq_disable(_T->lock),
+		    raw_spin_unlock_irq_enable(_T->lock))
 
-DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq(_T->lock))
+DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq_disable(_T->lock))
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_bh, raw_spinlock_t,
 		    raw_spin_lock_bh(_T->lock),
@@ -580,12 +580,11 @@ DEFINE_LOCK_GUARD_1(raw_spinlock_bh, raw_spinlock_t,
 DEFINE_LOCK_GUARD_1_COND(raw_spinlock_bh, _try, raw_spin_trylock_bh(_T->lock))
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irqsave, raw_spinlock_t,
-		    raw_spin_lock_irqsave(_T->lock, _T->flags),
-		    raw_spin_unlock_irqrestore(_T->lock, _T->flags),
-		    unsigned long flags)
+		    raw_spin_lock_irq_disable(_T->lock),
+		    raw_spin_unlock_irq_enable(_T->lock))
 
 DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irqsave, _try,
-			 raw_spin_trylock_irqsave(_T->lock, _T->flags))
+			 raw_spin_trylock_irq_disable(_T->lock))
 
 DEFINE_LOCK_GUARD_1(spinlock, spinlock_t,
 		    spin_lock(_T->lock),
@@ -594,11 +593,11 @@ DEFINE_LOCK_GUARD_1(spinlock, spinlock_t,
 DEFINE_LOCK_GUARD_1_COND(spinlock, _try, spin_trylock(_T->lock))
 
 DEFINE_LOCK_GUARD_1(spinlock_irq, spinlock_t,
-		    spin_lock_irq(_T->lock),
-		    spin_unlock_irq(_T->lock))
+		    spin_lock_irq_disable(_T->lock),
+		    spin_unlock_irq_enable(_T->lock))
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irq, _try,
-			 spin_trylock_irq(_T->lock))
+			 spin_trylock_irq_disable(_T->lock))
 
 DEFINE_LOCK_GUARD_1(spinlock_bh, spinlock_t,
 		    spin_lock_bh(_T->lock),
@@ -608,12 +607,11 @@ DEFINE_LOCK_GUARD_1_COND(spinlock_bh, _try,
 			 spin_trylock_bh(_T->lock))
 
 DEFINE_LOCK_GUARD_1(spinlock_irqsave, spinlock_t,
-		    spin_lock_irqsave(_T->lock, _T->flags),
-		    spin_unlock_irqrestore(_T->lock, _T->flags),
-		    unsigned long flags)
+		    spin_lock_irq_disable(_T->lock),
+		    spin_unlock_irq_enable(_T->lock))
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irqsave, _try,
-			 spin_trylock_irqsave(_T->lock, _T->flags))
+			 spin_trylock_irq_disable(_T->lock))
 
 DEFINE_LOCK_GUARD_1(read_lock, rwlock_t,
 		    read_lock(_T->lock),
Signature · 1 line
2.52.0
maintainer
#44 lore raw
In reply to Lyude Paul
Quoted 8 lines · Lyude Paul

This is the latest patch series for adding rust bindings for controlling local processor interrupts, adding support for spinlocks in rust that are acquired with local processor interrupts disabled, and implementing local interrupt controls through refcounting in the kernel.

The previous version of this patch series can be found here:

https://lkml.org/lkml/2025/12/15/1190

Please use lore.kernel.org (or patch.msgid.link) links instead.

Best,
Gary

Quoted 100 lines

This patch series applies on top of the rust-next branch.

There's a few big changes from the last time. Mainly that we've addressed all(?) of the open questions on this patch series:

* Thanks to Joel Fernandes, we now have a seperate per-CPU counter for tracking NMI nesting - which ensures that we don't have to sacrifice NMI nest level bits in order to store a counter for refcounted IRQs. These patches have been included at the start of the series.
* We've been able to prove that being able to convert the kernel over to this new interface is indeed possible, more on this below.
* Also thank to Joel, we also now have actual benchmarks for how this affects performance:
https://lore.kernel.org/rust-for-linux/[email protected]/
* Also some small changes to the kunit test I added, mainly just making sure I don't forget to include a MODULE_DESCRIPTION or MODULE_LICENSE.

Regarding the conversion plan: we've had some success at getting kernels to boot after attempting to convert the entire kernel from the non-refcounted API to the new refcounted API. It will definitely take quite a lot of work to get this right though, at least in the kernel core side of things. To give readers an idea of what I mean, here's a few of the issues that we ended up running into:

On my end, I tried running a number of coccinelle conversions for this. At first I did actually try simply rewiring local_irq_disable()/local_irq_enable() to local_interrupt_enable()/local_interrupt_disable(). This wasn't really workable though, as it causes the kernel to crash very early on in a number of ways that I haven't fully untangled. Doing this with coccinelle on the other hand allowed me to convert individual files at a time, along with specific usage patterns of the old API, and as a result this ended up giving me a pretty good idea of where our issues are coming from. This coccinelle script, while still leaving most of the kernel unconverted, was at least able to be run on almost all of kernel/ while still allowing us to boot on x86_64

@depends on patch && !report@
@@
- local_irq_disable();
+ local_interrupt_disable();
...
- local_irq_enable();
+ local_interrupt_enable();

There were two files in kernel/ that were exceptions to this:

* kernel/softirq.c
* kernel/main.c (I figured out at least one fix to an issue here)

The reason this worked is because it seems like the vast majority of the issues we're seeing come from "unbalanced"/"misordered" usages of the old irq API. And there seems to be a few reasons for this:

* The first simple reason: occasionally the enable/disable was split across a function, which this script didn't handle.
* The second more complicated reason: some portions of the kernel core end up calling processor instructions that modify the processor's local interrupt flags independently of the kernel. In x86_64's case, I believe we came to the conclusion the iret instruction (interrupt return) was modifying the interrupt flag state. There's possibly a few more instances like this elsewhere.

Boqun also took a stab at this on aarch64, and ended up having similar findings. In their case, they discovered one of the culprits being raw_spin_rq_unlock_irq(). Here the reason is that on aarch64 preempt_count is per-thread and not just per-cpu, and when context switching you generally disable interrupts from one task and restore it in the other task. So in order to fix it, we'll need to make some modifications to the aarch64 context-switching code.

So - with this being said, we decided that the best way of converting it is likely to just leave us with 3 APIs for the time being - and have new drivers and code use the new API while we go through and convert the rest of the kernel.

FULL CHANGELOG BELOW

Boqun Feng (5):
preempt: Introduce HARDIRQ_DISABLE_BITS
preempt: Introduce __preempt_count_{sub, add}_return()
irq & spin_lock: Add counted interrupt disabling/enabling
rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers
locking: Switch to _irq_{disable,enable}() variants in cleanup guards

Joel Fernandes (1):
preempt: Track NMI nesting to separate per-CPU counter

Lyude Paul (10):
openrisc: Include <linux/cpumask.h> in smp.h
irq: Add KUnit test for refcounted interrupt enable/disable
rust: Introduce interrupt module
rust: sync: Add SpinLockIrq
rust: sync: Introduce lock::Lock::lock_with() and friends
rust: sync: Expose lock::Backend
rust: sync: lock/global: Rename B to G in trait bounds
rust: sync: Add a lifetime parameter to lock::global::GlobalGuard
rust: sync: lock/global: Add Backend parameter to GlobalGuard
rust: sync: lock/global: Add ContextualBackend support to GlobalLock

#45 lore raw
In reply to Lyude Paul
Quoted 5 lines · Lyude Paul

This is the latest patch series for adding rust bindings for controlling local processor interrupts, adding support for spinlocks in rust that are acquired with local processor interrupts disabled, and implementing local interrupt controls through refcounting in the kernel.

If there is no objection, I plan to queue patch #1 to #6 and patch #16 first, hence kindly ping for any review on these.

Regards,
Boqun

maintainer
#46 lore raw
Quoted 8 lines · Boqun Feng
Quoted 5 lines · Lyude Paul

This is the latest patch series for adding rust bindings for controlling local processor interrupts, adding support for spinlocks in rust that are acquired with local processor interrupts disabled, and implementing local interrupt controls through refcounting in the kernel.

If there is no objection, I plan to queue patch #1 to #6 and patch #16

Hearing none, I just queued them into rust-sync:

https://git.kernel.org/pub/scm/linux/kernel/git/boqun/linux.git/ rust-sync

For more tests and reviews. Thanks!

@Lyude, for the rest of them, please resolve comments and rebase on rust-sync. Thank you for the great work!

Regards,
Boqun

Quoted 4 lines

first, hence kindly ping for any review on these.

Regards,
Boqun