Rust synchronization changes for v7.3
Subsystems (17)
Boqun Feng
maintainerHi Peter,
Please pull this Rust synchronization changes for v7.3 into tip.
The major changes are the introduction of preempt_count backed interrupt disabling level tracking, i.e. the local_interrupt_{enable, disable}(), along with the user (the Rust SpinLockIrq). This should unblock a few drivers in Rust. For other changes, please refer the following change log.
The following changes since commit 5e601ab3615c86be7c4068ce992f94654693a032:
futex: Optimise the size check get_futex_key() (2026-07-29 13:17:52 +0200)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/boqun/linux.git tags/rust-sync.20260731a
for you to fetch changes up to 93ff5657a5f1d0d8577b58e66bc116a1e0f38a87:
rust: revocable: Use LKMM atomics instead of Rust atomics (2026-07-30 13:38:53 -0700)
----------------------------------------------------------------
Rust synchronization changes for v7.3:
* Safe SpinLockIrq
- Introduce refcounted interrupt disabling and corresponding spinlock primitives, which tracks the interrupt disabling "level" in preempt_count, as a result, this set of primitives can automatically restore the interrupt disabling status when the outermost critical section exits.
- Based on the above, a SpinLockIrq is introduced on the Rust side, the guard of which will automatically restore the interrupt disabling status when dropped. This avoid playing unsafe games when using irq disabled spinlocks.
* Add synchronize_rcu() safe abstraction.
* Generic memory barrier: this extends our Rust barrier primitives to include more barriers for DMA as well.
* Replace the atomic usage in Revocable with LKMM atomics. -----BEGIN PGP SIGNATURE-----
iQFhBAABCABLFiEEj5IosQTPz8XU1wRHSXnow7UH+rgFAmptA2MbFIAAAAAABAAO bWFudTIsMi41KzEuMTIsMCwzERxib3F1bkBrZXJuZWwub3JnAAoJEEl56MO1B/q4 UBAH/108a8dCkDaK3MUuS0SeuqY57EPwLCTE/k4gJK35XlF0fCpEPfFtqfEs6xkG E0rMhSx3bDmtfccEo16LOkmbcrSx3NSsc9KlSRiYRywptT+CRld5FPqmJqDd2nCh mYYUsNppEXFd4rUiGfKAcVeLHD+PCio0/ElNN7X3TCfQgtWUuDp4pcB23m9t6ste Re4idhKoTmqeKTX3N8AuQ0QMsr1FNNLu0ESpUv1n6/DrEB1g8XevF1J2PKYr9fxt aahCMk2qQF3vDCi4Ydh9o9AWqSkinU2yIW64ZBWRC2S7QBVRIrPAEtX4V/n7d/n+ 2irbb1ZuVu5zFD0/CyNkgyQM/og=
=DzLT
-----END PGP SIGNATURE-----
Regards,
Boqun
----------------------------------------------------------------
Boqun Feng (10):
preempt: Introduce HARDIRQ_DISABLE_BITS
preempt: Introduce __preempt_count_{sub, add}_return() irq & spin_lock: Add counted interrupt disabling/enabling
locking: Switch to _irq_{disable,enable}() variants in cleanup guards
sched: Remove the unused preempt_offset parameter of __cant_sleep()
sched: Avoid signed comparison of preempt_count() in __cant_migrate()
preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS
arm64: sched/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
irq: Optimize reschedule check in local_interrupt_enable()
rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers
Gary Guo (3):
rust: sync: Add helpers for mb, dma_mb and friends
rust: sync: Add generic memory barriers
rust: revocable: Use LKMM atomics instead of Rust atomics
Heiko Carstens (1):
s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
Joel Fernandes (1):
preempt: Track NMI nesting to separate per-CPU counter
Lyude Paul (6):
openrisc: Include <linux/cpumask.h> in smp.h
irq: Add KUnit test for refcounted interrupt enable/disable
rust: Introduce interrupt module
rust: sync: use super::* in spinlock.rs
rust: sync: Add SpinLockIrq
rust: sync: Introduce SpinLockIrq::lock_with() and friends
Philipp Stanner (3):
rust: sync: Add abstraction for synchronize_rcu()
rust: revocable: Use safe synchronize_rcu() abstraction
rust: sync: Use safe synchronize_rcu() abstraction in poll
arch/arm64/Kconfig | 1 + arch/arm64/include/asm/preempt.h | 18 ++ arch/openrisc/include/asm/smp.h | 2 + arch/s390/Kconfig | 1 + arch/s390/include/asm/lowcore.h | 13 +- arch/s390/include/asm/preempt.h | 49 ++-- arch/x86/Kconfig | 1 + arch/x86/include/asm/preempt.h | 61 +++-- arch/x86/kernel/cpu/common.c | 2 +- include/asm-generic/preempt.h | 14 ++ include/linux/hardirq.h | 41 ++- include/linux/interrupt_rc.h | 76 ++++++ include/linux/kernel.h | 4 +- include/linux/preempt.h | 35 ++- include/linux/spinlock.h | 35 ++- include/linux/spinlock_api_smp.h | 41 +++ include/linux/spinlock_api_up.h | 16 ++ include/linux/spinlock_rt.h | 18 ++ kernel/Kconfig.preempt | 4 + kernel/irq/Makefile | 1 + kernel/irq/refcount_interrupt_test.c | 109 ++++++++ kernel/locking/spinlock.c | 29 +++ kernel/sched/core.c | 18 +- kernel/softirq.c | 22 +- lib/locking-selftest.c | 2 +- rust/helpers/barrier.c | 30 +++ rust/helpers/helpers.c | 1 + rust/helpers/interrupt.c | 18 ++ rust/helpers/spinlock.c | 15 ++ rust/helpers/sync.c | 5 + rust/kernel/interrupt.rs | 89 +++++++ rust/kernel/lib.rs | 1 + rust/kernel/revocable.rs | 28 ++- rust/kernel/sync.rs | 9 +- rust/kernel/sync/atomic/ordering.rs | 2 +- rust/kernel/sync/barrier.rs | 127 +++++++--- rust/kernel/sync/lock/global.rs | 3 + rust/kernel/sync/lock/spinlock.rs | 331 ++++++++++++++++++++++++- rust/kernel/sync/poll.rs | 10 +- rust/kernel/sync/rcu.rs | 16 ++ tools/testing/selftests/bpf/bpf_experimental.h | 7 +- 41 files changed, 1184 insertions(+), 121 deletions(-) create mode 100644 include/linux/interrupt_rc.h create mode 100644 kernel/irq/refcount_interrupt_test.c create mode 100644 rust/helpers/interrupt.c create mode 100644 rust/kernel/interrupt.rs
Boqun Feng
maintainerPATCH 1/24 preempt: Track NMI nesting to separate per-CPU counter
From: Joel Fernandes <[email protected]>
Move NMI nesting tracking from the preempt_count bits to a separate per-CPU counter (nmi_nesting). This is to free up the NMI bits in the preempt_count, allowing those bits to be repurposed for other uses.
Reduce NMI_BITS from 4 to 1, using it only to detect if we're in an NMI. The per-CPU counter currently caps nesting at 15.
[boqun: Solve Steven Rostedt's comment on the BUG_ON() condition] [boqun: Use preempt_count_set() in __nmi_exit() to avoid underflow]
--- include/linux/hardirq.h | 17 +++++++++++++---- include/linux/preempt.h | 9 +++++++-- kernel/softirq.c | 2 ++ tools/testing/selftests/bpf/bpf_experimental.h | 2 +- 4 files changed, 23 insertions(+), 7 deletions(-)
include/linux/hardirq.h +13 -4
@@ -10,6 +10,8 @@#include <linux/vtime.h> #include <asm/hardirq.h> +DECLARE_PER_CPU(unsigned int, nmi_nesting); +extern void synchronize_irq (unsigned int irq);extern bool synchronize_hardirq (unsigned int irq);
@@ -102,14 +104,17 @@ void irq_exit_rcu(void);*/ /* -* nmi_enter() can nest up to 15 times; see NMI_BITS. +* nmi_enter() can nest - nesting is tracked in a per-CPU counter. */ #define __nmi_enter () \ do { \ lockdep_off(); \ arch_nmi_enter(); \ - BUG_ON(in_nmi() == NMI_MASK); \ - __preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET); \ +/* Maximum NMI nesting is 15. */ \ +BUG_ON (__this_cpu_read (nmi_nesting) >=15 ); \ +__this_cpu_inc (nmi_nesting); \ +__preempt_count_add (HARDIRQ_OFFSET ); \ +preempt_count_set (preempt_count () |NMI_MASK ); \ } while (0)#define nmi_enter () \
@@ -124,8 +129,12 @@ void irq_exit_rcu(void);#define __nmi_exit () \ do { \ + unsigned int nesting; \ BUG_ON(!in_nmi()); \ - __preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET); \ + __preempt_count_sub(HARDIRQ_OFFSET); \ + nesting = __this_cpu_dec_return(nmi_nesting); \ + if (!nesting) \ + preempt_count_set(preempt_count() & ~NMI_MASK); \ arch_nmi_exit(); \ lockdep_on(); \ } while (0)
include/linux/preempt.h +7 -2
@@ -17,6 +17,8 @@* * - bits 0-7 are the preemption count (max preemption depth: 256) * - bits 8-15 are the softirq count (max # of softirqs: 256) +* - bits 16-19 are the hardirq count (max # of hardirqs: 16) +* - bit 20 is the NMI flag (no nesting count, tracked separately) * * The hardirq count could in theory be the same as the number of * interrupts in the system, but we run all interrupt handlers with
@@ -24,16 +26,19 @@* there are a few palaeontologic drivers which reenable interrupts in * the handler, so we need more than one bit here. * +* NMI nesting depth is tracked in a separate per-CPU variable +* (nmi_nesting) to save bits in preempt_count. +* * PREEMPT_MASK: 0x000000ff * SOFTIRQ_MASK: 0x0000ff00 * HARDIRQ_MASK: 0x000f0000 -* NMI_MASK: 0x00f00000 +* NMI_MASK: 0x00100000 * PREEMPT_NEED_RESCHED: 0x80000000 */ #define PREEMPT_BITS 8#define SOFTIRQ_BITS 8#define HARDIRQ_BITS 4 -#define NMI_BITS 4 +#define NMI_BITS 1#define PREEMPT_SHIFT 0#define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS)
kernel/softirq.c +2 -0
@@ -88,6 +88,8 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);EXPORT_PER_CPU_SYMBOL_GPL (hardirq_context);#endif +DEFINE_PER_CPU(unsigned int, nmi_nesting); +/* * SOFTIRQ_OFFSET usage: *
tools/testing/selftests/bpf/bpf_experimental.h +1 -1
@@ -367,7 +367,7 @@ extern int bpf_cgroup_read_xattr(struct cgroup *cgroup, const char *name__str,#define PREEMPT_BITS 8#define SOFTIRQ_BITS 8#define HARDIRQ_BITS 4 -#define NMI_BITS 4 +#define NMI_BITS 1#define PREEMPT_SHIFT 0#define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS)
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 2/24 preempt: Introduce HARDIRQ_DISABLE_BITS
In reply to Boqun FengFrom: Boqun Feng <[email protected]>
In order to support preempt_disable()-like interrupt disabling, that is, using part of preempt_count() to track interrupt disabling nested level, change the preempt_count() layout to contain 8-bit HARDIRQ_DISABLE count.
--- include/linux/preempt.h | 16 +++++++++++----- tools/testing/selftests/bpf/bpf_experimental.h | 5 ++++- 2 files changed, 15 insertions(+), 6 deletions(-)
include/linux/preempt.h +11 -5
@@ -17,8 +17,9 @@* * - bits 0-7 are the preemption count (max preemption depth: 256) * - bits 8-15 are the softirq count (max # of softirqs: 256) -* - bits 16-19 are the hardirq count (max # of hardirqs: 16) -* - bit 20 is the NMI flag (no nesting count, tracked separately) +* - bits 16-23 are the hardirq disable count (max # of hardirq disable: 256) +* - bits 24-27 are the hardirq count (max # of hardirqs: 16) +* - bit 28 is the NMI flag (no nesting count, tracked separately) * * The hardirq count could in theory be the same as the number of * interrupts in the system, but we run all interrupt handlers with
@@ -31,29 +32,34 @@* * PREEMPT_MASK: 0x000000ff * SOFTIRQ_MASK: 0x0000ff00 -* HARDIRQ_MASK: 0x000f0000 -* NMI_MASK: 0x00100000 +* HARDIRQ_DISABLE_MASK: 0x00ff0000 +* HARDIRQ_MASK: 0x0f000000 +* NMI_MASK: 0x10000000 * PREEMPT_NEED_RESCHED: 0x80000000 */ #define PREEMPT_BITS 8#define SOFTIRQ_BITS 8 +#define HARDIRQ_DISABLE_BITS 8#define HARDIRQ_BITS 4#define NMI_BITS 1#define PREEMPT_SHIFT 0#define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS) -#define HARDIRQ_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS) +#define HARDIRQ_DISABLE_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS) +#define HARDIRQ_SHIFT (HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS)#define NMI_SHIFT (HARDIRQ_SHIFT + HARDIRQ_BITS)#define __IRQ_MASK (x) ((1UL << (x))-1)#define PREEMPT_MASK (__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)#define SOFTIRQ_MASK (__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT) +#define HARDIRQ_DISABLE_MASK (__IRQ_MASK(HARDIRQ_DISABLE_BITS) << HARDIRQ_DISABLE_SHIFT)#define HARDIRQ_MASK (__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)#define NMI_MASK (__IRQ_MASK(NMI_BITS) << NMI_SHIFT)#define PREEMPT_OFFSET (1UL << PREEMPT_SHIFT)#define SOFTIRQ_OFFSET (1UL << SOFTIRQ_SHIFT) +#define HARDIRQ_DISABLE_OFFSET (1UL << HARDIRQ_DISABLE_SHIFT)#define HARDIRQ_OFFSET (1UL << HARDIRQ_SHIFT)#define NMI_OFFSET (1UL << NMI_SHIFT)
tools/testing/selftests/bpf/bpf_experimental.h +4 -1
@@ -366,17 +366,20 @@ extern int bpf_cgroup_read_xattr(struct cgroup *cgroup, const char *name__str,#define PREEMPT_BITS 8#define SOFTIRQ_BITS 8 +#define HARDIRQ_DISABLE_BITS 8#define HARDIRQ_BITS 4#define NMI_BITS 1#define PREEMPT_SHIFT 0#define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS) -#define HARDIRQ_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS) +#define HARDIRQ_DISABLE_SHIFT (SOFTIRQ_SHIFT + SOFTIRQ_BITS) +#define HARDIRQ_SHIFT (HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS)#define NMI_SHIFT (HARDIRQ_SHIFT + HARDIRQ_BITS)#define __IRQ_MASK (x) ((1UL << (x))-1)#define SOFTIRQ_MASK (__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT) +#define HARDIRQ_DISABLE_MASK (__IRQ_MASK(HARDIRQ_DISABLE_BITS) << HARDIRQ_DISABLE_SHIFT)#define HARDIRQ_MASK (__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)#define NMI_MASK (__IRQ_MASK(NMI_BITS) << NMI_SHIFT)
Signature · 1 line
2.50.1 (Apple Git-155)
Peter Zijlstra
maintainerRe: [PATCH 02/24] preempt: Introduce HARDIRQ_DISABLE_BITS
Quoted 10 lines · Boqun Feng
From: Boqun Feng <[email protected]>
In order to support preempt_disable()-like interrupt disabling, that is, using part of
preempt_count()to track interrupt disabling nested level, change thepreempt_count()layout to contain 8-bitHARDIRQ_DISABLEcount.Signed-off-by Lyude Paul <[email protected]>Signed-off-by Boqun Feng <[email protected]>
This is not a valid SoB chain.
Boqun Feng
maintainerPATCH 3/24 preempt: Introduce __preempt_count_{sub, add}_return()
In reply to Boqun FengFrom: Boqun Feng <[email protected]>
In order to use preempt_count() to tracking the interrupt disable nesting level, __preempt_count_{add,sub}_return() are introduced, as their name suggest, these primitives return the new value of the preempt_count() after changing it. The following example shows the usage of it in local_interrupt_disable():
// increase the HARDIRQ_DISABLE bit
new_count = __preempt_count_add_return(HARDIRQ_DISABLE_OFFSET);
// if it's the first-time increment, then disable the interrupt
// at hardware level.
if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET) {
local_irq_save(flags);
raw_cpu_write(local_interrupt_disable_state.flags, flags);
}Having these primitives will avoid a read of preempt_count() after changing preempt_count() on certain architectures.
--- arch/arm64/include/asm/preempt.h | 18 ++++++++++++++++++ arch/s390/include/asm/preempt.h | 10 ++++++++++ arch/x86/include/asm/preempt.h | 10 ++++++++++ include/asm-generic/preempt.h | 14 ++++++++++++++ 4 files changed, 52 insertions(+)
arch/arm64/include/asm/preempt.h +18 -0
@@ -55,6 +55,24 @@ static inline void __preempt_count_sub(int val)WRITE_ONCE (current_thread_info ()->preempt.count, pc); } +static inline int __preempt_count_add_return (int val) +{ +u32 pc =READ_ONCE (current_thread_info ()->preempt.count); + pc += val; +WRITE_ONCE (current_thread_info ()->preempt.count, pc); + +return pc; +} + +static inline int __preempt_count_sub_return (int val) +{ +u32 pc =READ_ONCE (current_thread_info ()->preempt.count); + pc -= val; +WRITE_ONCE (current_thread_info ()->preempt.count, pc); + +return pc; +} +static inline bool __preempt_count_dec_and_test (void ) {struct thread_info *ti =current_thread_info ();
arch/s390/include/asm/preempt.h +10 -0
@@ -139,6 +139,16 @@ static __always_inline bool should_resched(int preempt_offset)return unlikely (READ_ONCE (get_lowcore ()->preempt_count) == preempt_offset); } +static __always_inline int __preempt_count_add_return(int val) +{ +return val +__atomic_add (val, &get_lowcore ()->preempt_count); +} + +static __always_inline int __preempt_count_sub_return(int val) +{ +return __preempt_count_add_return (-val); +} +#define init_task_preempt_count (p) do { } while (0)/* Deferred to CPU bringup time */ #define init_idle_preempt_count (p, cpu) do { } while (0)
arch/x86/include/asm/preempt.h +10 -0
@@ -85,6 +85,16 @@ static __always_inline void __preempt_count_sub(int val)raw_cpu_add_4 (__preempt_count, -val); } +static __always_inline int __preempt_count_add_return(int val) +{ +return raw_cpu_add_return_4 (__preempt_count, val); +} + +static __always_inline int __preempt_count_sub_return(int val) +{ +return raw_cpu_add_return_4 (__preempt_count, -val); +} +/* * Because we keep PREEMPT_NEED_RESCHED set when we do _not_ need to reschedule * a decrement which hits zero means we have no preempt_count and should
include/asm-generic/preempt.h +14 -0
@@ -59,6 +59,20 @@ static __always_inline void __preempt_count_sub(int val) *preempt_count_ptr () -= val; } +static __always_inline int __preempt_count_add_return(int val) +{ + *preempt_count_ptr () += val; + +return *preempt_count_ptr (); +} + +static __always_inline int __preempt_count_sub_return(int val) +{ + *preempt_count_ptr () -= val; + +return *preempt_count_ptr (); +} + static __always_inline bool __preempt_count_dec_and_test(void) {/*
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 4/24 openrisc: Include <linux/cpumask.h> in smp.h
In reply to Boqun FengFrom: Lyude Paul <[email protected]>
While OpenRISC currently doesn't fail to build upstream, it appears that include <asm/smp.h> in the right headers is enough to break that - primarily because OpenRISC's asm/smp.h header doesn't actually provide any definition for struct cpumask. Which means the only reason we aren't failing to build kernel is because we've been lucky enough that every spot including asm/smp.h already has definitions for struct cpumask pulled in.
This became evident when trying to work on a patch series for adding ref-counted interrupt enable/disables to the kernel, where introducing a new interrupt_rc.h header suddenly introduced a build error on OpenRISC:
In file included from include/linux/interrupt_rc.h:17,
from include/linux/spinlock.h:60,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:7,
from include/linux/mm.h:7,
from arch/openrisc/include/asm/pgalloc.h:20,
from arch/openrisc/include/asm/io.h:18,
from include/linux/io.h:12,
from drivers/irqchip/irq-ompic.c:61:
arch/openrisc/include/asm/smp.h:21:59: warning: 'struct cpumask' declared inside parameter list will not be visible outside of this definition or declaration
21 | extern void arch_send_call_function_ipi_mask(const struct cpumask *mask);
| ^~~~~~~
arch/openrisc/include/asm/smp.h:23:54: warning: 'struct cpumask' declared inside parameter list will not be visible outside of this definition or declaration
23 | extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));
| ^~~~~~~
drivers/irqchip/irq-ompic.c: In function 'ompic_of_init':
Quoted 1 line
Quoted 1 line
drivers/irqchip/irq-ompic.c:191:28: error: passing argument 1 of
'set_smp_cross_call' from incompatible pointer type [-Werror=incompatible-pointer-types]
191 | set_smp_cross_call(ompic_raise_softirq);
| ^~~~~~~~~~~~~~~~~~~
| |
| void (*)(const struct cpumask *, unsigned int)
arch/openrisc/include/asm/smp.h:23:32: note: expected 'void (*)(const struct cpumask *, unsigned int)' but argument is of type 'void
(*)(const struct cpumask *, unsigned int)'
23 | extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));
To fix this, let's take an example from the smp.h headers of other architectures (x86, hexagon, arm64, probably more): just include linux/cpumask.h at the top.
--- arch/openrisc/include/asm/smp.h | 2 ++ 1 file changed, 2 insertions(+)
arch/openrisc/include/asm/smp.h +2 -0
@@ -9,6 +9,8 @@#ifndef __ASM_OPENRISC_SMP_H#define __ASM_OPENRISC_SMP_H +#include <linux/cpumask.h> +#include <asm/spr.h> #include <asm/spr_defs.h>
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 5/24 irq & spin_lock: Add counted interrupt disabling/enabling
In reply to Boqun FengFrom: Boqun Feng <[email protected]>
Currently the nested interrupt disabling and enabling is present by _irqsave() and _irqrestore() APIs, which are relatively unsafe, for example:
<interrupts are enabled as beginning> spin_lock_irqsave(l1, flag1); spin_lock_irqsave(l2, flag2); spin_unlock_irqrestore(l1, flags1); <l2 is still held but interrupts are enabled> // accesses to interrupt-disable protect data will cause races.
This is even easier to triggered with guard facilities:
unsigned long flag2;
scoped_guard(spin_lock_irqsave, l1) {
spin_lock_irqsave(l2, flag2);
}
// l2 locked but interrupts are enabled.
spin_unlock_irqrestore(l2, flag2);(Hand-to-hand locking critical sections are not uncommon for a fine-grained lock design)
And because this unsafety, Rust cannot easily wrap the interrupt-disabling locks in a safe API, which complicates the design.
To resolve this, introduce a new set of interrupt disabling APIs:
* local_interrupt_disable();
* local_interrupt_enable();
They work like local_irq_save() and local_irq_restore() except that 1) the outermost local_interrupt_disable() call save the interrupt state into a percpu variable, so that the outermost local_interrupt_enable() can restore the state, and 2) a percpu counter is added to record the nest level of these calls, so that interrupts are not accidentally enabled inside the outermost critical section.
Also add the corresponding spin_lock primitives: spin_lock_irq_disable() and spin_unlock_irq_enable(), as a result, code as follow:
spin_lock_irq_disable(l1); spin_lock_irq_disable(l2); spin_unlock_irq_enable(l1); // Interrupts are still disabled. spin_unlock_irq_enable(l2);
doesn't have the issue that interrupts are accidentally enabled.
This also makes the wrapper of interrupt-disabling locks on Rust easier to design.
--- include/linux/interrupt_rc.h | 67 ++++++++++++++++++++++++++++++++ include/linux/preempt.h | 4 ++ include/linux/spinlock.h | 23 +++++++++++ include/linux/spinlock_api_smp.h | 41 +++++++++++++++++++ include/linux/spinlock_api_up.h | 16 ++++++++ include/linux/spinlock_rt.h | 18 +++++++++ kernel/locking/spinlock.c | 29 ++++++++++++++ kernel/softirq.c | 14 ++++++- 8 files changed, 211 insertions(+), 1 deletion(-) create mode 100644 include/linux/interrupt_rc.h
include/linux/interrupt_rc.h new file +67 -0
@@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* include/linux/interrupt_rc.h - refcounted local processor interrupt +* management. +* +* Since the implementation of this API currently depends on +* local_irq_save()/local_irq_restore(), we split this into it's own header to +* make it easier to include without hitting circular header dependencies. +*/ + +#ifndef __LINUX_INTERRUPT_RC_H +#define __LINUX_INTERRUPT_RC_H + +#include <linux/irqflags.h> +#include <asm/processor.h> +#ifdef CONFIG_SMP +#include <asm/smp.h> +#endif + +/* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}() */ +struct interrupt_disable_state { +unsigned long flags; +}; + +DECLARE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state); + +static inline void local_interrupt_disable (void ) +{ +unsigned long flags; +int new_count; + +WARN_ON_ONCE (in_nmi ()); + + new_count =hardirq_disable_enter (); + +/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */ + +if ((new_count &HARDIRQ_DISABLE_MASK ) ==HARDIRQ_DISABLE_OFFSET ) { +local_irq_save (flags); +raw_cpu_write (local_interrupt_disable_state.flags, flags); + } +} + +static inline void local_interrupt_enable (void ) +{ +int new_count; + + new_count =hardirq_disable_exit (); + +if ((new_count &HARDIRQ_DISABLE_MASK ) ==0 ) { +unsigned long flags; + + flags =raw_cpu_read (local_interrupt_disable_state.flags); +local_irq_restore (flags); +/* +* TODO: re-read preempt count can be avoided, but it needs +* should_resched() taking another parameter as the current +* preempt count +*/ +#ifdef CONFIG_PREEMPTION +if (should_resched (0 )) +__preempt_schedule (); +#endif + } +} + +#endif /* !__LINUX_INTERRUPT_RC_H */
include/linux/preempt.h +4 -0
@@ -151,6 +151,10 @@ static __always_inline unsigned char interrupt_context_level(void)#define in_softirq () (softirq_count())#define in_interrupt () (irq_count()) +#define hardirq_disable_count () ((preempt_count() & HARDIRQ_DISABLE_MASK) >> HARDIRQ_DISABLE_SHIFT) +#define hardirq_disable_enter () __preempt_count_add_return(HARDIRQ_DISABLE_OFFSET) +#define hardirq_disable_exit () __preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) +/* * The preempt_count offset after preempt_disable(); */
include/linux/spinlock.h +23 -0
@@ -57,6 +57,7 @@#include <linux/linkage.h> #include <linux/compiler.h> #include <linux/irqflags.h> +#include <linux/interrupt_rc.h> #include <linux/thread_info.h> #include <linux/stringify.h> #include <linux/bottom_half.h>
@@ -273,9 +274,11 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)#endif #define raw_spin_lock_irq (lock) _raw_spin_lock_irq(lock) +#define raw_spin_lock_irq_disable (lock) _raw_spin_lock_irq_disable(lock)#define raw_spin_lock_bh (lock) _raw_spin_lock_bh(lock)#define raw_spin_unlock (lock) _raw_spin_unlock(lock)#define raw_spin_unlock_irq (lock) _raw_spin_unlock_irq(lock) +#define raw_spin_unlock_irq_enable (lock) _raw_spin_unlock_irq_enable(lock)#define raw_spin_unlock_irqrestore (lock, flags) \ do { \
@@ -290,6 +293,8 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)#define raw_spin_trylock_irqsave (lock, flags) _raw_spin_trylock_irqsave(lock, &(flags)) +#define raw_spin_trylock_irq_disable (lock) _raw_spin_trylock_irq_disable(lock) +#ifndef CONFIG_PREEMPT_RT /* Include rwlock functions for !RT */ #include <linux/rwlock.h>
@@ -372,6 +377,12 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)raw_spin_lock_irq (&lock->rlock); } +static __always_inline void spin_lock_irq_disable(spinlock_t *lock) +__acquires (lock) __no_context_analysis +{ +raw_spin_lock_irq_disable (&lock->rlock); +} +#define spin_lock_irqsave (lock, flags) \ do { \ raw_spin_lock_irqsave(spinlock_check(lock), flags); \
@@ -402,6 +413,12 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)raw_spin_unlock_irq (&lock->rlock); } +static __always_inline void spin_unlock_irq_enable(spinlock_t *lock) +__releases (lock) __no_context_analysis +{ +raw_spin_unlock_irq_enable (&lock->rlock); +} + static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags)__releases (lock) __no_context_analysis {
@@ -427,6 +444,12 @@ static __always_inline bool _spin_trylock_irqsave(spinlock_t *lock, unsigned lon }#define spin_trylock_irqsave (lock, flags) _spin_trylock_irqsave(lock, &(flags)) +static __always_inline int spin_trylock_irq_disable(spinlock_t *lock) +__cond_acquires (true, lock) __no_context_analysis +{ +return raw_spin_trylock_irq_disable (&lock->rlock); +} +/** * spin_is_locked() - Check whether a spinlock is locked. * @lock: Pointer to the spinlock.
include/linux/spinlock_api_smp.h +41 -0
83 lines not on this page · raw message
include/linux/spinlock_api_up.h +16 -0
@@ -42,6 +42,9 @@#define __LOCK_IRQSAVE (lock, flags, ...) \ do { local_irq_save(flags); __LOCK(lock, ##__VA_ARGS__); } while (0) +#define __LOCK_IRQ_DISABLE (lock, ...) \ + do { local_interrupt_disable(); __LOCK(lock, ##__VA_ARGS__); } while (0) +#define ___UNLOCK_ (lock) \ do { __release(lock); (void)(lock); } while (0)
@@ -61,6 +64,10 @@#define __UNLOCK_IRQRESTORE (lock, flags, ...) \ do { local_irq_restore(flags); __UNLOCK(lock, ##__VA_ARGS__); } while (0) +#define __UNLOCK_IRQ_ENABLE (lock, ...) \ + do { __UNLOCK(lock, ##__VA_ARGS__); local_interrupt_enable(); } while (0) + +#define _raw_spin_lock (lock) __LOCK(lock)#define _raw_spin_lock_nested (lock, subclass) __LOCK(lock)#define _raw_read_lock (lock) __LOCK(lock, shared)
@@ -70,6 +77,7 @@#define _raw_read_lock_bh (lock) __LOCK_BH(lock, shared)#define _raw_write_lock_bh (lock) __LOCK_BH(lock)#define _raw_spin_lock_irq (lock) __LOCK_IRQ(lock) +#define _raw_spin_lock_irq_disable (lock) __LOCK_IRQ_DISABLE(lock)#define _raw_read_lock_irq (lock) __LOCK_IRQ(lock, shared)#define _raw_write_lock_irq (lock) __LOCK_IRQ(lock)#define _raw_spin_lock_irqsave (lock, flags) __LOCK_IRQSAVE(lock, flags)
@@ -97,6 +105,13 @@ static __always_inline int _raw_spin_trylock_irq(raw_spinlock_t *lock)return 1 ; } +static __always_inline int _raw_spin_trylock_irq_disable(raw_spinlock_t *lock) +__cond_acquires (true, lock) +{ +__LOCK_IRQ_DISABLE (lock); +return 1 ; +} + static __always_inline int _raw_spin_trylock_irqsave(raw_spinlock_t *lock, unsigned long *flags)__cond_acquires (true, lock) {
@@ -132,6 +147,7 @@ static __always_inline int _raw_write_trylock_irqsave(rwlock_t *lock, unsigned l#define _raw_write_unlock_bh (lock) __UNLOCK_BH(lock)#define _raw_read_unlock_bh (lock) __UNLOCK_BH(lock, shared)#define _raw_spin_unlock_irq (lock) __UNLOCK_IRQ(lock) +#define _raw_spin_unlock_irq_enable (lock) __UNLOCK_IRQ_ENABLE(lock)#define _raw_read_unlock_irq (lock) __UNLOCK_IRQ(lock, shared)#define _raw_write_unlock_irq (lock) __UNLOCK_IRQ(lock)#define _raw_spin_unlock_irqrestore (lock, flags) \
include/linux/spinlock_rt.h +18 -0
@@ -96,6 +96,12 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)rt_spin_lock (lock); } +static __always_inline void spin_lock_irq_disable(spinlock_t *lock) +__acquires (lock) +{ +rt_spin_lock (lock); +} +#define spin_lock_irqsave (lock, flags) \ do { \ typecheck(unsigned long, flags); \
@@ -122,6 +128,12 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)rt_spin_unlock (lock); } +static __always_inline void spin_unlock_irq_enable(spinlock_t *lock) +__releases (lock) +{ +rt_spin_unlock (lock); +} + static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags) __releases(lock)
@@ -131,6 +143,12 @@ static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,#define spin_trylock (lock) rt_spin_trylock(lock) +static __always_inline int spin_trylock_irq_disable(spinlock_t *lock) +__cond_acquires (true, lock) +{ +return rt_spin_trylock (lock); +} +#define spin_trylock_bh (lock) rt_spin_trylock_bh(lock)#define spin_trylock_irq (lock) rt_spin_trylock(lock)
kernel/locking/spinlock.c +29 -0
@@ -129,6 +129,19 @@ static void __lockfunc __raw_##op##_lock_bh(locktype##_t *lock) \*/ BUILD_LOCK_OPS (spin, raw_spinlock, __acquires); +/* No rwlock_t variants for now, so just build this function by hand */ +static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock) +{ +for (;;) { +local_interrupt_disable (); +if (likely (do_raw_spin_trylock (lock))) +break ; +local_interrupt_enable (); + +arch_spin_relax (&lock->raw_lock); + } +} +#ifndef CONFIG_PREEMPT_RT BUILD_LOCK_OPS (read, rwlock, __acquires_shared);BUILD_LOCK_OPS (write, rwlock, __acquires);
@@ -176,6 +189,14 @@ noinline void __lockfunc _raw_spin_lock_irq(raw_spinlock_t *lock)EXPORT_SYMBOL (_raw_spin_lock_irq);#endif +#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ +noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock) +{ +__raw_spin_lock_irq_disable (lock); +} +EXPORT_SYMBOL_GPL (_raw_spin_lock_irq_disable); +#endif +#ifndef CONFIG_INLINE_SPIN_LOCK_BH noinline void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock) {
@@ -208,6 +229,14 @@ noinline void __lockfunc _raw_spin_unlock_irq(raw_spinlock_t *lock)EXPORT_SYMBOL (_raw_spin_unlock_irq);#endif +#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ +noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock) +{ +__raw_spin_unlock_irq_enable (lock); +} +EXPORT_SYMBOL_GPL (_raw_spin_unlock_irq_enable); +#endif +#ifndef CONFIG_INLINE_SPIN_UNLOCK_BH noinline void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock) {
kernel/softirq.c +13 -1
@@ -88,6 +88,9 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);EXPORT_PER_CPU_SYMBOL_GPL (hardirq_context);#endif +DEFINE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state); +EXPORT_PER_CPU_SYMBOL_GPL (local_interrupt_disable_state); + DEFINE_PER_CPU(unsigned int, nmi_nesting);/*
@@ -728,7 +731,16 @@ static inline void __irq_exit_rcu(void)#endif account_hardirq_exit (current);preempt_count_sub (HARDIRQ_OFFSET ); -if (!in_interrupt () &&local_softirq_pending ()) { +/* +* Interrupts may happen between hardirq_disable_enter() and +* local_irq_save() in local_interrupt_disable(), if irq_exit() invokes +* softirq here, we may have a softirq handler calling +* local_interrupt_disable() but it won't disable the irq because +* hardirq disabling count is already 1, hence we need to prevent +* invoking softirq when a local_interrupt_disable() is ongoing. +*/ +if (!in_interrupt () && !hardirq_disable_count () && +local_softirq_pending ()) {/* * If we left hrtimers unarmed, make sure to arm them now, * before enabling interrupts to run SoftIRQ.
Signature · 1 line
2.50.1 (Apple Git-155)
Peter Zijlstra
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
Quoted 50 lines · Boqun Feng
From: Boqun Feng <[email protected]>
Currently the nested interrupt disabling and enabling is present by
_irqsave()and_irqrestore()APIs, which are relatively unsafe, for example:<interrupts are enabled as beginning> spin_lock_irqsave(l1, flag1); spin_lock_irqsave(l2, flag2); spin_unlock_irqrestore(l1, flags1); <l2 is still held but interrupts are enabled> // accesses to interrupt-disable protect data will cause races.This is even easier to triggered with guard facilities:
unsigned long flag2; scoped_guard(spin_lock_irqsave, l1) { spin_lock_irqsave(l2, flag2); } // l2 locked but interrupts are enabled. spin_unlock_irqrestore(l2, flag2);(Hand-to-hand locking critical sections are not uncommon for a fine-grained lock design)
And because this unsafety, Rust cannot easily wrap the interrupt-disabling locks in a safe API, which complicates the design.
To resolve this, introduce a new set of interrupt disabling APIs:
*
local_interrupt_disable();
*local_interrupt_enable();They work like
local_irq_save()andlocal_irq_restore()except that 1) the outermostlocal_interrupt_disable()call save the interrupt state into a percpu variable, so that the outermostlocal_interrupt_enable()can restore the state, and 2) a percpu counter is added to record the nest level of these calls, so that interrupts are not accidentally enabled inside the outermost critical section.Also add the corresponding
spin_lockprimitives:spin_lock_irq_disable()andspin_unlock_irq_enable(), as a result, code as follow:spin_lock_irq_disable(l1); spin_lock_irq_disable(l2); spin_unlock_irq_enable(l1); // Interrupts are still disabled. spin_unlock_irq_enable(l2);
Quoted 8 lines
doesn't have the issue that interrupts are accidentally enabled.
This also makes the wrapper of interrupt-disabling locks on Rust easier to design.
Signed-off-by Lyude Paul <[email protected]>Signed-off-by Boqun Feng <[email protected]>
Same SoB chain issue.
Boqun Feng
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
On Mon, Aug 03, 2026 at 11:19:19AM +0200, Peter Zijlstra wrote:
[...]
Quoted 6 lines
Quoted 4 lines
Signed-off-by Lyude Paul <[email protected]>Signed-off-by Boqun Feng <[email protected]>Same SoB chain issue.
Fixed, thank you!
Regards,
Boqun
Peter Zijlstra
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
In reply to Boqun FengQuoted 17 lines · Boqun Feng
+++
b/include/linux/interrupt_rc.hinclude/linux/interrupt_rc.h new file +15
-0@@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* include/linux/interrupt_rc.h - refcounted local processor interrupt +* management. +* +* Since the implementation of this API currently depends on +* local_irq_save()/local_irq_restore(), we split this into it's own header to +* make it easier to include without hitting circular header dependencies. +*/ + +#ifndef __LINUX_INTERRUPT_RC_H +#define __LINUX_INTERRUPT_RC_H + +#include <linux/irqflags.h> +#include <asm/processor.h>
include/linux/interrupt_rc.h line 25
Quoted 10 lines
include/linux/interrupt_rc.h new file +10
-0+#ifdef CONFIG_SMP +#include <asm/smp.h> +#endif + +/* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}() */ +struct interrupt_disable_state { +unsigned long flags; +}; + +DECLARE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state);
I'm a bit confused by the need for this struct; afaict there aren't any additional members in the rest of the series. So at this point this is pointless wrappery, no?
Having this variable exported is unfortunate.
Peter Zijlstra
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
Quoted 33 lines · Peter Zijlstra
Quoted 17 lines · Boqun Feng
+++
b/include/linux/interrupt_rc.h(quoted diff) +15
-0@@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * include/linux/interrupt_rc.h - refcounted local processor interrupt + * management. + * + * Since the implementation of this API currently depends on + * local_irq_save()/local_irq_restore(), we split this into it's own header to + * make it easier to include without hitting circular header dependencies. + */ + +#ifndef __LINUX_INTERRUPT_RC_H +#define __LINUX_INTERRUPT_RC_H + +#include <linux/irqflags.h> +#include <asm/processor.h>Review line 25
Quoted 10 lines
(quoted diff) +10
-0+#ifdef CONFIG_SMP +#include <asm/smp.h> +#endif + +/* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}() */ +struct interrupt_disable_state { + unsigned long flags; +}; + +DECLARE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state);I'm a bit confused by the need for this struct; afaict there aren't any additional members in the rest of the series. So at this point this is pointless wrappery, no?
Quoted 1 line
Having this variable exported is unfortunate.
One option to get rid of that EXPORT would be something like so I suppose.
---
include/linux/interrupt_rc.h +39 -16
@@ -12,21 +12,52 @@#define __LINUX_INTERRUPT_RC_H#include <linux/irqflags.h> +#include <linux/preempt.h> #include <asm/processor.h> #ifdef CONFIG_SMP #include <asm/smp.h> #endif +#ifndef MODULE /* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}() */ -struct interrupt_disable_state { +DECLARE_PER_CPU(unsigned long, local_interrupt_disable_state); + +static __always_inline void __local_interrupt_disable(void) +{unsigned long flags; -}; -DECLARE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state); +local_irq_save (flags); +raw_cpu_write (local_interrupt_disable_state, flags); +} + +static __always_inline void __local_interrupt_enable(void) +{ +unsigned long flags =raw_cpu_read (local_interrupt_disable_state); +local_irq_restore (flags); +} + +#ifndef INSTANTIATE_EXPORTED_INTERRUPT_DISABLE +static __always_inline void _local_interrupt_disable(void) +{ +__local_interrupt_disable (); +} + +static __always_inline void _local_interrupt_enable(void) +{ +__local_interrupt_enable (); +} +#else +extern void _local_interrupt_disable (void ); +extern void _local_interrupt_enable (void ); +#endif + +#else /* !MODULE */ +extern void _local_interrupt_disable (void ); +extern void _local_interrupt_enable (void ); +#endif /* !MODULE */ static inline void local_interrupt_disable (void ) { -unsigned long flags;unsigned long new_count;WARN_ON_ONCE (in_nmi ());
@@ -35,10 +66,8 @@ static inline void local_interrupt_disab/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */ -if ((new_count &HARDIRQ_DISABLE_MASK ) ==HARDIRQ_DISABLE_OFFSET ) { -local_irq_save (flags); -raw_cpu_write (local_interrupt_disable_state.flags, flags); - } +if ((new_count &HARDIRQ_DISABLE_MASK ) ==HARDIRQ_DISABLE_OFFSET ) +_local_interrupt_disable (); }#ifdef CONFIG_PREEMPTION
@@ -63,14 +92,8 @@ static inline void local_interrupt_enabl new_count =hardirq_disable_exit (); -if ((new_count &HARDIRQ_DISABLE_MASK ) ==0 ) { -unsigned long flags; - - flags =raw_cpu_read (local_interrupt_disable_state.flags); -local_irq_restore (flags); - -local_interrupt_enable_reched (new_count); - } +if ((new_count &HARDIRQ_DISABLE_MASK ) ==0 ) + _local_interrupt_enable(); }#endif /* !__LINUX_INTERRUPT_RC_H */
kernel/softirq.c +12 -2
@@ -9,6 +9,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE #include <linux/export.h> #include <linux/kernel_stat.h> #include <linux/interrupt.h>
@@ -88,8 +89,17 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enablEXPORT_PER_CPU_SYMBOL_GPL (hardirq_context);#endif -DEFINE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state); -EXPORT_PER_CPU_SYMBOL_GPL (local_interrupt_disable_state); +DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state); + +void _local_interrupt_disable (void ) +{ +__local_interrupt_disable (); +} + +void _local_interrupt_enable (void ) +{ +__local_interrupt_enable (); +}#ifndef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS /*
Boqun Feng
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
Quoted 81 lines · Peter Zijlstra
Quoted 33 lines · Peter Zijlstra
Quoted 17 lines · Boqun Feng
+++
b/include/linux/interrupt_rc.h(quoted diff) +15
-0@@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * include/linux/interrupt_rc.h - refcounted local processor interrupt + * management. + * + * Since the implementation of this API currently depends on + * local_irq_save()/local_irq_restore(), we split this into it's own header to + * make it easier to include without hitting circular header dependencies. + */ + +#ifndef __LINUX_INTERRUPT_RC_H +#define __LINUX_INTERRUPT_RC_H + +#include <linux/irqflags.h> +#include <asm/processor.h>Review line 25
Quoted 10 lines
(quoted diff) +10
-0+#ifdef CONFIG_SMP +#include <asm/smp.h> +#endif + +/* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}() */ +struct interrupt_disable_state { + unsigned long flags; +}; + +DECLARE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state);I'm a bit confused by the need for this struct; afaict there aren't any additional members in the rest of the series. So at this point this is pointless wrappery, no?
Quoted 1 line
Having this variable exported is unfortunate.
One option to get rid of that EXPORT would be something like so I suppose.
---
include/linux/interrupt_rc.h +24
-3@@ -12,21 +12,52 @@#define __LINUX_INTERRUPT_RC_H#include <linux/irqflags.h> +#include <linux/preempt.h> #include <asm/processor.h> #ifdef CONFIG_SMP #include <asm/smp.h> #endif +#ifndef MODULE /* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}() */ -struct interrupt_disable_state { +DECLARE_PER_CPU(unsigned long, local_interrupt_disable_state); + +static __always_inline void __local_interrupt_disable(void) +{ unsigned long flags; -}; -DECLARE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state); + local_irq_save(flags); + raw_cpu_write(local_interrupt_disable_state, flags); +} + +static __always_inline void __local_interrupt_enable(void) +{ + unsigned long flags = raw_cpu_read(local_interrupt_disable_state); + local_irq_restore(flags); +} + +#ifndef INSTANTIATE_EXPORTED_INTERRUPT_DISABLE +static __always_inline void _local_interrupt_disable(void) +{ + __local_interrupt_disable(); +} + +static __always_inline void _local_interrupt_enable(void) +{
include/linux/interrupt_rc.h line 56
Quoted 10 lines
include/linux/interrupt_rc.h +10
-0+ __local_interrupt_enable(); +} +#else +extern void _local_interrupt_disable (void ); +extern void _local_interrupt_enable (void ); +#endif + +#else /* !MODULE */ +extern void _local_interrupt_disable (void ); +extern void _local_interrupt_enable (void );
I think the "extern" keywords are not necessary and ..
Quoted 56 lines
include/linux/interrupt_rc.h +5
-13+#endif /* !MODULE */ static inline void local_interrupt_disable (void ) { -unsigned long flags;unsigned long new_count;WARN_ON_ONCE (in_nmi ());@@ -35,10 +66,8 @@ static inline void local_interrupt_disab/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */ -if ((new_count &HARDIRQ_DISABLE_MASK ) ==HARDIRQ_DISABLE_OFFSET ) { -local_irq_save (flags); -raw_cpu_write (local_interrupt_disable_state.flags, flags); - } +if ((new_count &HARDIRQ_DISABLE_MASK ) ==HARDIRQ_DISABLE_OFFSET ) +_local_interrupt_disable (); }#ifdef CONFIG_PREEMPTION @@ -63,14 +92,8 @@ static inline void local_interrupt_enabl new_count =hardirq_disable_exit (); -if ((new_count &HARDIRQ_DISABLE_MASK ) ==0 ) { -unsigned long flags; - - flags =raw_cpu_read (local_interrupt_disable_state.flags); -local_irq_restore (flags); - -local_interrupt_enable_reched (new_count); - } +if ((new_count &HARDIRQ_DISABLE_MASK ) ==0 ) + _local_interrupt_enable(); }#endif /* !__LINUX_INTERRUPT_RC_H */ kernel/softirq.c +3
-2@@ -9,6 +9,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE #include <linux/export.h> #include <linux/kernel_stat.h> #include <linux/interrupt.h> @@ -88,8 +89,17 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enablEXPORT_PER_CPU_SYMBOL_GPL (hardirq_context);#endif -DEFINE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state); -EXPORT_PER_CPU_SYMBOL_GPL (local_interrupt_disable_state); +DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state); +
kernel/softirq.c EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabl · line 103
Quoted 10 lines
kernel/softirq.c +9
-0+void _local_interrupt_disable (void ) +{ +__local_interrupt_disable (); +} + +void _local_interrupt_enable (void ) +{ +__local_interrupt_enable (); +}
We need to EXPORT_SYMBOL_GPL() for these two functions.
Right?
Regards,
Boqun
Quoted 2 lines
kernel/softirq.c +0
-0#ifndef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS /*
Peter Zijlstra
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
Quoted 33 lines · Boqun Feng
Quoted 33 lines
include/linux/interrupt_rc.h +27
-3+#ifndef MODULE /* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}() */ -struct interrupt_disable_state { +DECLARE_PER_CPU(unsigned long, local_interrupt_disable_state); + +static __always_inline void __local_interrupt_disable(void) +{unsigned long flags; -}; -DECLARE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state); +local_irq_save (flags); +raw_cpu_write (local_interrupt_disable_state, flags); +} + +static __always_inline void __local_interrupt_enable(void) +{ +unsigned long flags =raw_cpu_read (local_interrupt_disable_state); +local_irq_restore (flags); +} + +#ifndef INSTANTIATE_EXPORTED_INTERRUPT_DISABLE +static __always_inline void _local_interrupt_disable(void) +{ +__local_interrupt_disable (); +} + +static __always_inline void _local_interrupt_enable(void) +{ +__local_interrupt_enable (); +} +#else +extern void _local_interrupt_disable (void );
Quoted 8 lines
include/linux/interrupt_rc.h line 56
Quoted 6 lines
include/linux/interrupt_rc.h +6
-0+extern void _local_interrupt_enable (void ); +#endif + +#else /* !MODULE */ +extern void _local_interrupt_disable (void ); +extern void _local_interrupt_enable (void );I think the "extern" keywords are not necessary and ..
Probably, but I like them (and I know that other people hate on then). To me it makes it clear the definition is not here and should be sought elsewhere.
The extern really is needed for variables, but I also use it with functions for the same.
Quoted 34 lines
Quoted 21 lines
include/linux/interrupt_rc.h +1
-0+#endif /* !MODULE */ kernel/softirq.c +3
-2@@ -9,6 +9,7 @@#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE #include <linux/export.h> #include <linux/kernel_stat.h> #include <linux/interrupt.h> @@ -88,8 +89,17 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enablEXPORT_PER_CPU_SYMBOL_GPL (hardirq_context);#endif -DEFINE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state); -EXPORT_PER_CPU_SYMBOL_GPL (local_interrupt_disable_state); +DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state); +kernel/softirq.c EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabl · line 103
Quoted 10 lines
kernel/softirq.c +9
-0+void _local_interrupt_disable (void ) +{ +__local_interrupt_disable (); +} + +void _local_interrupt_enable (void ) +{ +__local_interrupt_enable (); +}We need to
EXPORT_SYMBOL_GPL()for these two functions.
Quoted 1 line
Right?
Quite so.
Peter Zijlstra
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
Quoted 21 lines · Peter Zijlstra
Quoted 21 lines
Quoted 8 lines
kernel/softirq.c +2
-2@@ -88,8 +89,17 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enablEXPORT_PER_CPU_SYMBOL_GPL (hardirq_context);#endif -DEFINE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state); -EXPORT_PER_CPU_SYMBOL_GPL (local_interrupt_disable_state); +DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state); +kernel/softirq.c EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabl · line 103
Quoted 10 lines
kernel/softirq.c +9
-0+void _local_interrupt_disable (void ) +{ +__local_interrupt_disable (); +} + +void _local_interrupt_enable (void ) +{ +__local_interrupt_enable (); +}We need to
EXPORT_SYMBOL_GPL()for these two functions.
Quoted 3 lines
Quoted 1 line
Right?
Quite so.
Although, I suspect people will come and complain that this should be !GPL, it being something 'trivial' and all that nonsense, but we'll get there when we get there I suppose.
Boqun Feng
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
Quoted 23 lines · Peter Zijlstra
Quoted 21 lines · Peter Zijlstra
Quoted 21 lines
Quoted 8 lines
kernel/softirq.c +2
-2@@ -88,8 +89,17 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enablEXPORT_PER_CPU_SYMBOL_GPL (hardirq_context);#endif -DEFINE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state); -EXPORT_PER_CPU_SYMBOL_GPL (local_interrupt_disable_state); +DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state); +kernel/softirq.c EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabl · line 103
Quoted 10 lines
kernel/softirq.c +9
-0+void _local_interrupt_disable (void ) +{ +__local_interrupt_disable (); +} + +void _local_interrupt_enable (void ) +{ +__local_interrupt_enable (); +}We need to
EXPORT_SYMBOL_GPL()for these two functions.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Right?
Quite so.
Although, I suspect people will come and complain that this should be !GPL, it being something 'trivial' and all that nonsense, but we'll get there when we get there I suppose.
I haven't thought about the GPL part, but after a look, I think we could do EXPORT_SYMBOL() (similar __local_bh_disable_ip) from the get-go.
Regards,
Boqun
Boqun Feng
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
In reply to Peter ZijlstraQuoted 2 lines · Peter Zijlstra
On Mon, Aug 03, 2026 at 09:41:02PM -0700, Boqun Feng wrote:
[...]
Quoted 11 lines
Quoted 10 lines
include/linux/interrupt_rc.h line 56
Quoted 8 lines
include/linux/interrupt_rc.h +8
-0+#else +extern void _local_interrupt_disable (void ); +extern void _local_interrupt_enable (void ); +#endif + +#else /* !MODULE */ +extern void _local_interrupt_disable (void ); +extern void _local_interrupt_enable (void );I think the "extern" keywords are not necessary and ..
Quoted 7 lines
Probably, but I like them (and I know that other people hate on then). To me it makes it clear the definition is not here and should be sought elsewhere.
The extern really is needed for variables, but I also use it with functions for the same.
Alright I will keep them, thanks!
Regards,
Boqun
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
+#endif /* !MODULE */
[...]
Peter Zijlstra
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
In reply to Boqun FengQuoted 10 lines · Boqun Feng
include/linux/interrupt_rc.h new file +4
-0@@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* include/linux/interrupt_rc.h - refcounted local processor interrupt +* management.
include/linux/interrupt_rc.h line 14
Quoted 10 lines
include/linux/interrupt_rc.h new file +10
-0+* +* Since the implementation of this API currently depends on +* local_irq_save()/local_irq_restore(), we split this into it's own header to +* make it easier to include without hitting circular header dependencies. +*/ + +#ifndef __LINUX_INTERRUPT_RC_H +#define __LINUX_INTERRUPT_RC_H + +#include <linux/irqflags.h>
My lsp/clangd insists on:
#include <linux/preempt.h>
Quoted 53 lines
include/linux/interrupt_rc.h new file +53
-0+#include <asm/processor.h> +#ifdef CONFIG_SMP +#include <asm/smp.h> +#endif + +/* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}() */ +struct interrupt_disable_state { +unsigned long flags; +}; + +DECLARE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state); + +static inline void local_interrupt_disable (void ) +{ +unsigned long flags; +int new_count; + +WARN_ON_ONCE (in_nmi ()); + + new_count =hardirq_disable_enter (); + +/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */ + +if ((new_count &HARDIRQ_DISABLE_MASK ) ==HARDIRQ_DISABLE_OFFSET ) { +local_irq_save (flags); +raw_cpu_write (local_interrupt_disable_state.flags, flags); + } +} + +static inline void local_interrupt_enable (void ) +{ +int new_count; + + new_count =hardirq_disable_exit (); + +if ((new_count &HARDIRQ_DISABLE_MASK ) ==0 ) { +unsigned long flags; + + flags =raw_cpu_read (local_interrupt_disable_state.flags); +local_irq_restore (flags); +/* +* TODO: re-read preempt count can be avoided, but it needs +* should_resched() taking another parameter as the current +* preempt count +*/ +#ifdef CONFIG_PREEMPTION +if (should_resched (0 )) +__preempt_schedule (); +#endif + } +} + +#endif /* !__LINUX_INTERRUPT_RC_H */
Peter Zijlstra
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
In reply to Boqun FengQuoted 26 lines · Boqun Feng
include/linux/interrupt_rc.h new file +26
-0+static inline void local_interrupt_disable (void ) +{ +unsigned long flags; +int new_count; + +WARN_ON_ONCE (in_nmi ()); + + new_count =hardirq_disable_enter (); + +/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */ + +if ((new_count &HARDIRQ_DISABLE_MASK ) ==HARDIRQ_DISABLE_OFFSET ) { +local_irq_save (flags); +raw_cpu_write (local_interrupt_disable_state.flags, flags); + } +} + +static inline void local_interrupt_enable (void ) +{ +int new_count; + + new_count =hardirq_disable_exit (); + +if ((new_count &HARDIRQ_DISABLE_MASK ) ==0 ) { +unsigned long flags; +
include/linux/interrupt_rc.h line 62
Quoted 10 lines
include/linux/interrupt_rc.h new file +10
-0+ flags =raw_cpu_read (local_interrupt_disable_state.flags); +local_irq_restore (flags); +/* +* TODO: re-read preempt count can be avoided, but it needs +* should_resched() taking another parameter as the current +* preempt count +*/ +#ifdef CONFIG_PREEMPTION +if (should_resched (0 )) +__preempt_schedule ();
I'm confused by this one. This should not be needed. If there was a preemption while IRQs were disabled, there should be a pending IPI.
Notably local_irq_restore() also doesn't do this, since otherwise you would not have needed to add this here, since you already call that.
Quoted 3 lines
include/linux/interrupt_rc.h new file +3
-0+#endif + } +}
Boqun Feng
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
Quoted 42 lines · Peter Zijlstra
Quoted 26 lines · Boqun Feng
include/linux/interrupt_rc.h new file +26
-0+static inline void local_interrupt_disable (void ) +{ +unsigned long flags; +int new_count; + +WARN_ON_ONCE (in_nmi ()); + + new_count =hardirq_disable_enter (); + +/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */ + +if ((new_count &HARDIRQ_DISABLE_MASK ) ==HARDIRQ_DISABLE_OFFSET ) { +local_irq_save (flags); +raw_cpu_write (local_interrupt_disable_state.flags, flags); + } +} + +static inline void local_interrupt_enable (void ) +{ +int new_count; + + new_count =hardirq_disable_exit (); + +if ((new_count &HARDIRQ_DISABLE_MASK ) ==0 ) { +unsigned long flags; +include/linux/interrupt_rc.h line 62
Quoted 10 lines
include/linux/interrupt_rc.h new file +10
-0+ flags =raw_cpu_read (local_interrupt_disable_state.flags); +local_irq_restore (flags); +/* +* TODO: re-read preempt count can be avoided, but it needs +* should_resched() taking another parameter as the current +* preempt count +*/ +#ifdef CONFIG_PREEMPTION +if (should_resched (0 )) +__preempt_schedule ();I'm confused by this one. This should not be needed. If there was a preemption while IRQs were disabled, there should be a pending IPI.
Quoted 3 lines
Notably
local_irq_restore()also doesn't do this, since otherwise you would not have needed to add this here, since you already call that.
I should probably have made a comment on this instead of keeping it in my head ;-)
Note that since we use preempt_count to record irq disabling, then in the spinlock version i.e. spin_{,un}lock_irq_{disable,enable}(), we don't need (and don't have) preempt_disable/enable() any more. So we need to rely on this for the following case:
spin_lock_irq_disable(); <set NEED_RESCHED for the current task> spin_unlock_irq_enable(): do_raw_spin_unlock(lock); local_interrupt_enable(); // <- need this to trigger a // reschedule, because no // preempt_enable here.
Make sense?
Regards,
Boqun
Quoted 3 lines
Quoted 3 lines
include/linux/interrupt_rc.h new file +3
-0+#endif + } +}
Peter Zijlstra
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
Quoted 63 lines · Boqun Feng
Quoted 42 lines · Peter Zijlstra
Quoted 26 lines · Boqun Feng
include/linux/interrupt_rc.h new file +26
-0+static inline void local_interrupt_disable (void ) +{ +unsigned long flags; +int new_count; + +WARN_ON_ONCE (in_nmi ()); + + new_count =hardirq_disable_enter (); + +/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */ + +if ((new_count &HARDIRQ_DISABLE_MASK ) ==HARDIRQ_DISABLE_OFFSET ) { +local_irq_save (flags); +raw_cpu_write (local_interrupt_disable_state.flags, flags); + } +} + +static inline void local_interrupt_enable (void ) +{ +int new_count; + + new_count =hardirq_disable_exit (); + +if ((new_count &HARDIRQ_DISABLE_MASK ) ==0 ) { +unsigned long flags; +include/linux/interrupt_rc.h line 62
Quoted 10 lines
include/linux/interrupt_rc.h new file +10
-0+ flags =raw_cpu_read (local_interrupt_disable_state.flags); +local_irq_restore (flags); +/* +* TODO: re-read preempt count can be avoided, but it needs +* should_resched() taking another parameter as the current +* preempt count +*/ +#ifdef CONFIG_PREEMPTION +if (should_resched (0 )) +__preempt_schedule ();I'm confused by this one. This should not be needed. If there was a preemption while IRQs were disabled, there should be a pending IPI.
Quoted 3 lines
Notably
local_irq_restore()also doesn't do this, since otherwise you would not have needed to add this here, since you already call that.I should probably have made a comment on this instead of keeping it in my head ;-)
Note that since we use
preempt_countto record irq disabling, then in the spinlock version i.e. spin_{,un}lock_irq_{disable,enable}(), we don't need (and don't have) preempt_disable/enable() any more. So we need to rely on this for the following case:spin_lock_irq_disable(); <set NEED_RESCHED for the current task> spin_unlock_irq_enable(): do_raw_spin_unlock(lock); local_interrupt_enable(); // <- need this to trigger a // reschedule, because no // preempt_enable here.
Quoted 1 line
Make sense?
No; for one, you cannot elide the preempt_disable/enable() from this thing.
It has always been valid to do:
raw_spin_lock_irq(lock); ... raw_spin_unlock(lock); ... local_irq_enable();
And since raw_spin_unlock() does have the preemption thing, raw_spin_lock_irq() must too. And the very same argument is still valid if you now write:
raw_spin_lock_irq_disable(lock); ... raw_spin_unlock(lock); ... local_interrupt_enable();
Secondly, nothing should 'set' NEED_RESCHED without also immediately causing a preemption.
Peter Zijlstra
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
Quoted 15 lines · Peter Zijlstra
No; for one, you cannot elide the preempt_disable/enable() from this thing.
It has always been valid to do:
raw_spin_lock_irq(lock); ... raw_spin_unlock(lock); ... local_irq_enable();And since
raw_spin_unlock()does have the preemption thing,raw_spin_lock_irq()must too. And the very same argument is still valid if you now write:
Quoted 8 lines
raw_spin_lock_irq_disable(lock); ... raw_spin_unlock(lock); ... local_interrupt_enable();Secondly, nothing should 'set'
NEED_RESCHEDwithout also immediately causing a preemption.
Notably, scheduler_ipi() is the one that folds TIF_NEED_RESCHED into PREEMPT_NEED_RESCHED for remote reschedules.
Other than that, set_preempt_need_resched() must always be used from hardirq context or inside the schedule() loop on current. When used from hardirq context, the interrupt-return path will handle the reschedule, when used inside the schedule() loop, the loop will.
No other fiddling of NEED_RESCHED should occur.
Peter Zijlstra
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
Quoted 23 lines · Peter Zijlstra
Quoted 21 lines · Peter Zijlstra
No; for one, you cannot elide the preempt_disable/enable() from this thing.
It has always been valid to do:
raw_spin_lock_irq(lock); ... raw_spin_unlock(lock); ... local_irq_enable();And since
raw_spin_unlock()does have the preemption thing,raw_spin_lock_irq()must too. And the very same argument is still valid if you now write:raw_spin_lock_irq_disable(lock); ... raw_spin_unlock(lock); ... local_interrupt_enable();
Quoted 5 lines
Quoted 2 lines
Secondly, nothing should 'set'
NEED_RESCHEDwithout also immediately causing a preemption.Notably,
scheduler_ipi()is the one that foldsTIF_NEED_RESCHEDintoPREEMPT_NEED_RESCHEDfor remote reschedules.
(and note that this is exactly the from hardirq context on current usage as per the below...)
Quoted 6 lines
Other than that,
set_preempt_need_resched()must always be used from hardirq context or inside theschedule()loop on current. When used from hardirq context, the interrupt-return path will handle the reschedule, when used inside theschedule()loop, the loop will.No other fiddling of
NEED_RESCHEDshould occur.
Boqun Feng
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
In reply to Peter ZijlstraQuoted 71 lines · Peter Zijlstra
Quoted 63 lines · Boqun Feng
Quoted 42 lines · Peter Zijlstra
Quoted 26 lines · Boqun Feng
include/linux/interrupt_rc.h new file +26
-0+static inline void local_interrupt_disable (void ) +{ +unsigned long flags; +int new_count; + +WARN_ON_ONCE (in_nmi ()); + + new_count =hardirq_disable_enter (); + +/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */ + +if ((new_count &HARDIRQ_DISABLE_MASK ) ==HARDIRQ_DISABLE_OFFSET ) { +local_irq_save (flags); +raw_cpu_write (local_interrupt_disable_state.flags, flags); + } +} + +static inline void local_interrupt_enable (void ) +{ +int new_count; + + new_count =hardirq_disable_exit (); + +if ((new_count &HARDIRQ_DISABLE_MASK ) ==0 ) { +unsigned long flags; +include/linux/interrupt_rc.h line 62
Quoted 10 lines
include/linux/interrupt_rc.h new file +10
-0+ flags =raw_cpu_read (local_interrupt_disable_state.flags); +local_irq_restore (flags); +/* +* TODO: re-read preempt count can be avoided, but it needs +* should_resched() taking another parameter as the current +* preempt count +*/ +#ifdef CONFIG_PREEMPTION +if (should_resched (0 )) +__preempt_schedule ();I'm confused by this one. This should not be needed. If there was a preemption while IRQs were disabled, there should be a pending IPI.
Quoted 3 lines
Notably
local_irq_restore()also doesn't do this, since otherwise you would not have needed to add this here, since you already call that.I should probably have made a comment on this instead of keeping it in my head ;-)
Note that since we use
preempt_countto record irq disabling, then in the spinlock version i.e. spin_{,un}lock_irq_{disable,enable}(), we don't need (and don't have) preempt_disable/enable() any more. So we need to rely on this for the following case:spin_lock_irq_disable(); <set NEED_RESCHED for the current task> spin_unlock_irq_enable(): do_raw_spin_unlock(lock); local_interrupt_enable(); // <- need this to trigger a // reschedule, because no // preempt_enable here.Quoted 1 line
Make sense?
No; for one, you cannot elide the preempt_disable/enable() from this thing.
It has always been valid to do:
Quoted 7 lines
raw_spin_lock_irq(lock); ... raw_spin_unlock(lock); ... local_irq_enable();And since
raw_spin_unlock()does have the preemption thing,
Good point!
However, IIRC, the elision was trying to optimize two preempt_count accesses (one for irq disable, one for preempt disable) into one in raw_spin_lock_irq_disable() or raw_spin_unlock_irq_enable(). We may still want to do it in the future, if that's the case, we will still have should_resched() check in raw_spin_unlock_irq_enable()?
Quoted 11 lines
raw_spin_lock_irq()must too. And the very same argument is still valid if you now write:raw_spin_lock_irq_disable(lock); ... raw_spin_unlock(lock); ... local_interrupt_enable();Secondly, nothing should 'set'
NEED_RESCHEDwithout also immediately causing a preemption.
Thanks for the information.
Regards,
Boqun
Peter Zijlstra
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
Quoted 13 lines · Boqun Feng
Quoted 11 lines · Peter Zijlstra
No; for one, you cannot elide the preempt_disable/enable() from this thing.
It has always been valid to do:
raw_spin_lock_irq(lock); ... raw_spin_unlock(lock); ... local_irq_enable();
Quoted 7 lines
Quoted 1 line
And since
raw_spin_unlock()does have the preemption thing,Good point!
However, IIRC, the elision was trying to optimize two
preempt_countaccesses (one for irq disable, one for preempt disable) into one inraw_spin_lock_irq_disable()orraw_spin_unlock_irq_enable().
It is relatively easy to fold the increments in raw_spin_lock_irq_disable(), the enable (see below) is going to be a little more tricky.
Quoted 3 lines
We may
still want to do it in the future, if that's the case, we will still haveshould_resched()check inraw_spin_unlock_irq_enable()?
So if you have raw_spin_unlock_irq_enable() *and* you've managed to fold the preempt_count() accesses of hardirq_disable_exit() and preempt_enable(), then yes. Consider:
preempt_disable(); <IRQ happens, set TIF_NEED_RESCHED and folds PREEMPT_NEED_RESCHED> raw_spin_lock_irq_disable(lock); preempt_enable(); // can't reschedule ... raw_spin_unlock_irq_enable(lock); // drops to 0 without an interrupt pending
So the folding of hardirq_disable_exit() and preempt_enable() needs to take care of *both* special cases: 1) the local_irq_restore() and 2) calling __preempt_schedule() when 0.
Re: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
In reply to Boqun FengQuoted 8 lines · Boqun Feng
include/linux/interrupt_rc.h new file +2
-0@@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/*
include/linux/interrupt_rc.h line 12
Quoted 10 lines
include/linux/interrupt_rc.h new file +10
-0+* include/linux/interrupt_rc.h - refcounted local processor interrupt +* management. +* +* Since the implementation of this API currently depends on +* local_irq_save()/local_irq_restore(), we split this into it's own header to +* make it easier to include without hitting circular header dependencies. +*/ + +#ifndef __LINUX_INTERRUPT_RC_H +#define __LINUX_INTERRUPT_RC_H
That's not the customary placement for header guards, please try to follow existing patterns as much as possible.
include/linux/interrupt_rc.h line 20
Quoted 1 line
include/linux/interrupt_rc.h new file +1
-0+/* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}() */
That's not the standard spelling of per-CPU.
include/linux/interrupt_rc.h line 23
Quoted 3 lines
include/linux/interrupt_rc.h new file +3
-0+struct interrupt_disable_state { +unsigned long flags; +};
Unless this structure is extended in the same series - and it isn't AFICT, why this layer of obfuscation of type and purpose?
include/linux/interrupt_rc.h line 63
Quoted 9 lines
include/linux/interrupt_rc.h new file +9
-0+/* +* TODO: re-read preempt count can be avoided, but it needs +* should_resched() taking another parameter as the current +* preempt count +*/ +#ifdef CONFIG_PREEMPTION +if (should_resched (0 )) +__preempt_schedule (); +#endif
Let's not add TODOs and call it an upstream-ready patch ...
preempt_count_sub(HARDIRQ_OFFSET);
include/linux/interrupt_rc.h
Quoted 8 lines
include/linux/interrupt_rc.h new file +7
-1-if (!in_interrupt () &&local_softirq_pending ()) { +/* +* Interrupts may happen between hardirq_disable_enter() and +* local_irq_save() in local_interrupt_disable(), if irq_exit() invokes +* softirq here, we may have a softirq handler calling +* local_interrupt_disable() but it won't disable the irq because +* hardirq disabling count is already 1, hence we need to prevent +* invoking softirq when a local_interrupt_disable() is ongoing.
Non-standard spelling of 'IRQ'.
include/linux/interrupt_rc.h
Quoted 6 lines
include/linux/interrupt_rc.h new file +3
-0+*/ +if (!in_interrupt () && !hardirq_disable_count () && +local_softirq_pending ()) {/* * If we left hrtimers unarmed, make sure to arm them now, * before enabling interrupts to run SoftIRQ.
Non-standard spelling of 'softirq' here.
There's also new spelling errors in some of the comments introduced by this patch.
This is a very low quality patch, it should never have been included and declared 'ready' for upstream. It's not even close!
Thanks,
Ingo
Boqun Feng
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
Quoted 16 lines · Ingo Molnar
Quoted 13 lines · Boqun Feng
include/linux/interrupt_rc.h new file +7
-0@@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* include/linux/interrupt_rc.h - refcounted local processor interrupt +* management. +* +* Since the implementation of this API currently depends on +* local_irq_save()/local_irq_restore(), we split this into it's own header to
Quoted 9 lines
include/linux/interrupt_rc.h line 12
Quoted 5 lines
include/linux/interrupt_rc.h new file +5
-0+* make it easier to include without hitting circular header dependencies. +*/ + +#ifndef __LINUX_INTERRUPT_RC_H +#define __LINUX_INTERRUPT_RC_HThat's not the customary placement for header guards, please try to follow existing patterns as much as possible.
Fixed locally, I follow what spinlock.h does.
Quoted 11 lines
include/linux/interrupt_rc.h line 20
Quoted 1 line
include/linux/interrupt_rc.h new file +1
-0+/* Per-cpu interrupt disabling state for local_interrupt_{disable,enable}() */ That's not the standard spelling of per-CPU.
include/linux/interrupt_rc.h line 23
Quoted 3 lines
include/linux/interrupt_rc.h new file +3
-0+struct interrupt_disable_state { +unsigned long flags; +};Unless this structure is extended in the same series - and it isn't AFICT, why this layer of obfuscation of type and purpose?
Will remove this layer in the next version.
Quoted 12 lines
include/linux/interrupt_rc.h line 63
Quoted 9 lines
include/linux/interrupt_rc.h new file +9
-0+/* +* TODO: re-read preempt count can be avoided, but it needs +* should_resched() taking another parameter as the current +* preempt count +*/ +#ifdef CONFIG_PREEMPTION +if (should_resched (0 )) +__preempt_schedule (); +#endif Let's not add TODOs and call it an upstream-ready patch ...
Per discussion with Peter [1], this part is not needed, so the TODO will be removed.
Quoted 12 lines
preempt_count_sub(HARDIRQ_OFFSET);include/linux/interrupt_rc.h
Quoted 8 lines
include/linux/interrupt_rc.h new file +7
-1-if (!in_interrupt () &&local_softirq_pending ()) { +/* +* Interrupts may happen between hardirq_disable_enter() and +* local_irq_save() in local_interrupt_disable(), if irq_exit() invokes +* softirq here, we may have a softirq handler calling +* local_interrupt_disable() but it won't disable the irq because +* hardirq disabling count is already 1, hence we need to prevent +* invoking softirq when a local_interrupt_disable() is ongoing. Non-standard spelling of 'IRQ'.
Quoted 9 lines
include/linux/interrupt_rc.h
Quoted 6 lines
include/linux/interrupt_rc.h new file +3
-0+*/ +if (!in_interrupt () && !hardirq_disable_count () && +local_softirq_pending ()) {/* * If we left hrtimers unarmed, make sure to arm them now, * before enabling interrupts to run SoftIRQ. Non-standard spelling of 'softirq' here.
Fixed.
Quoted 3 lines
There's also new spelling errors in some of the comments introduced by this patch.
Will try to find and fix these as well.
[1]: https://lore.kernel.org/rust-for-linux/[email protected]/
Regards,
Boqun
Quoted 6 lines
This is a very low quality patch, it should never have been included and declared 'ready' for upstream. It's not even close!
Thanks,
Ingo
Boqun Feng
maintainerRe: [PATCH 05/24] irq & spin_lock: Add counted interrupt disabling/enabling
On Mon, Aug 03, 2026 at 12:47:17PM -0700, Boqun Feng wrote:
[...]
Quoted 3 lines
Quoted 3 lines
This is a very low quality patch, it should never have been included and declared 'ready' for upstream. It's not even close!
And just to be clear about the 'ready' decision: I did do a thorough test (with real workloads that I have), and tried to fix any format issues that I could find out. The latest version of the series can be found at:
https://lore.kernel.org/rust-for-linux/[email protected]/
Of course, there are still a few things I missed and it's sometimes hard for one to find opportunities to simplify their own code (e.g. the struct interrupt_disable_state you and Peter pointed out). Thanks for taking a look! I will post a new version for review.
Regards,
Boqun
Quoted 3 lines
Quoted 3 lines
Thanks,
Ingo
Boqun Feng
maintainerPATCH 6/24 irq: Add KUnit test for refcounted interrupt enable/disable
In reply to Boqun FengFrom: Lyude Paul <[email protected]>
While making changes to the refcounted interrupt patch series, at some point on my local branch I broke something and ended up writing some kunit tests for testing refcounted interrupts as a result. So, let's include these tests now that we have refcounted interrupts.
--- kernel/irq/Makefile | 1 + kernel/irq/refcount_interrupt_test.c | 109 +++++++++++++++++++++++++++ 2 files changed, 110 insertions(+) create mode 100644 kernel/irq/refcount_interrupt_test.c
kernel/irq/Makefile +1 -0
@@ -16,3 +16,4 @@ obj-$(CONFIG_SMP) += affinity.o obj-$(CONFIG_GENERIC_IRQ_DEBUGFS) += debugfs.o obj-$(CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR) += matrix.o obj-$(CONFIG_IRQ_KUNIT_TEST) += irq_test.o +obj-$(CONFIG_KUNIT) += refcount_interrupt_test.o
kernel/irq/refcount_interrupt_test.c new file +109 -0
109 lines not on this page · raw message
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 7/24 locking: Switch to _irq_{disable,enable}() variants in cleanup guards
In reply to Boqun FengFrom: Boqun Feng <[email protected]>
The semantics of various irq disabling guards match what *_irq_{disable,enable}() provide, i.e. the interrupt disabling is properly nested, therefore it's OK to switch to use *_irq_{disable,enable}() primitives.
--- include/linux/spinlock.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
include/linux/spinlock.h +6 -6
@@ -572,12 +572,12 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, __acquires(_T), __releases(*(raw#define class_raw_spinlock_nested_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, _T)DEFINE_LOCK_GUARD_1 (raw_spinlock_irq, raw_spinlock_t, -raw_spin_lock_irq (_T->lock), - raw_spin_unlock_irq(_T->lock)) +raw_spin_lock_irq_disable (_T->lock), + raw_spin_unlock_irq_enable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, __acquires(_T), __releases(*(raw_spinlock_t **)_T))#define class_raw_spinlock_irq_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, _T) -DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq(_T->lock)) +DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq_disable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))#define class_raw_spinlock_irq_try_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, _T)
@@ -618,13 +618,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_try, __acquires(_T), __releases(*(spinlock_t#define class_spinlock_try_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_try, _T)DEFINE_LOCK_GUARD_1 (spinlock_irq, spinlock_t, -spin_lock_irq (_T->lock), - spin_unlock_irq(_T->lock)) +spin_lock_irq_disable (_T->lock), + spin_unlock_irq_enable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq, __acquires(_T), __releases(*(spinlock_t **)_T))#define class_spinlock_irq_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq, _T)DEFINE_LOCK_GUARD_1_COND (spinlock_irq, _try, - spin_trylock_irq(_T->lock)) + spin_trylock_irq_disable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq_try, __acquires(_T), __releases(*(spinlock_t **)_T))#define class_spinlock_irq_try_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq_try, _T)
Signature · 1 line
2.50.1 (Apple Git-155)
Peter Zijlstra
maintainerRe: [PATCH 07/24] locking: Switch to _irq_{disable,enable}() variants in cleanup guards
Quoted 39 lines · Boqun Feng
From: Boqun Feng <[email protected]>
The semantics of various irq disabling guards match what *_irq_{disable,enable}() provide, i.e. the interrupt disabling is properly nested, therefore it's OK to switch to use *_irq_{disable,enable}() primitives.
Signed-off-by Boqun Feng <[email protected]>--- include/linux/spinlock.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)include/linux/spinlock.h +3
-5@@ -572,12 +572,12 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, __acquires(_T), __releases(*(raw#define class_raw_spinlock_nested_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, _T)DEFINE_LOCK_GUARD_1 (raw_spinlock_irq, raw_spinlock_t, -raw_spin_lock_irq (_T->lock), - raw_spin_unlock_irq(_T->lock)) +raw_spin_lock_irq_disable (_T->lock), + raw_spin_unlock_irq_enable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, __acquires(_T), __releases(*(raw_spinlock_t **)_T))#define class_raw_spinlock_irq_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, _T) -DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq(_T->lock)) +DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq_disable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))#define class_raw_spinlock_irq_try_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, _T)@@ -618,13 +618,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_try, __acquires(_T), __releases(*(spinlock_t#define class_spinlock_try_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_try, _T) DEFINE_LOCK_GUARD_1(spinlock_irq, spinlock_t, -spin_lock_irq (_T->lock), -spin_unlock_irq (_T->lock))
include/linux/spinlock.h DECLARE_LOCK_GUARD_1_ATTRS(spinlock_try, __acquires(_T), __releases(*(spinlock_t · line 629
Quoted 10 lines
include/linux/spinlock.h +3
-1+ spin_lock_irq_disable(_T->lock), + spin_unlock_irq_enable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq, __acquires(_T), __releases(*(spinlock_t **)_T)) #define class_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq, _T) DEFINE_LOCK_GUARD_1_COND(spinlock_irq, _try, - spin_trylock_irq(_T->lock)) + spin_trylock_irq_disable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq_try, __acquires(_T), __releases(*(spinlock_t **)_T))#define class_spinlock_irq_try_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq_try, _T)
What about the _irqsave() guards?
Is the goal to replace _irqsave guard usage with _irq and then remove the _irqsave guards?
If so, this should probably we mentioned somewhere.
Boqun Feng
maintainerRe: [PATCH 07/24] locking: Switch to _irq_{disable,enable}() variants in cleanup guards
Quoted 53 lines · Peter Zijlstra
Quoted 39 lines · Boqun Feng
From: Boqun Feng <[email protected]>
The semantics of various irq disabling guards match what *_irq_{disable,enable}() provide, i.e. the interrupt disabling is properly nested, therefore it's OK to switch to use *_irq_{disable,enable}() primitives.
Signed-off-by Boqun Feng <[email protected]>--- include/linux/spinlock.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)include/linux/spinlock.h +3
-5@@ -572,12 +572,12 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, __acquires(_T), __releases(*(raw#define class_raw_spinlock_nested_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, _T)DEFINE_LOCK_GUARD_1 (raw_spinlock_irq, raw_spinlock_t, -raw_spin_lock_irq (_T->lock), - raw_spin_unlock_irq(_T->lock)) +raw_spin_lock_irq_disable (_T->lock), + raw_spin_unlock_irq_enable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, __acquires(_T), __releases(*(raw_spinlock_t **)_T))#define class_raw_spinlock_irq_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, _T) -DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq(_T->lock)) +DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq_disable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))#define class_raw_spinlock_irq_try_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, _T)@@ -618,13 +618,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_try, __acquires(_T), __releases(*(spinlock_t#define class_spinlock_try_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_try, _T) DEFINE_LOCK_GUARD_1(spinlock_irq, spinlock_t, -spin_lock_irq (_T->lock), -spin_unlock_irq (_T->lock))include/linux/spinlock.h DECLARE_LOCK_GUARD_1_ATTRS(spinlock_try, __acquires(_T), __releases(*(spinlock_t · line 629
Quoted 10 lines
include/linux/spinlock.h +3
-1+ spin_lock_irq_disable(_T->lock), + spin_unlock_irq_enable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq, __acquires(_T), __releases(*(spinlock_t **)_T)) #define class_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq, _T) DEFINE_LOCK_GUARD_1_COND(spinlock_irq, _try, - spin_trylock_irq(_T->lock)) + spin_trylock_irq_disable(_T->lock)) DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq_try, __acquires(_T), __releases(*(spinlock_t **)_T))#define class_spinlock_irq_try_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq_try, _T)What about the
_irqsave()guards?
Quoted 3 lines
Is the goal to replace
_irqsaveguard usage with_irqand then remove the_irqsaveguards?
Yes, that's the goal. I had that in previous version. However in 1abbecd1d2d2 ("sched/fair: Convert cfs bandwidth throttling to use guards"), we have a user that explicitly plays with the .flags in guard. Lyude also spotted that too. We could adjust that user to the new API (Lyude already has the diff for that) but I decided to simply drop that part for now given the current size of the changes. But if you think it's a must for merge this, I will add it.
Quoted 1 line
If so, this should probably we mentioned somewhere.
I will at least add this part. Thanks!
Regards,
Boqun
Peter Zijlstra
maintainerRe: [PATCH 07/24] locking: Switch to _irq_{disable,enable}() variants in cleanup guards
Quoted 6 lines · Boqun Feng
Quoted 5 lines
What about the
_irqsave()guards?Is the goal to replace
_irqsaveguard usage with_irqand then remove the_irqsaveguards?
Quoted 7 lines
Yes, that's the goal. I had that in previous version. However in
1abbecd1d2d2("sched/fair: Convert cfs bandwidth throttling to use guards"), we have a user that explicitly plays with the .flags in guard. Lyude also spotted that too. We could adjust that user to the new API (Lyude already has the diff for that) but I decided to simply drop that part for now given the current size of the changes. But if you think it's a must for merge this, I will add it.
Well, but this new stuff should be able to clean that up, no?
---
kernel/sched/fair.c +6 -6
@@ -7248,7 +7248,7 @@ static bool distribute_cfs_runtime(struc* period the timer is deactivated until scheduling resumes; cfs_b->idle is * used to track this state. */ -static int do_sched_cfs_period_timer (struct cfs_bandwidth *cfs_b,int overrun,unsigned long flags) +static int do_sched_cfs_period_timer (struct cfs_bandwidth *cfs_b,int overrun) __must_hold(&cfs_b->lock) {int throttled;
@@ -7283,10 +7283,10 @@ static int do_sched_cfs_period_timer(str* This check is repeated as we release cfs_b->lock while we unthrottle. */ while (throttled && cfs_b->runtime >0 ) { -raw_spin_unlock_irqrestore (&cfs_b->lock, flags); +raw_spin_unlock_irq_enable (&cfs_b->lock);/* we can't nest cfs_b->lock while distributing bandwidth */ throttled =distribute_cfs_runtime (cfs_b); -raw_spin_lock_irqsave (&cfs_b->lock, flags); +raw_spin_lock_irq_disable (&cfs_b->lock); }/*
@@ -7394,7 +7394,7 @@ static __always_inline void return_cfs_rstatic void do_sched_cfs_slack_timer (struct cfs_bandwidth *cfs_b) {/* confirm we're still not at a refresh boundary */ - scoped_guard(raw_spinlock_irqsave, &cfs_b->lock) { + scoped_guard(raw_spinlock_irq, &cfs_b->lock) {u64 runtime =0 , slice =sched_cfs_bandwidth_slice (); cfs_b->slack_started = false;
@@ -7479,14 +7479,14 @@ static enum hrtimer_restart sched_cfs_peint idle =0 ;int count =0 ; -CLASS (raw_spinlock_irqsave, cfsb_guard)(&cfs_b->lock); +guard (raw_spinlock_irq)(&cfs_b->lock);for (;;) { overrun =hrtimer_forward_now (timer, cfs_b->period);if (!overrun)break ; - idle =do_sched_cfs_period_timer (cfs_b, overrun, cfsb_guard.flags); + idle =do_sched_cfs_period_timer (cfs_b, overrun);if (++count >3 ) {u64 new, old =ktime_to_ns (cfs_b->period);
Boqun Feng
maintainerRe: [PATCH 07/24] locking: Switch to _irq_{disable,enable}() variants in cleanup guards
Quoted 16 lines · Peter Zijlstra
Quoted 6 lines · Boqun Feng
Quoted 5 lines
What about the
_irqsave()guards?Is the goal to replace
_irqsaveguard usage with_irqand then remove the_irqsaveguards?Quoted 7 lines
Yes, that's the goal. I had that in previous version. However in
1abbecd1d2d2("sched/fair: Convert cfs bandwidth throttling to use guards"), we have a user that explicitly plays with the .flags in guard. Lyude also spotted that too. We could adjust that user to the new API (Lyude already has the diff for that) but I decided to simply drop that part for now given the current size of the changes. But if you think it's a must for merge this, I will add it.
Quoted 2 lines
Well, but this new stuff should be able to clean that up, no?
Yeah... I admit that the decision was made based on how much time I could spend on this cycle, rather than "what the best we can do here".
Thank you. I fold the following.
Regards,
Boqun
Quoted 51 lines
---
kernel/sched/fair.c +6
-6@@ -7248,7 +7248,7 @@ static bool distribute_cfs_runtime(struc* period the timer is deactivated until scheduling resumes; cfs_b->idle is * used to track this state. */ -static int do_sched_cfs_period_timer (struct cfs_bandwidth *cfs_b,int overrun,unsigned long flags) +static int do_sched_cfs_period_timer (struct cfs_bandwidth *cfs_b,int overrun) __must_hold(&cfs_b->lock) {int throttled;@@ -7283,10 +7283,10 @@ static int do_sched_cfs_period_timer(str* This check is repeated as we release cfs_b->lock while we unthrottle. */ while (throttled && cfs_b->runtime >0 ) { -raw_spin_unlock_irqrestore (&cfs_b->lock, flags); +raw_spin_unlock_irq_enable (&cfs_b->lock);/* we can't nest cfs_b->lock while distributing bandwidth */ throttled =distribute_cfs_runtime (cfs_b); -raw_spin_lock_irqsave (&cfs_b->lock, flags); +raw_spin_lock_irq_disable (&cfs_b->lock); }/* @@ -7394,7 +7394,7 @@ static __always_inline void return_cfs_rstatic void do_sched_cfs_slack_timer (struct cfs_bandwidth *cfs_b) {/* confirm we're still not at a refresh boundary */ - scoped_guard(raw_spinlock_irqsave, &cfs_b->lock) { + scoped_guard(raw_spinlock_irq, &cfs_b->lock) {u64 runtime =0 , slice =sched_cfs_bandwidth_slice (); cfs_b->slack_started = false;@@ -7479,14 +7479,14 @@ static enum hrtimer_restart sched_cfs_peint idle =0 ;int count =0 ; -CLASS (raw_spinlock_irqsave, cfsb_guard)(&cfs_b->lock); +guard (raw_spinlock_irq)(&cfs_b->lock);for (;;) { overrun =hrtimer_forward_now (timer, cfs_b->period);if (!overrun)break ; - idle =do_sched_cfs_period_timer (cfs_b, overrun, cfsb_guard.flags); + idle =do_sched_cfs_period_timer (cfs_b, overrun);if (++count >3 ) {u64 new, old =ktime_to_ns (cfs_b->period);
Boqun Feng
maintainerPATCH 8/24 sched: Remove the unused preempt_offset parameter of __cant_sleep()
In reply to Boqun FengThe preempt_offset is always 0 in all the callsites of __cant_sleep(), hence remove it. It also allows us to clear the code a bit by stopping using a "preempt_count() > .." comparison.
--- include/linux/kernel.h | 4 ++-- kernel/sched/core.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
include/linux/kernel.h +2 -2
@@ -72,7 +72,7 @@ extern int dynamic_might_resched(void);#ifdef CONFIG_DEBUG_ATOMIC_SLEEP extern void __might_resched (const char *file,int line,unsigned int offsets);extern void __might_sleep (const char *file,int line); -extern void __cant_sleep (const char *file,int line,int preempt_offset); +extern void __cant_sleep (const char *file,int line);extern void __cant_migrate (const char *file,int line);/**
@@ -95,7 +95,7 @@ extern void __cant_migrate(const char *file, int line);* this macro will print a stack trace if it is executed with preemption enabled */ # define cant_sleep () \ - do { __cant_sleep(__FILE__, __LINE__, 0); } while (0) + do { __cant_sleep(__FILE__, __LINE__); } while (0)# define sched_annotate_sleep () (current->task_state_change = 0)/**
kernel/sched/core.c +2 -2
@@ -9199,7 +9199,7 @@ void __might_resched(const char *file, int line, unsigned int offsets) }EXPORT_SYMBOL (__might_resched); -void __cant_sleep (const char *file,int line,int preempt_offset) +void __cant_sleep (const char *file,int line) {static unsigned long prev_jiffy;
@@ -9209,7 +9209,7 @@ void __cant_sleep(const char *file, int line, int preempt_offset)if (!IS_ENABLED (CONFIG_PREEMPT_COUNT ))return ; -if (preempt_count () > preempt_offset) +if (preempt_count ())return ;if (time_before (jiffies, prev_jiffy +HZ ) && prev_jiffy)
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 9/24 sched: Avoid signed comparison of preempt_count() in __cant_migrate()
In reply to Boqun FengCurrently preempt_count() is always a non-negative int on all archs (PREEMPT_NEED_RESCHED archs will mask out the MSB when return preempt_count()), hence the checking in __cant_migrate() is in fact just checking whether preempt_count() is 0 or not. In a future change, we are going to use all the 32 bits of preempt_count(), which would make negative int values possible from preempt_count(). Therefore convert the "> 0" comparison into a zero checking to prepare for the future change. No functional changes are intended.
--- kernel/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -9241,7 +9241,7 @@ void __cant_migrate(const char *file, int line)if (!IS_ENABLED (CONFIG_PREEMPT_COUNT ))return ; -if (preempt_count () >0 ) +if (preempt_count ())return ;if (time_before (jiffies, prev_jiffy +HZ ) && prev_jiffy)
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 10/24 preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS
In reply to Boqun FengWith the changes that enable preempt count to tracking irq disabling nesting, we don't have enough bits in 32bit preempt count implementation, as a result we move NMI nesting bits out of the 32bit preempt count. However on the architectures that can support 64bit preempt count implementation, we can keep the NMI nesting bits in the 32bit preempt count and avoid maintaining NMI nesting bits out of the same cache line.
Therefore HAS_SEPARATE_PREEMPT_RESCHED_BITS is introduced to allow architectures to select this. Note that under this kconfig, preempt count is maintained in a 64bit word however preempt_count() still remains as an int because all the effective bits still fit in (previously we mask out NEED_RESCHED bit in preempt_count()). This should make no functional changes for existing preempt_count() users.
Enable this for x86_64 along with the introduction of the Kconfig.
[boqun: Undo the __preempt_count_{add,sub}() optimization in 32bit preempt count since it may introduce {over,under}flow]
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/preempt.h | 55 +++++++++++++++++++++++----------- arch/x86/kernel/cpu/common.c | 2 +- include/linux/hardirq.h | 48 +++++++++++++++++++++-------- include/linux/preempt.h | 20 +++++++------ kernel/Kconfig.preempt | 4 +++ kernel/sched/core.c | 12 ++++++-- kernel/softirq.c | 6 ++++ lib/locking-selftest.c | 2 +- 9 files changed, 107 insertions(+), 43 deletions(-)
arch/x86/Kconfig +1 -0
@@ -326,6 +326,7 @@ config X86 select USER_STACKTRACE_SUPPORT select HAVE_ARCH_KCSAN if X86_64 select PROC_PID_ARCH_STATUS if PROC_FS + select HAS_SEPARATE_PREEMPT_RESCHED_BITS if X86_64 && PREEMPT_COUNT select HAVE_ARCH_NODE_DEV_GROUP if X86_SGX select FUNCTION_ALIGNMENT_16B if X86_64 || X86_ALIGNMENT_16 select FUNCTION_ALIGNMENT_4B
arch/x86/include/asm/preempt.h +38 -17
123 lines not on this page · raw message
arch/x86/kernel/cpu/common.c +1 -1
@@ -2236,7 +2236,7 @@ DEFINE_PER_CPU_CACHE_HOT(struct task_struct *, current_task) = &init_task;EXPORT_PER_CPU_SYMBOL (current_task);EXPORT_PER_CPU_SYMBOL (const_current_task); -DEFINE_PER_CPU_CACHE_HOT (int, __preempt_count) =INIT_PREEMPT_COUNT ; +DEFINE_PER_CPU_CACHE_HOT(unsigned long, __preempt_count) = INIT_PREEMPT_COUNT;EXPORT_PER_CPU_SYMBOL (__preempt_count); DEFINE_PER_CPU_CACHE_HOT(unsigned long, cpu_current_top_of_stack) = TOP_OF_INIT_STACK;
include/linux/hardirq.h +35 -13
79 lines not on this page · raw message
include/linux/preempt.h +11 -9
@@ -30,18 +30,20 @@* NMI nesting depth is tracked in a separate per-CPU variable * (nmi_nesting) to save bits in preempt_count. * -* PREEMPT_MASK: 0x000000ff -* SOFTIRQ_MASK: 0x0000ff00 -* HARDIRQ_DISABLE_MASK: 0x00ff0000 -* HARDIRQ_MASK: 0x0f000000 -* NMI_MASK: 0x10000000 -* PREEMPT_NEED_RESCHED: 0x80000000 +* 32bit HAS_SEPARATE_PREEMPT_RESCHED_BITS +* +* PREEMPT_MASK: 0x000000ff 0x00000000000000ff +* SOFTIRQ_MASK: 0x0000ff00 0x000000000000ff00 +* HARDIRQ_DISABLE_MASK: 0x00ff0000 0x0000000000ff0000 +* HARDIRQ_MASK: 0x0f000000 0x000000000f000000 +* NMI_MASK: 0x10000000 0x00000000f0000000 +* PREEMPT_NEED_RESCHED: 0x80000000 0x8000000000000000 */ #define PREEMPT_BITS 8#define SOFTIRQ_BITS 8#define HARDIRQ_DISABLE_BITS 8#define HARDIRQ_BITS 4 -#define NMI_BITS 1 +#define NMI_BITS (1 + 3*IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS))#define PREEMPT_SHIFT 0#define SOFTIRQ_SHIFT (PREEMPT_SHIFT + PREEMPT_BITS)
@@ -116,8 +118,8 @@ static __always_inline unsigned char interrupt_context_level(void)* preempt_count() is commonly implemented with READ_ONCE(). */ -#define nmi_count () (preempt_count() & NMI_MASK) -#define hardirq_count () (preempt_count() & HARDIRQ_MASK) +#define nmi_count () (preempt_count() & NMI_MASK) +#define hardirq_count () (preempt_count() & HARDIRQ_MASK)#ifdef CONFIG_PREEMPT_RT # define softirq_count () (current->softirq_disable_cnt & SOFTIRQ_MASK)# define irq_count () ((preempt_count() & (NMI_MASK | HARDIRQ_MASK)) | softirq_count())
kernel/Kconfig.preempt +4 -0
@@ -122,6 +122,10 @@ config PREEMPT_RT_NEEDS_BH_LOCK config PREEMPT_COUNT bool +config HAS_SEPARATE_PREEMPT_RESCHED_BITS + bool + depends on PREEMPT_COUNT && 64BIT + config PREEMPTION bool select PREEMPT_COUNT
kernel/sched/core.c +10 -2
@@ -5973,8 +5973,13 @@ void preempt_count_add(int val)#ifdef CONFIG_DEBUG_PREEMPT /* * Underflow? +* +* Cannot detect underflow based on the current preempt_count() value +* if using HAS_SEPARATE_PREEMPT_RESCHED_BITS because preempt count takes all 32 +* bits. */ -if (DEBUG_LOCKS_WARN_ON ((preempt_count () <0 ))) +if (!IS_ENABLED (CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS ) && +DEBUG_LOCKS_WARN_ON ((preempt_count () <0 )))return ;#endif __preempt_count_add (val);
@@ -6006,7 +6011,10 @@ void preempt_count_sub(int val)/* * Underflow? */ -if (DEBUG_LOCKS_WARN_ON (val >preempt_count ())) +unsigned int uval = val; +unsigned int pc =preempt_count (); + +if (DEBUG_LOCKS_WARN_ON (pc - uval > pc))return ;/* * Is the spinlock portion underflowing?
kernel/softirq.c +6 -0
@@ -91,7 +91,13 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context); DEFINE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state);EXPORT_PER_CPU_SYMBOL_GPL (local_interrupt_disable_state); +#ifndef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS +/* +* Any 32bit architecture that still cares about performance should +* probably ensure this is near preempt_count. +*/ DEFINE_PER_CPU(unsigned int, nmi_nesting); +#endif /* * SOFTIRQ_OFFSET usage:
lib/locking-selftest.c +1 -1
@@ -1429,7 +1429,7 @@ static int unexpected_testcase_failures;static void dotest (void (*testcase_fn)(void ),int expected,int lockclass_mask) { -int saved_preempt_count =preempt_count (); +long saved_preempt_count =preempt_count ();#ifdef CONFIG_PREEMPT_RT int saved_mgd_count = current->migration_disabled;int saved_rcu_count = current->rcu_read_lock_nesting;
Signature · 1 line
2.50.1 (Apple Git-155)
Peter Zijlstra
maintainerRe: [PATCH 10/24] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS
Quoted 12 lines · Boqun Feng
arch/x86/include/asm/preempt.h +1
-1@@ -24,18 +34,26 @@ DECLARE_PER_CPU_CACHE_HOT(int, __preempt_count);*/ static __always_inline int preempt_count(void) { -return raw_cpu_read_4 (__preempt_count) & ~PREEMPT_NEED_RESCHED ; +return __pc_op (read, __preempt_count) & ~PREEMPT_NEED_RESCHED ; }
arch/x86/include/asm/preempt.h DECLARE_PER_CPU_CACHE_HOT(int, __preempt_count); · line 47
Quoted 10 lines
arch/x86/include/asm/preempt.h +8
-1-static __always_inline void preempt_count_set(int pc) +/* +* unsigned long preempt count parameter works for both 32bit and 64bit cases: +* +* - For 32bit, "int" (the return of preempt_count()) and "unsigned long" have +* the same size. +* - For 64bit, the effective bits of a preempt count sits in 32bit, and we +* reserve the NEED_RESCHED bit from the old count. +*/
The 64bit comment doesn't really make sense to me.
Quoted 27 lines
arch/x86/include/asm/preempt.h +4
-3+static __always_inline void preempt_count_set(unsigned long pc) { -int old, new; +unsigned long old, new; - old =raw_cpu_read_4 (__preempt_count); + old =__pc_op (read, __preempt_count);do { new = (old &PREEMPT_NEED_RESCHED ) | (pc & ~PREEMPT_NEED_RESCHED ); - }while (!raw_cpu_try_cmpxchg_4 (__preempt_count, &old, new)); + }while (!__pc_op (try_cmpxchg, __preempt_count, &old, new)); }include/linux/preempt.h +0
-5@@ -30,18 +30,20 @@* NMI nesting depth is tracked in a separate per-CPU variable * (nmi_nesting) to save bits in preempt_count. * -* PREEMPT_MASK: 0x000000ff -* SOFTIRQ_MASK: 0x0000ff00 -* HARDIRQ_DISABLE_MASK: 0x00ff0000 -* HARDIRQ_MASK: 0x0f000000 -* NMI_MASK: 0x10000000
include/linux/preempt.h line 41
Quoted 10 lines
include/linux/preempt.h +8
-1-* PREEMPT_NEED_RESCHED: 0x80000000 +* 32bit HAS_SEPARATE_PREEMPT_RESCHED_BITS +* +* PREEMPT_MASK: 0x000000ff 0x00000000000000ff +* SOFTIRQ_MASK: 0x0000ff00 0x000000000000ff00 +* HARDIRQ_DISABLE_MASK: 0x00ff0000 0x0000000000ff0000 +* HARDIRQ_MASK: 0x0f000000 0x000000000f000000 +* NMI_MASK: 0x10000000 0x00000000f0000000 +* PREEMPT_NEED_RESCHED: 0x80000000 0x8000000000000000 */
Perhaps add a comment about how HAS_SEPARATE_PREEMPT_RESCHED_BITS really is about having PREEMPT_NEED_RESCHED in its own word, rather than preempt_count() being 64bit.
Because as presented it is very easy to confuse these two options. Ideally it would explain the LOAD-STORE issue with NEED_RESCHED and point to ARM64 or something.
Boqun Feng
maintainerRe: [PATCH 10/24] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS
Quoted 15 lines · Peter Zijlstra
Quoted 13 lines · Boqun Feng
arch/x86/include/asm/preempt.h +1
-1@@ -24,18 +34,26 @@ DECLARE_PER_CPU_CACHE_HOT(int, __preempt_count);*/ static __always_inline int preempt_count(void) { -return raw_cpu_read_4 (__preempt_count) & ~PREEMPT_NEED_RESCHED ; +return __pc_op (read, __preempt_count) & ~PREEMPT_NEED_RESCHED ; }
Quoted 12 lines
arch/x86/include/asm/preempt.h DECLARE_PER_CPU_CACHE_HOT(int, __preempt_count); · line 47
Quoted 9 lines
arch/x86/include/asm/preempt.h +8
-1-static __always_inline void preempt_count_set(int pc) +/* +* unsigned long preempt count parameter works for both 32bit and 64bit cases: +* +* - For 32bit, "int" (the return of preempt_count()) and "unsigned long" have +* the same size. +* - For 64bit, the effective bits of a preempt count sits in 32bit, and we +* reserve the NEED_RESCHED bit from the old count. +*/ The 64bit comment doesn't really make sense to me.
Ah, I meant "preserve" instead of "reserve"..
+/*
+ * unsigned long preempt count parameter works for both 32bit and 64bit cases:
+ *
+ * - For 32bit, "int" (the return of preempt_count()) and "unsigned long" have
+ * the same size.
+ * - For 64bit, the effective bits of a preempt count sits in 32bit, and we
+ * preserve the NEED_RESCHED bit from the old count.
+ */
It means we won't change the NEED_RESCHED bit regardless that pc has it set or not.
Quoted 42 lines
Quoted 27 lines
arch/x86/include/asm/preempt.h +4
-3+static __always_inline void preempt_count_set(unsigned long pc) { -int old, new; +unsigned long old, new; - old =raw_cpu_read_4 (__preempt_count); + old =__pc_op (read, __preempt_count);do { new = (old &PREEMPT_NEED_RESCHED ) | (pc & ~PREEMPT_NEED_RESCHED ); - }while (!raw_cpu_try_cmpxchg_4 (__preempt_count, &old, new)); + }while (!__pc_op (try_cmpxchg, __preempt_count, &old, new)); }include/linux/preempt.h +0
-5@@ -30,18 +30,20 @@* NMI nesting depth is tracked in a separate per-CPU variable * (nmi_nesting) to save bits in preempt_count. * -* PREEMPT_MASK: 0x000000ff -* SOFTIRQ_MASK: 0x0000ff00 -* HARDIRQ_DISABLE_MASK: 0x00ff0000 -* HARDIRQ_MASK: 0x0f000000 -* NMI_MASK: 0x10000000 include/linux/preempt.h line 41
Quoted 10 lines
include/linux/preempt.h +8
-1-* PREEMPT_NEED_RESCHED: 0x80000000 +* 32bit HAS_SEPARATE_PREEMPT_RESCHED_BITS +* +* PREEMPT_MASK: 0x000000ff 0x00000000000000ff +* SOFTIRQ_MASK: 0x0000ff00 0x000000000000ff00 +* HARDIRQ_DISABLE_MASK: 0x00ff0000 0x0000000000ff0000 +* HARDIRQ_MASK: 0x0f000000 0x000000000f000000 +* NMI_MASK: 0x10000000 0x00000000f0000000 +* PREEMPT_NEED_RESCHED: 0x80000000 0x8000000000000000 */ Perhaps add a comment about how
HAS_SEPARATE_PREEMPT_RESCHED_BITSreally is about havingPREEMPT_NEED_RESCHEDin its own word, rather thanpreempt_count()being 64bit.
Quoted 3 lines
Because as presented it is very easy to confuse these two options. Ideally it would explain the LOAD-STORE issue with
NEED_RESCHEDand point to ARM64 or something.
How about organizing the comments as following:
+ *
+ * PREEMPT_MASK: 0x000000ff
+ * SOFTIRQ_MASK: 0x0000ff00
+ * HARDIRQ_DISABLE_MASK: 0x00ff0000
+ * HARDIRQ_MASK: 0x0f000000
+ *
+ * Depending on HAS_SEPARATE_PREEMPT_RESCHED_BITS, NEED_RESCHED bit
+ * is put in a separate 32bits.
+ *
+ * HAS_SEPARATE_PREEMPT_RESCHED_BITS=n:
+ *
+ * NMI_MASK: 0x10000000
+ * PREEMPT_NEED_RESCHED: 0x80000000
+ *
+ * HAS_SEPARATE_PREEMPT_RESCHED_BITS=y:
+ *
+ * (NMI_MASK can use all the 4 bits)
+ *
+ * NMI_MASK: 0xf0000000
Thoughts?
Regards,
Boqun
Peter Zijlstra
maintainerRe: [PATCH 10/24] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS
Quoted 9 lines · Boqun Feng
Quoted 9 lines
Quoted 8 lines
+/*
+ * unsigned long preempt count parameter works for both 32bit and 64bit cases:
+ *
+ * - For 32bit, "int" (the return ofpreempt_count()) and "unsigned long" have
+ * the same size.
+ * - For 64bit, the effective bits of a preempt count sits in 32bit, and we
+ * reserve theNEED_RESCHEDbit from the old count.
+ */
Quoted 4 lines
Quoted 2 lines
The 64bit comment doesn't really make sense to me.
Ah, I meant "preserve" instead of "reserve"..
Ah, yes, that makes sense.
Quoted 54 lines
Quoted 28 lines
Quoted 13 lines
include/linux/preempt.h +0
-5@@ -30,18 +30,20 @@* NMI nesting depth is tracked in a separate per-CPU variable * (nmi_nesting) to save bits in preempt_count. * -* PREEMPT_MASK: 0x000000ff -* SOFTIRQ_MASK: 0x0000ff00 -* HARDIRQ_DISABLE_MASK: 0x00ff0000 -* HARDIRQ_MASK: 0x0f000000 -* NMI_MASK: 0x10000000 include/linux/preempt.h line 41
Quoted 10 lines
include/linux/preempt.h +8
-1-* PREEMPT_NEED_RESCHED: 0x80000000 +* 32bit HAS_SEPARATE_PREEMPT_RESCHED_BITS +* +* PREEMPT_MASK: 0x000000ff 0x00000000000000ff +* SOFTIRQ_MASK: 0x0000ff00 0x000000000000ff00 +* HARDIRQ_DISABLE_MASK: 0x00ff0000 0x0000000000ff0000 +* HARDIRQ_MASK: 0x0f000000 0x000000000f000000 +* NMI_MASK: 0x10000000 0x00000000f0000000 +* PREEMPT_NEED_RESCHED: 0x80000000 0x8000000000000000 */ Perhaps add a comment about how
HAS_SEPARATE_PREEMPT_RESCHED_BITSreally is about havingPREEMPT_NEED_RESCHEDin its own word, rather thanpreempt_count()being 64bit.Quoted 3 lines
Because as presented it is very easy to confuse these two options. Ideally it would explain the LOAD-STORE issue with
NEED_RESCHEDand point to ARM64 or something.How about organizing the comments as following:
+ *
+ *PREEMPT_MASK:0x000000ff
+ *SOFTIRQ_MASK:0x0000ff00
+ *HARDIRQ_DISABLE_MASK:0x00ff0000
+ *HARDIRQ_MASK:0x0f000000
+ *
+ * Depending onHAS_SEPARATE_PREEMPT_RESCHED_BITS,NEED_RESCHEDbit
+ * is put in a separate 32bits.
+ *
+ * HAS_SEPARATE_PREEMPT_RESCHED_BITS=n:
+ *
+ *NMI_MASK:0x10000000
+ *PREEMPT_NEED_RESCHED:0x80000000
+ *
+ * HAS_SEPARATE_PREEMPT_RESCHED_BITS=y:
+ *
+ * (NMI_MASKcan use all the 4 bits)
+ *
+ *NMI_MASK:0xf0000000
Quoted 1 line
Thoughts?
Perhaps add something like:
"Having PREEMPT_NEED_RESCHED in a separate word allows 64bit load-store architectures to 'set' PREEMPT_NEED_RESCHED without messing up the otherwise symmetric modifications used on preempt_count and still load the whole thing (single-copy) atomically, without having to resort to full atomic operations."
Boqun Feng
maintainerRe: [PATCH 10/24] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS
Quoted 74 lines · Peter Zijlstra
Quoted 9 lines · Boqun Feng
Quoted 9 lines
Quoted 8 lines
+/*
+ * unsigned long preempt count parameter works for both 32bit and 64bit cases:
+ *
+ * - For 32bit, "int" (the return ofpreempt_count()) and "unsigned long" have
+ * the same size.
+ * - For 64bit, the effective bits of a preempt count sits in 32bit, and we
+ * reserve theNEED_RESCHEDbit from the old count.
+ */Quoted 4 lines
Quoted 2 lines
The 64bit comment doesn't really make sense to me.
Ah, I meant "preserve" instead of "reserve"..
Ah, yes, that makes sense.
Quoted 54 lines
Quoted 28 lines
Quoted 13 lines
include/linux/preempt.h +0
-5@@ -30,18 +30,20 @@* NMI nesting depth is tracked in a separate per-CPU variable * (nmi_nesting) to save bits in preempt_count. * -* PREEMPT_MASK: 0x000000ff -* SOFTIRQ_MASK: 0x0000ff00 -* HARDIRQ_DISABLE_MASK: 0x00ff0000 -* HARDIRQ_MASK: 0x0f000000 -* NMI_MASK: 0x10000000 include/linux/preempt.h line 41
Quoted 10 lines
include/linux/preempt.h +8
-1-* PREEMPT_NEED_RESCHED: 0x80000000 +* 32bit HAS_SEPARATE_PREEMPT_RESCHED_BITS +* +* PREEMPT_MASK: 0x000000ff 0x00000000000000ff +* SOFTIRQ_MASK: 0x0000ff00 0x000000000000ff00 +* HARDIRQ_DISABLE_MASK: 0x00ff0000 0x0000000000ff0000 +* HARDIRQ_MASK: 0x0f000000 0x000000000f000000 +* NMI_MASK: 0x10000000 0x00000000f0000000 +* PREEMPT_NEED_RESCHED: 0x80000000 0x8000000000000000 */ Perhaps add a comment about how
HAS_SEPARATE_PREEMPT_RESCHED_BITSreally is about havingPREEMPT_NEED_RESCHEDin its own word, rather thanpreempt_count()being 64bit.Quoted 3 lines
Because as presented it is very easy to confuse these two options. Ideally it would explain the LOAD-STORE issue with
NEED_RESCHEDand point to ARM64 or something.How about organizing the comments as following:
+ *
+ *PREEMPT_MASK:0x000000ff
+ *SOFTIRQ_MASK:0x0000ff00
+ *HARDIRQ_DISABLE_MASK:0x00ff0000
+ *HARDIRQ_MASK:0x0f000000
+ *
+ * Depending onHAS_SEPARATE_PREEMPT_RESCHED_BITS,NEED_RESCHEDbit
+ * is put in a separate 32bits.
+ *
+ * HAS_SEPARATE_PREEMPT_RESCHED_BITS=n:
+ *
+ *NMI_MASK:0x10000000
+ *PREEMPT_NEED_RESCHED:0x80000000
+ *
+ * HAS_SEPARATE_PREEMPT_RESCHED_BITS=y:
+ *
+ * (NMI_MASKcan use all the 4 bits)
+ *
+ *NMI_MASK:0xf0000000Quoted 1 line
Thoughts?
Quoted 9 lines
Perhaps add something like:
"Having
PREEMPT_NEED_RESCHEDin a separate word allows 64bit load-store architectures to 'set'PREEMPT_NEED_RESCHEDwithout messing up the otherwise symmetric modifications used onpreempt_countand still load the whole thing (single-copy) atomically, without having to resort to full atomic operations."
Will do this as well. Thanks!
Regards,
Boqun
Boqun Feng
maintainerPATCH 11/24 arm64: sched/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
In reply to Boqun FengARM64 already uses 64bit preempt count and the need reschedule bit is maintained in a separate 32bit than the preempt count. Therefore preempt count has enough bits to represent 16 level of NMI nesting, hence enable it for ARM64. This saves a per-CPU variable and additional instructions in the NMI path.
--- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+)
arch/arm64/Kconfig +1 -0
@@ -247,6 +247,7 @@ config ARM64 select PCI_SYSCALL if PCI select POWER_RESET select POWER_SUPPLY + select HAS_SEPARATE_PREEMPT_RESCHED_BITS select SPARSE_IRQ select SWIOTLB select SYSCTL_EXCEPTION_TRACE
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 12/24 s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
In reply to Boqun FengFrom: Heiko Carstens <[email protected]>
Convert s390's preempt_count to 64 bit, and change the preempt primitives accordingly.
--- arch/s390/Kconfig | 1 + arch/s390/include/asm/lowcore.h | 13 +++++++---- arch/s390/include/asm/preempt.h | 41 +++++++++++++++------------------ 3 files changed, 29 insertions(+), 26 deletions(-)
arch/s390/Kconfig +1 -0
@@ -273,6 +273,7 @@ config S390 select PCI_MSI if PCI select PCI_MSI_ARCH_FALLBACKS if PCI_MSI select PCI_QUIRKS if PCI + select HAS_SEPARATE_PREEMPT_RESCHED_BITS select SPARSE_IRQ select SWIOTLB select SYSCTL_EXCEPTION_TRACE
arch/s390/include/asm/lowcore.h +9 -4
@@ -160,10 +160,15 @@ struct lowcore {/* SMP info area */ __u32 cpu_nr;/* 0x03a0 */ __u32 softirq_pending;/* 0x03a4 */ -__s32 preempt_count;/* 0x03a8 */ -__u32 spinlock_lockval;/* 0x03ac */ -__u32 spinlock_index;/* 0x03b0 */ -__u8 pad_0x03b4[0x03b8 -0x03b4 ];/* 0x03b4 */ +union { +struct { +__u32 need_resched;/* 0x03a8 */ +__u32 count;/* 0x03ac */ + } preempt; +__u64 preempt_count;/* 0x03a8 */ + }; +__u32 spinlock_lockval;/* 0x03b0 */ +__u32 spinlock_index;/* 0x03b4 */ __u64 percpu_offset;/* 0x03b8 */ __u8 percpu_register;/* 0x03c0 */ __u8 pad_0x03c1[0x0400 -0x03c1 ];/* 0x03c1 */
arch/s390/include/asm/preempt.h +19 -22
104 lines not on this page · raw message
Signature · 1 line
2.50.1 (Apple Git-155)
Peter Zijlstra
maintainerRe: [PATCH 12/24] s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
arch/s390/include/asm/preempt.h line 30
Quoted 12 lines · Boqun Feng
arch/s390/include/asm/preempt.h +3
-3@@ -26,25 +23,25 @@*/ static __always_inline int preempt_count(void) { -unsigned long lc_preempt, count; +unsigned long lc_preempt; +int count; - BUILD_BUG_ON(sizeof_field(struct lowcore, preempt_count) != sizeof(int)); - lc_preempt = offsetof(struct lowcore , preempt_count); + lc_preempt = offsetof(struct lowcore, preempt.count);/* READ_ONCE(get_lowcore()->preempt_count) & ~PREEMPT_NEED_RESCHED */
If I'm not mistaken, this comment is no longer accurate, since NEED_RESCHED lives in the high word and you're only loading the low word.
Quoted 13 lines
arch/s390/include/asm/preempt.h +3
-3asm_inline ( -ALTERNATIVE ("llgt %[count],%[offzero](%%r0)\n" , -"llgt %[count],%[offalt](%%r0)\n" , +ALTERNATIVE ("ly %[count],%[offzero](%%r0)\n" , +"ly %[count],%[offalt](%%r0)\n" ,ALT_FEATURE (MFEATURE_LOWCORE )) : [count] "=d" (count) : [offzero] "i" (lc_preempt), [offalt] "i" (lc_preempt + LOWCORE_ALT_ADDRESS), -"m" (((struct lowcore *)0 )->preempt_count)); +"m" (((struct lowcore *)0 )->preempt.count));return count; }
Heiko Carstens
maintainerRe: [PATCH 12/24] s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Boqun Feng
arch/s390/include/asm/preempt.h +2
-1@@ -26,25 +23,25 @@*/ static __always_inline int preempt_count(void) { -unsigned long lc_preempt, count; +unsigned long lc_preempt; +int count;
Quoted 8 lines
arch/s390/include/asm/preempt.h line 30
Quoted 4 lines
arch/s390/include/asm/preempt.h +1
-2- BUILD_BUG_ON(sizeof_field(struct lowcore, preempt_count) != sizeof(int)); - lc_preempt = offsetof(struct lowcore , preempt_count); + lc_preempt = offsetof(struct lowcore, preempt.count);/* READ_ONCE(get_lowcore()->preempt_count) & ~PREEMPT_NEED_RESCHED */ If I'm not mistaken, this comment is no longer accurate, since
NEED_RESCHEDlives in the high word and you're only loading the low word.
True. Something like
/* READ_ONCE(get_lowcore()->preempt.count) (without PREEMPT_NEED_RESCHED) */
might be better.
Quoted 13 lines
Quoted 13 lines
arch/s390/include/asm/preempt.h +3
-3asm_inline ( -ALTERNATIVE ("llgt %[count],%[offzero](%%r0)\n" , -"llgt %[count],%[offalt](%%r0)\n" , +ALTERNATIVE ("ly %[count],%[offzero](%%r0)\n" , +"ly %[count],%[offalt](%%r0)\n" ,ALT_FEATURE (MFEATURE_LOWCORE )) : [count] "=d" (count) : [offzero] "i" (lc_preempt), [offalt] "i" (lc_preempt + LOWCORE_ALT_ADDRESS), -"m" (((struct lowcore *)0 )->preempt_count)); +"m" (((struct lowcore *)0 )->preempt.count));return count; }
Boqun Feng
maintainerRe: [PATCH 12/24] s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
Quoted 20 lines · Heiko Carstens
Quoted 10 lines · Peter Zijlstra
Quoted 8 lines · Boqun Feng
arch/s390/include/asm/preempt.h +2
-1@@ -26,25 +23,25 @@*/ static __always_inline int preempt_count(void) { -unsigned long lc_preempt, count; +unsigned long lc_preempt; +int count;Quoted 8 lines
arch/s390/include/asm/preempt.h line 30
Quoted 4 lines
arch/s390/include/asm/preempt.h +1
-2- BUILD_BUG_ON(sizeof_field(struct lowcore, preempt_count) != sizeof(int)); - lc_preempt = offsetof(struct lowcore , preempt_count); + lc_preempt = offsetof(struct lowcore, preempt.count);/* READ_ONCE(get_lowcore()->preempt_count) & ~PREEMPT_NEED_RESCHED */ If I'm not mistaken, this comment is no longer accurate, since
NEED_RESCHEDlives in the high word and you're only loading the low word.
Quoted 6 lines
True. Something like
/* READ_ONCE(get_lowcore()->preempt.count) (without
PREEMPT_NEED_RESCHED) */might be better.
Thanks, I will update it.
Regards,
Boqun
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
arch/s390/include/asm/preempt.h +3
-3asm_inline ( -ALTERNATIVE ("llgt %[count],%[offzero](%%r0)\n" , -"llgt %[count],%[offalt](%%r0)\n" , +ALTERNATIVE ("ly %[count],%[offzero](%%r0)\n" , +"ly %[count],%[offalt](%%r0)\n" ,ALT_FEATURE (MFEATURE_LOWCORE )) : [count] "=d" (count) : [offzero] "i" (lc_preempt), [offalt] "i" (lc_preempt + LOWCORE_ALT_ADDRESS), -"m" (((struct lowcore *)0 )->preempt_count)); +"m" (((struct lowcore *)0 )->preempt.count));return count; }
Boqun Feng
maintainerPATCH 13/24 irq: Optimize reschedule check in local_interrupt_enable()
In reply to Boqun FengIn local_interrupt_enable(), we could avoid re-reading preempt count because of should_resched() by using the result from hardirq_disable_exit(), however this means __preempt_count_add_return() and __preempt_count_sub_return() need to return all the preempt count bits (including the PREEMPT_NEED_RESCHED bit), since the only user of __preempt_count_{add,sub}_return() is hardirq_disable_{enter,exit}(), hence make them return "unsigned long" to optimize this.
--- arch/arm64/include/asm/preempt.h | 12 ++++++------ arch/s390/include/asm/preempt.h | 4 ++-- arch/x86/include/asm/preempt.h | 4 ++-- include/asm-generic/preempt.h | 4 ++-- include/linux/interrupt_rc.h | 31 ++++++++++++++++++++----------- 5 files changed, 32 insertions(+), 23 deletions(-)
arch/arm64/include/asm/preempt.h +6 -6
@@ -55,20 +55,20 @@ static inline void __preempt_count_sub(int val)WRITE_ONCE (current_thread_info ()->preempt.count, pc); } -static inline int __preempt_count_add_return (int val) +static inline unsigned long __preempt_count_add_return (int val) { -u32 pc =READ_ONCE (current_thread_info ()->preempt.count); +u64 pc =READ_ONCE (current_thread_info ()->preempt_count); pc += val; -WRITE_ONCE (current_thread_info ()->preempt.count, pc); +WRITE_ONCE (current_thread_info ()->preempt_count, pc);return pc; } -static inline int __preempt_count_sub_return (int val) +static inline unsigned long __preempt_count_sub_return (int val) { -u32 pc =READ_ONCE (current_thread_info ()->preempt.count); +u64 pc =READ_ONCE (current_thread_info ()->preempt_count); pc -= val; -WRITE_ONCE (current_thread_info ()->preempt.count, pc); +WRITE_ONCE (current_thread_info ()->preempt_count, pc);return pc; }
arch/s390/include/asm/preempt.h +2 -2
@@ -136,12 +136,12 @@ static __always_inline bool should_resched(int preempt_offset)return unlikely (READ_ONCE (get_lowcore ()->preempt_count) == preempt_offset); } -static __always_inline int __preempt_count_add_return(int val) +static __always_inline unsigned long __preempt_count_add_return (int val) {return val +__atomic64_add (val, (long *)&get_lowcore ()->preempt_count); } -static __always_inline int __preempt_count_sub_return(int val) +static __always_inline unsigned long __preempt_count_sub_return (int val) {return __preempt_count_add_return (-val); }
arch/x86/include/asm/preempt.h +2 -2
@@ -103,12 +103,12 @@ static __always_inline void __preempt_count_sub(int val)__pc_op (add, __preempt_count, -val); } -static __always_inline int __preempt_count_add_return(int val) +static __always_inline unsigned long __preempt_count_add_return (int val) {return __pc_op (add_return, __preempt_count, val); } -static __always_inline int __preempt_count_sub_return(int val) +static __always_inline unsigned long __preempt_count_sub_return (int val) {return __pc_op (add_return, __preempt_count, -val); }
include/asm-generic/preempt.h +2 -2
@@ -59,14 +59,14 @@ static __always_inline void __preempt_count_sub(int val) *preempt_count_ptr () -= val; } -static __always_inline int __preempt_count_add_return(int val) +static __always_inline unsigned long __preempt_count_add_return (int val) { *preempt_count_ptr () += val;return *preempt_count_ptr (); } -static __always_inline int __preempt_count_sub_return(int val) +static __always_inline unsigned long __preempt_count_sub_return (int val) { *preempt_count_ptr () -= val;
include/linux/interrupt_rc.h +20 -11
@@ -27,7 +27,7 @@ DECLARE_PER_CPU(struct interrupt_disable_state, local_interrupt_disable_state);static inline void local_interrupt_disable (void ) {unsigned long flags; -int new_count; +unsigned long new_count;WARN_ON_ONCE (in_nmi ());
@@ -41,9 +41,25 @@ static inline void local_interrupt_disable(void) } } +#ifdef CONFIG_PREEMPTION +static inline void local_interrupt_enable_reched (unsigned long pc) +{ +if (pc) +return ; +/* No PREEMPT_NEED_RESCHED bit? Check tif_need_resched() */ +#ifndef PREEMPT_NEED_RESCHED +if (!tif_need_resched ()) +return ; +#endif +__preempt_schedule (); +} +#else +static inline void local_interrupt_enable_reched (unsigned long pc) {} +#endif +static inline void local_interrupt_enable (void ) { -int new_count; +unsigned long new_count; new_count =hardirq_disable_exit ();
@@ -52,15 +68,8 @@ static inline void local_interrupt_enable(void) flags =raw_cpu_read (local_interrupt_disable_state.flags);local_irq_restore (flags); -/* -* TODO: re-read preempt count can be avoided, but it needs -* should_resched() taking another parameter as the current -* preempt count -*/ -#ifdef CONFIG_PREEMPTION -if (should_resched (0 )) -__preempt_schedule (); -#endif + +local_interrupt_enable_reched (new_count); } }
Signature · 1 line
2.50.1 (Apple Git-155)
Peter Zijlstra
maintainerRe: [PATCH 13/24] irq: Optimize reschedule check in local_interrupt_enable()
Quoted 22 lines · Boqun Feng
In
local_interrupt_enable(), we could avoid re-reading preempt count because ofshould_resched()by using the result fromhardirq_disable_exit(), however this means__preempt_count_add_return()and__preempt_count_sub_return()need to return all the preempt count bits (including thePREEMPT_NEED_RESCHEDbit), since the only user of __preempt_count_{add,sub}_return() is hardirq_disable_{enter,exit}(), hence make them return "unsigned long" to optimize this.static inline void
local_interrupt_enable(void)
{include/linux/interrupt_rc.h +1
-4-int new_count; +unsigned long new_count; new_count =hardirq_disable_exit ();@@ -52,15 +68,8 @@ static inline void local_interrupt_enable(void) flags =raw_cpu_read (local_interrupt_disable_state.flags);local_irq_restore (flags); -/* -* TODO: re-read preempt count can be avoided, but it needs -* should_resched() taking another parameter as the current
include/linux/interrupt_rc.h static inline void local_interrupt_enable(void) · line 74
Quoted 10 lines
include/linux/interrupt_rc.h +2
-6-* preempt count -*/ -#ifdef CONFIG_PREEMPTION -if (should_resched (0 )) -__preempt_schedule (); -#endif + +local_interrupt_enable_reched (new_count); } }
As stated before; this whole thing is a mystery to me, this should not be needed.
Boqun Feng
maintainerRe: [PATCH 13/24] irq: Optimize reschedule check in local_interrupt_enable()
Quoted 28 lines · Peter Zijlstra
Quoted 27 lines · Boqun Feng
In
local_interrupt_enable(), we could avoid re-reading preempt count because ofshould_resched()by using the result fromhardirq_disable_exit(), however this means__preempt_count_add_return()and__preempt_count_sub_return()need to return all the preempt count bits (including thePREEMPT_NEED_RESCHEDbit), since the only user of __preempt_count_{add,sub}_return() is hardirq_disable_{enter,exit}(), hence make them return "unsigned long" to optimize this.static inline void
local_interrupt_enable(void)
{include/linux/interrupt_rc.h +1
-9-int new_count; +unsigned long new_count; new_count =hardirq_disable_exit ();@@ -52,15 +68,8 @@ static inline void local_interrupt_enable(void) flags =raw_cpu_read (local_interrupt_disable_state.flags);local_irq_restore (flags); -/* -* TODO: re-read preempt count can be avoided, but it needs -* should_resched() taking another parameter as the current -* preempt count -*/ -#ifdef CONFIG_PREEMPTION -if (should_resched (0 )) -__preempt_schedule ();
Quoted 8 lines
include/linux/interrupt_rc.h static inline void local_interrupt_enable(void) · line 74
Quoted 5 lines
include/linux/interrupt_rc.h +2
-1-#endif + +local_interrupt_enable_reched (new_count); } }As stated before; this whole thing is a mystery to me, this should not be needed.
I will drop this patch entirely per the discussion.
Regards,
Boqun
Boqun Feng
maintainerPATCH 14/24 rust: Introduce interrupt module
In reply to Boqun FengFrom: Lyude Paul <[email protected]>
This introduces a module for dealing with interrupt-disabled contexts, including the ability to enable and disable interrupts along with the ability to annotate functions as expecting that IRQs are already disabled on the local CPU.
[Boqun: This is based on Lyude's work on interrupt disable abstraction, I port to the new local_interrupt_disable() mechanism to make it work as a guard type. I cannot even take the credit of this design, since Lyude also brought up the same idea in zulip. Anyway, this is only for POC purpose, and of course all bugs are mine]
--- rust/helpers/helpers.c | 1 + rust/helpers/interrupt.c | 18 ++++++++ rust/helpers/sync.c | 5 +++ rust/kernel/interrupt.rs | 89 ++++++++++++++++++++++++++++++++++++++++ rust/kernel/lib.rs | 1 + 5 files changed, 114 insertions(+) create mode 100644 rust/helpers/interrupt.c create mode 100644 rust/kernel/interrupt.rs
rust/helpers/helpers.c +1 -0
@@ -65,6 +65,7 @@#include "irq.c" #include "fs.c" #include "gpu.c" +#include "interrupt.c" #include "io.c" #include "jump_label.c" #include "kunit.c"
rust/helpers/interrupt.c new file +18 -0
@@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/spinlock.h> + +__rust_helper void rust_helper_local_interrupt_disable(void) +{ +local_interrupt_disable (); +} + +__rust_helper void rust_helper_local_interrupt_enable(void) +{ +local_interrupt_enable (); +} + +__rust_helper bool rust_helper_irqs_disabled(void) +{ +return irqs_disabled (); +}
rust/helpers/sync.c +5 -0
@@ -11,3 +11,8 @@ __rust_helper void rust_helper_lockdep_unregister_key(struct lock_class_key *k) {lockdep_unregister_key (k); } + +__rust_helper void rust_helper_lockdep_assert_irqs_disabled(void) +{ +lockdep_assert_irqs_disabled (); +}
rust/kernel/interrupt.rs new file +89 -0
89 lines not on this page · raw message
rust/kernel/lib.rs +1 -0
@@ -82,6 +82,7 @@pub mod impl_flags;pub mod init;pub mod interop; +pub mod interrupt;pub mod io;pub mod ioctl;pub mod iommu;
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 15/24 rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers
In reply to Boqun FengFrom: Boqun Feng <[email protected]>
spin_lock_irq_disable() and spin_unlock_irq_enable() are inline functions, to use them in Rust, helpers are introduced. This is for interrupt disabling lock abstraction in Rust.
--- rust/helpers/spinlock.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
rust/helpers/spinlock.c +15 -0
@@ -36,3 +36,18 @@ __rust_helper void rust_helper_spin_assert_is_held(spinlock_t *lock) {lockdep_assert_held (lock); } + +__rust_helper void rust_helper_spin_lock_irq_disable(spinlock_t *lock) +{ +spin_lock_irq_disable (lock); +} + +__rust_helper void rust_helper_spin_unlock_irq_enable(spinlock_t *lock) +{ +spin_unlock_irq_enable (lock); +} + +__rust_helper int rust_helper_spin_trylock_irq_disable(spinlock_t *lock) +{ +return spin_trylock_irq_disable (lock); +}
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 16/24 rust: sync: use super::* in spinlock.rs
In reply to Boqun FengFrom: Lyude Paul <[email protected]>
No functional changes.
--- rust/kernel/sync/lock/spinlock.rs | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
rust/kernel/sync/lock/spinlock.rs +4 -5
@@ -3,6 +3,7 @@//! A kernel spinlock. //! //! This module allows Rust code to use the kernel's `spinlock_t`. +use super ::*;/// Creates a [`SpinLock`] initialiser with the given name and a newly-created lock class. ///
@@ -82,7 +83,7 @@ macro_rules! new_spinlock {/// ``` /// /// [`spinlock_t`]: srctree/include/linux/spinlock.h -pub type SpinLock <T > =super ::Lock <T ,SpinLockBackend >; +pub type SpinLock <T > =Lock <T ,SpinLockBackend >;/// A kernel `spinlock_t` lock backend. pub struct SpinLockBackend ;
@@ -91,13 +92,11 @@ macro_rules! new_spinlock {/// /// This is simply a type alias for a [`Guard`] returned from locking a [`SpinLock`]. It will unlock /// the [`SpinLock`] upon being dropped. -/// -/// [`Guard`]: super::Guard -pub type SpinLockGuard <'a,T > =super ::Guard <'a,T ,SpinLockBackend >; +pub type SpinLockGuard <'a,T > =Guard <'a,T ,SpinLockBackend >;// SAFETY: The underlying kernel `spinlock_t` object ensures mutual exclusion. `relock` uses the // default implementation that always calls the same locking method. -unsafe impl super ::Backend for SpinLockBackend { +unsafe impl Backend for SpinLockBackend {type State = bindings::spinlock_t ;type GuardState = ();
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 17/24 rust: sync: Add SpinLockIrq
In reply to Boqun FengFrom: Lyude Paul <[email protected]>
A variant of SpinLock that ensures interrupts are disabled in the critical section. lock() will ensure that either interrupts are already disabled or disable them. unlock() will reverse the respective operation.
[Boqun: Port to use spin_lock_irq_disable() andspin_unlock_irq_enable()]
--- rust/kernel/sync.rs | 9 +- rust/kernel/sync/lock/global.rs | 3 + rust/kernel/sync/lock/spinlock.rs | 243 ++++++++++++++++++++++++++++++ 3 files changed, 254 insertions(+), 1 deletion(-)
rust/kernel/sync.rs +8 -1
@@ -27,7 +27,14 @@pub use condvar::{new_condvar,CondVar ,CondVarTimeoutResult };pub use lock::global::{global_lock,GlobalGuard ,GlobalLock ,GlobalLockBackend ,GlobalLockedBy };pub use lock::mutex::{new_mutex,Mutex ,MutexGuard }; -pub use lock::spinlock::{new_spinlock,SpinLock ,SpinLockGuard }; +pub use lock::spinlock::{ + new_spinlock, + new_spinlock_irq, +SpinLock , +SpinLockGuard , +SpinLockIrq , +SpinLockIrqGuard ,// +};pub use locked_by::LockedBy ;pub use refcount::Refcount ;pub use set_once::SetOnce ;
rust/kernel/sync/lock/global.rs +3 -0
@@ -306,4 +306,7 @@ macro_rules! global_lock_inner { (backend SpinLock) => { $crate::sync::lock::spinlock::SpinLockBackend }; + (backend SpinLockIrq) => { + $crate::sync::lock::spinlock::SpinLockIrqBackend + }; }
rust/kernel/sync/lock/spinlock.rs +243 -0
252 lines not on this page · raw message
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 18/24 rust: sync: Introduce SpinLockIrq::lock_with() and friends
In reply to Boqun FengFrom: Lyude Paul <[email protected]>
SpinLockIrq and SpinLock use the exact same underlying C structure, with the only real difference being that the former uses the irq_disable() and irq_enable() variants for locking/unlocking. These variants can introduce some minor overhead in contexts where we already know that local processor interrupts are disabled, and as such we want a way to be able to skip modifying processor interrupt state in said contexts in order to avoid some overhead - just like the current C API allows us to do.
In order to do this, we add some special functions for SpinLockIrq:lock_with() and try_lock_with(), which allow acquiring the lock without changing the interrupt state - as long as the caller can provide a LocalInterruptDisabled reference to prove that local processor interrupts have been disabled.
In some hacked-together benchmarks we ran, most of the time this did actually seem to lead to a noticeable difference in overhead:
From an aarch64 VM running on a MacBook M4:
lock() when irq is disabled, 100 times cost Delta { nanos: 500 }
lock_with() when irq is disabled, 100 times cost Delta { nanos: 292 }
lock() when irq is enabled, 100 times cost Delta { nanos: 834 }
lock() when irq is disabled, 100 times cost Delta { nanos: 459 }
lock_with() when irq is disabled, 100 times cost Delta { nanos: 291 }
lock() when irq is enabled, 100 times cost Delta { nanos: 709 } From an x86_64 VM (qemu/kvm) running on a i7-13700H
lock() when irq is disabled, 100 times cost Delta { nanos: 1002 }
lock_with() when irq is disabled, 100 times cost Delta { nanos: 729 }
lock() when irq is enabled, 100 times cost Delta { nanos: 1516 }
lock() when irq is disabled, 100 times cost Delta { nanos: 754 }
lock_with() when irq is disabled, 100 times cost Delta { nanos: 966 }
lock() when irq is enabled, 100 times cost Delta { nanos: 1227 }
(note that there were some runs on x86_64 where lock() on irq disabled
vs. lock_with() on irq disabled had equivalent benchmarks, but it very
much appeared to be a minority of test runs.While it's not clear how this affects real-world workloads yet, let's add this for the time being so we can find out.
This makes it so that a SpinLockIrq will work like a SpinLock if interrupts are disabled. So a function:
(&'a SpinLockIrq, &'a InterruptDisabled) -> Guard<'a, .., SpinLockBackend>
makes sense. Note that due to Guard and InterruptDisabled having the same lifetime, interrupts cannot be enabled while the Guard exists.
--- rust/kernel/sync/lock/spinlock.rs | 81 ++++++++++++++++++++++++++++++- 1 file changed, 80 insertions(+), 1 deletion(-)
rust/kernel/sync/lock/spinlock.rs +80 -1
@@ -4,7 +4,10 @@//! //! This module allows Rust code to use the kernel's `spinlock_t`. use super ::*; -use crate ::prelude::*; +use crate ::{ + interrupt::LocalInterruptDisabled , + prelude::*,// +};/// Creates a [`SpinLock`] initialiser with the given name and a newly-created lock class. ///
@@ -224,6 +227,45 @@ macro_rules! new_spinlock_irq {/// # Ok::<(), Error>(()) /// ``` /// +/// The next example demonstrates locking a [`SpinLockIrq`] using [`lock_with()`] in a function +/// which can only be called when local processor interrupts are already disabled. +/// +/// ``` +/// use kernel::sync::{new_spinlock_irq, SpinLockIrq}; +/// use kernel::interrupt::*; +/// +/// struct Inner { +/// a: u32, +/// } +/// +/// #[pin_data] +/// struct Example { +/// #[pin] +/// inner: SpinLockIrq<Inner>, +/// } +/// +/// impl Example { +/// fn new() -> impl PinInit<Self> { +/// pin_init!(Self { +/// inner <- new_spinlock_irq!(Inner { a: 20 }), +/// }) +/// } +/// } +/// +/// // Accessing an `Example` from a function that can only be called in no-interrupt contexts. +/// fn noirq_work(e: &Example, interrupt_disabled: &LocalInterruptDisabled) { +/// // Because we know interrupts are disabled from interrupt_disable, we can skip toggling +/// // interrupt state using lock_with() and the provided token +/// assert_eq!(e.inner.lock_with(interrupt_disabled).a, 20); +/// } +/// +/// # let e = KBox::pin_init(Example::new(), GFP_KERNEL)?; +/// # let interrupt_guard = local_interrupt_disable(); +/// # noirq_work(&e, &interrupt_guard); +/// # +/// # Ok::<(), Error>(()) +/// ``` +/// /// [`lock()`]: SpinLockIrq::lock /// [`lock_with()`]: SpinLockIrq::lock_with pub type SpinLockIrq <T > =super ::Lock <T ,SpinLockIrqBackend >;
@@ -291,6 +333,43 @@ unsafe fn assert_is_held(ptr: *mut Self::State) { } } +impl <T : ?Sized >Lock <T ,SpinLockIrqBackend > { +/// Casts the lock as a `Lock<T, SpinLockBackend>`. +#[inline] +fn as_lock_in_interrupt <'a>(&'a self, _context: &'aLocalInterruptDisabled ) -> &'aSpinLock <T > { +// SAFETY: +// - `Lock<T, SpinLockBackend>` and `Lock<T, SpinLockIrqBackend>` both have identical data +// layouts. +// - As long as local interrupts are disabled (which is proven to be true by _context), it +// is safe to treat a lock with SpinLockIrqBackend as a SpinLockBackend lock. +unsafe { core::mem::transmute (self) } + } + +/// Acquires the lock without modifying local interrupt state. +/// +/// This function should be used in place of the more expensive [`Lock::lock()`] function when +/// possible for [`SpinLockIrq`] locks. +#[inline] +pub fn lock_with <'a>(&'a self, context: &'aLocalInterruptDisabled ) ->SpinLockGuard <'a,T > { + self.as_lock_in_interrupt (context).lock () + } + +/// Tries to acquire the lock without modifying local interrupt state. +/// +/// This function should be used in place of the more expensive [`Lock::try_lock()`] function +/// when possible for [`SpinLockIrq`] locks. +/// +/// Returns a guard that can be used to access the data protected by the lock if successful. +#[must_use = "if unused, the lock will be immediately unlocked" ] +#[inline] +pub fn try_lock_with <'a>( + &'a self, + context: &'aLocalInterruptDisabled , + ) ->Option <SpinLockGuard <'a,T >> { + self.as_lock_in_interrupt (context).try_lock () + } +} +#[kunit_tests(rust_spinlock_irq_condvar)] mod tests {use super ::*;
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 19/24 rust: sync: Add abstraction for synchronize_rcu()
In reply to Boqun FengFrom: Philipp Stanner <[email protected]>
synchronize_rcu() is a frequently used C function which is always safe to be called.
Add a safe abstraction for synchronize_rcu().
--- rust/kernel/sync/rcu.rs | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
rust/kernel/sync/rcu.rs +16 -0
@@ -50,3 +50,19 @@ fn drop(&mut self) {pub fn read_lock () ->Guard {Guard ::new () } + +/// Wait for one RCU grace period. +/// +/// Waits for all RCU read-side critical sections (such as those established by +/// a [`Guard`]) at the moment of the function call to finish. +/// +/// Does not prevent new read-side critical sections from starting, which may +/// begin and run while this call is blocking. +/// +/// Note that this is one of the RCU primitives which must not be called in +/// atomic context. +#[inline] +pub fn synchronize_rcu () { +// SAFETY: `synchronize_rcu()` is always safe to be called from process context. +unsafe { bindings::synchronize_rcu () }; +}
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 20/24 rust: revocable: Use safe synchronize_rcu() abstraction
In reply to Boqun FengFrom: Philipp Stanner <[email protected]>
We now have a safe wrapper for the foreign function synchronize_rcu().
Use it in revocable.rs.
--- rust/kernel/revocable.rs | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
rust/kernel/revocable.rs +6 -3
@@ -7,7 +7,11 @@use pin_init::Wrapper ; -use crate ::{bindings, prelude::*, sync::rcu, types::Opaque }; +use crate ::{ + prelude::*, + sync::rcu, + types::Opaque ,// +};use core::{ marker::PhantomData , ops::Deref ,
@@ -161,8 +165,7 @@ unsafe fn revoke_internal<const SYNC: bool>(&self) -> bool {if revoke {if SYNC { -// SAFETY: Just an FFI call, there are no further requirements. -unsafe { bindings::synchronize_rcu () }; + rcu::synchronize_rcu (); }// SAFETY: We know `self.data` is valid because only one CPU can succeed the
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 21/24 rust: sync: Use safe synchronize_rcu() abstraction in poll
In reply to Boqun FengFrom: Philipp Stanner <[email protected]>
We now have a safe wrapper for the foreign function synchronize_rcu().
Use it in poll.rs.
--- rust/kernel/sync/poll.rs | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
rust/kernel/sync/poll.rs +6 -4
@@ -8,7 +8,11 @@ bindings, fs::File, prelude::*, - sync::{CondVar, LockClassKey}, + sync::{ + rcu::synchronize_rcu, + CondVar, + LockClassKey, // + }, // };use core::{marker::PhantomData , ops::Deref };
@@ -99,8 +103,6 @@ fn drop(self: Pin<&mut Self>) {unsafe { bindings::__wake_up_pollfree (self.inner.wait_queue_head.get ()) };// Wait for epoll items to be properly removed. -// -// SAFETY: Just an FFI call. -unsafe { bindings::synchronize_rcu () }; +synchronize_rcu (); } }
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 22/24 rust: sync: Add helpers for mb, dma_mb and friends
In reply to Boqun FengFrom: Gary Guo <[email protected]>
They supplement the existing smp_mb, smp_rmb and smp_wmb.
--- rust/helpers/barrier.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+)
rust/helpers/barrier.c +30 -0
@@ -2,6 +2,36 @@#include <asm/barrier.h> +__rust_helper void rust_helper_mb(void) +{ +mb (); +} + +__rust_helper void rust_helper_rmb(void) +{ +rmb (); +} + +__rust_helper void rust_helper_wmb(void) +{ +wmb (); +} + +__rust_helper void rust_helper_dma_mb(void) +{ +dma_mb (); +} + +__rust_helper void rust_helper_dma_rmb(void) +{ +dma_rmb (); +} + +__rust_helper void rust_helper_dma_wmb(void) +{ +dma_wmb (); +} + __rust_helper void rust_helper_smp_mb(void) {smp_mb ();
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 23/24 rust: sync: Add generic memory barriers
In reply to Boqun FengFrom: Gary Guo <[email protected]>
Implement a generic interface for memory barriers (full system/DMA/SMP). The interface uses a parameter to force user to specify their intent with barriers.
Provide Read, Write, Full orderings which map to the existing rmb(), wmb() and mb(). Generic is used here instead of providing individual standalone functions to reduce code duplication; for example, the CONFIG_SMP check in smp_mb is uniformly implemented for all SMP barriers. This could extend to virt_mb's if they're introduced in the future. It would also make it easier if new ordering types are introduced in the future (e.g. Acquire, Release).
--- rust/kernel/sync/atomic/ordering.rs | 2 +- rust/kernel/sync/barrier.rs | 127 ++++++++++++++++++++++------ 2 files changed, 100 insertions(+), 29 deletions(-)
rust/kernel/sync/atomic/ordering.rs +1 -1
@@ -15,7 +15,7 @@//! - It provides ordering between the annotated operation and all the following memory accesses. //! - It provides ordering between all the preceding memory accesses and all the following memory //! accesses. -//! - All the orderings are the same strength as a full memory barrier (i.e. `smp_mb()`). +//! - All the orderings are the same strength as a full memory barrier (i.e. `smp_mb(Full)`). //! - [`Relaxed`] provides no ordering except the dependency orderings. Dependency orderings are //! described in "DEPENDENCY RELATIONS" in [`LKMM`]'s [`explanation`]. //!
rust/kernel/sync/barrier.rs +99 -28
148 lines not on this page · raw message
Signature · 1 line
2.50.1 (Apple Git-155)
Boqun Feng
maintainerPATCH 24/24 rust: revocable: Use LKMM atomics instead of Rust atomics
In reply to Boqun FengFrom: Gary Guo <[email protected]>
Kernel code should use LKMM atomics. The existing code is AtomicBool with the need to use xchg, so convert it to AtomicFlag.
--- rust/kernel/revocable.rs | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
rust/kernel/revocable.rs +13 -8
@@ -9,14 +9,19 @@use crate ::{ prelude::*, - sync::rcu, + sync::{ + atomic::{ +AtomicFlag , +Relaxed ,// + }, + rcu,// + }, types::Opaque ,// };use core::{ marker::PhantomData , ops::Deref , - ptr::drop_in_place, - sync::atomic::{AtomicBool ,Ordering }, + ptr::drop_in_place,// };/// An object that can become inaccessible at runtime.
@@ -69,7 +74,7 @@/// ``` #[pin_data( PinnedDrop )] pub struct Revocable <T > { - is_available:AtomicBool , + is_available:AtomicFlag ,#[pin] data:Opaque <T >, }
@@ -88,7 +93,7 @@ impl<T> Revocable<T> {/// Creates a new revocable instance of the given data. pub fn new <E >(data:impl PinInit <T ,E >) ->impl PinInit <Self ,E > {try_pin_init! (Self { - is_available:AtomicBool ::new (true ), + is_available:AtomicFlag ::new (true ), data <-Opaque ::pin_init (data), }?E ) }
@@ -102,7 +107,7 @@ pub fn new<E>(data: impl PinInit<T, E>) -> impl PinInit<Self, E> {/// because another CPU may be waiting to complete the revocation of this object. pub fn try_access (&self) ->Option <RevocableGuard <'_,T >> {let guard = rcu::read_lock (); -if self.is_available.load (Ordering ::Relaxed ) { +if self.is_available.load (Relaxed ) {// Since `self.is_available` is true, data is initialised and has to remain valid // because the RCU read side lock prevents it from being dropped. Some (RevocableGuard ::new (self.data.get (), guard))
@@ -120,7 +125,7 @@ pub fn try_access(&self) -> Option<RevocableGuard<'_, T>> {/// allowed to sleep because another CPU may be waiting to complete the revocation of this /// object. pub fn try_access_with_guard <'a>(&'a self, _guard: &'a rcu::Guard ) ->Option <&'aT > { -if self.is_available.load (Ordering ::Relaxed ) { +if self.is_available.load (Relaxed ) {// SAFETY: Since `self.is_available` is true, data is initialised and has to remain // valid because the RCU read side lock prevents it from being dropped. Some (unsafe { &*self.data.get () })
@@ -161,7 +166,7 @@ pub unsafe fn access(&self) -> &T {/// /// Callers must ensure that there are no more concurrent users of the revocable object. unsafe fn revoke_internal <const SYNC :bool >(&self) ->bool { -let revoke = self.is_available.swap (false ,Ordering ::Relaxed ); +let revoke = self.is_available.xchg (false ,Relaxed );if revoke {if SYNC {
Signature · 1 line
2.50.1 (Apple Git-155)