Skip to contentSkip to conversation

Refcounted interrupt disable and SpinLockIrq for Rust

PATCH Applied Boqun Feng 6 participants 110 messages Aug 4 – Sep 4

Subsystems (20)
x86 Kselftest ACPI Library code +16 more subsystems

v4 of 5 Applied Latest is v5 →

lore
Share & tools
lore

Boqun Feng

author maintainer
#0 lore raw

v4 cover Refcounted interrupt disable and SpinLockIrq for Rust

Hi Peter & Ingo,

Please take a look at this updated series for SpinLockIrq.

Changes since the PR attempt [1]:

* Removed the preempt_count check in local_interrupt_enable() per Peter.
* Simplified the local_interrupt_disable_state struct to avoid unnecessary wrapper (Thanks Peter).
* Adjusted the user side changes in do_sched_cfs_*_timer().
* Improved the documentation on the preempt_count layout regarding HAS_SEPARATE_PREEMPT_RESCHED_BITS
* Updated the asm block comment for s390 (Thanks Heiko!).
* SoB chain/typo/format fixes.

(Lyude, I dropped the Link tags of your original patchset, since I did some significant changes and let's just use the new links).

[1]: https://lore.kernel.org/rust-for-linux/[email protected]/

Thanks!

Regards,
Boqun

Boqun Feng (9):
preempt: Introduce HARDIRQ_DISABLE_BITS
preempt: Introduce __preempt_count_{sub,add}_return()
irq & spin_lock: Add counted interrupt disabling/enabling
locking: Switch to _irq_{disable,enable}() variants in cleanup guards
sched: Remove the unused preempt_offset parameter of __cant_sleep()
sched: Avoid signed comparison of preempt_count() in __cant_migrate()
preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS
arm64: sched/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS
rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers

Heiko Carstens (1):
s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

Joel Fernandes (1):
preempt: Track NMI nesting to separate per-CPU counter

Lyude Paul (6):
openrisc: Include <linux/cpumask.h> in smp.h
irq: Add KUnit test for refcounted interrupt enable/disable
rust: Introduce interrupt module
rust: sync: Use super::* in spinlock.rs
rust: sync: Add SpinLockIrq
rust: sync: Introduce SpinLockIrq::lock_with() and friends

 arch/arm64/Kconfig                            |   1 +
 arch/arm64/include/asm/preempt.h              |  20 ++
 arch/openrisc/include/asm/smp.h               |   2 +
 arch/s390/Kconfig                             |   1 +
 arch/s390/include/asm/lowcore.h               |  13 +-
 arch/s390/include/asm/preempt.h               |  51 +--
 arch/x86/Kconfig                              |   1 +
 arch/x86/include/asm/preempt.h                |  61 +++-
 arch/x86/kernel/cpu/common.c                  |   2 +-
 include/asm-generic/preempt.h                 |  14 +
 include/linux/hardirq.h                       |  40 ++-
 include/linux/interrupt_rc.h                  |  82 +++++
 include/linux/kernel.h                        |   4 +-
 include/linux/preempt.h                       |  44 ++-
 include/linux/spinlock.h                      |  49 ++-
 include/linux/spinlock_api_smp.h              |  43 +++
 include/linux/spinlock_api_up.h               |  15 +
 include/linux/spinlock_rt.h                   |  18 +
 kernel/Kconfig.preempt                        |   4 +
 kernel/irq/Makefile                           |   1 +
 kernel/irq/refcount_interrupt_test.c          | 109 ++++++
 kernel/locking/spinlock.c                     |  31 ++
 kernel/sched/core.c                           |  18 +-
 kernel/sched/fair.c                           |  12 +-
 kernel/softirq.c                              |  36 +-
 lib/locking-selftest.c                        |   2 +-
 rust/helpers/helpers.c                        |   1 +
 rust/helpers/interrupt.c                      |  18 +
 rust/helpers/spinlock.c                       |  15 +
 rust/helpers/sync.c                           |   5 +
 rust/kernel/interrupt.rs                      |  89 +++++
 rust/kernel/lib.rs                            |   1 +
 rust/kernel/sync.rs                           |   9 +-
 rust/kernel/sync/lock/global.rs               |   3 +
 rust/kernel/sync/lock/spinlock.rs             | 329 +++++++++++++++++-
 .../testing/selftests/bpf/bpf_experimental.h  |   7 +-
 36 files changed, 1060 insertions(+), 91 deletions(-)
 create mode 100644 include/linux/interrupt_rc.h
 create mode 100644 kernel/irq/refcount_interrupt_test.c
 create mode 100644 rust/helpers/interrupt.c
 create mode 100644 rust/kernel/interrupt.rs
Signature · 1 line
2.50.1 (Apple Git-155)
author maintainer
#1 lore raw

PATCH v4 1/17 preempt: Track NMI nesting to separate per-CPU counter

From: Joel Fernandes <[email protected]>

Move NMI nesting tracking from the preempt_count bits to a separate per-CPU counter (nmi_nesting). This is to free up the NMI bits in the preempt_count, allowing those bits to be repurposed for other uses.

Reduce NMI_BITS from 4 to 1, using it only to detect if we're in an NMI. The per-CPU counter currently caps nesting at 15.

[boqun: Address Steven Rostedt's comment on the BUG_ON() condition] [boqun: Use preempt_count_set() in __nmi_exit() to avoid underflow]

---
 include/linux/hardirq.h                        | 17 +++++++++++++----
 include/linux/preempt.h                        |  9 +++++++--
 kernel/softirq.c                               |  2 ++
 tools/testing/selftests/bpf/bpf_experimental.h |  2 +-
 4 files changed, 23 insertions(+), 7 deletions(-)
include/linux/hardirq.h +13 -4
index d57cab4d4c06..8d4895531a45 100644
@@ -10,6 +10,8 @@
 #include <linux/vtime.h>
 #include <asm/hardirq.h>
 
+DECLARE_PER_CPU(unsigned int, nmi_nesting);
+
 extern void synchronize_irq(unsigned int irq);
 extern bool synchronize_hardirq(unsigned int irq);
 
@@ -102,14 +104,17 @@ void irq_exit_rcu(void);
  */
 
 /*
- * nmi_enter() can nest up to 15 times; see NMI_BITS.
+ * nmi_enter() can nest - nesting is tracked in a per-CPU counter.
  */
 #define __nmi_enter()						\
 	do {							\
 		lockdep_off();					\
 		arch_nmi_enter();				\
-		BUG_ON(in_nmi() == NMI_MASK);			\
-		__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		/* Maximum NMI nesting is 15. */		\
+		BUG_ON(__this_cpu_read(nmi_nesting) >= 15);	\
+		__this_cpu_inc(nmi_nesting);			\
+		__preempt_count_add(HARDIRQ_OFFSET);		\
+		preempt_count_set(preempt_count() | NMI_MASK);	\
 	} while (0)
 
 #define nmi_enter()						\
@@ -124,8 +129,12 @@ void irq_exit_rcu(void);
 
 #define __nmi_exit()						\
 	do {							\
+		unsigned int nesting;				\
 		BUG_ON(!in_nmi());				\
-		__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		__preempt_count_sub(HARDIRQ_OFFSET);		\
+		nesting = __this_cpu_dec_return(nmi_nesting);	\
+		if (!nesting)					\
+			preempt_count_set(preempt_count() & ~NMI_MASK);	\
 		arch_nmi_exit();				\
 		lockdep_on();					\
 	} while (0)
include/linux/preempt.h +7 -2
index d964f965c8ff..586f96688325 100644
@@ -17,6 +17,8 @@
  *
  * - bits 0-7 are the preemption count (max preemption depth: 256)
  * - bits 8-15 are the softirq count (max # of softirqs: 256)
+ * - bits 16-19 are the hardirq count (max # of hardirqs: 16)
+ * - bit 20 is the NMI flag (no nesting count, tracked separately)
  *
  * The hardirq count could in theory be the same as the number of
  * interrupts in the system, but we run all interrupt handlers with
@@ -24,16 +26,19 @@
  * there are a few palaeontologic drivers which reenable interrupts in
  * the handler, so we need more than one bit here.
  *
+ * NMI nesting depth is tracked in a separate per-CPU variable
+ * (nmi_nesting) to save bits in preempt_count.
+ *
  *         PREEMPT_MASK:	0x000000ff
  *         SOFTIRQ_MASK:	0x0000ff00
  *         HARDIRQ_MASK:	0x000f0000
- *             NMI_MASK:	0x00f00000
+ *             NMI_MASK:	0x00100000
  * PREEMPT_NEED_RESCHED:	0x80000000
  */
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
 #define HARDIRQ_BITS	4
-#define NMI_BITS	4
+#define NMI_BITS	1
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
kernel/softirq.c +2 -0
index 4425d8dce44b..10af5ed859e7 100644
@@ -88,6 +88,8 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
+DEFINE_PER_CPU(unsigned int, nmi_nesting);
+
 /*
  * SOFTIRQ_OFFSET usage:
  *
tools/testing/selftests/bpf/bpf_experimental.h +1 -1
index 67ff7882299e..e4e12001fce9 100644
@@ -367,7 +367,7 @@ extern int bpf_cgroup_read_xattr(struct cgroup *cgroup, const char *name__str,
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
 #define HARDIRQ_BITS	4
-#define NMI_BITS	4
+#define NMI_BITS	1
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
Signature · 1 line
2.50.1 (Apple Git-155)
merge bot Applied
#2 lore raw

[tip: locking/core] preempt: Track NMI nesting to separate per-CPU counter

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     b54aa0edf0594a6e1138e626bfd8a352ed582ae9
Gitweb:        https://git.kernel.org/tip/b54aa0edf0594a6e1138e626bfd8a352ed582ae9
Author:        Joel Fernandes <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:23 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 08 Aug 2026 22:44:05 +02:00

preempt: Track NMI nesting to separate per-CPU counter

Move NMI nesting tracking from the preempt_count bits to a separate per-CPU counter (nmi_nesting). This is to free up the NMI bits in the preempt_count, allowing those bits to be repurposed for other uses.

Reduce NMI_BITS from 4 to 1, using it only to detect if we're in an NMI. The per-CPU counter currently caps nesting at 15.

[boqun: Address Steven Rostedt's comment on the BUG_ON() condition] [boqun: Use preempt_count_set() in __nmi_exit() to avoid underflow]

---
 include/linux/hardirq.h                        | 17 +++++++++++++----
 include/linux/preempt.h                        |  9 +++++++--
 kernel/softirq.c                               |  2 ++-
 tools/testing/selftests/bpf/bpf_experimental.h |  2 +-
 4 files changed, 23 insertions(+), 7 deletions(-)
include/linux/hardirq.h +13 -4
index d57cab4..8d48955 100644
@@ -10,6 +10,8 @@
 #include <linux/vtime.h>
 #include <asm/hardirq.h>
 
+DECLARE_PER_CPU(unsigned int, nmi_nesting);
+
 extern void synchronize_irq(unsigned int irq);
 extern bool synchronize_hardirq(unsigned int irq);
 
@@ -102,14 +104,17 @@ void irq_exit_rcu(void);
  */
 
 /*
- * nmi_enter() can nest up to 15 times; see NMI_BITS.
+ * nmi_enter() can nest - nesting is tracked in a per-CPU counter.
  */
 #define __nmi_enter()						\
 	do {							\
 		lockdep_off();					\
 		arch_nmi_enter();				\
-		BUG_ON(in_nmi() == NMI_MASK);			\
-		__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		/* Maximum NMI nesting is 15. */		\
+		BUG_ON(__this_cpu_read(nmi_nesting) >= 15);	\
+		__this_cpu_inc(nmi_nesting);			\
+		__preempt_count_add(HARDIRQ_OFFSET);		\
+		preempt_count_set(preempt_count() | NMI_MASK);	\
 	} while (0)
 
 #define nmi_enter()						\
@@ -124,8 +129,12 @@ void irq_exit_rcu(void);
 
 #define __nmi_exit()						\
 	do {							\
+		unsigned int nesting;				\
 		BUG_ON(!in_nmi());				\
-		__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);	\
+		__preempt_count_sub(HARDIRQ_OFFSET);		\
+		nesting = __this_cpu_dec_return(nmi_nesting);	\
+		if (!nesting)					\
+			preempt_count_set(preempt_count() & ~NMI_MASK);	\
 		arch_nmi_exit();				\
 		lockdep_on();					\
 	} while (0)
include/linux/preempt.h +7 -2
index d964f96..586f966 100644
@@ -17,6 +17,8 @@
  *
  * - bits 0-7 are the preemption count (max preemption depth: 256)
  * - bits 8-15 are the softirq count (max # of softirqs: 256)
+ * - bits 16-19 are the hardirq count (max # of hardirqs: 16)
+ * - bit 20 is the NMI flag (no nesting count, tracked separately)
  *
  * The hardirq count could in theory be the same as the number of
  * interrupts in the system, but we run all interrupt handlers with
@@ -24,16 +26,19 @@
  * there are a few palaeontologic drivers which reenable interrupts in
  * the handler, so we need more than one bit here.
  *
+ * NMI nesting depth is tracked in a separate per-CPU variable
+ * (nmi_nesting) to save bits in preempt_count.
+ *
  *         PREEMPT_MASK:	0x000000ff
  *         SOFTIRQ_MASK:	0x0000ff00
  *         HARDIRQ_MASK:	0x000f0000
- *             NMI_MASK:	0x00f00000
+ *             NMI_MASK:	0x00100000
  * PREEMPT_NEED_RESCHED:	0x80000000
  */
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
 #define HARDIRQ_BITS	4
-#define NMI_BITS	4
+#define NMI_BITS	1
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
kernel/softirq.c +2 -0
index 4425d8d..10af5ed 100644
@@ -88,6 +88,8 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
+DEFINE_PER_CPU(unsigned int, nmi_nesting);
+
 /*
  * SOFTIRQ_OFFSET usage:
  *
tools/testing/selftests/bpf/bpf_experimental.h +1 -1
index 67ff788..e4e1200 100644
@@ -367,7 +367,7 @@ extern int bpf_cgroup_read_xattr(struct cgroup *cgroup, const char *name__str,
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
 #define HARDIRQ_BITS	4
-#define NMI_BITS	4
+#define NMI_BITS	1
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
author maintainer
#3 lore raw

PATCH v4 2/17 preempt: Introduce HARDIRQ_DISABLE_BITS

In reply to Boqun Feng

In order to support preempt_disable()-like interrupt disabling, that is, using part of preempt_count() to track interrupt disabling nesting level, change the preempt_count() layout to contain 8-bit HARDIRQ_DISABLE count.

---
 include/linux/preempt.h                        | 16 +++++++++++-----
 tools/testing/selftests/bpf/bpf_experimental.h |  5 ++++-
 2 files changed, 15 insertions(+), 6 deletions(-)
include/linux/preempt.h +11 -5
index 586f96688325..e2d3079d3f5f 100644
@@ -17,8 +17,9 @@
  *
  * - bits 0-7 are the preemption count (max preemption depth: 256)
  * - bits 8-15 are the softirq count (max # of softirqs: 256)
- * - bits 16-19 are the hardirq count (max # of hardirqs: 16)
- * - bit 20 is the NMI flag (no nesting count, tracked separately)
+ * - bits 16-23 are the hardirq disable count (max # of hardirq disable: 256)
+ * - bits 24-27 are the hardirq count (max # of hardirqs: 16)
+ * - bit 28 is the NMI flag (no nesting count, tracked separately)
  *
  * The hardirq count could in theory be the same as the number of
  * interrupts in the system, but we run all interrupt handlers with
@@ -31,29 +32,34 @@
  *
  *         PREEMPT_MASK:	0x000000ff
  *         SOFTIRQ_MASK:	0x0000ff00
- *         HARDIRQ_MASK:	0x000f0000
- *             NMI_MASK:	0x00100000
+ * HARDIRQ_DISABLE_MASK:	0x00ff0000
+ *         HARDIRQ_MASK:	0x0f000000
+ *             NMI_MASK:	0x10000000
  * PREEMPT_NEED_RESCHED:	0x80000000
  */
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
+#define HARDIRQ_DISABLE_BITS	8
 #define HARDIRQ_BITS	4
 #define NMI_BITS	1
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
-#define HARDIRQ_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_DISABLE_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_SHIFT	(HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS)
 #define NMI_SHIFT	(HARDIRQ_SHIFT + HARDIRQ_BITS)
 
 #define __IRQ_MASK(x)	((1UL << (x))-1)
 
 #define PREEMPT_MASK	(__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)
 #define SOFTIRQ_MASK	(__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
+#define HARDIRQ_DISABLE_MASK	(__IRQ_MASK(HARDIRQ_DISABLE_BITS) << HARDIRQ_DISABLE_SHIFT)
 #define HARDIRQ_MASK	(__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
 #define NMI_MASK	(__IRQ_MASK(NMI_BITS)     << NMI_SHIFT)
 
 #define PREEMPT_OFFSET	(1UL << PREEMPT_SHIFT)
 #define SOFTIRQ_OFFSET	(1UL << SOFTIRQ_SHIFT)
+#define HARDIRQ_DISABLE_OFFSET	(1UL << HARDIRQ_DISABLE_SHIFT)
 #define HARDIRQ_OFFSET	(1UL << HARDIRQ_SHIFT)
 #define NMI_OFFSET	(1UL << NMI_SHIFT)
 
tools/testing/selftests/bpf/bpf_experimental.h +4 -1
index e4e12001fce9..0159a3d365c8 100644
@@ -366,17 +366,20 @@ extern int bpf_cgroup_read_xattr(struct cgroup *cgroup, const char *name__str,
 
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
+#define HARDIRQ_DISABLE_BITS	8
 #define HARDIRQ_BITS	4
 #define NMI_BITS	1
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
-#define HARDIRQ_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_DISABLE_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_SHIFT	(HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS)
 #define NMI_SHIFT	(HARDIRQ_SHIFT + HARDIRQ_BITS)
 
 #define __IRQ_MASK(x)	((1UL << (x))-1)
 
 #define SOFTIRQ_MASK	(__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
+#define HARDIRQ_DISABLE_MASK	(__IRQ_MASK(HARDIRQ_DISABLE_BITS) << HARDIRQ_DISABLE_SHIFT)
 #define HARDIRQ_MASK	(__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
 #define NMI_MASK	(__IRQ_MASK(NMI_BITS)     << NMI_SHIFT)
 
Signature · 1 line
2.50.1 (Apple Git-155)
maintainer
#4 lore raw

Re: [PATCH v4 02/17] preempt: Introduce HARDIRQ_DISABLE_BITS

Quoted 7 lines · Boqun Feng

In order to support preempt_disable()-like interrupt disabling, that is, using part of preempt_count() to track interrupt disabling nesting level, change the preempt_count() layout to contain 8-bit HARDIRQ_DISABLE count.

This still isn't right. Either it needs a: 'From: Lyude ....' or so Co-developed-by or however you spell that.

Since current from is Boqun, first SoB should be Boqun.

author maintainer
#5 lore raw

Re: [PATCH v4 02/17] preempt: Introduce HARDIRQ_DISABLE_BITS

Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Boqun Feng

In order to support preempt_disable()-like interrupt disabling, that is, using part of preempt_count() to track interrupt disabling nesting level, change the preempt_count() layout to contain 8-bit HARDIRQ_DISABLE count.

Quoted 7 lines
Quoted 2 lines

This still isn't right. Either it needs a: 'From: Lyude ....' or so Co-developed-by or however you spell that.

Since current from is Boqun, first SoB should be Boqun.

Got it, it happened when I switched mail addresses, so it used to be

From: Boqun Feng <[email protected]>

...

;-)

Will fix this.

Regards,
Boqun

merge bot Applied
#6 lore raw

[tip: locking/core] preempt: Introduce HARDIRQ_DISABLE_BITS

In reply to Boqun Feng
The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     9634d860ac15dad51a62955566424c9d2996b15c
Gitweb:        https://git.kernel.org/tip/9634d860ac15dad51a62955566424c9d2996b15c
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:24 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 08 Aug 2026 22:44:06 +02:00

preempt: Introduce HARDIRQ_DISABLE_BITS

In order to support preempt_disable()-like interrupt disabling, that is, using part of preempt_count() to track interrupt disabling nesting level, change the preempt_count() layout to contain 8-bit HARDIRQ_DISABLE count.

---
 include/linux/preempt.h                        | 16 +++++++++++-----
 tools/testing/selftests/bpf/bpf_experimental.h |  5 ++++-
 2 files changed, 15 insertions(+), 6 deletions(-)
include/linux/preempt.h +11 -5
index 586f966..e2d3079 100644
@@ -17,8 +17,9 @@
  *
  * - bits 0-7 are the preemption count (max preemption depth: 256)
  * - bits 8-15 are the softirq count (max # of softirqs: 256)
- * - bits 16-19 are the hardirq count (max # of hardirqs: 16)
- * - bit 20 is the NMI flag (no nesting count, tracked separately)
+ * - bits 16-23 are the hardirq disable count (max # of hardirq disable: 256)
+ * - bits 24-27 are the hardirq count (max # of hardirqs: 16)
+ * - bit 28 is the NMI flag (no nesting count, tracked separately)
  *
  * The hardirq count could in theory be the same as the number of
  * interrupts in the system, but we run all interrupt handlers with
@@ -31,29 +32,34 @@
  *
  *         PREEMPT_MASK:	0x000000ff
  *         SOFTIRQ_MASK:	0x0000ff00
- *         HARDIRQ_MASK:	0x000f0000
- *             NMI_MASK:	0x00100000
+ * HARDIRQ_DISABLE_MASK:	0x00ff0000
+ *         HARDIRQ_MASK:	0x0f000000
+ *             NMI_MASK:	0x10000000
  * PREEMPT_NEED_RESCHED:	0x80000000
  */
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
+#define HARDIRQ_DISABLE_BITS	8
 #define HARDIRQ_BITS	4
 #define NMI_BITS	1
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
-#define HARDIRQ_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_DISABLE_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_SHIFT	(HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS)
 #define NMI_SHIFT	(HARDIRQ_SHIFT + HARDIRQ_BITS)
 
 #define __IRQ_MASK(x)	((1UL << (x))-1)
 
 #define PREEMPT_MASK	(__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)
 #define SOFTIRQ_MASK	(__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
+#define HARDIRQ_DISABLE_MASK	(__IRQ_MASK(HARDIRQ_DISABLE_BITS) << HARDIRQ_DISABLE_SHIFT)
 #define HARDIRQ_MASK	(__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
 #define NMI_MASK	(__IRQ_MASK(NMI_BITS)     << NMI_SHIFT)
 
 #define PREEMPT_OFFSET	(1UL << PREEMPT_SHIFT)
 #define SOFTIRQ_OFFSET	(1UL << SOFTIRQ_SHIFT)
+#define HARDIRQ_DISABLE_OFFSET	(1UL << HARDIRQ_DISABLE_SHIFT)
 #define HARDIRQ_OFFSET	(1UL << HARDIRQ_SHIFT)
 #define NMI_OFFSET	(1UL << NMI_SHIFT)
 
tools/testing/selftests/bpf/bpf_experimental.h +4 -1
index e4e1200..0159a3d 100644
@@ -366,17 +366,20 @@ extern int bpf_cgroup_read_xattr(struct cgroup *cgroup, const char *name__str,
 
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
+#define HARDIRQ_DISABLE_BITS	8
 #define HARDIRQ_BITS	4
 #define NMI_BITS	1
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
-#define HARDIRQ_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_DISABLE_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
+#define HARDIRQ_SHIFT	(HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS)
 #define NMI_SHIFT	(HARDIRQ_SHIFT + HARDIRQ_BITS)
 
 #define __IRQ_MASK(x)	((1UL << (x))-1)
 
 #define SOFTIRQ_MASK	(__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
+#define HARDIRQ_DISABLE_MASK	(__IRQ_MASK(HARDIRQ_DISABLE_BITS) << HARDIRQ_DISABLE_SHIFT)
 #define HARDIRQ_MASK	(__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
 #define NMI_MASK	(__IRQ_MASK(NMI_BITS)     << NMI_SHIFT)
 
author maintainer
#7 lore raw

PATCH v4 3/17 preempt: Introduce __preempt_count_{sub,add}_return()

In reply to Boqun Feng

In order to use preempt_count() to track the interrupt disable nesting level, __preempt_count_{add,sub}_return() are introduced, as their names suggest, these primitives return the new value of the preempt_count() after changing it. The following example shows the usage of it in local_interrupt_disable():

	// increase the HARDIRQ_DISABLE bit
	new_count = __preempt_count_add_return(HARDIRQ_DISABLE_OFFSET);

	// if it's the first-time increment, then disable the interrupt
	// at hardware level.
	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET) {
		local_irq_save(flags);
		raw_cpu_write(local_interrupt_disable_state, flags);
	}

Having these primitives will avoid a read of preempt_count() after changing preempt_count() on certain architectures.

---
 arch/arm64/include/asm/preempt.h | 20 ++++++++++++++++++++
 arch/s390/include/asm/preempt.h  | 10 ++++++++++
 arch/x86/include/asm/preempt.h   | 10 ++++++++++
 include/asm-generic/preempt.h    | 14 ++++++++++++++
 4 files changed, 54 insertions(+)
arch/arm64/include/asm/preempt.h +20 -0
index 932ea4b62042..9ecc2766a9f2 100644
@@ -55,6 +55,26 @@ static inline void __preempt_count_sub(int val)
 	WRITE_ONCE(current_thread_info()->preempt.count, pc);
 }
 
+static inline int __preempt_count_add_return(int val)
+{
+	u32 pc = READ_ONCE(current_thread_info()->preempt.count);
+
+	pc += val;
+	WRITE_ONCE(current_thread_info()->preempt.count, pc);
+
+	return pc;
+}
+
+static inline int __preempt_count_sub_return(int val)
+{
+	u32 pc = READ_ONCE(current_thread_info()->preempt.count);
+
+	pc -= val;
+	WRITE_ONCE(current_thread_info()->preempt.count, pc);
+
+	return pc;
+}
+
 static inline bool __preempt_count_dec_and_test(void)
 {
 	struct thread_info *ti = current_thread_info();
arch/s390/include/asm/preempt.h +10 -0
index 6e5821bb047e..0a25d4648b4c 100644
@@ -139,6 +139,16 @@ static __always_inline bool should_resched(int preempt_offset)
 	return unlikely(READ_ONCE(get_lowcore()->preempt_count) == preempt_offset);
 }
 
+static __always_inline int __preempt_count_add_return(int val)
+{
+	return val + __atomic_add(val, &get_lowcore()->preempt_count);
+}
+
+static __always_inline int __preempt_count_sub_return(int val)
+{
+	return __preempt_count_add_return(-val);
+}
+
 #define init_task_preempt_count(p)	do { } while (0)
 /* Deferred to CPU bringup time */
 #define init_idle_preempt_count(p, cpu)	do { } while (0)
arch/x86/include/asm/preempt.h +10 -0
index 578441db09f0..1220656f3370 100644
@@ -85,6 +85,16 @@ static __always_inline void __preempt_count_sub(int val)
 	raw_cpu_add_4(__preempt_count, -val);
 }
 
+static __always_inline int __preempt_count_add_return(int val)
+{
+	return raw_cpu_add_return_4(__preempt_count, val);
+}
+
+static __always_inline int __preempt_count_sub_return(int val)
+{
+	return raw_cpu_add_return_4(__preempt_count, -val);
+}
+
 /*
  * Because we keep PREEMPT_NEED_RESCHED set when we do _not_ need to reschedule
  * a decrement which hits zero means we have no preempt_count and should
include/asm-generic/preempt.h +14 -0
index 51f8f3881523..c8683c046615 100644
@@ -59,6 +59,20 @@ static __always_inline void __preempt_count_sub(int val)
 	*preempt_count_ptr() -= val;
 }
 
+static __always_inline int __preempt_count_add_return(int val)
+{
+	*preempt_count_ptr() += val;
+
+	return *preempt_count_ptr();
+}
+
+static __always_inline int __preempt_count_sub_return(int val)
+{
+	*preempt_count_ptr() -= val;
+
+	return *preempt_count_ptr();
+}
+
 static __always_inline bool __preempt_count_dec_and_test(void)
 {
 	/*
Signature · 1 line
2.50.1 (Apple Git-155)
merge bot Applied
#8 lore raw

[tip: locking/core] preempt: Introduce __preempt_count_{sub,add}_return()

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     4ad87eee546ef97e3a5ad80414327f41f48a4e70
Gitweb:        https://git.kernel.org/tip/4ad87eee546ef97e3a5ad80414327f41f48a4e70
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:25 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 08 Aug 2026 22:44:06 +02:00

preempt: Introduce __preempt_count_{sub,add}_return()

In order to use preempt_count() to track the interrupt disable nesting level, __preempt_count_{add,sub}_return() are introduced, as their names suggest, these primitives return the new value of the preempt_count() after changing it. The following example shows the usage of it in local_interrupt_disable():

	// increase the HARDIRQ_DISABLE bit
	new_count = __preempt_count_add_return(HARDIRQ_DISABLE_OFFSET);

	// if it's the first-time increment, then disable the interrupt
	// at hardware level.
	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET) {
		local_irq_save(flags);
		raw_cpu_write(local_interrupt_disable_state, flags);
	}

Having these primitives will avoid a read of preempt_count() after changing preempt_count() on certain architectures.

---
 arch/arm64/include/asm/preempt.h | 20 ++++++++++++++++++++
 arch/s390/include/asm/preempt.h  | 10 ++++++++++
 arch/x86/include/asm/preempt.h   | 10 ++++++++++
 include/asm-generic/preempt.h    | 14 ++++++++++++++
 4 files changed, 54 insertions(+)
arch/arm64/include/asm/preempt.h +20 -0
index 932ea4b..9ecc276 100644
@@ -55,6 +55,26 @@ static inline void __preempt_count_sub(int val)
 	WRITE_ONCE(current_thread_info()->preempt.count, pc);
 }
 
+static inline int __preempt_count_add_return(int val)
+{
+	u32 pc = READ_ONCE(current_thread_info()->preempt.count);
+
+	pc += val;
+	WRITE_ONCE(current_thread_info()->preempt.count, pc);
+
+	return pc;
+}
+
+static inline int __preempt_count_sub_return(int val)
+{
+	u32 pc = READ_ONCE(current_thread_info()->preempt.count);
+
+	pc -= val;
+	WRITE_ONCE(current_thread_info()->preempt.count, pc);
+
+	return pc;
+}
+
 static inline bool __preempt_count_dec_and_test(void)
 {
 	struct thread_info *ti = current_thread_info();
arch/s390/include/asm/preempt.h +10 -0
index 6e5821b..0a25d46 100644
@@ -139,6 +139,16 @@ static __always_inline bool should_resched(int preempt_offset)
 	return unlikely(READ_ONCE(get_lowcore()->preempt_count) == preempt_offset);
 }
 
+static __always_inline int __preempt_count_add_return(int val)
+{
+	return val + __atomic_add(val, &get_lowcore()->preempt_count);
+}
+
+static __always_inline int __preempt_count_sub_return(int val)
+{
+	return __preempt_count_add_return(-val);
+}
+
 #define init_task_preempt_count(p)	do { } while (0)
 /* Deferred to CPU bringup time */
 #define init_idle_preempt_count(p, cpu)	do { } while (0)
arch/x86/include/asm/preempt.h +10 -0
index 578441d..1220656 100644
@@ -85,6 +85,16 @@ static __always_inline void __preempt_count_sub(int val)
 	raw_cpu_add_4(__preempt_count, -val);
 }
 
+static __always_inline int __preempt_count_add_return(int val)
+{
+	return raw_cpu_add_return_4(__preempt_count, val);
+}
+
+static __always_inline int __preempt_count_sub_return(int val)
+{
+	return raw_cpu_add_return_4(__preempt_count, -val);
+}
+
 /*
  * Because we keep PREEMPT_NEED_RESCHED set when we do _not_ need to reschedule
  * a decrement which hits zero means we have no preempt_count and should
include/asm-generic/preempt.h +14 -0
index 51f8f38..c8683c0 100644
@@ -59,6 +59,20 @@ static __always_inline void __preempt_count_sub(int val)
 	*preempt_count_ptr() -= val;
 }
 
+static __always_inline int __preempt_count_add_return(int val)
+{
+	*preempt_count_ptr() += val;
+
+	return *preempt_count_ptr();
+}
+
+static __always_inline int __preempt_count_sub_return(int val)
+{
+	*preempt_count_ptr() -= val;
+
+	return *preempt_count_ptr();
+}
+
 static __always_inline bool __preempt_count_dec_and_test(void)
 {
 	/*
author maintainer
#9 lore raw

PATCH v4 4/17 openrisc: Include <linux/cpumask.h> in smp.h

In reply to Boqun Feng

From: Lyude Paul <[email protected]>

While OpenRISC currently doesn't fail to build upstream, it appears that including <asm/smp.h> in the right headers is enough to break that - primarily because OpenRISC's asm/smp.h header doesn't actually provide any definition for struct cpumask. Which means the only reason we aren't failing to build the kernel is because we've been lucky enough that every spot including asm/smp.h already has definitions for struct cpumask pulled in.

This became evident when trying to work on a patch series for adding ref-counted interrupt enable/disable to the kernel, where introducing a new interrupt_rc.h header suddenly introduced a build error on OpenRISC:

In file included from include/linux/interrupt_rc.h:17,
from include/linux/spinlock.h:60,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:7,
from include/linux/mm.h:7,
from arch/openrisc/include/asm/pgalloc.h:20,
from arch/openrisc/include/asm/io.h:18,
from include/linux/io.h:12,
from drivers/irqchip/irq-ompic.c:61:
arch/openrisc/include/asm/smp.h:21:59: warning: 'struct cpumask' declared inside parameter list will not be visible outside of this definition or declaration
21 | extern void arch_send_call_function_ipi_mask(const struct cpumask *mask);
| ^~~~~~~
arch/openrisc/include/asm/smp.h:23:54: warning: 'struct cpumask' declared inside parameter list will not be visible outside of this definition or declaration
23 | extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));
| ^~~~~~~
drivers/irqchip/irq-ompic.c: In function 'ompic_of_init':

Quoted 1 line
Quoted 1 line

drivers/irqchip/irq-ompic.c:191:28: error: passing argument 1 of

'set_smp_cross_call' from incompatible pointer type [-Werror=incompatible-pointer-types]
191 | set_smp_cross_call(ompic_raise_softirq);
| ^~~~~~~~~~~~~~~~~~~
| |
| void (*)(const struct cpumask *, unsigned int)
arch/openrisc/include/asm/smp.h:23:32: note: expected 'void (*)(const struct cpumask *, unsigned int)' but argument is of type 'void
(*)(const struct cpumask *, unsigned int)'
23 | extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));

To fix this, let's take an example from the smp.h headers of other architectures (x86, hexagon, arm64, probably more): just include linux/cpumask.h at the top.

---
 arch/openrisc/include/asm/smp.h | 2 ++
 1 file changed, 2 insertions(+)
arch/openrisc/include/asm/smp.h +2 -0
index 007296f160ef..84653aaffa96 100644
@@ -9,6 +9,8 @@
 #ifndef __ASM_OPENRISC_SMP_H
 #define __ASM_OPENRISC_SMP_H
 
+#include <linux/cpumask.h>
+
 #include <asm/spr.h>
 #include <asm/spr_defs.h>
 
Signature · 1 line
2.50.1 (Apple Git-155)
merge bot Applied
#10 lore raw

[tip: locking/core] openrisc: Include <linux/cpumask.h> in smp.h

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     e35e7700029b936356d736d6378cf596019e4cc8
Gitweb:        https://git.kernel.org/tip/e35e7700029b936356d736d6378cf596019e4cc8
Author:        Lyude Paul <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:26 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 08 Aug 2026 22:44:06 +02:00

openrisc: Include <linux/cpumask.h> in smp.h

While OpenRISC currently doesn't fail to build upstream, it appears that including <asm/smp.h> in the right headers is enough to break that - primarily because OpenRISC's asm/smp.h header doesn't actually provide any definition for struct cpumask. Which means the only reason we aren't failing to build the kernel is because we've been lucky enough that every spot including asm/smp.h already has definitions for struct cpumask pulled in.

This became evident when trying to work on a patch series for adding ref-counted interrupt enable/disable to the kernel, where introducing a new interrupt_rc.h header suddenly introduced a build error on OpenRISC:

In file included from include/linux/interrupt_rc.h:17,
from include/linux/spinlock.h:60,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:7,
from include/linux/mm.h:7,
from arch/openrisc/include/asm/pgalloc.h:20,
from arch/openrisc/include/asm/io.h:18,
from include/linux/io.h:12,
from drivers/irqchip/irq-ompic.c:61:
arch/openrisc/include/asm/smp.h:21:59: warning: 'struct cpumask' declared inside parameter list will not be visible outside of this definition or declaration
21 | extern void arch_send_call_function_ipi_mask(const struct cpumask *mask);
| ^~~~~~~
arch/openrisc/include/asm/smp.h:23:54: warning: 'struct cpumask' declared inside parameter list will not be visible outside of this definition or declaration
23 | extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));
| ^~~~~~~
drivers/irqchip/irq-ompic.c: In function 'ompic_of_init':

Quoted 1 line
Quoted 1 line

drivers/irqchip/irq-ompic.c:191:28: error: passing argument 1 of

'set_smp_cross_call' from incompatible pointer type [-Werror=incompatible-pointer-types]
191 | set_smp_cross_call(ompic_raise_softirq);
| ^~~~~~~~~~~~~~~~~~~
| |
| void (*)(const struct cpumask *, unsigned int)
arch/openrisc/include/asm/smp.h:23:32: note: expected 'void (*)(const struct cpumask *, unsigned int)' but argument is of type 'void
(*)(const struct cpumask *, unsigned int)'
23 | extern void set_smp_cross_call(void (*)(const struct cpumask *, unsigned int));

To fix this, let's take an example from the smp.h headers of other architectures (x86, hexagon, arm64, probably more): just include linux/cpumask.h at the top.

---
 arch/openrisc/include/asm/smp.h | 2 ++
 1 file changed, 2 insertions(+)
arch/openrisc/include/asm/smp.h +2 -0
index 007296f..84653aa 100644
@@ -9,6 +9,8 @@
 #ifndef __ASM_OPENRISC_SMP_H
 #define __ASM_OPENRISC_SMP_H
 
+#include <linux/cpumask.h>
+
 #include <asm/spr.h>
 #include <asm/spr_defs.h>
 
author maintainer
#11 lore raw

PATCH v4 5/17 irq & spin_lock: Add counted interrupt disabling/enabling

In reply to Boqun Feng

Currently the nested interrupt disabling and enabling is represented by _irqsave() and _irqrestore() APIs, which are relatively unsafe, for example:

	<interrupts are enabled as beginning>
	spin_lock_irqsave(l1, flag1);
	spin_lock_irqsave(l2, flag2);
	spin_unlock_irqrestore(l1, flags1);
	<l2 is still held but interrupts are enabled>
	// accesses to interrupt-disable protected data will cause races

This is even easier to trigger with guard facilities:

	unsigned long flag2;

	scoped_guard(spin_lock_irqsave, l1) {
		spin_lock_irqsave(l2, flag2);
	}
	// l2 locked but interrupts are enabled.
	spin_unlock_irqrestore(l2, flag2);

(Hand-to-hand locking critical sections are not uncommon for a fine-grained lock design)

And because of this unsafety, Rust cannot easily wrap the interrupt-disabling locks in a safe API, which complicates the design.

To resolve this, introduce a new set of interrupt disabling APIs:

* local_interrupt_disable();
* local_interrupt_enable();

They work like local_irq_save() and local_irq_restore() except that 1) the outermost local_interrupt_disable() call saves the interrupt state into a per-CPU variable, so that the outermost local_interrupt_enable() can restore the state, and 2) a per-CPU counter is added to record the nest level of these calls, so that interrupts are not accidentally enabled inside the outermost critical section.

Also add the corresponding spin_lock primitives: spin_lock_irq_disable() and spin_unlock_irq_enable(), as a result, code as follows:

	spin_lock_irq_disable(l1);
	spin_lock_irq_disable(l2);
	spin_unlock_irq_enable(l1);
	// Interrupts are still disabled.
	spin_unlock_irq_enable(l2);

doesn't have the issue that interrupts are accidentally enabled.

This also makes the wrapper of interrupt-disabling locks on Rust easier to design.

[boqun: Apply Peter's feedback and fix spell errors reported by Ingo]
---
 include/linux/interrupt_rc.h     | 82 ++++++++++++++++++++++++++++++++
 include/linux/preempt.h          |  4 ++
 include/linux/spinlock.h         | 23 +++++++++
 include/linux/spinlock_api_smp.h | 43 +++++++++++++++++
 include/linux/spinlock_api_up.h  | 15 ++++++
 include/linux/spinlock_rt.h      | 18 +++++++
 kernel/locking/spinlock.c        | 31 ++++++++++++
 kernel/softirq.c                 | 28 ++++++++++-
 8 files changed, 242 insertions(+), 2 deletions(-)
 create mode 100644 include/linux/interrupt_rc.h
include/linux/interrupt_rc.h new file +82 -0
new file mode 100644
index 000000000000..b9a7f05ecf42
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __LINUX_INTERRUPT_RC_H
+#define __LINUX_INTERRUPT_RC_H
+
+/*
+ * include/linux/interrupt_rc.h - refcounted local processor interrupt
+ * management.
+ *
+ * Since the implementation of this API currently depends on
+ * local_irq_save()/local_irq_restore(), we split this into its own header to
+ * make it easier to include without hitting circular header dependencies.
+ */
+
+#include <linux/irqflags.h>
+#include <linux/preempt.h>
+#include <linux/processor.h>
+#include <linux/smp.h>
+
+#ifndef MODULE
+/* Per-CPU interrupt disabling state for local_interrupt_{disable,enable}(). */
+DECLARE_PER_CPU(unsigned long, local_interrupt_disable_state);
+
+static __always_inline void __local_interrupt_disable(void)
+{
+	unsigned long flags;
+
+	local_irq_save(flags);
+	raw_cpu_write(local_interrupt_disable_state, flags);
+}
+
+static __always_inline void __local_interrupt_enable(void)
+{
+	unsigned long flags = raw_cpu_read(local_interrupt_disable_state);
+
+	local_irq_restore(flags);
+}
+
+#ifndef INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
+static __always_inline void _local_interrupt_disable(void)
+{
+	__local_interrupt_disable();
+}
+
+static __always_inline void _local_interrupt_enable(void)
+{
+	__local_interrupt_enable();
+}
+#else
+extern void _local_interrupt_disable(void);
+extern void _local_interrupt_enable(void);
+#endif
+
+#else /* !MODULE */
+extern void _local_interrupt_disable(void);
+extern void _local_interrupt_enable(void);
+#endif /* !MODULE */
+
+static inline void local_interrupt_disable(void)
+{
+	int new_count;
+
+	WARN_ON_ONCE(in_nmi());
+
+	new_count = hardirq_disable_enter();
+
+	/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
+		_local_interrupt_disable();
+}
+
+static inline void local_interrupt_enable(void)
+{
+	int new_count;
+
+	new_count = hardirq_disable_exit();
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == 0)
+		_local_interrupt_enable();
+}
+
+#endif /* !__LINUX_INTERRUPT_RC_H */
include/linux/preempt.h +4 -0
index e2d3079d3f5f..33fc4c814a9f 100644
@@ -151,6 +151,10 @@ static __always_inline unsigned char interrupt_context_level(void)
 #define in_softirq()		(softirq_count())
 #define in_interrupt()		(irq_count())
 
+#define hardirq_disable_count()	((preempt_count() & HARDIRQ_DISABLE_MASK) >> HARDIRQ_DISABLE_SHIFT)
+#define hardirq_disable_enter()	__preempt_count_add_return(HARDIRQ_DISABLE_OFFSET)
+#define hardirq_disable_exit()	__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET)
+
 /*
  * The preempt_count offset after preempt_disable();
  */
include/linux/spinlock.h +23 -0
index 241277cd34cf..3d405cc4c121 100644
@@ -57,6 +57,7 @@
 #include <linux/linkage.h>
 #include <linux/compiler.h>
 #include <linux/irqflags.h>
+#include <linux/interrupt_rc.h>
 #include <linux/thread_info.h>
 #include <linux/stringify.h>
 #include <linux/bottom_half.h>
@@ -273,9 +274,11 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
 #endif
 
 #define raw_spin_lock_irq(lock)		_raw_spin_lock_irq(lock)
+#define raw_spin_lock_irq_disable(lock)	_raw_spin_lock_irq_disable(lock)
 #define raw_spin_lock_bh(lock)		_raw_spin_lock_bh(lock)
 #define raw_spin_unlock(lock)		_raw_spin_unlock(lock)
 #define raw_spin_unlock_irq(lock)	_raw_spin_unlock_irq(lock)
+#define raw_spin_unlock_irq_enable(lock)	_raw_spin_unlock_irq_enable(lock)
 
 #define raw_spin_unlock_irqrestore(lock, flags)		\
 	do {							\
@@ -290,6 +293,8 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
 
 #define raw_spin_trylock_irqsave(lock, flags) _raw_spin_trylock_irqsave(lock, &(flags))
 
+#define raw_spin_trylock_irq_disable(lock)	_raw_spin_trylock_irq_disable(lock)
+
 #ifndef CONFIG_PREEMPT_RT
 /* Include rwlock functions for !RT */
 #include <linux/rwlock.h>
@@ -372,6 +377,12 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)
 	raw_spin_lock_irq(&lock->rlock);
 }
 
+static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
+	__acquires(lock) __no_context_analysis
+{
+	raw_spin_lock_irq_disable(&lock->rlock);
+}
+
 #define spin_lock_irqsave(lock, flags)				\
 do {								\
 	raw_spin_lock_irqsave(spinlock_check(lock), flags);	\
@@ -402,6 +413,12 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)
 	raw_spin_unlock_irq(&lock->rlock);
 }
 
+static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
+	__releases(lock) __no_context_analysis
+{
+	raw_spin_unlock_irq_enable(&lock->rlock);
+}
+
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags)
 	__releases(lock) __no_context_analysis
 {
@@ -427,6 +444,12 @@ static __always_inline bool _spin_trylock_irqsave(spinlock_t *lock, unsigned lon
 }
 #define spin_trylock_irqsave(lock, flags) _spin_trylock_irqsave(lock, &(flags))
 
+static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
+	__cond_acquires(true, lock) __no_context_analysis
+{
+	return raw_spin_trylock_irq_disable(&lock->rlock);
+}
+
 /**
  * spin_is_locked() - Check whether a spinlock is locked.
  * @lock: Pointer to the spinlock.
include/linux/spinlock_api_smp.h +43 -0

85 lines not on this page · raw message

include/linux/spinlock_api_up.h +15 -0
index a9d5c7c66e03..d03d3065ee04 100644
@@ -42,6 +42,9 @@
 #define __LOCK_IRQSAVE(lock, flags, ...) \
   do { local_irq_save(flags); __LOCK(lock, ##__VA_ARGS__); } while (0)
 
+#define __LOCK_IRQ_DISABLE(lock, ...) \
+  do { local_interrupt_disable(); __LOCK(lock, ##__VA_ARGS__); } while (0)
+
 #define ___UNLOCK_(lock) \
   do { __release(lock); (void)(lock); } while (0)
 
@@ -61,6 +64,9 @@
 #define __UNLOCK_IRQRESTORE(lock, flags, ...) \
   do { local_irq_restore(flags); __UNLOCK(lock, ##__VA_ARGS__); } while (0)
 
+#define __UNLOCK_IRQ_ENABLE(lock, ...) \
+  do { __UNLOCK(lock, ##__VA_ARGS__); local_interrupt_enable(); } while (0)
+
 #define _raw_spin_lock(lock)			__LOCK(lock)
 #define _raw_spin_lock_nested(lock, subclass)	__LOCK(lock)
 #define _raw_read_lock(lock)			__LOCK(lock, shared)
@@ -70,6 +76,7 @@
 #define _raw_read_lock_bh(lock)			__LOCK_BH(lock, shared)
 #define _raw_write_lock_bh(lock)		__LOCK_BH(lock)
 #define _raw_spin_lock_irq(lock)		__LOCK_IRQ(lock)
+#define _raw_spin_lock_irq_disable(lock)	__LOCK_IRQ_DISABLE(lock)
 #define _raw_read_lock_irq(lock)		__LOCK_IRQ(lock, shared)
 #define _raw_write_lock_irq(lock)		__LOCK_IRQ(lock)
 #define _raw_spin_lock_irqsave(lock, flags)	__LOCK_IRQSAVE(lock, flags)
@@ -97,6 +104,13 @@ static __always_inline int _raw_spin_trylock_irq(raw_spinlock_t *lock)
 	return 1;
 }
 
+static __always_inline int _raw_spin_trylock_irq_disable(raw_spinlock_t *lock)
+	__cond_acquires(true, lock)
+{
+	__LOCK_IRQ_DISABLE(lock);
+	return 1;
+}
+
 static __always_inline int _raw_spin_trylock_irqsave(raw_spinlock_t *lock, unsigned long *flags)
 	__cond_acquires(true, lock)
 {
@@ -132,6 +146,7 @@ static __always_inline int _raw_write_trylock_irqsave(rwlock_t *lock, unsigned l
 #define _raw_write_unlock_bh(lock)		__UNLOCK_BH(lock)
 #define _raw_read_unlock_bh(lock)		__UNLOCK_BH(lock, shared)
 #define _raw_spin_unlock_irq(lock)		__UNLOCK_IRQ(lock)
+#define _raw_spin_unlock_irq_enable(lock)	__UNLOCK_IRQ_ENABLE(lock)
 #define _raw_read_unlock_irq(lock)		__UNLOCK_IRQ(lock, shared)
 #define _raw_write_unlock_irq(lock)		__UNLOCK_IRQ(lock)
 #define _raw_spin_unlock_irqrestore(lock, flags) \
include/linux/spinlock_rt.h +18 -0
index 373618a4243c..560d06384e0c 100644
@@ -96,6 +96,12 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)
 	rt_spin_lock(lock);
 }
 
+static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
+	__acquires(lock)
+{
+	rt_spin_lock(lock);
+}
+
 #define spin_lock_irqsave(lock, flags)			 \
 	do {						 \
 		typecheck(unsigned long, flags);	 \
@@ -122,6 +128,12 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)
 	rt_spin_unlock(lock);
 }
 
+static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
+	__releases(lock)
+{
+	rt_spin_unlock(lock);
+}
+
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,
 						   unsigned long flags)
 	__releases(lock)
@@ -131,6 +143,12 @@ static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,
 
 #define spin_trylock(lock)	rt_spin_trylock(lock)
 
+static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
+	__cond_acquires(true, lock)
+{
+	return rt_spin_trylock(lock);
+}
+
 #define spin_trylock_bh(lock)	rt_spin_trylock_bh(lock)
 
 #define spin_trylock_irq(lock)	rt_spin_trylock(lock)
kernel/locking/spinlock.c +31 -0
index b42d293da38b..83a17eaf5717 100644
@@ -129,6 +129,21 @@ static void __lockfunc __raw_##op##_lock_bh(locktype##_t *lock)		\
  */
 BUILD_LOCK_OPS(spin, raw_spinlock, __acquires);
 
+/* No rwlock_t variants for now, so just build this function by hand */
+static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	for (;;) {
+		preempt_disable();
+		local_interrupt_disable();
+		if (likely(do_raw_spin_trylock(lock)))
+			break;
+		local_interrupt_enable();
+		preempt_enable();
+
+		arch_spin_relax(&lock->raw_lock);
+	}
+}
+
 #ifndef CONFIG_PREEMPT_RT
 BUILD_LOCK_OPS(read, rwlock, __acquires_shared);
 BUILD_LOCK_OPS(write, rwlock, __acquires);
@@ -176,6 +191,14 @@ noinline void __lockfunc _raw_spin_lock_irq(raw_spinlock_t *lock)
 EXPORT_SYMBOL(_raw_spin_lock_irq);
 #endif
 
+#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ
+noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	__raw_spin_lock_irq_disable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable);
+#endif
+
 #ifndef CONFIG_INLINE_SPIN_LOCK_BH
 noinline void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock)
 {
@@ -208,6 +231,14 @@ noinline void __lockfunc _raw_spin_unlock_irq(raw_spinlock_t *lock)
 EXPORT_SYMBOL(_raw_spin_unlock_irq);
 #endif
 
+#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ
+noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
+{
+	__raw_spin_unlock_irq_enable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable);
+#endif
+
 #ifndef CONFIG_INLINE_SPIN_UNLOCK_BH
 noinline void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock)
 {
kernel/softirq.c +26 -2
index 10af5ed859e7..0c9b2269a8d6 100644
@@ -9,6 +9,7 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
 #include <linux/export.h>
 #include <linux/kernel_stat.h>
 #include <linux/interrupt.h>
@@ -88,6 +89,20 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
+DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state);
+
+void _local_interrupt_disable(void)
+{
+	__local_interrupt_disable();
+}
+EXPORT_SYMBOL(_local_interrupt_disable);
+
+void _local_interrupt_enable(void)
+{
+	__local_interrupt_enable();
+}
+EXPORT_SYMBOL(_local_interrupt_enable);
+
 DEFINE_PER_CPU(unsigned int, nmi_nesting);
 
 /*
@@ -728,10 +743,19 @@ static inline void __irq_exit_rcu(void)
 #endif
 	account_hardirq_exit(current);
 	preempt_count_sub(HARDIRQ_OFFSET);
-	if (!in_interrupt() && local_softirq_pending()) {
+	/*
+	 * Interrupts may happen between hardirq_disable_enter() and
+	 * local_irq_save() in local_interrupt_disable(), if irq_exit() invokes
+	 * softirq here, we may have a softirq handler calling
+	 * local_interrupt_disable() but it won't disable the IRQ because
+	 * hardirq disabling count is already 1, hence we need to prevent
+	 * invoking softirq when a local_interrupt_disable() is ongoing.
+	 */
+	if (!in_interrupt() && !hardirq_disable_count() &&
+	    local_softirq_pending()) {
 		/*
 		 * If we left hrtimers unarmed, make sure to arm them now,
-		 * before enabling interrupts to run SoftIRQ.
+		 * before enabling interrupts to run softirq.
 		 */
 		hrtimer_rearm_deferred();
 		invoke_softirq();
Signature · 1 line
2.50.1 (Apple Git-155)
author maintainer
#12 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

On Tue, Aug 04, 2026 at 09:14:27AM -0700, Boqun Feng wrote:
[...]

include/linux/spinlock_api_smp.h static __always_inline bool _raw_spin_trylock_irq(raw_spinlock_t *lock) · line 126

Quoted 6 lines
include/linux/spinlock_api_smp.h +6 -0
+static __always_inline bool _raw_spin_trylock_irq_disable(raw_spinlock_t *lock)
+	__cond_acquires(true, lock)
+{
+	local_interrupt_disable();
+	if (_raw_spin_trylock(lock)) {
+		spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);

This spin_acquire() needs to be removed as spotted by sashiko.

Regards,
Boqun

Quoted 6 lines
include/linux/spinlock_api_smp.h +6 -0
+		return true;
+	}
+	local_interrupt_enable();
+	return false;
+}
+

[...]

author maintainer
#13 lore raw

PATCH v4 5/17 irq & spin_lock: Add counted interrupt disabling/enabling

In reply to Boqun Feng

Currently the nested interrupt disabling and enabling is represented by _irqsave() and _irqrestore() APIs, which are relatively unsafe, for example:

	<interrupts are enabled as beginning>
	spin_lock_irqsave(l1, flag1);
	spin_lock_irqsave(l2, flag2);
	spin_unlock_irqrestore(l1, flags1);
	<l2 is still held but interrupts are enabled>
	// accesses to interrupt-disable protected data will cause races

This is even easier to trigger with guard facilities:

	unsigned long flag2;

	scoped_guard(spin_lock_irqsave, l1) {
		spin_lock_irqsave(l2, flag2);
	}
	// l2 locked but interrupts are enabled.
	spin_unlock_irqrestore(l2, flag2);

(Hand-to-hand locking critical sections are not uncommon for a fine-grained lock design)

And because of this unsafety, Rust cannot easily wrap the interrupt-disabling locks in a safe API, which complicates the design.

To resolve this, introduce a new set of interrupt disabling APIs:

* local_interrupt_disable();
* local_interrupt_enable();

They work like local_irq_save() and local_irq_restore() except that 1) the outermost local_interrupt_disable() call saves the interrupt state into a per-CPU variable, so that the outermost local_interrupt_enable() can restore the state, and 2) a per-CPU counter is added to record the nest level of these calls, so that interrupts are not accidentally enabled inside the outermost critical section.

Also add the corresponding spin_lock primitives: spin_lock_irq_disable() and spin_unlock_irq_enable(), as a result, code as follows:

	spin_lock_irq_disable(l1);
	spin_lock_irq_disable(l2);
	spin_unlock_irq_enable(l1);
	// Interrupts are still disabled.
	spin_unlock_irq_enable(l2);

doesn't have the issue that interrupts are accidentally enabled.

This also makes the wrapper of interrupt-disabling locks on Rust easier to design.

[boqun: Apply Peter's feedback and fix spell errors reported by Ingo]
[boqun: Address the duplicate spin_acquire() spotted by sashiko]
---
 include/linux/interrupt_rc.h     | 82 ++++++++++++++++++++++++++++++++
 include/linux/preempt.h          |  4 ++
 include/linux/spinlock.h         | 23 +++++++++
 include/linux/spinlock_api_smp.h | 41 ++++++++++++++++
 include/linux/spinlock_api_up.h  | 15 ++++++
 include/linux/spinlock_rt.h      | 18 +++++++
 kernel/locking/spinlock.c        | 31 ++++++++++++
 kernel/softirq.c                 | 28 ++++++++++-
 8 files changed, 240 insertions(+), 2 deletions(-)
 create mode 100644 include/linux/interrupt_rc.h
include/linux/interrupt_rc.h new file +82 -0
new file mode 100644
index 000000000000..b9a7f05ecf42
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __LINUX_INTERRUPT_RC_H
+#define __LINUX_INTERRUPT_RC_H
+
+/*
+ * include/linux/interrupt_rc.h - refcounted local processor interrupt
+ * management.
+ *
+ * Since the implementation of this API currently depends on
+ * local_irq_save()/local_irq_restore(), we split this into its own header to
+ * make it easier to include without hitting circular header dependencies.
+ */
+
+#include <linux/irqflags.h>
+#include <linux/preempt.h>
+#include <linux/processor.h>
+#include <linux/smp.h>
+
+#ifndef MODULE
+/* Per-CPU interrupt disabling state for local_interrupt_{disable,enable}(). */
+DECLARE_PER_CPU(unsigned long, local_interrupt_disable_state);
+
+static __always_inline void __local_interrupt_disable(void)
+{
+	unsigned long flags;
+
+	local_irq_save(flags);
+	raw_cpu_write(local_interrupt_disable_state, flags);
+}
+
+static __always_inline void __local_interrupt_enable(void)
+{
+	unsigned long flags = raw_cpu_read(local_interrupt_disable_state);
+
+	local_irq_restore(flags);
+}
+
+#ifndef INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
+static __always_inline void _local_interrupt_disable(void)
+{
+	__local_interrupt_disable();
+}
+
+static __always_inline void _local_interrupt_enable(void)
+{
+	__local_interrupt_enable();
+}
+#else
+extern void _local_interrupt_disable(void);
+extern void _local_interrupt_enable(void);
+#endif
+
+#else /* !MODULE */
+extern void _local_interrupt_disable(void);
+extern void _local_interrupt_enable(void);
+#endif /* !MODULE */
+
+static inline void local_interrupt_disable(void)
+{
+	int new_count;
+
+	WARN_ON_ONCE(in_nmi());
+
+	new_count = hardirq_disable_enter();
+
+	/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
+		_local_interrupt_disable();
+}
+
+static inline void local_interrupt_enable(void)
+{
+	int new_count;
+
+	new_count = hardirq_disable_exit();
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == 0)
+		_local_interrupt_enable();
+}
+
+#endif /* !__LINUX_INTERRUPT_RC_H */
include/linux/preempt.h +4 -0
index e2d3079d3f5f..33fc4c814a9f 100644
@@ -151,6 +151,10 @@ static __always_inline unsigned char interrupt_context_level(void)
 #define in_softirq()		(softirq_count())
 #define in_interrupt()		(irq_count())
 
+#define hardirq_disable_count()	((preempt_count() & HARDIRQ_DISABLE_MASK) >> HARDIRQ_DISABLE_SHIFT)
+#define hardirq_disable_enter()	__preempt_count_add_return(HARDIRQ_DISABLE_OFFSET)
+#define hardirq_disable_exit()	__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET)
+
 /*
  * The preempt_count offset after preempt_disable();
  */
include/linux/spinlock.h +23 -0
index 241277cd34cf..3d405cc4c121 100644
@@ -57,6 +57,7 @@
 #include <linux/linkage.h>
 #include <linux/compiler.h>
 #include <linux/irqflags.h>
+#include <linux/interrupt_rc.h>
 #include <linux/thread_info.h>
 #include <linux/stringify.h>
 #include <linux/bottom_half.h>
@@ -273,9 +274,11 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
 #endif
 
 #define raw_spin_lock_irq(lock)		_raw_spin_lock_irq(lock)
+#define raw_spin_lock_irq_disable(lock)	_raw_spin_lock_irq_disable(lock)
 #define raw_spin_lock_bh(lock)		_raw_spin_lock_bh(lock)
 #define raw_spin_unlock(lock)		_raw_spin_unlock(lock)
 #define raw_spin_unlock_irq(lock)	_raw_spin_unlock_irq(lock)
+#define raw_spin_unlock_irq_enable(lock)	_raw_spin_unlock_irq_enable(lock)
 
 #define raw_spin_unlock_irqrestore(lock, flags)		\
 	do {							\
@@ -290,6 +293,8 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
 
 #define raw_spin_trylock_irqsave(lock, flags) _raw_spin_trylock_irqsave(lock, &(flags))
 
+#define raw_spin_trylock_irq_disable(lock)	_raw_spin_trylock_irq_disable(lock)
+
 #ifndef CONFIG_PREEMPT_RT
 /* Include rwlock functions for !RT */
 #include <linux/rwlock.h>
@@ -372,6 +377,12 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)
 	raw_spin_lock_irq(&lock->rlock);
 }
 
+static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
+	__acquires(lock) __no_context_analysis
+{
+	raw_spin_lock_irq_disable(&lock->rlock);
+}
+
 #define spin_lock_irqsave(lock, flags)				\
 do {								\
 	raw_spin_lock_irqsave(spinlock_check(lock), flags);	\
@@ -402,6 +413,12 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)
 	raw_spin_unlock_irq(&lock->rlock);
 }
 
+static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
+	__releases(lock) __no_context_analysis
+{
+	raw_spin_unlock_irq_enable(&lock->rlock);
+}
+
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags)
 	__releases(lock) __no_context_analysis
 {
@@ -427,6 +444,12 @@ static __always_inline bool _spin_trylock_irqsave(spinlock_t *lock, unsigned lon
 }
 #define spin_trylock_irqsave(lock, flags) _spin_trylock_irqsave(lock, &(flags))
 
+static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
+	__cond_acquires(true, lock) __no_context_analysis
+{
+	return raw_spin_trylock_irq_disable(&lock->rlock);
+}
+
 /**
  * spin_is_locked() - Check whether a spinlock is locked.
  * @lock: Pointer to the spinlock.
include/linux/spinlock_api_smp.h +41 -0

83 lines not on this page · raw message

include/linux/spinlock_api_up.h +15 -0
index a9d5c7c66e03..d03d3065ee04 100644
@@ -42,6 +42,9 @@
 #define __LOCK_IRQSAVE(lock, flags, ...) \
   do { local_irq_save(flags); __LOCK(lock, ##__VA_ARGS__); } while (0)
 
+#define __LOCK_IRQ_DISABLE(lock, ...) \
+  do { local_interrupt_disable(); __LOCK(lock, ##__VA_ARGS__); } while (0)
+
 #define ___UNLOCK_(lock) \
   do { __release(lock); (void)(lock); } while (0)
 
@@ -61,6 +64,9 @@
 #define __UNLOCK_IRQRESTORE(lock, flags, ...) \
   do { local_irq_restore(flags); __UNLOCK(lock, ##__VA_ARGS__); } while (0)
 
+#define __UNLOCK_IRQ_ENABLE(lock, ...) \
+  do { __UNLOCK(lock, ##__VA_ARGS__); local_interrupt_enable(); } while (0)
+
 #define _raw_spin_lock(lock)			__LOCK(lock)
 #define _raw_spin_lock_nested(lock, subclass)	__LOCK(lock)
 #define _raw_read_lock(lock)			__LOCK(lock, shared)
@@ -70,6 +76,7 @@
 #define _raw_read_lock_bh(lock)			__LOCK_BH(lock, shared)
 #define _raw_write_lock_bh(lock)		__LOCK_BH(lock)
 #define _raw_spin_lock_irq(lock)		__LOCK_IRQ(lock)
+#define _raw_spin_lock_irq_disable(lock)	__LOCK_IRQ_DISABLE(lock)
 #define _raw_read_lock_irq(lock)		__LOCK_IRQ(lock, shared)
 #define _raw_write_lock_irq(lock)		__LOCK_IRQ(lock)
 #define _raw_spin_lock_irqsave(lock, flags)	__LOCK_IRQSAVE(lock, flags)
@@ -97,6 +104,13 @@ static __always_inline int _raw_spin_trylock_irq(raw_spinlock_t *lock)
 	return 1;
 }
 
+static __always_inline int _raw_spin_trylock_irq_disable(raw_spinlock_t *lock)
+	__cond_acquires(true, lock)
+{
+	__LOCK_IRQ_DISABLE(lock);
+	return 1;
+}
+
 static __always_inline int _raw_spin_trylock_irqsave(raw_spinlock_t *lock, unsigned long *flags)
 	__cond_acquires(true, lock)
 {
@@ -132,6 +146,7 @@ static __always_inline int _raw_write_trylock_irqsave(rwlock_t *lock, unsigned l
 #define _raw_write_unlock_bh(lock)		__UNLOCK_BH(lock)
 #define _raw_read_unlock_bh(lock)		__UNLOCK_BH(lock, shared)
 #define _raw_spin_unlock_irq(lock)		__UNLOCK_IRQ(lock)
+#define _raw_spin_unlock_irq_enable(lock)	__UNLOCK_IRQ_ENABLE(lock)
 #define _raw_read_unlock_irq(lock)		__UNLOCK_IRQ(lock, shared)
 #define _raw_write_unlock_irq(lock)		__UNLOCK_IRQ(lock)
 #define _raw_spin_unlock_irqrestore(lock, flags) \
include/linux/spinlock_rt.h +18 -0
index 373618a4243c..560d06384e0c 100644
@@ -96,6 +96,12 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)
 	rt_spin_lock(lock);
 }
 
+static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
+	__acquires(lock)
+{
+	rt_spin_lock(lock);
+}
+
 #define spin_lock_irqsave(lock, flags)			 \
 	do {						 \
 		typecheck(unsigned long, flags);	 \
@@ -122,6 +128,12 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)
 	rt_spin_unlock(lock);
 }
 
+static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
+	__releases(lock)
+{
+	rt_spin_unlock(lock);
+}
+
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,
 						   unsigned long flags)
 	__releases(lock)
@@ -131,6 +143,12 @@ static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,
 
 #define spin_trylock(lock)	rt_spin_trylock(lock)
 
+static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
+	__cond_acquires(true, lock)
+{
+	return rt_spin_trylock(lock);
+}
+
 #define spin_trylock_bh(lock)	rt_spin_trylock_bh(lock)
 
 #define spin_trylock_irq(lock)	rt_spin_trylock(lock)
kernel/locking/spinlock.c +31 -0
index b42d293da38b..83a17eaf5717 100644
@@ -129,6 +129,21 @@ static void __lockfunc __raw_##op##_lock_bh(locktype##_t *lock)		\
  */
 BUILD_LOCK_OPS(spin, raw_spinlock, __acquires);
 
+/* No rwlock_t variants for now, so just build this function by hand */
+static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	for (;;) {
+		preempt_disable();
+		local_interrupt_disable();
+		if (likely(do_raw_spin_trylock(lock)))
+			break;
+		local_interrupt_enable();
+		preempt_enable();
+
+		arch_spin_relax(&lock->raw_lock);
+	}
+}
+
 #ifndef CONFIG_PREEMPT_RT
 BUILD_LOCK_OPS(read, rwlock, __acquires_shared);
 BUILD_LOCK_OPS(write, rwlock, __acquires);
@@ -176,6 +191,14 @@ noinline void __lockfunc _raw_spin_lock_irq(raw_spinlock_t *lock)
 EXPORT_SYMBOL(_raw_spin_lock_irq);
 #endif
 
+#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ
+noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	__raw_spin_lock_irq_disable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable);
+#endif
+
 #ifndef CONFIG_INLINE_SPIN_LOCK_BH
 noinline void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock)
 {
@@ -208,6 +231,14 @@ noinline void __lockfunc _raw_spin_unlock_irq(raw_spinlock_t *lock)
 EXPORT_SYMBOL(_raw_spin_unlock_irq);
 #endif
 
+#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ
+noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
+{
+	__raw_spin_unlock_irq_enable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable);
+#endif
+
 #ifndef CONFIG_INLINE_SPIN_UNLOCK_BH
 noinline void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock)
 {
kernel/softirq.c +26 -2
index 10af5ed859e7..0c9b2269a8d6 100644
@@ -9,6 +9,7 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
 #include <linux/export.h>
 #include <linux/kernel_stat.h>
 #include <linux/interrupt.h>
@@ -88,6 +89,20 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
+DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state);
+
+void _local_interrupt_disable(void)
+{
+	__local_interrupt_disable();
+}
+EXPORT_SYMBOL(_local_interrupt_disable);
+
+void _local_interrupt_enable(void)
+{
+	__local_interrupt_enable();
+}
+EXPORT_SYMBOL(_local_interrupt_enable);
+
 DEFINE_PER_CPU(unsigned int, nmi_nesting);
 
 /*
@@ -728,10 +743,19 @@ static inline void __irq_exit_rcu(void)
 #endif
 	account_hardirq_exit(current);
 	preempt_count_sub(HARDIRQ_OFFSET);
-	if (!in_interrupt() && local_softirq_pending()) {
+	/*
+	 * Interrupts may happen between hardirq_disable_enter() and
+	 * local_irq_save() in local_interrupt_disable(), if irq_exit() invokes
+	 * softirq here, we may have a softirq handler calling
+	 * local_interrupt_disable() but it won't disable the IRQ because
+	 * hardirq disabling count is already 1, hence we need to prevent
+	 * invoking softirq when a local_interrupt_disable() is ongoing.
+	 */
+	if (!in_interrupt() && !hardirq_disable_count() &&
+	    local_softirq_pending()) {
 		/*
 		 * If we left hrtimers unarmed, make sure to arm them now,
-		 * before enabling interrupts to run SoftIRQ.
+		 * before enabling interrupts to run softirq.
 		 */
 		hrtimer_rearm_deferred();
 		invoke_softirq();
Signature · 1 line
2.50.1 (Apple Git-155)
merge bot Applied
#14 lore raw

[tip: locking/core] irq & spin_lock: Add counted interrupt disabling/enabling

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     a58d7a2f53e80422ea9156e7fcda93dc331ccc45
Gitweb:        https://git.kernel.org/tip/a58d7a2f53e80422ea9156e7fcda93dc331ccc45
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 11:26:57 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 08 Aug 2026 22:44:06 +02:00

irq & spin_lock: Add counted interrupt disabling/enabling

Currently the nested interrupt disabling and enabling is represented by _irqsave() and _irqrestore() APIs, which are relatively unsafe, for example:

	<interrupts are enabled as beginning>
	spin_lock_irqsave(l1, flag1);
	spin_lock_irqsave(l2, flag2);
	spin_unlock_irqrestore(l1, flags1);
	<l2 is still held but interrupts are enabled>
	// accesses to interrupt-disable protected data will cause races

This is even easier to trigger with guard facilities:

	unsigned long flag2;

	scoped_guard(spin_lock_irqsave, l1) {
		spin_lock_irqsave(l2, flag2);
	}
	// l2 locked but interrupts are enabled.
	spin_unlock_irqrestore(l2, flag2);

(Hand-to-hand locking critical sections are not uncommon for a fine-grained lock design)

And because of this unsafety, Rust cannot easily wrap the interrupt-disabling locks in a safe API, which complicates the design.

To resolve this, introduce a new set of interrupt disabling APIs:

* local_interrupt_disable();
* local_interrupt_enable();

They work like local_irq_save() and local_irq_restore() except that 1) the outermost local_interrupt_disable() call saves the interrupt state into a per-CPU variable, so that the outermost local_interrupt_enable() can restore the state, and 2) a per-CPU counter is added to record the nest level of these calls, so that interrupts are not accidentally enabled inside the outermost critical section.

Also add the corresponding spin_lock primitives: spin_lock_irq_disable() and spin_unlock_irq_enable(), as a result, code as follows:

	spin_lock_irq_disable(l1);
	spin_lock_irq_disable(l2);
	spin_unlock_irq_enable(l1);
	// Interrupts are still disabled.
	spin_unlock_irq_enable(l2);

doesn't have the issue that interrupts are accidentally enabled.

This also makes the wrapper of interrupt-disabling locks on Rust easier to design.

[boqun: Apply Peter's feedback and fix spell errors reported by Ingo]
[boqun: Address the duplicate spin_acquire() spotted by sashiko]
---
 include/linux/interrupt_rc.h     | 82 +++++++++++++++++++++++++++++++-
 include/linux/preempt.h          |  4 ++-
 include/linux/spinlock.h         | 23 +++++++++-
 include/linux/spinlock_api_smp.h | 41 ++++++++++++++++-
 include/linux/spinlock_api_up.h  | 15 ++++++-
 include/linux/spinlock_rt.h      | 18 +++++++-
 kernel/locking/spinlock.c        | 31 ++++++++++++-
 kernel/softirq.c                 | 28 ++++++++++-
 8 files changed, 240 insertions(+), 2 deletions(-)
 create mode 100644 include/linux/interrupt_rc.h
include/linux/interrupt_rc.h new file +82 -0
new file mode 100644
index 0000000..b9a7f05
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __LINUX_INTERRUPT_RC_H
+#define __LINUX_INTERRUPT_RC_H
+
+/*
+ * include/linux/interrupt_rc.h - refcounted local processor interrupt
+ * management.
+ *
+ * Since the implementation of this API currently depends on
+ * local_irq_save()/local_irq_restore(), we split this into its own header to
+ * make it easier to include without hitting circular header dependencies.
+ */
+
+#include <linux/irqflags.h>
+#include <linux/preempt.h>
+#include <linux/processor.h>
+#include <linux/smp.h>
+
+#ifndef MODULE
+/* Per-CPU interrupt disabling state for local_interrupt_{disable,enable}(). */
+DECLARE_PER_CPU(unsigned long, local_interrupt_disable_state);
+
+static __always_inline void __local_interrupt_disable(void)
+{
+	unsigned long flags;
+
+	local_irq_save(flags);
+	raw_cpu_write(local_interrupt_disable_state, flags);
+}
+
+static __always_inline void __local_interrupt_enable(void)
+{
+	unsigned long flags = raw_cpu_read(local_interrupt_disable_state);
+
+	local_irq_restore(flags);
+}
+
+#ifndef INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
+static __always_inline void _local_interrupt_disable(void)
+{
+	__local_interrupt_disable();
+}
+
+static __always_inline void _local_interrupt_enable(void)
+{
+	__local_interrupt_enable();
+}
+#else
+extern void _local_interrupt_disable(void);
+extern void _local_interrupt_enable(void);
+#endif
+
+#else /* !MODULE */
+extern void _local_interrupt_disable(void);
+extern void _local_interrupt_enable(void);
+#endif /* !MODULE */
+
+static inline void local_interrupt_disable(void)
+{
+	int new_count;
+
+	WARN_ON_ONCE(in_nmi());
+
+	new_count = hardirq_disable_enter();
+
+	/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
+		_local_interrupt_disable();
+}
+
+static inline void local_interrupt_enable(void)
+{
+	int new_count;
+
+	new_count = hardirq_disable_exit();
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == 0)
+		_local_interrupt_enable();
+}
+
+#endif /* !__LINUX_INTERRUPT_RC_H */
include/linux/preempt.h +4 -0
index e2d3079..33fc4c8 100644
@@ -151,6 +151,10 @@ static __always_inline unsigned char interrupt_context_level(void)
 #define in_softirq()		(softirq_count())
 #define in_interrupt()		(irq_count())
 
+#define hardirq_disable_count()	((preempt_count() & HARDIRQ_DISABLE_MASK) >> HARDIRQ_DISABLE_SHIFT)
+#define hardirq_disable_enter()	__preempt_count_add_return(HARDIRQ_DISABLE_OFFSET)
+#define hardirq_disable_exit()	__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET)
+
 /*
  * The preempt_count offset after preempt_disable();
  */
include/linux/spinlock.h +23 -0
index 241277c..3d405cc 100644
@@ -57,6 +57,7 @@
 #include <linux/linkage.h>
 #include <linux/compiler.h>
 #include <linux/irqflags.h>
+#include <linux/interrupt_rc.h>
 #include <linux/thread_info.h>
 #include <linux/stringify.h>
 #include <linux/bottom_half.h>
@@ -273,9 +274,11 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
 #endif
 
 #define raw_spin_lock_irq(lock)		_raw_spin_lock_irq(lock)
+#define raw_spin_lock_irq_disable(lock)	_raw_spin_lock_irq_disable(lock)
 #define raw_spin_lock_bh(lock)		_raw_spin_lock_bh(lock)
 #define raw_spin_unlock(lock)		_raw_spin_unlock(lock)
 #define raw_spin_unlock_irq(lock)	_raw_spin_unlock_irq(lock)
+#define raw_spin_unlock_irq_enable(lock)	_raw_spin_unlock_irq_enable(lock)
 
 #define raw_spin_unlock_irqrestore(lock, flags)		\
 	do {							\
@@ -290,6 +293,8 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
 
 #define raw_spin_trylock_irqsave(lock, flags) _raw_spin_trylock_irqsave(lock, &(flags))
 
+#define raw_spin_trylock_irq_disable(lock)	_raw_spin_trylock_irq_disable(lock)
+
 #ifndef CONFIG_PREEMPT_RT
 /* Include rwlock functions for !RT */
 #include <linux/rwlock.h>
@@ -372,6 +377,12 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)
 	raw_spin_lock_irq(&lock->rlock);
 }
 
+static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
+	__acquires(lock) __no_context_analysis
+{
+	raw_spin_lock_irq_disable(&lock->rlock);
+}
+
 #define spin_lock_irqsave(lock, flags)				\
 do {								\
 	raw_spin_lock_irqsave(spinlock_check(lock), flags);	\
@@ -402,6 +413,12 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)
 	raw_spin_unlock_irq(&lock->rlock);
 }
 
+static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
+	__releases(lock) __no_context_analysis
+{
+	raw_spin_unlock_irq_enable(&lock->rlock);
+}
+
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags)
 	__releases(lock) __no_context_analysis
 {
@@ -427,6 +444,12 @@ static __always_inline bool _spin_trylock_irqsave(spinlock_t *lock, unsigned lon
 }
 #define spin_trylock_irqsave(lock, flags) _spin_trylock_irqsave(lock, &(flags))
 
+static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
+	__cond_acquires(true, lock) __no_context_analysis
+{
+	return raw_spin_trylock_irq_disable(&lock->rlock);
+}
+
 /**
  * spin_is_locked() - Check whether a spinlock is locked.
  * @lock: Pointer to the spinlock.
include/linux/spinlock_api_smp.h +41 -0

83 lines not on this page · raw message

include/linux/spinlock_api_up.h +15 -0
index a9d5c7c..d03d306 100644
@@ -42,6 +42,9 @@
 #define __LOCK_IRQSAVE(lock, flags, ...) \
   do { local_irq_save(flags); __LOCK(lock, ##__VA_ARGS__); } while (0)
 
+#define __LOCK_IRQ_DISABLE(lock, ...) \
+  do { local_interrupt_disable(); __LOCK(lock, ##__VA_ARGS__); } while (0)
+
 #define ___UNLOCK_(lock) \
   do { __release(lock); (void)(lock); } while (0)
 
@@ -61,6 +64,9 @@
 #define __UNLOCK_IRQRESTORE(lock, flags, ...) \
   do { local_irq_restore(flags); __UNLOCK(lock, ##__VA_ARGS__); } while (0)
 
+#define __UNLOCK_IRQ_ENABLE(lock, ...) \
+  do { __UNLOCK(lock, ##__VA_ARGS__); local_interrupt_enable(); } while (0)
+
 #define _raw_spin_lock(lock)			__LOCK(lock)
 #define _raw_spin_lock_nested(lock, subclass)	__LOCK(lock)
 #define _raw_read_lock(lock)			__LOCK(lock, shared)
@@ -70,6 +76,7 @@
 #define _raw_read_lock_bh(lock)			__LOCK_BH(lock, shared)
 #define _raw_write_lock_bh(lock)		__LOCK_BH(lock)
 #define _raw_spin_lock_irq(lock)		__LOCK_IRQ(lock)
+#define _raw_spin_lock_irq_disable(lock)	__LOCK_IRQ_DISABLE(lock)
 #define _raw_read_lock_irq(lock)		__LOCK_IRQ(lock, shared)
 #define _raw_write_lock_irq(lock)		__LOCK_IRQ(lock)
 #define _raw_spin_lock_irqsave(lock, flags)	__LOCK_IRQSAVE(lock, flags)
@@ -97,6 +104,13 @@ static __always_inline int _raw_spin_trylock_irq(raw_spinlock_t *lock)
 	return 1;
 }
 
+static __always_inline int _raw_spin_trylock_irq_disable(raw_spinlock_t *lock)
+	__cond_acquires(true, lock)
+{
+	__LOCK_IRQ_DISABLE(lock);
+	return 1;
+}
+
 static __always_inline int _raw_spin_trylock_irqsave(raw_spinlock_t *lock, unsigned long *flags)
 	__cond_acquires(true, lock)
 {
@@ -132,6 +146,7 @@ static __always_inline int _raw_write_trylock_irqsave(rwlock_t *lock, unsigned l
 #define _raw_write_unlock_bh(lock)		__UNLOCK_BH(lock)
 #define _raw_read_unlock_bh(lock)		__UNLOCK_BH(lock, shared)
 #define _raw_spin_unlock_irq(lock)		__UNLOCK_IRQ(lock)
+#define _raw_spin_unlock_irq_enable(lock)	__UNLOCK_IRQ_ENABLE(lock)
 #define _raw_read_unlock_irq(lock)		__UNLOCK_IRQ(lock, shared)
 #define _raw_write_unlock_irq(lock)		__UNLOCK_IRQ(lock)
 #define _raw_spin_unlock_irqrestore(lock, flags) \
include/linux/spinlock_rt.h +18 -0
index 373618a..560d063 100644
@@ -96,6 +96,12 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)
 	rt_spin_lock(lock);
 }
 
+static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
+	__acquires(lock)
+{
+	rt_spin_lock(lock);
+}
+
 #define spin_lock_irqsave(lock, flags)			 \
 	do {						 \
 		typecheck(unsigned long, flags);	 \
@@ -122,6 +128,12 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)
 	rt_spin_unlock(lock);
 }
 
+static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
+	__releases(lock)
+{
+	rt_spin_unlock(lock);
+}
+
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,
 						   unsigned long flags)
 	__releases(lock)
@@ -131,6 +143,12 @@ static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,
 
 #define spin_trylock(lock)	rt_spin_trylock(lock)
 
+static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
+	__cond_acquires(true, lock)
+{
+	return rt_spin_trylock(lock);
+}
+
 #define spin_trylock_bh(lock)	rt_spin_trylock_bh(lock)
 
 #define spin_trylock_irq(lock)	rt_spin_trylock(lock)
kernel/locking/spinlock.c +31 -0
index b42d293..83a17ea 100644
@@ -129,6 +129,21 @@ static void __lockfunc __raw_##op##_lock_bh(locktype##_t *lock)		\
  */
 BUILD_LOCK_OPS(spin, raw_spinlock, __acquires);
 
+/* No rwlock_t variants for now, so just build this function by hand */
+static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	for (;;) {
+		preempt_disable();
+		local_interrupt_disable();
+		if (likely(do_raw_spin_trylock(lock)))
+			break;
+		local_interrupt_enable();
+		preempt_enable();
+
+		arch_spin_relax(&lock->raw_lock);
+	}
+}
+
 #ifndef CONFIG_PREEMPT_RT
 BUILD_LOCK_OPS(read, rwlock, __acquires_shared);
 BUILD_LOCK_OPS(write, rwlock, __acquires);
@@ -176,6 +191,14 @@ noinline void __lockfunc _raw_spin_lock_irq(raw_spinlock_t *lock)
 EXPORT_SYMBOL(_raw_spin_lock_irq);
 #endif
 
+#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ
+noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	__raw_spin_lock_irq_disable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable);
+#endif
+
 #ifndef CONFIG_INLINE_SPIN_LOCK_BH
 noinline void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock)
 {
@@ -208,6 +231,14 @@ noinline void __lockfunc _raw_spin_unlock_irq(raw_spinlock_t *lock)
 EXPORT_SYMBOL(_raw_spin_unlock_irq);
 #endif
 
+#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ
+noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
+{
+	__raw_spin_unlock_irq_enable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable);
+#endif
+
 #ifndef CONFIG_INLINE_SPIN_UNLOCK_BH
 noinline void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock)
 {
kernel/softirq.c +26 -2
index 10af5ed..0c9b226 100644
@@ -9,6 +9,7 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
 #include <linux/export.h>
 #include <linux/kernel_stat.h>
 #include <linux/interrupt.h>
@@ -88,6 +89,20 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
+DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state);
+
+void _local_interrupt_disable(void)
+{
+	__local_interrupt_disable();
+}
+EXPORT_SYMBOL(_local_interrupt_disable);
+
+void _local_interrupt_enable(void)
+{
+	__local_interrupt_enable();
+}
+EXPORT_SYMBOL(_local_interrupt_enable);
+
 DEFINE_PER_CPU(unsigned int, nmi_nesting);
 
 /*
@@ -728,10 +743,19 @@ static inline void __irq_exit_rcu(void)
 #endif
 	account_hardirq_exit(current);
 	preempt_count_sub(HARDIRQ_OFFSET);
-	if (!in_interrupt() && local_softirq_pending()) {
+	/*
+	 * Interrupts may happen between hardirq_disable_enter() and
+	 * local_irq_save() in local_interrupt_disable(), if irq_exit() invokes
+	 * softirq here, we may have a softirq handler calling
+	 * local_interrupt_disable() but it won't disable the IRQ because
+	 * hardirq disabling count is already 1, hence we need to prevent
+	 * invoking softirq when a local_interrupt_disable() is ongoing.
+	 */
+	if (!in_interrupt() && !hardirq_disable_count() &&
+	    local_softirq_pending()) {
 		/*
 		 * If we left hrtimers unarmed, make sure to arm them now,
-		 * before enabling interrupts to run SoftIRQ.
+		 * before enabling interrupts to run softirq.
 		 */
 		hrtimer_rearm_deferred();
 		invoke_softirq();
merge bot Applied
#15 lore raw

[tip: locking/core] irq,spin_lock: Add counted interrupt disabling/enabling

In reply to Boqun Feng
The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     e901c1510e24726dcbd6340ee927b3ac8b992043
Gitweb:        https://git.kernel.org/tip/e901c1510e24726dcbd6340ee927b3ac8b992043
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 11:26:57 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 10 Aug 2026 10:50:18 +02:00

irq,spin_lock: Add counted interrupt disabling/enabling

Currently the nested interrupt disabling and enabling is represented by _irqsave() and _irqrestore() APIs, which are relatively unsafe, for example:

	<interrupts are enabled as beginning>
	spin_lock_irqsave(l1, flag1);
	spin_lock_irqsave(l2, flag2);
	spin_unlock_irqrestore(l1, flags1);
	<l2 is still held but interrupts are enabled>
	// accesses to interrupt-disable protected data will cause races

This is even easier to trigger with guard facilities:

	unsigned long flag2;

	scoped_guard(spin_lock_irqsave, l1) {
		spin_lock_irqsave(l2, flag2);
	}
	// l2 locked but interrupts are enabled.
	spin_unlock_irqrestore(l2, flag2);

(Hand-to-hand locking critical sections are not uncommon for a fine-grained lock design)

And because of this unsafety, Rust cannot easily wrap the interrupt-disabling locks in a safe API, which complicates the design.

To resolve this, introduce a new set of interrupt disabling APIs:

* local_interrupt_disable();
* local_interrupt_enable();

They work like local_irq_save() and local_irq_restore() except that 1) the outermost local_interrupt_disable() call saves the interrupt state into a per-CPU variable, so that the outermost local_interrupt_enable() can restore the state, and 2) a per-CPU counter is added to record the nest level of these calls, so that interrupts are not accidentally enabled inside the outermost critical section.

Also add the corresponding spin_lock primitives: spin_lock_irq_disable() and spin_unlock_irq_enable(), as a result, code as follows:

	spin_lock_irq_disable(l1);
	spin_lock_irq_disable(l2);
	spin_unlock_irq_enable(l1);
	// Interrupts are still disabled.
	spin_unlock_irq_enable(l2);

doesn't have the issue that interrupts are accidentally enabled.

This also makes the wrapper of interrupt-disabling locks on Rust easier to design.

[boqun: Apply Peter's feedback and fix spell errors reported by Ingo]
[boqun: Address the duplicate spin_acquire() spotted by sashiko]
---
 include/linux/interrupt_rc.h     | 82 +++++++++++++++++++++++++++++++-
 include/linux/preempt.h          |  4 ++-
 include/linux/spinlock.h         | 23 +++++++++-
 include/linux/spinlock_api_smp.h | 41 ++++++++++++++++-
 include/linux/spinlock_api_up.h  | 15 ++++++-
 include/linux/spinlock_rt.h      | 18 +++++++-
 kernel/locking/spinlock.c        | 31 ++++++++++++-
 kernel/softirq.c                 | 28 ++++++++++-
 8 files changed, 240 insertions(+), 2 deletions(-)
 create mode 100644 include/linux/interrupt_rc.h
include/linux/interrupt_rc.h new file +82 -0
new file mode 100644
index 0000000..b9a7f05
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __LINUX_INTERRUPT_RC_H
+#define __LINUX_INTERRUPT_RC_H
+
+/*
+ * include/linux/interrupt_rc.h - refcounted local processor interrupt
+ * management.
+ *
+ * Since the implementation of this API currently depends on
+ * local_irq_save()/local_irq_restore(), we split this into its own header to
+ * make it easier to include without hitting circular header dependencies.
+ */
+
+#include <linux/irqflags.h>
+#include <linux/preempt.h>
+#include <linux/processor.h>
+#include <linux/smp.h>
+
+#ifndef MODULE
+/* Per-CPU interrupt disabling state for local_interrupt_{disable,enable}(). */
+DECLARE_PER_CPU(unsigned long, local_interrupt_disable_state);
+
+static __always_inline void __local_interrupt_disable(void)
+{
+	unsigned long flags;
+
+	local_irq_save(flags);
+	raw_cpu_write(local_interrupt_disable_state, flags);
+}
+
+static __always_inline void __local_interrupt_enable(void)
+{
+	unsigned long flags = raw_cpu_read(local_interrupt_disable_state);
+
+	local_irq_restore(flags);
+}
+
+#ifndef INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
+static __always_inline void _local_interrupt_disable(void)
+{
+	__local_interrupt_disable();
+}
+
+static __always_inline void _local_interrupt_enable(void)
+{
+	__local_interrupt_enable();
+}
+#else
+extern void _local_interrupt_disable(void);
+extern void _local_interrupt_enable(void);
+#endif
+
+#else /* !MODULE */
+extern void _local_interrupt_disable(void);
+extern void _local_interrupt_enable(void);
+#endif /* !MODULE */
+
+static inline void local_interrupt_disable(void)
+{
+	int new_count;
+
+	WARN_ON_ONCE(in_nmi());
+
+	new_count = hardirq_disable_enter();
+
+	/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
+		_local_interrupt_disable();
+}
+
+static inline void local_interrupt_enable(void)
+{
+	int new_count;
+
+	new_count = hardirq_disable_exit();
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == 0)
+		_local_interrupt_enable();
+}
+
+#endif /* !__LINUX_INTERRUPT_RC_H */
include/linux/preempt.h +4 -0
index e2d3079..33fc4c8 100644
@@ -151,6 +151,10 @@ static __always_inline unsigned char interrupt_context_level(void)
 #define in_softirq()		(softirq_count())
 #define in_interrupt()		(irq_count())
 
+#define hardirq_disable_count()	((preempt_count() & HARDIRQ_DISABLE_MASK) >> HARDIRQ_DISABLE_SHIFT)
+#define hardirq_disable_enter()	__preempt_count_add_return(HARDIRQ_DISABLE_OFFSET)
+#define hardirq_disable_exit()	__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET)
+
 /*
  * The preempt_count offset after preempt_disable();
  */
include/linux/spinlock.h +23 -0
index 241277c..3d405cc 100644
@@ -57,6 +57,7 @@
 #include <linux/linkage.h>
 #include <linux/compiler.h>
 #include <linux/irqflags.h>
+#include <linux/interrupt_rc.h>
 #include <linux/thread_info.h>
 #include <linux/stringify.h>
 #include <linux/bottom_half.h>
@@ -273,9 +274,11 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
 #endif
 
 #define raw_spin_lock_irq(lock)		_raw_spin_lock_irq(lock)
+#define raw_spin_lock_irq_disable(lock)	_raw_spin_lock_irq_disable(lock)
 #define raw_spin_lock_bh(lock)		_raw_spin_lock_bh(lock)
 #define raw_spin_unlock(lock)		_raw_spin_unlock(lock)
 #define raw_spin_unlock_irq(lock)	_raw_spin_unlock_irq(lock)
+#define raw_spin_unlock_irq_enable(lock)	_raw_spin_unlock_irq_enable(lock)
 
 #define raw_spin_unlock_irqrestore(lock, flags)		\
 	do {							\
@@ -290,6 +293,8 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
 
 #define raw_spin_trylock_irqsave(lock, flags) _raw_spin_trylock_irqsave(lock, &(flags))
 
+#define raw_spin_trylock_irq_disable(lock)	_raw_spin_trylock_irq_disable(lock)
+
 #ifndef CONFIG_PREEMPT_RT
 /* Include rwlock functions for !RT */
 #include <linux/rwlock.h>
@@ -372,6 +377,12 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)
 	raw_spin_lock_irq(&lock->rlock);
 }
 
+static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
+	__acquires(lock) __no_context_analysis
+{
+	raw_spin_lock_irq_disable(&lock->rlock);
+}
+
 #define spin_lock_irqsave(lock, flags)				\
 do {								\
 	raw_spin_lock_irqsave(spinlock_check(lock), flags);	\
@@ -402,6 +413,12 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)
 	raw_spin_unlock_irq(&lock->rlock);
 }
 
+static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
+	__releases(lock) __no_context_analysis
+{
+	raw_spin_unlock_irq_enable(&lock->rlock);
+}
+
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags)
 	__releases(lock) __no_context_analysis
 {
@@ -427,6 +444,12 @@ static __always_inline bool _spin_trylock_irqsave(spinlock_t *lock, unsigned lon
 }
 #define spin_trylock_irqsave(lock, flags) _spin_trylock_irqsave(lock, &(flags))
 
+static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
+	__cond_acquires(true, lock) __no_context_analysis
+{
+	return raw_spin_trylock_irq_disable(&lock->rlock);
+}
+
 /**
  * spin_is_locked() - Check whether a spinlock is locked.
  * @lock: Pointer to the spinlock.
include/linux/spinlock_api_smp.h +41 -0

83 lines not on this page · raw message

include/linux/spinlock_api_up.h +15 -0
index a9d5c7c..d03d306 100644
@@ -42,6 +42,9 @@
 #define __LOCK_IRQSAVE(lock, flags, ...) \
   do { local_irq_save(flags); __LOCK(lock, ##__VA_ARGS__); } while (0)
 
+#define __LOCK_IRQ_DISABLE(lock, ...) \
+  do { local_interrupt_disable(); __LOCK(lock, ##__VA_ARGS__); } while (0)
+
 #define ___UNLOCK_(lock) \
   do { __release(lock); (void)(lock); } while (0)
 
@@ -61,6 +64,9 @@
 #define __UNLOCK_IRQRESTORE(lock, flags, ...) \
   do { local_irq_restore(flags); __UNLOCK(lock, ##__VA_ARGS__); } while (0)
 
+#define __UNLOCK_IRQ_ENABLE(lock, ...) \
+  do { __UNLOCK(lock, ##__VA_ARGS__); local_interrupt_enable(); } while (0)
+
 #define _raw_spin_lock(lock)			__LOCK(lock)
 #define _raw_spin_lock_nested(lock, subclass)	__LOCK(lock)
 #define _raw_read_lock(lock)			__LOCK(lock, shared)
@@ -70,6 +76,7 @@
 #define _raw_read_lock_bh(lock)			__LOCK_BH(lock, shared)
 #define _raw_write_lock_bh(lock)		__LOCK_BH(lock)
 #define _raw_spin_lock_irq(lock)		__LOCK_IRQ(lock)
+#define _raw_spin_lock_irq_disable(lock)	__LOCK_IRQ_DISABLE(lock)
 #define _raw_read_lock_irq(lock)		__LOCK_IRQ(lock, shared)
 #define _raw_write_lock_irq(lock)		__LOCK_IRQ(lock)
 #define _raw_spin_lock_irqsave(lock, flags)	__LOCK_IRQSAVE(lock, flags)
@@ -97,6 +104,13 @@ static __always_inline int _raw_spin_trylock_irq(raw_spinlock_t *lock)
 	return 1;
 }
 
+static __always_inline int _raw_spin_trylock_irq_disable(raw_spinlock_t *lock)
+	__cond_acquires(true, lock)
+{
+	__LOCK_IRQ_DISABLE(lock);
+	return 1;
+}
+
 static __always_inline int _raw_spin_trylock_irqsave(raw_spinlock_t *lock, unsigned long *flags)
 	__cond_acquires(true, lock)
 {
@@ -132,6 +146,7 @@ static __always_inline int _raw_write_trylock_irqsave(rwlock_t *lock, unsigned l
 #define _raw_write_unlock_bh(lock)		__UNLOCK_BH(lock)
 #define _raw_read_unlock_bh(lock)		__UNLOCK_BH(lock, shared)
 #define _raw_spin_unlock_irq(lock)		__UNLOCK_IRQ(lock)
+#define _raw_spin_unlock_irq_enable(lock)	__UNLOCK_IRQ_ENABLE(lock)
 #define _raw_read_unlock_irq(lock)		__UNLOCK_IRQ(lock, shared)
 #define _raw_write_unlock_irq(lock)		__UNLOCK_IRQ(lock)
 #define _raw_spin_unlock_irqrestore(lock, flags) \
include/linux/spinlock_rt.h +18 -0
index 373618a..560d063 100644
@@ -96,6 +96,12 @@ static __always_inline void spin_lock_irq(spinlock_t *lock)
 	rt_spin_lock(lock);
 }
 
+static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
+	__acquires(lock)
+{
+	rt_spin_lock(lock);
+}
+
 #define spin_lock_irqsave(lock, flags)			 \
 	do {						 \
 		typecheck(unsigned long, flags);	 \
@@ -122,6 +128,12 @@ static __always_inline void spin_unlock_irq(spinlock_t *lock)
 	rt_spin_unlock(lock);
 }
 
+static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
+	__releases(lock)
+{
+	rt_spin_unlock(lock);
+}
+
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,
 						   unsigned long flags)
 	__releases(lock)
@@ -131,6 +143,12 @@ static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,
 
 #define spin_trylock(lock)	rt_spin_trylock(lock)
 
+static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
+	__cond_acquires(true, lock)
+{
+	return rt_spin_trylock(lock);
+}
+
 #define spin_trylock_bh(lock)	rt_spin_trylock_bh(lock)
 
 #define spin_trylock_irq(lock)	rt_spin_trylock(lock)
kernel/locking/spinlock.c +31 -0
index b42d293..83a17ea 100644
@@ -129,6 +129,21 @@ static void __lockfunc __raw_##op##_lock_bh(locktype##_t *lock)		\
  */
 BUILD_LOCK_OPS(spin, raw_spinlock, __acquires);
 
+/* No rwlock_t variants for now, so just build this function by hand */
+static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	for (;;) {
+		preempt_disable();
+		local_interrupt_disable();
+		if (likely(do_raw_spin_trylock(lock)))
+			break;
+		local_interrupt_enable();
+		preempt_enable();
+
+		arch_spin_relax(&lock->raw_lock);
+	}
+}
+
 #ifndef CONFIG_PREEMPT_RT
 BUILD_LOCK_OPS(read, rwlock, __acquires_shared);
 BUILD_LOCK_OPS(write, rwlock, __acquires);
@@ -176,6 +191,14 @@ noinline void __lockfunc _raw_spin_lock_irq(raw_spinlock_t *lock)
 EXPORT_SYMBOL(_raw_spin_lock_irq);
 #endif
 
+#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ
+noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	__raw_spin_lock_irq_disable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable);
+#endif
+
 #ifndef CONFIG_INLINE_SPIN_LOCK_BH
 noinline void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock)
 {
@@ -208,6 +231,14 @@ noinline void __lockfunc _raw_spin_unlock_irq(raw_spinlock_t *lock)
 EXPORT_SYMBOL(_raw_spin_unlock_irq);
 #endif
 
+#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ
+noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
+{
+	__raw_spin_unlock_irq_enable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable);
+#endif
+
 #ifndef CONFIG_INLINE_SPIN_UNLOCK_BH
 noinline void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock)
 {
kernel/softirq.c +26 -2
index 10af5ed..0c9b226 100644
@@ -9,6 +9,7 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
 #include <linux/export.h>
 #include <linux/kernel_stat.h>
 #include <linux/interrupt.h>
@@ -88,6 +89,20 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
+DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state);
+
+void _local_interrupt_disable(void)
+{
+	__local_interrupt_disable();
+}
+EXPORT_SYMBOL(_local_interrupt_disable);
+
+void _local_interrupt_enable(void)
+{
+	__local_interrupt_enable();
+}
+EXPORT_SYMBOL(_local_interrupt_enable);
+
 DEFINE_PER_CPU(unsigned int, nmi_nesting);
 
 /*
@@ -728,10 +743,19 @@ static inline void __irq_exit_rcu(void)
 #endif
 	account_hardirq_exit(current);
 	preempt_count_sub(HARDIRQ_OFFSET);
-	if (!in_interrupt() && local_softirq_pending()) {
+	/*
+	 * Interrupts may happen between hardirq_disable_enter() and
+	 * local_irq_save() in local_interrupt_disable(), if irq_exit() invokes
+	 * softirq here, we may have a softirq handler calling
+	 * local_interrupt_disable() but it won't disable the IRQ because
+	 * hardirq disabling count is already 1, hence we need to prevent
+	 * invoking softirq when a local_interrupt_disable() is ongoing.
+	 */
+	if (!in_interrupt() && !hardirq_disable_count() &&
+	    local_softirq_pending()) {
 		/*
 		 * If we left hrtimers unarmed, make sure to arm them now,
-		 * before enabling interrupts to run SoftIRQ.
+		 * before enabling interrupts to run softirq.
 		 */
 		hrtimer_rearm_deferred();
 		invoke_softirq();
#16 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

In reply to Boqun Feng
Quoted 134 lines · Boqun Feng

Currently the nested interrupt disabling and enabling is represented by _irqsave() and _irqrestore() APIs, which are relatively unsafe, for example:

	<interrupts are enabled as beginning>
	spin_lock_irqsave(l1, flag1);
	spin_lock_irqsave(l2, flag2);
	spin_unlock_irqrestore(l1, flags1);
	<l2 is still held but interrupts are enabled>
	// accesses to interrupt-disable protected data will cause races

This is even easier to trigger with guard facilities:

	unsigned long flag2;

	scoped_guard(spin_lock_irqsave, l1) {
		spin_lock_irqsave(l2, flag2);
	}
	// l2 locked but interrupts are enabled.
	spin_unlock_irqrestore(l2, flag2);

(Hand-to-hand locking critical sections are not uncommon for a fine-grained lock design)

And because of this unsafety, Rust cannot easily wrap the interrupt-disabling locks in a safe API, which complicates the design.

To resolve this, introduce a new set of interrupt disabling APIs:

* local_interrupt_disable();
* local_interrupt_enable();

They work like local_irq_save() and local_irq_restore() except that 1) the outermost local_interrupt_disable() call saves the interrupt state into a per-CPU variable, so that the outermost local_interrupt_enable() can restore the state, and 2) a per-CPU counter is added to record the nest level of these calls, so that interrupts are not accidentally enabled inside the outermost critical section.

Also add the corresponding spin_lock primitives: spin_lock_irq_disable() and spin_unlock_irq_enable(), as a result, code as follows:

	spin_lock_irq_disable(l1);
	spin_lock_irq_disable(l2);
	spin_unlock_irq_enable(l1);
	// Interrupts are still disabled.
	spin_unlock_irq_enable(l2);

doesn't have the issue that interrupts are accidentally enabled.

This also makes the wrapper of interrupt-disabling locks on Rust easier to design.

[boqun: Apply Peter's feedback and fix spell errors reported by Ingo]
---
  include/linux/interrupt_rc.h     | 82 ++++++++++++++++++++++++++++++++
  include/linux/preempt.h          |  4 ++
  include/linux/spinlock.h         | 23 +++++++++
  include/linux/spinlock_api_smp.h | 43 +++++++++++++++++
  include/linux/spinlock_api_up.h  | 15 ++++++
  include/linux/spinlock_rt.h      | 18 +++++++
  kernel/locking/spinlock.c        | 31 ++++++++++++
  kernel/softirq.c                 | 28 ++++++++++-
  8 files changed, 242 insertions(+), 2 deletions(-)

create mode 100644 include/linux/interrupt_rc.h

include/linux/interrupt_rc.h new file +60 -0
new file mode 100644
index 000000000000..b9a7f05ecf42
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __LINUX_INTERRUPT_RC_H
+#define __LINUX_INTERRUPT_RC_H
+
+/*
+ * include/linux/interrupt_rc.h - refcounted local processor interrupt
+ * management.
+ *
+ * Since the implementation of this API currently depends on
+ * local_irq_save()/local_irq_restore(), we split this into its own header to
+ * make it easier to include without hitting circular header dependencies.
+ */
+
+#include <linux/irqflags.h>
+#include <linux/preempt.h>
+#include <linux/processor.h>
+#include <linux/smp.h>
+
+#ifndef MODULE
+/* Per-CPU interrupt disabling state for local_interrupt_{disable,enable}(). */
+DECLARE_PER_CPU(unsigned long, local_interrupt_disable_state);
+
+static __always_inline void __local_interrupt_disable(void)
+{
+	unsigned long flags;
+
+	local_irq_save(flags);
+	raw_cpu_write(local_interrupt_disable_state, flags);
+}
+
+static __always_inline void __local_interrupt_enable(void)
+{
+	unsigned long flags = raw_cpu_read(local_interrupt_disable_state);
+
+	local_irq_restore(flags);
+}
+
+#ifndef INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
+static __always_inline void _local_interrupt_disable(void)
+{
+	__local_interrupt_disable();
+}
+
+static __always_inline void _local_interrupt_enable(void)
+{
+	__local_interrupt_enable();
+}
+#else
+extern void _local_interrupt_disable(void);
+extern void _local_interrupt_enable(void);
+#endif
+
+#else /* !MODULE */
+extern void _local_interrupt_disable(void);
+extern void _local_interrupt_enable(void);
+#endif /* !MODULE */
+
+static inline void local_interrupt_disable(void)
+{
+	int new_count;

include/linux/interrupt_rc.h line 70

Quoted 10 lines
include/linux/interrupt_rc.h new file +10 -0
+
+	WARN_ON_ONCE(in_nmi());
+
+	new_count = hardirq_disable_enter();
+
+	/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
+		_local_interrupt_disable();
+}

Maximum nesting possible is 256 right? Whats is stopping here to do more than that? Should there be a warn_on?

Quoted 12 lines
include/linux/interrupt_rc.h new file +12 -0
+
+static inline void local_interrupt_enable(void)
+{
+	int new_count;
+
+	new_count = hardirq_disable_exit();
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == 0)
+		_local_interrupt_enable();
+}
+
+#endif /* !__LINUX_INTERRUPT_RC_H */
author maintainer
#17 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

Quoted 141 lines · Shrikanth Hegde
Quoted 138 lines · Boqun Feng

Currently the nested interrupt disabling and enabling is represented by _irqsave() and _irqrestore() APIs, which are relatively unsafe, for example:

	<interrupts are enabled as beginning>
	spin_lock_irqsave(l1, flag1);
	spin_lock_irqsave(l2, flag2);
	spin_unlock_irqrestore(l1, flags1);
	<l2 is still held but interrupts are enabled>
	// accesses to interrupt-disable protected data will cause races

This is even easier to trigger with guard facilities:

	unsigned long flag2;

	scoped_guard(spin_lock_irqsave, l1) {
		spin_lock_irqsave(l2, flag2);
	}
	// l2 locked but interrupts are enabled.
	spin_unlock_irqrestore(l2, flag2);

(Hand-to-hand locking critical sections are not uncommon for a fine-grained lock design)

And because of this unsafety, Rust cannot easily wrap the interrupt-disabling locks in a safe API, which complicates the design.

To resolve this, introduce a new set of interrupt disabling APIs:

* local_interrupt_disable();
* local_interrupt_enable();

They work like local_irq_save() and local_irq_restore() except that 1) the outermost local_interrupt_disable() call saves the interrupt state into a per-CPU variable, so that the outermost local_interrupt_enable() can restore the state, and 2) a per-CPU counter is added to record the nest level of these calls, so that interrupts are not accidentally enabled inside the outermost critical section.

Also add the corresponding spin_lock primitives: spin_lock_irq_disable() and spin_unlock_irq_enable(), as a result, code as follows:

	spin_lock_irq_disable(l1);
	spin_lock_irq_disable(l2);
	spin_unlock_irq_enable(l1);
	// Interrupts are still disabled.
	spin_unlock_irq_enable(l2);

doesn't have the issue that interrupts are accidentally enabled.

This also makes the wrapper of interrupt-disabling locks on Rust easier to design.

[boqun: Apply Peter's feedback and fix spell errors reported by Ingo]
---
  include/linux/interrupt_rc.h     | 82 ++++++++++++++++++++++++++++++++
  include/linux/preempt.h          |  4 ++
  include/linux/spinlock.h         | 23 +++++++++
  include/linux/spinlock_api_smp.h | 43 +++++++++++++++++
  include/linux/spinlock_api_up.h  | 15 ++++++
  include/linux/spinlock_rt.h      | 18 +++++++
  kernel/locking/spinlock.c        | 31 ++++++++++++
  kernel/softirq.c                 | 28 ++++++++++-
  8 files changed, 242 insertions(+), 2 deletions(-)

create mode 100644 include/linux/interrupt_rc.h

include/linux/interrupt_rc.h new file +64 -0
new file mode 100644
index 000000000000..b9a7f05ecf42
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __LINUX_INTERRUPT_RC_H
+#define __LINUX_INTERRUPT_RC_H
+
+/*
+ * include/linux/interrupt_rc.h - refcounted local processor interrupt
+ * management.
+ *
+ * Since the implementation of this API currently depends on
+ * local_irq_save()/local_irq_restore(), we split this into its own header to
+ * make it easier to include without hitting circular header dependencies.
+ */
+
+#include <linux/irqflags.h>
+#include <linux/preempt.h>
+#include <linux/processor.h>
+#include <linux/smp.h>
+
+#ifndef MODULE
+/* Per-CPU interrupt disabling state for local_interrupt_{disable,enable}(). */
+DECLARE_PER_CPU(unsigned long, local_interrupt_disable_state);
+
+static __always_inline void __local_interrupt_disable(void)
+{
+	unsigned long flags;
+
+	local_irq_save(flags);
+	raw_cpu_write(local_interrupt_disable_state, flags);
+}
+
+static __always_inline void __local_interrupt_enable(void)
+{
+	unsigned long flags = raw_cpu_read(local_interrupt_disable_state);
+
+	local_irq_restore(flags);
+}
+
+#ifndef INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
+static __always_inline void _local_interrupt_disable(void)
+{
+	__local_interrupt_disable();
+}
+
+static __always_inline void _local_interrupt_enable(void)
+{
+	__local_interrupt_enable();
+}
+#else
+extern void _local_interrupt_disable(void);
+extern void _local_interrupt_enable(void);
+#endif
+
+#else /* !MODULE */
+extern void _local_interrupt_disable(void);
+extern void _local_interrupt_enable(void);
+#endif /* !MODULE */
+
+static inline void local_interrupt_disable(void)
+{
+	int new_count;
+
+	WARN_ON_ONCE(in_nmi());
+
+	new_count = hardirq_disable_enter();
Quoted 8 lines

include/linux/interrupt_rc.h line 70

Quoted 6 lines
include/linux/interrupt_rc.h new file +6 -0
+
+	/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
+		_local_interrupt_disable();
+}

Maximum nesting possible is 256 right? Whats is stopping here to do more than that?

Yes. Currently similar as softirq, we don't detect the overflow.

Quoted 1 line

Should there be a warn_on?

A simple warn_on could be problematic because warn_on() itself may take an irq-disabling lock, and that may trigger another overflow on top of the existing overflow. It's a bit tricky to do a proper detection. But I'm open to ideas.

Regards,
Boqun

Quoted 13 lines
Quoted 13 lines
include/linux/interrupt_rc.h new file +12 -0
+
+static inline void local_interrupt_enable(void)
+{
+	int new_count;
+
+	new_count = hardirq_disable_exit();
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == 0)
+		_local_interrupt_enable();
+}
+
+#endif /* !__LINUX_INTERRUPT_RC_H */
maintainer
#18 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

Quoted 8 lines · Boqun Feng
Quoted 7 lines

include/linux/interrupt_rc.h line 70

Quoted 5 lines
include/linux/interrupt_rc.h new file +5 -0
+	/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
+		_local_interrupt_disable();
+}

Maximum nesting possible is 256 right? Whats is stopping here to do more than that?

Quoted 8 lines

Yes. Currently similar as softirq, we don't detect the overflow.

Quoted 1 line

Should there be a warn_on?

A simple warn_on could be problematic because warn_on() itself may take an irq-disabling lock, and that may trigger another overflow on top of the existing overflow. It's a bit tricky to do a proper detection. But I'm open to ideas.

DEBUG_PREEMPT's preempt_count_add() does:

        DEBUG_LOCKS_WARN_ON((preempt_count() & PREEMPT_MASK) >= PREEMPT_MASK - 10);
author maintainer
#19 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · Boqun Feng
Quoted 7 lines

include/linux/interrupt_rc.h line 70

Quoted 5 lines
include/linux/interrupt_rc.h new file +5 -0
+	/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
+		_local_interrupt_disable();
+}

Maximum nesting possible is 256 right? Whats is stopping here to do more than that?

Yes. Currently similar as softirq, we don't detect the overflow.

Quoted 1 line

Should there be a warn_on?

Quoted 9 lines
Quoted 4 lines

A simple warn_on could be problematic because warn_on() itself may take an irq-disabling lock, and that may trigger another overflow on top of the existing overflow. It's a bit tricky to do a proper detection. But I'm open to ideas.

DEBUG_PREEMPT's preempt_count_add() does:

        DEBUG_LOCKS_WARN_ON((preempt_count() & PREEMPT_MASK) >= PREEMPT_MASK - 10);

Yeah, but this is behind a kconfig (DEBUG_PREEMPT), so not sure whether it's what Shrikanth asked here.

Regards,
Boqun

Quoted 1 line
#20 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

Quoted 20 lines · Boqun Feng
Quoted 19 lines · Peter Zijlstra
Quoted 16 lines · Boqun Feng
Quoted 7 lines

include/linux/interrupt_rc.h line 70

Quoted 5 lines
include/linux/interrupt_rc.h new file +5 -0
+	/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
+		_local_interrupt_disable();
+}

Maximum nesting possible is 256 right? Whats is stopping here to do more than that?

Yes. Currently similar as softirq, we don't detect the overflow.

Quoted 1 line

Should there be a warn_on?

A simple warn_on could be problematic because warn_on() itself may take an irq-disabling lock, and that may trigger another overflow on top of the existing overflow. It's a bit tricky to do a proper detection. But I'm open to ideas.

Quoted 8 lines
Quoted 4 lines

DEBUG_PREEMPT's preempt_count_add() does:

         DEBUG_LOCKS_WARN_ON((preempt_count() & PREEMPT_MASK) >= PREEMPT_MASK - 10);

Yeah, but this is behind a kconfig (DEBUG_PREEMPT), so not sure whether it's what Shrikanth asked here.

I was suggesting to have a mechanism which allows to debug it/ find the callers.

Quoted 4 lines

Regards,
Boqun

Quoted 1 line
author maintainer
#21 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

Quoted 28 lines · Shrikanth Hegde
Quoted 25 lines · Boqun Feng
Quoted 23 lines · Peter Zijlstra
Quoted 8 lines · Boqun Feng
Quoted 7 lines

include/linux/interrupt_rc.h line 70

Quoted 5 lines
include/linux/interrupt_rc.h new file +5 -0
+	/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
+		_local_interrupt_disable();
+}

Maximum nesting possible is 256 right? Whats is stopping here to do more than that?

Quoted 8 lines

Yes. Currently similar as softirq, we don't detect the overflow.

Quoted 1 line

Should there be a warn_on?

A simple warn_on could be problematic because warn_on() itself may take an irq-disabling lock, and that may trigger another overflow on top of the existing overflow. It's a bit tricky to do a proper detection. But I'm open to ideas.

DEBUG_PREEMPT's preempt_count_add() does:

         DEBUG_LOCKS_WARN_ON((preempt_count() & PREEMPT_MASK) >= PREEMPT_MASK - 10);
Quoted 7 lines
Quoted 3 lines

Yeah, but this is behind a kconfig (DEBUG_PREEMPT), so not sure whether it's what Shrikanth asked here.

I was suggesting to have a mechanism which allows to debug it/ find the callers.

Ok, I can reuse the DEBUG_PREEMPT kconfig and the DEBUG_LOCKS_WARN_ON() here, but it'll be similar to the detection here, not a 256 maximum nesting but a 256 - 10 value (and I'm not going to explain why we think 10 is a good buffer, unless you think we should have a discussion about it ;-) )

(But still who is going to add the detection for BH count over? ;-) )

Regards,
Boqun

Quoted 5 lines
Quoted 4 lines

Regards,
Boqun

Quoted 1 line
author maintainer
#22 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

Quoted 37 lines · Boqun Feng
Quoted 28 lines · Shrikanth Hegde
Quoted 25 lines · Boqun Feng
Quoted 23 lines · Peter Zijlstra
Quoted 8 lines · Boqun Feng
Quoted 7 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+	/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
+
+	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
+		_local_interrupt_disable();
+}

Maximum nesting possible is 256 right? Whats is stopping here to do more than that?

Quoted 8 lines

Yes. Currently similar as softirq, we don't detect the overflow.

Quoted 1 line

Should there be a warn_on?

A simple warn_on could be problematic because warn_on() itself may take an irq-disabling lock, and that may trigger another overflow on top of the existing overflow. It's a bit tricky to do a proper detection. But I'm open to ideas.

DEBUG_PREEMPT's preempt_count_add() does:

         DEBUG_LOCKS_WARN_ON((preempt_count() & PREEMPT_MASK) >= PREEMPT_MASK - 10);
Quoted 7 lines
Quoted 3 lines

Yeah, but this is behind a kconfig (DEBUG_PREEMPT), so not sure whether it's what Shrikanth asked here.

I was suggesting to have a mechanism which allows to debug it/ find the callers.

Quoted 6 lines

Ok, I can reuse the DEBUG_PREEMPT kconfig and the DEBUG_LOCKS_WARN_ON() here, but it'll be similar to the detection here, not a 256 maximum nesting but a 256 - 10 value (and I'm not going to explain why we think 10 is a good buffer, unless you think we should have a discussion about it ;-) )

Something as below? Going to send it to kernel build bot and see if it works for all configs.

----------------->8

include/linux/interrupt_rc.h +12 -0
index b9a7f05ecf42..39f30bc65548 100644
@@ -12,6 +12,7 @@
  */
 
 #include <linux/irqflags.h>
+#include <linux/debug_locks.h>
 #include <linux/preempt.h>
 #include <linux/processor.h>
 #include <linux/smp.h>
@@ -63,6 +64,12 @@ static inline void local_interrupt_disable(void)
 
        new_count = hardirq_disable_enter();
 
+       /* Is hardirq disable count overflow soon? */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT))
+               DEBUG_LOCKS_WARN_ON((new_count & HARDIRQ_DISABLE_MASK) +
+                                   (10 << HARDIRQ_DISABLE_SHIFT) >
+                                   HARDIRQ_DISABLE_MASK);
+
        /* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
 
        if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
@@ -73,6 +80,11 @@ static inline void local_interrupt_enable(void)
 {
        int new_count;
 
+       /* Unpaired local_interrupt_enable()? Warn and abort. */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) &&
+           DEBUG_LOCKS_WARN_ON((preempt_count() & HARDIRQ_DISABLE_MASK) == 0))
+               return;
+
        new_count = hardirq_disable_exit();
 
        if ((new_count & HARDIRQ_DISABLE_MASK) == 0) > 
#23 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

Hi Boqun,

Quoted 18 lines

Something as below? Going to send it to kernel build bot and see if it works for all configs.

----------------->8

include/linux/interrupt_rc.h +1 -0
index b9a7f05ecf42..39f30bc65548 100644
@@ -12,6 +12,7 @@
   */
 
  #include <linux/irqflags.h>
+#include <linux/debug_locks.h>
  #include <linux/preempt.h>
  #include <linux/processor.h>
  #include <linux/smp.h>

include/linux/interrupt_rc.h static inline void local_interrupt_disable(void) · line 72

Quoted 9 lines
include/linux/interrupt_rc.h +6 -0
@@ -63,6 +64,12 @@ static inline void local_interrupt_disable(void)
 
         new_count = hardirq_disable_enter();
 
+       /* Is hardirq disable count overflow soon? */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT))
+               DEBUG_LOCKS_WARN_ON((new_count & HARDIRQ_DISABLE_MASK) +
+                                   (10 << HARDIRQ_DISABLE_SHIFT) >
+                                   HARDIRQ_DISABLE_MASK);
+

This needs a return here right? Else we will see warning for 10 times and then overflow happens and we will call _local_interrupt_disable. No?

Not sure, if below is any better? (Igore whitespace mangling)

if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) &&
DEBUG_LOCKS_WARN_ON((preempt_count() & HARDIRQ_DISABLE_MASK) >=
HARDIRQ_DISABLE_MASK - (10 << HARDIRQ_DISABLE_SHIFT)))
return;

Quoted 15 lines
include/linux/interrupt_rc.h +5 -0
         /* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
 
         if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
@@ -73,6 +80,11 @@ static inline void local_interrupt_enable(void)
  {
         int new_count;
 
+       /* Unpaired local_interrupt_enable()? Warn and abort. */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) &&
+           DEBUG_LOCKS_WARN_ON((preempt_count() & HARDIRQ_DISABLE_MASK) == 0))
+               return;
+
         new_count = hardirq_disable_exit();
 
         if ((new_count & HARDIRQ_DISABLE_MASK) == 0) >
author maintainer
#24 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

Quoted 24 lines · Shrikanth Hegde

Hi Boqun,

Quoted 21 lines

Something as below? Going to send it to kernel build bot and see if it works for all configs.

----------------->8

include/linux/interrupt_rc.h +1 -0
index b9a7f05ecf42..39f30bc65548 100644
@@ -12,6 +12,7 @@
   */
 
  #include <linux/irqflags.h>
+#include <linux/debug_locks.h>
  #include <linux/preempt.h>
  #include <linux/processor.h>
  #include <linux/smp.h>
@@ -63,6 +64,12 @@ static inline void local_interrupt_disable(void)
 
         new_count = hardirq_disable_enter();
 
Quoted 10 lines

include/linux/interrupt_rc.h static inline void local_interrupt_disable(void) · line 72

Quoted 6 lines
include/linux/interrupt_rc.h +6 -0
+       /* Is hardirq disable count overflow soon? */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT))
+               DEBUG_LOCKS_WARN_ON((new_count & HARDIRQ_DISABLE_MASK) +
+                                   (10 << HARDIRQ_DISABLE_SHIFT) >
+                                   HARDIRQ_DISABLE_MASK);
+

This needs a return here right? Else we will see warning for 10 times and then overflow happens and we will call _local_interrupt_disable. No?

DEBUG_LOCKS_WARN_ON() uses debug_locks_off() to avoid this, so we won't see it 10 times. The reason not using return here, because we would introduce unpaired local_interrupt_disable() if we returned:

	// hardirq disable count is n
	local_interrupt_disable(); // hardirq disable count is n + 1

	local_interrupt_disable(); <- trigger the warning, if we return
				   // hardirq disable count is n + 1

	local_interrupt_enable(); // hardirq disable count is n

	local_interrupt_enable(); // hardirq disable count is n - 1

Regards,
Boqun

Quoted 23 lines

Not sure, if below is any better? (Igore whitespace mangling)

if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) &&
DEBUG_LOCKS_WARN_ON((preempt_count() & HARDIRQ_DISABLE_MASK) >=
HARDIRQ_DISABLE_MASK - (10 << HARDIRQ_DISABLE_SHIFT)))
return;

Quoted 15 lines
include/linux/interrupt_rc.h +5 -0
         /* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
 
         if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
@@ -73,6 +80,11 @@ static inline void local_interrupt_enable(void)
  {
         int new_count;
 
+       /* Unpaired local_interrupt_enable()? Warn and abort. */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) &&
+           DEBUG_LOCKS_WARN_ON((preempt_count() & HARDIRQ_DISABLE_MASK) == 0))
+               return;
+
         new_count = hardirq_disable_exit();
 
         if ((new_count & HARDIRQ_DISABLE_MASK) == 0) >
#25 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

Quoted 36 lines · Boqun Feng
Quoted 24 lines · Shrikanth Hegde

Hi Boqun,

Quoted 21 lines

Something as below? Going to send it to kernel build bot and see if it works for all configs.

----------------->8

include/linux/interrupt_rc.h +1 -0
index b9a7f05ecf42..39f30bc65548 100644
@@ -12,6 +12,7 @@
    */
 
   #include <linux/irqflags.h>
+#include <linux/debug_locks.h>
   #include <linux/preempt.h>
   #include <linux/processor.h>
   #include <linux/smp.h>
@@ -63,6 +64,12 @@ static inline void local_interrupt_disable(void)
 
          new_count = hardirq_disable_enter();
 
Quoted 10 lines

include/linux/interrupt_rc.h static inline void local_interrupt_disable(void) · line 72

Quoted 6 lines
include/linux/interrupt_rc.h +6 -0
+       /* Is hardirq disable count overflow soon? */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT))
+               DEBUG_LOCKS_WARN_ON((new_count & HARDIRQ_DISABLE_MASK) +
+                                   (10 << HARDIRQ_DISABLE_SHIFT) >
+                                   HARDIRQ_DISABLE_MASK);
+

This needs a return here right? Else we will see warning for 10 times and then overflow happens and we will call _local_interrupt_disable. No?

Quoted 3 lines

DEBUG_LOCKS_WARN_ON() uses debug_locks_off() to avoid this, so we won't see it 10 times. The reason not using return here, because we would introduce unpaired local_interrupt_disable() if we returned:

Yes, it could be a weird case if the overflow actually happens. So just warning maybe enough to catch such callers.

Maybe your kunit test can actually help test the behavior with the loop count.

Quoted 6 lines
	// hardirq disable count is n
	local_interrupt_disable(); // hardirq disable count is n + 1

	local_interrupt_disable(); <- trigger the warning, if we return
				   // hardirq disable count is n + 1

Likely i am missing to understand.

Isn't the count incremented earlier than return? I.e even if return happens it should be n + 2 right?

Quoted 31 lines
	local_interrupt_enable(); // hardirq disable count is n

	local_interrupt_enable(); // hardirq disable count is n - 1

Regards,
Boqun

Quoted 23 lines

Not sure, if below is any better? (Igore whitespace mangling)

if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) &&
DEBUG_LOCKS_WARN_ON((preempt_count() & HARDIRQ_DISABLE_MASK) >=
HARDIRQ_DISABLE_MASK - (10 << HARDIRQ_DISABLE_SHIFT)))
return;

Quoted 15 lines
include/linux/interrupt_rc.h +5 -0
          /* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
 
          if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
@@ -73,6 +80,11 @@ static inline void local_interrupt_enable(void)
   {
          int new_count;
 
+       /* Unpaired local_interrupt_enable()? Warn and abort. */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) &&
+           DEBUG_LOCKS_WARN_ON((preempt_count() & HARDIRQ_DISABLE_MASK) == 0))
+               return;
+
          new_count = hardirq_disable_exit();
 
          if ((new_count & HARDIRQ_DISABLE_MASK) == 0) >
author maintainer
#26 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

Quoted 28 lines · Shrikanth Hegde
Quoted 25 lines · Boqun Feng
Quoted 24 lines · Shrikanth Hegde

Hi Boqun,

Quoted 21 lines

Something as below? Going to send it to kernel build bot and see if it works for all configs.

----------------->8

include/linux/interrupt_rc.h +1 -0
index b9a7f05ecf42..39f30bc65548 100644
@@ -12,6 +12,7 @@
    */
 
   #include <linux/irqflags.h>
+#include <linux/debug_locks.h>
   #include <linux/preempt.h>
   #include <linux/processor.h>
   #include <linux/smp.h>
@@ -63,6 +64,12 @@ static inline void local_interrupt_disable(void)
 
          new_count = hardirq_disable_enter();
 
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

include/linux/interrupt_rc.h static inline void local_interrupt_disable(void) · line 72

Quoted 6 lines
include/linux/interrupt_rc.h +6 -0
+       /* Is hardirq disable count overflow soon? */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT))
+               DEBUG_LOCKS_WARN_ON((new_count & HARDIRQ_DISABLE_MASK) +
+                                   (10 << HARDIRQ_DISABLE_SHIFT) >
+                                   HARDIRQ_DISABLE_MASK);
+

This needs a return here right? Else we will see warning for 10 times and then overflow happens and we will call _local_interrupt_disable. No?

Oh, seems I overlooked something... could you elaborate on this? What's the scenario in your mind? You said we hit 10 times warning and *then* overflow?

Quoted 13 lines
Quoted 5 lines
Quoted 1 line

DEBUG_LOCKS_WARN_ON() uses debug_locks_off() to avoid this, so we won't see it 10 times. The reason not using return here, because we would introduce unpaired local_interrupt_disable() if we returned:

Yes, it could be a weird case if the overflow actually happens. So just warning maybe enough to catch such callers.

Maybe your kunit test can actually help test the behavior with the loop count.

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
	// hardirq disable count is n
	local_interrupt_disable(); // hardirq disable count is n + 1

	local_interrupt_disable(); <- trigger the warning, if we return
				   // hardirq disable count is n + 1

Likely i am missing to understand.

No, it was me who misunderstand ;-)

Quoted 3 lines

Isn't the count incremented earlier than return? I.e even if return happens it should be n + 2 right?

Yeah, you're right, but then why do we want to return earlier? Since the following if:

	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)

will be false, and we will just return from the function, no?

Regards,
Boqun

Quoted 32 lines
Quoted 31 lines
	local_interrupt_enable(); // hardirq disable count is n

	local_interrupt_enable(); // hardirq disable count is n - 1

Regards,
Boqun

Quoted 23 lines

Not sure, if below is any better? (Igore whitespace mangling)

if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) &&
DEBUG_LOCKS_WARN_ON((preempt_count() & HARDIRQ_DISABLE_MASK) >=
HARDIRQ_DISABLE_MASK - (10 << HARDIRQ_DISABLE_SHIFT)))
return;

Quoted 15 lines
include/linux/interrupt_rc.h +5 -0
          /* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
 
          if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
@@ -73,6 +80,11 @@ static inline void local_interrupt_enable(void)
   {
          int new_count;
 
+       /* Unpaired local_interrupt_enable()? Warn and abort. */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) &&
+           DEBUG_LOCKS_WARN_ON((preempt_count() & HARDIRQ_DISABLE_MASK) == 0))
+               return;
+
          new_count = hardirq_disable_exit();
 
          if ((new_count & HARDIRQ_DISABLE_MASK) == 0) >
#27 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

Hi Boqun.

Quoted 39 lines · Boqun Feng
Quoted 28 lines · Shrikanth Hegde
Quoted 25 lines · Boqun Feng
Quoted 24 lines · Shrikanth Hegde

Hi Boqun,

Quoted 21 lines

Something as below? Going to send it to kernel build bot and see if it works for all configs.

----------------->8

include/linux/interrupt_rc.h +1 -0
index b9a7f05ecf42..39f30bc65548 100644
@@ -12,6 +12,7 @@
     */
 
    #include <linux/irqflags.h>
+#include <linux/debug_locks.h>
    #include <linux/preempt.h>
    #include <linux/processor.h>
    #include <linux/smp.h>
@@ -63,6 +64,12 @@ static inline void local_interrupt_disable(void)
 
           new_count = hardirq_disable_enter();
 
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

include/linux/interrupt_rc.h static inline void local_interrupt_disable(void) · line 72

Quoted 6 lines
include/linux/interrupt_rc.h +6 -0
+       /* Is hardirq disable count overflow soon? */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT))
+               DEBUG_LOCKS_WARN_ON((new_count & HARDIRQ_DISABLE_MASK) +
+                                   (10 << HARDIRQ_DISABLE_SHIFT) >
+                                   HARDIRQ_DISABLE_MASK);
+

This needs a return here right? Else we will see warning for 10 times and then overflow happens and we will call _local_interrupt_disable. No?

Quoted 4 lines

Oh, seems I overlooked something... could you elaborate on this? What's the scenario in your mind? You said we hit 10 times warning and *then* overflow?

(the “10 times” wording was for the possible wraparound. I didn't know DEBUG_LOCKS_WARN_ON() will turn debug_locks off after the first warning, I thought it will print 10 times.)

Main concern is the wraparound itself. If the HARDIRQ_DISABLE field reaches 0xff and we increment it once more, the field becomes zero after masking (ff + 1 -> 100 in the shifted field). Then local_interrupt_enable() can observe:

        (new_count & HARDIRQ_DISABLE_MASK) == 0

and treat it as the outermost enable, potentially calling _local_interrupt_enable() while there are still outstanding logical local_interrupt_disable() users.

So I think the useful debug check is one that catches the count before it gets close enough to wrap. Thing I wanted to avoid is silently making the HARDIRQ_DISABLE field look like zero after overflow.

Whether it returns or just warns, I am not sure. As recovery may not be easy. It is meant more to be a damage contol than recovery.

My line of thought is,

If we return after debug checks in local_interrupt_disable(), we wont advance the preempt count further, so it wont wrap around. Now, there will be corresponding local_interrupt_enable(), at some point it will reach 0, we enable the interrupts. There will be some more local_interrupt_enable() still, but they will be caught with your debug check in local_interrupt_enable() and preempt count won't be decremented further.
So if we put return we may have a damage control.

Quoted 70 lines
Quoted 13 lines
Quoted 5 lines
Quoted 1 line

DEBUG_LOCKS_WARN_ON() uses debug_locks_off() to avoid this, so we won't see it 10 times. The reason not using return here, because we would introduce unpaired local_interrupt_disable() if we returned:

Yes, it could be a weird case if the overflow actually happens. So just warning maybe enough to catch such callers.

Maybe your kunit test can actually help test the behavior with the loop count.

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
	// hardirq disable count is n
	local_interrupt_disable(); // hardirq disable count is n + 1

	local_interrupt_disable(); <- trigger the warning, if we return
				   // hardirq disable count is n + 1

Likely i am missing to understand.

No, it was me who misunderstand ;-)

Quoted 3 lines

Isn't the count incremented earlier than return? I.e even if return happens it should be n + 2 right?

Yeah, you're right, but then why do we want to return earlier? Since the following if:

	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)

will be false, and we will just return from the function, no?

Regards,
Boqun

Quoted 32 lines
Quoted 31 lines
	local_interrupt_enable(); // hardirq disable count is n

	local_interrupt_enable(); // hardirq disable count is n - 1

Regards,
Boqun

Quoted 23 lines

Not sure, if below is any better? (Igore whitespace mangling)

if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) &&
DEBUG_LOCKS_WARN_ON((preempt_count() & HARDIRQ_DISABLE_MASK) >=
HARDIRQ_DISABLE_MASK - (10 << HARDIRQ_DISABLE_SHIFT)))
return;

Quoted 15 lines
include/linux/interrupt_rc.h +5 -0
           /* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
 
           if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
@@ -73,6 +80,11 @@ static inline void local_interrupt_enable(void)
    {
           int new_count;
 
+       /* Unpaired local_interrupt_enable()? Warn and abort. */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) &&
+           DEBUG_LOCKS_WARN_ON((preempt_count() & HARDIRQ_DISABLE_MASK) == 0))
+               return;
+
           new_count = hardirq_disable_exit();
 
           if ((new_count & HARDIRQ_DISABLE_MASK) == 0) >
author maintainer
#28 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

Quoted 59 lines · Shrikanth Hegde

Hi Boqun.

Quoted 39 lines · Boqun Feng
Quoted 28 lines · Shrikanth Hegde
Quoted 25 lines · Boqun Feng
Quoted 24 lines · Shrikanth Hegde

Hi Boqun,

Quoted 21 lines

Something as below? Going to send it to kernel build bot and see if it works for all configs.

----------------->8

include/linux/interrupt_rc.h +1 -0
index b9a7f05ecf42..39f30bc65548 100644
@@ -12,6 +12,7 @@
     */
 
    #include <linux/irqflags.h>
+#include <linux/debug_locks.h>
    #include <linux/preempt.h>
    #include <linux/processor.h>
    #include <linux/smp.h>
@@ -63,6 +64,12 @@ static inline void local_interrupt_disable(void)
 
           new_count = hardirq_disable_enter();
 
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

include/linux/interrupt_rc.h static inline void local_interrupt_disable(void) · line 72

Quoted 6 lines
include/linux/interrupt_rc.h +6 -0
+       /* Is hardirq disable count overflow soon? */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT))
+               DEBUG_LOCKS_WARN_ON((new_count & HARDIRQ_DISABLE_MASK) +
+                                   (10 << HARDIRQ_DISABLE_SHIFT) >
+                                   HARDIRQ_DISABLE_MASK);
+

This needs a return here right? Else we will see warning for 10 times and then overflow happens and we will call _local_interrupt_disable. No?

Quoted 4 lines

Oh, seems I overlooked something... could you elaborate on this? What's the scenario in your mind? You said we hit 10 times warning and *then* overflow?

(the "10 times" wording was for the possible wraparound. I didn't know DEBUG_LOCKS_WARN_ON() will turn debug_locks off after the first warning, I thought it will print 10 times.)

Main concern is the wraparound itself. If the HARDIRQ_DISABLE field reaches 0xff and we increment it once more, the field becomes zero after masking (ff + 1 -> 100 in the shifted field). Then local_interrupt_enable() can observe:

        (new_count & HARDIRQ_DISABLE_MASK) == 0
Quoted 8 lines

and treat it as the outermost enable, potentially calling _local_interrupt_enable() while there are still outstanding logical local_interrupt_disable() users.

So I think the useful debug check is one that catches the count before it gets close enough to wrap. Thing I wanted to avoid is silently making the HARDIRQ_DISABLE field look like zero after overflow.

Given that the detection is only on debug kernel, I think the damage of keeping increment after warn triggered is low (this is assuming that we can detect all the issues with the debug kernel, similar treatment as the preempt_count nowadays). However, your suggestion does look better for the corner cases and it's a better damage control, so I will add the return part, thank you!

For the future, we need better detection and report for these counters for sure.

Regards,
Boqun

Quoted 84 lines

Whether it returns or just warns, I am not sure. As recovery may not be easy. It is meant more to be a damage contol than recovery.

My line of thought is,

If we return after debug checks in local_interrupt_disable(), we wont advance the preempt count further, so it wont wrap around. Now, there will be corresponding local_interrupt_enable(), at some point it will reach 0, we enable the interrupts. There will be some more local_interrupt_enable() still, but they will be caught with your debug check in local_interrupt_enable() and preempt count won't be decremented further.
So if we put return we may have a damage control.

Quoted 70 lines
Quoted 13 lines
Quoted 5 lines
Quoted 1 line

DEBUG_LOCKS_WARN_ON() uses debug_locks_off() to avoid this, so we won't see it 10 times. The reason not using return here, because we would introduce unpaired local_interrupt_disable() if we returned:

Yes, it could be a weird case if the overflow actually happens. So just warning maybe enough to catch such callers.

Maybe your kunit test can actually help test the behavior with the loop count.

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
	// hardirq disable count is n
	local_interrupt_disable(); // hardirq disable count is n + 1

	local_interrupt_disable(); <- trigger the warning, if we return
				   // hardirq disable count is n + 1

Likely i am missing to understand.

No, it was me who misunderstand ;-)

Quoted 3 lines

Isn't the count incremented earlier than return? I.e even if return happens it should be n + 2 right?

Yeah, you're right, but then why do we want to return earlier? Since the following if:

	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)

will be false, and we will just return from the function, no?

Regards,
Boqun

Quoted 32 lines
Quoted 31 lines
	local_interrupt_enable(); // hardirq disable count is n

	local_interrupt_enable(); // hardirq disable count is n - 1

Regards,
Boqun

Quoted 23 lines

Not sure, if below is any better? (Igore whitespace mangling)

if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) &&
DEBUG_LOCKS_WARN_ON((preempt_count() & HARDIRQ_DISABLE_MASK) >=
HARDIRQ_DISABLE_MASK - (10 << HARDIRQ_DISABLE_SHIFT)))
return;

Quoted 15 lines
include/linux/interrupt_rc.h +5 -0
           /* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
 
           if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
@@ -73,6 +80,11 @@ static inline void local_interrupt_enable(void)
    {
           int new_count;
 
+       /* Unpaired local_interrupt_enable()? Warn and abort. */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT) &&
+           DEBUG_LOCKS_WARN_ON((preempt_count() & HARDIRQ_DISABLE_MASK) == 0))
+               return;
+
           new_count = hardirq_disable_exit();
 
           if ((new_count & HARDIRQ_DISABLE_MASK) == 0) >
author maintainer
#29 lore raw

Re: [PATCH v4 05/17] irq & spin_lock: Add counted interrupt disabling/enabling

In reply to Shrikanth Hegde
Quoted 43 lines · Shrikanth Hegde
Quoted 36 lines · Boqun Feng
Quoted 24 lines · Shrikanth Hegde

Hi Boqun,

Quoted 21 lines

Something as below? Going to send it to kernel build bot and see if it works for all configs.

----------------->8

include/linux/interrupt_rc.h +1 -0
index b9a7f05ecf42..39f30bc65548 100644
@@ -12,6 +12,7 @@
    */
 
   #include <linux/irqflags.h>
+#include <linux/debug_locks.h>
   #include <linux/preempt.h>
   #include <linux/processor.h>
   #include <linux/smp.h>
@@ -63,6 +64,12 @@ static inline void local_interrupt_disable(void)
 
          new_count = hardirq_disable_enter();
 
Quoted 10 lines

include/linux/interrupt_rc.h static inline void local_interrupt_disable(void) · line 72

Quoted 6 lines
include/linux/interrupt_rc.h +6 -0
+       /* Is hardirq disable count overflow soon? */
+       if (IS_ENABLED(CONFIG_DEBUG_PREEMPT))
+               DEBUG_LOCKS_WARN_ON((new_count & HARDIRQ_DISABLE_MASK) +
+                                   (10 << HARDIRQ_DISABLE_SHIFT) >
+                                   HARDIRQ_DISABLE_MASK);
+

This needs a return here right? Else we will see warning for 10 times and then overflow happens and we will call _local_interrupt_disable. No?

Quoted 3 lines

DEBUG_LOCKS_WARN_ON() uses debug_locks_off() to avoid this, so we won't see it 10 times. The reason not using return here, because we would introduce unpaired local_interrupt_disable() if we returned:

Quoted 6 lines

Yes, it could be a weird case if the overflow actually happens. So just warning maybe enough to catch such callers.

Maybe your kunit test can actually help test the behavior with the loop count.

I'm sure that we can have more tests, but this is a good start, thank you!

Regards,
Boqun

------------->8
Subject: [PATCH] irq: Add max local_interrupt_disable() nesting level kunit
test case

To confirm the max nesting level of local_interrupt_disable() works, a kunit test is added to the whole test suite.

Note that when DEBUG_PREEMPT=y, it'll generate a warning which is expected.

---
 kernel/irq/refcount_interrupt_test.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
kernel/irq/refcount_interrupt_test.c +22 -0
index ca904dba24b9..38dfccbaa4d4 100644
@@ -52,6 +52,27 @@ static void test_multiple_irq_change(struct kunit *test)
 	TEST_IRQ_ON();
 }
 
+static void test_max_nesting_irq_change(struct kunit *test)
+{
+	for (int i = 0; i < __IRQ_MASK(HARDIRQ_DISABLE_BITS); i++) {
+		local_interrupt_disable();
+		TEST_IRQ_OFF();
+	}
+
+
+	for (int i = 0; i < __IRQ_MASK(HARDIRQ_DISABLE_BITS); i++) {
+		TEST_IRQ_OFF();
+		local_interrupt_enable();
+	}
+
+	TEST_IRQ_ON();
+
+	local_interrupt_disable();
+	TEST_IRQ_OFF();
+	local_interrupt_enable();
+	TEST_IRQ_ON();
+}
+
 static void test_irq_save(struct kunit *test)
 {
 	unsigned long flags;
@@ -79,6 +100,7 @@ static struct kunit_case test_cases[] = {
 	KUNIT_CASE(test_single_irq_change),
 	KUNIT_CASE(test_nested_irq_change),
 	KUNIT_CASE(test_multiple_irq_change),
+	KUNIT_CASE(test_max_nesting_irq_change),
 	KUNIT_CASE(test_irq_save),
 	{},
 };
Signature · 1 line
2.50.1 (Apple Git-155)
author maintainer
#30 lore raw

PATCH v4 6/17 irq: Add KUnit test for refcounted interrupt enable/disable

In reply to Boqun Feng

From: Lyude Paul <[email protected]>

While making changes to the refcounted interrupt patch series, at some point on my local branch I broke something and ended up writing some kunit tests for testing refcounted interrupts as a result. So, let's include these tests now that we have refcounted interrupts.

---
 kernel/irq/Makefile                  |   1 +
 kernel/irq/refcount_interrupt_test.c | 109 +++++++++++++++++++++++++++
 2 files changed, 110 insertions(+)
 create mode 100644 kernel/irq/refcount_interrupt_test.c
kernel/irq/Makefile +1 -0
index 86a2e5ae08f9..44c4d6fc502a 100644
@@ -16,3 +16,4 @@ obj-$(CONFIG_SMP) += affinity.o
 obj-$(CONFIG_GENERIC_IRQ_DEBUGFS) += debugfs.o
 obj-$(CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR) += matrix.o
 obj-$(CONFIG_IRQ_KUNIT_TEST) += irq_test.o
+obj-$(CONFIG_KUNIT) += refcount_interrupt_test.o
kernel/irq/refcount_interrupt_test.c new file +109 -0

109 lines not on this page · raw message

Signature · 1 line
2.50.1 (Apple Git-155)
merge bot Applied
#31 lore raw

[tip: locking/core] irq: Add KUnit test for refcounted interrupt enable/disable

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     417f9684716b6577c717b09337e14040a1a067f2
Gitweb:        https://git.kernel.org/tip/417f9684716b6577c717b09337e14040a1a067f2
Author:        Lyude Paul <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:28 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 08 Aug 2026 22:44:07 +02:00

irq: Add KUnit test for refcounted interrupt enable/disable

While making changes to the refcounted interrupt patch series, at some point on my local branch I broke something and ended up writing some kunit tests for testing refcounted interrupts as a result. So, let's include these tests now that we have refcounted interrupts.

---
 kernel/irq/Makefile                  |   1 +-
 kernel/irq/refcount_interrupt_test.c | 109 ++++++++++++++++++++++++++-
 2 files changed, 110 insertions(+)
 create mode 100644 kernel/irq/refcount_interrupt_test.c
kernel/irq/Makefile +1 -0
index 86a2e5a..44c4d6f 100644
@@ -16,3 +16,4 @@ obj-$(CONFIG_SMP) += affinity.o
 obj-$(CONFIG_GENERIC_IRQ_DEBUGFS) += debugfs.o
 obj-$(CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR) += matrix.o
 obj-$(CONFIG_IRQ_KUNIT_TEST) += irq_test.o
+obj-$(CONFIG_KUNIT) += refcount_interrupt_test.o
kernel/irq/refcount_interrupt_test.c new file +109 -0

109 lines not on this page · raw message

merge bot Applied
#32 lore raw

[tip: locking/core] irq: Add KUnit test for refcounted interrupt enable/disable

In reply to Boqun Feng
The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     07a88e2bcd5b5bd7881b2e12b6aad1897a7ee1de
Gitweb:        https://git.kernel.org/tip/07a88e2bcd5b5bd7881b2e12b6aad1897a7ee1de
Author:        Lyude Paul <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:28 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 10 Aug 2026 10:50:18 +02:00

irq: Add KUnit test for refcounted interrupt enable/disable

While making changes to the refcounted interrupt patch series, at some point on my local branch I broke something and ended up writing some kunit tests for testing refcounted interrupts as a result. So, let's include these tests now that we have refcounted interrupts.

---
 kernel/irq/Makefile                  |   1 +-
 kernel/irq/refcount_interrupt_test.c | 109 ++++++++++++++++++++++++++-
 2 files changed, 110 insertions(+)
 create mode 100644 kernel/irq/refcount_interrupt_test.c
kernel/irq/Makefile +1 -0
index 86a2e5a..44c4d6f 100644
@@ -16,3 +16,4 @@ obj-$(CONFIG_SMP) += affinity.o
 obj-$(CONFIG_GENERIC_IRQ_DEBUGFS) += debugfs.o
 obj-$(CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR) += matrix.o
 obj-$(CONFIG_IRQ_KUNIT_TEST) += irq_test.o
+obj-$(CONFIG_KUNIT) += refcount_interrupt_test.o
kernel/irq/refcount_interrupt_test.c new file +109 -0

109 lines not on this page · raw message

author maintainer
#33 lore raw

PATCH v4 7/17 locking: Switch to _irq_{disable,enable}() variants in cleanup guards

In reply to Boqun Feng

The semantics of various IRQ disabling guards match what *_irq_{disable,enable}() provide, i.e. the interrupt disabling is properly nested, therefore it's OK to switch to use *_irq_{disable,enable}() primitives.

[boqun: Adjust the user-side changes in do_sched_cfs_*_timer() provided by Peter and Lyude]

---
 include/linux/spinlock.h | 26 ++++++++++++--------------
 kernel/sched/fair.c      | 12 ++++++------
 2 files changed, 18 insertions(+), 20 deletions(-)
include/linux/spinlock.h +12 -14
index 3d405cc4c121..799a8f7d2741 100644
@@ -572,12 +572,12 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, __acquires(_T), __releases(*(raw
 #define class_raw_spinlock_nested_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, _T)
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irq, raw_spinlock_t,
-		    raw_spin_lock_irq(_T->lock),
-		    raw_spin_unlock_irq(_T->lock))
+		    raw_spin_lock_irq_disable(_T->lock),
+		    raw_spin_unlock_irq_enable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, _T)
 
-DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq(_T->lock))
+DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq_disable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, _T)
 
@@ -592,14 +592,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_bh_try, __acquires(_T), __releases(*(raw
 #define class_raw_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_bh_try, _T)
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irqsave, raw_spinlock_t,
-		    raw_spin_lock_irqsave(_T->lock, _T->flags),
-		    raw_spin_unlock_irqrestore(_T->lock, _T->flags),
-		    unsigned long flags)
+		    raw_spin_lock_irq_disable(_T->lock),
+		    raw_spin_unlock_irq_enable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave, _T)
 
 DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irqsave, _try,
-			 raw_spin_trylock_irqsave(_T->lock, _T->flags))
+			 raw_spin_trylock_irq_disable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave_try, _T)
 
@@ -618,13 +617,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_try, __acquires(_T), __releases(*(spinlock_t
 #define class_spinlock_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_try, _T)
 
 DEFINE_LOCK_GUARD_1(spinlock_irq, spinlock_t,
-		    spin_lock_irq(_T->lock),
-		    spin_unlock_irq(_T->lock))
+		    spin_lock_irq_disable(_T->lock),
+		    spin_unlock_irq_enable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq, _T)
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irq, _try,
-			 spin_trylock_irq(_T->lock))
+			 spin_trylock_irq_disable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq_try, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq_try, _T)
 
@@ -640,14 +639,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_bh_try, __acquires(_T), __releases(*(spinloc
 #define class_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_bh_try, _T)
 
 DEFINE_LOCK_GUARD_1(spinlock_irqsave, spinlock_t,
-		    spin_lock_irqsave(_T->lock, _T->flags),
-		    spin_unlock_irqrestore(_T->lock, _T->flags),
-		    unsigned long flags)
+		    spin_lock_irq_disable(_T->lock),
+		    spin_unlock_irq_enable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irqsave, _T)
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irqsave, _try,
-			 spin_trylock_irqsave(_T->lock, _T->flags))
+			 spin_trylock_irq_disable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave_try, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irqsave_try, _T)
 
kernel/sched/fair.c +6 -6
index d78467ec6ee1..a46c4ff49f74 100644
@@ -7120,7 +7120,7 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
  * period the timer is deactivated until scheduling resumes; cfs_b->idle is
  * used to track this state.
  */
-static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun, unsigned long flags)
+static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun)
 	__must_hold(&cfs_b->lock)
 {
 	int throttled;
@@ -7155,10 +7155,10 @@ static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun, u
 	 * This check is repeated as we release cfs_b->lock while we unthrottle.
 	 */
 	while (throttled && cfs_b->runtime > 0) {
-		raw_spin_unlock_irqrestore(&cfs_b->lock, flags);
+		raw_spin_unlock_irq_enable(&cfs_b->lock);
 		/* we can't nest cfs_b->lock while distributing bandwidth */
 		throttled = distribute_cfs_runtime(cfs_b);
-		raw_spin_lock_irqsave(&cfs_b->lock, flags);
+		raw_spin_lock_irq_disable(&cfs_b->lock);
 	}
 
 	/*
@@ -7266,7 +7266,7 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq)
 static void do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b)
 {
 	/* confirm we're still not at a refresh boundary */
-	scoped_guard(raw_spinlock_irqsave, &cfs_b->lock) {
+	scoped_guard(raw_spinlock_irq, &cfs_b->lock) {
 		u64 runtime = 0, slice = sched_cfs_bandwidth_slice();
 
 		cfs_b->slack_started = false;
@@ -7351,14 +7351,14 @@ static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer)
 	int idle = 0;
 	int count = 0;
 
-	CLASS(raw_spinlock_irqsave, cfsb_guard)(&cfs_b->lock);
+	guard(raw_spinlock_irq)(&cfs_b->lock);
 
 	for (;;) {
 		overrun = hrtimer_forward_now(timer, cfs_b->period);
 		if (!overrun)
 			break;
 
-		idle = do_sched_cfs_period_timer(cfs_b, overrun, cfsb_guard.flags);
+		idle = do_sched_cfs_period_timer(cfs_b, overrun);
 
 		if (++count > 3) {
 			u64 new, old = ktime_to_ns(cfs_b->period);
Signature · 1 line
2.50.1 (Apple Git-155)
merge bot Applied
#34 lore raw

[tip: locking/core] locking: Switch to _irq_{disable,enable}() variants in cleanup guards

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     d512f73c83c3a74073833e7d29584e890f90db36
Gitweb:        https://git.kernel.org/tip/d512f73c83c3a74073833e7d29584e890f90db36
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:29 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 08 Aug 2026 22:44:07 +02:00

locking: Switch to _irq_{disable,enable}() variants in cleanup guards

The semantics of various IRQ disabling guards match what *_irq_{disable,enable}() provide, i.e. the interrupt disabling is properly nested, therefore it's OK to switch to use *_irq_{disable,enable}() primitives.

[boqun: Adjust the user-side changes in do_sched_cfs_*_timer() provided by Peter and Lyude]

---
 include/linux/spinlock.h | 26 ++++++++++++--------------
 kernel/sched/fair.c      | 12 ++++++------
 2 files changed, 18 insertions(+), 20 deletions(-)
include/linux/spinlock.h +12 -14
index 3d405cc..799a8f7 100644
@@ -572,12 +572,12 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, __acquires(_T), __releases(*(raw
 #define class_raw_spinlock_nested_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, _T)
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irq, raw_spinlock_t,
-		    raw_spin_lock_irq(_T->lock),
-		    raw_spin_unlock_irq(_T->lock))
+		    raw_spin_lock_irq_disable(_T->lock),
+		    raw_spin_unlock_irq_enable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, _T)
 
-DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq(_T->lock))
+DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq_disable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, _T)
 
@@ -592,14 +592,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_bh_try, __acquires(_T), __releases(*(raw
 #define class_raw_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_bh_try, _T)
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irqsave, raw_spinlock_t,
-		    raw_spin_lock_irqsave(_T->lock, _T->flags),
-		    raw_spin_unlock_irqrestore(_T->lock, _T->flags),
-		    unsigned long flags)
+		    raw_spin_lock_irq_disable(_T->lock),
+		    raw_spin_unlock_irq_enable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave, _T)
 
 DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irqsave, _try,
-			 raw_spin_trylock_irqsave(_T->lock, _T->flags))
+			 raw_spin_trylock_irq_disable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave_try, _T)
 
@@ -618,13 +617,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_try, __acquires(_T), __releases(*(spinlock_t
 #define class_spinlock_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_try, _T)
 
 DEFINE_LOCK_GUARD_1(spinlock_irq, spinlock_t,
-		    spin_lock_irq(_T->lock),
-		    spin_unlock_irq(_T->lock))
+		    spin_lock_irq_disable(_T->lock),
+		    spin_unlock_irq_enable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq, _T)
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irq, _try,
-			 spin_trylock_irq(_T->lock))
+			 spin_trylock_irq_disable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq_try, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq_try, _T)
 
@@ -640,14 +639,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_bh_try, __acquires(_T), __releases(*(spinloc
 #define class_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_bh_try, _T)
 
 DEFINE_LOCK_GUARD_1(spinlock_irqsave, spinlock_t,
-		    spin_lock_irqsave(_T->lock, _T->flags),
-		    spin_unlock_irqrestore(_T->lock, _T->flags),
-		    unsigned long flags)
+		    spin_lock_irq_disable(_T->lock),
+		    spin_unlock_irq_enable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irqsave, _T)
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irqsave, _try,
-			 spin_trylock_irqsave(_T->lock, _T->flags))
+			 spin_trylock_irq_disable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave_try, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irqsave_try, _T)
 
kernel/sched/fair.c +6 -6
index d78467e..a46c4ff 100644
@@ -7120,7 +7120,7 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
  * period the timer is deactivated until scheduling resumes; cfs_b->idle is
  * used to track this state.
  */
-static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun, unsigned long flags)
+static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun)
 	__must_hold(&cfs_b->lock)
 {
 	int throttled;
@@ -7155,10 +7155,10 @@ static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun, u
 	 * This check is repeated as we release cfs_b->lock while we unthrottle.
 	 */
 	while (throttled && cfs_b->runtime > 0) {
-		raw_spin_unlock_irqrestore(&cfs_b->lock, flags);
+		raw_spin_unlock_irq_enable(&cfs_b->lock);
 		/* we can't nest cfs_b->lock while distributing bandwidth */
 		throttled = distribute_cfs_runtime(cfs_b);
-		raw_spin_lock_irqsave(&cfs_b->lock, flags);
+		raw_spin_lock_irq_disable(&cfs_b->lock);
 	}
 
 	/*
@@ -7266,7 +7266,7 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq)
 static void do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b)
 {
 	/* confirm we're still not at a refresh boundary */
-	scoped_guard(raw_spinlock_irqsave, &cfs_b->lock) {
+	scoped_guard(raw_spinlock_irq, &cfs_b->lock) {
 		u64 runtime = 0, slice = sched_cfs_bandwidth_slice();
 
 		cfs_b->slack_started = false;
@@ -7351,14 +7351,14 @@ static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer)
 	int idle = 0;
 	int count = 0;
 
-	CLASS(raw_spinlock_irqsave, cfsb_guard)(&cfs_b->lock);
+	guard(raw_spinlock_irq)(&cfs_b->lock);
 
 	for (;;) {
 		overrun = hrtimer_forward_now(timer, cfs_b->period);
 		if (!overrun)
 			break;
 
-		idle = do_sched_cfs_period_timer(cfs_b, overrun, cfsb_guard.flags);
+		idle = do_sched_cfs_period_timer(cfs_b, overrun);
 
 		if (++count > 3) {
 			u64 new, old = ktime_to_ns(cfs_b->period);
merge bot Applied
#35 lore raw

[tip: locking/core] locking: Switch to _irq_{disable,enable}() variants in cleanup guards

In reply to Boqun Feng
The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     1b086687483371621e684e2a05b2bcd2cf07b634
Gitweb:        https://git.kernel.org/tip/1b086687483371621e684e2a05b2bcd2cf07b634
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:29 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 10 Aug 2026 10:50:18 +02:00

locking: Switch to _irq_{disable,enable}() variants in cleanup guards

The semantics of various IRQ disabling guards match what *_irq_{disable,enable}() provide, i.e. the interrupt disabling is properly nested, therefore it's OK to switch to use *_irq_{disable,enable}() primitives.

[boqun: Adjust the user-side changes in do_sched_cfs_*_timer() provided by Peter and Lyude]

---
 include/linux/spinlock.h | 26 ++++++++++++--------------
 kernel/sched/fair.c      | 12 ++++++------
 2 files changed, 18 insertions(+), 20 deletions(-)
include/linux/spinlock.h +12 -14
index 3d405cc..799a8f7 100644
@@ -572,12 +572,12 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, __acquires(_T), __releases(*(raw
 #define class_raw_spinlock_nested_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, _T)
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irq, raw_spinlock_t,
-		    raw_spin_lock_irq(_T->lock),
-		    raw_spin_unlock_irq(_T->lock))
+		    raw_spin_lock_irq_disable(_T->lock),
+		    raw_spin_unlock_irq_enable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, _T)
 
-DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq(_T->lock))
+DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq_disable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, _T)
 
@@ -592,14 +592,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_bh_try, __acquires(_T), __releases(*(raw
 #define class_raw_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_bh_try, _T)
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irqsave, raw_spinlock_t,
-		    raw_spin_lock_irqsave(_T->lock, _T->flags),
-		    raw_spin_unlock_irqrestore(_T->lock, _T->flags),
-		    unsigned long flags)
+		    raw_spin_lock_irq_disable(_T->lock),
+		    raw_spin_unlock_irq_enable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave, _T)
 
 DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irqsave, _try,
-			 raw_spin_trylock_irqsave(_T->lock, _T->flags))
+			 raw_spin_trylock_irq_disable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave_try, _T)
 
@@ -618,13 +617,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_try, __acquires(_T), __releases(*(spinlock_t
 #define class_spinlock_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_try, _T)
 
 DEFINE_LOCK_GUARD_1(spinlock_irq, spinlock_t,
-		    spin_lock_irq(_T->lock),
-		    spin_unlock_irq(_T->lock))
+		    spin_lock_irq_disable(_T->lock),
+		    spin_unlock_irq_enable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq, _T)
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irq, _try,
-			 spin_trylock_irq(_T->lock))
+			 spin_trylock_irq_disable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq_try, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq_try, _T)
 
@@ -640,14 +639,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_bh_try, __acquires(_T), __releases(*(spinloc
 #define class_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_bh_try, _T)
 
 DEFINE_LOCK_GUARD_1(spinlock_irqsave, spinlock_t,
-		    spin_lock_irqsave(_T->lock, _T->flags),
-		    spin_unlock_irqrestore(_T->lock, _T->flags),
-		    unsigned long flags)
+		    spin_lock_irq_disable(_T->lock),
+		    spin_unlock_irq_enable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irqsave, _T)
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irqsave, _try,
-			 spin_trylock_irqsave(_T->lock, _T->flags))
+			 spin_trylock_irq_disable(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave_try, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irqsave_try, _T)
 
kernel/sched/fair.c +6 -6
index d78467e..a46c4ff 100644
@@ -7120,7 +7120,7 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
  * period the timer is deactivated until scheduling resumes; cfs_b->idle is
  * used to track this state.
  */
-static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun, unsigned long flags)
+static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun)
 	__must_hold(&cfs_b->lock)
 {
 	int throttled;
@@ -7155,10 +7155,10 @@ static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun, u
 	 * This check is repeated as we release cfs_b->lock while we unthrottle.
 	 */
 	while (throttled && cfs_b->runtime > 0) {
-		raw_spin_unlock_irqrestore(&cfs_b->lock, flags);
+		raw_spin_unlock_irq_enable(&cfs_b->lock);
 		/* we can't nest cfs_b->lock while distributing bandwidth */
 		throttled = distribute_cfs_runtime(cfs_b);
-		raw_spin_lock_irqsave(&cfs_b->lock, flags);
+		raw_spin_lock_irq_disable(&cfs_b->lock);
 	}
 
 	/*
@@ -7266,7 +7266,7 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq)
 static void do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b)
 {
 	/* confirm we're still not at a refresh boundary */
-	scoped_guard(raw_spinlock_irqsave, &cfs_b->lock) {
+	scoped_guard(raw_spinlock_irq, &cfs_b->lock) {
 		u64 runtime = 0, slice = sched_cfs_bandwidth_slice();
 
 		cfs_b->slack_started = false;
@@ -7351,14 +7351,14 @@ static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer)
 	int idle = 0;
 	int count = 0;
 
-	CLASS(raw_spinlock_irqsave, cfsb_guard)(&cfs_b->lock);
+	guard(raw_spinlock_irq)(&cfs_b->lock);
 
 	for (;;) {
 		overrun = hrtimer_forward_now(timer, cfs_b->period);
 		if (!overrun)
 			break;
 
-		idle = do_sched_cfs_period_timer(cfs_b, overrun, cfsb_guard.flags);
+		idle = do_sched_cfs_period_timer(cfs_b, overrun);
 
 		if (++count > 3) {
 			u64 new, old = ktime_to_ns(cfs_b->period);
maintainer
#36 lore raw

Re: [tip: locking/core] locking: Switch to _irq_{disable,enable}() variants in cleanup guards

Quoted 16 lines · On Mon, Aug 10, 2026 at 08:57:43AM -0000, tip-bot2 for Boqun Feng wrote:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     1b086687483371621e684e2a05b2bcd2cf07b634
Gitweb:        https://git.kernel.org/tip/1b086687483371621e684e2a05b2bcd2cf07b634
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:29 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 10 Aug 2026 10:50:18 +02:00

locking: Switch to _irq_{disable,enable}() variants in cleanup guards

The semantics of various IRQ disabling guards match what *_irq_{disable,enable}() provide, i.e. the interrupt disabling is properly nested, therefore it's OK to switch to use *_irq_{disable,enable}() primitives.

Quoted 6 lines

[boqun: Adjust the user-side changes in do_sched_cfs_*_timer() provided by Peter and Lyude]

I'm going to revert this patch, the rest can stay, but this patch is broken vs existing code.

Notably, the one found by syzbot is postix_timer_delete() doing spin_unlock_irq()+spin_lock_irq() inside a scoped_guard(spinlock_irq).

We're going to need stronger tools/debug before trying this again.

maintainer
#37 lore raw

PATCH 1/1 locking: Revert switching guards to _irq_{disable,enable}()

While the guards are properly nested, not all wrapped code is nice, as already highlighted by that fair.c hunk.

Syzbot found another instance of this pattern in posix_timer_delete(), which does spin_unlock_irq()+spin_lock_irq() inside scoped_guard(spinlock_irq). Combined with this patch, that goes sideways most spectacular.

Undo this change, until we've developed stronger tools / debug for such issues.

---
 include/linux/spinlock.h |   26 ++++++++++++++------------
 kernel/sched/fair.c      |   12 ++++++------
 2 files changed, 20 insertions(+), 18 deletions(-)
include/linux/spinlock.h +14 -12
@@ -572,12 +572,12 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_
 #define class_raw_spinlock_nested_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, _T)
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irq, raw_spinlock_t,
-		    raw_spin_lock_irq_disable(_T->lock),
-		    raw_spin_unlock_irq_enable(_T->lock))
+		    raw_spin_lock_irq(_T->lock),
+		    raw_spin_unlock_irq(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, _T)
 
-DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq_disable(_T->lock))
+DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, _T)
 
@@ -592,13 +592,14 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_
 #define class_raw_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_bh_try, _T)
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irqsave, raw_spinlock_t,
-		    raw_spin_lock_irq_disable(_T->lock),
-		    raw_spin_unlock_irq_enable(_T->lock))
+		    raw_spin_lock_irqsave(_T->lock, _T->flags),
+		    raw_spin_unlock_irqrestore(_T->lock, _T->flags),
+		    unsigned long flags)
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave, _T)
 
 DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irqsave, _try,
-			 raw_spin_trylock_irq_disable(_T->lock))
+			 raw_spin_trylock_irqsave(_T->lock, _T->flags))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave_try, _T)
 
@@ -617,13 +618,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_try,
 #define class_spinlock_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_try, _T)
 
 DEFINE_LOCK_GUARD_1(spinlock_irq, spinlock_t,
-		    spin_lock_irq_disable(_T->lock),
-		    spin_unlock_irq_enable(_T->lock))
+		    spin_lock_irq(_T->lock),
+		    spin_unlock_irq(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq, _T)
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irq, _try,
-			 spin_trylock_irq_disable(_T->lock))
+			 spin_trylock_irq(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq_try, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq_try, _T)
 
@@ -639,13 +640,14 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_bh_t
 #define class_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_bh_try, _T)
 
 DEFINE_LOCK_GUARD_1(spinlock_irqsave, spinlock_t,
-		    spin_lock_irq_disable(_T->lock),
-		    spin_unlock_irq_enable(_T->lock))
+		    spin_lock_irqsave(_T->lock, _T->flags),
+		    spin_unlock_irqrestore(_T->lock, _T->flags),
+		    unsigned long flags)
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irqsave, _T)
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irqsave, _try,
-			 spin_trylock_irq_disable(_T->lock))
+			 spin_trylock_irqsave(_T->lock, _T->flags))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave_try, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irqsave_try, _T)
 
kernel/sched/fair.c +6 -6
@@ -7253,7 +7253,7 @@ static bool distribute_cfs_runtime(struc
  * period the timer is deactivated until scheduling resumes; cfs_b->idle is
  * used to track this state.
  */
-static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun)
+static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun, unsigned long flags)
 	__must_hold(&cfs_b->lock)
 {
 	int throttled;
@@ -7288,10 +7288,10 @@ static int do_sched_cfs_period_timer(str
 	 * This check is repeated as we release cfs_b->lock while we unthrottle.
 	 */
 	while (throttled && cfs_b->runtime > 0) {
-		raw_spin_unlock_irq_enable(&cfs_b->lock);
+		raw_spin_unlock_irqrestore(&cfs_b->lock, flags);
 		/* we can't nest cfs_b->lock while distributing bandwidth */
 		throttled = distribute_cfs_runtime(cfs_b);
-		raw_spin_lock_irq_disable(&cfs_b->lock);
+		raw_spin_lock_irqsave(&cfs_b->lock, flags);
 	}
 
 	/*
@@ -7399,7 +7399,7 @@ static __always_inline void return_cfs_r
 static void do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b)
 {
 	/* confirm we're still not at a refresh boundary */
-	scoped_guard(raw_spinlock_irq, &cfs_b->lock) {
+	scoped_guard(raw_spinlock_irqsave, &cfs_b->lock) {
 		u64 runtime = 0, slice = sched_cfs_bandwidth_slice();
 
 		cfs_b->slack_started = false;
@@ -7484,14 +7484,14 @@ static enum hrtimer_restart sched_cfs_pe
 	int idle = 0;
 	int count = 0;
 
-	guard(raw_spinlock_irq)(&cfs_b->lock);
+	CLASS(raw_spinlock_irqsave, cfsb_guard)(&cfs_b->lock);
 
 	for (;;) {
 		overrun = hrtimer_forward_now(timer, cfs_b->period);
 		if (!overrun)
 			break;
 
-		idle = do_sched_cfs_period_timer(cfs_b, overrun);
+		idle = do_sched_cfs_period_timer(cfs_b, overrun, cfsb_guard.flags);
 
 		if (++count > 3) {
 			u64 new, old = ktime_to_ns(cfs_b->period);
merge bot Applied
#38 lore raw

[tip: locking/urgent] locking: Revert switching guards to _irq_{disable,enable}()

The following commit has been merged into the locking/urgent branch of tip:

The following commit has been merged into the locking/urgent branch of tip:

Commit-ID:     46094a7708b7945cb7eba9eb887e3ea9757440a7
Gitweb:        https://git.kernel.org/tip/46094a7708b7945cb7eba9eb887e3ea9757440a7
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Mon, 24 Aug 2026 12:49:10 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 24 Aug 2026 12:58:54 +02:00

locking: Revert switching guards to _irq_{disable,enable}()

Revert commit 1b0866874833 ("locking: Switch to _irq_{disable,enable}() variants in cleanup guards").

While the guards are properly nested, not all wrapped code is nice, as already highlighted by that fair.c hunk.

Syzbot found another instance of this pattern in posix_timer_delete(), which does spin_unlock_irq()+spin_lock_irq() inside scoped_guard(spinlock_irq). Combined with this patch, that goes sideways most spectacular.

Undo this until we've developed stronger tools / debug for such issues.

Fixes 1b0866874833 ("locking: Switch to _irq_{disable,enable}() variants in cleanup guards")
---
 include/linux/spinlock.h | 26 ++++++++++++++------------
 kernel/sched/fair.c      | 12 ++++++------
 2 files changed, 20 insertions(+), 18 deletions(-)
include/linux/spinlock.h +14 -12
index 799a8f7..3d405cc 100644
@@ -572,12 +572,12 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, __acquires(_T), __releases(*(raw
 #define class_raw_spinlock_nested_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, _T)
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irq, raw_spinlock_t,
-		    raw_spin_lock_irq_disable(_T->lock),
-		    raw_spin_unlock_irq_enable(_T->lock))
+		    raw_spin_lock_irq(_T->lock),
+		    raw_spin_unlock_irq(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, _T)
 
-DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq_disable(_T->lock))
+DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, _T)
 
@@ -592,13 +592,14 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_bh_try, __acquires(_T), __releases(*(raw
 #define class_raw_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_bh_try, _T)
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irqsave, raw_spinlock_t,
-		    raw_spin_lock_irq_disable(_T->lock),
-		    raw_spin_unlock_irq_enable(_T->lock))
+		    raw_spin_lock_irqsave(_T->lock, _T->flags),
+		    raw_spin_unlock_irqrestore(_T->lock, _T->flags),
+		    unsigned long flags)
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave, _T)
 
 DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irqsave, _try,
-			 raw_spin_trylock_irq_disable(_T->lock))
+			 raw_spin_trylock_irqsave(_T->lock, _T->flags))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave_try, _T)
 
@@ -617,13 +618,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_try, __acquires(_T), __releases(*(spinlock_t
 #define class_spinlock_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_try, _T)
 
 DEFINE_LOCK_GUARD_1(spinlock_irq, spinlock_t,
-		    spin_lock_irq_disable(_T->lock),
-		    spin_unlock_irq_enable(_T->lock))
+		    spin_lock_irq(_T->lock),
+		    spin_unlock_irq(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq, _T)
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irq, _try,
-			 spin_trylock_irq_disable(_T->lock))
+			 spin_trylock_irq(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq_try, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq_try, _T)
 
@@ -639,13 +640,14 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_bh_try, __acquires(_T), __releases(*(spinloc
 #define class_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_bh_try, _T)
 
 DEFINE_LOCK_GUARD_1(spinlock_irqsave, spinlock_t,
-		    spin_lock_irq_disable(_T->lock),
-		    spin_unlock_irq_enable(_T->lock))
+		    spin_lock_irqsave(_T->lock, _T->flags),
+		    spin_unlock_irqrestore(_T->lock, _T->flags),
+		    unsigned long flags)
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irqsave, _T)
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irqsave, _try,
-			 spin_trylock_irq_disable(_T->lock))
+			 spin_trylock_irqsave(_T->lock, _T->flags))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave_try, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irqsave_try, _T)
 
kernel/sched/fair.c +6 -6
index 6d881e5..8dff370 100644
@@ -7253,7 +7253,7 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b)
  * period the timer is deactivated until scheduling resumes; cfs_b->idle is
  * used to track this state.
  */
-static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun)
+static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun, unsigned long flags)
 	__must_hold(&cfs_b->lock)
 {
 	int throttled;
@@ -7288,10 +7288,10 @@ static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun)
 	 * This check is repeated as we release cfs_b->lock while we unthrottle.
 	 */
 	while (throttled && cfs_b->runtime > 0) {
-		raw_spin_unlock_irq_enable(&cfs_b->lock);
+		raw_spin_unlock_irqrestore(&cfs_b->lock, flags);
 		/* we can't nest cfs_b->lock while distributing bandwidth */
 		throttled = distribute_cfs_runtime(cfs_b);
-		raw_spin_lock_irq_disable(&cfs_b->lock);
+		raw_spin_lock_irqsave(&cfs_b->lock, flags);
 	}
 
 	/*
@@ -7399,7 +7399,7 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq)
 static void do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b)
 {
 	/* confirm we're still not at a refresh boundary */
-	scoped_guard(raw_spinlock_irq, &cfs_b->lock) {
+	scoped_guard(raw_spinlock_irqsave, &cfs_b->lock) {
 		u64 runtime = 0, slice = sched_cfs_bandwidth_slice();
 
 		cfs_b->slack_started = false;
@@ -7484,14 +7484,14 @@ static enum hrtimer_restart sched_cfs_period_timer(struct hrtimer *timer)
 	int idle = 0;
 	int count = 0;
 
-	guard(raw_spinlock_irq)(&cfs_b->lock);
+	CLASS(raw_spinlock_irqsave, cfsb_guard)(&cfs_b->lock);
 
 	for (;;) {
 		overrun = hrtimer_forward_now(timer, cfs_b->period);
 		if (!overrun)
 			break;
 
-		idle = do_sched_cfs_period_timer(cfs_b, overrun);
+		idle = do_sched_cfs_period_timer(cfs_b, overrun, cfsb_guard.flags);
 
 		if (++count > 3) {
 			u64 new, old = ktime_to_ns(cfs_b->period);
author maintainer Acked-by on v1 1/1
#39 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra

While the guards are properly nested, not all wrapped code is nice, as already highlighted by that fair.c hunk.

Syzbot found another instance of this pattern in posix_timer_delete(), which does spin_unlock_irq()+spin_lock_irq() inside scoped_guard(spinlock_irq). Combined with this patch, that goes sideways most spectacular.

Undo this change, until we've developed stronger tools / debug for such issues.

Mainly hand-waving, but if we make _irq(), irqsave(), _disable() __acquires() different contexts, we may be able to catch these issues at compile time. I will explore a bit on this.

Quoted 1 line

Regards,
Boqun

Quoted 126 lines
---
 include/linux/spinlock.h |   26 ++++++++++++++------------
 kernel/sched/fair.c      |   12 ++++++------
 2 files changed, 20 insertions(+), 18 deletions(-)
include/linux/spinlock.h +14 -12
@@ -572,12 +572,12 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_
 #define class_raw_spinlock_nested_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_nested, _T)
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irq, raw_spinlock_t,
-		    raw_spin_lock_irq_disable(_T->lock),
-		    raw_spin_unlock_irq_enable(_T->lock))
+		    raw_spin_lock_irq(_T->lock),
+		    raw_spin_unlock_irq(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq, _T)
 
-DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq_disable(_T->lock))
+DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irq, _try, raw_spin_trylock_irq(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irq_try, _T)
 
@@ -592,13 +592,14 @@ DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_
 #define class_raw_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_bh_try, _T)
 
 DEFINE_LOCK_GUARD_1(raw_spinlock_irqsave, raw_spinlock_t,
-		    raw_spin_lock_irq_disable(_T->lock),
-		    raw_spin_unlock_irq_enable(_T->lock))
+		    raw_spin_lock_irqsave(_T->lock, _T->flags),
+		    raw_spin_unlock_irqrestore(_T->lock, _T->flags),
+		    unsigned long flags)
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave, _T)
 
 DEFINE_LOCK_GUARD_1_COND(raw_spinlock_irqsave, _try,
-			 raw_spin_trylock_irq_disable(_T->lock))
+			 raw_spin_trylock_irqsave(_T->lock, _T->flags))
 DECLARE_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave_try, __acquires(_T), __releases(*(raw_spinlock_t **)_T))
 #define class_raw_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(raw_spinlock_irqsave_try, _T)
 
@@ -617,13 +618,13 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_try,
 #define class_spinlock_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_try, _T)
 
 DEFINE_LOCK_GUARD_1(spinlock_irq, spinlock_t,
-		    spin_lock_irq_disable(_T->lock),
-		    spin_unlock_irq_enable(_T->lock))
+		    spin_lock_irq(_T->lock),
+		    spin_unlock_irq(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irq_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq, _T)
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irq, _try,
-			 spin_trylock_irq_disable(_T->lock))
+			 spin_trylock_irq(_T->lock))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irq_try, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irq_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irq_try, _T)
 
@@ -639,13 +640,14 @@ DECLARE_LOCK_GUARD_1_ATTRS(spinlock_bh_t
 #define class_spinlock_bh_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_bh_try, _T)
 
 DEFINE_LOCK_GUARD_1(spinlock_irqsave, spinlock_t,
-		    spin_lock_irq_disable(_T->lock),
-		    spin_unlock_irq_enable(_T->lock))
+		    spin_lock_irqsave(_T->lock, _T->flags),
+		    spin_unlock_irqrestore(_T->lock, _T->flags),
+		    unsigned long flags)
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irqsave_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irqsave, _T)
 
 DEFINE_LOCK_GUARD_1_COND(spinlock_irqsave, _try,
-			 spin_trylock_irq_disable(_T->lock))
+			 spin_trylock_irqsave(_T->lock, _T->flags))
 DECLARE_LOCK_GUARD_1_ATTRS(spinlock_irqsave_try, __acquires(_T), __releases(*(spinlock_t **)_T))
 #define class_spinlock_irqsave_try_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(spinlock_irqsave_try, _T)
 
kernel/sched/fair.c +6 -6
@@ -7253,7 +7253,7 @@ static bool distribute_cfs_runtime(struc
  * period the timer is deactivated until scheduling resumes; cfs_b->idle is
  * used to track this state.
  */
-static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun)
+static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun, unsigned long flags)
 	__must_hold(&cfs_b->lock)
 {
 	int throttled;
@@ -7288,10 +7288,10 @@ static int do_sched_cfs_period_timer(str
 	 * This check is repeated as we release cfs_b->lock while we unthrottle.
 	 */
 	while (throttled && cfs_b->runtime > 0) {
-		raw_spin_unlock_irq_enable(&cfs_b->lock);
+		raw_spin_unlock_irqrestore(&cfs_b->lock, flags);
 		/* we can't nest cfs_b->lock while distributing bandwidth */
 		throttled = distribute_cfs_runtime(cfs_b);
-		raw_spin_lock_irq_disable(&cfs_b->lock);
+		raw_spin_lock_irqsave(&cfs_b->lock, flags);
 	}
 
 	/*
@@ -7399,7 +7399,7 @@ static __always_inline void return_cfs_r
 static void do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b)
 {
 	/* confirm we're still not at a refresh boundary */
-	scoped_guard(raw_spinlock_irq, &cfs_b->lock) {
+	scoped_guard(raw_spinlock_irqsave, &cfs_b->lock) {
 		u64 runtime = 0, slice = sched_cfs_bandwidth_slice();
 
 		cfs_b->slack_started = false;
@@ -7484,14 +7484,14 @@ static enum hrtimer_restart sched_cfs_pe
 	int idle = 0;
 	int count = 0;
 
-	guard(raw_spinlock_irq)(&cfs_b->lock);
+	CLASS(raw_spinlock_irqsave, cfsb_guard)(&cfs_b->lock);
 
 	for (;;) {
 		overrun = hrtimer_forward_now(timer, cfs_b->period);
 		if (!overrun)
 			break;
 
-		idle = do_sched_cfs_period_timer(cfs_b, overrun);
+		idle = do_sched_cfs_period_timer(cfs_b, overrun, cfsb_guard.flags);
 
 		if (++count > 3) {
 			u64 new, old = ktime_to_ns(cfs_b->period);
maintainer
#40 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 9 lines · Boqun Feng
Quoted 8 lines · Peter Zijlstra

While the guards are properly nested, not all wrapped code is nice, as already highlighted by that fair.c hunk.

Syzbot found another instance of this pattern in posix_timer_delete(), which does spin_unlock_irq()+spin_lock_irq() inside scoped_guard(spinlock_irq). Combined with this patch, that goes sideways most spectacular.

Quoted 6 lines
Quoted 2 lines

Undo this change, until we've developed stronger tools / debug for such issues.

Mainly hand-waving, but if we make _irq(), irqsave(), _disable() __acquires() different contexts, we may be able to catch these issues at compile time. I will explore a bit on this.

No.

Just do a wholesale conversion of all functions which affect the CPU interrupt disabled state directly (local_irq_*) and indirectly (locking functions etc.)

Anything else is just a whack a mole game.

TBH, I do not understand why you thought that you can get away with this lazy approach especially after you discovered the same nasty problem in do_sched_cfs_period_timer(). The resolution of that got buried in

1b0866874833 ("locking: Switch to _irq_{disable,enable}() variants in cleanup guards")

without even being mentioned.

When I was discussing the non-sensical syzbot messages earlier today with Peter it immediately occurred to me that this undocumented change in do_sched_cfs_period_timer() is not the only pattern which causes this to go belly up. It took me five seconds to find the posix timer one.

TBH, my hope really was that the RUST people take the only valid engineering principle "Correctness first" serious, but sadly they seem to be the same lazy sods than everyone else who want to push their agenda through no matter what.

Thanks,

tglx

author maintainer
#41 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 17 lines · Thomas Gleixner
Quoted 9 lines · Boqun Feng
Quoted 8 lines · Peter Zijlstra

While the guards are properly nested, not all wrapped code is nice, as already highlighted by that fair.c hunk.

Syzbot found another instance of this pattern in posix_timer_delete(), which does spin_unlock_irq()+spin_lock_irq() inside scoped_guard(spinlock_irq). Combined with this patch, that goes sideways most spectacular.

Quoted 6 lines
Quoted 2 lines

Undo this change, until we've developed stronger tools / debug for such issues.

Mainly hand-waving, but if we make _irq(), irqsave(), _disable() __acquires() different contexts, we may be able to catch these issues at compile time. I will explore a bit on this.

Quoted 8 lines

No.

Just do a wholesale conversion of all functions which affect the CPU interrupt disabled state directly (local_irq_*) and indirectly (locking functions etc.)

Anything else is just a whack a mole game.

Alright. But I'm afraid that's just another type of whack-a-mole games.

As I mentioned here [1], we are a few unpaired local_irq_disable() + local_irq_enable(), we can spend time to clean them up, but no guarantee people will not introduce more, plus we have code that does spin_lock_irqsave(); spin_unlock_irq(); spin_lock_irq(); spin_unlock_irqrestore(); and expect it works. A more reasonable approach to me is introducing the new API and fixing the problematic usage one-by-one and then when we are certain about only a few cases left, we do a flag day change.

Trying to do it (new API and whole conversion) in one go is easier said than done. Of course I might miss something subtle here, looking forwards to your suggestion.

Quoted 8 lines

TBH, I do not understand why you thought that you can get away with this lazy approach especially after you discovered the same nasty problem in do_sched_cfs_period_timer(). The resolution of that got buried in

1b0866874833 ("locking: Switch to _irq_{disable,enable}() variants in cleanup guards")

without even being mentioned.

I have this in the commit log:

[boqun: Adjust the user-side changes in do_sched_cfs_*_timer() provided
by Peter and Lyude]

but sure, I should have done a better job mentioning it.

A bit more context of switching the guard implementation: I wanted to have some test/usage coverage other than Rust for the new API, and since the guard() API is relatively new, so I thought people will not use it "creatively" (but obviously I was wrong). Hence I add the conversation for the guard APIs only. It is not a lazy approach IMO, but rather a way to test how the new API works. Of course, a bug is a bug, I don't have any excuse on that.

Quoted 10 lines

When I was discussing the non-sensical syzbot messages earlier today with Peter it immediately occurred to me that this undocumented change in do_sched_cfs_period_timer() is not the only pattern which causes this to go belly up. It took me five seconds to find the posix timer one.

TBH, my hope really was that the RUST people take the only valid engineering principle "Correctness first" serious, but sadly they seem to be the same lazy sods than everyone else who want to push their agenda through no matter what.

There seems some misunderstandings here. The only "lazy" part is we defer the whole conversion because of the problems I mentioned above, and that is because Correctness is valued.

[1]: https://lore.kernel.org/rust-for-linux/[email protected]/

Regards,
Boqun

Quoted 3 lines

Thanks,

tglx

author maintainer
#42 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 11 lines · Boqun Feng
Quoted 10 lines · Thomas Gleixner
Quoted 9 lines · Boqun Feng
Quoted 8 lines · Peter Zijlstra

While the guards are properly nested, not all wrapped code is nice, as already highlighted by that fair.c hunk.

Syzbot found another instance of this pattern in posix_timer_delete(), which does spin_unlock_irq()+spin_lock_irq() inside scoped_guard(spinlock_irq). Combined with this patch, that goes sideways most spectacular.

I'm not saying the posix_timer_delete() implementation has any problem, but TBH allowing spin_unlock_irq()+spin_lock_irq() inside scoped_guard(spinlock_irq) is questionable design, and can result into foot-gun code like:

	scoped_guard(spinlock_irq) {
		...
		spin_unlock_irq();
		if (cond)
			return; // BOOM, double unlock
		spin_lock_irq();
	}

Sure, if handling carefully, it won't cause problem, but it undermines the easy-to-use and less-err-prone features of scoped_guard().

Regards,
Boqun

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Undo this change, until we've developed stronger tools / debug for such issues.

Mainly hand-waving, but if we make _irq(), irqsave(), _disable() __acquires() different contexts, we may be able to catch these issues at compile time. I will explore a bit on this.

[...]

author maintainer
#43 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 26 lines · Boqun Feng
Quoted 11 lines · Boqun Feng
Quoted 10 lines · Thomas Gleixner
Quoted 9 lines · Boqun Feng
Quoted 8 lines · Peter Zijlstra

While the guards are properly nested, not all wrapped code is nice, as already highlighted by that fair.c hunk.

Syzbot found another instance of this pattern in posix_timer_delete(), which does spin_unlock_irq()+spin_lock_irq() inside scoped_guard(spinlock_irq). Combined with this patch, that goes sideways most spectacular.

I'm not saying the posix_timer_delete() implementation has any problem, but TBH allowing spin_unlock_irq()+spin_lock_irq() inside scoped_guard(spinlock_irq) is questionable design, and can result into foot-gun code like:

	scoped_guard(spinlock_irq) {
		...
		spin_unlock_irq();
		if (cond)
			return; // BOOM, double unlock
		spin_lock_irq();
	}
Quoted 3 lines

Sure, if handling carefully, it won't cause problem, but it undermines the easy-to-use and less-err-prone features of scoped_guard().

One idea is since each scoped_guard() creates a scope guard, the guard should be used as token for the inner unlock (guard_drop()) and lock (guard_retake()). So the above code become:

	scoped_guard(spinlock_irq, ...) {
		guard_drop(spinlock_irq, &scope);
		// ^ scope is modified to know that no more unlock is
		// needed.
		if (cond)
			return; // no double unlock
		guard_retake(spinlock_irq, &scope, ...);
	}

The following shows the idea (only compile test). Also applies to lock_timer as well.

Regards,
Boqun

--------------------------------->8

include/linux/cleanup.h +25 -1
index b1b5698cbf1b..459c533e4cc8 100644
@@ -249,6 +249,19 @@ const volatile void * __must_check_fn(const volatile void *val)
  */
 #define retain_and_null_ptr(p)		((void)__get_and_null(p, NULL))
 
+/*
+ * Drops a scoped guard
+ */
+#define guard_drop(name, p)	class_##name##_destructor(p)
+
+/*
+ * Re-takes a scoped guard
+ */
+#define guard_retake(name, p, ...)					\
+do {									\
+	class_##name##_retake(p, __VA_ARGS__);				\
+} while (0)
+
 /*
  * DEFINE_CLASS(name, type, exit, init, init_args...):
  *	helper to define the destructor and constructor for a type.
@@ -492,7 +505,10 @@ typedef struct {							\
 static __always_inline void class_##_name##_destructor(class_##_name##_t *_T) \
 	__no_context_analysis						\
 {									\
-	_unlock;							\
+	if ((_T)->lock) {						\
+		_unlock;						\
+		(_T)->lock = NULL;					\
+	}								\
 }									\
 									\
 __DEFINE_GUARD_LOCK_PTR(_name, &_T->lock)
@@ -505,6 +521,14 @@ class_##_name##_t class_##_name##_constructor(_type *l)			\
 	class_##_name##_t _t = { .lock = l }, *_T = &_t;		\
 	__VA_ARGS__;							\
 	return _t;							\
+}									\
+static __always_inline __nonnull_args(2)				\
+void class_##_name##_retake(class_##_name##_t *_T, _type *l)		\
+	__no_context_analysis						\
+{									\
+	BUG_ON((_T)->lock);						\
+	(_T)->lock = l;							\
+	__VA_ARGS__;							\
 }
 
 #define __DEFINE_LOCK_GUARD_0(_name, ...)				\
kernel/time/posix-timers.c +13 -5
index 436ba794cc0b..b34e8292e989 100644
@@ -1026,7 +1026,8 @@ static inline void posix_timer_cleanup_ignored(struct k_itimer *tmr)
 	}
 }
 
-static void posix_timer_delete(struct k_itimer *timer)
+static void posix_timer_delete(struct k_itimer *timer,
+			       class_spinlock_irq_t *guard)
 {
 	/*
 	 * Invalidate the timer, remove it from the linked list and remove
@@ -1057,9 +1058,10 @@ static void posix_timer_delete(struct k_itimer *timer)
 
 	while (timer->kclock->timer_del(timer) == TIMER_RETRY) {
 		guard(rcu)();
-		spin_unlock_irq(&timer->it_lock);
+
+		guard_drop(spinlock_irq, guard);
 		timer_wait_running(timer);
-		spin_lock_irq(&timer->it_lock);
+		guard_retake(spinlock_irq, guard, &timer->it_lock);
 	}
 }
 
@@ -1069,8 +1071,14 @@ SYSCALL_DEFINE1(timer_delete, timer_t, timer_id)
 	struct k_itimer *timer;
 
 	scoped_timer_get_or_fail(timer_id) {
+		// Needs a better to "cast" a guard of "lock_timer" to
+		// "spinlock_irq".
+		class_spinlock_irq_t guard = {
+			.lock = &scoped_timer->it_lock,
+		};
+
 		timer = scoped_timer;
-		posix_timer_delete(timer);
+		posix_timer_delete(timer, &guard);
 	}
 	/* Remove it from the hash, which frees up the timer ID */
 	posix_timer_unhash_and_free(timer);
@@ -1101,7 +1109,7 @@ void exit_itimers(struct task_struct *tsk)
 	/* The timers are not longer accessible via tsk::signal */
 	hlist_for_each_entry_safe(timer, next, &timers, list) {
 		scoped_guard (spinlock_irq, &timer->it_lock)
-			posix_timer_delete(timer);
+			posix_timer_delete(timer, &scope);
 		posix_timer_unhash_and_free(timer);
 		cond_resched();
 	}
maintainer
#44 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

In reply to Boqun Feng
Quoted 24 lines · Boqun Feng
Quoted 23 lines · Thomas Gleixner
Quoted 9 lines · Boqun Feng
Quoted 8 lines · Peter Zijlstra

While the guards are properly nested, not all wrapped code is nice, as already highlighted by that fair.c hunk.

Syzbot found another instance of this pattern in posix_timer_delete(), which does spin_unlock_irq()+spin_lock_irq() inside scoped_guard(spinlock_irq). Combined with this patch, that goes sideways most spectacular.

Quoted 6 lines
Quoted 2 lines

Undo this change, until we've developed stronger tools / debug for such issues.

Mainly hand-waving, but if we make _irq(), irqsave(), _disable() __acquires() different contexts, we may be able to catch these issues at compile time. I will explore a bit on this.

No.

Just do a wholesale conversion of all functions which affect the CPU interrupt disabled state directly (local_irq_*) and indirectly (locking functions etc.)

Quoted 5 lines
Quoted 2 lines

Anything else is just a whack a mole game.

Alright. But I'm afraid that's just another type of whack-a-mole games.

I don't think so.

Quoted 5 lines

As I mentioned here [1], we are a few unpaired local_irq_disable() + local_irq_enable(), we can spend time to clean them up, but no guarantee people will not introduce more, plus we have code that does spin_lock_irqsave(); spin_unlock_irq(); spin_lock_irq(); spin_unlock_irqrestore(); and expect it works.

It actually works and there are reasons why this needs to work in certain cases. It needs some support with a different set of helper functions for sure.

I played around with changing local_irq_disab/enable/save/restore almost two decades ago when cli/sti was expensive, so we could do a lazy disable approach. It went nowhere because it turned out to be too complex to handle the interrupts which hit a lazy disabled region later, but the principle itself worked.

I dealt with the above example by doing:

  oldcnt = irq_save()   	return cnt++;
  irq_restore(oldcnt)		cnt = oldcnt;
  irq_disable()  		cnt=1;
  irq_enable()  		cnt=0;

See below.

Quoted 3 lines

A more reasonable approach to me is introducing the new API and fixing the problematic usage one-by-one and then when we are certain about only a few cases left, we do a flag day change.

You already did a flag day change which causes problems, no?

The main problem is that you cover only half of it and there are completely correct cases where this simply blows up in your face:

local_irq_disable(); // does not affect CNT
....
guard(raw_spinlock)(&l1); // does not affect CNT
foo()
guard(raw_spinlock_irqsave)(&l2); // observes CNT = 0

so the unlocking of &l2 will enable interrupts prematurely.

That's a very common scheme in interrupt handling. Functions which know they are always invoked with interrupts disabled use raw_spinlock() while others which can be invoked from different contexts use the irqsave() variant.

Also the lack of rwlock support is a red flag. Again completely valid code:

     read_lock_irq()
     ...
     guard(spinlock_irqsave)();

Same issue as above.

There are more subtle problems lurking around the corner.

Quoted 3 lines

Trying to do it (new API and whole conversion) in one go is easier said than done. Of course I might miss something subtle here, looking forwards to your suggestion.

I did not say it's easy and I did not say that you have to do both in one go, which is impossible.

You have to do it in stages, which means you put the infrastructure in place first and then once that is settled you build the new API on top if required at all. Building a new API first and hoping that it works out without actually addressing the underlying issues first is just a recipe for disaster.

You really want to start at the places which deal with the actual interrupt flags of the CPU and that's definitely not locking. That's only a couple of functions plus a few related helpers:

raw_local_irq_disable()
raw_local_irq_enable()
raw_local_irq_save()
raw_local_irq_restore()

If you actually look at the usage of the 'flags' argument of raw_local_irq_save() and raw_local_irq_restore() then you'll notice that it's a completely opaque cookie. Validating that there is no user which is actually interested in seeing the real flags should be trivial enough. A quick skim of x86 revealed exactly zero places, but I might have missed one of course.

So you can get away with:

raw_local_irq_save(flags)

{
	flags = count;
	if (!count)
        	arch_local_irq_disable();
        count++;
}

raw_local_irq_restore(flags)

{
	if (!(count = flags))
        	arch_local_irq_enable();
}

raw_local_irq_disable()

{
        arch_local_irq_disable();
        count = 1;
}

raw_local_irq_enable()

{
        count = 0;
        arch_local_irq_enable();
}

To make this work you need to deal with the obvious race conditions between modifying the counter and modifying the CPU flag, which is relevant for all hardware initiated context changes (syscalls, interrupts, exceptions, NMI).

In enter_from_user_mode() is trivial. All you need to add is an unconditional

        count = 1;

because interrupts are enabled when a task runs in user space. On entry to the kernel (syscall, interrupt, exception, NMI) the CPU disables interrupts so you have to reflect that in the software counter.

exit_to_user_mode() requires then obviously:

        count = 0;

irqentry_enter_from_kernel_mode() is a bit more tricky because count and the actual interrupt flags state in the CPU can be out of sync as you can see in all four related functions above. But that's easy enough to cure:

        irqentry_state_t ret = {
                .exit_rcu = false,
        };

        ret.irqdisable_cnt = count;
	count = 1;

Setting it to 1 is the correct thing to do as this is fresh context and it's safe for exception handlers which conditionally enable interrupts because they explicitly rely on checking regs->eflags to figure out whether the interrupted context had interrupts enabled.

That also makes this horrible hack in __irq_exit_rcu() go away because the state is fully consistent.

In irqentry_exit_to_kernel_mode_after_preempt()

       count = state.irqdisable_cnt;

In irqentry_nmi_enter() and irqentry_nmi_exit() you need exactly the same.

With that you have a fully consistent and working system. Not what you are aiming for in the very end, but a first step to cover the existing code base fully without nasty to debug surprises.

Now you need to handle the oddball cases which nest an interrupt enable/disable pair into a irqsave/restore region like the one in the scheduler and the other in posix timers.

First of all, most of these places can be found by code analysis. When I saw the one in the scheduler I whipped up a trivial coccinelle script which found the one in posix timers immediately.

Then you can obviously add debug variants of those functions which are conditional by an explicit config switch and emit warnings which are easy enough to distinguish so that automated testing failures do not result in a "paper over the problem" frenzy.

For dealing with those cases you want something like this:

raw_local_irq_enable_nested()

{
	cur = count;
        count = 0;
        arch_local_irq_enable();
        return cur;        
}

raw_local_irq_disable_nested(oldcnt)

{
        arch_local_irq_disable();
        count = oldcnt; 
}

Once all this headache is gone, you can modify the underlying machinery without touching any other code at all and make the debug code a real (lockdep) warning which has to be treated like any other splat.

See?

Thanks,

tglx

author maintainer
#45 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

On Thu, Aug 27, 2026 at 10:30:50AM +0200, Thomas Gleixner wrote:
[...]

Quoted 41 lines
Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

Mainly hand-waving, but if we make _irq(), irqsave(), _disable() __acquires() different contexts, we may be able to catch these issues at compile time. I will explore a bit on this.

No.

Just do a wholesale conversion of all functions which affect the CPU interrupt disabled state directly (local_irq_*) and indirectly (locking functions etc.)

Quoted 5 lines
Quoted 2 lines

Anything else is just a whack a mole game.

Alright. But I'm afraid that's just another type of whack-a-mole games.

I don't think so.

Quoted 5 lines

As I mentioned here [1], we are a few unpaired local_irq_disable() + local_irq_enable(), we can spend time to clean them up, but no guarantee people will not introduce more, plus we have code that does spin_lock_irqsave(); spin_unlock_irq(); spin_lock_irq(); spin_unlock_irqrestore(); and expect it works.

It actually works and there are reasons why this needs to work in certain cases. It needs some support with a different set of helper functions for sure.

I played around with changing local_irq_disab/enable/save/restore almost two decades ago when cli/sti was expensive, so we could do a lazy disable approach. It went nowhere because it turned out to be too complex to handle the interrupts which hit a lazy disabled region later, but the principle itself worked.

I dealt with the above example by doing:

  oldcnt = irq_save()   	return cnt++;
  irq_restore(oldcnt)		cnt = oldcnt;
  irq_disable()  		cnt=1;
  irq_enable()  		cnt=0;
Quoted 8 lines

See below.

Quoted 3 lines

A more reasonable approach to me is introducing the new API and fixing the problematic usage one-by-one and then when we are certain about only a few cases left, we do a flag day change.

You already did a flag day change which causes problems, no?

(I will reply a few things here, and will read through your suggestions below, and reply them latter.)

Right, but that was an attempt to see if we could switch to scoped_guard() implementation to the new infrastructure (see below) in this stage. Clearly we cannot because I overlooked cases like posix_timer_delete(), but itself is not trying to introduce the new API.

Quoted 11 lines

The main problem is that you cover only half of it and there are completely correct cases where this simply blows up in your face:

local_irq_disable(); // does not affect CNT
....
guard(raw_spinlock)(&l1); // does not affect CNT
foo()
guard(raw_spinlock_irqsave)(&l2); // observes CNT = 0

so the unlocking of &l2 will enable interrupts prematurely.

If we are talking the switch in this patch, then no, the unlocking of &l2 will NOT enable interrupts prematurely. The above code expands as the following (using pseudo code to describe how raw_spin_lock_irq_disable(), raw_spin_lock_irq_enable(), local_interrupt_disable(), and local_interrupt_enable() work)

local_irq_disable(); // does not affect CNT
....
guard(raw_spinlock)(&l1); // does not affect CNT

      foo()
        guard(raw_spinlock_irqsave)(&l2): 
          raw_spin_lock_irq_disable():
            local_interrupt_disable():
	      CNT++;
              this_cpu(state) = local_irq_save(); // record the current state
            raw_spin_lock(&l2);
	  ...
          raw_spin_lock_irq_enable():
            raw_spin_unlock(&l2);
            local_interrupt_enable():
              CNT--;
              if (CNT == 0)
                local_irq_restore(this_cpu(state)); // recover the previous state

So local_interrupt_disable() and local_interrupt_enable() only recover to the previous state, as a result it'll not enable interrupt prematurely here. In other words, the following code works:

    local_irq_disable();
    local_interrupt_disable();
    local_interrupt_enable();  // interrupt is not re-enabled here
                               // similar to how preempt_disable() does
			       // in a nested preemption disable
			       // critical section.
    local_irq_enable();

These functions are the infrastructure thing you talk about below:

(they are introduced in commit e901c1510e24 ("irq,spin_lock: Add counted interrupt disabling/enabling"))

* local_interrupt_disable()
* local_interrupt_enable()
* raw_spin_lock_irq_disable()
* raw_spin_lock_irq_enable()

They currently work when nested in local_irq_disable() or local_irq_save() because of the per-CPU irq state tracking when CNT reaches 0->1 or 1->0.

Quoted 6 lines

That's a very common scheme in interrupt handling. Functions which know they are always invoked with interrupts disabled use raw_spinlock() while others which can be invoked from different contexts use the irqsave() variant.

Also the lack of rwlock support is a red flag. Again completely valid

We could use local_interrupt_disable() and local_interrupt_enable() to implement a new API for rwlock when the support is needed in the future.

Quoted 8 lines

code:

     read_lock_irq()
     ...
     guard(spinlock_irqsave)();

Same issue as above.

Similar as above, no issue in this case.

Quoted 6 lines

There are more subtle problems lurking around the corner.

Quoted 3 lines

Trying to do it (new API and whole conversion) in one go is easier said than done. Of course I might miss something subtle here, looking forwards to your suggestion.

Quoted 9 lines

I did not say it's easy and I did not say that you have to do both in one go, which is impossible.

You have to do it in stages, which means you put the infrastructure in place first and then once that is settled you build the new API on top if required at all. Building a new API first and hoping that it works out without actually addressing the underlying issues first is just a recipe for disaster.

I agree and that is actually what I did here: adding the infrastructure local_interrupt_disable() and local_interrupt_enable() and gradually using that infrastructure to support building new API (or existing API). `
The part that went wrong for this particular patch was I was missing the usage similar to posix_timer_delete() cases where users want to drop the lock under scoped_guard context, I have a proposal in another reply, and I think that might be better way, but of course the infrastructure can support without it, we just need to postpone the implementation switch of scoped_guard() until it's ready.

All I'm trying to say here is I'm doing this slow and steady :)

[I will take a deep look for the following later, I feel I need to reply above in case I or the patch confused you somehow]

Regards,
Boqun

Quoted 130 lines

interrupt flags of the CPU and that's definitely not locking. That's only a couple of functions plus a few related helpers:

raw_local_irq_disable()
raw_local_irq_enable()
raw_local_irq_save()
raw_local_irq_restore()

If you actually look at the usage of the 'flags' argument of raw_local_irq_save() and raw_local_irq_restore() then you'll notice that it's a completely opaque cookie. Validating that there is no user which is actually interested in seeing the real flags should be trivial enough. A quick skim of x86 revealed exactly zero places, but I might have missed one of course.

So you can get away with:

raw_local_irq_save(flags)

{
	flags = count;
	if (!count)
        	arch_local_irq_disable();
        count++;
}

raw_local_irq_restore(flags)

{
	if (!(count = flags))
        	arch_local_irq_enable();
}

raw_local_irq_disable()

{
        arch_local_irq_disable();
        count = 1;
}

raw_local_irq_enable()

{
        count = 0;
        arch_local_irq_enable();
}

To make this work you need to deal with the obvious race conditions between modifying the counter and modifying the CPU flag, which is relevant for all hardware initiated context changes (syscalls, interrupts, exceptions, NMI).

In enter_from_user_mode() is trivial. All you need to add is an unconditional

        count = 1;

because interrupts are enabled when a task runs in user space. On entry to the kernel (syscall, interrupt, exception, NMI) the CPU disables interrupts so you have to reflect that in the software counter.

exit_to_user_mode() requires then obviously:

        count = 0;

irqentry_enter_from_kernel_mode() is a bit more tricky because count and the actual interrupt flags state in the CPU can be out of sync as you can see in all four related functions above. But that's easy enough to cure:

        irqentry_state_t ret = {
                .exit_rcu = false,
        };

        ret.irqdisable_cnt = count;
	count = 1;

Setting it to 1 is the correct thing to do as this is fresh context and it's safe for exception handlers which conditionally enable interrupts because they explicitly rely on checking regs->eflags to figure out whether the interrupted context had interrupts enabled.

That also makes this horrible hack in __irq_exit_rcu() go away because the state is fully consistent.

In irqentry_exit_to_kernel_mode_after_preempt()

       count = state.irqdisable_cnt;

In irqentry_nmi_enter() and irqentry_nmi_exit() you need exactly the same.

With that you have a fully consistent and working system. Not what you are aiming for in the very end, but a first step to cover the existing code base fully without nasty to debug surprises.

Now you need to handle the oddball cases which nest an interrupt enable/disable pair into a irqsave/restore region like the one in the scheduler and the other in posix timers.

First of all, most of these places can be found by code analysis. When I saw the one in the scheduler I whipped up a trivial coccinelle script which found the one in posix timers immediately.

Then you can obviously add debug variants of those functions which are conditional by an explicit config switch and emit warnings which are easy enough to distinguish so that automated testing failures do not result in a "paper over the problem" frenzy.

For dealing with those cases you want something like this:

raw_local_irq_enable_nested()

{
	cur = count;
        count = 0;
        arch_local_irq_enable();
        return cur;        
}

raw_local_irq_disable_nested(oldcnt)

{
        arch_local_irq_disable();
        count = oldcnt; 
}

Once all this headache is gone, you can modify the underlying machinery without touching any other code at all and make the debug code a real (lockdep) warning which has to be treated like any other splat.

See?

Thanks,

tglx

maintainer
#46 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 41 lines · Boqun Feng
Quoted 11 lines · Thomas Gleixner

The main problem is that you cover only half of it and there are completely correct cases where this simply blows up in your face:

local_irq_disable(); // does not affect CNT
....
guard(raw_spinlock)(&l1); // does not affect CNT
foo()
guard(raw_spinlock_irqsave)(&l2); // observes CNT = 0

so the unlocking of &l2 will enable interrupts prematurely.

If we are talking the switch in this patch, then no, the unlocking of &l2 will NOT enable interrupts prematurely. The above code expands as the following (using pseudo code to describe how raw_spin_lock_irq_disable(), raw_spin_lock_irq_enable(), local_interrupt_disable(), and local_interrupt_enable() work)

   local_irq_disable();                          // does not affect CNT
   ....
   guard(raw_spinlock)(&l1);                     // does not affect CNT
      foo()
        guard(raw_spinlock_irqsave)(&l2): 
          raw_spin_lock_irq_disable():
            local_interrupt_disable():
	      CNT++;
              this_cpu(state) = local_irq_save(); // record the current state
            raw_spin_lock(&l2);
	  ...
          raw_spin_lock_irq_enable():
            raw_spin_unlock(&l2);
            local_interrupt_enable():
              CNT--;
              if (CNT == 0)
                local_irq_restore(this_cpu(state)); // recover the previous state

So local_interrupt_disable() and local_interrupt_enable() only recover to the previous state, as a result it'll not enable interrupt prematurely here. In other words, the following code works:

Quoted 7 lines
    local_irq_disable();
    local_interrupt_disable();
    local_interrupt_enable();  // interrupt is not re-enabled here
                               // similar to how preempt_disable() does
			       // in a nested preemption disable
			       // critical section.
    local_irq_enable();

Fair enough. I misread that part.

But my main observation that the counter is inconsistent still stands and I think that's a fundamental flaw because there is no way that code can rely on that counter until everything has been converted over and the interrupt/exception/nmi/syscall entry/exit code has been fixed up.

Just let me look at local_interrupt_disable() and __irq_exit_rcu() again.

local_interrupt_disable()
new_count = hardirq_disable_enter();

/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */

   if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
      _local_interrupt_disable();

This is absolutely not ok. Why?

The counter is incremented _before_ interrupts are actually disabled. Now in __irq_exit_rcu():

        if (!in_interrupt() && !hardirq_disable_count() &&
            local_softirq_pending()) {

which prevents soft interrupt handling in a completely legitimate situation. As a consequence _nothing_ will handle the pending soft interrupt until:

- an interrupt coming in which observes consistent state

- a local_bh_enable() processes them

That explains that recently quite a few more spurious 'local soft irq pending' printk's have been observed by people as there is no guarantee that either one of those events happens _before_ a CPU reaches idle. Even in the non-idle case deferring this to the next 'by chance' handling is fundamentally broken. This needs to be removed ASAP.

But coming back to the problem underneath. The ordering in local_interrupt_disable() is simply wrong. You need to disable first and then update the counter. Reverse order for enable() obviously update counter and enable, which you got right.

And to make stuff work correctly you need the fixups I pointed out in my previous reply to the various entry/exit functions. It's exactly the same problem as we handle in interrupt/exception/nmi entry/exit code vs. RCU, lockdep, tracing etc.

So if disable() does:

   if (!count)
   	arch_local_irq_disable();
   count++;

then an interrupt hitting before arch_local_irq_disable() will always observe the correct state. After that it can't be delivered.

Now with exceptions that's a different story because they can hit after local_irq_disable() and before the count is incremented.

I thought some more about the state handling there and I think we can avoid irq_state_t completely:

irqentry_enter_from_kernel_mode()

     count++;
     ...

irqentry_exit_to_kernel_mode_after_preempt()

     ...
     count--;

For a regular interrupt which hit _before_ disable() managed to disable it at the CPU level, this will go from 0 -> 1 and on return from 1 -> 0.

For an exception which hits between disabling and incrementing the counter this will go from 0 -> 1 as well, but there is nothing which can be done about that and exception handlers need to consult regs->eflags to figure out the state of the context they interrupted. If the exception hits afterwards then it will set the correct state. But it does not matter in that case because everything there needs to do irqsave() so interrupts can't be enabled accidentaly. The only exception to that rule is the conditional enable:

   if (regs->eflags & X86_EFLAGS_IF)
   	local_irq_enable();

And for that to work correctly you want overall consistent counter state. Otherwise your counter is just a random number generator.

With that fixed the disable race becomes:

disable()

     if (!count)

-> Interrupt before interrupts are disabled in the CPU.

irqentry_enter_from_kernel_mode()
count++; // Correct state because the CPU disabled interrupts

  ...
  __irq_exit_rcu()
     if (!in_interrupt() && local_softirq_pending()) {
        handle_softirqs()
          ...
          local_irq_enable();                -> Count goes to 0
          ...
          guard(spinlock_irq)(&lock)
            local_interrupt_disable()
              // Observes count == 0
              if (!count)
                 arch_local_irq_disable();
     ...

irqentry_exit_to_kernel_mode_after_preempt()

     ...
     count--;

And yes, this only works correctly when _all_ state is consistent. You can't get it to work properly with half of it without creating hard to debug problems.

Thanks,

tglx

author maintainer
#47 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 90 lines · Thomas Gleixner
Quoted 41 lines · Boqun Feng
Quoted 11 lines · Thomas Gleixner

The main problem is that you cover only half of it and there are completely correct cases where this simply blows up in your face:

local_irq_disable(); // does not affect CNT
....
guard(raw_spinlock)(&l1); // does not affect CNT
foo()
guard(raw_spinlock_irqsave)(&l2); // observes CNT = 0

so the unlocking of &l2 will enable interrupts prematurely.

If we are talking the switch in this patch, then no, the unlocking of &l2 will NOT enable interrupts prematurely. The above code expands as the following (using pseudo code to describe how raw_spin_lock_irq_disable(), raw_spin_lock_irq_enable(), local_interrupt_disable(), and local_interrupt_enable() work)

   local_irq_disable();                          // does not affect CNT
   ....
   guard(raw_spinlock)(&l1);                     // does not affect CNT
      foo()
        guard(raw_spinlock_irqsave)(&l2): 
          raw_spin_lock_irq_disable():
            local_interrupt_disable():
	      CNT++;
              this_cpu(state) = local_irq_save(); // record the current state
            raw_spin_lock(&l2);
	  ...
          raw_spin_lock_irq_enable():
            raw_spin_unlock(&l2);
            local_interrupt_enable():
              CNT--;
              if (CNT == 0)
                local_irq_restore(this_cpu(state)); // recover the previous state

So local_interrupt_disable() and local_interrupt_enable() only recover to the previous state, as a result it'll not enable interrupt prematurely here. In other words, the following code works:

Quoted 7 lines
    local_irq_disable();
    local_interrupt_disable();
    local_interrupt_enable();  // interrupt is not re-enabled here
                               // similar to how preempt_disable() does
			       // in a nested preemption disable
			       // critical section.
    local_irq_enable();

Fair enough. I misread that part.

But my main observation that the counter is inconsistent still stands and I think that's a fundamental flaw because there is no way that code can rely on that counter until everything has been converted over and the interrupt/exception/nmi/syscall entry/exit code has been fixed up.

Just let me look at local_interrupt_disable() and __irq_exit_rcu() again.

local_interrupt_disable()
new_count = hardirq_disable_enter();

   /* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */

   if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
      _local_interrupt_disable();

This is absolutely not ok. Why?

The counter is incremented _before_ interrupts are actually disabled. Now in __irq_exit_rcu():

        if (!in_interrupt() && !hardirq_disable_count() &&
            local_softirq_pending()) {

which prevents soft interrupt handling in a completely legitimate situation. As a consequence _nothing_ will handle the pending soft interrupt until:

- an interrupt coming in which observes consistent state

- a local_bh_enable() processes them

That explains that recently quite a few more spurious 'local soft irq pending' printk's have been observed by people as there is no guarantee that either one of those events happens _before_ a CPU reaches idle. Even in the non-idle case deferring this to the next 'by chance' handling is fundamentally broken. This needs to be removed ASAP.

Quoted 5 lines

But coming back to the problem underneath. The ordering in local_interrupt_disable() is simply wrong. You need to disable first and then update the counter. Reverse order for enable() obviously update counter and enable, which you got right.

Noted, the reason that I used the current order is to optimize local_interrupt_disable() from re-disabling interrupt every time:

	https://lore.kernel.org/rust-for-linux/87a5eu7gvw.ffs@tglx/

but looks like we cannot do it without the fixups you mention below. For now I will reverse the order and remove the additional checking in softirq to fix the softirq pending issue.

Regards,
Boqun

Quoted 14 lines

And to make stuff work correctly you need the fixups I pointed out in my previous reply to the various entry/exit functions. It's exactly the same problem as we handle in interrupt/exception/nmi entry/exit code vs. RCU, lockdep, tracing etc.

So if disable() does:

   if (!count)
   	arch_local_irq_disable();
   count++;

then an interrupt hitting before arch_local_irq_disable() will always observe the correct state. After that it can't be delivered.

[...]

maintainer
#48 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 11 lines · Boqun Feng
Quoted 5 lines · Thomas Gleixner

But coming back to the problem underneath. The ordering in local_interrupt_disable() is simply wrong. You need to disable first and then update the counter. Reverse order for enable() obviously update counter and enable, which you got right.

Noted, the reason that I used the current order is to optimize local_interrupt_disable() from re-disabling interrupt every time:

	https://lore.kernel.org/rust-for-linux/87a5eu7gvw.ffs@tglx/

Yes. I gave you the wrong order, but I expected you to actually think it through and not blindly copy it. :)

Quoted 1 line

but looks like we cannot do it without the fixups you mention below.

But that does not mean it can't be done. Checking for 0 first and incrementing after the actual disable is still achieving the same result of touching the CPU only once, no?

Quoted 2 lines

For now I will reverse the order and remove the additional checking in softirq to fix the softirq pending issue.

That "fixes" another nasty bug which was latent for weeks and people could not get a handle on it because it was absolutely not reproducible. Given all that I'm absolutely not convinced that there isn't another pile of latent surprises lurking.

Aside of that I'm worried about having this new counter exposed in the current state of affairs. Nothing prevents arbitrary code from using hardirq_disable_count(), which is definitely faster than irqs_disabled(), but returns a random value depending on context. That's just another recipe for latent and hard to debug disasters to happen as you already demonstrated in __irq_exit_rcu().

It's not the end of the world to bite the bullet and undo the whole pile, except for the then unused expansion of preempt count, go back to the drawing board and come up with a consistent and better overall solution.

I know that hurts, I've been there myself more than once. But at the end I was always happy that we decided to rip it out instead of trying to debug and duct tape it to death.

A inconsistent and fragile facility is worse than having none.

Thanks,

tglx

author maintainer
#49 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 8 lines · Thomas Gleixner
Quoted 7 lines · Boqun Feng
Quoted 5 lines · Thomas Gleixner

But coming back to the problem underneath. The ordering in local_interrupt_disable() is simply wrong. You need to disable first and then update the counter. Reverse order for enable() obviously update counter and enable, which you got right.

Quoted 8 lines
Quoted 4 lines

Noted, the reason that I used the current order is to optimize local_interrupt_disable() from re-disabling interrupt every time:

	https://lore.kernel.org/rust-for-linux/87a5eu7gvw.ffs@tglx/

Yes. I gave you the wrong order, but I expected you to actually think it through and not blindly copy it. :)

No, not blaming you :) I was just providing a bit more context.

I did think through a few parts to make it work, but TBH I lack of the sensitivity for the impact that no interrupt happen on one CPU for a while, so I didn't think this part very seriously. And I just liked the idea we could skip disabling IRQ if possible.

Quoted 6 lines
Quoted 1 line

but looks like we cannot do it without the fixups you mention below.

But that does not mean it can't be done. Checking for 0 first and incrementing after the actual disable is still achieving the same result of touching the CPU only once, no?

Yeah, that should work. But I need to think a bit hard on this.

Quoted 8 lines
Quoted 2 lines

For now I will reverse the order and remove the additional checking in softirq to fix the softirq pending issue.

That "fixes" another nasty bug which was latent for weeks and people could not get a handle on it because it was absolutely not reproducible. Given all that I'm absolutely not convinced that there isn't another pile of latent surprises lurking.

Quoted 4 lines

Aside of that I'm worried about having this new counter exposed in the current state of affairs. Nothing prevents arbitrary code from using hardirq_disable_count(), which is definitely faster than irqs_disabled(), but returns a random value depending on context. That's

Random how? Are you saying in the current (wrong) order? Because after reversing the order, hardirq_disable_count() != 0 means the interrupt has been disabled, no?

But I checked, actually with the reverse order, we don't need hardirq_disable_count(), so we can remove it entirely. Will send a follow up patch on this.

Quoted 8 lines

just another recipe for latent and hard to debug disasters to happen as you already demonstrated in __irq_exit_rcu().

It's not the end of the world to bite the bullet and undo the whole pile, except for the then unused expansion of preempt count, go back to the drawing board and come up with a consistent and better overall solution.

Quoted 6 lines

I know that hurts, I've been there myself more than once. But at the end I was always happy that we decided to rip it out instead of trying to debug and duct tape it to death.

A inconsistent and fragile facility is worse than having none.

To be honest, it doesn't hurt myself if we have to redo the work, I would always like to do it correct. So I don't mind doing that. But it might hurt others who want to develop real drivers with Rust because no SpinLockIrq for them until the redo finishes. That's the major reason that I would like to keep local_interrupt_disable() and spin_lock_irq_disable().

(I also feel like with the order fix and hardirq_disable_count() remove, the design is robust enough to exist and evolve, but I may miss something subtle?)

Alternatively, we can move the current API to be Rust use only (we can make the implementation in Rust even, if we maintain the state and counter in Rust) in this way, there is only a limit set interactions from the new things with the existing kernel, and Rust can always make the guard work properly.

But honestly, it'll be just duplicating what we already have here to the Rust side. So it's not my own desire that I want to keep the current things in tree, it's more that I also look at this from a different angle, and it make some sense engineer-wise: the semantics of local_interrupt_disable() is so easy and straightforward that I feel it's unfair to block the potential user especially when the users can guarantee the correct usages with the type system.

Anyway, that's just my two cents.

Regards,
Boqun

Quoted 10 lines

Thanks,

tglx

maintainer
#50 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 14 lines · Boqun Feng
Quoted 8 lines · Thomas Gleixner
Quoted 2 lines

For now I will reverse the order and remove the additional checking in softirq to fix the softirq pending issue.

That "fixes" another nasty bug which was latent for weeks and people could not get a handle on it because it was absolutely not reproducible. Given all that I'm absolutely not convinced that there isn't another pile of latent surprises lurking.

Quoted 4 lines

Aside of that I'm worried about having this new counter exposed in the current state of affairs. Nothing prevents arbitrary code from using hardirq_disable_count(), which is definitely faster than irqs_disabled(), but returns a random value depending on context. That's

Quoted 7 lines

Random how? Are you saying in the current (wrong) order? Because after reversing the order, hardirq_disable_count() != 0 means the interrupt has been disabled, no?

But I checked, actually with the reverse order, we don't need hardirq_disable_count(), so we can remove it entirely. Will send a follow up patch on this.

The point is that the counter is only valid when used within the limits of the current coverage. Other than that it is not:

        spin_lock_irq() // or any other non-covered mechanism
            // observes 0
            cnt = preempt_count() & HARDIRQ_DISABLE_MASK;

That's inconsistent and therefore it is a random number, no?

You have no way to prevent that this happens and if it does it becomes a nightmare to debug for everyone. Guess who got the bug reports about preemption counter issues and local softirq pending messages in his inbox and dealt with them.

There is a world outside of your safe rust zone and that needs to be safe too. This half finished attempt to make Rust work is absolutely not and I have zero interrest to deal with the fallout.

It's not safe and no extra hacks will make it safe. Which means it is not ready. So the only sensible thing is to revert everything which touches that section of preempt_count() and provides interfaces.

As this annoyed me, I rumaged through my poison cabinet and found the old patches again. They obviously don't apply anymore but I found the hints which corners need some care. With the generic entry code that also got way simpler.

So I sat down and reverted

1b0866874833 ("locking: Switch to _irq_{disable,enable}() variants in cleanup guards") e901c1510e24 ("irq,spin_lock: Add counted interrupt disabling/enabling")

and then hacked it up just to see how far I get before vanishing to bed.

Three hours later it surprisingly booted right away into a full distro kernel and survived kernel builds and a few test cases. :)

Obviously I did not do any serious testing on it, but I wanted to share it as a starting point and food for thoughts.

Yes, it needs to be enabled per architecture as the preempt counter initialization is architecture specific and it requires generic entry code. But those are not uncommon prerequisites and an incentive for architecture people to get their act together.

But it is fully consistent and the fully refcounted thing can be built on top of it. If you look carefuly you'll notice that __raw_local_irq_disable/enable() are just optimized versions of __raw_local_irq_save/restore() as they don't have the conditionals, so they can be unified completely at least for debug builds or in general when it turns out that the overhead is neglible.

There is a wide range of optimizations possible with that especially by combining preempt/interrupt modifications into one operation and rescheduling without changing the preemption counter in the first place. Which is what I hinted to in the mail you linked earlier. I'm so tempted to hack that up tomorrow once my brain is less fried than now and after I exposed it to some serious testing.

Thanks,

tglx
---

arch/x86/Kconfig +1 -0
@@ -318,6 +318,7 @@ config X86
 	select PCI_DOMAINS			if PCI
 	select PCI_LOCKLESS_CONFIG		if PCI
 	select PERF_EVENTS
+	select PREEMPT_COUNT_IRQFLAGS
 	select RTC_LIB
 	select RTC_MC146818_LIB
 	select SPARSE_IRQ
arch/x86/include/asm/preempt.h +2 -2
@@ -61,8 +61,8 @@ static __always_inline void preempt_coun
  */
 #define init_task_preempt_count(p) do { } while (0)
 
-#define init_idle_preempt_count(p, cpu) do { \
-	per_cpu(__preempt_count, (cpu)) = PREEMPT_DISABLED; \
+#define init_idle_preempt_count(p, cpu) do {						\
+	per_cpu(__preempt_count, (cpu)) = PREEMPT_DISABLED | HARDIRQ_DISABLE_OFFSET;	\
 } while (0)
 
 /*
include/linux/irq-entry-common.h +6 -0
@@ -97,6 +97,7 @@ static __always_inline bool arch_in_rcu_
  */
 static __always_inline void enter_from_user_mode(struct pt_regs *regs)
 {
+	__preempt_count_inc_hardirqs_disable();
 	arch_enter_from_user_mode(regs);
 	lockdep_hardirqs_off(CALLER_ADDR0);
 
@@ -275,6 +276,7 @@ static __always_inline void exit_to_user
 	user_enter_irqoff();
 	arch_exit_to_user_mode();
 	lockdep_hardirqs_on(CALLER_ADDR0);
+	__preempt_count_dec_hardirqs_disable();
 }
 
 /**
@@ -385,6 +387,8 @@ static __always_inline irqentry_state_t
 		.exit_rcu = false,
 	};
 
+	__preempt_count_inc_hardirqs_disable();
+
 	/*
 	 * If this entry hit the idle task invoke ct_irq_enter() whether
 	 * RCU is watching or not.
@@ -498,6 +502,7 @@ irqentry_exit_to_kernel_mode_after_preem
 			instrumentation_end();
 			ct_irq_exit();
 			lockdep_hardirqs_on(CALLER_ADDR0);
+			__preempt_count_dec_hardirqs_disable();
 			return;
 		}
 
@@ -514,6 +519,7 @@ irqentry_exit_to_kernel_mode_after_preem
 		if (state.exit_rcu)
 			ct_irq_exit();
 	}
+	__preempt_count_dec_hardirqs_disable();
 }
 
 /**
include/linux/irqflags.h +102 -8

139 lines not on this page · raw message

include/linux/preempt.h +36 -17

80 lines not on this page · raw message

kernel/Kconfig.preempt +3 -0
@@ -152,6 +152,9 @@ config PREEMPT_DYNAMIC
 	  Interesting if you want the same pre-built kernel should be used for
 	  both Server and Desktop workloads.
 
+config PREEMPT_COUNT_IRQFLAGS
+	bool
+
 config SCHED_CORE
 	bool "Core Scheduling for SMT"
 	depends on SCHED_SMT
kernel/entry/common.c +2 -0
@@ -171,6 +171,7 @@ irqentry_state_t noinstr irqentry_nmi_en
 {
 	irqentry_state_t irq_state;
 
+	__preempt_count_inc_hardirqs_disable();
 	irq_state.lockdep = lockdep_hardirqs_enabled();
 
 	__nmi_enter();
@@ -202,4 +203,5 @@ void noinstr irqentry_nmi_exit(struct pt
 	if (irq_state.lockdep)
 		lockdep_hardirqs_on(CALLER_ADDR0);
 	__nmi_exit();
+	__preempt_count_dec_hardirqs_disable();
 }
author maintainer
#51 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 55 lines · Thomas Gleixner
Quoted 14 lines · Boqun Feng
Quoted 8 lines · Thomas Gleixner
Quoted 2 lines

For now I will reverse the order and remove the additional checking in softirq to fix the softirq pending issue.

That "fixes" another nasty bug which was latent for weeks and people could not get a handle on it because it was absolutely not reproducible. Given all that I'm absolutely not convinced that there isn't another pile of latent surprises lurking.

Quoted 4 lines

Aside of that I'm worried about having this new counter exposed in the current state of affairs. Nothing prevents arbitrary code from using hardirq_disable_count(), which is definitely faster than irqs_disabled(), but returns a random value depending on context. That's

Quoted 7 lines

Random how? Are you saying in the current (wrong) order? Because after reversing the order, hardirq_disable_count() != 0 means the interrupt has been disabled, no?

But I checked, actually with the reverse order, we don't need hardirq_disable_count(), so we can remove it entirely. Will send a follow up patch on this.

The point is that the counter is only valid when used within the limits of the current coverage. Other than that it is not:

        spin_lock_irq() // or any other non-covered mechanism
            // observes 0
            cnt = preempt_count() & HARDIRQ_DISABLE_MASK;

That's inconsistent and therefore it is a random number, no?

You have no way to prevent that this happens and if it does it becomes a nightmare to debug for everyone. Guess who got the bug reports about preemption counter issues and local softirq pending messages in his inbox and dealt with them.

There is a world outside of your safe rust zone and that needs to be safe too. This half finished attempt to make Rust work is absolutely not and I have zero interrest to deal with the fallout.

It's not safe and no extra hacks will make it safe. Which means it is not ready. So the only sensible thing is to revert everything which touches that section of preempt_count() and provides interfaces.

As this annoyed me, I rumaged through my poison cabinet and found the old patches again. They obviously don't apply anymore but I found the hints which corners need some care. With the generic entry code that also got way simpler.

So I sat down and reverted

1b0866874833 ("locking: Switch to _irq_{disable,enable}() variants in cleanup guards") e901c1510e24 ("irq,spin_lock: Add counted interrupt disabling/enabling")

Quoted 8 lines

and then hacked it up just to see how far I get before vanishing to bed.

Three hours later it surprisingly booted right away into a full distro kernel and survived kernel builds and a few test cases. :)

Obviously I did not do any serious testing on it, but I wanted to share it as a starting point and food for thoughts.

My biggest concern is how you are going to handle the oddballs I mentioned in another thread, especially when you need to fix the users. Because this would lead to a all-or-nothing solution: unless we resolve all the oddballs, we cannot enable this.

Quoted 12 lines

Yes, it needs to be enabled per architecture as the preempt counter initialization is architecture specific and it requires generic entry code. But those are not uncommon prerequisites and an incentive for architecture people to get their act together.

But it is fully consistent and the fully refcounted thing can be built on top of it. If you look carefuly you'll notice that __raw_local_irq_disable/enable() are just optimized versions of __raw_local_irq_save/restore() as they don't have the conditionals, so they can be unified completely at least for debug builds or in general when it turns out that the overhead is neglible.

Quoted 7 lines

There is a wide range of optimizations possible with that especially by combining preempt/interrupt modifications into one operation and rescheduling without changing the preemption counter in the first place. Which is what I hinted to in the mail you linked earlier. I'm so tempted to hack that up tomorrow once my brain is less fried than now and after I exposed it to some serious testing.

If you did, looking forwards to it, I can help enable that for other architectures if needed.

Regards,
Boqun

Quoted 326 lines

Thanks,

tglx
---

arch/x86/Kconfig +1 -0
@@ -318,6 +318,7 @@ config X86
 	select PCI_DOMAINS			if PCI
 	select PCI_LOCKLESS_CONFIG		if PCI
 	select PERF_EVENTS
+	select PREEMPT_COUNT_IRQFLAGS
 	select RTC_LIB
 	select RTC_MC146818_LIB
 	select SPARSE_IRQ
arch/x86/include/asm/preempt.h +2 -2
@@ -61,8 +61,8 @@ static __always_inline void preempt_coun
  */
 #define init_task_preempt_count(p) do { } while (0)
 
-#define init_idle_preempt_count(p, cpu) do { \
-	per_cpu(__preempt_count, (cpu)) = PREEMPT_DISABLED; \
+#define init_idle_preempt_count(p, cpu) do {						\
+	per_cpu(__preempt_count, (cpu)) = PREEMPT_DISABLED | HARDIRQ_DISABLE_OFFSET;	\
 } while (0)
 
 /*
include/linux/irq-entry-common.h +6 -0
@@ -97,6 +97,7 @@ static __always_inline bool arch_in_rcu_
  */
 static __always_inline void enter_from_user_mode(struct pt_regs *regs)
 {
+	__preempt_count_inc_hardirqs_disable();
 	arch_enter_from_user_mode(regs);
 	lockdep_hardirqs_off(CALLER_ADDR0);
 
@@ -275,6 +276,7 @@ static __always_inline void exit_to_user
 	user_enter_irqoff();
 	arch_exit_to_user_mode();
 	lockdep_hardirqs_on(CALLER_ADDR0);
+	__preempt_count_dec_hardirqs_disable();
 }
 
 /**
@@ -385,6 +387,8 @@ static __always_inline irqentry_state_t
 		.exit_rcu = false,
 	};
 
+	__preempt_count_inc_hardirqs_disable();
+
 	/*
 	 * If this entry hit the idle task invoke ct_irq_enter() whether
 	 * RCU is watching or not.
@@ -498,6 +502,7 @@ irqentry_exit_to_kernel_mode_after_preem
 			instrumentation_end();
 			ct_irq_exit();
 			lockdep_hardirqs_on(CALLER_ADDR0);
+			__preempt_count_dec_hardirqs_disable();
 			return;
 		}
 
@@ -514,6 +519,7 @@ irqentry_exit_to_kernel_mode_after_preem
 		if (state.exit_rcu)
 			ct_irq_exit();
 	}
+	__preempt_count_dec_hardirqs_disable();
 }
 
 /**
include/linux/irqflags.h +102 -8
@@ -13,6 +13,7 @@
 #define _LINUX_TRACE_IRQFLAGS_H
 
 #include <linux/irqflags_types.h>
+#include <linux/preempt.h>
 #include <linux/typecheck.h>
 #include <linux/cleanup.h>
 #include <asm/irqflags.h>
@@ -165,31 +166,124 @@ extern void warn_bogus_irq_restore(void)
 /*
  * Wrap the arch provided IRQ routines to provide appropriate checks.
  */
-#define raw_local_irq_disable()		arch_local_irq_disable()
-#define raw_local_irq_enable()		arch_local_irq_enable()
+#ifdef CONFIG_PREEMPT_COUNT_IRQFLAGS
+static __always_inline void raw_local_irq_disable(void)
+{
+	arch_local_irq_disable();
+	preempt_count_add(HARDIRQ_DISABLE_OFFSET);
+}
+
+static __always_inline void raw_local_irq_enable(void)
+{
+	preempt_count_sub(HARDIRQ_DISABLE_OFFSET);
+	arch_local_irq_enable();
+}
+
+static __always_inline unsigned long __raw_local_irq_save(void)
+{
+	unsigned long cnt = preempt_count();
+
+	if (!(cnt & HARDIRQ_DISABLE_MASK))
+		arch_local_irq_disable();
+	preempt_count_add(HARDIRQ_DISABLE_OFFSET);
+
+	// Probably not even needed unless something feeds 'flags' into
+	// irqs_disabled_flags()
+	return cnt;
+}
+
+static __always_inline void __raw_local_irq_restore(unsigned long cnt)
+{
+	if (!(__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) & HARDIRQ_DISABLE_MASK))
+		arch_local_irq_enable();
+}
+
+static __always_inline unsigned long __raw_local_save_flags(void)
+{
+	return preempt_count() & HARDIRQ_DISABLE_MASK;
+}
+
+static __always_inline bool __raw_irqs_disabled_flags(unsigned long cnt)
+{
+	return !!cnt;
+}
+
+static __always_inline bool raw_irqs_disabled(void)
+{
+	return preempt_count() & HARDIRQ_DISABLE_MASK;
+}
+
+static __always_inline void raw_safe_halt(void)
+{
+	preempt_count_sub(HARDIRQ_DISABLE_OFFSET);
+	arch_safe_halt();
+}
+
+#else
+
+static __always_inline void raw_local_irq_disable(void)
+{
+	arch_local_irq_disable();
+}
+
+static __always_inline void raw_local_irq_enable(void)
+{
+	arch_local_irq_enable();
+}
+
+static __always_inline unsigned long __raw_local_irq_save(void)
+{
+	return arch_local_irq_save();
+}
+
+static __always_inline void __raw_local_irq_restore(unsigned long flags)
+{
+	arch_local_irq_restore(flags);
+}
+
+static __always_inline unsigned long __raw_local_save_flags(void)
+{
+	return arch_local_save_flags();
+}
+
+static __always_inline bool __raw_irqs_disabled_flags(unsigned long flags)
+{
+	return arch_irqs_disabled_flags(flags);
+}
+
+static __always_inline bool raw_irqs_disabled(void)
+{
+	return arch_irqs_disabled();
+}
+
+static __always_inline void raw_safe_halt(void)
+{
+	arch_safe_halt();
+}
+
+#endif
+
 #define raw_local_irq_save(flags)			\
 	do {						\
 		typecheck(unsigned long, flags);	\
-		flags = arch_local_irq_save();		\
+		flags = __raw_local_irq_save();		\
 	} while (0)
 #define raw_local_irq_restore(flags)			\
 	do {						\
 		typecheck(unsigned long, flags);	\
 		raw_check_bogus_irq_restore();		\
-		arch_local_irq_restore(flags);		\
+		__raw_local_irq_restore(flags);	\
 	} while (0)
 #define raw_local_save_flags(flags)			\
 	do {						\
 		typecheck(unsigned long, flags);	\
-		flags = arch_local_save_flags();	\
+		flags = __raw_local_save_flags();	\
 	} while (0)
 #define raw_irqs_disabled_flags(flags)			\
 	({						\
 		typecheck(unsigned long, flags);	\
-		arch_irqs_disabled_flags(flags);	\
+		__raw_irqs_disabled_flags(flags);	\
 	})
-#define raw_irqs_disabled()		(arch_irqs_disabled())
-#define raw_safe_halt()			arch_safe_halt()
 
 /*
  * The local_irq_*() APIs are equal to the raw_local_irq*()
include/linux/preempt.h +36 -17
@@ -54,31 +54,31 @@
  *             NMI_MASK:	0xf0000000
  * (PREEMPT_NEED_RESCHED is in a different word)
  */
-#define PREEMPT_BITS	8
-#define SOFTIRQ_BITS	8
+#define PREEMPT_BITS		8
+#define SOFTIRQ_BITS		8
 #define HARDIRQ_DISABLE_BITS	8
-#define HARDIRQ_BITS	4
-#define NMI_BITS	(1 + 3*IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS))
+#define HARDIRQ_BITS		4
+#define NMI_BITS		(1 + 3*IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS))
 
-#define PREEMPT_SHIFT	0
-#define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
+#define PREEMPT_SHIFT		0
+#define SOFTIRQ_SHIFT		(PREEMPT_SHIFT + PREEMPT_BITS)
 #define HARDIRQ_DISABLE_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
-#define HARDIRQ_SHIFT	(HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS)
-#define NMI_SHIFT	(HARDIRQ_SHIFT + HARDIRQ_BITS)
+#define HARDIRQ_SHIFT		(HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS)
+#define NMI_SHIFT		(HARDIRQ_SHIFT + HARDIRQ_BITS)
 
-#define __IRQ_MASK(x)	((1UL << (x))-1)
+#define __IRQ_MASK(x)		((1UL << (x))-1)
 
-#define PREEMPT_MASK	(__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)
-#define SOFTIRQ_MASK	(__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
+#define PREEMPT_MASK		(__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)
+#define SOFTIRQ_MASK		(__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
 #define HARDIRQ_DISABLE_MASK	(__IRQ_MASK(HARDIRQ_DISABLE_BITS) << HARDIRQ_DISABLE_SHIFT)
-#define HARDIRQ_MASK	(__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
-#define NMI_MASK	(__IRQ_MASK(NMI_BITS)     << NMI_SHIFT)
+#define HARDIRQ_MASK		(__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
+#define NMI_MASK		(__IRQ_MASK(NMI_BITS)     << NMI_SHIFT)
 
-#define PREEMPT_OFFSET	(1UL << PREEMPT_SHIFT)
-#define SOFTIRQ_OFFSET	(1UL << SOFTIRQ_SHIFT)
+#define PREEMPT_OFFSET		(1UL << PREEMPT_SHIFT)
+#define SOFTIRQ_OFFSET		(1UL << SOFTIRQ_SHIFT)
 #define HARDIRQ_DISABLE_OFFSET	(1UL << HARDIRQ_DISABLE_SHIFT)
-#define HARDIRQ_OFFSET	(1UL << HARDIRQ_SHIFT)
-#define NMI_OFFSET	(1UL << NMI_SHIFT)
+#define HARDIRQ_OFFSET		(1UL << HARDIRQ_SHIFT)
+#define NMI_OFFSET		(1UL << NMI_SHIFT)
 
 #define SOFTIRQ_DISABLE_OFFSET	(2 * SOFTIRQ_OFFSET)
 
@@ -90,7 +90,11 @@
  *
  * Reset by start_kernel()->sched_init()->init_idle()->init_idle_preempt_count().
  */
+#ifdef CONFIG_PREEMPT_COUNT_IRQFLAGS
+#define INIT_PREEMPT_COUNT	(PREEMPT_OFFSET + HARDIRQ_DISABLE_OFFSET)
+#else
 #define INIT_PREEMPT_COUNT	PREEMPT_OFFSET
+#endif
 
 /*
  * Initial preempt_count value; reflects the preempt_count schedule invariant
@@ -322,6 +326,21 @@ do { \
 
 #endif /* CONFIG_PREEMPT_COUNT */
 
+#ifdef CONFIG_PREEMPT_COUNT_IRQFLAGS
+static __always_inline void __preempt_count_inc_hardirqs_disable(void)
+{
+	__preempt_count_add(HARDIRQ_DISABLE_OFFSET);
+}
+
+static __always_inline void __preempt_count_dec_hardirqs_disable(void)
+{
+	__preempt_count_sub(HARDIRQ_DISABLE_OFFSET);
+}
+#else
+static __always_inline void __preempt_count_inc_hardirqs_disable(void) { }
+static __always_inline void __preempt_count_dec_hardirqs_disable(void) { }
+#endif
+
 #ifdef MODULE
 /*
  * Modules have no business playing preemption tricks.
kernel/Kconfig.preempt +3 -0
@@ -152,6 +152,9 @@ config PREEMPT_DYNAMIC
 	  Interesting if you want the same pre-built kernel should be used for
 	  both Server and Desktop workloads.
 
+config PREEMPT_COUNT_IRQFLAGS
+	bool
+
 config SCHED_CORE
 	bool "Core Scheduling for SMT"
 	depends on SCHED_SMT
kernel/entry/common.c +2 -0
@@ -171,6 +171,7 @@ irqentry_state_t noinstr irqentry_nmi_en
 {
 	irqentry_state_t irq_state;
 
+	__preempt_count_inc_hardirqs_disable();
 	irq_state.lockdep = lockdep_hardirqs_enabled();
 
 	__nmi_enter();
@@ -202,4 +203,5 @@ void noinstr irqentry_nmi_exit(struct pt
 	if (irq_state.lockdep)
 		lockdep_hardirqs_on(CALLER_ADDR0);
 	__nmi_exit();
+	__preempt_count_dec_hardirqs_disable();
 }
maintainer
#52 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

In reply to Thomas Gleixner
Quoted 55 lines · Thomas Gleixner
include/linux/irq-entry-common.h +6 -0
@@ -97,6 +97,7 @@ static __always_inline bool arch_in_rcu_
  */
 static __always_inline void enter_from_user_mode(struct pt_regs *regs)
 {
+	__preempt_count_inc_hardirqs_disable();
 	arch_enter_from_user_mode(regs);
 	lockdep_hardirqs_off(CALLER_ADDR0);
 
@@ -275,6 +276,7 @@ static __always_inline void exit_to_user
 	user_enter_irqoff();
 	arch_exit_to_user_mode();
 	lockdep_hardirqs_on(CALLER_ADDR0);
+	__preempt_count_dec_hardirqs_disable();
 }
 
 /**
@@ -385,6 +387,8 @@ static __always_inline irqentry_state_t
 		.exit_rcu = false,
 	};
 
+	__preempt_count_inc_hardirqs_disable();
+
 	/*
 	 * If this entry hit the idle task invoke ct_irq_enter() whether
 	 * RCU is watching or not.
@@ -498,6 +502,7 @@ irqentry_exit_to_kernel_mode_after_preem
 			instrumentation_end();
 			ct_irq_exit();
 			lockdep_hardirqs_on(CALLER_ADDR0);
+			__preempt_count_dec_hardirqs_disable();
 			return;
 		}
 
@@ -514,6 +519,7 @@ irqentry_exit_to_kernel_mode_after_preem
 		if (state.exit_rcu)
 			ct_irq_exit();
 	}
+	__preempt_count_dec_hardirqs_disable();
 }
 
 /**
kernel/entry/common.c +1 -0
@@ -171,6 +171,7 @@ irqentry_state_t noinstr irqentry_nmi_en
 {
 	irqentry_state_t irq_state;
 
+	__preempt_count_inc_hardirqs_disable();
 	irq_state.lockdep = lockdep_hardirqs_enabled();
 
 	__nmi_enter();

kernel/entry/common.c void noinstr irqentry_nmi_exit(struct pt · line 207

Quoted 5 lines
kernel/entry/common.c +1 -0
@@ -202,4 +203,5 @@ void noinstr irqentry_nmi_exit(struct pt
 	if (irq_state.lockdep)
 		lockdep_hardirqs_on(CALLER_ADDR0);
 	__nmi_exit();
+	__preempt_count_dec_hardirqs_disable();
 }

So I was thinking about this, and can't we get away with not doing this? That is, simply leave DISABLED_OFFSET set while in userspace?

We always exit to userspace with IRQs disabled, and every entry will disable them anyway, so they match up, might as well make use of that, no?

maintainer
#53 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

In reply to Thomas Gleixner
Quoted 10 lines · Thomas Gleixner

On Thu, Aug 27 2026 at 12:41, Boqun Feng wrote:
So I sat down and reverted

1b0866874833 ("locking: Switch to _irq_{disable,enable}() variants in cleanup guards") e901c1510e24 ("irq,spin_lock: Add counted interrupt disabling/enabling")

and then hacked it up just to see how far I get before vanishing to bed.

Three hours later it surprisingly booted right away into a full distro kernel and survived kernel builds and a few test cases. :)

/FACMEPALM

Yesterday night I was really surprised but too tired to think about it.

When I came around today to look at it again I was more than embarrassed to figure out that the KVM script rebuilt the wrong branch over and over. So the build numbers kept increasing...

Brown paperbag time ...

Of course the real thing did _NOT_ boot at all, so I sat down and figured out what's going wrong and added a pile of debug to it, which is sadly non-existing in this magic local_interrupt_dis/enable() code.

The overall fallout is moderate. Some of it are actual (but harmless) bugs and the rest are the oddball cases we talked about before.

It builds and boots now for real, but of course your mileage will vary depending on hardware and .config. Combo patch on top of the reverts is below.

The whole pile can be retrieved from git via:

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git irqflags

I have some thoughts about how to deal with the overall disaster, but that has to wait until my brain is truly awake again...

Thanks,

tglx
---

arch/x86/Kconfig +1 -0
index 15fd9ec5ecac..c7229a7cfa7c 100644
@@ -133,6 +133,7 @@ config X86
 	select ARCH_USES_CFI_TRAPS		if X86_64 && CFI
 	select ARCH_SUPPORTS_LTO_CLANG
 	select ARCH_SUPPORTS_LTO_CLANG_THIN
+	select ARCH_SUPPORTS_PREEMPT_COUNT_IRQFLAGS
 	select ARCH_SUPPORTS_RT
 	select ARCH_USE_BUILTIN_BSWAP
 	select ARCH_USE_CMPXCHG_LOCKREF
arch/x86/include/asm/hardirq.h +2 -0
index dea60d66d976..34bdf24b939b 100644
@@ -113,4 +113,6 @@ static __always_inline bool kvm_get_cpu_l1tf_flush_l1d(void)
 static __always_inline void kvm_set_cpu_l1tf_flush_l1d(void) { }
 #endif /* IS_ENABLED(CONFIG_KVM_INTEL) */
 
+#define __ARCH_IRQ_EXIT_IRQS_DISABLED 1
+
 #endif /* _ASM_X86_HARDIRQ_H */
arch/x86/include/asm/preempt.h +12 -2
index fafb6f8cdac3..8b4d4cbae52e 100644
@@ -61,10 +61,20 @@ static __always_inline void preempt_count_set(unsigned long pc)
  */
 #define init_task_preempt_count(p) do { } while (0)
 
-#define init_idle_preempt_count(p, cpu) do { \
-	per_cpu(__preempt_count, (cpu)) = PREEMPT_DISABLED; \
+#ifdef CONFIG_PREEMPT_COUNT_IRQFLAGS
+
+#define init_idle_preempt_count(p, cpu) do {						\
+	per_cpu(__preempt_count, (cpu)) = PREEMPT_DISABLED | HARDIRQ_DISABLE_OFFSET;	\
 } while (0)
 
+#else
+
+#define init_idle_preempt_count(p, cpu) do {						\
+	per_cpu(__preempt_count, (cpu)) = PREEMPT_DISABLED;				\
+} while (0)
+
+#endif
+
 /*
  * We fold the NEED_RESCHED bit into the preempt count such that
  * preempt_enable() can decrement and test for needing to reschedule with a
arch/x86/kernel/kvm.c +3 -3
index 6b0a5861ccb8..0b5a05cb543f 100644
@@ -256,9 +256,9 @@ noinstr u32 kvm_read_and_reset_apf_flags(void)
 {
 	u32 flags = 0;
 
-	if (__this_cpu_read(async_pf_enabled)) {
-		flags = __this_cpu_read(apf_reason.flags);
-		__this_cpu_write(apf_reason.flags, 0);
+	if (raw_cpu_read(async_pf_enabled)) {
+		flags = raw_cpu_read(apf_reason.flags);
+		raw_cpu_write(apf_reason.flags, 0);
 	}
 
 	return flags;
arch/x86/kernel/process.c +1 -1
index 346c438ac880..b0be24a386f3 100644
@@ -824,7 +824,7 @@ void __noreturn stop_this_cpu(void *dummy)
 	struct cpuinfo_x86 *c = this_cpu_ptr(&cpu_info);
 	unsigned int cpu = smp_processor_id();
 
-	local_irq_disable();
+	raw_force_local_irq_disable();
 
 	/*
 	 * Remove this CPU from the online mask and disable it
arch/x86/kernel/reboot.c +6 -5
index 0fed6d0d7e32..40104170be24 100644
@@ -98,7 +98,7 @@ static int __init set_efi_reboot(const struct dmi_system_id *d)
 
 void __noreturn machine_real_restart(unsigned int type)
 {
-	local_irq_disable();
+	raw_force_local_irq_disable();
 
 	/*
 	 * Write zero to CMOS register number 0x0f, which the BIOS POST
@@ -535,7 +535,7 @@ static inline void nmi_shootdown_cpus_on_restart(void);
 #if IS_ENABLED(CONFIG_KVM_X86)
 static void emergency_reboot_disable_virtualization(void)
 {
-	local_irq_disable();
+	raw_force_local_irq_disable();
 
 	/*
 	 * Disable virtualization on all CPUs before rebooting to avoid hanging
@@ -699,7 +699,7 @@ void native_machine_shutdown(void)
 	 * not receive the per-cpu timer interrupt which may trigger
 	 * scheduler's load balance.
 	 */
-	local_irq_disable();
+	raw_force_local_irq_disable();
 	stop_other_cpus();
 #endif
 
@@ -823,7 +823,8 @@ static int crash_nmi_callback(unsigned int val, struct pt_regs *regs)
 	 */
 	if (cpu == crashing_cpu)
 		return NMI_HANDLED;
-	local_irq_disable();
+
+	raw_force_local_irq_disable();
 
 	if (shootdown_callback)
 		shootdown_callback(cpu, regs);
@@ -865,7 +866,7 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback)
 {
 	unsigned long msecs;
 
-	local_irq_disable();
+	raw_force_local_irq_disable();
 
 	/*
 	 * Avoid certain doom if a shootdown already occurred; re-registering
arch/x86/mm/fault.c +2 -1
index aa88370ce739..3164eab7cecd 100644
@@ -1486,7 +1486,8 @@ handle_page_fault(struct pt_regs *regs, unsigned long error_code,
 	 * page fault handling might have reenabled interrupts,
 	 * make sure to disable them again.
 	 */
-	local_irq_disable();
+	if (!irqs_disabled())
+		local_irq_disable();
 }
 
 DEFINE_IDTENTRY_RAW_ERRORCODE(exc_page_fault)
drivers/acpi/sleep.c +1 -1
index 132a9df98471..f13e88fb7a78 100644
@@ -1095,7 +1095,7 @@ static int acpi_power_off(struct sys_off_data *data)
 {
 	/* acpi_sleep_prepare(ACPI_STATE_S5) should have already been called */
 	pr_debug("%s called\n", __func__);
-	local_irq_disable();
+	raw_force_local_irq_disable();
 	acpi_enter_sleep_state(ACPI_STATE_S5);
 	return NOTIFY_DONE;
 }
include/linux/interrupt.h +2 -1
index 3bf969ad8fe0..b54afbd6e073 100644
@@ -594,13 +594,14 @@ struct softirq_action
 
 asmlinkage void do_softirq(void);
 asmlinkage void __do_softirq(void);
+void do_softirq_irqsoff(void);
 
 #ifdef CONFIG_PREEMPT_RT
 extern void do_softirq_post_smp_call_flush(unsigned int was_pending);
 #else
 static inline void do_softirq_post_smp_call_flush(unsigned int unused)
 {
-	do_softirq();
+	do_softirq_irqsoff();
 }
 #endif
 
include/linux/irq-entry-common.h +6 -0
index 0bb6c03481fa..ba5210fab31f 100644
@@ -97,6 +97,7 @@ static __always_inline bool arch_in_rcu_eqs(void) { return false; }
  */
 static __always_inline void enter_from_user_mode(struct pt_regs *regs)
 {
+	__preempt_count_inc_hardirqs_disable();
 	arch_enter_from_user_mode(regs);
 	lockdep_hardirqs_off(CALLER_ADDR0);
 
@@ -275,6 +276,7 @@ static __always_inline void exit_to_user_mode(void)
 	user_enter_irqoff();
 	arch_exit_to_user_mode();
 	lockdep_hardirqs_on(CALLER_ADDR0);
+	__preempt_count_dec_hardirqs_disable();
 }
 
 /**
@@ -385,6 +387,8 @@ static __always_inline irqentry_state_t irqentry_enter_from_kernel_mode(struct p
 		.exit_rcu = false,
 	};
 
+	__preempt_count_inc_hardirqs_disable();
+
 	/*
 	 * If this entry hit the idle task invoke ct_irq_enter() whether
 	 * RCU is watching or not.
@@ -498,6 +502,7 @@ irqentry_exit_to_kernel_mode_after_preempt(struct pt_regs *regs, irqentry_state_
 			instrumentation_end();
 			ct_irq_exit();
 			lockdep_hardirqs_on(CALLER_ADDR0);
+			__preempt_count_dec_hardirqs_disable();
 			return;
 		}
 
@@ -514,6 +519,7 @@ irqentry_exit_to_kernel_mode_after_preempt(struct pt_regs *regs, irqentry_state_
 		if (state.exit_rcu)
 			ct_irq_exit();
 	}
+	__preempt_count_dec_hardirqs_disable();
 }
 
 /**
include/linux/irqflags.h +126 -9

165 lines not on this page · raw message

include/linux/preempt.h +56 -20

117 lines not on this page · raw message

include/linux/uaccess.h +2 -2
index eddbbb65ccc4..086de4c18575 100644
@@ -296,9 +296,9 @@ static inline bool pagefault_disabled(void)
  * stick to pagefault_disabled().
  * Please NEVER use preempt_disable() to disable the fault handler. With
  * !CONFIG_PREEMPT_COUNT, this is like a NOP. So the handler won't be disabled.
- * in_atomic() will report different values based on !CONFIG_PREEMPT_COUNT.
+ * fault_in_atomic() will report different values based on !CONFIG_PREEMPT_COUNT.
  */
-#define faulthandler_disabled() (pagefault_disabled() || in_atomic())
+#define faulthandler_disabled() (pagefault_disabled() || fault_in_atomic())
 
 DEFINE_LOCK_GUARD_0(pagefault, pagefault_disable(), pagefault_enable())
 
init/main.c +0 -1
index 2613d3f9b3ce..fa84ce260b04 100644
@@ -991,7 +991,6 @@ void start_kernel(void)
 
 	cgroup_init_early();
 
-	local_irq_disable();
 	early_boot_irqs_disabled = true;
 
 	/*
kernel/Kconfig.preempt +9 -0
index f294dad43bd7..c44607990219 100644
@@ -152,6 +152,15 @@ config PREEMPT_DYNAMIC
 	  Interesting if you want the same pre-built kernel should be used for
 	  both Server and Desktop workloads.
 
+config ARCH_SUPPORTS_PREEMPT_COUNT_IRQFLAGS
+	bool
+
+config PREEMPT_COUNT_IRQFLAGS
+	bool "Enable reference counted interrupt disable/enable mechanisms"
+	depends on ARCH_SUPPORTS_PREEMPT_COUNT_IRQFLAGS
+	help
+	  FIXME: Add some useful blurb
+
 config SCHED_CORE
 	bool "Core Scheduling for SMT"
 	depends on SCHED_SMT
kernel/entry/common.c +2 -0
index e3d381fd3d25..3c93ce7f86f6 100644
@@ -171,6 +171,7 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs)
 {
 	irqentry_state_t irq_state;
 
+	__preempt_count_inc_hardirqs_disable();
 	irq_state.lockdep = lockdep_hardirqs_enabled();
 
 	__nmi_enter();
@@ -202,4 +203,5 @@ void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t irq_state)
 	if (irq_state.lockdep)
 		lockdep_hardirqs_on(CALLER_ADDR0);
 	__nmi_exit();
+	__preempt_count_dec_hardirqs_disable();
 }
kernel/sched/core.c +1 -1
index f78275192036..b6d14feaaf56 100644
@@ -5343,7 +5343,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)
 	 *
 	 * Also, see FORK_PREEMPT_COUNT.
 	 */
-	if (WARN_ONCE(preempt_count() != 2*PREEMPT_DISABLE_OFFSET,
+	if (WARN_ONCE(preempt_count() != SCHED_PREEMPT_COUNT,
 		      "corrupted preempt_count: %s/%d/0x%x\n",
 		      current->comm, current->pid, preempt_count()))
 		preempt_count_set(FORK_PREEMPT_COUNT);
kernel/sched/idle.c +2 -1
index eb73b65ce6c4..7132320035eb 100644
@@ -380,7 +380,8 @@ static void do_idle(void)
 	 * RCU relies on this call to be done outside of an RCU read-side
 	 * critical section.
 	 */
-	flush_smp_call_function_queue();
+	scoped_guard(irq)
+		flush_smp_call_function_queue();
 	schedule_idle();
 
 	if (unlikely(klp_patch_pending(current)))
kernel/smp.c +2 -4
index b696bcc60c08..d51e4bc8da1b 100644
@@ -665,19 +665,17 @@ static void __flush_smp_call_function_queue(bool warn_cpu_offline)
 void flush_smp_call_function_queue(void)
 {
 	unsigned int was_pending;
-	unsigned long flags;
 
 	if (llist_empty(this_cpu_ptr(&call_single_queue)))
 		return;
 
-	local_irq_save(flags);
+	lockdep_assert_irqs_disabled();
+
 	/* Get the already pending soft interrupts for RT enabled kernels */
 	was_pending = local_softirq_pending();
 	__flush_smp_call_function_queue(true);
 	if (local_softirq_pending())
 		do_softirq_post_smp_call_flush(was_pending);
-
-	local_irq_restore(flags);
 }
 
 static int __smp_call_function_single(int cpu, smp_call_func_t func,
kernel/softirq.c +13 -9
index e1a773e3eb4e..33f82d03ca2b 100644
@@ -455,7 +455,10 @@ void __local_bh_enable_ip(unsigned long ip, unsigned int cnt)
 		 * Run softirq if any pending. And do it in its own stack
 		 * as we may be calling this deep in a task call stack already.
 		 */
-		do_softirq();
+		if (IS_ENABLED(CONFIG_TRACE_IRQFLAGS))
+			do_softirq_irqsoff();
+		else
+			do_softirq();
 	}
 
 	preempt_count_dec();
@@ -517,20 +520,21 @@ static inline void invoke_softirq(void)
 
 asmlinkage __visible void do_softirq(void)
 {
-	__u32 pending;
-	unsigned long flags;
-
 	if (in_interrupt())
 		return;
 
-	local_irq_save(flags);
+	guard(irqsave)();
+	if (local_softirq_pending())
+		do_softirq_own_stack();
+}
 
-	pending = local_softirq_pending();
+void do_softirq_irqsoff(void)
+{
+	if (in_interrupt())
+		return;
 
-	if (pending)
+	if (local_softirq_pending())
 		do_softirq_own_stack();
-
-	local_irq_restore(flags);
 }
 
 #endif /* !CONFIG_PREEMPT_RT */
kernel/time/hrtimer.c +1 -1
index 530d61257b9a..977dd8928934 100644
@@ -2068,7 +2068,7 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base, struct hrtimer_cloc
 
 	lockdep_hrtimer_exit(expires_in_hardirq);
 	trace_hrtimer_expire_exit(timer);
-	raw_spin_lock_irq(&cpu_base->lock);
+	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 
 	/*
 	 * Note: We clear the running state after enqueue_hrtimer and
author maintainer
#54 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 326 lines · Thomas Gleixner
Quoted 10 lines · Thomas Gleixner

On Thu, Aug 27 2026 at 12:41, Boqun Feng wrote:
So I sat down and reverted

1b0866874833 ("locking: Switch to _irq_{disable,enable}() variants in cleanup guards") e901c1510e24 ("irq,spin_lock: Add counted interrupt disabling/enabling")

and then hacked it up just to see how far I get before vanishing to bed.

Three hours later it surprisingly booted right away into a full distro kernel and survived kernel builds and a few test cases. :)

/FACMEPALM

Yesterday night I was really surprised but too tired to think about it.

When I came around today to look at it again I was more than embarrassed to figure out that the KVM script rebuilt the wrong branch over and over. So the build numbers kept increasing...

Brown paperbag time ...

Of course the real thing did _NOT_ boot at all, so I sat down and figured out what's going wrong and added a pile of debug to it, which is sadly non-existing in this magic local_interrupt_dis/enable() code.

The overall fallout is moderate. Some of it are actual (but harmless) bugs and the rest are the oddball cases we talked about before.

It builds and boots now for real, but of course your mileage will vary depending on hardware and .config. Combo patch on top of the reverts is below.

The whole pile can be retrieved from git via:

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git irqflags

I have some thoughts about how to deal with the overall disaster, but that has to wait until my brain is truly awake again...

Thanks,

tglx
---

arch/x86/Kconfig +1 -0
index 15fd9ec5ecac..c7229a7cfa7c 100644
@@ -133,6 +133,7 @@ config X86
 	select ARCH_USES_CFI_TRAPS		if X86_64 && CFI
 	select ARCH_SUPPORTS_LTO_CLANG
 	select ARCH_SUPPORTS_LTO_CLANG_THIN
+	select ARCH_SUPPORTS_PREEMPT_COUNT_IRQFLAGS
 	select ARCH_SUPPORTS_RT
 	select ARCH_USE_BUILTIN_BSWAP
 	select ARCH_USE_CMPXCHG_LOCKREF
arch/x86/include/asm/hardirq.h +2 -0
index dea60d66d976..34bdf24b939b 100644
@@ -113,4 +113,6 @@ static __always_inline bool kvm_get_cpu_l1tf_flush_l1d(void)
 static __always_inline void kvm_set_cpu_l1tf_flush_l1d(void) { }
 #endif /* IS_ENABLED(CONFIG_KVM_INTEL) */
 
+#define __ARCH_IRQ_EXIT_IRQS_DISABLED 1
+
 #endif /* _ASM_X86_HARDIRQ_H */
arch/x86/include/asm/preempt.h +12 -2
index fafb6f8cdac3..8b4d4cbae52e 100644
@@ -61,10 +61,20 @@ static __always_inline void preempt_count_set(unsigned long pc)
  */
 #define init_task_preempt_count(p) do { } while (0)
 
-#define init_idle_preempt_count(p, cpu) do { \
-	per_cpu(__preempt_count, (cpu)) = PREEMPT_DISABLED; \
+#ifdef CONFIG_PREEMPT_COUNT_IRQFLAGS
+
+#define init_idle_preempt_count(p, cpu) do {						\
+	per_cpu(__preempt_count, (cpu)) = PREEMPT_DISABLED | HARDIRQ_DISABLE_OFFSET;	\
 } while (0)
 
+#else
+
+#define init_idle_preempt_count(p, cpu) do {						\
+	per_cpu(__preempt_count, (cpu)) = PREEMPT_DISABLED;				\
+} while (0)
+
+#endif
+
 /*
  * We fold the NEED_RESCHED bit into the preempt count such that
  * preempt_enable() can decrement and test for needing to reschedule with a
arch/x86/kernel/kvm.c +3 -3
index 6b0a5861ccb8..0b5a05cb543f 100644
@@ -256,9 +256,9 @@ noinstr u32 kvm_read_and_reset_apf_flags(void)
 {
 	u32 flags = 0;
 
-	if (__this_cpu_read(async_pf_enabled)) {
-		flags = __this_cpu_read(apf_reason.flags);
-		__this_cpu_write(apf_reason.flags, 0);
+	if (raw_cpu_read(async_pf_enabled)) {
+		flags = raw_cpu_read(apf_reason.flags);
+		raw_cpu_write(apf_reason.flags, 0);
 	}
 
 	return flags;
arch/x86/kernel/process.c +1 -1
index 346c438ac880..b0be24a386f3 100644
@@ -824,7 +824,7 @@ void __noreturn stop_this_cpu(void *dummy)
 	struct cpuinfo_x86 *c = this_cpu_ptr(&cpu_info);
 	unsigned int cpu = smp_processor_id();
 
-	local_irq_disable();
+	raw_force_local_irq_disable();
 
 	/*
 	 * Remove this CPU from the online mask and disable it
arch/x86/kernel/reboot.c +6 -5
index 0fed6d0d7e32..40104170be24 100644
@@ -98,7 +98,7 @@ static int __init set_efi_reboot(const struct dmi_system_id *d)
 
 void __noreturn machine_real_restart(unsigned int type)
 {
-	local_irq_disable();
+	raw_force_local_irq_disable();
 
 	/*
 	 * Write zero to CMOS register number 0x0f, which the BIOS POST
@@ -535,7 +535,7 @@ static inline void nmi_shootdown_cpus_on_restart(void);
 #if IS_ENABLED(CONFIG_KVM_X86)
 static void emergency_reboot_disable_virtualization(void)
 {
-	local_irq_disable();
+	raw_force_local_irq_disable();
 
 	/*
 	 * Disable virtualization on all CPUs before rebooting to avoid hanging
@@ -699,7 +699,7 @@ void native_machine_shutdown(void)
 	 * not receive the per-cpu timer interrupt which may trigger
 	 * scheduler's load balance.
 	 */
-	local_irq_disable();
+	raw_force_local_irq_disable();
 	stop_other_cpus();
 #endif
 
@@ -823,7 +823,8 @@ static int crash_nmi_callback(unsigned int val, struct pt_regs *regs)
 	 */
 	if (cpu == crashing_cpu)
 		return NMI_HANDLED;
-	local_irq_disable();
+
+	raw_force_local_irq_disable();
 
 	if (shootdown_callback)
 		shootdown_callback(cpu, regs);
@@ -865,7 +866,7 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback)
 {
 	unsigned long msecs;
 
-	local_irq_disable();
+	raw_force_local_irq_disable();
 
 	/*
 	 * Avoid certain doom if a shootdown already occurred; re-registering
arch/x86/mm/fault.c +2 -1
index aa88370ce739..3164eab7cecd 100644
@@ -1486,7 +1486,8 @@ handle_page_fault(struct pt_regs *regs, unsigned long error_code,
 	 * page fault handling might have reenabled interrupts,
 	 * make sure to disable them again.
 	 */
-	local_irq_disable();
+	if (!irqs_disabled())
+		local_irq_disable();
 }
 
 DEFINE_IDTENTRY_RAW_ERRORCODE(exc_page_fault)
drivers/acpi/sleep.c +1 -1
index 132a9df98471..f13e88fb7a78 100644
@@ -1095,7 +1095,7 @@ static int acpi_power_off(struct sys_off_data *data)
 {
 	/* acpi_sleep_prepare(ACPI_STATE_S5) should have already been called */
 	pr_debug("%s called\n", __func__);
-	local_irq_disable();
+	raw_force_local_irq_disable();
 	acpi_enter_sleep_state(ACPI_STATE_S5);
 	return NOTIFY_DONE;
 }
include/linux/interrupt.h +2 -1
index 3bf969ad8fe0..b54afbd6e073 100644
@@ -594,13 +594,14 @@ struct softirq_action
 
 asmlinkage void do_softirq(void);
 asmlinkage void __do_softirq(void);
+void do_softirq_irqsoff(void);
 
 #ifdef CONFIG_PREEMPT_RT
 extern void do_softirq_post_smp_call_flush(unsigned int was_pending);
 #else
 static inline void do_softirq_post_smp_call_flush(unsigned int unused)
 {
-	do_softirq();
+	do_softirq_irqsoff();
 }
 #endif
 
include/linux/irq-entry-common.h +6 -0
index 0bb6c03481fa..ba5210fab31f 100644
@@ -97,6 +97,7 @@ static __always_inline bool arch_in_rcu_eqs(void) { return false; }
  */
 static __always_inline void enter_from_user_mode(struct pt_regs *regs)
 {
+	__preempt_count_inc_hardirqs_disable();
 	arch_enter_from_user_mode(regs);
 	lockdep_hardirqs_off(CALLER_ADDR0);
 
@@ -275,6 +276,7 @@ static __always_inline void exit_to_user_mode(void)
 	user_enter_irqoff();
 	arch_exit_to_user_mode();
 	lockdep_hardirqs_on(CALLER_ADDR0);
+	__preempt_count_dec_hardirqs_disable();
 }
 
 /**
@@ -385,6 +387,8 @@ static __always_inline irqentry_state_t irqentry_enter_from_kernel_mode(struct p
 		.exit_rcu = false,
 	};
 
+	__preempt_count_inc_hardirqs_disable();
+
 	/*
 	 * If this entry hit the idle task invoke ct_irq_enter() whether
 	 * RCU is watching or not.
@@ -498,6 +502,7 @@ irqentry_exit_to_kernel_mode_after_preempt(struct pt_regs *regs, irqentry_state_
 			instrumentation_end();
 			ct_irq_exit();
 			lockdep_hardirqs_on(CALLER_ADDR0);
+			__preempt_count_dec_hardirqs_disable();
 			return;
 		}
 
@@ -514,6 +519,7 @@ irqentry_exit_to_kernel_mode_after_preempt(struct pt_regs *regs, irqentry_state_
 		if (state.exit_rcu)
 			ct_irq_exit();
 	}
+	__preempt_count_dec_hardirqs_disable();
 }
 
 /**
include/linux/irqflags.h +41 -3
index 57b074e0cfbb..dd55786768d1 100644
@@ -13,6 +13,7 @@
 #define _LINUX_TRACE_IRQFLAGS_H
 
 #include <linux/irqflags_types.h>
+#include <linux/preempt.h>
 #include <linux/typecheck.h>
 #include <linux/cleanup.h>
 #include <asm/irqflags.h>
@@ -163,33 +164,149 @@ extern void warn_bogus_irq_restore(void);
 #endif
 
 /*
- * Wrap the arch provided IRQ routines to provide appropriate checks.
+ * Wrap the architecture specific routines to provide appropriate checks.
  */
-#define raw_local_irq_disable()		arch_local_irq_disable()
-#define raw_local_irq_enable()		arch_local_irq_enable()
+#ifdef CONFIG_PREEMPT_COUNT_IRQFLAGS
+
+// FIXME: Convert this into a proper debug mechanism
+#define debug_assert(c)					\
+do {							\
+	WARN_ON(!(c));					\
+} while (0)
+
+static __always_inline void raw_local_irq_disable(void)
+{
+	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK) == 0);
+	arch_local_irq_disable();
+	__preempt_count_add(HARDIRQ_DISABLE_OFFSET);
+}
+
+static __always_inline void raw_force_local_irq_disable(void)
+{
+	arch_local_irq_disable();
+	__preempt_count_add(HARDIRQ_DISABLE_OFFSET);
+}
+
+static __always_inline void raw_local_irq_enable(void)
+{
+	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET);
+	__preempt_count_sub(HARDIRQ_DISABLE_OFFSET);
+	arch_local_irq_enable();
+}
+
+static __always_inline unsigned long __raw_local_irq_save(void)
+{
+	unsigned int cnt = preempt_count() & HARDIRQ_DISABLE_MASK;
+
+	debug_assert(cnt != HARDIRQ_DISABLE_MASK);
+
+	if (!cnt)
+		arch_local_irq_disable();
+	__preempt_count_add(HARDIRQ_DISABLE_OFFSET);
+
+	return cnt;

include/linux/irqflags.h extern void warn_bogus_irq_restore(void); · line 217

Quoted 10 lines
include/linux/irqflags.h +10 -0
+}
+
+static __always_inline void __raw_local_irq_restore(unsigned long cnt)
+{
+	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK) == (cnt + HARDIRQ_DISABLE_OFFSET));
+
+	if (!(__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) & HARDIRQ_DISABLE_MASK))
+		arch_local_irq_enable();
+}
+

So we can change the semantics of local_irq_disable(), local_irq_enable(), local_irq_save() and local_irq_restore()? Nice! local_irq_{en,dis}able() are no longer idempotent, and local_irq_{save,restore}() have to pair with each other or a local_irq_{en,dis}able(). This would make things much easier. And TBH, I never think this is an option because there could be so much code not obeying this (at least not on day 1). Now I see your point on getting the design correct at the first place :D

Regards,
Boqun

Quoted 407 lines
include/linux/irqflags.h +75 -6
+static __always_inline unsigned long __raw_local_save_flags(void)
+{
+	return preempt_count() & HARDIRQ_DISABLE_MASK;
+}
+
+static __always_inline bool __raw_irqs_disabled_flags(unsigned long cnt)
+{
+	return !!cnt;
+}
+
+static __always_inline bool raw_irqs_disabled(void)
+{
+	return preempt_count() & HARDIRQ_DISABLE_MASK;
+}
+
+static __always_inline void raw_safe_halt(void)
+{
+	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET);
+	__preempt_count_sub(HARDIRQ_DISABLE_OFFSET);
+	arch_safe_halt();
+}
+
+#else
+
+static __always_inline void raw_local_irq_disable(void)
+{
+	arch_local_irq_disable();
+}
+
+static __always_inline void raw_force_local_irq_disable(void)
+{
+	arch_local_irq_disable();
+}
+
+static __always_inline void raw_local_irq_enable(void)
+{
+	arch_local_irq_enable();
+}
+
+static __always_inline unsigned long __raw_local_irq_save(void)
+{
+	return arch_local_irq_save();
+}
+
+static __always_inline void __raw_local_irq_restore(unsigned long flags)
+{
+	arch_local_irq_restore(flags);
+}
+
+static __always_inline unsigned long __raw_local_save_flags(void)
+{
+	return arch_local_save_flags();
+}
+
+static __always_inline bool __raw_irqs_disabled_flags(unsigned long flags)
+{
+	return arch_irqs_disabled_flags(flags);
+}
+
+static __always_inline bool raw_irqs_disabled(void)
+{
+	return arch_irqs_disabled();
+}
+
+static __always_inline void raw_safe_halt(void)
+{
+	arch_safe_halt();
+}
+
+#endif
+
 #define raw_local_irq_save(flags)			\
 	do {						\
 		typecheck(unsigned long, flags);	\
-		flags = arch_local_irq_save();		\
+		flags = __raw_local_irq_save();		\
 	} while (0)
 #define raw_local_irq_restore(flags)			\
 	do {						\
 		typecheck(unsigned long, flags);	\
 		raw_check_bogus_irq_restore();		\
-		arch_local_irq_restore(flags);		\
+		__raw_local_irq_restore(flags);		\
 	} while (0)
 #define raw_local_save_flags(flags)			\
 	do {						\
 		typecheck(unsigned long, flags);	\
-		flags = arch_local_save_flags();	\
+		flags = __raw_local_save_flags();	\
 	} while (0)
 #define raw_irqs_disabled_flags(flags)			\
 	({						\
 		typecheck(unsigned long, flags);	\
-		arch_irqs_disabled_flags(flags);	\
+		__raw_irqs_disabled_flags(flags);	\
 	})
-#define raw_irqs_disabled()		(arch_irqs_disabled())
-#define raw_safe_halt()			arch_safe_halt()
 
 /*
  * The local_irq_*() APIs are equal to the raw_local_irq*()
include/linux/preempt.h +56 -20
index 2e689de7b29a..c953cdfb3cc2 100644
@@ -54,31 +54,31 @@
  *             NMI_MASK:	0xf0000000
  * (PREEMPT_NEED_RESCHED is in a different word)
  */
-#define PREEMPT_BITS	8
-#define SOFTIRQ_BITS	8
+#define PREEMPT_BITS		8
+#define SOFTIRQ_BITS		8
 #define HARDIRQ_DISABLE_BITS	8
-#define HARDIRQ_BITS	4
-#define NMI_BITS	(1 + 3*IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS))
+#define HARDIRQ_BITS		4
+#define NMI_BITS		(1 + 3 * IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS))
 
-#define PREEMPT_SHIFT	0
-#define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
+#define PREEMPT_SHIFT		0
+#define SOFTIRQ_SHIFT		(PREEMPT_SHIFT + PREEMPT_BITS)
 #define HARDIRQ_DISABLE_SHIFT	(SOFTIRQ_SHIFT + SOFTIRQ_BITS)
-#define HARDIRQ_SHIFT	(HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS)
-#define NMI_SHIFT	(HARDIRQ_SHIFT + HARDIRQ_BITS)
+#define HARDIRQ_SHIFT		(HARDIRQ_DISABLE_SHIFT + HARDIRQ_DISABLE_BITS)
+#define NMI_SHIFT		(HARDIRQ_SHIFT + HARDIRQ_BITS)
 
-#define __IRQ_MASK(x)	((1UL << (x))-1)
+#define __IRQ_MASK(x)		((1UL << (x))-1)
 
-#define PREEMPT_MASK	(__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)
-#define SOFTIRQ_MASK	(__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
+#define PREEMPT_MASK		(__IRQ_MASK(PREEMPT_BITS) << PREEMPT_SHIFT)
+#define SOFTIRQ_MASK		(__IRQ_MASK(SOFTIRQ_BITS) << SOFTIRQ_SHIFT)
 #define HARDIRQ_DISABLE_MASK	(__IRQ_MASK(HARDIRQ_DISABLE_BITS) << HARDIRQ_DISABLE_SHIFT)
-#define HARDIRQ_MASK	(__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
-#define NMI_MASK	(__IRQ_MASK(NMI_BITS)     << NMI_SHIFT)
+#define HARDIRQ_MASK		(__IRQ_MASK(HARDIRQ_BITS) << HARDIRQ_SHIFT)
+#define NMI_MASK		(__IRQ_MASK(NMI_BITS)     << NMI_SHIFT)
 
-#define PREEMPT_OFFSET	(1UL << PREEMPT_SHIFT)
-#define SOFTIRQ_OFFSET	(1UL << SOFTIRQ_SHIFT)
+#define PREEMPT_OFFSET		(1UL << PREEMPT_SHIFT)
+#define SOFTIRQ_OFFSET		(1UL << SOFTIRQ_SHIFT)
 #define HARDIRQ_DISABLE_OFFSET	(1UL << HARDIRQ_DISABLE_SHIFT)
-#define HARDIRQ_OFFSET	(1UL << HARDIRQ_SHIFT)
-#define NMI_OFFSET	(1UL << NMI_SHIFT)
+#define HARDIRQ_OFFSET		(1UL << HARDIRQ_SHIFT)
+#define NMI_OFFSET		(1UL << NMI_SHIFT)
 
 #define SOFTIRQ_DISABLE_OFFSET	(2 * SOFTIRQ_OFFSET)
 
@@ -90,18 +90,29 @@
  *
  * Reset by start_kernel()->sched_init()->init_idle()->init_idle_preempt_count().
  */
+
+#ifdef CONFIG_PREEMPT_COUNT_IRQFLAGS
+
+#define INIT_PREEMPT_COUNT	(PREEMPT_OFFSET + HARDIRQ_DISABLE_OFFSET)
+#define SCHED_PREEMPT_COUNT	(2 * PREEMPT_DISABLE_OFFSET + HARDIRQ_DISABLE_OFFSET)
+
+#else
+
 #define INIT_PREEMPT_COUNT	PREEMPT_OFFSET
+#define SCHED_PREEMPT_COUNT	(2 * PREEMPT_DISABLE_OFFSET)
+
+#endif
 
 /*
  * Initial preempt_count value; reflects the preempt_count schedule invariant
  * which states that during context switches:
  *
- *    preempt_count() == 2*PREEMPT_DISABLE_OFFSET
+ *    preempt_count() == SCHED_PREEMPT_COUNT
  *
- * Note: PREEMPT_DISABLE_OFFSET is 0 for !PREEMPT_COUNT kernels.
+ * Note: SCHED_PREEMPT_COUNT is 0 for !PREEMPT_COUNT kernels.
  * Note: See finish_task_switch().
  */
-#define FORK_PREEMPT_COUNT	(2*PREEMPT_DISABLE_OFFSET + PREEMPT_ENABLED)
+#define FORK_PREEMPT_COUNT	(SCHED_PREEMPT_COUNT + PREEMPT_ENABLED)
 
 /* preempt_count() and related functions, depends on PREEMPT_NEED_RESCHED */
 #include <asm/preempt.h>
@@ -168,6 +179,16 @@ static __always_inline unsigned char interrupt_context_level(void)
 #define in_softirq()		(softirq_count())
 #define in_interrupt()		(irq_count())
 
+/*
+ * Check whether a fault happened in an atomic context. Depending on
+ * CONFIG_PREEMPT_COUNT and CONFIG_PREEMPTION this check might be useless.
+ */
+#ifdef CONFIG_PREEMPT_COUNT_IRQFLAGS
+# define fault_in_atomic()	(preempt_count() != HARDIRQ_DISABLE_OFFSET)
+#else
+# define fault_in_atomic()	in_atomic()
+#endif
+
 /*
  * The preempt_count offset after preempt_disable();
  */
@@ -322,6 +343,21 @@ do { \
 
 #endif /* CONFIG_PREEMPT_COUNT */
 
+#ifdef CONFIG_PREEMPT_COUNT_IRQFLAGS
+static __always_inline void __preempt_count_inc_hardirqs_disable(void)
+{
+	__preempt_count_add(HARDIRQ_DISABLE_OFFSET);
+}
+
+static __always_inline void __preempt_count_dec_hardirqs_disable(void)
+{
+	__preempt_count_sub(HARDIRQ_DISABLE_OFFSET);
+}
+#else
+static __always_inline void __preempt_count_inc_hardirqs_disable(void) { }
+static __always_inline void __preempt_count_dec_hardirqs_disable(void) { }
+#endif
+
 #ifdef MODULE
 /*
  * Modules have no business playing preemption tricks.
include/linux/uaccess.h +2 -2
index eddbbb65ccc4..086de4c18575 100644
@@ -296,9 +296,9 @@ static inline bool pagefault_disabled(void)
  * stick to pagefault_disabled().
  * Please NEVER use preempt_disable() to disable the fault handler. With
  * !CONFIG_PREEMPT_COUNT, this is like a NOP. So the handler won't be disabled.
- * in_atomic() will report different values based on !CONFIG_PREEMPT_COUNT.
+ * fault_in_atomic() will report different values based on !CONFIG_PREEMPT_COUNT.
  */
-#define faulthandler_disabled() (pagefault_disabled() || in_atomic())
+#define faulthandler_disabled() (pagefault_disabled() || fault_in_atomic())
 
 DEFINE_LOCK_GUARD_0(pagefault, pagefault_disable(), pagefault_enable())
 
init/main.c +0 -1
index 2613d3f9b3ce..fa84ce260b04 100644
@@ -991,7 +991,6 @@ void start_kernel(void)
 
 	cgroup_init_early();
 
-	local_irq_disable();
 	early_boot_irqs_disabled = true;
 
 	/*
kernel/Kconfig.preempt +9 -0
index f294dad43bd7..c44607990219 100644
@@ -152,6 +152,15 @@ config PREEMPT_DYNAMIC
 	  Interesting if you want the same pre-built kernel should be used for
 	  both Server and Desktop workloads.
 
+config ARCH_SUPPORTS_PREEMPT_COUNT_IRQFLAGS
+	bool
+
+config PREEMPT_COUNT_IRQFLAGS
+	bool "Enable reference counted interrupt disable/enable mechanisms"
+	depends on ARCH_SUPPORTS_PREEMPT_COUNT_IRQFLAGS
+	help
+	  FIXME: Add some useful blurb
+
 config SCHED_CORE
 	bool "Core Scheduling for SMT"
 	depends on SCHED_SMT
kernel/entry/common.c +2 -0
index e3d381fd3d25..3c93ce7f86f6 100644
@@ -171,6 +171,7 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs)
 {
 	irqentry_state_t irq_state;
 
+	__preempt_count_inc_hardirqs_disable();
 	irq_state.lockdep = lockdep_hardirqs_enabled();
 
 	__nmi_enter();
@@ -202,4 +203,5 @@ void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t irq_state)
 	if (irq_state.lockdep)
 		lockdep_hardirqs_on(CALLER_ADDR0);
 	__nmi_exit();
+	__preempt_count_dec_hardirqs_disable();
 }
kernel/sched/core.c +1 -1
index f78275192036..b6d14feaaf56 100644
@@ -5343,7 +5343,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)
 	 *
 	 * Also, see FORK_PREEMPT_COUNT.
 	 */
-	if (WARN_ONCE(preempt_count() != 2*PREEMPT_DISABLE_OFFSET,
+	if (WARN_ONCE(preempt_count() != SCHED_PREEMPT_COUNT,
 		      "corrupted preempt_count: %s/%d/0x%x\n",
 		      current->comm, current->pid, preempt_count()))
 		preempt_count_set(FORK_PREEMPT_COUNT);
kernel/sched/idle.c +2 -1
index eb73b65ce6c4..7132320035eb 100644
@@ -380,7 +380,8 @@ static void do_idle(void)
 	 * RCU relies on this call to be done outside of an RCU read-side
 	 * critical section.
 	 */
-	flush_smp_call_function_queue();
+	scoped_guard(irq)
+		flush_smp_call_function_queue();
 	schedule_idle();
 
 	if (unlikely(klp_patch_pending(current)))
kernel/smp.c +2 -4
index b696bcc60c08..d51e4bc8da1b 100644
@@ -665,19 +665,17 @@ static void __flush_smp_call_function_queue(bool warn_cpu_offline)
 void flush_smp_call_function_queue(void)
 {
 	unsigned int was_pending;
-	unsigned long flags;
 
 	if (llist_empty(this_cpu_ptr(&call_single_queue)))
 		return;
 
-	local_irq_save(flags);
+	lockdep_assert_irqs_disabled();
+
 	/* Get the already pending soft interrupts for RT enabled kernels */
 	was_pending = local_softirq_pending();
 	__flush_smp_call_function_queue(true);
 	if (local_softirq_pending())
 		do_softirq_post_smp_call_flush(was_pending);
-
-	local_irq_restore(flags);
 }
 
 static int __smp_call_function_single(int cpu, smp_call_func_t func,
kernel/softirq.c +13 -9
index e1a773e3eb4e..33f82d03ca2b 100644
@@ -455,7 +455,10 @@ void __local_bh_enable_ip(unsigned long ip, unsigned int cnt)
 		 * Run softirq if any pending. And do it in its own stack
 		 * as we may be calling this deep in a task call stack already.
 		 */
-		do_softirq();
+		if (IS_ENABLED(CONFIG_TRACE_IRQFLAGS))
+			do_softirq_irqsoff();
+		else
+			do_softirq();
 	}
 
 	preempt_count_dec();
@@ -517,20 +520,21 @@ static inline void invoke_softirq(void)
 
 asmlinkage __visible void do_softirq(void)
 {
-	__u32 pending;
-	unsigned long flags;
-
 	if (in_interrupt())
 		return;
 
-	local_irq_save(flags);
+	guard(irqsave)();
+	if (local_softirq_pending())
+		do_softirq_own_stack();
+}
 
-	pending = local_softirq_pending();
+void do_softirq_irqsoff(void)
+{
+	if (in_interrupt())
+		return;
 
-	if (pending)
+	if (local_softirq_pending())
 		do_softirq_own_stack();
-
-	local_irq_restore(flags);
 }
 
 #endif /* !CONFIG_PREEMPT_RT */
kernel/time/hrtimer.c +1 -1
index 530d61257b9a..977dd8928934 100644
@@ -2068,7 +2068,7 @@ static void __run_hrtimer(struct hrtimer_cpu_base *cpu_base, struct hrtimer_cloc
 
 	lockdep_hrtimer_exit(expires_in_hardirq);
 	trace_hrtimer_expire_exit(timer);
-	raw_spin_lock_irq(&cpu_base->lock);
+	raw_spin_lock_irqsave(&cpu_base->lock, flags);
 
 	/*
 	 * Note: We clear the running state after enqueue_hrtimer and
maintainer
#55 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 10 lines · Boqun Feng

include/linux/irqflags.h extern void warn_bogus_irq_restore(void); · line 217

Quoted 8 lines · Thomas Gleixner
include/linux/irqflags.h +8 -0
+static __always_inline void __raw_local_irq_restore(unsigned long cnt)
+{
+	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK) == (cnt + HARDIRQ_DISABLE_OFFSET));
+
+	if (!(__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) & HARDIRQ_DISABLE_MASK))
+		arch_local_irq_enable();
+}
+
Quoted 8 lines

So we can change the semantics of local_irq_disable(), local_irq_enable(), local_irq_save() and local_irq_restore()? Nice! local_irq_{en,dis}able() are no longer idempotent, and local_irq_{save,restore}() have to pair with each other or a local_irq_{en,dis}able(). This would make things much easier. And TBH, I never think this is an option because there could be so much code not obeying this (at least not on day 1). Now I see your point on getting the design correct at the first place :D

Compared to the insanities I had to handle almost 20 years ago when I tried that this has become much easier because lockdep and RT made quite some of the nastier lock/local_irq games go away.

There are probably a few other places in cpu idle drivers or in dark half maintained driver implementations which might need some care, but I expect the overall fallout to be managable. The debug mechanisms should help to identify them quickly.

author maintainer
#56 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 20 lines · Thomas Gleixner
Quoted 10 lines · Boqun Feng

include/linux/irqflags.h extern void warn_bogus_irq_restore(void); · line 217

Quoted 8 lines · Thomas Gleixner
include/linux/irqflags.h +8 -0
+static __always_inline void __raw_local_irq_restore(unsigned long cnt)
+{
+	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK) == (cnt + HARDIRQ_DISABLE_OFFSET));
+
+	if (!(__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) & HARDIRQ_DISABLE_MASK))
+		arch_local_irq_enable();
+}
+
Quoted 8 lines

So we can change the semantics of local_irq_disable(), local_irq_enable(), local_irq_save() and local_irq_restore()? Nice! local_irq_{en,dis}able() are no longer idempotent, and local_irq_{save,restore}() have to pair with each other or a local_irq_{en,dis}able(). This would make things much easier. And TBH, I never think this is an option because there could be so much code not obeying this (at least not on day 1). Now I see your point on getting the design correct at the first place :D

Quoted 9 lines

Compared to the insanities I had to handle almost 20 years ago when I tried that this has become much easier because lockdep and RT made quite some of the nastier lock/local_irq games go away.

There are probably a few other places in cpu idle drivers or in dark half maintained driver implementations which might need some care, but I expect the overall fallout to be managable. The debug mechanisms should help to identify them quickly.

Yeah, that's something I was missing, I was too afraid to change API semantics, but with reasonable debug mechanisms, at least users would have a pointer to resolve the use case issues. Lesson learned.

I will fix the Rust's side compile error.

Regards,
Boqun

maintainer
#57 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

In reply to Thomas Gleixner
Quoted 82 lines · Thomas Gleixner
arch/x86/kernel/process.c +1 -1
index 346c438ac880..b0be24a386f3 100644
@@ -824,7 +824,7 @@ void __noreturn stop_this_cpu(void *dummy)
 	struct cpuinfo_x86 *c = this_cpu_ptr(&cpu_info);
 	unsigned int cpu = smp_processor_id();
 
-	local_irq_disable();
+	raw_force_local_irq_disable();
 
 	/*
 	 * Remove this CPU from the online mask and disable it
arch/x86/kernel/reboot.c +6 -5
index 0fed6d0d7e32..40104170be24 100644
@@ -98,7 +98,7 @@ static int __init set_efi_reboot(const struct dmi_system_id *d)
 
 void __noreturn machine_real_restart(unsigned int type)
 {
-	local_irq_disable();
+	raw_force_local_irq_disable();
 
 	/*
 	 * Write zero to CMOS register number 0x0f, which the BIOS POST
@@ -535,7 +535,7 @@ static inline void nmi_shootdown_cpus_on_restart(void);
 #if IS_ENABLED(CONFIG_KVM_X86)
 static void emergency_reboot_disable_virtualization(void)
 {
-	local_irq_disable();
+	raw_force_local_irq_disable();
 
 	/*
 	 * Disable virtualization on all CPUs before rebooting to avoid hanging
@@ -699,7 +699,7 @@ void native_machine_shutdown(void)
 	 * not receive the per-cpu timer interrupt which may trigger
 	 * scheduler's load balance.
 	 */
-	local_irq_disable();
+	raw_force_local_irq_disable();
 	stop_other_cpus();
 #endif
 
@@ -823,7 +823,8 @@ static int crash_nmi_callback(unsigned int val, struct pt_regs *regs)
 	 */
 	if (cpu == crashing_cpu)
 		return NMI_HANDLED;
-	local_irq_disable();
+
+	raw_force_local_irq_disable();
 
 	if (shootdown_callback)
 		shootdown_callback(cpu, regs);
@@ -865,7 +866,7 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback)
 {
 	unsigned long msecs;
 
-	local_irq_disable();
+	raw_force_local_irq_disable();
 
 	/*
 	 * Avoid certain doom if a shootdown already occurred; re-registering
drivers/acpi/sleep.c +1 -1
index 132a9df98471..f13e88fb7a78 100644
@@ -1095,7 +1095,7 @@ static int acpi_power_off(struct sys_off_data *data)
 {
 	/* acpi_sleep_prepare(ACPI_STATE_S5) should have already been called */
 	pr_debug("%s called\n", __func__);
-	local_irq_disable();
+	raw_force_local_irq_disable();
 	acpi_enter_sleep_state(ACPI_STATE_S5);
 	return NOTIFY_DONE;
 }
include/linux/irqflags.h +0 -0
index 57b074e0cfbb..dd55786768d1 100644

include/linux/irqflags.h extern void warn_bogus_irq_restore(void); · line 188

Quoted 6 lines
include/linux/irqflags.h +5 -0
 
+static __always_inline void raw_force_local_irq_disable(void)
+{
+	arch_local_irq_disable();
+	__preempt_count_add(HARDIRQ_DISABLE_OFFSET);
+}

So this thing is on all sorts of don't care, we're going down paths. It needs to ensure IRQs really are off, and preempt_count has at least one DISABLE_OFFSET on.

*However* if something like acpi_power_off() were to 'fail' to enter S5 and continue on with the notifier, things are now unbalanced. Probably not a problem, since the next handler will likely do raw_force_load_irq_disable() again.

At the least this wants a comment I suppose.

init/main.c void start_kernel(void) · line 994

Quoted 10 lines
init/main.c +0 -1
index 2613d3f9b3ce..fa84ce260b04 100644
@@ -991,7 +991,6 @@ void start_kernel(void)
 
 	cgroup_init_early();
 
-	local_irq_disable();
 	early_boot_irqs_disabled = true;

If we want to preserve the paranoia of having that statement in the first place, it could be replaced with something like:

	WARN_ON_ONCE(!irqs_disabled());

I suppose (lockdep isn't available yet).

maintainer
#58 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 16 lines · Peter Zijlstra

include/linux/irqflags.h extern void warn_bogus_irq_restore(void); · line 188

Quoted 10 lines · Thomas Gleixner
include/linux/irqflags.h +5 -0
index 57b074e0cfbb..dd55786768d1 100644
 
+static __always_inline void raw_force_local_irq_disable(void)
+{
+	arch_local_irq_disable();
+	__preempt_count_add(HARDIRQ_DISABLE_OFFSET);
+}

So this thing is on all sorts of don't care, we're going down paths. It needs to ensure IRQs really are off, and preempt_count has at least one DISABLE_OFFSET on.

Quoted 6 lines

*However* if something like acpi_power_off() were to 'fail' to enter S5 and continue on with the notifier, things are now unbalanced. Probably not a problem, since the next handler will likely do raw_force_load_irq_disable() again.

At the least this wants a comment I suppose.

Right. That stuff needs some eyeballs.

Quoted 8 lines
Quoted 8 lines
init/main.c +0 -0
index 2613d3f9b3ce..fa84ce260b04 100644
@@ -991,7 +991,6 @@ void start_kernel(void)
 
 	cgroup_init_early();
 
Quoted 7 lines

init/main.c void start_kernel(void) · line 994

Quoted 2 lines
init/main.c +0 -1
-	local_irq_disable();
 	early_boot_irqs_disabled = true;

If we want to preserve the paranoia of having that statement in the first place, it could be replaced with something like:

	WARN_ON_ONCE(!irqs_disabled());

Right.

Quoted 1 line

I suppose (lockdep isn't available yet).

Good question.

author maintainer
#59 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

In reply to Thomas Gleixner

On Sat, Aug 29, 2026 at 01:11:56AM +0200, Thomas Gleixner wrote:
[...]

Quoted 10 lines
include/linux/irqflags.h +10 -0
+static __always_inline unsigned long __raw_local_irq_save(void)
+{
+	unsigned int cnt = preempt_count() & HARDIRQ_DISABLE_MASK;
+
+	debug_assert(cnt != HARDIRQ_DISABLE_MASK);
+
+	if (!cnt)
+		arch_local_irq_disable();
+	__preempt_count_add(HARDIRQ_DISABLE_OFFSET);
+

include/linux/irqflags.h extern void warn_bogus_irq_restore(void); · line 216

Quoted 10 lines
include/linux/irqflags.h +10 -0
+	return cnt;
+}
+
+static __always_inline void __raw_local_irq_restore(unsigned long cnt)
+{
+	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK) == (cnt + HARDIRQ_DISABLE_OFFSET));
+
+	if (!(__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) & HARDIRQ_DISABLE_MASK))
+		arch_local_irq_enable();
+}

And while we are at, we can just introduce a raw_local_irq_restore_auto() (definitely needs a better name), which doesn't need a cnt:

static __always_inline void __raw_local_irq_restore_auto(void)

{
	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK));

	if (!(__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) & HARDIRQ_DISABLE_MASK))
		arch_local_irq_enable();
}

And we can slowly convert irq_restore() users to use it?

Regards,
Boqun

Quoted 24 lines
include/linux/irqflags.h +24 -0
+
+static __always_inline unsigned long __raw_local_save_flags(void)
+{
+	return preempt_count() & HARDIRQ_DISABLE_MASK;
+}
+
+static __always_inline bool __raw_irqs_disabled_flags(unsigned long cnt)
+{
+	return !!cnt;
+}
+
+static __always_inline bool raw_irqs_disabled(void)
+{
+	return preempt_count() & HARDIRQ_DISABLE_MASK;
+}
+
+static __always_inline void raw_safe_halt(void)
+{
+	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET);
+	__preempt_count_sub(HARDIRQ_DISABLE_OFFSET);
+	arch_safe_halt();
+}
+
+#else

[..]

author maintainer
#60 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 28 lines · Boqun Feng

On Sat, Aug 29, 2026 at 01:11:56AM +0200, Thomas Gleixner wrote:
[...]

Quoted 10 lines
(quoted diff) +10 -0
+static __always_inline unsigned long __raw_local_irq_save(void)
+{
+	unsigned int cnt = preempt_count() & HARDIRQ_DISABLE_MASK;
+
+	debug_assert(cnt != HARDIRQ_DISABLE_MASK);
+
+	if (!cnt)
+		arch_local_irq_disable();
+	__preempt_count_add(HARDIRQ_DISABLE_OFFSET);
+

Review

Quoted 10 lines
(quoted diff) +10 -0
+	return cnt;
+}
+
+static __always_inline void __raw_local_irq_restore(unsigned long cnt)
+{
+	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK) == (cnt + HARDIRQ_DISABLE_OFFSET));
+
+	if (!(__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) & HARDIRQ_DISABLE_MASK))
+		arch_local_irq_enable();
+}

And while we are at, we can just introduce a raw_local_irq_restore_auto() (definitely needs a better name), which doesn't need a cnt:

static __always_inline void __raw_local_irq_restore_auto(void)

Quoted 9 lines
{
	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK));

	if (!(__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) & HARDIRQ_DISABLE_MASK))
		arch_local_irq_enable();
}

And we can slowly convert irq_restore() users to use it?

I decided to use local_irq_resume(), not sure whether it's a good name either...

If we are OK to not fully revert on commit e901c1510e24 ("irq,spin_lock:
Add counted interrupt disabling/enabling"), I think the following will resolve the 0day built errors on your irqflags branch (I rebased onto tip/locking/urgent with rest of your series on irqflags branch). A few things to notice:

* local_interrupt_*() can obviously be removed at all, this patch is just for the idea, so I didn't do that. Similarly I didn't introduce a spin_lock_irqresume().

* I haven't found a way that we can do a local_irq_resume() when !CONFIG_PREEMPT_COUNT_IRQFLAGS, and if we cannot, it's going to make part of Rust code depends on CONFIG_PREEMPT_COUNT_IRQFLAGS=y

* Further cleanups on hardirq_disable_*() are needed.

Thoughts?

Regards,
Boqun

------------------------------>8
Subject: [PATCH] interrupt: Add {raw_}local_irq_resume()

With the new PREEMPT_COUNT_IRQFLAGS design, local_interrupt_{dis,en}able() can be implemented by the same machinery in local_irq_{save,restore}(). Add a new API local_irq_resume() to skip the need of passing a previous flags/count to local_irq_restore(). Map local_interrupt_{dis,en}able() to local_irq{save,resume}().

---
 include/linux/interrupt_rc.h         | 79 ----------------------------
 include/linux/irqflags.h             | 39 ++++++++++++++
 include/linux/spinlock.h             |  1 -
 kernel/irq/refcount_interrupt_test.c |  2 +-
 kernel/softirq.c                     | 15 ------
 5 files changed, 40 insertions(+), 96 deletions(-)
 delete mode 100644 include/linux/interrupt_rc.h
include/linux/interrupt_rc.h deleted +0 -79
deleted file mode 100644
index e68e1bedba66..000000000000
@@ -1,79 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef __LINUX_INTERRUPT_RC_H
-#define __LINUX_INTERRUPT_RC_H
-
-/*
- * include/linux/interrupt_rc.h - refcounted local processor interrupt
- * management.
- *
- * Since the implementation of this API currently depends on
- * local_irq_save()/local_irq_restore(), we split this into its own header to
- * make it easier to include without hitting circular header dependencies.
- */
-
-#include <linux/irqflags.h>
-#include <linux/preempt.h>
-#include <linux/processor.h>
-#include <linux/smp.h>
-
-#ifndef MODULE
-/* Per-CPU interrupt disabling state for local_interrupt_{disable,enable}(). */
-DECLARE_PER_CPU(unsigned long, local_interrupt_disable_state);
-
-static __always_inline void __local_interrupt_save_state(unsigned long flags)
-{
-	raw_cpu_write(local_interrupt_disable_state, flags);
-}
-
-static __always_inline void __local_interrupt_enable(void)
-{
-	unsigned long flags = raw_cpu_read(local_interrupt_disable_state);
-
-	local_irq_restore(flags);
-}
-
-#ifndef INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
-static __always_inline void _local_interrupt_save_state(unsigned long flags)
-{
-	__local_interrupt_save_state(flags);
-}
-
-static __always_inline void _local_interrupt_enable(void)
-{
-	__local_interrupt_enable();
-}
-#else
-extern void _local_interrupt_save_state(unsigned long flags);
-extern void _local_interrupt_enable(void);
-#endif
-
-#else /* !MODULE */
-extern void _local_interrupt_save_state(unsigned long flags);
-extern void _local_interrupt_enable(void);
-#endif /* !MODULE */
-
-static inline void local_interrupt_disable(void)
-{
-	int new_count;
-	unsigned long flags;
-
-	WARN_ON_ONCE(in_nmi());
-
-	local_irq_save(flags);
-	new_count = hardirq_disable_enter();
-
-	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
-		_local_interrupt_save_state(flags);
-}
-
-static inline void local_interrupt_enable(void)
-{
-	int new_count;
-
-	new_count = hardirq_disable_exit();
-
-	if ((new_count & HARDIRQ_DISABLE_MASK) == 0)
-		_local_interrupt_enable();
-}
-
-#endif /* !__LINUX_INTERRUPT_RC_H */
include/linux/irqflags.h +39 -0
index dd55786768d1..6d4e5f1c73a8 100644
@@ -215,6 +215,15 @@ static __always_inline void __raw_local_irq_restore(unsigned long cnt)
 		arch_local_irq_enable();
 }
 
+/* Same as raw_local_irq_restore() but don't need user to pass a count. */
+static __always_inline void raw_local_irq_resume(void)
+{
+	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK));
+
+	if (!(__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) & HARDIRQ_DISABLE_MASK))
+		arch_local_irq_enable();
+}
+
 static __always_inline unsigned long __raw_local_save_flags(void)
 {
 	return preempt_count() & HARDIRQ_DISABLE_MASK;
@@ -264,6 +273,15 @@ static __always_inline void __raw_local_irq_restore(unsigned long flags)
 	arch_local_irq_restore(flags);
 }
 
+static __always_inline void raw_local_irq_resume(void)
+{
+	/*
+	 * local_irq_resume() is not supported when
+	 * CONFIG_PREEMPT_COUNT_IRQFLAGS = n
+	 */
+	BUG();
+}
+
 static __always_inline unsigned long __raw_local_save_flags(void)
 {
 	return arch_local_save_flags();
@@ -342,6 +360,14 @@ static __always_inline void raw_safe_halt(void)
 		raw_local_irq_restore(flags);		\
 	} while (0)
 
+#define local_irq_resume()						\
+	do {								\
+		if ((preempt_count() & HARDIRQ_DISABLE_MASK) ==		\
+				HARDIRQ_DISABLE_OFFSET)			\
+			trace_hardirqs_on();				\
+		raw_local_irq_resume();					\
+	} while (0)
+
 #define safe_halt()				\
 	do {					\
 		trace_hardirqs_on();		\
@@ -355,10 +381,23 @@ static __always_inline void raw_safe_halt(void)
 #define local_irq_disable()	do { raw_local_irq_disable(); } while (0)
 #define local_irq_save(flags)	do { raw_local_irq_save(flags); } while (0)
 #define local_irq_restore(flags) do { raw_local_irq_restore(flags); } while (0)
+#define local_irq_resume()	do { raw_local_irq_resume(); } while (0)
 #define safe_halt()		do { raw_safe_halt(); } while (0)
 
 #endif /* CONFIG_TRACE_IRQFLAGS */
 
+static __always_inline void local_interrupt_disable(void)
+{
+	unsigned long flags;
+
+	local_irq_save(flags);
+}
+
+static __always_inline void local_interrupt_enable(void)
+{
+	local_irq_resume();
+}
+
 #define local_save_flags(flags)	raw_local_save_flags(flags)
 
 /*
include/linux/spinlock.h +0 -1
index 3d405cc4c121..c619502501e2 100644
@@ -57,7 +57,6 @@
 #include <linux/linkage.h>
 #include <linux/compiler.h>
 #include <linux/irqflags.h>
-#include <linux/interrupt_rc.h>
 #include <linux/thread_info.h>
 #include <linux/stringify.h>
 #include <linux/bottom_half.h>
kernel/irq/refcount_interrupt_test.c +1 -1
index ca904dba24b9..cbd3b5b9cfa3 100644
@@ -4,7 +4,7 @@
  */
 
 #include <kunit/test.h>
-#include <linux/interrupt_rc.h>
+#include <linux/irqflags.h>
 
 #define TEST_IRQ_ON() KUNIT_EXPECT_FALSE(test, irqs_disabled())
 #define TEST_IRQ_OFF() KUNIT_EXPECT_TRUE(test, irqs_disabled())
kernel/softirq.c +0 -15
index d124ae6fd1e4..47fb46f9d6f6 100644
@@ -9,7 +9,6 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
-#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
 #include <linux/export.h>
 #include <linux/kernel_stat.h>
 #include <linux/interrupt.h>
@@ -89,20 +88,6 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabled);
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
-DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state);
-
-void _local_interrupt_save_state(unsigned long flags)
-{
-	__local_interrupt_save_state(flags);
-}
-EXPORT_SYMBOL(_local_interrupt_save_state);
-
-void _local_interrupt_enable(void)
-{
-	__local_interrupt_enable();
-}
-EXPORT_SYMBOL(_local_interrupt_enable);
-
 #ifndef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS
 /*
  * Any 32bit architecture that still cares about performance should
Signature · 1 line
2.50.1 (Apple Git-155)
maintainer
#61 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 5 lines · Boqun Feng
Quoted 4 lines · Boqun Feng

And while we are at, we can just introduce a raw_local_irq_restore_auto() (definitely needs a better name), which doesn't need a cnt:

Quoted 7 lines
Quoted 7 lines

static __always_inline void __raw_local_irq_restore_auto(void)

{
	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK));

	if (!(__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) & HARDIRQ_DISABLE_MASK))
		arch_local_irq_enable();
}

Can we just make the thing work in the first place?

Quoted 1 line
Quoted 1 line

And we can slowly convert irq_restore() users to use it?

Somewhere down the road.

Quoted 8 lines

I decided to use local_irq_resume(), not sure whether it's a good name either...

If we are OK to not fully revert on commit e901c1510e24 ("irq,spin_lock:
Add counted interrupt disabling/enabling"), I think the following will resolve the 0day built errors on your irqflags branch (I rebased onto tip/locking/urgent with rest of your series on irqflags branch). A few things to notice:

The zero day failure is not due to that. It's because I reverted that refcounted muck as well in my git tree.

I just rebased the series on top of tip locking/urgent, which only has the irq,spinlock revert and your patches on top.

Quoted 3 lines

* I haven't found a way that we can do a local_irq_resume() when !CONFIG_PREEMPT_COUNT_IRQFLAGS, and if we cannot, it's going to make part of Rust code depends on CONFIG_PREEMPT_COUNT_IRQFLAGS=y

That's a good thing as it might make people actually get their act together. And you can work around that in interrupt_rc.h itself.

Can you please stop bouncing around like a rubber ball and take your time?

First of all I want to move interrupt_rc.h and the whole spinlock muck into rust/helpers/ now. Why?

Simply because it is Rust only and we don't want to expose any of this stuff to random driver writers. See attached patch.

I've rebased my devel branch on top of tip/locking/urgent and applied that patch so the robots can have their field day.

The actual PREEMPT_COUNT_IRQFLAGS thing will be 7.4 material obviously and as this is confined to Rust then it's trivial enough to work around it locally without exposing more stuff. See tiny delta patch below.

So the only side effect of that is that the Rust implementation will not be fully integrated into the preempt count magic, but it should just work, no?

And when an architecture supports the real thing then it gets all the benefits with bells and whistels.

I really want to get the PREEMPT_COUNT_IRQFLAGS design right first and then we can think about simplifications and cleanups and remove the whole Rust magic once all architectures which support Rust play along.

Thanks,

tglx

---
 rust/helpers/interrupt_rc.h |    6 ++++++
 1 file changed, 6 insertions(+)
rust/helpers/interrupt_rc.h +6 -0
@@ -38,8 +38,14 @@ extern void _local_interrupt_save_state(
 extern void _local_interrupt_enable(void);
 #endif
 
+#ifdef CONFIG_PREEMPT_COUNT_IRQFLAGS
 #define hardirq_disable_enter()	__preempt_count_add_return(HARDIRQ_DISABLE_OFFSET)
 #define hardirq_disable_exit()	__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET)
+#else
+DECLARE_PER_CPU(unsigned int, local_interrupt_cnt);
+#define hardirq_disable_enter()	__this_cpu_add_return(local_interrupt_count, HARDIRQ_DISABLE_OFFSET)
+#define hardirq_disable_exit()	__this_cpu_sub_return(local_interrupt_count, HARDIRQ_DISABLE_OFFSET)
+#endif
 
 static inline void local_interrupt_disable(void)
 {

Subject: movestuff
From: Thomas Gleixner <[email protected]>
Date: Sun, 30 Aug 2026 20:45:30 +0200

---
 include/linux/spinlock.h         |   23 -------
 include/linux/spinlock_api_smp.h |   41 --------------
 include/linux/spinlock_api_up.h  |   15 -----
 include/linux/spinlock_rt.h      |   18 ------
 kernel/locking/spinlock.c        |   31 ----------
 kernel/softirq.c                 |   15 -----
 rust/helpers/interrupt.c         |   21 ++++++-
 rust/helpers/interrupt_rc.h      |   68 +++++++++++++++++++++++
 rust/helpers/spinlock.c          |   39 +++++++++++++
 rust/helpers/spinlock.h          |  114 +++++++++++++++++++++++++++++++++++++++
 10 files changed, 240 insertions(+), 145 deletions(-)
include/linux/spinlock.h +0 -23
@@ -57,7 +57,6 @@
 #include <linux/linkage.h>
 #include <linux/compiler.h>
 #include <linux/irqflags.h>
-#include <linux/interrupt_rc.h>
 #include <linux/thread_info.h>
 #include <linux/stringify.h>
 #include <linux/bottom_half.h>
@@ -274,11 +273,9 @@ static inline void do_raw_spin_unlock(ra
 #endif
 
 #define raw_spin_lock_irq(lock)		_raw_spin_lock_irq(lock)
-#define raw_spin_lock_irq_disable(lock)	_raw_spin_lock_irq_disable(lock)
 #define raw_spin_lock_bh(lock)		_raw_spin_lock_bh(lock)
 #define raw_spin_unlock(lock)		_raw_spin_unlock(lock)
 #define raw_spin_unlock_irq(lock)	_raw_spin_unlock_irq(lock)
-#define raw_spin_unlock_irq_enable(lock)	_raw_spin_unlock_irq_enable(lock)
 
 #define raw_spin_unlock_irqrestore(lock, flags)		\
 	do {							\
@@ -293,8 +290,6 @@ static inline void do_raw_spin_unlock(ra
 
 #define raw_spin_trylock_irqsave(lock, flags) _raw_spin_trylock_irqsave(lock, &(flags))
 
-#define raw_spin_trylock_irq_disable(lock)	_raw_spin_trylock_irq_disable(lock)
-
 #ifndef CONFIG_PREEMPT_RT
 /* Include rwlock functions for !RT */
 #include <linux/rwlock.h>
@@ -377,12 +372,6 @@ static __always_inline void spin_lock_ir
 	raw_spin_lock_irq(&lock->rlock);
 }
 
-static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
-	__acquires(lock) __no_context_analysis
-{
-	raw_spin_lock_irq_disable(&lock->rlock);
-}
-
 #define spin_lock_irqsave(lock, flags)				\
 do {								\
 	raw_spin_lock_irqsave(spinlock_check(lock), flags);	\
@@ -413,12 +402,6 @@ static __always_inline void spin_unlock_
 	raw_spin_unlock_irq(&lock->rlock);
 }
 
-static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
-	__releases(lock) __no_context_analysis
-{
-	raw_spin_unlock_irq_enable(&lock->rlock);
-}
-
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags)
 	__releases(lock) __no_context_analysis
 {
@@ -444,12 +427,6 @@ static __always_inline bool _spin_tryloc
 }
 #define spin_trylock_irqsave(lock, flags) _spin_trylock_irqsave(lock, &(flags))
 
-static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
-	__cond_acquires(true, lock) __no_context_analysis
-{
-	return raw_spin_trylock_irq_disable(&lock->rlock);
-}
-
 /**
  * spin_is_locked() - Check whether a spinlock is locked.
  * @lock: Pointer to the spinlock.
include/linux/spinlock_api_smp.h +0 -41

83 lines not on this page · raw message

include/linux/spinlock_api_up.h +0 -15
@@ -42,9 +42,6 @@
 #define __LOCK_IRQSAVE(lock, flags, ...) \
   do { local_irq_save(flags); __LOCK(lock, ##__VA_ARGS__); } while (0)
 
-#define __LOCK_IRQ_DISABLE(lock, ...) \
-  do { local_interrupt_disable(); __LOCK(lock, ##__VA_ARGS__); } while (0)
-
 #define ___UNLOCK_(lock) \
   do { __release(lock); (void)(lock); } while (0)
 
@@ -64,9 +61,6 @@
 #define __UNLOCK_IRQRESTORE(lock, flags, ...) \
   do { local_irq_restore(flags); __UNLOCK(lock, ##__VA_ARGS__); } while (0)
 
-#define __UNLOCK_IRQ_ENABLE(lock, ...) \
-  do { __UNLOCK(lock, ##__VA_ARGS__); local_interrupt_enable(); } while (0)
-
 #define _raw_spin_lock(lock)			__LOCK(lock)
 #define _raw_spin_lock_nested(lock, subclass)	__LOCK(lock)
 #define _raw_read_lock(lock)			__LOCK(lock, shared)
@@ -76,7 +70,6 @@
 #define _raw_read_lock_bh(lock)			__LOCK_BH(lock, shared)
 #define _raw_write_lock_bh(lock)		__LOCK_BH(lock)
 #define _raw_spin_lock_irq(lock)		__LOCK_IRQ(lock)
-#define _raw_spin_lock_irq_disable(lock)	__LOCK_IRQ_DISABLE(lock)
 #define _raw_read_lock_irq(lock)		__LOCK_IRQ(lock, shared)
 #define _raw_write_lock_irq(lock)		__LOCK_IRQ(lock)
 #define _raw_spin_lock_irqsave(lock, flags)	__LOCK_IRQSAVE(lock, flags)
@@ -104,13 +97,6 @@ static __always_inline int _raw_spin_try
 	return 1;
 }
 
-static __always_inline int _raw_spin_trylock_irq_disable(raw_spinlock_t *lock)
-	__cond_acquires(true, lock)
-{
-	__LOCK_IRQ_DISABLE(lock);
-	return 1;
-}
-
 static __always_inline int _raw_spin_trylock_irqsave(raw_spinlock_t *lock, unsigned long *flags)
 	__cond_acquires(true, lock)
 {
@@ -146,7 +132,6 @@ static __always_inline int _raw_write_tr
 #define _raw_write_unlock_bh(lock)		__UNLOCK_BH(lock)
 #define _raw_read_unlock_bh(lock)		__UNLOCK_BH(lock, shared)
 #define _raw_spin_unlock_irq(lock)		__UNLOCK_IRQ(lock)
-#define _raw_spin_unlock_irq_enable(lock)	__UNLOCK_IRQ_ENABLE(lock)
 #define _raw_read_unlock_irq(lock)		__UNLOCK_IRQ(lock, shared)
 #define _raw_write_unlock_irq(lock)		__UNLOCK_IRQ(lock)
 #define _raw_spin_unlock_irqrestore(lock, flags) \
include/linux/spinlock_rt.h +0 -18
@@ -96,12 +96,6 @@ static __always_inline void spin_lock_ir
 	rt_spin_lock(lock);
 }
 
-static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
-	__acquires(lock)
-{
-	rt_spin_lock(lock);
-}
-
 #define spin_lock_irqsave(lock, flags)			 \
 	do {						 \
 		typecheck(unsigned long, flags);	 \
@@ -128,12 +122,6 @@ static __always_inline void spin_unlock_
 	rt_spin_unlock(lock);
 }
 
-static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
-	__releases(lock)
-{
-	rt_spin_unlock(lock);
-}
-
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,
 						   unsigned long flags)
 	__releases(lock)
@@ -143,12 +131,6 @@ static __always_inline void spin_unlock_
 
 #define spin_trylock(lock)	rt_spin_trylock(lock)
 
-static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
-	__cond_acquires(true, lock)
-{
-	return rt_spin_trylock(lock);
-}
-
 #define spin_trylock_bh(lock)	rt_spin_trylock_bh(lock)
 
 #define spin_trylock_irq(lock)	rt_spin_trylock(lock)
kernel/locking/spinlock.c +0 -31
@@ -129,21 +129,6 @@ static void __lockfunc __raw_##op##_lock
  */
 BUILD_LOCK_OPS(spin, raw_spinlock, __acquires);
 
-/* No rwlock_t variants for now, so just build this function by hand */
-static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
-{
-	for (;;) {
-		preempt_disable();
-		local_interrupt_disable();
-		if (likely(do_raw_spin_trylock(lock)))
-			break;
-		local_interrupt_enable();
-		preempt_enable();
-
-		arch_spin_relax(&lock->raw_lock);
-	}
-}
-
 #ifndef CONFIG_PREEMPT_RT
 BUILD_LOCK_OPS(read, rwlock, __acquires_shared);
 BUILD_LOCK_OPS(write, rwlock, __acquires);
@@ -191,14 +176,6 @@ noinline void __lockfunc _raw_spin_lock_
 EXPORT_SYMBOL(_raw_spin_lock_irq);
 #endif
 
-#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ
-noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
-{
-	__raw_spin_lock_irq_disable(lock);
-}
-EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable);
-#endif
-
 #ifndef CONFIG_INLINE_SPIN_LOCK_BH
 noinline void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock)
 {
@@ -231,14 +208,6 @@ noinline void __lockfunc _raw_spin_unloc
 EXPORT_SYMBOL(_raw_spin_unlock_irq);
 #endif
 
-#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ
-noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
-{
-	__raw_spin_unlock_irq_enable(lock);
-}
-EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable);
-#endif
-
 #ifndef CONFIG_INLINE_SPIN_UNLOCK_BH
 noinline void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock)
 {
kernel/softirq.c +0 -15
@@ -9,7 +9,6 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
-#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
 #include <linux/export.h>
 #include <linux/kernel_stat.h>
 #include <linux/interrupt.h>
@@ -89,20 +88,6 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabl
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
-DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state);
-
-void _local_interrupt_save_state(unsigned long flags)
-{
-	__local_interrupt_save_state(flags);
-}
-EXPORT_SYMBOL(_local_interrupt_save_state);
-
-void _local_interrupt_enable(void)
-{
-	__local_interrupt_enable();
-}
-EXPORT_SYMBOL(_local_interrupt_enable);
-
 #ifndef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS
 /*
  * Any 32bit architecture that still cares about performance should
rust/helpers/interrupt.c +20 -1
@@ -1,6 +1,25 @@
 // SPDX-License-Identifier: GPL-2.0
 
-#include <linux/spinlock.h>
+#include <linux/export.h>
+#include <linux/percpu.h>
+
+#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
+#include "interrupt_rc.h"
+#include "spinlock.h"
+
+DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state);
+
+void _local_interrupt_save_state(unsigned long flags)
+{
+	__local_interrupt_save_state(flags);
+}
+EXPORT_SYMBOL(_local_interrupt_save_state);
+
+void _local_interrupt_enable(void)
+{
+	__local_interrupt_enable();
+}
+EXPORT_SYMBOL(_local_interrupt_enable);
 
 __rust_helper void rust_helper_local_interrupt_disable(void)
 {
rust/helpers/interrupt_rc.h new file +68 -0

68 lines not on this page · raw message

rust/helpers/spinlock.c +38 -1
@@ -1,6 +1,43 @@
 // SPDX-License-Identifier: GPL-2.0
 
-#include <linux/spinlock.h>
+#include <linux/export.h>
+#include "spinlock.h"
+
+#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC)
+/* The __lock_function inlines are taken from "spinlock.h" */
+#else
+
+/* No rwlock_t variants for now, so just build this function by hand */
+static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	for (;;) {
+		preempt_disable();
+		local_interrupt_disable();
+		if (likely(do_raw_spin_trylock(lock)))
+			break;
+		local_interrupt_enable();
+		preempt_enable();
+
+		arch_spin_relax(&lock->raw_lock);
+	}
+}
+#endif
+
+#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ
+noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	__raw_spin_lock_irq_disable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable);
+#endif
+
+#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ
+noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
+{
+	__raw_spin_unlock_irq_enable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable);
+#endif
 
 __rust_helper void rust_helper___spin_lock_init(spinlock_t *lock,
 						const char *name,
rust/helpers/spinlock.h new file +114 -0

114 lines not on this page · raw message

author maintainer
#62 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 10 lines · Thomas Gleixner
Quoted 9 lines · Boqun Feng
Quoted 8 lines · Boqun Feng

And while we are at, we can just introduce a raw_local_irq_restore_auto() (definitely needs a better name), which doesn't need a cnt:

static __always_inline void __raw_local_irq_restore_auto(void)

{
	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK));
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
	if (!(__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) & HARDIRQ_DISABLE_MASK))
		arch_local_irq_enable();
}

Can we just make the thing work in the first place?

Yes, for sure.

Quoted 7 lines
Quoted 1 line
Quoted 1 line

And we can slowly convert irq_restore() users to use it?

Somewhere down the road.

Quoted 3 lines

I decided to use local_irq_resume(), not sure whether it's a good name either...

Quoted 9 lines
Quoted 5 lines

If we are OK to not fully revert on commit e901c1510e24 ("irq,spin_lock:
Add counted interrupt disabling/enabling"), I think the following will resolve the 0day built errors on your irqflags branch (I rebased onto tip/locking/urgent with rest of your series on irqflags branch). A few things to notice:

The zero day failure is not due to that. It's because I reverted that refcounted muck as well in my git tree.

Right, that's why I said "if we are OK to not fully revert commit e901c1510e24 ...", but your movestuff.patch should works as well.

Quoted 7 lines

I just rebased the series on top of tip locking/urgent, which only has the irq,spinlock revert and your patches on top.

Quoted 3 lines

* I haven't found a way that we can do a local_irq_resume() when !CONFIG_PREEMPT_COUNT_IRQFLAGS, and if we cannot, it's going to make part of Rust code depends on CONFIG_PREEMPT_COUNT_IRQFLAGS=y

Quoted 6 lines

That's a good thing as it might make people actually get their act together. And you can work around that in interrupt_rc.h itself.

Can you please stop bouncing around like a rubber ball and take your time?

Apologies. I was just trying to help (and explore myself) the integration of PREEMPT_COUNT_IRQFLAGS with local_interrupt_disable(), it's merely some food for thoughts...

Quoted 6 lines

First of all I want to move interrupt_rc.h and the whole spinlock muck into rust/helpers/ now. Why?

Simply because it is Rust only and we don't want to expose any of this stuff to random driver writers. See attached patch.

Sure. I will give the movestuff.patch some test.

Quoted 11 lines

I've rebased my devel branch on top of tip/locking/urgent and applied that patch so the robots can have their field day.

The actual PREEMPT_COUNT_IRQFLAGS thing will be 7.4 material obviously and as this is confined to Rust then it's trivial enough to work around it locally without exposing more stuff. See tiny delta patch below.

So the only side effect of that is that the Rust implementation will not be fully integrated into the preempt count magic, but it should just work, no?

Right, that works. It's similar to the "alternatively" approach I mentioned here [1].

Quoted 7 lines

And when an architecture supports the real thing then it gets all the benefits with bells and whistels.

I really want to get the PREEMPT_COUNT_IRQFLAGS design right first and then we can think about simplifications and cleanups and remove the whole Rust magic once all architectures which support Rust play along.

Yeah, that's a better plan.

[1]: https://lore.kernel.org/lkml/[email protected]/

Regards,
Boqun

Quoted 5 lines

Thanks,

tglx

---

[...]

maintainer
#63 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 5 lines · Boqun Feng
Quoted 4 lines · Thomas Gleixner

The actual PREEMPT_COUNT_IRQFLAGS thing will be 7.4 material obviously and as this is confined to Rust then it's trivial enough to work around it locally without exposing more stuff. See tiny delta patch below.

Quoted 7 lines
Quoted 4 lines

So the only side effect of that is that the Rust implementation will not be fully integrated into the preempt count magic, but it should just work, no?

Right, that works. It's similar to the "alternatively" approach I mentioned here [1].

But thinking more about it. It actually just works with the preempt count bits independent of PREEMPT_COUNT_IRQFLAGS.

For PREEMPT_COUNT_IRQFLAGS=n, Rust is the only one using it.

For PREEMPT_COUNT_IRQFLAGS=y, the Rust part integrates with the core implementation. And once all Rust supporting architectures are enabling PREEMPT_COUNT_IRQFLAGS the Rust extra magic goes away.

No?

Thanks,

tglx

author maintainer
#64 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 17 lines · Thomas Gleixner
Quoted 5 lines · Boqun Feng
Quoted 4 lines · Thomas Gleixner

The actual PREEMPT_COUNT_IRQFLAGS thing will be 7.4 material obviously and as this is confined to Rust then it's trivial enough to work around it locally without exposing more stuff. See tiny delta patch below.

Quoted 7 lines
Quoted 4 lines

So the only side effect of that is that the Rust implementation will not be fully integrated into the preempt count magic, but it should just work, no?

Right, that works. It's similar to the "alternatively" approach I mentioned here [1].

But thinking more about it. It actually just works with the preempt count bits independent of PREEMPT_COUNT_IRQFLAGS.

Quoted 8 lines

For PREEMPT_COUNT_IRQFLAGS=n, Rust is the only one using it.

For PREEMPT_COUNT_IRQFLAGS=y, the Rust part integrates with the core implementation. And once all Rust supporting architectures are enabling PREEMPT_COUNT_IRQFLAGS the Rust extra magic goes away.

No?

Right, that's why I thought fully revert on commit e901c1510e24 might not be needed.

To me, the only "magic" part when PREEMPT_COUNT_IRQFLAGS=n are 1) preempt_count() is inconsistent on HARDIRQ_DISABLE_MASK about interrupt disabling as you pointed out earlier and 2) local_interrupt_{en,dis}able() has to save the current irq disabling state/flag to work with local_irq_*(). PREEMPT_COUNT_IRQFLAGS=y resolves both of them.

And if PREEMPT_COUNT_IRQFLAGS=y is the future (i.e. it'll be always y), then we will likely have local_interrupt_{en,dis}able() (or a different name) as a general API for everyone. So the API (and its semantics) is not Rust-specific considering the future direction. Hence previously I said that we can move them to Rust only but seems a bit unnecessary to me.

Hope this makes sense.

Regards,
Boqun

Quoted 5 lines

Thanks,

tglx

maintainer
#65 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 3 lines · Thomas Gleixner

Right, that's why I thought fully revert on commit e901c1510e24 might not be needed.

It's gone already and as I told you before the reason is that the issues were not restricted to the ordering parts v.s. count/hardware and the fallout in the softirq code. The whole issue with nested unlock/lock inside a guard are not solved by reordering local_interrupt_disable(). Not to talk about the lack of proper debug features for it.

Quoted 6 lines

And if PREEMPT_COUNT_IRQFLAGS=y is the future (i.e. it'll be always y), then we will likely have local_interrupt_{en,dis}able() (or a different name) as a general API for everyone. So the API (and its semantics) is not Rust-specific considering the future direction. Hence previously I said that we can move them to Rust only but seems a bit unnecessary to me.

No. We need a proper strategy to pull that off and not exposing the functionality and the name right now outside of Rust makes that way simpler. Changing Rust is one thing, chasing down a pile of random use cases which crept in _before_ the design and strategy is settled is a completely different story.

Thanks,

tglx

author maintainer
#66 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 17 lines · Thomas Gleixner
Quoted 3 lines · Thomas Gleixner

Right, that's why I thought fully revert on commit e901c1510e24 might not be needed.

It's gone already and as I told you before the reason is that the issues were not restricted to the ordering parts v.s. count/hardware and the fallout in the softirq code. The whole issue with nested unlock/lock inside a guard are not solved by reordering local_interrupt_disable(). Not to talk about the lack of proper debug features for it.

Quoted 6 lines

And if PREEMPT_COUNT_IRQFLAGS=y is the future (i.e. it'll be always y), then we will likely have local_interrupt_{en,dis}able() (or a different name) as a general API for everyone. So the API (and its semantics) is not Rust-specific considering the future direction. Hence previously I said that we can move them to Rust only but seems a bit unnecessary to me.

Quoted 6 lines

No. We need a proper strategy to pull that off and not exposing the functionality and the name right now outside of Rust makes that way simpler. Changing Rust is one thing, chasing down a pile of random use cases which crept in _before_ the design and strategy is settled is a completely different story.

Fair enough.

Not trying to keep interrupt_{en,dis}able() from moving, but after some thoughts, I think I figured out a few debugs we can add for PREEMPT_COUNT_IRQFLAGS=n case, things we want to avoid:

* interrupt_disable(); irq_disable(); irq_enable(); interrupt_enable();

* interrupt_disable(); irq_enable(); irq_disable(); interrupt_enable();

* irq_save(flags); interrupt_disable(); irq_restore(flags); interrupt_enable();

on top of your current work, we can do the following when PREEMPT_COUNT_IRQFLAGS=n. It'll help find a few random use cases that break. (even when interrupt_disable() are Rust-only, Rust code can still call a function which does irq_enable(); irq_disable(); while in a interrupt_disable() critical section, so it makes sense to catch them).

Thoughts?

Regards,
Boqun

---------------------->8

include/linux/irqflags.h +20 -0
index dd55786768d1..a41d188dceef 100644
@@ -241,6 +241,14 @@ static __always_inline void raw_safe_halt(void)
 
 static __always_inline void raw_local_irq_disable(void)
 {
+	/*
+	 * Assuming local_irq_{en,dis}able() always paired, then
+	 * local_irq_disable() should not be used inside an
+	 * local_interrupt_disable() critical section. Because the paired
+	 * local_irq_enable() would enable the interrupt inside a
+	 * local_interrupt_disable() critical section.
+	 */
+	debug_assert(!(preempt_count() & HARDIRQ_DISABLE_MASK));
 	arch_local_irq_disable();
 }
 
@@ -251,6 +259,12 @@ static __always_inline void raw_force_local_irq_disable(void)
 
 static __always_inline void raw_local_irq_enable(void)
 {
+	/*
+	 * local_irq_enable() should not be called inside a
+	 * local_interrupt_disable() critical section, but it would enable the
+	 * interrupt unexpectedly.
+	 */
+	debug_assert(!(preempt_count() & HARDIRQ_DISABLE_MASK));
 	arch_local_irq_enable();
 }
 
@@ -261,6 +275,12 @@ static __always_inline unsigned long __raw_local_irq_save(void)
 
 static __always_inline void __raw_local_irq_restore(unsigned long flags)
 {
+	/*
+	 * local_irq_restore() should not enable interrupt unexpectedly inside
+	 * a local_interrupt_disable() critical section
+	 */
+	debug_assert(!(preempt_count() & HARDIRQ_DISABLE_MASK) ||
+		     arch_irqs_disabled_flags(flags));
 	arch_local_irq_restore(flags);
 }
maintainer
#67 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

include/linux/irqflags.h static __always_inline void raw_safe_halt(void) · line 252

Quoted 12 lines · Boqun Feng

On Tue, Sep 01, 2026 at 03:43:47PM +0200, Thomas Gleixner wrote:
static __always_inline void raw_local_irq_disable(void) {

include/linux/irqflags.h +8 -0
+	/*
+	 * Assuming local_irq_{en,dis}able() always paired, then
+	 * local_irq_disable() should not be used inside an
+	 * local_interrupt_disable() critical section. Because the paired
+	 * local_irq_enable() would enable the interrupt inside a
+	 * local_interrupt_disable() critical section.
+	 */
+	debug_assert(!(preempt_count() & HARDIRQ_DISABLE_MASK));
 	arch_local_irq_disable();

The problem with pure debug_assert()s is that the damage is already done. I learned that the hard way when I was chasing the last issue in the #UD handler of x86 that this starts to recurse up to the point where the system falls apart.

The modified version of this debug stuff in my devel branch does actual fixups to prevent the subsequent damage. It's a hack and I did not come around yet to make it actually less horrible.

maintainer
#68 lore raw

PATCH 1/1 irq: Move local_irq_enable/disable() into Rust

In reply to Thomas Gleixner

After reverting the spinlock conversion and a lengthy discussion it's the best to confine the reference counted interrupt disable/enable mechanism to Rust which is the only user.

This should become the new norm, but that needs more thoughts and cleaning up the confined usage in Rust at some point is way simpler than chasing random places which adopt it in the meanwhile.

---
 include/linux/interrupt_rc.h         |   82 -------------------------
 include/linux/spinlock.h             |   23 -------
 include/linux/spinlock_api_smp.h     |   41 ------------
 include/linux/spinlock_api_up.h      |   15 ----
 include/linux/spinlock_rt.h          |   18 -----
 kernel/irq/refcount_interrupt_test.c |    2 
 kernel/locking/spinlock.c            |   31 ---------
 kernel/softirq.c                     |   15 ----
 rust/helpers/interrupt.c             |   21 ++++++
 rust/helpers/interrupt_rc.h          |   68 ++++++++++++++++++++
 rust/helpers/spinlock.c              |   39 +++++++++++
 rust/helpers/spinlock.h              |  114 +++++++++++++++++++++++++++++++++++
 12 files changed, 241 insertions(+), 228 deletions(-)
include/linux/interrupt_rc.h deleted +0 -82
@@ -1,82 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef __LINUX_INTERRUPT_RC_H
-#define __LINUX_INTERRUPT_RC_H
-
-/*
- * include/linux/interrupt_rc.h - refcounted local processor interrupt
- * management.
- *
- * Since the implementation of this API currently depends on
- * local_irq_save()/local_irq_restore(), we split this into its own header to
- * make it easier to include without hitting circular header dependencies.
- */
-
-#include <linux/irqflags.h>
-#include <linux/preempt.h>
-#include <linux/processor.h>
-#include <linux/smp.h>
-
-#ifndef MODULE
-/* Per-CPU interrupt disabling state for local_interrupt_{disable,enable}(). */
-DECLARE_PER_CPU(unsigned long, local_interrupt_disable_state);
-
-static __always_inline void __local_interrupt_disable(void)
-{
-	unsigned long flags;
-
-	local_irq_save(flags);
-	raw_cpu_write(local_interrupt_disable_state, flags);
-}
-
-static __always_inline void __local_interrupt_enable(void)
-{
-	unsigned long flags = raw_cpu_read(local_interrupt_disable_state);
-
-	local_irq_restore(flags);
-}
-
-#ifndef INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
-static __always_inline void _local_interrupt_disable(void)
-{
-	__local_interrupt_disable();
-}
-
-static __always_inline void _local_interrupt_enable(void)
-{
-	__local_interrupt_enable();
-}
-#else
-extern void _local_interrupt_disable(void);
-extern void _local_interrupt_enable(void);
-#endif
-
-#else /* !MODULE */
-extern void _local_interrupt_disable(void);
-extern void _local_interrupt_enable(void);
-#endif /* !MODULE */
-
-static inline void local_interrupt_disable(void)
-{
-	int new_count;
-
-	WARN_ON_ONCE(in_nmi());
-
-	new_count = hardirq_disable_enter();
-
-	/* Interrupts can happen here, but it's OK, see __irq_exit_rcu(). */
-
-	if ((new_count & HARDIRQ_DISABLE_MASK) == HARDIRQ_DISABLE_OFFSET)
-		_local_interrupt_disable();
-}
-
-static inline void local_interrupt_enable(void)
-{
-	int new_count;
-
-	new_count = hardirq_disable_exit();
-
-	if ((new_count & HARDIRQ_DISABLE_MASK) == 0)
-		_local_interrupt_enable();
-}
-
-#endif /* !__LINUX_INTERRUPT_RC_H */
include/linux/spinlock.h +0 -23
@@ -57,7 +57,6 @@
 #include <linux/linkage.h>
 #include <linux/compiler.h>
 #include <linux/irqflags.h>
-#include <linux/interrupt_rc.h>
 #include <linux/thread_info.h>
 #include <linux/stringify.h>
 #include <linux/bottom_half.h>
@@ -274,11 +273,9 @@ static inline void do_raw_spin_unlock(ra
 #endif
 
 #define raw_spin_lock_irq(lock)		_raw_spin_lock_irq(lock)
-#define raw_spin_lock_irq_disable(lock)	_raw_spin_lock_irq_disable(lock)
 #define raw_spin_lock_bh(lock)		_raw_spin_lock_bh(lock)
 #define raw_spin_unlock(lock)		_raw_spin_unlock(lock)
 #define raw_spin_unlock_irq(lock)	_raw_spin_unlock_irq(lock)
-#define raw_spin_unlock_irq_enable(lock)	_raw_spin_unlock_irq_enable(lock)
 
 #define raw_spin_unlock_irqrestore(lock, flags)		\
 	do {							\
@@ -293,8 +290,6 @@ static inline void do_raw_spin_unlock(ra
 
 #define raw_spin_trylock_irqsave(lock, flags) _raw_spin_trylock_irqsave(lock, &(flags))
 
-#define raw_spin_trylock_irq_disable(lock)	_raw_spin_trylock_irq_disable(lock)
-
 #ifndef CONFIG_PREEMPT_RT
 /* Include rwlock functions for !RT */
 #include <linux/rwlock.h>
@@ -377,12 +372,6 @@ static __always_inline void spin_lock_ir
 	raw_spin_lock_irq(&lock->rlock);
 }
 
-static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
-	__acquires(lock) __no_context_analysis
-{
-	raw_spin_lock_irq_disable(&lock->rlock);
-}
-
 #define spin_lock_irqsave(lock, flags)				\
 do {								\
 	raw_spin_lock_irqsave(spinlock_check(lock), flags);	\
@@ -413,12 +402,6 @@ static __always_inline void spin_unlock_
 	raw_spin_unlock_irq(&lock->rlock);
 }
 
-static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
-	__releases(lock) __no_context_analysis
-{
-	raw_spin_unlock_irq_enable(&lock->rlock);
-}
-
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags)
 	__releases(lock) __no_context_analysis
 {
@@ -444,12 +427,6 @@ static __always_inline bool _spin_tryloc
 }
 #define spin_trylock_irqsave(lock, flags) _spin_trylock_irqsave(lock, &(flags))
 
-static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
-	__cond_acquires(true, lock) __no_context_analysis
-{
-	return raw_spin_trylock_irq_disable(&lock->rlock);
-}
-
 /**
  * spin_is_locked() - Check whether a spinlock is locked.
  * @lock: Pointer to the spinlock.
include/linux/spinlock_api_smp.h +0 -41

83 lines not on this page · raw message

include/linux/spinlock_api_up.h +0 -15
@@ -42,9 +42,6 @@
 #define __LOCK_IRQSAVE(lock, flags, ...) \
   do { local_irq_save(flags); __LOCK(lock, ##__VA_ARGS__); } while (0)
 
-#define __LOCK_IRQ_DISABLE(lock, ...) \
-  do { local_interrupt_disable(); __LOCK(lock, ##__VA_ARGS__); } while (0)
-
 #define ___UNLOCK_(lock) \
   do { __release(lock); (void)(lock); } while (0)
 
@@ -64,9 +61,6 @@
 #define __UNLOCK_IRQRESTORE(lock, flags, ...) \
   do { local_irq_restore(flags); __UNLOCK(lock, ##__VA_ARGS__); } while (0)
 
-#define __UNLOCK_IRQ_ENABLE(lock, ...) \
-  do { __UNLOCK(lock, ##__VA_ARGS__); local_interrupt_enable(); } while (0)
-
 #define _raw_spin_lock(lock)			__LOCK(lock)
 #define _raw_spin_lock_nested(lock, subclass)	__LOCK(lock)
 #define _raw_read_lock(lock)			__LOCK(lock, shared)
@@ -76,7 +70,6 @@
 #define _raw_read_lock_bh(lock)			__LOCK_BH(lock, shared)
 #define _raw_write_lock_bh(lock)		__LOCK_BH(lock)
 #define _raw_spin_lock_irq(lock)		__LOCK_IRQ(lock)
-#define _raw_spin_lock_irq_disable(lock)	__LOCK_IRQ_DISABLE(lock)
 #define _raw_read_lock_irq(lock)		__LOCK_IRQ(lock, shared)
 #define _raw_write_lock_irq(lock)		__LOCK_IRQ(lock)
 #define _raw_spin_lock_irqsave(lock, flags)	__LOCK_IRQSAVE(lock, flags)
@@ -104,13 +97,6 @@ static __always_inline int _raw_spin_try
 	return 1;
 }
 
-static __always_inline int _raw_spin_trylock_irq_disable(raw_spinlock_t *lock)
-	__cond_acquires(true, lock)
-{
-	__LOCK_IRQ_DISABLE(lock);
-	return 1;
-}
-
 static __always_inline int _raw_spin_trylock_irqsave(raw_spinlock_t *lock, unsigned long *flags)
 	__cond_acquires(true, lock)
 {
@@ -146,7 +132,6 @@ static __always_inline int _raw_write_tr
 #define _raw_write_unlock_bh(lock)		__UNLOCK_BH(lock)
 #define _raw_read_unlock_bh(lock)		__UNLOCK_BH(lock, shared)
 #define _raw_spin_unlock_irq(lock)		__UNLOCK_IRQ(lock)
-#define _raw_spin_unlock_irq_enable(lock)	__UNLOCK_IRQ_ENABLE(lock)
 #define _raw_read_unlock_irq(lock)		__UNLOCK_IRQ(lock, shared)
 #define _raw_write_unlock_irq(lock)		__UNLOCK_IRQ(lock)
 #define _raw_spin_unlock_irqrestore(lock, flags) \
include/linux/spinlock_rt.h +0 -18
@@ -96,12 +96,6 @@ static __always_inline void spin_lock_ir
 	rt_spin_lock(lock);
 }
 
-static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
-	__acquires(lock)
-{
-	rt_spin_lock(lock);
-}
-
 #define spin_lock_irqsave(lock, flags)			 \
 	do {						 \
 		typecheck(unsigned long, flags);	 \
@@ -128,12 +122,6 @@ static __always_inline void spin_unlock_
 	rt_spin_unlock(lock);
 }
 
-static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
-	__releases(lock)
-{
-	rt_spin_unlock(lock);
-}
-
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,
 						   unsigned long flags)
 	__releases(lock)
@@ -143,12 +131,6 @@ static __always_inline void spin_unlock_
 
 #define spin_trylock(lock)	rt_spin_trylock(lock)
 
-static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
-	__cond_acquires(true, lock)
-{
-	return rt_spin_trylock(lock);
-}
-
 #define spin_trylock_bh(lock)	rt_spin_trylock_bh(lock)
 
 #define spin_trylock_irq(lock)	rt_spin_trylock(lock)
kernel/irq/refcount_interrupt_test.c +1 -1
@@ -4,7 +4,7 @@
  */
 
 #include <kunit/test.h>
-#include <linux/interrupt_rc.h>
+#include <../../rust/helpers/interrupt_rc.h>
 
 #define TEST_IRQ_ON() KUNIT_EXPECT_FALSE(test, irqs_disabled())
 #define TEST_IRQ_OFF() KUNIT_EXPECT_TRUE(test, irqs_disabled())
kernel/locking/spinlock.c +0 -31
@@ -129,21 +129,6 @@ static void __lockfunc __raw_##op##_lock
  */
 BUILD_LOCK_OPS(spin, raw_spinlock, __acquires);
 
-/* No rwlock_t variants for now, so just build this function by hand */
-static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
-{
-	for (;;) {
-		preempt_disable();
-		local_interrupt_disable();
-		if (likely(do_raw_spin_trylock(lock)))
-			break;
-		local_interrupt_enable();
-		preempt_enable();
-
-		arch_spin_relax(&lock->raw_lock);
-	}
-}
-
 #ifndef CONFIG_PREEMPT_RT
 BUILD_LOCK_OPS(read, rwlock, __acquires_shared);
 BUILD_LOCK_OPS(write, rwlock, __acquires);
@@ -191,14 +176,6 @@ noinline void __lockfunc _raw_spin_lock_
 EXPORT_SYMBOL(_raw_spin_lock_irq);
 #endif
 
-#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ
-noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
-{
-	__raw_spin_lock_irq_disable(lock);
-}
-EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable);
-#endif
-
 #ifndef CONFIG_INLINE_SPIN_LOCK_BH
 noinline void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock)
 {
@@ -231,14 +208,6 @@ noinline void __lockfunc _raw_spin_unloc
 EXPORT_SYMBOL(_raw_spin_unlock_irq);
 #endif
 
-#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ
-noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
-{
-	__raw_spin_unlock_irq_enable(lock);
-}
-EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable);
-#endif
-
 #ifndef CONFIG_INLINE_SPIN_UNLOCK_BH
 noinline void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock)
 {
kernel/softirq.c +0 -15
@@ -9,7 +9,6 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
-#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
 #include <linux/export.h>
 #include <linux/kernel_stat.h>
 #include <linux/interrupt.h>
@@ -89,20 +88,6 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabl
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
-DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state);
-
-void _local_interrupt_disable(void)
-{
-	__local_interrupt_disable();
-}
-EXPORT_SYMBOL(_local_interrupt_disable);
-
-void _local_interrupt_enable(void)
-{
-	__local_interrupt_enable();
-}
-EXPORT_SYMBOL(_local_interrupt_enable);
-
 #ifndef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS
 /*
  * Any 32bit architecture that still cares about performance should
rust/helpers/interrupt.c +20 -1
@@ -1,6 +1,25 @@
 // SPDX-License-Identifier: GPL-2.0
 
-#include <linux/spinlock.h>
+#include <linux/export.h>
+#include <linux/percpu.h>
+
+#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
+#include "interrupt_rc.h"
+#include "spinlock.h"
+
+DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state);
+
+void _local_interrupt_save_state(unsigned long flags)
+{
+	__local_interrupt_save_state(flags);
+}
+EXPORT_SYMBOL(_local_interrupt_save_state);
+
+void _local_interrupt_enable(void)
+{
+	__local_interrupt_enable();
+}
+EXPORT_SYMBOL(_local_interrupt_enable);
 
 __rust_helper void rust_helper_local_interrupt_disable(void)
 {
rust/helpers/interrupt_rc.h new file +68 -0

68 lines not on this page · raw message

rust/helpers/spinlock.c +38 -1
@@ -1,6 +1,43 @@
 // SPDX-License-Identifier: GPL-2.0
 
-#include <linux/spinlock.h>
+#include <linux/export.h>
+#include "spinlock.h"
+
+#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC)
+/* The __lock_function inlines are taken from "spinlock.h" */
+#else
+
+/* No rwlock_t variants for now, so just build this function by hand */
+static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	for (;;) {
+		preempt_disable();
+		local_interrupt_disable();
+		if (likely(do_raw_spin_trylock(lock)))
+			break;
+		local_interrupt_enable();
+		preempt_enable();
+
+		arch_spin_relax(&lock->raw_lock);
+	}
+}
+#endif
+
+#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ
+noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	__raw_spin_lock_irq_disable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable);
+#endif
+
+#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ
+noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
+{
+	__raw_spin_unlock_irq_enable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable);
+#endif
 
 __rust_helper void rust_helper___spin_lock_init(spinlock_t *lock,
 						const char *name,
rust/helpers/spinlock.h new file +114 -0

114 lines not on this page · raw message

maintainer
#69 lore raw

PATCH 1/1 irq: Move local_irq_enable/disable() into Rust

After reverting the spinlock conversion and a lengthy discussion it's the best to confine the reference counted interrupt disable/enable mechanism to Rust which is the only user.

This should become the new norm, but that needs more thoughts and cleaning up the confined usage in Rust at some point is way simpler than chasing random places which adopt it in the meanwhile.

---
Resend because I fatfingered the Subject line ... Sorry for the noise in case you got the original busted one.

Applies against tip locking/urgent

---
 include/linux/spinlock.h             |   23 -------
 include/linux/spinlock_api_smp.h     |   41 ------------
 include/linux/spinlock_api_up.h      |   15 ----
 include/linux/spinlock_rt.h          |   18 -----
 kernel/irq/refcount_interrupt_test.c |    2 
 kernel/locking/spinlock.c            |   31 ---------
 kernel/softirq.c                     |   15 ----
 rust/helpers/interrupt.c             |   21 ++++++
 rust/helpers/interrupt_rc.h          |   68 ++++++++++++++++++++
 rust/helpers/spinlock.c              |   39 +++++++++++
 rust/helpers/spinlock.h              |  114 +++++++++++++++++++++++++++++++++++
 11 files changed, 241 insertions(+), 146 deletions(-)
include/linux/spinlock.h +0 -23
@@ -57,7 +57,6 @@
 #include <linux/linkage.h>
 #include <linux/compiler.h>
 #include <linux/irqflags.h>
-#include <linux/interrupt_rc.h>
 #include <linux/thread_info.h>
 #include <linux/stringify.h>
 #include <linux/bottom_half.h>
@@ -274,11 +273,9 @@ static inline void do_raw_spin_unlock(ra
 #endif
 
 #define raw_spin_lock_irq(lock)		_raw_spin_lock_irq(lock)
-#define raw_spin_lock_irq_disable(lock)	_raw_spin_lock_irq_disable(lock)
 #define raw_spin_lock_bh(lock)		_raw_spin_lock_bh(lock)
 #define raw_spin_unlock(lock)		_raw_spin_unlock(lock)
 #define raw_spin_unlock_irq(lock)	_raw_spin_unlock_irq(lock)
-#define raw_spin_unlock_irq_enable(lock)	_raw_spin_unlock_irq_enable(lock)
 
 #define raw_spin_unlock_irqrestore(lock, flags)		\
 	do {							\
@@ -293,8 +290,6 @@ static inline void do_raw_spin_unlock(ra
 
 #define raw_spin_trylock_irqsave(lock, flags) _raw_spin_trylock_irqsave(lock, &(flags))
 
-#define raw_spin_trylock_irq_disable(lock)	_raw_spin_trylock_irq_disable(lock)
-
 #ifndef CONFIG_PREEMPT_RT
 /* Include rwlock functions for !RT */
 #include <linux/rwlock.h>
@@ -377,12 +372,6 @@ static __always_inline void spin_lock_ir
 	raw_spin_lock_irq(&lock->rlock);
 }
 
-static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
-	__acquires(lock) __no_context_analysis
-{
-	raw_spin_lock_irq_disable(&lock->rlock);
-}
-
 #define spin_lock_irqsave(lock, flags)				\
 do {								\
 	raw_spin_lock_irqsave(spinlock_check(lock), flags);	\
@@ -413,12 +402,6 @@ static __always_inline void spin_unlock_
 	raw_spin_unlock_irq(&lock->rlock);
 }
 
-static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
-	__releases(lock) __no_context_analysis
-{
-	raw_spin_unlock_irq_enable(&lock->rlock);
-}
-
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags)
 	__releases(lock) __no_context_analysis
 {
@@ -444,12 +427,6 @@ static __always_inline bool _spin_tryloc
 }
 #define spin_trylock_irqsave(lock, flags) _spin_trylock_irqsave(lock, &(flags))
 
-static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
-	__cond_acquires(true, lock) __no_context_analysis
-{
-	return raw_spin_trylock_irq_disable(&lock->rlock);
-}
-
 /**
  * spin_is_locked() - Check whether a spinlock is locked.
  * @lock: Pointer to the spinlock.
include/linux/spinlock_api_smp.h +0 -41

83 lines not on this page · raw message

include/linux/spinlock_api_up.h +0 -15
@@ -42,9 +42,6 @@
 #define __LOCK_IRQSAVE(lock, flags, ...) \
   do { local_irq_save(flags); __LOCK(lock, ##__VA_ARGS__); } while (0)
 
-#define __LOCK_IRQ_DISABLE(lock, ...) \
-  do { local_interrupt_disable(); __LOCK(lock, ##__VA_ARGS__); } while (0)
-
 #define ___UNLOCK_(lock) \
   do { __release(lock); (void)(lock); } while (0)
 
@@ -64,9 +61,6 @@
 #define __UNLOCK_IRQRESTORE(lock, flags, ...) \
   do { local_irq_restore(flags); __UNLOCK(lock, ##__VA_ARGS__); } while (0)
 
-#define __UNLOCK_IRQ_ENABLE(lock, ...) \
-  do { __UNLOCK(lock, ##__VA_ARGS__); local_interrupt_enable(); } while (0)
-
 #define _raw_spin_lock(lock)			__LOCK(lock)
 #define _raw_spin_lock_nested(lock, subclass)	__LOCK(lock)
 #define _raw_read_lock(lock)			__LOCK(lock, shared)
@@ -76,7 +70,6 @@
 #define _raw_read_lock_bh(lock)			__LOCK_BH(lock, shared)
 #define _raw_write_lock_bh(lock)		__LOCK_BH(lock)
 #define _raw_spin_lock_irq(lock)		__LOCK_IRQ(lock)
-#define _raw_spin_lock_irq_disable(lock)	__LOCK_IRQ_DISABLE(lock)
 #define _raw_read_lock_irq(lock)		__LOCK_IRQ(lock, shared)
 #define _raw_write_lock_irq(lock)		__LOCK_IRQ(lock)
 #define _raw_spin_lock_irqsave(lock, flags)	__LOCK_IRQSAVE(lock, flags)
@@ -104,13 +97,6 @@ static __always_inline int _raw_spin_try
 	return 1;
 }
 
-static __always_inline int _raw_spin_trylock_irq_disable(raw_spinlock_t *lock)
-	__cond_acquires(true, lock)
-{
-	__LOCK_IRQ_DISABLE(lock);
-	return 1;
-}
-
 static __always_inline int _raw_spin_trylock_irqsave(raw_spinlock_t *lock, unsigned long *flags)
 	__cond_acquires(true, lock)
 {
@@ -146,7 +132,6 @@ static __always_inline int _raw_write_tr
 #define _raw_write_unlock_bh(lock)		__UNLOCK_BH(lock)
 #define _raw_read_unlock_bh(lock)		__UNLOCK_BH(lock, shared)
 #define _raw_spin_unlock_irq(lock)		__UNLOCK_IRQ(lock)
-#define _raw_spin_unlock_irq_enable(lock)	__UNLOCK_IRQ_ENABLE(lock)
 #define _raw_read_unlock_irq(lock)		__UNLOCK_IRQ(lock, shared)
 #define _raw_write_unlock_irq(lock)		__UNLOCK_IRQ(lock)
 #define _raw_spin_unlock_irqrestore(lock, flags) \
include/linux/spinlock_rt.h +0 -18
@@ -96,12 +96,6 @@ static __always_inline void spin_lock_ir
 	rt_spin_lock(lock);
 }
 
-static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
-	__acquires(lock)
-{
-	rt_spin_lock(lock);
-}
-
 #define spin_lock_irqsave(lock, flags)			 \
 	do {						 \
 		typecheck(unsigned long, flags);	 \
@@ -128,12 +122,6 @@ static __always_inline void spin_unlock_
 	rt_spin_unlock(lock);
 }
 
-static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
-	__releases(lock)
-{
-	rt_spin_unlock(lock);
-}
-
 static __always_inline void spin_unlock_irqrestore(spinlock_t *lock,
 						   unsigned long flags)
 	__releases(lock)
@@ -143,12 +131,6 @@ static __always_inline void spin_unlock_
 
 #define spin_trylock(lock)	rt_spin_trylock(lock)
 
-static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
-	__cond_acquires(true, lock)
-{
-	return rt_spin_trylock(lock);
-}
-
 #define spin_trylock_bh(lock)	rt_spin_trylock_bh(lock)
 
 #define spin_trylock_irq(lock)	rt_spin_trylock(lock)
kernel/irq/refcount_interrupt_test.c +1 -1
@@ -4,7 +4,7 @@
  */
 
 #include <kunit/test.h>
-#include <linux/interrupt_rc.h>
+#include <../../rust/helpers/interrupt_rc.h>
 
 #define TEST_IRQ_ON() KUNIT_EXPECT_FALSE(test, irqs_disabled())
 #define TEST_IRQ_OFF() KUNIT_EXPECT_TRUE(test, irqs_disabled())
kernel/locking/spinlock.c +0 -31
@@ -129,21 +129,6 @@ static void __lockfunc __raw_##op##_lock
  */
 BUILD_LOCK_OPS(spin, raw_spinlock, __acquires);
 
-/* No rwlock_t variants for now, so just build this function by hand */
-static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
-{
-	for (;;) {
-		preempt_disable();
-		local_interrupt_disable();
-		if (likely(do_raw_spin_trylock(lock)))
-			break;
-		local_interrupt_enable();
-		preempt_enable();
-
-		arch_spin_relax(&lock->raw_lock);
-	}
-}
-
 #ifndef CONFIG_PREEMPT_RT
 BUILD_LOCK_OPS(read, rwlock, __acquires_shared);
 BUILD_LOCK_OPS(write, rwlock, __acquires);
@@ -191,14 +176,6 @@ noinline void __lockfunc _raw_spin_lock_
 EXPORT_SYMBOL(_raw_spin_lock_irq);
 #endif
 
-#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ
-noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
-{
-	__raw_spin_lock_irq_disable(lock);
-}
-EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable);
-#endif
-
 #ifndef CONFIG_INLINE_SPIN_LOCK_BH
 noinline void __lockfunc _raw_spin_lock_bh(raw_spinlock_t *lock)
 {
@@ -231,14 +208,6 @@ noinline void __lockfunc _raw_spin_unloc
 EXPORT_SYMBOL(_raw_spin_unlock_irq);
 #endif
 
-#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ
-noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
-{
-	__raw_spin_unlock_irq_enable(lock);
-}
-EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable);
-#endif
-
 #ifndef CONFIG_INLINE_SPIN_UNLOCK_BH
 noinline void __lockfunc _raw_spin_unlock_bh(raw_spinlock_t *lock)
 {
kernel/softirq.c +0 -15
@@ -9,7 +9,6 @@
 
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
-#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
 #include <linux/export.h>
 #include <linux/kernel_stat.h>
 #include <linux/interrupt.h>
@@ -89,20 +88,6 @@ EXPORT_PER_CPU_SYMBOL_GPL(hardirqs_enabl
 EXPORT_PER_CPU_SYMBOL_GPL(hardirq_context);
 #endif
 
-DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state);
-
-void _local_interrupt_save_state(unsigned long flags)
-{
-	__local_interrupt_save_state(flags);
-}
-EXPORT_SYMBOL(_local_interrupt_save_state);
-
-void _local_interrupt_enable(void)
-{
-	__local_interrupt_enable();
-}
-EXPORT_SYMBOL(_local_interrupt_enable);
-
 #ifndef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS
 /*
  * Any 32bit architecture that still cares about performance should
rust/helpers/interrupt.c +20 -1
@@ -1,6 +1,25 @@
 // SPDX-License-Identifier: GPL-2.0
 
-#include <linux/spinlock.h>
+#include <linux/export.h>
+#include <linux/percpu.h>
+
+#define INSTANTIATE_EXPORTED_INTERRUPT_DISABLE
+#include "interrupt_rc.h"
+#include "spinlock.h"
+
+DEFINE_PER_CPU(unsigned long, local_interrupt_disable_state);
+
+void _local_interrupt_save_state(unsigned long flags)
+{
+	__local_interrupt_save_state(flags);
+}
+EXPORT_SYMBOL(_local_interrupt_save_state);
+
+void _local_interrupt_enable(void)
+{
+	__local_interrupt_enable();
+}
+EXPORT_SYMBOL(_local_interrupt_enable);
 
 __rust_helper void rust_helper_local_interrupt_disable(void)
 {
rust/helpers/interrupt_rc.h new file +68 -0

68 lines not on this page · raw message

rust/helpers/spinlock.c +38 -1
@@ -1,6 +1,43 @@
 // SPDX-License-Identifier: GPL-2.0
 
-#include <linux/spinlock.h>
+#include <linux/export.h>
+#include "spinlock.h"
+
+#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC)
+/* The __lock_function inlines are taken from "spinlock.h" */
+#else
+
+/* No rwlock_t variants for now, so just build this function by hand */
+static void __lockfunc __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	for (;;) {
+		preempt_disable();
+		local_interrupt_disable();
+		if (likely(do_raw_spin_trylock(lock)))
+			break;
+		local_interrupt_enable();
+		preempt_enable();
+
+		arch_spin_relax(&lock->raw_lock);
+	}
+}
+#endif
+
+#ifndef CONFIG_INLINE_SPIN_LOCK_IRQ
+noinline void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+{
+	__raw_spin_lock_irq_disable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_lock_irq_disable);
+#endif
+
+#ifndef CONFIG_INLINE_SPIN_UNLOCK_IRQ
+noinline void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
+{
+	__raw_spin_unlock_irq_enable(lock);
+}
+EXPORT_SYMBOL_GPL(_raw_spin_unlock_irq_enable);
+#endif
 
 __rust_helper void rust_helper___spin_lock_init(spinlock_t *lock,
 						const char *name,
rust/helpers/spinlock.h new file +114 -0

114 lines not on this page · raw message

author maintainer
#70 lore raw

Re: [PATCH] irq: Move local_irq_enable/disable() into Rust

[Cc Miguel, Lyude, Alice, Gary]

Quoted 8 lines · Thomas Gleixner

After reverting the spinlock conversion and a lengthy discussion it's the best to confine the reference counted interrupt disable/enable mechanism to Rust which is the only user.

This should become the new norm, but that needs more thoughts and cleaning up the confined usage in Rust at some point is way simpler than chasing random places which adopt it in the meanwhile.

Thank you for doing this! I think the subject should be:

irq: Move local_interrupt_{dis,en}able() into Rust

to be accurate about the name of the functions.

Quoted 11 lines

---
Resend because I fatfingered the Subject line ... Sorry for the noise in case you got the original busted one.

Applies against tip locking/urgent

---
 include/linux/spinlock.h             |   23 -------
 include/linux/spinlock_api_smp.h     |   41 ------------
 include/linux/spinlock_api_up.h      |   15 ----
 include/linux/spinlock_rt.h          |   18 -----

Seems we are missing a deletion of include/linux/interrupt_rc.h here.

Quoted 9 lines
 kernel/irq/refcount_interrupt_test.c |    2 
 kernel/locking/spinlock.c            |   31 ---------
 kernel/softirq.c                     |   15 ----
 rust/helpers/interrupt.c             |   21 ++++++
 rust/helpers/interrupt_rc.h          |   68 ++++++++++++++++++++
 rust/helpers/spinlock.c              |   39 +++++++++++
 rust/helpers/spinlock.h              |  114 +++++++++++++++++++++++++++++++++++
 11 files changed, 241 insertions(+), 146 deletions(-)

[...]

Quoted 26 lines
rust/helpers/spinlock.h new file +23 -0
@@ -0,0 +1,114 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __RUST_HELPERS_SPINLOCK_H
+#define __RUST_HELPERS_SPINLOCK_H
+
+#include <linux/spinlock.h>
+#include "interrupt_rc.h"
+
+#ifdef CONFIG_SMP
+void __lockfunc _raw_spin_lock_irq_disable(raw_spinlock_t *lock)	__acquires(lock);
+void __lockfunc _raw_spin_unlock_irq_enable(raw_spinlock_t *lock)	__releases(lock);
+
+/* Use the same config as spin_lock_irq() temporarily. */
+#ifdef CONFIG_INLINE_SPIN_LOCK_IRQ
+#define _raw_spin_lock_irq_disable(lock) __raw_spin_lock_irq_disable(lock)
+#endif
+
+/* Use the same config as spin_unlock_irq() temporarily. */
+#ifdef CONFIG_INLINE_SPIN_UNLOCK_IRQ
+#define _raw_spin_unlock_irq_enable(lock) __raw_spin_unlock_irq_enable(lock)
+#endif
+
+static __always_inline bool _raw_spin_trylock_irq_disable(raw_spinlock_t *lock)
+	__cond_acquires(true, lock)

rust/helpers/spinlock.h line 33

Quoted 10 lines
rust/helpers/spinlock.h new file +10 -0
+{
+	local_interrupt_disable();
+	if (_raw_spin_trylock(lock))
+		return true;
+	local_interrupt_enable();
+	return false;
+}
+
+static inline void __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+	__acquires(lock) __no_context_analysis

I think we need to put the

#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC) #endif

around this. Because in the #else branch of rust/helpers/spinlock.c we have an out-of-line definition of the same function.

Regards,
Boqun

Quoted 33 lines
rust/helpers/spinlock.h new file +33 -0
+{
+	local_interrupt_disable();
+	preempt_disable();
+	spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
+	LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
+}
+
+static inline void __raw_spin_unlock_irq_enable(raw_spinlock_t *lock)
+	__releases(lock)
+{
+	spin_release(&lock->dep_map, _RET_IP_);
+	do_raw_spin_unlock(lock);
+	local_interrupt_enable();
+	preempt_enable();
+}
+
+#else  /* CONFIG_SMP */
+
+#define __LOCK_IRQ_DISABLE(lock, ...)						\
+	do { local_interrupt_disable(); __LOCK(lock, ##__VA_ARGS__); } while (0)
+#define __UNLOCK_IRQ_ENABLE(lock, ...)						\
+	do { __UNLOCK(lock, ##__VA_ARGS__); local_interrupt_enable(); } while (0)
+
+#define _raw_spin_lock_irq_disable(lock)	__LOCK_IRQ_DISABLE(lock)
+#define _raw_spin_unlock_irq_enable(lock)	__UNLOCK_IRQ_ENABLE(lock)
+
+static __always_inline int _raw_spin_trylock_irq_disable(raw_spinlock_t *lock)
+	__cond_acquires(true, lock)
+{
+	__LOCK_IRQ_DISABLE(lock);
+	return 1;
+}
+

[...]

maintainer
#71 lore raw

Re: [PATCH] irq: Move local_irq_enable/disable() into Rust

Quoted 12 lines · Boqun Feng

[Cc Miguel, Lyude, Alice, Gary]

Quoted 8 lines · Thomas Gleixner

After reverting the spinlock conversion and a lengthy discussion it's the best to confine the reference counted interrupt disable/enable mechanism to Rust which is the only user.

This should become the new norm, but that needs more thoughts and cleaning up the confined usage in Rust at some point is way simpler than chasing random places which adopt it in the meanwhile.

Quoted 5 lines

Thank you for doing this! I think the subject should be:

irq: Move local_interrupt_{dis,en}able() into Rust

to be accurate about the name of the functions.

It actually also fails to mention the spinlock part :(

Quoted 6 lines
Quoted 4 lines
 include/linux/spinlock.h             |   23 -------
 include/linux/spinlock_api_smp.h     |   41 ------------
 include/linux/spinlock_api_up.h      |   15 ----
 include/linux/spinlock_rt.h          |   18 -----

Seems we are missing a deletion of include/linux/interrupt_rc.h here.

Weird. I'm sure I deleted it, but ...

Quoted 15 lines

rust/helpers/spinlock.h line 33

Quoted 12 lines
rust/helpers/spinlock.h new file +12 -0
+static __always_inline bool _raw_spin_trylock_irq_disable(raw_spinlock_t *lock)
+	__cond_acquires(true, lock)
+{
+	local_interrupt_disable();
+	if (_raw_spin_trylock(lock))
+		return true;
+	local_interrupt_enable();
+	return false;
+}
+
+static inline void __raw_spin_lock_irq_disable(raw_spinlock_t *lock)
+	__acquires(lock) __no_context_analysis

I think we need to put the

Quoted 2 lines

#if !defined(CONFIG_GENERIC_LOCKBREAK) || defined(CONFIG_DEBUG_LOCK_ALLOC) #endif

Duh yes. Missed that completely.

Quoted 2 lines

around this. Because in the #else branch of rust/helpers/spinlock.c we have an out-of-line definition of the same function.

Right.

Thanks,

tglx

author maintainer
#72 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

In reply to Thomas Gleixner

On Sun, Aug 30, 2026 at 09:57:30PM +0200, Thomas Gleixner wrote:
[...]

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+#ifdef CONFIG_RT

0 day probably will find it, but FWIW this one should be CONFIG_PREEMPT_RT.

Regards,
Boqun

Quoted 20 lines
rust/helpers/spinlock.h new file +20 -0
+static __always_inline void spin_lock_irq_disable(spinlock_t *lock)
+	__acquires(lock)
+{
+	rt_spin_lock(lock);
+}
+
+static __always_inline void spin_unlock_irq_enable(spinlock_t *lock)
+	__releases(lock)
+{
+	rt_spin_unlock(lock);
+}
+
+static __always_inline int spin_trylock_irq_disable(spinlock_t *lock)
+	__cond_acquires(true, lock)
+{
+	return rt_spin_trylock(lock);
+}
+
+#else  /* CONFIG_RT */
+

[...]

maintainer
#73 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 7 lines · Boqun Feng

On Sun, Aug 30, 2026 at 09:57:30PM +0200, Thomas Gleixner wrote:
[...]

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+#ifdef CONFIG_RT

0 day probably will find it, but FWIW this one should be CONFIG_PREEMPT_RT.

Indeed

maintainer
#74 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

In reply to Boqun Feng
Quoted 13 lines · Boqun Feng

include/linux/irqflags.h extern void warn_bogus_irq_restore(void); · line 216

Quoted 7 lines · Thomas Gleixner
include/linux/irqflags.h +7 -0
+static __always_inline void __raw_local_irq_restore(unsigned long cnt)
+{
+	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK) == (cnt + HARDIRQ_DISABLE_OFFSET));
+
+	if (!(__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) & HARDIRQ_DISABLE_MASK))
+		arch_local_irq_enable();
+}

And while we are at, we can just introduce a raw_local_irq_restore_auto() (definitely needs a better name), which doesn't need a cnt:

Quoted 7 lines

static __always_inline void __raw_local_irq_restore_auto(void)

{
	debug_assert((preempt_count() & HARDIRQ_DISABLE_MASK));

	if (!(__preempt_count_sub_return(HARDIRQ_DISABLE_OFFSET) & HARDIRQ_DISABLE_MASK))
		arch_local_irq_enable();
}

Yes, we can add something like this once we got the design and the debug infrastructure in place. For now the count is helpful to debug stuff and we want to have something equivalent at least for lockdep builds.

Thanks,

tglx

maintainer
#75 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

In reply to Thomas Gleixner
Quoted 23 lines · Thomas Gleixner

On Tue, Aug 25 2026 at 16:28, Boqun Feng wrote:
So you can get away with:

raw_local_irq_save(flags)

{
	flags = count;
	if (!count)
        	arch_local_irq_disable();
        count++;
}

raw_local_irq_restore(flags)

{
	if (!(count = flags))
        	arch_local_irq_enable();
}

raw_local_irq_disable()

{
        arch_local_irq_disable();
        count = 1;
}
Quoted 5 lines

raw_local_irq_enable()

{
        count = 0;
        arch_local_irq_enable();
}

Actually it can be done way simpler because the nasty case of

         scoped_guard(lock_irqsave, lock) {
         	unlock_irq(lock);
                lock_irq(lock);
         }

is only valid for a single lock guard, because if it's nested then the outer lock would lose the interrupt disabled protection. Anything else would be a bug on its own and would have long ago blown up in our face.

So we can completely ignore flags.

raw_local_irq_save(flags)
{
if (!count)

         	arch_local_irq_disable();
        count++;
}

raw_local_irq_restore(flags)
{
if (!--count)

         	arch_local_irq_enable();
}

raw_local_irq_disable()

{
         arch_local_irq_disable();
         count++;
}

raw_local_irq_enable()

{
         count--;
         arch_local_irq_enable();
}

with a copious amount of debug machinery to catch any oddballs.

Also note that this never uses irqsave/restore because historically that has been way slower than CLI/STI.

A decade+ ago this used to be up to 30%, but micro architectures optimized for it. Still on a SKL it's ~14% and on a Zen3 ~8% slower.

Thanks,

tglx

author maintainer
#76 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 32 lines · Thomas Gleixner
Quoted 23 lines · Thomas Gleixner

On Tue, Aug 25 2026 at 16:28, Boqun Feng wrote:
So you can get away with:

raw_local_irq_save(flags)

{
	flags = count;
	if (!count)
        	arch_local_irq_disable();
        count++;
}

raw_local_irq_restore(flags)

{
	if (!(count = flags))
        	arch_local_irq_enable();
}

raw_local_irq_disable()

{
        arch_local_irq_disable();
        count = 1;
}
Quoted 5 lines

raw_local_irq_enable()

{
        count = 0;
        arch_local_irq_enable();
}

Actually it can be done way simpler because the nasty case of

Quoted 9 lines
         scoped_guard(lock_irqsave, lock) {
         	unlock_irq(lock);
                lock_irq(lock);
         }

is only valid for a single lock guard, because if it's nested then the outer lock would lose the interrupt disabled protection. Anything else would be a bug on its own and would have long ago blown up in our face.

Right.

Quoted 9 lines

So we can completely ignore flags.

raw_local_irq_save(flags)
{
if (!count)

         	arch_local_irq_disable();
        count++;
}
Quoted 6 lines

raw_local_irq_restore(flags)
{
if (!--count)

         	arch_local_irq_enable();
}

These are just local_interrupt_{disable,enable}() (replacing arch_local_irq_save() with arch_local_irq_disable()) :)

Quoted 6 lines

raw_local_irq_disable()

{
         arch_local_irq_disable();
         count++;
}
Quoted 8 lines

raw_local_irq_enable()

{
         count--;
         arch_local_irq_enable();
}

with a copious amount of debug machinery to catch any oddballs.

Ok, so brainstorm on the oddballs:

# 1: double disable

	local_irq_disable();
	local_irq_disable();
	local_irq_enable();

# 2: double enable

	local_irq_disable();
	local_irq_enable();
	local_irq_enable();

I think these mean we should probably do count = 1 and count = 0 in irq_{enable,disable}() than count++ and count--?

# 3: only restore once

	local_irq_save(flag1);
	local_irq_save(flag2);
	local_irq_restore(flag1);

# 4: keep restoring

	local_irq_save(flag1);
	local_irq_restore(flag1);
	local_irq_restore(flag1);

these are a bit tricky, I guess we could only fix the users? But we should not postpone the infrastructure because of these?

Quoted 6 lines

Also note that this never uses irqsave/restore because historically that has been way slower than CLI/STI.

A decade+ ago this used to be up to 30%, but micro architectures optimized for it. Still on a SKL it's ~14% and on a Zen3 ~8% slower.

Yes, if we go to the level to unify all irq disabling with counter tracking then I think using arch_local_irq_disable() is possible and makes a lot of senses.

Regards,
Boqun

Quoted 4 lines

Thanks,

tglx

maintainer
#77 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 8 lines · Boqun Feng

Ok, so brainstorm on the oddballs:

# 1: double disable

	local_irq_disable();
	local_irq_disable();
	local_irq_enable();
Quoted 8 lines

# 2: double enable

	local_irq_disable();
	local_irq_enable();
	local_irq_enable();

I think these mean we should probably do count = 1 and count = 0 in irq_{enable,disable}() than count++ and count--?

Could yeah, but ideally we'd take this opportunity to finally get rid of them. We've ran into them a number of times, and I'm sure get fixed up a bunch at some point, but never made the push to clean them out.

As is, lockdep only counts the redundant ones. They're a stat nobody ever looks at.

Quoted 6 lines

# 3: only restore once

	local_irq_save(flag1);
	local_irq_save(flag2);
	local_irq_restore(flag1);
Quoted 8 lines

# 4: keep restoring

	local_irq_save(flag1);
	local_irq_restore(flag1);
	local_irq_restore(flag1);

these are a bit tricky, I guess we could only fix the users? But we should not postpone the infrastructure because of these?

Yeah, so I do have a solution for that, but it is too horrible to write in this small margin and all that :-) Thomas will kick my ass.

Best we simply detect and clean up.

#78 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

In reply to Boqun Feng

On Thu, 27 Aug 2026 14:33:05 -0700
Boqun Feng <[email protected]> wrote:

....

Quoted 3 lines

Yes, if we go to the level to unify all irq disabling with counter tracking then I think using arch_local_irq_disable() is possible and makes a lot of senses.

Where are you thinking of keeping the counter?
On non-x86 accessing it may be expensive.
The best bet is probably in 'current'.

Doesn't that make this valid?

	int c = current->irq_disable_count;
	if (c) {
		current->irq_disable_count = c + 1;
		return c;
	}
	disable_irq(); // asm("cli")
	interrupt_disable_barrier(); // ISTR arm needs this
	current->irq_disable_count = 1;
	return 0;
}

The task can be preempted in the middle - but that doesn't matter.

If spin_lock_irqsave() returns current->irq_disable_count then any existing code that does lock chaining works unaltered.

David

Quoted 9 lines

Regards,
Boqun

Quoted 4 lines

Thanks,

tglx

author maintainer
#79 lore raw

Re: [PATCH] locking: Revert switching guards to _irq_{disable,enable}()

Quoted 12 lines · David Laight

On Thu, 27 Aug 2026 14:33:05 -0700
Boqun Feng <[email protected]> wrote:

....

Quoted 3 lines

Yes, if we go to the level to unify all irq disabling with counter tracking then I think using arch_local_irq_disable() is possible and makes a lot of senses.

Where are you thinking of keeping the counter?
On non-x86 accessing it may be expensive.
The best bet is probably in 'current'.

The counter remains in preempt_count() as we already did for local_interrupt_disable()?

Quoted 12 lines

Doesn't that make this valid?

	int c = current->irq_disable_count;
	if (c) {
		current->irq_disable_count = c + 1;
		return c;
	}
	disable_irq(); // asm("cli")
	interrupt_disable_barrier(); // ISTR arm needs this
	current->irq_disable_count = 1;
	return 0;
}
Quoted 5 lines

The task can be preempted in the middle - but that doesn't matter.

If spin_lock_irqsave() returns current->irq_disable_count then any existing code that does lock chaining works unaltered.

yes, but someone could be creative and do:

	spin_lock_irqsave(l1, flag1);
	spin_lock_irqsave(l2, flag2);
	spin_unlock(l2);
	spin_unlock_irqrestore(l1, flag1);

basically, perfectly nesting and paired critical sections always work, it's the unknown oddballs that we need to worry about.

Regards,
Boqun

Quoted 12 lines

David

Quoted 9 lines

Regards,
Boqun

Quoted 4 lines

Thanks,

tglx

author maintainer
#80 lore raw

PATCH v4 8/17 sched: Remove the unused preempt_offset parameter of __cant_sleep()

In reply to Boqun Feng

The preempt_offset is always 0 in all the callsites of __cant_sleep(), hence remove it. It also allows us to clear up the code a bit by no longer using a "preempt_count() > .." comparison.

---
 include/linux/kernel.h | 4 ++--
 kernel/sched/core.c    | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
include/linux/kernel.h +2 -2
index e5570a16cbb1..24414c79e59a 100644
@@ -72,7 +72,7 @@ extern int dynamic_might_resched(void);
 #ifdef CONFIG_DEBUG_ATOMIC_SLEEP
 extern void __might_resched(const char *file, int line, unsigned int offsets);
 extern void __might_sleep(const char *file, int line);
-extern void __cant_sleep(const char *file, int line, int preempt_offset);
+extern void __cant_sleep(const char *file, int line);
 extern void __cant_migrate(const char *file, int line);
 
 /**
@@ -95,7 +95,7 @@ extern void __cant_migrate(const char *file, int line);
  * this macro will print a stack trace if it is executed with preemption enabled
  */
 # define cant_sleep() \
-	do { __cant_sleep(__FILE__, __LINE__, 0); } while (0)
+	do { __cant_sleep(__FILE__, __LINE__); } while (0)
 # define sched_annotate_sleep()	(current->task_state_change = 0)
 
 /**
kernel/sched/core.c +2 -2
index 96226707c2f6..aa116daf21bd 100644
@@ -9199,7 +9199,7 @@ void __might_resched(const char *file, int line, unsigned int offsets)
 }
 EXPORT_SYMBOL(__might_resched);
 
-void __cant_sleep(const char *file, int line, int preempt_offset)
+void __cant_sleep(const char *file, int line)
 {
 	static unsigned long prev_jiffy;
 
@@ -9209,7 +9209,7 @@ void __cant_sleep(const char *file, int line, int preempt_offset)
 	if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
 		return;
 
-	if (preempt_count() > preempt_offset)
+	if (preempt_count())
 		return;
 
 	if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy)
Signature · 1 line
2.50.1 (Apple Git-155)
merge bot Applied
#81 lore raw

[tip: locking/core] sched: Remove the unused preempt_offset parameter of __cant_sleep()

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     1cda40c48e295f8ad16bd5401591d7e6e5bcb591
Gitweb:        https://git.kernel.org/tip/1cda40c48e295f8ad16bd5401591d7e6e5bcb591
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:30 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 08 Aug 2026 22:44:07 +02:00

sched: Remove the unused preempt_offset parameter of __cant_sleep()

The preempt_offset is always 0 in all the callsites of __cant_sleep(), hence remove it. It also allows us to clear up the code a bit by no longer using a "preempt_count() > .." comparison.

---
 include/linux/kernel.h | 4 ++--
 kernel/sched/core.c    | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
include/linux/kernel.h +2 -2
index e5570a1..24414c7 100644
@@ -72,7 +72,7 @@ extern int dynamic_might_resched(void);
 #ifdef CONFIG_DEBUG_ATOMIC_SLEEP
 extern void __might_resched(const char *file, int line, unsigned int offsets);
 extern void __might_sleep(const char *file, int line);
-extern void __cant_sleep(const char *file, int line, int preempt_offset);
+extern void __cant_sleep(const char *file, int line);
 extern void __cant_migrate(const char *file, int line);
 
 /**
@@ -95,7 +95,7 @@ extern void __cant_migrate(const char *file, int line);
  * this macro will print a stack trace if it is executed with preemption enabled
  */
 # define cant_sleep() \
-	do { __cant_sleep(__FILE__, __LINE__, 0); } while (0)
+	do { __cant_sleep(__FILE__, __LINE__); } while (0)
 # define sched_annotate_sleep()	(current->task_state_change = 0)
 
 /**
kernel/sched/core.c +2 -2
index 9622670..aa116da 100644
@@ -9199,7 +9199,7 @@ void __might_resched(const char *file, int line, unsigned int offsets)
 }
 EXPORT_SYMBOL(__might_resched);
 
-void __cant_sleep(const char *file, int line, int preempt_offset)
+void __cant_sleep(const char *file, int line)
 {
 	static unsigned long prev_jiffy;
 
@@ -9209,7 +9209,7 @@ void __cant_sleep(const char *file, int line, int preempt_offset)
 	if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
 		return;
 
-	if (preempt_count() > preempt_offset)
+	if (preempt_count())
 		return;
 
 	if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy)
merge bot Applied
#82 lore raw

[tip: locking/core] sched: Remove the unused preempt_offset parameter of __cant_sleep()

In reply to Boqun Feng
The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     ac4231a77973fc20808ed84c4af343eca2342d4b
Gitweb:        https://git.kernel.org/tip/ac4231a77973fc20808ed84c4af343eca2342d4b
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:30 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 10 Aug 2026 10:50:19 +02:00

sched: Remove the unused preempt_offset parameter of __cant_sleep()

The preempt_offset is always 0 in all the callsites of __cant_sleep(), hence remove it. It also allows us to clear up the code a bit by no longer using a "preempt_count() > .." comparison.

---
 include/linux/kernel.h | 4 ++--
 kernel/sched/core.c    | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
include/linux/kernel.h +2 -2
index e5570a1..24414c7 100644
@@ -72,7 +72,7 @@ extern int dynamic_might_resched(void);
 #ifdef CONFIG_DEBUG_ATOMIC_SLEEP
 extern void __might_resched(const char *file, int line, unsigned int offsets);
 extern void __might_sleep(const char *file, int line);
-extern void __cant_sleep(const char *file, int line, int preempt_offset);
+extern void __cant_sleep(const char *file, int line);
 extern void __cant_migrate(const char *file, int line);
 
 /**
@@ -95,7 +95,7 @@ extern void __cant_migrate(const char *file, int line);
  * this macro will print a stack trace if it is executed with preemption enabled
  */
 # define cant_sleep() \
-	do { __cant_sleep(__FILE__, __LINE__, 0); } while (0)
+	do { __cant_sleep(__FILE__, __LINE__); } while (0)
 # define sched_annotate_sleep()	(current->task_state_change = 0)
 
 /**
kernel/sched/core.c +2 -2
index 9622670..aa116da 100644
@@ -9199,7 +9199,7 @@ void __might_resched(const char *file, int line, unsigned int offsets)
 }
 EXPORT_SYMBOL(__might_resched);
 
-void __cant_sleep(const char *file, int line, int preempt_offset)
+void __cant_sleep(const char *file, int line)
 {
 	static unsigned long prev_jiffy;
 
@@ -9209,7 +9209,7 @@ void __cant_sleep(const char *file, int line, int preempt_offset)
 	if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
 		return;
 
-	if (preempt_count() > preempt_offset)
+	if (preempt_count())
 		return;
 
 	if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy)
author maintainer
#83 lore raw

PATCH v4 9/17 sched: Avoid signed comparison of preempt_count() in __cant_migrate()

In reply to Boqun Feng

Currently preempt_count() is always a non-negative int on all archs (PREEMPT_NEED_RESCHED archs will mask out the MSB when returning preempt_count()), hence the checking in __cant_migrate() is in fact just checking whether preempt_count() is 0 or not. In a future change, we are going to use all the 32 bits of preempt_count(), which would make negative int values possible from preempt_count(). Therefore convert the "> 0" comparison into a zero check to prepare for the future change. No functional changes are intended.

---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index aa116daf21bd..9b3f1764fa9e 100644
@@ -9241,7 +9241,7 @@ void __cant_migrate(const char *file, int line)
 	if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
 		return;
 
-	if (preempt_count() > 0)
+	if (preempt_count())
 		return;
 
 	if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy)
Signature · 1 line
2.50.1 (Apple Git-155)
merge bot Applied
#84 lore raw

[tip: locking/core] sched: Avoid signed comparison of preempt_count() in __cant_migrate()

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     31f35d04a29a7554bb3b9ad827dd37c2942e31d4
Gitweb:        https://git.kernel.org/tip/31f35d04a29a7554bb3b9ad827dd37c2942e31d4
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:31 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 08 Aug 2026 22:44:08 +02:00

sched: Avoid signed comparison of preempt_count() in __cant_migrate()

Currently preempt_count() is always a non-negative int on all archs (PREEMPT_NEED_RESCHED archs will mask out the MSB when returning preempt_count()), hence the checking in __cant_migrate() is in fact just checking whether preempt_count() is 0 or not. In a future change, we are going to use all the 32 bits of preempt_count(), which would make negative int values possible from preempt_count(). Therefore convert the "> 0" comparison into a zero check to prepare for the future change. No functional changes are intended.

---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index aa116da..9b3f176 100644
@@ -9241,7 +9241,7 @@ void __cant_migrate(const char *file, int line)
 	if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
 		return;
 
-	if (preempt_count() > 0)
+	if (preempt_count())
 		return;
 
 	if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy)
merge bot Applied
#85 lore raw

[tip: locking/core] sched: Avoid signed comparison of preempt_count() in __cant_migrate()

In reply to Boqun Feng
The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     560fcaa92ef983315023eb4ebfc1ebae1132fb2a
Gitweb:        https://git.kernel.org/tip/560fcaa92ef983315023eb4ebfc1ebae1132fb2a
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:31 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 10 Aug 2026 10:50:19 +02:00

sched: Avoid signed comparison of preempt_count() in __cant_migrate()

Currently preempt_count() is always a non-negative int on all archs (PREEMPT_NEED_RESCHED archs will mask out the MSB when returning preempt_count()), hence the checking in __cant_migrate() is in fact just checking whether preempt_count() is 0 or not. In a future change, we are going to use all the 32 bits of preempt_count(), which would make negative int values possible from preempt_count(). Therefore convert the "> 0" comparison into a zero check to prepare for the future change. No functional changes are intended.

---
 kernel/sched/core.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
index aa116da..9b3f176 100644
@@ -9241,7 +9241,7 @@ void __cant_migrate(const char *file, int line)
 	if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
 		return;
 
-	if (preempt_count() > 0)
+	if (preempt_count())
 		return;
 
 	if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy)
author maintainer
#86 lore raw

PATCH v4 10/17 preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS

In reply to Boqun Feng

With the changes that enable preempt count to track IRQ disabling nesting, we don't have enough bits in 32-bit preempt count implementation, as a result we move NMI nesting bits out of the 32-bit preempt count. However on the architectures that can support 64-bit preempt count implementation, we can keep the NMI nesting bits in the 32-bit preempt count and avoid maintaining NMI nesting bits outside of the same cache line.

Therefore HAS_SEPARATE_PREEMPT_RESCHED_BITS is introduced to allow architectures to select this. Note that under this Kconfig, preempt count is maintained in a 64-bit word however preempt_count() still remains as an int because all the effective bits still fit in (previously we mask out NEED_RESCHED bit in preempt_count()). This should make no functional changes for existing preempt_count() users.

Enable this for x86_64 along with the introduction of the Kconfig.

[boqun: Undo the __preempt_count_{add,sub}() optimization in 32-bit preempt count since it may introduce {over,under}flow]

---
 arch/x86/Kconfig               |  1 +
 arch/x86/include/asm/preempt.h | 55 +++++++++++++++++++++++-----------
 arch/x86/kernel/cpu/common.c   |  2 +-
 include/linux/hardirq.h        | 47 +++++++++++++++++++++--------
 include/linux/preempt.h        | 23 ++++++++++++--
 kernel/Kconfig.preempt         |  4 +++
 kernel/sched/core.c            | 12 ++++++--
 kernel/softirq.c               |  6 ++++
 lib/locking-selftest.c         |  2 +-
 9 files changed, 115 insertions(+), 37 deletions(-)
arch/x86/Kconfig +1 -0
index bdad90f210e4..6a7067d20a6a 100644
@@ -326,6 +326,7 @@ config X86
 	select USER_STACKTRACE_SUPPORT
 	select HAVE_ARCH_KCSAN			if X86_64
 	select PROC_PID_ARCH_STATUS		if PROC_FS
+	select HAS_SEPARATE_PREEMPT_RESCHED_BITS		if X86_64 && PREEMPT_COUNT
 	select HAVE_ARCH_NODE_DEV_GROUP		if X86_SGX
 	select FUNCTION_ALIGNMENT_16B		if X86_64 || X86_ALIGNMENT_16
 	select FUNCTION_ALIGNMENT_4B
arch/x86/include/asm/preempt.h +38 -17

123 lines not on this page · raw message

arch/x86/kernel/cpu/common.c +1 -1
index a3df21d26460..73a6d9f6a78e 100644
@@ -2236,7 +2236,7 @@ DEFINE_PER_CPU_CACHE_HOT(struct task_struct *, current_task) = &init_task;
 EXPORT_PER_CPU_SYMBOL(current_task);
 EXPORT_PER_CPU_SYMBOL(const_current_task);
 
-DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) = INIT_PREEMPT_COUNT;
+DEFINE_PER_CPU_CACHE_HOT(unsigned long, __preempt_count) = INIT_PREEMPT_COUNT;
 EXPORT_PER_CPU_SYMBOL(__preempt_count);
 
 DEFINE_PER_CPU_CACHE_HOT(unsigned long, cpu_current_top_of_stack) = TOP_OF_INIT_STACK;
include/linux/hardirq.h +34 -13

78 lines not on this page · raw message

include/linux/preempt.h +20 -3
index 33fc4c814a9f..8299657f0f86 100644
@@ -34,14 +34,31 @@
  *         SOFTIRQ_MASK:	0x0000ff00
  * HARDIRQ_DISABLE_MASK:	0x00ff0000
  *         HARDIRQ_MASK:	0x0f000000
+ *
+ * When HAS_SEPARATE_PREEMPT_RESCHED_BITS=y, PREEMPT_NEED_RESCHED is put in a
+ * separate word and that allows 64bit load-store architectures to 'set'
+ * PREEMPT_NEED_RESCHED without messing up the otherwise symmetric
+ * modifications used on preempt_count and still load the whole thing
+ * (single-copy) atomically, without having to resort to full atomic
+ * operations.
+ *
+ * Because of the above, NMI_MASK bits are different depending on
+ * HAS_SEPARATE_PREEMPT_RESCHED_BITS:
+ *
+ * - HAS_SEPARATE_PREEMPT_RESCHED_BITS=n:
+ *
  *             NMI_MASK:	0x10000000
  * PREEMPT_NEED_RESCHED:	0x80000000
+ *
+ * - HAS_SEPARATE_PREEMPT_RESCHED_BITS=y:
+ *             NMI_MASK:	0xf0000000
+ * (PREEMPT_NEED_RESCHED is in a different word)
  */
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
 #define HARDIRQ_DISABLE_BITS	8
 #define HARDIRQ_BITS	4
-#define NMI_BITS	1
+#define NMI_BITS	(1 + 3*IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS))
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
@@ -116,8 +133,8 @@ static __always_inline unsigned char interrupt_context_level(void)
  * preempt_count() is commonly implemented with READ_ONCE().
  */
 
-#define nmi_count()	(preempt_count() & NMI_MASK)
-#define hardirq_count()	(preempt_count() & HARDIRQ_MASK)
+#define nmi_count()		(preempt_count() & NMI_MASK)
+#define hardirq_count()		(preempt_count() & HARDIRQ_MASK)
 #ifdef CONFIG_PREEMPT_RT
 # define softirq_count()	(current->softirq_disable_cnt & SOFTIRQ_MASK)
 # define irq_count()		((preempt_count() & (NMI_MASK | HARDIRQ_MASK)) | softirq_count())
kernel/Kconfig.preempt +4 -0
index 88c594c6d7fc..35f546a042b1 100644
@@ -122,6 +122,10 @@ config PREEMPT_RT_NEEDS_BH_LOCK
 config PREEMPT_COUNT
        bool
 
+config HAS_SEPARATE_PREEMPT_RESCHED_BITS
+	bool
+	depends on PREEMPT_COUNT && 64BIT
+
 config PREEMPTION
        bool
        select PREEMPT_COUNT
kernel/sched/core.c +10 -2
index 9b3f1764fa9e..6d88343c3bad 100644
@@ -5973,8 +5973,13 @@ void preempt_count_add(int val)
 #ifdef CONFIG_DEBUG_PREEMPT
 	/*
 	 * Underflow?
+	 *
+	 * Cannot detect underflow based on the current preempt_count() value
+	 * if using HAS_SEPARATE_PREEMPT_RESCHED_BITS because preempt count takes all 32
+	 * bits.
 	 */
-	if (DEBUG_LOCKS_WARN_ON((preempt_count() < 0)))
+	if (!IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS) &&
+	    DEBUG_LOCKS_WARN_ON((preempt_count() < 0)))
 		return;
 #endif
 	__preempt_count_add(val);
@@ -6006,7 +6011,10 @@ void preempt_count_sub(int val)
 	/*
 	 * Underflow?
 	 */
-	if (DEBUG_LOCKS_WARN_ON(val > preempt_count()))
+	unsigned int uval = val;
+	unsigned int pc = preempt_count();
+
+	if (DEBUG_LOCKS_WARN_ON(pc - uval > pc))
 		return;
 	/*
 	 * Is the spinlock portion underflowing?
kernel/softirq.c +6 -0
index 0c9b2269a8d6..7980a4a232f9 100644
@@ -103,7 +103,13 @@ void _local_interrupt_enable(void)
 }
 EXPORT_SYMBOL(_local_interrupt_enable);
 
+#ifndef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS
+/*
+ * Any 32bit architecture that still cares about performance should
+ * probably ensure this is near preempt_count.
+ */
 DEFINE_PER_CPU(unsigned int, nmi_nesting);
+#endif
 
 /*
  * SOFTIRQ_OFFSET usage:
lib/locking-selftest.c +1 -1
index bfafe1204c7b..c3d976c801bb 100644
@@ -1429,7 +1429,7 @@ static int unexpected_testcase_failures;
 
 static void dotest(void (*testcase_fn)(void), int expected, int lockclass_mask)
 {
-	int saved_preempt_count = preempt_count();
+	long saved_preempt_count = preempt_count();
 #ifdef CONFIG_PREEMPT_RT
 	int saved_mgd_count = current->migration_disabled;
 	int saved_rcu_count = current->rcu_read_lock_nesting;
Signature · 1 line
2.50.1 (Apple Git-155)
#87 lore raw

Re: [PATCH v4 10/17] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS

Hi Boqun,

Quoted 8 lines · Boqun Feng

With the changes that enable preempt count to track IRQ disabling nesting, we don't have enough bits in 32-bit preempt count implementation, as a result we move NMI nesting bits out of the 32-bit preempt count. However on the architectures that can support 64-bit preempt count implementation, we can keep the NMI nesting bits in the 32-bit preempt count and avoid maintaining NMI nesting bits outside of the same cache line.

[...]

Quoted 21 lines
include/linux/hardirq.h +6 -2
@@ -10,8 +10,6 @@
  #include <linux/vtime.h>
  #include <asm/hardirq.h>
 
-DECLARE_PER_CPU(unsigned int, nmi_nesting);
-
  extern void synchronize_irq(unsigned int irq);
  extern bool synchronize_hardirq(unsigned int irq);
 
@@ -94,6 +92,37 @@ void irq_exit_rcu(void);
  #define arch_nmi_exit()		do { } while (0)
  #endif
 
+#ifdef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS
+static __always_inline void __preempt_count_nmi_enter(void)
+{
+	__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);
+}
+

include/linux/hardirq.h void irq_exit_rcu(void); · line 110

Quoted 10 lines
include/linux/hardirq.h +10 -0
+static __always_inline void __preempt_count_nmi_exit(void)
+{
+	__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);
+}
+#else
+DECLARE_PER_CPU(unsigned int, nmi_nesting);
+
+#define __preempt_count_nmi_enter()				\
+	do {							\
+		__preempt_count_add(HARDIRQ_OFFSET);		\

nit: This limit is because to have the same behavior as other case when NMI_BITS=4 right?
It is not easy to infer that from comment.

include/linux/hardirq.h void irq_exit_rcu(void); · line 114

Quoted 4 lines
include/linux/hardirq.h +4 -0
+		/* Maximum NMI nesting is 15. */		\
+		BUG_ON(__this_cpu_read(nmi_nesting) >= 15);	\
+		__this_cpu_inc(nmi_nesting);			\
+		preempt_count_set(preempt_count() | NMI_MASK);  \

Is there a reason preempt count updates are split rather than folded into a single preempt_count update?

Quoted 49 lines
include/linux/hardirq.h +14 -11
+	} while (0)
+
+#define __preempt_count_nmi_exit()				\
+	do {							\
+		__preempt_count_sub(HARDIRQ_OFFSET);		\
+		if (!__this_cpu_dec_return(nmi_nesting))	\
+			preempt_count_set(preempt_count() & ~NMI_MASK); \
+	} while (0)
+
+#endif
+
  /*
   * NMI vs Tracing
   * --------------
@@ -110,18 +139,14 @@ void irq_exit_rcu(void);
  	do {							\
  		lockdep_off();					\
  		arch_nmi_enter();				\
-		/* Maximum NMI nesting is 15. */		\
-		BUG_ON(__this_cpu_read(nmi_nesting) >= 15);	\
-		__this_cpu_inc(nmi_nesting);			\
-		__preempt_count_add(HARDIRQ_OFFSET);		\
-		preempt_count_set(preempt_count() | NMI_MASK);	\
+		__preempt_count_nmi_enter();			\
  	} while (0)
 
  #define nmi_enter()						\
  	do {							\
  		__nmi_enter();					\
  		lockdep_hardirq_enter();			\
-		ct_nmi_enter();				\
+		ct_nmi_enter();					\
  		instrumentation_begin();			\
  		ftrace_nmi_enter();				\
  		instrumentation_end();				\
@@ -129,12 +154,8 @@ void irq_exit_rcu(void);
 
  #define __nmi_exit()						\
  	do {							\
-		unsigned int nesting;				\
  		BUG_ON(!in_nmi());				\
-		__preempt_count_sub(HARDIRQ_OFFSET);		\
-		nesting = __this_cpu_dec_return(nmi_nesting);	\
-		if (!nesting)					\
-			preempt_count_set(preempt_count() & ~NMI_MASK);	\
+		__preempt_count_nmi_exit();			\
  		arch_nmi_exit();				\
  		lockdep_on();					\
  	} while (0)
author maintainer
#88 lore raw

Re: [PATCH v4 10/17] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS

Quoted 38 lines · Shrikanth Hegde

Hi Boqun,

Quoted 8 lines · Boqun Feng

With the changes that enable preempt count to track IRQ disabling nesting, we don't have enough bits in 32-bit preempt count implementation, as a result we move NMI nesting bits out of the 32-bit preempt count. However on the architectures that can support 64-bit preempt count implementation, we can keep the NMI nesting bits in the 32-bit preempt count and avoid maintaining NMI nesting bits outside of the same cache line.

[...]

Quoted 24 lines
include/linux/hardirq.h +12 -2
@@ -10,8 +10,6 @@
  #include <linux/vtime.h>
  #include <asm/hardirq.h>
-DECLARE_PER_CPU(unsigned int, nmi_nesting);
-
  extern void synchronize_irq(unsigned int irq);
  extern bool synchronize_hardirq(unsigned int irq);
@@ -94,6 +92,37 @@ void irq_exit_rcu(void);
  #define arch_nmi_exit()		do { } while (0)
  #endif
+#ifdef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS
+static __always_inline void __preempt_count_nmi_enter(void)
+{
+	__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);
+}
+
+static __always_inline void __preempt_count_nmi_exit(void)
+{
+	__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);
+}
+#else
+DECLARE_PER_CPU(unsigned int, nmi_nesting);
Quoted 9 lines

include/linux/hardirq.h void irq_exit_rcu(void); · line 110

Quoted 4 lines
include/linux/hardirq.h +4 -0
+
+#define __preempt_count_nmi_enter()				\
+	do {							\
+		__preempt_count_add(HARDIRQ_OFFSET);		\

nit: This limit is because to have the same behavior as other case when NMI_BITS=4 right?
It is not easy to infer that from comment.

It's sort of design by implementation IIUC, previously because of NMI_BITS=4, we could only support nesting level being 15. And here we just want to keep the same behavior here.

If your question is why 15 was a good number before this change, I guess would be it's just a number that is neither too big or too small.

Quoted 9 lines

include/linux/hardirq.h void irq_exit_rcu(void); · line 114

Quoted 4 lines
include/linux/hardirq.h +4 -0
+		/* Maximum NMI nesting is 15. */		\
+		BUG_ON(__this_cpu_read(nmi_nesting) >= 15);	\
+		__this_cpu_inc(nmi_nesting);			\
+		preempt_count_set(preempt_count() | NMI_MASK);  \

Is there a reason preempt count updates are split rather than folded into a single preempt_count update?

Keeping the implementation simple is one reason, most architectures could utilize the HAS_SEPARATE_PREEMPT_RESCHED_BITS for better performance. So that reduces the importance of having something complicated but saves one access here. But if you see an optimization that can be done here, please do share!

Peter had proposed one optimization here:

	#define __preempt_count_nmi_enter()				\
		do {							\
			unsigned int _o = NMI_MASK + HARDIRQ_OFFSET;	\
			/* Maximum NMI nesting is 15. */		\
			BUG_ON(__this_cpu_read(nmi_nesting) >= 15);	\
			__this_cpu_inc(nmi_nesting);			\
			_o -= (preempt_count() & NMI_MASK);		\
			__preempt_count_add(_o);			\
		} while (0)
	
	#define __preempt_count_nmi_exit()				\
		do {							\
			unsigned int _o = HARDIRQ_OFFSET;		\
			if (!__this_cpu_dec_return(nmi_nesting))	\
				_o += NMI_MASK;				\
			__preempt_count_sub(_o);			\
		} while (0)

but it has a problem considering this:

	// outermost NMI handler
	// nmi_nesting == 0

	nmi_enter();
	// ^ nmi_nesting == 1 and NMI_MASK is set.
	...
	nmi_exit():
	  if (!__this_cpu_dec_return(nmi_nesting)) // return true
	    _o += NMI_MASK;
	  <NMI start>
	  nmi_enter();
	  // ^ nmi_nesting == 1 and NMI_MASK is set.
	  nmi_exit();
	  // ^ nmi_nesting == 0 and NMI_MASK is *unset*.
	  <NMI end>

	  preempt_count_sub(_o); // _o == HARDIRQ_OFFSET + NMI_MASK,
	  			 // underflow

(Now think about this, the __preempt_count_nmi_enter() does seems fine, maybe we can keep that, too tired to remember whether there is any subtly here... will take another look tomorrow)

Regards,
Boqun

Quoted 47 lines
Quoted 47 lines
include/linux/hardirq.h +14 -11
+	} while (0)
+
+#define __preempt_count_nmi_exit()				\
+	do {							\
+		__preempt_count_sub(HARDIRQ_OFFSET);		\
+		if (!__this_cpu_dec_return(nmi_nesting))	\
+			preempt_count_set(preempt_count() & ~NMI_MASK); \
+	} while (0)
+
+#endif
+
  /*
   * NMI vs Tracing
   * --------------
@@ -110,18 +139,14 @@ void irq_exit_rcu(void);
  	do {							\
  		lockdep_off();					\
  		arch_nmi_enter();				\
-		/* Maximum NMI nesting is 15. */		\
-		BUG_ON(__this_cpu_read(nmi_nesting) >= 15);	\
-		__this_cpu_inc(nmi_nesting);			\
-		__preempt_count_add(HARDIRQ_OFFSET);		\
-		preempt_count_set(preempt_count() | NMI_MASK);	\
+		__preempt_count_nmi_enter();			\
  	} while (0)
  #define nmi_enter()						\
  	do {							\
  		__nmi_enter();					\
  		lockdep_hardirq_enter();			\
-		ct_nmi_enter();				\
+		ct_nmi_enter();					\
  		instrumentation_begin();			\
  		ftrace_nmi_enter();				\
  		instrumentation_end();				\
@@ -129,12 +154,8 @@ void irq_exit_rcu(void);
  #define __nmi_exit()						\
  	do {							\
-		unsigned int nesting;				\
  		BUG_ON(!in_nmi());				\
-		__preempt_count_sub(HARDIRQ_OFFSET);		\
-		nesting = __this_cpu_dec_return(nmi_nesting);	\
-		if (!nesting)					\
-			preempt_count_set(preempt_count() & ~NMI_MASK);	\
+		__preempt_count_nmi_exit();			\
  		arch_nmi_exit();				\
  		lockdep_on();					\
  	} while (0)
#89 lore raw

Re: [PATCH v4 10/17] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS

Hi Boqun.

Quoted 8 lines
Quoted 4 lines

nit: This limit is because to have the same behavior as other case when NMI_BITS=4 right?
It is not easy to infer that from comment.

It's sort of design by implementation IIUC, previously because of NMI_BITS=4, we could only support nesting level being 15. And here we just want to keep the same behavior here.

That I understood.

Quoted 3 lines

If your question is why 15 was a good number before this change, I guess would be it's just a number that is neither too big or too small.

Its more about below comment. Something like below is better?

/* NMI nesting is represented in 4 bits. */

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
include/linux/hardirq.h +4 -0
+		/* Maximum NMI nesting is 15. */		\
+		BUG_ON(__this_cpu_read(nmi_nesting) >= 15);	\
+		__this_cpu_inc(nmi_nesting);			\
+		preempt_count_set(preempt_count() | NMI_MASK);  \
author maintainer
#90 lore raw

Re: [PATCH v4 10/17] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS

Quoted 15 lines · Shrikanth Hegde

Hi Boqun.

Quoted 8 lines
Quoted 4 lines

nit: This limit is because to have the same behavior as other case when NMI_BITS=4 right?
It is not easy to infer that from comment.

It's sort of design by implementation IIUC, previously because of NMI_BITS=4, we could only support nesting level being 15. And here we just want to keep the same behavior here.

That I understood.

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

If your question is why 15 was a good number before this change, I guess would be it's just a number that is neither too big or too small.

Its more about below comment. Something like below is better?

/* NMI nesting is represented in 4 bits. */

Sounds good, I will apply this, thank you!

Regards,
Boqun

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
include/linux/hardirq.h +4 -0
+		/* Maximum NMI nesting is 15. */		\
+		BUG_ON(__this_cpu_read(nmi_nesting) >= 15);	\
+		__this_cpu_inc(nmi_nesting);			\
+		preempt_count_set(preempt_count() | NMI_MASK);  \
author maintainer
#91 lore raw

Re: [PATCH v4 10/17] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS

In reply to Boqun Feng
Quoted 113 lines · Boqun Feng
Quoted 38 lines · Shrikanth Hegde

Hi Boqun,

Quoted 8 lines · Boqun Feng

With the changes that enable preempt count to track IRQ disabling nesting, we don't have enough bits in 32-bit preempt count implementation, as a result we move NMI nesting bits out of the 32-bit preempt count. However on the architectures that can support 64-bit preempt count implementation, we can keep the NMI nesting bits in the 32-bit preempt count and avoid maintaining NMI nesting bits outside of the same cache line.

[...]

Quoted 24 lines
include/linux/hardirq.h +12 -2
@@ -10,8 +10,6 @@
  #include <linux/vtime.h>
  #include <asm/hardirq.h>
-DECLARE_PER_CPU(unsigned int, nmi_nesting);
-
  extern void synchronize_irq(unsigned int irq);
  extern bool synchronize_hardirq(unsigned int irq);
@@ -94,6 +92,37 @@ void irq_exit_rcu(void);
  #define arch_nmi_exit()		do { } while (0)
  #endif
+#ifdef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS
+static __always_inline void __preempt_count_nmi_enter(void)
+{
+	__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);
+}
+
+static __always_inline void __preempt_count_nmi_exit(void)
+{
+	__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);
+}
+#else
+DECLARE_PER_CPU(unsigned int, nmi_nesting);
Quoted 9 lines

include/linux/hardirq.h void irq_exit_rcu(void); · line 110

Quoted 4 lines
include/linux/hardirq.h +4 -0
+
+#define __preempt_count_nmi_enter()				\
+	do {							\
+		__preempt_count_add(HARDIRQ_OFFSET);		\

nit: This limit is because to have the same behavior as other case when NMI_BITS=4 right?
It is not easy to infer that from comment.

It's sort of design by implementation IIUC, previously because of NMI_BITS=4, we could only support nesting level being 15. And here we just want to keep the same behavior here.

If your question is why 15 was a good number before this change, I guess would be it's just a number that is neither too big or too small.

Quoted 9 lines

include/linux/hardirq.h void irq_exit_rcu(void); · line 114

Quoted 4 lines
include/linux/hardirq.h +4 -0
+		/* Maximum NMI nesting is 15. */		\
+		BUG_ON(__this_cpu_read(nmi_nesting) >= 15);	\
+		__this_cpu_inc(nmi_nesting);			\
+		preempt_count_set(preempt_count() | NMI_MASK);  \

Is there a reason preempt count updates are split rather than folded into a single preempt_count update?

Keeping the implementation simple is one reason, most architectures could utilize the HAS_SEPARATE_PREEMPT_RESCHED_BITS for better performance. So that reduces the importance of having something complicated but saves one access here. But if you see an optimization that can be done here, please do share!

Peter had proposed one optimization here:

	#define __preempt_count_nmi_enter()				\
		do {							\
			unsigned int _o = NMI_MASK + HARDIRQ_OFFSET;	\
			/* Maximum NMI nesting is 15. */		\
			BUG_ON(__this_cpu_read(nmi_nesting) >= 15);	\
			__this_cpu_inc(nmi_nesting);			\
			_o -= (preempt_count() & NMI_MASK);		\
			__preempt_count_add(_o);			\
		} while (0)
	
	#define __preempt_count_nmi_exit()				\
		do {							\
			unsigned int _o = HARDIRQ_OFFSET;		\
			if (!__this_cpu_dec_return(nmi_nesting))	\
				_o += NMI_MASK;				\
			__preempt_count_sub(_o);			\
		} while (0)

but it has a problem considering this:

	// outermost NMI handler
	// nmi_nesting == 0

	nmi_enter();
	// ^ nmi_nesting == 1 and NMI_MASK is set.
	...
	nmi_exit():
	  if (!__this_cpu_dec_return(nmi_nesting)) // return true
	    _o += NMI_MASK;
	  <NMI start>
	  nmi_enter();
	  // ^ nmi_nesting == 1 and NMI_MASK is set.
	  nmi_exit();
	  // ^ nmi_nesting == 0 and NMI_MASK is *unset*.
	  <NMI end>

	  preempt_count_sub(_o); // _o == HARDIRQ_OFFSET + NMI_MASK,
	  			 // underflow
Quoted 4 lines

(Now think about this, the __preempt_count_nmi_enter() does seems fine, maybe we can keep that, too tired to remember whether there is any subtly here... will take another look tomorrow)

Ok, now I remember the issue of the preempt_count_add() implemented __preempt_count_nmi_enter(), considering this:

// outermost NMI handler
// nmi_nesting == 0

nmi_enter():

	  __preempt_count_nmi_enter():
	    unsigned int _o = NMI_MASK + HARDIRQ_OFFSET;
	    ...
	    __this_cpu_inc(nmi_nesting);
	    // ^ nmi_nesting == 1
	    _o -= (preempt_count() & NMI_MASK);	
	    // ^ _o == NMI_MASK + HARDIRQ_OFFSET because the NMI_MASK bit was not set
	    <NMI start>
	    nmi_enter();
	    // ^ nmi_nesting == 2 and NMI_MASK is set.
	    nmi_exti();
	    // ^ nmi_nesting == 1 so NMI_MASK is *still* set
	    <NMI end>
	    __preempt_count_add(_o);	
	    // ^ NMI_MASK overflows because of the addition.

Make sense?

But maybe there are clever ways that I don't know. Please do tell!

Regards,
Boqun

Quoted 51 lines

Regards,
Boqun

Quoted 47 lines
Quoted 47 lines
include/linux/hardirq.h +14 -11
+	} while (0)
+
+#define __preempt_count_nmi_exit()				\
+	do {							\
+		__preempt_count_sub(HARDIRQ_OFFSET);		\
+		if (!__this_cpu_dec_return(nmi_nesting))	\
+			preempt_count_set(preempt_count() & ~NMI_MASK); \
+	} while (0)
+
+#endif
+
  /*
   * NMI vs Tracing
   * --------------
@@ -110,18 +139,14 @@ void irq_exit_rcu(void);
  	do {							\
  		lockdep_off();					\
  		arch_nmi_enter();				\
-		/* Maximum NMI nesting is 15. */		\
-		BUG_ON(__this_cpu_read(nmi_nesting) >= 15);	\
-		__this_cpu_inc(nmi_nesting);			\
-		__preempt_count_add(HARDIRQ_OFFSET);		\
-		preempt_count_set(preempt_count() | NMI_MASK);	\
+		__preempt_count_nmi_enter();			\
  	} while (0)
  #define nmi_enter()						\
  	do {							\
  		__nmi_enter();					\
  		lockdep_hardirq_enter();			\
-		ct_nmi_enter();				\
+		ct_nmi_enter();					\
  		instrumentation_begin();			\
  		ftrace_nmi_enter();				\
  		instrumentation_end();				\
@@ -129,12 +154,8 @@ void irq_exit_rcu(void);
  #define __nmi_exit()						\
  	do {							\
-		unsigned int nesting;				\
  		BUG_ON(!in_nmi());				\
-		__preempt_count_sub(HARDIRQ_OFFSET);		\
-		nesting = __this_cpu_dec_return(nmi_nesting);	\
-		if (!nesting)					\
-			preempt_count_set(preempt_count() & ~NMI_MASK);	\
+		__preempt_count_nmi_exit();			\
  		arch_nmi_exit();				\
  		lockdep_on();					\
  	} while (0)
#92 lore raw

Re: [PATCH v4 10/17] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS

In reply to Boqun Feng

Hi.

Quoted 25 lines
include/linux/preempt.h +15 -0
index 33fc4c814a9f..8299657f0f86 100644
@@ -34,14 +34,31 @@
   *         SOFTIRQ_MASK:	0x0000ff00
   * HARDIRQ_DISABLE_MASK:	0x00ff0000
   *         HARDIRQ_MASK:	0x0f000000
+ *
+ * When HAS_SEPARATE_PREEMPT_RESCHED_BITS=y, PREEMPT_NEED_RESCHED is put in a
+ * separate word and that allows 64bit load-store architectures to 'set'
+ * PREEMPT_NEED_RESCHED without messing up the otherwise symmetric
+ * modifications used on preempt_count and still load the whole thing
+ * (single-copy) atomically, without having to resort to full atomic
+ * operations.
+ *
+ * Because of the above, NMI_MASK bits are different depending on
+ * HAS_SEPARATE_PREEMPT_RESCHED_BITS:
+ *
+ * - HAS_SEPARATE_PREEMPT_RESCHED_BITS=n:
+ *
   *             NMI_MASK:	0x10000000
   * PREEMPT_NEED_RESCHED:	0x80000000
+ *
+ * - HAS_SEPARATE_PREEMPT_RESCHED_BITS=y:

include/linux/preempt.h line 62

Quoted 10 lines
include/linux/preempt.h +3 -1
+ *             NMI_MASK:	0xf0000000
+ * (PREEMPT_NEED_RESCHED is in a different word)
   */
  #define PREEMPT_BITS	8
  #define SOFTIRQ_BITS	8
  #define HARDIRQ_DISABLE_BITS	8
  #define HARDIRQ_BITS	4
-#define NMI_BITS	1
+#define NMI_BITS	(1 + 3*IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS))
 

Shouldn't testing/selftests/bpf/bpf_experimental.h also be updated with same?

Quoted 82 lines
include/linux/preempt.h +2 -2
  #define PREEMPT_SHIFT	0
  #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
@@ -116,8 +133,8 @@ static __always_inline unsigned char interrupt_context_level(void)
   * preempt_count() is commonly implemented with READ_ONCE().
   */
 
-#define nmi_count()	(preempt_count() & NMI_MASK)
-#define hardirq_count()	(preempt_count() & HARDIRQ_MASK)
+#define nmi_count()		(preempt_count() & NMI_MASK)
+#define hardirq_count()		(preempt_count() & HARDIRQ_MASK)
  #ifdef CONFIG_PREEMPT_RT
  # define softirq_count()	(current->softirq_disable_cnt & SOFTIRQ_MASK)
  # define irq_count()		((preempt_count() & (NMI_MASK | HARDIRQ_MASK)) | softirq_count())
kernel/Kconfig.preempt +4 -0
index 88c594c6d7fc..35f546a042b1 100644
@@ -122,6 +122,10 @@ config PREEMPT_RT_NEEDS_BH_LOCK
  config PREEMPT_COUNT
         bool
 
+config HAS_SEPARATE_PREEMPT_RESCHED_BITS
+	bool
+	depends on PREEMPT_COUNT && 64BIT
+
  config PREEMPTION
         bool
         select PREEMPT_COUNT
kernel/sched/core.c +10 -2
index 9b3f1764fa9e..6d88343c3bad 100644
@@ -5973,8 +5973,13 @@ void preempt_count_add(int val)
  #ifdef CONFIG_DEBUG_PREEMPT
  	/*
  	 * Underflow?
+	 *
+	 * Cannot detect underflow based on the current preempt_count() value
+	 * if using HAS_SEPARATE_PREEMPT_RESCHED_BITS because preempt count takes all 32
+	 * bits.
  	 */
-	if (DEBUG_LOCKS_WARN_ON((preempt_count() < 0)))
+	if (!IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS) &&
+	    DEBUG_LOCKS_WARN_ON((preempt_count() < 0)))
  		return;
  #endif
  	__preempt_count_add(val);
@@ -6006,7 +6011,10 @@ void preempt_count_sub(int val)
  	/*
  	 * Underflow?
  	 */
-	if (DEBUG_LOCKS_WARN_ON(val > preempt_count()))
+	unsigned int uval = val;
+	unsigned int pc = preempt_count();
+
+	if (DEBUG_LOCKS_WARN_ON(pc - uval > pc))
  		return;
  	/*
  	 * Is the spinlock portion underflowing?
kernel/softirq.c +6 -0
index 0c9b2269a8d6..7980a4a232f9 100644
@@ -103,7 +103,13 @@ void _local_interrupt_enable(void)
  }
  EXPORT_SYMBOL(_local_interrupt_enable);
 
+#ifndef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS
+/*
+ * Any 32bit architecture that still cares about performance should
+ * probably ensure this is near preempt_count.
+ */
  DEFINE_PER_CPU(unsigned int, nmi_nesting);
+#endif
 
  /*
   * SOFTIRQ_OFFSET usage:
lib/locking-selftest.c +0 -0
index bfafe1204c7b..c3d976c801bb 100644

lib/locking-selftest.c static int unexpected_testcase_failures; · line 1432

Quoted 5 lines
lib/locking-selftest.c +1 -1
@@ -1429,7 +1429,7 @@ static int unexpected_testcase_failures;
 
  static void dotest(void (*testcase_fn)(void), int expected, int lockclass_mask)
  {
-	int saved_preempt_count = preempt_count();
+	long saved_preempt_count = preempt_count();

nit: Isn't preempt_count still returns int?

Quoted 3 lines
lib/locking-selftest.c +0 -0
  #ifdef CONFIG_PREEMPT_RT
  	int saved_mgd_count = current->migration_disabled;
  	int saved_rcu_count = current->rcu_read_lock_nesting;
author maintainer
#93 lore raw

Re: [PATCH v4 10/17] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS

Quoted 32 lines · Shrikanth Hegde

Hi.

Quoted 30 lines
include/linux/preempt.h +17 -0
index 33fc4c814a9f..8299657f0f86 100644
@@ -34,14 +34,31 @@
   *         SOFTIRQ_MASK:	0x0000ff00
   * HARDIRQ_DISABLE_MASK:	0x00ff0000
   *         HARDIRQ_MASK:	0x0f000000
+ *
+ * When HAS_SEPARATE_PREEMPT_RESCHED_BITS=y, PREEMPT_NEED_RESCHED is put in a
+ * separate word and that allows 64bit load-store architectures to 'set'
+ * PREEMPT_NEED_RESCHED without messing up the otherwise symmetric
+ * modifications used on preempt_count and still load the whole thing
+ * (single-copy) atomically, without having to resort to full atomic
+ * operations.
+ *
+ * Because of the above, NMI_MASK bits are different depending on
+ * HAS_SEPARATE_PREEMPT_RESCHED_BITS:
+ *
+ * - HAS_SEPARATE_PREEMPT_RESCHED_BITS=n:
+ *
   *             NMI_MASK:	0x10000000
   * PREEMPT_NEED_RESCHED:	0x80000000
+ *
+ * - HAS_SEPARATE_PREEMPT_RESCHED_BITS=y:
+ *             NMI_MASK:	0xf0000000
+ * (PREEMPT_NEED_RESCHED is in a different word)
   */
  #define PREEMPT_BITS	8
  #define SOFTIRQ_BITS	8
Quoted 9 lines

include/linux/preempt.h line 61

Quoted 4 lines
include/linux/preempt.h +1 -1
  #define HARDIRQ_DISABLE_BITS	8
  #define HARDIRQ_BITS	4
-#define NMI_BITS	1
+#define NMI_BITS	(1 + 3*IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS))

Shouldn't testing/selftests/bpf/bpf_experimental.h also be updated with same?

Yeah, sashiko found out the same thing.

Quoted 77 lines
Quoted 77 lines
include/linux/preempt.h +2 -2
  #define PREEMPT_SHIFT	0
  #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
@@ -116,8 +133,8 @@ static __always_inline unsigned char interrupt_context_level(void)
   * preempt_count() is commonly implemented with READ_ONCE().
   */
-#define nmi_count()	(preempt_count() & NMI_MASK)
-#define hardirq_count()	(preempt_count() & HARDIRQ_MASK)
+#define nmi_count()		(preempt_count() & NMI_MASK)
+#define hardirq_count()		(preempt_count() & HARDIRQ_MASK)
  #ifdef CONFIG_PREEMPT_RT
  # define softirq_count()	(current->softirq_disable_cnt & SOFTIRQ_MASK)
  # define irq_count()		((preempt_count() & (NMI_MASK | HARDIRQ_MASK)) | softirq_count())
kernel/Kconfig.preempt +4 -0
index 88c594c6d7fc..35f546a042b1 100644
@@ -122,6 +122,10 @@ config PREEMPT_RT_NEEDS_BH_LOCK
  config PREEMPT_COUNT
         bool
+config HAS_SEPARATE_PREEMPT_RESCHED_BITS
+	bool
+	depends on PREEMPT_COUNT && 64BIT
+
  config PREEMPTION
         bool
         select PREEMPT_COUNT
kernel/sched/core.c +10 -2
index 9b3f1764fa9e..6d88343c3bad 100644
@@ -5973,8 +5973,13 @@ void preempt_count_add(int val)
  #ifdef CONFIG_DEBUG_PREEMPT
  	/*
  	 * Underflow?
+	 *
+	 * Cannot detect underflow based on the current preempt_count() value
+	 * if using HAS_SEPARATE_PREEMPT_RESCHED_BITS because preempt count takes all 32
+	 * bits.
  	 */
-	if (DEBUG_LOCKS_WARN_ON((preempt_count() < 0)))
+	if (!IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS) &&
+	    DEBUG_LOCKS_WARN_ON((preempt_count() < 0)))
  		return;
  #endif
  	__preempt_count_add(val);
@@ -6006,7 +6011,10 @@ void preempt_count_sub(int val)
  	/*
  	 * Underflow?
  	 */
-	if (DEBUG_LOCKS_WARN_ON(val > preempt_count()))
+	unsigned int uval = val;
+	unsigned int pc = preempt_count();
+
+	if (DEBUG_LOCKS_WARN_ON(pc - uval > pc))
  		return;
  	/*
  	 * Is the spinlock portion underflowing?
kernel/softirq.c +6 -0
index 0c9b2269a8d6..7980a4a232f9 100644
@@ -103,7 +103,13 @@ void _local_interrupt_enable(void)
  }
  EXPORT_SYMBOL(_local_interrupt_enable);
+#ifndef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS
+/*
+ * Any 32bit architecture that still cares about performance should
+ * probably ensure this is near preempt_count.
+ */
  DEFINE_PER_CPU(unsigned int, nmi_nesting);
+#endif
  /*
   * SOFTIRQ_OFFSET usage:
lib/locking-selftest.c +0 -1
index bfafe1204c7b..c3d976c801bb 100644
--- a/lib/locking-selftest.c
Quoted 9 lines

lib/locking-selftest.c static int unexpected_testcase_failures; · line 1431

Quoted 6 lines

+++ b/lib/locking-selftest.c

lib/locking-selftest.c +1 -1
@@ -1429,7 +1429,7 @@ static int unexpected_testcase_failures;
  static void dotest(void (*testcase_fn)(void), int expected, int lockclass_mask)
  {
-	int saved_preempt_count = preempt_count();
+	long saved_preempt_count = preempt_count();

nit: Isn't preempt_count still returns int?

Right, let me fix that. Thank you!

Regards,
Boqun

Quoted 4 lines
Quoted 3 lines
lib/locking-selftest.c +0 -0
  #ifdef CONFIG_PREEMPT_RT
  	int saved_mgd_count = current->migration_disabled;
  	int saved_rcu_count = current->rcu_read_lock_nesting;
merge bot Applied
#94 lore raw

[tip: locking/core] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS

In reply to Boqun Feng
The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     a1ad9bd8e22a26a15f57100e9feef92386808ebe
Gitweb:        https://git.kernel.org/tip/a1ad9bd8e22a26a15f57100e9feef92386808ebe
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:32 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 08 Aug 2026 22:44:08 +02:00

preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS

With the changes that enable preempt count to track IRQ disabling nesting, we don't have enough bits in 32-bit preempt count implementation, as a result we move NMI nesting bits out of the 32-bit preempt count. However on the architectures that can support 64-bit preempt count implementation, we can keep the NMI nesting bits in the 32-bit preempt count and avoid maintaining NMI nesting bits outside of the same cache line.

Therefore HAS_SEPARATE_PREEMPT_RESCHED_BITS is introduced to allow architectures to select this. Note that under this Kconfig, preempt count is maintained in a 64-bit word however preempt_count() still remains as an int because all the effective bits still fit in (previously we mask out NEED_RESCHED bit in preempt_count()). This should make no functional changes for existing preempt_count() users.

Enable this for x86_64 along with the introduction of the Kconfig.

[boqun: Undo the __preempt_count_{add,sub}() optimization in 32-bit preempt count since it may introduce {over,under}flow]

---
 arch/x86/Kconfig               |  1 +-
 arch/x86/include/asm/preempt.h | 55 ++++++++++++++++++++++-----------
 arch/x86/kernel/cpu/common.c   |  2 +-
 include/linux/hardirq.h        | 47 ++++++++++++++++++++--------
 include/linux/preempt.h        | 23 ++++++++++++--
 kernel/Kconfig.preempt         |  4 ++-
 kernel/sched/core.c            | 12 +++++--
 kernel/softirq.c               |  6 ++++-
 lib/locking-selftest.c         |  2 +-
 9 files changed, 115 insertions(+), 37 deletions(-)
arch/x86/Kconfig +1 -0
index bdad90f..6a7067d 100644
@@ -326,6 +326,7 @@ config X86
 	select USER_STACKTRACE_SUPPORT
 	select HAVE_ARCH_KCSAN			if X86_64
 	select PROC_PID_ARCH_STATUS		if PROC_FS
+	select HAS_SEPARATE_PREEMPT_RESCHED_BITS		if X86_64 && PREEMPT_COUNT
 	select HAVE_ARCH_NODE_DEV_GROUP		if X86_SGX
 	select FUNCTION_ALIGNMENT_16B		if X86_64 || X86_ALIGNMENT_16
 	select FUNCTION_ALIGNMENT_4B
arch/x86/include/asm/preempt.h +38 -17

123 lines not on this page · raw message

arch/x86/kernel/cpu/common.c +1 -1
index a3df21d..73a6d9f 100644
@@ -2236,7 +2236,7 @@ DEFINE_PER_CPU_CACHE_HOT(struct task_struct *, current_task) = &init_task;
 EXPORT_PER_CPU_SYMBOL(current_task);
 EXPORT_PER_CPU_SYMBOL(const_current_task);
 
-DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) = INIT_PREEMPT_COUNT;
+DEFINE_PER_CPU_CACHE_HOT(unsigned long, __preempt_count) = INIT_PREEMPT_COUNT;
 EXPORT_PER_CPU_SYMBOL(__preempt_count);
 
 DEFINE_PER_CPU_CACHE_HOT(unsigned long, cpu_current_top_of_stack) = TOP_OF_INIT_STACK;
include/linux/hardirq.h +34 -13

78 lines not on this page · raw message

include/linux/preempt.h +20 -3
index 33fc4c8..8299657 100644
@@ -34,14 +34,31 @@
  *         SOFTIRQ_MASK:	0x0000ff00
  * HARDIRQ_DISABLE_MASK:	0x00ff0000
  *         HARDIRQ_MASK:	0x0f000000
+ *
+ * When HAS_SEPARATE_PREEMPT_RESCHED_BITS=y, PREEMPT_NEED_RESCHED is put in a
+ * separate word and that allows 64bit load-store architectures to 'set'
+ * PREEMPT_NEED_RESCHED without messing up the otherwise symmetric
+ * modifications used on preempt_count and still load the whole thing
+ * (single-copy) atomically, without having to resort to full atomic
+ * operations.
+ *
+ * Because of the above, NMI_MASK bits are different depending on
+ * HAS_SEPARATE_PREEMPT_RESCHED_BITS:
+ *
+ * - HAS_SEPARATE_PREEMPT_RESCHED_BITS=n:
+ *
  *             NMI_MASK:	0x10000000
  * PREEMPT_NEED_RESCHED:	0x80000000
+ *
+ * - HAS_SEPARATE_PREEMPT_RESCHED_BITS=y:
+ *             NMI_MASK:	0xf0000000
+ * (PREEMPT_NEED_RESCHED is in a different word)
  */
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
 #define HARDIRQ_DISABLE_BITS	8
 #define HARDIRQ_BITS	4
-#define NMI_BITS	1
+#define NMI_BITS	(1 + 3*IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS))
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
@@ -116,8 +133,8 @@ static __always_inline unsigned char interrupt_context_level(void)
  * preempt_count() is commonly implemented with READ_ONCE().
  */
 
-#define nmi_count()	(preempt_count() & NMI_MASK)
-#define hardirq_count()	(preempt_count() & HARDIRQ_MASK)
+#define nmi_count()		(preempt_count() & NMI_MASK)
+#define hardirq_count()		(preempt_count() & HARDIRQ_MASK)
 #ifdef CONFIG_PREEMPT_RT
 # define softirq_count()	(current->softirq_disable_cnt & SOFTIRQ_MASK)
 # define irq_count()		((preempt_count() & (NMI_MASK | HARDIRQ_MASK)) | softirq_count())
kernel/Kconfig.preempt +4 -0
index 88c594c..35f546a 100644
@@ -122,6 +122,10 @@ config PREEMPT_RT_NEEDS_BH_LOCK
 config PREEMPT_COUNT
        bool
 
+config HAS_SEPARATE_PREEMPT_RESCHED_BITS
+	bool
+	depends on PREEMPT_COUNT && 64BIT
+
 config PREEMPTION
        bool
        select PREEMPT_COUNT
kernel/sched/core.c +10 -2
index 9b3f176..6d88343 100644
@@ -5973,8 +5973,13 @@ void preempt_count_add(int val)
 #ifdef CONFIG_DEBUG_PREEMPT
 	/*
 	 * Underflow?
+	 *
+	 * Cannot detect underflow based on the current preempt_count() value
+	 * if using HAS_SEPARATE_PREEMPT_RESCHED_BITS because preempt count takes all 32
+	 * bits.
 	 */
-	if (DEBUG_LOCKS_WARN_ON((preempt_count() < 0)))
+	if (!IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS) &&
+	    DEBUG_LOCKS_WARN_ON((preempt_count() < 0)))
 		return;
 #endif
 	__preempt_count_add(val);
@@ -6006,7 +6011,10 @@ void preempt_count_sub(int val)
 	/*
 	 * Underflow?
 	 */
-	if (DEBUG_LOCKS_WARN_ON(val > preempt_count()))
+	unsigned int uval = val;
+	unsigned int pc = preempt_count();
+
+	if (DEBUG_LOCKS_WARN_ON(pc - uval > pc))
 		return;
 	/*
 	 * Is the spinlock portion underflowing?
kernel/softirq.c +6 -0
index 0c9b226..7980a4a 100644
@@ -103,7 +103,13 @@ void _local_interrupt_enable(void)
 }
 EXPORT_SYMBOL(_local_interrupt_enable);
 
+#ifndef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS
+/*
+ * Any 32bit architecture that still cares about performance should
+ * probably ensure this is near preempt_count.
+ */
 DEFINE_PER_CPU(unsigned int, nmi_nesting);
+#endif
 
 /*
  * SOFTIRQ_OFFSET usage:
lib/locking-selftest.c +1 -1
index bfafe12..c3d976c 100644
@@ -1429,7 +1429,7 @@ static int unexpected_testcase_failures;
 
 static void dotest(void (*testcase_fn)(void), int expected, int lockclass_mask)
 {
-	int saved_preempt_count = preempt_count();
+	long saved_preempt_count = preempt_count();
 #ifdef CONFIG_PREEMPT_RT
 	int saved_mgd_count = current->migration_disabled;
 	int saved_rcu_count = current->rcu_read_lock_nesting;
merge bot Applied
#95 lore raw

[tip: locking/core] preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS

In reply to Boqun Feng
The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     3b0e2a22d4086ed7c1294584c4417f7d19f1ac67
Gitweb:        https://git.kernel.org/tip/3b0e2a22d4086ed7c1294584c4417f7d19f1ac67
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:32 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 10 Aug 2026 10:50:19 +02:00

preempt: Introduce HAS_SEPARATE_PREEMPT_RESCHED_BITS

With the changes that enable preempt count to track IRQ disabling nesting, we don't have enough bits in 32-bit preempt count implementation, as a result we move NMI nesting bits out of the 32-bit preempt count. However on the architectures that can support 64-bit preempt count implementation, we can keep the NMI nesting bits in the 32-bit preempt count and avoid maintaining NMI nesting bits outside of the same cache line.

Therefore HAS_SEPARATE_PREEMPT_RESCHED_BITS is introduced to allow architectures to select this. Note that under this Kconfig, preempt count is maintained in a 64-bit word however preempt_count() still remains as an int because all the effective bits still fit in (previously we mask out NEED_RESCHED bit in preempt_count()). This should make no functional changes for existing preempt_count() users.

Enable this for x86_64 along with the introduction of the Kconfig.

[boqun: Undo the __preempt_count_{add,sub}() optimization in 32-bit preempt count since it may introduce {over,under}flow]

---
 arch/x86/Kconfig               |  1 +-
 arch/x86/include/asm/preempt.h | 55 ++++++++++++++++++++++-----------
 arch/x86/kernel/cpu/common.c   |  2 +-
 include/linux/hardirq.h        | 47 ++++++++++++++++++++--------
 include/linux/preempt.h        | 23 ++++++++++++--
 kernel/Kconfig.preempt         |  4 ++-
 kernel/sched/core.c            | 12 +++++--
 kernel/softirq.c               |  6 ++++-
 lib/locking-selftest.c         |  2 +-
 9 files changed, 115 insertions(+), 37 deletions(-)
arch/x86/Kconfig +1 -0
index bdad90f..6a7067d 100644
@@ -326,6 +326,7 @@ config X86
 	select USER_STACKTRACE_SUPPORT
 	select HAVE_ARCH_KCSAN			if X86_64
 	select PROC_PID_ARCH_STATUS		if PROC_FS
+	select HAS_SEPARATE_PREEMPT_RESCHED_BITS		if X86_64 && PREEMPT_COUNT
 	select HAVE_ARCH_NODE_DEV_GROUP		if X86_SGX
 	select FUNCTION_ALIGNMENT_16B		if X86_64 || X86_ALIGNMENT_16
 	select FUNCTION_ALIGNMENT_4B
arch/x86/include/asm/preempt.h +38 -17

123 lines not on this page · raw message

arch/x86/kernel/cpu/common.c +1 -1
index a3df21d..73a6d9f 100644
@@ -2236,7 +2236,7 @@ DEFINE_PER_CPU_CACHE_HOT(struct task_struct *, current_task) = &init_task;
 EXPORT_PER_CPU_SYMBOL(current_task);
 EXPORT_PER_CPU_SYMBOL(const_current_task);
 
-DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) = INIT_PREEMPT_COUNT;
+DEFINE_PER_CPU_CACHE_HOT(unsigned long, __preempt_count) = INIT_PREEMPT_COUNT;
 EXPORT_PER_CPU_SYMBOL(__preempt_count);
 
 DEFINE_PER_CPU_CACHE_HOT(unsigned long, cpu_current_top_of_stack) = TOP_OF_INIT_STACK;
include/linux/hardirq.h +34 -13

78 lines not on this page · raw message

include/linux/preempt.h +20 -3
index 33fc4c8..8299657 100644
@@ -34,14 +34,31 @@
  *         SOFTIRQ_MASK:	0x0000ff00
  * HARDIRQ_DISABLE_MASK:	0x00ff0000
  *         HARDIRQ_MASK:	0x0f000000
+ *
+ * When HAS_SEPARATE_PREEMPT_RESCHED_BITS=y, PREEMPT_NEED_RESCHED is put in a
+ * separate word and that allows 64bit load-store architectures to 'set'
+ * PREEMPT_NEED_RESCHED without messing up the otherwise symmetric
+ * modifications used on preempt_count and still load the whole thing
+ * (single-copy) atomically, without having to resort to full atomic
+ * operations.
+ *
+ * Because of the above, NMI_MASK bits are different depending on
+ * HAS_SEPARATE_PREEMPT_RESCHED_BITS:
+ *
+ * - HAS_SEPARATE_PREEMPT_RESCHED_BITS=n:
+ *
  *             NMI_MASK:	0x10000000
  * PREEMPT_NEED_RESCHED:	0x80000000
+ *
+ * - HAS_SEPARATE_PREEMPT_RESCHED_BITS=y:
+ *             NMI_MASK:	0xf0000000
+ * (PREEMPT_NEED_RESCHED is in a different word)
  */
 #define PREEMPT_BITS	8
 #define SOFTIRQ_BITS	8
 #define HARDIRQ_DISABLE_BITS	8
 #define HARDIRQ_BITS	4
-#define NMI_BITS	1
+#define NMI_BITS	(1 + 3*IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS))
 
 #define PREEMPT_SHIFT	0
 #define SOFTIRQ_SHIFT	(PREEMPT_SHIFT + PREEMPT_BITS)
@@ -116,8 +133,8 @@ static __always_inline unsigned char interrupt_context_level(void)
  * preempt_count() is commonly implemented with READ_ONCE().
  */
 
-#define nmi_count()	(preempt_count() & NMI_MASK)
-#define hardirq_count()	(preempt_count() & HARDIRQ_MASK)
+#define nmi_count()		(preempt_count() & NMI_MASK)
+#define hardirq_count()		(preempt_count() & HARDIRQ_MASK)
 #ifdef CONFIG_PREEMPT_RT
 # define softirq_count()	(current->softirq_disable_cnt & SOFTIRQ_MASK)
 # define irq_count()		((preempt_count() & (NMI_MASK | HARDIRQ_MASK)) | softirq_count())
kernel/Kconfig.preempt +4 -0
index 88c594c..35f546a 100644
@@ -122,6 +122,10 @@ config PREEMPT_RT_NEEDS_BH_LOCK
 config PREEMPT_COUNT
        bool
 
+config HAS_SEPARATE_PREEMPT_RESCHED_BITS
+	bool
+	depends on PREEMPT_COUNT && 64BIT
+
 config PREEMPTION
        bool
        select PREEMPT_COUNT
kernel/sched/core.c +10 -2
index 9b3f176..6d88343 100644
@@ -5973,8 +5973,13 @@ void preempt_count_add(int val)
 #ifdef CONFIG_DEBUG_PREEMPT
 	/*
 	 * Underflow?
+	 *
+	 * Cannot detect underflow based on the current preempt_count() value
+	 * if using HAS_SEPARATE_PREEMPT_RESCHED_BITS because preempt count takes all 32
+	 * bits.
 	 */
-	if (DEBUG_LOCKS_WARN_ON((preempt_count() < 0)))
+	if (!IS_ENABLED(CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS) &&
+	    DEBUG_LOCKS_WARN_ON((preempt_count() < 0)))
 		return;
 #endif
 	__preempt_count_add(val);
@@ -6006,7 +6011,10 @@ void preempt_count_sub(int val)
 	/*
 	 * Underflow?
 	 */
-	if (DEBUG_LOCKS_WARN_ON(val > preempt_count()))
+	unsigned int uval = val;
+	unsigned int pc = preempt_count();
+
+	if (DEBUG_LOCKS_WARN_ON(pc - uval > pc))
 		return;
 	/*
 	 * Is the spinlock portion underflowing?
kernel/softirq.c +6 -0
index 0c9b226..7980a4a 100644
@@ -103,7 +103,13 @@ void _local_interrupt_enable(void)
 }
 EXPORT_SYMBOL(_local_interrupt_enable);
 
+#ifndef CONFIG_HAS_SEPARATE_PREEMPT_RESCHED_BITS
+/*
+ * Any 32bit architecture that still cares about performance should
+ * probably ensure this is near preempt_count.
+ */
 DEFINE_PER_CPU(unsigned int, nmi_nesting);
+#endif
 
 /*
  * SOFTIRQ_OFFSET usage:
lib/locking-selftest.c +1 -1
index bfafe12..c3d976c 100644
@@ -1429,7 +1429,7 @@ static int unexpected_testcase_failures;
 
 static void dotest(void (*testcase_fn)(void), int expected, int lockclass_mask)
 {
-	int saved_preempt_count = preempt_count();
+	long saved_preempt_count = preempt_count();
 #ifdef CONFIG_PREEMPT_RT
 	int saved_mgd_count = current->migration_disabled;
 	int saved_rcu_count = current->rcu_read_lock_nesting;
author maintainer
#96 lore raw

PATCH v4 11/17 arm64: sched/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

In reply to Boqun Feng

Arm64 already uses 64-bit preempt count and the need reschedule bit is maintained in a separate 32-bit word from the preempt count. Therefore preempt count has enough bits to represent 16 levels of NMI nesting, hence enable it for arm64. This saves a per-CPU variable and additional instructions in the NMI path.

---
 arch/arm64/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/Kconfig +1 -0
index b3afe0688919..349c3533cd1e 100644
@@ -247,6 +247,7 @@ config ARM64
 	select PCI_SYSCALL if PCI
 	select POWER_RESET
 	select POWER_SUPPLY
+	select HAS_SEPARATE_PREEMPT_RESCHED_BITS
 	select SPARSE_IRQ
 	select SWIOTLB
 	select SYSCTL_EXCEPTION_TRACE
Signature · 1 line
2.50.1 (Apple Git-155)
merge bot Applied
#97 lore raw

[tip: locking/core] arm64: sched/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     c9bae17d96b93ce05ff2eeef08165c24c7f5e7e3
Gitweb:        https://git.kernel.org/tip/c9bae17d96b93ce05ff2eeef08165c24c7f5e7e3
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:33 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 08 Aug 2026 22:44:08 +02:00

arm64: sched/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

Arm64 already uses 64-bit preempt count and the need reschedule bit is maintained in a separate 32-bit word from the preempt count. Therefore preempt count has enough bits to represent 16 levels of NMI nesting, hence enable it for arm64. This saves a per-CPU variable and additional instructions in the NMI path.

---
 arch/arm64/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/Kconfig +1 -0
index b3afe06..349c353 100644
@@ -247,6 +247,7 @@ config ARM64
 	select PCI_SYSCALL if PCI
 	select POWER_RESET
 	select POWER_SUPPLY
+	select HAS_SEPARATE_PREEMPT_RESCHED_BITS
 	select SPARSE_IRQ
 	select SWIOTLB
 	select SYSCTL_EXCEPTION_TRACE
merge bot Applied
#98 lore raw

[tip: locking/core] arm64: sched/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

In reply to Boqun Feng
The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     264bbd32a31ba80e634bec5c1df8514c0e97fe7e
Gitweb:        https://git.kernel.org/tip/264bbd32a31ba80e634bec5c1df8514c0e97fe7e
Author:        Boqun Feng <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:33 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 10 Aug 2026 10:50:20 +02:00

arm64: sched/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

Arm64 already uses 64-bit preempt count and the need reschedule bit is maintained in a separate 32-bit word from the preempt count. Therefore preempt count has enough bits to represent 16 levels of NMI nesting, hence enable it for arm64. This saves a per-CPU variable and additional instructions in the NMI path.

---
 arch/arm64/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/Kconfig +1 -0
index b3afe06..349c353 100644
@@ -247,6 +247,7 @@ config ARM64
 	select PCI_SYSCALL if PCI
 	select POWER_RESET
 	select POWER_SUPPLY
+	select HAS_SEPARATE_PREEMPT_RESCHED_BITS
 	select SPARSE_IRQ
 	select SWIOTLB
 	select SYSCTL_EXCEPTION_TRACE
author maintainer
#99 lore raw

PATCH v4 12/17 s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

In reply to Boqun Feng

From: Heiko Carstens <[email protected]>

Convert s390's preempt_count to 64 bit, and change the preempt primitives accordingly.

[boqun: Apply the corrected comment for asm block]
---
 arch/s390/Kconfig               |  1 +
 arch/s390/include/asm/lowcore.h | 13 +++++++---
 arch/s390/include/asm/preempt.h | 43 +++++++++++++++------------------
 3 files changed, 30 insertions(+), 27 deletions(-)
arch/s390/Kconfig +1 -0
index 84404e6778d5..378fcd2b6181 100644
@@ -273,6 +273,7 @@ config S390
 	select PCI_MSI			if PCI
 	select PCI_MSI_ARCH_FALLBACKS	if PCI_MSI
 	select PCI_QUIRKS		if PCI
+	select HAS_SEPARATE_PREEMPT_RESCHED_BITS
 	select SPARSE_IRQ
 	select SWIOTLB
 	select SYSCTL_EXCEPTION_TRACE
arch/s390/include/asm/lowcore.h +9 -4
index 3b3ecc647993..5cef215d30e7 100644
@@ -160,10 +160,15 @@ struct lowcore {
 	/* SMP info area */
 	__u32	cpu_nr;				/* 0x03a0 */
 	__u32	softirq_pending;		/* 0x03a4 */
-	__s32	preempt_count;			/* 0x03a8 */
-	__u32	spinlock_lockval;		/* 0x03ac */
-	__u32	spinlock_index;			/* 0x03b0 */
-	__u8	pad_0x03b4[0x03b8-0x03b4];	/* 0x03b4 */
+	union {
+		struct {
+			__u32	need_resched;	/* 0x03a8 */
+			__u32	count;		/* 0x03ac */
+		} preempt;
+		__u64	preempt_count;		/* 0x03a8 */
+	};
+	__u32	spinlock_lockval;		/* 0x03b0 */
+	__u32	spinlock_index;			/* 0x03b4 */
 	__u64	percpu_offset;			/* 0x03b8 */
 	__u8	percpu_register;		/* 0x03c0 */
 	__u8	pad_0x03c1[0x0400-0x03c1];	/* 0x03c1 */
arch/s390/include/asm/preempt.h +20 -23

105 lines not on this page · raw message

Signature · 1 line
2.50.1 (Apple Git-155)
#100 lore raw

Re: [PATCH v4 12/17] s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

+maddy, christophe,

arch/s390/include/asm/lowcore.h struct lowcore { · line 169

Quoted 13 lines · Boqun Feng

From: Heiko Carstens <[email protected]>

Convert s390's preempt_count to 64 bit, and change the preempt primitives accordingly.

arch/s390/include/asm/lowcore.h +7 -0
+	union {
+		struct {
+			__u32	need_resched;	/* 0x03a8 */
+			__u32	count;		/* 0x03ac */
+		} preempt;
+		__u64	preempt_count;		/* 0x03a8 */
+	};

This is interesting. So arm64, s390 will have similar implementation w.r.t to preempt_count.

I guess PoC that i was trying for ppc64 is worth pursuing even more, since ppc64 is one missing major arch which still uses old method of querying tif_need_resched.
With similar change ppc64 can set HAS_SEPARATE_PREEMPT_RESCHED_BITS too.

maintainer
#101 lore raw

Re: [PATCH v4 12/17] s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

Quoted 21 lines · Shrikanth Hegde

+maddy, christophe,

arch/s390/include/asm/lowcore.h struct lowcore { · line 169

Quoted 13 lines · Boqun Feng

From: Heiko Carstens <[email protected]>

Convert s390's preempt_count to 64 bit, and change the preempt primitives accordingly.

arch/s390/include/asm/lowcore.h +7 -0
+	union {
+		struct {
+			__u32	need_resched;	/* 0x03a8 */
+			__u32	count;		/* 0x03ac */
+		} preempt;
+		__u64	preempt_count;		/* 0x03a8 */
+	};

This is interesting. So arm64, s390 will have similar implementation w.r.t to preempt_count.

Quoted 4 lines

I guess PoC that i was trying for ppc64 is worth pursuing even more, since ppc64 is one missing major arch which still uses old method of querying tif_need_resched.
With similar change ppc64 can set HAS_SEPARATE_PREEMPT_RESCHED_BITS too.

Yes. PPC should be able to the same thing ARM64 does.

#102 lore raw

Re: [PATCH v4 12/17] s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

Hi Peter.

Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Shrikanth Hegde

+maddy, christophe,

arch/s390/include/asm/lowcore.h struct lowcore { · line 169

Quoted 13 lines · Boqun Feng

From: Heiko Carstens <[email protected]>

Convert s390's preempt_count to 64 bit, and change the preempt primitives accordingly.

arch/s390/include/asm/lowcore.h +7 -0
+	union {
+		struct {
+			__u32	need_resched;	/* 0x03a8 */
+			__u32	count;		/* 0x03ac */
+		} preempt;
+		__u64	preempt_count;		/* 0x03a8 */
+	};

This is interesting. So arm64, s390 will have similar implementation w.r.t to preempt_count.

Quoted 6 lines
Quoted 4 lines

I guess PoC that i was trying for ppc64 is worth pursuing even more, since ppc64 is one missing major arch which still uses old method of querying tif_need_resched.
With similar change ppc64 can set HAS_SEPARATE_PREEMPT_RESCHED_BITS too.

Yes. PPC should be able to the same thing ARM64 does.

Yes. I am thinking of moving it to PACA instead of keeping it in thread_info as PACA is a natural per-CPU storage.

If i hit any major hurdle there, will use thread info similar to arm64.

merge bot Applied
#103 lore raw

[tip: locking/core] s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

In reply to Boqun Feng
The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     c915500b3a095ee7647337f0ed13efe8b1e6e99e
Gitweb:        https://git.kernel.org/tip/c915500b3a095ee7647337f0ed13efe8b1e6e99e
Author:        Heiko Carstens <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:34 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Sat, 08 Aug 2026 22:44:09 +02:00

s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

Convert s390's preempt_count to 64 bit, and change the preempt primitives accordingly.

---
 arch/s390/Kconfig               |  1 +-
 arch/s390/include/asm/lowcore.h | 13 ++++++----
 arch/s390/include/asm/preempt.h | 43 ++++++++++++++------------------
 3 files changed, 30 insertions(+), 27 deletions(-)
arch/s390/Kconfig +1 -0
index 84404e6..378fcd2 100644
@@ -273,6 +273,7 @@ config S390
 	select PCI_MSI			if PCI
 	select PCI_MSI_ARCH_FALLBACKS	if PCI_MSI
 	select PCI_QUIRKS		if PCI
+	select HAS_SEPARATE_PREEMPT_RESCHED_BITS
 	select SPARSE_IRQ
 	select SWIOTLB
 	select SYSCTL_EXCEPTION_TRACE
arch/s390/include/asm/lowcore.h +9 -4
index 3b3ecc6..5cef215 100644
@@ -160,10 +160,15 @@ struct lowcore {
 	/* SMP info area */
 	__u32	cpu_nr;				/* 0x03a0 */
 	__u32	softirq_pending;		/* 0x03a4 */
-	__s32	preempt_count;			/* 0x03a8 */
-	__u32	spinlock_lockval;		/* 0x03ac */
-	__u32	spinlock_index;			/* 0x03b0 */
-	__u8	pad_0x03b4[0x03b8-0x03b4];	/* 0x03b4 */
+	union {
+		struct {
+			__u32	need_resched;	/* 0x03a8 */
+			__u32	count;		/* 0x03ac */
+		} preempt;
+		__u64	preempt_count;		/* 0x03a8 */
+	};
+	__u32	spinlock_lockval;		/* 0x03b0 */
+	__u32	spinlock_index;			/* 0x03b4 */
 	__u64	percpu_offset;			/* 0x03b8 */
 	__u8	percpu_register;		/* 0x03c0 */
 	__u8	pad_0x03c1[0x0400-0x03c1];	/* 0x03c1 */
arch/s390/include/asm/preempt.h +20 -23

105 lines not on this page · raw message

merge bot Applied
#104 lore raw

[tip: locking/core] s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

In reply to Boqun Feng
The following commit has been merged into the locking/core branch of tip:

The following commit has been merged into the locking/core branch of tip:

Commit-ID:     3484440a7b194ed80fa4ae2d25fe1a3fcade98f3
Gitweb:        https://git.kernel.org/tip/3484440a7b194ed80fa4ae2d25fe1a3fcade98f3
Author:        Heiko Carstens <[email protected]>
AuthorDate:    Tue, 04 Aug 2026 09:14:34 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 10 Aug 2026 10:50:20 +02:00

s390/preempt: Enable HAS_SEPARATE_PREEMPT_RESCHED_BITS

Convert s390's preempt_count to 64 bit, and change the preempt primitives accordingly.

---
 arch/s390/Kconfig               |  1 +-
 arch/s390/include/asm/lowcore.h | 13 ++++++----
 arch/s390/include/asm/preempt.h | 43 ++++++++++++++------------------
 3 files changed, 30 insertions(+), 27 deletions(-)
arch/s390/Kconfig +1 -0
index 84404e6..378fcd2 100644
@@ -273,6 +273,7 @@ config S390
 	select PCI_MSI			if PCI
 	select PCI_MSI_ARCH_FALLBACKS	if PCI_MSI
 	select PCI_QUIRKS		if PCI
+	select HAS_SEPARATE_PREEMPT_RESCHED_BITS
 	select SPARSE_IRQ
 	select SWIOTLB
 	select SYSCTL_EXCEPTION_TRACE
arch/s390/include/asm/lowcore.h +9 -4
index 3b3ecc6..5cef215 100644
@@ -160,10 +160,15 @@ struct lowcore {
 	/* SMP info area */
 	__u32	cpu_nr;				/* 0x03a0 */
 	__u32	softirq_pending;		/* 0x03a4 */
-	__s32	preempt_count;			/* 0x03a8 */
-	__u32	spinlock_lockval;		/* 0x03ac */
-	__u32	spinlock_index;			/* 0x03b0 */
-	__u8	pad_0x03b4[0x03b8-0x03b4];	/* 0x03b4 */
+	union {
+		struct {
+			__u32	need_resched;	/* 0x03a8 */
+			__u32	count;		/* 0x03ac */
+		} preempt;
+		__u64	preempt_count;		/* 0x03a8 */
+	};
+	__u32	spinlock_lockval;		/* 0x03b0 */
+	__u32	spinlock_index;			/* 0x03b4 */
 	__u64	percpu_offset;			/* 0x03b8 */
 	__u8	percpu_register;		/* 0x03c0 */
 	__u8	pad_0x03c1[0x0400-0x03c1];	/* 0x03c1 */
arch/s390/include/asm/preempt.h +20 -23

105 lines not on this page · raw message

author maintainer
#105 lore raw

PATCH v4 13/17 rust: Introduce interrupt module

In reply to Boqun Feng

From: Lyude Paul <[email protected]>

This introduces a module for dealing with interrupt-disabled contexts, including the ability to enable and disable interrupts along with the ability to annotate functions as expecting that IRQs are already disabled on the local CPU.

---
 rust/helpers/helpers.c   |  1 +
 rust/helpers/interrupt.c | 18 ++++++++
 rust/helpers/sync.c      |  5 +++
 rust/kernel/interrupt.rs | 89 ++++++++++++++++++++++++++++++++++++++++
 rust/kernel/lib.rs       |  1 +
 5 files changed, 114 insertions(+)
 create mode 100644 rust/helpers/interrupt.c
 create mode 100644 rust/kernel/interrupt.rs
rust/helpers/helpers.c +1 -0
index 998e31052e66..0d85b5e68ec2 100644
@@ -65,6 +65,7 @@
 #include "irq.c"
 #include "fs.c"
 #include "gpu.c"
+#include "interrupt.c"
 #include "io.c"
 #include "jump_label.c"
 #include "kunit.c"
rust/helpers/interrupt.c new file +18 -0
new file mode 100644
index 000000000000..51b319bd4c00
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/spinlock.h>
+
+__rust_helper void rust_helper_local_interrupt_disable(void)
+{
+	local_interrupt_disable();
+}
+
+__rust_helper void rust_helper_local_interrupt_enable(void)
+{
+	local_interrupt_enable();
+}
+
+__rust_helper bool rust_helper_irqs_disabled(void)
+{
+	return irqs_disabled();
+}
rust/helpers/sync.c +5 -0
index 82d6aff73b04..4f474fe847c4 100644
@@ -11,3 +11,8 @@ __rust_helper void rust_helper_lockdep_unregister_key(struct lock_class_key *k)
 {
 	lockdep_unregister_key(k);
 }
+
+__rust_helper void rust_helper_lockdep_assert_irqs_disabled(void)
+{
+	lockdep_assert_irqs_disabled();
+}
rust/kernel/interrupt.rs new file +89 -0

89 lines not on this page · raw message

rust/kernel/lib.rs +1 -0
index 9512af7156df..2ee6c24d39c2 100644
@@ -82,6 +82,7 @@
 pub mod impl_flags;
 pub mod init;
 pub mod interop;
+pub mod interrupt;
 pub mod io;
 pub mod ioctl;
 pub mod iommu;
Signature · 1 line
2.50.1 (Apple Git-155)
author maintainer
#106 lore raw

PATCH v4 14/17 rust: helper: Add spin_{un,}lock_irq_{enable,disable}() helpers

In reply to Boqun Feng

spin_lock_irq_disable() and spin_unlock_irq_enable() are inline functions, to use them in Rust helpers are introduced. This is for interrupt disabling lock abstraction in Rust.

---
 rust/helpers/spinlock.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
rust/helpers/spinlock.c +15 -0
index 4d13062cf253..d53400c15022 100644
@@ -36,3 +36,18 @@ __rust_helper void rust_helper_spin_assert_is_held(spinlock_t *lock)
 {
 	lockdep_assert_held(lock);
 }
+
+__rust_helper void rust_helper_spin_lock_irq_disable(spinlock_t *lock)
+{
+	spin_lock_irq_disable(lock);
+}
+
+__rust_helper void rust_helper_spin_unlock_irq_enable(spinlock_t *lock)
+{
+	spin_unlock_irq_enable(lock);
+}
+
+__rust_helper int rust_helper_spin_trylock_irq_disable(spinlock_t *lock)
+{
+	return spin_trylock_irq_disable(lock);
+}
Signature · 1 line
2.50.1 (Apple Git-155)
author maintainer
#107 lore raw

PATCH v4 15/17 rust: sync: Use super::* in spinlock.rs

In reply to Boqun Feng

From: Lyude Paul <[email protected]>

No functional changes.

---
 rust/kernel/sync/lock/spinlock.rs | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
rust/kernel/sync/lock/spinlock.rs +4 -5
index ef76fa07ca3a..d75af32218ba 100644
@@ -3,6 +3,7 @@
 //! A kernel spinlock.
 //!
 //! This module allows Rust code to use the kernel's `spinlock_t`.
+use super::*;
 
 /// Creates a [`SpinLock`] initialiser with the given name and a newly-created lock class.
 ///
@@ -82,7 +83,7 @@ macro_rules! new_spinlock {
 /// ```
 ///
 /// [`spinlock_t`]: srctree/include/linux/spinlock.h
-pub type SpinLock<T> = super::Lock<T, SpinLockBackend>;
+pub type SpinLock<T> = Lock<T, SpinLockBackend>;
 
 /// A kernel `spinlock_t` lock backend.
 pub struct SpinLockBackend;
@@ -91,13 +92,11 @@ macro_rules! new_spinlock {
 ///
 /// This is simply a type alias for a [`Guard`] returned from locking a [`SpinLock`]. It will unlock
 /// the [`SpinLock`] upon being dropped.
-///
-/// [`Guard`]: super::Guard
-pub type SpinLockGuard<'a, T> = super::Guard<'a, T, SpinLockBackend>;
+pub type SpinLockGuard<'a, T> = Guard<'a, T, SpinLockBackend>;
 
 // SAFETY: The underlying kernel `spinlock_t` object ensures mutual exclusion. `relock` uses the
 // default implementation that always calls the same locking method.
-unsafe impl super::Backend for SpinLockBackend {
+unsafe impl Backend for SpinLockBackend {
     type State = bindings::spinlock_t;
     type GuardState = ();
 
Signature · 1 line
2.50.1 (Apple Git-155)
author maintainer
#108 lore raw

PATCH v4 16/17 rust: sync: Add SpinLockIrq

In reply to Boqun Feng

From: Lyude Paul <[email protected]>

A variant of SpinLock that ensures interrupts are disabled in the critical section. lock() will ensure that either interrupts are already disabled or disable them. unlock() will reverse the respective operation.

[Boqun: Port to use spin_lock_irq_disable() and
spin_unlock_irq_enable()]

---
 rust/kernel/sync.rs               |   9 +-
 rust/kernel/sync/lock/global.rs   |   3 +
 rust/kernel/sync/lock/spinlock.rs | 230 ++++++++++++++++++++++++++++++
 3 files changed, 241 insertions(+), 1 deletion(-)
rust/kernel/sync.rs +8 -1
index 993dbf2caa0e..df4f2604ff9b 100644
@@ -27,7 +27,14 @@
 pub use condvar::{new_condvar, CondVar, CondVarTimeoutResult};
 pub use lock::global::{global_lock, GlobalGuard, GlobalLock, GlobalLockBackend, GlobalLockedBy};
 pub use lock::mutex::{new_mutex, Mutex, MutexGuard};
-pub use lock::spinlock::{new_spinlock, SpinLock, SpinLockGuard};
+pub use lock::spinlock::{
+    new_spinlock,
+    new_spinlock_irq,
+    SpinLock,
+    SpinLockGuard,
+    SpinLockIrq,
+    SpinLockIrqGuard, //
+};
 pub use locked_by::LockedBy;
 pub use refcount::Refcount;
 pub use set_once::SetOnce;
rust/kernel/sync/lock/global.rs +3 -0
index ec2dd84316fc..ebb10521d8bd 100644
@@ -306,4 +306,7 @@ macro_rules! global_lock_inner {
     (backend SpinLock) => {
         $crate::sync::lock::spinlock::SpinLockBackend
     };
+    (backend SpinLockIrq) => {
+        $crate::sync::lock::spinlock::SpinLockIrqBackend
+    };
 }
rust/kernel/sync/lock/spinlock.rs +230 -0

239 lines not on this page · raw message

Signature · 1 line
2.50.1 (Apple Git-155)
author maintainer
#109 lore raw

PATCH v4 17/17 rust: sync: Introduce SpinLockIrq::lock_with() and friends

In reply to Boqun Feng

From: Lyude Paul <[email protected]>

SpinLockIrq and SpinLock use the exact same underlying C structure, with the only real difference being that the former uses the irq_disable() and irq_enable() variants for locking/unlocking. These variants can introduce some minor overhead in contexts where we already know that local processor interrupts are disabled, and as such we want a way to be able to skip modifying processor interrupt state in said contexts in order to avoid some overhead - just like the current C API allows us to do.

In order to do this, we add some special functions for SpinLockIrq:
lock_with() and try_lock_with(), which allow acquiring the lock without changing the interrupt state - as long as the caller can provide a LocalInterruptDisabled reference to prove that local processor interrupts have been disabled.

In some hacked-together benchmarks we ran, most of the time this did actually seem to lead to a noticeable difference in overhead:

From an aarch64 VM running on a MacBook M4:

    lock() when irq is disabled, 100 times cost Delta { nanos: 500 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 292 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 834 }

    lock() when irq is disabled, 100 times cost Delta { nanos: 459 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 291 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 709 }

From an x86_64 VM (qemu/kvm) running on a i7-13700H

    lock() when irq is disabled, 100 times cost Delta { nanos: 1002 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 729 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 1516 }

    lock() when irq is disabled, 100 times cost Delta { nanos: 754 }
    lock_with() when irq is disabled, 100 times cost Delta { nanos: 966 }
    lock() when irq is enabled, 100 times cost Delta { nanos: 1227 }

    (note that there were some runs on x86_64 where lock() on irq
    disabled vs. lock_with() on irq disabled had equivalent benchmarks,
    but it very much appeared to be a minority of test runs.)

While it's not clear how this affects real-world workloads yet, let's add this for the time being so we can find out.

This makes it so that a SpinLockIrq will work like a SpinLock if interrupts are disabled. So a function:

        (&'a SpinLockIrq, &'a LocalInterruptDisabled) -> Guard<'a, .., SpinLockBackend>

makes sense. Note that due to Guard and LocalInterruptDisabled having the same lifetime, interrupts cannot be enabled while the Guard exists.

---
 rust/kernel/sync/lock/spinlock.rs | 92 ++++++++++++++++++++++++++++++-
 1 file changed, 91 insertions(+), 1 deletion(-)
rust/kernel/sync/lock/spinlock.rs +91 -1
index 872544948e5d..aafc80125f59 100644
@@ -4,7 +4,10 @@
 //!
 //! This module allows Rust code to use the kernel's `spinlock_t`.
 use super::*;
-use crate::prelude::*;
+use crate::{
+    interrupt::LocalInterruptDisabled,
+    prelude::*, //
+};
 
 /// Creates a [`SpinLock`] initialiser with the given name and a newly-created lock class.
 ///
@@ -160,6 +163,16 @@ macro_rules! new_spinlock_irq {
 
 /// A variant of `SpinLock` that ensures interrupts are disabled in the critical section.
 ///
+/// This lock can be acquired in two ways:
+///
+/// - Using [`lock()`] like any other type of lock, in which case the bindings will modify the
+///   interrupt state to ensure that local processor interrupts remain disabled for at least as
+///   long as the [`SpinLockIrqGuard`] exists.
+/// - Using [`lock_with()`] in contexts where a [`LocalInterruptDisabled`] token is present and
+///   local processor interrupts are already known to be disabled, in which case the local
+///   interrupt state will not be touched. This method should be preferred if a
+///   [`LocalInterruptDisabled`] token is present in the scope.
+///
 /// For more info on spinlocks, see [`SpinLock`]. For more information on interrupts,
 /// [see the interrupt module](kernel::interrupt).
 ///
@@ -213,7 +226,47 @@ macro_rules! new_spinlock_irq {
 /// # Ok::<(), Error>(())
 /// ```
 ///
+/// The next example demonstrates locking a [`SpinLockIrq`] using [`lock_with()`] in a function
+/// which can only be called when local processor interrupts are already disabled.
+///
+/// ```
+/// use kernel::sync::{new_spinlock_irq, SpinLockIrq};
+/// use kernel::interrupt::*;
+///
+/// struct Inner {
+///     a: u32,
+/// }
+///
+/// #[pin_data]
+/// struct Example {
+///     #[pin]
+///     inner: SpinLockIrq<Inner>,
+/// }
+///
+/// impl Example {
+///     fn new() -> impl PinInit<Self> {
+///         pin_init!(Self {
+///             inner <- new_spinlock_irq!(Inner { a: 20 }),
+///         })
+///     }
+/// }
+///
+/// // Accessing an `Example` from a function that can only be called in no-interrupt contexts.
+/// fn noirq_work(e: &Example, interrupt_disabled: &LocalInterruptDisabled) {
+///     // Because we know interrupts are disabled from interrupt_disable, we can skip toggling
+///     // interrupt state using lock_with() and the provided token
+///     assert_eq!(e.inner.lock_with(interrupt_disabled).a, 20);
+/// }
+///
+/// # let e = KBox::pin_init(Example::new(), GFP_KERNEL)?;
+/// # let interrupt_guard = local_interrupt_disable();
+/// # noirq_work(&e, &interrupt_guard);
+/// #
+/// # Ok::<(), Error>(())
+/// ```
+///
 /// [`lock()`]: SpinLockIrq::lock
+/// [`lock_with()`]: SpinLockIrq::lock_with
 pub type SpinLockIrq<T> = super::Lock<T, SpinLockIrqBackend>;
 
 /// A kernel `spinlock_t` lock backend that can only be acquired in interrupt disabled contexts.
@@ -278,6 +331,43 @@ unsafe fn assert_is_held(ptr: *mut Self::State) {
     }
 }
 
+impl<T: ?Sized> Lock<T, SpinLockIrqBackend> {
+    /// Casts the lock as a `Lock<T, SpinLockBackend>`.
+    #[inline]
+    fn as_lock_in_interrupt<'a>(&'a self, _context: &'a LocalInterruptDisabled) -> &'a SpinLock<T> {
+        // SAFETY:
+        // - `Lock<T, SpinLockBackend>` and `Lock<T, SpinLockIrqBackend>` both have identical data
+        //   layouts.
+        // - As long as local interrupts are disabled (which is proven to be true by _context), it
+        //   is safe to treat a lock with SpinLockIrqBackend as a SpinLockBackend lock.
+        unsafe { core::mem::transmute(self) }
+    }
+
+    /// Acquires the lock without modifying local interrupt state.
+    ///
+    /// This function should be used in place of the more expensive [`Lock::lock()`] function when
+    /// possible for [`SpinLockIrq`] locks.
+    #[inline]
+    pub fn lock_with<'a>(&'a self, context: &'a LocalInterruptDisabled) -> SpinLockGuard<'a, T> {
+        self.as_lock_in_interrupt(context).lock()
+    }
+
+    /// Tries to acquire the lock without modifying local interrupt state.
+    ///
+    /// This function should be used in place of the more expensive [`Lock::try_lock()`] function
+    /// when possible for [`SpinLockIrq`] locks.
+    ///
+    /// Returns a guard that can be used to access the data protected by the lock if successful.
+    #[must_use = "if unused, the lock will be immediately unlocked"]
+    #[inline]
+    pub fn try_lock_with<'a>(
+        &'a self,
+        context: &'a LocalInterruptDisabled,
+    ) -> Option<SpinLockGuard<'a, T>> {
+        self.as_lock_in_interrupt(context).try_lock()
+    }
+}
+
 #[kunit_tests(rust_spinlock_irq_condvar)]
 mod tests {
     use super::*;
Signature · 1 line
2.50.1 (Apple Git-155)