Skip to contentSkip to conversation

rcu,srcu: Make call_rcu()/call_srcu() safe from any context

PATCH In review Puranjay Mohan 6 participants 19 messages Aug 10 – Sep 10

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

v4 cover rcu,srcu: Make call_rcu()/call_srcu() safe from any context

call_rcu() and call_srcu() only ever touch their per-CPU callback lists with interrupts disabled: the enqueue runs under local_irq_save() (and the nocb locks when offloaded), and so do callback invocation and grace-period work. That is fine as long as call_rcu() itself is invoked with interrupts enabled, but it is not always. An NMI handler can call call_rcu(), and instrumentation can reenter it. The case that prompted this is a BPF program attached to rcu_segcblist_enqueue() that frees an
object: the free reaches call_rcu_tasks_trace(), which is call_srcu() under the hood, back on the same CPU with the srcu_data lock already held, and it deadlocks on that lock. Either way, enqueuing directly can corrupt the list or deadlock.

Rather than scatter context checks through the enqueue, make it defer whenever interrupts are disabled: stage the callback on a per-CPU lockless list and re-issue it from an irq_work once interrupts are back on, going straight to the enqueue helper so the re-issue cannot defer again. Only the drain side takes a lock; the staging is a bare llist_add() and stays safe from NMI. This is behind a new hidden CONFIG_RCU_DEFER, which is set wherever a reentrant enqueue is possible (HAVE_NMI, KPROBES, FUNCTION_TRACER or TRACEPOINTS); without it call_rcu() enqueues exactly as before. The gate is bare irqs_disabled(), so callers that merely hold interrupts off are deferred too and pay one irq_work hop.

CPU offline is the awkward part. A callback can be deferred very late in the outgoing CPU's teardown -- from do_idle() or cpuhp_ap_report_dead(), past the CPUHP_AP_SMPCFD_DYING flush that would otherwise run the irq_work -- so the irq_work can no longer run there to re-issue it. rcu_barrier() and srcu_barrier() therefore drain every CPU's deferred list themselves before they wait. They drain rather than wait the irq_work out because irq_work_sync() parks on an rcuwait, which holds a single waiter, so two concurrent barriers would clobber each other's wakeup. rcutree_migrate_callbacks() drains the outgoing CPU's list too, so a late deferral still lands on a callback list even when nobody calls a barrier. To keep those drainers from stepping on each other, the drain holds a per-CPU raw lock across the llist_del_all() and the re-issue, so a drainer never returns having pulled callbacks off the deferred list but not yet put them on a callback list. Every lock the re-issue touches (nocb, rcu_node, srcu_data) is already raw, so the nesting is fine.

The drain re-issues with interrupts disabled, so instrumentation on the enqueue path can re-enter call_rcu()/call_srcu() from inside it, stage another callback, re-raise the irq_work, and the drain never finishes. A per-CPU flag catches that: a deferral that arrives while that CPU is inside its own irq_work drain, and is not from an NMI, is dropped rather than staged, with a WARN_ONCE() under CONFIG_PROVE_RCU whose backtrace names the instrumentation responsible. Dropping leaks that callback, and can strand state its caller tied to it, but the alternative is a CPU that never leaves the drain, and the producer is a BPF program that emits one callback per enqueue, so there is nothing finite to wait for. Only the irq_work drain sets the flag: a direct drain from a barrier or from CPU-offline re-issues onto the current CPU, so anything staged during it is picked up by that CPU's own irq_work rather than feeding the drain in progress, and no legitimate callback is dropped. Instrumenting the irq_work machinery itself can still loop, as it can for any irq_work user, and is not something this series can fix.

The irq_work is IRQ_WORK_INIT_HARD in all four flavors. It is not needed for correctness, but a non-HARD irq_work runs from a kthread on PREEMPT_RT and can be delayed under load, letting deferred callbacks pile up; running the re-issue in hard-irq context keeps that from turning into an OOM.

Patches 1 and 2 do Tree and Tiny RCU, 3 and 4 Tree and Tiny SRCU. Patch 5 teaches rcutorture to issue ->call() from a perf-overflow NMI -- the nmi_calls parameter, on by default -- on the flavors that advertise it, and checks that every callback issued from NMI is later invoked. Patch 6 adds the BPF reentry reproducer described above.

Changelog:
v3: https://lore.kernel.org/rcu/[email protected]/
Changes in v4:
- rcu_barrier() and srcu_barrier() no longer skip a CPU whose deferred list looks empty. The lockless llist_empty() test added in v3 skipped the lock that makes a drain conclusive: a concurrent drainer can already have emptied the list without having re-issued yet, so the barrier could return before those callbacks reached a callback list.
- Tree SRCU: drain before cleanup_srcu_struct()'s "just leak it" early returns, and in srcu_module_going() before it frees any ->sda. Deferral skips check_init_srcu_struct(), so srcu_module_going() could skip cleanup_srcu_struct() and free ->sda with a staged srcu_data still chained on a per-CPU list.
- Dropped ->defer_exp: a deferred expedited call_srcu() now completes as a normal grace period. The flag lived on the srcu_data rather than on the callback, so it expedited whole batches; only srcu_expedite_current() with interrupts already disabled can reach this, so the downgrade costs a delay and nothing else.
- rcutorture: use a fixed perf sample period rather than .freq, which sets TICK_DEP_BIT_PERF_EVENTS for every CPU and pinned the tick for whole runs of TREE04 and TREE07. Also document nmi_calls, and say on the console when there is no PMU, since the issued==invoked check then compares zero to zero.
- kasan_record_aux_stack() moved to __call_rcu_common(), so a use-after-free report names the caller rather than the irq_work.
- Known gap, documented in patch 1: a callback deferred past the CPUHP_AP_SMPCFD_DYING flush leaves ->defer_work claimed with its self-IPI lost. rcutree_migrate_callbacks() still re-issues the callback, but the first deferral after that CPU comes back raises no IPI and waits for the next irq_work there, or for rcu_barrier().

v2: https://lore.kernel.org/rcu/[email protected]/
Changes in v3:
- Barriers no longer call irq_work_sync() on an online CPU's ->defer_work. irq_work_sync() waits on an rcuwait, which holds exactly one task, so two concurrent rcu_barrier()s syncing the same per-CPU irq_work could lose a wakeup and hang. Both flushes now drain every CPU directly, guarded by a lockless llist_empty() test so the no-deferrals case stays cheap.
- The re-entry guard is now set only by the irq_work drain, which always runs on the CPU owning the list it drains. In v2 a barrier draining a remote CPU set the flag on the draining CPU, so an unrelated irqs-off call_rcu() there was dropped and leaked even though it could not have fed the drain.
- The drain clears ->next before re-issuing. A double call_rcu() on a head that is already debug-object-active makes llist_add() self-link it, and rcu_do_enqueue()'s double-free path returns without clearing ->next, so the drain span looped forever with interrupts disabled.
- An expedited call_srcu() is no longer silently downgraded: ->defer_exp records it per srcu_data and the batch is re-issued expedited. Only srcu_expedite_current() is affected, __synchronize_srcu() sleeps and so is never deferred.
- The drop is now WARN_ONCE() under CONFIG_PROVE_RCU rather than an unconditional WARN, so instrumentation cannot reboot a panic_on_warn kernel; the backtrace is what identifies the offending program.
- Tiny RCU and Tiny SRCU use IRQ_WORK_INIT_HARD like the Tree flavors, and READ_ONCE()/WRITE_ONCE() on their draining flags. TINY_SRCU is "default y if !SMP" with no PREEMPT_RT dependency, so it really can be built on RT where a non-HARD irq_work waits on the irq_workd kthread.
- Tiny SRCU: cleanup_srcu_struct() drains *and* irq_work_sync()s ->defer_iw. That irq_work is embedded in the srcu_struct the caller is about to free, unlike the Tree flavors' static per-CPU ones.
- rcutorture: drive the perf counters from CPU-hotplug callbacks. The one-shot for_each_online_cpu() loop lost them at the first CPU offline, after which the end-of-test issued==invoked check compared 0 == 0.
- rcutorture: per-CPU rcu_head instead of one global, so several CPUs can race the drain; count the call before issuing it so mid-run stats cannot show nmi-cbs > nmi-calls; release the perf events on the torture_cleanup_begin() early-return path; print nmi_calls in the module banner; report when nmi_calls is set but no NMI ->call() ever happened; and drop sample_freq to 100, since 1000 made perf lower the system-wide perf_event_max_sample_rate tenfold.
- selftests/bpf: the old ASSERT_EQ(reentered, 1) could not fail, and an atomic allocation failure in the nested task-storage delete made the test pass without ever re-entering call_srcu(). It now records and asserts the helper return values, probes for rcu_segcblist_enqueue() up front so a Tiny kernel skips instead of failing, and restores the CPU affinity it changes.

Testing: rcutorture rcu/srcu/srcud/tasks-tracing with nmi_calls, hotplug, barriers and nocb toggling issued ~81000 callbacks from NMI with none lost, under PROVE_LOCKING, PROVE_RAW_LOCK_NESTING, DEBUG_OBJECTS_RCU_HEAD and RCU_LAZY, with no lockdep reports; SRCU-T, SRCU-U, TINY01 and TINY02 pass; the BPF reproducer passes.

v1: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Fixed the re-entry livelock Zqiang spotted: a BPF program on the enqueue path re-enters call_srcu() from inside srcu_defer_drain(), stages another callback and re-raises the irq_work, so the drain never finishes. A per-CPU flag now drops such a deferral, with a warning, unless it comes from an NMI.
- cleanup_srcu_struct(): drain the deferred callbacks before syncing ->irq_work rather than after, since re-issuing one can start a grace period and re-queue that irq_work (Zqiang).
- Tiny SRCU: sync ->defer_iw in cleanup_srcu_struct() as well, so a deferred callback is re-issued onto ->srcu_cb_head where the leak checks can see it instead of being stranded on a soon-to-be-freed srcu_struct.
- Added Kumar's ack to the BPF selftest patch.

Testing: rcutorture rcu, srcu, srcud and tasks-tracing, each with nmi_calls, CPU hotplug and barriers, and nocb toggling for rcu, all End of test: SUCCESS with the count issued from NMI equal to the count invoked (17987, 2924, 2661 and 4412 respectively) and clean consoles, under PROVE_LOCKING, PROVE_RAW_LOCK_NESTING and DEBUG_OBJECTS_RCU_HEAD; SRCU-T, SRCU-U, TINY01 and TINY02 pass; the BPF reproducer passes. Build-tested x86_64 with TREE_RCU+TREE_SRCU, TINY_RCU+TINY_SRCU and TREE_RCU+TINY_SRCU, each with CONFIG_RCU_DEFER=y and =n, plus arm64 TREE_RCU+TREE_SRCU with RCU_NOCB_CPU and RCU_LAZY.

Puranjay Mohan (6):
rcu: Make call_rcu() safe to call from any context
rcu: Make Tiny call_rcu() safe to call from any context
srcu: Make call_srcu() safe to call from any context
srcu: Make Tiny call_srcu() safe to call from any context
rcutorture: Exercise ->call() from NMI context selftests/bpf: Add a call_srcu() re-entry reproducer

 .../admin-guide/kernel-parameters.txt         |   7 +
 include/linux/srcutiny.h                      |  12 +-
 include/linux/srcutree.h                      |   4 +
 kernel/rcu/Kconfig                            |   6 +
 kernel/rcu/rcu.h                              |  14 ++
 kernel/rcu/rcutorture.c                       | 150 +++++++++++++++-
 kernel/rcu/srcutiny.c                         |  93 +++++++++-
 kernel/rcu/srcutree.c                         | 170 +++++++++++++++++-
 kernel/rcu/tiny.c                             | 127 ++++++++++---
 kernel/rcu/tree.c                             | 133 ++++++++++++--
 kernel/rcu/tree.h                             |   6 +
 .../selftests/bpf/prog_tests/rcu_reentry.c    |  93 ++++++++++
 .../testing/selftests/bpf/progs/rcu_reentry.c |  51 ++++++
 13 files changed, 818 insertions(+), 48 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/rcu_reentry.c
 create mode 100644 tools/testing/selftests/bpf/progs/rcu_reentry.c

base-commit: 9dc303e69bcd49f9668ca090ae45325269531fbb

Signature · 1 line
2.53.0-Meta
#1 lore raw

PATCH v4 1/6 rcu: Make call_rcu() safe to call from any context

RCU's per-CPU callback list is only touched with interrupts disabled: the enqueue runs under local_irq_save() (and the nocb locks when offloaded), as do callback invocation and grace-period work. A call_rcu() that arrives with interrupts already disabled, whether from an NMI or from instrumentation that re-enters RCU, can interrupt one of those and corrupt the list or deadlock.

Defer instead: stage the callback on a per-CPU llist and raise an irq_work that re-issues it once interrupts are on, straight to the enqueue so it cannot defer again. The gate is bare irqs_disabled(), so callers that merely hold interrupts off are deferred too and pay one irq_work hop. Skip it while the scheduler is down (RCU_SCHEDULER_INACTIVE): irq_work is not usable that early, rcu_init() already calls call_rcu(), and the per-CPU deferral state is not initialised until rcu_init_one() runs later in it.

rcu_barrier() drains every CPU's ->defer_head before it scans the lists, and rcutree_migrate_callbacks() drains an outgoing CPU's. A drain re-issues onto the draining CPU, so a barrier moves other CPUs' staged callbacks onto
its own ->cblist; call_rcu() promises no CPU affinity for invocation. ->defer_lock is held across llist_del_all() and the whole re-issue so the drainers
serialize: one that finds the list empty can conclude that everything staged before it is already on a callback list. Interrupts stay off for the batch. Where the arch has an irq_work self-IPI that is what one interrupts-disabled region could stage, normally a single callback; where arch_irq_work_has_interrupt() is false the drain waits for the tick, so several regions can accumulate first.

The drain clears ->next before re-issuing. A double call_rcu() on a head that is already debug-object-active self-links the staged node, and rcu_do_enqueue()'s duplicate path returns without clearing it, so the drain would spin. A re-add behind other staged callbacks makes a longer cycle, which that does not bound; a double call_rcu() stays undefined. llist_del_all() yields newest-first, so a batch is re-issued in reverse call order; nothing depends on call_rcu() ordering. The re-issue drops the lazy hint, since staging records only ->func, so a deferred callback loses its batching on CONFIG_RCU_LAZY. kasan_record_aux_stack() moves to __call_rcu_common() so a use-after-free report names the caller rather than the irq_work.

The re-issue runs with interrupts disabled, so instrumentation on the enqueue path can re-enter call_rcu(), stage another callback and re-raise the irq_work, livelocking the drain. A per-CPU flag guards it: a deferral that arrives while this CPU is draining, and is not from an NMI, is dropped. The WARN_ONCE() is under CONFIG_PROVE_RCU, so a production kernel drops it silently. That leaks the callback and can strand state the caller tied to it, since a one-shot flag only the callback clears never resets, but the alternative is an unbounded loop.

A callback deferred past the CPUHP_AP_SMPCFD_DYING irq_work flush leaves ->defer_work claimed with its self-IPI lost. rcutree_migrate_callbacks() still re-issues the callback, but the first deferral after that CPU comes back raises no IPI and waits for the next irq_work there, or for rcu_barrier(). Unqueueing an irq_work is not something the API offers.

The irq_work is IRQ_WORK_INIT_HARD so the re-issue stays prompt on PREEMPT_RT, where a non-HARD irq_work runs in a kthread that can be delayed under load. A hidden CONFIG_RCU_DEFER gates the deferral code and its IRQ_WORK dependency, though the rcu_data members are unconditional; without it call_rcu() enqueues directly as before. Under CONFIG_PROVE_RCU, warn if the direct path is reached from an NMI.

---
 kernel/rcu/Kconfig |   6 +++
 kernel/rcu/rcu.h   |  11 ++++
 kernel/rcu/tree.c  | 131 +++++++++++++++++++++++++++++++++++++++++----
 kernel/rcu/tree.h  |   6 +++
 4 files changed, 143 insertions(+), 11 deletions(-)
kernel/rcu/Kconfig +6 -0
index f15da8038d0ba..1a5fb3156c062 100644
@@ -175,6 +175,12 @@ config RCU_STALL_COMMON
 config RCU_NEED_SEGCBLIST
 	def_bool ( TREE_RCU || TREE_SRCU || TASKS_RCU_GENERIC )
 
+# The deferral (and the IRQ_WORK it uses) is only needed where call_rcu() /
+# call_srcu() can be invoked while a callback-list operation is in flight.
+config RCU_DEFER
+	def_bool HAVE_NMI || KPROBES || FUNCTION_TRACER || TRACEPOINTS
+	select IRQ_WORK
+
 config RCU_FANOUT
 	int "Tree-based hierarchical RCU fanout value"
 	range 2 64 if 64BIT
kernel/rcu/rcu.h +11 -0
index 39a9f6fa9a7b2..91e33571a554d 100644
@@ -572,6 +572,17 @@ static inline void tasks_cblist_init_generic(void) { }
 #define RCU_SCHEDULER_INIT	1
 #define RCU_SCHEDULER_RUNNING	2
 
+/*
+ * Defer whenever interrupts are disabled, since a callback-list operation may
+ * be in flight on this CPU.  Not before the scheduler is up: irq_work is not
+ * usable that early, and rcu_init() itself calls call_rcu().
+ */
+static inline bool should_rcu_defer(void)
+{
+	return IS_ENABLED(CONFIG_RCU_DEFER) && irqs_disabled() &&
+	       rcu_scheduler_active != RCU_SCHEDULER_INACTIVE;
+}
+
 enum rcutorture_type {
 	RCU_FLAVOR,
 	RCU_TASKS_FLAVOR,
kernel/rcu/tree.c +120 -11

180 lines not on this page · raw message

kernel/rcu/tree.h +6 -0
index eedfa43059e80..b7cac7a13b4f2 100644
@@ -229,6 +229,12 @@ struct rcu_data {
 	struct rcu_head barrier_head;
 	int exp_watching_snap;		/* Double-check need for IPI. */
 
+	/* Deferral of an NMI/reentrant call_rcu(); see __call_rcu_common(). */
+	struct llist_head defer_head;
+	struct irq_work defer_work;
+	raw_spinlock_t defer_lock;
+	bool defer_draining;
+
 	/* 5) Callback offloading. */
 #ifdef CONFIG_RCU_NOCB_CPU
 	struct swait_queue_head nocb_cb_wq; /* For nocb kthreads to sleep on. */
Signature · 1 line
2.53.0-Meta
#2 lore raw

Re: [PATCH v4 1/6] rcu: Make call_rcu() safe to call from any context

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Deferred callbacks bypass `debug_rcu_head_queue()` check, causing immediate list corruption on…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Deferred callbacks bypass debug_rcu_head_queue() check, causing immediate list corruption on double call_rcu() and defeating CONFIG_DEBUG_OBJECTS_RCU_HEAD. --

--- Patch [1]: [PATCH v4 1/6] rcu: Make call_rcu() safe to call from any context --- commit 5205f0afe602a3f93ee865a26d94981f93553b7d
Author: Puranjay Mohan <[email protected]>

rcu: Make call_rcu() safe to call from any context

This patch defers call_rcu() via a per-CPU llist and irq_work when called with interrupts disabled. This avoids deadlocks or list corruption if call_rcu() is invoked from NMI or instrumentation.

Quoted 4 lines
kernel/rcu/tree.c +0 -0
index 96848fc1f02b8..ff9a2395c9e8e 100644

[ ... ]

Quoted 23 lines
kernel/rcu/tree.c +7 -9
-static void
-__call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
+/*
+ * Also called by __rcu_defer_drain() to re-issue a deferred callback, so it
+ * must not re-check the deferral condition.  Either caller may have interrupts
+ * already disabled, and a drain of a remote CPU re-issues onto the draining
+ * CPU.
+ */
+static void rcu_do_enqueue(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
 {
 	static atomic_t doublefrees;
 	unsigned long flags;
 	bool lazy;
 	struct rcu_data *rdp;
 
-	/* Misaligned rcu_head! */
-	WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1));
-
-	/* Avoid NULL dereference if callback is NULL. */
-	if (WARN_ON_ONCE(!func))
-		return;
-
 	if (debug_rcu_head_queue(head)) {

[ ... ]

Quoted 18 lines
kernel/rcu/tree.c +18 -0
+static void call_rcu_defer(struct rcu_head *head, rcu_callback_t func)
+{
+	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
+
+	/*
+	 * Instrumentation on the enqueue path can re-enter here from inside the
+	 * drain.  Re-queuing would livelock it, so drop the callback; an NMI
+	 * cannot loop, so let it through.
+	 */
+	if (READ_ONCE(rdp->defer_draining) && !in_nmi()) {
+		WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU),
+			  "call_rcu() re-entered during callback drain; leaking callback\n");
+		return;
+	}
+	head->func = func;
+	if (llist_add((struct llist_node *)head, &rdp->defer_head))
+		irq_work_queue(&rdp->defer_work);
+}

[ ... ]

Quoted 5 lines
kernel/rcu/tree.c +5 -0
+static void
+__call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
+{
+	/* Misaligned rcu_head! */
+	WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1));

kernel/rcu/tree.c __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) · line 3291

Quoted 10 lines
kernel/rcu/tree.c +10 -0
+
+	/* Avoid NULL dereference if callback is NULL. */
+	if (WARN_ON_ONCE(!func))
+		return;
+
+	/* Record the caller: the irq_work's stack says nothing about it. */
+	kasan_record_aux_stack(head);
+
+	if (should_rcu_defer()) {
+		call_rcu_defer(head, func);

[Severity: High]
Does this bypass the debug check for double call_rcu(), potentially corrupting the callback list?

By moving debug_rcu_head_queue() from __call_rcu_common() into rcu_do_enqueue(), deferred callbacks are added to the defer_head list before being validated.

If a double call_rcu() occurs, llist_add() in call_rcu_defer() will overwrite head->next. If the callback was already in a normal cblist, its next pointer is overwritten with defer_head->first, immediately splitting the cblist and splicing it into the deferred list.

Could we check debug_rcu_head_queue() here before deferring, so that CONFIG_DEBUG_OBJECTS_RCU_HEAD can safely catch the bug without list corruption?

Quoted 11 lines
kernel/rcu/tree.c +11 -0
+		return;
+	}
+
+	/*
+	 * Only reachable from an NMI when deferral is off: before the scheduler
+	 * is up, or with CONFIG_RCU_DEFER=n.  The enqueue can then race.
+	 */
+	WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi());
+
+	rcu_do_enqueue(head, func, lazy_in);
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#3 lore raw

Re: [PATCH v4 1/6] rcu: Make call_rcu() safe to call from any context

Quoted 92 lines · On Mon, Aug 10, 2026 at 1:43 PM <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Deferred callbacks bypass debug_rcu_head_queue() check, causing immediate list corruption on double call_rcu() and defeating CONFIG_DEBUG_OBJECTS_RCU_HEAD. --

--- Patch [1]: [PATCH v4 1/6] rcu: Make call_rcu() safe to call from any context --- commit 5205f0afe602a3f93ee865a26d94981f93553b7d
Author: Puranjay Mohan <[email protected]>

rcu: Make call_rcu() safe to call from any context

This patch defers call_rcu() via a per-CPU llist and irq_work when called with interrupts disabled. This avoids deadlocks or list corruption if call_rcu() is invoked from NMI or instrumentation.

Quoted 4 lines
kernel/rcu/tree.c +0 -0
index 96848fc1f02b8..ff9a2395c9e8e 100644

[ ... ]

Quoted 23 lines
kernel/rcu/tree.c +7 -9
-static void
-__call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
+/*
+ * Also called by __rcu_defer_drain() to re-issue a deferred callback, so it
+ * must not re-check the deferral condition.  Either caller may have interrupts
+ * already disabled, and a drain of a remote CPU re-issues onto the draining
+ * CPU.
+ */
+static void rcu_do_enqueue(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
 {
      static atomic_t doublefrees;
      unsigned long flags;
      bool lazy;
      struct rcu_data *rdp;
 
-     /* Misaligned rcu_head! */
-     WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1));
-
-     /* Avoid NULL dereference if callback is NULL. */
-     if (WARN_ON_ONCE(!func))
-             return;
-
      if (debug_rcu_head_queue(head)) {

[ ... ]

Quoted 18 lines
kernel/rcu/tree.c +18 -0
+static void call_rcu_defer(struct rcu_head *head, rcu_callback_t func)
+{
+     struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
+
+     /*
+      * Instrumentation on the enqueue path can re-enter here from inside the
+      * drain.  Re-queuing would livelock it, so drop the callback; an NMI
+      * cannot loop, so let it through.
+      */
+     if (READ_ONCE(rdp->defer_draining) && !in_nmi()) {
+             WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU),
+                       "call_rcu() re-entered during callback drain; leaking callback\n");
+             return;
+     }
+     head->func = func;
+     if (llist_add((struct llist_node *)head, &rdp->defer_head))
+             irq_work_queue(&rdp->defer_work);
+}

[ ... ]

Quoted 5 lines
kernel/rcu/tree.c +5 -0
+static void
+__call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
+{
+     /* Misaligned rcu_head! */
+     WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1));

kernel/rcu/tree.c __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) · line 3291

Quoted 10 lines
kernel/rcu/tree.c +10 -0
+
+     /* Avoid NULL dereference if callback is NULL. */
+     if (WARN_ON_ONCE(!func))
+             return;
+
+     /* Record the caller: the irq_work's stack says nothing about it. */
+     kasan_record_aux_stack(head);
+
+     if (should_rcu_defer()) {
+             call_rcu_defer(head, func);

[Severity: High]
Does this bypass the debug check for double call_rcu(), potentially corrupting the callback list?

By moving debug_rcu_head_queue() from __call_rcu_common() into rcu_do_enqueue(), deferred callbacks are added to the defer_head list before being validated.

Quoted 7 lines

If a double call_rcu() occurs, llist_add() in call_rcu_defer() will overwrite head->next. If the callback was already in a normal cblist, its next pointer is overwritten with defer_head->first, immediately splitting the cblist and splicing it into the deferred list.

Could we check debug_rcu_head_queue() here before deferring, so that CONFIG_DEBUG_OBJECTS_RCU_HEAD can safely catch the bug without list corruption?

No we can't debug_rcu_head_queue() can't be done from NMI and doubling calling call_rcu() from NMI or interrupts disabled will result in leaking some callbacks but it will not cause a deadlock (validated in testing using a kernel module that does multiple call_rcu() from NMI).

#4 lore raw

Re: [PATCH v4 1/6] rcu: Make call_rcu() safe to call from any context

In reply to Puranjay Mohan
Quoted 6 lines · Puranjay Mohan
kernel/rcu/rcu.h +0 -0
@@ -572,6 +572,17 @@ static inline void tasks_cblist_init_generic(void) { }
 #define RCU_SCHEDULER_INIT	1
 #define RCU_SCHEDULER_RUNNING	2
 

kernel/rcu/rcu.h static inline void tasks_cblist_init_generic(void) { } · line 584

Quoted 10 lines
kernel/rcu/rcu.h +10 -0
+/*
+ * Defer whenever interrupts are disabled, since a callback-list operation may
+ * be in flight on this CPU.  Not before the scheduler is up: irq_work is not
+ * usable that early, and rcu_init() itself calls call_rcu().
+ */
+static inline bool should_rcu_defer(void)
+{
+	return IS_ENABLED(CONFIG_RCU_DEFER) && irqs_disabled() &&
+	       rcu_scheduler_active != RCU_SCHEDULER_INACTIVE;
+}

Why does the description say that the defer part is for usage from NMI and the test here has irqs_disabled() instead of in_nmi()?

Quoted 18 lines
kernel/rcu/rcu.h +1 -0
+
 enum rcutorture_type {
 	RCU_FLAVOR,
 	RCU_TASKS_FLAVOR,
kernel/rcu/tree.c +6 -0
index 21b6ce1dffb63..3bf3a250f9de8 100644
@@ -3206,6 +3204,103 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
 	local_irq_restore(flags);
 }
 
+/*
+ * Re-issue deferred callbacks straight to the enqueue so they cannot defer
+ * again.  ->defer_lock serializes the drainers: this CPU's irq_work,
+ * rcu_defer_flush() and rcutree_migrate_callbacks().
+ */
+static void __rcu_defer_drain(struct rcu_data *rdp)

kernel/rcu/tree.c __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) · line 3222

Quoted 10 lines
kernel/rcu/tree.c +10 -0
+{
+	struct llist_node *node, *next;
+	unsigned long flags;
+
+	if (!IS_ENABLED(CONFIG_RCU_DEFER))
+		return;
+
+	raw_spin_lock_irqsave(&rdp->defer_lock, flags);
+	llist_for_each_safe(node, next, llist_del_all(&rdp->defer_head)) {
+		struct rcu_head *head = (struct rcu_head *)node;

Why do you need the lock. This is still not clear to me despite the comment. You can do llist_del_all() towards another list and then feed it into rcu_do_enqueue() one by one. And you use the LAZY part.

Quoted 7 lines
kernel/rcu/tree.c +7 -0
+
+		/* Bounds a node self-linked by a double call_rcu(). */
+		head->next = NULL;
+		rcu_do_enqueue(head, head->func, false);
+	}
+	raw_spin_unlock_irqrestore(&rdp->defer_lock, flags);
+}

…

kernel/rcu/tree.c rcu_boot_init_percpu_data(int cpu) · line 4335

Quoted 7 lines
kernel/rcu/tree.c +3 -0
@@ -4231,6 +4330,9 @@ rcu_boot_init_percpu_data(int cpu)
 	rdp->rcu_onl_gp_state = RCU_GP_CLEANED;
 	rdp->last_sched_clock = jiffies;
 	rdp->cpu = cpu;
+	init_llist_head(&rdp->defer_head);
+	raw_spin_lock_init(&rdp->defer_lock);
+	rdp->defer_work = IRQ_WORK_INIT_HARD(rcu_defer_drain);

Why is this IRQ_WORK_INIT_HARD() instead, say, IRQ_WORK_INIT_LAZY()? Is there a requirement that the RCU callback needs to complete asap and not be delayed to the next tick? This would give kind of the LAZY part.

kernel/rcu/tree.c rcu_boot_init_percpu_data(int cpu) · line 4338

Quoted 3 lines
kernel/rcu/tree.c +0 -0
 	rcu_boot_init_nocb_percpu_data(rdp);
 }
 

Sebastian

#5 lore raw

Re: [PATCH v4 1/6] rcu: Make call_rcu() safe to call from any context

Quoted 8 lines · Sebastian Andrzej Siewior
Quoted 6 lines · Puranjay Mohan
kernel/rcu/rcu.h +0 -0
@@ -572,6 +572,17 @@ static inline void tasks_cblist_init_generic(void) { }
 #define RCU_SCHEDULER_INIT   1
 #define RCU_SCHEDULER_RUNNING        2
 
Quoted 13 lines

kernel/rcu/rcu.h static inline void tasks_cblist_init_generic(void) { } · line 584

Quoted 10 lines
kernel/rcu/rcu.h +10 -0
+/*
+ * Defer whenever interrupts are disabled, since a callback-list operation may
+ * be in flight on this CPU.  Not before the scheduler is up: irq_work is not
+ * usable that early, and rcu_init() itself calls call_rcu().
+ */
+static inline bool should_rcu_defer(void)
+{
+     return IS_ENABLED(CONFIG_RCU_DEFER) && irqs_disabled() &&
+            rcu_scheduler_active != RCU_SCHEDULER_INACTIVE;
+}

Why does the description say that the defer part is for usage from NMI and the test here has irqs_disabled() instead of in_nmi()?

We defer for both irq disabled sections, hard irq, and in_nmi() and checking for irqs_disabled() covers all three.

Quoted 25 lines
Quoted 25 lines
kernel/rcu/rcu.h +1 -0
+
 enum rcutorture_type {
      RCU_FLAVOR,
      RCU_TASKS_FLAVOR,
kernel/rcu/tree.c +12 -0
index 21b6ce1dffb63..3bf3a250f9de8 100644
@@ -3206,6 +3204,103 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
      local_irq_restore(flags);
 }
 
+/*
+ * Re-issue deferred callbacks straight to the enqueue so they cannot defer
+ * again.  ->defer_lock serializes the drainers: this CPU's irq_work,
+ * rcu_defer_flush() and rcutree_migrate_callbacks().
+ */
+static void __rcu_defer_drain(struct rcu_data *rdp)
+{
+     struct llist_node *node, *next;
+     unsigned long flags;
+
+     if (!IS_ENABLED(CONFIG_RCU_DEFER))
+             return;
Quoted 8 lines

kernel/rcu/tree.c __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) · line 3222

Quoted 4 lines
kernel/rcu/tree.c +4 -0
+
+     raw_spin_lock_irqsave(&rdp->defer_lock, flags);
+     llist_for_each_safe(node, next, llist_del_all(&rdp->defer_head)) {
+             struct rcu_head *head = (struct rcu_head *)node;

Why do you need the lock. This is still not clear to me despite the comment. You can do llist_del_all() towards another list and then feed it into rcu_do_enqueue() one by one. And you use the LAZY part.

Because rcu_barrier() can call this for each cpu and at the same time rcu_defer_drain() can call it too, so this would cause a race where the irq work can remove the callbacks (llist_del_all) and before it can enqueue them, rcu_barrier will see that the list is already empty and will not wait for these callbacks. We want the llist_del_all() and rcu_do_enqueue() to happen atomically so rcu_barrier() can work correctly.

Quoted 9 lines
Quoted 8 lines
kernel/rcu/tree.c +7 -0
+
+             /* Bounds a node self-linked by a double call_rcu(). */
+             head->next = NULL;
+             rcu_do_enqueue(head, head->func, false);
+     }
+     raw_spin_unlock_irqrestore(&rdp->defer_lock, flags);
+}

…

Quoted 12 lines

kernel/rcu/tree.c rcu_boot_init_percpu_data(int cpu) · line 4335

Quoted 7 lines
kernel/rcu/tree.c +3 -0
@@ -4231,6 +4330,9 @@ rcu_boot_init_percpu_data(int cpu)
      rdp->rcu_onl_gp_state = RCU_GP_CLEANED;
      rdp->last_sched_clock = jiffies;
      rdp->cpu = cpu;
+     init_llist_head(&rdp->defer_head);
+     raw_spin_lock_init(&rdp->defer_lock);
+     rdp->defer_work = IRQ_WORK_INIT_HARD(rcu_defer_drain);

Why is this IRQ_WORK_INIT_HARD() instead, say, IRQ_WORK_INIT_LAZY()? Is there a requirement that the RCU callback needs to complete asap and not be delayed to the next tick? This would give kind of the LAZY part.

In discussions with Paul, we concluded that we need HARD because in low memory situations delaying rcu callbacks can be problematic. Given that we are already deferring the callbacks, it would be nice to have them enqueued asap. But if there are downsides to this we may change it later.

Quoted 5 lines

kernel/rcu/tree.c rcu_boot_init_percpu_data(int cpu) · line 4338

Quoted 3 lines
kernel/rcu/tree.c +0 -0
      rcu_boot_init_nocb_percpu_data(rdp);
 }
 

Sebastian

#6 lore raw

Re: [PATCH v4 1/6] rcu: Make call_rcu() safe to call from any context

Quoted 15 lines · Puranjay Mohan

On Wed, Aug 26, 2026 at 3:46 PM Sebastian Andrzej Siewior

Quoted 5 lines
Quoted 5 lines

+/*
+ * Defer whenever interrupts are disabled, since a callback-list operation may
+ * be in flight on this CPU. Not before the scheduler is up: irq_work is not
+ * usable that early, and rcu_init() itself calls call_rcu().
+ */

Quoted 8 lines

kernel/rcu/rcu.h static inline void tasks_cblist_init_generic(void) { } · line 584

Quoted 5 lines
kernel/rcu/rcu.h +5 -0
+static inline bool should_rcu_defer(void)
+{
+     return IS_ENABLED(CONFIG_RCU_DEFER) && irqs_disabled() &&
+            rcu_scheduler_active != RCU_SCHEDULER_INACTIVE;
+}

Why does the description say that the defer part is for usage from NMI and the test here has irqs_disabled() instead of in_nmi()?

Quoted 2 lines

We defer for both irq disabled sections, hard irq, and in_nmi() and checking for irqs_disabled() covers all three.

But it is wrong to talk about NMI and do this for other reasons not mentioning why this was needed/ made sense.
Can this be fixed?
Also what is the reasoning for doing it from any IRQ disabled region?

Quoted 34 lines
Quoted 25 lines
Quoted 25 lines
kernel/rcu/rcu.h +1 -0
+
 enum rcutorture_type {
      RCU_FLAVOR,
      RCU_TASKS_FLAVOR,
kernel/rcu/tree.c +12 -0
index 21b6ce1dffb63..3bf3a250f9de8 100644
@@ -3206,6 +3204,103 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
      local_irq_restore(flags);
 }
 
+/*
+ * Re-issue deferred callbacks straight to the enqueue so they cannot defer
+ * again.  ->defer_lock serializes the drainers: this CPU's irq_work,
+ * rcu_defer_flush() and rcutree_migrate_callbacks().
+ */
+static void __rcu_defer_drain(struct rcu_data *rdp)
+{
+     struct llist_node *node, *next;
+     unsigned long flags;
+
+     if (!IS_ENABLED(CONFIG_RCU_DEFER))
+             return;
Quoted 8 lines

kernel/rcu/tree.c __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) · line 3222

Quoted 4 lines
kernel/rcu/tree.c +4 -0
+
+     raw_spin_lock_irqsave(&rdp->defer_lock, flags);
+     llist_for_each_safe(node, next, llist_del_all(&rdp->defer_head)) {
+             struct rcu_head *head = (struct rcu_head *)node;

Why do you need the lock. This is still not clear to me despite the comment. You can do llist_del_all() towards another list and then feed it into rcu_do_enqueue() one by one. And you use the LAZY part.

Quoted 7 lines

Because rcu_barrier() can call this for each cpu and at the same time rcu_defer_drain() can call it too, so this would cause a race where the irq work can remove the callbacks (llist_del_all) and before it can enqueue them, rcu_barrier will see that the list is already empty and will not wait for these callbacks. We want the llist_del_all() and rcu_do_enqueue() to happen atomically so rcu_barrier() can work correctly.

So you collect a bunch of callbacks and spent time re-arranging everything with irqs off.
What is wrong with keeping it in the llist and consuming it like the regular rcu_segcblist?

Quoted 22 lines
Quoted 9 lines
Quoted 8 lines
kernel/rcu/tree.c +7 -0
+
+             /* Bounds a node self-linked by a double call_rcu(). */
+             head->next = NULL;
+             rcu_do_enqueue(head, head->func, false);
+     }
+     raw_spin_unlock_irqrestore(&rdp->defer_lock, flags);
+}

…

Quoted 12 lines

kernel/rcu/tree.c rcu_boot_init_percpu_data(int cpu) · line 4335

Quoted 7 lines
kernel/rcu/tree.c +3 -0
@@ -4231,6 +4330,9 @@ rcu_boot_init_percpu_data(int cpu)
      rdp->rcu_onl_gp_state = RCU_GP_CLEANED;
      rdp->last_sched_clock = jiffies;
      rdp->cpu = cpu;
+     init_llist_head(&rdp->defer_head);
+     raw_spin_lock_init(&rdp->defer_lock);
+     rdp->defer_work = IRQ_WORK_INIT_HARD(rcu_defer_drain);

Why is this IRQ_WORK_INIT_HARD() instead, say, IRQ_WORK_INIT_LAZY()? Is there a requirement that the RCU callback needs to complete asap and not be delayed to the next tick? This would give kind of the LAZY part.

Quoted 5 lines

In discussions with Paul, we concluded that we need HARD because in low memory situations delaying rcu callbacks can be problematic. Given that we are already deferring the callbacks, it would be nice to have them enqueued asap. But if there are downsides to this we may change it later.

It does not look like you defer them for long at all. Every callback enqueued in an IRQ-off region will be immediately re-enqueued to the regular list the moment interrupts are enabled again.
Except on architectures which don't implement irq-work interrupts where it will be delayed to the next HZ tick. And the next HZ tick does not sound like a long time either.

Quoted 3 lines
Quoted 3 lines

kernel/rcu/tree.c rcu_boot_init_percpu_data(int cpu) · line 4338

Quoted 3 lines
kernel/rcu/tree.c +0 -0
      rcu_boot_init_nocb_percpu_data(rdp);
 }
 

Sebastian

#7 lore raw

Re: [PATCH v4 1/6] rcu: Make call_rcu() safe to call from any context

Quoted 17 lines · Sebastian Andrzej Siewior
Quoted 15 lines · Puranjay Mohan

On Wed, Aug 26, 2026 at 3:46 PM Sebastian Andrzej Siewior

Quoted 13 lines

kernel/rcu/rcu.h static inline void tasks_cblist_init_generic(void) { } · line 584

Quoted 10 lines
kernel/rcu/rcu.h +10 -0
+/*
+ * Defer whenever interrupts are disabled, since a callback-list operation may
+ * be in flight on this CPU.  Not before the scheduler is up: irq_work is not
+ * usable that early, and rcu_init() itself calls call_rcu().
+ */
+static inline bool should_rcu_defer(void)
+{
+     return IS_ENABLED(CONFIG_RCU_DEFER) && irqs_disabled() &&
+            rcu_scheduler_active != RCU_SCHEDULER_INACTIVE;
+}

Why does the description say that the defer part is for usage from NMI and the test here has irqs_disabled() instead of in_nmi()?

Quoted 7 lines
Quoted 2 lines

We defer for both irq disabled sections, hard irq, and in_nmi() and checking for irqs_disabled() covers all three.

But it is wrong to talk about NMI and do this for other reasons not mentioning why this was needed/ made sense.
Can this be fixed?
Also what is the reasoning for doing it from any IRQ disabled region?

So, the main reason for this patchset is to make call_(s)rcu() work from BPF programs, and these BPF programs can run from any context like an NMI perf event interrupt or maybe attached to a function through ftrace which is in the call chain of call_rcu() itself. call_rcu() takes locks with irqs disabled or it modifies lists non-atomically with irqs disabled. This means that a BPF program that runs from NMI with the lock already taken will deadlock, or if it runs between these non-atomic list modifications, it will corrupt the lists. A BPF program can also synchronously re-enter call_rcu() if it is attached to a function in the call chain of call_rcu() that is called with a lock taken. So, all these cases are detected with irqs_disabled() and on detection we do stuff that doesn't require taking any locks and is safe even if re-entered.

Quoted 42 lines
Quoted 34 lines
Quoted 25 lines
Quoted 25 lines
kernel/rcu/rcu.h +1 -0
+
 enum rcutorture_type {
      RCU_FLAVOR,
      RCU_TASKS_FLAVOR,
kernel/rcu/tree.c +12 -0
index 21b6ce1dffb63..3bf3a250f9de8 100644
@@ -3206,6 +3204,103 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
      local_irq_restore(flags);
 }
 
+/*
+ * Re-issue deferred callbacks straight to the enqueue so they cannot defer
+ * again.  ->defer_lock serializes the drainers: this CPU's irq_work,
+ * rcu_defer_flush() and rcutree_migrate_callbacks().
+ */
+static void __rcu_defer_drain(struct rcu_data *rdp)
+{
+     struct llist_node *node, *next;
+     unsigned long flags;
+
+     if (!IS_ENABLED(CONFIG_RCU_DEFER))
+             return;
Quoted 8 lines

kernel/rcu/tree.c __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) · line 3222

Quoted 4 lines
kernel/rcu/tree.c +4 -0
+
+     raw_spin_lock_irqsave(&rdp->defer_lock, flags);
+     llist_for_each_safe(node, next, llist_del_all(&rdp->defer_head)) {
+             struct rcu_head *head = (struct rcu_head *)node;

Why do you need the lock. This is still not clear to me despite the comment. You can do llist_del_all() towards another list and then feed it into rcu_do_enqueue() one by one. And you use the LAZY part.

Quoted 7 lines

Because rcu_barrier() can call this for each cpu and at the same time rcu_defer_drain() can call it too, so this would cause a race where the irq work can remove the callbacks (llist_del_all) and before it can enqueue them, rcu_barrier will see that the list is already empty and will not wait for these callbacks. We want the llist_del_all() and rcu_do_enqueue() to happen atomically so rcu_barrier() can work correctly.

Quoted 4 lines

So you collect a bunch of callbacks and spent time re-arranging everything with irqs off.
What is wrong with keeping it in the llist and consuming it like the regular rcu_segcblist?

I don't understand your last point, all the collection and delayed addition into rcu_segcblist is because it is not safe to do it directly as I explained above.

Quoted 29 lines
Quoted 23 lines
Quoted 10 lines
Quoted 9 lines
kernel/rcu/tree.c +7 -0
+
+             /* Bounds a node self-linked by a double call_rcu(). */
+             head->next = NULL;
+             rcu_do_enqueue(head, head->func, false);
+     }
+     raw_spin_unlock_irqrestore(&rdp->defer_lock, flags);
+}

…

Quoted 12 lines

kernel/rcu/tree.c rcu_boot_init_percpu_data(int cpu) · line 4335

Quoted 7 lines
kernel/rcu/tree.c +3 -0
@@ -4231,6 +4330,9 @@ rcu_boot_init_percpu_data(int cpu)
      rdp->rcu_onl_gp_state = RCU_GP_CLEANED;
      rdp->last_sched_clock = jiffies;
      rdp->cpu = cpu;
+     init_llist_head(&rdp->defer_head);
+     raw_spin_lock_init(&rdp->defer_lock);
+     rdp->defer_work = IRQ_WORK_INIT_HARD(rcu_defer_drain);

Why is this IRQ_WORK_INIT_HARD() instead, say, IRQ_WORK_INIT_LAZY()? Is there a requirement that the RCU callback needs to complete asap and not be delayed to the next tick? This would give kind of the LAZY part.

Quoted 5 lines

In discussions with Paul, we concluded that we need HARD because in low memory situations delaying rcu callbacks can be problematic. Given that we are already deferring the callbacks, it would be nice to have them enqueued asap. But if there are downsides to this we may change it later.

Quoted 6 lines

It does not look like you defer them for long at all. Every callback enqueued in an IRQ-off region will be immediately re-enqueued to the regular list the moment interrupts are enabled again.
Except on architectures which don't implement irq-work interrupts where it will be delayed to the next HZ tick. And the next HZ tick does not sound like a long time either.

Yeah, I get your point, and when a CPU goes offline will its LAZY irq works be drained or left in the queue? Because we want to make sure this irq work is executed before a CPU goes offline.

#8 lore raw

Re: [PATCH v4 1/6] rcu: Make call_rcu() safe to call from any context

Quoted 10 lines · Puranjay Mohan
Quoted 5 lines

But it is wrong to talk about NMI and do this for other reasons not mentioning why this was needed/ made sense.
Can this be fixed?
Also what is the reasoning for doing it from any IRQ disabled region?

So, the main reason for this patchset is to make call_(s)rcu() work from BPF programs, and these BPF programs can run from any context like an NMI perf event interrupt or maybe attached to a function through ftrace which is in the call chain of call_rcu() itself.

This I expected.

Quoted 9 lines

call_rcu() takes locks with irqs disabled or it modifies lists non-atomically with irqs disabled. This means that a BPF program that runs from NMI with the lock already taken will deadlock, or if it runs between these non-atomic list modifications, it will corrupt the lists. A BPF program can also synchronously re-enter call_rcu() if it is attached to a function in the call chain of call_rcu() that is called with a lock taken. So, all these cases are detected with irqs_disabled() and on detection we do stuff that doesn't require taking any locks and is safe even if re-entered.

Attaching a BPF program to call_rcu() and re-entering sounds great. But there is nothing unique about call_rcu(), is there? We do have certain amount of functions which can be invoked from NMI and their usage is limited.

Anyway. If this is the sole reason for stunt then it _has_ to be part of the commit message. Now it looks like an ooopsie where you talk about NMI and do hardirq check.

Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
Quoted 7 lines
Quoted 7 lines
kernel/rcu/tree.c +7 -0
+static void __rcu_defer_drain(struct rcu_data *rdp)
+{
+     struct llist_node *node, *next;
+     unsigned long flags;
+
+     if (!IS_ENABLED(CONFIG_RCU_DEFER))
+             return;
Quoted 8 lines

kernel/rcu/tree.c __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) · line 3222

Quoted 4 lines
kernel/rcu/tree.c +4 -0
+
+     raw_spin_lock_irqsave(&rdp->defer_lock, flags);
+     llist_for_each_safe(node, next, llist_del_all(&rdp->defer_head)) {
+             struct rcu_head *head = (struct rcu_head *)node;

Why do you need the lock. This is still not clear to me despite the comment. You can do llist_del_all() towards another list and then feed it into rcu_do_enqueue() one by one. And you use the LAZY part.

Because rcu_barrier() can call this for each cpu and at the same time rcu_defer_drain() can call it too, so this would cause a race where the irq work can remove the callbacks (llist_del_all) and before it can enqueue them, rcu_barrier will see that the list is already empty and will not wait for these callbacks. We want the llist_del_all() and rcu_do_enqueue() to happen atomically so rcu_barrier() can work correctly.

Quoted 8 lines
Quoted 4 lines

So you collect a bunch of callbacks and spent time re-arranging everything with irqs off.
What is wrong with keeping it in the llist and consuming it like the regular rcu_segcblist?

I don't understand your last point, all the collection and delayed addition into rcu_segcblist is because it is not safe to do it directly as I explained above.

That is correct. My point is the more callbacks you add, the more time you spend to re-add them here. It is not just the one NMI callback injected from NMI, it is everyone within interrupts disabled region. After each one you trigger an IRQ. And with LAZY-irq-work or on architectures which don't have an IRQ-work interrupt, you accumulate every callback from IRQ disabled region since the last HZ tick.

The question is why not have an additional list which is handled the same way in terms of grace period like the current one?

Quoted 10 lines
Quoted 6 lines

It does not look like you defer them for long at all. Every callback enqueued in an IRQ-off region will be immediately re-enqueued to the regular list the moment interrupts are enabled again.
Except on architectures which don't implement irq-work interrupts where it will be delayed to the next HZ tick. And the next HZ tick does not sound like a long time either.

Yeah, I get your point, and when a CPU goes offline will its LAZY irq works be drained or left in the queue? Because we want to make sure this irq work is executed before a CPU goes offline.

Hmm. There is smpcfd_dying_cpu() running before rcutree_dying_cpu(). smpcfd_dying_cpu() flushes irq-work near the end. On PREEMPT_RT we have this thread handling them and this one is parked smpboot_park_threads() a bit earlier. So here smpcfd_dying_cpu() will wake the parked thread which then waits until the CPU comes back on.
We can't run the work directly due to the IRQ-off context. We could move them to another CPU but then the irq-work might expect to be run on the other CPU…
But we could flush the queue before parking at the very least I suppose…

Sebastian

#9 lore raw

Re: [PATCH v4 1/6] rcu: Make call_rcu() safe to call from any context

In reply to Puranjay Mohan
Quoted 12 lines · Puranjay Mohan
---
 kernel/rcu/Kconfig |   6 +++
 kernel/rcu/rcu.h   |  11 ++++
 kernel/rcu/tree.c  | 131 +++++++++++++++++++++++++++++++++++++++++----
 kernel/rcu/tree.h  |   6 +++
 4 files changed, 143 insertions(+), 11 deletions(-)
kernel/rcu/Kconfig +0 -0
index f15da8038d0ba..1a5fb3156c062 100644

kernel/rcu/Kconfig config RCU_STALL_COMMON · line 182

Quoted 8 lines
kernel/rcu/Kconfig +5 -0
@@ -175,6 +175,12 @@ config RCU_STALL_COMMON
 config RCU_NEED_SEGCBLIST
 	def_bool ( TREE_RCU || TREE_SRCU || TASKS_RCU_GENERIC )
 
+# The deferral (and the IRQ_WORK it uses) is only needed where call_rcu() /
+# call_srcu() can be invoked while a callback-list operation is in flight.
+config RCU_DEFER
+	def_bool HAVE_NMI || KPROBES || FUNCTION_TRACER || TRACEPOINTS
+	select IRQ_WORK

nit: I don't think this config belongs to RCU.
Do we really need this when majority of kernels enable those?

Quoted 19 lines
kernel/rcu/Kconfig +12 -0
 config RCU_FANOUT
 	int "Tree-based hierarchical RCU fanout value"
 	range 2 64 if 64BIT
@@ -3206,6 +3204,103 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
 	local_irq_restore(flags);
 }
 
+/*
+ * Stage @head for this CPU's irq_work to re-issue once interrupts are on.  Only
+ * the drain side takes a lock, so this stays safe from NMI.
+ */
+static void call_rcu_defer(struct rcu_head *head, rcu_callback_t func)
+{
+	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
+
+	/*
+	 * Instrumentation on the enqueue path can re-enter here from inside the
+	 * drain.  Re-queuing would livelock it, so drop the callback; an NMI
+	 * cannot loop, so let it through.

kernel/rcu/Kconfig __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) · line 3228

Quoted 10 lines
kernel/rcu/Kconfig +10 -0
+
+	 */
+	if (READ_ONCE(rdp->defer_draining) && !in_nmi()) {
+		WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU),
+			  "call_rcu() re-entered during callback drain; leaking callback\n");
+		return;
+	}
+	head->func = func;
+	if (llist_add((struct llist_node *)head, &rdp->defer_head))
+		irq_work_queue(&rdp->defer_work);

Should RCU wait for the IRQ work to be processed during e.g.) rcu_barrier()?

Signature · 2 lines
Cheers,
Harry / Hyeonggon
#10 lore raw

Re: [PATCH v4 1/6] rcu: Make call_rcu() safe to call from any context

Quoted 17 lines · Harry Yoo
Quoted 15 lines · Puranjay Mohan
---
 kernel/rcu/Kconfig |   6 +++
 kernel/rcu/rcu.h   |  11 ++++
 kernel/rcu/tree.c  | 131 +++++++++++++++++++++++++++++++++++++++++----
 kernel/rcu/tree.h  |   6 +++
 4 files changed, 143 insertions(+), 11 deletions(-)
kernel/rcu/Kconfig +0 -0
index f15da8038d0ba..1a5fb3156c062 100644
@@ -175,6 +175,12 @@ config RCU_STALL_COMMON
 config RCU_NEED_SEGCBLIST
      def_bool ( TREE_RCU || TREE_SRCU || TASKS_RCU_GENERIC )
 
Quoted 8 lines

kernel/rcu/Kconfig config RCU_STALL_COMMON · line 182

Quoted 5 lines
kernel/rcu/Kconfig +5 -0
+# The deferral (and the IRQ_WORK it uses) is only needed where call_rcu() /
+# call_srcu() can be invoked while a callback-list operation is in flight.
+config RCU_DEFER
+     def_bool HAVE_NMI || KPROBES || FUNCTION_TRACER || TRACEPOINTS
+     select IRQ_WORK

nit: I don't think this config belongs to RCU.
Do we really need this when majority of kernels enable those?

Paul was worried about some embedded systems paying a performance penalty as they might not enable these but would still use RCU.

Quoted 25 lines
Quoted 25 lines
kernel/rcu/Kconfig +17 -0
 config RCU_FANOUT
      int "Tree-based hierarchical RCU fanout value"
      range 2 64 if 64BIT
@@ -3206,6 +3204,103 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in)
      local_irq_restore(flags);
 }
 
+/*
+ * Stage @head for this CPU's irq_work to re-issue once interrupts are on.  Only
+ * the drain side takes a lock, so this stays safe from NMI.
+ */
+static void call_rcu_defer(struct rcu_head *head, rcu_callback_t func)
+{
+     struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
+
+     /*
+      * Instrumentation on the enqueue path can re-enter here from inside the
+      * drain.  Re-queuing would livelock it, so drop the callback; an NMI
+      * cannot loop, so let it through.
+
+      */
+     if (READ_ONCE(rdp->defer_draining) && !in_nmi()) {
+             WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU),
+                       "call_rcu() re-entered during callback drain; leaking callback\n");
Quoted 8 lines

kernel/rcu/Kconfig __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) · line 3228

Quoted 5 lines
kernel/rcu/Kconfig +5 -0
+             return;
+     }
+     head->func = func;
+     if (llist_add((struct llist_node *)head, &rdp->defer_head))
+             irq_work_queue(&rdp->defer_work);

Should RCU wait for the IRQ work to be processed during e.g.) rcu_barrier()?

Yes it should and it does, rcu_barrier() now enqueues these deferred callbacks from this list before doing the other barrier logic.

Thanks

#11 lore raw

PATCH v4 2/6 rcu: Make Tiny call_rcu() safe to call from any context

In reply to Puranjay Mohan

Give Tiny call_rcu() the same treatment as Tree RCU. When interrupts are disabled and the scheduler is up, stage the callback on a lockless list that an irq_work re-issues later. One global list and irq_work suffice since Tiny RCU is uniprocessor, and there is no CPU-offline drain.

The re-issue runs with interrupts disabled and can be re-entered by instrumentation, so a draining flag drops a deferring call_rcu() seen mid-drain (unless from an NMI), as in Tree RCU. Gated by CONFIG_RCU_DEFER, though the deferral state is unconditional.

Interrupts stay off for the whole batch, but the re-issue is a tail append with no locks. TINY_RCU implies !SMP, where arch_irq_work_has_interrupt() is false, so the drain always waits for the tick and a batch is whatever one tick's worth of interrupts-disabled call_rcu()s staged. As in Tree RCU the drain clears ->next before re-issuing, which bounds a node self-linked by a double call_rcu(): rcu_do_enqueue()'s duplicate path returns without clearing it. A longer cycle is not bounded; a double call_rcu() stays undefined.

The idle-task reschedule moves out of the enqueue helper so that a drain does it once for the batch rather than once per callback, which would otherwise take the runqueue lock N times with interrupts disabled.

---
 kernel/rcu/tiny.c | 127 +++++++++++++++++++++++++++++++++++++---------
 1 file changed, 104 insertions(+), 23 deletions(-)
kernel/rcu/tiny.c +104 -23
index dccccd6be9411..656b6a682e31a 100644
@@ -11,6 +11,8 @@
  */
 #include <linux/completion.h>
 #include <linux/interrupt.h>
+#include <linux/irq_work.h>
+#include <linux/llist.h>
 #include <linux/notifier.h>
 #include <linux/rcupdate_wait.h>
 #include <linux/kernel.h>
@@ -42,8 +44,100 @@ static struct rcu_ctrlblk rcu_ctrlblk = {
 	.gp_seq		= 0 - 300UL,
 };
 
+/*
+ * The callback list is only accessed with interrupts disabled, so a call_rcu()
+ * that arrives with interrupts off stages the callback on a lockless list that
+ * an irq_work re-issues later.  One global list and irq_work suffice, as Tiny
+ * RCU is uniprocessor.
+ */
+static void rcu_defer_drain(struct irq_work *iw);
+static LLIST_HEAD(rcu_defer_list);
+static struct irq_work rcu_defer_iw = IRQ_WORK_INIT_HARD(rcu_defer_drain);
+static bool rcu_defer_draining;
+
+/*
+ * Also called by __rcu_defer_drain() to re-issue a deferred callback, so it
+ * must not re-check the deferral condition.
+ */
+static void rcu_do_enqueue(struct rcu_head *head, rcu_callback_t func)
+{
+	static atomic_t doublefrees;
+	unsigned long flags;
+
+	if (debug_rcu_head_queue(head)) {
+		if (atomic_inc_return(&doublefrees) < 4) {
+			pr_err("%s(): Double-freed CB %p->%pS()!!!  ", __func__, head, head->func);
+			mem_dump_obj(head);
+		}
+		return;
+	}
+
+	head->func = func;
+	head->next = NULL;
+
+	local_irq_save(flags);
+	*rcu_ctrlblk.curtail = head;
+	rcu_ctrlblk.curtail = &head->next;
+	local_irq_restore(flags);
+}
+
+/* Force scheduling for rcu_qs() when enqueuing from the idle task. */
+static void rcu_resched_if_idle(void)
+{
+	if (unlikely(is_idle_task(current)))
+		resched_cpu(0);
+}
+
+static void __rcu_defer_drain(void)
+{
+	struct llist_node *node, *next;
+	bool drained = false;
+	unsigned long flags;
+
+	if (!IS_ENABLED(CONFIG_RCU_DEFER))
+		return;
+
+	/* Re-issued newest-first; nothing depends on call_rcu() ordering. */
+	local_irq_save(flags);
+	llist_for_each_safe(node, next, llist_del_all(&rcu_defer_list)) {
+		struct rcu_head *head = (struct rcu_head *)node;
+
+		/* Bounds a node self-linked by a double call_rcu(). */
+		head->next = NULL;
+		rcu_do_enqueue(head, head->func);
+		drained = true;
+	}
+	local_irq_restore(flags);
+
+	if (drained)
+		rcu_resched_if_idle();
+}
+
+/* Only the irq_work drain can be re-fed by its own re-issue; see Tree RCU. */
+static void rcu_defer_drain(struct irq_work *iw)
+{
+	WRITE_ONCE(rcu_defer_draining, true);
+	__rcu_defer_drain();
+	WRITE_ONCE(rcu_defer_draining, false);
+}
+
+static void call_rcu_defer(struct rcu_head *head, rcu_callback_t func)
+{
+	/* A re-entrant call_rcu() during the drain would livelock it; drop it. */
+	if (READ_ONCE(rcu_defer_draining) && !in_nmi()) {
+		WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU),
+			  "call_rcu() re-entered during callback drain; leaking callback\n");
+		return;
+	}
+	head->func = func;
+	if (llist_add((struct llist_node *)head, &rcu_defer_list))
+		irq_work_queue(&rcu_defer_iw);
+}
+
 void rcu_barrier(void)
 {
+	/* Register any deferred callbacks so the wait below covers them. */
+	__rcu_defer_drain();
 	wait_rcu_gp(call_rcu_hurry);
 }
 EXPORT_SYMBOL(rcu_barrier);
@@ -157,29 +251,19 @@ EXPORT_SYMBOL_GPL(synchronize_rcu);
  */
 void call_rcu(struct rcu_head *head, rcu_callback_t func)
 {
-	static atomic_t doublefrees;
-	unsigned long flags;
-
-	if (debug_rcu_head_queue(head)) {
-		if (atomic_inc_return(&doublefrees) < 4) {
-			pr_err("%s(): Double-freed CB %p->%pS()!!!  ", __func__, head, head->func);
-			mem_dump_obj(head);
-		}
+	if (should_rcu_defer()) {
+		call_rcu_defer(head, func);
 		return;
 	}
 
-	head->func = func;
-	head->next = NULL;
+	/*
+	 * Only reachable from an NMI when deferral is off: before the scheduler
+	 * is up, or with CONFIG_RCU_DEFER=n.  The enqueue can then race.
+	 */
+	WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && in_nmi());
 
-	local_irq_save(flags);
-	*rcu_ctrlblk.curtail = head;
-	rcu_ctrlblk.curtail = &head->next;
-	local_irq_restore(flags);
-
-	if (unlikely(is_idle_task(current))) {
-		/* force scheduling for rcu_qs() */
-		resched_cpu(0);
-	}
+	rcu_do_enqueue(head, func);
+	rcu_resched_if_idle();
 }
 EXPORT_SYMBOL_GPL(call_rcu);
 
@@ -211,10 +295,7 @@ unsigned long start_poll_synchronize_rcu(void)
 {
 	unsigned long gp_seq = get_state_synchronize_rcu();
 
-	if (unlikely(is_idle_task(current))) {
-		/* force scheduling for rcu_qs() */
-		resched_cpu(0);
-	}
+	rcu_resched_if_idle();
 	return gp_seq;
 }
 EXPORT_SYMBOL_GPL(start_poll_synchronize_rcu);
Signature · 1 line
2.53.0-Meta
#12 lore raw

Re: [PATCH v4 2/6] rcu: Make Tiny call_rcu() safe to call from any context

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Moving `debug_rcu_head_queue()` into `rcu_do_enqueue()` bypasses debugobjects tracking for…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Moving debug_rcu_head_queue() into rcu_do_enqueue() bypasses debugobjects tracking for deferred RCU callbacks, masking use-after-free bugs and causing list corruption on double frees. --

--- Patch [2]: [PATCH v4 2/6] rcu: Make Tiny call_rcu() safe to call from any context --- commit b9d1d9e1aa870c19c3d9b30019cdb1bc509b6703
Author: Puranjay Mohan <[email protected]>

rcu: Make Tiny call_rcu() safe to call from any context

This commit updates Tiny call_rcu() to handle interrupts-disabled contexts when the scheduler is active. It defers the callback on a lockless list which is re-issued later via an irq_work, similar to Tree RCU.

Quoted 4 lines
kernel/rcu/tiny.c +0 -0
index dccccd6be9411..656b6a682e31a 100644

[ ... ]

Quoted 6 lines
kernel/rcu/tiny.c +0 -2
@@ -157,29 +251,19 @@ EXPORT_SYMBOL_GPL(synchronize_rcu);
  */
 void call_rcu(struct rcu_head *head, rcu_callback_t func)
 {
-	static atomic_t doublefrees;
-	unsigned long flags;

kernel/rcu/tiny.c EXPORT_SYMBOL_GPL(synchronize_rcu); · line 257

Quoted 10 lines
kernel/rcu/tiny.c +2 -6
-
-	if (debug_rcu_head_queue(head)) {
-		if (atomic_inc_return(&doublefrees) < 4) {
-			pr_err("%s(): Double-freed CB %p->%pS()!!!  ", __func__, head, head->func);
-			mem_dump_obj(head);
-		}
+	if (should_rcu_defer()) {
+		call_rcu_defer(head, func);
 		return;
 	}

[Severity: High]
Does moving debug_rcu_head_queue() into rcu_do_enqueue() bypass debugobjects tracking for deferred RCU callbacks?

Execution is routed to call_rcu_defer() before debug_rcu_head_queue() can activate the object in the debugobjects tracker. Adding to the lockless list in call_rcu_defer() overwrites head->next without verifying if the RCU head is already enqueued.

If a caller erroneously frees the object or double-queues it during the deferral window before the timer tick issues the irq_work drain, could this mask use-after-free bugs or create cyclic structures on double frees?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#13 lore raw

Re: [PATCH v4 2/6] rcu: Make Tiny call_rcu() safe to call from any context

Quoted 41 lines · On Mon, Aug 10, 2026 at 1:43 PM <[email protected]> wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Moving debug_rcu_head_queue() into rcu_do_enqueue() bypasses debugobjects tracking for deferred RCU callbacks, masking use-after-free bugs and causing list corruption on double frees. --

--- Patch [2]: [PATCH v4 2/6] rcu: Make Tiny call_rcu() safe to call from any context --- commit b9d1d9e1aa870c19c3d9b30019cdb1bc509b6703
Author: Puranjay Mohan <[email protected]>

rcu: Make Tiny call_rcu() safe to call from any context

This commit updates Tiny call_rcu() to handle interrupts-disabled contexts when the scheduler is active. It defers the callback on a lockless list which is re-issued later via an irq_work, similar to Tree RCU.

Quoted 4 lines
kernel/rcu/tiny.c +0 -0
index dccccd6be9411..656b6a682e31a 100644

[ ... ]

Quoted 6 lines
kernel/rcu/tiny.c +0 -2
@@ -157,29 +251,19 @@ EXPORT_SYMBOL_GPL(synchronize_rcu);
  */
 void call_rcu(struct rcu_head *head, rcu_callback_t func)
 {
-     static atomic_t doublefrees;
-     unsigned long flags;

kernel/rcu/tiny.c EXPORT_SYMBOL_GPL(synchronize_rcu); · line 257

Quoted 10 lines
kernel/rcu/tiny.c +2 -6
-
-     if (debug_rcu_head_queue(head)) {
-             if (atomic_inc_return(&doublefrees) < 4) {
-                     pr_err("%s(): Double-freed CB %p->%pS()!!!  ", __func__, head, head->func);
-                     mem_dump_obj(head);
-             }
+     if (should_rcu_defer()) {
+             call_rcu_defer(head, func);
              return;
      }

[Severity: High]
Does moving debug_rcu_head_queue() into rcu_do_enqueue() bypass debugobjects tracking for deferred RCU callbacks?

Quoted 8 lines

Execution is routed to call_rcu_defer() before debug_rcu_head_queue() can activate the object in the debugobjects tracker. Adding to the lockless list in call_rcu_defer() overwrites head->next without verifying if the RCU head is already enqueued.

If a caller erroneously frees the object or double-queues it during the deferral window before the timer tick issues the irq_work drain, could this mask use-after-free bugs or create cyclic structures on double frees?

double calling call_rcu() is already a BUG and doing it from NMI is one level more. debug_rcu_head_queue() can't be called from NMI, until we have something that can work from NMI we have to go with the current implementation.

#14 lore raw

PATCH v4 3/6 srcu: Make call_srcu() safe to call from any context

In reply to Puranjay Mohan

call_srcu() has the same constraint as call_rcu():
srcu_gp_start_if_needed() enqueues under raw_spin_lock_irqsave() and may walk the srcu_node tree, as do callback invocation and grace-period work, so a call_srcu() with interrupts already disabled can race an operation in flight on this CPU. call_rcu_tasks_trace() is call_srcu() under the hood, so a sleepable BPF program freeing an object can reach this.

Defer as call_rcu() does: stage the callback on the srcu_data's ->defer_cbs, chain that srcu_data onto a per-CPU list, and raise a per-CPU irq_work that re-issues it straight to the enqueue helper, never back through __call_srcu(). The irqs-enabled path is unchanged; as for call_rcu() the gate is bare irqs_disabled(), so callers that merely hold interrupts off are deferred too and pay one irq_work hop, including call_rcu_tasks_trace() from the BPF memalloc irq_work.

The irq_work is per-CPU rather than per-srcu_struct and statically initialized, so deferral never runs check_init_srcu_struct(); it is IRQ_WORK_INIT_HARD as for call_rcu(). srcu_barrier() flushes it first, and rcutree_migrate_callbacks() calls srcu_offline_drain() for an outgoing CPU. cleanup_srcu_struct() drains before its "just leak it" early returns, and srcu_module_going() before freeing any ->sda, since a staged srcu_data left chained on a per-CPU list would dangle. Staging is two steps, the callback onto
->defer_cbs and then the srcu_data onto the per-CPU list, so a flusher can find the per-CPU list empty while a callback whose call_srcu() has not returned sits on ->defer_cbs; the staging CPU's own irq_work takes that one.

The per-CPU srcu_defer ->lock, not any srcu_data's, is held with interrupts off across the whole nested drain: the chain of srcu_datas staged on that CPU and, for each, its callbacks, with srcu_do_enqueue() taking that srcu_data's ->lock and possibly starting a grace period for every one. That is what serializes the drainers. The bound is as for call_rcu(): what one interrupts-disabled region could stage, normally a single callback, or whatever accumulates before the tick where arch_irq_work_has_interrupt() is false. Both lists are drained newest-first; nothing depends on call_srcu() ordering.

The drain clears ->next before re-issuing, which bounds a node self-linked by a double call_srcu(); a longer cycle is not bounded, and a double call_srcu() stays undefined, as for call_rcu().

A callback deferred past the CPUHP_AP_SMPCFD_DYING irq_work flush leaves that CPU's srcu_defer ->iw claimed with its self-IPI lost, as for call_rcu(). srcu_offline_drain() still re-issues the callback, but the irq_work cannot be un-queued, and here the claim is shared by every srcu_struct on the CPU.

As in call_rcu(), the re-issue runs with interrupts disabled and can be re-entered by instrumentation, so a per-CPU flag, set only while that CPU is inside its own irq_work drain, drops a deferring call_srcu() seen mid-drain unless it comes from an NMI. Such a drop can strand state the caller associated with the callback, not just the callback itself.

Staging records only the callback, so a deferred expedited call_srcu() completes as a normal grace period. Only srcu_expedite_current() can hit that, and only when invoked with interrupts already disabled.

Gated by CONFIG_RCU_DEFER, though the srcu_data members and the per-CPU srcu_defer are unconditional. Under CONFIG_PROVE_RCU, warn if the direct path is reached from an NMI.

---
 include/linux/srcutree.h |   4 +
 kernel/rcu/rcu.h         |   3 +
 kernel/rcu/srcutree.c    | 170 ++++++++++++++++++++++++++++++++++++++-
 kernel/rcu/tree.c        |   2 +
 4 files changed, 175 insertions(+), 4 deletions(-)
include/linux/srcutree.h +4 -0
index 75e54e4f963fa..1ce759fb70948 100644
@@ -13,6 +13,8 @@
 
 #include <linux/rcu_node_tree.h>
 #include <linux/completion.h>
+#include <linux/irq_work_types.h>
+#include <linux/llist.h>
 
 struct srcu_node;
 struct srcu_struct;
@@ -41,6 +43,8 @@ struct srcu_data {
 	bool srcu_cblist_invoking;		/* Invoking these CBs? */
 	struct timer_list delay_work;		/* Delay for CB invoking */
 	struct work_struct work;		/* Context for CB invoking. */
+	struct llist_head defer_cbs;		/* Callbacks deferred on re-entry. */
+	struct llist_node defer_link;		/* Links onto the per-CPU deferral drain list */
 	struct rcu_head srcu_barrier_head;	/* For srcu_barrier() use. */
 	struct rcu_head srcu_ec_head;		/* For srcu_expedite_current() use. */
 	int srcu_ec_state;			/*  State for srcu_expedite_current(). */
kernel/rcu/rcu.h +3 -0
index 91e33571a554d..d60444bf3a02d 100644
@@ -583,6 +583,9 @@ static inline bool should_rcu_defer(void)
 	       rcu_scheduler_active != RCU_SCHEDULER_INACTIVE;
 }
 
+/* Drain an outgoing CPU's deferred SRCU callbacks; see rcutree_migrate_callbacks(). */
+void srcu_offline_drain(int cpu);
+
 enum rcutorture_type {
 	RCU_FLAVOR,
 	RCU_TASKS_FLAVOR,
kernel/rcu/srcutree.c +166 -4

226 lines not on this page · raw message

kernel/rcu/tree.c +2 -0
index 3bf3a250f9de8..7a9b18825e029 100644
@@ -4636,6 +4636,8 @@ void rcutree_migrate_callbacks(int cpu)
 	 * cover.  Drain them here, before the early returns.
 	 */
 	__rcu_defer_drain(rdp);
+	/* Likewise for the outgoing CPU's deferred call_srcu() callbacks. */
+	srcu_offline_drain(cpu);
 
 	if (rcu_rdp_is_offloaded(rdp))
 		return;
Signature · 1 line
2.53.0-Meta
#15 lore raw

PATCH v4 4/6 srcu: Make Tiny call_srcu() safe to call from any context

In reply to Puranjay Mohan

Give Tiny call_srcu() the same treatment as Tree SRCU. When interrupts are disabled and the scheduler is up, stage the callback on the srcu_struct's lockless list for an irq_work to re-issue later. Tiny SRCU is uniprocessor, so there is no CPU-offline drain. A draining flag drops a deferring call_srcu() that re-enters mid-drain (unless from an NMI), as in Tree SRCU; such a drop can strand state the caller tied to the callback, not just the callback itself.

Interrupts stay off for the whole batch. TINY_SRCU implies !SMP, where arch_irq_work_has_interrupt() is false, so the drain always waits for the tick and a batch is whatever one tick's worth of interrupts-disabled call_srcu()s staged. Unlike the other three flavors srcu_do_enqueue() here has no debug_rcu_head_queue(), so nothing reports a double call_srcu(); termination of the drain rests on srcu_do_enqueue() clearing ->next, and the callback list self-links at the tail exactly as a double call_srcu() made it before.

srcu_barrier() (now out of line) and cleanup_srcu_struct() drain the deferred list first, so a deferred callback is re-issued onto the callback list and invoked by the grace-period work that cleanup_srcu_struct() flushes, rather than stranded on a soon-to-be-freed srcu_struct. cleanup_srcu_struct() also syncs ->defer_iw, since that irq_work is embedded in the srcu_struct the caller is about to free.

The draining flag is global rather than per-srcu_struct: a re-entrant call_srcu(B) inside a drain of A raises B's own ->defer_iw, whose drain can stage back onto A, so a per-srcu_struct flag would not break the chain. The cost is that a drain of A also drops a non-NMI call_srcu() to any other srcu_struct for its duration.

Gated by CONFIG_RCU_DEFER like Tree SRCU, though the srcu_struct members are unconditional.

---
 include/linux/srcutiny.h | 12 ++++--
 kernel/rcu/srcutiny.c    | 93 ++++++++++++++++++++++++++++++++++++++--
 2 files changed, 97 insertions(+), 8 deletions(-)
include/linux/srcutiny.h +8 -4
index fbcf13bc12d15..85b5de438450b 100644
@@ -12,6 +12,7 @@
 #define _LINUX_SRCU_TINY_H
 
 #include <linux/irq_work_types.h>
+#include <linux/llist.h>
 #include <linux/swait.h>
 
 struct srcu_struct {
@@ -26,6 +27,8 @@ struct srcu_struct {
 	struct rcu_head **srcu_cb_tail;	/* Pending callbacks: Tail. */
 	struct work_struct srcu_work;	/* For driving grace periods. */
 	struct irq_work srcu_irq_work;	/* Defer schedule_work() to irq work. */
+	struct llist_head defer_cbs;	/* Callbacks deferred on re-entry. */
+	struct irq_work defer_iw;	/* Re-issues defer_cbs later. */
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 	struct lockdep_map dep_map;
 #endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
@@ -33,6 +36,7 @@ struct srcu_struct {
 
 void srcu_drive_gp(struct work_struct *wp);
 void srcu_tiny_irq_work(struct irq_work *irq_work);
+void srcu_defer_drain(struct irq_work *irq_work);
 
 #define __SRCU_STRUCT_INIT(name, __ignored, ___ignored, ____ignored)	\
 {									\
@@ -40,6 +44,9 @@ void srcu_tiny_irq_work(struct irq_work *irq_work);
 	.srcu_cb_tail = &name.srcu_cb_head,				\
 	.srcu_work = __WORK_INITIALIZER(name.srcu_work, srcu_drive_gp),	\
 	.srcu_irq_work = { .func = srcu_tiny_irq_work },		\
+	.defer_cbs = LLIST_HEAD_INIT(name.defer_cbs),			\
+	.defer_iw = { .node = { .u_flags = IRQ_WORK_HARD_IRQ },		\
+		      .func = srcu_defer_drain },			\
 	__SRCU_DEP_MAP_INIT(name)					\
 }
 
@@ -131,10 +138,7 @@ static inline void synchronize_srcu_expedited(struct srcu_struct *ssp)
 	synchronize_srcu(ssp);
 }
 
-static inline void srcu_barrier(struct srcu_struct *ssp)
-{
-	synchronize_srcu(ssp);
-}
+void srcu_barrier(struct srcu_struct *ssp);
 
 static inline void srcu_expedite_current(struct srcu_struct *ssp) { }
 #define srcu_check_read_flavor(ssp, read_flavor) do { } while (0)
kernel/rcu/srcutiny.c +89 -4

136 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#16 lore raw

PATCH v4 5/6 rcutorture: Exercise ->call() from NMI context

In reply to Puranjay Mohan

call_rcu() and call_srcu() are now safe to invoke from NMI, but rcutorture never does, leaving the deferral path untested.

Add an ->nmi_capable flag to rcu_torture_ops. For flavors that set it, arm a per-CPU hardware perf counter whose overflow handler submits a callback via ->call(). The handler acts only when in_nmi(), so only a genuine NMI exercises the deferral path. One preallocated callback per CPU is kept in flight, guarded by an atomic, to avoid allocating in NMI. The counter uses a fixed sample period rather than a frequency: a frequency-based event sets TICK_DEP_BIT_PERF_EVENTS and would pin the tick for the whole run on NO_HZ_FULL kernels.

Report the count issued from NMI ("nmi-calls:") and the count invoked ("nmi-cbs:"). rcu_torture_cleanup() disables the counters and then calls cb_barrier(), which drains every deferred callback, so the two counts must then match; a mismatch fails the test. This relies on rcu_barrier()/srcu_barrier() flushing deferred callbacks, as added earlier in the series.

Set ->nmi_capable on the NMI-safe flavors: rcu, srcu, srcud, and tasks-tracing (call_srcu() under the hood). Tasks and Tasks Rude are left alone, as call_rcu_tasks_generic() is not yet NMI-safe.

Enabled by default; the nmi_calls parameter disables it, which helps rule NMI handling in or out when triaging a failure. Requires CONFIG_PERF_EVENTS and a hardware PMU: without one nothing is issued from NMI and the end-of-test check compares zero against zero, so a pass does not by itself mean the path ran. That is the case under kvm.sh, which boots qemu with -cpu kvm64 and no vPMU; rcu_torture_nmi_cleanup() says so on the console.

---
 .../admin-guide/kernel-parameters.txt         |   7 +
 kernel/rcu/rcutorture.c                       | 150 +++++++++++++++++-
 2 files changed, 155 insertions(+), 2 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +7 -0
index 6ebc1cd26c7c8..a3d7a4f8bca33 100644
@@ -6198,6 +6198,13 @@ Kernel parameters
 			stress RCU, they don't participate in the actual
 			test, hence the "fake".
 
+	rcutorture.nmi_calls= [KNL]
+			Enable issuing RCU callbacks from an NMI, on the
+			RCU flavors that support it, to exercise the
+			any-context callback path.  Requires
+			CONFIG_PERF_EVENTS and a hardware PMU; without
+			both, nothing is issued.  Defaults to enabled.
+
 	rcutorture.nocbs_nthreads= [KNL]
 			Set number of RCU callback-offload togglers.
 			Zero (the default) disables toggling.
kernel/rcu/rcutorture.c +148 -2

246 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#17 lore raw

PATCH v4 6/6 selftests/bpf: Add a call_srcu() re-entry reproducer

In reply to Puranjay Mohan

Re-enter call_srcu() from a BPF program to exercise its any-context safety, via call_rcu_tasks_trace(), which is call_srcu() on rcu_tasks_trace_srcu_struct.

An fentry program on rcu_segcblist_enqueue() fires mid-enqueue: that function is reached from srcu_gp_start_if_needed() with the srcu_data ->lock held. The program does a task-storage delete, whose only deferred work is call_rcu_tasks_trace(), re-entering the enqueue on the same CPU. The handler matches on TID and fires once; pinning the thread removes the migration window between picking the srcu_data and taking its lock.

Without the fix the nested call re-takes the same sdp lock and self-deadlocks; with it the nested __call_srcu() sees interrupts disabled and defers via irq_work, so the delete returns and the test passes.

The test skips where it does not apply: Tiny RCU has no rcu_segcblist_enqueue() to attach to, and a UP+PREEMPT kernel pairs Tree RCU with Tiny SRCU, so the attach succeeds but call_srcu() never reaches the enqueue. Tiny SRCU is told apart by srcu_expedite_current(), which it stubs out, so on Tree SRCU a zero hit count fails rather than skips and the reproducer cannot quietly stop reproducing.

---
 .../selftests/bpf/prog_tests/rcu_reentry.c    | 93 +++++++++++++++++++
 .../testing/selftests/bpf/progs/rcu_reentry.c | 51 ++++++++++
 2 files changed, 144 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/rcu_reentry.c
 create mode 100644 tools/testing/selftests/bpf/progs/rcu_reentry.c
tools/testing/selftests/bpf/prog_tests/rcu_reentry.c new file +93 -0
new file mode 100644
index 0000000000000..de23a14b3d408
@@ -0,0 +1,93 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Exercise re-entry into call_srcu() from BPF; see progs/rcu_reentry.c. */
+#define _GNU_SOURCE
+#include <sched.h>
+#include <test_progs.h>
+#include "task_local_storage_helpers.h"
+#include "trace_helpers.h"
+#include "rcu_reentry.skel.h"
+
+/* Tiny RCU has no rcu_segcblist_enqueue() to attach to. */
+static bool have_attach_target(void)
+{
+	unsigned long long addr;
+
+	return kallsyms_find("rcu_segcblist_enqueue", &addr) == 0;
+}
+
+/* Tiny SRCU stubs out srcu_expedite_current(); Tree SRCU exports it. */
+static bool have_tree_srcu(void)
+{
+	unsigned long long addr;
+
+	return kallsyms_find("srcu_expedite_current", &addr) == 0;
+}
+
+void test_rcu_reentry(void)
+{
+	struct rcu_reentry *skel;
+	int err, pidfd = -1, map_fd;
+	cpu_set_t set, old_set;
+	bool affinity_saved;
+	__u64 val = 1;
+	int cpu;
+
+	if (!have_attach_target()) {
+		test__skip();
+		return;
+	}
+
+	skel = rcu_reentry__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "skel_open_and_load"))
+		return;
+
+	err = rcu_reentry__attach(skel);
+	if (!ASSERT_OK(err, "skel_attach"))
+		goto out;
+
+	/* Keep the re-entry on a single CPU; a cpuset may exclude CPU 0. */
+	affinity_saved = !sched_getaffinity(0, sizeof(old_set), &old_set);
+	cpu = sched_getcpu();
+	if (!ASSERT_GE(cpu, 0, "getcpu"))
+		goto out;
+	CPU_ZERO(&set);
+	CPU_SET(cpu, &set);
+	if (!ASSERT_OK(sched_setaffinity(0, sizeof(set), &set), "setaffinity"))
+		goto out;
+
+	pidfd = sys_pidfd_open(getpid(), 0);
+	if (!ASSERT_GE(pidfd, 0, "pidfd_open"))
+		goto restore;
+	map_fd = bpf_map__fd(skel->maps.task_stg);
+	err = bpf_map_update_elem(map_fd, &pidfd, &val, BPF_NOEXIST);
+	if (!ASSERT_OK(err, "boot_create"))
+		goto restore;
+
+	/* Arm the handler for this thread, then trigger call_rcu_tasks_trace(). */
+	skel->bss->target_pid = syscall(__NR_gettid);
+	err = bpf_map_delete_elem(map_fd, &pidfd);
+	if (!ASSERT_OK(err, "boot_delete"))
+		goto restore;
+
+	/*
+	 * Only Tree SRCU reaches rcu_segcblist_enqueue() from call_srcu(); a
+	 * UP+PREEMPT kernel pairs Tree RCU with Tiny SRCU, so the attach
+	 * succeeds but nothing fires.  On Tree SRCU it must fire.
+	 */
+	if (!skel->bss->hits) {
+		if (have_tree_srcu())
+			ASSERT_GT(skel->bss->hits, 0, "prog_fired");
+		else
+			test__skip();
+		goto restore;
+	}
+	ASSERT_EQ(skel->bss->get_errs, 0, "nested_storage_get");
+	ASSERT_EQ(skel->bss->del_errs, 0, "nested_storage_delete");
+restore:
+	if (affinity_saved)
+		sched_setaffinity(0, sizeof(old_set), &old_set);
+out:
+	if (pidfd >= 0)
+		close(pidfd);
+	rcu_reentry__destroy(skel);
+}
tools/testing/selftests/bpf/progs/rcu_reentry.c new file +51 -0

51 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
maintainer
#18 lore raw
In reply to Puranjay Mohan
Quoted 11 lines · Puranjay Mohan

call_rcu() and call_srcu() only ever touch their per-CPU callback lists with interrupts disabled: the enqueue runs under local_irq_save() (and the nocb locks when offloaded), and so do callback invocation and grace-period work. That is fine as long as call_rcu() itself is invoked with interrupts enabled, but it is not always. An NMI handler can call call_rcu(), and instrumentation can reenter it. The case that prompted this is a BPF program attached to rcu_segcblist_enqueue() that frees an
object: the free reaches call_rcu_tasks_trace(), which is call_srcu() under the hood, back on the same CPU with the srcu_data lock already held, and it deadlocks on that lock. Either way, enqueuing directly can corrupt the list or deadlock.

Queued for testing and further review, thank you!

Thanx, Paul

Quoted 195 lines

Rather than scatter context checks through the enqueue, make it defer whenever interrupts are disabled: stage the callback on a per-CPU lockless list and re-issue it from an irq_work once interrupts are back on, going straight to the enqueue helper so the re-issue cannot defer again. Only the drain side takes a lock; the staging is a bare llist_add() and stays safe from NMI. This is behind a new hidden CONFIG_RCU_DEFER, which is set wherever a reentrant enqueue is possible (HAVE_NMI, KPROBES, FUNCTION_TRACER or TRACEPOINTS); without it call_rcu() enqueues exactly as before. The gate is bare irqs_disabled(), so callers that merely hold interrupts off are deferred too and pay one irq_work hop.

CPU offline is the awkward part. A callback can be deferred very late in the outgoing CPU's teardown -- from do_idle() or cpuhp_ap_report_dead(), past the CPUHP_AP_SMPCFD_DYING flush that would otherwise run the irq_work -- so the irq_work can no longer run there to re-issue it. rcu_barrier() and srcu_barrier() therefore drain every CPU's deferred list themselves before they wait. They drain rather than wait the irq_work out because irq_work_sync() parks on an rcuwait, which holds a single waiter, so two concurrent barriers would clobber each other's wakeup. rcutree_migrate_callbacks() drains the outgoing CPU's list too, so a late deferral still lands on a callback list even when nobody calls a barrier. To keep those drainers from stepping on each other, the drain holds a per-CPU raw lock across the llist_del_all() and the re-issue, so a drainer never returns having pulled callbacks off the deferred list but not yet put them on a callback list. Every lock the re-issue touches (nocb, rcu_node, srcu_data) is already raw, so the nesting is fine.

The drain re-issues with interrupts disabled, so instrumentation on the enqueue path can re-enter call_rcu()/call_srcu() from inside it, stage another callback, re-raise the irq_work, and the drain never finishes. A per-CPU flag catches that: a deferral that arrives while that CPU is inside its own irq_work drain, and is not from an NMI, is dropped rather than staged, with a WARN_ONCE() under CONFIG_PROVE_RCU whose backtrace names the instrumentation responsible. Dropping leaks that callback, and can strand state its caller tied to it, but the alternative is a CPU that never leaves the drain, and the producer is a BPF program that emits one callback per enqueue, so there is nothing finite to wait for. Only the irq_work drain sets the flag: a direct drain from a barrier or from CPU-offline re-issues onto the current CPU, so anything staged during it is picked up by that CPU's own irq_work rather than feeding the drain in progress, and no legitimate callback is dropped. Instrumenting the irq_work machinery itself can still loop, as it can for any irq_work user, and is not something this series can fix.

The irq_work is IRQ_WORK_INIT_HARD in all four flavors. It is not needed for correctness, but a non-HARD irq_work runs from a kthread on PREEMPT_RT and can be delayed under load, letting deferred callbacks pile up; running the re-issue in hard-irq context keeps that from turning into an OOM.

Patches 1 and 2 do Tree and Tiny RCU, 3 and 4 Tree and Tiny SRCU. Patch 5 teaches rcutorture to issue ->call() from a perf-overflow NMI -- the nmi_calls parameter, on by default -- on the flavors that advertise it, and checks that every callback issued from NMI is later invoked. Patch 6 adds the BPF reentry reproducer described above.

Changelog:
v3: https://lore.kernel.org/rcu/[email protected]/
Changes in v4:
- rcu_barrier() and srcu_barrier() no longer skip a CPU whose deferred list looks empty. The lockless llist_empty() test added in v3 skipped the lock that makes a drain conclusive: a concurrent drainer can already have emptied the list without having re-issued yet, so the barrier could return before those callbacks reached a callback list.
- Tree SRCU: drain before cleanup_srcu_struct()'s "just leak it" early returns, and in srcu_module_going() before it frees any ->sda. Deferral skips check_init_srcu_struct(), so srcu_module_going() could skip cleanup_srcu_struct() and free ->sda with a staged srcu_data still chained on a per-CPU list.
- Dropped ->defer_exp: a deferred expedited call_srcu() now completes as a normal grace period. The flag lived on the srcu_data rather than on the callback, so it expedited whole batches; only srcu_expedite_current() with interrupts already disabled can reach this, so the downgrade costs a delay and nothing else.
- rcutorture: use a fixed perf sample period rather than .freq, which sets TICK_DEP_BIT_PERF_EVENTS for every CPU and pinned the tick for whole runs of TREE04 and TREE07. Also document nmi_calls, and say on the console when there is no PMU, since the issued==invoked check then compares zero to zero.
- kasan_record_aux_stack() moved to __call_rcu_common(), so a use-after-free report names the caller rather than the irq_work.
- Known gap, documented in patch 1: a callback deferred past the CPUHP_AP_SMPCFD_DYING flush leaves ->defer_work claimed with its self-IPI lost. rcutree_migrate_callbacks() still re-issues the callback, but the first deferral after that CPU comes back raises no IPI and waits for the next irq_work there, or for rcu_barrier().

v2: https://lore.kernel.org/rcu/[email protected]/
Changes in v3:
- Barriers no longer call irq_work_sync() on an online CPU's ->defer_work. irq_work_sync() waits on an rcuwait, which holds exactly one task, so two concurrent rcu_barrier()s syncing the same per-CPU irq_work could lose a wakeup and hang. Both flushes now drain every CPU directly, guarded by a lockless llist_empty() test so the no-deferrals case stays cheap.
- The re-entry guard is now set only by the irq_work drain, which always runs on the CPU owning the list it drains. In v2 a barrier draining a remote CPU set the flag on the draining CPU, so an unrelated irqs-off call_rcu() there was dropped and leaked even though it could not have fed the drain.
- The drain clears ->next before re-issuing. A double call_rcu() on a head that is already debug-object-active makes llist_add() self-link it, and rcu_do_enqueue()'s double-free path returns without clearing ->next, so the drain span looped forever with interrupts disabled.
- An expedited call_srcu() is no longer silently downgraded: ->defer_exp records it per srcu_data and the batch is re-issued expedited. Only srcu_expedite_current() is affected, __synchronize_srcu() sleeps and so is never deferred.
- The drop is now WARN_ONCE() under CONFIG_PROVE_RCU rather than an unconditional WARN, so instrumentation cannot reboot a panic_on_warn kernel; the backtrace is what identifies the offending program.
- Tiny RCU and Tiny SRCU use IRQ_WORK_INIT_HARD like the Tree flavors, and READ_ONCE()/WRITE_ONCE() on their draining flags. TINY_SRCU is "default y if !SMP" with no PREEMPT_RT dependency, so it really can be built on RT where a non-HARD irq_work waits on the irq_workd kthread.
- Tiny SRCU: cleanup_srcu_struct() drains *and* irq_work_sync()s ->defer_iw. That irq_work is embedded in the srcu_struct the caller is about to free, unlike the Tree flavors' static per-CPU ones.
- rcutorture: drive the perf counters from CPU-hotplug callbacks. The one-shot for_each_online_cpu() loop lost them at the first CPU offline, after which the end-of-test issued==invoked check compared 0 == 0.
- rcutorture: per-CPU rcu_head instead of one global, so several CPUs can race the drain; count the call before issuing it so mid-run stats cannot show nmi-cbs > nmi-calls; release the perf events on the torture_cleanup_begin() early-return path; print nmi_calls in the module banner; report when nmi_calls is set but no NMI ->call() ever happened; and drop sample_freq to 100, since 1000 made perf lower the system-wide perf_event_max_sample_rate tenfold.
- selftests/bpf: the old ASSERT_EQ(reentered, 1) could not fail, and an atomic allocation failure in the nested task-storage delete made the test pass without ever re-entering call_srcu(). It now records and asserts the helper return values, probes for rcu_segcblist_enqueue() up front so a Tiny kernel skips instead of failing, and restores the CPU affinity it changes.

Testing: rcutorture rcu/srcu/srcud/tasks-tracing with nmi_calls, hotplug, barriers and nocb toggling issued ~81000 callbacks from NMI with none lost, under PROVE_LOCKING, PROVE_RAW_LOCK_NESTING, DEBUG_OBJECTS_RCU_HEAD and RCU_LAZY, with no lockdep reports; SRCU-T, SRCU-U, TINY01 and TINY02 pass; the BPF reproducer passes.

v1: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Fixed the re-entry livelock Zqiang spotted: a BPF program on the enqueue path re-enters call_srcu() from inside srcu_defer_drain(), stages another callback and re-raises the irq_work, so the drain never finishes. A per-CPU flag now drops such a deferral, with a warning, unless it comes from an NMI.
- cleanup_srcu_struct(): drain the deferred callbacks before syncing ->irq_work rather than after, since re-issuing one can start a grace period and re-queue that irq_work (Zqiang).
- Tiny SRCU: sync ->defer_iw in cleanup_srcu_struct() as well, so a deferred callback is re-issued onto ->srcu_cb_head where the leak checks can see it instead of being stranded on a soon-to-be-freed srcu_struct.
- Added Kumar's ack to the BPF selftest patch.

Testing: rcutorture rcu, srcu, srcud and tasks-tracing, each with nmi_calls, CPU hotplug and barriers, and nocb toggling for rcu, all End of test: SUCCESS with the count issued from NMI equal to the count invoked (17987, 2924, 2661 and 4412 respectively) and clean consoles, under PROVE_LOCKING, PROVE_RAW_LOCK_NESTING and DEBUG_OBJECTS_RCU_HEAD; SRCU-T, SRCU-U, TINY01 and TINY02 pass; the BPF reproducer passes. Build-tested x86_64 with TREE_RCU+TREE_SRCU, TINY_RCU+TINY_SRCU and TREE_RCU+TINY_SRCU, each with CONFIG_RCU_DEFER=y and =n, plus arm64 TREE_RCU+TREE_SRCU with RCU_NOCB_CPU and RCU_LAZY.

Puranjay Mohan (6):
rcu: Make call_rcu() safe to call from any context
rcu: Make Tiny call_rcu() safe to call from any context
srcu: Make call_srcu() safe to call from any context
srcu: Make Tiny call_srcu() safe to call from any context
rcutorture: Exercise ->call() from NMI context selftests/bpf: Add a call_srcu() re-entry reproducer

 .../admin-guide/kernel-parameters.txt         |   7 +
 include/linux/srcutiny.h                      |  12 +-
 include/linux/srcutree.h                      |   4 +
 kernel/rcu/Kconfig                            |   6 +
 kernel/rcu/rcu.h                              |  14 ++
 kernel/rcu/rcutorture.c                       | 150 +++++++++++++++-
 kernel/rcu/srcutiny.c                         |  93 +++++++++-
 kernel/rcu/srcutree.c                         | 170 +++++++++++++++++-
 kernel/rcu/tiny.c                             | 127 ++++++++++---
 kernel/rcu/tree.c                             | 133 ++++++++++++--
 kernel/rcu/tree.h                             |   6 +
 .../selftests/bpf/prog_tests/rcu_reentry.c    |  93 ++++++++++
 .../testing/selftests/bpf/progs/rcu_reentry.c |  51 ++++++
 13 files changed, 818 insertions(+), 48 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/rcu_reentry.c
 create mode 100644 tools/testing/selftests/bpf/progs/rcu_reentry.c

base-commit: 9dc303e69bcd49f9668ca090ae45325269531fbb

Signature · 1 line
2.53.0-Meta