Skip to contentSkip to conversation

RCU fixes for v6.13

PATCH In review Frederic Weisbecker 2 participants 10 messages Nov 6, 2024 – Nov 11, 2024

Subsystems (5)
lore
Share & tools
lore

Frederic Weisbecker

author maintainer
#0 lore raw

cover RCU fixes for v6.13

Hello,

Please find below the general RCU fixes targeted for the upcoming merge window.

Michal Schmidt (1):
rcu/srcutiny: don't return before reenabling preemption

Paul E. McKenney (6):
doc: Add rcuog kthreads to kernel-per-CPU-kthreads.rst
rcu: Allow short-circuiting of synchronize_rcu_tasks_rude()
rcu: Permit start_poll_synchronize_rcu*() with interrupts disabled
rcutorture: Test start-poll primitives with interrupts disabled
doc: Remove kernel-parameters.txt entry for rcutorture.read_exit
rcu-tasks: Remove open-coded one-byte cmpxchg() emulation

Uladzislau Rezki (Sony) (1):
rcu/kvfree: Fix data-race in __mod_timer / kvfree_call_rcu

 .../admin-guide/kernel-parameters.txt         |  5 -----
 .../admin-guide/kernel-per-CPU-kthreads.rst   |  2 +-
 kernel/rcu/rcutorture.c                       | 10 +++++++++
 kernel/rcu/srcutiny.c                         |  2 +-
 kernel/rcu/tasks.h                            | 20 +++---------------
 kernel/rcu/tree.c                             | 21 +++++++++++--------
 6 files changed, 27 insertions(+), 33 deletions(-)

Thanks.

Signature · 1 line
2.46.0
author maintainer
#1 lore raw

PATCH 1/8 doc: Add rcuog kthreads to kernel-per-CPU-kthreads.rst

From: "Paul E. McKenney" <[email protected]>

This commit adds the rcuog kthreads to the list of callback-offloading kthreads that can be affinitied away from worker CPUs.

---
 Documentation/admin-guide/kernel-per-CPU-kthreads.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/admin-guide/kernel-per-CPU-kthreads.rst +1 -1
index b6aeae3327ce..ea7fa2a8bbf0 100644
@@ -315,7 +315,7 @@ To reduce its OS jitter, do at least one of the following:
 	to do.
 
 Name:
-  rcuop/%d and rcuos/%d
+  rcuop/%d, rcuos/%d, and rcuog/%d
 
 Purpose:
   Offload RCU callbacks from the corresponding CPU.
Signature · 1 line
2.46.0
author maintainer
#2 lore raw

PATCH 2/8 rcu: Allow short-circuiting of synchronize_rcu_tasks_rude()

In reply to Frederic Weisbecker

From: "Paul E. McKenney" <[email protected]>

There are now architectures for which all deep-idle and entry-exit functions are properly inlined or marked noinstr. Such architectures do not need synchronize_rcu_tasks_rude(), or will not once RCU Tasks has been modified to pay attention to idle tasks. This commit therefore allows a CONFIG_ARCH_HAS_NOINSTR_MARKINGS Kconfig option to turn synchronize_rcu_tasks_rude() into a no-op.

To facilitate testing, kernels built by rcutorture scripting will enable RCU Tasks Trace even on systems that do not need it.

[ paulmck: Apply Peter Zijlstra feedback. ]

---
 kernel/rcu/tasks.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/rcu/tasks.h +2 -1
index 6333f4ccf024..dd9730fd44fb 100644
@@ -1398,7 +1398,8 @@ static void call_rcu_tasks_rude(struct rcu_head *rhp, rcu_callback_t func)
  */
 void synchronize_rcu_tasks_rude(void)
 {
-	synchronize_rcu_tasks_generic(&rcu_tasks_rude);
+	if (!IS_ENABLED(CONFIG_ARCH_WANTS_NO_INSTR) || IS_ENABLED(CONFIG_FORCE_TASKS_RUDE_RCU))
+		synchronize_rcu_tasks_generic(&rcu_tasks_rude);
 }
 EXPORT_SYMBOL_GPL(synchronize_rcu_tasks_rude);
 
Signature · 1 line
2.46.0
author maintainer
#3 lore raw

PATCH 3/8 rcu: Permit start_poll_synchronize_rcu*() with interrupts disabled

In reply to Frederic Weisbecker

From: "Paul E. McKenney" <[email protected]>

The header comment for both start_poll_synchronize_rcu() and start_poll_synchronize_rcu_full() state that interrupts must be enabled when calling these two functions, and there is a lockdep assertion in start_poll_synchronize_rcu_common() enforcing this restriction. However, there is no need for this restrictions, as can be seen in call_rcu(), which does wakeups when interrupts are disabled.

This commit therefore removes the lockdep assertion and the comments.

---
 kernel/rcu/tree.c | 7 -------
 1 file changed, 7 deletions(-)
kernel/rcu/tree.c +0 -7
index 38b121a77366..13829cf38f52 100644
@@ -4194,7 +4194,6 @@ static void start_poll_synchronize_rcu_common(void)
 	struct rcu_data *rdp;
 	struct rcu_node *rnp;
 
-	lockdep_assert_irqs_enabled();
 	local_irq_save(flags);
 	rdp = this_cpu_ptr(&rcu_data);
 	rnp = rdp->mynode;
@@ -4219,9 +4218,6 @@ static void start_poll_synchronize_rcu_common(void)
  * grace period has elapsed in the meantime.  If the needed grace period
  * is not already slated to start, notifies RCU core of the need for that
  * grace period.
- *
- * Interrupts must be enabled for the case where it is necessary to awaken
- * the grace-period kthread.
  */
 unsigned long start_poll_synchronize_rcu(void)
 {
@@ -4242,9 +4238,6 @@ EXPORT_SYMBOL_GPL(start_poll_synchronize_rcu);
  * grace period (whether normal or expedited) has elapsed in the meantime.
  * If the needed grace period is not already slated to start, notifies
  * RCU core of the need for that grace period.
- *
- * Interrupts must be enabled for the case where it is necessary to awaken
- * the grace-period kthread.
  */
 void start_poll_synchronize_rcu_full(struct rcu_gp_oldstate *rgosp)
 {
Signature · 1 line
2.46.0
author maintainer
#4 lore raw

PATCH 4/8 rcutorture: Test start-poll primitives with interrupts disabled

In reply to Frederic Weisbecker

From: "Paul E. McKenney" <[email protected]>

This commit tests the ->start_poll() and ->start_poll_full() functions with interrupts disabled, but only for RCU variants setting the ->start_poll_irqsoff flag.

---
 kernel/rcu/rcutorture.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
kernel/rcu/rcutorture.c +10 -0
index bb75dbf5c800..b4cb7623a8bf 100644
@@ -393,6 +393,7 @@ struct rcu_torture_ops {
 	int slow_gps;
 	int no_pi_lock;
 	int debug_objects;
+	int start_poll_irqsoff;
 	const char *name;
 };
 
@@ -581,6 +582,7 @@ static struct rcu_torture_ops rcu_ops = {
 	.can_boost		= IS_ENABLED(CONFIG_RCU_BOOST),
 	.extendables		= RCUTORTURE_MAX_EXTEND,
 	.debug_objects		= 1,
+	.start_poll_irqsoff	= 1,
 	.name			= "rcu"
 };
 
@@ -1695,14 +1697,22 @@ rcu_torture_fakewriter(void *arg)
 				cur_ops->cond_sync_exp_full(&gp_snap_full);
 				break;
 			case RTWS_POLL_GET:
+				if (cur_ops->start_poll_irqsoff)
+					local_irq_disable();
 				gp_snap = cur_ops->start_gp_poll();
+				if (cur_ops->start_poll_irqsoff)
+					local_irq_enable();
 				while (!cur_ops->poll_gp_state(gp_snap)) {
 					torture_hrtimeout_jiffies(torture_random(&rand) % 16,
 								  &rand);
 				}
 				break;
 			case RTWS_POLL_GET_FULL:
+				if (cur_ops->start_poll_irqsoff)
+					local_irq_disable();
 				cur_ops->start_gp_poll_full(&gp_snap_full);
+				if (cur_ops->start_poll_irqsoff)
+					local_irq_enable();
 				while (!cur_ops->poll_gp_state_full(&gp_snap_full)) {
 					torture_hrtimeout_jiffies(torture_random(&rand) % 16,
 								  &rand);
Signature · 1 line
2.46.0
author maintainer
#5 lore raw

PATCH 5/8 doc: Remove kernel-parameters.txt entry for rcutorture.read_exit

In reply to Frederic Weisbecker

From: "Paul E. McKenney" <[email protected]>

There is only ever the one read-exit task, and there is no module parameter named rcutorture.read_exit, so remove the bogus documentation. Instead, use rcutorture.read_exit_burst to enable/disable read-exit race testing.

Cc Alexei Starovoitov <[email protected]>
Cc Andrii Nakryiko <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Kent Overstreet <[email protected]>
---
 Documentation/admin-guide/kernel-parameters.txt | 5 -----
 1 file changed, 5 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +0 -5
index 1518343bbe22..7edc5a5ba9c9 100644
@@ -5412,11 +5412,6 @@
 			Set time (jiffies) between CPU-hotplug operations,
 			or zero to disable CPU-hotplug testing.
 
-	rcutorture.read_exit= [KNL]
-			Set the number of read-then-exit kthreads used
-			to test the interaction of RCU updaters and
-			task-exit processing.
-
 	rcutorture.read_exit_burst= [KNL]
 			The number of times in a given read-then-exit
 			episode that a set of read-then-exit kthreads
Signature · 1 line
2.46.0
author maintainer
#6 lore raw

PATCH 6/8 rcu-tasks: Remove open-coded one-byte cmpxchg() emulation

In reply to Frederic Weisbecker

From: "Paul E. McKenney" <[email protected]>

This commit removes the open-coded one-byte cmpxchg() emulation from rcu_trc_cmpxchg_need_qs(), replacing it with just cmpxchg() given the latter's new-found ability to handle single-byte arguments across all architectures.

---
 kernel/rcu/tasks.h | 17 +----------------
 1 file changed, 1 insertion(+), 16 deletions(-)
kernel/rcu/tasks.h +1 -16
index dd9730fd44fb..c789d994e7eb 100644
@@ -1541,22 +1541,7 @@ static void rcu_st_need_qs(struct task_struct *t, u8 v)
  */
 u8 rcu_trc_cmpxchg_need_qs(struct task_struct *t, u8 old, u8 new)
 {
-	union rcu_special ret;
-	union rcu_special trs_old = READ_ONCE(t->trc_reader_special);
-	union rcu_special trs_new = trs_old;
-
-	if (trs_old.b.need_qs != old)
-		return trs_old.b.need_qs;
-	trs_new.b.need_qs = new;
-
-	// Although cmpxchg() appears to KCSAN to update all four bytes,
-	// only the .b.need_qs byte actually changes.
-	instrument_atomic_read_write(&t->trc_reader_special.b.need_qs,
-				     sizeof(t->trc_reader_special.b.need_qs));
-	// Avoid false-positive KCSAN failures.
-	ret.s = data_race(cmpxchg(&t->trc_reader_special.s, trs_old.s, trs_new.s));
-
-	return ret.b.need_qs;
+	return cmpxchg(&t->trc_reader_special.b.need_qs, old, new);
 }
 EXPORT_SYMBOL_GPL(rcu_trc_cmpxchg_need_qs);
 
Signature · 1 line
2.46.0
author maintainer
#7 lore raw

PATCH 7/8 rcu/srcutiny: don't return before reenabling preemption

In reply to Frederic Weisbecker

From: Michal Schmidt <[email protected]>

Code after the return statement is dead. Enable preemption before returning from srcu_drive_gp().

This will be important when/if PREEMPT_AUTO (lazy resched) gets merged.

Fixes 65b4a59557f6 ("srcu: Make Tiny SRCU explicitly disable preemption")
---
 kernel/rcu/srcutiny.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/srcutiny.c +1 -1
index 549c03336ee9..4dcbf8aa80ff 100644
@@ -122,8 +122,8 @@ void srcu_drive_gp(struct work_struct *wp)
 	ssp = container_of(wp, struct srcu_struct, srcu_work);
 	preempt_disable();  // Needed for PREEMPT_AUTO
 	if (ssp->srcu_gp_running || ULONG_CMP_GE(ssp->srcu_idx, READ_ONCE(ssp->srcu_idx_max))) {
+		preempt_enable();
 		return; /* Already running or nothing to do. */
-		preempt_enable();
 	}
 
 	/* Remove recently arrived callbacks and wait for readers. */
Signature · 1 line
2.46.0
author maintainer
#8 lore raw

PATCH 8/8 rcu/kvfree: Fix data-race in __mod_timer / kvfree_call_rcu

In reply to Frederic Weisbecker

From: "Uladzislau Rezki (Sony)" <[email protected]>

KCSAN reports a data race when access the krcp->monitor_work.timer.expires variable in the schedule_delayed_monitor_work() function:

<snip>

BUG: KCSAN: data-race in __mod_timer / kvfree_call_rcu

read to 0xffff888237d1cce8 of 8 bytes by task 10149 on cpu 1:

 schedule_delayed_monitor_work kernel/rcu/tree.c:3520 [inline]
 kvfree_call_rcu+0x3b8/0x510 kernel/rcu/tree.c:3839
 trie_update_elem+0x47c/0x620 kernel/bpf/lpm_trie.c:441
 bpf_map_update_value+0x324/0x350 kernel/bpf/syscall.c:203
 generic_map_update_batch+0x401/0x520 kernel/bpf/syscall.c:1849
 bpf_map_do_batch+0x28c/0x3f0 kernel/bpf/syscall.c:5143
 __sys_bpf+0x2e5/0x7a0
 __do_sys_bpf kernel/bpf/syscall.c:5741 [inline]
 __se_sys_bpf kernel/bpf/syscall.c:5739 [inline]
 __x64_sys_bpf+0x43/0x50 kernel/bpf/syscall.c:5739
 x64_sys_call+0x2625/0x2d60 arch/x86/include/generated/asm/syscalls_64.h:322
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xc9/0x1c0 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f

write to 0xffff888237d1cce8 of 8 bytes by task 56 on cpu 0:

 __mod_timer+0x578/0x7f0 kernel/time/timer.c:1173
 add_timer_global+0x51/0x70 kernel/time/timer.c:1330
 __queue_delayed_work+0x127/0x1a0 kernel/workqueue.c:2523
 queue_delayed_work_on+0xdf/0x190 kernel/workqueue.c:2552
 queue_delayed_work include/linux/workqueue.h:677 [inline]
 schedule_delayed_monitor_work kernel/rcu/tree.c:3525 [inline]
 kfree_rcu_monitor+0x5e8/0x660 kernel/rcu/tree.c:3643
 process_one_work kernel/workqueue.c:3229 [inline]
 process_scheduled_works+0x483/0x9a0 kernel/workqueue.c:3310
 worker_thread+0x51d/0x6f0 kernel/workqueue.c:3391
 kthread+0x1d1/0x210 kernel/kthread.c:389
 ret_from_fork+0x4b/0x60 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244

Reported by Kernel Concurrency Sanitizer on:

CPU: 0 UID: 0 PID: 56 Comm: kworker/u8:4 Not tainted 6.12.0-rc2-syzkaller-00050-g5b7c893ed5ed #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Workqueue: events_unbound kfree_rcu_monitor

<snip>

kfree_rcu_monitor() rearms the work if a "krcp" has to be still offloaded and this is done without holding krcp->lock, whereas the kvfree_call_rcu() holds it.

Fix it by acquiring the "krcp->lock" for kfree_rcu_monitor() so both functions do not race anymore.

Reported-by [email protected]
Fixes 8fc5494ad5fa ("rcu/kvfree: Move need_offload_krc() out of krcp->lock")
---
 kernel/rcu/tree.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
kernel/rcu/tree.c +12 -2
index 13829cf38f52..ff98233d4aa5 100644
@@ -3511,7 +3511,7 @@ static int krc_count(struct kfree_rcu_cpu *krcp)
 }
 
 static void
-schedule_delayed_monitor_work(struct kfree_rcu_cpu *krcp)
+__schedule_delayed_monitor_work(struct kfree_rcu_cpu *krcp)
 {
 	long delay, delay_left;
 
@@ -3525,6 +3525,16 @@ schedule_delayed_monitor_work(struct kfree_rcu_cpu *krcp)
 	queue_delayed_work(system_unbound_wq, &krcp->monitor_work, delay);
 }
 
+static void
+schedule_delayed_monitor_work(struct kfree_rcu_cpu *krcp)
+{
+	unsigned long flags;
+
+	raw_spin_lock_irqsave(&krcp->lock, flags);
+	__schedule_delayed_monitor_work(krcp);
+	raw_spin_unlock_irqrestore(&krcp->lock, flags);
+}
+
 static void
 kvfree_rcu_drain_ready(struct kfree_rcu_cpu *krcp)
 {
@@ -3836,7 +3846,7 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
 
 	// Set timer to drain after KFREE_DRAIN_JIFFIES.
 	if (rcu_scheduler_active == RCU_SCHEDULER_RUNNING)
-		schedule_delayed_monitor_work(krcp);
+		__schedule_delayed_monitor_work(krcp);
 
 unlock_return:
 	krc_this_cpu_unlock(krcp, flags);
Signature · 1 line
2.46.0
Reviewed-by on the v1 series
#9 lore raw
In reply to Frederic Weisbecker
Quoted 19 lines · Frederic Weisbecker

Hello,

Please find below the general RCU fixes targeted for the upcoming merge window.

Michal Schmidt (1):
rcu/srcutiny: don't return before reenabling preemption

Paul E. McKenney (6):
doc: Add rcuog kthreads to kernel-per-CPU-kthreads.rst
rcu: Allow short-circuiting of synchronize_rcu_tasks_rude()
rcu: Permit start_poll_synchronize_rcu*() with interrupts disabled
rcutorture: Test start-poll primitives with interrupts disabled
doc: Remove kernel-parameters.txt entry for rcutorture.read_exit
rcu-tasks: Remove open-coded one-byte cmpxchg() emulation

Uladzislau Rezki (Sony) (1):
rcu/kvfree: Fix data-race in __mod_timer / kvfree_call_rcu

Quoted 8 lines
 .../admin-guide/kernel-parameters.txt         |  5 -----
 .../admin-guide/kernel-per-CPU-kthreads.rst   |  2 +-
 kernel/rcu/rcutorture.c                       | 10 +++++++++
 kernel/rcu/srcutiny.c                         |  2 +-
 kernel/rcu/tasks.h                            | 20 +++---------------
 kernel/rcu/tree.c                             | 21 +++++++++++--------
 6 files changed, 27 insertions(+), 33 deletions(-)

For this series:

- Neeraj

Quoted 1 line

Thanks.