RCU fixes for v6.13
Subsystems (5)
Frederic Weisbecker
maintainercover RCU fixes for v6.13
Hello,
Please find below the general RCU fixes targeted for the upcoming merge window.
Michal Schmidt (1):
rcu/srcutiny: don't return before reenabling preemption
Paul E. McKenney (6):
doc: Add rcuog kthreads to kernel-per-CPU-kthreads.rst
rcu: Allow short-circuiting of synchronize_rcu_tasks_rude()
rcu: Permit start_poll_synchronize_rcu*() with interrupts disabled
rcutorture: Test start-poll primitives with interrupts disabled
doc: Remove kernel-parameters.txt entry for rcutorture.read_exit
rcu-tasks: Remove open-coded one-byte cmpxchg() emulation
Uladzislau Rezki (Sony) (1):
rcu/kvfree: Fix data-race in __mod_timer / kvfree_call_rcu
.../admin-guide/kernel-parameters.txt | 5 ----- .../admin-guide/kernel-per-CPU-kthreads.rst | 2 +- kernel/rcu/rcutorture.c | 10 +++++++++ kernel/rcu/srcutiny.c | 2 +- kernel/rcu/tasks.h | 20 +++--------------- kernel/rcu/tree.c | 21 +++++++++++-------- 6 files changed, 27 insertions(+), 33 deletions(-)
Thanks.
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 1/8 doc: Add rcuog kthreads to kernel-per-CPU-kthreads.rst
From: "Paul E. McKenney" <[email protected]>
This commit adds the rcuog kthreads to the list of callback-offloading kthreads that can be affinitied away from worker CPUs.
--- Documentation/admin-guide/kernel-per-CPU-kthreads.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/admin-guide/kernel-per-CPU-kthreads.rst +1 -1
@@ -315,7 +315,7 @@ To reduce its OS jitter, do at least one of the following: to do. Name: - rcuop/%d and rcuos/%d + rcuop/%d, rcuos/%d, and rcuog/%d Purpose: Offload RCU callbacks from the corresponding CPU.
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 2/8 rcu: Allow short-circuiting of synchronize_rcu_tasks_rude()
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
There are now architectures for which all deep-idle and entry-exit functions are properly inlined or marked noinstr. Such architectures do not need synchronize_rcu_tasks_rude(), or will not once RCU Tasks has been modified to pay attention to idle tasks. This commit therefore allows a CONFIG_ARCH_HAS_NOINSTR_MARKINGS Kconfig option to turn synchronize_rcu_tasks_rude() into a no-op.
To facilitate testing, kernels built by rcutorture scripting will enable RCU Tasks Trace even on systems that do not need it.
[ paulmck: Apply Peter Zijlstra feedback. ]
--- kernel/rcu/tasks.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/rcu/tasks.h +2 -1
@@ -1398,7 +1398,8 @@ static void call_rcu_tasks_rude(struct rcu_head *rhp, rcu_callback_t func)*/ void synchronize_rcu_tasks_rude (void ) { -synchronize_rcu_tasks_generic (&rcu_tasks_rude); +if (!IS_ENABLED (CONFIG_ARCH_WANTS_NO_INSTR ) ||IS_ENABLED (CONFIG_FORCE_TASKS_RUDE_RCU )) +synchronize_rcu_tasks_generic (&rcu_tasks_rude); }EXPORT_SYMBOL_GPL (synchronize_rcu_tasks_rude);
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 3/8 rcu: Permit start_poll_synchronize_rcu*() with interrupts disabled
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
The header comment for both start_poll_synchronize_rcu() and start_poll_synchronize_rcu_full() state that interrupts must be enabled when calling these two functions, and there is a lockdep assertion in start_poll_synchronize_rcu_common() enforcing this restriction. However, there is no need for this restrictions, as can be seen in call_rcu(), which does wakeups when interrupts are disabled.
This commit therefore removes the lockdep assertion and the comments.
--- kernel/rcu/tree.c | 7 ------- 1 file changed, 7 deletions(-)
kernel/rcu/tree.c +0 -7
@@ -4194,7 +4194,6 @@ static void start_poll_synchronize_rcu_common(void)struct rcu_data *rdp;struct rcu_node *rnp; -lockdep_assert_irqs_enabled ();local_irq_save (flags); rdp =this_cpu_ptr (&rcu_data); rnp = rdp->mynode;
@@ -4219,9 +4218,6 @@ static void start_poll_synchronize_rcu_common(void)* grace period has elapsed in the meantime. If the needed grace period * is not already slated to start, notifies RCU core of the need for that * grace period. -* -* Interrupts must be enabled for the case where it is necessary to awaken -* the grace-period kthread. */ unsigned long start_poll_synchronize_rcu (void ) {
@@ -4242,9 +4238,6 @@ EXPORT_SYMBOL_GPL(start_poll_synchronize_rcu);* grace period (whether normal or expedited) has elapsed in the meantime. * If the needed grace period is not already slated to start, notifies * RCU core of the need for that grace period. -* -* Interrupts must be enabled for the case where it is necessary to awaken -* the grace-period kthread. */ void start_poll_synchronize_rcu_full (struct rcu_gp_oldstate *rgosp) {
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 4/8 rcutorture: Test start-poll primitives with interrupts disabled
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
This commit tests the ->start_poll() and ->start_poll_full() functions with interrupts disabled, but only for RCU variants setting the ->start_poll_irqsoff flag.
--- kernel/rcu/rcutorture.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
kernel/rcu/rcutorture.c +10 -0
@@ -393,6 +393,7 @@ struct rcu_torture_ops {int slow_gps;int no_pi_lock;int debug_objects; +int start_poll_irqsoff;const char *name; };
@@ -581,6 +582,7 @@ static struct rcu_torture_ops rcu_ops = { .can_boost = IS_ENABLED(CONFIG_RCU_BOOST), .extendables = RCUTORTURE_MAX_EXTEND, .debug_objects = 1, + .start_poll_irqsoff = 1, .name = "rcu" };
@@ -1695,14 +1697,22 @@ rcu_torture_fakewriter(void *arg) cur_ops->cond_sync_exp_full (&gp_snap_full);break ;case RTWS_POLL_GET : +if (cur_ops->start_poll_irqsoff) +local_irq_disable (); gp_snap = cur_ops->start_gp_poll (); +if (cur_ops->start_poll_irqsoff) +local_irq_enable ();while (!cur_ops->poll_gp_state (gp_snap)) {torture_hrtimeout_jiffies (torture_random (&rand) %16 , &rand); }break ;case RTWS_POLL_GET_FULL : +if (cur_ops->start_poll_irqsoff) +local_irq_disable (); cur_ops->start_gp_poll_full (&gp_snap_full); +if (cur_ops->start_poll_irqsoff) +local_irq_enable ();while (!cur_ops->poll_gp_state_full (&gp_snap_full)) {torture_hrtimeout_jiffies (torture_random (&rand) %16 , &rand);
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 5/8 doc: Remove kernel-parameters.txt entry for rcutorture.read_exit
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
There is only ever the one read-exit task, and there is no module parameter named rcutorture.read_exit, so remove the bogus documentation. Instead, use rcutorture.read_exit_burst to enable/disable read-exit race testing.
--- Documentation/admin-guide/kernel-parameters.txt | 5 ----- 1 file changed, 5 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +0 -5
@@ -5412,11 +5412,6 @@ Set time (jiffies) between CPU-hotplug operations, or zero to disable CPU-hotplug testing. - rcutorture.read_exit= [KNL] - Set the number of read-then-exit kthreads used - to test the interaction of RCU updaters and - task-exit processing. - rcutorture.read_exit_burst= [KNL] The number of times in a given read-then-exit episode that a set of read-then-exit kthreads
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 6/8 rcu-tasks: Remove open-coded one-byte cmpxchg() emulation
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
This commit removes the open-coded one-byte cmpxchg() emulation from rcu_trc_cmpxchg_need_qs(), replacing it with just cmpxchg() given the latter's new-found ability to handle single-byte arguments across all architectures.
--- kernel/rcu/tasks.h | 17 +---------------- 1 file changed, 1 insertion(+), 16 deletions(-)
kernel/rcu/tasks.h +1 -16
@@ -1541,22 +1541,7 @@ static void rcu_st_need_qs(struct task_struct *t, u8 v)*/ u8 rcu_trc_cmpxchg_need_qs (struct task_struct *t,u8 old,u8 new) { -union rcu_special ret; -union rcu_special trs_old =READ_ONCE (t->trc_reader_special); -union rcu_special trs_new = trs_old; - -if (trs_old.b.need_qs != old) -return trs_old.b.need_qs; - trs_new.b.need_qs = new; - -// Although cmpxchg() appears to KCSAN to update all four bytes, -// only the .b.need_qs byte actually changes. -instrument_atomic_read_write (&t->trc_reader_special.b.need_qs, -sizeof (t->trc_reader_special.b.need_qs)); -// Avoid false-positive KCSAN failures. - ret.s =data_race (cmpxchg (&t->trc_reader_special.s, trs_old.s, trs_new.s)); - -return ret.b.need_qs; +return cmpxchg (&t->trc_reader_special.b.need_qs, old, new); }EXPORT_SYMBOL_GPL (rcu_trc_cmpxchg_need_qs);
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 7/8 rcu/srcutiny: don't return before reenabling preemption
In reply to Frederic WeisbeckerFrom: Michal Schmidt <[email protected]>
Code after the return statement is dead. Enable preemption before returning from srcu_drive_gp().
This will be important when/if PREEMPT_AUTO (lazy resched) gets merged.
--- kernel/rcu/srcutiny.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/srcutiny.c +1 -1
@@ -122,8 +122,8 @@ void srcu_drive_gp(struct work_struct *wp) ssp = container_of(wp, struct srcu_struct, srcu_work);preempt_disable ();// Needed for PREEMPT_AUTO if (ssp->srcu_gp_running ||ULONG_CMP_GE (ssp->srcu_idx,READ_ONCE (ssp->srcu_idx_max))) { +preempt_enable ();return ;/* Already running or nothing to do. */ -preempt_enable (); }/* Remove recently arrived callbacks and wait for readers. */
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH 8/8 rcu/kvfree: Fix data-race in __mod_timer / kvfree_call_rcu
In reply to Frederic WeisbeckerFrom: "Uladzislau Rezki (Sony)" <[email protected]>
KCSAN reports a data race when access the krcp->monitor_work.timer.expires variable in the schedule_delayed_monitor_work() function:
<snip>
BUG: KCSAN: data-race in __mod_timer / kvfree_call_rcu
read to 0xffff888237d1cce8 of 8 bytes by task 10149 on cpu 1:
schedule_delayed_monitor_work kernel/rcu/tree.c:3520 [inline] kvfree_call_rcu+0x3b8/0x510 kernel/rcu/tree.c:3839 trie_update_elem+0x47c/0x620 kernel/bpf/lpm_trie.c:441 bpf_map_update_value+0x324/0x350 kernel/bpf/syscall.c:203 generic_map_update_batch+0x401/0x520 kernel/bpf/syscall.c:1849 bpf_map_do_batch+0x28c/0x3f0 kernel/bpf/syscall.c:5143 __sys_bpf+0x2e5/0x7a0 __do_sys_bpf kernel/bpf/syscall.c:5741 [inline] __se_sys_bpf kernel/bpf/syscall.c:5739 [inline] __x64_sys_bpf+0x43/0x50 kernel/bpf/syscall.c:5739 x64_sys_call+0x2625/0x2d60 arch/x86/include/generated/asm/syscalls_64.h:322 do_syscall_x64 arch/x86/entry/common.c:52 [inline] do_syscall_64+0xc9/0x1c0 arch/x86/entry/common.c:83 entry_SYSCALL_64_after_hwframe+0x77/0x7f
write to 0xffff888237d1cce8 of 8 bytes by task 56 on cpu 0:
__mod_timer+0x578/0x7f0 kernel/time/timer.c:1173 add_timer_global+0x51/0x70 kernel/time/timer.c:1330 __queue_delayed_work+0x127/0x1a0 kernel/workqueue.c:2523 queue_delayed_work_on+0xdf/0x190 kernel/workqueue.c:2552 queue_delayed_work include/linux/workqueue.h:677 [inline] schedule_delayed_monitor_work kernel/rcu/tree.c:3525 [inline] kfree_rcu_monitor+0x5e8/0x660 kernel/rcu/tree.c:3643 process_one_work kernel/workqueue.c:3229 [inline] process_scheduled_works+0x483/0x9a0 kernel/workqueue.c:3310 worker_thread+0x51d/0x6f0 kernel/workqueue.c:3391 kthread+0x1d1/0x210 kernel/kthread.c:389 ret_from_fork+0x4b/0x60 arch/x86/kernel/process.c:147 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
Reported by Kernel Concurrency Sanitizer on:
CPU: 0 UID: 0 PID: 56 Comm: kworker/u8:4 Not tainted 6.12.0-rc2-syzkaller-00050-g5b7c893ed5ed #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024 Workqueue: events_unbound kfree_rcu_monitor
<snip>
kfree_rcu_monitor() rearms the work if a "krcp" has to be still offloaded and this is done without holding krcp->lock, whereas the kvfree_call_rcu() holds it.
Fix it by acquiring the "krcp->lock" for kfree_rcu_monitor() so both functions do not race anymore.
--- kernel/rcu/tree.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
kernel/rcu/tree.c +12 -2
@@ -3511,7 +3511,7 @@ static int krc_count(struct kfree_rcu_cpu *krcp) }static void -schedule_delayed_monitor_work (struct kfree_rcu_cpu *krcp) +__schedule_delayed_monitor_work (struct kfree_rcu_cpu *krcp) {long delay, delay_left;
@@ -3525,6 +3525,16 @@ schedule_delayed_monitor_work(struct kfree_rcu_cpu *krcp)queue_delayed_work (system_unbound_wq, &krcp->monitor_work, delay); } +static void +schedule_delayed_monitor_work (struct kfree_rcu_cpu *krcp) +{ +unsigned long flags; + +raw_spin_lock_irqsave (&krcp->lock, flags); +__schedule_delayed_monitor_work (krcp); +raw_spin_unlock_irqrestore (&krcp->lock, flags); +} +static void kvfree_rcu_drain_ready (struct kfree_rcu_cpu *krcp) {
@@ -3836,7 +3846,7 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)// Set timer to drain after KFREE_DRAIN_JIFFIES. if (rcu_scheduler_active ==RCU_SCHEDULER_RUNNING ) -schedule_delayed_monitor_work (krcp); +__schedule_delayed_monitor_work (krcp); unlock_return:krc_this_cpu_unlock (krcp, flags);
Signature · 1 line
2.46.0
Quoted 19 lines · Frederic Weisbecker
Hello,
Please find below the general RCU fixes targeted for the upcoming merge window.
Michal Schmidt (1):
rcu/srcutiny: don't return before reenabling preemptionPaul E. McKenney (6):
doc: Add rcuog kthreads tokernel-per-CPU-kthreads.rst
rcu: Allow short-circuiting ofsynchronize_rcu_tasks_rude()
rcu: Permit start_poll_synchronize_rcu*() with interrupts disabled
rcutorture: Test start-poll primitives with interrupts disabled
doc: Remove kernel-parameters.txt entry for rcutorture.read_exit
rcu-tasks: Remove open-coded one-bytecmpxchg()emulationUladzislau Rezki (Sony) (1):
rcu/kvfree: Fix data-race in__mod_timer/kvfree_call_rcu
Quoted 8 lines
.../admin-guide/kernel-parameters.txt | 5 ----- .../admin-guide/kernel-per-CPU-kthreads.rst | 2 +- kernel/rcu/rcutorture.c | 10 +++++++++ kernel/rcu/srcutiny.c | 2 +- kernel/rcu/tasks.h | 20 +++--------------- kernel/rcu/tree.c | 21 +++++++++++-------- 6 files changed, 27 insertions(+), 33 deletions(-)
For this series:
- Neeraj
Quoted 1 line
Thanks.