Miscellaneous fixes for v5.11
Subsystems (4)
Paul E. McKenney
maintainercover Miscellaneous fixes for v5.11
Hello!
This series contains miscellaneous fixes:
1. Don't invoke try_invoke_on_locked_down_task() with irqs disabled.
2. Move x86's rcu_cpu_starting() earlier.
3. Panic after fixed number of stalls.
4. Update list.h comment to explicitly note circular lists.
5. Implement rcu_segcblist_is_offloaded() config dependent.
6. Fix single-CPU check in rcu_blocking_is_gp().
7. Clarify nocb kthreads naming in RCU_NOCB_CPU config.
8. Add a warning if CPU being onlined did not report QS already.
9. Make struct kernel_param_ops definitions const.
10. Fix ftrace recursion.
11. nocb: Avoid raising softirq for offloaded ready-to-execute CBs.
12. Prevent lockdep-RCU splats on lock acquisition/release.
13. Fix a typo in rcu_blocking_is_gp() header comment.
14. Do not report strict GPs for outgoing CPUs.
15. Defer kvfree_rcu() allocation to a clean context.
16. Take early exit on memory-allocation failure.
Thanx, Paul
------------------------------------------------------------------------
arch/x86/kernel/cpu/mtrr/mtrr.c | 2 arch/x86/kernel/smpboot.c | 1 include/linux/kernel.h | 1 include/linux/list.h | 2 kernel/rcu/Kconfig | 20 ++-- kernel/rcu/rcu_segcblist.h | 2 kernel/rcu/srcutree.c | 4 kernel/rcu/tree.c | 188 +++++++++++++++++++++++++--------------- kernel/rcu/tree.h | 2 kernel/rcu/tree_plugin.h | 2 kernel/rcu/tree_stall.h | 28 ++++- kernel/sysctl.c | 11 ++ 12 files changed, 178 insertions(+), 85 deletions(-)
[email protected]
maintainerPATCH 1/16 rcu: Don't invoke try_invoke_on_locked_down_task() with irqs disabled
From: "Paul E. McKenney" <[email protected]>
The try_invoke_on_locked_down_task() function requires that interrupts be enabled, but it is called with interrupts disabled from rcu_print_task_stall(), resulting in an "IRQs not enabled as expected" diagnostic. This commit therefore updates rcu_print_task_stall() to accumulate a list of the first few tasks while holding the current leaf rcu_node structure's ->lock, then releases that lock and only then uses try_invoke_on_locked_down_task() to attempt to obtain per-task detailed information. Of course, as soon as ->lock is released, the task might exit, so the get_task_struct() function is used to prevent the task structure from going away in the meantime.
--- kernel/rcu/tree_stall.h | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)
kernel/rcu/tree_stall.h +17 -5
@@ -249,13 +249,16 @@ static bool check_slow_task(struct task_struct *t, void *arg)/* * Scan the current list of tasks blocked within RCU read-side critical -* sections, printing out the tid of each. +* sections, printing out the tid of each of the first few of them. */ -static int rcu_print_task_stall (struct rcu_node *rnp) +static int rcu_print_task_stall (struct rcu_node *rnp,unsigned long flags) + __releases(rnp->lock) { +int i =0 ;int ndetected =0 ;struct rcu_stall_chk_rdr rscr;struct task_struct *t; +struct task_struct *ts[8 ];if (!rcu_preempt_blocked_readers_cgp (rnp))return 0 ;
@@ -264,6 +267,14 @@ static int rcu_print_task_stall(struct rcu_node *rnp) t =list_entry (rnp->gp_tasks->prev, struct task_struct, rcu_node_entry);list_for_each_entry_continue (t, &rnp->blkd_tasks, rcu_node_entry) { +get_task_struct (t); + ts[i++] = t; +if (i >=ARRAY_SIZE (ts)) +break ; + } +raw_spin_unlock_irqrestore_rcu_node (rnp, flags); +for (i--; i; i--) { + t = ts[i];if (!try_invoke_on_locked_down_task (t, check_slow_task, &rscr))pr_cont (" P%d" , t->pid);else
@@ -273,6 +284,7 @@ static int rcu_print_task_stall(struct rcu_node *rnp)".q" [rscr.rs.b.need_qs],".e" [rscr.rs.b.exp_hint], ".l"[rscr.on_blkd_list]); +put_task_struct (t); ndetected++; }pr_cont ("\n" );
@@ -293,8 +305,9 @@ static void rcu_print_detail_task_stall_rnp(struct rcu_node *rnp)* Because preemptible RCU does not exist, we never have to check for * tasks blocked within RCU read-side critical sections. */ -static int rcu_print_task_stall (struct rcu_node *rnp) +static int rcu_print_task_stall (struct rcu_node *rnp,unsigned long flags) { +raw_spin_unlock_irqrestore_rcu_node (rnp, flags);return 0 ; }#endif /* #else #ifdef CONFIG_PREEMPT_RCU */
@@ -472,7 +485,6 @@ static void print_other_cpu_stall(unsigned long gp_seq, unsigned long gps)pr_err ("INFO: %s detected stalls on CPUs/tasks:\n" , rcu_state.name);rcu_for_each_leaf_node (rnp) {raw_spin_lock_irqsave_rcu_node (rnp, flags); - ndetected +=rcu_print_task_stall (rnp);if (rnp->qsmask !=0 ) {for_each_leaf_node_possible_cpu (rnp, cpu)if (rnp->qsmask &leaf_node_cpu_bit (rnp, cpu)) {
@@ -480,7 +492,7 @@ static void print_other_cpu_stall(unsigned long gp_seq, unsigned long gps) ndetected++; } } -raw_spin_unlock_irqrestore_rcu_node (rnp, flags); + ndetected +=rcu_print_task_stall (rnp, flags);// Releases rnp->lock. }for_each_possible_cpu (cpu)
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 2/16 x86/smpboot: Move rcu_cpu_starting() earlier
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The call to rcu_cpu_starting() in mtrr_ap_init() is not early enough in the CPU-hotplug onlining process, which results in lockdep splats as follows:
=============================
WARNING: suspicious RCU usage
5.9.0+ #268 Not tainted
-----------------------------kernel/kprobes.c:300 RCU-list traversed in non-reader section!!
other info that might help us debug this:
RCU used illegally from offline CPU!rcu_scheduler_active = 1, debug_locks = 1
no locks held by swapper/1/0.
stack backtrace: CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.9.0+ #268 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.2-1ubuntu1 04/01/2014 Call Trace: dump_stack+0x77/0x97 __is_insn_slot_addr+0x15d/0x170 kernel_text_address+0xba/0xe0 ? get_stack_info+0x22/0xa0 __kernel_text_address+0x9/0x30 show_trace_log_lvl+0x17d/0x380 ? dump_stack+0x77/0x97 dump_stack+0x77/0x97 __lock_acquire+0xdf7/0x1bf0 lock_acquire+0x258/0x3d0 ? vprintk_emit+0x6d/0x2c0 _raw_spin_lock+0x27/0x40 ? vprintk_emit+0x6d/0x2c0 vprintk_emit+0x6d/0x2c0 printk+0x4d/0x69 start_secondary+0x1c/0x100 secondary_startup_64_no_verify+0xb8/0xbb
This is avoided by moving the call to rcu_cpu_starting up near the beginning of the start_secondary() function. Note that the raw_smp_processor_id() is required in order to avoid calling into lockdep before RCU has declared the CPU to be watched for readers.
--- arch/x86/kernel/cpu/mtrr/mtrr.c | 2 -- arch/x86/kernel/smpboot.c | 1 + 2 files changed, 1 insertion(+), 2 deletions(-)
arch/x86/kernel/cpu/mtrr/mtrr.c +0 -2
@@ -794,8 +794,6 @@ void mtrr_ap_init(void)if (!use_intel () || mtrr_aps_delayed_init)return ; -rcu_cpu_starting (smp_processor_id ()); -/* * Ideally we should hold mtrr_mutex here to avoid mtrr entries * changed, but this routine will be called in cpu boot time,
arch/x86/kernel/smpboot.c +1 -0
@@ -229,6 +229,7 @@ static void notrace start_secondary(void *unused)#endif cpu_init_exception_handling ();cpu_init (); +rcu_cpu_starting (raw_smp_processor_id ()); x86_cpuinit.early_percpu_clock_init ();preempt_disable ();smp_callin ();
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 3/16 rcu: Panic after fixed number of stalls
In reply to Paul E. McKenneyFrom: chao <[email protected]>
Some stalls are transient, so that system fully recovers. This commit therefore allows users to configure the number of stalls that must happen in order to trigger kernel panic.
--- include/linux/kernel.h | 1 + kernel/rcu/tree_stall.h | 6 ++++++ kernel/sysctl.c | 11 +++++++++++ 3 files changed, 18 insertions(+)
include/linux/kernel.h +1 -0
@@ -536,6 +536,7 @@ extern int panic_on_warn;extern unsigned long panic_on_taint;extern bool panic_on_taint_nousertaint;extern int sysctl_panic_on_rcu_stall; +extern int sysctl_max_rcu_stall_to_panic;extern int sysctl_panic_on_stackoverflow;extern bool crash_kexec_post_notifiers;
kernel/rcu/tree_stall.h +6 -0
@@ -13,6 +13,7 @@/* panic() on RCU Stall sysctl. */ int sysctl_panic_on_rcu_stall__read_mostly ; +int sysctl_max_rcu_stall_to_panic__read_mostly ;#ifdef CONFIG_PROVE_RCU #define RCU_STALL_DELAY_DELTA (5 * HZ)
@@ -106,6 +107,11 @@ early_initcall(check_cpu_stall_init);/* If so specified via sysctl, panic, yielding cleaner stall-warning output. */ static void panic_on_rcu_stall (void ) { +static int cpu_stall; + +if (++cpu_stall < sysctl_max_rcu_stall_to_panic) +return ; +if (sysctl_panic_on_rcu_stall)panic ("RCU Stall\n" ); }
kernel/sysctl.c +11 -0
@@ -2650,6 +2650,17 @@ static struct ctl_table kern_table[] = { .extra2 = SYSCTL_ONE, },#endif +#if defined(CONFIG_TREE_RCU) + { + .procname = "max_rcu_stall_to_panic", + .data = &sysctl_max_rcu_stall_to_panic, + .maxlen = sizeof(sysctl_max_rcu_stall_to_panic), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ONE, + .extra2 = SYSCTL_INT_MAX, + }, +#endif #ifdef CONFIG_STACKLEAK_RUNTIME_DISABLE { .procname = "stack_erasing",
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 4/16 list.h: Update comment to explicitly note circular lists
In reply to Paul E. McKenneyFrom: Asif Rasheed <[email protected]>
The students in the Operating System Lecture Section at the American University of Sharjah were confused by the header comment in include/linux/list.h, which says "Simple doubly linked list implementation". This comment means "simple" as in "not complex", but "simple" is often used in this context to mean "not circular". This commit therefore avoids this ambiguity by explicitly calling out "circular".
--- include/linux/list.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/list.h +1 -1
@@ -9,7 +9,7 @@#include <linux/kernel.h> /* -* Simple doubly linked list implementation. +* Circular doubly linked list implementation. * * Some of the internal functions ("__xxx") are useful when * manipulating whole lists rather than single entries, as
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 5/16 rcu: Implement rcu_segcblist_is_offloaded() config dependent
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
This commit simplifies the use of the rcu_segcblist_is_offloaded() API so that its callers no longer need to check the RCU_NOCB_CPU Kconfig option. Note that rcu_segcblist_is_offloaded() is defined in the header file, which means that the generated code should be just as efficient as before.
--- kernel/rcu/rcu_segcblist.h | 2 +- kernel/rcu/tree.c | 21 +++++++-------------- 2 files changed, 8 insertions(+), 15 deletions(-)
kernel/rcu/rcu_segcblist.h +1 -1
@@ -62,7 +62,7 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)/* Is the specified rcu_segcblist offloaded? */ static inline bool rcu_segcblist_is_offloaded (struct rcu_segcblist *rsclp) { -return rsclp->offloaded; +return IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && rsclp->offloaded; }/*
kernel/rcu/tree.c +7 -14
@@ -1603,8 +1603,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp) {bool ret = false;bool need_qs; -const bool offloaded =IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -rcu_segcblist_is_offloaded (&rdp->cblist); +const bool offloaded =rcu_segcblist_is_offloaded (&rdp->cblist);raw_lockdep_assert_held_rcu_node (rnp);
@@ -2048,8 +2047,7 @@ static void rcu_gp_cleanup(void) needgp = true; }/* Advance CBs to reduce false positives below. */ - offloaded =IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -rcu_segcblist_is_offloaded (&rdp->cblist); + offloaded =rcu_segcblist_is_offloaded (&rdp->cblist);if ((offloaded || !rcu_accelerate_cbs (rnp, rdp)) && needgp) {WRITE_ONCE (rcu_state.gp_flags,RCU_GP_FLAG_INIT );WRITE_ONCE (rcu_state.gp_req_activity, jiffies);
@@ -2248,8 +2246,7 @@ rcu_report_qs_rdp(struct rcu_data *rdp)unsigned long flags;unsigned long mask;bool needwake = false; -const bool offloaded =IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -rcu_segcblist_is_offloaded (&rdp->cblist); +const bool offloaded =rcu_segcblist_is_offloaded (&rdp->cblist);struct rcu_node *rnp;WARN_ON_ONCE (rdp->cpu !=smp_processor_id ());
@@ -2417,8 +2414,7 @@ static void rcu_do_batch(struct rcu_data *rdp) {int div;unsigned long flags; -const bool offloaded =IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -rcu_segcblist_is_offloaded (&rdp->cblist); +const bool offloaded =rcu_segcblist_is_offloaded (&rdp->cblist);struct rcu_head *rhp;struct rcu_cblist rcl =RCU_CBLIST_INITIALIZER (rcl);long bl, count;
@@ -2675,8 +2671,7 @@ static __latent_entropy void rcu_core(void)unsigned long flags;struct rcu_data *rdp =raw_cpu_ptr (&rcu_data);struct rcu_node *rnp = rdp->mynode; -const bool offloaded =IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -rcu_segcblist_is_offloaded (&rdp->cblist); +const bool offloaded =rcu_segcblist_is_offloaded (&rdp->cblist);if (cpu_is_offline (smp_processor_id ()))return ;
@@ -2978,8 +2973,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func) rcu_segcblist_n_cbs(&rdp->cblist));/* Go handle any RCU core processing required. */ -if (IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -unlikely (rcu_segcblist_is_offloaded (&rdp->cblist))) { +if (unlikely (rcu_segcblist_is_offloaded (&rdp->cblist))) {__call_rcu_nocb_wake (rdp, was_alldone, flags);/* unlocks */ }else {__call_rcu_core (rdp, head, flags);
@@ -3712,8 +3706,7 @@ static int rcu_pending(int user)/* Has RCU gone idle with this CPU needing another grace period? */ if (!gp_in_progress &&rcu_segcblist_is_enabled (&rdp->cblist) && - (!IS_ENABLED (CONFIG_RCU_NOCB_CPU ) || - !rcu_segcblist_is_offloaded (&rdp->cblist)) && + !rcu_segcblist_is_offloaded (&rdp->cblist) && !rcu_segcblist_restempty (&rdp->cblist,RCU_NEXT_READY_TAIL ))return 1 ;
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 6/16 rcu: Fix single-CPU check in rcu_blocking_is_gp()
In reply to Paul E. McKenneyFrom: Neeraj Upadhyay <[email protected]>
Currently, for CONFIG_PREEMPTION=n kernels, rcu_blocking_is_gp() uses num_online_cpus() to determine whether there is only one CPU online. When there is only a single CPU online, the simple fact that synchronize_rcu() could be legally called implies that a full grace period has elapsed. Therefore, in the single-CPU case, synchronize_rcu() simply returns immediately. Unfortunately, num_online_cpus() is unreliable while a CPU-hotplug operation is transitioning to or from single-CPU operation because:
1. num_online_cpus() uses atomic_read(&__num_online_cpus) to
locklessly sample the number of online CPUs. The hotplug locks are not held, which means that an incoming CPU can concurrently update this count. This in turn means that an RCU read-side critical section on the incoming CPU might observe updates prior to the grace period, but also that this critical section might extend beyond the end of the optimized synchronize_rcu(). This breaks RCU's fundamental guarantee.
2. In addition, num_online_cpus() does no ordering, thus providing
another way that RCU's fundamental guarantee can be broken by the current code.
3. The most probable failure mode happens on outgoing CPUs.
The outgoing CPU updates the count of online CPUs in the CPUHP_TEARDOWN_CPU stop-machine handler, which is fine in and of itself due to preemption being disabled at the call to num_online_cpus(). Unfortunately, after that stop-machine handler returns, the CPU takes one last trip through the scheduler (which has RCU readers) and, after the resulting context switch, one final dive into the idle loop. During this time, RCU needs to keep track of two CPUs, but num_online_cpus() will say that there is only one, which in turn means that the surviving CPU will incorrectly ignore the outgoing CPU's RCU read-side critical sections.
This problem is illustrated by the following litmus test in which P0() corresponds to synchronize_rcu() and P1() corresponds to the incoming CPU. The herd7 tool confirms that the "exists" clause can be satisfied, thus demonstrating that this breakage can happen according to the Linux kernel memory model.
{
int x = 0;
atomic_t numonline = ATOMIC_INIT(1);
}
P0(int *x, atomic_t *numonline)
{
int r0;
WRITE_ONCE(*x, 1);
r0 = atomic_read(numonline);
if (r0 == 1) {
smp_mb();
} else {
synchronize_rcu();
}
WRITE_ONCE(*x, 2);
}
P1(int *x, atomic_t *numonline)
{
int r0; int r1; atomic_inc(numonline);
smp_mb();
rcu_read_lock();
r0 = READ_ONCE(*x);
smp_rmb();
r1 = READ_ONCE(*x);
rcu_read_unlock();}
locations [x;numonline;]
exists (1:r0=0 /\ 1:r1=2)
It is important to note that these problems arise only when the system is transitioning to or from single-CPU operation.
One solution would be to hold the CPU-hotplug locks while sampling num_online_cpus(), which was in fact the intent of the (redundant) preempt_disable() and preempt_enable() surrounding this call to num_online_cpus(). Actually blocking CPU hotplug would not only result in excessive overhead, but would also unnecessarily impede CPU-hotplug operations.
This commit therefore follows long-standing RCU tradition by maintaining a separate RCU-specific set of CPU-hotplug books.
This separate set of books is implemented by a new ->n_online_cpus field in the rcu_state structure that maintains RCU's count of the online CPUs. This count is incremented early in the CPU-online process, so that the critical transition away from single-CPU operation will occur when there is only a single CPU. Similarly for the critical transition to single-CPU operation, the counter is decremented late in the CPU-offline process, again while there is only a single CPU. Because there is only ever a single CPU when the ->n_online_cpus field undergoes the critical 1->2 and 2->1 transitions, full memory ordering and mutual exclusion is provided implicitly and, better yet, for free.
In the case where the CPU is coming online, nothing will happen until the current CPU helps it come online. Therefore, the new CPU will see all accesses prior to the optimized grace period, which means that RCU does not need to further delay this new CPU. In the case where the CPU is going offline, the outgoing CPU is totally out of the picture before the optimized grace period starts, which means that this outgoing CPU cannot see any of the accesses following that grace period. Again, RCU needs no further interaction with the outgoing CPU.
This does mean that synchronize_rcu() will unnecessarily do a few grace periods the hard way just before the second CPU comes online and just after the second-to-last CPU goes offline, but it is not worth optimizing this uncommon case.
--- kernel/rcu/tree.c | 19 +++++++++++++++++-- kernel/rcu/tree.h | 1 + 2 files changed, 18 insertions(+), 2 deletions(-)
kernel/rcu/tree.c +17 -2
@@ -2396,6 +2396,7 @@ int rcutree_dead_cpu(unsigned int cpu)if (!IS_ENABLED (CONFIG_HOTPLUG_CPU ))return 0 ; +WRITE_ONCE (rcu_state.n_online_cpus, rcu_state.n_online_cpus -1 );/* Adjust any no-longer-needed kthreads. */ rcu_boost_kthread_setaffinity (rnp,-1 );/* Do any needed no-CB deferred wakeups from this CPU. */
@@ -3577,7 +3578,20 @@ static int rcu_blocking_is_gp(void)return rcu_scheduler_active ==RCU_SCHEDULER_INACTIVE ;might_sleep ();/* Check for RCU read-side critical section. */ preempt_disable (); - ret =num_online_cpus () <=1 ; +/* +* If the rcu_state.n_online_cpus counter is equal to one, +* there is only one CPU, and that CPU sees all prior accesses +* made by any CPU that was online at the time of its access. +* Furthermore, if this counter is equal to one, its value cannot +* change until after the preempt_enable() below. +* +* Furthermore, if rcu_state.n_online_cpus is equal to one here, +* all later CPUs (both this one and any that come online later +* on) are guaranteed to see all accesses prior to this point +* in the code, without the need for additional memory barriers. +* Those memory barriers are provided by CPU-hotplug code. +*/ + ret =READ_ONCE (rcu_state.n_online_cpus) <=1 ;preempt_enable ();return ret; }
@@ -3622,7 +3636,7 @@ void synchronize_rcu(void)lock_is_held (&rcu_sched_lock_map), "Illegal synchronize_rcu() in RCU read-side critical section");if (rcu_blocking_is_gp ()) -return ; +return ;// Context allows vacuous grace periods. if (rcu_gp_is_expedited ())synchronize_rcu_expedited ();else
@@ -3962,6 +3976,7 @@ int rcutree_prepare_cpu(unsigned int cpu)raw_spin_unlock_irqrestore_rcu_node (rnp, flags);rcu_prepare_kthreads (cpu);rcu_spawn_cpu_nocb_kthread (cpu); +WRITE_ONCE (rcu_state.n_online_cpus, rcu_state.n_online_cpus +1 );return 0 ; }
kernel/rcu/tree.h +1 -0
@@ -298,6 +298,7 @@ struct rcu_state {/* Hierarchy levels (+1 to */ /* shut bogus gcc warning) */ int ncpus;/* # CPUs seen so far. */ +int n_online_cpus;/* # CPUs online for RCU. */ /* The following fields are guarded by the root rcu_node's lock. */
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 7/16 rcu: Clarify nocb kthreads naming in RCU_NOCB_CPU config
In reply to Paul E. McKenneyFrom: Neeraj Upadhyay <[email protected]>
This commit clarifies that the "p" and the "s" in the in the RCU_NOCB_CPU config-option description refer to the "x" in the "rcuox/N" kthread name.
--- kernel/rcu/Kconfig | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-)
kernel/rcu/Kconfig +12 -8
@@ -221,19 +221,23 @@ config RCU_NOCB_CPU Use this option to reduce OS jitter for aggressive HPC or real-time workloads. It can also be used to offload RCU callback invocation to energy-efficient CPUs in battery-powered - asymmetric multiprocessors. + asymmetric multiprocessors. The price of this reduced jitter + is that the overhead of call_rcu() increases and that some + workloads will incur significant increases in context-switch + rates. This option offloads callback invocation from the set of CPUs specified at boot time by the rcu_nocbs parameter. For each such CPU, a kthread ("rcuox/N") will be created to invoke callbacks, where the "N" is the CPU being offloaded, and where - the "p" for RCU-preempt (PREEMPTION kernels) and "s" for RCU-sched - (!PREEMPTION kernels). Nothing prevents this kthread from running - on the specified CPUs, but (1) the kthreads may be preempted - between each callback, and (2) affinity or cgroups can be used - to force the kthreads to run on whatever set of CPUs is desired. - - Say Y here if you want to help to debug reduced OS jitter. + the "x" is "p" for RCU-preempt (PREEMPTION kernels) and "s" for + RCU-sched (!PREEMPTION kernels). Nothing prevents this kthread + from running on the specified CPUs, but (1) the kthreads may be + preempted between each callback, and (2) affinity or cgroups can + be used to force the kthreads to run on whatever set of CPUs is + desired. + + Say Y here if you need reduced OS jitter, despite added overhead. Say N here if you are unsure. config TASKS_TRACE_RCU_READ_MB
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 8/16 rcu/tree: Add a warning if CPU being onlined did not report QS already
In reply to Paul E. McKenneyFrom: "Joel Fernandes (Google)" <[email protected]>
Currently, rcu_cpu_starting() checks to see if the RCU core expects a quiescent state from the incoming CPU. However, the current interaction between RCU quiescent-state reporting and CPU-hotplug operations should mean that the incoming CPU never needs to report a quiescent state. First, the outgoing CPU reports a quiescent state if needed. Second, the race where the CPU is leaving just as RCU is initializing a new grace period is handled by an explicit check for this condition. Third, the CPU's leaf rcu_node structure's ->lock serializes these checks.
This means that if rcu_cpu_starting() ever feels the need to report a quiescent state, then there is a bug somewhere in the CPU hotplug code or the RCU grace-period handling code. This commit therefore adds a WARN_ON_ONCE() to bring that bug to everyone's attention.
--- kernel/rcu/tree.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/rcu/tree.c +3 -1
@@ -4075,7 +4075,9 @@ void rcu_cpu_starting(unsigned int cpu)rcu_gpnum_ovf (rnp, rdp);/* Offline-induced counter wrap? */ rdp->rcu_onl_gp_seq =READ_ONCE (rcu_state.gp_seq); rdp->rcu_onl_gp_flags =READ_ONCE (rcu_state.gp_flags); -if (rnp->qsmask & mask) {/* RCU waiting on incoming CPU? */ + +/* An incoming CPU should never be blocking a grace period. */ +if (WARN_ON_ONCE (rnp->qsmask & mask)) {/* RCU waiting on incoming CPU? */ rcu_disable_urgency_upon_qs (rdp);/* Report QS -after- changing ->qsmaskinitnext! */ rcu_report_qs_rnp (mask, rnp, rnp->gp_seq, flags);
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 9/16 rcu/tree: Make struct kernel_param_ops definitions const
In reply to Paul E. McKenneyFrom: Joe Perches <[email protected]>
These should be const, so make it so.
--- kernel/rcu/tree.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/rcu/tree.c +2 -2
@@ -546,12 +546,12 @@ static int param_set_next_fqs_jiffies(const char *val, const struct kernel_paramreturn ret; } -static struct kernel_param_ops first_fqs_jiffies_ops = { +static const struct kernel_param_ops first_fqs_jiffies_ops = { .set = param_set_first_fqs_jiffies, .get = param_get_ulong, }; -static struct kernel_param_ops next_fqs_jiffies_ops = { +static const struct kernel_param_ops next_fqs_jiffies_ops = { .set = param_set_next_fqs_jiffies, .get = param_get_ulong, };
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 10/16 rcu,ftrace: Fix ftrace recursion
In reply to Paul E. McKenneyFrom: Peter Zijlstra <[email protected]>
Kim reported that perf-ftrace made his box unhappy. It turns out that commit:
ff5c4f5cad33 ("rcu/tree: Mark the idle relevant functions noinstr")
removed one too many notrace qualifiers, probably due to there not being a helpful comment.
This commit therefore reinstates the notrace and adds a comment to avoid losing it again.
--- kernel/rcu/tree.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/rcu/tree.c +4 -1
@@ -1093,8 +1093,11 @@ static void rcu_disable_urgency_upon_qs(struct rcu_data *rdp)* CPU can safely enter RCU read-side critical sections. In other words, * if the current CPU is not in its idle loop or is in an interrupt or * NMI handler, return true. +* +* Make notrace because it can be called by the internal functions of +* ftrace, and making this notrace removes unnecessary recursion calls. */ -bool rcu_is_watching (void ) +notrace bool rcu_is_watching(void) {bool ret;
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 11/16 rcu/tree: nocb: Avoid raising softirq for offloaded ready-to-execute CBs
In reply to Paul E. McKenneyFrom: "Joel Fernandes (Google)" <[email protected]>
Testing showed that rcu_pending() can return 1 when offloaded callbacks are ready to execute. This invokes RCU core processing, for example, by raising RCU_SOFTIRQ, eventually resulting in a call to rcu_core(). However, rcu_core() explicitly avoids in any way manipulating offloaded callbacks, which are instead handled by the rcuog and rcuoc kthreads, which work independently of rcu_core().
One exception to this independence is that rcu_core() invokes do_nocb_deferred_wakeup(), however, rcu_pending() also checks rcu_nocb_need_deferred_wakeup() in order to correctly handle this case, invoking rcu_core() when needed.
This commit therefore avoids needlessly invoking RCU core processing by checking rcu_segcblist_ready_cbs() only on non-offloaded CPUs. This reduces overhead, for example, by reducing softirq activity.
This change passed 30 minute tests of TREE01 through TREE09 each.
On TREE08, there is at most 150us from the time that rcu_pending() chose not to invoke RCU core processing to the time when the ready callbacks were invoked by the rcuoc kthread. This provides further evidence that there is no need to invoke rcu_core() for offloaded callbacks that are ready to invoke.
--- kernel/rcu/tree.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/rcu/tree.c +2 -1
@@ -3718,7 +3718,8 @@ static int rcu_pending(int user)return 1 ;/* Does this CPU have callbacks ready to invoke? */ -if (rcu_segcblist_ready_cbs (&rdp->cblist)) +if (!rcu_segcblist_is_offloaded (&rdp->cblist) && +rcu_segcblist_ready_cbs (&rdp->cblist))return 1 ;/* Has RCU gone idle with this CPU needing another grace period? */
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 12/16 rcu: Prevent lockdep-RCU splats on lock acquisition/release
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The rcu_cpu_starting() and rcu_report_dead() functions transition the current CPU between online and offline state from an RCU perspective. Unfortunately, this means that the rcu_cpu_starting() function's lock acquisition and the rcu_report_dead() function's lock releases happen while the CPU is offline from an RCU perspective, which can result in lockdep-RCU splats about using RCU from an offline CPU. And this situation can also result in too-short grace periods, especially in guest OSes that are subject to vCPU preemption.
This commit therefore uses sequence-count-like synchronization to forgive use of RCU while RCU thinks a CPU is offline across the full extent of the rcu_cpu_starting() and rcu_report_dead() function's lock acquisitions and releases.
One approach would have been to use the actual sequence-count primitives provided by the Linux kernel. Unfortunately, the resulting code looks completely broken and wrong, and is likely to result in patches that break RCU in an attempt to address this appearance of broken wrongness. Plus there is no net savings in lines of code, given the additional explicit memory barriers required.
Therefore, this sequence count is instead implemented by a new ->ofl_seq field in the rcu_node structure. If this counter's value is an odd number, RCU forgives RCU read-side critical sections on other CPUs covered by the same rcu_node structure, even if those CPUs are offline from an RCU perspective. In addition, if a given leaf rcu_node structure's ->ofl_seq counter value is an odd number, rcu_gp_init() delays starting the grace period until that counter value changes.
--- kernel/rcu/tree.c | 21 ++++++++++++++++++++- kernel/rcu/tree.h | 1 + 2 files changed, 21 insertions(+), 1 deletion(-)
kernel/rcu/tree.c +20 -1
@@ -1152,7 +1152,7 @@ bool rcu_lockdep_current_cpu_online(void)preempt_disable_notrace (); rdp =this_cpu_ptr (&rcu_data); rnp = rdp->mynode; -if (rdp->grpmask &rcu_rnp_online_cpus (rnp)) +if (rdp->grpmask &rcu_rnp_online_cpus (rnp) ||READ_ONCE (rnp->ofl_seq) &0x1 ) ret = true;preempt_enable_notrace ();return ret;
@@ -1717,6 +1717,7 @@ static void rcu_strict_gp_boundary(void *unused)*/ static bool rcu_gp_init (void ) { +unsigned long firstseq;unsigned long flags;unsigned long oldmask;unsigned long mask;
@@ -1760,6 +1761,12 @@ static bool rcu_gp_init(void)*/ rcu_state.gp_state =RCU_GP_ONOFF ;rcu_for_each_leaf_node (rnp) { +smp_mb ();// Pair with barriers used when updating ->ofl_seq to odd values. + firstseq =READ_ONCE (rnp->ofl_seq); +if (firstseq &0x1 ) +while (firstseq ==READ_ONCE (rnp->ofl_seq)) +schedule_timeout_idle (1 );// Can't wake unless RCU is watching. +smp_mb ();// Pair with barriers used when updating ->ofl_seq to even values. raw_spin_lock (&rcu_state.ofl_lock);raw_spin_lock_irq_rcu_node (rnp); if (rnp->qsmaskinit == rnp->qsmaskinitnext &&
@@ -4069,6 +4076,9 @@ void rcu_cpu_starting(unsigned int cpu) rnp = rdp->mynode; mask = rdp->grpmask; +WRITE_ONCE (rnp->ofl_seq, rnp->ofl_seq +1 ); +WARN_ON_ONCE (!(rnp->ofl_seq &0x1 )); +smp_mb ();// Pair with rcu_gp_cleanup()'s ->ofl_seq barrier(). raw_spin_lock_irqsave_rcu_node (rnp, flags);WRITE_ONCE (rnp->qsmaskinitnext, rnp->qsmaskinitnext | mask); newcpu = !(rnp->expmaskinitnext & mask);
@@ -4088,6 +4098,9 @@ void rcu_cpu_starting(unsigned int cpu) }else {raw_spin_unlock_irqrestore_rcu_node (rnp, flags); } +smp_mb ();// Pair with rcu_gp_cleanup()'s ->ofl_seq barrier(). +WRITE_ONCE (rnp->ofl_seq, rnp->ofl_seq +1 ); +WARN_ON_ONCE (rnp->ofl_seq &0x1 );smp_mb ();/* Ensure RCU read-side usage follows above initialization. */ }
@@ -4115,6 +4128,9 @@ void rcu_report_dead(unsigned int cpu)/* Remove outgoing CPU from mask in the leaf rcu_node structure. */ mask = rdp->grpmask; +WRITE_ONCE (rnp->ofl_seq, rnp->ofl_seq +1 ); +WARN_ON_ONCE (!(rnp->ofl_seq &0x1 )); +smp_mb ();// Pair with rcu_gp_cleanup()'s ->ofl_seq barrier(). raw_spin_lock (&rcu_state.ofl_lock);raw_spin_lock_irqsave_rcu_node (rnp, flags);/* Enforce GP memory-order guarantee. */ rdp->rcu_ofl_gp_seq =READ_ONCE (rcu_state.gp_seq);
@@ -4127,6 +4143,9 @@ void rcu_report_dead(unsigned int cpu)WRITE_ONCE (rnp->qsmaskinitnext, rnp->qsmaskinitnext & ~mask);raw_spin_unlock_irqrestore_rcu_node (rnp, flags);raw_spin_unlock (&rcu_state.ofl_lock); +smp_mb ();// Pair with rcu_gp_cleanup()'s ->ofl_seq barrier(). +WRITE_ONCE (rnp->ofl_seq, rnp->ofl_seq +1 ); +WARN_ON_ONCE (rnp->ofl_seq &0x1 ); rdp->cpu_started = false; }
kernel/rcu/tree.h +1 -0
@@ -56,6 +56,7 @@ struct rcu_node {/* Initialized from ->qsmaskinitnext at the */ /* beginning of each grace period. */ unsigned long qsmaskinitnext; +unsigned long ofl_seq;/* CPU-hotplug operation sequence count. */ /* Online CPUs for next grace period. */ unsigned long expmask;/* CPUs or groups that need to check in */ /* to allow the current expedited GP */
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 13/16 rcu: Fix a typo in rcu_blocking_is_gp() header comment
In reply to Paul E. McKenneyFrom: Zhouyi Zhou <[email protected]>
This commit fixes a typo in the rcu_blocking_is_gp() function's header comment.
--- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
@@ -3572,7 +3572,7 @@ void __init kfree_rcu_scheduler_running(void)* During early boot, any blocking grace-period wait automatically * implies a grace period. Later on, this is never the case for PREEMPTION. * -* Howevr, because a context switch is a grace period for !PREEMPTION, any +* However, because a context switch is a grace period for !PREEMPTION, any * blocking grace-period wait automatically implies a grace period if * there is only one CPU online at any point time during execution of * either synchronize_rcu() or synchronize_rcu_expedited(). It is OK to
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 14/16 rcu: Do not report strict GPs for outgoing CPUs
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
An outgoing CPU is marked offline in a stop-machine handler and most of that CPU's services stop at that point, including IRQ work queues. However, that CPU must take another pass through the scheduler and through a number of CPU-hotplug notifiers, many of which contain RCU readers. In the past, these readers were not a problem because the outgoing CPU has interrupts disabled, so that rcu_read_unlock_special() would not be invoked, and thus RCU would never attempt to queue IRQ work on the outgoing CPU.
This changed with the advent of the CONFIG_RCU_STRICT_GRACE_PERIOD Kconfig option, in which rcu_read_unlock_special() is invoked upon exit from almost all RCU read-side critical sections. Worse yet, because interrupts are disabled, rcu_read_unlock_special() cannot immediately report a quiescent state and will therefore attempt to defer this reporting, for example, by queueing IRQ work. Which fails with a splat because the CPU is already marked as being offline.
But it turns out that there is no need to report this quiescent state because rcu_report_dead() will do this job shortly after the outgoing CPU makes its final dive into the idle loop. This commit therefore makes rcu_read_unlock_special() refrain from queuing IRQ work onto outgoing CPUs.
--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +1 -1
@@ -628,7 +628,7 @@ static void rcu_read_unlock_special(struct task_struct *t)set_tsk_need_resched (current);set_preempt_need_resched ();if (IS_ENABLED (CONFIG_IRQ_WORK ) && irqs_were_disabled && - !rdp->defer_qs_iw_pending && exp) { + !rdp->defer_qs_iw_pending && exp &&cpu_online (rdp->cpu)) {// Get scheduler to re-evaluate and call hooks. // If !IRQ_WORK, FQS scan will eventually IPI. init_irq_work(&rdp->defer_qs_iw,
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 15/16 rcu/tree: Defer kvfree_rcu() allocation to a clean context
In reply to Paul E. McKenneyFrom: "Uladzislau Rezki (Sony)" <[email protected]>
The current memmory-allocation interface causes the following difficulties for kvfree_rcu():
a) If built with CONFIG_PROVE_RAW_LOCK_NESTING, the lockdep will complain about violation of the nesting rules, as in "BUG: Invalid wait context". This Kconfig option checks for proper raw_spinlock vs. spinlock nesting, in particular, it is not legal to acquire a spinlock_t while holding a raw_spinlock_t.
This is a problem because kfree_rcu() uses raw_spinlock_t whereas the "page allocator" internally deals with spinlock_t to access to its zones. The code also can be broken from higher level of view:
<snip>
raw_spin_lock(&some_lock);
kfree_rcu(some_pointer, some_field_offset);<snip>
b) If built with CONFIG_PREEMPT_RT, spinlock_t is converted into sleeplock. This means that invoking the page allocator from atomic contexts results in "BUG: scheduling while atomic".
c) Please note that call_rcu() is already invoked from raw atomic context, so it is only reasonable to expaect that kfree_rcu() and kvfree_rcu() will also be called from atomic raw context.
This commit therefore defers page allocation to a clean context using the combination of an hrtimer and a workqueue. The hrtimer stage is required in order to avoid deadlocks with the scheduler. This deferred allocation is required only when kvfree_rcu()'s per-CPU page cache is empty.
--- kernel/rcu/tree.c | 109 +++++++++++++++++++++++++++++++++--------------------- 1 file changed, 66 insertions(+), 43 deletions(-)
kernel/rcu/tree.c +66 -43
@@ -177,7 +177,7 @@ module_param(rcu_unlock_delay, int, 0444);* per-CPU. Object size is equal to one page. This value * can be changed at boot time. */ -static int rcu_min_cached_objs =2 ; +static int rcu_min_cached_objs =5 ;module_param (rcu_min_cached_objs, int,0444 );/* Retrieve RCU kthreads priority for rcutorture */
@@ -3089,6 +3089,9 @@ struct kfree_rcu_cpu_work {* In order to save some per-cpu space the list is singular. * Even though it is lockless an access has to be protected by the * per-cpu lock. +* @page_cache_work: A work to refill the cache when it is empty +* @work_in_progress: Indicates that page_cache_work is running +* @hrtimer: A hrtimer for scheduling a page_cache_work * @nr_bkv_objs: number of allocated objects at @bkvcache. * * This is a per-CPU structure. The reason that it is not included in
@@ -3105,6 +3108,11 @@ struct kfree_rcu_cpu {bool monitor_todo;bool initialized;int count; + +struct work_struct page_cache_work; +atomic_t work_in_progress; +struct hrtimer hrtimer; +struct llist_head bkvcache;int nr_bkv_objs; };
@@ -3222,10 +3230,10 @@ static void kfree_rcu_work(struct work_struct *work) }rcu_lock_release (&rcu_callback_map); - krcp =krc_this_cpu_lock (&flags); +raw_spin_lock_irqsave (&krcp->lock, flags);if (put_cached_bnode (krcp, bkvhead[i])) bkvhead[i] =NULL ; -krc_this_cpu_unlock (krcp, flags); +raw_spin_unlock_irqrestore (&krcp->lock, flags);if (bkvhead[i])free_page ((unsigned long ) bkvhead[i]);
@@ -3352,6 +3360,57 @@ static void kfree_rcu_monitor(struct work_struct *work)raw_spin_unlock_irqrestore (&krcp->lock, flags); } +static enum hrtimer_restart +schedule_page_work_fn (struct hrtimer *t) +{ +struct kfree_rcu_cpu *krcp = + container_of(t, struct kfree_rcu_cpu, hrtimer); + +queue_work (system_highpri_wq, &krcp->page_cache_work); +return HRTIMER_NORESTART ; +} + +static void fill_page_cache_func (struct work_struct *work) +{ +struct kvfree_rcu_bulk_data *bnode; +struct kfree_rcu_cpu *krcp = + container_of(work, struct kfree_rcu_cpu, + page_cache_work); +unsigned long flags; +bool pushed; +int i; + +for (i =0 ; i < rcu_min_cached_objs; i++) { + bnode = (struct kvfree_rcu_bulk_data *) +__get_free_page (GFP_KERNEL | __GFP_NOWARN); + +if (bnode) { +raw_spin_lock_irqsave (&krcp->lock, flags); + pushed =put_cached_bnode (krcp, bnode); +raw_spin_unlock_irqrestore (&krcp->lock, flags); + +if (!pushed) { +free_page ((unsigned long ) bnode); +break ; + } + } + } + +atomic_set (&krcp->work_in_progress,0 ); +} + +static void +run_page_cache_worker (struct kfree_rcu_cpu *krcp) +{ +if (rcu_scheduler_active ==RCU_SCHEDULER_RUNNING && + !atomic_xchg (&krcp->work_in_progress,1 )) { +hrtimer_init (&krcp->hrtimer,CLOCK_MONOTONIC , +HRTIMER_MODE_REL ); + krcp->hrtimer.function = schedule_page_work_fn; +hrtimer_start (&krcp->hrtimer,0 ,HRTIMER_MODE_REL ); + } +} +static inline bool kvfree_call_rcu_add_ptr_to_bulk (struct kfree_rcu_cpu *krcp,void *ptr) {
@@ -3368,32 +3427,8 @@ kvfree_call_rcu_add_ptr_to_bulk(struct kfree_rcu_cpu *krcp, void *ptr)if (!krcp->bkvhead[idx] || krcp->bkvhead[idx]->nr_records ==KVFREE_BULK_MAX_ENTR ) { bnode =get_cached_bnode (krcp); -if (!bnode) { -/* -* To keep this path working on raw non-preemptible -* sections, prevent the optional entry into the -* allocator as it uses sleeping locks. In fact, even -* if the caller of kfree_rcu() is preemptible, this -* path still is not, as krcp->lock is a raw spinlock. -* With additional page pre-allocation in the works, -* hitting this return is going to be much less likely. -*/ -if (IS_ENABLED (CONFIG_PREEMPT_RT )) -return false; - -/* -* NOTE: For one argument of kvfree_rcu() we can -* drop the lock and get the page in sleepable -* context. That would allow to maintain an array -* for the CONFIG_PREEMPT_RT as well if no cached -* pages are available. -*/ - bnode = (struct kvfree_rcu_bulk_data *) -__get_free_page (GFP_NOWAIT | __GFP_NOWARN); - } -/* Switch to emergency path. */ -if (unlikely (!bnode)) +if (!bnode)return false;/* Initialize the new block. */
@@ -3457,12 +3492,10 @@ void kvfree_call_rcu(struct rcu_head *head, rcu_callback_t func) goto unlock_return; } -/* -* Under high memory pressure GFP_NOWAIT can fail, -* in that case the emergency path is maintained. -*/ success =kvfree_call_rcu_add_ptr_to_bulk (krcp, ptr);if (!success) { +run_page_cache_worker (krcp); +if (head ==NULL )// Inline if kvfree_rcu(one_arg) call. goto unlock_return;
@@ -4482,24 +4515,14 @@ static void __init kfree_rcu_batch_init(void)for_each_possible_cpu (cpu) {struct kfree_rcu_cpu *krcp =per_cpu_ptr (&krc, cpu); -struct kvfree_rcu_bulk_data *bnode;for (i =0 ; i <KFREE_N_BATCHES ; i++) {INIT_RCU_WORK (&krcp->krw_arr[i].rcu_work, kfree_rcu_work); krcp->krw_arr[i].krcp = krcp; } -for (i =0 ; i < rcu_min_cached_objs; i++) { - bnode = (struct kvfree_rcu_bulk_data *) -__get_free_page (GFP_NOWAIT | __GFP_NOWARN); - -if (bnode) -put_cached_bnode (krcp, bnode); -else -pr_err ("Failed to preallocate for %d CPU!\n" , cpu); - } -INIT_DELAYED_WORK (&krcp->monitor_work, kfree_rcu_monitor); +INIT_WORK (&krcp->page_cache_work, fill_page_cache_func); krcp->initialized = true; }if (register_shrinker (&kfree_rcu_shrinker))
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 16/16 srcu: Take early exit on memory-allocation failure
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
It turns out that init_srcu_struct() can be invoked from usermode tasks, and that fatal signals received by these tasks can cause memory-allocation failures. These failures are not handled well by init_srcu_struct(), so much so that NULL pointer dereferences can result. This commit therefore causes init_srcu_struct() to take an early exit upon detection of memory-allocation failure.
--- kernel/rcu/srcutree.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/rcu/srcutree.c +3 -1
@@ -177,11 +177,13 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static)INIT_DELAYED_WORK (&ssp->work, process_srcu);if (!is_static) ssp->sda = alloc_percpu(struct srcu_data); +if (!ssp->sda) +return -ENOMEM ;init_srcu_struct_nodes (ssp, is_static); ssp->srcu_gp_seq_needed_exp =0 ; ssp->srcu_last_gp_end =ktime_get_mono_fast_ns ();smp_store_release (&ssp->srcu_gp_seq_needed,0 );/* Init done. */ -return ssp->sda ?0 : -ENOMEM ; +return 0 ; }#ifdef CONFIG_DEBUG_LOCK_ALLOC
Signature · 1 line
2.9.5