NOCBs and NO_HZ_FULL changes for 3.18
cover NOCBs and NO_HZ_FULL changes for 3.18
Hello!
This series includes changes affecting NOCBs and NO_HZ_FULL operation.
1. Fix for bug in earlier patch (b58cc46c5f6b, Don't offload
callbacks unless specifically requested) that properly handles early-boot callbacks. (This bug affects unusual configurations, see commit log for more information.)
2. Properly check the return value of zalloc_cpumask_var(), courtesy
of Pranith Kumar.
3. Use bool constant for __call_rcu_nocb() return value, courtesy
of Pranith Kumar.
4. Use bool constant for rcu_nocb_adopt_orphan_cbs() return value,
courtesy of Pranith Kumar.
5. Use bool constant for __call_rcu_nocb() return value in the case
where callbacks are not offloaded, courtesy of Pranith Kumar.
6. Use bool constant for rcu_nocb_adopt_orphan_cbs() return value in
the case where callbacks are not offloaded, courtesy of Pranith Kumar.
7. Aggregate RCU's kthread creation so that it is all done from a
single early_initcall() function.
8. Create rcuo kthreads only for CPUs that actually have come online
at least once since boot.
9. Check have_rcu_nocb_mask instead of for a NULL rcu_nocb_mask,
given that this latter is not necessarily a pointer, courtesy of Pranith Kumar.
10. Eliminate redundant rcu_sysidle_state variable.
11. Don't bother tracking sysidle state if there are no nohz_full=
CPUs in the system.
12. Avoid misordering in __call_rcu_nocb_enqueue().
13. Handle NOCB callbacks from irq-disabled idle code.
14. Avoid misordering in nocb_leader_wait().
Thanx, Paul
------------------------------------------------------------------------
b/include/linux/rcupdate.h | 8 + b/init/Kconfig | 4 b/init/main.c | 1 b/kernel/rcu/tree.c | 4 b/kernel/rcu/tree.h | 1 b/kernel/rcu/tree_plugin.h | 92 +++++++++++++++------- kernel/rcu/tree.c | 3 kernel/rcu/tree.h | 6 + kernel/rcu/tree_plugin.h | 182 ++++++++++++++++++++++++++++++++------------- 9 files changed, 216 insertions(+), 85 deletions(-)
PATCH 1/14 rcu: Fix attempt to avoid unsolicited offloading of callbacks
From: "Paul E. McKenney" <[email protected]>
Commit b58cc46c5f6b (rcu: Don't offload callbacks unless specifically requested) failed to adjust the callback lists of the CPUs that are known to be no-CBs CPUs only because they are also nohz_full= CPUs. This failure can result in callbacks that are posted during early boot getting stranded on nxtlist for CPUs whose no-CBs property becomes apparent late, and there can also be spurious warnings about offline CPUs posting callbacks.
This commit fixes these problems by adding an early-boot rcu_init_nohz() that properly initializes the no-CBs CPUs.
Note that kernels built with CONFIG_RCU_NOCB_CPU_ALL=y or with CONFIG_RCU_NOCB_CPU=n do not exhibit this bug. Neither do kernels booted without the nohz_full= boot parameter.
--- include/linux/rcupdate.h | 8 +++++ init/Kconfig | 4 +-- init/main.c | 1 + kernel/rcu/tree_plugin.h | 92 ++++++++++++++++++++++++++++++++---------------- 4 files changed, 72 insertions(+), 33 deletions(-)
include/linux/rcupdate.h +8 -0
@@ -269,6 +269,14 @@ static inline void rcu_user_hooks_switch(struct task_struct *prev, struct task_struct *next) { }#endif /* CONFIG_RCU_USER_QS */ +#ifdef CONFIG_RCU_NOCB_CPU +void rcu_init_nohz (void ); +#else /* #ifdef CONFIG_RCU_NOCB_CPU */ +static inline void rcu_init_nohz (void ) +{ +} +#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */ +/** * RCU_NONIDLE - Indicate idle-loop code that needs RCU readers * @a: Code that RCU needs to pay attention to.
init/Kconfig +2 -2
@@ -737,7 +737,7 @@ choice config RCU_NOCB_CPU_NONE bool "No build_forced no-CBs CPUs" - depends on RCU_NOCB_CPU && !NO_HZ_FULL_ALL + depends on RCU_NOCB_CPU help This option does not force any of the CPUs to be no-CBs CPUs. Only CPUs designated by the rcu_nocbs= boot parameter will be
@@ -751,7 +751,7 @@ config RCU_NOCB_CPU_NONE config RCU_NOCB_CPU_ZERO bool "CPU 0 is a build_forced no-CBs CPU" - depends on RCU_NOCB_CPU && !NO_HZ_FULL_ALL + depends on RCU_NOCB_CPU help This option forces CPU 0 to be a no-CBs CPU, so that its RCU callbacks are invoked by a per-CPU kthread whose name begins
init/main.c +1 -0
@@ -578,6 +578,7 @@ asmlinkage __visible void __init start_kernel(void)idr_init_cache ();rcu_init ();tick_nohz_init (); +rcu_init_nohz ();context_tracking_init ();radix_tree_init ();/* init some links before init_ISA_irqs() */
kernel/rcu/tree_plugin.h +61 -31
110 lines not on this page · raw message
Signature · 1 line
1.8.1.5
PATCH 2/14 rcu: Check the return value of zalloc_cpumask_var()
From: Pranith Kumar <[email protected]>
This commit checks the return value of the zalloc_cpumask_var() used for allocating cpumask for rcu_nocb_mask.
--- kernel/rcu/tree_plugin.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +4 -1
@@ -2440,7 +2440,10 @@ void __init rcu_init_nohz(void)#endif /* #if defined(CONFIG_NO_HZ_FULL) */ if (!have_rcu_nocb_mask && need_rcu_nocb_mask) { -zalloc_cpumask_var (&rcu_nocb_mask,GFP_KERNEL ); +if (!zalloc_cpumask_var (&rcu_nocb_mask,GFP_KERNEL )) { +pr_info ("rcu_nocb_mask allocation failed, callback offloading disabled.\n" ); +return ; + } have_rcu_nocb_mask = true; }if (!have_rcu_nocb_mask)
Signature · 1 line
1.8.1.5
PATCH 3/14 rcu: Use true/false for return in __call_rcu_nocb()
In reply to Paul E. McKenneyFrom: Pranith Kumar <[email protected]>
Return true/false instead of 0/1 in __call_rcu_nocb() as this returns a bool type.
--- kernel/rcu/tree_plugin.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/rcu/tree_plugin.h +2 -2
@@ -2123,7 +2123,7 @@ static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp, {if (!rcu_is_nocb_cpu (rdp->cpu)) -return 0 ; +return false;__call_rcu_nocb_enqueue (rdp, rhp, &rhp->next,1 , lazy, flags);if (__is_kfree_rcu_offset ((unsigned long )rhp->func)) trace_rcu_kfree_callback(rdp->rsp->name, rhp,
@@ -2134,7 +2134,7 @@ static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp,trace_rcu_callback (rdp->rsp->name, rhp, -atomic_long_read (&rdp->nocb_q_count_lazy), -atomic_long_read (&rdp->nocb_q_count)); -return 1 ; +return true; }/*
Signature · 1 line
1.8.1.5
PATCH 4/14 rcu: Use true/false for return in rcu_nocb_adopt_orphan_cbs()
In reply to Paul E. McKenneyFrom: Pranith Kumar <[email protected]>
Return true/false in rcu_nocb_adopt_orphan_cbs() instead of 0/1 as this function has return type of bool.
--- kernel/rcu/tree_plugin.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/rcu/tree_plugin.h +2 -2
@@ -2150,7 +2150,7 @@ static bool __maybe_unused rcu_nocb_adopt_orphan_cbs(struct rcu_state *rsp,/* If this is not a no-CBs CPU, tell the caller to do it the old way. */ if (!rcu_is_nocb_cpu (smp_processor_id ())) -return 0 ; +return false; rsp->qlen =0 ; rsp->qlen_lazy =0 ;
@@ -2169,7 +2169,7 @@ static bool __maybe_unused rcu_nocb_adopt_orphan_cbs(struct rcu_state *rsp, rsp->orphan_nxtlist =NULL ; rsp->orphan_nxttail = &rsp->orphan_nxtlist; } -return 1 ; +return true; }/*
Signature · 1 line
1.8.1.5
PATCH 5/14 rcu: Use false for return in __call_rcu_nocb()
In reply to Paul E. McKenneyFrom: Pranith Kumar <[email protected]>
Return false instead of 0 in __call_rcu_nocb() as this has bool as return type.
--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +1 -1
@@ -2574,7 +2574,7 @@ static void rcu_init_one_nocb(struct rcu_node *rnp)static bool __call_rcu_nocb (struct rcu_data *rdp,struct rcu_head *rhp,bool lazy,unsigned long flags) { -return 0 ; +return false; } static bool __maybe_unused rcu_nocb_adopt_orphan_cbs(struct rcu_state *rsp,
Signature · 1 line
1.8.1.5
PATCH 6/14 rcu: Return false instead of 0 in rcu_nocb_adopt_orphan_cbs()
In reply to Paul E. McKenneyFrom: Pranith Kumar <[email protected]>
Return false instead of 0 in rcu_nocb_adopt_orphan_cbs() as this has bool as return type.
--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +1 -1
@@ -2581,7 +2581,7 @@ static bool __maybe_unused rcu_nocb_adopt_orphan_cbs(struct rcu_state *rsp, struct rcu_data *rdp, unsigned long flags) { -return 0 ; +return false; }static void __init rcu_boot_init_nocb_percpu_data (struct rcu_data *rdp)
Signature · 1 line
1.8.1.5
PATCH 7/14 rcu: Rationalize kthread spawning
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Currently, RCU spawns kthreads from several different early_initcall() functions. Although this has served RCU well for quite some time, as more kthreads are added a more deterministic approach is required. This commit therefore causes all of RCU's early-boot kthreads to be spawned from a single early_initcall() function.
--- kernel/rcu/tree.c | 4 +++- kernel/rcu/tree.h | 1 + kernel/rcu/tree_plugin.h | 12 +++--------- 3 files changed, 7 insertions(+), 10 deletions(-)
kernel/rcu/tree.c +3 -1
@@ -3489,7 +3489,7 @@ static int rcu_pm_notify(struct notifier_block *self, }/* -* Spawn the kthread that handles this RCU flavor's grace periods. +* Spawn the kthreads that handle each RCU flavor's grace periods. */ static int __init rcu_spawn_gp_kthread (void ) {
@@ -3498,6 +3498,7 @@ static int __init rcu_spawn_gp_kthread(void)struct rcu_state *rsp;struct task_struct *t; + rcu_scheduler_fully_active =1 ;for_each_rcu_flavor (rsp) { t =kthread_run (rcu_gp_kthread, rsp,"%s" , rsp->name);BUG_ON (IS_ERR (t));
@@ -3507,6 +3508,7 @@ static int __init rcu_spawn_gp_kthread(void)raw_spin_unlock_irqrestore (&rnp->lock, flags);rcu_spawn_nocb_kthreads (rsp); } +rcu_spawn_boost_kthreads ();return 0 ; }early_initcall (rcu_spawn_gp_kthread);
kernel/rcu/tree.h +1 -0
@@ -572,6 +572,7 @@ static void rcu_preempt_do_callbacks(void);static int rcu_spawn_one_boost_kthread (struct rcu_state *rsp,struct rcu_node *rnp);#endif /* #ifdef CONFIG_RCU_BOOST */ +static void __init rcu_spawn_boost_kthreads (void );static void rcu_prepare_kthreads (int cpu);static void rcu_cleanup_after_idle (int cpu);static void rcu_prepare_for_idle (int cpu);
kernel/rcu/tree_plugin.h +3 -9
@@ -1435,14 +1435,13 @@ static struct smp_hotplug_thread rcu_cpu_thread_spec = { };/* -* Spawn all kthreads -- called as soon as the scheduler is running. +* Spawn boost kthreads -- called as soon as the scheduler is running. */ -static int __init rcu_spawn_kthreads (void ) +static void __init rcu_spawn_boost_kthreads (void ) {struct rcu_node *rnp;int cpu; - rcu_scheduler_fully_active =1 ;for_each_possible_cpu (cpu)per_cpu (rcu_cpu_has_work, cpu) =0 ;BUG_ON (smpboot_register_percpu_thread (&rcu_cpu_thread_spec));
@@ -1452,9 +1451,7 @@ static int __init rcu_spawn_kthreads(void)rcu_for_each_leaf_node (rcu_state_p, rnp) (void )rcu_spawn_one_boost_kthread (rcu_state_p, rnp); } -return 0 ; } -early_initcall (rcu_spawn_kthreads);static void rcu_prepare_kthreads (int cpu) {
@@ -1492,12 +1489,9 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu) { } -static int __init rcu_scheduler_really_started (void ) +static void __init rcu_spawn_boost_kthreads (void ) { - rcu_scheduler_fully_active =1 ; -return 0 ; } -early_initcall (rcu_scheduler_really_started);static void rcu_prepare_kthreads (int cpu) {
Signature · 1 line
1.8.1.5
PATCH 8/14 rcu: Create rcuo kthreads only for onlined CPUs
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
RCU currently uses for_each_possible_cpu() to spawn rcuo kthreads, which can result in more rcuo kthreads than one would expect, for example, derRichard reported 64 CPUs worth of rcuo kthreads on an 8-CPU image. This commit therefore creates rcuo kthreads only for those CPUs that actually come online.
This was reported by derRichard on the OFTC IRC network.
--- kernel/rcu/tree.c | 3 +- kernel/rcu/tree.h | 6 +++- kernel/rcu/tree_plugin.h | 90 ++++++++++++++++++++++++++++++++++++++++++------ 3 files changed, 86 insertions(+), 13 deletions(-)
kernel/rcu/tree.c +2 -1
@@ -3442,6 +3442,7 @@ static int rcu_cpu_notify(struct notifier_block *self,case CPU_UP_PREPARE_FROZEN :rcu_prepare_cpu (cpu);rcu_prepare_kthreads (cpu); +rcu_spawn_all_nocb_kthreads (cpu);break ;case CPU_ONLINE :case CPU_DOWN_FAILED :
@@ -3506,8 +3507,8 @@ static int __init rcu_spawn_gp_kthread(void)raw_spin_lock_irqsave (&rnp->lock, flags); rsp->gp_kthread = t;raw_spin_unlock_irqrestore (&rnp->lock, flags); -rcu_spawn_nocb_kthreads (rsp); } +rcu_spawn_nocb_kthreads ();rcu_spawn_boost_kthreads ();return 0 ; }
kernel/rcu/tree.h +5 -1
@@ -593,7 +593,11 @@ static bool rcu_nocb_adopt_orphan_cbs(struct rcu_state *rsp,static bool rcu_nocb_need_deferred_wakeup (struct rcu_data *rdp);static void do_nocb_deferred_wakeup (struct rcu_data *rdp);static void rcu_boot_init_nocb_percpu_data (struct rcu_data *rdp); -static void rcu_spawn_nocb_kthreads (struct rcu_state *rsp); +static void rcu_spawn_all_nocb_kthreads (int cpu); +static void __init rcu_spawn_nocb_kthreads (void ); +#ifdef CONFIG_RCU_NOCB_CPU +static void __init rcu_organize_nocb_kthreads (struct rcu_state *rsp); +#endif /* #ifdef CONFIG_RCU_NOCB_CPU */ static void __maybe_unused rcu_kick_nohz_cpu (int cpu);static bool init_nocb_callback_list (struct rcu_data *rdp);static void rcu_sysidle_enter (struct rcu_dynticks *rdtp,int irq);
kernel/rcu/tree_plugin.h +79 -11
126 lines not on this page · raw message
Signature · 1 line
1.8.1.5
PATCH 9/14 rcu: Check for have_rcu_nocb_mask instead of rcu_nocb_mask
In reply to Paul E. McKenneyFrom: Pranith Kumar <[email protected]>
If we configure a kernel with CONFIG_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_NONE=y and CONFIG_CPUMASK_OFFSTACK=n and do not pass in a rcu_nocb= boot parameter, the cpumask rcu_nocb_mask can be garbage instead of NULL.
Hence this commit replaces checks for rcu_nocb_mask == NULL with a check for have_rcu_nocb_mask.
--- kernel/rcu/tree_plugin.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/rcu/tree_plugin.h +3 -3
@@ -2578,7 +2578,7 @@ static void __init rcu_organize_nocb_kthreads(struct rcu_state *rsp)struct rcu_data *rdp_leader =NULL ;/* Suppress misguided gcc warn. */ struct rcu_data *rdp_prev =NULL ; -if (rcu_nocb_mask ==NULL ) +if (!have_rcu_nocb_mask)return ;if (ls ==-1 ) { ls =int_sqrt (nr_cpu_ids);
@@ -2608,9 +2608,9 @@ static void __init rcu_organize_nocb_kthreads(struct rcu_state *rsp)/* Prevent __call_rcu() from enqueuing callbacks on no-CBs CPUs */ static bool init_nocb_callback_list (struct rcu_data *rdp) { -if (rcu_nocb_mask ==NULL || - !cpumask_test_cpu (rdp->cpu, rcu_nocb_mask)) +if (!rcu_is_nocb_cpu (rdp->cpu))return false; + rdp->nxttail[RCU_NEXT_TAIL ] =NULL ;return true; }
Signature · 1 line
1.8.1.5
PATCH 10/14 rcu: Eliminate redundant rcu_sysidle_state variable
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Now that we have rcu_state_p, which references rcu_preempt_state for TREE_PREEMPT_RCU and rcu_sched_state for TREE_RCU, we don't need a separate rcu_sysidle_state variable. This commit therefore eliminates rcu_preempt_state in favor of rcu_state_p.
--- kernel/rcu/tree_plugin.h | 23 ++++++----------------- 1 file changed, 6 insertions(+), 17 deletions(-)
kernel/rcu/tree_plugin.h +6 -17
@@ -2690,16 +2690,6 @@ static void __maybe_unused rcu_kick_nohz_cpu(int cpu)#ifdef CONFIG_NO_HZ_FULL_SYSIDLE -/* -* Define RCU flavor that holds sysidle state. This needs to be the -* most active flavor of RCU. -*/ -#ifdef CONFIG_PREEMPT_RCU -static struct rcu_state *rcu_sysidle_state = &rcu_preempt_state; -#else /* #ifdef CONFIG_PREEMPT_RCU */ -static struct rcu_state *rcu_sysidle_state = &rcu_sched_state; -#endif /* #else #ifdef CONFIG_PREEMPT_RCU */ -static int full_sysidle_state;/* Current system-idle state. */ #define RCU_SYSIDLE_NOT 0/* Some CPU is not idle. */ #define RCU_SYSIDLE_SHORT 1/* All CPUs idle for brief period. */
@@ -2841,7 +2831,7 @@ static void rcu_sysidle_check_cpu(struct rcu_data *rdp, bool *isidle,* not the flavor of RCU that tracks sysidle state, or if this * is an offline or the timekeeping CPU, nothing to do. */ -if (!*isidle || rdp->rsp != rcu_sysidle_state || +if (!*isidle || rdp->rsp != rcu_state_p ||cpu_is_offline (rdp->cpu) || rdp->cpu == tick_do_timer_cpu)return ;if (rcu_gp_in_progress (rdp->rsp))
@@ -2867,7 +2857,7 @@ static void rcu_sysidle_check_cpu(struct rcu_data *rdp, bool *isidle,*/ static bool is_sysidle_rcu_state (struct rcu_state *rsp) { -return rsp == rcu_sysidle_state; +return rsp == rcu_state_p; }/*
@@ -2945,7 +2935,7 @@ static void rcu_sysidle_cancel(void)static void rcu_sysidle_report (struct rcu_state *rsp,int isidle,unsigned long maxj,bool gpkt) { -if (rsp != rcu_sysidle_state) +if (rsp != rcu_state_p)return ;/* Wrong flavor, ignore. */ if (gpkt && nr_cpu_ids <=CONFIG_NO_HZ_FULL_SYSIDLE_SMALL )return ;/* Running state machine from timekeeping CPU. */
@@ -3014,13 +3004,12 @@ bool rcu_sys_is_idle(void)/* Scan all the CPUs looking for nonidle CPUs. */ for_each_possible_cpu (cpu) { - rdp =per_cpu_ptr (rcu_sysidle_state->rda, cpu); + rdp =per_cpu_ptr (rcu_state_p->rda, cpu);rcu_sysidle_check_cpu (rdp, &isidle, &maxj);if (!isidle)break ; } -rcu_sysidle_report (rcu_sysidle_state, - isidle, maxj, false); +rcu_sysidle_report (rcu_state_p, isidle, maxj, false); oldrss = rss; rss =ACCESS_ONCE (full_sysidle_state); }
@@ -3047,7 +3036,7 @@ bool rcu_sys_is_idle(void)* provided by the memory allocator. */ if (nr_cpu_ids >CONFIG_NO_HZ_FULL_SYSIDLE_SMALL && - !rcu_gp_in_progress (rcu_sysidle_state) && + !rcu_gp_in_progress (rcu_state_p) && !rsh.inuse &&xchg (&rsh.inuse,1 ) ==0 )call_rcu (&rsh.rh, rcu_sysidle_cb);return false;
Signature · 1 line
1.8.1.5
PATCH 11/14 rcu: Don't track sysidle state if no nohz_full= CPUs
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
If there are no nohz_full= CPUs, then there is currently no reason to track sysidle state. This commit therefore short-circuits this state tracking if !tick_nohz_full_enabled().
Note that these checks will need to be revisited if nohz_full= state can ever be changed at runtime.
--- kernel/rcu/tree_plugin.h | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +18 -1
@@ -2707,6 +2707,10 @@ static void rcu_sysidle_enter(struct rcu_dynticks *rdtp, int irq) {unsigned long j; +/* If there are no nohz_full= CPUs, no need to track this. */ +if (!tick_nohz_full_enabled ()) +return ; +/* Adjust nesting, check for fully idle. */ if (irq) { rdtp->dynticks_idle_nesting--;
@@ -2772,6 +2776,10 @@ void rcu_sysidle_force_exit(void)*/ static void rcu_sysidle_exit (struct rcu_dynticks *rdtp,int irq) { +/* If there are no nohz_full= CPUs, no need to track this. */ +if (!tick_nohz_full_enabled ()) +return ; +/* Adjust nesting, check for already non-idle. */ if (irq) { rdtp->dynticks_idle_nesting++;
@@ -2826,6 +2834,10 @@ static void rcu_sysidle_check_cpu(struct rcu_data *rdp, bool *isidle,unsigned long j;struct rcu_dynticks *rdtp = rdp->dynticks; +/* If there are no nohz_full= CPUs, don't check system-wide idleness. */ +if (!tick_nohz_full_enabled ()) +return ; +/* * If some other CPU has already reported non-idle, if this is * not the flavor of RCU that tracks sysidle state, or if this
@@ -2952,6 +2964,10 @@ static void rcu_sysidle_report(struct rcu_state *rsp, int isidle,static void rcu_sysidle_report_gp (struct rcu_state *rsp,int isidle,unsigned long maxj) { +/* If there are no nohz_full= CPUs, no need to track this. */ +if (!tick_nohz_full_enabled ()) +return ; +rcu_sysidle_report (rsp, isidle, maxj, true); }
@@ -2978,7 +2994,8 @@ static void rcu_sysidle_cb(struct rcu_head *rhp)/* * Check to see if the system is fully idle, other than the timekeeping CPU. -* The caller must have disabled interrupts. +* The caller must have disabled interrupts. This is not intended to be +* called unless tick_nohz_full_enabled(). */ bool rcu_sys_is_idle (void ) {
Signature · 1 line
1.8.1.5
PATCH 12/14 rcu: Avoid misordering in __call_rcu_nocb_enqueue()
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The NOCB leader wakeup ordering depends on the store to the header happening before the check for the leader already being awake. However, because atomic_long_add() does not return a value, it does not provide ordering guarantees, the incorrect comment in wake_nocb_leader() notwithstanding. This commit therefore adds a smp_mb__after_atomic() after the final atomic_long_add() to provide the needed ordering guarantee.
--- kernel/rcu/tree_plugin.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +2 -1
@@ -2042,7 +2042,7 @@ static void wake_nocb_leader(struct rcu_data *rdp, bool force)if (!ACCESS_ONCE (rdp_leader->nocb_kthread))return ;if (!ACCESS_ONCE (rdp_leader->nocb_leader_wake) || force) { -/* Prior xchg orders against prior callback enqueue. */ +/* Prior smp_mb__after_atomic() orders against prior enqueue. */ ACCESS_ONCE (rdp_leader->nocb_leader_wake) = true;wake_up (&rdp_leader->nocb_wq); }
@@ -2071,6 +2071,7 @@ static void __call_rcu_nocb_enqueue(struct rcu_data *rdp,ACCESS_ONCE (*old_rhpp) = rhp;atomic_long_add (rhcount, &rdp->nocb_q_count);atomic_long_add (rhcount_lazy, &rdp->nocb_q_count_lazy); +smp_mb__after_atomic ();/* Store *old_rhpp before _wake test. */ /* If we are not being polled and there is a kthread, awaken it ... */ t =ACCESS_ONCE (rdp->nocb_kthread);
Signature · 1 line
1.8.1.5
PATCH 13/14 rcu: Handle NOCB callbacks from irq-disabled idle code
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
If an RCU callback is queued on a no-CBs CPU from idle code with irqs disabled, and if that CPU stays idle forever after, the callback will never be invoked. This commit therefore adds a check for this situation in ____call_rcu_nocb(), invoking the RCU core solely for the purpose of the ensuing return-to-idle transition. (If the CPU doesn't return to idle, the next scheduling-clock interrupt will fix things up.)
--- kernel/rcu/tree_plugin.h | 11 +++++++++++ 1 file changed, 11 insertions(+)
kernel/rcu/tree_plugin.h +11 -0
@@ -2129,6 +2129,17 @@ static bool __call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *rhp,trace_rcu_callback (rdp->rsp->name, rhp, -atomic_long_read (&rdp->nocb_q_count_lazy), -atomic_long_read (&rdp->nocb_q_count)); + +/* +* If called from an extended quiescent state with interrupts +* disabled, invoke the RCU core in order to allow the idle-entry +* deferred-wakeup check to function. +*/ +if (irqs_disabled_flags (flags) && + !rcu_is_watching () && +cpu_online (smp_processor_id ())) +invoke_rcu_core (); +return true; }
Signature · 1 line
1.8.1.5
PATCH 14/14 rcu: Avoid misordering in nocb_leader_wait()
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The NOCB follower wakeup ordering depends on the store to the tail pointer happening before the wakeup. However, because atomic_long_add() does not return a value, it does not provide ordering guarantees, and the locking in wake_up() only guarantees that the store will happen before the unlock, which might be too late. Even though this is only a theoretical issue, this commit adds a smp_mb__after_atomic() after the final atomic_long_add() to provide the needed ordering guarantee.
--- kernel/rcu/tree_plugin.h | 1 + 1 file changed, 1 insertion(+)
kernel/rcu/tree_plugin.h +1 -0
@@ -2306,6 +2306,7 @@ wait_again:atomic_long_add (rdp->nocb_gp_count, &rdp->nocb_follower_count);atomic_long_add (rdp->nocb_gp_count_lazy, &rdp->nocb_follower_count_lazy); +smp_mb__after_atomic ();/* Store *tail before wakeup. */ if (rdp != my_rdp && tail == &rdp->nocb_follower_head) {/* * List was empty, wake up the follower.
Signature · 1 line
1.8.1.5