Expedited-grace-period changes for 4.5
Subsystems (4)
v2 cover Expedited-grace-period changes for 4.5
Hello!
This series contains follow-on cleanup changes to RCU's expedited grace-period functionality. The patches in this series are as follows:
1. Short-circuit synchronize_sched_expedited() if there is only one CPU.
2. Update comment to clarify the role of ->expmaskinitnext.
3. Separate concerns by moving smp_mb() from rcu_seq_snap() to
rcu_exp_gp_seq_snap().
4. Safe a few lines of code by inverting the sense of
sync_rcu_exp_select_cpus() "if" statement.
5. Reduce expedited GP memory contention via per-CPU variables.
6. Get rid of (some) empty stall-warning messages by resolving
stall-warning ties.
7. Add more diagnostics to expedited stall warning messages.
8. Add rcu_normal kernel parameter to suppress expediting for
the benefit of aggressive real-time systems.
9-10. Allow expedited grace periods to be disabled just before init
is spawned, allowing them to speed up boot without interfering with run-time real-time workloads.
11. Remove TINY_RCU bloat from pointless (to TINY_RCU) boot parameters.
Changes since v1:
o Add patch 11 removing TINY_RCU bloat.
o Get cover letter and patches correctly associated. :-/
Thanx, Paul
------------------------------------------------------------------------
Documentation/kernel-parameters.txt | 37 ++++++++++++---- include/linux/rcupdate.h | 15 ++++++ init/main.c | 2 kernel/ksysfs.c | 26 ++++++++++- kernel/rcu/srcu.c | 2 kernel/rcu/tree.c | 79 +++++++++++++++++++++++------------- kernel/rcu/tree.h | 10 ++-- kernel/rcu/tree_plugin.h | 6 ++ kernel/rcu/tree_trace.c | 18 +++++--- kernel/rcu/update.c | 24 +++++++++- 10 files changed, 165 insertions(+), 54 deletions(-)
PATCH v2 1/11 rcu: Short-circuit synchronize_sched_expedited() if only one CPU
If there is only one CPU, then invoking synchronize_sched_expedited() is by definition a grace period. This commit checks for this condition and does a short-circuit return in that case.
--- kernel/rcu/tree.c | 4 ++++ 1 file changed, 4 insertions(+)
kernel/rcu/tree.c +4 -0
@@ -3822,6 +3822,10 @@ void synchronize_sched_expedited(void)struct rcu_node *rnp;struct rcu_state *rsp = &rcu_sched_state; +/* If only one CPU, this is automatically a grace period. */ +if (rcu_blocking_is_gp ()) +return ; +/* Take a snapshot of the sequence number. */ s =rcu_exp_gp_seq_snap (rsp);
Signature · 1 line
2.5.2
PATCH v2 2/11 rcu: Clarify role of ->expmaskinitnext
In reply to Paul E. McKenneyAnalogy with the ->qsmaskinitnext field might lead one to believe that ->expmaskinitnext tracks online CPUs. This belief is incorrect: Any CPU that has ever been online will have its bit set in the ->expmaskinitnext field. This commit therefore adds a comment to make this clear, at least to people who read comments.
--- kernel/rcu/tree.h | 2 ++ 1 file changed, 2 insertions(+)
kernel/rcu/tree.h +2 -0
@@ -178,6 +178,8 @@ struct rcu_node {/* beginning of each expedited GP. */ unsigned long expmaskinitnext;/* Online CPUs for next expedited GP. */ +/* Any CPU that has ever been online will */ +/* have its bit set. */ unsigned long grpmask;/* Mask to apply to parent qsmask. */ /* Only one bit will be set in this mask. */ int grplo;/* lowest-numbered CPU or group here. */
Signature · 1 line
2.5.2
PATCH v2 3/11 rcu: Move smp_mb() from rcu_seq_snap() to rcu_exp_gp_seq_snap()
In reply to Paul E. McKenneyThe memory barrier in rcu_seq_snap() is needed only for grace periods, so this commit moves it to the grace-period-oriented wrapper rcu_exp_gp_seq_snap().
--- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
@@ -3347,7 +3347,6 @@ static unsigned long rcu_seq_snap(unsigned long *sp) {unsigned long s; -smp_mb ();/* Caller's modifications seen first by other CPUs. */ s = (READ_ONCE (*sp) +3 ) & ~0x1 ;smp_mb ();/* Above access must not bleed into critical section. */ return s;
@@ -3374,6 +3373,7 @@ static void rcu_exp_gp_seq_end(struct rcu_state *rsp) }static unsigned long rcu_exp_gp_seq_snap (struct rcu_state *rsp) { +smp_mb ();/* Caller's modifications seen first by other CPUs. */ return rcu_seq_snap (&rsp->expedited_sequence); }static bool rcu_exp_gp_seq_done (struct rcu_state *rsp,unsigned long s)
Signature · 1 line
2.5.2
Re: [PATCH v2 tip/core/rcu 03/11] rcu: Move smp_mb() from rcu_seq_snap() to rcu_exp_gp_seq_snap()
Quoted 3 lines · Paul E. McKenney
The memory barrier in
rcu_seq_snap()is needed only for grace periods, so this commit moves it to the grace-period-oriented wrapperrcu_exp_gp_seq_snap().
This is a bit short for a memory barrier changelog. I think I remember enough of the code to see this is indeed so, but who knows who will remember what in another few weeks :-)
Re: [PATCH v2 tip/core/rcu 03/11] rcu: Move smp_mb() from rcu_seq_snap() to rcu_exp_gp_seq_snap()
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Paul E. McKenney
The memory barrier in
rcu_seq_snap()is needed only for grace periods, so this commit moves it to the grace-period-oriented wrapperrcu_exp_gp_seq_snap().This is a bit short for a memory barrier changelog. I think I remember enough of the code to see this is indeed so, but who knows who will remember what in another few weeks :-)
;-)
Would it help to add something like the following?
This change does not add or remove a memory barrier, but instead associates it with the right level of abstraction.
Thanx, Paul
PATCH v2 4/11 rcu: Invert sync_rcu_exp_select_cpus() "if" statement
In reply to Paul E. McKenneyThis commit saves a couple lines of code and reduces indentation by inverting the sense of an "if" statement in the function sync_rcu_exp_select_cpus().
--- kernel/rcu/tree.c | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-)
kernel/rcu/tree.c +14 -16
@@ -3716,24 +3716,22 @@ retry_ipi: ret =smp_call_function_single (cpu, func, rsp,0 );if (!ret) { mask_ofl_ipi &= ~mask; - }else { -/* Failed, raced with offline. */ -raw_spin_lock_irqsave_rcu_node (rnp, flags); -if (cpu_online (cpu) && - (rnp->expmask & mask)) { -raw_spin_unlock_irqrestore (&rnp->lock, - flags); -schedule_timeout_uninterruptible (1 ); -if (cpu_online (cpu) && - (rnp->expmask & mask)) - goto retry_ipi; -raw_spin_lock_irqsave_rcu_node (rnp, - flags); - } -if (!(rnp->expmask & mask)) - mask_ofl_ipi &= ~mask; +continue ; + } +/* Failed, raced with offline. */ +raw_spin_lock_irqsave_rcu_node (rnp, flags); +if (cpu_online (cpu) && + (rnp->expmask & mask)) {raw_spin_unlock_irqrestore (&rnp->lock, flags); +schedule_timeout_uninterruptible (1 ); +if (cpu_online (cpu) && + (rnp->expmask & mask)) + goto retry_ipi; +raw_spin_lock_irqsave_rcu_node (rnp, flags); } +if (!(rnp->expmask & mask)) + mask_ofl_ipi &= ~mask; +raw_spin_unlock_irqrestore (&rnp->lock, flags); }/* Report quiescent states for those that went offline. */ mask_ofl_test |= mask_ofl_ipi;
Signature · 1 line
2.5.2
PATCH v2 5/11 rcu: Reduce expedited GP memory contention via per-CPU variables
In reply to Paul E. McKenneyCurrently, the piggybacked-work checks carried out by sync_exp_work_done() atomically increment a small set of variables (the ->expedited_workdone0, ->expedited_workdone1, ->expedited_workdone2, ->expedited_workdone3 fields in the rcu_state structure), which will form a memory-contention bottleneck given a sufficiently large number of CPUs concurrently invoking either synchronize_rcu_expedited() or synchronize_sched_expedited().
This commit therefore moves these for fields to the per-CPU rcu_data structure, eliminating the memory contention. The show_rcuexp() function also changes to sum up each field in the rcu_data structures.
--- kernel/rcu/tree.c | 11 +++++------ kernel/rcu/tree.h | 8 ++++---- kernel/rcu/tree_trace.c | 18 ++++++++++++------ 3 files changed, 21 insertions(+), 16 deletions(-)
kernel/rcu/tree.c +5 -6
@@ -3585,7 +3585,7 @@ static bool sync_exp_work_done(struct rcu_state *rsp, struct rcu_node *rnp,*/ static struct rcu_node *exp_funnel_lock (struct rcu_state *rsp,unsigned long s) { -struct rcu_data *rdp; +struct rcu_data *rdp =per_cpu_ptr (rsp->rda,raw_smp_processor_id ());struct rcu_node *rnp0;struct rcu_node *rnp1 =NULL ;
@@ -3599,7 +3599,7 @@ static struct rcu_node *exp_funnel_lock(struct rcu_state *rsp, unsigned long s)if (!mutex_is_locked (&rnp0->exp_funnel_mutex)) {if (mutex_trylock (&rnp0->exp_funnel_mutex)) {if (sync_exp_work_done (rsp, rnp0,NULL , - &rsp->expedited_workdone0, s)) + &rdp->expedited_workdone0, s))return NULL ;return rnp0; }
@@ -3613,14 +3613,13 @@ static struct rcu_node *exp_funnel_lock(struct rcu_state *rsp, unsigned long s)* can be inexact, as it is just promoting locality and is not * strictly needed for correctness. */ - rdp =per_cpu_ptr (rsp->rda,raw_smp_processor_id ()); -if (sync_exp_work_done (rsp,NULL ,NULL , &rsp->expedited_workdone1, s)) +if (sync_exp_work_done (rsp,NULL ,NULL , &rdp->expedited_workdone1, s))return NULL ;mutex_lock (&rdp->exp_funnel_mutex); rnp0 = rdp->mynode;for (; rnp0 !=NULL ; rnp0 = rnp0->parent) {if (sync_exp_work_done (rsp, rnp1, rdp, - &rsp->expedited_workdone2, s)) + &rdp->expedited_workdone2, s))return NULL ;mutex_lock (&rnp0->exp_funnel_mutex);if (rnp1)
@@ -3630,7 +3629,7 @@ static struct rcu_node *exp_funnel_lock(struct rcu_state *rsp, unsigned long s) rnp1 = rnp0; }if (sync_exp_work_done (rsp, rnp1, rdp, - &rsp->expedited_workdone3, s)) + &rdp->expedited_workdone3, s))return NULL ;return rnp1; }
kernel/rcu/tree.h +4 -4
@@ -386,6 +386,10 @@ struct rcu_data {struct rcu_head oom_head;#endif /* #ifdef CONFIG_RCU_FAST_NO_HZ */ struct mutex exp_funnel_mutex; +atomic_long_t expedited_workdone0;/* # done by others #0. */ +atomic_long_t expedited_workdone1;/* # done by others #1. */ +atomic_long_t expedited_workdone2;/* # done by others #2. */ +atomic_long_t expedited_workdone3;/* # done by others #3. */ /* 7) Callback offloading. */ #ifdef CONFIG_RCU_NOCB_CPU
@@ -500,10 +504,6 @@ struct rcu_state {/* End of fields guarded by barrier_mutex. */ unsigned long expedited_sequence;/* Take a ticket. */ -atomic_long_t expedited_workdone0;/* # done by others #0. */ -atomic_long_t expedited_workdone1;/* # done by others #1. */ -atomic_long_t expedited_workdone2;/* # done by others #2. */ -atomic_long_t expedited_workdone3;/* # done by others #3. */ atomic_long_t expedited_normal;/* # fallbacks to normal. */ atomic_t expedited_need_qs;/* # CPUs left to check in. */ wait_queue_head_t expedited_wq;/* Wait for check-ins. */
kernel/rcu/tree_trace.c +12 -6
@@ -183,14 +183,20 @@ static const struct file_operations rcudata_fops = {static int show_rcuexp (struct seq_file *m,void *v) { +int cpu;struct rcu_state *rsp = (struct rcu_state *)m->private; - +struct rcu_data *rdp; +unsigned long s0 =0 , s1 =0 , s2 =0 , s3 =0 ; + +for_each_possible_cpu (cpu) { + rdp =per_cpu_ptr (rsp->rda, cpu); + s0 +=atomic_long_read (&rdp->expedited_workdone0); + s1 +=atomic_long_read (&rdp->expedited_workdone1); + s2 +=atomic_long_read (&rdp->expedited_workdone2); + s3 +=atomic_long_read (&rdp->expedited_workdone3); + }seq_printf (m,"s=%lu wd0=%lu wd1=%lu wd2=%lu wd3=%lu n=%lu enq=%d sc=%lu\n" , - rsp->expedited_sequence, -atomic_long_read (&rsp->expedited_workdone0), -atomic_long_read (&rsp->expedited_workdone1), -atomic_long_read (&rsp->expedited_workdone2), -atomic_long_read (&rsp->expedited_workdone3), + rsp->expedited_sequence, s0, s1, s2, s3,atomic_long_read (&rsp->expedited_normal),atomic_read (&rsp->expedited_need_qs), rsp->expedited_sequence /2 );
Signature · 1 line
2.5.2
PATCH v2 6/11 rcu: Make expedited grace periods resolve stall-warning ties
In reply to Paul E. McKenneyCurrently, if a grace period ends just as the stall-warning timeout fires, an empty stall warning will be printed. This is not helpful, so this commit avoids these useless warnings by rechecking completion after awakening in synchronize_sched_expedited_wait().
--- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
@@ -3757,7 +3757,7 @@ static void synchronize_sched_expedited_wait(struct rcu_state *rsp) rsp->expedited_wq,sync_rcu_preempt_exp_done (rnp_root), jiffies_stall); -if (ret >0 ) +if (ret >0 ||sync_rcu_preempt_exp_done (rnp_root))return ;if (ret <0 ) {/* Hit a signal, disable CPU stall warnings. */
Signature · 1 line
2.5.2
PATCH v2 7/11 rcu: Add more diagnostics to expedited stall warning messages.
In reply to Paul E. McKenneyThis commit adds print statements that check the rcu_node structure to find which ->expmask bits and which ->exp_tasks structures are blocking the current expedited grace period.
--- kernel/rcu/tree.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
kernel/rcu/tree.c +21 -3
@@ -3745,6 +3745,7 @@ static void synchronize_sched_expedited_wait(struct rcu_state *rsp)unsigned long jiffies_stall;unsigned long jiffies_start;unsigned long mask; +int ndetected;struct rcu_node *rnp;struct rcu_node *rnp_root =rcu_get_root (rsp);int ret;
@@ -3767,14 +3768,16 @@ static void synchronize_sched_expedited_wait(struct rcu_state *rsp) }pr_err ("INFO: %s detected expedited stalls on CPUs/tasks: {" , rsp->name); + ndetected =0 ;rcu_for_each_leaf_node (rsp, rnp) { - (void )rcu_print_task_exp_stall (rnp); + ndetected =rcu_print_task_exp_stall (rnp); mask =1 ;for (cpu = rnp->grplo; cpu <= rnp->grphi; cpu++, mask <<=1 ) {struct rcu_data *rdp;if (!(rnp->expmask & mask))continue ; + ndetected++; rdp =per_cpu_ptr (rsp->rda, cpu); pr_cont(" %d-%c%c%c", cpu, "O."[cpu_online(cpu)],
@@ -3783,8 +3786,23 @@ static void synchronize_sched_expedited_wait(struct rcu_state *rsp) } mask <<=1 ; } -pr_cont (" } %lu jiffies s: %lu\n" , - jiffies - jiffies_start, rsp->expedited_sequence); +pr_cont (" } %lu jiffies s: %lu root: %#lx/%c\n" , + jiffies - jiffies_start, rsp->expedited_sequence, + rnp_root->expmask,".T" [!!rnp_root->exp_tasks]); +if (!ndetected) { +pr_err ("blocking rcu_node structures:" ); +rcu_for_each_node_breadth_first (rsp, rnp) { +if (rnp == rnp_root) +continue ;/* printed unconditionally */ +if (sync_rcu_preempt_exp_done (rnp)) +continue ; +pr_cont (" l=%u:%d-%d:%#lx/%c" , + rnp->level, rnp->grplo, rnp->grphi, + rnp->expmask, +".T" [!!rnp->exp_tasks]); + } +pr_cont ("\n" ); + }rcu_for_each_leaf_node (rsp, rnp) { mask =1 ;for (cpu = rnp->grplo; cpu <= rnp->grphi; cpu++, mask <<=1 ) {
Signature · 1 line
2.5.2
PATCH v2 8/11 rcu: Add rcu_normal kernel parameter to suppress expediting
In reply to Paul E. McKenneyAlthough expedited grace periods can be quite useful, and although their OS jitter has been greatly reduced, they can still pose problems for extreme real-time workloads. This commit therefore adds a rcu_normal kernel boot parameter (which can also be manipulated via sysfs) to suppress expedited grace periods, that is, to treat requests for expedited grace periods as if they were requests for normal grace periods. If both rcu_expedited and rcu_normal are specified, rcu_normal wins. This means that if you are relying on expedited grace periods to speed up boot, you will want to specify rcu_expedited on the kernel command line, and then specify rcu_normal via sysfs once boot completes.
--- Documentation/kernel-parameters.txt | 19 ++++++++++++++----- include/linux/rcupdate.h | 6 ++++++ kernel/ksysfs.c | 22 ++++++++++++++++++++-- kernel/rcu/srcu.c | 2 +- kernel/rcu/tree.c | 6 ++++++ kernel/rcu/tree_plugin.h | 6 ++++++ kernel/rcu/update.c | 12 ++++++++++++ 7 files changed, 65 insertions(+), 8 deletions(-)
Documentation/kernel-parameters.txt +14 -5
@@ -3296,6 +3296,12 @@ bytes respectively. Such letter suffixes can also be entirely omitted. rcutorture.verbose= [KNL] Enable additional printk() statements. + rcupdate.rcu_cpu_stall_suppress= [KNL] + Suppress RCU CPU stall warning messages. + + rcupdate.rcu_cpu_stall_timeout= [KNL] + Set timeout for RCU CPU stall warning messages. + rcupdate.rcu_expedited= [KNL] Use expedited grace-period primitives, for example, synchronize_rcu_expedited() instead
@@ -3303,11 +3309,14 @@ bytes respectively. Such letter suffixes can also be entirely omitted. but can increase CPU utilization, degrade real-time latency, and degrade energy efficiency. - rcupdate.rcu_cpu_stall_suppress= [KNL] - Suppress RCU CPU stall warning messages. - - rcupdate.rcu_cpu_stall_timeout= [KNL] - Set timeout for RCU CPU stall warning messages. + rcupdate.rcu_normal= [KNL] + Use only normal grace-period primitives, + for example, synchronize_rcu() instead of + synchronize_rcu_expedited(). This improves + real-time latency, CPU utilization, and energy + efficiency, but can expose users to increased + grace-period latency. This parameter overrides + rcupdate.rcu_expedited. rcupdate.rcu_task_stall_timeout= [KNL] Set timeout in jiffies for RCU task stall warning
include/linux/rcupdate.h +6 -0
@@ -49,9 +49,14 @@#include <asm/barrier.h> extern int rcu_expedited;/* for sysctl */ +extern int rcu_normal;/* also for sysctl */ #ifdef CONFIG_TINY_RCU /* Tiny RCU doesn't expedite, as its purpose in life is instead to be tiny. */ +static inline bool rcu_gp_is_normal (void )/* Internal RCU use. */ +{ +return true; +}static inline bool rcu_gp_is_expedited (void )/* Internal RCU use. */ {return false;
@@ -65,6 +70,7 @@ static inline void rcu_unexpedite_gp(void) { }#else /* #ifdef CONFIG_TINY_RCU */ +bool rcu_gp_is_normal (void );/* Internal RCU use. */ bool rcu_gp_is_expedited (void );/* Internal RCU use. */ void rcu_expedite_gp (void );void rcu_unexpedite_gp (void );
kernel/ksysfs.c +20 -2
@@ -20,7 +20,7 @@#include <linux/capability.h> #include <linux/compiler.h> -#include <linux/rcupdate.h> /* rcu_expedited */ +#include <linux/rcupdate.h> /* rcu_expedited and rcu_normal */ #define KERNEL_ATTR_RO (_name) \ static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
@@ -148,7 +148,7 @@ int rcu_expedited;static ssize_t rcu_expedited_show (struct kobject *kobj,struct kobj_attribute *attr,char *buf) { -return sprintf (buf,"%d\n" , rcu_expedited); +return sprintf (buf,"%d\n" ,READ_ONCE (rcu_expedited)); } static ssize_t rcu_expedited_store(struct kobject *kobj, struct kobj_attribute *attr,
@@ -161,6 +161,23 @@ static ssize_t rcu_expedited_store(struct kobject *kobj, }KERNEL_ATTR_RW (rcu_expedited); +int rcu_normal; +static ssize_t rcu_normal_show (struct kobject *kobj, +struct kobj_attribute *attr,char *buf) +{ +return sprintf (buf,"%d\n" ,READ_ONCE (rcu_normal)); +} +static ssize_t rcu_normal_store (struct kobject *kobj, +struct kobj_attribute *attr, +const char *buf,size_t count) +{ +if (kstrtoint (buf,0 , &rcu_normal)) +return -EINVAL ; + +return count; +} +KERNEL_ATTR_RW (rcu_normal); +/* * Make /sys/kernel/notes give the raw contents of our kernel .notes section. */
@@ -203,6 +220,7 @@ static struct attribute * kernel_attrs[] = { &vmcoreinfo_attr.attr, #endif &rcu_expedited_attr.attr, + &rcu_normal_attr.attr,NULL };
kernel/rcu/srcu.c +1 -1
@@ -489,7 +489,7 @@ static void __synchronize_srcu(struct srcu_struct *sp, int trycount)*/ void synchronize_srcu (struct srcu_struct *sp) { -__synchronize_srcu (sp,rcu_gp_is_expedited () +__synchronize_srcu (sp, (rcu_gp_is_expedited () && !rcu_gp_is_normal ()) ?SYNCHRONIZE_SRCU_EXP_TRYCOUNT :SYNCHRONIZE_SRCU_TRYCOUNT ); }
kernel/rcu/tree.c +6 -0
@@ -3841,6 +3841,12 @@ void synchronize_sched_expedited(void)if (rcu_blocking_is_gp ())return ; +/* If expedited grace periods are prohibited, fall back to normal. */ +if (rcu_gp_is_normal ()) { +wait_rcu_gp (call_rcu_sched); +return ; + } +/* Take a snapshot of the sequence number. */ s =rcu_exp_gp_seq_snap (rsp);
kernel/rcu/tree_plugin.h +6 -0
@@ -746,6 +746,12 @@ void synchronize_rcu_expedited(void)struct rcu_state *rsp = rcu_state_p;unsigned long s; +/* If expedited grace periods are prohibited, fall back to normal. */ +if (rcu_gp_is_normal ()) { +wait_rcu_gp (call_rcu); +return ; + } + s =rcu_exp_gp_seq_snap (rsp); rnp_unlock =exp_funnel_lock (rsp, s);
kernel/rcu/update.c +12 -0
@@ -61,6 +61,7 @@ MODULE_ALIAS("rcupdate");#define MODULE_PARAM_PREFIX "rcupdate."module_param (rcu_expedited, int,0 ); +module_param (rcu_normal, int,0 );#if defined(CONFIG_DEBUG_LOCK_ALLOC ) && defined(CONFIG_PREEMPT_COUNT )/**
@@ -113,6 +114,17 @@ EXPORT_SYMBOL(rcu_read_lock_sched_held);#ifndef CONFIG_TINY_RCU +/* +* Should expedited grace-period primitives always fall back to their +* non-expedited counterparts? Intended for use within RCU. Note +* that if the user specifies both rcu_expedited and rcu_normal, then +* rcu_normal wins. +*/ +bool rcu_gp_is_normal (void ) +{ +return READ_ONCE (rcu_normal); +} +static atomic_t rcu_expedited_nesting =ATOMIC_INIT (IS_ENABLED (CONFIG_RCU_EXPEDITE_BOOT ) ?1 :0 );
Signature · 1 line
2.5.2
PATCH v2 9/11 rcu: Wire up rcu_end_inkernel_boot()
In reply to Paul E. McKenneyThis commit adds the invocation of rcu_end_inkernel_boot() just before init is invoked. This allows the CONFIG_RCU_EXPEDITE_BOOT Kconfig option to do something useful and prepares for the upcoming rcupdate.rcu_normal_after_boot kernel parameter.
--- init/main.c | 2 ++ 1 file changed, 2 insertions(+)
init/main.c +2 -0
@@ -943,6 +943,8 @@ static int __ref kernel_init(void *unused)flush_delayed_fput (); +rcu_end_inkernel_boot (); +if (ramdisk_execute_command) { ret =run_init_process (ramdisk_execute_command);if (!ret)
Signature · 1 line
2.5.2
PATCH v2 10/11 rcu: Allow expedited grace periods to be disabled at init
In reply to Paul E. McKenneyExpedited grace periods can speed up boot, but are undesirable in aggressive real-time systems. This commit therefore introduces a kernel parameter rcupdate.rcu_normal_after_boot that disables expedited grace periods just before init is spawned.
--- Documentation/kernel-parameters.txt | 5 +++++ kernel/rcu/update.c | 5 +++++ 2 files changed, 10 insertions(+)
Documentation/kernel-parameters.txt +5 -0
@@ -3318,6 +3318,11 @@ bytes respectively. Such letter suffixes can also be entirely omitted. grace-period latency. This parameter overrides rcupdate.rcu_expedited. + rcupdate.rcu_normal_after_boot= [KNL] + Once boot has completed (that is, after + rcu_end_inkernel_boot() has been invoked), use + only normal grace-period primitives. + rcupdate.rcu_task_stall_timeout= [KNL] Set timeout in jiffies for RCU task stall warning messages. Disable with a value less than or equal
kernel/rcu/update.c +5 -0
@@ -63,6 +63,9 @@ MODULE_ALIAS("rcupdate");module_param (rcu_expedited, int,0 );module_param (rcu_normal, int,0 ); +static int rcu_normal_after_boot; +module_param (rcu_normal_after_boot, int,0 ); +#if defined(CONFIG_DEBUG_LOCK_ALLOC ) && defined(CONFIG_PREEMPT_COUNT )/** * rcu_read_lock_sched_held() - might we be in RCU-sched read-side critical section?
@@ -178,6 +181,8 @@ void rcu_end_inkernel_boot(void) {if (IS_ENABLED (CONFIG_RCU_EXPEDITE_BOOT ))rcu_unexpedite_gp (); +if (rcu_normal_after_boot) +WRITE_ONCE (rcu_normal,1 ); }#ifdef CONFIG_PREEMPT_RCU
Signature · 1 line
2.5.2
PATCH v2 11/11 rcu: Remove TINY_RCU bloat from pointless boot parameters
In reply to Paul E. McKenneyThe rcu_expedited, rcu_normal, and rcu_normal_after_boot kernel boot parameters are pointless in the case of TINY_RCU because in that case synchronous grace periods, both expedited and normal, are no-ops. However, these three symbols contribute several hundred bytes of bloat. This commit therefore uses CPP directives to avoid compiling this code in TINY_RCU kernels.
--- Documentation/kernel-parameters.txt | 13 ++++++++----- include/linux/rcupdate.h | 9 ++++++++- kernel/ksysfs.c | 4 ++++ kernel/rcu/update.c | 7 ++++--- 4 files changed, 24 insertions(+), 9 deletions(-)
Documentation/kernel-parameters.txt +8 -5
@@ -3308,20 +3308,23 @@ bytes respectively. Such letter suffixes can also be entirely omitted. of synchronize_rcu(). This reduces latency, but can increase CPU utilization, degrade real-time latency, and degrade energy efficiency. + No effect on CONFIG_TINY_RCU kernels. rcupdate.rcu_normal= [KNL] Use only normal grace-period primitives, for example, synchronize_rcu() instead of synchronize_rcu_expedited(). This improves - real-time latency, CPU utilization, and energy - efficiency, but can expose users to increased - grace-period latency. This parameter overrides - rcupdate.rcu_expedited. + real-time latency, CPU utilization, and + energy efficiency, but can expose users to + increased grace-period latency. This parameter + overrides rcupdate.rcu_expedited. No effect on + CONFIG_TINY_RCU kernels. rcupdate.rcu_normal_after_boot= [KNL] Once boot has completed (that is, after rcu_end_inkernel_boot() has been invoked), use - only normal grace-period primitives. + only normal grace-period primitives. No effect + on CONFIG_TINY_RCU kernels. rcupdate.rcu_task_stall_timeout= [KNL] Set timeout in jiffies for RCU task stall warning
include/linux/rcupdate.h +8 -1
@@ -48,8 +48,10 @@#include <asm/barrier.h> +#ifndef CONFIG_TINY_RCU extern int rcu_expedited;/* for sysctl */ extern int rcu_normal;/* also for sysctl */ +#endif /* #ifndef CONFIG_TINY_RCU */ #ifdef CONFIG_TINY_RCU /* Tiny RCU doesn't expedite, as its purpose in life is instead to be tiny. */
@@ -327,7 +329,6 @@ static inline int rcu_preempt_depth(void)/* Internal to kernel */ void rcu_init (void ); -void rcu_end_inkernel_boot (void );void rcu_sched_qs (void );void rcu_bh_qs (void );void rcu_check_callbacks (int user);
@@ -335,6 +336,12 @@ struct notifier_block;int rcu_cpu_notify (struct notifier_block *self,unsigned long action,void *hcpu); +#ifndef CONFIG_TINY_RCU +void rcu_end_inkernel_boot (void ); +#else /* #ifndef CONFIG_TINY_RCU */ +static inline void rcu_end_inkernel_boot (void ) { } +#endif /* #ifndef CONFIG_TINY_RCU */ +#ifdef CONFIG_RCU_STALL_COMMON void rcu_sysrq_start (void );void rcu_sysrq_end (void );
kernel/ksysfs.c +4 -0
@@ -144,6 +144,7 @@ static ssize_t fscaps_show(struct kobject *kobj, }KERNEL_ATTR_RO (fscaps); +#ifndef CONFIG_TINY_RCU int rcu_expedited;static ssize_t rcu_expedited_show (struct kobject *kobj,struct kobj_attribute *attr,char *buf)
@@ -177,6 +178,7 @@ static ssize_t rcu_normal_store(struct kobject *kobj, return count; }KERNEL_ATTR_RW (rcu_normal); +#endif /* #ifndef CONFIG_TINY_RCU */ /* * Make /sys/kernel/notes give the raw contents of our kernel .notes section.
@@ -219,8 +221,10 @@ static struct attribute * kernel_attrs[] = { &kexec_crash_size_attr.attr, &vmcoreinfo_attr.attr, #endif +#ifndef CONFIG_TINY_RCU &rcu_expedited_attr.attr, &rcu_normal_attr.attr, +#endifNULL };
kernel/rcu/update.c +4 -3
@@ -60,11 +60,12 @@ MODULE_ALIAS("rcupdate");#endif #define MODULE_PARAM_PREFIX "rcupdate." +#ifndef CONFIG_TINY_RCU module_param (rcu_expedited, int,0 );module_param (rcu_normal, int,0 ); -static int rcu_normal_after_boot;module_param (rcu_normal_after_boot, int,0 ); +#endif /* #ifndef CONFIG_TINY_RCU */ #if defined(CONFIG_DEBUG_LOCK_ALLOC ) && defined(CONFIG_PREEMPT_COUNT )/**
@@ -172,8 +173,6 @@ void rcu_unexpedite_gp(void) }EXPORT_SYMBOL_GPL (rcu_unexpedite_gp); -#endif /* #ifndef CONFIG_TINY_RCU */ -/* * Inform RCU of the end of the in-kernel boot sequence. */
@@ -185,6 +184,8 @@ void rcu_end_inkernel_boot(void) WRITE_ONCE(rcu_normal, 1); } +#endif /* #ifndef CONFIG_TINY_RCU */ +#ifdef CONFIG_PREEMPT_RCU /*
Signature · 1 line
2.5.2