RCU flavor consolidation cleanups for v5.1
Subsystems (4)
cover RCU flavor consolidation cleanups for v5.1
Hello!
This series contains additional cleanups for RCU flavor consolidation:
1-2. Replace scheduler _sched() APIs with _rcu() counterparts.
3. Renaming and commenting changes due to only one rcuo kthread
per CPU.
4. Make expedited IPI handler return after handling critical section.
This is a maintainability and performance change.
5. Inline force_quiescent_state() into rcu_force_quiescent_state().
6. Eliminate RCU_BH_FLAVOR and RCU_SCHED_FLAVOR.
7. Inline rcu_kthread_do_work() into its sole remaining caller.
8. Determine expedited-GP IPI handler at build time.
9. Consolidate PREEMPT and !PREEMPT synchronize_rcu_expedited().
10. Consolidate PREEMPT and !PREEMPT synchronize_rcu().
11. Inline _synchronize_rcu_expedited() into synchronize_rcu_expedited().
12. Discard separate per-CPU callback counts.
Thanx, Paul
------------------------------------------------------------------------
Documentation/RCU/Design/Expedited-Grace-Periods/ExpSchedFlow.svg | 18 Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.html | 26 - Documentation/RCU/stallwarn.txt | 15 kernel/rcu/rcu.h | 2 kernel/rcu/tree.c | 123 +++-- kernel/rcu/tree.h | 9 kernel/rcu/tree_exp.h | 244 ++++------ kernel/rcu/tree_plugin.h | 129 ----- kernel/sched/cpufreq.c | 4 kernel/sched/cpufreq_schedutil.c | 2 kernel/sched/sched.h | 2 kernel/sched/topology.c | 4 12 files changed, 237 insertions(+), 341 deletions(-)
PATCH 1/12 sched: Replace call_rcu_sched() with call_rcu()
Now that call_rcu()'s callback is not invoked until after all preempt-disable regions of code have completed (in addition to explicitly marked RCU read-side critical sections), call_rcu() can be used in place of call_rcu_sched(). This commit therefore makes that change.
While in the area, this commit also updates an outdated header comment for for_each_domain().
--- kernel/sched/sched.h | 2 +- kernel/sched/topology.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
kernel/sched/sched.h +1 -1
@@ -1260,7 +1260,7 @@ extern void sched_ttwu_pending(void);/* * The domain tree (rq->sd) is protected by RCU's quiescent state transition. -* See detach_destroy_domains: synchronize_sched for details. +* See destroy_sched_domains: call_rcu for details. * * The domain tree of any CPU may only be accessed from within * preempt-disabled sections.
kernel/sched/topology.c +2 -2
@@ -442,7 +442,7 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)raw_spin_unlock_irqrestore (&rq->lock, flags);if (old_rd) -call_rcu_sched (&old_rd->rcu, free_rootdomain); +call_rcu (&old_rd->rcu, free_rootdomain); }void sched_get_rd (struct root_domain *rd)
@@ -455,7 +455,7 @@ void sched_put_rd(struct root_domain *rd)if (!atomic_dec_and_test (&rd->refcount))return ; -call_rcu_sched (&rd->rcu, free_rootdomain); +call_rcu (&rd->rcu, free_rootdomain); }static int init_rootdomain (struct root_domain *rd)
Signature · 1 line
2.17.1
PATCH 2/12 sched: Replace synchronize_sched() with synchronize_rcu()
In reply to Paul E. McKenneyNow that synchronize_rcu() waits for preempt-disable regions of code as well as RCU read-side critical sections, synchronize_sched() can be replaced by synchronize_rcu(), in fact, synchronize_sched() is now completely equivalent to synchronize_rcu(). This commit therefore replaces synchronize_sched() with synchronize_rcu() so that synchronize_sched() can eventually be removed entirely.
--- kernel/sched/cpufreq.c | 4 ++-- kernel/sched/cpufreq_schedutil.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-)
kernel/sched/cpufreq.c +2 -2
@@ -48,8 +48,8 @@ EXPORT_SYMBOL_GPL(cpufreq_add_update_util_hook);* * Clear the update_util_data pointer for the given CPU. * -* Callers must use RCU-sched callbacks to free any memory that might be -* accessed via the old update_util_data pointer or invoke synchronize_sched() +* Callers must use RCU callbacks to free any memory that might be +* accessed via the old update_util_data pointer or invoke synchronize_rcu() * right after this function to avoid use-after-free. */ void cpufreq_remove_update_util_hook (int cpu)
kernel/sched/cpufreq_schedutil.c +1 -1
@@ -859,7 +859,7 @@ static void sugov_stop(struct cpufreq_policy *policy)for_each_cpu (cpu, policy->cpus)cpufreq_remove_update_util_hook (cpu); -synchronize_sched (); +synchronize_rcu ();if (!policy->fast_switch_enabled) {irq_work_sync (&sg_policy->irq_work);
Signature · 1 line
2.17.1
PATCH 3/12 rcu: Rename and comment changes due to only one rcuo kthread per CPU
In reply to Paul E. McKenneyGiven RCU flavor consolidation, the name rcu_spawn_all_nocb_kthreads() is quite misleading. It no longer ever creates more than one kthread, and it does so only for the specified CPU. This commit therefore changes this name to the more descriptive rcu_spawn_cpu_nocb_kthread(), and also fixes up a similar issue in its header comment while in the area.
--- kernel/rcu/tree.c | 2 +- kernel/rcu/tree.h | 2 +- kernel/rcu/tree_plugin.h | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-)
kernel/rcu/tree.c +1 -1
@@ -3299,7 +3299,7 @@ int rcutree_prepare_cpu(unsigned int cpu)trace_rcu_grace_period (rcu_state.name, rdp->gp_seq,TPS ("cpuonl" ));raw_spin_unlock_irqrestore_rcu_node (rnp, flags);rcu_prepare_kthreads (cpu); -rcu_spawn_all_nocb_kthreads (cpu); +rcu_spawn_cpu_nocb_kthread (cpu);return 0 ; }
kernel/rcu/tree.h +1 -1
@@ -451,7 +451,7 @@ static bool rcu_nocb_adopt_orphan_cbs(struct rcu_data *my_rdp,static int rcu_nocb_need_deferred_wakeup (struct rcu_data *rdp);static void do_nocb_deferred_wakeup (struct rcu_data *rdp);static void rcu_boot_init_nocb_percpu_data (struct rcu_data *rdp); -static void rcu_spawn_all_nocb_kthreads (int cpu); +static void rcu_spawn_cpu_nocb_kthread (int cpu);static void __init rcu_spawn_nocb_kthreads (void );#ifdef CONFIG_RCU_NOCB_CPU static void __init rcu_organize_nocb_kthreads (void );
kernel/rcu/tree_plugin.h +4 -4
@@ -2517,9 +2517,9 @@ static void rcu_spawn_one_nocb_kthread(int cpu)/* * If the specified CPU is a no-CBs CPU that does not already have its -* rcuo kthreads, spawn them. +* rcuo kthread, spawn it. */ -static void rcu_spawn_all_nocb_kthreads (int cpu) +static void rcu_spawn_cpu_nocb_kthread (int cpu) {if (rcu_scheduler_fully_active)rcu_spawn_one_nocb_kthread (cpu);
@@ -2536,7 +2536,7 @@ static void __init rcu_spawn_nocb_kthreads(void)int cpu;for_each_online_cpu (cpu) -rcu_spawn_all_nocb_kthreads (cpu); +rcu_spawn_cpu_nocb_kthread (cpu); }/* How many follower CPU IDs per leader? Default of -1 for sqrt(nr_cpu_ids). */
@@ -2670,7 +2670,7 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp) { } -static void rcu_spawn_all_nocb_kthreads (int cpu) +static void rcu_spawn_cpu_nocb_kthread (int cpu) { }
Signature · 1 line
2.17.1
PATCH 4/12 rcu: Make expedited IPI handler return after handling critical section
In reply to Paul E. McKenneyDuring expedited RCU grace-period initialization, IPIs are sent to all non-idle online CPUs. The IPI handler checks to see if the CPU is in quiescent state, reporting one if so. This handler looks at three different cases: (1) The CPU is not in an rcu_read_lock()-based critical section, (2) The CPU is in the process of exiting an rcu_read_lock()-based critical section, and (3) The CPU is in an rcu_read_lock()-based critical section. In case (2), execution falls through into case (3).
This is harmless from a functionality viewpoint, but can result in needless overhead during an improbable corner case. This commit therefore adds the "return" statement needed to prevent fall-through.
--- kernel/rcu/tree_exp.h | 1 + 1 file changed, 1 insertion(+)
kernel/rcu/tree_exp.h +1 -0
@@ -697,6 +697,7 @@ static void sync_rcu_exp_handler(void *unused)WRITE_ONCE (t->rcu_read_unlock_special.b.exp_hint, true); }raw_spin_unlock_irqrestore_rcu_node (rnp, flags); +return ; }/*
Signature · 1 line
2.17.1
PATCH 5/12 rcu: Inline force_quiescent_state() into rcu_force_quiescent_state()
In reply to Paul E. McKenneyGiven that rcu_force_quiescent_state() is a simple wrapper around force_quiescent_state(), this commit saves a few lines of code by inlining force_quiescent_state() into rcu_force_quiescent_state(), and changing all references to force_quiescent_state() to instead invoke rcu_force_quiescent_state().
--- kernel/rcu/tree.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-)
kernel/rcu/tree.c +6 -15
@@ -479,7 +479,6 @@ module_param_cb(jiffies_till_next_fqs, &next_fqs_jiffies_ops, &jiffies_till_nextmodule_param (rcu_kick_kthreads, bool,0644 );static void force_qs_rnp (int (*f)(struct rcu_data *rdp)); -static void force_quiescent_state (void );static int rcu_pending (void );/*
@@ -503,15 +502,6 @@ unsigned long rcu_exp_batches_completed(void) }EXPORT_SYMBOL_GPL (rcu_exp_batches_completed); -/* -* Force a quiescent state. -*/ -void rcu_force_quiescent_state (void ) -{ -force_quiescent_state (); -} -EXPORT_SYMBOL_GPL (rcu_force_quiescent_state); -/* * Convert a ->gp_state value to a character string. */
@@ -1310,7 +1300,7 @@ static void print_other_cpu_stall(unsigned long gp_seq)panic_on_rcu_stall (); -force_quiescent_state ();/* Kick them all. */ +rcu_force_quiescent_state ();/* Kick them all. */ }static void print_cpu_stall (void )
@@ -2578,7 +2568,7 @@ static void force_qs_rnp(int (*f)(struct rcu_data *rdp))* Force quiescent states on reluctant CPUs, and also detect which * CPUs are in dyntick-idle mode. */ -static void force_quiescent_state (void ) +void rcu_force_quiescent_state (void ) {unsigned long flags;bool ret;
@@ -2610,6 +2600,7 @@ static void force_quiescent_state(void)raw_spin_unlock_irqrestore_rcu_node (rnp_old, flags);rcu_gp_kthread_wake (); } +EXPORT_SYMBOL_GPL (rcu_force_quiescent_state);/* * This function checks for grace-period requests that fail to motivate
@@ -2801,9 +2792,9 @@ static void __call_rcu_core(struct rcu_data *rdp, struct rcu_head *head,/* * Force the grace period if too many callbacks or too long waiting. -* Enforce hysteresis, and don't invoke force_quiescent_state() +* Enforce hysteresis, and don't invoke rcu_force_quiescent_state() * if some other CPU has recently done so. Also, don't bother -* invoking force_quiescent_state() if the newly enqueued callback +* invoking rcu_force_quiescent_state() if the newly enqueued callback * is the only one waiting for a grace period to complete. */ if (unlikely(rcu_segcblist_n_cbs(&rdp->cblist) >
@@ -2820,7 +2811,7 @@ static void __call_rcu_core(struct rcu_data *rdp, struct rcu_head *head, rdp->blimit =LONG_MAX ;if (rcu_state.n_force_qs == rdp->n_force_qs_snap &&rcu_segcblist_first_pend_cb (&rdp->cblist) != head) -force_quiescent_state (); +rcu_force_quiescent_state (); rdp->n_force_qs_snap = rcu_state.n_force_qs; rdp->qlen_last_fqs_check =rcu_segcblist_n_cbs (&rdp->cblist); }
Signature · 1 line
2.17.1
PATCH 6/12 rcu: Eliminate RCU_BH_FLAVOR and RCU_SCHED_FLAVOR
In reply to Paul E. McKenneyNow that the RCU flavors have been consolidated, RCU_BH_FLAVOR and RCU_SCHED_FLAVOR are no longer used. This commit therefore saves a few lines by removing them.
--- kernel/rcu/rcu.h | 2 -- kernel/rcu/tree.c | 2 -- 2 files changed, 4 deletions(-)
kernel/rcu/rcu.h +0 -2
@@ -462,8 +462,6 @@ void rcu_request_urgent_qs_task(struct task_struct *t);enum rcutorture_type {RCU_FLAVOR , -RCU_BH_FLAVOR , -RCU_SCHED_FLAVOR ,RCU_TASKS_FLAVOR ,SRCU_FLAVOR ,INVALID_RCU_FLAVOR
kernel/rcu/tree.c +0 -2
@@ -556,8 +556,6 @@ void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, {switch (test_type) {case RCU_FLAVOR : -case RCU_BH_FLAVOR : -case RCU_SCHED_FLAVOR : *flags =READ_ONCE (rcu_state.gp_flags); *gp_seq =rcu_seq_current (&rcu_state.gp_seq);break ;
Signature · 1 line
2.17.1
PATCH 7/12 rcu: Inline rcu_kthread_do_work() into its sole remaining caller
In reply to Paul E. McKenneyThe rcu_kthread_do_work() function has a single-line body and only one remaining caller. This commit therefore saves a few lines of code by inlining rcu_kthread_do_work() into its sole remaining caller.
--- kernel/rcu/tree_plugin.h | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
kernel/rcu/tree_plugin.h +1 -6
@@ -1369,11 +1369,6 @@ static int rcu_spawn_one_boost_kthread(struct rcu_node *rnp)return 0 ; } -static void rcu_kthread_do_work (void ) -{ -rcu_do_batch (this_cpu_ptr (&rcu_data)); -} -static void rcu_cpu_kthread_setup (unsigned int cpu) {struct sched_param sp;
@@ -1413,7 +1408,7 @@ static void rcu_cpu_kthread(unsigned int cpu) *workp =0 ;local_irq_enable ();if (work) -rcu_kthread_do_work (); +rcu_do_batch (this_cpu_ptr (&rcu_data));local_bh_enable ();if (*workp ==0 ) {trace_rcu_utilization (TPS ("End CPU kthread@rcu_wait" ));
Signature · 1 line
2.17.1
PATCH 8/12 rcu: Determine expedited-GP IPI handler at build time
In reply to Paul E. McKenneyBack when there could be multiple RCU flavors running in the same kernel at the same time, it was necessary to specify the expedited grace-period IPI handler at runtime. Now that there is only one RCU flavor, the IPI handler can be determined at build time. There is therefore no longer any reason for the RCU-preempt and RCU-sched IPI handlers to have different names, nor is there any reason to pass these handlers in function arguments and in the data structures enclosing workqueues.
This commit therefore makes all these changes, pushing the specification of the expedited grace-period IPI handler down to the point of use.
--- .../Expedited-Grace-Periods/ExpSchedFlow.svg | 18 ++++++----- .../Expedited-Grace-Periods.html | 26 ++++++++-------- kernel/rcu/tree.h | 1 - kernel/rcu/tree_exp.h | 30 +++++++++---------- 4 files changed, 38 insertions(+), 37 deletions(-)
Documentation/RCU/Design/Expedited-Grace-Periods/ExpSchedFlow.svg +11 -7
@@ -328,13 +328,13 @@ inkscape:window-height="1148" id="namedview90" showgrid="true" - inkscape:zoom="0.80021373" - inkscape:cx="462.49289" - inkscape:cy="473.6718" + inkscape:zoom="0.69092787" + inkscape:cx="476.34085" + inkscape:cy="712.80957" inkscape:window-x="770" inkscape:window-y="24" inkscape:window-maximized="0" - inkscape:current-layer="g4114-9-3-9" + inkscape:current-layer="g4" inkscape:snap-grids="false" fit-margin-top="5" fit-margin-right="5"
@@ -813,14 +813,18 @@ <text sodipodi:linespacing="125%" id="text4110-5-7-6-2-4-0" - y="841.88086" + y="670.74316" x="1460.1007" style="font-size:267.24359131px;font-style:normal;font-weight:normal;text-align:center;line-height:125%;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans" xml:space="preserve"><tspan - y="841.88086" + y="670.74316" + x="1460.1007" + sodipodi:role="line" + id="tspan4925-1-2-4-5">Request</tspan><tspan + y="1004.7976" x="1460.1007" sodipodi:role="line" - id="tspan4925-1-2-4-5">reched_cpu()</tspan></text> + id="tspan3100">context switch</tspan></text> </g> </g> </svg>
Documentation/RCU/Design/Expedited-Grace-Periods/Expedited-Grace-Periods.html +13 -13
@@ -72,10 +72,10 @@ will ignore it because idle and offline CPUs are already residing in quiescent states. Otherwise, the expedited grace period will use <tt>smp_call_function_single()</tt> to send the CPU an IPI, which -is handled by <tt>sync_rcu_exp_handler()</tt>. +is handled by <tt>rcu_exp_handler()</tt>. <p> -However, because this is preemptible RCU, <tt>sync_rcu_exp_handler()</tt> +However, because this is preemptible RCU, <tt>rcu_exp_handler()</tt> can check to see if the CPU is currently running in an RCU read-side critical section. If not, the handler can immediately report a quiescent state.
@@ -145,19 +145,18 @@ expedited grace period is shown in the following diagram: <p><img src="ExpSchedFlow.svg" alt="ExpSchedFlow.svg" width="55%"> <p> -As with RCU-preempt's <tt>synchronize_rcu_expedited()</tt>, +As with RCU-preempt, RCU-sched's <tt>synchronize_sched_expedited()</tt> ignores offline and idle CPUs, again because they are in remotely detectable quiescent states. -However, the <tt>synchronize_rcu_expedited()</tt> handler -is <tt>sync_sched_exp_handler()</tt>, and because the +However, because the <tt>rcu_read_lock_sched()</tt> and <tt>rcu_read_unlock_sched()</tt> leave no trace of their invocation, in general it is not possible to tell whether or not the current CPU is in an RCU read-side critical section. -The best that <tt>sync_sched_exp_handler()</tt> can do is to check +The best that RCU-sched's <tt>rcu_exp_handler()</tt> can do is to check for idle, on the off-chance that the CPU went idle while the IPI was in flight. -If the CPU is idle, then <tt>sync_sched_exp_handler()</tt> reports +If the CPU is idle, then <tt>rcu_exp_handler()</tt> reports the quiescent state. <p> Otherwise, the handler forces a future context switch by setting the
@@ -298,19 +297,18 @@ Instead, the task pushing the grace period forward will include the idle CPUs in the mask passed to <tt>rcu_report_exp_cpu_mult()</tt>. <p> -For RCU-sched, there is an additional check for idle in the IPI -handler, <tt>sync_sched_exp_handler()</tt>. +For RCU-sched, there is an additional check: If the IPI has interrupted the idle loop, then -<tt>sync_sched_exp_handler()</tt> invokes <tt>rcu_report_exp_rdp()</tt> +<tt>rcu_exp_handler()</tt> invokes <tt>rcu_report_exp_rdp()</tt> to report the corresponding quiescent state. <p> For RCU-preempt, there is no specific check for idle in the -IPI handler (<tt>sync_rcu_exp_handler()</tt>), but because +IPI handler (<tt>rcu_exp_handler()</tt>), but because RCU read-side critical sections are not permitted within the -idle loop, if <tt>sync_rcu_exp_handler()</tt> sees that the CPU is within +idle loop, if <tt>rcu_exp_handler()</tt> sees that the CPU is within RCU read-side critical section, the CPU cannot possibly be idle. -Otherwise, <tt>sync_rcu_exp_handler()</tt> invokes +Otherwise, <tt>rcu_exp_handler()</tt> invokes <tt>rcu_report_exp_rdp()</tt> to report the corresponding quiescent state, regardless of whether or not that quiescent state was due to the CPU being idle.
@@ -625,6 +623,8 @@ checks, but only during the mid-boot dead zone. <p> With this refinement, synchronous grace periods can now be used from task context pretty much any time during the life of the kernel. +That is, aside from some points in the suspend, hibernate, or shutdown +code path. <h3><a name="Summary"> Summary</a></h3>
kernel/rcu/tree.h +0 -1
@@ -36,7 +36,6 @@/* Communicate arguments to a workqueue handler. */ struct rcu_exp_work { -smp_call_func_t rew_func;unsigned long rew_s;struct work_struct rew_work; };
kernel/rcu/tree_exp.h +14 -16
@@ -22,6 +22,8 @@#include <linux/lockdep.h> +static void rcu_exp_handler (void *unused); +/* * Record the start of an expedited grace period. */
@@ -344,7 +346,6 @@ static void sync_rcu_exp_select_node_cpus(struct work_struct *wp) {int cpu;unsigned long flags; -smp_call_func_t func;unsigned long mask_ofl_test;unsigned long mask_ofl_ipi;int ret;
@@ -352,7 +353,6 @@ static void sync_rcu_exp_select_node_cpus(struct work_struct *wp) container_of(wp, struct rcu_exp_work, rew_work); struct rcu_node *rnp = container_of(rewp, struct rcu_node, rew); - func = rewp->rew_func;raw_spin_lock_irqsave_rcu_node (rnp, flags);/* Each pass checks a CPU for identity, offline, and idle. */
@@ -396,7 +396,7 @@ static void sync_rcu_exp_select_node_cpus(struct work_struct *wp) mask_ofl_test |= mask;continue ; } - ret =smp_call_function_single (cpu, func,NULL ,0 ); + ret =smp_call_function_single (cpu, rcu_exp_handler,NULL ,0 );if (!ret) { mask_ofl_ipi &= ~mask;continue ;
@@ -426,7 +426,7 @@ static void sync_rcu_exp_select_node_cpus(struct work_struct *wp)* Select the nodes that the upcoming expedited grace period needs * to wait for. */ -static void sync_rcu_exp_select_cpus (smp_call_func_t func) +static void sync_rcu_exp_select_cpus (void ) {int cpu;struct rcu_node *rnp;
@@ -440,7 +440,6 @@ static void sync_rcu_exp_select_cpus(smp_call_func_t func) rnp->exp_need_flush = false;if (!READ_ONCE (rnp->expmask))continue ;/* Avoid early boot non-existent wq. */ - rnp->rew.rew_func = func;if (!READ_ONCE (rcu_par_gp_wq) || rcu_scheduler_active !=RCU_SCHEDULER_RUNNING ||rcu_is_last_leaf_node (rnp)) {
@@ -580,10 +579,10 @@ static void rcu_exp_wait_wake(unsigned long s)* Common code to drive an expedited grace period forward, used by * workqueues and mid-boot-time tasks. */ -static void rcu_exp_sel_wait_wake (smp_call_func_t func,unsigned long s) +static void rcu_exp_sel_wait_wake (unsigned long s) {/* Initialize the rcu_node tree in preparation for the wait. */ -sync_rcu_exp_select_cpus (func); +sync_rcu_exp_select_cpus ();/* Wait and clean up, including waking everyone. */ rcu_exp_wait_wake (s);
@@ -597,14 +596,14 @@ static void wait_rcu_exp_gp(struct work_struct *wp)struct rcu_exp_work *rewp; rewp = container_of(wp, struct rcu_exp_work, rew_work); -rcu_exp_sel_wait_wake (rewp->rew_func, rewp->rew_s); +rcu_exp_sel_wait_wake (rewp->rew_s); }/* * Given a smp_call_function() handler, kick off the specified * implementation of expedited grace period. */ -static void _synchronize_rcu_expedited (smp_call_func_t func) +static void _synchronize_rcu_expedited (void ) {struct rcu_data *rdp;struct rcu_exp_work rew;
@@ -625,10 +624,9 @@ static void _synchronize_rcu_expedited(smp_call_func_t func)/* Ensure that load happens before action based on it. */ if (unlikely (rcu_scheduler_active ==RCU_SCHEDULER_INIT )) {/* Direct call during scheduler init and early_initcalls(). */ -rcu_exp_sel_wait_wake (func, s); +rcu_exp_sel_wait_wake (s); }else {/* Marshall arguments & schedule the expedited grace period. */ - rew.rew_func = func; rew.rew_s = s;INIT_WORK_ONSTACK (&rew.rew_work, wait_rcu_exp_gp);queue_work (rcu_gp_wq, &rew.rew_work);
@@ -654,7 +652,7 @@ static void _synchronize_rcu_expedited(smp_call_func_t func)* ->expmask fields in the rcu_node tree. Otherwise, immediately * report the quiescent state. */ -static void sync_rcu_exp_handler (void *unused) +static void rcu_exp_handler (void *unused) {unsigned long flags;struct rcu_data *rdp =this_cpu_ptr (&rcu_data);
@@ -760,14 +758,14 @@ void synchronize_rcu_expedited(void)if (rcu_scheduler_active ==RCU_SCHEDULER_INACTIVE )return ; - _synchronize_rcu_expedited(sync_rcu_exp_handler); + _synchronize_rcu_expedited(); }EXPORT_SYMBOL_GPL (synchronize_rcu_expedited);#else /* #ifdef CONFIG_PREEMPT_RCU */ /* Invoked on each online non-idle CPU for expedited quiescent state. */ -static void sync_sched_exp_handler (void *unused) +static void rcu_exp_handler (void *unused) {struct rcu_data *rdp;struct rcu_node *rnp;
@@ -799,7 +797,7 @@ static void sync_sched_exp_online_cleanup(int cpu) rnp = rdp->mynode;if (!(READ_ONCE (rnp->expmask) & rdp->grpmask))return ; - ret =smp_call_function_single (cpu, sync_sched_exp_handler,NULL ,0 ); + ret =smp_call_function_single (cpu, rcu_exp_handler,NULL ,0 );WARN_ON_ONCE (ret); }
@@ -835,7 +833,7 @@ void synchronize_rcu_expedited(void)if (rcu_blocking_is_gp ())return ; -_synchronize_rcu_expedited (sync_sched_exp_handler); +_synchronize_rcu_expedited (); }EXPORT_SYMBOL_GPL (synchronize_rcu_expedited);
Signature · 1 line
2.17.1
PATCH 9/12 rcu: Consolidate PREEMPT and !PREEMPT synchronize_rcu_expedited()
In reply to Paul E. McKenneyThe CONFIG_PREEMPT=n and CONFIG_PREEMPT=y implementations of synchronize_rcu_expedited() are quite similar, and with small modifications to rcu_blocking_is_gp() can be made identical. This commit therefore makes this change in order to save a few lines of code and to reduce the amount of duplicate code.
--- kernel/rcu/tree_exp.h | 105 ++++++++++++++++++++---------------------- 1 file changed, 49 insertions(+), 56 deletions(-)
kernel/rcu/tree_exp.h +49 -56
@@ -643,6 +643,33 @@ static void _synchronize_rcu_expedited(void)mutex_unlock (&rcu_state.exp_mutex); } +/* +* During early boot, any blocking grace-period wait automatically +* implies a grace period. Later on, this is never the case for PREEMPT. +* +* Howevr, because a context switch is a grace period for !PREEMPT, any +* blocking grace-period wait automatically implies a grace period if +* there is only one CPU online at any point time during execution of +* either synchronize_rcu() or synchronize_rcu_expedited(). It is OK to +* occasionally incorrectly indicate that there are multiple CPUs online +* when there was in fact only one the whole time, as this just adds some +* overhead: RCU still operates correctly. +*/ +static int rcu_blocking_is_gp (void ) +{ +int ret; + +if (rcu_scheduler_active ==RCU_SCHEDULER_INACTIVE ) +return true; +if (IS_ENABLED (CONFIG_PREEMPT )) +return false; +might_sleep ();/* Check for RCU read-side critical section. */ +preempt_disable (); + ret =num_online_cpus () <=1 ; +preempt_enable (); +return ret; +} +#ifdef CONFIG_PREEMPT_RCU /*
@@ -729,39 +756,6 @@ static void sync_sched_exp_online_cleanup(int cpu) { } -/** -* synchronize_rcu_expedited - Brute-force RCU grace period -* -* Wait for an RCU-preempt grace period, but expedite it. The basic -* idea is to IPI all non-idle non-nohz online CPUs. The IPI handler -* checks whether the CPU is in an RCU-preempt critical section, and -* if so, it sets a flag that causes the outermost rcu_read_unlock() -* to report the quiescent state. On the other hand, if the CPU is -* not in an RCU read-side critical section, the IPI handler reports -* the quiescent state immediately. -* -* Although this is a greate improvement over previous expedited -* implementations, it is still unfriendly to real-time workloads, so is -* thus not recommended for any sort of common-case code. In fact, if -* you are using synchronize_rcu_expedited() in a loop, please restructure -* your code to batch your updates, and then Use a single synchronize_rcu() -* instead. -* -* This has the same semantics as (but is more brutal than) synchronize_rcu(). -*/ -void synchronize_rcu_expedited (void ) -{ -RCU_LOCKDEP_WARN (lock_is_held (&rcu_bh_lock_map) || -lock_is_held (&rcu_lock_map) || -lock_is_held (&rcu_sched_lock_map), -"Illegal synchronize_rcu_expedited() in RCU read-side critical section" ); - -if (rcu_scheduler_active ==RCU_SCHEDULER_INACTIVE ) -return ; -_synchronize_rcu_expedited (); -} -EXPORT_SYMBOL_GPL (synchronize_rcu_expedited); -#else /* #ifdef CONFIG_PREEMPT_RCU */ /* Invoked on each online non-idle CPU for expedited quiescent state. */
@@ -801,27 +795,28 @@ static void sync_sched_exp_online_cleanup(int cpu)WARN_ON_ONCE (ret); } -/* -* Because a context switch is a grace period for !PREEMPT, any -* blocking grace-period wait automatically implies a grace period if -* there is only one CPU online at any point time during execution of -* either synchronize_rcu() or synchronize_rcu_expedited(). It is OK to -* occasionally incorrectly indicate that there are multiple CPUs online -* when there was in fact only one the whole time, as this just adds some -* overhead: RCU still operates correctly. -*/ -static int rcu_blocking_is_gp (void ) -{ -int ret; - -might_sleep ();/* Check for RCU read-side critical section. */ -preempt_disable (); - ret =num_online_cpus () <=1 ; -preempt_enable (); -return ret; -} +#endif /* #else #ifdef CONFIG_PREEMPT_RCU */ -/* PREEMPT=n implementation of synchronize_rcu_expedited(). */ +/** +* synchronize_rcu_expedited - Brute-force RCU grace period +* +* Wait for an RCU grace period, but expedite it. The basic idea is to +* IPI all non-idle non-nohz online CPUs. The IPI handler checks whether +* the CPU is in an RCU critical section, and if so, it sets a flag that +* causes the outermost rcu_read_unlock() to report the quiescent state +* for RCU-preempt or asks the scheduler for help for RCU-sched. On the +* other hand, if the CPU is not in an RCU read-side critical section, +* the IPI handler reports the quiescent state immediately. +* +* Although this is a greate improvement over previous expedited +* implementations, it is still unfriendly to real-time workloads, so is +* thus not recommended for any sort of common-case code. In fact, if +* you are using synchronize_rcu_expedited() in a loop, please restructure +* your code to batch your updates, and then Use a single synchronize_rcu() +* instead. +* +* This has the same semantics as (but is more brutal than) synchronize_rcu(). +*/ void synchronize_rcu_expedited (void ) { RCU_LOCKDEP_WARN(lock_is_held(&rcu_bh_lock_map) ||
@@ -829,12 +824,10 @@ void synchronize_rcu_expedited(void)lock_is_held (&rcu_sched_lock_map), "Illegal synchronize_rcu_expedited() in RCU read-side critical section"); -/* If only one CPU, this is automatically a grace period. */ +/* Is the state is such that the call is a grace period? */ if (rcu_blocking_is_gp ())return ;_synchronize_rcu_expedited (); }EXPORT_SYMBOL_GPL (synchronize_rcu_expedited); - -#endif /* #else #ifdef CONFIG_PREEMPT_RCU */
Signature · 1 line
2.17.1
PATCH 10/12 rcu: Consolidate PREEMPT and !PREEMPT synchronize_rcu()
In reply to Paul E. McKenneyNow that rcu_blocking_is_gp() makes the correct immediate-return decision for both PREEMPT and !PREEMPT, a single implementation of synchronize_rcu() will work correctly under both configurations. This commit therefore eliminates a few lines of code by consolidating the two implementations of synchronize_rcu().
--- kernel/rcu/tree.c | 73 ++++++++++++++++++++++++++++++++++++++++ kernel/rcu/tree_exp.h | 27 --------------- kernel/rcu/tree_plugin.h | 64 ----------------------------------- 3 files changed, 73 insertions(+), 91 deletions(-)
kernel/rcu/tree.c +73 -0
@@ -2950,6 +2950,79 @@ void kfree_call_rcu(struct rcu_head *head, rcu_callback_t func) }EXPORT_SYMBOL_GPL (kfree_call_rcu); +/* +* During early boot, any blocking grace-period wait automatically +* implies a grace period. Later on, this is never the case for PREEMPT. +* +* Howevr, because a context switch is a grace period for !PREEMPT, any +* blocking grace-period wait automatically implies a grace period if +* there is only one CPU online at any point time during execution of +* either synchronize_rcu() or synchronize_rcu_expedited(). It is OK to +* occasionally incorrectly indicate that there are multiple CPUs online +* when there was in fact only one the whole time, as this just adds some +* overhead: RCU still operates correctly. +*/ +static int rcu_blocking_is_gp (void ) +{ +int ret; + +if (IS_ENABLED (CONFIG_PREEMPT )) +return rcu_scheduler_active ==RCU_SCHEDULER_INACTIVE ; +might_sleep ();/* Check for RCU read-side critical section. */ +preempt_disable (); + ret =num_online_cpus () <=1 ; +preempt_enable (); +return ret; +} + +/** +* synchronize_rcu - wait until a grace period has elapsed. +* +* Control will return to the caller some time after a full grace +* period has elapsed, in other words after all currently executing RCU +* read-side critical sections have completed. Note, however, that +* upon return from synchronize_rcu(), the caller might well be executing +* concurrently with new RCU read-side critical sections that began while +* synchronize_rcu() was waiting. RCU read-side critical sections are +* delimited by rcu_read_lock() and rcu_read_unlock(), and may be nested. +* In addition, regions of code across which interrupts, preemption, or +* softirqs have been disabled also serve as RCU read-side critical +* sections. This includes hardware interrupt handlers, softirq handlers, +* and NMI handlers. +* +* Note that this guarantee implies further memory-ordering guarantees. +* On systems with more than one CPU, when synchronize_rcu() returns, +* each CPU is guaranteed to have executed a full memory barrier since +* the end of its last RCU read-side critical section whose beginning +* preceded the call to synchronize_rcu(). In addition, each CPU having +* an RCU read-side critical section that extends beyond the return from +* synchronize_rcu() is guaranteed to have executed a full memory barrier +* after the beginning of synchronize_rcu() and before the beginning of +* that RCU read-side critical section. Note that these guarantees include +* CPUs that are offline, idle, or executing in user mode, as well as CPUs +* that are executing in the kernel. +* +* Furthermore, if CPU A invoked synchronize_rcu(), which returned +* to its caller on CPU B, then both CPU A and CPU B are guaranteed +* to have executed a full memory barrier during the execution of +* synchronize_rcu() -- even if CPU A and CPU B are the same CPU (but +* again only if the system has more than one CPU). +*/ +void synchronize_rcu (void ) +{ +RCU_LOCKDEP_WARN (lock_is_held (&rcu_bh_lock_map) || +lock_is_held (&rcu_lock_map) || +lock_is_held (&rcu_sched_lock_map), +"Illegal synchronize_rcu() in RCU read-side critical section" ); +if (rcu_blocking_is_gp ()) +return ; +if (rcu_gp_is_expedited ()) +synchronize_rcu_expedited (); +else +wait_rcu_gp (call_rcu); +} +EXPORT_SYMBOL_GPL (synchronize_rcu); +/** * get_state_synchronize_rcu - Snapshot current RCU state *
kernel/rcu/tree_exp.h +0 -27
@@ -643,33 +643,6 @@ static void _synchronize_rcu_expedited(void)mutex_unlock (&rcu_state.exp_mutex); } -/* -* During early boot, any blocking grace-period wait automatically -* implies a grace period. Later on, this is never the case for PREEMPT. -* -* Howevr, because a context switch is a grace period for !PREEMPT, any -* blocking grace-period wait automatically implies a grace period if -* there is only one CPU online at any point time during execution of -* either synchronize_rcu() or synchronize_rcu_expedited(). It is OK to -* occasionally incorrectly indicate that there are multiple CPUs online -* when there was in fact only one the whole time, as this just adds some -* overhead: RCU still operates correctly. -*/ -static int rcu_blocking_is_gp (void ) -{ -int ret; - -if (rcu_scheduler_active ==RCU_SCHEDULER_INACTIVE ) -return true; -if (IS_ENABLED (CONFIG_PREEMPT )) -return false; -might_sleep ();/* Check for RCU read-side critical section. */ -preempt_disable (); - ret =num_online_cpus () <=1 ; -preempt_enable (); -return ret; -} -#ifdef CONFIG_PREEMPT_RCU /*
kernel/rcu/tree_plugin.h +0 -64
76 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH 11/12 rcu: Inline _synchronize_rcu_expedited() into synchronize_rcu_expedited()
In reply to Paul E. McKenneyNow that _synchronize_rcu_expedited() has only one caller, and given that this is a tail call, this commit inlines _synchronize_rcu_expedited() into synchronize_rcu_expedited().
--- kernel/rcu/tree_exp.h | 81 +++++++++++++++++++------------------------ 1 file changed, 36 insertions(+), 45 deletions(-)
kernel/rcu/tree_exp.h +36 -45
@@ -599,50 +599,6 @@ static void wait_rcu_exp_gp(struct work_struct *wp)rcu_exp_sel_wait_wake (rewp->rew_s); } -/* -* Given a smp_call_function() handler, kick off the specified -* implementation of expedited grace period. -*/ -static void _synchronize_rcu_expedited (void ) -{ -struct rcu_data *rdp; -struct rcu_exp_work rew; -struct rcu_node *rnp; -unsigned long s; - -/* If expedited grace periods are prohibited, fall back to normal. */ -if (rcu_gp_is_normal ()) { -wait_rcu_gp (call_rcu); -return ; - } - -/* Take a snapshot of the sequence number. */ - s =rcu_exp_gp_seq_snap (); -if (exp_funnel_lock (s)) -return ;/* Someone else did our work for us. */ - -/* Ensure that load happens before action based on it. */ -if (unlikely (rcu_scheduler_active ==RCU_SCHEDULER_INIT )) { -/* Direct call during scheduler init and early_initcalls(). */ -rcu_exp_sel_wait_wake (s); - }else { -/* Marshall arguments & schedule the expedited grace period. */ - rew.rew_s = s; -INIT_WORK_ONSTACK (&rew.rew_work, wait_rcu_exp_gp); -queue_work (rcu_gp_wq, &rew.rew_work); - } - -/* Wait for expedited grace period to complete. */ - rdp =per_cpu_ptr (&rcu_data,raw_smp_processor_id ()); - rnp =rcu_get_root (); -wait_event (rnp->exp_wq[rcu_seq_ctr (s) &0x3 ], -sync_exp_work_done (s)); -smp_mb ();/* Workqueue actions happen before return. */ - -/* Let the next expedited grace period start. */ -mutex_unlock (&rcu_state.exp_mutex); -} -#ifdef CONFIG_PREEMPT_RCU /*
@@ -792,6 +748,11 @@ static void sync_sched_exp_online_cleanup(int cpu)*/ void synchronize_rcu_expedited (void ) { +struct rcu_data *rdp; +struct rcu_exp_work rew; +struct rcu_node *rnp; +unsigned long s; + RCU_LOCKDEP_WARN(lock_is_held(&rcu_bh_lock_map) || lock_is_held(&rcu_lock_map) || lock_is_held(&rcu_sched_lock_map),
@@ -801,6 +762,36 @@ void synchronize_rcu_expedited(void)if (rcu_blocking_is_gp ())return ; -_synchronize_rcu_expedited (); +/* If expedited grace periods are prohibited, fall back to normal. */ +if (rcu_gp_is_normal ()) { +wait_rcu_gp (call_rcu); +return ; + } + +/* Take a snapshot of the sequence number. */ + s =rcu_exp_gp_seq_snap (); +if (exp_funnel_lock (s)) +return ;/* Someone else did our work for us. */ + +/* Ensure that load happens before action based on it. */ +if (unlikely (rcu_scheduler_active ==RCU_SCHEDULER_INIT )) { +/* Direct call during scheduler init and early_initcalls(). */ +rcu_exp_sel_wait_wake (s); + }else { +/* Marshall arguments & schedule the expedited grace period. */ + rew.rew_s = s; +INIT_WORK_ONSTACK (&rew.rew_work, wait_rcu_exp_gp); +queue_work (rcu_gp_wq, &rew.rew_work); + } + +/* Wait for expedited grace period to complete. */ + rdp =per_cpu_ptr (&rcu_data,raw_smp_processor_id ()); + rnp =rcu_get_root (); +wait_event (rnp->exp_wq[rcu_seq_ctr (s) &0x3 ], +sync_exp_work_done (s)); +smp_mb ();/* Workqueue actions happen before return. */ + +/* Let the next expedited grace period start. */ +mutex_unlock (&rcu_state.exp_mutex); }EXPORT_SYMBOL_GPL (synchronize_rcu_expedited);
Signature · 1 line
2.17.1
PATCH 12/12 rcu: Discard separate per-CPU callback counts
In reply to Paul E. McKenneyBack when there were multiple flavors of RCU, it was necessary to separately count lazy and non-lazy callbacks for each CPU. These counts were used in CONFIG_RCU_FAST_NO_HZ kernels to determine how long a newly idle CPU should be allowed to sleep before handling its RCU callbacks. But now that there is only one flavor, the callback counts for a given CPU's sole rcu_data structure are the counts for that CPU.
This commit therefore removes the rcu_data structure's ->nonlazy_posted and ->nonlazy_posted_snap fields, the rcu_idle_count_callbacks_posted() and rcu_cpu_has_callbacks() functions, repurposes the rcu_data structure's ->all_lazy field to record the laziness state at the beginning of the latest idle sojourn, and modifies CONFIG_RCU_FAST_NO_HZ RCU CPU stall warnings accordingly.
--- Documentation/RCU/stallwarn.txt | 15 +++++----- kernel/rcu/tree.c | 25 ----------------- kernel/rcu/tree.h | 6 +--- kernel/rcu/tree_plugin.h | 50 ++++++++------------------------- 4 files changed, 21 insertions(+), 75 deletions(-)
Documentation/RCU/stallwarn.txt +8 -7
@@ -219,17 +219,18 @@ an estimate of the total number of RCU callbacks queued across all CPUs In kernels with CONFIG_RCU_FAST_NO_HZ, more information is printed for each CPU: - 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 nonlazy_posted: 25 .D + 0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 Nonlazy posted: ..D The "last_accelerate:" prints the low-order 16 bits (in hex) of the jiffies counter when this CPU last invoked rcu_try_advance_all_cbs() from rcu_needs_cpu() or last invoked rcu_accelerate_cbs() from -rcu_prepare_for_idle(). The "nonlazy_posted:" prints the number -of non-lazy callbacks posted since the last call to rcu_needs_cpu(). -Finally, an "L" indicates that there are currently no non-lazy callbacks -("." is printed otherwise, as shown above) and "D" indicates that -dyntick-idle processing is enabled ("." is printed otherwise, for example, -if disabled via the "nohz=" kernel boot parameter). +rcu_prepare_for_idle(). The "Nonlazy posted:" indicates lazy-callback +status, so that an "l" indicates that all callbacks were lazy at the start +of the last idle period and an "L" indicates that there are currently +no non-lazy callbacks (in both cases, "." is printed otherwise, as +shown above) and "D" indicates that dyntick-idle processing is enabled +("." is printed otherwise, for example, if disabled via the "nohz=" +kernel boot parameter). If the grace period ends just as the stall warning starts printing, there will be a spurious stall-warning message, which will include
kernel/rcu/tree.c +0 -25
@@ -2878,9 +2878,6 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func, int cpu, bool lazy)rcu_segcblist_init (&rdp->cblist); }rcu_segcblist_enqueue (&rdp->cblist, head, lazy); -if (!lazy) -rcu_idle_count_callbacks_posted (); -if (__is_kfree_rcu_offset ((unsigned long )func)) trace_rcu_kfree_callback(rcu_state.name, head, (unsigned long)func,
@@ -3110,28 +3107,6 @@ static int rcu_pending(void)return 0 ; } -/* -* Return true if the specified CPU has any callback. If all_lazy is -* non-NULL, store an indication of whether all callbacks are lazy. -* (If there are no callbacks, all of them are deemed to be lazy.) -*/ -static bool rcu_cpu_has_callbacks (bool *all_lazy) -{ -bool al = true; -bool hc = false; -struct rcu_data *rdp; - - rdp =this_cpu_ptr (&rcu_data); -if (!rcu_segcblist_empty (&rdp->cblist)) { - hc = true; -if (rcu_segcblist_n_nonlazy_cbs (&rdp->cblist)) - al = false; - } -if (all_lazy) - *all_lazy = al; -return hc; -} -/* * Helper function for rcu_barrier() tracing. If tracing is disabled, * the compiler is expected to optimize this away.
kernel/rcu/tree.h +1 -5
@@ -193,10 +193,7 @@ struct rcu_data {bool rcu_need_heavy_qs;/* GP old, so heavy quiescent state! */ bool rcu_urgent_qs;/* GP old need light quiescent state. */ #ifdef CONFIG_RCU_FAST_NO_HZ -bool all_lazy;/* Are all CPU's CBs lazy? */ -unsigned long nonlazy_posted;/* # times non-lazy CB posted to CPU. */ -unsigned long nonlazy_posted_snap; -/* Nonlazy_posted snapshot. */ +bool all_lazy;/* All CPU's CBs lazy at idle start? */ unsigned long last_accelerate;/* Last jiffy CBs were accelerated. */ unsigned long last_advance_all;/* Last jiffy CBs were all advanced. */ int tick_nohz_enabled_snap;/* Previously seen value from sysfs. */
@@ -430,7 +427,6 @@ static void __init rcu_spawn_boost_kthreads(void);static void rcu_prepare_kthreads (int cpu);static void rcu_cleanup_after_idle (void );static void rcu_prepare_for_idle (void ); -static void rcu_idle_count_callbacks_posted (void );static bool rcu_preempt_has_tasks (struct rcu_node *rnp);static bool rcu_preempt_need_deferred_qs (struct task_struct *t);static void rcu_preempt_deferred_qs (struct task_struct *t);
kernel/rcu/tree_plugin.h +12 -38
104 lines not on this page · raw message
Signature · 1 line
2.17.1
[PATCH tip/core/rcu 13/12] Remove wrapper definitions for obsolete RCU update functions
In reply to Paul E. McKenneyNone of synchronize_rcu_bh, synchronize_rcu_bh_expedited, call_rcu_bh, rcu_barrier_bh, synchronize_sched, synchronize_sched_expedited, call_rcu_sched, rcu_barrier_sched, get_state_synchronize_sched, and cond_synchronize_sched are actually used. This commit therefore removes their trivial wrapper-function definitions.
[ paulmck: The intent was to defer this to the following release, but
there are no more uses of these functions, so full speed ahead!!! ]
include/linux/rcupdate.h +0 -53
@@ -896,57 +896,4 @@ rcu_head_after_call_rcu(struct rcu_head *rhp, rcu_callback_t f) return false; } - -/* Transitional pre-consolidation compatibility definitions. */ - -static inline void synchronize_rcu_bh (void ) -{ -synchronize_rcu (); -} - -static inline void synchronize_rcu_bh_expedited (void ) -{ -synchronize_rcu_expedited (); -} - -static inline void call_rcu_bh (struct rcu_head *head,rcu_callback_t func) -{ -call_rcu (head, func); -} - -static inline void rcu_barrier_bh (void ) -{ -rcu_barrier (); -} - -static inline void synchronize_sched (void ) -{ -synchronize_rcu (); -} - -static inline void synchronize_sched_expedited (void ) -{ -synchronize_rcu_expedited (); -} - -static inline void call_rcu_sched (struct rcu_head *head,rcu_callback_t func) -{ -call_rcu (head, func); -} - -static inline void rcu_barrier_sched (void ) -{ -rcu_barrier (); -} - -static inline unsigned long get_state_synchronize_sched (void ) -{ -return get_state_synchronize_rcu (); -} - -static inline void cond_synchronize_sched (unsigned long oldstate) -{ -cond_synchronize_rcu (oldstate); -} -#endif /* __LINUX_RCUPDATE_H */