preemptible RCU
Subsystems (4)
cover preemptible RCU
Hello!
This patchset is an update of that posted by Dipankar last January
(http://lkml.org/lkml/2007/1/15/133). This is work in progress, not yet ready for inclusion. It passes rcutorture on i386, x86_64, and ppc64 boxes as well as kernbench, so should be safe for experimentation. As with Dipankar's previous post, this variant of preemptible rcu_read_lock() and rcu_read_unlock may be invoked from NMI/SMI handlers, and do not contain any heavyweight atomic operations or memory barriers (although they do still momentarily disable IRQs). This patchset features a fully parallel grace-period computation, which will become increasingly important with upcoming multicore/multi-threaded CPUs. In addition, this patchset provides a preemptible-RCU variant of synchronize_sched() that avoids the previous deadlock with CPU hotplug -- this variant may eventually prove unnecessary, but is offered in the spirit of separating concerns.
Next steps: (1) Integrate with CPU hotplug. (2) Re-merge RCU priority boosting. (3) Fix some naming issues. Longer term work includes optimized dyntick operation and eliminating the interrupt disabling in rcu_read_lock() and rcu_read_unlock().
Thanx, Paul
This patch re-organizes the RCU code to enable multiple implementations of RCU. Users of RCU continues to include rcupdate.h and the RCU interfaces remain the same. This is in preparation for subsequently merging the preepmtpible RCU implementation.
---
include/linux/rcuclassic.h | 149 ++++++++++++ include/linux/rcupdate.h | 151 +++--------- kernel/Makefile | 2 kernel/rcuclassic.c | 558 +++++++++++++++++++++++++++++++++++++++++++++ kernel/rcupdate.c | 558 ++------------------------------------------- 5 files changed, 778 insertions(+), 640 deletions(-)
include/linux/rcuclassic.h +149 -0
@@ -0,0 +1,149 @@ +/* +* Read-Copy Update mechanism for mutual exclusion (classic version) +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. +* +* Copyright IBM Corporation, 2001 +* +* Author: Dipankar Sarma <[email protected]> +* +* Based on the original work by Paul McKenney <[email protected]> +* and inputs from Rusty Russell, Andrea Arcangeli and Andi Kleen. +* Papers: +* http://www.rdrop.com/users/paulmck/paper/rclockpdcsproof.pdf +* http://lse.sourceforge.net/locking/rclock_OLS.2001.05.01c.sc.pdf (OLS2001) +* +* For detailed explanation of Read-Copy Update mechanism see - +* Documentation/RCU +* +*/ + +#ifndef __LINUX_RCUCLASSIC_H +#define __LINUX_RCUCLASSIC_H + +#ifdef __KERNEL__ + +#include <linux/cache.h> +#include <linux/spinlock.h> +#include <linux/threads.h> +#include <linux/percpu.h> +#include <linux/cpumask.h> +#include <linux/seqlock.h> + + +/* Global control variables for rcupdate callback mechanism. */ +struct rcu_ctrlblk { +long cur;/* Current batch number. */ +long completed;/* Number of the last completed batch */ +int next_pending;/* Is the next batch already waiting? */ + +int signaled; + +spinlock_t lock____cacheline_internodealigned_in_smp ; +cpumask_t cpumask;/* CPUs that need to switch in order */ +/* for current batch to proceed. */ +}____cacheline_internodealigned_in_smp ; + +/* Is batch a before batch b ? */ +static inline int rcu_batch_before (long a,long b) +{ +return (a - b) <0 ; +} + +/* Is batch a after batch b ? */ +static inline int rcu_batch_after (long a,long b) +{ +return (a - b) >0 ; +} + +/* +* Per-CPU data for Read-Copy UPdate. +* nxtlist - new callbacks are added here +* curlist - current batch for which quiescent cycle started if any +*/ +struct rcu_data { +/* 1) quiescent state handling : */ +long quiescbatch;/* Batch # for grace period */ +int passed_quiesc;/* User-mode/idle loop etc. */ +int qs_pending;/* core waits for quiesc state */ + +/* 2) batch handling */ +long batch;/* Batch # for current RCU batch */ +struct rcu_head *nxtlist; +struct rcu_head **nxttail; +long qlen;/* # of queued callbacks */ +struct rcu_head *curlist; +struct rcu_head **curtail; +struct rcu_head *donelist; +struct rcu_head **donetail; +long blimit;/* Upper limit on a processed batch */ +int cpu; +struct rcu_head barrier; +}; + +DECLARE_PER_CPU(struct rcu_data, rcu_data); +DECLARE_PER_CPU(struct rcu_data, rcu_bh_data); + +/* +* Increment the quiescent state counter. +* The counter is a bit degenerated: We do not need to know +* how many quiescent states passed, just if there was at least +* one since the start of the grace period. Thus just a flag. +*/ +static inline void rcu_qsctr_inc (int cpu) +{ +struct rcu_data *rdp = &per_cpu (rcu_data, cpu); + rdp->passed_quiesc =1 ; +} +static inline void rcu_bh_qsctr_inc (int cpu) +{ +struct rcu_data *rdp = &per_cpu (rcu_bh_data, cpu); + rdp->passed_quiesc =1 ; +} + +extern int rcu_pending (int cpu); +extern int rcu_needs_cpu (int cpu); + +#define __rcu_read_lock () \ + do { \ + preempt_disable(); \ + __acquire(RCU); \ + } while(0) +#define __rcu_read_unlock () \ + do { \ + __release(RCU); \ + preempt_enable(); \ + } while(0) +#define __rcu_read_lock_bh () \ + do { \ + local_bh_disable(); \ + __acquire(RCU_BH); \ + } while(0) +#define __rcu_read_unlock_bh () \ + do { \ + __release(RCU_BH); \ + local_bh_enable(); \ + } while(0) + +#define __synchronize_sched () synchronize_rcu() + +extern void __rcu_init (void ); +extern void rcu_check_callbacks (int cpu,int user); +extern void rcu_restart_cpu (int cpu); +extern long rcu_batches_completed (void ); +extern long rcu_batches_completed_bh (void ); + +#endif /* __KERNEL__ */ +#endif /* __LINUX_RCUCLASSIC_H */
include/linux/rcupdate.h +45 -106
204 lines not on this page · raw message
kernel/Makefile +1 -1
@@ -6,7 +6,7 @@ obj-y = sched.o fork.o exec_domain.o exit.o itimer.o time.o softirq.o resource.o \ sysctl.o capability.o ptrace.o timer.o user.o \ signal.o sys.o kmod.o workqueue.o pid.o \ - rcupdate.o extable.o params.o posix-timers.o \ + rcupdate.o rcuclassic.o extable.o params.o posix-timers.o \ kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \ hrtimer.o rwsem.o latency.o nsproxy.o srcu.o die_notifier.o
kernel/rcuclassic.c +558 -0
558 lines not on this page · raw message
kernel/rcupdate.c +25 -533
615 lines not on this page · raw message
RFC PATCH 1/4 RCU: Fix barriers
In reply to Paul E. McKenneyFix rcu_barrier() to work properly in preemptive kernel environment. Also, the ordering of callback must be preserved while moving callbacks to another CPU during CPU hotplug.
---
rcuclassic.c | 2 +- rcupdate.c | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-)
kernel/rcuclassic.c +1 -1
@@ -349,9 +349,9 @@ static void __rcu_offline_cpu(struct rcuif (rcp->cur != rcp->completed)cpu_quiet (rdp->cpu, rcp);spin_unlock_bh (&rcp->lock); +rcu_move_batch (this_rdp, rdp->donelist, rdp->donetail);rcu_move_batch (this_rdp, rdp->curlist, rdp->curtail);rcu_move_batch (this_rdp, rdp->nxtlist, rdp->nxttail); -rcu_move_batch (this_rdp, rdp->donelist, rdp->donetail); }static void rcu_offline_cpu (int cpu)
kernel/rcupdate.c +10 -0
@@ -115,7 +115,17 @@ void rcu_barrier(void)mutex_lock (&rcu_barrier_mutex);init_completion (&rcu_barrier_completion);atomic_set (&rcu_barrier_cpu_count,0 ); +/* +* The queueing of callbacks in all CPUs must be atomic with +* respect to RCU, otherwise one CPU may queue a callback, +* wait for a grace period, decrement barrier count and call +* complete(), while other CPUs have not yet queued anything. +* So, we need to make sure that grace periods cannot complete +* until all the callbacks are queued. +*/ +rcu_read_lock ();on_each_cpu (rcu_barrier_func,NULL ,0 ,1 ); +rcu_read_unlock ();wait_for_completion (&rcu_barrier_completion);mutex_unlock (&rcu_barrier_mutex); }
RFC PATCH 3/4 RCU: preemptible RCU
In reply to Paul E. McKenneyThis patch implements a new version of RCU which allows its read-side critical sections to be preempted. It uses a set of counter pairs to keep track of the read-side critical sections and flips them when all tasks exit read-side critical section. The details of this implementation can be found in this paper -
http://www.rdrop.com/users/paulmck/RCU/OLSrtRCU.2006.08.11a.pdf
This patch was developed as a part of the -rt kernel development and meant to provide better latencies when read-side critical sections of RCU don't disable preemption. As a consequence of keeping track of RCU readers, the readers have a slight overhead (optimizations in the paper). This implementation co-exists with the "classic" RCU implementations and can be switched to at compiler.
Also includes RCU tracing summarized in debugfs and RCU_SOFTIRQ for the preemptible variant of RCU.
---
include/linux/interrupt.h | 1 include/linux/rcuclassic.h | 2 include/linux/rcupdate.h | 7 include/linux/rcupreempt.h | 78 +++ include/linux/rcupreempt_trace.h | 100 +++++ include/linux/sched.h | 5 kernel/Kconfig.preempt | 38 + kernel/Makefile | 7 kernel/fork.c | 4 kernel/rcupreempt.c | 768 +++++++++++++++++++++++++++++++++++++++ kernel/rcupreempt_trace.c | 330 ++++++++++++++++ 11 files changed, 1337 insertions(+), 3 deletions(-)
include/linux/interrupt.h +1 -0
@@ -269,6 +269,7 @@ enum#ifdef CONFIG_HIGH_RES_TIMERS HRTIMER_SOFTIRQ , #endif + RCU_SOFTIRQ, /* Preferable RCU should always be the last softirq */ };/* softirq mask and active fields moved to irq_cpustat_t in
include/linux/rcuclassic.h +0 -2
@@ -142,8 +142,6 @@ extern int rcu_needs_cpu(int cpu);extern void __rcu_init (void );extern void rcu_check_callbacks (int cpu,int user);extern void rcu_restart_cpu (int cpu); -extern long rcu_batches_completed (void ); -extern long rcu_batches_completed_bh (void );#endif /* __KERNEL__ */ #endif /* __LINUX_RCUCLASSIC_H */
include/linux/rcupdate.h +7 -0
@@ -52,7 +52,11 @@ struct rcu_head {void (*func)(struct rcu_head *head); }; +#ifdef CONFIG_CLASSIC_RCU #include <linux/rcuclassic.h> +#else /* #ifdef CONFIG_CLASSIC_RCU */ +#include <linux/rcupreempt.h> +#endif /* #else #ifdef CONFIG_CLASSIC_RCU */ #define RCU_HEAD_INIT { .next = NULL, .func = NULL }#define RCU_HEAD (head) struct rcu_head head = RCU_HEAD_INIT
@@ -218,10 +222,13 @@ extern void FASTCALL(call_rcu_bh(struct/* Exported common interfaces */ extern void synchronize_rcu (void );extern void rcu_barrier (void ); +extern long rcu_batches_completed (void ); +extern long rcu_batches_completed_bh (void );/* Internal to kernel */ extern void rcu_init (void );extern void rcu_check_callbacks (int cpu,int user); +extern int rcu_needs_cpu (int cpu);#endif /* __KERNEL__ */ #endif /* __LINUX_RCUPDATE_H */
include/linux/rcupreempt.h +78 -0
78 lines not on this page · raw message
include/linux/rcupreempt_trace.h +100 -0
100 lines not on this page · raw message
include/linux/sched.h +5 -0
@@ -850,6 +850,11 @@ struct task_struct {cpumask_t cpus_allowed;unsigned int time_slice, first_time_slice; +#ifdef CONFIG_PREEMPT_RCU +int rcu_read_lock_nesting; +int rcu_flipctr_idx; +#endif /* #ifdef CONFIG_PREEMPT_RCU */ +#if defined(CONFIG_SCHEDSTATS ) || defined(CONFIG_TASK_DELAY_ACCT )struct sched_info sched_info;#endif
kernel/fork.c +4 -0
@@ -1032,6 +1032,10 @@ static struct task_struct *copy_process(INIT_LIST_HEAD (&p->children);INIT_LIST_HEAD (&p->sibling); +#ifdef CONFIG_PREEMPT_RCU + p->rcu_read_lock_nesting =0 ; + p->rcu_flipctr_idx =0 ; +#endif /* #ifdef CONFIG_PREEMPT_RCU */ p->vfork_done =NULL ;spin_lock_init (&p->alloc_lock);
kernel/Kconfig.preempt +38 -0
@@ -63,3 +63,41 @@ config PREEMPT_BKL Say Y here if you are building a kernel for a desktop system. Say N if you are unsure. +choice + prompt "RCU implementation type:" + default CLASSIC_RCU + +config CLASSIC_RCU + bool "Classic RCU" + help + This option selects the classic RCU implementation that is + designed for best read-side performance on non-realtime + systems. + + Say Y if you are unsure. + +config PREEMPT_RCU + bool "Preemptible RCU" + depends on PREEMPT + help + This option reduces the latency of the kernel by making certain + RCU sections preemptible. Normally RCU code is non-preemptible, if + this option is selected then read-only RCU sections become + preemptible. This helps latency, but may expose bugs due to + now-naive assumptions about each RCU read-side critical section + remaining on a given CPU through its execution. + + Say N if you are unsure. + +endchoice + +config RCU_TRACE + bool "Enable tracing for RCU - currently stats in debugfs" + select DEBUG_FS + default y + help + This option provides tracing in RCU which presents stats + in debugfs for debugging RCU implementation. + + Say Y here if you want to enable RCU tracing + Say N if you are unsure.
kernel/Makefile +6 -1
@@ -6,7 +6,7 @@ obj-y = sched.o fork.o exec_domain.o exit.o itimer.o time.o softirq.o resource.o \ sysctl.o capability.o ptrace.o timer.o user.o \ signal.o sys.o kmod.o workqueue.o pid.o \ - rcupdate.o rcuclassic.o extable.o params.o posix-timers.o \ + rcupdate.o extable.o params.o posix-timers.o \ kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \ hrtimer.o rwsem.o latency.o nsproxy.o srcu.o die_notifier.o
@@ -46,6 +46,11 @@ obj-$(CONFIG_DETECT_SOFTLOCKUP) += softl obj-$(CONFIG_GENERIC_HARDIRQS) += irq/ obj-$(CONFIG_SECCOMP) += seccomp.o obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o +obj-$(CONFIG_CLASSIC_RCU) += rcuclassic.o +obj-$(CONFIG_PREEMPT_RCU) += rcupreempt.o +ifeq ($(CONFIG_PREEMPT_RCU),y) +obj-$(CONFIG_RCU_TRACE) += rcupreempt_trace.o +endif obj-$(CONFIG_RELAY) += relay.o obj-$(CONFIG_SYSCTL) += utsname_sysctl.o obj-$(CONFIG_UTS_NS) += utsname.o
kernel/rcupreempt.c +768 -0
768 lines not on this page · raw message
kernel/rcupreempt_trace.c +330 -0
330 lines not on this page · raw message
Re: [PATCH 3/4 RFC] RCU: preemptible RCU
Quoted 15 lines · Paul E. McKenney
This patch implements a new version of RCU which allows its read-side critical sections to be preempted. It uses a set of counter pairs to keep track of the read-side critical sections and flips them when all tasks exit read-side critical section. The details of this implementation can be found in this paper -
http://www.rdrop.com/users/paulmck/RCU/OLSrtRCU.2006.08.11a.pdf
This patch was developed as a part of the -rt kernel development and meant to provide better latencies when read-side critical sections of RCU don't disable preemption. As a consequence of keeping track of RCU readers, the readers have a slight overhead (optimizations in the paper). This implementation co-exists with the "classic" RCU implementations and can be switched to at compiler.
Quoted 2 lines
Also includes RCU tracing summarized in debugfs and
RCU_SOFTIRQfor the preemptible variant of RCU.
Whickedly complex but very cool stuff!
Unfortunately I have nothing to contribute other than praise at the complex yet elegant way you dodged the need for synchronisation.
Re: [PATCH 3/4 RFC] RCU: preemptible RCU
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Paul E. McKenney
This patch implements a new version of RCU which allows its read-side critical sections to be preempted. It uses a set of counter pairs to keep track of the read-side critical sections and flips them when all tasks exit read-side critical section. The details of this implementation can be found in this paper -
http://www.rdrop.com/users/paulmck/RCU/OLSrtRCU.2006.08.11a.pdf
This patch was developed as a part of the -rt kernel development and meant to provide better latencies when read-side critical sections of RCU don't disable preemption. As a consequence of keeping track of RCU readers, the readers have a slight overhead (optimizations in the paper). This implementation co-exists with the "classic" RCU implementations and can be switched to at compiler.
Quoted 7 lines
Quoted 2 lines
Also includes RCU tracing summarized in debugfs and
RCU_SOFTIRQfor the preemptible variant of RCU.Whickedly complex but very cool stuff!
Unfortunately I have nothing to contribute other than praise at the complex yet elegant way you dodged the need for synchronisation.
I am glad you like it! Heck, I may have to print this email out, frame it, and hang it on my cube wall. ;-)
Thanx, Paul
Re: [PATCH 3/4 RFC] RCU: preemptible RCU
In reply to Paul E. McKenney[ Recipients list trimmed ]
Quoted 6 lines · Paul E. McKenney
This patch was developed as a part of the -rt kernel development and meant to provide better latencies when read-side critical sections of RCU don't disable preemption. As a consequence of keeping track of RCU readers, the readers have a slight overhead (optimizations in the paper). This implementation co-exists with the "classic" RCU implementations and can be switched to at compiler.
I cannot parse the last sentence :-)
Did you mean that this new implementation can be chosen (over the classic implementation) at compile-time?
Regards.
Re: [PATCH 3/4 RFC] RCU: preemptible RCU
Quoted 4 lines · John Sigler
[ Recipients list trimmed ]
Paul E. McKenney wrote:
Quoted 8 lines
Quoted 6 lines
This patch was developed as a part of the -rt kernel development and meant to provide better latencies when read-side critical sections of RCU don't disable preemption. As a consequence of keeping track of RCU readers, the readers have a slight overhead (optimizations in the paper). This implementation co-exists with the "classic" RCU implementations and can be switched to at compiler.
I cannot parse the last sentence :-)
Oops... :-/
Quoted 2 lines
Did you mean that this new implementation can be chosen (over the classic implementation) at compile-time?
Yes, via config variables. CONFIG_CLASSIC_RCU selects the "classic" RCU implementation, and CONFIG_PREEMPT_RCU selects the preemptible RCU. CONFIG_PREEMPT_CPU depends on CONFIG_PREEMPT, so that you only are permitted to specify CONFIG_PREEMPT_RT in preemptible kernels.
Thanx, Paul
RFC PATCH 4/4 RCU: synchronize_sched() without migration
In reply to Paul E. McKenneyThe combination of CPU hotplug and PREEMPT_RCU has resulted in deadlocks due to the migration-based implementation of synchronize_sched() in -rt. This experimental patch maps synchronize_sched() back onto Classic RCU, eliminating the migration, thus hopefully also eliminating the deadlocks. It is not clear that this is a good long-term approach, but it will at least permit people doing CPU hotplug in -rt kernels additional wiggle room in their design and implementation.
The basic approach is to cause the -rt kernel to incorporate rcuclassic.c as well as rcupreempt.c, but to #ifdef out the conflicting portions of rcuclassic.c so that only the code needed to implement synchronize_sched() remains in a PREEMPT_RT build. Invocations of grace-period detection from the scheduling-clock interrupt go to rcuclassic.c, which then invokes the corresponding functions in rcupreempt.c (with _rt suffix added to keep the linker happy). Also applies the RCU_SOFTIRQ to classic RCU. The bulk of this patch just moves code around.
If this patch does turn out to be the right approach, the #ifdefs in kernel/rcuclassic.c might be dealt with. ;-)
---
include/linux/rcuclassic.h | 78 +++++--------------------------------- include/linux/rcupdate.h | 30 ++++++++++++-- include/linux/rcupreempt.h | 27 ++++++------- kernel/Makefile | 2 kernel/rcuclassic.c | 92 ++++++++++++++++++++++++++++++++++++--------- kernel/rcupdate.c | 21 +++++++--- kernel/rcupreempt.c | 50 +++++------------------- 7 files changed, 154 insertions(+), 146 deletions(-)
include/linux/rcuclassic.h +12 -66
@@ -43,79 +43,19 @@#include <linux/seqlock.h> -/* Global control variables for rcupdate callback mechanism. */ -struct rcu_ctrlblk { -long cur;/* Current batch number. */ -long completed;/* Number of the last completed batch */ -int next_pending;/* Is the next batch already waiting? */ - -int signaled; - -spinlock_t lock____cacheline_internodealigned_in_smp ; -cpumask_t cpumask;/* CPUs that need to switch in order */ -/* for current batch to proceed. */ -}____cacheline_internodealigned_in_smp ; - -/* Is batch a before batch b ? */ -static inline int rcu_batch_before (long a,long b) -{ -return (a - b) <0 ; -} - -/* Is batch a after batch b ? */ -static inline int rcu_batch_after (long a,long b) -{ -return (a - b) >0 ; -} - -/* -* Per-CPU data for Read-Copy UPdate. -* nxtlist - new callbacks are added here -* curlist - current batch for which quiescent cycle started if any -*/ -struct rcu_data { -/* 1) quiescent state handling : */ -long quiescbatch;/* Batch # for grace period */ -int passed_quiesc;/* User-mode/idle loop etc. */ -int qs_pending;/* core waits for quiesc state */ - -/* 2) batch handling */ -long batch;/* Batch # for current RCU batch */ -struct rcu_head *nxtlist; -struct rcu_head **nxttail; -long qlen;/* # of queued callbacks */ -struct rcu_head *curlist; -struct rcu_head **curtail; -struct rcu_head *donelist; -struct rcu_head **donetail; -long blimit;/* Upper limit on a processed batch */ -int cpu; -struct rcu_head barrier; -}; - -DECLARE_PER_CPU(struct rcu_data, rcu_data); -DECLARE_PER_CPU(struct rcu_data, rcu_bh_data); +DECLARE_PER_CPU (int, rcu_data_bh_passed_quiesc);/* -* Increment the quiescent state counter. +* Increment the bottom-half quiescent state counter. * The counter is a bit degenerated: We do not need to know * how many quiescent states passed, just if there was at least * one since the start of the grace period. Thus just a flag. */ -static inline void rcu_qsctr_inc (int cpu) -{ -struct rcu_data *rdp = &per_cpu (rcu_data, cpu); - rdp->passed_quiesc =1 ; -}static inline void rcu_bh_qsctr_inc (int cpu) { -struct rcu_data *rdp = &per_cpu (rcu_bh_data, cpu); - rdp->passed_quiesc =1 ; +per_cpu (rcu_data_bh_passed_quiesc, cpu) =1 ; } -extern int rcu_pending (int cpu); -extern int rcu_needs_cpu (int cpu); -#define __rcu_read_lock () \ do { \ preempt_disable(); \
@@ -139,9 +79,15 @@ extern int rcu_needs_cpu(int cpu);#define __synchronize_sched () synchronize_rcu() -extern void __rcu_init (void ); -extern void rcu_check_callbacks (int cpu,int user); -extern void rcu_restart_cpu (int cpu); +#define rcu_advance_callbacks_rt (cpu, user) +#define rcu_check_callbacks_rt (cpu, user) +#define rcu_init_rt () +#define rcu_needs_cpu_rt (cpu) 0 +#define rcu_pending_rt (cpu) 0 +#define rcu_process_callbacks_rt (unused) + +extern void FASTCALL (call_rcu_classic (struct rcu_head *head, +void (*func)(struct rcu_head *head)));#endif /* __KERNEL__ */ #endif /* __LINUX_RCUCLASSIC_H */
include/linux/rcupdate.h +26 -4
@@ -197,8 +197,11 @@ struct rcu_head {* delimited by rcu_read_lock() and rcu_read_unlock(), * and may be nested. */ -extern void FASTCALL (call_rcu (struct rcu_head *head, -void (*func)(struct rcu_head *head))); +#ifdef CONFIG_CLASSIC_RCU +#define call_rcu (head, func) call_rcu_classic(head, func) +#else /* #ifdef CONFIG_CLASSIC_RCU */ +#define call_rcu (head, func) call_rcu_preempt(head, func) +#endif /* #else #ifdef CONFIG_CLASSIC_RCU */ /** * call_rcu_bh - Queue an RCU for invocation after a quicker grace period.
@@ -226,9 +229,28 @@ extern long rcu_batches_completed(void);extern long rcu_batches_completed_bh (void );/* Internal to kernel */ -extern void rcu_init (void );extern void rcu_check_callbacks (int cpu,int user); -extern int rcu_needs_cpu (int cpu); +extern long rcu_batches_completed (void ); +extern long rcu_batches_completed_bh (void ); +extern void rcu_check_callbacks (int cpu,int user); +extern void rcu_init (void ); +extern int rcu_needs_cpu (int cpu); +extern int rcu_pending (int cpu); +struct softirq_action ; +extern void rcu_restart_cpu (int cpu); + +DECLARE_PER_CPU (int, rcu_data_passed_quiesc); + +/* +* Increment the quiescent state counter. +* The counter is a bit degenerated: We do not need to know +* how many quiescent states passed, just if there was at least +* one since the start of the grace period. Thus just a flag. +*/ +static inline void rcu_qsctr_inc (int cpu) +{ +per_cpu (rcu_data_passed_quiesc, cpu) =1 ; +}#endif /* __KERNEL__ */ #endif /* __LINUX_RCUPDATE_H */
include/linux/rcupreempt.h +14 -13
@@ -42,25 +42,26 @@#include <linux/cpumask.h> #include <linux/seqlock.h> -#define rcu_qsctr_inc (cpu) -#define rcu_bh_qsctr_inc (cpu)#define call_rcu_bh (head, rcu) call_rcu(head, rcu) - -extern void __rcu_read_lock (void ); -extern void __rcu_read_unlock (void ); -extern int rcu_pending (int cpu); -extern int rcu_needs_cpu (int cpu); - +#define rcu_bh_qsctr_inc (cpu)#define __rcu_read_lock_bh () { rcu_read_lock(); local_bh_disable(); }#define __rcu_read_unlock_bh () { local_bh_enable(); rcu_read_unlock(); } -#define __rcu_read_lock_nesting () (current->rcu_read_lock_nesting) +extern void FASTCALL (call_rcu_classic (struct rcu_head *head, +void (*func)(struct rcu_head *head))); +extern void FASTCALL (call_rcu_preempt (struct rcu_head *head, +void (*func)(struct rcu_head *head))); +extern void __rcu_read_lock (void ); +extern void __rcu_read_unlock (void );extern void __synchronize_sched (void ); - -extern void __rcu_init (void ); -extern void rcu_check_callbacks (int cpu,int user); -extern void rcu_restart_cpu (int cpu); +extern void rcu_advance_callbacks_rt (int cpu,int user); +extern void rcu_check_callbacks_rt (int cpu,int user); +extern void rcu_init_rt (void ); +extern int rcu_needs_cpu_rt (int cpu); +extern int rcu_pending_rt (int cpu); +struct softirq_action ; +extern void rcu_process_callbacks_rt (struct softirq_action *unused);#ifdef CONFIG_RCU_TRACE struct rcupreempt_trace ;
kernel/Makefile +1 -1
@@ -47,7 +47,7 @@ obj-$(CONFIG_GENERIC_HARDIRQS) += irq/ obj-$(CONFIG_SECCOMP) += seccomp.o obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o obj-$(CONFIG_CLASSIC_RCU) += rcuclassic.o -obj-$(CONFIG_PREEMPT_RCU) += rcupreempt.o +obj-$(CONFIG_PREEMPT_RCU) += rcuclassic.o rcupreempt.o ifeq ($(CONFIG_PREEMPT_RCU),y) obj-$(CONFIG_RCU_TRACE) += rcupreempt_trace.o endif
kernel/rcuclassic.c +74 -18
207 lines not on this page · raw message
kernel/rcupdate.c +16 -5
@@ -52,6 +52,7 @@ struct rcu_synchronize {struct completion completion; }; +DEFINE_PER_CPU (int, rcu_data_passed_quiesc); static DEFINE_PER_CPU(struct rcu_head, rcu_barrier_head) = {NULL};static atomic_t rcu_barrier_cpu_count;static DEFINE_MUTEX (rcu_barrier_mutex);
@@ -87,6 +88,21 @@ void synchronize_rcu(void)wait_for_completion (&rcu.completion); } +#ifdef CONFIG_PREEMPT_RCU + +/* +* Map synchronize_sched() to the classic RCU implementation. +*/ +void __synchronize_sched (void ) +{ +struct rcu_synchronize rcu; + +init_completion (&rcu.completion); +call_rcu_classic (&rcu.head, wakeme_after_rcu); +wait_for_completion (&rcu.completion); +} +#endif /* #ifdef CONFIG_PREEMPT_RCU */ +static void rcu_barrier_callback (struct rcu_head *notused) {if (atomic_dec_and_test (&rcu_barrier_cpu_count))
@@ -130,10 +146,5 @@ void rcu_barrier(void)mutex_unlock (&rcu_barrier_mutex); } -void __init rcu_init (void ) -{ -__rcu_init (); -} -EXPORT_SYMBOL_GPL (rcu_barrier);EXPORT_SYMBOL_GPL (synchronize_rcu);
kernel/rcupreempt.c +11 -39
121 lines not on this page · raw message
Re: [PATCH 4/4 RFC] RCU: synchronize_sched() without migration
Quoted 17 lines · Paul E. McKenney
The combination of CPU hotplug and
PREEMPT_RCUhas resulted in deadlocks due to the migration-based implementation ofsynchronize_sched()in -rt. This experimental patch mapssynchronize_sched()back onto Classic RCU, eliminating the migration, thus hopefully also eliminating the deadlocks. It is not clear that this is a good long-term approach, but it will at least permit people doing CPU hotplug in -rt kernels additional wiggle room in their design and implementation.The basic approach is to cause the -rt kernel to incorporate
rcuclassic.cas well asrcupreempt.c, but to #ifdef out the conflicting portions ofrcuclassic.cso that only the code needed to implementsynchronize_sched()remains in aPREEMPT_RTbuild. Invocations of grace-period detection from the scheduling-clock interrupt go torcuclassic.c, which then invokes the corresponding functions inrcupreempt.c(with_rtsuffix added to keep the linker happy). Also applies theRCU_SOFTIRQto classic RCU. The bulk of this patch just moves code around.
Quoted 2 lines
If this patch does turn out to be the right approach, the #ifdefs in
kernel/rcuclassic.cmight be dealt with. ;-)
I think the right thing to do would be to fix cpu hotplug to use a simple reference count, as the latest patches Gautham is working on. That simplifies the implementation from the earlier version with a per-cpu reference counter with RCU. No freezer, no lock.
Thanks
Dipankar
Re: [PATCH 4/4 RFC] RCU: synchronize_sched() without migration
Quoted 18 lines · Dipankar Sarma
Quoted 17 lines · Paul E. McKenney
The combination of CPU hotplug and
PREEMPT_RCUhas resulted in deadlocks due to the migration-based implementation ofsynchronize_sched()in -rt. This experimental patch mapssynchronize_sched()back onto Classic RCU, eliminating the migration, thus hopefully also eliminating the deadlocks. It is not clear that this is a good long-term approach, but it will at least permit people doing CPU hotplug in -rt kernels additional wiggle room in their design and implementation.The basic approach is to cause the -rt kernel to incorporate
rcuclassic.cas well asrcupreempt.c, but to #ifdef out the conflicting portions ofrcuclassic.cso that only the code needed to implementsynchronize_sched()remains in aPREEMPT_RTbuild. Invocations of grace-period detection from the scheduling-clock interrupt go torcuclassic.c, which then invokes the corresponding functions inrcupreempt.c(with_rtsuffix added to keep the linker happy). Also applies theRCU_SOFTIRQto classic RCU. The bulk of this patch just moves code around.
Quoted 8 lines
Quoted 2 lines
If this patch does turn out to be the right approach, the #ifdefs in
kernel/rcuclassic.cmight be dealt with. ;-)I think the right thing to do would be to fix cpu hotplug to use a simple reference count, as the latest patches Gautham is working on. That simplifies the implementation from the earlier version with a per-cpu reference counter with RCU. No freezer, no lock.
Works for me -- I will just keep this patch for testing of the CPU hotplug stuff I am adding to preemptible RCU. Speaking of which, any advise on good testing methods for CPU hotplug would be greatly appreciated!!!
Thanx, Paul