PREEMPT/PREEMPT_RT question
Subsystems (3)
Hello!
OK, counter-flip RCU actually survives a pair of overnight runs on CONFIG_PREEMPT running on 4-CPU machines, and also survives five kernbenches and an LTP on another 4-CPU machine. (Overnight-run script later in this message, FWIW.)
So, time to get serious about a bit of code cleanup:
o The heavyweight atomic operations in rcu_read_lock() and
rcu_read_unlock() are not needed in UP kernels, since interrupts are disabled.
Is there already something like smp_atomic_inc() and smp_atomic_dec() that generate atomic_inc()/atomic_dec() in SMP kernels, but ++/-- in UP kernels? If not, any reasons not to add them, for example, as follows?
#ifdef CONFIG_SMP
#define smp_atomic_inc(v) atomic_inc(v) #define smp_atomic_dec(v) atomic_dec(v) #else /* #ifdef CONFIG_SMP */ #define smp_atomic_inc(v) ((v)++) #define smp_atomic_dec(v) ((v)++) #endif /* #else #ifdef CONFIG_SMP */
Since interrupts must be disabled for these to be safe, my guess is that I should define them locally in rcupdate.c. If there turns out to be a general need for them, they can be moved somewhere more public.
Objections?
o In order to get things to work in both CONFIG_PREEMPT and
CONFIG_PREEMPT_RT, I ended up using the following: #ifdef CONFIG_PREEMPT_RT #define rcu_spinlock_t _raw_spinlock_t #define rcu_spin_lock(l, f) _raw_spin_lock(l) #define rcu_spin_trylock(l, f) _raw_spin_trylock(l) #define rcu_spin_unlock(l, f) _raw_spin_unlock(l) #define RCU_SPIN_LOCK_UNLOCKED RAW_SPIN_LOCK_UNLOCKED #else /* #ifdef CONFIG_PREEMPT_RT */ #define rcu_spinlock_t spinlock_t #define rcu_spin_lock(l, f) spin_lock_irqsave(l, f) #define rcu_spin_trylock(l, f) spin_trylock_irqsave(l, f) #define rcu_spin_unlock(l, f) spin_unlock_irqrestore(l, f) #define RCU_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED #endif /* #else #ifdef CONFIG_PREEMPT_RT */ Then using rcu_spin_lock() &c everywhere. The problem is that (as near as I can tell) the only way to prevent interrupts from running on the current CPU in CONFIG_PREEMPT kernels is to use the _irq spinlock primitives, but _raw_spin_lock() does the job in CONFIG_PREEMPT_RT (since interrupts are run in process context, right). I could use _irq in both, but that would unnecessarily degrade interrupt latency in CONFIG_PREEMPT_RT. Suggestions???
Some remaining shortcomings of the current code:
o Untested on CONFIG_PREEMPT_RT (working on this, suggestions
for 4-CPU-stable CONFIG_PREEMPT_RT versions most welcome).
o Uses atomic operations and memory barriers in the
rcu_read_lock() and rcu_read_unlock() fastpaths. Getting rid of the atomic operations in UP is easy, in SMP in the common case is reasonably straightforward, getting rid of memory barriers is harder, but is likely doable.
o Global callback management slows things down on SMP
(need to move to per-CPU callback management).
o Gross debug code needs pruning (working on this).
o Current implementation is probably too aggressive about
finishing out grace periods -- one of the machines averaged one grace period every 29 milliseconds over a 15-hour time period. This likely understates the aggressiveness, since the test is quite bursty.
o Given the small amount testing, there are undoubtably
more bugs.
All that aside, only a moderately severe degree of insanity is required to run this code.
Improvements since last time:
o Boots and runs reasonably reliably.
o Got rid of the workqueue stuff, simplifying grace-period
detection code. Also allows synchronize_sched() to be common code between "classic" and realtime RCU implementations.
o Enabled preemption in RCU read-side critical sections
(even in CONFIG_PREEMPT kernels).
Test script (assumes that you have a linux-2.6.11.tar.bz2 file in the current directory), relies on the fact that dcache's heavy use of RCU:
#!/bin/sh while : do i=0 while expr $i '<' 10 do tar -xjf linux-2.6.11.tar.bz2 echo tar status $? mv linux-2.6.11 argh$i echo $i : `grep dentry_cache /proc/slabinfo` i=`expr $i + 1` done for i in argh[0-9] do rm -rf $i & done wait echo rm : `grep dentry_cache /proc/slabinfo` done
Current (ugly) patch:
arch/i386/Kconfig +25 -0
@@ -523,6 +523,31 @@ config PREEMPT Say Y here if you are building a kernel for a desktop, embedded or real-time system. Say N if you are unsure. +config PREEMPT_RCU + bool "Preemptible RCU read-side critical sections" + depends on PREEMPT + default y + help + This option reduces the latency of the kernel when reacting to + real-time or interactive events by allowing a low priority process to + be preempted even if it is in an RCU read-side critical section. + This allows applications to run more reliably even when the system is + under load. + + Say Y here if you enjoy debugging random oopses. + Say N if, for whatever reason, you want your kernel to actually work. + +config RCU_STATS + bool "/proc stats for preemptible RCU read-side critical sections" + depends on PREEMPT_RCU + default y + help + This option provides /proc stats to provide debugging info for + the preemptible realtime RCU implementation. + + Say Y here if you want to see RCU stats in /proc + Say N if you are unsure. + config PREEMPT_BKL bool "Preempt The Big Kernel Lock" depends on PREEMPT
fs/proc/proc_misc.c +37 -0
@@ -549,6 +549,38 @@ void create_seq_entry(char *name, mode_t entry->proc_fops = f; } +#ifdef CONFIG_RCU_STATS +int rcu_read_proc (char *page,char **start,off_t off, +int count,int *eof,void *data) +{ +int len; +extern int rcu_read_proc_data (char *page); + + len =rcu_read_proc_data (page); +return proc_calc_metrics (page, start, off, count, eof, len); +} + +int rcu_read_proc_gp (char *page,char **start,off_t off, +int count,int *eof,void *data) +{ +int len; +extern int rcu_read_proc_gp_data (char *page); + + len =rcu_read_proc_gp_data (page); +return proc_calc_metrics (page, start, off, count, eof, len); +} + +int rcu_read_proc_ptrs (char *page,char **start,off_t off, +int count,int *eof,void *data) +{ +int len; +extern int rcu_read_proc_ptrs_data (char *page); + + len =rcu_read_proc_ptrs_data (page); +return proc_calc_metrics (page, start, off, count, eof, len); +} +#endif /* #ifdef CONFIG_RCU_STATS */ +void __init proc_misc_init (void ) {struct proc_dir_entry *entry;
@@ -571,6 +603,11 @@ void __init proc_misc_init(void) {"cmdline", cmdline_read_proc}, {"locks", locks_read_proc}, {"execdomains", execdomains_read_proc}, +#ifdef CONFIG_RCU_STATS + {"rcustats", rcu_read_proc}, + {"rcugp", rcu_read_proc_gp}, + {"rcuptrs", rcu_read_proc_ptrs}, +#endif /* #ifdef CONFIG_RCU_STATS */ {NULL,} };for (p = simple_ones; p->name; p++)
include/linux/rcupdate.h +25 -0
@@ -59,6 +59,7 @@ struct rcu_head { } while (0) +#ifndef CONFIG_PREEMPT_RCU /* Global control variables for rcupdate callback mechanism. */ struct rcu_ctrlblk {
@@ -192,6 +193,18 @@ static inline int rcu_pending(int cpu)*/ #define rcu_read_unlock () preempt_enable() +#else /* #ifndef CONFIG_PREEMPT_RCU */ + +#define rcu_qsctr_inc (cpu) +#define rcu_bh_qsctr_inc (cpu) +#define call_rcu_bh (head, rcu) call_rcu(head, rcu) + +extern void rcu_read_lock (void ); +extern void rcu_read_unlock (void ); +extern int rcu_pending (int cpu); + +#endif /* #else #ifndef CONFIG_PREEMPT_RCU */ +/* * So where is rcu_write_lock()? It does not exist, as there is no * way for writers to lock out RCU readers. This is a feature, not
@@ -213,14 +226,22 @@ static inline int rcu_pending(int cpu)* can use just rcu_read_lock(). * */ +#ifndef CONFIG_PREEMPT_RCU #define rcu_read_lock_bh () local_bh_disable() +#else /* #ifndef CONFIG_PREEMPT_RCU */ +#define rcu_read_lock_bh () { rcu_read_lock(); local_bh_disable(); } +#endif /* #else #ifndef CONFIG_PREEMPT_RCU */ /* * rcu_read_unlock_bh - marks the end of a softirq-only RCU critical section * * See rcu_read_lock_bh() for more information. */ +#ifndef CONFIG_PREEMPT_RCU #define rcu_read_unlock_bh () local_bh_enable() +#else /* #ifndef CONFIG_PREEMPT_RCU */ +#define rcu_read_unlock_bh () { local_bh_enable(); rcu_read_unlock(); } +#endif /* #else #ifndef CONFIG_PREEMPT_RCU */ /** * rcu_dereference - fetch an RCU-protected pointer in an
@@ -269,7 +290,11 @@ static inline int rcu_pending(int cpu)* synchronize_kernel() API. In contrast, synchronize_rcu() only * guarantees that rcu_read_lock() sections will have completed. */ +#ifndef CONFIG_PREEMPT_RCU #define synchronize_sched () synchronize_rcu() +#else /* #ifndef CONFIG_PREEMPT_RCU */ +extern void synchronize_sched (void ); +#endif /* #else #ifndef CONFIG_PREEMPT_RCU */ extern void rcu_init (void );extern void rcu_check_callbacks (int cpu,int user);
include/linux/sched.h +5 -0
@@ -740,6 +740,11 @@ struct task_struct {nodemask_t mems_allowed;int cpuset_mems_generation;#endif +#ifdef CONFIG_PREEMPT_RCU +int rcu_read_lock_nesting; +atomic_t *rcu_flipctr1; +atomic_t *rcu_flipctr2; +#endif };static inline pid_t process_group (struct task_struct *tsk)
kernel/rcupdate.c +479 -29
538 lines not on this page · raw message
Steven Rostedt
reviewerQuoted 10 lines · Paul E. McKenney
Hello!
OK, counter-flip RCU actually survives a pair of overnight runs on
CONFIG_PREEMPTrunning on 4-CPU machines, and also survives five kernbenches and an LTP on another 4-CPU machine. (Overnight-run script later in this message, FWIW.)So, time to get serious about a bit of code cleanup:
o The heavyweight atomic operations in
rcu_read_lock()and
Quoted 15 lines
rcu_read_unlock() are not needed in UP kernels, since interrupts are disabled. Is there already something like smp_atomic_inc() and smp_atomic_dec() that generate atomic_inc()/atomic_dec() in SMP kernels, but ++/-- in UP kernels? If not, any reasons not to add them, for example, as follows? #ifdef CONFIG_SMP #define smp_atomic_inc(v) atomic_inc(v) #define smp_atomic_dec(v) atomic_dec(v) #else /* #ifdef CONFIG_SMP */ #define smp_atomic_inc(v) ((v)++) #define smp_atomic_dec(v) ((v)++) #endif /* #else #ifdef CONFIG_SMP */
What's the problem with atomic_inc? At least on x86, atomic inc is defined as:
static __inline__ void atomic_inc(atomic_t *v)
{
__asm__ __volatile__(
LOCK "incl %0"
:"=m" (v->counter)
:"m" (v->counter));
}With LOCK defined as:
#ifdef CONFIG_SMP
#define LOCK "lock ; "
#else
#define LOCK ""
#endif
So is there a difference on UP between x.counter++ and atomic_inc(&x)?
Quoted 28 lines
Since interrupts must be disabled for these to be safe, my guess is that I should define them locally in
rcupdate.c. If there turns out to be a general need for them, they can be moved somewhere more public.Objections?
o In order to get things to work in both
CONFIG_PREEMPTandCONFIG_PREEMPT_RT, I ended up using the following: #ifdef CONFIG_PREEMPT_RT #define rcu_spinlock_t _raw_spinlock_t #define rcu_spin_lock(l, f) _raw_spin_lock(l) #define rcu_spin_trylock(l, f) _raw_spin_trylock(l) #define rcu_spin_unlock(l, f) _raw_spin_unlock(l) #define RCU_SPIN_LOCK_UNLOCKED RAW_SPIN_LOCK_UNLOCKED #else /* #ifdef CONFIG_PREEMPT_RT */ #define rcu_spinlock_t spinlock_t #define rcu_spin_lock(l, f) spin_lock_irqsave(l, f) #define rcu_spin_trylock(l, f) spin_trylock_irqsave(l, f) #define rcu_spin_unlock(l, f) spin_unlock_irqrestore(l, f) #define RCU_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED #endif /* #else #ifdef CONFIG_PREEMPT_RT */
Quoted 7 lines
Then using rcu_spin_lock() &c everywhere. The problem is that (as near as I can tell) the only way to prevent interrupts from running on the current CPU in CONFIG_PREEMPT kernels is to use the _irq spinlock primitives, but _raw_spin_lock() does the job in CONFIG_PREEMPT_RT (since interrupts are run in process context, right). I could use _irq in both, but that would unnecessarily degrade interrupt latency in CONFIG_PREEMPT_RT.
Yep interrupts are threads in CONFIG_PREEMPT_RT. I guess you could also just use local_irq_save with spin_lock, since now local_irq_save no longer disables interrupts in PREEMPT_RT.
-- Steve
Quoted 40 lines · Steven Rostedt
Quoted 10 lines · Paul E. McKenney
Hello!
OK, counter-flip RCU actually survives a pair of overnight runs on
CONFIG_PREEMPTrunning on 4-CPU machines, and also survives five kernbenches and an LTP on another 4-CPU machine. (Overnight-run script later in this message, FWIW.)So, time to get serious about a bit of code cleanup:
o The heavyweight atomic operations in
rcu_read_lock()andQuoted 15 lines
rcu_read_unlock() are not needed in UP kernels, since interrupts are disabled. Is there already something like smp_atomic_inc() and smp_atomic_dec() that generate atomic_inc()/atomic_dec() in SMP kernels, but ++/-- in UP kernels? If not, any reasons not to add them, for example, as follows? #ifdef CONFIG_SMP #define smp_atomic_inc(v) atomic_inc(v) #define smp_atomic_dec(v) atomic_dec(v) #else /* #ifdef CONFIG_SMP */ #define smp_atomic_inc(v) ((v)++) #define smp_atomic_dec(v) ((v)++) #endif /* #else #ifdef CONFIG_SMP */What's the problem with
atomic_inc? At least on x86, atomic inc is defined as:static
__inline__voidatomic_inc(atomic_t *v){ __asm__ __volatile__( LOCK "incl %0" :"=m" (v->counter) :"m" (v->counter)); }With LOCK defined as:
Quoted 7 lines
#ifdef
CONFIG_SMP
#define LOCK "lock ; "
#else
#define LOCK ""
#endifSo is there a difference on UP between x.counter++ and
atomic_inc(&x)?
On x86, you are right. The full list of architectures that are atomic only in SMP are i386 (as you noted), parisc, sparc, and x86_64.
The architectures that appear to always be atomic are: alpha, ia64, m32r (but unfamiliar with this one), mips, ppc, ppc64, s390 (I think...), and sparc64. Most of these architectures need to disable interrupts to provide "universal" atomic_inc() semantics in UP kernels, due to their RISC-style atomic instructions.
The following architectures avoid the issue entirely by refusing to support SMP: arm, arm26, cris, frv, h8300, m68k, m68knommu, sh, sh64, and v850. Many of them disable interrupts in their atomic_inc() implementations.
The advantage of smp_atomic_inc() is that architectures could dispense with interrupt disabling. The disadvantage is that it is yet another contribution to Linux's combinatorial explosion of primitives.
For the moment, I will grit my teeth and keep atomic_inc() and atomic_dec().
Other thoughts?
Quoted 36 lines
Quoted 28 lines
Since interrupts must be disabled for these to be safe, my guess is that I should define them locally in
rcupdate.c. If there turns out to be a general need for them, they can be moved somewhere more public.Objections?
o In order to get things to work in both
CONFIG_PREEMPTandCONFIG_PREEMPT_RT, I ended up using the following: #ifdef CONFIG_PREEMPT_RT #define rcu_spinlock_t _raw_spinlock_t #define rcu_spin_lock(l, f) _raw_spin_lock(l) #define rcu_spin_trylock(l, f) _raw_spin_trylock(l) #define rcu_spin_unlock(l, f) _raw_spin_unlock(l) #define RCU_SPIN_LOCK_UNLOCKED RAW_SPIN_LOCK_UNLOCKED #else /* #ifdef CONFIG_PREEMPT_RT */ #define rcu_spinlock_t spinlock_t #define rcu_spin_lock(l, f) spin_lock_irqsave(l, f) #define rcu_spin_trylock(l, f) spin_trylock_irqsave(l, f) #define rcu_spin_unlock(l, f) spin_unlock_irqrestore(l, f) #define RCU_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED #endif /* #else #ifdef CONFIG_PREEMPT_RT */Quoted 7 lines
Then using rcu_spin_lock() &c everywhere. The problem is that (as near as I can tell) the only way to prevent interrupts from running on the current CPU in CONFIG_PREEMPT kernels is to use the _irq spinlock primitives, but _raw_spin_lock() does the job in CONFIG_PREEMPT_RT (since interrupts are run in process context, right). I could use _irq in both, but that would unnecessarily degrade interrupt latency in CONFIG_PREEMPT_RT.
Quoted 3 lines
Yep interrupts are threads in
CONFIG_PREEMPT_RT. I guess you could also just uselocal_irq_savewithspin_lock, since nowlocal_irq_saveno longer disables interrupts inPREEMPT_RT.
By this you mean the following?
local_irq_save(flags); _raw_spin_lock(&mylock); /* critical section */ _raw_spin_unlock(&mylock); local_irq_restore(flags); Thanx, Paul
Steven Rostedt
reviewerQuoted 17 lines · Paul E. McKenney
Quoted 2 lines
So is there a difference on UP between x.counter++ and
atomic_inc(&x)?On x86, you are right. The full list of architectures that are atomic only in SMP are i386 (as you noted), parisc, sparc, and
x86_64.The architectures that appear to always be atomic are: alpha, ia64, m32r (but unfamiliar with this one), mips, ppc, ppc64, s390 (I think...), and sparc64. Most of these architectures need to disable interrupts to provide "universal"
atomic_inc()semantics in UP kernels, due to their RISC-style atomic instructions.The following architectures avoid the issue entirely by refusing to support SMP: arm, arm26, cris, frv, h8300, m68k, m68knommu, sh, sh64, and v850. Many of them disable interrupts in their
atomic_inc()implementations.
Quoted 7 lines
The advantage of
smp_atomic_inc()is that architectures could dispense with interrupt disabling. The disadvantage is that it is yet another contribution to Linux's combinatorial explosion of primitives.For the moment, I will grit my teeth and keep
atomic_inc()andatomic_dec().Other thoughts?
How did I know that you would mention mips and the like :-)
Yeah, mips has the crazy Load Linked and Store Conditional crap, so it is a little more complex than the simple add one. And I think PPC does too, although it has been a while since I've used them. And older mips don't have the LL SC command so the only option is to turn off interrupts (of course those that don't have the LL and SC are not SMP compatible). So, I will admit that having a smp_atomic_inc might be nice. I was just being a narrow minded x86 hacker ;-)
Quoted 6 lines
Quoted 3 lines
Yep interrupts are threads in
CONFIG_PREEMPT_RT. I guess you could also just uselocal_irq_savewithspin_lock, since nowlocal_irq_saveno longer disables interrupts inPREEMPT_RT.By this you mean the following?
Quoted 7 lines
local_irq_save(flags); _raw_spin_lock(&mylock); /* critical section */ _raw_spin_unlock(&mylock); local_irq_restore(flags);
Yeah, that on PREEMP_RT would not turn off interrupts but just stops preemption. This is fine as long as the mylock is not used in any SA_NODELAY interrupt.
-- Steve
Quoted 28 lines · Steven Rostedt
Quoted 17 lines · Paul E. McKenney
Quoted 2 lines
So is there a difference on UP between x.counter++ and
atomic_inc(&x)?On x86, you are right. The full list of architectures that are atomic only in SMP are i386 (as you noted), parisc, sparc, and
x86_64.The architectures that appear to always be atomic are: alpha, ia64, m32r (but unfamiliar with this one), mips, ppc, ppc64, s390 (I think...), and sparc64. Most of these architectures need to disable interrupts to provide "universal"
atomic_inc()semantics in UP kernels, due to their RISC-style atomic instructions.The following architectures avoid the issue entirely by refusing to support SMP: arm, arm26, cris, frv, h8300, m68k, m68knommu, sh, sh64, and v850. Many of them disable interrupts in their
atomic_inc()implementations.Quoted 7 lines
The advantage of
smp_atomic_inc()is that architectures could dispense with interrupt disabling. The disadvantage is that it is yet another contribution to Linux's combinatorial explosion of primitives.For the moment, I will grit my teeth and keep
atomic_inc()andatomic_dec().Other thoughts?
How did I know that you would mention mips and the like :-)
Quoted 7 lines
Yeah, mips has the crazy Load Linked and Store Conditional crap, so it is a little more complex than the simple add one. And I think PPC does too, although it has been a while since I've used them. And older mips don't have the LL SC command so the only option is to turn off interrupts (of course those that don't have the LL and SC are not SMP compatible). So, I will admit that having a
smp_atomic_incmight be nice. I was just being a narrow minded x86 hacker ;-)
I am sure that LL/SC seemed like a good idea at the time. ;-)
To be fair, LL/SC does allow allow some things to be done more easily than with cmpxchg, since it allows you to tell that the value changed even if it later changed back. Helps with some linked-list operations.
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines
Yep interrupts are threads in
CONFIG_PREEMPT_RT. I guess you could also just uselocal_irq_savewithspin_lock, since nowlocal_irq_saveno longer disables interrupts inPREEMPT_RT.By this you mean the following?
local_irq_save(flags); _raw_spin_lock(&mylock);
Quoted 8 lines
Quoted 4 lines
/* critical section */ _raw_spin_unlock(&mylock); local_irq_restore(flags);Yeah, that on
PREEMP_RTwould not turn off interrupts but just stops preemption. This is fine as long as the mylock is not used in anySA_NODELAYinterrupt.
Cool! Is the scheduling-clock interrupt an SA_NODELAY interrupt?
Thanx, Paul
Steven Rostedt
reviewerQuoted 9 lines · Paul E. McKenney
Quoted 8 lines
Yeah, mips has the crazy Load Linked and Store Conditional crap, so it is a little more complex than the simple add one. And I think PPC does too, although it has been a while since I've used them. And older mips don't have the LL SC command so the only option is to turn off interrupts (of course those that don't have the LL and SC are not SMP compatible). So, I will admit that having a
smp_atomic_incmight be nice. I was just being a narrow minded x86 hacker ;-)
Quoted 5 lines
I am sure that LL/SC seemed like a good idea at the time. ;-)
To be fair, LL/SC does allow allow some things to be done more easily than with cmpxchg, since it allows you to tell that the value changed even if it later changed back. Helps with some linked-list operations.
I was being a little harsh in my statements. I didn't really mind the LL and SC but a true atomic inc would have been nice. I actually had to port Linux to a MIPS board once that didn't have the LL or SC, and that was even more painful.
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Yep interrupts are threads in
CONFIG_PREEMPT_RT. I guess you could also just uselocal_irq_savewithspin_lock, since nowlocal_irq_saveno longer disables interrupts inPREEMPT_RT.By this you mean the following?
local_irq_save(flags); _raw_spin_lock(&mylock); /* critical section */
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
_raw_spin_unlock(&mylock); local_irq_restore(flags);Yeah, that on
PREEMP_RTwould not turn off interrupts but just stops preemption. This is fine as long as the mylock is not used in anySA_NODELAYinterrupt.Cool! Is the scheduling-clock interrupt an
SA_NODELAYinterrupt?
If you are talking about scheduler_tick, then yes, it is called by the timer interrupt which is a SA_NODELAY interrupt. If you don't want to get interrupted by the timer interrupt, then you will need to disable interrupts for both. Since currently, the timer interrupt is the only true hard interrupt in the PREEMPT_RT and that may not change.
-- Steve
Quoted 11 lines · Steven Rostedt
Quoted 10 lines · Paul E. McKenney
Quoted 7 lines
Yeah, mips has the crazy Load Linked and Store Conditional crap, so it is a little more complex than the simple add one. And I think PPC does too, although it has been a while since I've used them. And older mips don't have the LL SC command so the only option is to turn off interrupts (of course those that don't have the LL and SC are not SMP compatible). So, I will admit that having a
smp_atomic_incmight be nice. I was just being a narrow minded x86 hacker ;-)I am sure that LL/SC seemed like a good idea at the time. ;-)
Quoted 8 lines
Quoted 3 lines
To be fair, LL/SC does allow allow some things to be done more easily than with cmpxchg, since it allows you to tell that the value changed even if it later changed back. Helps with some linked-list operations.
I was being a little harsh in my statements. I didn't really mind the LL and SC but a true atomic inc would have been nice. I actually had to port Linux to a MIPS board once that didn't have the LL or SC, and that was even more painful.
I could believe that!
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines
Quoted 3 lines
Yep interrupts are threads in
CONFIG_PREEMPT_RT. I guess you could also just uselocal_irq_savewithspin_lock, since nowlocal_irq_saveno longer disables interrupts inPREEMPT_RT.By this you mean the following?
Quoted 7 lines
local_irq_save(flags); _raw_spin_lock(&mylock); /* critical section */ _raw_spin_unlock(&mylock); local_irq_restore(flags);Yeah, that on
PREEMP_RTwould not turn off interrupts but just stops preemption. This is fine as long as the mylock is not used in anySA_NODELAYinterrupt.
Quoted 7 lines
Quoted 1 line
Cool! Is the scheduling-clock interrupt an
SA_NODELAYinterrupt?If you are talking about
scheduler_tick, then yes, it is called by the timer interrupt which is aSA_NODELAYinterrupt. If you don't want to get interrupted by the timer interrupt, then you will need to disable interrupts for both. Since currently, the timer interrupt is the only true hard interrupt in thePREEMPT_RTand that may not change.
OK, so if I take a spinlock in something invoked from scheduler_tick(), then any other acquisitions of that spinlock must disable hardware interrupts, right?
Thanx, Paul
Steven Rostedt
reviewerQuoted 9 lines · Paul E. McKenney
Quoted 5 lines
If you are talking about
scheduler_tick, then yes, it is called by the timer interrupt which is aSA_NODELAYinterrupt. If you don't want to get interrupted by the timer interrupt, then you will need to disable interrupts for both. Since currently, the timer interrupt is the only true hard interrupt in thePREEMPT_RTand that may not change.OK, so if I take a spinlock in something invoked from
scheduler_tick(), then any other acquisitions of that spinlock must disable hardware interrupts, right?
Yes, otherwise you could have a local CPU deadlock on a SMP machine. And I would also say the same is true for any lock that is grabbed by the timer interrupt or one of the functions it calls.
-- Steve
Quoted 8 lines · Steven Rostedt
Quoted 6 lines · Paul E. McKenney
Quoted 5 lines
If you are talking about
scheduler_tick, then yes, it is called by the timer interrupt which is aSA_NODELAYinterrupt. If you don't want to get interrupted by the timer interrupt, then you will need to disable interrupts for both. Since currently, the timer interrupt is the only true hard interrupt in thePREEMPT_RTand that may not change.
Quoted 7 lines
Quoted 3 lines
OK, so if I take a spinlock in something invoked from
scheduler_tick(), then any other acquisitions of that spinlock must disable hardware interrupts, right?Yes, otherwise you could have a local CPU deadlock on a SMP machine. And I would also say the same is true for any lock that is grabbed by the timer interrupt or one of the functions it calls.
OK, I do indeed have critical sections spanning rcu_check_callbacks() and process-level code. Since rcu_check_callbacks() is called from account_user_vtime() and update_process_times(), both of which call scheduler_tick(), looks like I need to disable hardware interrupts.
Looks to me like I need to use _raw_spin_lock() and _raw_spin_unlock() from within rcu_check_callbacks(), but to instead use _raw_spin_lock_irqsave() and _raw_spin_lock_irqrestore() outside of rcu_check_callbacks() for locks acquired within rcu_check_callbacks().
Of course, for fliplock, which is only acquired conditionally from a rcu_check_callbacks() callee, I can just use spin_trylock() and spin_unlock(). Though _raw_spin_lock()/_raw_spin_unlock() might be faster?
No -wonder- I have been seeing hangs in CONFIG_PREEMPT_RT!!! ;-)
Thanx, Paul
Quoted 5 lines · Paul E. McKenney
Hello!
OK, counter-flip RCU actually survives a pair of overnight runs on
CONFIG_PREEMPTrunning on 4-CPU machines, and also survives five kernbenches and an LTP on another 4-CPU machine. [...]
cool!
Quoted 5 lines
So, time to get serious about a bit of code cleanup:
o The heavyweight atomic operations in
rcu_read_lock()and
rcu_read_unlock()are not needed in UP kernels, since interrupts are disabled.
atomic_*() ops should already be lightweight on UP.
Quoted 22 lines
o In order to get things to work in both
CONFIG_PREEMPTandCONFIG_PREEMPT_RT, I ended up using the following: #ifdef CONFIG_PREEMPT_RT #define rcu_spinlock_t _raw_spinlock_t #define rcu_spin_lock(l, f) _raw_spin_lock(l) #define rcu_spin_trylock(l, f) _raw_spin_trylock(l) #define rcu_spin_unlock(l, f) _raw_spin_unlock(l) #define RCU_SPIN_LOCK_UNLOCKED RAW_SPIN_LOCK_UNLOCKED #else /* #ifdef CONFIG_PREEMPT_RT */ #define rcu_spinlock_t spinlock_t #define rcu_spin_lock(l, f) spin_lock_irqsave(l, f) #define rcu_spin_trylock(l, f) spin_trylock_irqsave(l, f) #define rcu_spin_unlock(l, f) spin_unlock_irqrestore(l, f) #define RCU_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED #endif /* #else #ifdef CONFIG_PREEMPT_RT */ Then using rcu_spin_lock() &c everywhere. The problem is
Quoted 8 lines
that (as near as I can tell) the only way to prevent interrupts from running on the current CPU in CONFIG_PREEMPT kernels is to use the _irq spinlock primitives, but _raw_spin_lock() does the job in CONFIG_PREEMPT_RT (since interrupts are run in process context, right). I could use _irq in both, but that would unnecessarily degrade interrupt latency in CONFIG_PREEMPT_RT. Suggestions???
in PREEMPT_RT, if you define a spinlock type as raw_spinlock_t (via DEFINE_RAW_SPINLOCK(lock)) then the spin_lock*() APIs automatically switch over to that type. I.e. no need to do the rcu_ stuff AFAICT. (unless i missed some detail about what you are trying to do.) Raw spinlocks under PREEMPT_RT pair with the raw IRQ flag, i.e. spin_lock_irq() will disable hard interrupts. Generally i'd suggest to go for the raw spinlock and raw-irq-flag-disabling variant, because RCU locking is core enough functionality to warrant atomic treatment. If there's any latency path in it, we can work on breaking it up later.
it's perfectly fine to disable raw interrupts with the RCU code, as long as you do O(1) amount of work. (where the constant factor isnt 2^(2^32) ;-)
Quoted 4 lines
Some remaining shortcomings of the current code:
o Untested on
CONFIG_PREEMPT_RT(working on this, suggestions
for 4-CPU-stableCONFIG_PREEMPT_RTversions most welcome).
latest (-51-28) has no known regressions, and i regularly boot on 4-way (and irregularly on 8-way) boxes. So if you see something strange on the latest -RT kernel, please report it.
Ingo
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · Paul E. McKenney
So, time to get serious about a bit of code cleanup:
o The heavyweight atomic operations in
rcu_read_lock()and
rcu_read_unlock()are not needed in UP kernels, since interrupts are disabled.atomic_*() ops should already be lightweight on UP.
Agreed, will worry about architectures where they might not be later.
Quoted 31 lines
Quoted 22 lines
o In order to get things to work in both
CONFIG_PREEMPTandCONFIG_PREEMPT_RT, I ended up using the following: #ifdef CONFIG_PREEMPT_RT #define rcu_spinlock_t _raw_spinlock_t #define rcu_spin_lock(l, f) _raw_spin_lock(l) #define rcu_spin_trylock(l, f) _raw_spin_trylock(l) #define rcu_spin_unlock(l, f) _raw_spin_unlock(l) #define RCU_SPIN_LOCK_UNLOCKED RAW_SPIN_LOCK_UNLOCKED #else /* #ifdef CONFIG_PREEMPT_RT */ #define rcu_spinlock_t spinlock_t #define rcu_spin_lock(l, f) spin_lock_irqsave(l, f) #define rcu_spin_trylock(l, f) spin_trylock_irqsave(l, f) #define rcu_spin_unlock(l, f) spin_unlock_irqrestore(l, f) #define RCU_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED #endif /* #else #ifdef CONFIG_PREEMPT_RT */ Then using rcu_spin_lock() &c everywhere. The problem isQuoted 8 lines
that (as near as I can tell) the only way to prevent interrupts from running on the current CPU in CONFIG_PREEMPT kernels is to use the _irq spinlock primitives, but _raw_spin_lock() does the job in CONFIG_PREEMPT_RT (since interrupts are run in process context, right). I could use _irq in both, but that would unnecessarily degrade interrupt latency in CONFIG_PREEMPT_RT. Suggestions???
Quoted 9 lines
in
PREEMPT_RT, if you define a spinlock type asraw_spinlock_t(viaDEFINE_RAW_SPINLOCK(lock)) then the spin_lock*() APIs automatically switch over to that type. I.e. no need to do thercu_stuff AFAICT. (unless i missed some detail about what you are trying to do.) Raw spinlocks underPREEMPT_RTpair with the raw IRQ flag, i.e.spin_lock_irq()will disable hard interrupts. Generally i'd suggest to go for the raw spinlock and raw-irq-flag-disabling variant, because RCU locking is core enough functionality to warrant atomic treatment. If there's any latency path in it, we can work on breaking it up later.
Fair enough!
The thing I am (perhaps foolishly) trying to do is get to a single source base that provides PREEMPT_RCU under both CONFIG_PREEMPT and CONFIG_PREEMPT_RT. Since you are OK with RCU disabling interrupts, I am happy.
Quoted 3 lines
it's perfectly fine to disable raw interrupts with the RCU code, as long as you do O(1) amount of work. (where the constant factor isnt 2^(2^32) ;-)
Will try to keep it down to a dull roar. ;-)
Quoted 8 lines
Quoted 4 lines
Some remaining shortcomings of the current code:
o Untested on
CONFIG_PREEMPT_RT(working on this, suggestions
for 4-CPU-stableCONFIG_PREEMPT_RTversions most welcome).latest (-51-28) has no known regressions, and i regularly boot on 4-way (and irregularly on 8-way) boxes. So if you see something strange on the latest -RT kernel, please report it.
And stock -51-7 just passed nine rounds of kernbench + LTP on some 4-CPU x86 machines, with one still running. Good stuff!!!
Thanx, Paul