Skip to contentSkip to conversation

PREEMPT/PREEMPT_RT question

PATCH Paul E. McKenney 3 participants 11 messages Jul 12, 2005 – Jul 13, 2005

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hello!

OK, counter-flip RCU actually survives a pair of overnight runs on CONFIG_PREEMPT running on 4-CPU machines, and also survives five kernbenches and an LTP on another 4-CPU machine. (Overnight-run script later in this message, FWIW.)

So, time to get serious about a bit of code cleanup:

o The heavyweight atomic operations in rcu_read_lock() and
rcu_read_unlock() are not needed in UP kernels, since interrupts are disabled.

Is there already something like smp_atomic_inc() and smp_atomic_dec() that generate atomic_inc()/atomic_dec() in SMP kernels, but ++/-- in UP kernels? If not, any reasons not to add them, for example, as follows?

#ifdef CONFIG_SMP
#define smp_atomic_inc(v) atomic_inc(v) #define smp_atomic_dec(v) atomic_dec(v) #else /* #ifdef CONFIG_SMP */ #define smp_atomic_inc(v) ((v)++) #define smp_atomic_dec(v) ((v)++) #endif /* #else #ifdef CONFIG_SMP */

Since interrupts must be disabled for these to be safe, my guess is that I should define them locally in rcupdate.c. If there turns out to be a general need for them, they can be moved somewhere more public.

Objections?

o In order to get things to work in both CONFIG_PREEMPT and

	CONFIG_PREEMPT_RT, I ended up using the following:

		#ifdef CONFIG_PREEMPT_RT

		#define rcu_spinlock_t _raw_spinlock_t
		#define rcu_spin_lock(l, f) _raw_spin_lock(l)
		#define rcu_spin_trylock(l, f) _raw_spin_trylock(l)
		#define rcu_spin_unlock(l, f) _raw_spin_unlock(l)
		#define RCU_SPIN_LOCK_UNLOCKED RAW_SPIN_LOCK_UNLOCKED

		#else /* #ifdef CONFIG_PREEMPT_RT */

		#define rcu_spinlock_t spinlock_t
		#define rcu_spin_lock(l, f) spin_lock_irqsave(l, f)
		#define rcu_spin_trylock(l, f) spin_trylock_irqsave(l, f)
		#define rcu_spin_unlock(l, f) spin_unlock_irqrestore(l, f)
		#define RCU_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED

		#endif /* #else #ifdef CONFIG_PREEMPT_RT */

	Then using rcu_spin_lock() &c everywhere.  The problem is
	that (as near as I can tell) the only way to prevent interrupts 
	from running on the current CPU in CONFIG_PREEMPT kernels is
	to use the _irq spinlock primitives, but _raw_spin_lock() does
	the job in CONFIG_PREEMPT_RT (since interrupts are run in process
	context, right).  I could use _irq in both, but that would
	unnecessarily degrade interrupt latency in CONFIG_PREEMPT_RT.

	Suggestions???

Some remaining shortcomings of the current code:

o Untested on CONFIG_PREEMPT_RT (working on this, suggestions
for 4-CPU-stable CONFIG_PREEMPT_RT versions most welcome).

o Uses atomic operations and memory barriers in the
rcu_read_lock() and rcu_read_unlock() fastpaths. Getting rid of the atomic operations in UP is easy, in SMP in the common case is reasonably straightforward, getting rid of memory barriers is harder, but is likely doable.

o Global callback management slows things down on SMP
(need to move to per-CPU callback management).

o Gross debug code needs pruning (working on this).

o Current implementation is probably too aggressive about
finishing out grace periods -- one of the machines averaged one grace period every 29 milliseconds over a 15-hour time period. This likely understates the aggressiveness, since the test is quite bursty.

o Given the small amount testing, there are undoubtably
more bugs.

All that aside, only a moderately severe degree of insanity is required to run this code.

Improvements since last time:

o Boots and runs reasonably reliably.

o Got rid of the workqueue stuff, simplifying grace-period
detection code. Also allows synchronize_sched() to be common code between "classic" and realtime RCU implementations.

o Enabled preemption in RCU read-side critical sections
(even in CONFIG_PREEMPT kernels).

Test script (assumes that you have a linux-2.6.11.tar.bz2 file in the current directory), relies on the fact that dcache's heavy use of RCU:

	#!/bin/sh

	while :
	do
		i=0
		while expr $i '<' 10
		do
			tar -xjf linux-2.6.11.tar.bz2
			echo tar status $?
			mv linux-2.6.11 argh$i
			echo $i : `grep dentry_cache /proc/slabinfo`
			i=`expr $i + 1`
		done
		for i in argh[0-9]
		do
			rm -rf $i &
		done
		wait
		echo rm : `grep dentry_cache /proc/slabinfo`
	done

Current (ugly) patch:

Not-signed-off-by <[email protected]>
arch/i386/Kconfig +25 -0
diff -urpN -X dontdiff linux-2.6.12-rc6/arch/i386/Kconfig linux-2.6.12-rc6-ctrRCU/arch/i386/Kconfig
@@ -523,6 +523,31 @@ config PREEMPT
 	  Say Y here if you are building a kernel for a desktop, embedded
 	  or real-time system.  Say N if you are unsure.
 
+config PREEMPT_RCU
+	bool "Preemptible RCU read-side critical sections"
+	depends on PREEMPT
+	default y
+	help
+	  This option reduces the latency of the kernel when reacting to
+	  real-time or interactive events by allowing a low priority process to
+	  be preempted even if it is in an RCU read-side critical section.
+	  This allows applications to run more reliably even when the system is
+	  under load.
+
+	  Say Y here if you enjoy debugging random oopses.
+	  Say N if, for whatever reason, you want your kernel to actually work.
+
+config RCU_STATS
+	bool "/proc stats for preemptible RCU read-side critical sections"
+	depends on PREEMPT_RCU
+	default y
+	help
+	  This option provides /proc stats to provide debugging info for
+	  the preemptible realtime RCU implementation.
+
+	  Say Y here if you want to see RCU stats in /proc
+	  Say N if you are unsure.
+
 config PREEMPT_BKL
 	bool "Preempt The Big Kernel Lock"
 	depends on PREEMPT
fs/proc/proc_misc.c +37 -0
diff -urpN -X dontdiff linux-2.6.12-rc6/fs/proc/proc_misc.c linux-2.6.12-rc6-ctrRCU/fs/proc/proc_misc.c
@@ -549,6 +549,38 @@ void create_seq_entry(char *name, mode_t
 		entry->proc_fops = f;
 }
 
+#ifdef CONFIG_RCU_STATS
+int rcu_read_proc(char *page, char **start, off_t off,
+		  int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_data(char *page);
+
+	len = rcu_read_proc_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_gp(char *page, char **start, off_t off,
+		     int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_gp_data(char *page);
+
+	len = rcu_read_proc_gp_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_ptrs(char *page, char **start, off_t off,
+		       int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_ptrs_data(char *page);
+
+	len = rcu_read_proc_ptrs_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+#endif /* #ifdef CONFIG_RCU_STATS */
+
 void __init proc_misc_init(void)
 {
 	struct proc_dir_entry *entry;
@@ -571,6 +603,11 @@ void __init proc_misc_init(void)
 		{"cmdline",	cmdline_read_proc},
 		{"locks",	locks_read_proc},
 		{"execdomains",	execdomains_read_proc},
+#ifdef CONFIG_RCU_STATS
+		{"rcustats",	rcu_read_proc},
+		{"rcugp",	rcu_read_proc_gp},
+		{"rcuptrs",	rcu_read_proc_ptrs},
+#endif /* #ifdef CONFIG_RCU_STATS */
 		{NULL,}
 	};
 	for (p = simple_ones; p->name; p++)
include/linux/rcupdate.h +25 -0
diff -urpN -X dontdiff linux-2.6.12-rc6/include/linux/rcupdate.h linux-2.6.12-rc6-ctrRCU/include/linux/rcupdate.h
@@ -59,6 +59,7 @@ struct rcu_head {
 } while (0)
 
 
+#ifndef CONFIG_PREEMPT_RCU
 
 /* Global control variables for rcupdate callback mechanism. */
 struct rcu_ctrlblk {
@@ -192,6 +193,18 @@ static inline int rcu_pending(int cpu)
  */
 #define rcu_read_unlock()	preempt_enable()
 
+#else /* #ifndef CONFIG_PREEMPT_RCU */
+
+#define rcu_qsctr_inc(cpu)
+#define rcu_bh_qsctr_inc(cpu)
+#define call_rcu_bh(head, rcu) call_rcu(head, rcu)
+
+extern void rcu_read_lock(void);
+extern void rcu_read_unlock(void);
+extern int rcu_pending(int cpu);
+
+#endif /* #else #ifndef CONFIG_PREEMPT_RCU */
+
 /*
  * So where is rcu_write_lock()?  It does not exist, as there is no
  * way for writers to lock out RCU readers.  This is a feature, not
@@ -213,14 +226,22 @@ static inline int rcu_pending(int cpu)
  * can use just rcu_read_lock().
  *
  */
+#ifndef CONFIG_PREEMPT_RCU
 #define rcu_read_lock_bh()	local_bh_disable()
+#else /* #ifndef CONFIG_PREEMPT_RCU */
+#define rcu_read_lock_bh()	{ rcu_read_lock(); local_bh_disable(); }
+#endif /* #else #ifndef CONFIG_PREEMPT_RCU */
 
 /*
  * rcu_read_unlock_bh - marks the end of a softirq-only RCU critical section
  *
  * See rcu_read_lock_bh() for more information.
  */
+#ifndef CONFIG_PREEMPT_RCU
 #define rcu_read_unlock_bh()	local_bh_enable()
+#else /* #ifndef CONFIG_PREEMPT_RCU */
+#define rcu_read_unlock_bh()	{ local_bh_enable(); rcu_read_unlock(); }
+#endif /* #else #ifndef CONFIG_PREEMPT_RCU */
 
 /**
  * rcu_dereference - fetch an RCU-protected pointer in an
@@ -269,7 +290,11 @@ static inline int rcu_pending(int cpu)
  * synchronize_kernel() API.  In contrast, synchronize_rcu() only
  * guarantees that rcu_read_lock() sections will have completed.
  */
+#ifndef CONFIG_PREEMPT_RCU
 #define synchronize_sched() synchronize_rcu()
+#else /* #ifndef CONFIG_PREEMPT_RCU */
+extern void synchronize_sched(void);
+#endif /* #else #ifndef CONFIG_PREEMPT_RCU */
 
 extern void rcu_init(void);
 extern void rcu_check_callbacks(int cpu, int user);
include/linux/sched.h +5 -0
diff -urpN -X dontdiff linux-2.6.12-rc6/include/linux/sched.h linux-2.6.12-rc6-ctrRCU/include/linux/sched.h
@@ -740,6 +740,11 @@ struct task_struct {
 	nodemask_t mems_allowed;
 	int cpuset_mems_generation;
 #endif
+#ifdef CONFIG_PREEMPT_RCU
+	int rcu_read_lock_nesting;
+	atomic_t *rcu_flipctr1;
+	atomic_t *rcu_flipctr2;
+#endif
 };
 
 static inline pid_t process_group(struct task_struct *tsk)
kernel/rcupdate.c +479 -29

538 lines not on this page · raw message

#1 lore raw
Quoted 10 lines · Paul E. McKenney

Hello!

OK, counter-flip RCU actually survives a pair of overnight runs on CONFIG_PREEMPT running on 4-CPU machines, and also survives five kernbenches and an LTP on another 4-CPU machine. (Overnight-run script later in this message, FWIW.)

So, time to get serious about a bit of code cleanup:

o The heavyweight atomic operations in rcu_read_lock() and

Quoted 15 lines
	rcu_read_unlock() are not needed in UP kernels, since
	interrupts are disabled.

	Is there already something like smp_atomic_inc() and
	smp_atomic_dec() that generate atomic_inc()/atomic_dec() in
	SMP kernels, but ++/-- in UP kernels?  If not, any reasons
	not to add them, for example, as follows?

		#ifdef CONFIG_SMP
		#define smp_atomic_inc(v) atomic_inc(v)
		#define smp_atomic_dec(v) atomic_dec(v)
		#else /* #ifdef CONFIG_SMP */
		#define smp_atomic_inc(v) ((v)++)
		#define smp_atomic_dec(v) ((v)++)
		#endif /* #else #ifdef CONFIG_SMP */

What's the problem with atomic_inc? At least on x86, atomic inc is defined as:

static __inline__ void atomic_inc(atomic_t *v)

{
	__asm__ __volatile__(
		LOCK "incl %0"
		:"=m" (v->counter)
		:"m" (v->counter));
}

With LOCK defined as:

#ifdef CONFIG_SMP
#define LOCK "lock ; "
#else
#define LOCK ""
#endif

So is there a difference on UP between x.counter++ and atomic_inc(&x)?

Quoted 28 lines

Since interrupts must be disabled for these to be safe, my guess is that I should define them locally in rcupdate.c. If there turns out to be a general need for them, they can be moved somewhere more public.

Objections?

o In order to get things to work in both CONFIG_PREEMPT and

	CONFIG_PREEMPT_RT, I ended up using the following:

		#ifdef CONFIG_PREEMPT_RT

		#define rcu_spinlock_t _raw_spinlock_t
		#define rcu_spin_lock(l, f) _raw_spin_lock(l)
		#define rcu_spin_trylock(l, f) _raw_spin_trylock(l)
		#define rcu_spin_unlock(l, f) _raw_spin_unlock(l)
		#define RCU_SPIN_LOCK_UNLOCKED RAW_SPIN_LOCK_UNLOCKED

		#else /* #ifdef CONFIG_PREEMPT_RT */

		#define rcu_spinlock_t spinlock_t
		#define rcu_spin_lock(l, f) spin_lock_irqsave(l, f)
		#define rcu_spin_trylock(l, f) spin_trylock_irqsave(l, f)
		#define rcu_spin_unlock(l, f) spin_unlock_irqrestore(l, f)
		#define RCU_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED

		#endif /* #else #ifdef CONFIG_PREEMPT_RT */
Quoted 7 lines
	Then using rcu_spin_lock() &c everywhere.  The problem is
	that (as near as I can tell) the only way to prevent interrupts 
	from running on the current CPU in CONFIG_PREEMPT kernels is
	to use the _irq spinlock primitives, but _raw_spin_lock() does
	the job in CONFIG_PREEMPT_RT (since interrupts are run in process
	context, right).  I could use _irq in both, but that would
	unnecessarily degrade interrupt latency in CONFIG_PREEMPT_RT.

Yep interrupts are threads in CONFIG_PREEMPT_RT. I guess you could also just use local_irq_save with spin_lock, since now local_irq_save no longer disables interrupts in PREEMPT_RT.

-- Steve

#2 lore raw
Quoted 40 lines · Steven Rostedt
Quoted 10 lines · Paul E. McKenney

Hello!

OK, counter-flip RCU actually survives a pair of overnight runs on CONFIG_PREEMPT running on 4-CPU machines, and also survives five kernbenches and an LTP on another 4-CPU machine. (Overnight-run script later in this message, FWIW.)

So, time to get serious about a bit of code cleanup:

o The heavyweight atomic operations in rcu_read_lock() and

Quoted 15 lines
	rcu_read_unlock() are not needed in UP kernels, since
	interrupts are disabled.

	Is there already something like smp_atomic_inc() and
	smp_atomic_dec() that generate atomic_inc()/atomic_dec() in
	SMP kernels, but ++/-- in UP kernels?  If not, any reasons
	not to add them, for example, as follows?

		#ifdef CONFIG_SMP
		#define smp_atomic_inc(v) atomic_inc(v)
		#define smp_atomic_dec(v) atomic_dec(v)
		#else /* #ifdef CONFIG_SMP */
		#define smp_atomic_inc(v) ((v)++)
		#define smp_atomic_dec(v) ((v)++)
		#endif /* #else #ifdef CONFIG_SMP */

What's the problem with atomic_inc? At least on x86, atomic inc is defined as:

static __inline__ void atomic_inc(atomic_t *v)

{
	__asm__ __volatile__(
		LOCK "incl %0"
		:"=m" (v->counter)
		:"m" (v->counter));
}

With LOCK defined as:

Quoted 7 lines

#ifdef CONFIG_SMP
#define LOCK "lock ; "
#else
#define LOCK ""
#endif

So is there a difference on UP between x.counter++ and atomic_inc(&x)?

On x86, you are right. The full list of architectures that are atomic only in SMP are i386 (as you noted), parisc, sparc, and x86_64.

The architectures that appear to always be atomic are: alpha, ia64, m32r (but unfamiliar with this one), mips, ppc, ppc64, s390 (I think...), and sparc64. Most of these architectures need to disable interrupts to provide "universal" atomic_inc() semantics in UP kernels, due to their RISC-style atomic instructions.

The following architectures avoid the issue entirely by refusing to support SMP: arm, arm26, cris, frv, h8300, m68k, m68knommu, sh, sh64, and v850. Many of them disable interrupts in their atomic_inc() implementations.

The advantage of smp_atomic_inc() is that architectures could dispense with interrupt disabling. The disadvantage is that it is yet another contribution to Linux's combinatorial explosion of primitives.

For the moment, I will grit my teeth and keep atomic_inc() and atomic_dec().

Other thoughts?

Quoted 36 lines
Quoted 28 lines

Since interrupts must be disabled for these to be safe, my guess is that I should define them locally in rcupdate.c. If there turns out to be a general need for them, they can be moved somewhere more public.

Objections?

o In order to get things to work in both CONFIG_PREEMPT and

	CONFIG_PREEMPT_RT, I ended up using the following:

		#ifdef CONFIG_PREEMPT_RT

		#define rcu_spinlock_t _raw_spinlock_t
		#define rcu_spin_lock(l, f) _raw_spin_lock(l)
		#define rcu_spin_trylock(l, f) _raw_spin_trylock(l)
		#define rcu_spin_unlock(l, f) _raw_spin_unlock(l)
		#define RCU_SPIN_LOCK_UNLOCKED RAW_SPIN_LOCK_UNLOCKED

		#else /* #ifdef CONFIG_PREEMPT_RT */

		#define rcu_spinlock_t spinlock_t
		#define rcu_spin_lock(l, f) spin_lock_irqsave(l, f)
		#define rcu_spin_trylock(l, f) spin_trylock_irqsave(l, f)
		#define rcu_spin_unlock(l, f) spin_unlock_irqrestore(l, f)
		#define RCU_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED

		#endif /* #else #ifdef CONFIG_PREEMPT_RT */
Quoted 7 lines
	Then using rcu_spin_lock() &c everywhere.  The problem is
	that (as near as I can tell) the only way to prevent interrupts 
	from running on the current CPU in CONFIG_PREEMPT kernels is
	to use the _irq spinlock primitives, but _raw_spin_lock() does
	the job in CONFIG_PREEMPT_RT (since interrupts are run in process
	context, right).  I could use _irq in both, but that would
	unnecessarily degrade interrupt latency in CONFIG_PREEMPT_RT.
Quoted 3 lines

Yep interrupts are threads in CONFIG_PREEMPT_RT. I guess you could also just use local_irq_save with spin_lock, since now local_irq_save no longer disables interrupts in PREEMPT_RT.

By this you mean the following?

	local_irq_save(flags);
	_raw_spin_lock(&mylock);

	/* critical section */

	_raw_spin_unlock(&mylock);
	local_irq_restore(flags);

							Thanx, Paul
#3 lore raw
Quoted 17 lines · Paul E. McKenney
Quoted 2 lines

So is there a difference on UP between x.counter++ and atomic_inc(&x)?

On x86, you are right. The full list of architectures that are atomic only in SMP are i386 (as you noted), parisc, sparc, and x86_64.

The architectures that appear to always be atomic are: alpha, ia64, m32r (but unfamiliar with this one), mips, ppc, ppc64, s390 (I think...), and sparc64. Most of these architectures need to disable interrupts to provide "universal" atomic_inc() semantics in UP kernels, due to their RISC-style atomic instructions.

The following architectures avoid the issue entirely by refusing to support SMP: arm, arm26, cris, frv, h8300, m68k, m68knommu, sh, sh64, and v850. Many of them disable interrupts in their atomic_inc() implementations.

Quoted 7 lines

The advantage of smp_atomic_inc() is that architectures could dispense with interrupt disabling. The disadvantage is that it is yet another contribution to Linux's combinatorial explosion of primitives.

For the moment, I will grit my teeth and keep atomic_inc() and atomic_dec().

Other thoughts?

How did I know that you would mention mips and the like :-)

Yeah, mips has the crazy Load Linked and Store Conditional crap, so it is a little more complex than the simple add one. And I think PPC does too, although it has been a while since I've used them. And older mips don't have the LL SC command so the only option is to turn off interrupts (of course those that don't have the LL and SC are not SMP compatible). So, I will admit that having a smp_atomic_inc might be nice. I was just being a narrow minded x86 hacker ;-)

Quoted 6 lines
Quoted 3 lines

Yep interrupts are threads in CONFIG_PREEMPT_RT. I guess you could also just use local_irq_save with spin_lock, since now local_irq_save no longer disables interrupts in PREEMPT_RT.

By this you mean the following?

Quoted 7 lines
	local_irq_save(flags);
	_raw_spin_lock(&mylock);

	/* critical section */

	_raw_spin_unlock(&mylock);
	local_irq_restore(flags);

Yeah, that on PREEMP_RT would not turn off interrupts but just stops preemption. This is fine as long as the mylock is not used in any SA_NODELAY interrupt.

-- Steve

#4 lore raw
Quoted 28 lines · Steven Rostedt
Quoted 17 lines · Paul E. McKenney
Quoted 2 lines

So is there a difference on UP between x.counter++ and atomic_inc(&x)?

On x86, you are right. The full list of architectures that are atomic only in SMP are i386 (as you noted), parisc, sparc, and x86_64.

The architectures that appear to always be atomic are: alpha, ia64, m32r (but unfamiliar with this one), mips, ppc, ppc64, s390 (I think...), and sparc64. Most of these architectures need to disable interrupts to provide "universal" atomic_inc() semantics in UP kernels, due to their RISC-style atomic instructions.

The following architectures avoid the issue entirely by refusing to support SMP: arm, arm26, cris, frv, h8300, m68k, m68knommu, sh, sh64, and v850. Many of them disable interrupts in their atomic_inc() implementations.

Quoted 7 lines

The advantage of smp_atomic_inc() is that architectures could dispense with interrupt disabling. The disadvantage is that it is yet another contribution to Linux's combinatorial explosion of primitives.

For the moment, I will grit my teeth and keep atomic_inc() and atomic_dec().

Other thoughts?

How did I know that you would mention mips and the like :-)

Quoted 7 lines

Yeah, mips has the crazy Load Linked and Store Conditional crap, so it is a little more complex than the simple add one. And I think PPC does too, although it has been a while since I've used them. And older mips don't have the LL SC command so the only option is to turn off interrupts (of course those that don't have the LL and SC are not SMP compatible). So, I will admit that having a smp_atomic_inc might be nice. I was just being a narrow minded x86 hacker ;-)

I am sure that LL/SC seemed like a good idea at the time. ;-)

To be fair, LL/SC does allow allow some things to be done more easily than with cmpxchg, since it allows you to tell that the value changed even if it later changed back. Helps with some linked-list operations.

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

Yep interrupts are threads in CONFIG_PREEMPT_RT. I guess you could also just use local_irq_save with spin_lock, since now local_irq_save no longer disables interrupts in PREEMPT_RT.

By this you mean the following?

	local_irq_save(flags);
	_raw_spin_lock(&mylock);
Quoted 8 lines
Quoted 4 lines
	/* critical section */

	_raw_spin_unlock(&mylock);
	local_irq_restore(flags);

Yeah, that on PREEMP_RT would not turn off interrupts but just stops preemption. This is fine as long as the mylock is not used in any SA_NODELAY interrupt.

Cool! Is the scheduling-clock interrupt an SA_NODELAY interrupt?

Thanx, Paul

#5 lore raw
Quoted 9 lines · Paul E. McKenney
Quoted 8 lines

Yeah, mips has the crazy Load Linked and Store Conditional crap, so it is a little more complex than the simple add one. And I think PPC does too, although it has been a while since I've used them. And older mips don't have the LL SC command so the only option is to turn off interrupts (of course those that don't have the LL and SC are not SMP compatible). So, I will admit that having a smp_atomic_inc might be nice. I was just being a narrow minded x86 hacker ;-)

Quoted 5 lines

I am sure that LL/SC seemed like a good idea at the time. ;-)

To be fair, LL/SC does allow allow some things to be done more easily than with cmpxchg, since it allows you to tell that the value changed even if it later changed back. Helps with some linked-list operations.

I was being a little harsh in my statements. I didn't really mind the LL and SC but a true atomic inc would have been nice. I actually had to port Linux to a MIPS board once that didn't have the LL or SC, and that was even more painful.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 3 lines

Yep interrupts are threads in CONFIG_PREEMPT_RT. I guess you could also just use local_irq_save with spin_lock, since now local_irq_save no longer disables interrupts in PREEMPT_RT.

By this you mean the following?

	local_irq_save(flags);
	_raw_spin_lock(&mylock);

	/* critical section */
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
	_raw_spin_unlock(&mylock);
	local_irq_restore(flags);

Yeah, that on PREEMP_RT would not turn off interrupts but just stops preemption. This is fine as long as the mylock is not used in any SA_NODELAY interrupt.

Cool! Is the scheduling-clock interrupt an SA_NODELAY interrupt?

If you are talking about scheduler_tick, then yes, it is called by the timer interrupt which is a SA_NODELAY interrupt. If you don't want to get interrupted by the timer interrupt, then you will need to disable interrupts for both. Since currently, the timer interrupt is the only true hard interrupt in the PREEMPT_RT and that may not change.

-- Steve

#6 lore raw
Quoted 11 lines · Steven Rostedt
Quoted 10 lines · Paul E. McKenney
Quoted 7 lines

Yeah, mips has the crazy Load Linked and Store Conditional crap, so it is a little more complex than the simple add one. And I think PPC does too, although it has been a while since I've used them. And older mips don't have the LL SC command so the only option is to turn off interrupts (of course those that don't have the LL and SC are not SMP compatible). So, I will admit that having a smp_atomic_inc might be nice. I was just being a narrow minded x86 hacker ;-)

I am sure that LL/SC seemed like a good idea at the time. ;-)

Quoted 8 lines
Quoted 3 lines

To be fair, LL/SC does allow allow some things to be done more easily than with cmpxchg, since it allows you to tell that the value changed even if it later changed back. Helps with some linked-list operations.

I was being a little harsh in my statements. I didn't really mind the LL and SC but a true atomic inc would have been nice. I actually had to port Linux to a MIPS board once that didn't have the LL or SC, and that was even more painful.

I could believe that!

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines
Quoted 3 lines

Yep interrupts are threads in CONFIG_PREEMPT_RT. I guess you could also just use local_irq_save with spin_lock, since now local_irq_save no longer disables interrupts in PREEMPT_RT.

By this you mean the following?

Quoted 7 lines
	local_irq_save(flags);
	_raw_spin_lock(&mylock);

	/* critical section */

	_raw_spin_unlock(&mylock);
	local_irq_restore(flags);

Yeah, that on PREEMP_RT would not turn off interrupts but just stops preemption. This is fine as long as the mylock is not used in any SA_NODELAY interrupt.

Quoted 7 lines
Quoted 1 line

Cool! Is the scheduling-clock interrupt an SA_NODELAY interrupt?

If you are talking about scheduler_tick, then yes, it is called by the timer interrupt which is a SA_NODELAY interrupt. If you don't want to get interrupted by the timer interrupt, then you will need to disable interrupts for both. Since currently, the timer interrupt is the only true hard interrupt in the PREEMPT_RT and that may not change.

OK, so if I take a spinlock in something invoked from scheduler_tick(), then any other acquisitions of that spinlock must disable hardware interrupts, right?

Thanx, Paul

#7 lore raw
Quoted 9 lines · Paul E. McKenney
Quoted 5 lines

If you are talking about scheduler_tick, then yes, it is called by the timer interrupt which is a SA_NODELAY interrupt. If you don't want to get interrupted by the timer interrupt, then you will need to disable interrupts for both. Since currently, the timer interrupt is the only true hard interrupt in the PREEMPT_RT and that may not change.

OK, so if I take a spinlock in something invoked from scheduler_tick(), then any other acquisitions of that spinlock must disable hardware interrupts, right?

Yes, otherwise you could have a local CPU deadlock on a SMP machine. And I would also say the same is true for any lock that is grabbed by the timer interrupt or one of the functions it calls.

-- Steve

#8 lore raw
Quoted 8 lines · Steven Rostedt
Quoted 6 lines · Paul E. McKenney
Quoted 5 lines

If you are talking about scheduler_tick, then yes, it is called by the timer interrupt which is a SA_NODELAY interrupt. If you don't want to get interrupted by the timer interrupt, then you will need to disable interrupts for both. Since currently, the timer interrupt is the only true hard interrupt in the PREEMPT_RT and that may not change.

Quoted 7 lines
Quoted 3 lines

OK, so if I take a spinlock in something invoked from scheduler_tick(), then any other acquisitions of that spinlock must disable hardware interrupts, right?

Yes, otherwise you could have a local CPU deadlock on a SMP machine. And I would also say the same is true for any lock that is grabbed by the timer interrupt or one of the functions it calls.

OK, I do indeed have critical sections spanning rcu_check_callbacks() and process-level code. Since rcu_check_callbacks() is called from account_user_vtime() and update_process_times(), both of which call scheduler_tick(), looks like I need to disable hardware interrupts.

Looks to me like I need to use _raw_spin_lock() and _raw_spin_unlock() from within rcu_check_callbacks(), but to instead use _raw_spin_lock_irqsave() and _raw_spin_lock_irqrestore() outside of rcu_check_callbacks() for locks acquired within rcu_check_callbacks().

Of course, for fliplock, which is only acquired conditionally from a rcu_check_callbacks() callee, I can just use spin_trylock() and spin_unlock(). Though _raw_spin_lock()/_raw_spin_unlock() might be faster?

No -wonder- I have been seeing hangs in CONFIG_PREEMPT_RT!!! ;-)

Thanx, Paul

#9 lore raw
In reply to Paul E. McKenney
Quoted 5 lines · Paul E. McKenney

Hello!

OK, counter-flip RCU actually survives a pair of overnight runs on CONFIG_PREEMPT running on 4-CPU machines, and also survives five kernbenches and an LTP on another 4-CPU machine. [...]

cool!

Quoted 5 lines

So, time to get serious about a bit of code cleanup:

o The heavyweight atomic operations in rcu_read_lock() and
rcu_read_unlock() are not needed in UP kernels, since interrupts are disabled.

atomic_*() ops should already be lightweight on UP.

Quoted 22 lines

o In order to get things to work in both CONFIG_PREEMPT and

	CONFIG_PREEMPT_RT, I ended up using the following:

		#ifdef CONFIG_PREEMPT_RT

		#define rcu_spinlock_t _raw_spinlock_t
		#define rcu_spin_lock(l, f) _raw_spin_lock(l)
		#define rcu_spin_trylock(l, f) _raw_spin_trylock(l)
		#define rcu_spin_unlock(l, f) _raw_spin_unlock(l)
		#define RCU_SPIN_LOCK_UNLOCKED RAW_SPIN_LOCK_UNLOCKED

		#else /* #ifdef CONFIG_PREEMPT_RT */

		#define rcu_spinlock_t spinlock_t
		#define rcu_spin_lock(l, f) spin_lock_irqsave(l, f)
		#define rcu_spin_trylock(l, f) spin_trylock_irqsave(l, f)
		#define rcu_spin_unlock(l, f) spin_unlock_irqrestore(l, f)
		#define RCU_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED

		#endif /* #else #ifdef CONFIG_PREEMPT_RT */

	Then using rcu_spin_lock() &c everywhere.  The problem is
Quoted 8 lines
	that (as near as I can tell) the only way to prevent interrupts 
	from running on the current CPU in CONFIG_PREEMPT kernels is
	to use the _irq spinlock primitives, but _raw_spin_lock() does
	the job in CONFIG_PREEMPT_RT (since interrupts are run in process
	context, right).  I could use _irq in both, but that would
	unnecessarily degrade interrupt latency in CONFIG_PREEMPT_RT.

	Suggestions???

in PREEMPT_RT, if you define a spinlock type as raw_spinlock_t (via DEFINE_RAW_SPINLOCK(lock)) then the spin_lock*() APIs automatically switch over to that type. I.e. no need to do the rcu_ stuff AFAICT. (unless i missed some detail about what you are trying to do.) Raw spinlocks under PREEMPT_RT pair with the raw IRQ flag, i.e. spin_lock_irq() will disable hard interrupts. Generally i'd suggest to go for the raw spinlock and raw-irq-flag-disabling variant, because RCU locking is core enough functionality to warrant atomic treatment. If there's any latency path in it, we can work on breaking it up later.

it's perfectly fine to disable raw interrupts with the RCU code, as long as you do O(1) amount of work. (where the constant factor isnt 2^(2^32) ;-)

Quoted 4 lines

Some remaining shortcomings of the current code:

o Untested on CONFIG_PREEMPT_RT (working on this, suggestions
for 4-CPU-stable CONFIG_PREEMPT_RT versions most welcome).

latest (-51-28) has no known regressions, and i regularly boot on 4-way (and irregularly on 8-way) boxes. So if you see something strange on the latest -RT kernel, please report it.

Ingo

#10 lore raw
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · Paul E. McKenney

So, time to get serious about a bit of code cleanup:

o The heavyweight atomic operations in rcu_read_lock() and
rcu_read_unlock() are not needed in UP kernels, since interrupts are disabled.

atomic_*() ops should already be lightweight on UP.

Agreed, will worry about architectures where they might not be later.

Quoted 31 lines
Quoted 22 lines

o In order to get things to work in both CONFIG_PREEMPT and

	CONFIG_PREEMPT_RT, I ended up using the following:

		#ifdef CONFIG_PREEMPT_RT

		#define rcu_spinlock_t _raw_spinlock_t
		#define rcu_spin_lock(l, f) _raw_spin_lock(l)
		#define rcu_spin_trylock(l, f) _raw_spin_trylock(l)
		#define rcu_spin_unlock(l, f) _raw_spin_unlock(l)
		#define RCU_SPIN_LOCK_UNLOCKED RAW_SPIN_LOCK_UNLOCKED

		#else /* #ifdef CONFIG_PREEMPT_RT */

		#define rcu_spinlock_t spinlock_t
		#define rcu_spin_lock(l, f) spin_lock_irqsave(l, f)
		#define rcu_spin_trylock(l, f) spin_trylock_irqsave(l, f)
		#define rcu_spin_unlock(l, f) spin_unlock_irqrestore(l, f)
		#define RCU_SPIN_LOCK_UNLOCKED SPIN_LOCK_UNLOCKED

		#endif /* #else #ifdef CONFIG_PREEMPT_RT */

	Then using rcu_spin_lock() &c everywhere.  The problem is
Quoted 8 lines
	that (as near as I can tell) the only way to prevent interrupts 
	from running on the current CPU in CONFIG_PREEMPT kernels is
	to use the _irq spinlock primitives, but _raw_spin_lock() does
	the job in CONFIG_PREEMPT_RT (since interrupts are run in process
	context, right).  I could use _irq in both, but that would
	unnecessarily degrade interrupt latency in CONFIG_PREEMPT_RT.

	Suggestions???
Quoted 9 lines

in PREEMPT_RT, if you define a spinlock type as raw_spinlock_t (via DEFINE_RAW_SPINLOCK(lock)) then the spin_lock*() APIs automatically switch over to that type. I.e. no need to do the rcu_ stuff AFAICT. (unless i missed some detail about what you are trying to do.) Raw spinlocks under PREEMPT_RT pair with the raw IRQ flag, i.e. spin_lock_irq() will disable hard interrupts. Generally i'd suggest to go for the raw spinlock and raw-irq-flag-disabling variant, because RCU locking is core enough functionality to warrant atomic treatment. If there's any latency path in it, we can work on breaking it up later.

Fair enough!

The thing I am (perhaps foolishly) trying to do is get to a single source base that provides PREEMPT_RCU under both CONFIG_PREEMPT and CONFIG_PREEMPT_RT. Since you are OK with RCU disabling interrupts, I am happy.

Quoted 3 lines

it's perfectly fine to disable raw interrupts with the RCU code, as long as you do O(1) amount of work. (where the constant factor isnt 2^(2^32) ;-)

Will try to keep it down to a dull roar. ;-)

Quoted 8 lines
Quoted 4 lines

Some remaining shortcomings of the current code:

o Untested on CONFIG_PREEMPT_RT (working on this, suggestions
for 4-CPU-stable CONFIG_PREEMPT_RT versions most welcome).

latest (-51-28) has no known regressions, and i regularly boot on 4-way (and irregularly on 8-way) boxes. So if you see something strange on the latest -RT kernel, please report it.

And stock -51-7 just passed nine rounds of kernbench + LTP on some 4-CPU x86 machines, with one still running. Good stuff!!!

Thanx, Paul