RCU and CONFIG_PREEMPT_RT sane patch

RFC PATCH In review Paul E. McKenney 2 participants 4 messages Aug 1, 2005 – Aug 4, 2005

Subsystems (3)

Scheduler Proc filesystem RCU

lore

Share & tools

lore

Paul E. McKenney

author

2005-08-01 17:11 UTC #0 lore raw

RFC PATCH 1/1 RCU and CONFIG_PREEMPT_RT sane patch

Hello!

The attached patch passes 20 hours of the internal-to-RCU torture test (see the code in the attached patch under CONFIG_RCU_TORTURE_TEST), which is a good improvement over last week's version, which would fail this test in less than two seconds -- this despite passing kernbench with flying colors and passing LTP 90% of the time. Sometimes there is no substitute for a good in-kernel stress test...

The attached version of the patch seems to me to be ready for a larger audience.

The following caveats still apply:

o Still have heavyweight operations in rcu_read_lock() and
rcu_read_unlock(). Will work on removing the atomic_inc() and atomic_dec() first, with the memory barriers later. The reason for this order of attack is that the atomic_inc() and atomic_dec() slow down UP kernels for some architectures, but the memory barriers no effect on UP.

o Global callback queues result in poor SMP performance. On
the list to fix. Will likely require handling CPU hotplug (current code is too stupid to need to care about CPU hotplug).

o Grace-period-detection code is probably too aggressive, but will
worry about that later. This will interact with OOM on systems with small memories.

o Applies against V0.7.51-27 of Ingo's patch.

Thanx, Paul

---

Signed-off-by <[email protected]>

 fs/proc/proc_misc.c      |   81 ++++
 include/linux/rcupdate.h |   27 +
 include/linux/sched.h    |    3 
 kernel/Kconfig.preempt   |   26 +
 kernel/rcupdate.c        |  798 +++++++++++++++++++++++++++++++++++++++++++----
 5 files changed, 873 insertions(+), 62 deletions(-)

fs/proc/proc_misc.c +81 -0

diff -urpN -X dontdiff linux-2.6.12-realtime-preempt-V0.7.51-27/fs/proc/proc_misc.c linux-2.6.12-realtime-preempt-V0.7.51-27-ctrRCUtt1/fs/proc/proc_misc.c
@@ -599,6 +599,78 @@ void create_seq_entry(char *name, mode_t
 		entry->proc_fops = f;
 }
 
+#ifdef CONFIG_RCU_STATS
+int rcu_read_proc(char *page, char **start, off_t off,
+		  int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_data(char *page);
+
+	len = rcu_read_proc_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_gp(char *page, char **start, off_t off,
+		     int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_gp_data(char *page);
+
+	len = rcu_read_proc_gp_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_ptrs(char *page, char **start, off_t off,
+		       int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_ptrs_data(char *page);
+
+	len = rcu_read_proc_ptrs_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_ctrs(char *page, char **start, off_t off,
+		       int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_ctrs_data(char *page);
+
+	len = rcu_read_proc_ctrs_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_torture_writer(char *page, char **start, off_t off,
+			         int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_torture_writer_data(char *page);
+
+	len = rcu_read_proc_torture_writer_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_torture_reader(char *page, char **start, off_t off,
+			         int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_torture_reader_data(char *page);
+
+	len = rcu_read_proc_torture_reader_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_torture_stats(char *page, char **start, off_t off,
+			        int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_torture_stats_data(char *page);
+
+	len = rcu_read_proc_torture_stats_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+#endif /* #ifdef CONFIG_RCU_STATS */
+
 void __init proc_misc_init(void)
 {
 	struct proc_dir_entry *entry;

@@ -621,6 +693,15 @@ void __init proc_misc_init(void)
 		{"cmdline",	cmdline_read_proc},
 		{"locks",	locks_read_proc},
 		{"execdomains",	execdomains_read_proc},
+#ifdef CONFIG_RCU_STATS
+		{"rcustats",	rcu_read_proc},
+		{"rcugp",	rcu_read_proc_gp},
+		{"rcuptrs",	rcu_read_proc_ptrs},
+		{"rcuctrs",	rcu_read_proc_ctrs},
+		{"rcutw",	rcu_read_proc_torture_writer},
+		{"rcutr",	rcu_read_proc_torture_reader},
+		{"rcuts",	rcu_read_proc_torture_stats},
+#endif /* #ifdef CONFIG_RCU_STATS */
 		{NULL,}
 	};
 	for (p = simple_ones; p->name; p++)

include/linux/rcupdate.h +25 -2

diff -urpN -X dontdiff linux-2.6.12-realtime-preempt-V0.7.51-27/include/linux/rcupdate.h linux-2.6.12-realtime-preempt-V0.7.51-27-ctrRCUtt1/include/linux/rcupdate.h
@@ -59,6 +59,7 @@ struct rcu_head {
 } while (0)
 
 
+#ifndef CONFIG_PREEMPT_RCU
 
 /* Global control variables for rcupdate callback mechanism. */
 struct rcu_ctrlblk {

@@ -209,6 +210,18 @@ static inline int rcu_pending(int cpu)
 # define rcu_read_unlock preempt_enable
 #endif
 
+#else /* #ifndef CONFIG_PREEMPT_RCU */
+
+#define rcu_qsctr_inc(cpu)
+#define rcu_bh_qsctr_inc(cpu)
+#define call_rcu_bh(head, rcu) call_rcu(head, rcu)
+
+extern void rcu_read_lock(void);
+extern void rcu_read_unlock(void);
+extern int rcu_pending(int cpu);
+
+#endif /* #else #ifndef CONFIG_PREEMPT_RCU */
+
 /*
  * So where is rcu_write_lock()?  It does not exist, as there is no
  * way for writers to lock out RCU readers.  This is a feature, not

@@ -230,16 +243,22 @@ static inline int rcu_pending(int cpu)
  * can use just rcu_read_lock().
  *
  */
-//#define rcu_read_lock_bh()	local_bh_disable()
+#ifndef CONFIG_PREEMPT_RCU
+#define rcu_read_lock_bh()	local_bh_disable()
+#else /* #ifndef CONFIG_PREEMPT_RCU */
 #define rcu_read_lock_bh()	{ rcu_read_lock(); local_bh_disable(); }
+#endif /* #else #ifndef CONFIG_PREEMPT_RCU */
 
 /*
  * rcu_read_unlock_bh - marks the end of a softirq-only RCU critical section
  *
  * See rcu_read_lock_bh() for more information.
  */
-//#define rcu_read_unlock_bh()	local_bh_enable()
+#ifndef CONFIG_PREEMPT_RCU
+#define rcu_read_unlock_bh()	local_bh_enable()
+#else /* #ifndef CONFIG_PREEMPT_RCU */
 #define rcu_read_unlock_bh()	{ local_bh_enable(); rcu_read_unlock(); }
+#endif /* #else #ifndef CONFIG_PREEMPT_RCU */
 
 /**
  * rcu_dereference - fetch an RCU-protected pointer in an

@@ -288,7 +307,11 @@ static inline int rcu_pending(int cpu)
  * synchronize_kernel() API.  In contrast, synchronize_rcu() only
  * guarantees that rcu_read_lock() sections will have completed.
  */
+#ifndef CONFIG_PREEMPT_RCU
 #define synchronize_sched() synchronize_rcu()
+#else /* #ifndef CONFIG_PREEMPT_RCU */
+extern void synchronize_sched(void);
+#endif /* #else #ifndef CONFIG_PREEMPT_RCU */
 
 extern void rcu_init(void);
 extern void rcu_check_callbacks(int cpu, int user);

include/linux/sched.h +2 -1

diff -urpN -X dontdiff linux-2.6.12-realtime-preempt-V0.7.51-27/include/linux/sched.h linux-2.6.12-realtime-preempt-V0.7.51-27-ctrRCUtt1/include/linux/sched.h
@@ -917,7 +917,8 @@ struct task_struct {
 #endif
 #ifdef CONFIG_PREEMPT_RCU
 	int rcu_read_lock_nesting;
-	struct rcu_data *rcu_data;
+	atomic_t *rcu_flipctr1;
+	atomic_t *rcu_flipctr2;
 #endif
 };

kernel/Kconfig.preempt +25 -1

diff -urpN -X dontdiff linux-2.6.12-realtime-preempt-V0.7.51-27/kernel/Kconfig.preempt linux-2.6.12-realtime-preempt-V0.7.51-27-ctrRCUtt1/kernel/Kconfig.preempt
@@ -127,10 +127,34 @@ config PREEMPT_RCU
 	  This option reduces the latency of the kernel by making certain
 	  RCU sections preemptible. Normally RCU code is non-preemptible, if
 	  this option is selected then read-only RCU sections become
-	  preemptible. This helps latency, but may increase memory utilization.
+	  preemptible. This helps latency, but may expose bugs due to
+	  now-naive assumptions about each RCU read-side critical section
+	  remaining on a given CPU through its execution.
 
 	  Say N if you are unsure.
 
+config RCU_STATS
+	bool "/proc stats for preemptible RCU read-side critical sections"
+	depends on PREEMPT_RCU
+	default y
+	help
+	  This option provides /proc stats to provide debugging info for
+	  the preemptible realtime RCU implementation.
+
+	  Say Y here if you want to see RCU stats in /proc
+	  Say N if you are unsure.
+
+config RCU_TORTURE_TEST
+	bool "/proc torture tests for RCU"
+	depends on RCU_STATS
+	default y
+	help
+	  This option provides /proc files that run torture tests on
+	  the preemptible realtime RCU implementation.
+
+	  Say Y here if you want to be able to run RCU torture tests.
+	  Say N if you are unsure.
+
 config SPINLOCK_BKL
 	bool "Old-Style Big Kernel Lock"
 	depends on (PREEMPT || SMP) && !PREEMPT_RT

kernel/rcupdate.c +740 ~~-58~~

847 lines not on this page · raw message

Ingo Molnar

2005-08-04 14:18 UTC #1 lore raw

Quoted 11 lines · Paul E. McKenney

Hello!
The attached patch passes 20 hours of the internal-to-RCU torture test (see the code in the attached patch under CONFIG_RCU_TORTURE_TEST), which is a good improvement over last week's version, which would fail this test in less than two seconds -- this despite passing kernbench with flying colors and passing LTP 90% of the time. Sometimes there is no substitute for a good in-kernel stress test...
The attached version of the patch seems to me to be ready for a larger audience.

Cool! I've merged your patch to the -52-12 version of the -RT patch. You can get it from the usual place:

http://redhat.com/~mingo/realtime-preempt/

also, i've attached a port against 2.6.13-rc4. I have done this because the PREEMPT_RCU patch in my tree is applied before the core-PREEMPT_RT patch. I have fixed up the CONFIG_PREEMPT compilation branch and have removed your #error define - it built and booted fine on an UP box but there are no guarantees ...

Ingo

make rcu_read_lock() critical sections preemptible.

 fs/proc/proc_misc.c      |   81 ++++
 include/linux/rcupdate.h |   50 ++-
 include/linux/sched.h    |    5 
 kernel/Kconfig.preempt   |   36 ++
 kernel/rcupdate.c        |  783 +++++++++++++++++++++++++++++++++++++++++++++--
 5 files changed, 918 insertions(+), 37 deletions(-)

fs/proc/proc_misc.c +81 -0

Index: linux/fs/proc/proc_misc.c
===================================================================
@@ -563,6 +563,78 @@ void create_seq_entry(char *name, mode_t
 		entry->proc_fops = f;
 }
 
+#ifdef CONFIG_RCU_STATS
+int rcu_read_proc(char *page, char **start, off_t off,
+		  int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_data(char *page);
+
+	len = rcu_read_proc_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_gp(char *page, char **start, off_t off,
+		     int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_gp_data(char *page);
+
+	len = rcu_read_proc_gp_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_ptrs(char *page, char **start, off_t off,
+		       int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_ptrs_data(char *page);
+
+	len = rcu_read_proc_ptrs_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_ctrs(char *page, char **start, off_t off,
+		       int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_ctrs_data(char *page);
+
+	len = rcu_read_proc_ctrs_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_torture_writer(char *page, char **start, off_t off,
+			         int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_torture_writer_data(char *page);
+
+	len = rcu_read_proc_torture_writer_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_torture_reader(char *page, char **start, off_t off,
+			         int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_torture_reader_data(char *page);
+
+	len = rcu_read_proc_torture_reader_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+
+int rcu_read_proc_torture_stats(char *page, char **start, off_t off,
+			        int count, int *eof, void *data)
+{
+	int len;
+	extern int rcu_read_proc_torture_stats_data(char *page);
+
+	len = rcu_read_proc_torture_stats_data(page);
+	return proc_calc_metrics(page, start, off, count, eof, len);
+}
+#endif /* #ifdef CONFIG_RCU_STATS */
+
 void __init proc_misc_init(void)
 {
 	struct proc_dir_entry *entry;

@@ -585,6 +657,15 @@ void __init proc_misc_init(void)
 		{"cmdline",	cmdline_read_proc},
 		{"locks",	locks_read_proc},
 		{"execdomains",	execdomains_read_proc},
+#ifdef CONFIG_RCU_STATS
+		{"rcustats",	rcu_read_proc},
+		{"rcugp",	rcu_read_proc_gp},
+		{"rcuptrs",	rcu_read_proc_ptrs},
+		{"rcuctrs",	rcu_read_proc_ctrs},
+		{"rcutw",	rcu_read_proc_torture_writer},
+		{"rcutr",	rcu_read_proc_torture_reader},
+		{"rcuts",	rcu_read_proc_torture_stats},
+#endif /* #ifdef CONFIG_RCU_STATS */
 		{NULL,}
 	};
 	for (p = simple_ones; p->name; p++)

include/linux/rcupdate.h +46 -4

105 lines not on this page · raw message

include/linux/sched.h +5 -0

Index: linux/include/linux/sched.h
===================================================================
@@ -623,6 +623,11 @@ struct task_struct {
 	cpumask_t cpus_allowed;
 	unsigned int time_slice, first_time_slice;
 
+#ifdef CONFIG_PREEMPT_RCU
+	int rcu_read_lock_nesting;
+	atomic_t *rcu_flipctr1;
+	atomic_t *rcu_flipctr2;
+#endif
 #ifdef CONFIG_SCHEDSTATS
 	struct sched_info sched_info;
 #endif

kernel/Kconfig.preempt +36 -0

Index: linux/kernel/Kconfig.preempt
===================================================================
@@ -63,3 +63,39 @@ config PREEMPT_BKL
 	  Say Y here if you are building a kernel for a desktop system.
 	  Say N if you are unsure.
 
+config PREEMPT_RCU
+	bool "Preemptible RCU"
+	default n
+	depends on PREEMPT
+	help
+	  This option reduces the latency of the kernel by making certain
+	  RCU sections preemptible. Normally RCU code is non-preemptible, if
+	  this option is selected then read-only RCU sections become
+	  preemptible. This helps latency, but may expose bugs due to
+	  now-naive assumptions about each RCU read-side critical section
+	  remaining on a given CPU through its execution.
+
+	  Say N if you are unsure.
+
+config RCU_STATS
+	bool "/proc stats for preemptible RCU read-side critical sections"
+	depends on PREEMPT_RCU
+	default y
+	help
+	  This option provides /proc stats to provide debugging info for
+	  the preemptible realtime RCU implementation.
+
+	  Say Y here if you want to see RCU stats in /proc
+	  Say N if you are unsure.
+
+config RCU_TORTURE_TEST
+	bool "/proc torture tests for RCU"
+	depends on RCU_STATS
+	default y
+	help
+	  This option provides /proc files that run torture tests on
+	  the preemptible realtime RCU implementation.
+
+	  Say Y here if you want to be able to run RCU torture tests.
+	  Say N if you are unsure.
+

kernel/rcupdate.c +750 ~~-33~~

821 lines not on this page · raw message

Paul E. McKenney

author

2005-08-04 14:53 UTC #2 lore raw

Quoted 18 lines · Ingo Molnar

Quoted 11 lines · Paul E. McKenney
Hello!
The attached patch passes 20 hours of the internal-to-RCU torture test (see the code in the attached patch under CONFIG_RCU_TORTURE_TEST), which is a good improvement over last week's version, which would fail this test in less than two seconds -- this despite passing kernbench with flying colors and passing LTP 90% of the time. Sometimes there is no substitute for a good in-kernel stress test...
The attached version of the patch seems to me to be ready for a larger audience.
Cool! I've merged your patch to the -52-12 version of the -RT patch. You can get it from the usual place:

Quoted 7 lines

http://redhat.com/~mingo/realtime-preempt/
also, i've attached a port against 2.6.13-rc4. I have done this because the PREEMPT_RCU patch in my tree is applied before the core-PREEMPT_RT patch. I have fixed up the CONFIG_PREEMPT compilation branch and have removed your #error define - it built and booted fine on an UP box but there are no guarantees ...

Sounds great!!! I will give it a thorough hammering in both the CONFIG_PREEMPT and CONFIG_PREEMPT_RT environments.

Thanx, Paul

Paul E. McKenney

author

2005-08-04 16:54 UTC #3 lore raw

Quoted 8 lines · Paul E. McKenney

Quoted 7 lines · Ingo Molnar
* Paul E. McKenney <[email protected]> wrote:
Cool! I've merged your patch to the -52-12 version of the -RT patch. You can get it from the usual place:
http://redhat.com/~mingo/realtime-preempt/

Quoted 8 lines

Quoted 5 lines
also, i've attached a port against 2.6.13-rc4. I have done this because the PREEMPT_RCU patch in my tree is applied before the core-PREEMPT_RT patch. I have fixed up the CONFIG_PREEMPT compilation branch and have removed your #error define - it built and booted fine on an UP box but there are no guarantees ...
Sounds great!!! I will give it a thorough hammering in both the CONFIG_PREEMPT and CONFIG_PREEMPT_RT environments.

A couple of minor things thus far:

o I get a linker error from V0.7.52-12:

net/built-in.o(.init.text+0x898): In function `sock_aio_read':
include/net/sock.h:625: undefined reference to `__you_cannot_kmalloc_that_much'

This is not making much sense to me, since line 625 of sock.h is a small function with a container_of() as its only statement. The other other kmalloc() I can find in sock_aio_read() is allocating a "struct sock_iocb", which is certainly small enough for kmalloc() to handle.

This may be a problem specific to the machine I compiled on, so am retrying on another. Though that machine had absolutely no problem compiling and running V0.7.51-27...

Enlightenment?

o I messed up the #ifdefs for CONFIG_PREEMPT_RCU, CONFIG_RCU_STATS,
and CONFIG_RCU_TORTURE_TEST so that it is not possible to build any of the following:

1. CONFIG_RCU_STATS without also CONFIG_RCU_TORTURE_TEST.

2. CONFIG_RCU_TORTURE_TEST without also CONFIG_RCU_STATS.

3. CONFIG_RCU_TORTURE_TEST without also CONFIG_PREEMPT_RCU.
Which makes it tough to torture-test stock RCU. ;-)

Am checking out fixes for these, will post once I have untangled and tested. Does not affect people who are not working directly on the RCU infrastructure itself.

Thanx, Paul