Make rcutorture more vicious: add stutter feature

Hello!

This patch takes a step towards making rcutorture more brutal by allowing the test to be automatically periodically paused, with the default being to run the test for five seconds then pause for five seconds and repeat. This behavior can be controlled using a new "stutter" module parameter, so that "stutter=0" gives the old default behavior of running continuously.

Starting and stopping rcutorture more heavily stresses RCU's interaction with the scheduler, as well as exercising more paths through the grace-period detection code.

Note that the default to "shuffle_interval" has also been adjusted from 5 seconds to 3 seconds to provide varying overlap with the "stutter" interval.

I am still unable to provoke the failures that Alexey has been seeing, even with this patch, but will be doing a few additional things to beef up rcutorture.

Hello again!

This patch re-institutes the ability to build rcutorture directly into the Linux kernel. The reason that this capability was removed was that this could result in your kernel being pretty much useless, as rcutorture would be running starting from early boot. This problem has been avoided by (1) making rcutorture run only three seconds of every six by default,
(2) adding a CONFIG_RCU_TORTURE_TEST_RUNNABLE that permits rcutorture to be quiesced at boot time, and (3) adding a sysctl in /proc named /proc/sys/kernel/rcutorture_runnable that permits rcutorture to be quiesced and unquiesced when built into the kernel.

Please note that this /proc file is -not- available when rcutorture is built as a module. Please also note that to get the earlier take-no-prisoners behavior, you must use the boot command line to set rcutorture's "stutter" parameter to zero.

The rcutorture quiescing mechanism is currently quite crude: loops in each rcutorture process that poll a global variable once per tick. Suggestions for improvement are welcome. The default action will be to reduce the polling rate to a few times per second.

Signed-off-by Paul E. McKenney <[email protected]>

Suggested-by Ingo Molnar <[email protected]>

Hello again!

This patch re-institutes the ability to build rcutorture directly into the Linux kernel. The reason that this capability was removed was that this could result in your kernel being pretty much useless, as rcutorture would be running starting from early boot. This problem has been avoided by (1) making rcutorture run only three seconds of every six by default, (2) adding a CONFIG_RCU_TORTURE_TEST_RUNNABLE that permits rcutorture to be quiesced at boot time, and (3) adding a sysctl in /proc named /proc/sys/kernel/rcutorture_runnable that permits rcutorture to be quiesced and unquiesced when built into the kernel.

Please note that this /proc file is -not- available when rcutorture is built as a module. Please also note that to get the earlier take-no-prisoners behavior, you must use the boot command line to set rcutorture's "stutter" parameter to zero.

The rcutorture quiescing mechanism is currently quite crude: loops in each rcutorture process that poll a global variable once per tick. Suggestions for improvement are welcome. The default action will be to reduce the polling rate to a few times per second.

Quoted 18 lines · Paul E. McKenney

Hello again!

This patch re-institutes the ability to build rcutorture directly into the Linux kernel. The reason that this capability was removed was that this could result in your kernel being pretty much useless, as rcutorture would be running starting from early boot. This problem has been avoided by (1) making rcutorture run only three seconds of every six by default, (2) adding a CONFIG_RCU_TORTURE_TEST_RUNNABLE that permits rcutorture to be quiesced at boot time, and (3) adding a sysctl in /proc named /proc/sys/kernel/rcutorture_runnable that permits rcutorture to be quiesced and unquiesced when built into the kernel.

Please note that this /proc file is -not- available when rcutorture is built as a module. Please also note that to get the earlier take-no-prisoners behavior, you must use the boot command line to set rcutorture's "stutter" parameter to zero.

Quoted 4 lines

The rcutorture quiescing mechanism is currently quite crude: loops in each rcutorture process that poll a global variable once per tick. Suggestions for improvement are welcome. The default action will be to reduce the polling rate to a few times per second.

applied to tip/core/rcu - thanks Paul!

FYI, yesterday's rcutorture-stutter feature survived a few hundred random bootup tests in -tip testing already.

a possible area for enhancement would be the following code:

static void
rcu_stutter_wait(void)

{
        while (stutter_pause_test || !rcutorture_runnable)
                schedule_timeout_interruptible(1);
}

will cause HZ number of IRQs on nohz systems, even if rcutorture is disabled. While it's fine to poll if rcutorture_runnable==1, the sleep should be at least 1 second when !rcutorture_runnable.

[ Or it might even make sense to completely stop the threads from a
sysctl handler - to make it event-driven and to cause no extra IRQs at all when rcutorture is disabled via /proc. ]

Quoted 5 lines · Ingo Molnar

Quoted 2 lines · Paul E. McKenney

Hello again!

 Documentation/RCU/torture.txt      |   21 
 Documentation/RCU/torture.txt.orig |    8 

 kernel/rcutorture.c                |   14 
 kernel/rcutorture.c.orig           |   59 

 kernel/sysctl.c                    |   13 
 kernel/sysctl.c.orig               | 2851 +++++++++++++++++++++++++++++++++++++

 lib/Kconfig.debug                  |   20 
 lib/Kconfig.debug.orig             |  679 ++++++++

 8 files changed, 3651 insertions(+), 14 deletions(-)

Quoted 9 lines · On Thu, 19 Jun 2008 08:58:54 -0700 Paul E. McKenney wrote:

Quoted 5 lines · Ingo Molnar

Quoted 2 lines · Paul E. McKenney

Hello again!

 Documentation/RCU/torture.txt      |   21 
 Documentation/RCU/torture.txt.orig |    8 

                                ~~~~~

Quoted 2 lines

 kernel/rcutorture.c                |   14 
 kernel/rcutorture.c.orig           |   59 

                       !!!!

Quoted 2 lines

 kernel/sysctl.c                    |   13 
 kernel/sysctl.c.orig               | 2851 +++++++++++++++++++++++++++++++++++++

                   ~~~~

Quoted 2 lines

 lib/Kconfig.debug                  |   20 
 lib/Kconfig.debug.orig             |  679 ++++++++

                     ~~~~

What's with all of these .orig files, eh??

Quoted 1 line

 8 files changed, 3651 insertions(+), 14 deletions(-)

Quoted 19 lines · Randy Dunlap

Quoted 9 lines · On Thu, 19 Jun 2008 08:58:54 -0700 Paul E. McKenney wrote:

Quoted 5 lines · Ingo Molnar

Quoted 2 lines · Paul E. McKenney

Hello again!

 Documentation/RCU/torture.txt      |   21 
 Documentation/RCU/torture.txt.orig |    8 

                                ~~~~~

Quoted 2 lines

 kernel/rcutorture.c                |   14 
 kernel/rcutorture.c.orig           |   59 

                       !!!!

Quoted 2 lines

 kernel/sysctl.c                    |   13 
 kernel/sysctl.c.orig               | 2851 +++++++++++++++++++++++++++++++++++++

Quoted 7 lines

                   ~~~~

Quoted 2 lines

 lib/Kconfig.debug                  |   20 
 lib/Kconfig.debug.orig             |  679 ++++++++

                     ~~~~

What's with all of these .orig files, eh??

Gah, typos generating the diffs. :-/

Quoted 1 line

Quoted 1 line

 8 files changed, 3651 insertions(+), 14 deletions(-)

How about the following instead? (I expect to be following up with a version that gets rid of the threads in response to writing a "0" to /proc/sys/kernel/rcutorture_runnable later on.)

Signed-off-by Paul E. McKenney <[email protected]>

---

 Documentation/RCU/torture.txt |   21 ++++++++++++++-------
 kernel/rcutorture.c           |   14 ++++++++++++--
 kernel/sysctl.c               |   13 +++++++++++++
 lib/Kconfig.debug             |   20 +++++++++++++++++++-
 4 files changed, 58 insertions(+), 10 deletions(-)

diff -urpNa -X dontdiff linux-2.6.26-rc4-rcut1-stutter/Documentation/RCU/torture.txt linux-2.6.26-rc4-rcut2-proc/Documentation/RCU/torture.txt

Documentation/RCU/torture.txt +14 -7

@@ -10,13 +10,20 @@ status messages via printk(), which can 
 command (perhaps grepping for "torture").  The test is started
 when the module is loaded, and stops when the module is unloaded.
 
-However, actually setting this config option to "y" results in the system
-running the test immediately upon boot, and ending only when the system
-is taken down.  Normally, one will instead want to build the system
-with CONFIG_RCU_TORTURE_TEST=m and to use modprobe and rmmod to control
-the test, perhaps using a script similar to the one shown at the end of
-this document.  Note that you will need CONFIG_MODULE_UNLOAD in order
-to be able to end the test.
+CONFIG_RCU_TORTURE_TEST_RUNNABLE
+
+It is also possible to specify CONFIG_RCU_TORTURE_TEST=y, which will
+result in the tests being loaded into the base kernel.  In this case,
+the CONFIG_RCU_TORTURE_TEST_RUNNABLE config option is used to specify
+whether the RCU torture tests are to be started immediately during
+boot or whether the /proc/sys/kernel/rcutorture_runnable file is used
+to enable them.  This /proc file can be used to repeatedly pause and
+restart the tests, regardless of the initial state specified by the
+CONFIG_RCU_TORTURE_TEST_RUNNABLE config option.
+
+You will normally -not- want to start the RCU torture tests during boot
+(and thus the default is CONFIG_RCU_TORTURE_TEST_RUNNABLE=n), but doing
+this can sometimes be useful in finding boot-time bugs.
 
 
 MODULE PARAMETERS

diff -urpNa -X dontdiff linux-2.6.26-rc4-rcut1-stutter/kernel/rcutorture.c linux-2.6.26-rc4-rcut2-proc/kernel/rcutorture.c

kernel/rcutorture.c +12 -2

@@ -125,6 +125,13 @@ static struct list_head rcu_torture_remo
 
 static int stutter_pause_test = 0;
 
+#if defined(MODULE) || defined(CONFIG_RCU_TORTURE_TEST_RUNNABLE)
+#define RCUTORTURE_RUNNABLE_INIT 1
+#else
+#define RCUTORTURE_RUNNABLE_INIT 0
+#endif
+int rcutorture_runnable = RCUTORTURE_RUNNABLE_INIT;
+
 /*
  * Allocate an element from the rcu_tortures pool.
  */

@@ -188,8 +195,11 @@ rcu_random(struct rcu_random_state *rrsp
 static void
 rcu_stutter_wait(void)
 {
-	while (stutter_pause_test)
-		schedule_timeout_interruptible(1);
+	while (stutter_pause_test || !rcutorture_runnable)
+		if (rcutorture_runnable)
+			schedule_timeout_interruptible(1);
+		else
+			schedule_timeout_interruptible(HZ);
 }
 
 /*

diff -urpNa -X dontdiff linux-2.6.26-rc4-rcut1-stutter/kernel/sysctl.c linux-2.6.26-rc4-rcut2-proc/kernel/sysctl.c

kernel/sysctl.c +13 -0

@@ -82,6 +82,9 @@ extern int maps_protect;
 extern int sysctl_stat_interval;
 extern int latencytop_enabled;
 extern int sysctl_nr_open_min, sysctl_nr_open_max;
+#ifdef CONFIG_RCU_TORTURE_TEST
+extern int rcutorture_runnable;
+#endif /* #ifdef CONFIG_RCU_TORTURE_TEST */
 
 /* Constants used for minimum and  maximum */
 #if defined(CONFIG_DETECT_SOFTLOCKUP) || defined(CONFIG_HIGHMEM)

@@ -813,6 +816,16 @@ static struct ctl_table kern_table[] = {
 		.child		= key_sysctls,
 	},
 #endif
+#ifdef CONFIG_RCU_TORTURE_TEST
+	{
+		.ctl_name       = CTL_UNNUMBERED,
+		.procname       = "rcutorture_runnable",
+		.data           = &rcutorture_runnable,
+		.maxlen         = sizeof(int),
+		.mode           = 0644,
+		.proc_handler   = &proc_dointvec,
+	},
+#endif
 /*
  * NOTE: do not add new entries to this table unless you have read
  * Documentation/sysctl/ctl_unnumbered.txt

diff -urpNa -X dontdiff linux-2.6.26-rc4-rcut1-stutter/lib/Kconfig.debug linux-2.6.26-rc4-rcut2-proc/lib/Kconfig.debug

lib/Kconfig.debug +13 -1

@@ -531,16 +531,34 @@ config BOOT_PRINTK_DELAY
 config RCU_TORTURE_TEST
 	tristate "torture tests for RCU"
 	depends on DEBUG_KERNEL
-	depends on m
 	default n
 	help
 	  This option provides a kernel module that runs torture tests
 	  on the RCU infrastructure.  The kernel module may be built
 	  after the fact on the running kernel to be tested, if desired.
 
+	  Say Y here if you want RCU torture tests to be built into
+	  the kernel.
 	  Say M if you want the RCU torture tests to build as a module.
 	  Say N if you are unsure.
 
+config RCU_TORTURE_TEST_RUNNABLE
+	bool "torture tests for RCU runnable by default"
+	depends on RCU_TORTURE_TEST = y
+	default n
+	help
+	  This option provides a way to build the RCU torture tests
+	  directly into the kernel without them starting up at boot
+	  time.  You can use /proc/sys/kernel/rcutorture_runnable
+	  to manually override this setting.  This /proc file is
+	  available only when the RCU torture tests have been built
+	  into the kernel.

This patch makes the non-module rcutorture a bit more friendly to the power-conservation code. This is a rather simple-minded approach. More sophisticated approaches would get rid of the rcutorture tasks while rcutorture execution was suppressed, but attempts thus far to do this have not gone well -- calling rcu_torture_init() from a /proc callout results in oopses.

Signed-off-by Paul E. McKenney <[email protected]>

---

 rcutorture.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff -urpNa -X dontdiff linux-2.6.26-rc4-rcut2-proc/kernel/rcutorture.c linux-2.6.26-rc4-rcut3-procq/kernel/rcutorture.c ---
linux-2.6.26-rc4-rcut2-proc/kernel/rcutorture.c 2008-06-22
10:29:04.000000000 -0700 +++
linux-2.6.26-rc4-rcut3-procq/kernel/rcutorture.c 2008-06-22
12:20:10.000000000 -0700 @@ -196,7 +196,10 @@ static void rcu_stutter_wait(void) { while (stutter_pause_test || !rcutorture_runnable)

Quoted 24 lines · On Sun, 22 Jun 2008 13:06:38 -0700 "Paul E. McKenney"…

This patch makes the non-module rcutorture a bit more friendly to the power-conservation code. This is a rather simple-minded approach. More sophisticated approaches would get rid of the rcutorture tasks while rcutorture execution was suppressed, but attempts thus far to do this have not gone well -- calling rcu_torture_init() from a /proc callout results in oopses.

Signed-off-by Paul E. McKenney <[email protected]>

---

 rcutorture.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff -urpNa -X dontdiff linux-2.6.26-rc4-rcut2-proc/kernel/rcutorture.c linux-2.6.26-rc4-rcut3-procq/kernel/rcutorture.c ---
linux-2.6.26-rc4-rcut2-proc/kernel/rcutorture.c 2008-06-22
10:29:04.000000000 -0700 +++
linux-2.6.26-rc4-rcut3-procq/kernel/rcutorture.c 2008-06-22
12:20:10.000000000 -0700 @@ -196,7 +196,10 @@ static void rcu_stutter_wait(void) { while (stutter_pause_test || !rcutorture_runnable)

(quoted diff) +1 -1

-		schedule_timeout_interruptible(1);
+		if (rcutorture_runnable)

Review

Quoted 4 lines

(quoted diff) +3 -0

+			schedule_timeout_interruptible(1);
+		else
+			schedule_timeout_interruptible(HZ);
 }

could you also make it use round_jiffies_relative() to make the power impact even less.....
(by coalescing various "once a second" timers like this)

Quoted 3 lines

could you also make it use round_jiffies_relative() to make the power impact even less..... (by coalescing various "once a second" timers like this)

Good point!!! As follows, on top of the previous one.

This patch aligns the rcutorture wakeup times to align with all other multiple-of-a-second wakeups to further decrease power consumption.

Suggested-by Arjan van de Ven <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Quoted 24 lines · On Sun, 22 Jun 2008 13:06:38 -0700 "Paul E. McKenney"…

This patch makes the non-module rcutorture a bit more friendly to the power-conservation code. This is a rather simple-minded approach. More sophisticated approaches would get rid of the rcutorture tasks while rcutorture execution was suppressed, but attempts thus far to do this have not gone well -- calling rcu_torture_init() from a /proc callout results in oopses.

Signed-off-by Paul E. McKenney <[email protected]>

---

 rcutorture.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff -urpNa -X dontdiff linux-2.6.26-rc4-rcut2-proc/kernel/rcutorture.c linux-2.6.26-rc4-rcut3-procq/kernel/rcutorture.c ---
linux-2.6.26-rc4-rcut2-proc/kernel/rcutorture.c 2008-06-22
10:29:04.000000000 -0700 +++
linux-2.6.26-rc4-rcut3-procq/kernel/rcutorture.c 2008-06-22
12:20:10.000000000 -0700 @@ -196,7 +196,10 @@ static void rcu_stutter_wait(void) { while (stutter_pause_test || !rcutorture_runnable)

kernel/rcutorture.c +1 -1

-		schedule_timeout_interruptible(1);
+		if (rcutorture_runnable)

kernel/rcutorture.c static void · line 203

Quoted 4 lines

kernel/rcutorture.c +3 -0

+			schedule_timeout_interruptible(1);
+		else
+			schedule_timeout_interruptible(HZ);
 }

could you also make it use round_jiffies_relative() to make the power impact even less.....
(by coalescing various "once a second" timers like this)

I'm a little concerned about how this will affect real-time performance, as queueing up lots of timers all at once can lead to long running timer expiration handlers. If just a schedule_timeout, I suppose we are only looking at a process wakeup, as opposed to a softirq context callback function?

Quoted 5 lines · On Mon, 23 Jun 2008 17:54:09 +0000 Darren Hart <[email protected]> wrote:

I'm a little concerned about how this will affect real-time performance, as queueing up lots of timers all at once can lead to long running timer expiration handlers. If just a schedule_timeout, I suppose we are only looking at a process wakeup, as opposed to a softirq context callback function?

in reality, the time it takes to deliver the interrupt (including waking the CPU up etc), is likely to be an order or two of magnitude higher than this kind of code loop....

Quoted 9 lines · Arjan van de Ven

Quoted 5 lines · On Mon, 23 Jun 2008 17:54:09 +0000 Darren Hart <[email protected]> wrote:

I'm a little concerned about how this will affect real-time performance, as queueing up lots of timers all at once can lead to long running timer expiration handlers. If just a schedule_timeout, I suppose we are only looking at a process wakeup, as opposed to a softirq context callback function?

Quoted 3 lines

in reality, the time it takes to deliver the interrupt (including waking the CPU up etc), is likely to be an order or two of magnitude higher than this kind of code loop....

Sure, if we just look at one of them. Any idea how many such items we're looking at rounding up to fire at the same time? Is it dozens, hundreds, thousands?

Quoted 9 lines · Arjan van de Ven

Quoted 5 lines · On Mon, 23 Jun 2008 17:54:09 +0000 Darren Hart <[email protected]> wrote:

I'm a little concerned about how this will affect real-time performance, as queueing up lots of timers all at once can lead to long running timer expiration handlers. If just a schedule_timeout, I suppose we are only looking at a process wakeup, as opposed to a softirq context callback function?

Quoted 3 lines

in reality, the time it takes to deliver the interrupt (including waking the CPU up etc), is likely to be an order or two of magnitude higher than this kind of code loop....

Sure, if we just look at one of them. Any idea how many such items we're looking at rounding up to fire at the same time? Is it dozens, hundreds, thousands?

Quoted 14 lines · Darren Hart

Quoted 12 lines · Arjan van de Ven

Quoted 5 lines · On Mon, 23 Jun 2008 17:54:09 +0000 Darren Hart <[email protected]> wrote:

I'm a little concerned about how this will affect real-time performance, as queueing up lots of timers all at once can lead to long running timer expiration handlers. If just a schedule_timeout, I suppose we are only looking at a process wakeup, as opposed to a softirq context callback function?

in reality, the time it takes to deliver the interrupt (including waking the CPU up etc), is likely to be an order or two of magnitude higher than this kind of code loop....

Quoted 3 lines

Sure, if we just look at one of them. Any idea how many such items we're looking at rounding up to fire at the same time? Is it dozens, hundreds, thousands?

Hello, Darren,

Wouldn't these timers be running at low priority, so that high-priority realtime tasks would preempt them?

Thanx, Paul

This patch makes the non-module rcutorture a bit more friendly to the power-conservation code. This is a rather simple-minded approach. More sophisticated approaches would get rid of the rcutorture tasks while rcutorture execution was suppressed, but attempts thus far to do this have not gone well -- calling rcu_torture_init() from a /proc callout results in oopses.

Quoted 6 lines · Paul E. McKenney

This patch makes the non-module rcutorture a bit more friendly to the power-conservation code. This is a rather simple-minded approach. More sophisticated approaches would get rid of the rcutorture tasks while rcutorture execution was suppressed, but attempts thus far to do this have not gone well -- calling rcu_torture_init() from a /proc callout results in oopses.

Hi Paul,

I applied your three patches

[PATCH] Make rcutorture more vicious: add stutter feature [PATCH] Make rcutorture more vicious: reinstate boot-time testing [PATCH -tip-rcu] Make rcutorture more vicious: make quiescent rcutorture less power-hungry

to v2.6.26-rc7 and gave it a quick testing in qemu. But it seems to hang during gdb self-tests at boot:

This patch allows torturing RCU from irq handlers (timers, in this case). A new module parameter irqreader enables such additional torturing, and is enabled by default. Variants of RCU that do not tolerate readers being called from irq handlers (e.g., SRCU) ignore irqreader.

(No failures observed, but only short tests thus far. And no failures observed thus far from about 10 hours of running stock rcutorture in parallel with 170 kernel builds on a two-CPU machine.)

+static int irqreader = 1; /* RCU readers from irq (timers). */

Quoted 9 lines · Paul E. McKenney

This patch allows torturing RCU from irq handlers (timers, in this case). A new module parameter irqreader enables such additional torturing, and is enabled by default. Variants of RCU that do not tolerate readers being called from irq handlers (e.g., SRCU) ignore irqreader.

(No failures observed, but only short tests thus far. And no failures observed thus far from about 10 hours of running stock rcutorture in parallel with 170 kernel builds on a two-CPU machine.)

applied to tip/core/rcu - thanks Paul.

Quoted 1 line

+static int irqreader = 1; /* RCU readers from irq (timers). */

Good - this means that in bzImage type of tests (built-in rcutorture, no modules) this new test variant will be activated by default, right?