RCU torture testing
Subsystems (3)
PATCH 1/1 RCU torture testing
Hello!
The attached patch adds CONFIG_RCU_TORTURE_TEST, which enables a /proc-based intense torture test of the RCU infrastructure. This is needed due to the continued changes to RCU infrastructure to accommodate dynamic ticks, CPU hotplug, and so on. Most of the code is in a separate file that is compiled only if the CONFIG variable is set. Documentation on how to run the test and interpret the output is also included.
This code has been tested on i386 and ppc64, and an earlier version of the code has seen extensive testing on a number of architectures as part of the PREEMPT_RT patchset.
---
Documentation/RCU/proc.txt | 104 ++++++++++++++ fs/proc/proc_misc.c | 37 +++++ include/linux/rcupdate.h | 6 kernel/Kconfig.preempt | 9 + kernel/Makefile | 1 kernel/rcupdate.c | 10 + kernel/rcutorture.c | 320 +++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 487 insertions(+)
Documentation/RCU/proc.txt +104 -0
@@ -0,0 +1,104 @@ +/proc Filesystem Entries for RCU + + +CONFIG_RCU_TORTURE_TEST + +The CONFIG_RCU_TORTURE_TEST config option is available for all RCU +implementations. It makes three /proc entries available, namely: rcutw, +rcutr, and rcuts. + + +/proc/rcutw + +Reading this entry starts a new torture test, or ends an earlier one +if one is already in progress (in other words, there can be only one +writer at a time). This sleeps uninterruptibly, so be sure to run +it in the background. One could argue that it would be good to have +multiple writers, but Linux uses RCU heavily enough that you will get +write-side contention whether you want it or not. If you want additional +write-side contention, repeatedly create and destroy several large file +trees in parallel. Or use some other RCU-protected update. + + +/proc/rcutr + +Reading this entry starts a new torture reader, which runs until sent +a signal (e.g., control-C). If testing an RCU implementation with +preemptible read-side critical sections, make sure to spawn at least +two /proc/rcutr instances for each CPU. + + +/proc/rcuts + +Displays the current state of the torture test: + + ggp = 20961 + rtc: c04496f4 ver: 8734 tfle: 0 rta: 8734 rtaf: 0 rtf: 8715 + Reader Pipe: 88024120 63914 0 0 0 0 0 0 0 0 0 + Reader Batch: 88024097 63937 0 0 0 0 0 0 0 0 + Free-Block Circulation: 8733 8731 8729 8727 8725 8723 8721 8719 8717 8715 0 + +The entries are as follows: + +o "ggp": The number of counter flips (or batches) since boot. + +o "rtc": The hexadecimal address of the structure currently visible + to readers. + +o "ver": The number of times since boot that the rcutw writer task + has changed the structure visible to readers. + +o "tfle": If non-zero, indicates that the "torture freelist" + containing structure to be placed into the "rtc" area is empty. + This condition is important, since it can fool you into thinking + that RCU is working when it is not. :-/ + +o "rta": Number of structures allocated from the torture freelist. + +o "rtaf": Number of allocations from the torture freelist that have + failed due to the list being empty. + +o "rtf": Number of frees into the torture freelist. + +o "Reader Pipe": Histogram of "ages" of structures seen by readers. + If any entries past the first two are non-zero, RCU is broken. + And /proc/rcuts prints "!!!" to make sure you notice. The age + of a newly allocated structure is zero, it becomes one when + removed from reader visibility, and is incremented once per + grace period subsequently -- and is freed after passing through + (RCU_TORTURE_PIPE_LEN-2) grace periods. + + The output displayed above was taken from a correctly working + RCU. If you want to see what it looks like when broken, break + it yourself. ;-) + +o "Reader Batch": Another histogram of "ages" of structures seen + by readers, but in terms of counter flips (or batches) rather + than in terms of grace periods. The legal number of non-zero + entries is again two. The reason for this separate view is + that it is easier to get the third entry to show up in the + "Reader Batch" list than in the "Reader Pipe" list. + +o "Free-Block Circulation": Shows the number of torture structures + that have reached a given point in the pipeline. The first element + should closely correspond to the number of structures allocated, + the second to the number that have been removed from reader view, + and all but the last remaining to the corresponding number of + passes through a grace period. The last entry should be zero, + as it is only incremented if a torture structure's counter + somehow gets incremented farther than it should. + + +Usage + +The following script may be used to torture RCU: + + #!/bin/sh + + cat /proc/rcutw& + cat /proc/rcutr& cat /proc/rcutr& cat /proc/rcutr& cat /proc/rcutr& + while : + do + cat /proc/rcuts + sleep 10 + done
fs/proc/proc_misc.c +37 -0
@@ -563,6 +563,38 @@ void create_seq_entry(char *name, mode_t entry->proc_fops = f; } +#ifdef CONFIG_RCU_TORTURE_TEST +int rcu_read_proc_torture_writer (char *page,char **start,off_t off, +int count,int *eof,void *data) +{ +int len; +extern int rcu_read_proc_torture_writer_data (char *page); + + len =rcu_read_proc_torture_writer_data (page); +return proc_calc_metrics (page, start, off, count, eof, len); +} + +int rcu_read_proc_torture_reader (char *page,char **start,off_t off, +int count,int *eof,void *data) +{ +int len; +extern int rcu_read_proc_torture_reader_data (char *page); + + len =rcu_read_proc_torture_reader_data (page); +return proc_calc_metrics (page, start, off, count, eof, len); +} + +int rcu_read_proc_torture_stats (char *page,char **start,off_t off, +int count,int *eof,void *data) +{ +int len; +extern int rcu_read_proc_torture_stats_data (char *page); + + len =rcu_read_proc_torture_stats_data (page); +return proc_calc_metrics (page, start, off, count, eof, len); +} +#endif /* #ifdef CONFIG_RCU_TORTURE_TEST */ +void __init proc_misc_init (void ) {struct proc_dir_entry *entry;
@@ -585,6 +617,11 @@ void __init proc_misc_init(void) {"cmdline", cmdline_read_proc}, {"locks", locks_read_proc}, {"execdomains", execdomains_read_proc}, +#ifdef CONFIG_RCU_TORTURE_TEST + {"rcutw", rcu_read_proc_torture_writer}, + {"rcutr", rcu_read_proc_torture_reader}, + {"rcuts", rcu_read_proc_torture_stats}, +#endif /* #ifdef CONFIG_RCU_TORTURE_TEST */ {NULL,} };for (p = simple_ones; p->name; p++)
include/linux/rcupdate.h +6 -0
@@ -274,6 +274,12 @@ static inline int rcu_pending(int cpu)extern void rcu_init (void );extern void rcu_check_callbacks (int cpu,int user);extern void rcu_restart_cpu (int cpu); +extern long rcu_batches_completed (void ); +#ifdef CONFIG_RCU_TORTURE_TEST +extern void rcu_torture_init (void ); +#else +static inline void rcu_torture_init (void ) { } +#endif /* Exported interfaces */ extern void FASTCALL(call_rcu(struct rcu_head *head,
kernel/Kconfig.preempt +9 -0
@@ -63,3 +63,12 @@ config PREEMPT_BKL Say Y here if you are building a kernel for a desktop system. Say N if you are unsure. +config RCU_TORTURE_TEST + bool "/proc torture tests for RCU" + default n + help + This option provides /proc files that run torture tests on the + RCU infrastructure. + + Say Y here if you want to be able to run RCU torture tests. + Say N if you are unsure.
kernel/Makefile +1 -0
@@ -32,6 +32,7 @@ obj-$(CONFIG_DETECT_SOFTLOCKUP) += softl obj-$(CONFIG_GENERIC_HARDIRQS) += irq/ obj-$(CONFIG_CRASH_DUMP) += crash_dump.o obj-$(CONFIG_SECCOMP) += seccomp.o +obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y) # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
kernel/rcupdate.c +10 -0
@@ -144,6 +144,15 @@ void fastcall call_rcu_bh(struct rcu_hea }/* +* Return the number of RCU batches processed thus far. Useful +* for debug and statistics. +*/ +long rcu_batches_completed (void ) +{ +return rcu_ctrlblk.completed; +} + +/* * Invoke the completed RCU callbacks. They are expected to be in * a per-cpu list. */
@@ -437,6 +446,7 @@ static struct notifier_block __devinitda*/ void __init rcu_init (void ) { +rcu_torture_init ();rcu_cpu_notify (&rcu_nb,CPU_UP_PREPARE , (void *)(long )smp_processor_id ());/* Register notifier for non-boot CPUs */
kernel/rcutorture.c +320 -0
320 lines not on this page · raw message
Documentation/RCU/proc.txt line 8
Quoted 3 lines · Paul E. McKenney
Documentation/RCU/proc.txt +3
-0+The CONFIG_RCU_TORTURE_TEST config option is available for all RCU +implementations. It makes three /proc entries available, namely: rcutw, +rcutr, and rcuts.
Ick, why /proc entries, this has nothing to do with processes, right? Please use debugfs instead, that is what it was created for.
thanks,
greg k-h
Hi!
Quoted 7 lines
The attached patch adds
CONFIG_RCU_TORTURE_TEST, which enables a /proc-based intense torture test of the RCU infrastructure. This is needed due to the continued changes to RCU infrastructure to accommodate dynamic ticks, CPU hotplug, and so on. Most of the code is in a separate file that is compiled only if the CONFIG variable is set. Documentation on how to run the test and interpret the output is also included.
Quoted 5 lines
This code has been tested on i386 and ppc64, and an earlier version of the code has seen extensive testing on a number of architectures as part of the
PREEMPT_RTpatchset.Signed-off-by <[email protected]>
Can you just run the tests from time to time inside IBM?
Pavel
Signature · 1 line
if you have sharp zaurus hardware you don't need... you know my address
Quoted 7 lines · Pavel Machek
Quoted 7 lines
The attached patch adds
CONFIG_RCU_TORTURE_TEST, which enables a /proc-based intense torture test of the RCU infrastructure. This is needed due to the continued changes to RCU infrastructure to accommodate dynamic ticks, CPU hotplug, and so on. Most of the code is in a separate file that is compiled only if the CONFIG variable is set. Documentation on how to run the test and interpret the output is also included.
Quoted 7 lines
Quoted 5 lines
This code has been tested on i386 and ppc64, and an earlier version of the code has seen extensive testing on a number of architectures as part of the
PREEMPT_RTpatchset.Signed-off-by <[email protected]>Can you just run the tests from time to time inside IBM?
actually, i think RCU_TORTURE_TEST is a prime example of why this should be in the main kernel. The torture-test found RCU bugs that never popped up during normal use, and RCU bugs are hillariously hard to debug. Having that code outside of the main kernel will most certainly result in a degradation of the test - while it's role for the development of new architectures (and changes to the rcu code) is very important. I agree it should move to debugfs, and maybe in the future we'll need some better framework for a 'in-kernel integrated testsuite'. I think we should encourage such testsuites. I was thinking about writing something similar for spinlocks and rwlocks and semaphores. (we occasionally break them and they are hard to debug too)
Ingo
Quoted 7 lines · Greg KH
Documentation/RCU/proc.txt line 8
Quoted 3 lines · Paul E. McKenney
Documentation/RCU/proc.txt +3
-0+The CONFIG_RCU_TORTURE_TEST config option is available for all RCU +implementations. It makes three /proc entries available, namely: rcutw, +rcutr, and rcuts.Ick, why /proc entries, this has nothing to do with processes, right? Please use debugfs instead, that is what it was created for.
OK, will look into that. At first glance, it does appear to require quite a bit more code to make use of than did the /proc filesystem, if you want the files to produce human-readable strings, as is appropriate in this case. I am looking at uhci-debug.c -- is there an example that better matches what I am trying to do?
Thanx, Paul
Quoted 9 lines · Pavel Machek
Hi!
Quoted 7 lines
The attached patch adds
CONFIG_RCU_TORTURE_TEST, which enables a /proc-based intense torture test of the RCU infrastructure. This is needed due to the continued changes to RCU infrastructure to accommodate dynamic ticks, CPU hotplug, and so on. Most of the code is in a separate file that is compiled only if the CONFIG variable is set. Documentation on how to run the test and interpret the output is also included.
Quoted 7 lines
Quoted 5 lines
This code has been tested on i386 and ppc64, and an earlier version of the code has seen extensive testing on a number of architectures as part of the
PREEMPT_RTpatchset.Signed-off-by <[email protected]>Can you just run the tests from time to time inside IBM?
In principle, I could, but in practice it is appropriate for non-IBMers to be able to test the RCU infrastructure easily and thoroughly when they work on it.
Thanx, Paul
Quoted 14 lines · Paul E. McKenney
Quoted 13 lines · Pavel Machek
Hi!
Quoted 11 lines
The attached patch adds
CONFIG_RCU_TORTURE_TEST, which enables a /proc-based intense torture test of the RCU infrastructure. This is needed due to the continued changes to RCU infrastructure to accommodate dynamic ticks, CPU hotplug, and so on. Most of the code is in a separate file that is compiled only if the CONFIG variable is set. Documentation on how to run the test and interpret the output is also included.This code has been tested on i386 and ppc64, and an earlier version of the code has seen extensive testing on a number of architectures as part of the
PREEMPT_RTpatchset.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Signed-off-by <[email protected]>Can you just run the tests from time to time inside IBM?
In principle, I could, but in practice it is appropriate for non-IBMers to be able to test the RCU infrastructure easily and thoroughly when they work on it.
how hard would it be to make the few parameters just be module options... and then fail module load if the test fails or something? (and spew loudly in dmesg :)
I'd be all in favor of having such a module in the kernel; in fact it would be nice if we roughly could standardize on an way to load/start and then find the result, I'd love to have a "make runtests" or something that would load such modules one by one
(and no that's not the task of ltp, ltp should test userspace; things that test in kernel code should really be part of the kernel)
Quoted 11 lines · Paul E. McKenney
Quoted 7 lines · Greg KH
Documentation/RCU/proc.txt line 8
Quoted 3 lines · Paul E. McKenney
Documentation/RCU/proc.txt +3
-0+The CONFIG_RCU_TORTURE_TEST config option is available for all RCU +implementations. It makes three /proc entries available, namely: rcutw, +rcutr, and rcuts.Ick, why /proc entries, this has nothing to do with processes, right? Please use debugfs instead, that is what it was created for.
OK, will look into that. At first glance, it does appear to require quite a bit more code to make use of than did the /proc filesystem,
I wasn't going for "least lines of code" here, like I did for sysfs. Although if you do have a "simple" datatype, it's less code than for procfs.
Quoted 3 lines
if you want the files to produce human-readable strings, as is appropriate in this case. I am looking at
uhci-debug.c-- is there an example that better matches what I am trying to do?
Just provide a "read" function, that's exactly what people do for proc these days, it shouldn't be tough to switch over.
thanks,
greg k-h
Quoted 11 lines · Arjan van de Ven
Quoted 6 lines · Paul E. McKenney
Quoted 1 line · Pavel Machek
Can you just run the tests from time to time inside IBM?
In principle, I could, but in practice it is appropriate for non-IBMers to be able to test the RCU infrastructure easily and thoroughly when they work on it.
how hard would it be to make the few parameters just be module options... and then fail module load if the test fails or something? (and spew loudly in dmesg :)
Good point -- all I really need for module parameters is the number of readers. I should be able to have module load start the test and module unload stop it (any problems with this approach?). And doing a module should remove the intrusions into rcupdate.c and rcupdate.h, which would be good.
I would rather avoid dmesg. But perhaps a read-only debugfs for output (as Greg suggested) combined with module parameters for input could make this straightforward.
Quoted 4 lines
I'd be all in favor of having such a module in the kernel; in fact it would be nice if we roughly could standardize on an way to load/start and then find the result, I'd love to have a "make runtests" or something that would load such modules one by one
Which would mean that each test needs to give unambiguous machine-readable indication of failure. I guess I will nominate the string "!!!". ;-)
Quoted 2 lines
(and no that's not the task of ltp, ltp should test userspace; things that test in kernel code should really be part of the kernel)
I agree that there is definitely a need for both user-level and in-kernel testing. User-level testing is needed to make sure that user programs get what they need, but there is no substitute for in-kernel testing when you need to apply maximum conceiveable stress on some kernel component.
Thanx, Paul
Quoted 4 lines
Good point -- all I really need for module parameters is the number of readers. I should be able to have module load start the test and module unload stop it (any problems with this approach?).
only one potential gotcha; it means you can't load the system to the extend that the shell doesn't get cpu time otherwise the admin can never issue the unload. Maybe a time limit on the testing? (optional as module parm for all I care)
Quoted 9 lines · Arjan van de Ven
Quoted 4 lines
Good point -- all I really need for module parameters is the number of readers. I should be able to have module load start the test and module unload stop it (any problems with this approach?).
only one potential gotcha; it means you can't load the system to the extend that the shell doesn't get cpu time otherwise the admin can never issue the unload. Maybe a time limit on the testing? (optional as module parm for all I care)
Makes sense to me! Not an issue for the RCU torture test, but it certainly might be for other tests.
Thanx, Paul
Quoted 9 lines · Arjan van de Ven
Quoted 4 lines
Good point -- all I really need for module parameters is the number of readers. I should be able to have module load start the test and module unload stop it (any problems with this approach?).
only one potential gotcha; it means you can't load the system to the extend that the shell doesn't get cpu time otherwise the admin can never issue the unload. Maybe a time limit on the testing? (optional as module parm for all I care)
OK... I tried both debugfs and modules, and the modules approach turned out to be much more tester-friendly. With debugfs, one must do a special build of the kernel to enable debugfs -- which will also include other test code that might skew results -- and the fact that debugfs does not yet seem to have a standard mount point makes scripting less effective.
In contrast, the modules approach can be used even on an already running kernel that you did not plan to test ahead of time.
So modules it is. Patch submitted separately.
Thanx, Paul