Add read-side scalability tests for v5.9
Subsystems (5)
Paul E. McKenney
maintainercover Add read-side scalability tests for v5.9
Hello!
This series adds a refscale set of read-side scalability tests for reference counting, reader-writer locking, and RCU.
1. Remove useless while loops around wait_event, courtesy of
Joel Fernandes.
2. Add a test to measure performance of read-side synchronization,
courtesy of Joel Fernandes.
3. Add comments explaining the high reader overhead.
4. Add refperf to the rcutorture scripting.
5. Add holdoff parameter to allow CPUs to come online.
6. Hoist function-pointer calls out of the loop.
7. Allow decimal nanoseconds.
8. Convert nreaders to a module parameter.
9. Provide module parameter to specify number of experiments.
10. Dynamically allocate experiment-summary output buffer.
11. Dynamically allocate thread-summary output buffer.
12. Make functions static.
13. Tune reader measurement interval.
14. Convert reader_task structure's "start" field to int.
15. More closely synchronize reader start times.
16. Add warmup and cooldown processing phases.
17. Label experiment-number column "Runs".
18. Output per-experiment data points.
19. Simplify initialization-time wakeup protocol.
20. Add read-side delay module parameter.
21. Adjust refperf.loop default value.
22. Document rcuperf's module parameters.
23. Work around 64-bit division, courtesy of Arnd Bergmann.
24. Change readdelay module parameter to nanoseconds.
25. Add test for RCU Tasks Trace readers..
26. Add test for RCU Tasks readers.
27. rcu-tasks: Fix synchronize_rcu_tasks_trace() header comment.
28. Rename RCU_REF_PERF_TEST to RCU_REF_SCALE_TEST.
29. Rename refperf.c to refscale.c and change internal names.
30. Change --torture type from refperf to refscale.
Thanx, Paul
------------------------------------------------------------------------
Documentation/admin-guide/kernel-parameters.txt | 53 include/linux/rcupdate_trace.h | 4 kernel/rcu/Kconfig.debug | 23 kernel/rcu/Makefile | 5 kernel/rcu/rcuperf.c | 25 kernel/rcu/refperf.c | 1037 ++++++++-- kernel/rcu/refscale.c | 182 - kernel/rcu/tasks.h | 9 tools/testing/selftests/rcutorture/bin/kvm-recheck-refperf.sh | 73 tools/testing/selftests/rcutorture/bin/kvm-recheck-refscale.sh | 8 tools/testing/selftests/rcutorture/bin/kvm.sh | 17 tools/testing/selftests/rcutorture/bin/parse-console.sh | 8 tools/testing/selftests/rcutorture/configs/refperf/CFLIST | 2 tools/testing/selftests/rcutorture/configs/refperf/CFcommon | 4 tools/testing/selftests/rcutorture/configs/refperf/NOPREEMPT | 18 tools/testing/selftests/rcutorture/configs/refperf/PREEMPT | 18 tools/testing/selftests/rcutorture/configs/refperf/ver_functions.sh | 20 tools/testing/selftests/rcutorture/configs/refscale/ver_functions.sh | 1 18 files changed, 1205 insertions(+), 302 deletions(-)
[email protected]
maintainerPATCH 1/30 rcuperf: Remove useless while loops around wait_event
From: "Joel Fernandes (Google)" <[email protected]>
wait_event() already retries if the condition for the wake up is not satisifed after wake up. Remove them from the rcuperf test.
--- kernel/rcu/rcuperf.c | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-)
kernel/rcu/rcuperf.c +4 -10
@@ -576,11 +576,8 @@ static int compute_real(int n)static int rcu_perf_shutdown (void *arg) { -do { -wait_event (shutdown_wq, -atomic_read (&n_rcu_perf_writer_finished) >= - nrealwriters); - }while (atomic_read (&n_rcu_perf_writer_finished) < nrealwriters); +wait_event (shutdown_wq, +atomic_read (&n_rcu_perf_writer_finished) >= nrealwriters);smp_mb ();/* Wake before output. */ rcu_perf_cleanup ();kernel_power_off ();
@@ -693,11 +690,8 @@ kfree_perf_cleanup(void)static int kfree_perf_shutdown (void *arg) { -do { -wait_event (shutdown_wq, -atomic_read (&n_kfree_perf_thread_ended) >= - kfree_nrealthreads); - }while (atomic_read (&n_kfree_perf_thread_ended) < kfree_nrealthreads); +wait_event (shutdown_wq, +atomic_read (&n_kfree_perf_thread_ended) >= kfree_nrealthreads);smp_mb ();/* Wake before output. */
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 2/30 refperf: Add a test to measure performance of read-side synchronization
In reply to Paul E. McKenneyFrom: "Joel Fernandes (Google)" <[email protected]>
Add a test for comparing the performance of RCU with various read-side synchronization mechanisms. The test has proved useful for collecting data and performing these comparisons.
Currently RCU, SRCU, reader-writer lock, reader-writer semaphore and reference counting can be measured using refperf.perf_type parameter. Each invocation of the test runs measures performance of a specific mechanism.
The maximum number of CPUs to concurrently run readers on is chosen by the test itself and is 75% of the total number of CPUs. So if you had 24 CPUs, the test runs with a maximum of 18 parallel readers.
A number of experiments are conducted, and in each experiment, the number of readers is increased by 1, upto the 75% of CPUs mark. During each experiment, all readers execute an empty loop with refperf.loops iterations and time the total loop duration. This is then averaged.
Example output:
Parameters "refperf.perf_type=srcu refperf.loops=2000000" looks like:
[ 3.347133] srcu-ref-perf: [ 3.347133] Threads Time(ns) [ 3.347133] 1 36 [ 3.347133] 2 34 [ 3.347133] 3 34 [ 3.347133] 4 34 [ 3.347133] 5 33 [ 3.347133] 6 33 [ 3.347133] 7 33 [ 3.347133] 8 33 [ 3.347133] 9 33 [ 3.347133] 10 33 [ 3.347133] 11 33 [ 3.347133] 12 33 [ 3.347133] 13 33 [ 3.347133] 14 33 [ 3.347133] 15 32 [ 3.347133] 16 33 [ 3.347133] 17 33 [ 3.347133] 18 34
--- kernel/rcu/Kconfig.debug | 19 ++ kernel/rcu/Makefile | 1 + kernel/rcu/refperf.c | 558 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 578 insertions(+) create mode 100644 kernel/rcu/refperf.c
kernel/rcu/Kconfig.debug +19 -0
@@ -61,6 +61,25 @@ config RCU_TORTURE_TEST Say M if you want the RCU torture tests to build as a module. Say N if you are unsure. +config RCU_REF_PERF_TEST + tristate "Performance tests for read-side synchronization (RCU and others)" + depends on DEBUG_KERNEL + select TORTURE_TEST + select SRCU + select TASKS_RCU + select TASKS_RUDE_RCU + select TASKS_TRACE_RCU + default n + help + This option provides a kernel module that runs performance tests + useful comparing RCU with various read-side synchronization mechanisms. + The kernel module may be built after the fact on the running kernel to be + tested, if desired. + + Say Y here if you want these performance tests built into the kernel. + Say M if you want to build it as a module instead. + Say N if you are unsure. + config RCU_CPU_STALL_TIMEOUT int "RCU CPU stall timeout in seconds" depends on RCU_STALL_COMMON
kernel/rcu/Makefile +1 -0
@@ -12,6 +12,7 @@ obj-$(CONFIG_TREE_SRCU) += srcutree.o obj-$(CONFIG_TINY_SRCU) += srcutiny.o obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o obj-$(CONFIG_RCU_PERF_TEST) += rcuperf.o +obj-$(CONFIG_RCU_REF_PERF_TEST) += refperf.o obj-$(CONFIG_TREE_RCU) += tree.o obj-$(CONFIG_TINY_RCU) += tiny.o obj-$(CONFIG_RCU_NEED_SEGCBLIST) += rcu_segcblist.o
kernel/rcu/refperf.c new file +558 -0
558 lines not on this page · raw message
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 3/30 rcuperf: Add comments explaining the high reader overhead
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit adds comments explaining why the readers have otherwise insane levels of measurement overhead, namely that they are intended as a test load for update-side performance measurements, not as a straight-up read-side performance test.
--- kernel/rcu/rcuperf.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/rcu/rcuperf.c +9 -2
@@ -69,6 +69,11 @@ MODULE_AUTHOR("Paul E. McKenney <[email protected]>");* value specified by nr_cpus for a read-only test. * * Various other use cases may of course be specified. +* +* Note that this test's readers are intended only as a test load for +* the writers. The reader performance statistics will be overly +* pessimistic due to the per-critical-section interrupt disabling, +* test-end checks, and the pair of calls through pointers. */ #ifdef MODULE
@@ -309,8 +314,10 @@ static void rcu_perf_wait_shutdown(void) }/* -* RCU perf reader kthread. Repeatedly does empty RCU read-side -* critical section, minimizing update-side interference. +* RCU perf reader kthread. Repeatedly does empty RCU read-side critical +* section, minimizing update-side interference. However, the point of +* this test is not to evaluate reader performance, but instead to serve +* as a test load for update-side performance testing. */ static int rcu_perf_reader (void *arg)
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 4/30 torture: Add refperf to the rcutorture scripting
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit updates the rcutorture scripting to include the new refperf torture-test module.
--- .../rcutorture/bin/kvm-recheck-refperf.sh | 67 ++++++++++++++++++++++ tools/testing/selftests/rcutorture/bin/kvm.sh | 9 +-- .../selftests/rcutorture/bin/parse-console.sh | 4 +- .../selftests/rcutorture/configs/refperf/CFLIST | 2 + .../selftests/rcutorture/configs/refperf/CFcommon | 2 + .../selftests/rcutorture/configs/refperf/NOPREEMPT | 18 ++++++ .../selftests/rcutorture/configs/refperf/PREEMPT | 18 ++++++ .../rcutorture/configs/refperf/ver_functions.sh | 16 ++++++ 8 files changed, 130 insertions(+), 6 deletions(-) create mode 100755 tools/testing/selftests/rcutorture/bin/kvm-recheck-refperf.sh create mode 100644 tools/testing/selftests/rcutorture/configs/refperf/CFLIST create mode 100644 tools/testing/selftests/rcutorture/configs/refperf/CFcommon create mode 100644 tools/testing/selftests/rcutorture/configs/refperf/NOPREEMPT create mode 100644 tools/testing/selftests/rcutorture/configs/refperf/PREEMPT create mode 100644 tools/testing/selftests/rcutorture/configs/refperf/ver_functions.sh
tools/testing/selftests/rcutorture/bin/kvm-recheck-refperf.sh new file +67 -0
@@ -0,0 +1,67 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0+ +# +# Analyze a given results directory for refperf performance measurements. +# +# Usage: kvm-recheck-refperf.sh resdir +# +# Copyright (C) IBM Corporation, 2016 +# +# Authors: Paul E. McKenney <[email protected]> + +i="$1" +if test -d "$i" -a -r "$i" +then + : +else + echo Unreadable results directory: $i + exit 1 +fi +PATH=`pwd`/tools/testing/selftests/rcutorture/bin:$PATH; export PATH +. functions.sh + +configfile=`echo $i | sed -e 's/^.*\///'` + +sed -e 's/^\[[^]]*]//' < $i/console.log | tr -d '\015' | +awk -v configfile="$configfile" ' +/^[ ]*Threads Time\(ns\) *$/ { + if (dataphase + 0 == 0) { + dataphase = 1; + # print configfile, $0; + } + next; +} + +/[^ ]*[0-9][0-9]* [0-9][0-9]*\.[0-9][0-9]*$/ { + if (dataphase == 1) { + # print $0; + readertimes[++n] = $2; + sum += $2; + } + next; +} + +{ + if (dataphase == 1) + dataphase == 2; + next; +} + +END { + print configfile " results:"; + newNR = asort(readertimes); + if (newNR <= 0) { + print "No refperf records found???" + exit; + } + medianidx = int(newNR / 2); + if (newNR == medianidx * 2) + medianvalue = (readertimes[medianidx - 1] + readertimes[medianidx]) / 2; + else + medianvalue = readertimes[medianidx]; + print "Average reader duration: " sum / newNR " nanoseconds"; + print "Minimum reader duration: " readertimes[1]; + print "Median reader duration: " medianvalue; + print "Maximum reader duration: " readertimes[newNR]; + print "Computed from refperf printk output."; +}'
tools/testing/selftests/rcutorture/bin/kvm.sh +5 -4
@@ -180,13 +180,14 @@ do shift ;; --torture) - checkarg --torture "(suite name)" "$#" "$2" '^\(lock\|rcu\|rcuperf\)$' '^--' + checkarg --torture "(suite name)" "$#" "$2" '^\(lock\|rcu\|rcuperf\|refperf\)$' '^--' TORTURE_SUITE=$2 shift - if test "$TORTURE_SUITE" = rcuperf + if test "$TORTURE_SUITE" = rcuperf || test "$TORTURE_SUITE" = refperf then - # If you really want jitter for rcuperf, specify - # it after specifying rcuperf. (But why?) + # If you really want jitter for refperf or + # rcuperf, specify it after specifying the rcuperf + # or the refperf. (But why jitter in these cases?) jitter=0 fi ;;
tools/testing/selftests/rcutorture/bin/parse-console.sh +2 -2
@@ -33,8 +33,8 @@ then fi cat /dev/null > $file.diags -# Check for proper termination, except that rcuperf runs don't indicate this. -if test "$TORTURE_SUITE" != rcuperf +# Check for proper termination, except for rcuperf and refperf. +if test "$TORTURE_SUITE" != rcuperf && test "$TORTURE_SUITE" != refperf then # check for abject failure
tools/testing/selftests/rcutorture/configs/refperf/CFLIST new file +2 -0
@@ -0,0 +1,2 @@ +NOPREEMPT +PREEMPT
tools/testing/selftests/rcutorture/configs/refperf/CFcommon new file +2 -0
@@ -0,0 +1,2 @@ +CONFIG_RCU_REF_PERF_TEST=y +CONFIG_PRINTK_TIME=y
tools/testing/selftests/rcutorture/configs/refperf/NOPREEMPT new file +18 -0
@@ -0,0 +1,18 @@ +CONFIG_SMP=y +CONFIG_PREEMPT_NONE=y +CONFIG_PREEMPT_VOLUNTARY=n +CONFIG_PREEMPT=n +#CHECK#CONFIG_PREEMPT_RCU=n +CONFIG_HZ_PERIODIC=n +CONFIG_NO_HZ_IDLE=y +CONFIG_NO_HZ_FULL=n +CONFIG_RCU_FAST_NO_HZ=n +CONFIG_HOTPLUG_CPU=n +CONFIG_SUSPEND=n +CONFIG_HIBERNATION=n +CONFIG_RCU_NOCB_CPU=n +CONFIG_DEBUG_LOCK_ALLOC=n +CONFIG_PROVE_LOCKING=n +CONFIG_RCU_BOOST=n +CONFIG_DEBUG_OBJECTS_RCU_HEAD=n +CONFIG_RCU_EXPERT=y
tools/testing/selftests/rcutorture/configs/refperf/PREEMPT new file +18 -0
@@ -0,0 +1,18 @@ +CONFIG_SMP=y +CONFIG_PREEMPT_NONE=n +CONFIG_PREEMPT_VOLUNTARY=n +CONFIG_PREEMPT=y +#CHECK#CONFIG_PREEMPT_RCU=y +CONFIG_HZ_PERIODIC=n +CONFIG_NO_HZ_IDLE=y +CONFIG_NO_HZ_FULL=n +CONFIG_RCU_FAST_NO_HZ=n +CONFIG_HOTPLUG_CPU=n +CONFIG_SUSPEND=n +CONFIG_HIBERNATION=n +CONFIG_RCU_NOCB_CPU=n +CONFIG_DEBUG_LOCK_ALLOC=n +CONFIG_PROVE_LOCKING=n +CONFIG_RCU_BOOST=n +CONFIG_DEBUG_OBJECTS_RCU_HEAD=n +CONFIG_RCU_EXPERT=y
tools/testing/selftests/rcutorture/configs/refperf/ver_functions.sh new file +16 -0
@@ -0,0 +1,16 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0+ +# +# Torture-suite-dependent shell functions for the rest of the scripts. +# +# Copyright (C) IBM Corporation, 2015 +# +# Authors: Paul E. McKenney <[email protected]> + +# per_version_boot_params bootparam-string config-file seconds +# +# Adds per-version torture-module parameters to kernels supporting them. +per_version_boot_params () { + echo $1 refperf.shutdown=1 \ + refperf.verbose=1 +}
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 5/30 refperf: Add holdoff parameter to allow CPUs to come online
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit adds an rcuperf module parameter named "holdoff" that defaults to 10 seconds if refperf is built in and to zero otherwise. The assumption is that all the CPUs are online by the time that the modprobe and insmod commands are going to do anything, and that normal systems will have all the CPUs online within ten seconds.
Larger systems may take many tens of seconds or even minutes to get to this point, hence this being a module parameter instead of being a hard-coded constant.
--- kernel/rcu/refperf.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
kernel/rcu/refperf.c +10 -3
@@ -57,7 +57,10 @@ MODULE_PARM_DESC(perf_type, "Type of test (rcu, srcu, refcnt, rwsem, rwlock.");torture_param (int, verbose,0 ,"Enable verbose debugging printk()s" ); -// Number of loops per experiment, all readers execute an operation concurrently +// Wait until there are multiple CPUs before starting test. +torture_param (int, holdoff,IS_BUILTIN (CONFIG_RCU_REF_PERF_TEST ) ?10 :0 , +"Holdoff time before test start (s)" ); +// Number of loops per experiment, all readers execute operations concurrently. torture_param (long, loops,10000000 ,"Number of loops per experiment." );#ifdef MODULE
@@ -248,6 +251,8 @@ ref_perf_reader(void *arg)set_cpus_allowed_ptr (current,cpumask_of (me % nr_cpu_ids));set_user_nice (current,MAX_NICE );atomic_inc (&n_init); +if (holdoff) +schedule_timeout_interruptible (holdoff *HZ ); repeat:VERBOSE_PERFOUT ("ref_perf_reader %ld: waiting to start next experiment on cpu %d" , me,smp_processor_id ());
@@ -357,6 +362,8 @@ static int main_func(void *arg)// Wait for all threads to start. wait_event (main_wq,atomic_read (&n_init) == (nreaders +1 )); +if (holdoff) +schedule_timeout_interruptible (holdoff *HZ );// Start exp readers up per experiment for (exp =0 ; exp < nreaders && !torture_must_stop (); exp++) {
@@ -420,8 +427,8 @@ static void ref_perf_print_module_parms(struct ref_perf_ops *cur_ops, const char *tag) { pr_alert("%s" PERF_FLAG -"--- %s: verbose=%d shutdown=%d loops=%ld\n" , perf_type, tag, - verbose, shutdown, loops); +"--- %s: verbose=%d shutdown=%d holdoff=%d loops=%ld\n" , perf_type, tag, + verbose, shutdown, holdoff, loops); } static void
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 6/30 refperf: Hoist function-pointer calls out of the loop
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Current runs show PREEMPT=n rcu_read_lock()/rcu_read_unlock() pairs consuming between 20 and 30 nanoseconds, when in fact the actual value is zero, give or take the barrier() asm's effect on compiler optimizations. The additional overhead is caused by function calls through pointers (especially in these days of Spectre mitigations) and perhaps also needless argument passing, a non-const loop limit, and an upcounting loop.
This commit therefore combines the ->readlock() and ->readunlock() function pointers into a single ->readsection() function pointer that takes the loop count as a const parameter and keeps any data passed from the read-lock to the read-unlock internal to this new function.
These changes reduce the measured overhead of the aforementioned PREEMPT=n rcu_read_lock()/rcu_read_unlock() pairs from between 20 and 30 nanoseconds to somewhere south of 500 picoseconds.
--- kernel/rcu/refperf.c | 92 ++++++++++++++++++++++------------------------------ 1 file changed, 38 insertions(+), 54 deletions(-)
kernel/rcu/refperf.c +38 -54
@@ -108,23 +108,20 @@ static int exp_idx;struct ref_perf_ops {void (*init)(void );void (*cleanup)(void ); -int (*readlock)(void ); -void (*readunlock)(int idx); +void (*readsection)(const int nloops);const char *name; };static struct ref_perf_ops *cur_ops; -// Definitions for RCU ref perf testing. -static int ref_rcu_read_lock (void )__acquires (RCU ) +static void ref_rcu_read_section (const int nloops) { -rcu_read_lock (); -return 0 ; -} +int i; -static void ref_rcu_read_unlock (int idx)__releases (RCU ) -{ -rcu_read_unlock (); +for (i = nloops; i >=0 ; i--) { +rcu_read_lock (); +rcu_read_unlock (); + } }static void rcu_sync_perf_init (void )
@@ -133,8 +130,7 @@ static void rcu_sync_perf_init(void)static struct ref_perf_ops rcu_ops = { .init = rcu_sync_perf_init, - .readlock = ref_rcu_read_lock, - .readunlock = ref_rcu_read_unlock, + .readsection = ref_rcu_read_section, .name ="rcu" };
@@ -143,42 +139,39 @@ static struct ref_perf_ops rcu_ops = {DEFINE_STATIC_SRCU (srcu_refctl_perf);static struct srcu_struct *srcu_ctlp = &srcu_refctl_perf; -static int srcu_ref_perf_read_lock (void )__acquires (srcu_ctlp) +static void srcu_ref_perf_read_section (int nloops) { -return srcu_read_lock (srcu_ctlp); -} +int i; +int idx; -static void srcu_ref_perf_read_unlock (int idx)__releases (srcu_ctlp) -{ -srcu_read_unlock (srcu_ctlp, idx); +for (i = nloops; i >=0 ; i--) { + idx =srcu_read_lock (srcu_ctlp); +srcu_read_unlock (srcu_ctlp, idx); + } }static struct ref_perf_ops srcu_ops = { .init = rcu_sync_perf_init, - .readlock = srcu_ref_perf_read_lock, - .readunlock = srcu_ref_perf_read_unlock, + .readsection = srcu_ref_perf_read_section, .name ="srcu" };// Definitions for reference count static atomic_t refcnt; -static int srcu_ref_perf_refcnt_lock (void ) +static void ref_perf_refcnt_section (const int nloops) { -atomic_inc (&refcnt); -return 0 ; -} +int i; -static void srcu_ref_perf_refcnt_unlock (int idx)__releases (srcu_ctlp) -{ -atomic_dec (&refcnt); -srcu_read_unlock (srcu_ctlp, idx); +for (i = nloops; i >=0 ; i--) { +atomic_inc (&refcnt); +atomic_dec (&refcnt); + } }static struct ref_perf_ops refcnt_ops = { .init = rcu_sync_perf_init, - .readlock = srcu_ref_perf_refcnt_lock, - .readunlock = srcu_ref_perf_refcnt_unlock, + .readsection = ref_perf_refcnt_section, .name ="refcnt" };
@@ -190,21 +183,19 @@ static void ref_perf_rwlock_init(void)rwlock_init (&test_rwlock); } -static int ref_perf_rwlock_lock (void ) +static void ref_perf_rwlock_section (const int nloops) { -read_lock (&test_rwlock); -return 0 ; -} +int i; -static void ref_perf_rwlock_unlock (int idx) -{ -read_unlock (&test_rwlock); +for (i = nloops; i >=0 ; i--) { +read_lock (&test_rwlock); +read_unlock (&test_rwlock); + } }static struct ref_perf_ops rwlock_ops = { .init = ref_perf_rwlock_init, - .readlock = ref_perf_rwlock_lock, - .readunlock = ref_perf_rwlock_unlock, + .readsection = ref_perf_rwlock_section, .name ="rwlock" };
@@ -216,21 +207,19 @@ static void ref_perf_rwsem_init(void)init_rwsem (&test_rwsem); } -static int ref_perf_rwsem_lock (void ) +static void ref_perf_rwsem_section (const int nloops) { -down_read (&test_rwsem); -return 0 ; -} +int i; -static void ref_perf_rwsem_unlock (int idx) -{ -up_read (&test_rwsem); +for (i = nloops; i >=0 ; i--) { +down_read (&test_rwsem); +up_read (&test_rwsem); + } }static struct ref_perf_ops rwsem_ops = { .init = ref_perf_rwsem_init, - .readlock = ref_perf_rwsem_lock, - .readunlock = ref_perf_rwsem_unlock, + .readsection = ref_perf_rwsem_section, .name ="rwsem" };
@@ -242,8 +231,6 @@ ref_perf_reader(void *arg)unsigned long flags;long me = (long )arg;struct reader_task *rt = &(reader_tasks[me]); -unsigned long spincnt; -int idx;u64 start;s64 duration;
@@ -275,10 +262,7 @@ ref_perf_reader(void *arg)VERBOSE_PERFOUT ("ref_perf_reader %ld: experiment %d started" , me, exp_idx); -for (spincnt =0 ; spincnt < loops; spincnt++) { - idx = cur_ops->readlock (); - cur_ops->readunlock (idx); - } + cur_ops->readsection (loops); duration =ktime_get_mono_fast_ns () - start;local_irq_restore (flags);
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 7/30 refperf: Allow decimal nanoseconds
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The CONFIG_PREEMPT=n rcu_read_lock()/rcu_read_unlock() pair's overhead, even including loop overhead, is far less than one nanosecond. Since logscale plots are not all that happy with zero values, provide picoseconds as decimals.
--- kernel/rcu/refperf.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/rcu/refperf.c +2 -2
@@ -375,7 +375,7 @@ static int main_func(void *arg)if (torture_must_stop ()) goto end; - reader_tasks[exp].result_avg =process_durations (exp) / ((exp +1 ) * loops); + reader_tasks[exp].result_avg =1000 *process_durations (exp) / ((exp +1 ) * loops); }// Print the average of all experiments
@@ -386,7 +386,7 @@ static int main_func(void *arg)strcat (buf,"Threads\tTime(ns)\n" );for (exp =0 ; exp < nreaders; exp++) { -sprintf (buf1,"%d\t%llu\n" , exp +1 , reader_tasks[exp].result_avg); +sprintf (buf1,"%d\t%llu.%03d\n" , exp +1 , reader_tasks[exp].result_avg /1000 , (int )(reader_tasks[exp].result_avg %1000 ));strcat (buf, buf1); }
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 8/30 refperf: Convert nreaders to a module parameter
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit converts nreaders to a module parameter, with the default of -1 specifying the old behavior of using 75% of the readers.
--- kernel/rcu/refperf.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
kernel/rcu/refperf.c +11 -5
@@ -62,6 +62,12 @@ torture_param(int, holdoff, IS_BUILTIN(CONFIG_RCU_REF_PERF_TEST) ? 10 : 0, "Holdoff time before test start (s)");// Number of loops per experiment, all readers execute operations concurrently. torture_param (long, loops,10000000 ,"Number of loops per experiment." ); +// Number of readers, with -1 defaulting to about 75% of the CPUs. +torture_param (int, nreaders,-1 ,"Number of readers, -1 for 75% of CPUs." ); +// Number of runs. +torture_param (int, nruns,30 ,"Number of experiments to run." ); +// Reader delay in nanoseconds, 0 for no delay. +torture_param (int, readdelay,0 ,"Read-side delay in nanoseconds." );#ifdef MODULE # define REFPERF_SHUTDOWN 0
@@ -93,7 +99,6 @@ static wait_queue_head_t main_wq;static int shutdown_start;static struct reader_task *reader_tasks; -static int nreaders;// Number of readers that are part of the current experiment. static atomic_t nreaders_exp;
@@ -411,8 +416,8 @@ static void ref_perf_print_module_parms(struct ref_perf_ops *cur_ops, const char *tag) { pr_alert("%s" PERF_FLAG -"--- %s: verbose=%d shutdown=%d holdoff=%d loops=%ld\n" , perf_type, tag, - verbose, shutdown, holdoff, loops); +"--- %s: verbose=%d shutdown=%d holdoff=%d loops=%ld nreaders=%d\n" , perf_type, tag, + verbose, shutdown, holdoff, loops, nreaders); } static void
@@ -501,8 +506,9 @@ ref_perf_init(void)schedule_timeout_uninterruptible (1 ); } -// Reader tasks (~75% of online CPUs). - nreaders = (num_online_cpus () >>1 ) + (num_online_cpus () >>2 ); +// Reader tasks (default to ~75% of online CPUs). +if (nreaders <0 ) + nreaders = (num_online_cpus () >>1 ) + (num_online_cpus () >>2 ); reader_tasks =kcalloc (nreaders,sizeof (reader_tasks[0 ]),GFP_KERNEL );if (!reader_tasks) {
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 9/30 refperf: Provide module parameter to specify number of experiments
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The current code uses the number of threads both to limit the number of threads and to specify the number of experiments, but also varies the number of threads as the experiments progress. This commit takes a different approach by adding an refperf.nruns module parameter that specifies the number of experiments, and furthermore uses the same number of threads for each experiment.
--- kernel/rcu/refperf.c | 43 +++++++++++++++++++++++-------------------- 1 file changed, 23 insertions(+), 20 deletions(-)
kernel/rcu/refperf.c +23 -20
@@ -83,12 +83,6 @@ struct reader_task {atomic_t start;wait_queue_head_t wq;u64 last_duration_ns; - -// The average latency When 1..<this reader> are concurrently -// running an experiment. For example, if this reader_task is -// of index 5 in the reader_tasks array, then result is for -// 6 cores. -u64 result_avg; };static struct task_struct *shutdown_task;
@@ -289,12 +283,12 @@ ref_perf_reader(void *arg)return 0 ; } -void reset_readers (int n) +void reset_readers (void ) {int i;struct reader_task *rt; -for (i =0 ; i < n; i++) { +for (i =0 ; i < nreaders; i++) { rt = &(reader_tasks[i]); rt->last_duration_ns =0 ;
@@ -314,7 +308,7 @@ u64 process_durations(int n)sprintf (buf,"Experiment #%d (Format: <THREAD-NUM>:<Total loop time in ns>)" , exp_idx); -for (i =0 ; i <= n && !torture_must_stop (); i++) { +for (i =0 ; i < n && !torture_must_stop (); i++) { rt = &(reader_tasks[i]);sprintf (buf1,"%d: %llu\t" , i, rt->last_duration_ns);
@@ -342,11 +336,15 @@ static int main_func(void *arg)int exp, r;char buf1[64 ];char buf[512 ]; +u64 *result_avg;set_cpus_allowed_ptr (current,cpumask_of (nreaders % nr_cpu_ids));set_user_nice (current,MAX_NICE );VERBOSE_PERFOUT ("main_func task started" ); + result_avg =kzalloc (nruns *sizeof (*result_avg),GFP_KERNEL ); +if (!result_avg) +VERBOSE_PERFOUT_ERRSTRING ("out of memory" );atomic_inc (&n_init);// Wait for all threads to start.
@@ -355,22 +353,24 @@ static int main_func(void *arg)schedule_timeout_interruptible (holdoff *HZ );// Start exp readers up per experiment -for (exp =0 ; exp < nreaders && !torture_must_stop (); exp++) { +for (exp =0 ; exp < nruns && !torture_must_stop (); exp++) { +if (!result_avg) +break ;if (torture_must_stop ()) goto end; -reset_readers (exp); -atomic_set (&nreaders_exp, exp +1 ); +reset_readers (); +atomic_set (&nreaders_exp, nreaders); exp_idx = exp; -for (r =0 ; r <= exp; r++) { +for (r =0 ; r < nreaders; r++) {atomic_set (&reader_tasks[r].start,1 );wake_up (&reader_tasks[r].wq); }VERBOSE_PERFOUT ("main_func: experiment started, waiting for %d readers" , - exp); + nreaders);wait_event (main_wq, !atomic_read (&nreaders_exp) ||torture_must_stop ());
@@ -380,7 +380,7 @@ static int main_func(void *arg)if (torture_must_stop ()) goto end; - reader_tasks[exp].result_avg =1000 *process_durations (exp) / ((exp +1 ) * loops); + result_avg[exp] =1000 *process_durations (nreaders) / (nreaders * loops); }// Print the average of all experiments
@@ -390,12 +390,15 @@ static int main_func(void *arg)strcat (buf,"\n" );strcat (buf,"Threads\tTime(ns)\n" ); -for (exp =0 ; exp < nreaders; exp++) { -sprintf (buf1,"%d\t%llu.%03d\n" , exp +1 , reader_tasks[exp].result_avg /1000 , (int )(reader_tasks[exp].result_avg %1000 )); +for (exp =0 ; exp < nruns; exp++) { +if (!result_avg) +break ; +sprintf (buf1,"%d\t%llu.%03d\n" , exp +1 , result_avg[exp] /1000 , (int )(result_avg[exp] %1000 ));strcat (buf, buf1); } -PERFOUT ("%s" , buf); +if (result_avg) +PERFOUT ("%s" , buf);// This will shutdown everything including us. if (shutdown) {
@@ -416,8 +419,8 @@ static void ref_perf_print_module_parms(struct ref_perf_ops *cur_ops, const char *tag) { pr_alert("%s" PERF_FLAG -"--- %s: verbose=%d shutdown=%d holdoff=%d loops=%ld nreaders=%d\n" , perf_type, tag, - verbose, shutdown, holdoff, loops, nreaders); +"--- %s: verbose=%d shutdown=%d holdoff=%d loops=%ld nreaders=%d nruns=%d\n" , perf_type, tag, + verbose, shutdown, holdoff, loops, nreaders, nruns); } static void
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 10/30 refperf: Dynamically allocate experiment-summary output buffer
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Currently, the buffer used to accumulate the experiment-summary output is fixed size, which will cause problems if someone decides to run one hundred experiments. This commit therefore dynamically allocates this buffer.
--- kernel/rcu/refperf.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
kernel/rcu/refperf.c +11 -5
@@ -333,9 +333,10 @@ u64 process_durations(int n)// point all the timestamps are printed. static int main_func (void *arg) { +bool errexit = false;int exp, r;char buf1[64 ]; -char buf[512 ]; +char *buf;u64 *result_avg;set_cpus_allowed_ptr (current,cpumask_of (nreaders % nr_cpu_ids));
@@ -343,8 +344,11 @@ static int main_func(void *arg)VERBOSE_PERFOUT ("main_func task started" ); result_avg =kzalloc (nruns *sizeof (*result_avg),GFP_KERNEL ); -if (!result_avg) + buf =kzalloc (64 + nruns *32 ,GFP_KERNEL ); +if (!result_avg || !buf) {VERBOSE_PERFOUT_ERRSTRING ("out of memory" ); + errexit = true; + }atomic_inc (&n_init);// Wait for all threads to start.
@@ -354,7 +358,7 @@ static int main_func(void *arg)// Start exp readers up per experiment for (exp =0 ; exp < nruns && !torture_must_stop (); exp++) { -if (!result_avg) +if (errexit)break ;if (torture_must_stop ()) goto end;
@@ -391,13 +395,13 @@ static int main_func(void *arg)strcat (buf,"Threads\tTime(ns)\n" );for (exp =0 ; exp < nruns; exp++) { -if (!result_avg) +if (errexit)break ;sprintf (buf1,"%d\t%llu.%03d\n" , exp +1 , result_avg[exp] /1000 , (int )(result_avg[exp] %1000 ));strcat (buf, buf1); } -if (result_avg) +if (!errexit)PERFOUT ("%s" , buf);// This will shutdown everything including us.
@@ -412,6 +416,8 @@ static int main_func(void *arg) end:torture_kthread_stopping ("main_func" ); +kfree (result_avg); +kfree (buf);return 0 ; }
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 11/30 refperf: Dynamically allocate thread-summary output buffer
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Currently, the buffer used to accumulate the thread-summary output is fixed size, which will cause problems if someone decides to run on a large number of PCUs. This commit therefore dynamically allocates this buffer.
--- kernel/rcu/refperf.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/rcu/refperf.c +5 -1
@@ -301,9 +301,12 @@ u64 process_durations(int n)int i;struct reader_task *rt;char buf1[64 ]; -char buf[512 ]; +char *buf;u64 sum =0 ; + buf =kmalloc (128 + nreaders *32 ,GFP_KERNEL ); +if (!buf) +return 0 ; buf[0 ] =0 ;sprintf (buf,"Experiment #%d (Format: <THREAD-NUM>:<Total loop time in ns>)" , exp_idx);
@@ -322,6 +325,7 @@ u64 process_durations(int n)PERFOUT ("%s\n" , buf); +kfree (buf);return sum; }
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 12/30 refperf: Make functions static
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Because the reset_readers() and process_durations() functions are used only within kernel/rcu/refperf.c, this commit makes them static.
--- kernel/rcu/refperf.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/rcu/refperf.c +2 -2
@@ -283,7 +283,7 @@ ref_perf_reader(void *arg)return 0 ; } -void reset_readers (void ) +static void reset_readers (void ) {int i;struct reader_task *rt;
@@ -296,7 +296,7 @@ void reset_readers(void) }// Print the results of each reader and return the sum of all their durations. -u64 process_durations (int n) +static u64 process_durations (int n) {int i;struct reader_task *rt;
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 13/30 refperf: Tune reader measurement interval
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit moves a printk() out of the measurement interval, converts a atomic_dec()/atomic_read() pair to atomic_dec_and_test(), and adds a smp_mb__before_atomic() to avoid potential wake/wait hangs. These changes have the added benefit of reducing the number of loops required for amortizing loop overhead for CONFIG_PREEMPT=n RCU measurements from 1,000,000 to 10,000. This reduction in turn shortens the test, reducing the probability of interference.
--- kernel/rcu/refperf.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
kernel/rcu/refperf.c +5 -6
@@ -252,15 +252,16 @@ ref_perf_reader(void *arg)// Make sure that the CPU is affinitized appropriately during testing. WARN_ON_ONCE (smp_processor_id () != me); +smp_mb__before_atomic ();atomic_dec (&rt->start); +VERBOSE_PERFOUT ("ref_perf_reader %ld: experiment %d started" , me, exp_idx); +// To prevent noise, keep interrupts disabled. This also has the // effect of preventing entries into slow path for rcu_read_unlock(). local_irq_save (flags); start =ktime_get_mono_fast_ns (); -VERBOSE_PERFOUT ("ref_perf_reader %ld: experiment %d started" , me, exp_idx); - cur_ops->readsection (loops); duration =ktime_get_mono_fast_ns () - start;
@@ -268,14 +269,12 @@ ref_perf_reader(void *arg) rt->last_duration_ns =WARN_ON_ONCE (duration <0 ) ?0 : duration; -atomic_dec (&nreaders_exp); +if (atomic_dec_and_test (&nreaders_exp)) +wake_up (&main_wq);VERBOSE_PERFOUT ("ref_perf_reader %ld: experiment %d ended, (readers remaining=%d)" , me, exp_idx,atomic_read (&nreaders_exp)); -if (!atomic_read (&nreaders_exp)) -wake_up (&main_wq); -if (!torture_must_stop ()) goto repeat; end:
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 14/30 refperf: Convert reader_task structure's "start" field to int
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit converts the reader_task structure's "start" field to int in order to demote a full barrier to an smp_load_acquire() and also to simplify the code a bit. While in the area, and to enlist the compiler's help in ensuring that nothing was missed, the field's name was changed to start_reader.
Also while in the area, change the main_func() store to use smp_store_release() to further fortify against wait/wake races.
--- kernel/rcu/refperf.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
kernel/rcu/refperf.c +4 -5
@@ -80,7 +80,7 @@ torture_param(bool, shutdown, REFPERF_SHUTDOWN,struct reader_task {struct task_struct *task; -atomic_t start; +int start_reader;wait_queue_head_t wq;u64 last_duration_ns; };
@@ -243,7 +243,7 @@ ref_perf_reader(void *arg)VERBOSE_PERFOUT ("ref_perf_reader %ld: waiting to start next experiment on cpu %d" , me,smp_processor_id ());// Wait for signal that this reader can start. -wait_event (rt->wq, (atomic_read (&nreaders_exp) &&atomic_read (&rt->start)) || +wait_event (rt->wq, (atomic_read (&nreaders_exp) &&smp_load_acquire (&rt->start_reader)) ||torture_must_stop ());if (torture_must_stop ())
@@ -252,8 +252,7 @@ ref_perf_reader(void *arg)// Make sure that the CPU is affinitized appropriately during testing. WARN_ON_ONCE (smp_processor_id () != me); -smp_mb__before_atomic (); -atomic_dec (&rt->start); +WRITE_ONCE (rt->start_reader,0 );VERBOSE_PERFOUT ("ref_perf_reader %ld: experiment %d started" , me, exp_idx);
@@ -372,7 +371,7 @@ static int main_func(void *arg) exp_idx = exp;for (r =0 ; r < nreaders; r++) { -atomic_set (&reader_tasks[r].start,1 ); +smp_store_release (&reader_tasks[r].start_reader,1 );wake_up (&reader_tasks[r].wq); }
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 15/30 refperf: More closely synchronize reader start times
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Currently, readers are awakened individually. On most systems, this results in significant wakeup delay from one reader to the next, which can result in the first and last reader having sole access to the synchronization primitive in question. If that synchronization primitive involves shared memory, those readers will rack up a huge number of operations in a very short time, causing large perturbations in the results.
This commit therefore has the readers busy-wait after being awakened, and uses a new n_started variable to synchronize their start times.
--- kernel/rcu/refperf.c | 5 +++++ 1 file changed, 5 insertions(+)
kernel/rcu/refperf.c +5 -0
@@ -99,6 +99,7 @@ static atomic_t nreaders_exp;// Use to wait for all threads to start. static atomic_t n_init; +static atomic_t n_started;// Track which experiment is currently running. static int exp_idx;
@@ -253,6 +254,9 @@ ref_perf_reader(void *arg)WARN_ON_ONCE (smp_processor_id () != me);WRITE_ONCE (rt->start_reader,0 ); +if (!atomic_dec_return (&n_started)) +while (atomic_read_acquire (&n_started)) +cpu_relax ();VERBOSE_PERFOUT ("ref_perf_reader %ld: experiment %d started" , me, exp_idx);
@@ -367,6 +371,7 @@ static int main_func(void *arg)reset_readers ();atomic_set (&nreaders_exp, nreaders); +atomic_set (&n_started, nreaders); exp_idx = exp;
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 16/30 refperf: Add warmup and cooldown processing phases
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit causes all the readers to start running unmeasured load until all readers have done at least one such run (thus having warmed up), then run the measured load, and then run unmeasured load until all readers have completed their measured load. This approach avoids any thread running measured load while other readers are idle.
--- kernel/rcu/refperf.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
kernel/rcu/refperf.c +18 -2
@@ -100,6 +100,8 @@ static atomic_t nreaders_exp;// Use to wait for all threads to start. static atomic_t n_init;static atomic_t n_started; +static atomic_t n_warmedup; +static atomic_t n_cooleddown;// Track which experiment is currently running. static int exp_idx;
@@ -260,8 +262,15 @@ ref_perf_reader(void *arg)VERBOSE_PERFOUT ("ref_perf_reader %ld: experiment %d started" , me, exp_idx); -// To prevent noise, keep interrupts disabled. This also has the -// effect of preventing entries into slow path for rcu_read_unlock(). + +// To reduce noise, do an initial cache-warming invocation, check +// in, and then keep warming until everyone has checked in. + cur_ops->readsection (loops); +if (!atomic_dec_return (&n_warmedup)) +while (atomic_read_acquire (&n_warmedup)) + cur_ops->readsection (loops); +// Also keep interrupts disabled. This also has the effect +// of preventing entries into slow path for rcu_read_unlock(). local_irq_save (flags); start =ktime_get_mono_fast_ns ();
@@ -271,6 +280,11 @@ ref_perf_reader(void *arg)local_irq_restore (flags); rt->last_duration_ns =WARN_ON_ONCE (duration <0 ) ?0 : duration; +// To reduce runtime-skew noise, do maintain-load invocations until +// everyone is done. +if (!atomic_dec_return (&n_cooleddown)) +while (atomic_read_acquire (&n_cooleddown)) + cur_ops->readsection (loops);if (atomic_dec_and_test (&nreaders_exp))wake_up (&main_wq);
@@ -372,6 +386,8 @@ static int main_func(void *arg)reset_readers ();atomic_set (&nreaders_exp, nreaders);atomic_set (&n_started, nreaders); +atomic_set (&n_warmedup, nreaders); +atomic_set (&n_cooleddown, nreaders); exp_idx = exp;
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 17/30 refperf: Label experiment-number column "Runs"
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The experiment-number column is currently labeled "Threads", which is misleading at best. This commit therefore relabels it as "Runs", and adjusts the scripts accordingly.
--- kernel/rcu/refperf.c | 2 +- tools/testing/selftests/rcutorture/bin/kvm-recheck-refperf.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
kernel/rcu/refperf.c +1 -1
@@ -415,7 +415,7 @@ static int main_func(void *arg) buf[0 ] =0 ;strcat (buf,"\n" ); -strcat (buf,"Threads\tTime(ns)\n" ); +strcat (buf,"Runs\tTime(ns)\n" );for (exp =0 ; exp < nruns; exp++) {if (errexit)
tools/testing/selftests/rcutorture/bin/kvm-recheck-refperf.sh +1 -1
@@ -24,7 +24,7 @@ configfile=`echo $i | sed -e 's/^.*\///'` sed -e 's/^\[[^]]*]//' < $i/console.log | tr -d '\015' | awk -v configfile="$configfile" ' -/^[ ]*Threads Time\(ns\) *$/ { +/^[ ]*Runs Time\(ns\) *$/ { if (dataphase + 0 == 0) { dataphase = 1; # print configfile, $0;
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 18/30 refperf: Output per-experiment data points
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Currently, it is necessary to manually edit the console output to see anything more than statistics, and sometimes the statistics can indicate outliers that need more investigation. This commit therefore dumps out the per-experiment measurements, sorted in ascending order, just before dumping out the statistics.
--- tools/testing/selftests/rcutorture/bin/kvm-recheck-refperf.sh | 4 ++++ 1 file changed, 4 insertions(+)
tools/testing/selftests/rcutorture/bin/kvm-recheck-refperf.sh +4 -0
@@ -59,6 +59,10 @@ END { medianvalue = (readertimes[medianidx - 1] + readertimes[medianidx]) / 2; else medianvalue = readertimes[medianidx]; + points = "Points:"; + for (i = 1; i <= newNR; i++) + points = points " " readertimes[i]; + print points; print "Average reader duration: " sum / newNR " nanoseconds"; print "Minimum reader duration: " readertimes[1]; print "Median reader duration: " medianvalue;
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 19/30 refperf: Simplify initialization-time wakeup protocol
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit moves the reader-launch wait loop from ref_perf_init() to main_func(), removing one layer of wakeup and allowing slightly faster system boot.
--- kernel/rcu/refperf.c | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-)
kernel/rcu/refperf.c +5 -12
@@ -369,13 +369,14 @@ static int main_func(void *arg)VERBOSE_PERFOUT_ERRSTRING ("out of memory" ); errexit = true; } -atomic_inc (&n_init); - -// Wait for all threads to start. -wait_event (main_wq,atomic_read (&n_init) == (nreaders +1 ));if (holdoff)schedule_timeout_interruptible (holdoff *HZ ); +// Wait for all threads to start. +atomic_inc (&n_init); +while (atomic_read (&n_init) < nreaders +1 ) +schedule_timeout_uninterruptible (1 ); +// Start exp readers up per experiment for (exp =0 ; exp < nruns && !torture_must_stop (); exp++) {if (errexit)
@@ -565,14 +566,6 @@ ref_perf_init(void) firsterr =torture_create_kthread (main_func,NULL , main_task);if (firsterr) goto unwind; -schedule_timeout_uninterruptible (1 ); - - -// Wait until all threads start -while (atomic_read (&n_init) < nreaders +1 ) -schedule_timeout_uninterruptible (1 ); - -wake_up (&main_wq);torture_init_end ();return 0 ;
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 20/30 refperf: Add read-side delay module parameter
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit adds a refperf.readdelay module parameter that controls the duration of each critical section. This parameter allows gathering data showing how the performance differences between the various primitives vary with critical-section length.
--- kernel/rcu/refperf.c | 108 ++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 89 insertions(+), 19 deletions(-)
kernel/rcu/refperf.c +89 -19
@@ -66,8 +66,8 @@ torture_param(long, loops, 10000000, "Number of loops per experiment.");torture_param (int, nreaders,-1 ,"Number of readers, -1 for 75% of CPUs." );// Number of runs. torture_param (int, nruns,30 ,"Number of experiments to run." ); -// Reader delay in nanoseconds, 0 for no delay. -torture_param (int, readdelay,0 ,"Read-side delay in nanoseconds." ); +// Reader delay in microseconds, 0 for no delay. +torture_param (int, readdelay,0 ,"Read-side delay in microseconds." );#ifdef MODULE # define REFPERF_SHUTDOWN 0
@@ -111,6 +111,7 @@ struct ref_perf_ops {void (*init)(void );void (*cleanup)(void );void (*readsection)(const int nloops); +void (*delaysection)(const int nloops,const int ndelay);const char *name; };
@@ -126,6 +127,17 @@ static void ref_rcu_read_section(const int nloops) } } +static void ref_rcu_delay_section (const int nloops,const int ndelay) +{ +int i; + +for (i = nloops; i >=0 ; i--) { +rcu_read_lock (); +udelay (ndelay); +rcu_read_unlock (); + } +} +static void rcu_sync_perf_init (void ) { }
@@ -133,6 +145,7 @@ static void rcu_sync_perf_init(void)static struct ref_perf_ops rcu_ops = { .init = rcu_sync_perf_init, .readsection = ref_rcu_read_section, + .delaysection = ref_rcu_delay_section, .name ="rcu" };
@@ -141,7 +154,7 @@ static struct ref_perf_ops rcu_ops = {DEFINE_STATIC_SRCU (srcu_refctl_perf);static struct srcu_struct *srcu_ctlp = &srcu_refctl_perf; -static void srcu_ref_perf_read_section (int nloops) +static void srcu_ref_perf_read_section (const int nloops) {int i;int idx;
@@ -152,16 +165,29 @@ static void srcu_ref_perf_read_section(int nloops) } } +static void srcu_ref_perf_delay_section (const int nloops,const int ndelay) +{ +int i; +int idx; + +for (i = nloops; i >=0 ; i--) { + idx =srcu_read_lock (srcu_ctlp); +udelay (ndelay); +srcu_read_unlock (srcu_ctlp, idx); + } +} +static struct ref_perf_ops srcu_ops = { .init = rcu_sync_perf_init, .readsection = srcu_ref_perf_read_section, + .delaysection = srcu_ref_perf_delay_section, .name ="srcu" };// Definitions for reference count static atomic_t refcnt; -static void ref_perf_refcnt_section (const int nloops) +static void ref_refcnt_section (const int nloops) {int i;
@@ -171,45 +197,69 @@ static void ref_perf_refcnt_section(const int nloops) } } +static void ref_refcnt_delay_section (const int nloops,const int ndelay) +{ +int i; + +for (i = nloops; i >=0 ; i--) { +atomic_inc (&refcnt); +udelay (ndelay); +atomic_dec (&refcnt); + } +} +static struct ref_perf_ops refcnt_ops = { .init = rcu_sync_perf_init, - .readsection = ref_perf_refcnt_section, + .readsection = ref_refcnt_section, + .delaysection = ref_refcnt_delay_section, .name ="refcnt" };// Definitions for rwlock static rwlock_t test_rwlock; -static void ref_perf_rwlock_init (void ) +static void ref_rwlock_init (void ) {rwlock_init (&test_rwlock); } -static void ref_perf_rwlock_section (const int nloops) +static void ref_rwlock_section (const int nloops) +{ +int i; + +for (i = nloops; i >=0 ; i--) { +read_lock (&test_rwlock); +read_unlock (&test_rwlock); + } +} + +static void ref_rwlock_delay_section (const int nloops,const int ndelay) {int i;for (i = nloops; i >=0 ; i--) {read_lock (&test_rwlock); +udelay (ndelay);read_unlock (&test_rwlock); } }static struct ref_perf_ops rwlock_ops = { - .init = ref_perf_rwlock_init, - .readsection = ref_perf_rwlock_section, + .init = ref_rwlock_init, + .readsection = ref_rwlock_section, + .delaysection = ref_rwlock_delay_section, .name ="rwlock" };// Definitions for rwsem static struct rw_semaphore test_rwsem; -static void ref_perf_rwsem_init (void ) +static void ref_rwsem_init (void ) {init_rwsem (&test_rwsem); } -static void ref_perf_rwsem_section (const int nloops) +static void ref_rwsem_section (const int nloops) {int i;
@@ -219,12 +269,32 @@ static void ref_perf_rwsem_section(const int nloops) } } +static void ref_rwsem_delay_section (const int nloops,const int ndelay) +{ +int i; + +for (i = nloops; i >=0 ; i--) { +down_read (&test_rwsem); +udelay (ndelay); +up_read (&test_rwsem); + } +} +static struct ref_perf_ops rwsem_ops = { - .init = ref_perf_rwsem_init, - .readsection = ref_perf_rwsem_section, + .init = ref_rwsem_init, + .readsection = ref_rwsem_section, + .delaysection = ref_rwsem_delay_section, .name ="rwsem" }; +static void rcu_perf_one_reader (void ) +{ +if (readdelay <=0 ) + cur_ops->readsection (loops); +else + cur_ops->delaysection (loops, readdelay); +} +// Reader kthread. Repeatedly does empty RCU read-side // critical section, minimizing update-side interference. static int
@@ -265,16 +335,16 @@ ref_perf_reader(void *arg)// To reduce noise, do an initial cache-warming invocation, check // in, and then keep warming until everyone has checked in. - cur_ops->readsection (loops); +rcu_perf_one_reader ();if (!atomic_dec_return (&n_warmedup))while (atomic_read_acquire (&n_warmedup)) - cur_ops->readsection (loops); +rcu_perf_one_reader ();// Also keep interrupts disabled. This also has the effect // of preventing entries into slow path for rcu_read_unlock(). local_irq_save (flags); start =ktime_get_mono_fast_ns (); - cur_ops->readsection (loops); +rcu_perf_one_reader (); duration =ktime_get_mono_fast_ns () - start;local_irq_restore (flags);
@@ -284,7 +354,7 @@ ref_perf_reader(void *arg)// everyone is done. if (!atomic_dec_return (&n_cooleddown))while (atomic_read_acquire (&n_cooleddown)) - cur_ops->readsection (loops); +rcu_perf_one_reader ();if (atomic_dec_and_test (&nreaders_exp))wake_up (&main_wq);
@@ -449,8 +519,8 @@ static void ref_perf_print_module_parms(struct ref_perf_ops *cur_ops, const char *tag) { pr_alert("%s" PERF_FLAG -"--- %s: verbose=%d shutdown=%d holdoff=%d loops=%ld nreaders=%d nruns=%d\n" , perf_type, tag, - verbose, shutdown, holdoff, loops, nreaders, nruns); +"--- %s: verbose=%d shutdown=%d holdoff=%d loops=%ld nreaders=%d nruns=%d readdelay=%d\n" , perf_type, tag, + verbose, shutdown, holdoff, loops, nreaders, nruns, readdelay); } static void
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 21/30 refperf: Adjust refperf.loop default value
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
With the various measurement optimizations, 10,000 loops normally suffices. This commit therefore reduces the refperf.loops default value from 10,000,000 to 10,000.
--- kernel/rcu/refperf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/refperf.c +1 -1
@@ -61,7 +61,7 @@ torture_param(int, verbose, 0, "Enable verbose debugging printk()s");torture_param (int, holdoff,IS_BUILTIN (CONFIG_RCU_REF_PERF_TEST ) ?10 :0 ,"Holdoff time before test start (s)" );// Number of loops per experiment, all readers execute operations concurrently. -torture_param (long, loops,10000000 ,"Number of loops per experiment." ); +torture_param (long, loops,10000 ,"Number of loops per experiment." );// Number of readers, with -1 defaulting to about 75% of the CPUs. torture_param (int, nreaders,-1 ,"Number of readers, -1 for 75% of CPUs." );// Number of runs.
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 22/30 doc: Document rcuperf's module parameters
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit adds documentation for the rcuperf module parameters.
--- Documentation/admin-guide/kernel-parameters.txt | 36 +++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +36 -0
@@ -4407,6 +4407,42 @@ reboot_cpu is s[mp]#### with #### being the processor to be used for rebooting. + refperf.holdoff= [KNL] + Set test-start holdoff period. The purpose of + this parameter is to delay the start of the + test until boot completes in order to avoid + interference. + + refperf.loops= [KNL] + Set the number of loops over the synchronization + primitive under test. Increasing this number + reduces noise due to loop start/end overhead, + but the default has already reduced the per-pass + noise to a handful of picoseconds on ca. 2020 + x86 laptops. + + refperf.nreaders= [KNL] + Set number of readers. The default value of -1 + selects N, where N is roughly 75% of the number + of CPUs. A value of zero is an interesting choice. + + refperf.nruns= [KNL] + Set number of runs, each of which is dumped onto + the console log. + + refperf.readdelay= [KNL] + Set the read-side critical-section duration, + measured in microseconds. + + refperf.shutdown= [KNL] + Shut down the system at the end of the performance + test. This defaults to 1 (shut it down) when + rcuperf is built into the kernel and to 0 (leave + it running) when rcuperf is built as a module. + + refperf.verbose= [KNL] + Enable additional printk() statements. + relax_domain_level= [KNL, SMP] Set scheduler's default relax_domain_level. See Documentation/admin-guide/cgroup-v1/cpusets.rst.
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 23/30 refperf: Work around 64-bit division
In reply to Paul E. McKenneyFrom: Arnd Bergmann <[email protected]>
A 64-bit division was introduced in refperf, breaking compilation on all 32-bit architectures:
kernel/rcu/refperf.o: in function `main_func':
refperf.c:(.text+0x57c): undefined reference to `__aeabi_uldivmod'
Fix this by using div_u64 to mark the expensive operation.
--- kernel/rcu/refperf.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
kernel/rcu/refperf.c +6 -2
@@ -478,7 +478,7 @@ static int main_func(void *arg)if (torture_must_stop ()) goto end; - result_avg[exp] =1000 *process_durations (nreaders) / (nreaders * loops); + result_avg[exp] =div_u64 (1000 *process_durations (nreaders), nreaders * loops); }// Print the average of all experiments
@@ -489,9 +489,13 @@ static int main_func(void *arg)strcat (buf,"Runs\tTime(ns)\n" );for (exp =0 ; exp < nruns; exp++) { +u64 avg; +u32 rem; +if (errexit)break ; -sprintf (buf1,"%d\t%llu.%03d\n" , exp +1 , result_avg[exp] /1000 , (int )(result_avg[exp] %1000 )); + avg =div_u64_rem (result_avg[exp],1000 , &rem); +sprintf (buf1,"%d\t%llu.%03u\n" , exp +1 , avg, rem);strcat (buf, buf1); }
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 24/30 refperf: Change readdelay module parameter to nanoseconds
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The current units of microseconds are too coarse, so this commit changes the units to nanoseconds. However, ndelay is used only for the nanoseconds with udelay being used for whole microseconds. For example, setting refperf.readdelay=1500 results in a udelay(1) followed by an ndelay(500).
--- kernel/rcu/refperf.c | 36 ++++++++++++++++++++++-------------- 1 file changed, 22 insertions(+), 14 deletions(-)
kernel/rcu/refperf.c +22 -14
@@ -66,8 +66,8 @@ torture_param(long, loops, 10000, "Number of loops per experiment.");torture_param (int, nreaders,-1 ,"Number of readers, -1 for 75% of CPUs." );// Number of runs. torture_param (int, nruns,30 ,"Number of experiments to run." ); -// Reader delay in microseconds, 0 for no delay. -torture_param (int, readdelay,0 ,"Read-side delay in microseconds." ); +// Reader delay in nanoseconds, 0 for no delay. +torture_param (int, readdelay,0 ,"Read-side delay in nanoseconds." );#ifdef MODULE # define REFPERF_SHUTDOWN 0
@@ -111,12 +111,20 @@ struct ref_perf_ops {void (*init)(void );void (*cleanup)(void );void (*readsection)(const int nloops); -void (*delaysection)(const int nloops,const int ndelay); +void (*delaysection)(const int nloops,const int udl,const int ndl);const char *name; };static struct ref_perf_ops *cur_ops; +static void un_delay (const int udl,const int ndl) +{ +if (udl) +udelay (udl); +if (ndl) +ndelay (ndl); +} +static void ref_rcu_read_section (const int nloops) {int i;
@@ -127,13 +135,13 @@ static void ref_rcu_read_section(const int nloops) } } -static void ref_rcu_delay_section (const int nloops,const int ndelay) +static void ref_rcu_delay_section (const int nloops,const int udl,const int ndl) {int i;for (i = nloops; i >=0 ; i--) {rcu_read_lock (); -udelay (ndelay); +un_delay (udl, ndl);rcu_read_unlock (); } }
@@ -165,14 +173,14 @@ static void srcu_ref_perf_read_section(const int nloops) } } -static void srcu_ref_perf_delay_section (const int nloops,const int ndelay) +static void srcu_ref_perf_delay_section (const int nloops,const int udl,const int ndl) {int i;int idx;for (i = nloops; i >=0 ; i--) { idx =srcu_read_lock (srcu_ctlp); -udelay (ndelay); +un_delay (udl, ndl);srcu_read_unlock (srcu_ctlp, idx); } }
@@ -197,13 +205,13 @@ static void ref_refcnt_section(const int nloops) } } -static void ref_refcnt_delay_section (const int nloops,const int ndelay) +static void ref_refcnt_delay_section (const int nloops,const int udl,const int ndl) {int i;for (i = nloops; i >=0 ; i--) {atomic_inc (&refcnt); -udelay (ndelay); +un_delay (udl, ndl);atomic_dec (&refcnt); } }
@@ -233,13 +241,13 @@ static void ref_rwlock_section(const int nloops) } } -static void ref_rwlock_delay_section (const int nloops,const int ndelay) +static void ref_rwlock_delay_section (const int nloops,const int udl,const int ndl) {int i;for (i = nloops; i >=0 ; i--) {read_lock (&test_rwlock); -udelay (ndelay); +un_delay (udl, ndl);read_unlock (&test_rwlock); } }
@@ -269,13 +277,13 @@ static void ref_rwsem_section(const int nloops) } } -static void ref_rwsem_delay_section (const int nloops,const int ndelay) +static void ref_rwsem_delay_section (const int nloops,const int udl,const int ndl) {int i;for (i = nloops; i >=0 ; i--) {down_read (&test_rwsem); -udelay (ndelay); +un_delay (udl, ndl);up_read (&test_rwsem); } }
@@ -292,7 +300,7 @@ static void rcu_perf_one_reader(void)if (readdelay <=0 ) cur_ops->readsection (loops);else - cur_ops->delaysection (loops, readdelay); + cur_ops->delaysection (loops, readdelay /1000 , readdelay %1000 ); }// Reader kthread. Repeatedly does empty RCU read-side
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 25/30 refperf: Add test for RCU Tasks Trace readers.
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit adds testing for RCU Tasks Trace readers to the refperf module.
--- kernel/rcu/refperf.c | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-)
kernel/rcu/refperf.c +31 -2
@@ -25,6 +25,7 @@#include <linux/notifier.h> #include <linux/percpu.h> #include <linux/rcupdate.h> +#include <linux/rcupdate_trace.h> #include <linux/reboot.h> #include <linux/sched.h> #include <linux/spinlock.h>
@@ -157,7 +158,6 @@ static struct ref_perf_ops rcu_ops = { .name = "rcu" }; -// Definitions for SRCU ref perf testing. DEFINE_STATIC_SRCU (srcu_refctl_perf);static struct srcu_struct *srcu_ctlp = &srcu_refctl_perf;
@@ -192,6 +192,35 @@ static struct ref_perf_ops srcu_ops = { .name = "srcu" }; +// Definitions for RCU Tasks Trace ref perf testing. +static void rcu_trace_ref_perf_read_section (const int nloops) +{ +int i; + +for (i = nloops; i >=0 ; i--) { +rcu_read_lock_trace (); +rcu_read_unlock_trace (); + } +} + +static void rcu_trace_ref_perf_delay_section (const int nloops,const int udl,const int ndl) +{ +int i; + +for (i = nloops; i >=0 ; i--) { +rcu_read_lock_trace (); +un_delay (udl, ndl); +rcu_read_unlock_trace (); + } +} + +static struct ref_perf_ops rcu_trace_ops = { + .init = rcu_sync_perf_init, + .readsection = rcu_trace_ref_perf_read_section, + .delaysection = rcu_trace_ref_perf_delay_section, + .name ="rcu-trace" +}; +// Definitions for reference count static atomic_t refcnt;
@@ -584,7 +613,7 @@ ref_perf_init(void)long i;int firsterr =0 ;static struct ref_perf_ops *perf_ops[] = { - &rcu_ops, &srcu_ops, &refcnt_ops, &rwlock_ops, &rwsem_ops, + &rcu_ops, &srcu_ops, &rcu_trace_ops, &refcnt_ops, &rwlock_ops, &rwsem_ops, };if (!torture_init_begin (perf_type, verbose))
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 26/30 refperf: Add test for RCU Tasks readers
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit adds testing for RCU Tasks readers to the refperf module. This also applies to RCU Rude readers, as both flavors have empty (as in non-existent) read-side markers.
--- kernel/rcu/refperf.c | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-)
kernel/rcu/refperf.c +27 -1
@@ -192,6 +192,31 @@ static struct ref_perf_ops srcu_ops = { .name = "srcu" }; +// Definitions for RCU Tasks ref perf testing: Empty read markers. +// These definitions also work for RCU Rude readers. +static void rcu_tasks_ref_perf_read_section (const int nloops) +{ +int i; + +for (i = nloops; i >=0 ; i--) +continue ; +} + +static void rcu_tasks_ref_perf_delay_section (const int nloops,const int udl,const int ndl) +{ +int i; + +for (i = nloops; i >=0 ; i--) +un_delay (udl, ndl); +} + +static struct ref_perf_ops rcu_tasks_ops = { + .init = rcu_sync_perf_init, + .readsection = rcu_tasks_ref_perf_read_section, + .delaysection = rcu_tasks_ref_perf_delay_section, + .name ="rcu-tasks" +}; +// Definitions for RCU Tasks Trace ref perf testing. static void rcu_trace_ref_perf_read_section (const int nloops) {
@@ -613,7 +638,8 @@ ref_perf_init(void)long i;int firsterr =0 ;static struct ref_perf_ops *perf_ops[] = { - &rcu_ops, &srcu_ops, &rcu_trace_ops, &refcnt_ops, &rwlock_ops, &rwsem_ops, + &rcu_ops, &srcu_ops, &rcu_trace_ops, &rcu_tasks_ops, + &refcnt_ops, &rwlock_ops, &rwsem_ops, };if (!torture_init_begin (perf_type, verbose))
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 27/30 rcu-tasks: Fix synchronize_rcu_tasks_trace() header comment
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The synchronize_rcu_tasks_trace() header comment incorrectly claims that any number of things delimit RCU Tasks Trace read-side critical sections, when in fact only rcu_read_lock_trace() and rcu_read_unlock_trace() do so. This commit therefore fixes this comment, and, while in the area, fixes a typo in the rcu_read_lock_trace() header comment.
--- include/linux/rcupdate_trace.h | 4 ++-- kernel/rcu/tasks.h | 9 ++++----- 2 files changed, 6 insertions(+), 7 deletions(-)
include/linux/rcupdate_trace.h +2 -2
@@ -36,8 +36,8 @@ void rcu_read_unlock_trace_special(struct task_struct *t, int nesting);/** * rcu_read_lock_trace - mark beginning of RCU-trace read-side critical section * -* When synchronize_rcu_trace() is invoked by one task, then that task -* is guaranteed to block until all other tasks exit their read-side +* When synchronize_rcu_tasks_trace() is invoked by one task, then that +* task is guaranteed to block until all other tasks exit their read-side * critical sections. Similarly, if call_rcu_trace() is invoked on one * task while other tasks are within RCU read-side critical sections, * invocation of the corresponding RCU callback is deferred until after
kernel/rcu/tasks.h +4 -5
@@ -1118,11 +1118,10 @@ EXPORT_SYMBOL_GPL(call_rcu_tasks_trace);* synchronize_rcu_tasks_trace - wait for a trace rcu-tasks grace period * * Control will return to the caller some time after a trace rcu-tasks -* grace period has elapsed, in other words after all currently -* executing rcu-tasks read-side critical sections have elapsed. These -* read-side critical sections are delimited by calls to schedule(), -* cond_resched_tasks_rcu_qs(), userspace execution, and (in theory, -* anyway) cond_resched(). +* grace period has elapsed, in other words after all currently executing +* rcu-tasks read-side critical sections have elapsed. These read-side +* critical sections are delimited by calls to rcu_read_lock_trace() +* and rcu_read_unlock_trace(). * * This is a very specialized primitive, intended only for a few uses in * tracing and other situations requiring manipulation of function preambles
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 28/30 refperf: Rename RCU_REF_PERF_TEST to RCU_REF_SCALE_TEST
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
The old Kconfig option name is all too easy to conflate with the unrelated "perf" feature, so this commit renames RCU_REF_PERF_TEST to RCU_REF_SCALE_TEST.
--- kernel/rcu/Kconfig.debug | 4 ++-- kernel/rcu/Makefile | 2 +- kernel/rcu/refperf.c | 6 +++--- tools/testing/selftests/rcutorture/configs/refperf/CFcommon | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-)
kernel/rcu/Kconfig.debug +2 -2
@@ -61,8 +61,8 @@ config RCU_TORTURE_TEST Say M if you want the RCU torture tests to build as a module. Say N if you are unsure. -config RCU_REF_PERF_TEST - tristate "Performance tests for read-side synchronization (RCU and others)" +config RCU_REF_SCALE_TEST + tristate "Scalability tests for read-side synchronization (RCU and others)" depends on DEBUG_KERNEL select TORTURE_TEST select SRCU
kernel/rcu/Makefile +1 -1
@@ -12,7 +12,7 @@ obj-$(CONFIG_TREE_SRCU) += srcutree.o obj-$(CONFIG_TINY_SRCU) += srcutiny.o obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o obj-$(CONFIG_RCU_PERF_TEST) += rcuperf.o -obj-$(CONFIG_RCU_REF_PERF_TEST) += refperf.o +obj-$(CONFIG_RCU_REF_SCALE_TEST) += refperf.o obj-$(CONFIG_TREE_RCU) += tree.o obj-$(CONFIG_TINY_RCU) += tiny.o obj-$(CONFIG_RCU_NEED_SEGCBLIST) += rcu_segcblist.o
kernel/rcu/refperf.c +3 -3
@@ -1,6 +1,6 @@// SPDX-License-Identifier: GPL-2.0+ // -// Performance test comparing RCU vs other mechanisms +// Scalability test comparing RCU vs other mechanisms // for acquiring references on objects. // // Copyright (C) Google, 2020.
@@ -59,7 +59,7 @@ MODULE_PARM_DESC(perf_type, "Type of test (rcu, srcu, refcnt, rwsem, rwlock.");torture_param (int, verbose,0 ,"Enable verbose debugging printk()s" );// Wait until there are multiple CPUs before starting test. -torture_param (int, holdoff,IS_BUILTIN (CONFIG_RCU_REF_PERF_TEST ) ?10 :0 , +torture_param (int, holdoff,IS_BUILTIN (CONFIG_RCU_REF_SCALE_TEST ) ?10 :0 ,"Holdoff time before test start (s)" );// Number of loops per experiment, all readers execute operations concurrently. torture_param (long, loops,10000 ,"Number of loops per experiment." );
@@ -656,7 +656,7 @@ ref_perf_init(void)for (i =0 ; i <ARRAY_SIZE (perf_ops); i++)pr_cont (" %s" , perf_ops[i]->name);pr_cont ("\n" ); -WARN_ON (!IS_MODULE (CONFIG_RCU_REF_PERF_TEST )); +WARN_ON (!IS_MODULE (CONFIG_RCU_REF_SCALE_TEST )); firsterr = -EINVAL ; cur_ops =NULL ; goto unwind;
tools/testing/selftests/rcutorture/configs/refperf/CFcommon +1 -1
@@ -1,2 +1,2 @@ -CONFIG_RCU_REF_PERF_TEST=y +CONFIG_RCU_REF_SCALE_TEST=y CONFIG_PRINTK_TIME=y
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 29/30 refperf: Rename refperf.c to refscale.c and change internal names
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit further avoids conflation of refperf with the kernel's perf feature by renaming kernel/rcu/refperf.c to kernel/rcu/refscale.c, and also by similarly renaming the functions and variables inside this file. This has the side effect of changing the names of the kernel boot parameters, so kernel-parameters.txt and ver_functions.sh are also updated.
The rcutorture --torture type remains refperf, and this will be addressed in a separate commit.
---
Documentation/admin-guide/kernel-parameters.txt | 17 +-
kernel/rcu/Makefile | 2 +-
kernel/rcu/{refperf.c => refscale.c} | 182 ++++++++++-----------
.../rcutorture/configs/refperf/ver_functions.sh | 4 +-
4 files changed, 104 insertions(+), 101 deletions(-)
rename kernel/rcu/{refperf.c => refscale.c} (74%)Documentation/admin-guide/kernel-parameters.txt +10 -7
@@ -4407,13 +4407,13 @@ reboot_cpu is s[mp]#### with #### being the processor to be used for rebooting. - refperf.holdoff= [KNL] + refscale.holdoff= [KNL] Set test-start holdoff period. The purpose of this parameter is to delay the start of the test until boot completes in order to avoid interference. - refperf.loops= [KNL] + refscale.loops= [KNL] Set the number of loops over the synchronization primitive under test. Increasing this number reduces noise due to loop start/end overhead,
@@ -4421,26 +4421,29 @@ noise to a handful of picoseconds on ca. 2020 x86 laptops. - refperf.nreaders= [KNL] + refscale.nreaders= [KNL] Set number of readers. The default value of -1 selects N, where N is roughly 75% of the number of CPUs. A value of zero is an interesting choice. - refperf.nruns= [KNL] + refscale.nruns= [KNL] Set number of runs, each of which is dumped onto the console log. - refperf.readdelay= [KNL] + refscale.readdelay= [KNL] Set the read-side critical-section duration, measured in microseconds. - refperf.shutdown= [KNL] + refscale.scale_type= [KNL] + Specify the read-protection implementation to test. + + refscale.shutdown= [KNL] Shut down the system at the end of the performance test. This defaults to 1 (shut it down) when rcuperf is built into the kernel and to 0 (leave it running) when rcuperf is built as a module. - refperf.verbose= [KNL] + refscale.verbose= [KNL] Enable additional printk() statements. relax_domain_level=
kernel/rcu/Makefile +1 -1
@@ -12,7 +12,7 @@ obj-$(CONFIG_TREE_SRCU) += srcutree.o obj-$(CONFIG_TINY_SRCU) += srcutiny.o obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o obj-$(CONFIG_RCU_PERF_TEST) += rcuperf.o -obj-$(CONFIG_RCU_REF_SCALE_TEST) += refperf.o +obj-$(CONFIG_RCU_REF_SCALE_TEST) += refscale.o obj-$(CONFIG_TREE_RCU) += tree.o obj-$(CONFIG_TINY_RCU) += tiny.o obj-$(CONFIG_RCU_NEED_SEGCBLIST) += rcu_segcblist.o
kernel/rcu/refscale.c renamed from kernel/rcu/refperf.c +91 -91
464 lines not on this page · raw message
tools/testing/selftests/rcutorture/configs/refperf/ver_functions.sh +2 -2
@@ -11,6 +11,6 @@ # # Adds per-version torture-module parameters to kernels supporting them. per_version_boot_params () { - echo $1 refperf.shutdown=1 \ - refperf.verbose=1 + echo $1 refscale.shutdown=1 \ + refscale.verbose=1 }
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 30/30 refscale: Change --torture type from refperf to refscale
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit renames the rcutorture config/refperf to config/refscale to further avoid conflation with the Linux kernel's perf feature.
---
.../bin/{kvm-recheck-refperf.sh => kvm-recheck-refscale.sh} | 8 ++++----
tools/testing/selftests/rcutorture/bin/kvm.sh | 8 ++++----
tools/testing/selftests/rcutorture/bin/parse-console.sh | 4 ++--
.../selftests/rcutorture/configs/{refperf => refscale}/CFLIST | 0
.../selftests/rcutorture/configs/{refperf => refscale}/CFcommon | 0
.../selftests/rcutorture/configs/{refperf => refscale}/NOPREEMPT | 0
.../selftests/rcutorture/configs/{refperf => refscale}/PREEMPT | 0
.../rcutorture/configs/{refperf => refscale}/ver_functions.sh | 0
8 files changed, 10 insertions(+), 10 deletions(-)
rename tools/testing/selftests/rcutorture/bin/{kvm-recheck-refperf.sh => kvm-recheck-refscale.sh} (87%)
rename tools/testing/selftests/rcutorture/configs/{refperf => refscale}/CFLIST (100%)
rename tools/testing/selftests/rcutorture/configs/{refperf => refscale}/CFcommon (100%)
rename tools/testing/selftests/rcutorture/configs/{refperf => refscale}/NOPREEMPT (100%)
rename tools/testing/selftests/rcutorture/configs/{refperf => refscale}/PREEMPT (100%)
rename tools/testing/selftests/rcutorture/configs/{refperf => refscale}/ver_functions.sh (100%)tools/testing/selftests/rcutorture/bin/kvm-recheck-refscale.sh renamed from tools/testing/selftests/rcutorture/bin/kvm-recheck-refperf.sh +4 -4
@@ -1,9 +1,9 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0+ # -# Analyze a given results directory for refperf performance measurements. +# Analyze a given results directory for refscale performance measurements. # -# Usage: kvm-recheck-refperf.sh resdir +# Usage: kvm-recheck-refscale.sh resdir # # Copyright (C) IBM Corporation, 2016 #
@@ -51,7 +51,7 @@ END { print configfile " results:"; newNR = asort(readertimes); if (newNR <= 0) { - print "No refperf records found???" + print "No refscale records found???" exit; } medianidx = int(newNR / 2);
@@ -67,5 +67,5 @@ END { print "Minimum reader duration: " readertimes[1]; print "Median reader duration: " medianvalue; print "Maximum reader duration: " readertimes[newNR]; - print "Computed from refperf printk output."; + print "Computed from refscale printk output."; }'
tools/testing/selftests/rcutorture/bin/kvm.sh +4 -4
@@ -180,14 +180,14 @@ do shift ;; --torture) - checkarg --torture "(suite name)" "$#" "$2" '^\(lock\|rcu\|rcuperf\|refperf\)$' '^--' + checkarg --torture "(suite name)" "$#" "$2" '^\(lock\|rcu\|rcuperf\|refscale\)$' '^--' TORTURE_SUITE=$2 shift - if test "$TORTURE_SUITE" = rcuperf || test "$TORTURE_SUITE" = refperf + if test "$TORTURE_SUITE" = rcuperf || test "$TORTURE_SUITE" = refscale then - # If you really want jitter for refperf or + # If you really want jitter for refscale or # rcuperf, specify it after specifying the rcuperf - # or the refperf. (But why jitter in these cases?) + # or the refscale. (But why jitter in these cases?) jitter=0 fi ;;
tools/testing/selftests/rcutorture/bin/parse-console.sh +2 -2
@@ -33,8 +33,8 @@ then fi cat /dev/null > $file.diags -# Check for proper termination, except for rcuperf and refperf. -if test "$TORTURE_SUITE" != rcuperf && test "$TORTURE_SUITE" != refperf +# Check for proper termination, except for rcuperf and refscale. +if test "$TORTURE_SUITE" != rcuperf && test "$TORTURE_SUITE" != refscale then # check for abject failure
tools/testing/selftests/rcutorture/configs/refscale/CFLIST renamed from tools/testing/selftests/rcutorture/configs/refperf/CFLIST +0 -0
tools/testing/selftests/rcutorture/configs/refscale/CFcommon renamed from tools/testing/selftests/rcutorture/configs/refperf/CFcommon +0 -0
tools/testing/selftests/rcutorture/configs/refscale/NOPREEMPT renamed from tools/testing/selftests/rcutorture/configs/refperf/NOPREEMPT +0 -0
tools/testing/selftests/rcutorture/configs/refscale/PREEMPT renamed from tools/testing/selftests/rcutorture/configs/refperf/PREEMPT +0 -0
tools/testing/selftests/rcutorture/configs/refscale/ver_functions.sh renamed from tools/testing/selftests/rcutorture/configs/refperf/ver_functions.sh +0 -0
Signature · 1 line
2.9.5