Skip to contentSkip to conversation

defer: misc updates

PATCH In review Akira Yokosawa 2 participants 15 messages May 31, 2020 – Jun 3, 2020 RCU

lore
Share & tools
lore
#0 lore raw

cover defer: misc updates

Hi Paul,

This is misc updates in response to your recent updates.

Patch 1/3 treats QQZ annotations for "nq" build.
Patch 2/3 adds a paragraph in #9 of FAQ.txt. The wording may need your retouch for fluency.
Patch 3/3 is an independent improvement of runlatex.sh. It will avoid a few redundant runs of pdflatex when you have some typo in labels/refs.

Another suggestion to Figures 9.25 and 9.29. Wouldn't these graphs look better with log scale x-axis?

X range can be 0.001 -- 10.

You'll need to add a few data points in sub-microsecond critical-section duration to show plausible shapes in those regions, though.

Thanks, Akira
--
Akira Yokosawa (3):
defer: Annotate consecutive QQZs as such for 'nq' build FAQ.txt: Advertise 'nq' build in #9 runlatex.sh: Give up early on undefined refs

 FAQ.txt               |  8 ++++++--
 defer/rcuusage.tex    | 20 +++++++++++---------
 utilities/runlatex.sh | 42 +++++++++++++++++++++++++-----------------
 3 files changed, 42 insertions(+), 28 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/3 defer: Annotate consecutive QQZs as such for 'nq' build

From ff1d191e40f23f5a0200ee3bbabe2073a6c03394 Mon Sep 17 00:00:00 2001
From: Akira Yokosawa <[email protected]>
Date: Sun, 31 May 2020 08:12:02 +0900
Subject: [PATCH 1/3] defer: Annotate consecutive QQZs as such for 'nq' build

---
 defer/rcuusage.tex | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)
defer/rcuusage.tex +11 -9
index d5329a39..72c1f331 100644
@@ -241,12 +241,13 @@ reader-writer locking are shown in
 Figure~\ref{fig:defer:Performance Advantage of RCU Over Reader-Writer Locking},
 which was generated on a 448-CPU 2.10\,GHz Intel x86 system.
 
-\QuickQuiz{
+\QuickQuizSeries{%
+\QuickQuizB{
 	WTF?
 	How the heck do you expect me to believe that RCU can have less
 	than a 300-picosecond overhead when the clock period at 2.10\,GHz
 	is almost 500\,picoseconds?
-}\QuickQuizAnswer{
+}\QuickQuizAnswerB{
 	First, consider that the inner loop used to
 	take this measurement is as follows:
 
@@ -290,13 +291,13 @@ which was generated on a 448-CPU 2.10\,GHz Intel x86 system.
 
 	It certainly is not just every day that a timing measurement
 	of 267 picoseconds turns out to be an overestimate!
-}\QuickQuizEnd
+}\QuickQuizEndB
 
-\QuickQuiz{
+\QuickQuizM{
 	Didn't an earlier release of this book show RCU read-side
 	overhead way down in the sub-picosecond range?
 	What happened???
-}\QuickQuizAnswer{
+}\QuickQuizAnswerM{
 	Excellent memory!!!
 	The overhead in some early releases was in fact roughly
 	100~femtoseconds.
@@ -328,12 +329,12 @@ which was generated on a 448-CPU 2.10\,GHz Intel x86 system.
 	So which change had the most effect, Linus's commit or the change in
 	the system?
 	This question is left as an exercise to the reader.
-}\QuickQuizEnd
+}\QuickQuizEndM
 
-\QuickQuiz{
+\QuickQuizE{
 	Why is there such large variation for the \co{rcu} trace in
 	Figure~\ref{fig:defer:Performance Advantage of RCU Over Reader-Writer Locking}?
-}\QuickQuizAnswer{
+}\QuickQuizAnswerE{
 	Keep in mind that this is a log-log plot, so those large-seeming
 	\co{rcu} variances in reality span only a few hundred picoseconds.
 	And that is such a short time that anything could cause it.
@@ -347,7 +348,8 @@ which was generated on a 448-CPU 2.10\,GHz Intel x86 system.
 	Attempting to reduce these variations by running the guest OSes
 	at real-time priority (as suggested by Joel Fernandes) is left
 	as an exercise for the reader.
-}\QuickQuizEnd
+}\QuickQuizEndE
+}                 % End of \QuickQuizSeries
 
 Note that reader-writer locking is more than an order of magnitude slower
 than RCU on a single CPU, and is more than \emph{four} orders of magnitude
Signature · 1 line
2.17.1
#2 lore raw

PATCH 2/3 FAQ.txt: Advertise 'nq' build in #9

In reply to Akira Yokosawa

From 4c3ca479e8aeb16fee229d9d52eb30b42c7618cc Mon Sep 17 00:00:00 2001
From: Akira Yokosawa <[email protected]>
Date: Sun, 31 May 2020 08:16:33 +0900
Subject: [PATCH 2/3] FAQ.txt: Advertise 'nq' build in #9

The experimental target "nq" can be a partial solution for those who don't like Quick Quizzes.

---
 FAQ.txt | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
FAQ.txt +6 -2
index e63117af..a284615f 100644
@@ -158,8 +158,12 @@
 
 	A.	Quite a few people like them a lot, so they will be
 		staying.  However, you can easily produce a copy of the
-		book that omits the Quick Quizzes by editing the Makefile
-		and qqz.sty files in the top-level directory.
+		book that hides the quiz part of Quick Quizzes in the
+		text by "make nq".
+
+		If the resulting perfbook-nq.pdf does not satisfy you,
+		you can edit the Makefile and qqz.sty files in the
+		top-level directory as you like.
 
 		One approach is to make the "\QuickQuiz" command in
 		qqz.sty be a no-op and to add line to the Makefile that
Signature · 1 line
2.17.1
#3 lore raw

PATCH 3/3 runlatex.sh: Give up early on undefined refs

In reply to Akira Yokosawa

From 073429a7f9e68b4b7f51c9b766bf8aa24da83081 Mon Sep 17 00:00:00 2001
From: Akira Yokosawa <[email protected]>
Date: Sun, 31 May 2020 08:18:34 +0900
Subject: [PATCH 3/3] runlatex.sh: Give up early on undefined refs

Successive "undefined refs"s mean true missing/misspelled labels/refs.

Add code to detect second "undefined refs" and to give up early.

Also add code to skip an unnecessary pdflatex run when .aux and .bbl files are up-to-date.
This can happen when you run "make" after removing perfbook.pdf, e.g., with a different LATEX_OPT setting.

---
 utilities/runlatex.sh | 42 +++++++++++++++++++++++++-----------------
 1 file changed, 25 insertions(+), 17 deletions(-)
utilities/runlatex.sh +25 -17
index 2aef9e2c..9687be95 100644
@@ -51,6 +51,18 @@ identical_warnings () {
 	return 1 ;
 }
 
+exerpt_warnings () {
+	if grep -q "LaTeX Warning:" $basename.log
+	then
+		echo "----- Excerpt around remaining warning messages -----"
+		grep -B 8 -A 5 "LaTeX Warning:" $basename.log | tee $basename-warning.log
+		echo "----- You can see $basename-warning.log for the warnings above. -----"
+		echo "----- If you need to, see $basename.log for details. -----"
+		rm -f $basename-warning-prev.log
+		exit 1
+	fi
+}
+
 iterate_latex () {
 	pdflatex $LATEX_OPT $basename > /dev/null 2>&1 < /dev/null || :
 	if grep -q '! Emergency stop.' $basename.log
@@ -83,26 +95,30 @@ basename=`echo $1 | sed -e 's/\.tex$//'`
 
 if ! test -r $basename-first.log
 then
-	if ! sh utilities/mpostcheck.sh
-	then
-		exit 1
-	fi
+	echo "No need to update aux and bbl files."
 	echo "pdflatex 1 for $basename.pdf"
-	iterate_latex
+	iter=1
+else
+	rm -f $basename-first.log
+	echo "pdflatex 2 for $basename.pdf # for possible bib update"
+	iter=2
 fi
-rm -f $basename-first.log
-iter=2
-echo "pdflatex 2 for $basename.pdf # for possible bib update"
 iterate_latex
 min_iter=2
 while grep -q 'LaTeX Warning: There were undefined references' $basename.log
 do
+	if test $undefined_refs
+	then
+		echo "Undefined refs remain, giving up."
+		exerpt_warnings
+	fi
 	if identical_warnings
 	then
 		break
 	fi
 	iter=`expr $iter + 1`
 	echo "pdflatex $iter for $basename.pdf # remaining undefined refs"
+	undefined_refs=1
 	iterate_latex
 done
 min_iter=3
@@ -116,15 +132,7 @@ do
 	echo "pdflatex $iter for $basename.pdf # label(s) may have changed"
 	iterate_latex
 done
-if grep -q "LaTeX Warning:" $basename.log
-then
-	echo "----- Excerpt around remaining warning messages -----"
-	grep -B 8 -A 5 "LaTeX Warning:" $basename.log | tee $basename-warning.log
-	echo "----- You can see $basename-warning.log for the warnings above. -----"
-	echo "----- If you need to, see $basename.log for details. -----"
-	rm -f $basename-warning-prev.log
-	exit 1
-fi
+exerpt_warnings
 rm -f $basename-warning.log $basename-warning-prev.log
 echo "'$basename.pdf' is ready."
 # cleveref version check (Ubuntu 18.04 LTS has buggy one
Signature · 1 line
2.17.1
maintainer
#4 lore raw
In reply to Akira Yokosawa
Quoted 5 lines · Akira Yokosawa

Hi Paul,

This is misc updates in response to your recent updates.

Patch 1/3 treats QQZ annotations for "nq" build.

Good reminder, thank you!

Quoted 4 lines

Patch 2/3 adds a paragraph in #9 of FAQ.txt. The wording may need your retouch for fluency.
Patch 3/3 is an independent improvement of runlatex.sh. It will avoid a few redundant runs of pdflatex when you have some typo in labels/refs.

Nice, queued and pushed, thank you!

Quoted 7 lines

Another suggestion to Figures 9.25 and 9.29. Wouldn't these graphs look better with log scale x-axis?

X range can be 0.001 -- 10.

You'll need to add a few data points in sub-microsecond critical-section duration to show plausible shapes in those regions, though.

I took a quick look and didn't find any nanosecond delay primitives in the Linux kernel, but yes, that would be nicer looking.

I don't expect to make further progress on this particular graph in the immediate future, but if you know of such a delay primitive, please don't keep it a secret! ;-)

Thanx, Paul

Quoted 15 lines

Thanks, Akira
--
Akira Yokosawa (3):
defer: Annotate consecutive QQZs as such for 'nq' build FAQ.txt: Advertise 'nq' build in #9 runlatex.sh: Give up early on undefined refs

 FAQ.txt               |  8 ++++++--
 defer/rcuusage.tex    | 20 +++++++++++---------
 utilities/runlatex.sh | 42 +++++++++++++++++++++++++-----------------
 3 files changed, 42 insertions(+), 28 deletions(-)
Signature · 1 line
2.17.1
#5 lore raw
Quoted 24 lines · Paul E. McKenney
Quoted 5 lines · Akira Yokosawa

Hi Paul,

This is misc updates in response to your recent updates.

Patch 1/3 treats QQZ annotations for "nq" build.

Good reminder, thank you!

Quoted 4 lines

Patch 2/3 adds a paragraph in #9 of FAQ.txt. The wording may need your retouch for fluency.
Patch 3/3 is an independent improvement of runlatex.sh. It will avoid a few redundant runs of pdflatex when you have some typo in labels/refs.

Nice, queued and pushed, thank you!

Quoted 7 lines

Another suggestion to Figures 9.25 and 9.29. Wouldn't these graphs look better with log scale x-axis?

X range can be 0.001 -- 10.

You'll need to add a few data points in sub-microsecond critical-section duration to show plausible shapes in those regions, though.

Quoted 6 lines

I took a quick look and didn't find any nanosecond delay primitives in the Linux kernel, but yes, that would be nicer looking.

I don't expect to make further progress on this particular graph in the immediate future, but if you know of such a delay primitive, please don't keep it a secret! ;-)

I find ndelay() defined in include/asm_generic/delay.h. I'm not sure if it works as you would expect, though.

Thanks, Akira

Quoted 18 lines

Thanx, Paul

Quoted 15 lines

Thanks, Akira
--
Akira Yokosawa (3):
defer: Annotate consecutive QQZs as such for 'nq' build FAQ.txt: Advertise 'nq' build in #9 runlatex.sh: Give up early on undefined refs

 FAQ.txt               |  8 ++++++--
 defer/rcuusage.tex    | 20 +++++++++++---------
 utilities/runlatex.sh | 42 +++++++++++++++++++++++++-----------------
 3 files changed, 42 insertions(+), 28 deletions(-)
Signature · 1 line
2.17.1
maintainer
#6 lore raw
Quoted 28 lines · Akira Yokosawa
Quoted 27 lines · Paul E. McKenney
Quoted 5 lines · Akira Yokosawa

Hi Paul,

This is misc updates in response to your recent updates.

Patch 1/3 treats QQZ annotations for "nq" build.

Good reminder, thank you!

Quoted 4 lines

Patch 2/3 adds a paragraph in #9 of FAQ.txt. The wording may need your retouch for fluency.
Patch 3/3 is an independent improvement of runlatex.sh. It will avoid a few redundant runs of pdflatex when you have some typo in labels/refs.

Nice, queued and pushed, thank you!

Quoted 7 lines

Another suggestion to Figures 9.25 and 9.29. Wouldn't these graphs look better with log scale x-axis?

X range can be 0.001 -- 10.

You'll need to add a few data points in sub-microsecond critical-section duration to show plausible shapes in those regions, though.

I took a quick look and didn't find any nanosecond delay primitives in the Linux kernel, but yes, that would be nicer looking.

Quoted 6 lines
Quoted 3 lines

I don't expect to make further progress on this particular graph in the immediate future, but if you know of such a delay primitive, please don't keep it a secret! ;-)

I find ndelay() defined in include/asm_generic/delay.h. I'm not sure if it works as you would expect, though.

I must be going blind, given that I missed that one!

I did try it out, and it suffers from about 10% timing errors. In contrast, udelay is usually less than 1%. But how about as shown below?

Thanx, Paul

------------------------------------------------------------------------

commit 7d9ab703b0a33ff5f8db330f0bac3dde9deead07

Author: Paul E. McKenney <[email protected]>
Date:   Sun May 31 18:14:57 2020 -0700

refperf: Change readdelay module parameter to nanoseconds

The current units of microseconds are too coarse, so this commit changes the units to nanoseconds. However, ndelay is used only for the nanoseconds with udelay being used for whole microseconds. For example, setting refperf.readdelay=1500 results in an ndelay(500) followed by a udelay(1).

kernel/rcu/refperf.c +36 -14
index 3b72925..96f8ba0 100644
@@ -66,8 +66,8 @@ torture_param(long, loops, 10000, "Number of loops per experiment.");
 torture_param(int, nreaders, -1, "Number of loops per experiment.");
 // Number of runs.
 torture_param(int, nruns, 30, "Number of experiments to run.");
-// Reader delay in microseconds, 0 for no delay.
-torture_param(int, readdelay, 0, "Read-side delay in microseconds.");
+// Reader delay in nanoseconds, 0 for no delay.
+torture_param(int, readdelay, 0, "Read-side delay in nanoseconds.");
 
 #ifdef MODULE
 # define REFPERF_SHUTDOWN 0
@@ -111,7 +111,8 @@ struct ref_perf_ops {
 	void (*init)(void);
 	void (*cleanup)(void);
 	void (*readsection)(const int nloops);
-	void (*delaysection)(const int nloops, const int ndelay);
+	void (*delaysection)(const int nloops,
+			     const int udelay, const int ndelay);
 	const char *name;
 };
 
@@ -127,13 +128,17 @@ static void ref_rcu_read_section(const int nloops)
 	}
 }
 
-static void ref_rcu_delay_section(const int nloops, const int ndelay)
+static void
+ref_rcu_delay_section(const int nloops, const int udelay, const int ndelay)
 {
 	int i;
 
 	for (i = nloops; i >= 0; i--) {
 		rcu_read_lock();
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);
 		rcu_read_unlock();
 	}
 }
@@ -165,14 +170,18 @@ static void srcu_ref_perf_read_section(const int nloops)
 	}
 }
 
-static void srcu_ref_perf_delay_section(const int nloops, const int ndelay)
+static void srcu_ref_perf_delay_section(const int nloops,
+					const int udelay, const int ndelay)
 {
 	int i;
 	int idx;
 
 	for (i = nloops; i >= 0; i--) {
 		idx = srcu_read_lock(srcu_ctlp);
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);
 		srcu_read_unlock(srcu_ctlp, idx);
 	}
 }
@@ -197,13 +206,17 @@ static void ref_refcnt_section(const int nloops)
 	}
 }
 
-static void ref_refcnt_delay_section(const int nloops, const int ndelay)
+static void
+ref_refcnt_delay_section(const int nloops, const int udelay, const int ndelay)
 {
 	int i;
 
 	for (i = nloops; i >= 0; i--) {
 		atomic_inc(&refcnt);
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);
 		atomic_dec(&refcnt);
 	}
 }
@@ -233,13 +246,17 @@ static void ref_rwlock_section(const int nloops)
 	}
 }
 
-static void ref_rwlock_delay_section(const int nloops, const int ndelay)
+static void
+ref_rwlock_delay_section(const int nloops, const int udelay, const int ndelay)
 {
 	int i;
 
 	for (i = nloops; i >= 0; i--) {
 		read_lock(&test_rwlock);
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);
 		read_unlock(&test_rwlock);
 	}
 }
@@ -269,13 +286,17 @@ static void ref_rwsem_section(const int nloops)
 	}
 }
 
-static void ref_rwsem_delay_section(const int nloops, const int ndelay)
+static void
+ref_rwsem_delay_section(const int nloops, const int udelay, const int ndelay)
 {
 	int i;
 
 	for (i = nloops; i >= 0; i--) {
 		down_read(&test_rwsem);
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);
 		up_read(&test_rwsem);
 	}
 }
@@ -292,7 +313,8 @@ static void rcu_perf_one_reader(void)
 	if (readdelay <= 0)
 		cur_ops->readsection(loops);
 	else
-		cur_ops->delaysection(loops, readdelay);
+		cur_ops->delaysection(loops,
+				      readdelay / 1000, readdelay % 1000);
 }
 
 // Reader kthread.  Repeatedly does empty RCU read-side
#7 lore raw
Quoted 29 lines · Paul E. McKenney
Quoted 28 lines · Akira Yokosawa
Quoted 27 lines · Paul E. McKenney
Quoted 5 lines · Akira Yokosawa

Hi Paul,

This is misc updates in response to your recent updates.

Patch 1/3 treats QQZ annotations for "nq" build.

Good reminder, thank you!

Quoted 4 lines

Patch 2/3 adds a paragraph in #9 of FAQ.txt. The wording may need your retouch for fluency.
Patch 3/3 is an independent improvement of runlatex.sh. It will avoid a few redundant runs of pdflatex when you have some typo in labels/refs.

Nice, queued and pushed, thank you!

Quoted 7 lines

Another suggestion to Figures 9.25 and 9.29. Wouldn't these graphs look better with log scale x-axis?

X range can be 0.001 -- 10.

You'll need to add a few data points in sub-microsecond critical-section duration to show plausible shapes in those regions, though.

I took a quick look and didn't find any nanosecond delay primitives in the Linux kernel, but yes, that would be nicer looking.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I don't expect to make further progress on this particular graph in the immediate future, but if you know of such a delay primitive, please don't keep it a secret! ;-)

I find ndelay() defined in include/asm_generic/delay.h. I'm not sure if it works as you would expect, though.

I must be going blind, given that I missed that one!

:-) :-)

Quoted 3 lines

I did try it out, and it suffers from about 10% timing errors. In contrast, udelay is usually less than 1%.

You mean udelay(1)'s error is less than 10ns, whereas ndelay(1000)'s error is about 100ns?

Looking at the definition of __udelay() and __ndelay() in arch/x86/lib/delay.c, the constant 0x10c7 has much effective bits than 0x0005. This is likely the cause of difference in errors.

Quoted 10 lines

But how about as shown below?

Thanx, Paul

------------------------------------------------------------------------

commit 7d9ab703b0a33ff5f8db330f0bac3dde9deead07

Author: Paul E. McKenney <[email protected]>
Date:   Sun May 31 18:14:57 2020 -0700
Quoted 7 lines

refperf: Change readdelay module parameter to nanoseconds

The current units of microseconds are too coarse, so this commit changes the units to nanoseconds. However, ndelay is used only for the nanoseconds with udelay being used for whole microseconds. For example, setting refperf.readdelay=1500 results in an ndelay(500) followed by a udelay(1).

Your code below looks opposite, udelay(1) + ndelay(500), doesn't it?

Quoted 117 lines
kernel/rcu/refperf.c +30 -12
index 3b72925..96f8ba0 100644
@@ -66,8 +66,8 @@ torture_param(long, loops, 10000, "Number of loops per experiment.");
 torture_param(int, nreaders, -1, "Number of loops per experiment.");
 // Number of runs.
 torture_param(int, nruns, 30, "Number of experiments to run.");
-// Reader delay in microseconds, 0 for no delay.
-torture_param(int, readdelay, 0, "Read-side delay in microseconds.");
+// Reader delay in nanoseconds, 0 for no delay.
+torture_param(int, readdelay, 0, "Read-side delay in nanoseconds.");
 
 #ifdef MODULE
 # define REFPERF_SHUTDOWN 0
@@ -111,7 +111,8 @@ struct ref_perf_ops {
 	void (*init)(void);
 	void (*cleanup)(void);
 	void (*readsection)(const int nloops);
-	void (*delaysection)(const int nloops, const int ndelay);
+	void (*delaysection)(const int nloops,
+			     const int udelay, const int ndelay);
 	const char *name;
 };
 
@@ -127,13 +128,17 @@ static void ref_rcu_read_section(const int nloops)
 	}
 }
 
-static void ref_rcu_delay_section(const int nloops, const int ndelay)
+static void
+ref_rcu_delay_section(const int nloops, const int udelay, const int ndelay)
 {
 	int i;
 
 	for (i = nloops; i >= 0; i--) {
 		rcu_read_lock();
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);
 		rcu_read_unlock();
 	}
 }
@@ -165,14 +170,18 @@ static void srcu_ref_perf_read_section(const int nloops)
 	}
 }
 
-static void srcu_ref_perf_delay_section(const int nloops, const int ndelay)
+static void srcu_ref_perf_delay_section(const int nloops,
+					const int udelay, const int ndelay)
 {
 	int i;
 	int idx;
 
 	for (i = nloops; i >= 0; i--) {
 		idx = srcu_read_lock(srcu_ctlp);
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);
 		srcu_read_unlock(srcu_ctlp, idx);
 	}
 }
@@ -197,13 +206,17 @@ static void ref_refcnt_section(const int nloops)
 	}
 }
 
-static void ref_refcnt_delay_section(const int nloops, const int ndelay)
+static void
+ref_refcnt_delay_section(const int nloops, const int udelay, const int ndelay)
 {
 	int i;
 
 	for (i = nloops; i >= 0; i--) {
 		atomic_inc(&refcnt);
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);
 		atomic_dec(&refcnt);
 	}
 }
@@ -233,13 +246,17 @@ static void ref_rwlock_section(const int nloops)
 	}
 }
 
-static void ref_rwlock_delay_section(const int nloops, const int ndelay)
+static void
+ref_rwlock_delay_section(const int nloops, const int udelay, const int ndelay)
 {
 	int i;
 
 	for (i = nloops; i >= 0; i--) {
 		read_lock(&test_rwlock);
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);
 		read_unlock(&test_rwlock);
 	}
 }
@@ -269,13 +286,17 @@ static void ref_rwsem_section(const int nloops)
 	}
 }
 
-static void ref_rwsem_delay_section(const int nloops, const int ndelay)
+static void
+ref_rwsem_delay_section(const int nloops, const int udelay, const int ndelay)

kernel/rcu/refperf.c static void ref_rwsem_section(const int nloops) · line 299

Quoted 10 lines
kernel/rcu/refperf.c +4 -1
 {
 	int i;
 
 	for (i = nloops; i >= 0; i--) {
 		down_read(&test_rwsem);
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);

Maybe defining a helper function/macro for this common pattern in rcu.h can ease maintenance cost. Say undelay(udl, ndl)?

Thanks, Akira

Quoted 14 lines
kernel/rcu/refperf.c +2 -1
 		up_read(&test_rwsem);
 	}
 }
@@ -292,7 +313,8 @@ static void rcu_perf_one_reader(void)
 	if (readdelay <= 0)
 		cur_ops->readsection(loops);
 	else
-		cur_ops->delaysection(loops, readdelay);
+		cur_ops->delaysection(loops,
+				      readdelay / 1000, readdelay % 1000);
 }
 
 // Reader kthread.  Repeatedly does empty RCU read-side
maintainer
#8 lore raw
Quoted 39 lines · Akira Yokosawa
Quoted 29 lines · Paul E. McKenney
Quoted 28 lines · Akira Yokosawa
Quoted 27 lines · Paul E. McKenney
Quoted 5 lines · Akira Yokosawa

Hi Paul,

This is misc updates in response to your recent updates.

Patch 1/3 treats QQZ annotations for "nq" build.

Good reminder, thank you!

Quoted 4 lines

Patch 2/3 adds a paragraph in #9 of FAQ.txt. The wording may need your retouch for fluency.
Patch 3/3 is an independent improvement of runlatex.sh. It will avoid a few redundant runs of pdflatex when you have some typo in labels/refs.

Nice, queued and pushed, thank you!

Quoted 7 lines

Another suggestion to Figures 9.25 and 9.29. Wouldn't these graphs look better with log scale x-axis?

X range can be 0.001 -- 10.

You'll need to add a few data points in sub-microsecond critical-section duration to show plausible shapes in those regions, though.

I took a quick look and didn't find any nanosecond delay primitives in the Linux kernel, but yes, that would be nicer looking.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I don't expect to make further progress on this particular graph in the immediate future, but if you know of such a delay primitive, please don't keep it a secret! ;-)

I find ndelay() defined in include/asm_generic/delay.h. I'm not sure if it works as you would expect, though.

I must be going blind, given that I missed that one!

Quoted 7 lines

:-) :-)

Quoted 2 lines

I did try it out, and it suffers from about 10% timing errors. In contrast, udelay is usually less than 1%.

You mean udelay(1)'s error is less than 10ns, whereas ndelay(1000)'s error is about 100ns?

Yuck. The 10% was a preliminary eyeballing. An overnight run showed it to be worst than that. 100ns gets me about 130ns, 200ns gets me about 270ns, and 500ns gets me about 600ns. So ndelay() is useful only for very short delays.

Quoted 3 lines

Looking at the definition of __udelay() and __ndelay() in arch/x86/lib/delay.c, the constant 0x10c7 has much effective bits than 0x0005. This is likely the cause of difference in errors.

That makes a lot of sense, and thank you for checking!

Quoted 12 lines
Quoted 12 lines

But how about as shown below?

Thanx, Paul

------------------------------------------------------------------------

commit 7d9ab703b0a33ff5f8db330f0bac3dde9deead07

Author: Paul E. McKenney <[email protected]>
Date:   Sun May 31 18:14:57 2020 -0700

refperf: Change readdelay module parameter to nanoseconds

Quoted 7 lines
Quoted 5 lines

The current units of microseconds are too coarse, so this commit changes the units to nanoseconds. However, ndelay is used only for the nanoseconds with udelay being used for whole microseconds. For example, setting refperf.readdelay=1500 results in an ndelay(500) followed by a udelay(1).

Your code below looks opposite, udelay(1) + ndelay(500), doesn't it?

Indeed it does! I will fix this, thank you.

Quoted 119 lines
Quoted 119 lines
kernel/rcu/refperf.c +30 -12
index 3b72925..96f8ba0 100644
@@ -66,8 +66,8 @@ torture_param(long, loops, 10000, "Number of loops per experiment.");
 torture_param(int, nreaders, -1, "Number of loops per experiment.");
 // Number of runs.
 torture_param(int, nruns, 30, "Number of experiments to run.");
-// Reader delay in microseconds, 0 for no delay.
-torture_param(int, readdelay, 0, "Read-side delay in microseconds.");
+// Reader delay in nanoseconds, 0 for no delay.
+torture_param(int, readdelay, 0, "Read-side delay in nanoseconds.");
 
 #ifdef MODULE
 # define REFPERF_SHUTDOWN 0
@@ -111,7 +111,8 @@ struct ref_perf_ops {
 	void (*init)(void);
 	void (*cleanup)(void);
 	void (*readsection)(const int nloops);
-	void (*delaysection)(const int nloops, const int ndelay);
+	void (*delaysection)(const int nloops,
+			     const int udelay, const int ndelay);
 	const char *name;
 };
 
@@ -127,13 +128,17 @@ static void ref_rcu_read_section(const int nloops)
 	}
 }
 
-static void ref_rcu_delay_section(const int nloops, const int ndelay)
+static void
+ref_rcu_delay_section(const int nloops, const int udelay, const int ndelay)
 {
 	int i;
 
 	for (i = nloops; i >= 0; i--) {
 		rcu_read_lock();
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);
 		rcu_read_unlock();
 	}
 }
@@ -165,14 +170,18 @@ static void srcu_ref_perf_read_section(const int nloops)
 	}
 }
 
-static void srcu_ref_perf_delay_section(const int nloops, const int ndelay)
+static void srcu_ref_perf_delay_section(const int nloops,
+					const int udelay, const int ndelay)
 {
 	int i;
 	int idx;
 
 	for (i = nloops; i >= 0; i--) {
 		idx = srcu_read_lock(srcu_ctlp);
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);
 		srcu_read_unlock(srcu_ctlp, idx);
 	}
 }
@@ -197,13 +206,17 @@ static void ref_refcnt_section(const int nloops)
 	}
 }
 
-static void ref_refcnt_delay_section(const int nloops, const int ndelay)
+static void
+ref_refcnt_delay_section(const int nloops, const int udelay, const int ndelay)
 {
 	int i;
 
 	for (i = nloops; i >= 0; i--) {
 		atomic_inc(&refcnt);
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);
 		atomic_dec(&refcnt);
 	}
 }
@@ -233,13 +246,17 @@ static void ref_rwlock_section(const int nloops)
 	}
 }
 
-static void ref_rwlock_delay_section(const int nloops, const int ndelay)
+static void
+ref_rwlock_delay_section(const int nloops, const int udelay, const int ndelay)
 {
 	int i;
 
 	for (i = nloops; i >= 0; i--) {
 		read_lock(&test_rwlock);
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);
 		read_unlock(&test_rwlock);
 	}
 }
@@ -269,13 +286,17 @@ static void ref_rwsem_section(const int nloops)
 	}
 }
 
-static void ref_rwsem_delay_section(const int nloops, const int ndelay)
+static void
+ref_rwsem_delay_section(const int nloops, const int udelay, const int ndelay)
 {
 	int i;
 
Quoted 10 lines

kernel/rcu/refperf.c static void ref_rwsem_section(const int nloops) · line 299

Quoted 7 lines
kernel/rcu/refperf.c +4 -1
 	for (i = nloops; i >= 0; i--) {
 		down_read(&test_rwsem);
-		udelay(ndelay);
+		if (udelay)
+			udelay(udelay);
+		if (ndelay)
+			ndelay(ndelay);

Maybe defining a helper function/macro for this common pattern in rcu.h can ease maintenance cost. Say undelay(udl, ndl)?

Good point! Updated and currently testing.

Thank you for your review and comments!

Thanx, Paul

Quoted 16 lines

Thanks, Akira

Quoted 14 lines
kernel/rcu/refperf.c +2 -1
 		up_read(&test_rwsem);
 	}
 }
@@ -292,7 +313,8 @@ static void rcu_perf_one_reader(void)
 	if (readdelay <= 0)
 		cur_ops->readsection(loops);
 	else
-		cur_ops->delaysection(loops, readdelay);
+		cur_ops->delaysection(loops,
+				      readdelay / 1000, readdelay % 1000);
 }
 
 // Reader kthread.  Repeatedly does empty RCU read-side
#9 lore raw
Quoted 45 lines · Paul E. McKenney
Quoted 44 lines · Akira Yokosawa
Quoted 29 lines · Paul E. McKenney
Quoted 28 lines · Akira Yokosawa
Quoted 27 lines · Paul E. McKenney
Quoted 5 lines · Akira Yokosawa

Hi Paul,

This is misc updates in response to your recent updates.

Patch 1/3 treats QQZ annotations for "nq" build.

Good reminder, thank you!

Quoted 4 lines

Patch 2/3 adds a paragraph in #9 of FAQ.txt. The wording may need your retouch for fluency.
Patch 3/3 is an independent improvement of runlatex.sh. It will avoid a few redundant runs of pdflatex when you have some typo in labels/refs.

Nice, queued and pushed, thank you!

Quoted 7 lines

Another suggestion to Figures 9.25 and 9.29. Wouldn't these graphs look better with log scale x-axis?

X range can be 0.001 -- 10.

You'll need to add a few data points in sub-microsecond critical-section duration to show plausible shapes in those regions, though.

I took a quick look and didn't find any nanosecond delay primitives in the Linux kernel, but yes, that would be nicer looking.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I don't expect to make further progress on this particular graph in the immediate future, but if you know of such a delay primitive, please don't keep it a secret! ;-)

I find ndelay() defined in include/asm_generic/delay.h. I'm not sure if it works as you would expect, though.

I must be going blind, given that I missed that one!

:-) :-)

Quoted 2 lines

I did try it out, and it suffers from about 10% timing errors. In contrast, udelay is usually less than 1%.

Quoted 7 lines
Quoted 2 lines

You mean udelay(1)'s error is less than 10ns, whereas ndelay(1000)'s error is about 100ns?

Yuck. The 10% was a preliminary eyeballing. An overnight run showed it to be worst than that. 100ns gets me about 130ns, 200ns gets me about 270ns, and 500ns gets me about 600ns. So ndelay() is useful only for very short delays.

To compensate the error, how about doing the appended? Yes, this is kind of ugly...

Another point you should be aware. It looks like arch/powerpc does not have __ndelay defined. Which means ndelay() would cause build error. Still, I might be missing something.

Thanks, Akira

kernel/rcu/refperf.c +1 -1
index 5db165ecd465..0a3764ea220c 100644
@@ -122,7 +122,7 @@ static void un_delay(const int udl, const int ndl)
        if (udl)
                udelay(udl);
        if (ndl)
-               ndelay(ndl);
+               ndelay((ndl * 859) / 1000); // 5 : 2^32/1000000000 (4.295)
 }
 
 static void ref_rcu_read_section(const int nloops)
maintainer
#10 lore raw
Quoted 54 lines · Akira Yokosawa
Quoted 45 lines · Paul E. McKenney
Quoted 44 lines · Akira Yokosawa
Quoted 29 lines · Paul E. McKenney
Quoted 28 lines · Akira Yokosawa
Quoted 27 lines · Paul E. McKenney
Quoted 5 lines · Akira Yokosawa

Hi Paul,

This is misc updates in response to your recent updates.

Patch 1/3 treats QQZ annotations for "nq" build.

Good reminder, thank you!

Quoted 4 lines

Patch 2/3 adds a paragraph in #9 of FAQ.txt. The wording may need your retouch for fluency.
Patch 3/3 is an independent improvement of runlatex.sh. It will avoid a few redundant runs of pdflatex when you have some typo in labels/refs.

Nice, queued and pushed, thank you!

Quoted 7 lines

Another suggestion to Figures 9.25 and 9.29. Wouldn't these graphs look better with log scale x-axis?

X range can be 0.001 -- 10.

You'll need to add a few data points in sub-microsecond critical-section duration to show plausible shapes in those regions, though.

I took a quick look and didn't find any nanosecond delay primitives in the Linux kernel, but yes, that would be nicer looking.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I don't expect to make further progress on this particular graph in the immediate future, but if you know of such a delay primitive, please don't keep it a secret! ;-)

I find ndelay() defined in include/asm_generic/delay.h. I'm not sure if it works as you would expect, though.

I must be going blind, given that I missed that one!

:-) :-)

Quoted 2 lines

I did try it out, and it suffers from about 10% timing errors. In contrast, udelay is usually less than 1%.

Quoted 7 lines
Quoted 2 lines

You mean udelay(1)'s error is less than 10ns, whereas ndelay(1000)'s error is about 100ns?

Yuck. The 10% was a preliminary eyeballing. An overnight run showed it to be worst than that. 100ns gets me about 130ns, 200ns gets me about 270ns, and 500ns gets me about 600ns. So ndelay() is useful only for very short delays.

Quoted 6 lines

To compensate the error, how about doing the appended? Yes, this is kind of ugly...

Another point you should be aware. It looks like arch/powerpc does not have __ndelay defined. Which means ndelay() would cause build error. Still, I might be missing something.

That is quite clever! It does turn ndelay(1) into ndelay(0), but it probably costs more than a nanosecond to do the integer division, so that shouldn't be a problem.

However, I believe that any such compensatory schemes should be done within ndelay() rather than by its users. Plus, as you imply, different architectures might need different adjustments. My concern is that different CPU generations within a given architecture might also need different adjustments. :-(

Thanx, Paul

Quoted 18 lines

Thanks, Akira

kernel/rcu/refperf.c +1 -1
index 5db165ecd465..0a3764ea220c 100644
@@ -122,7 +122,7 @@ static void un_delay(const int udl, const int ndl)
        if (udl)
                udelay(udl);
        if (ndl)
-               ndelay(ndl);
+               ndelay((ndl * 859) / 1000); // 5 : 2^32/1000000000 (4.295)
 }
 
 static void ref_rcu_read_section(const int nloops)
#11 lore raw
Quoted 62 lines · Paul E. McKenney
Quoted 54 lines · Akira Yokosawa
Quoted 45 lines · Paul E. McKenney
Quoted 44 lines · Akira Yokosawa
Quoted 29 lines · Paul E. McKenney
Quoted 28 lines · Akira Yokosawa
Quoted 27 lines · Paul E. McKenney
Quoted 5 lines · Akira Yokosawa

Hi Paul,

This is misc updates in response to your recent updates.

Patch 1/3 treats QQZ annotations for "nq" build.

Good reminder, thank you!

Quoted 4 lines

Patch 2/3 adds a paragraph in #9 of FAQ.txt. The wording may need your retouch for fluency.
Patch 3/3 is an independent improvement of runlatex.sh. It will avoid a few redundant runs of pdflatex when you have some typo in labels/refs.

Nice, queued and pushed, thank you!

Quoted 7 lines

Another suggestion to Figures 9.25 and 9.29. Wouldn't these graphs look better with log scale x-axis?

X range can be 0.001 -- 10.

You'll need to add a few data points in sub-microsecond critical-section duration to show plausible shapes in those regions, though.

I took a quick look and didn't find any nanosecond delay primitives in the Linux kernel, but yes, that would be nicer looking.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I don't expect to make further progress on this particular graph in the immediate future, but if you know of such a delay primitive, please don't keep it a secret! ;-)

I find ndelay() defined in include/asm_generic/delay.h. I'm not sure if it works as you would expect, though.

I must be going blind, given that I missed that one!

:-) :-)

Quoted 2 lines

I did try it out, and it suffers from about 10% timing errors. In contrast, udelay is usually less than 1%.

Quoted 7 lines
Quoted 2 lines

You mean udelay(1)'s error is less than 10ns, whereas ndelay(1000)'s error is about 100ns?

Yuck. The 10% was a preliminary eyeballing. An overnight run showed it to be worst than that. 100ns gets me about 130ns, 200ns gets me about 270ns, and 500ns gets me about 600ns. So ndelay() is useful only for very short delays.

Quoted 6 lines

To compensate the error, how about doing the appended? Yes, this is kind of ugly...

Another point you should be aware. It looks like arch/powerpc does not have __ndelay defined. Which means ndelay() would cause build error. Still, I might be missing something.

Quoted 6 lines

That is quite clever! It does turn ndelay(1) into ndelay(0), but it probably costs more than a nanosecond to do the integer division, so that shouldn't be a problem.

However, I believe that any such compensatory schemes should be done within ndelay() rather than by its users.

I'm not brave enough to change the behavior of ndelay() seeing the number of call sites in kernel code base, especially under drivers/.

Looking at the updated Figures 9.25 and 9.29, the timing error of ndelay() results in the discrepancy of "rcu" plots from the ideal orthogonal lines in sub-microseconds regions (0.1, 0.2, and 0.5us). I don't think you like such misleading plots.

You could instead compensate the x-values you give to ndelay().

On x86, you know the resolution of xdelay() is 1.164153ns. Which means if you want a time delay of 100ns, ndelay(86) will be 100.117ns.
ndelay(172) will be 200.234ns and ndelay(429) will be 499.422ns. ndelay(430) will be 500.586ns, which is the 2nd closest. If you don't want to exceed 500ns, ndelay(430) would be your choice.

I think this level of tweak is worthwhile, especially it will result in a better looking plot of RCU scaling.

Thoughts?

Thanks, Akira

PS: The bumps in Figures 9.25 and 9.29 in the sub-microsecond region might be the effect of difference of instruction stream. As we have seen in Figure 9.22, slight changes in the code path, e.g. jump target alignment, can cause 10% -- 20% of performance difference.

Enforce inlining un_delay() might or might not help. Just guessing.

Quoted 25 lines

Plus, as you imply, different
architectures might need different adjustments. My concern is that different CPU generations within a given architecture might also need different adjustments. :-(

Thanx, Paul

Quoted 18 lines

Thanks, Akira

kernel/rcu/refperf.c +1 -1
index 5db165ecd465..0a3764ea220c 100644
@@ -122,7 +122,7 @@ static void un_delay(const int udl, const int ndl)
        if (udl)
                udelay(udl);
        if (ndl)
-               ndelay(ndl);
+               ndelay((ndl * 859) / 1000); // 5 : 2^32/1000000000 (4.295)
 }
 
 static void ref_rcu_read_section(const int nloops)
maintainer
#12 lore raw
Quoted 87 lines · Akira Yokosawa
Quoted 62 lines · Paul E. McKenney
Quoted 54 lines · Akira Yokosawa
Quoted 45 lines · Paul E. McKenney
Quoted 44 lines · Akira Yokosawa
Quoted 29 lines · Paul E. McKenney
Quoted 28 lines · Akira Yokosawa
Quoted 27 lines · Paul E. McKenney
Quoted 5 lines · Akira Yokosawa

Hi Paul,

This is misc updates in response to your recent updates.

Patch 1/3 treats QQZ annotations for "nq" build.

Good reminder, thank you!

Quoted 4 lines

Patch 2/3 adds a paragraph in #9 of FAQ.txt. The wording may need your retouch for fluency.
Patch 3/3 is an independent improvement of runlatex.sh. It will avoid a few redundant runs of pdflatex when you have some typo in labels/refs.

Nice, queued and pushed, thank you!

Quoted 7 lines

Another suggestion to Figures 9.25 and 9.29. Wouldn't these graphs look better with log scale x-axis?

X range can be 0.001 -- 10.

You'll need to add a few data points in sub-microsecond critical-section duration to show plausible shapes in those regions, though.

I took a quick look and didn't find any nanosecond delay primitives in the Linux kernel, but yes, that would be nicer looking.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I don't expect to make further progress on this particular graph in the immediate future, but if you know of such a delay primitive, please don't keep it a secret! ;-)

I find ndelay() defined in include/asm_generic/delay.h. I'm not sure if it works as you would expect, though.

I must be going blind, given that I missed that one!

:-) :-)

Quoted 2 lines

I did try it out, and it suffers from about 10% timing errors. In contrast, udelay is usually less than 1%.

Quoted 7 lines
Quoted 2 lines

You mean udelay(1)'s error is less than 10ns, whereas ndelay(1000)'s error is about 100ns?

Yuck. The 10% was a preliminary eyeballing. An overnight run showed it to be worst than that. 100ns gets me about 130ns, 200ns gets me about 270ns, and 500ns gets me about 600ns. So ndelay() is useful only for very short delays.

Quoted 6 lines

To compensate the error, how about doing the appended? Yes, this is kind of ugly...

Another point you should be aware. It looks like arch/powerpc does not have __ndelay defined. Which means ndelay() would cause build error. Still, I might be missing something.

Quoted 6 lines

That is quite clever! It does turn ndelay(1) into ndelay(0), but it probably costs more than a nanosecond to do the integer division, so that shouldn't be a problem.

However, I believe that any such compensatory schemes should be done within ndelay() rather than by its users.

I'm not brave enough to change the behavior of ndelay() seeing the number of call sites in kernel code base, especially under drivers/.

Looking at the updated Figures 9.25 and 9.29, the timing error of ndelay() results in the discrepancy of "rcu" plots from the ideal orthogonal lines in sub-microseconds regions (0.1, 0.2, and 0.5us). I don't think you like such misleading plots.

You could instead compensate the x-values you give to ndelay().

On x86, you know the resolution of xdelay() is 1.164153ns. Which means if you want a time delay of 100ns, ndelay(86) will be 100.117ns.
ndelay(172) will be 200.234ns and ndelay(429) will be 499.422ns. ndelay(430) will be 500.586ns, which is the 2nd closest. If you don't want to exceed 500ns, ndelay(430) would be your choice.

Quoted 4 lines

I think this level of tweak is worthwhile, especially it will result in a better looking plot of RCU scaling.

Thoughts?

Huh.

What we could do is to do a calibration pass where we sample a fine-grained timesource, spin on a series of ndelay() calls that last for a few microseconds, then resample the fine-grained timestamp. We could then do a binary search so as to compute a corrected ndelay argument. We would then need to verify the corrected argument.

This procedure would be architecture independent, and might also account for instruction-stream differences.

Is there a better way? Seems like there should be. ;-)

Thanx, Paul

Quoted 34 lines

PS: The bumps in Figures 9.25 and 9.29 in the sub-microsecond region might be the effect of difference of instruction stream. As we have seen in Figure 9.22, slight changes in the code path, e.g. jump target alignment, can cause 10% -- 20% of performance difference.

Enforce inlining un_delay() might or might not help. Just guessing.

Quoted 25 lines

Plus, as you imply, different
architectures might need different adjustments. My concern is that different CPU generations within a given architecture might also need different adjustments. :-(

Thanx, Paul

Quoted 18 lines

Thanks, Akira

kernel/rcu/refperf.c +1 -1
index 5db165ecd465..0a3764ea220c 100644
@@ -122,7 +122,7 @@ static void un_delay(const int udl, const int ndl)
        if (udl)
                udelay(udl);
        if (ndl)
-               ndelay(ndl);
+               ndelay((ndl * 859) / 1000); // 5 : 2^32/1000000000 (4.295)
 }
 
 static void ref_rcu_read_section(const int nloops)
#13 lore raw
Quoted 95 lines · Paul E. McKenney
Quoted 87 lines · Akira Yokosawa
Quoted 62 lines · Paul E. McKenney
Quoted 54 lines · Akira Yokosawa
Quoted 45 lines · Paul E. McKenney
Quoted 44 lines · Akira Yokosawa
Quoted 29 lines · Paul E. McKenney
Quoted 28 lines · Akira Yokosawa
Quoted 27 lines · Paul E. McKenney
Quoted 5 lines · Akira Yokosawa

Hi Paul,

This is misc updates in response to your recent updates.

Patch 1/3 treats QQZ annotations for "nq" build.

Good reminder, thank you!

Quoted 4 lines

Patch 2/3 adds a paragraph in #9 of FAQ.txt. The wording may need your retouch for fluency.
Patch 3/3 is an independent improvement of runlatex.sh. It will avoid a few redundant runs of pdflatex when you have some typo in labels/refs.

Nice, queued and pushed, thank you!

Quoted 7 lines

Another suggestion to Figures 9.25 and 9.29. Wouldn't these graphs look better with log scale x-axis?

X range can be 0.001 -- 10.

You'll need to add a few data points in sub-microsecond critical-section duration to show plausible shapes in those regions, though.

I took a quick look and didn't find any nanosecond delay primitives in the Linux kernel, but yes, that would be nicer looking.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I don't expect to make further progress on this particular graph in the immediate future, but if you know of such a delay primitive, please don't keep it a secret! ;-)

I find ndelay() defined in include/asm_generic/delay.h. I'm not sure if it works as you would expect, though.

I must be going blind, given that I missed that one!

:-) :-)

Quoted 2 lines

I did try it out, and it suffers from about 10% timing errors. In contrast, udelay is usually less than 1%.

Quoted 7 lines
Quoted 2 lines

You mean udelay(1)'s error is less than 10ns, whereas ndelay(1000)'s error is about 100ns?

Yuck. The 10% was a preliminary eyeballing. An overnight run showed it to be worst than that. 100ns gets me about 130ns, 200ns gets me about 270ns, and 500ns gets me about 600ns. So ndelay() is useful only for very short delays.

Quoted 6 lines

To compensate the error, how about doing the appended? Yes, this is kind of ugly...

Another point you should be aware. It looks like arch/powerpc does not have __ndelay defined. Which means ndelay() would cause build error. Still, I might be missing something.

Quoted 6 lines

That is quite clever! It does turn ndelay(1) into ndelay(0), but it probably costs more than a nanosecond to do the integer division, so that shouldn't be a problem.

However, I believe that any such compensatory schemes should be done within ndelay() rather than by its users.

I'm not brave enough to change the behavior of ndelay() seeing the number of call sites in kernel code base, especially under drivers/.

Looking at the updated Figures 9.25 and 9.29, the timing error of ndelay() results in the discrepancy of "rcu" plots from the ideal orthogonal lines in sub-microseconds regions (0.1, 0.2, and 0.5us). I don't think you like such misleading plots.

You could instead compensate the x-values you give to ndelay().

On x86, you know the resolution of xdelay() is 1.164153ns. Which means if you want a time delay of 100ns, ndelay(86) will be 100.117ns.
ndelay(172) will be 200.234ns and ndelay(429) will be 499.422ns. ndelay(430) will be 500.586ns, which is the 2nd closest. If you don't want to exceed 500ns, ndelay(430) would be your choice.

Quoted 4 lines

I think this level of tweak is worthwhile, especially it will result in a better looking plot of RCU scaling.

Thoughts?

Huh.

Quoted 8 lines

What we could do is to do a calibration pass where we sample a fine-grained timesource, spin on a series of ndelay() calls that last for a few microseconds, then resample the fine-grained timestamp. We could then do a binary search so as to compute a corrected ndelay argument. We would then need to verify the corrected argument.

This procedure would be architecture independent, and might also account for instruction-stream differences.

This calibration part could be implemented and tested on a small system, assuming you have sub-microsecond ndelay() and fine-grained timer.

For example, powerpc I mentioned earlier uses the fallback definition in linux/delay.h:

	#ifndef ndelay
	static inline void ndelay(unsigned long x)
	{
		udelay(DIV_ROUND_UP(x, 1000));
	}
	#define ndelay(x) ndelay(x)
	#endif
Quoted 2 lines

Is there a better way? Seems like there should be. ;-)

There can be someone already has done a similar thing.

Thanks, Akira

Quoted 37 lines

Thanx, Paul

Quoted 34 lines

PS: The bumps in Figures 9.25 and 9.29 in the sub-microsecond region might be the effect of difference of instruction stream. As we have seen in Figure 9.22, slight changes in the code path, e.g. jump target alignment, can cause 10% -- 20% of performance difference.

Enforce inlining un_delay() might or might not help. Just guessing.

Quoted 25 lines

Plus, as you imply, different
architectures might need different adjustments. My concern is that different CPU generations within a given architecture might also need different adjustments. :-(

Thanx, Paul

Quoted 18 lines

Thanks, Akira

kernel/rcu/refperf.c +1 -1
index 5db165ecd465..0a3764ea220c 100644
@@ -122,7 +122,7 @@ static void un_delay(const int udl, const int ndl)
        if (udl)
                udelay(udl);
        if (ndl)
-               ndelay(ndl);
+               ndelay((ndl * 859) / 1000); // 5 : 2^32/1000000000 (4.295)
 }
 
 static void ref_rcu_read_section(const int nloops)
maintainer
#14 lore raw
Quoted 102 lines · Akira Yokosawa
Quoted 101 lines · Paul E. McKenney
Quoted 87 lines · Akira Yokosawa
Quoted 62 lines · Paul E. McKenney
Quoted 54 lines · Akira Yokosawa
Quoted 45 lines · Paul E. McKenney
Quoted 44 lines · Akira Yokosawa
Quoted 29 lines · Paul E. McKenney
Quoted 28 lines · Akira Yokosawa
Quoted 27 lines · Paul E. McKenney
Quoted 5 lines · Akira Yokosawa

Hi Paul,

This is misc updates in response to your recent updates.

Patch 1/3 treats QQZ annotations for "nq" build.

Good reminder, thank you!

Quoted 4 lines

Patch 2/3 adds a paragraph in #9 of FAQ.txt. The wording may need your retouch for fluency.
Patch 3/3 is an independent improvement of runlatex.sh. It will avoid a few redundant runs of pdflatex when you have some typo in labels/refs.

Nice, queued and pushed, thank you!

Quoted 7 lines

Another suggestion to Figures 9.25 and 9.29. Wouldn't these graphs look better with log scale x-axis?

X range can be 0.001 -- 10.

You'll need to add a few data points in sub-microsecond critical-section duration to show plausible shapes in those regions, though.

I took a quick look and didn't find any nanosecond delay primitives in the Linux kernel, but yes, that would be nicer looking.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I don't expect to make further progress on this particular graph in the immediate future, but if you know of such a delay primitive, please don't keep it a secret! ;-)

I find ndelay() defined in include/asm_generic/delay.h. I'm not sure if it works as you would expect, though.

I must be going blind, given that I missed that one!

:-) :-)

Quoted 2 lines

I did try it out, and it suffers from about 10% timing errors. In contrast, udelay is usually less than 1%.

Quoted 7 lines
Quoted 2 lines

You mean udelay(1)'s error is less than 10ns, whereas ndelay(1000)'s error is about 100ns?

Yuck. The 10% was a preliminary eyeballing. An overnight run showed it to be worst than that. 100ns gets me about 130ns, 200ns gets me about 270ns, and 500ns gets me about 600ns. So ndelay() is useful only for very short delays.

Quoted 6 lines

To compensate the error, how about doing the appended? Yes, this is kind of ugly...

Another point you should be aware. It looks like arch/powerpc does not have __ndelay defined. Which means ndelay() would cause build error. Still, I might be missing something.

Quoted 6 lines

That is quite clever! It does turn ndelay(1) into ndelay(0), but it probably costs more than a nanosecond to do the integer division, so that shouldn't be a problem.

However, I believe that any such compensatory schemes should be done within ndelay() rather than by its users.

I'm not brave enough to change the behavior of ndelay() seeing the number of call sites in kernel code base, especially under drivers/.

Looking at the updated Figures 9.25 and 9.29, the timing error of ndelay() results in the discrepancy of "rcu" plots from the ideal orthogonal lines in sub-microseconds regions (0.1, 0.2, and 0.5us). I don't think you like such misleading plots.

You could instead compensate the x-values you give to ndelay().

On x86, you know the resolution of xdelay() is 1.164153ns. Which means if you want a time delay of 100ns, ndelay(86) will be 100.117ns.
ndelay(172) will be 200.234ns and ndelay(429) will be 499.422ns. ndelay(430) will be 500.586ns, which is the 2nd closest. If you don't want to exceed 500ns, ndelay(430) would be your choice.

Quoted 4 lines

I think this level of tweak is worthwhile, especially it will result in a better looking plot of RCU scaling.

Thoughts?

Huh.

What we could do is to do a calibration pass where we sample a fine-grained timesource, spin on a series of ndelay() calls that last for a few microseconds, then resample the fine-grained timestamp. We could then do a binary search so as to compute a corrected ndelay argument. We would then need to verify the corrected argument.

Quoted 5 lines
Quoted 2 lines

This procedure would be architecture independent, and might also account for instruction-stream differences.

This calibration part could be implemented and tested on a small system, assuming you have sub-microsecond ndelay() and fine-grained timer.

Just to be clear, my thought is to do a short calibration cycle on the system running the actual test as part of refperf initialization.

Quoted 10 lines

For example, powerpc I mentioned earlier uses the fallback definition in linux/delay.h:

	#ifndef ndelay
	static inline void ndelay(unsigned long x)
	{
		udelay(DIV_ROUND_UP(x, 1000));
	}
	#define ndelay(x) ndelay(x)
	#endif

Indeed, any calibration would need to be careful of this!

Quoted 3 lines
Quoted 1 line

Is there a better way? Seems like there should be. ;-)

There can be someone already has done a similar thing.

Quite possibly.

Thanx, Paul

Quoted 39 lines

Thanks, Akira

Quoted 37 lines

Thanx, Paul

Quoted 34 lines

PS: The bumps in Figures 9.25 and 9.29 in the sub-microsecond region might be the effect of difference of instruction stream. As we have seen in Figure 9.22, slight changes in the code path, e.g. jump target alignment, can cause 10% -- 20% of performance difference.

Enforce inlining un_delay() might or might not help. Just guessing.

Quoted 25 lines

Plus, as you imply, different
architectures might need different adjustments. My concern is that different CPU generations within a given architecture might also need different adjustments. :-(

Thanx, Paul

Quoted 18 lines

Thanks, Akira

kernel/rcu/refperf.c +1 -1
index 5db165ecd465..0a3764ea220c 100644
@@ -122,7 +122,7 @@ static void un_delay(const int udl, const int ndl)
        if (udl)
                udelay(udl);
        if (ndl)
-               ndelay(ndl);
+               ndelay((ndl * 859) / 1000); // 5 : 2^32/1000000000 (4.295)
 }
 
 static void ref_rcu_read_section(const int nloops)