Skip to contentSkip to conversation

perf test stat+shadow_stat.sh: Add threshold for rounding errors

PATCH Applied [email protected] 3 participants 5 messages Sep 19, 2023 – Sep 26, 2023 perf events

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 perf test stat+shadow_stat.sh: Add threshold for rounding errors

From: Veronika Molnarova <[email protected]>

The test was failing in specific scenarios due to imperfection of FP arithmetics. The bc command wasn't correctly rounding the result of division causing the failure.

Replace the bc with awk which should work with more decimal places and add a threshold to catch any possible rounding errors. The acceptable rounding error is set to 0.01 when the test passes with a warning message.

---
Resending the patches with sign-off as it didn't go through last time, also tried to fix the found issues.

 tools/perf/tests/shell/stat+shadow_stat.sh | 30 +++++++++++++++++-----
 1 file changed, 24 insertions(+), 6 deletions(-)
tools/perf/tests/shell/stat+shadow_stat.sh +24 -6
index 0e9cba84e75..6be3b4f80e5 100755
@@ -4,6 +4,8 @@
 
 set -e
 
+THRESHOLD=0.015
+
 # skip if system-wide mode is forbidden
 perf stat -a true > /dev/null 2>&1 || exit 2
 
@@ -33,10 +35,18 @@ test_global_aggr()
 		fi
 
 		# use printf for rounding and a leading zero
-		res=`printf "%.2f" "$(echo "scale=6; $num / $cyc" | bc -q)"`
+		res=`echo $num $cyc | awk '{printf "%.2f", $1 / $2}'`
 		if [ "$ipc" != "$res" ]; then
-			echo "IPC is different: $res != $ipc  ($num / $cyc)"
-			exit 1
+			# check the difference from the real result for FP imperfections
+			diff=`echo $ipc $res $THRESHOLD | \
+			awk '{x = ($1 - $2) < 0 ? ($2 - $1) : ($1 - $2); print (x > $3)}'`
+
+			if [ $diff -eq 1 ]; then
+				echo "IPC is different: $res != $ipc  ($num / $cyc)"
+				exit 1
+			fi
+
+			echo "Warning: Difference of IPC is under the threshold"
 		fi
 	done
 }
@@ -67,10 +77,18 @@ test_no_aggr()
 		fi
 
 		# use printf for rounding and a leading zero
-		res=`printf "%.2f" "$(echo "scale=6; $num / $cyc" | bc -q)"`
+		res=`echo $num $cyc | awk '{printf "%.2f", $1 / $2}'`
 		if [ "$ipc" != "$res" ]; then
-			echo "IPC is different for $cpu: $res != $ipc  ($num / $cyc)"
-			exit 1
+			# check difference from the real result for FP imperfections
+			diff=`echo $ipc $res $THRESHOLD | \
+			awk '{x = ($1 - $2) < 0 ? ($2 - $1) : ($1 - $2); print (x > $3)}'`
+
+			if [ $diff -eq 1 ]; then
+				echo "IPC is different: $res != $ipc  ($num / $cyc)"
+				exit 1
+			fi
+
+			echo "Warning: Difference of IPC is under the threshold"
 		fi
 	done
 }
Signature · 1 line
2.41.0
#1 lore raw

PATCH 2/2 perf test lock_contention.sh: Skip test if not enough CPUs

From: Veronika Molnarova <[email protected]>

Machines with less then 4 CPUs weren't consistently triggering lock events required for the test.

Skip the test on those machines. The limit of 4 CPUs is set as it generates around 100 lock events for a test.

---
Resending the patches with sign-off as it didn't go through last time, also tried to fix the found issues.

 tools/perf/tests/shell/lock_contention.sh | 6 ++++++
 1 file changed, 6 insertions(+)
tools/perf/tests/shell/lock_contention.sh +6 -0
index f2cc187b618..768c10f1d6b 100755
@@ -32,6 +32,12 @@ check() {
 		err=2
 		exit
 	fi
+
+	if [ `nproc` -lt 4 ]; then
+		echo "[Skip] Low number of CPUs (`nproc`), lock event cannot be triggered certainly"
+		err=2
+		exit
+	fi
 }
 
 test_record()
Signature · 1 line
2.41.0
Acked-by on v1 2/2
#2 lore raw

Re: [PATCH 2/2] perf test lock_contention.sh: Skip test if not enough CPUs

Quoted 9 lines · On Tue, 19 Sep 2023, [email protected] wrote:

From: Veronika Molnarova <[email protected]>

Machines with less then 4 CPUs weren't consistently triggering lock events required for the test.

Skip the test on those machines. The limit of 4 CPUs is set as it generates around 100 lock events for a test.

Quoted 28 lines

---
Resending the patches with sign-off as it didn't go through last time, also tried to fix the found issues.

 tools/perf/tests/shell/lock_contention.sh | 6 ++++++
 1 file changed, 6 insertions(+)
tools/perf/tests/shell/lock_contention.sh +6 -0
index f2cc187b618..768c10f1d6b 100755
@@ -32,6 +32,12 @@ check() {
 		err=2
 		exit
 	fi
+
+	if [ `nproc` -lt 4 ]; then
+		echo "[Skip] Low number of CPUs (`nproc`), lock event cannot be triggered certainly"
+		err=2
+		exit
+	fi
 }
 
 test_record()
Signature · 1 line
2.41.0
Acked-by on v1 1/2
#3 lore raw
In reply to [email protected]
Quoted 6 lines · On Tue, 19 Sep 2023, [email protected] wrote:

From: Veronika Molnarova <[email protected]>

The test was failing in specific scenarios due to imperfection of FP arithmetics. The bc command wasn't correctly rounding the result of division causing the failure.

Quoted 6 lines

Replace the bc with awk which should work with more decimal places and add a threshold to catch any possible rounding errors. The acceptable rounding error is set to 0.01 when the test passes with a warning message.

Quoted 68 lines

---
Resending the patches with sign-off as it didn't go through last time, also tried to fix the found issues.

 tools/perf/tests/shell/stat+shadow_stat.sh | 30 +++++++++++++++++-----
 1 file changed, 24 insertions(+), 6 deletions(-)
tools/perf/tests/shell/stat+shadow_stat.sh +24 -6
index 0e9cba84e75..6be3b4f80e5 100755
@@ -4,6 +4,8 @@
 
 set -e
 
+THRESHOLD=0.015
+
 # skip if system-wide mode is forbidden
 perf stat -a true > /dev/null 2>&1 || exit 2
 
@@ -33,10 +35,18 @@ test_global_aggr()
 		fi
 
 		# use printf for rounding and a leading zero
-		res=`printf "%.2f" "$(echo "scale=6; $num / $cyc" | bc -q)"`
+		res=`echo $num $cyc | awk '{printf "%.2f", $1 / $2}'`
 		if [ "$ipc" != "$res" ]; then
-			echo "IPC is different: $res != $ipc  ($num / $cyc)"
-			exit 1
+			# check the difference from the real result for FP imperfections
+			diff=`echo $ipc $res $THRESHOLD | \
+			awk '{x = ($1 - $2) < 0 ? ($2 - $1) : ($1 - $2); print (x > $3)}'`
+
+			if [ $diff -eq 1 ]; then
+				echo "IPC is different: $res != $ipc  ($num / $cyc)"
+				exit 1
+			fi
+
+			echo "Warning: Difference of IPC is under the threshold"
 		fi
 	done
 }
@@ -67,10 +77,18 @@ test_no_aggr()
 		fi
 
 		# use printf for rounding and a leading zero
-		res=`printf "%.2f" "$(echo "scale=6; $num / $cyc" | bc -q)"`
+		res=`echo $num $cyc | awk '{printf "%.2f", $1 / $2}'`
 		if [ "$ipc" != "$res" ]; then
-			echo "IPC is different for $cpu: $res != $ipc  ($num / $cyc)"
-			exit 1
+			# check difference from the real result for FP imperfections
+			diff=`echo $ipc $res $THRESHOLD | \
+			awk '{x = ($1 - $2) < 0 ? ($2 - $1) : ($1 - $2); print (x > $3)}'`
+
+			if [ $diff -eq 1 ]; then
+				echo "IPC is different: $res != $ipc  ($num / $cyc)"
+				exit 1
+			fi
+
+			echo "Warning: Difference of IPC is under the threshold"
 		fi
 	done
 }
Signature · 1 line
2.41.0
Applied
#4 lore raw
Quoted 8 lines · Michael Petlan
Quoted 6 lines · On Tue, 19 Sep 2023, [email protected] wrote:

From: Veronika Molnarova <[email protected]>

The test was failing in specific scenarios due to imperfection of FP arithmetics. The bc command wasn't correctly rounding the result of division causing the failure.

Quoted 8 lines
Quoted 6 lines

Replace the bc with awk which should work with more decimal places and add a threshold to catch any possible rounding errors. The acceptable rounding error is set to 0.01 when the test passes with a warning message.

Applied both to perf-tools-next, thanks!