perf test: Allow tolerance for leader sampling test
Subsystems (2)
PATCH v3 1/1 perf test: Allow tolerance for leader sampling test
V3: Added check for missing samples as suggested by Chun-Tse.
V2: Changed bc invocation to return 0 on success and 1 on error.
There is a known issue that the leader sampling is inconsistent, since throttle only affect leader, not the slave. The detail is in [1]. To maintain test coverage, this patch sets a tolerance rate of 80% to accommodate the throttled samples and prevent test failures due to throttling.
[1] lore.kernel.org/[email protected]
--- tools/perf/tests/shell/record.sh | 33 ++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-)
tools/perf/tests/shell/record.sh +27 -6
@@ -238,22 +238,43 @@ test_leader_sampling() { err=1 return fi + perf script -i "${perfdata}" | grep brstack > $script_output + # Check if the two instruction counts are equal in each record. + # However, the throttling code doesn't consider event grouping. During throttling, only the + # leader is stopped, causing the slave's counts significantly higher. To temporarily solve this, + # let's set the tolerance rate to 80%. + # TODO: Revert the code for tolerance once the throttling mechanism is fixed. index=0 - perf script -i "${perfdata}" > $script_output + valid_counts=0 + invalid_counts=0 + tolerance_rate=0.8 while IFS= read -r line do - # Check if the two instruction counts are equal in each record cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}') if [ $(($index%2)) -ne 0 ] && [ ${cycles}x != ${prev_cycles}x ] then - echo "Leader sampling [Failed inconsistent cycles count]" - err=1 - return + invalid_counts=$(($invalid_counts+1)) + else + valid_counts=$(($valid_counts+1)) fi index=$(($index+1)) prev_cycles=$cycles done < $script_output - echo "Basic leader sampling test [Success]" + total_counts=$(bc <<< "$invalid_counts+$valid_counts") + if (( $(bc <<< "$total_counts <= 0") )) + then + echo "Leader sampling [No sample generated]" + err=1 + return + fi + isok=$(bc <<< "scale=2; if (($invalid_counts/$total_counts) < (1-$tolerance_rate)) { 0 } else { 1 };") + if [ $isok -eq 1 ] + then + echo "Leader sampling [Failed inconsistent cycles count]" + err=1 + else + echo "Basic leader sampling test [Success]" + fi } test_topdown_leader_sampling() {
Signature · 1 line
2.49.0
Quoted 10 lines · Thomas Richter
V3: Added check for missing samples as suggested by Chun-Tse.
V2: Changed bc invocation to return 0 on success and 1 on error.There is a known issue that the leader sampling is inconsistent, since throttle only affect leader, not the slave. The detail is in [1]. To maintain test coverage, this patch sets a tolerance rate of 80% to accommodate the throttled samples and prevent test failures due to throttling.
Quoted 7 lines
[1] lore.kernel.org/[email protected]
Signed-off-by Chun-Tse Shao <[email protected]>Suggested-by Ian Rogers <[email protected]>Suggested-by Thomas Richter <[email protected]>Tested-by Thomas Richter <[email protected]>Signed-off-by Thomas Richter <[email protected]>
Quoted 61 lines
--- tools/perf/tests/shell/record.sh | 33 ++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-)tools/perf/tests/shell/record.sh +27
-7@@ -238,22 +238,43 @@ test_leader_sampling() { err=1 return fi + perf script -i "${perfdata}" | grep brstack > $script_output + # Check if the two instruction counts are equal in each record. + # However, the throttling code doesn't consider event grouping. During throttling, only the + # leader is stopped, causing the slave's counts significantly higher. To temporarily solve this, + # let's set the tolerance rate to 80%. + # TODO: Revert the code for tolerance once the throttling mechanism is fixed. index=0 - perf script -i "${perfdata}" > $script_output + valid_counts=0 + invalid_counts=0 + tolerance_rate=0.8 while IFS= read -r line do - # Check if the two instruction counts are equal in each record cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}') if [ $(($index%2)) -ne 0 ] && [ ${cycles}x != ${prev_cycles}x ] then - echo "Leader sampling [Failed inconsistent cycles count]" - err=1 - return + invalid_counts=$(($invalid_counts+1)) + else + valid_counts=$(($valid_counts+1)) fi index=$(($index+1)) prev_cycles=$cycles done < $script_output - echo "Basic leader sampling test [Success]" + total_counts=$(bc <<< "$invalid_counts+$valid_counts") + if (( $(bc <<< "$total_counts <= 0") )) + then + echo "Leader sampling [No sample generated]" + err=1 + return + fi + isok=$(bc <<< "scale=2; if (($invalid_counts/$total_counts) < (1-$tolerance_rate)) { 0 } else { 1 };") + if [ $isok -eq 1 ] + then + echo "Leader sampling [Failed inconsistent cycles count]" + err=1 + else + echo "Basic leader sampling test [Success]" + fi } test_topdown_leader_sampling() { --2.49.0
Namhyung Kim
maintainerHello,
Quoted 54 lines · Thomas Richter
V3: Added check for missing samples as suggested by Chun-Tse.
V2: Changed bc invocation to return 0 on success and 1 on error.There is a known issue that the leader sampling is inconsistent, since throttle only affect leader, not the slave. The detail is in [1]. To maintain test coverage, this patch sets a tolerance rate of 80% to accommodate the throttled samples and prevent test failures due to throttling.
[1] lore.kernel.org/[email protected]
Signed-off-by Chun-Tse Shao <[email protected]>Suggested-by Ian Rogers <[email protected]>Suggested-by Thomas Richter <[email protected]>Tested-by Thomas Richter <[email protected]>Signed-off-by Thomas Richter <[email protected]>--- tools/perf/tests/shell/record.sh | 33 ++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-)tools/perf/tests/shell/record.sh +12
-5@@ -238,22 +238,43 @@ test_leader_sampling() { err=1 return fi + perf script -i "${perfdata}" | grep brstack > $script_output + # Check if the two instruction counts are equal in each record. + # However, the throttling code doesn't consider event grouping. During throttling, only the + # leader is stopped, causing the slave's counts significantly higher. To temporarily solve this, + # let's set the tolerance rate to 80%. + # TODO: Revert the code for tolerance once the throttling mechanism is fixed. index=0 - perf script -i "${perfdata}" > $script_output + valid_counts=0 + invalid_counts=0 + tolerance_rate=0.8 while IFS= read -r line do - # Check if the two instruction counts are equal in each record cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}') if [ $(($index%2)) -ne 0 ] && [ ${cycles}x != ${prev_cycles}x ] then - echo "Leader sampling [Failed inconsistent cycles count]" - err=1 - return + invalid_counts=$(($invalid_counts+1)) + else + valid_counts=$(($valid_counts+1)) fi index=$(($index+1)) prev_cycles=$cycles
tools/perf/tests/shell/record.sh test_leader_sampling() { · line 270
Quoted 10 lines
tools/perf/tests/shell/record.sh +8
-1done < $script_output - echo "Basic leader sampling test [Success]" + total_counts=$(bc <<< "$invalid_counts+$valid_counts") + if (( $(bc <<< "$total_counts <= 0") )) + then + echo "Leader sampling [No sample generated]" + err=1 + return + fi + isok=$(bc <<< "scale=2; if (($invalid_counts/$total_counts) < (1-$tolerance_rate)) { 0 } else { 1 };")
Is 'scale=2' really needed? Does something similar to the above like
if (( $(bc <<< "($invalid_counts / $total_counts) < (1 - $tolerance_rate)") ))
work?
Thanks,
Namhyung
Quoted 13 lines
tools/perf/tests/shell/record.sh +7
-0+ if [ $isok -eq 1 ] + then + echo "Leader sampling [Failed inconsistent cycles count]" + err=1 + else + echo "Basic leader sampling test [Success]" + fi } test_topdown_leader_sampling() {Signature · 1 line
2.49.0
Quoted 70 lines · Namhyung Kim
Hello,
Quoted 54 lines · Thomas Richter
V3: Added check for missing samples as suggested by Chun-Tse.
V2: Changed bc invocation to return 0 on success and 1 on error.There is a known issue that the leader sampling is inconsistent, since throttle only affect leader, not the slave. The detail is in [1]. To maintain test coverage, this patch sets a tolerance rate of 80% to accommodate the throttled samples and prevent test failures due to throttling.
[1] lore.kernel.org/[email protected]
Signed-off-by Chun-Tse Shao <[email protected]>Suggested-by Ian Rogers <[email protected]>Suggested-by Thomas Richter <[email protected]>Tested-by Thomas Richter <[email protected]>Signed-off-by Thomas Richter <[email protected]>--- tools/perf/tests/shell/record.sh | 33 ++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-)tools/perf/tests/shell/record.sh +12
-5@@ -238,22 +238,43 @@ test_leader_sampling() { err=1 return fi + perf script -i "${perfdata}" | grep brstack > $script_output + # Check if the two instruction counts are equal in each record. + # However, the throttling code doesn't consider event grouping. During throttling, only the + # leader is stopped, causing the slave's counts significantly higher. To temporarily solve this, + # let's set the tolerance rate to 80%. + # TODO: Revert the code for tolerance once the throttling mechanism is fixed. index=0 - perf script -i "${perfdata}" > $script_output + valid_counts=0 + invalid_counts=0 + tolerance_rate=0.8 while IFS= read -r line do - # Check if the two instruction counts are equal in each record cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}') if [ $(($index%2)) -ne 0 ] && [ ${cycles}x != ${prev_cycles}x ] then - echo "Leader sampling [Failed inconsistent cycles count]" - err=1 - return + invalid_counts=$(($invalid_counts+1)) + else + valid_counts=$(($valid_counts+1)) fi index=$(($index+1)) prev_cycles=$cyclestools/perf/tests/shell/record.sh test_leader_sampling() { · line 270
Quoted 10 lines
tools/perf/tests/shell/record.sh +8
-1done < $script_output - echo "Basic leader sampling test [Success]" + total_counts=$(bc <<< "$invalid_counts+$valid_counts") + if (( $(bc <<< "$total_counts <= 0") )) + then + echo "Leader sampling [No sample generated]" + err=1 + return + fi + isok=$(bc <<< "scale=2; if (($invalid_counts/$total_counts) < (1-$tolerance_rate)) { 0 } else { 1 };")Is 'scale=2' really needed? Does something similar to the above like
Quoted 8 lines
if (( $(bc <<< "($invalid_counts / $total_counts) < (1 - $tolerance_rate)") ))work?
Thanks,
Namhyung
From the man page of bc:
NUMBERS
The most basic element in bc is the number. Numbers are arbitrary precision numbers. This
precision is both in the integer part and the fractional part. All numbers are represented internally in decimal and all computation is done in decimal. (This version truncates re‐ sults from divide and multiply operations.) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This can be proved with:
# bc <<< "2/27" 0 # bc <<< "scale=2;2/27" .07 #
Without scale there is no fractional part and integer arithmetic will lead to wrong results.
I think scale=2 is necessary or we need to use something different like awk.
Thanks
Signature · 9 lines
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany -- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Wolfgang Wendt Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
Namhyung Kim
maintainerQuoted 97 lines · Thomas Richter
Quoted 70 lines · Namhyung Kim
Hello,
Quoted 54 lines · Thomas Richter
V3: Added check for missing samples as suggested by Chun-Tse.
V2: Changed bc invocation to return 0 on success and 1 on error.There is a known issue that the leader sampling is inconsistent, since throttle only affect leader, not the slave. The detail is in [1]. To maintain test coverage, this patch sets a tolerance rate of 80% to accommodate the throttled samples and prevent test failures due to throttling.
[1] lore.kernel.org/[email protected]
Signed-off-by Chun-Tse Shao <[email protected]>Suggested-by Ian Rogers <[email protected]>Suggested-by Thomas Richter <[email protected]>Tested-by Thomas Richter <[email protected]>Signed-off-by Thomas Richter <[email protected]>--- tools/perf/tests/shell/record.sh | 33 ++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-)tools/perf/tests/shell/record.sh +12
-5@@ -238,22 +238,43 @@ test_leader_sampling() { err=1 return fi + perf script -i "${perfdata}" | grep brstack > $script_output + # Check if the two instruction counts are equal in each record. + # However, the throttling code doesn't consider event grouping. During throttling, only the + # leader is stopped, causing the slave's counts significantly higher. To temporarily solve this, + # let's set the tolerance rate to 80%. + # TODO: Revert the code for tolerance once the throttling mechanism is fixed. index=0 - perf script -i "${perfdata}" > $script_output + valid_counts=0 + invalid_counts=0 + tolerance_rate=0.8 while IFS= read -r line do - # Check if the two instruction counts are equal in each record cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}') if [ $(($index%2)) -ne 0 ] && [ ${cycles}x != ${prev_cycles}x ] then - echo "Leader sampling [Failed inconsistent cycles count]" - err=1 - return + invalid_counts=$(($invalid_counts+1)) + else + valid_counts=$(($valid_counts+1)) fi index=$(($index+1)) prev_cycles=$cyclestools/perf/tests/shell/record.sh test_leader_sampling() { · line 270
Quoted 10 lines
tools/perf/tests/shell/record.sh +8
-1done < $script_output - echo "Basic leader sampling test [Success]" + total_counts=$(bc <<< "$invalid_counts+$valid_counts") + if (( $(bc <<< "$total_counts <= 0") )) + then + echo "Leader sampling [No sample generated]" + err=1 + return + fi + isok=$(bc <<< "scale=2; if (($invalid_counts/$total_counts) < (1-$tolerance_rate)) { 0 } else { 1 };")Is 'scale=2' really needed? Does something similar to the above like
Quoted 8 lines
if (( $(bc <<< "($invalid_counts / $total_counts) < (1 - $tolerance_rate)") ))work?
Thanks,
NamhyungFrom the man page of bc:
NUMBERS
The most basic element in bc is the number. Numbers are arbitrary precision numbers. This
precision is both in the integer part and the fractional part. All numbers are represented internally in decimal and all computation is done in decimal. (This version truncates re‐ sults from divide and multiply operations.) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^This can be proved with:
# bc <<< "2/27" 0 # bc <<< "scale=2;2/27" .07 #
Quoted 3 lines
Without scale there is no fractional part and integer arithmetic will lead to wrong results.
I think scale=2 is necessary or we need to use something different like awk.
Ok, thanks for checking it. Right, the scale=2 is necessary.
$ bc <<< '(1 / 10) < (1 - 0.8)' 1
$ bc <<< '(3 / 10) < (1 - 0.8)' 1
$ bc <<< 'scale=2; (1 / 10) < (1 - 0.8)' 1
$ bc <<< 'scale=2; (3 / 10) < (1 - 0.8)' 0
Thanks,
Namhyung