perf test: Fix errors when deleting data in perf record tests
Namhyung Kim
maintainerPATCH 1/3 perf test: Fix errors when deleting data in perf record tests
perf record with multithreads would create a data directory instead of a file. But the cleanup and retry logic expect a file and try to delete them with 'rm -f' which would fail. Add '-r' flag to remove directories.
rm: cannot remove '/tmp/__perf_test.perf.data.ZVHTw.old': Is a directory
--- tools/perf/tests/shell/lib/perf_record.sh | 2 +- tools/perf/tests/shell/record.sh | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
tools/perf/tests/shell/lib/perf_record.sh +1 -1
@@ -30,7 +30,7 @@ perf_record_with_retry() { fi for duration in 0.01 0.1 0.3 1.0 2.0; do - rm -f "${perfdata}".old + rm -rf "${perfdata}".old ${cmd_prefix} "$@" -o "${perfdata}" ${testprog_base} ${duration} > "$logfile" 2>&1 local record_exit=$?
tools/perf/tests/shell/record.sh +2 -2
@@ -39,8 +39,8 @@ default_fd_limit=$(ulimit -Sn) min_fd_limit=$(($(getconf _NPROCESSORS_ONLN) * 16)) cleanup() { - rm -f "${perfdata}" - rm -f "${perfdata}".old + rm -rf "${perfdata}" + rm -rf "${perfdata}".old rm -f "${script_output}" perf_record_cleanup
Signature · 1 line
2.56.0.rc1.310.g51773c2048-goog
Namhyung Kim
maintainerPATCH 2/3 perf test: Fix record tests on Intel Broadwell
I noticed it failed for leader sampling tests on Broadwell. It's because the period of groups members are a little bit different. Maybe there's an hardware issue but anyway it's close enough to treat them as same. I printed the value like below.
$ perf test record ... prev: 462507, curr: 462506 prev: 494783, curr: 494782 prev: 454350, curr: 454349 prev: 502782, curr: 502781 prev: 566693, curr: 566692 prev: 606558, curr: 606557 prev: 559770, curr: 559769 prev: 595737, curr: 595736 prev: 643386, curr: 643385 prev: 663672, curr: 663671 prev: 620622, curr: 620621 prev: 650419, curr: 650418 Leader sampling [Failed inconsistent cycles count]
Let's update the logic to compare values to allow 1% of errors.
Also I think it should compare only if sees the non-leader samples so split the check of (($index % 2)).
--- tools/perf/tests/shell/record.sh | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
tools/perf/tests/shell/record.sh +7 -4
@@ -550,11 +550,14 @@ test_leader_sampling() { while IFS= read -r line do cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}') - if [ $(($index%2)) -ne 0 ] && [ ${cycles}x != ${prev_cycles}x ] + if [ $(($index%2)) -ne 0 ] then - invalid_counts=$(($invalid_counts+1)) - else - valid_counts=$(($valid_counts+1)) + if (( $(bc <<< "scale=4; r = ${cycles} / ${prev_cycles}; r >= 0.99 && r <= 1.01") )) + then + valid_counts=$(($valid_counts+1)) + else + invalid_counts=$(($invalid_counts+1)) + fi fi index=$(($index+1)) prev_cycles=$cycles
Signature · 1 line
2.56.0.rc1.310.g51773c2048-goog
[email protected]
AI reviewRe: [PATCH 2/3] perf test: Fix record tests on Intel Broadwell
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 21 lines
I noticed it failed for leader sampling tests on Broadwell. It's because the period of groups members are a little bit different. Maybe there's an hardware issue but anyway it's close enough to treat them as same. I printed the value like below.
$ perf test record ... prev: 462507, curr: 462506 prev: 494783, curr: 494782 prev: 454350, curr: 454349 prev: 502782, curr: 502781 prev: 566693, curr: 566692 prev: 606558, curr: 606557 prev: 559770, curr: 559769 prev: 595737, curr: 595736 prev: 643386, curr: 643385 prev: 663672, curr: 663671 prev: 620622, curr: 620621 prev: 650419, curr: 650418 Leader sampling [Failed inconsistent cycles count]
Quoted 6 lines
Let's update the logic to compare values to allow 1% of errors.
Also I think it should compare only if sees the non-leader samples so split the check of (($index % 2)).
Signed-off-by Namhyung Kim <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Re: [PATCH 2/3] perf test: Fix record tests on Intel Broadwell
In reply to Namhyung KimQuoted 22 lines · Namhyung Kim
I noticed it failed for leader sampling tests on Broadwell. It's because the period of groups members are a little bit different. Maybe there's an hardware issue but anyway it's close enough to treat them as same. I printed the value like below.
$ perf test record ... prev: 462507, curr: 462506 prev: 494783, curr: 494782 prev: 454350, curr: 454349 prev: 502782, curr: 502781 prev: 566693, curr: 566692 prev: 606558, curr: 606557 prev: 559770, curr: 559769 prev: 595737, curr: 595736 prev: 643386, curr: 643385 prev: 663672, curr: 663671 prev: 620622, curr: 620621 prev: 650419, curr: 650418 Leader sampling [Failed inconsistent cycles count]
Quoted 6 lines
Let's update the logic to compare values to allow 1% of errors.
Also I think it should compare only if sees the non-leader samples so split the check of (($index % 2)).
Signed-off-by Namhyung Kim <[email protected]>
+Dapeng Mi
Thanks,
Ian
Quoted 30 lines
--- tools/perf/tests/shell/record.sh | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)tools/perf/tests/shell/record.sh +7
-5@@ -550,11 +550,14 @@ test_leader_sampling() { while IFS= read -r line do cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}') - if [ $(($index%2)) -ne 0 ] && [ ${cycles}x != ${prev_cycles}x ] + if [ $(($index%2)) -ne 0 ] then - invalid_counts=$(($invalid_counts+1)) - else - valid_counts=$(($valid_counts+1)) + if (( $(bc <<< "scale=4; r = ${cycles} / ${prev_cycles}; r >= 0.99 && r <= 1.01") )) + then + valid_counts=$(($valid_counts+1)) + else + invalid_counts=$(($invalid_counts+1)) + fi fi index=$(($index+1)) prev_cycles=$cycles --2.56.0.rc1.310.g51773c2048-goog
Mi, Dapeng
Reviewed-byRe: [PATCH 2/3] perf test: Fix record tests on Intel Broadwell
Quoted 6 lines · Ian Rogers
Quoted 5 lines · Namhyung Kim
I noticed it failed for leader sampling tests on Broadwell. It's because the period of groups members are a little bit different. Maybe there's an hardware issue but anyway it's close enough to treat them as same. I printed the value like below.
Quoted 15 lines
Quoted 7 lines
$ perf test record ... prev: 462507, curr: 462506 prev: 494783, curr: 494782 prev: 454350, curr: 454349 prev: 502782, curr: 502781 prev: 566693, curr: 566692Quoted 8 lines
prev: 606558, curr: 606557 prev: 559770, curr: 559769 prev: 595737, curr: 595736 prev: 643386, curr: 643385 prev: 663672, curr: 663671 prev: 620622, curr: 620621 prev: 650419, curr: 650418 Leader sampling [Failed inconsistent cycles count]
It looks a hardware overcount issue. I don't see this issue on recent platforms, like SPR+, so I suppose the issue got fixed.
BTW, recently I found the SAMPLE_READ may report invalid counts in some PEBS based corner sampling cases, like ACR events + SAMPLE_READ, or multiple PEBS events simultaneously run SAMPLE_READ, I'm working on a patch series to fix the issues.
Thanks.
Quoted 43 lines
Quoted 6 lines
Let's update the logic to compare values to allow 1% of errors.
Also I think it should compare only if sees the non-leader samples so split the check of (($index % 2)).
Signed-off-by Namhyung Kim <[email protected]>+Dapeng Mi
Tested-by Ian Rogers <[email protected]>Thanks,
IanQuoted 30 lines
--- tools/perf/tests/shell/record.sh | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)tools/perf/tests/shell/record.sh +7
-5@@ -550,11 +550,14 @@ test_leader_sampling() { while IFS= read -r line do cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}') - if [ $(($index%2)) -ne 0 ] && [ ${cycles}x != ${prev_cycles}x ] + if [ $(($index%2)) -ne 0 ] then - invalid_counts=$(($invalid_counts+1)) - else - valid_counts=$(($valid_counts+1)) + if (( $(bc <<< "scale=4; r = ${cycles} / ${prev_cycles}; r >= 0.99 && r <= 1.01") )) + then + valid_counts=$(($valid_counts+1)) + else + invalid_counts=$(($invalid_counts+1)) + fi fi index=$(($index+1)) prev_cycles=$cycles --2.56.0.rc1.310.g51773c2048-goog
Namhyung Kim
maintainerPATCH 3/3 perf test: Fix record tests on hybrid machines
In reply to Namhyung KimCurrently the leader sampling test looks for "cycles:" string to match the event and retrieve the value before the event name. But it won't work on hybrid systems as the event name is different like below.
$ perf script | grep -m 10 brstack
perf 3322152 1873980.978431: 625379 cpu_atom/cycles/: 558200cc40cf brstack_bench+0x31 (/home/namhyung/project/linux/tools/perf/perf)
perf 3322152 1873980.978431: 625379 cpu_atom/cycles/: 558200cc40cf brstack_bench+0x31 (/home/namhyung/project/linux/tools/perf/perf)
perf 3322152 1873980.978598: 612177 cpu_atom/cycles/: 558200cc4082 brstack_foo+0x17 (/home/namhyung/project/linux/tools/perf/perf)
perf 3322152 1873980.978598: 612177 cpu_atom/cycles/: 558200cc4082 brstack_foo+0x17 (/home/namhyung/project/linux/tools/perf/perf)
perf 3322152 1873980.978762: 591859 cpu_atom/cycles/: 558200cc403d brstack_bar+0x0 (/home/namhyung/project/linux/tools/perf/perf)
perf 3322152 1873980.978762: 591859 cpu_atom/cycles/: 558200cc403d brstack_bar+0x0 (/home/namhyung/project/linux/tools/perf/perf)
perf 3322152 1873980.978948: 628505 cpu_atom/cycles/: 558200cc403e brstack_bar+0x1 (/home/namhyung/project/linux/tools/perf/perf)
perf 3322152 1873980.978948: 628505 cpu_atom/cycles/: 558200cc403e brstack_bar+0x1 (/home/namhyung/project/linux/tools/perf/perf)
perf 3322152 1873980.979146: 669816 cpu_atom/cycles/: 558200cc40e5 brstack_bench+0x47 (/home/namhyung/project/linux/tools/perf/perf)
perf 3322152 1873980.979146: 669816 cpu_atom/cycles/: 558200cc40e5 brstack_bench+0x47 (/home/namhyung/project/linux/tools/perf/perf)But we can control the output format of perf script using -F option. So we knows where the period is in the output. To filter by symbol name, "ip" and "sym" fields are required as well.
$ perf script -F period,ip,sym | grep -m 10 brstack
625379 558200cc40cf brstack_bench
625379 558200cc40cf brstack_bench
612177 558200cc4082 brstack_foo
612177 558200cc4082 brstack_foo
591859 558200cc403d brstack_bar
591859 558200cc403d brstack_bar
628505 558200cc403e brstack_bar
628505 558200cc403e brstack_bar
669816 558200cc40e5 brstack_bench
669816 558200cc40e5 brstack_bench--- tools/perf/tests/shell/record.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/tests/shell/record.sh +2 -2
@@ -537,7 +537,7 @@ test_leader_sampling() { err=1 return fi - perf script -i "${perfdata}" | grep brstack > $script_output + perf script -i "${perfdata}" -F period,ip,sym | grep brstack > $script_output # Check if the two instruction counts are equal in each record. # However, the throttling code doesn't consider event grouping. During throttling, only the # leader is stopped, causing the slave's counts significantly higher. To temporarily solve this,
@@ -549,7 +549,7 @@ test_leader_sampling() { tolerance_rate=0.8 while IFS= read -r line do - cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}') + cycles=$(echo $line | awk '{ print $1 }') if [ $(($index%2)) -ne 0 ] then if (( $(bc <<< "scale=4; r = ${cycles} / ${prev_cycles}; r >= 0.99 && r <= 1.01") ))
Signature · 1 line
2.56.0.rc1.310.g51773c2048-goog
[email protected]
AI reviewRe: [PATCH 3/3] perf test: Fix record tests on hybrid machines
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 20 lines
Currently the leader sampling test looks for "cycles:" string to match the event and retrieve the value before the event name. But it won't work on hybrid systems as the event name is different like below.
$ perf script | grep -m 10 brstack perf 3322152 1873980.978431: 625379 cpu_atom/cycles/: 558200cc40cf brstack_bench+0x31 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978431: 625379 cpu_atom/cycles/: 558200cc40cf brstack_bench+0x31 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978598: 612177 cpu_atom/cycles/: 558200cc4082 brstack_foo+0x17 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978598: 612177 cpu_atom/cycles/: 558200cc4082 brstack_foo+0x17 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978762: 591859 cpu_atom/cycles/: 558200cc403d brstack_bar+0x0 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978762: 591859 cpu_atom/cycles/: 558200cc403d brstack_bar+0x0 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978948: 628505 cpu_atom/cycles/: 558200cc403e brstack_bar+0x1 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978948: 628505 cpu_atom/cycles/: 558200cc403e brstack_bar+0x1 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.979146: 669816 cpu_atom/cycles/: 558200cc40e5 brstack_bench+0x47 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.979146: 669816 cpu_atom/cycles/: 558200cc40e5 brstack_bench+0x47 (/home/namhyung/project/linux/tools/perf/perf) [ ... ] 628505 558200cc403e brstack_bar 669816 558200cc40e5 brstack_bench 669816 558200cc40e5 brstack_bench
Quoted 1 line
Signed-off-by Namhyung Kim <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Re: [PATCH 3/3] perf test: Fix record tests on hybrid machines
In reply to Namhyung KimQuoted 33 lines · Namhyung Kim
Currently the leader sampling test looks for "cycles:" string to match the event and retrieve the value before the event name. But it won't work on hybrid systems as the event name is different like below.
$ perf script | grep -m 10 brstack perf 3322152 1873980.978431: 625379 cpu_atom/cycles/: 558200cc40cf brstack_bench+0x31 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978431: 625379 cpu_atom/cycles/: 558200cc40cf brstack_bench+0x31 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978598: 612177 cpu_atom/cycles/: 558200cc4082 brstack_foo+0x17 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978598: 612177 cpu_atom/cycles/: 558200cc4082 brstack_foo+0x17 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978762: 591859 cpu_atom/cycles/: 558200cc403d brstack_bar+0x0 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978762: 591859 cpu_atom/cycles/: 558200cc403d brstack_bar+0x0 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978948: 628505 cpu_atom/cycles/: 558200cc403e brstack_bar+0x1 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.978948: 628505 cpu_atom/cycles/: 558200cc403e brstack_bar+0x1 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.979146: 669816 cpu_atom/cycles/: 558200cc40e5 brstack_bench+0x47 (/home/namhyung/project/linux/tools/perf/perf) perf 3322152 1873980.979146: 669816 cpu_atom/cycles/: 558200cc40e5 brstack_bench+0x47 (/home/namhyung/project/linux/tools/perf/perf)But we can control the output format of perf script using -F option. So we knows where the period is in the output. To filter by symbol name, "ip" and "sym" fields are required as well.
$ perf script -F period,ip,sym | grep -m 10 brstack 625379 558200cc40cf brstack_bench 625379 558200cc40cf brstack_bench 612177 558200cc4082 brstack_foo 612177 558200cc4082 brstack_foo 591859 558200cc403d brstack_bar 591859 558200cc403d brstack_bar 628505 558200cc403e brstack_bar 628505 558200cc403e brstack_bar 669816 558200cc40e5 brstack_bench 669816 558200cc40e5 brstack_bench
Quoted 1 line
Signed-off-by Namhyung Kim <[email protected]>
Thanks,
Ian
Quoted 29 lines
--- tools/perf/tests/shell/record.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)tools/perf/tests/shell/record.sh +2
-3@@ -537,7 +537,7 @@ test_leader_sampling() { err=1 return fi - perf script -i "${perfdata}" | grep brstack > $script_output + perf script -i "${perfdata}" -F period,ip,sym | grep brstack > $script_output # Check if the two instruction counts are equal in each record. # However, the throttling code doesn't consider event grouping. During throttling, only the # leader is stopped, causing the slave's counts significantly higher. To temporarily solve this,@@ -549,7 +549,7 @@ test_leader_sampling() { tolerance_rate=0.8 while IFS= read -r line do - cycles=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="cycles:") print $(i-1)}') + cycles=$(echo $line | awk '{ print $1 }') if [ $(($index%2)) -ne 0 ] then if (( $(bc <<< "scale=4; r = ${cycles} / ${prev_cycles}; r >= 0.99 && r <= 1.01") )) --2.56.0.rc1.310.g51773c2048-goog
[email protected]
AI reviewSashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines
perf record with multithreads would create a data directory instead of a file. But the cleanup and retry logic expect a file and try to delete them with 'rm -f' which would fail. Add '-r' flag to remove directories.
rm: cannot remove '
/tmp/__perf_test.perf.data.ZVHTw.old': Is a directorySigned-off-by Namhyung Kim <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 8 lines · Namhyung Kim
perf record with multithreads would create a data directory instead of a file. But the cleanup and retry logic expect a file and try to delete them with 'rm -f' which would fail. Add '-r' flag to remove directories.
rm: cannot remove '
/tmp/__perf_test.perf.data.ZVHTw.old': Is a directorySigned-off-by Namhyung Kim <[email protected]>
Thanks,
Ian
Quoted 36 lines
--- tools/perf/tests/shell/lib/perf_record.sh | 2 +- tools/perf/tests/shell/record.sh | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)tools/perf/tests/shell/lib/perf_record.sh +1
-1@@ -30,7 +30,7 @@ perf_record_with_retry() { fi for duration in 0.01 0.1 0.3 1.0 2.0; do - rm -f "${perfdata}".old + rm -rf "${perfdata}".old ${cmd_prefix} "$@" -o "${perfdata}" ${testprog_base} ${duration} > "$logfile" 2>&1 local record_exit=$?tools/perf/tests/shell/record.sh +2
-3@@ -39,8 +39,8 @@ default_fd_limit=$(ulimit -Sn) min_fd_limit=$(($(getconf _NPROCESSORS_ONLN) * 16)) cleanup() { - rm -f "${perfdata}" - rm -f "${perfdata}".old + rm -rf "${perfdata}" + rm -rf "${perfdata}".old rm -f "${script_output}" perf_record_cleanup --2.56.0.rc1.310.g51773c2048-goog
Quoted 5 lines · Namhyung Kim
perf record with multithreads would create a data directory instead of a file. But the cleanup and retry logic expect a file and try to delete them with 'rm -f' which would fail. Add '-r' flag to remove directories.
rm: cannot remove '
/tmp/__perf_test.perf.data.ZVHTw.old': Is a directory
Thanks, applied the series to perf-tools-next, for v7.4.
- Arnaldo
Quoted 36 lines
Signed-off-by Namhyung Kim <[email protected]>--- tools/perf/tests/shell/lib/perf_record.sh | 2 +- tools/perf/tests/shell/record.sh | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)tools/perf/tests/shell/lib/perf_record.sh +1
-1@@ -30,7 +30,7 @@ perf_record_with_retry() { fi for duration in 0.01 0.1 0.3 1.0 2.0; do - rm -f "${perfdata}".old + rm -rf "${perfdata}".old ${cmd_prefix} "$@" -o "${perfdata}" ${testprog_base} ${duration} > "$logfile" 2>&1 local record_exit=$?tools/perf/tests/shell/record.sh +2
-2@@ -39,8 +39,8 @@ default_fd_limit=$(ulimit -Sn) min_fd_limit=$(($(getconf _NPROCESSORS_ONLN) * 16)) cleanup() { - rm -f "${perfdata}" - rm -f "${perfdata}".old + rm -rf "${perfdata}" + rm -rf "${perfdata}".old rm -f "${script_output}" perf_record_cleanupSignature · 1 line
2.56.0.rc1.310.g51773c2048-goog