perf test: test_intel_pt.sh: Add per-thread test
Adrian Hunter
reviewercover perf test: test_intel_pt.sh: Add per-thread test
Hi
Here is a new per-thread test for the test_intel_pt.sh test script.
The first 9 patches are tidy-ups for the script, mostly based on results from the shellcheck utility.
The 10th patch adds debug prints that the script will capture to help verify correct operation.
The final patch actually adds the new test.
Adrian Hunter (11):
perf test: test_intel_pt.sh: Add cleanup function
perf test: test_intel_pt.sh: Use a temp directory
perf test: test_intel_pt.sh: Fix redirection
perf test: test_intel_pt.sh: Stop using expr
perf test: test_intel_pt.sh: Stop using backticks
perf test: test_intel_pt.sh: Use grep -c instead of grep plus wc -l perf test: test_intel_pt.sh: Use quotes around variable expansion perf test: test_intel_pt.sh: Fix return checking
perf test: test_intel_pt.sh: Add more output in preparation for more tests perf tools: Add debug messages and comments for testing
perf test: test_intel_pt.sh: Add per-thread test
tools/lib/perf/evlist.c | 2 + tools/perf/builtin-record.c | 8 + tools/perf/tests/shell/test_intel_pt.sh | 307 ++++++++++++++++++++++++++++++-- tools/perf/util/evsel.c | 2 + 4 files changed, 304 insertions(+), 15 deletions(-)
Regards
Adrian
Adrian Hunter
reviewerPATCH 1/11 perf test: test_intel_pt.sh: Add cleanup function
Add a cleanup function that will still clean up if the script is terminated prematurely.
--- tools/perf/tests/shell/test_intel_pt.sh | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)
tools/perf/tests/shell/test_intel_pt.sh +16 -2
@@ -14,6 +14,21 @@ err_cnt=0 tmpfile=`mktemp` perfdatafile=`mktemp` +cleanup() +{ + trap - EXIT TERM INT + rm -f ${tmpfile} + rm -f ${perfdatafile} +} + +trap_cleanup() +{ + cleanup + exit 1 +} + +trap trap_cleanup EXIT TERM INT + can_cpu_wide() { perf record -o ${tmpfile} -B -N --no-bpf-event -e dummy:u -C $1 true 2>&1 >/dev/null || return 2
@@ -57,8 +72,7 @@ test_system_wide_side_band count_result $? -rm -f ${tmpfile} -rm -f ${perfdatafile} +cleanup if [ ${err_cnt} -gt 0 ] ; then exit 1
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 2/11 perf test: test_intel_pt.sh: Use a temp directory
In reply to Adrian HunterCreate a directory for temporary files so that mktemp needs to be used only once. It also enables more temp files to be added without having to add them also to the cleanup.
--- tools/perf/tests/shell/test_intel_pt.sh | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
tools/perf/tests/shell/test_intel_pt.sh +10 -4
@@ -11,14 +11,20 @@ skip_cnt=0 ok_cnt=0 err_cnt=0 -tmpfile=`mktemp` -perfdatafile=`mktemp` +temp_dir=$(mktemp -d /tmp/perf-test-intel-pt-sh.XXXXXXXXXX) + +tmpfile="${temp_dir}/tmp-perf.data" +perfdatafile="${temp_dir}/test-perf.data" cleanup() { trap - EXIT TERM INT - rm -f ${tmpfile} - rm -f ${perfdatafile} + sane=$(echo "${temp_dir}" | cut -b 1-26) + if [ "${sane}" = "/tmp/perf-test-intel-pt-sh" ] ; then + echo "--- Cleaning up ---" + rm -f "${temp_dir}/"* + rmdir "${temp_dir}" + fi } trap_cleanup()
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 3/11 perf test: test_intel_pt.sh: Fix redirection
In reply to Adrian HunterAs reported by shellcheck, 2>&1 must come after >/dev/null
--- tools/perf/tests/shell/test_intel_pt.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/test_intel_pt.sh +1 -1
@@ -37,7 +37,7 @@ trap trap_cleanup EXIT TERM INT can_cpu_wide() { - perf record -o ${tmpfile} -B -N --no-bpf-event -e dummy:u -C $1 true 2>&1 >/dev/null || return 2 + perf record -o ${tmpfile} -B -N --no-bpf-event -e dummy:u -C $1 true >/dev/null 2>&1 || return 2 return 0 }
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 4/11 perf test: test_intel_pt.sh: Stop using expr
In reply to Adrian HunterAs suggested by shellcheck, stop using expr.
--- tools/perf/tests/shell/test_intel_pt.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/tests/shell/test_intel_pt.sh +3 -3
@@ -64,14 +64,14 @@ test_system_wide_side_band() count_result() { if [ $1 -eq 2 ] ; then - skip_cnt=`expr ${skip_cnt} \+ 1` + skip_cnt=$((skip_cnt + 1)) return fi if [ $1 -eq 0 ] ; then - ok_cnt=`expr ${ok_cnt} \+ 1` + ok_cnt=$((ok_cnt + 1)) return fi - err_cnt=`expr ${err_cnt} \+ 1` + err_cnt=$((err_cnt + 1)) } test_system_wide_side_band
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 5/11 perf test: test_intel_pt.sh: Stop using backticks
In reply to Adrian HunterAs suggested by shellcheck, stop using backticks.
--- tools/perf/tests/shell/test_intel_pt.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/test_intel_pt.sh +1 -1
@@ -51,7 +51,7 @@ test_system_wide_side_band() perf record -B -N --no-bpf-event -o ${perfdatafile} -e intel_pt//u -C 0 -- taskset --cpu-list 1 uname # Should get MMAP events from CPU 1 because they can be needed to decode - mmap_cnt=`perf script -i ${perfdatafile} --no-itrace --show-mmap-events -C 1 2>/dev/null | grep MMAP | wc -l` + mmap_cnt=$(perf script -i ${perfdatafile} --no-itrace --show-mmap-events -C 1 2>/dev/null | grep MMAP | wc -l) if [ ${mmap_cnt} -gt 0 ] ; then return 0
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 6/11 perf test: test_intel_pt.sh: Use grep -c instead of grep plus wc -l
In reply to Adrian HunterAs suggested by shellcheck, use grep -c instead of grep plus wc -l
--- tools/perf/tests/shell/test_intel_pt.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/tests/shell/test_intel_pt.sh +1 -1
@@ -51,7 +51,7 @@ test_system_wide_side_band() perf record -B -N --no-bpf-event -o ${perfdatafile} -e intel_pt//u -C 0 -- taskset --cpu-list 1 uname # Should get MMAP events from CPU 1 because they can be needed to decode - mmap_cnt=$(perf script -i ${perfdatafile} --no-itrace --show-mmap-events -C 1 2>/dev/null | grep MMAP | wc -l) + mmap_cnt=$(perf script -i ${perfdatafile} --no-itrace --show-mmap-events -C 1 2>/dev/null | grep -c MMAP) if [ ${mmap_cnt} -gt 0 ] ; then return 0
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 7/11 perf test: test_intel_pt.sh: Use quotes around variable expansion
In reply to Adrian HunterAs suggested by shellcheck, use quotes around variable expansion.
--- tools/perf/tests/shell/test_intel_pt.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
tools/perf/tests/shell/test_intel_pt.sh +6 -6
@@ -37,7 +37,7 @@ trap trap_cleanup EXIT TERM INT can_cpu_wide() { - perf record -o ${tmpfile} -B -N --no-bpf-event -e dummy:u -C $1 true >/dev/null 2>&1 || return 2 + perf record -o "${tmpfile}" -B -N --no-bpf-event -e dummy:u -C "$1" true >/dev/null 2>&1 || return 2 return 0 }
@@ -48,12 +48,12 @@ test_system_wide_side_band() can_cpu_wide 1 || return $? # Record on CPU 0 a task running on CPU 1 - perf record -B -N --no-bpf-event -o ${perfdatafile} -e intel_pt//u -C 0 -- taskset --cpu-list 1 uname + perf record -B -N --no-bpf-event -o "${perfdatafile}" -e intel_pt//u -C 0 -- taskset --cpu-list 1 uname # Should get MMAP events from CPU 1 because they can be needed to decode - mmap_cnt=$(perf script -i ${perfdatafile} --no-itrace --show-mmap-events -C 1 2>/dev/null | grep -c MMAP) + mmap_cnt=$(perf script -i "${perfdatafile}" --no-itrace --show-mmap-events -C 1 2>/dev/null | grep -c MMAP) - if [ ${mmap_cnt} -gt 0 ] ; then + if [ "${mmap_cnt}" -gt 0 ] ; then return 0 fi
@@ -63,11 +63,11 @@ test_system_wide_side_band() count_result() { - if [ $1 -eq 2 ] ; then + if [ "$1" -eq 2 ] ; then skip_cnt=$((skip_cnt + 1)) return fi - if [ $1 -eq 0 ] ; then + if [ "$1" -eq 0 ] ; then ok_cnt=$((ok_cnt + 1)) return fi
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 8/11 perf test: test_intel_pt.sh: Fix return checking
In reply to Adrian HunterThe use of set -e will cause a function that returns non-zero to terminate the script unless the result is consumed by || for example. That is OK if there is only 1 test function, but not if there are more. Prepare for more by using ||.
--- tools/perf/tests/shell/test_intel_pt.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
tools/perf/tests/shell/test_intel_pt.sh +3 -3
@@ -72,11 +72,11 @@ count_result() return fi err_cnt=$((err_cnt + 1)) + ret=0 } -test_system_wide_side_band - -count_result $? +ret=0 +test_system_wide_side_band || ret=$? ; count_result $ret cleanup
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 9/11 perf test: test_intel_pt.sh: Add more output in preparation for more tests
In reply to Adrian HunterWhen there are more tests it won't be obvious which test failed. Add more output so that it is.
--- tools/perf/tests/shell/test_intel_pt.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
tools/perf/tests/shell/test_intel_pt.sh +11 -1
@@ -37,12 +37,19 @@ trap trap_cleanup EXIT TERM INT can_cpu_wide() { - perf record -o "${tmpfile}" -B -N --no-bpf-event -e dummy:u -C "$1" true >/dev/null 2>&1 || return 2 + echo "Checking for CPU-wide recording on CPU $1" + if ! perf record -o "${tmpfile}" -B -N --no-bpf-event -e dummy:u -C "$1" true >/dev/null 2>&1 ; then + echo "No so skipping" + return 2 + fi + echo OK return 0 } test_system_wide_side_band() { + echo "--- Test system-wide sideband ---" + # Need CPU 0 and CPU 1 can_cpu_wide 0 || return $? can_cpu_wide 1 || return $?
@@ -54,6 +61,7 @@ test_system_wide_side_band() mmap_cnt=$(perf script -i "${perfdatafile}" --no-itrace --show-mmap-events -C 1 2>/dev/null | grep -c MMAP) if [ "${mmap_cnt}" -gt 0 ] ; then + echo OK return 0 fi
@@ -80,6 +88,8 @@ test_system_wide_side_band || ret=$? ; count_result $ret cleanup +echo "--- Done ---" + if [ ${err_cnt} -gt 0 ] ; then exit 1 fi
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 10/11 perf tools: Add debug messages and comments for testing
In reply to Adrian HunterAdd debug messages to enable scripts to track aspects of perf record behaviour. The messages will be consumed after perf record has run, with the exception of "perf record has started" which is consequently flushed.
Put comments so developers know which messages are also being used by test scripts.
--- tools/lib/perf/evlist.c | 2 ++ tools/perf/builtin-record.c | 8 ++++++++ tools/perf/util/evsel.c | 2 ++ 3 files changed, 12 insertions(+)
tools/lib/perf/evlist.c +2 -0
@@ -486,6 +486,7 @@ mmap_per_evsel(struct perf_evlist *evlist, struct perf_evlist_mmap_ops *ops,if (ops->idx) ops->idx (evlist, evsel, mp, idx); +/* Debug message used by test scripts */ pr_debug ("idx %d: mmapping fd %d\n" , idx, *output);if (ops->mmap (map, mp, *output, evlist_cpu) <0 )return -1 ;
@@ -495,6 +496,7 @@ mmap_per_evsel(struct perf_evlist *evlist, struct perf_evlist_mmap_ops *ops,if (!idx)perf_evlist__set_mmap_first (evlist, map, overwrite); }else { +/* Debug message used by test scripts */ pr_debug ("idx %d: set output fd %d -> %d\n" , idx, fd, *output);if (ioctl (fd,PERF_EVENT_IOC_SET_OUTPUT , *output) !=0 )return -1 ;
tools/perf/builtin-record.c +8 -0
@@ -2428,10 +2428,14 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)record__uniquify_name (rec); +/* Debug message used by test scripts */ +pr_debug3 ("perf record opening and mmapping events\n" );if (record__open (rec) !=0 ) { err =-1 ; goto out_free_threads; } +/* Debug message used by test scripts */ +pr_debug3 ("perf record done opening and mmapping events\n" ); session->header.env.comp_mmap_len = session->evlist->core.mmap_len;if (rec->opts.kcore) {
@@ -2574,6 +2578,10 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)if (err) goto out_child; +/* Debug message used by test scripts */ +pr_debug3 ("perf record has started\n" ); +fflush (stderr); +trigger_ready (&auxtrace_snapshot_trigger);trigger_ready (&switch_output_trigger);perf_hooks__invoke_record_start ();
tools/perf/util/evsel.c +2 -0
@@ -2089,6 +2089,7 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,test_attr__ready (); +/* Debug message used by test scripts */ pr_debug2_peo ("sys_perf_event_open: pid %d cpu %d group_fd %d flags %#lx" , pid,perf_cpu_map__cpu (cpus, idx).cpu, group_fd, evsel->open_flags);
@@ -2114,6 +2115,7 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus, fd, group_fd, evsel->open_flags); } +/* Debug message used by test scripts */ pr_debug2_peo (" = %d\n" , fd);if (evsel->bpf_fd >=0 ) {
Signature · 1 line
2.25.1
Adrian Hunter
reviewerPATCH 11/11 perf test: test_intel_pt.sh: Add per-thread test
In reply to Adrian HunterWhen tracing the kernel with Intel PT, text_poke events are recorded per-cpu. In per-thread mode that results in a mixture of per-thread and per-cpu events and mmaps. Check that happens correctly.
The debug output from perf record -vvv is recorded and then awk used to process the debug messages that indicate what file descriptors were opened and whether they were mmapped or set-output.
--- tools/perf/tests/shell/test_intel_pt.sh | 247 ++++++++++++++++++++++++ 1 file changed, 247 insertions(+)
tools/perf/tests/shell/test_intel_pt.sh +247 -0
@@ -4,6 +4,8 @@ set -e +tenths=date\ +%s%1N + # Skip if no Intel PT perf list | grep -q 'intel_pt//' || exit 2
@@ -15,6 +17,10 @@ temp_dir=$(mktemp -d /tmp/perf-test-intel-pt-sh.XXXXXXXXXX) tmpfile="${temp_dir}/tmp-perf.data" perfdatafile="${temp_dir}/test-perf.data" +outfile="${temp_dir}/test-out.txt" +errfile="${temp_dir}/test-err.txt" +workload="${temp_dir}/workload" +awkscript="${temp_dir}/awkscript" cleanup() {
@@ -35,6 +41,37 @@ trap_cleanup() trap trap_cleanup EXIT TERM INT +have_workload=false +cat << _end_of_file_ | /usr/bin/cc -o "${workload}" -xc - -pthread && have_workload=true +#include <time.h> +#include <pthread.h> + +void work(void) { + struct timespec tm = { + .tv_nsec = 1000000, + }; + int i; + + /* Run for about 30 seconds */ + for (i = 0; i < 30000; i++) + nanosleep(&tm, NULL); +} + +void *threadfunc(void *arg) { + work(); + return NULL; +} + +int main(void) { + pthread_t th; + + pthread_create(&th, NULL, threadfunc, NULL); + work(); + pthread_join(th, NULL); + return 0; +} +_end_of_file_ + can_cpu_wide() { echo "Checking for CPU-wide recording on CPU $1"
@@ -69,6 +106,214 @@ test_system_wide_side_band() return 1 } +can_kernel() +{ + perf record -o "${tmpfile}" -B -N --no-bpf-event -e dummy:k true >/dev/null 2>&1 || return 2 + return 0 +} + +wait_for_threads() +{ + start_time=$($tenths) + while [ -e "/proc/$1/task" ] ; do + th_cnt=$(find "/proc/$1/task" -mindepth 1 -maxdepth 1 -printf x | wc -c) + if [ "${th_cnt}" -ge "$2" ] ; then + return 0 + fi + # Wait at most 5 seconds + if [ $(($($tenths) - start_time)) -ge 50 ] ; then + echo "PID $1 does not have $2 threads" + return 1 + fi + done + return 1 +} + +wait_for_perf_to_start() +{ + echo "Waiting for \"perf record has started\" message" + start_time=$($tenths) + while [ -e "/proc/$1" ] ; do + if grep -q "perf record has started" "${errfile}" ; then + echo OK + break + fi + # Wait at most 5 seconds + if [ $(($($tenths) - start_time)) -ge 50 ] ; then + echo "perf recording did not start" + return 1 + fi + done + return 0 +} + +wait_for_process_to_exit() +{ + start_time=$($tenths) + while [ -e "/proc/$1" ] ; do + # Wait at most 5 seconds + if [ $(($($tenths) - start_time)) -ge 50 ] ; then + echo "PID $1 did not exit as expected" + return 1 + fi + done + return 0 +} + +is_running() +{ + start_time=$($tenths) + while [ -e "/proc/$1" ] ; do + # Check for at least 0.3s + if [ $(($($tenths) - start_time)) -gt 3 ] ; then + return 0 + fi + done + echo "PID $1 exited prematurely" + return 1 +} + +test_per_thread() +{ + k="$1" + desc="$2" + + echo "--- Test per-thread ${desc}recording ---" + + if ! $have_workload ; then + echo "No workload, so skipping" + return 2 + fi + + if [ "${k}" = "k" ] ; then + can_kernel || return 2 + fi + + cat <<- "_end_of_file_" > "${awkscript}" + BEGIN { + s = "[ ]*" + u = s"[0-9]+"s + d = s"[0-9-]+"s + x = s"[0-9a-fA-FxX]+"s + mmapping = "idx"u": mmapping fd"u + set_output = "idx"u": set output fd"u"->"u + perf_event_open = "sys_perf_event_open: pid"d"cpu"d"group_fd"d"flags"x"="u + } + + /perf record opening and mmapping events/ { + if (!done) + active = 1 + } + + /perf record done opening and mmapping events/ { + active = 0 + done = 1 + } + + $0 ~ perf_event_open && active { + match($0, perf_event_open) + $0 = substr($0, RSTART, RLENGTH) + pid = $3 + cpu = $5 + fd = $11 + print "pid " pid " cpu " cpu " fd " fd " : " $0 + fd_array[fd] = fd + pid_array[fd] = pid + cpu_array[fd] = cpu + } + + $0 ~ mmapping && active { + match($0, mmapping) + $0 = substr($0, RSTART, RLENGTH) + fd = $5 + print "fd " fd " : " $0 + if (fd in fd_array) { + mmap_array[fd] = 1 + } else { + print "Unknown fd " fd + exit 1 + } + } + + $0 ~ set_output && active { + match($0, set_output) + $0 = substr($0, RSTART, RLENGTH) + fd = $6 + fd_to = $8 + print "fd " fd " fd_to " fd_to " : " $0 + if (fd in fd_array) { + if (fd_to in fd_array) { + set_output_array[fd] = fd_to + } else { + print "Unknown fd " fd_to + exit 1 + } + } else { + print "Unknown fd " fd + exit 1 + } + } + + END { + print "Checking " length(fd_array) " fds" + for (fd in fd_array) { + if (fd in mmap_array) { + pid = pid_array[fd] + if (pid != -1) { + if (pid in pids) { + print "More than 1 mmap for PID " pid + exit 1 + } + pids[pid] = 1 + } + cpu = cpu_array[fd] + if (cpu != -1) { + if (cpu in cpus) { + print "More than 1 mmap for CPU " cpu + exit 1 + } + cpus[cpu] = 1 + } + } else if (!(fd in set_output_array)) { + print "No mmap for fd " fd + exit 1 + } + } + n = length(pids) + if (n != thread_cnt) { + print "Expected " thread_cnt " per-thread mmaps - found " n + exit 1 + } + } + _end_of_file_ + + $workload & + w1=$! + $workload & + w2=$! + echo "Workload PIDs are $w1 and $w2" + wait_for_threads ${w1} 2 + wait_for_threads ${w2} 2 + + perf record -B -N --no-bpf-event -o "${perfdatafile}" -e intel_pt//u"${k}" -vvv --per-thread -p "${w1},${w2}" 2>"${errfile}" >"${outfile}" & + ppid=$! + echo "perf PID is $ppid" + wait_for_perf_to_start ${ppid} || return 1 + + kill ${w1} + wait_for_process_to_exit ${w1} || return 1 + is_running ${ppid} || return 1 + + kill ${w2} + wait_for_process_to_exit ${w2} || return 1 + wait_for_process_to_exit ${ppid} || return 1 + + awk -v thread_cnt=4 -f "${awkscript}" "${errfile}" || return 1 + + echo OK + return 0 +} + count_result() { if [ "$1" -eq 2 ] ; then
@@ -85,6 +330,8 @@ count_result() ret=0 test_system_wide_side_band || ret=$? ; count_result $ret +test_per_thread "" "" || ret=$? ; count_result $ret +test_per_thread "k" "(incl. kernel) " || ret=$? ; count_result $ret cleanup
Signature · 1 line
2.25.1
Namhyung Kim
maintainerQuoted 86 lines · Adrian Hunter
When tracing the kernel with Intel PT,
text_pokeevents are recorded per-cpu. In per-thread mode that results in a mixture of per-thread and per-cpu events and mmaps. Check that happens correctly.The debug output from perf record -vvv is recorded and then awk used to process the debug messages that indicate what file descriptors were opened and whether they were mmapped or set-output.
Signed-off-by Adrian Hunter <[email protected]>--- tools/perf/tests/shell/test_intel_pt.sh | 247 ++++++++++++++++++++++++ 1 file changed, 247 insertions(+)tools/perf/tests/shell/test_intel_pt.sh +43
-0@@ -4,6 +4,8 @@ set -e +tenths=date\ +%s%1N + # Skip if no Intel PT perf list | grep -q 'intel_pt//' || exit 2@@ -15,6 +17,10 @@ temp_dir=$(mktemp -d /tmp/perf-test-intel-pt-sh.XXXXXXXXXX) tmpfile="${temp_dir}/tmp-perf.data" perfdatafile="${temp_dir}/test-perf.data" +outfile="${temp_dir}/test-out.txt" +errfile="${temp_dir}/test-err.txt" +workload="${temp_dir}/workload" +awkscript="${temp_dir}/awkscript" cleanup() {@@ -35,6 +41,37 @@ trap_cleanup() trap trap_cleanup EXIT TERM INT +have_workload=false +cat << _end_of_file_ | /usr/bin/cc -o "${workload}" -xc - -pthread && have_workload=true +#include <time.h> +#include <pthread.h> + +void work(void) { + struct timespec tm = { + .tv_nsec = 1000000, + }; + int i; + + /* Run for about 30 seconds */ + for (i = 0; i < 30000; i++) + nanosleep(&tm, NULL); +} + +void *threadfunc(void *arg) { + work(); + return NULL; +} + +int main(void) { + pthread_t th; + + pthread_create(&th, NULL, threadfunc, NULL); + work(); + pthread_join(th, NULL); + return 0; +} +_end_of_file_ + can_cpu_wide() { echo "Checking for CPU-wide recording on CPU $1"@@ -69,6 +106,214 @@ test_system_wide_side_band() return 1 } +can_kernel() +{ + perf record -o "${tmpfile}" -B -N --no-bpf-event -e dummy:k true >/dev/null 2>&1 || return 2 + return 0 +} +
tools/perf/tests/shell/test_intel_pt.sh test_system_wide_side_band() · line 124
Quoted 10 lines
tools/perf/tests/shell/test_intel_pt.sh +10
-0+wait_for_threads() +{ + start_time=$($tenths) + while [ -e "/proc/$1/task" ] ; do + th_cnt=$(find "/proc/$1/task" -mindepth 1 -maxdepth 1 -printf x | wc -c) + if [ "${th_cnt}" -ge "$2" ] ; then + return 0 + fi + # Wait at most 5 seconds + if [ $(($($tenths) - start_time)) -ge 50 ] ; then
It's a shame that we would use this syntax but I couldn't find any better. :(
Quoted 40 lines
tools/perf/tests/shell/test_intel_pt.sh +40
-0+ echo "PID $1 does not have $2 threads" + return 1 + fi + done + return 1 +} + +wait_for_perf_to_start() +{ + echo "Waiting for \"perf record has started\" message" + start_time=$($tenths) + while [ -e "/proc/$1" ] ; do + if grep -q "perf record has started" "${errfile}" ; then + echo OK + break + fi + # Wait at most 5 seconds + if [ $(($($tenths) - start_time)) -ge 50 ] ; then + echo "perf recording did not start" + return 1 + fi + done + return 0 +} + +wait_for_process_to_exit() +{ + start_time=$($tenths) + while [ -e "/proc/$1" ] ; do + # Wait at most 5 seconds + if [ $(($($tenths) - start_time)) -ge 50 ] ; then + echo "PID $1 did not exit as expected" + return 1 + fi + done + return 0 +} + +is_running() +{
tools/perf/tests/shell/test_intel_pt.sh test_system_wide_side_band() · line 174
Quoted 10 lines
tools/perf/tests/shell/test_intel_pt.sh +10
-0+ start_time=$($tenths) + while [ -e "/proc/$1" ] ; do + # Check for at least 0.3s + if [ $(($($tenths) - start_time)) -gt 3 ] ; then + return 0 + fi + done + echo "PID $1 exited prematurely" + return 1 +}
Can we move this into the lib directory with an optional argument specifying the timeout?
Thanks,
Namhyung
Quoted 157 lines
tools/perf/tests/shell/test_intel_pt.sh +144
-1+ +test_per_thread() +{ + k="$1" + desc="$2" + + echo "--- Test per-thread ${desc}recording ---" + + if ! $have_workload ; then + echo "No workload, so skipping" + return 2 + fi + + if [ "${k}" = "k" ] ; then + can_kernel || return 2 + fi + + cat <<- "_end_of_file_" > "${awkscript}" + BEGIN { + s = "[ ]*" + u = s"[0-9]+"s + d = s"[0-9-]+"s + x = s"[0-9a-fA-FxX]+"s + mmapping = "idx"u": mmapping fd"u + set_output = "idx"u": set output fd"u"->"u + perf_event_open = "sys_perf_event_open: pid"d"cpu"d"group_fd"d"flags"x"="u + } + + /perf record opening and mmapping events/ { + if (!done) + active = 1 + } + + /perf record done opening and mmapping events/ { + active = 0 + done = 1 + } + + $0 ~ perf_event_open && active { + match($0, perf_event_open) + $0 = substr($0, RSTART, RLENGTH) + pid = $3 + cpu = $5 + fd = $11 + print "pid " pid " cpu " cpu " fd " fd " : " $0 + fd_array[fd] = fd + pid_array[fd] = pid + cpu_array[fd] = cpu + } + + $0 ~ mmapping && active { + match($0, mmapping) + $0 = substr($0, RSTART, RLENGTH) + fd = $5 + print "fd " fd " : " $0 + if (fd in fd_array) { + mmap_array[fd] = 1 + } else { + print "Unknown fd " fd + exit 1 + } + } + + $0 ~ set_output && active { + match($0, set_output) + $0 = substr($0, RSTART, RLENGTH) + fd = $6 + fd_to = $8 + print "fd " fd " fd_to " fd_to " : " $0 + if (fd in fd_array) { + if (fd_to in fd_array) { + set_output_array[fd] = fd_to + } else { + print "Unknown fd " fd_to + exit 1 + } + } else { + print "Unknown fd " fd + exit 1 + } + } + + END { + print "Checking " length(fd_array) " fds" + for (fd in fd_array) { + if (fd in mmap_array) { + pid = pid_array[fd] + if (pid != -1) { + if (pid in pids) { + print "More than 1 mmap for PID " pid + exit 1 + } + pids[pid] = 1 + } + cpu = cpu_array[fd] + if (cpu != -1) { + if (cpu in cpus) { + print "More than 1 mmap for CPU " cpu + exit 1 + } + cpus[cpu] = 1 + } + } else if (!(fd in set_output_array)) { + print "No mmap for fd " fd + exit 1 + } + } + n = length(pids) + if (n != thread_cnt) { + print "Expected " thread_cnt " per-thread mmaps - found " n + exit 1 + } + } + _end_of_file_ + + $workload & + w1=$! + $workload & + w2=$! + echo "Workload PIDs are $w1 and $w2" + wait_for_threads ${w1} 2 + wait_for_threads ${w2} 2 + + perf record -B -N --no-bpf-event -o "${perfdatafile}" -e intel_pt//u"${k}" -vvv --per-thread -p "${w1},${w2}" 2>"${errfile}" >"${outfile}" & + ppid=$! + echo "perf PID is $ppid" + wait_for_perf_to_start ${ppid} || return 1 + + kill ${w1} + wait_for_process_to_exit ${w1} || return 1 + is_running ${ppid} || return 1 + + kill ${w2} + wait_for_process_to_exit ${w2} || return 1 + wait_for_process_to_exit ${ppid} || return 1 + + awk -v thread_cnt=4 -f "${awkscript}" "${errfile}" || return 1 + + echo OK + return 0 +} + count_result() { if [ "$1" -eq 2 ] ; then@@ -85,6 +330,8 @@ count_result() ret=0 test_system_wide_side_band || ret=$? ; count_result $ret +test_per_thread "" "" || ret=$? ; count_result $ret +test_per_thread "k" "(incl. kernel) " || ret=$? ; count_result $ret cleanup --2.25.1
Quoted 14 lines · Adrian Hunter
Hi
Here is a new per-thread test for the
test_intel_pt.shtest script.The first 9 patches are tidy-ups for the script, mostly based on results from the shellcheck utility.
The 10th patch adds debug prints that the script will capture to help verify correct operation.
The final patch actually adds the new test.
Quoted 12 lines
Adrian Hunter (11):
perf test:test_intel_pt.sh: Add cleanup function
perf test:test_intel_pt.sh: Use a temp directory
perf test:test_intel_pt.sh: Fix redirection
perf test:test_intel_pt.sh: Stop using expr
perf test:test_intel_pt.sh: Stop using backticks
perf test:test_intel_pt.sh: Use grep -c instead of grep plus wc -l perf test:test_intel_pt.sh: Use quotes around variable expansion perf test:test_intel_pt.sh: Fix return checking
perf test:test_intel_pt.sh: Add more output in preparation for more tests perf tools: Add debug messages and comments for testing
perf test:test_intel_pt.sh: Add per-thread test
I don't think I understood all the black magic in patch 11. :) At least we can move some helper functions to the lib and reuse them in other tests. I'll test that later..
So for patch 01-10,
Thanks,
Namhyung
Quoted 10 lines
tools/lib/perf/evlist.c | 2 + tools/perf/builtin-record.c | 8 + tools/perf/tests/shell/test_intel_pt.sh | 307 ++++++++++++++++++++++++++++++-- tools/perf/util/evsel.c | 2 + 4 files changed, 304 insertions(+), 15 deletions(-)Regards
Adrian
Adrian Hunter
reviewerQuoted 28 lines · Namhyung Kim
Quoted 14 lines · Adrian Hunter
Hi
Here is a new per-thread test for the
test_intel_pt.shtest script.The first 9 patches are tidy-ups for the script, mostly based on results from the shellcheck utility.
The 10th patch adds debug prints that the script will capture to help verify correct operation.
The final patch actually adds the new test.
Quoted 12 lines
Adrian Hunter (11):
perf test:test_intel_pt.sh: Add cleanup function
perf test:test_intel_pt.sh: Use a temp directory
perf test:test_intel_pt.sh: Fix redirection
perf test:test_intel_pt.sh: Stop using expr
perf test:test_intel_pt.sh: Stop using backticks
perf test:test_intel_pt.sh: Use grep -c instead of grep plus wc -l perf test:test_intel_pt.sh: Use quotes around variable expansion perf test:test_intel_pt.sh: Fix return checking
perf test:test_intel_pt.sh: Add more output in preparation for more tests perf tools: Add debug messages and comments for testing
perf test:test_intel_pt.sh: Add per-thread test
Quoted 1 line
I don't think I understood all the black magic in patch 11. :)
It is not that bad :-)
Consider the output from the test:
$ perf test -v " intel pt" Couldn't bump rlimit(MEMLOCK), failures may take place when creating BPF maps, etc 102: Miscellaneous Intel PT testing : --- start --- test child forked, pid 155646 <SNIP> --- Test per-thread recording --- Workload PIDs are 155669 and 155670 perf PID is 155681 Waiting for "perf record has started" message OK pid 155669 cpu -1 fd 5 : sys_perf_event_open: pid 155669 cpu -1 group_fd -1 flags 0x8 = 5
awk has matched the debug message and determined the values for pid, cpu and fd
pid 155673 cpu -1 fd 6 : sys_perf_event_open: pid 155673 cpu -1 group_fd -1 flags 0x8 = 6 pid 155670 cpu -1 fd 7 : sys_perf_event_open: pid 155670 cpu -1 group_fd -1 flags 0x8 = 7 pid 155672 cpu -1 fd 8 : sys_perf_event_open: pid 155672 cpu -1 group_fd -1 flags 0x8 = 8 pid 155669 cpu -1 fd 9 : sys_perf_event_open: pid 155669 cpu -1 group_fd -1 flags 0x8 = 9 pid 155673 cpu -1 fd 10 : sys_perf_event_open: pid 155673 cpu -1 group_fd -1 flags 0x8 = 10 pid 155670 cpu -1 fd 11 : sys_perf_event_open: pid 155670 cpu -1 group_fd -1 flags 0x8 = 11 pid 155672 cpu -1 fd 12 : sys_perf_event_open: pid 155672 cpu -1 group_fd -1 flags 0x8 = 12
fd 5 : idx 0: mmapping fd 5
awk has matched the debug message and determined the values for fd
fd 9 fd_to 5 : idx 0: set output fd 9 -> 5
awk has matched the debug message and determined the values for fd and fd_to
fd 6 : idx 1: mmapping fd 6
fd 10 fd_to 6 : idx 1: set output fd 10 -> 6 fd 7 : idx 2: mmapping fd 7
fd 11 fd_to 7 : idx 2: set output fd 11 -> 7 fd 8 : idx 3: mmapping fd 8
fd 12 fd_to 8 : idx 3: set output fd 12 -> 8 Checking 8 fds
Now awk is checking:
- every fd is mmapped or set-output
- there is 1 mmap per unique pid
- there is 1 mmap per unique cpu
- there are the right number of different pids
i.e.
END {
print "Checking " length(fd_array) " fds"
for (fd in fd_array) {
if (fd in mmap_array) {
pid = pid_array[fd]
if (pid != -1) {
if (pid in pids) {
print "More than 1 mmap for PID " pid
exit 1
}
pids[pid] = 1
}
cpu = cpu_array[fd]
if (cpu != -1) {
if (cpu in cpus) {
print "More than 1 mmap for CPU " cpu
exit 1
}
cpus[cpu] = 1
}
} else if (!(fd in set_output_array)) {
print "No mmap for fd " fd
exit 1
}
}
n = length(pids)
if (n != thread_cnt) {
print "Expected " thread_cnt " per-thread mmaps - found " n
exit 1
}
}Quoted 21 lines
At least we can move some helper functions to the lib and reuse them in other tests. I'll test that later..
So for patch 01-10,
Acked-by Namhyung Kim <[email protected]>Thanks,
NamhyungQuoted 10 lines
tools/lib/perf/evlist.c | 2 + tools/perf/builtin-record.c | 8 + tools/perf/tests/shell/test_intel_pt.sh | 307 ++++++++++++++++++++++++++++++-- tools/perf/util/evsel.c | 2 + 4 files changed, 304 insertions(+), 15 deletions(-)Regards
Adrian
Em Wed, Sep 14, 2022 at 11:25:07AM +0300, Adrian Hunter escreveu:
Quoted 29 lines · Namhyung Kim
Quoted 28 lines · Adrian Hunter
Quoted 26 lines
Hi
Here is a new per-thread test for the
test_intel_pt.shtest script.The first 9 patches are tidy-ups for the script, mostly based on results from the shellcheck utility.
The 10th patch adds debug prints that the script will capture to help verify correct operation.
The final patch actually adds the new test.
Adrian Hunter (11):
perf test:test_intel_pt.sh: Add cleanup function
perf test:test_intel_pt.sh: Use a temp directory
perf test:test_intel_pt.sh: Fix redirection
perf test:test_intel_pt.sh: Stop using expr
perf test:test_intel_pt.sh: Stop using backticks
perf test:test_intel_pt.sh: Use grep -c instead of grep plus wc -l perf test:test_intel_pt.sh: Use quotes around variable expansion perf test:test_intel_pt.sh: Fix return checking
perf test:test_intel_pt.sh: Add more output in preparation for more tests perf tools: Add debug messages and comments for testing
perf test:test_intel_pt.sh: Add per-thread test
Quoted 3 lines
Quoted 1 line
I don't think I understood all the black magic in patch 11. :)
It is not that bad :-)
:-)
Thanks, applied the series.
- Arnaldo
Quoted 100 lines
Consider the output from the test:
$ perf test -v " intel pt" Couldn't bump rlimit(MEMLOCK), failures may take place when creating BPF maps, etc 102: Miscellaneous Intel PT testing : --- start --- test child forked, pid 155646 <SNIP> --- Test per-thread recording --- Workload PIDs are 155669 and 155670 perf PID is 155681 Waiting for "perf record has started" message OK pid 155669 cpu -1 fd 5 : sys_perf_event_open: pid 155669 cpu -1 group_fd -1 flags 0x8 = 5awk has matched the debug message and determined the values for pid, cpu and fd
pid 155673 cpu -1 fd 6 : sys_perf_event_open: pid 155673 cpu -1 group_fd -1 flags 0x8 = 6 pid 155670 cpu -1 fd 7 : sys_perf_event_open: pid 155670 cpu -1 group_fd -1 flags 0x8 = 7 pid 155672 cpu -1 fd 8 : sys_perf_event_open: pid 155672 cpu -1 group_fd -1 flags 0x8 = 8 pid 155669 cpu -1 fd 9 : sys_perf_event_open: pid 155669 cpu -1 group_fd -1 flags 0x8 = 9 pid 155673 cpu -1 fd 10 : sys_perf_event_open: pid 155673 cpu -1 group_fd -1 flags 0x8 = 10 pid 155670 cpu -1 fd 11 : sys_perf_event_open: pid 155670 cpu -1 group_fd -1 flags 0x8 = 11 pid 155672 cpu -1 fd 12 : sys_perf_event_open: pid 155672 cpu -1 group_fd -1 flags 0x8 = 12fd 5 : idx 0: mmapping fd 5
awk has matched the debug message and determined the values for fd
fd 9 fd_to 5 : idx 0: set output fd 9 -> 5awk has matched the debug message and determined the values for fd and
fd_tofd 6 : idx 1: mmapping fd 6
fd 10fd_to6 : idx 1: set output fd 10 -> 6 fd 7 : idx 2: mmapping fd 7
fd 11fd_to7 : idx 2: set output fd 11 -> 7 fd 8 : idx 3: mmapping fd 8
fd 12fd_to8 : idx 3: set output fd 12 -> 8 Checking 8 fdsNow awk is checking:
- every fd is mmapped or set-output
- there is 1 mmap per unique pid
- there is 1 mmap per unique cpu
- there are the right number of different pids
i.e.END { print "Checking " length(fd_array) " fds" for (fd in fd_array) { if (fd in mmap_array) { pid = pid_array[fd] if (pid != -1) { if (pid in pids) { print "More than 1 mmap for PID " pid exit 1 } pids[pid] = 1 } cpu = cpu_array[fd] if (cpu != -1) { if (cpu in cpus) { print "More than 1 mmap for CPU " cpu exit 1 } cpus[cpu] = 1 } } else if (!(fd in set_output_array)) { print "No mmap for fd " fd exit 1 } } n = length(pids) if (n != thread_cnt) { print "Expected " thread_cnt " per-thread mmaps - found " n exit 1 } }Quoted 21 lines
At least we can move some helper functions to the lib and reuse them in other tests. I'll test that later..
So for patch 01-10,
Acked-by Namhyung Kim <[email protected]>Thanks,
NamhyungQuoted 10 lines
tools/lib/perf/evlist.c | 2 + tools/perf/builtin-record.c | 8 + tools/perf/tests/shell/test_intel_pt.sh | 307 ++++++++++++++++++++++++++++++-- tools/perf/util/evsel.c | 2 + 4 files changed, 304 insertions(+), 15 deletions(-)Regards
Adrian
Signature · 1 line
- Arnaldo