perf/test: Skip leader sampling for s390
Subsystems (2)
PATCH 1/1 perf/test: Skip leader sampling for s390
In tree linux-next
the perf test case 114 'perf record tests' has a subtest named 'Basic leader sampling test' which always fails on s390. Root cause is this invocation
# perf record -vv -e '{cycles,cycles}:Su' -- perf test -w brstack ...
In the debug output the following 2 event are installed:
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 136
config 0 (PERF_COUNT_HW_CPU_CYCLES)
{ sample_period, sample_freq } 4000
sample_type IP|TID|TIME|READ|CPU|PERIOD|IDENTIFIER
read_format ID|GROUP|LOST
disabled 1
exclude_kernel 1
exclude_hv 1
freq 1
sample_id_all 1
------------------------------------------------------------
sys_perf_event_open: pid -1 cpu 0 group_fd -1 flags 0x8 = 5
------------------------------------------------------------
perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 136
config 0 (PERF_COUNT_HW_CPU_CYCLES)
sample_type IP|TID|TIME|READ|CPU|PERIOD|IDENTIFIER
read_format ID|GROUP|LOST
exclude_kernel 1
exclude_hv 1
sample_id_all 1 ------------------------------------------------------------
sys_perf_event_open: pid -1 cpu 0 group_fd 5 flags 0x8 = 6
...
The first event is the group leader and is installed as sampling event. The secound one is group member and is installed as counting event.
Namhyung Kim confirms this observation:
Quoted 8 lines
Yep, the syntax '{event1,event2}:S' is for group leader sampling which reduces the overhead of PMU interrupts. The idea is that those events are scheduled together so sampling is enabled only for the leader (usually the first) event and it reads counts from the member events using
PERF_SAMPLE_READ.So they should have the same counts if it uses the same events in a group.
However this does not work on s390. s390 has one dedicated sampling PMU which supports only one event. A different PMU is used for counting. Both run concurrently using different setups and frequencies.
On s390x a sampling event is setup using a preset trigger and a large buffer. The hardware
- writes a samples (64 bytes) into this buffer
when a given number of CPU instructions has been executed.
- and triggers an interrupt when the buffer gets full.
The trigger has just a few possible values.
On s390x the counting event cycles is used to read out the numer of CPU cycles executed.
On s390 above invocation created 2 events executed on 2 different PMU and the result are diffent values from two independently running PMUs which do not match in a consistent and reliably as on Intel:
# ./perf record -e '{cycles,cycles}:Su' -- perf test -w brstack
...
# ./perf script
perf 2799437 92568.845118: 5508000 cycles: 3ffbcb898b6 do_lookup_x+0x196
perf 2799437 92568.845119: 1377000 cycles: 3ffbcb898b6 do_lookup_x+0x196
perf 2799437 92568.845120: 4131000 cycles: 3ffbcb897e8 do_lookup_x+0xc8
perf 2799437 92568.845121: 1377000 cycles: 3ffbcb8a37c _dl_lookup_symbol
perf 2799437 92568.845122: 1377000 cycles: 3ffbcb89558 check_match+0x18
perf 2799437 92568.845123: 2754000 cycles: 3ffbcb89b2a do_lookup_x+0x40a
perf 2799437 92568.845124: 1377000 cycles: 3ffbcb89b1e do_lookup_x+0x3feAs can be seen the result match very often but not all the time make this test on s390 failing very, very often.
This patch bypasses this test on s390.
Output before:
# ./perf test 114 114: perf record tests : FAILED! #
Output after:
# ./perf test 114 114: perf record tests : Ok #
--- tools/perf/tests/shell/record.sh | 6 ++++++ 1 file changed, 6 insertions(+)
tools/perf/tests/shell/record.sh +6 -0
@@ -231,6 +231,12 @@ test_cgroup() { test_leader_sampling() { echo "Basic leader sampling test" + if [ "$(uname -m)" = s390x ] + then + echo "Leader sampling skipped" + ((skipped+=1)) + return + fi if ! perf record -o "${perfdata}" -e "{cycles,cycles}:Su" -- \ perf test -w brstack 2> /dev/null then
Signature · 1 line
2.45.2
Namhyung Kim
maintainerHello,
Quoted 91 lines · Thomas Richter
In tree linux-next
the perf test case 114 'perf record tests' has a subtest named 'Basic leader sampling test' which always fails on s390. Root cause is this invocation# perf record -vv -e '{cycles,cycles}:Su' -- perf test -w brstack...
In the debug output the following 2 event are installed:------------------------------------------------------------
perf_event_attr:type 0 (PERF_TYPE_HARDWARE) size 136 config 0 (PERF_COUNT_HW_CPU_CYCLES) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|CPU|PERIOD|IDENTIFIER read_format ID|GROUP|LOST disabled 1 exclude_kernel 1 exclude_hv 1 freq 1 sample_id_all 1 ------------------------------------------------------------ sys_perf_event_open: pid -1 cpu 0 group_fd -1 flags 0x8 = 5 ------------------------------------------------------------ perf_event_attr: type 0 (PERF_TYPE_HARDWARE) size 136 config 0 (PERF_COUNT_HW_CPU_CYCLES) sample_type IP|TID|TIME|READ|CPU|PERIOD|IDENTIFIER read_format ID|GROUP|LOST exclude_kernel 1 exclude_hv 1 sample_id_all 1------------------------------------------------------------
sys_perf_event_open: pid -1 cpu 0group_fd5 flags0x8= 6
...The first event is the group leader and is installed as sampling event. The secound one is group member and is installed as counting event.
Namhyung Kim confirms this observation:
Quoted 8 lines
Yep, the syntax '{event1,event2}:S' is for group leader sampling which reduces the overhead of PMU interrupts. The idea is that those events are scheduled together so sampling is enabled only for the leader (usually the first) event and it reads counts from the member events using
PERF_SAMPLE_READ.So they should have the same counts if it uses the same events in a group.
However this does not work on s390. s390 has one dedicated sampling PMU which supports only one event. A different PMU is used for counting. Both run concurrently using different setups and frequencies.
On s390x a sampling event is setup using a preset trigger and a large buffer. The hardware
- writes a samples (64 bytes) into this buffer
when a given number of CPU instructions has been executed.
- and triggers an interrupt when the buffer gets full.
The trigger has just a few possible values.On s390x the counting event cycles is used to read out the numer of CPU cycles executed.
On s390 above invocation created 2 events executed on 2 different PMU and the result are diffent values from two independently running PMUs which do not match in a consistent and reliably as on Intel:
# ./perf record -e '{cycles,cycles}:Su' -- perf test -w brstack ... # ./perf script perf 2799437 92568.845118: 5508000 cycles: 3ffbcb898b6 do_lookup_x+0x196 perf 2799437 92568.845119: 1377000 cycles: 3ffbcb898b6 do_lookup_x+0x196 perf 2799437 92568.845120: 4131000 cycles: 3ffbcb897e8 do_lookup_x+0xc8 perf 2799437 92568.845121: 1377000 cycles: 3ffbcb8a37c _dl_lookup_symbol perf 2799437 92568.845122: 1377000 cycles: 3ffbcb89558 check_match+0x18 perf 2799437 92568.845123: 2754000 cycles: 3ffbcb89b2a do_lookup_x+0x40a perf 2799437 92568.845124: 1377000 cycles: 3ffbcb89b1e do_lookup_x+0x3feAs can be seen the result match very often but not all the time make this test on s390 failing very, very often.
This patch bypasses this test on s390.
Output before:
# ./perf test 114 114: perf record tests : FAILED! #
Quoted 7 lines
Output after:
# ./perf test 114 114: perf record tests : Ok #Signed-off-by Thomas Richter <[email protected]>Acked-by Sumanth Korikkar <[email protected]>
Thanks for the fix. I think Ian saw the same problem on other archs too. Maybe we need to enable it on supported archs only.
Thanks,
Namhyung
Quoted 24 lines
--- tools/perf/tests/shell/record.sh | 6 ++++++ 1 file changed, 6 insertions(+)tools/perf/tests/shell/record.sh +6
-0@@ -231,6 +231,12 @@ test_cgroup() { test_leader_sampling() { echo "Basic leader sampling test" + if [ "$(uname -m)" = s390x ] + then + echo "Leader sampling skipped" + ((skipped+=1)) + return + fi if ! perf record -o "${perfdata}" -e "{cycles,cycles}:Su" -- \ perf test -w brstack 2> /dev/null thenSignature · 1 line
2.45.2
Ian Rogers
reviewerQuoted 75 lines · Namhyung Kim
Hello,
Quoted 71 lines · Thomas Richter
In tree linux-next
the perf test case 114 'perf record tests' has a subtest named 'Basic leader sampling test' which always fails on s390. Root cause is this invocation# perf record -vv -e '{cycles,cycles}:Su' -- perf test -w brstack...
In the debug output the following 2 event are installed:------------------------------------------------------------
perf_event_attr:type 0 (PERF_TYPE_HARDWARE) size 136 config 0 (PERF_COUNT_HW_CPU_CYCLES) { sample_period, sample_freq } 4000 sample_type IP|TID|TIME|READ|CPU|PERIOD|IDENTIFIER read_format ID|GROUP|LOST disabled 1 exclude_kernel 1 exclude_hv 1 freq 1 sample_id_all 1 ------------------------------------------------------------ sys_perf_event_open: pid -1 cpu 0 group_fd -1 flags 0x8 = 5 ------------------------------------------------------------ perf_event_attr: type 0 (PERF_TYPE_HARDWARE) size 136 config 0 (PERF_COUNT_HW_CPU_CYCLES) sample_type IP|TID|TIME|READ|CPU|PERIOD|IDENTIFIER read_format ID|GROUP|LOST exclude_kernel 1 exclude_hv 1 sample_id_all 1------------------------------------------------------------
sys_perf_event_open: pid -1 cpu 0group_fd5 flags0x8= 6
...The first event is the group leader and is installed as sampling event. The secound one is group member and is installed as counting event.
Namhyung Kim confirms this observation:
Quoted 8 lines
Yep, the syntax '{event1,event2}:S' is for group leader sampling which reduces the overhead of PMU interrupts. The idea is that those events are scheduled together so sampling is enabled only for the leader (usually the first) event and it reads counts from the member events using
PERF_SAMPLE_READ.So they should have the same counts if it uses the same events in a group.
However this does not work on s390. s390 has one dedicated sampling PMU which supports only one event. A different PMU is used for counting. Both run concurrently using different setups and frequencies.
On s390x a sampling event is setup using a preset trigger and a large buffer. The hardware
- writes a samples (64 bytes) into this buffer
when a given number of CPU instructions has been executed.
- and triggers an interrupt when the buffer gets full.
The trigger has just a few possible values.On s390x the counting event cycles is used to read out the numer of CPU cycles executed.
On s390 above invocation created 2 events executed on 2 different PMU and the result are diffent values from two independently running PMUs which do not match in a consistent and reliably as on Intel:
# ./perf record -e '{cycles,cycles}:Su' -- perf test -w brstack
Hi Thomas,
Thanks for reporting this! Could you try adding --count=100000 so that we're not using frequency mode and we expect the counts to look like 100,000. For example, on my x86 laptop:
```
$ perf record --count=100000 -e '{cycles,cycles}:Su' -- perf test -w brstack
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.047 MB perf.data (712 samples) ]
$ perf script
perf 635952 290271.436115: 100007 cycles:
ffffffffada00080 [unknown] ([unknown])
perf 635952 290271.436115: 100007 cycles:
ffffffffada00080 [unknown] ([unknown])
perf 635952 290271.436650: 100525 cycles:
7f86352b01b3 _dl_map_object_from_fd+0x553
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
perf 635952 290271.436650: 100525 cycles:
7f86352b01b3 _dl_map_object_from_fd+0x553
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
perf 635952 290271.437088: 99866 cycles:
7f86352cb827 strchr+0x27
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
perf 635952 290271.437088: 99866 cycles:
7f86352cb827 strchr+0x27
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
perf 635952 290271.437376: 99912 cycles:
7f86352cba74 strcmp+0x54
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
perf 635952 290271.437376: 99912 cycles:
7f86352cba74 strcmp+0x54
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
perf 635952 290271.437509: 100279 cycles:
7f86352cba3a strcmp+0x1a
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
perf 635952 290271.437509: 100279 cycles:
7f86352cba3a strcmp+0x1a
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
perf 635952 290271.437559: 99760 cycles:
7f86352bc39f _dl_check_map_versions+0x50f
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
perf 635952 290271.437559: 99760 cycles:
7f86352bc39f _dl_check_map_versions+0x50f
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
...
```I'm particularly concerned if we see the cycles count very deviant from the 100000.
Quoted 10 lines
Quoted 10 lines
... # ./perf script perf 2799437 92568.845118: 5508000 cycles: 3ffbcb898b6 do_lookup_x+0x196 perf 2799437 92568.845119: 1377000 cycles: 3ffbcb898b6 do_lookup_x+0x196 perf 2799437 92568.845120: 4131000 cycles: 3ffbcb897e8 do_lookup_x+0xc8 perf 2799437 92568.845121: 1377000 cycles: 3ffbcb8a37c _dl_lookup_symbol perf 2799437 92568.845122: 1377000 cycles: 3ffbcb89558 check_match+0x18 perf 2799437 92568.845123: 2754000 cycles: 3ffbcb89b2a do_lookup_x+0x40a perf 2799437 92568.845124: 1377000 cycles: 3ffbcb89b1e do_lookup_x+0x3fe
Quoted 2 lines
Quoted 2 lines
As can be seen the result match very often but not all the time make this test on s390 failing very, very often.
Actually this is much more deviation than I'd expect. If we use task-clock softer/timer based event I see:
```
$ perf record --count=100000 -e '{task-clock,task-clock}:Su' -- perf
test -w brstack
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.047 MB perf.data (712 samples) ]
$ perf script
perf 636643 290571.807049: 801858 task-clock:
7fdf48643439 _dl_map_object_from_fd+0x7d9
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
perf 636643 290571.807049: 804012 task-clock:
7fdf48643439 _dl_map_object_from_fd+0x7d9
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
perf 636643 290571.807549: 499833 task-clock:
7fdf4863eb9b _dl_map_object_deps+0x3eb
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
perf 636643 290571.807549: 498236 task-clock:
7fdf4863eb9b _dl_map_object_deps+0x3eb
(/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
```So the count deviates by a few hundred, but your output seems to deviate by 4 million.
So, I think the test needs to be more tolerant that should help your case. As Namhyung mentions I think there may be another bug lurking.
Thanks,
Ian
Quoted 45 lines
Quoted 7 lines
This patch bypasses this test on s390.
Output before:
# ./perf test 114 114: perf record tests : FAILED! #Quoted 7 lines
Output after:
# ./perf test 114 114: perf record tests : Ok #Signed-off-by Thomas Richter <[email protected]>Acked-by Sumanth Korikkar <[email protected]>Thanks for the fix. I think Ian saw the same problem on other archs too. Maybe we need to enable it on supported archs only.
Thanks,
NamhyungQuoted 24 lines
--- tools/perf/tests/shell/record.sh | 6 ++++++ 1 file changed, 6 insertions(+)tools/perf/tests/shell/record.sh +6
-1@@ -231,6 +231,12 @@ test_cgroup() { test_leader_sampling() { echo "Basic leader sampling test" + if [ "$(uname -m)" = s390x ] + then + echo "Leader sampling skipped" + ((skipped+=1)) + return + fi if ! perf record -o "${perfdata}" -e "{cycles,cycles}:Su" -- \ perf test -w brstack 2> /dev/null then --2.45.2
Quoted 1 line · Ian Rogers
perf record --count=100000 -e '{cycles,cycles}:Su' -- perf test -w brstack
Ian, Namhyung,
here is my output using this command:
# ./perf record --count=100000 -e '{cycles,cycles}:Su' -- perf test -w brstack
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.106 MB perf.data (1080 samples) ]
# ./perf script
perf 22194 484835.185113: 100000 cycles: 3ff9e407c8c _dl_map_object_from_fd+0xa3c (/usr/lib/ld64.so.1)
perf 22194 484835.185114: 100000 cycles: 3ff9e408940 _dl_map_object+0x110 (/usr/lib/ld64.so.1)
perf 22194 484835.185116: 400000 cycles: 3ff9e40890e _dl_map_object+0xde (/usr/lib/ld64.so.1)
perf 22194 484835.185117: 900000 cycles: 3ff9e40b572 _dl_name_match_p+0x42 (/usr/lib/ld64.so.1)
perf 22194 484835.185118: 500000 cycles: 3ff9e407c8c _dl_map_object_from_fd+0xa3c (/usr/lib/ld64.so.1)
perf 22194 484835.185119: 100000 cycles: 3ff9e40b53e _dl_name_match_p+0xe (/usr/lib/ld64.so.1)
perf 22194 484835.185120: 100000 cycles: 3ff9e40890e _dl_map_object+0xde (/usr/lib/ld64.so.1)
perf 22194 484835.185121: 100000 cycles: 3ff9e408904 _dl_map_object+0xd4 (/usr/lib/ld64.so.1)
perf 22194 484835.185122: 100000 cycles: 3ff9e40369a _dl_map_object_deps+0xbba (/usr/lib/ld64.so.1)
perf 22194 484835.185123: 100000 cycles: 3ff9e413460 _dl_check_map_versions+0x100 (/usr/lib/ld64.so.1)
perf 22194 484835.185124: 500000 cycles: 3ff9e40b53e _dl_name_match_p+0xe (/usr/lib/ld64.so.1)
perf 22194 484835.185125: 100000 cycles: 3ff9e40e7e0 _dl_relocate_object+0x550 (/usr/lib/ld64.so.1)
perf 22194 484835.185126: 200000 cycles: 3ff9e40e7e0 _dl_relocate_object+0x550 (/usr/lib/ld64.so.1)
perf 22194 484835.185127: 200000 cycles: 3ff9e409558 check_match+0x18 (/usr/lib/ld64.so.1)
perf 22194 484835.185128: 200000 cycles: 3ff9e409894 do_lookup_x+0x174 (/usr/lib/ld64.so.1)
perf 22194 484835.185129: 100000 cycles: 3ff9e409910 do_lookup_x+0x1f0 (/usr/lib/ld64.so.1)
perf 22194 484835.185130: 100000 cycles: 3ff9e409b1e do_lookup_x+0x3fe (/usr/lib/ld64.so.1)
perf 22194 484835.185131: 100000 cycles: 3ff9e409894 do_lookup_x+0x174 (/usr/lib/ld64.so.1)
perf 22194 484835.185132: 100000 cycles: 3ff9e409558 check_match+0x18 (/usr/lib/ld64.so.1)
perf 22194 484835.187445: 100000 cycles: 3ff9e409ad4 do_lookup_x+0x3b4 (/usr/lib/ld64.so.1)The difference when using counts instead of frequency is similar. Most of time the numbers are identical, but sometime they do not match.
Using task-clock as event, I have similar results. The counts vary a bit, but the numbers are pretty close. They vary by just a few hundred at the most:
# perf record --count=100000 -e '{task-clock,task-clock}:Su' -- perf test -w brstack
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.016 MB perf.data (246 samples) ]
]# ./perf script
perf 22223 485235.378380: 402070 task-clock: 3ffbed874c6 _dl_map_object_from_fd+0x276 (/usr/lib/ld64.so.1)
perf 22223 485235.378380: 404640 task-clock: 3ffbed874c6 _dl_map_object_from_fd+0x276 (/usr/lib/ld64.so.1)
perf 22223 485235.378779: 399960 task-clock: 3ffbed888de _dl_map_object+0xae (/usr/lib/ld64.so.1)
perf 22223 485235.378779: 397689 task-clock: 3ffbed888de _dl_map_object+0xae (/usr/lib/ld64.so.1)
perf 22223 485235.378879: 100055 task-clock: 3ffbed8e7e0 _dl_relocate_object+0x550 (/usr/lib/ld64.so.1)
perf 22223 485235.378879: 100100 task-clock: 3ffbed8e7e0 _dl_relocate_object+0x550 (/usr/lib/ld64.so.1)
perf 22223 485235.378979: 99981 task-clock: 3ffbed895ae check_match+0x6e (/usr/lib/ld64.so.1)
perf 22223 485235.378979: 99876 task-clock: 3ffbed895ae check_match+0x6e (/usr/lib/ld64.so.1)
perf 22223 485235.379079: 99950 task-clock: 3ffbed8974c do_lookup_x+0x2c (/usr/lib/ld64.so.1)
perf 22223 485235.379079: 99957 task-clock: 3ffbed8974c do_lookup_x+0x2c (/usr/lib/ld64.so.1)
perf 22223 485235.379179: 100051 task-clock: 3ffbed8e7f0 _dl_relocate_object+0x560 (/usr/lib/ld64.so.1)
perf 22223 485235.379179: 100004 task-clock: 3ffbed8e7f0 _dl_relocate_object+0x560 (/usr/lib/ld64.so.1)
perf 22223 485235.379279: 99933 task-clock: 3ffbed8e7ea _dl_relocate_object+0x55a (/usr/lib/ld64.so.1)
perf 22223 485235.379279: 99952 task-clock: 3ffbed8e7ea _dl_relocate_object+0x55a (/usr/lib/ld64.so.1)Thanks for your help
Signature · 9 lines
Thomas Richter, Dept 3303, IBM s390 Linux Development, Boeblingen, Germany -- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Wolfgang Wendt Geschäftsführung: David Faller Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294
We believe we know the problem, appreciate Stephan Eranian's investigation. It comes from throttling. While the sampling is too high, the generic code does not modify event scheduling. perf_event_overflow() simply returns 1, and subsequently, pmu_stop() only stops the leader event, not the slave events because the arch layer does not consider groups. Also, the event_stop() callback only operates on a single event, not the siblings.
This would impact all architectures. Perhaps we can extend the evnet_stop() callback to include a new argument to also stop the siblings. We also welcome all suggestions and open to discuss any potential solutions.
Thanks,
CT
Hi,
Thanks CT for the post. Indeed this is a long-standing bug impacting (most likely)
all architectures. The rate throttling code does not consider event grouping. It stops the sampling event in place (on x86) at the hardware level, not the generic
scheduling layer. But if the event is in a group, it may make sense to also stop all the other events in the group, i.e., stop the group. Otherwise you may get discrepancies between samples of the "slave events". Similarly, the time_running and time_enable logic is not modified during throttling.
Interested in hearing potential ways of solving this in a portable manner.
Quoted 16 lines · Chun-Tse Shao
We believe we know the problem, appreciate Stephan Eranian's investigation. It comes from throttling. While the sampling is too high, the generic code does not modify event scheduling.
perf_event_overflow()simply returns 1, and subsequently,pmu_stop()only stops the leader event, not the slave events because the arch layer does not consider groups. Also, theevent_stop()callback only operates on a single event, not the siblings.This would impact all architectures. Perhaps we can extend the
evnet_stop()callback to include a new argument to also stop the siblings. We also welcome all suggestions and open to discuss any potential solutions.Thanks,
CTCc Stephane Eranian <[email protected]>