Prefer sysfs/JSON events also when no PMU is provided
Subsystems (2)
Ian Rogers
reviewerv2 cover Prefer sysfs/JSON events also when no PMU is provided
At the RISC-V summit the topic of avoiding event data being in the RISC-V PMU kernel driver came up. There is a preference for sysfs/JSON events being the priority when no PMU is provided so that legacy events maybe supported via json. Originally Mark Rutland also expressed at LPC 2023 that doing this would resolve bugs on ARM Apple M? processors, but James Clark more recently tested this and believes the driver issues there may not have existed or have been resolved. In any case, it is inconsistent that with a PMU event names avoid legacy encodings, but when wildcarding PMUs (ie without a PMU with the event name) the legacy encodings have priority.
The patch doing this work was reverted in a v6.10 release candidate as, even though the patch was posted for weeks and had been on linux-next for weeks without issue, Linus was in the habit of using explicit legacy events with unsupported precision options on his Neoverse-N1. This machine has SLC PMU events for bus and CPU cycles where ARM decided to call the events bus_cycles and cycles, the latter being also a legacy event name. ARM haven't renamed the cycles event to a more consistent cpu_cycles and avoided the problem. With these changes the problematic event will now be skipped, a large warning produced, and perf record will continue for the other PMU events. This solution was proposed by Arnaldo.
Two minor changes have been added to help with the error message and to work around issues occurring with "perf stat metrics (shadow stat) test".
The patches have only been tested on my x86 non-hybrid laptop.
v2: Rebase and add tested-by tags from James Clark, Leo Yan and Atish
Patra who have tested on RISC-V and ARM CPUs, including the problem case from before.
Ian Rogers (4):
perf evsel: Add pmu_name helper
perf stat: Fix find_stat for mixed legacy/non-legacy events perf record: Skip don't fail for events that don't open perf parse-events: Reapply "Prefer sysfs/JSON hardware events over
legacy"
tools/perf/builtin-record.c | 22 +++++++--- tools/perf/util/evsel.c | 10 +++++ tools/perf/util/evsel.h | 1 + tools/perf/util/parse-events.c | 26 +++++++++--- tools/perf/util/parse-events.l | 76 +++++++++++++++++----------------- tools/perf/util/parse-events.y | 60 ++++++++++++++++++--------- tools/perf/util/pmus.c | 20 +++++++-- tools/perf/util/stat-shadow.c | 3 +- 8 files changed, 145 insertions(+), 73 deletions(-)
Signature · 1 line
2.47.0.277.g8800431eea-goog
Ian Rogers
reviewerPATCH v2 1/4 perf evsel: Add pmu_name helper
Add helper to get the name of the evsel's PMU. This handles the case where there's no sysfs PMU via parse_events event_type helper.
--- tools/perf/util/evsel.c | 10 ++++++++++ tools/perf/util/evsel.h | 1 + 2 files changed, 11 insertions(+)
tools/perf/util/evsel.c +10 -0
@@ -237,6 +237,16 @@ int evsel__object_config(size_t object_size, int (*init)(struct evsel *evsel),return 0 ; } +const char *evsel__pmu_name (const struct evsel *evsel) +{ +struct perf_pmu *pmu =evsel__find_pmu (evsel); + +if (pmu) +return pmu->name; + +return event_type (evsel->core.attr.type); +} +#define FD (e, x, y) (*(int *)xyarray__entry(e->core.fd, x, y))int __evsel__sample_size (u64 sample_type)
tools/perf/util/evsel.h +1 -0
@@ -232,6 +232,7 @@ int evsel__object_config(size_t object_size, void (*fini)(struct evsel *evsel));struct perf_pmu *evsel__find_pmu (const struct evsel *evsel); +const char *evsel__pmu_name (const struct evsel *evsel);bool evsel__is_aux_event (const struct evsel *evsel);struct evsel *evsel__new_idx (struct perf_event_attr *attr,int idx);
Signature · 1 line
2.47.0.277.g8800431eea-goog
Ian Rogers
reviewerPATCH v2 2/4 perf stat: Fix find_stat for mixed legacy/non-legacy events
In reply to Ian RogersLegacy events typically don't have a PMU when added leading to mismatched legacy/non-legacy cases in find_stat. Use evsel__find_pmu to make sure the evsel PMU is looked up. Update the evsel__find_pmu code to look for the PMU using the extended config type or, for legacy hardware/hw_cache events on non-hybrid systems, just use the core PMU.
Before:
```
$ perf stat -e cycles,cpu/instructions/ -a sleep 1
Performance counter stats for 'system wide':
215,309,764 cycles
44,326,491 cpu/instructions/
1.002555314 seconds time elapsed
```
After:
```
$ perf stat -e cycles,cpu/instructions/ -a sleep 1
Performance counter stats for 'system wide':
990,676,332 cycles
1,235,762,487 cpu/instructions/ # 1.25 insn per cycle
1.002667198 seconds time elapsed
```calculation on the hybrid")
--- tools/perf/util/pmus.c | 20 +++++++++++++++++--- tools/perf/util/stat-shadow.c | 3 ++- 2 files changed, 19 insertions(+), 4 deletions(-)
tools/perf/util/pmus.c +17 -3
@@ -710,11 +710,25 @@ char *perf_pmus__default_pmu_name(void)struct perf_pmu *evsel__find_pmu (const struct evsel *evsel) {struct perf_pmu *pmu = evsel->pmu; +bool legacy_core_type; -if (!pmu) { - pmu =perf_pmus__find_by_type (evsel->core.attr.type); - ((struct evsel *)evsel)->pmu = pmu; +if (pmu) +return pmu; + + pmu =perf_pmus__find_by_type (evsel->core.attr.type); + legacy_core_type = + evsel->core.attr.type ==PERF_TYPE_HARDWARE || + evsel->core.attr.type ==PERF_TYPE_HW_CACHE ; +if (!pmu && legacy_core_type) { +if (perf_pmus__supports_extended_type ()) { +u32 type = evsel->core.attr.config >>PERF_PMU_TYPE_SHIFT ; + + pmu =perf_pmus__find_by_type (type); + }else { + pmu =perf_pmus__find_core_pmu (); + } } + ((struct evsel *)evsel)->pmu = pmu;return pmu; }
tools/perf/util/stat-shadow.c +2 -1
@@ -151,6 +151,7 @@ static double find_stat(const struct evsel *evsel, int aggr_idx, enum stat_type {struct evsel *cur;int evsel_ctx =evsel_context (evsel); +struct perf_pmu *evsel_pmu =evsel__find_pmu (evsel);evlist__for_each_entry (evsel->evlist, cur) {struct perf_stat_aggr *aggr;
@@ -177,7 +178,7 @@ static double find_stat(const struct evsel *evsel, int aggr_idx, enum stat_type* Except the SW CLOCK events, * ignore if not the PMU we're looking for. */ -if ((type !=STAT_NSECS ) && (evsel->pmu != cur->pmu)) +if ((type !=STAT_NSECS ) && (evsel_pmu !=evsel__find_pmu (cur)))continue ; aggr = &cur->stats->aggr[aggr_idx];
Signature · 1 line
2.47.0.277.g8800431eea-goog
Ian Rogers
reviewerPATCH v2 3/4 perf record: Skip don't fail for events that don't open
In reply to Ian RogersWhilst for many tools it is an expected behavior that failure to open a perf event is a failure, ARM decided to name PMU events the same as legacy events and then failed to rename such events on a server uncore SLC PMU. As perf's default behavior when no PMU is specified is to open the event on all PMUs that advertise/"have" the event, this yielded failures when trying to make the priority of legacy and sysfs/json events uniform - something requested by RISC-V and ARM. A legacy event user on ARM hardware may find their event opened on an uncore PMU which for perf record will fail. Arnaldo suggested skipping such events which this patch implements. Rather than have the skipping conditional on running on ARM, the skipping is done on all architectures as such a fundamental behavioral difference could lead to problems with tools built/depending on perf.
An example of perf record failing to open events on x86 is:
```
$ perf record -e data_read,cycles,LLC-prefetch-read -a sleep 0.1
Error:
Failure to open event 'data_read' on PMU 'uncore_imc_free_running_0' which will be removed.
The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (data_read).
"dmesg | grep -i perf" may provide additional information.
Error:
Failure to open event 'data_read' on PMU 'uncore_imc_free_running_1' which will be removed.
The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (data_read).
"dmesg | grep -i perf" may provide additional information.
Error:
Failure to open event 'LLC-prefetch-read' on PMU 'cpu' which will be removed.
The LLC-prefetch-read event is not supported.
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 2.188 MB perf.data (87 samples) ]
$ perf report --stats
Aggregated stats:
TOTAL events: 17255
MMAP events: 284 ( 1.6%)
COMM events: 1961 (11.4%)
EXIT events: 1 ( 0.0%)
FORK events: 1960 (11.4%)
SAMPLE events: 87 ( 0.5%)
MMAP2 events: 12836 (74.4%)
KSYMBOL events: 83 ( 0.5%)
BPF_EVENT events: 36 ( 0.2%)
FINISHED_ROUND events: 2 ( 0.0%)
ID_INDEX events: 1 ( 0.0%)
THREAD_MAP events: 1 ( 0.0%)
CPU_MAP events: 1 ( 0.0%)
TIME_CONV events: 1 ( 0.0%)
FINISHED_INIT events: 1 ( 0.0%)
cycles stats:
SAMPLE events: 87
```Note, if all events fail to open then the data file will contain no samples. This is deliberate as at the point the events are opened there are other events, such as the dummy event for sideband data, and these events will succeed in opening even if the user specified ones don't. Having a mix of open and broken events leads to a problem of identifying different sources of events.
The issue with legacy events is that on RISC-V they want the driver to not have mappings from legacy to non-legacy config encodings for each vendor/model due to size, complexity and difficulty to update. It was reported that on ARM Apple-M? CPUs the legacy mapping in the driver was broken and the sysfs/json events should always take precedent, however, it isn't clear this is still the case. It is the case that without working around this issue a legacy event like cycles without a PMU can encode differently than when specified with a PMU - the non-PMU version favoring legacy encodings, the PMU one avoiding legacy encodings.
The patch removes events and then adjusts the idx value for each evsel. This is done so that the dense xyarrays used for file descriptors, etc. don't contain broken entries. As event opening happens relatively late in the record process, use of the idx value before the open will have become corrupted, so it is expected there are latent bugs hidden behind this change - the change is best effort. As the only vendor that has broken event names is ARM, this will principally effect ARM users. They will also experience warning messages like those above because of the uncore PMU advertising legacy event names.
--- tools/perf/builtin-record.c | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)
tools/perf/builtin-record.c +17 -5
@@ -1364,6 +1364,7 @@ static int record__open(struct record *rec)struct perf_session *session = rec->session;struct record_opts *opts = &rec->opts;int rc =0 ; +bool skipped = false;evlist__for_each_entry (evlist, pos) { try_again:
@@ -1379,15 +1380,26 @@ static int record__open(struct record *rec) pos =evlist__reset_weak_group (evlist, pos, true); goto try_again; } - rc = -errno;evsel__open_strerror (pos, &opts->target, errno, msg,sizeof (msg)); -ui__error ("%s\n" , msg); - goto out; +ui__error ("Failure to open event '%s' on PMU '%s' which will be removed.\n%s\n" , +evsel__name (pos),evsel__pmu_name (pos), msg); + pos->skippable = true; + skipped = true; + }else { + pos->supported = true; } - - pos->supported = true; } +if (skipped) { +struct evsel *tmp; +int idx =0 ; + +evlist__for_each_entry_safe (evlist, tmp, pos) { +if (pos->skippable) +evlist__remove (evlist, pos); + pos->core.idx = idx++; + } + }if (symbol_conf.kptr_restrict && !evlist__exclude_kernel (evlist)) {pr_warning ( "WARNING: Kernel address maps (/proc/{kallsyms,modules}) are restricted,\n"
Signature · 1 line
2.47.0.277.g8800431eea-goog
Ian Rogers
reviewerPATCH v2 4/4 perf parse-events: Reapply "Prefer sysfs/JSON hardware events over legacy"
In reply to Ian RogersOriginally posted and merged from:
https://lore.kernel.org/r/[email protected]
This reverts commit 4f1b067359ac8364cdb7f9fda41085fa85789d0f although the patch is now smaller due to related fixes being applied in commit 22a4db3c3603 ("perf evsel: Add alternate_hw_config and use in evsel__match").
The original commit message was:
It was requested that RISC-V be able to add events to the perf tool so the PMU driver didn't need to map legacy events to config encodings:
https://lore.kernel.org/lkml/[email protected]/
This change makes the priority of events specified without a PMU the same as those specified with a PMU, namely sysfs and JSON events are checked first before using the legacy encoding.
The hw_term is made more generic as a hardware_event that encodes a pair of string and int value, allowing parse_events_multi_pmu_add to fall back on a known encoding when the sysfs/JSON adding fails for core events. As this covers PE_VALUE_SYM_HW, that token is removed and related code simplified.
--- tools/perf/util/parse-events.c | 26 +++++++++--- tools/perf/util/parse-events.l | 76 +++++++++++++++++----------------- tools/perf/util/parse-events.y | 60 ++++++++++++++++++--------- 3 files changed, 98 insertions(+), 64 deletions(-)
tools/perf/util/parse-events.c +20 -6
@@ -1549,8 +1549,8 @@ int parse_events_multi_pmu_add(struct parse_events_state *parse_state,struct list_head *list =NULL ;struct perf_pmu *pmu =NULL ;YYLTYPE *loc = loc_; -int ok =0 ; -const char *config; +int ok =0 , core_ok =0 ; +const char *tmp;struct parse_events_terms parsed_terms; *listp =NULL ;
@@ -1563,15 +1563,15 @@ int parse_events_multi_pmu_add(struct parse_events_state *parse_state,return ret; } - config =strdup (event_name); -if (!config) + tmp =strdup (event_name); +if (!tmp) goto out_err;if (parse_events_term__num (&term,PARSE_EVENTS__TERM_TYPE_USER , - config,/*num=*/ 1 ,/*novalue=*/ true, + tmp,/*num=*/ 1 ,/*novalue=*/ true, loc,/*loc_val=*/ NULL ) <0 ) { -zfree (&config); +zfree (&tmp); goto out_err; }list_add_tail (&term->list, &parsed_terms.terms);
@@ -1602,6 +1602,8 @@ int parse_events_multi_pmu_add(struct parse_events_state *parse_state,pr_debug ("%s -> %s/%s/\n" , event_name, pmu->name, sb.buf);strbuf_release (&sb); ok++; +if (pmu->is_core) + core_ok++; } }
@@ -1618,6 +1620,18 @@ int parse_events_multi_pmu_add(struct parse_events_state *parse_state, } } +if (hw_config !=PERF_COUNT_HW_MAX && !core_ok) { +/* +* The event wasn't found on core PMUs but it has a hardware +* config version to try. +*/ +if (!parse_events_add_numeric (parse_state, list, +PERF_TYPE_HARDWARE , hw_config, + const_parsed_terms, +/*wildcard=*/ true)) + ok++; + } + out_err:parse_events_terms__exit (&parsed_terms);if (ok)
tools/perf/util/parse-events.l +38 -38
107 lines not on this page · raw message
tools/perf/util/parse-events.y +40 -20
130 lines not on this page · raw message
Signature · 1 line
2.47.0.277.g8800431eea-goog
Quoted 27 lines · Ian Rogers
At the RISC-V summit the topic of avoiding event data being in the RISC-V PMU kernel driver came up. There is a preference for sysfs/JSON events being the priority when no PMU is provided so that legacy events maybe supported via json. Originally Mark Rutland also expressed at LPC 2023 that doing this would resolve bugs on ARM Apple M? processors, but James Clark more recently tested this and believes the driver issues there may not have existed or have been resolved. In any case, it is inconsistent that with a PMU event names avoid legacy encodings, but when wildcarding PMUs (ie without a PMU with the event name) the legacy encodings have priority.
The patch doing this work was reverted in a v6.10 release candidate as, even though the patch was posted for weeks and had been on linux-next for weeks without issue, Linus was in the habit of using explicit legacy events with unsupported precision options on his Neoverse-N1. This machine has SLC PMU events for bus and CPU cycles where ARM decided to call the events
bus_cyclesand cycles, the latter being also a legacy event name. ARM haven't renamed the cycles event to a more consistentcpu_cyclesand avoided the problem. With these changes the problematic event will now be skipped, a large warning produced, and perf record will continue for the other PMU events. This solution was proposed by Arnaldo.
Quoted 5 lines
Two minor changes have been added to help with the error message and to work around issues occurring with "perf stat metrics (shadow stat) test".
The patches have only been tested on my x86 non-hybrid laptop.
Hi,
After applying this patch series,we observed a regression while running the perf test suite on powerpc system. Specifically, test case for "Check branch stack sampling" now fails. Upon investigation, identified that patch "perf record: Skip don't fail for events that don't open" is causing the breakage. This test case uses branch-filter as "save_type" and it is supposed to be skipped in powerpc. Snippet of code:
skip the test if the hardware doesn't support branch stack sampling
and if the architecture doesn't support filter types: any,save_type,u
if ! perf record -o- --no-buildid --branch-filter any,save_type,u -- true > /dev/null 2>&1 ; then
echo "skip: system doesn't support filter types: any,save_type,u"
exit 2
fi
Before applying the patch, running the command:
./perf record -o- --no-buildid --branch-filter any,save_type,u -- true cycles:PH: PMU Hardware or event type doesn't support branch stack sampling.
# echo $? 255
would return 255 (indicating not supported) with the error.
After applying the patch, the same command now returns 0, which is incorrect. The output is as follows:
# ./perf record -o- --no-buildid --branch-filter any,save_type,u -- true Lowering default frequency rate from 4000 to 2000. Please consider tweaking /proc/sys/kernel/perf_event_max_sample_rate. Error: Failure to open event 'cycles:PH' on PMU 'cpu' which will be removed. cycles:PH: PMU Hardware or event type doesn't support branch stack sampling. libperf: Miscounted nr_mmaps 0 vs 8 WARNING: No sample_id_all support, falling back to unordered processing [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.008 MB - ] # echo $? 0
Also there were some junk result in the output which I have skipped in above result. The patch appears to alter behavior such that the unsupported or failed event open still proceeds and leading to this.
Ian ,
Is this behaviour expected ?
Thank you,
Aditya
Quoted 26 lines
v2: Rebase and add tested-by tags from James Clark, Leo Yan and Atish
Patra who have tested on RISC-V and ARM CPUs, including the problem case from before.Ian Rogers (4):
perf evsel: Addpmu_namehelper
perf stat: Fixfind_statfor mixed legacy/non-legacy events perf record: Skip don't fail for events that don't open perf parse-events: Reapply "Prefer sysfs/JSON hardware events over
legacy"tools/perf/builtin-record.c | 22 +++++++--- tools/perf/util/evsel.c | 10 +++++ tools/perf/util/evsel.h | 1 + tools/perf/util/parse-events.c | 26 +++++++++--- tools/perf/util/parse-events.l | 76 +++++++++++++++++----------------- tools/perf/util/parse-events.y | 60 ++++++++++++++++++--------- tools/perf/util/pmus.c | 20 +++++++-- tools/perf/util/stat-shadow.c | 3 +- 8 files changed, 145 insertions(+), 73 deletions(-)Signature · 1 line
2.47.0.277.g8800431eea-goog
Ian Rogers
reviewerQuoted 68 lines · Aditya Bodkhe
Quoted 28 lines · Ian Rogers
At the RISC-V summit the topic of avoiding event data being in the RISC-V PMU kernel driver came up. There is a preference for sysfs/JSON events being the priority when no PMU is provided so that legacy events maybe supported via json. Originally Mark Rutland also expressed at LPC 2023 that doing this would resolve bugs on ARM Apple M? processors, but James Clark more recently tested this and believes the driver issues there may not have existed or have been resolved. In any case, it is inconsistent that with a PMU event names avoid legacy encodings, but when wildcarding PMUs (ie without a PMU with the event name) the legacy encodings have priority.
The patch doing this work was reverted in a v6.10 release candidate as, even though the patch was posted for weeks and had been on linux-next for weeks without issue, Linus was in the habit of using explicit legacy events with unsupported precision options on his Neoverse-N1. This machine has SLC PMU events for bus and CPU cycles where ARM decided to call the events
bus_cyclesand cycles, the latter being also a legacy event name. ARM haven't renamed the cycles event to a more consistentcpu_cyclesand avoided the problem. With these changes the problematic event will now be skipped, a large warning produced, and perf record will continue for the other PMU events. This solution was proposed by Arnaldo.Quoted 5 lines
Two minor changes have been added to help with the error message and to work around issues occurring with "perf stat metrics (shadow stat) test".
The patches have only been tested on my x86 non-hybrid laptop.
Hi,
After applying this patch series,we observed a regression while running the perf test suite on powerpc system. Specifically, test case for "Check branch stack sampling" now fails. Upon investigation, identified that patch "perf record: Skip don't fail for events that don't open" is causing the breakage. This test case uses branch-filter as "save_type" and it is supposed to be skipped in powerpc. Snippet of code:skip the test if the hardware doesn't support branch stack sampling
and if the architecture doesn't support filter types: any,save_type,u
if ! perf record -o- --no-buildid --branch-filter any,save_type,u -- true >/dev/null2>&1 ; then
echo "skip: system doesn't support filter types: any,save_type,u"
exit 2
fiBefore applying the patch, running the command:
./perf record -o- --no-buildid --branch-filter any,save_type,u -- true cycles:PH: PMU Hardware or event type doesn't support branch stack sampling.# echo $? 255would return 255 (indicating not supported) with the error.
After applying the patch, the same command now returns 0, which is incorrect. The output is as follows:# ./perf record -o- --no-buildid --branch-filter any,save_type,u -- true Lowering default frequency rate from 4000 to 2000. Please consider tweaking /proc/sys/kernel/perf_event_max_sample_rate. Error: Failure to open event 'cycles:PH' on PMU 'cpu' which will be removed. cycles:PH: PMU Hardware or event type doesn't support branch stack sampling. libperf: Miscounted nr_mmaps 0 vs 8 WARNING: No sample_id_all support, falling back to unordered processing [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.008 MB - ] # echo $? 0
Quoted 4 lines
Also there were some junk result in the output which I have skipped in above result. The patch appears to alter behavior such that the unsupported or failed event open still proceeds and leading to this.
Ian ,
Is this behaviour expected ?
Thanks Aditya, sorry for my slow response! Breaking tests isn't expected (thanks for the report!), warning when encountering an event that won't be recorded is expected. James Clark mentioned that we should probably make perf record fail if it has no events to record, my issue there was that the dummy event (for sideband things like mmap events) is always opened and I was worried about turning "perf record -e dummy .." into a failure. We should be able to determine if the dummy event was created by the tool or specified the user, so I should look to add that to the patch series and it should address this test regression.
Thanks!
Ian