Bug fixes on topdown events reordering
v5 cover Bug fixes on topdown events reordering
Changes:
v5 -> v6:
* no function change.
* rebase patchset to latest code of perf-tool-next tree.
* Add Kan's reviewed-by tag.
History:
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
Dapeng Mi (6):
perf x86/topdown: Complete topdown slots/metrics events check perf x86/topdown: Correct leader selection with sample_read enabled perf x86/topdown: Don't move topdown metric events in group perf tests: Add leader sampling test in record tests perf tests: Add topdown events counting and sampling tests perf tests: Add more topdown events regroup tests
tools/perf/arch/x86/util/evlist.c | 68 ++++++++++++++++++++++++++++-- tools/perf/arch/x86/util/evsel.c | 3 +- tools/perf/arch/x86/util/topdown.c | 64 +++++++++++++++++++++++++++- tools/perf/arch/x86/util/topdown.h | 2 + tools/perf/tests/shell/record.sh | 45 ++++++++++++++++++++ tools/perf/tests/shell/stat.sh | 28 +++++++++++- 6 files changed, 201 insertions(+), 9 deletions(-)
base-commit: 1de5b5dcb8353f36581c963df2d359a5f151a0be
Signature · 1 line
2.40.1
PATCH v5 1/6 perf x86/topdown: Complete topdown slots/metrics events check
It's not complete to check whether an event is a topdown slots or topdown metrics event by only comparing the event name since user may assign the event by RAW format, e.g.
perf stat -e '{instructions,cpu/r400/,cpu/r8300/}' sleep 1
Performance counter stats for 'sleep 1':
<not counted> instructions
<not counted> cpu/r400/
<not supported> cpu/r8300/ 1.002917796 seconds time elapsed
0.002955000 seconds user
0.000000000 seconds sysThe RAW format slots and topdown-be-bound events are not recognized and not regroup the events, and eventually cause error.
Thus add two helpers arch_is_topdown_slots()/arch_is_topdown_metrics() to detect whether an event is topdown slots/metrics event by comparing the event config directly, and use these two helpers to replace the original event name comparisons.
--- tools/perf/arch/x86/util/evlist.c | 8 ++--- tools/perf/arch/x86/util/evsel.c | 3 +- tools/perf/arch/x86/util/topdown.c | 48 +++++++++++++++++++++++++++++- tools/perf/arch/x86/util/topdown.h | 2 ++ 4 files changed, 55 insertions(+), 6 deletions(-)
tools/perf/arch/x86/util/evlist.c +4 -4
@@ -78,14 +78,14 @@ int arch_evlist__cmp(const struct evsel *lhs, const struct evsel *rhs)if (topdown_sys_has_perf_metrics () && (arch_evsel__must_be_in_group (lhs) ||arch_evsel__must_be_in_group (rhs))) {/* Ensure the topdown slots comes first. */ -if (strcasestr (lhs->name,"slots" ) && !strcasestr (lhs->name,"uops_retired.slots" )) +if (arch_is_topdown_slots (lhs))return -1 ; -if (strcasestr (rhs->name,"slots" ) && !strcasestr (rhs->name,"uops_retired.slots" )) +if (arch_is_topdown_slots (rhs))return 1 ;/* Followed by topdown events. */ -if (strcasestr (lhs->name,"topdown" ) && !strcasestr (rhs->name,"topdown" )) +if (arch_is_topdown_metrics (lhs) && !arch_is_topdown_metrics (rhs))return -1 ; -if (!strcasestr (lhs->name,"topdown" ) &&strcasestr (rhs->name,"topdown" )) +if (!arch_is_topdown_metrics (lhs) &&arch_is_topdown_metrics (rhs))return 1 ; }
tools/perf/arch/x86/util/evsel.c +2 -1
@@ -6,6 +6,7 @@#include "util/pmu.h" #include "util/pmus.h" #include "linux/string.h" +#include "topdown.h" #include "evsel.h" #include "util/debug.h" #include "env.h"
@@ -44,7 +45,7 @@ bool arch_evsel__must_be_in_group(const struct evsel *evsel) strcasestr(evsel->name, "uops_retired.slots"))return false; -return strcasestr (evsel->name,"topdown" ) ||strcasestr (evsel->name,"slots" ); +return arch_is_topdown_metrics (evsel) ||arch_is_topdown_slots (evsel); }int arch_evsel__hw_name (struct evsel *evsel,char *bf,size_t size)
tools/perf/arch/x86/util/topdown.c +47 -1
60 lines not on this page · raw message
tools/perf/arch/x86/util/topdown.h +2 -0
@@ -3,5 +3,7 @@#define _TOPDOWN_H 1bool topdown_sys_has_perf_metrics (void ); +bool arch_is_topdown_slots (const struct evsel *evsel); +bool arch_is_topdown_metrics (const struct evsel *evsel);#endif
Signature · 1 line
2.40.1
Ian Rogers
reviewerRe: [Patch v5 1/6] perf x86/topdown: Complete topdown slots/metrics events check
Quoted 84 lines · Dapeng Mi
It's not complete to check whether an event is a topdown slots or topdown metrics event by only comparing the event name since user may assign the event by RAW format, e.g.
perf stat -e '{instructions,cpu/r400/,cpu/r8300/}' sleep 1
Performance counter stats for 'sleep 1':
<not counted> instructions <not counted> cpu/r400/ <not supported> cpu/r8300/1.002917796 seconds time elapsed 0.002955000 seconds user 0.000000000 seconds sysThe RAW format slots and topdown-be-bound events are not recognized and not regroup the events, and eventually cause error.
Thus add two helpers arch_is_topdown_slots()/arch_is_topdown_metrics() to detect whether an event is topdown slots/metrics event by comparing the event config directly, and use these two helpers to replace the original event name comparisons.
Reviewed-by Kan Liang <[email protected]>Signed-off-by Dapeng Mi <[email protected]>--- tools/perf/arch/x86/util/evlist.c | 8 ++--- tools/perf/arch/x86/util/evsel.c | 3 +- tools/perf/arch/x86/util/topdown.c | 48 +++++++++++++++++++++++++++++- tools/perf/arch/x86/util/topdown.h | 2 ++ 4 files changed, 55 insertions(+), 6 deletions(-)tools/perf/arch/x86/util/evlist.c +4
-4@@ -78,14 +78,14 @@ int arch_evlist__cmp(const struct evsel *lhs, const struct evsel *rhs)if (topdown_sys_has_perf_metrics () && (arch_evsel__must_be_in_group (lhs) ||arch_evsel__must_be_in_group (rhs))) {/* Ensure the topdown slots comes first. */ -if (strcasestr (lhs->name,"slots" ) && !strcasestr (lhs->name,"uops_retired.slots" )) +if (arch_is_topdown_slots (lhs))return -1 ; -if (strcasestr (rhs->name,"slots" ) && !strcasestr (rhs->name,"uops_retired.slots" )) +if (arch_is_topdown_slots (rhs))return 1 ;/* Followed by topdown events. */ -if (strcasestr (lhs->name,"topdown" ) && !strcasestr (rhs->name,"topdown" )) +if (arch_is_topdown_metrics (lhs) && !arch_is_topdown_metrics (rhs))return -1 ; -if (!strcasestr (lhs->name,"topdown" ) &&strcasestr (rhs->name,"topdown" )) +if (!arch_is_topdown_metrics (lhs) &&arch_is_topdown_metrics (rhs))return 1 ; }tools/perf/arch/x86/util/evsel.c +2
-1@@ -6,6 +6,7 @@#include "util/pmu.h" #include "util/pmus.h" #include "linux/string.h" +#include "topdown.h" #include "evsel.h" #include "util/debug.h" #include "env.h" @@ -44,7 +45,7 @@ bool arch_evsel__must_be_in_group(const struct evsel *evsel) strcasestr(evsel->name, "uops_retired.slots"))return false; -return strcasestr (evsel->name,"topdown" ) ||strcasestr (evsel->name,"slots" ); +return arch_is_topdown_metrics (evsel) ||arch_is_topdown_slots (evsel); }int arch_evsel__hw_name (struct evsel *evsel,char *bf,size_t size)tools/perf/arch/x86/util/topdown.c +0
-0
tools/perf/arch/x86/util/topdown.c bool topdown_sys_has_perf_metrics(void) · line 36
Quoted 5 lines
tools/perf/arch/x86/util/topdown.c +2
-0@@ -32,6 +32,52 @@ bool topdown_sys_has_perf_metrics(void) }#define TOPDOWN_SLOTS 0x0400 +bool arch_is_topdown_slots (const struct evsel *evsel) +{
Perhaps: assert(evsel__find_pmu(evsel)->is_core);
Quoted 34 lines
tools/perf/arch/x86/util/topdown.c +34
-0+if (evsel->core.attr.config ==TOPDOWN_SLOTS ) +return true; + +return false; +} + +static int compare_topdown_event (void *vstate,struct pmu_event_info *info) +{ +int *config = vstate; +int event =0 ; +int umask =0 ; +char *str; + +if (!strcasestr (info->name,"topdown" )) +return 0 ; + + str =strcasestr (info->str,"event=" ); +if (str) +sscanf (str,"event=%x" , &event); + + str =strcasestr (info->str,"umask=" ); +if (str) +sscanf (str,"umask=%x" , &umask); + +if (event ==0 && *config == (event | umask <<8 )) +return 1 ; + +return 0 ; +} + +bool arch_is_topdown_metrics (const struct evsel *evsel) +{ +struct perf_pmu *pmu =evsel__find_pmu (evsel); +int config = evsel->core.attr.config;
tools/perf/arch/x86/util/topdown.c bool topdown_sys_has_perf_metrics(void) · line 80
Quoted 10 lines
tools/perf/arch/x86/util/topdown.c +10
-0+ +if (!pmu || !pmu->is_core) +return false; + +if (perf_pmu__for_each_event (pmu, false, &config, + compare_topdown_event)) +return true; + +return false; +}
Doing a linear search over every event is painful perf_pmu__have_event will try to binary search. The search rejects all events without "topdown" in their name, it then sees if the event code is 0 and the event|umask match the sysfs/json event. Is there ever a case the "topdown" string isn't at the beginning of the string? If it is it should be possible to binary search to the start of the topdown events.
Strictly you shouldn't hard code the config positions of event and umask, they are in the format list:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/pmu.h?h=perf-tools-next#n113
There is code doing similar to this here:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/pmu.c?h=perf-tools-next#n2285
but it avoids the scanf, uses formats...
It seems reasonable this code should reject all non-zero configs before searching all core PMU events. It could also use perf_pmu__name_from_config. So:
```
bool evsel__is_topdown_metric_event(const struct evsel *evsel)
{
struct perf_pmu *pmu;
const char *name_from_config;
if (evsel->core.attr.config & 0xFF != 0) /* All topdown events have
an event code of 0. */
return false;
pmu = evsel__find_pmu(evsel);
if (!pmu || !pmu->is_core)
return false;
name_from_config = perf_pmu__name_from_config(pmu, config);
return name_from_config && !strcasestr(name_from_config, "topdown);
}
```We could tweak perf_pmu__name_from_config to be pased a "filter". In this case the filter would skip events without topdown in their name, without doing a config comparison.
If later we change perf_pmu__name_from_config to say sort events in a list by config, then this code could take advantage of that. Perhaps for now there should just be an optional "prefix" that can be used to binary search to the events.
``` name_from_config = perf_pmu__name_from_config(pmu, config, /*prefix=*/"topdown"); ```
Thanks,
Ian
Quoted 26 lines
tools/perf/arch/x86/util/topdown.c +1
-1/* * Check whether a topdown group supports sample-read. @@ -44,7 +90,7 @@ bool arch_topdown_sample_read(struct evsel *leader)if (!evsel__sys_has_perf_metrics (leader))return false; -if (leader->core.attr.config ==TOPDOWN_SLOTS ) +if (arch_is_topdown_slots (leader))return true;return false;tools/perf/arch/x86/util/topdown.h +2
-1@@ -3,5 +3,7 @@#define _TOPDOWN_H 1bool topdown_sys_has_perf_metrics (void ); +bool arch_is_topdown_slots (const struct evsel *evsel); +bool arch_is_topdown_metrics (const struct evsel *evsel);#endif --2.40.1
Re: [Patch v5 1/6] perf x86/topdown: Complete topdown slots/metrics events check
Quoted 86 lines · Ian Rogers
Quoted 85 lines · Dapeng Mi
It's not complete to check whether an event is a topdown slots or topdown metrics event by only comparing the event name since user may assign the event by RAW format, e.g.
perf stat -e '{instructions,cpu/r400/,cpu/r8300/}' sleep 1
Performance counter stats for 'sleep 1':
<not counted> instructions <not counted> cpu/r400/ <not supported> cpu/r8300/1.002917796 seconds time elapsed 0.002955000 seconds user 0.000000000 seconds sysThe RAW format slots and topdown-be-bound events are not recognized and not regroup the events, and eventually cause error.
Thus add two helpers arch_is_topdown_slots()/arch_is_topdown_metrics() to detect whether an event is topdown slots/metrics event by comparing the event config directly, and use these two helpers to replace the original event name comparisons.
Reviewed-by Kan Liang <[email protected]>Signed-off-by Dapeng Mi <[email protected]>--- tools/perf/arch/x86/util/evlist.c | 8 ++--- tools/perf/arch/x86/util/evsel.c | 3 +- tools/perf/arch/x86/util/topdown.c | 48 +++++++++++++++++++++++++++++- tools/perf/arch/x86/util/topdown.h | 2 ++ 4 files changed, 55 insertions(+), 6 deletions(-)tools/perf/arch/x86/util/evlist.c +4
-4@@ -78,14 +78,14 @@ int arch_evlist__cmp(const struct evsel *lhs, const struct evsel *rhs)if (topdown_sys_has_perf_metrics () && (arch_evsel__must_be_in_group (lhs) ||arch_evsel__must_be_in_group (rhs))) {/* Ensure the topdown slots comes first. */ -if (strcasestr (lhs->name,"slots" ) && !strcasestr (lhs->name,"uops_retired.slots" )) +if (arch_is_topdown_slots (lhs))return -1 ; -if (strcasestr (rhs->name,"slots" ) && !strcasestr (rhs->name,"uops_retired.slots" )) +if (arch_is_topdown_slots (rhs))return 1 ;/* Followed by topdown events. */ -if (strcasestr (lhs->name,"topdown" ) && !strcasestr (rhs->name,"topdown" )) +if (arch_is_topdown_metrics (lhs) && !arch_is_topdown_metrics (rhs))return -1 ; -if (!strcasestr (lhs->name,"topdown" ) &&strcasestr (rhs->name,"topdown" )) +if (!arch_is_topdown_metrics (lhs) &&arch_is_topdown_metrics (rhs))return 1 ; }tools/perf/arch/x86/util/evsel.c +2
-1@@ -6,6 +6,7 @@#include "util/pmu.h" #include "util/pmus.h" #include "linux/string.h" +#include "topdown.h" #include "evsel.h" #include "util/debug.h" #include "env.h" @@ -44,7 +45,7 @@ bool arch_evsel__must_be_in_group(const struct evsel *evsel) strcasestr(evsel->name, "uops_retired.slots"))return false; -return strcasestr (evsel->name,"topdown" ) ||strcasestr (evsel->name,"slots" ); +return arch_is_topdown_metrics (evsel) ||arch_is_topdown_slots (evsel); }int arch_evsel__hw_name (struct evsel *evsel,char *bf,size_t size)tools/perf/arch/x86/util/topdown.c +0
-0@@ -32,6 +32,52 @@ bool topdown_sys_has_perf_metrics(void) }
Quoted 4 lines
tools/perf/arch/x86/util/topdown.c bool topdown_sys_has_perf_metrics(void) · line 36
Quoted 3 lines
tools/perf/arch/x86/util/topdown.c +2
-0#define TOPDOWN_SLOTS 0x0400 +bool arch_is_topdown_slots (const struct evsel *evsel) +{Perhaps: assert(evsel__find_pmu(evsel)->is_core);
assert? we don't need an assert as this helper just detects if a evsel is slots event and it's reasonable if the evsel is not. But it's nice to add the is_core check. Thanks.
Quoted 43 lines
Quoted 43 lines
tools/perf/arch/x86/util/topdown.c +42
-0+if (evsel->core.attr.config ==TOPDOWN_SLOTS ) +return true; + +return false; +} + +static int compare_topdown_event (void *vstate,struct pmu_event_info *info) +{ +int *config = vstate; +int event =0 ; +int umask =0 ; +char *str; + +if (!strcasestr (info->name,"topdown" )) +return 0 ; + + str =strcasestr (info->str,"event=" ); +if (str) +sscanf (str,"event=%x" , &event); + + str =strcasestr (info->str,"umask=" ); +if (str) +sscanf (str,"umask=%x" , &umask); + +if (event ==0 && *config == (event | umask <<8 )) +return 1 ; + +return 0 ; +} + +bool arch_is_topdown_metrics (const struct evsel *evsel) +{ +struct perf_pmu *pmu =evsel__find_pmu (evsel); +int config = evsel->core.attr.config; + +if (!pmu || !pmu->is_core) +return false; + +if (perf_pmu__for_each_event (pmu, false, &config, + compare_topdown_event)) +return true; +
Quoted 8 lines
tools/perf/arch/x86/util/topdown.c bool topdown_sys_has_perf_metrics(void) · line 80
Quoted 2 lines
tools/perf/arch/x86/util/topdown.c +2
-0+return false; +}Doing a linear search over every event is painful
perf_pmu__have_eventwill try to binary search. The search rejects all events without "topdown" in their name, it then sees if the event code is 0 and the event|umask match the sysfs/json event. Is there ever a case the "topdown" string isn't at the beginning of the string? If it is it
Currently all topdown events would start with "topdown" prefix.
Quoted 22 lines
should be possible to binary search to the start of the topdown events.
Strictly you shouldn't hard code the config positions of event and umask, they are in the format list:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/pmu.h?h=perf-tools-next#n113
There is code doing similar to this here:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/pmu.c?h=perf-tools-next#n2285
but it avoids the scanf, uses formats...
It seems reasonable this code should reject all non-zero configs before searching all core PMU events. It could also useperf_pmu__name_from_config. So:``` bool evsel__is_topdown_metric_event(const struct evsel *evsel) { struct perf_pmu *pmu; const char *name_from_config; if (evsel->core.attr.config & 0xFF != 0) /* All topdown events have an event code of 0. */ return false;
Quoted 8 lines
pmu = evsel__find_pmu(evsel);
if (!pmu || !pmu->is_core)
return false;
name_from_config = perf_pmu__name_from_config(pmu, config);
return name_from_config && !strcasestr(name_from_config, "topdown);
}
```Good point! would cook a patch to implement this. Thanks.
Quoted 42 lines
We could tweak
perf_pmu__name_from_configto be pased a "filter". In this case the filter would skip events without topdown in their name, without doing a config comparison.If later we change
perf_pmu__name_from_configto say sort events in a list by config, then this code could take advantage of that. Perhaps for now there should just be an optional "prefix" that can be used to binary search to the events.``` name_from_config = perf_pmu__name_from_config(pmu, config, /*prefix=*/"topdown"); ```Thanks,
IanQuoted 26 lines
tools/perf/arch/x86/util/topdown.c +1
-1/* * Check whether a topdown group supports sample-read. @@ -44,7 +90,7 @@ bool arch_topdown_sample_read(struct evsel *leader)if (!evsel__sys_has_perf_metrics (leader))return false; -if (leader->core.attr.config ==TOPDOWN_SLOTS ) +if (arch_is_topdown_slots (leader))return true;return false;tools/perf/arch/x86/util/topdown.h +2
-1@@ -3,5 +3,7 @@#define _TOPDOWN_H 1bool topdown_sys_has_perf_metrics (void ); +bool arch_is_topdown_slots (const struct evsel *evsel); +bool arch_is_topdown_metrics (const struct evsel *evsel);#endif --2.40.1
PATCH v5 2/6 perf x86/topdown: Correct leader selection with sample_read enabled
In reply to Dapeng MiAddresses an issue where, in the absence of a topdown metrics event within a sampling group, the slots event was incorrectly bypassed as the sampling leader when sample_read was enabled.
perf record -e '{slots,branches}:S' -c 10000 -vv sleep 1
In this case, the slots event should be sampled as leader but the branches event is sampled in fact like the verbose output shows.
perf_event_attr:
type 4 (cpu) size 168 config 0x400 (slots) sample_type IP|TID|TIME|READ|CPU|IDENTIFIER read_format ID|GROUP|LOST disabled 1 sample_id_all 1 exclude_guest 1
------------------------------------------------------------sys_perf_event_open: pid -1 cpu 0 group_fd -1 flags 0x8 = 5
------------------------------------------------------------perf_event_attr:
type 0 (PERF_TYPE_HARDWARE)
size 168
config 0x4 (PERF_COUNT_HW_BRANCH_INSTRUCTIONS)
{ sample_period, sample_freq } 10000
sample_type IP|TID|TIME|READ|CPU|IDENTIFIER
read_format ID|GROUP|LOST
sample_id_all 1
exclude_guest 1The sample period of slots event instead of branches event is reset to 0.
This fix ensures the slots event remains the leader under these conditions.
--- tools/perf/arch/x86/util/topdown.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-)
tools/perf/arch/x86/util/topdown.c +16 -2
@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0 #include "api/fs/fs.h" #include "util/evsel.h" +#include "util/evlist.h" #include "util/pmu.h" #include "util/pmus.h" #include "util/topdown.h"
@@ -87,11 +88,24 @@ bool arch_is_topdown_metrics(const struct evsel *evsel)*/ bool arch_topdown_sample_read (struct evsel *leader) { +struct evsel *evsel; +if (!evsel__sys_has_perf_metrics (leader))return false; -if (arch_is_topdown_slots (leader)) -return true; +if (!arch_is_topdown_slots (leader)) +return false; + +/* +* If slots event as leader event but no topdown metric events +* in group, slots event should still sample as leader. +*/ +evlist__for_each_entry (leader->evlist, evsel) { +if (evsel->core.leader != leader->core.leader) +return false; +if (evsel != leader &&arch_is_topdown_metrics (evsel)) +return true; + }return false; }
Signature · 1 line
2.40.1
PATCH v5 3/6 perf x86/topdown: Don't move topdown metric events in group
In reply to Dapeng Miwhen running below perf command, we say error is reported.
perf record -e "{slots,instructions,topdown-retiring}:S" -vv -C0 sleep 1
------------------------------------------------------------perf_event_attr:
type 4 (cpu) size 168 config 0x400 (slots) sample_type IP|TID|TIME|READ|CPU|PERIOD|IDENTIFIER read_format ID|GROUP|LOST disabled 1 sample_id_all 1 exclude_guest 1
------------------------------------------------------------sys_perf_event_open: pid -1 cpu 0 group_fd -1 flags 0x8 = 5
------------------------------------------------------------perf_event_attr:
type 4 (cpu)
size 168
config 0x8000 (topdown-retiring)
{ sample_period, sample_freq } 4000
sample_type IP|TID|TIME|READ|CPU|PERIOD|IDENTIFIER
read_format ID|GROUP|LOST
freq 1
sample_id_all 1
exclude_guest 1------------------------------------------------------------sys_perf_event_open: pid -1 cpu 0 group_fd 5 flags 0x8 sys_perf_event_open failed, error -22
Error:
The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (topdown-retiring).
The reason of error is that the events are regrouped and topdown-retiring event is moved to closely after the slots event and topdown-retiring event needs to do the sampling, but Intel PMU driver doesn't support to sample topdown metrics events.
For topdown metrics events, it just requires to be in a group which has slots event as leader. It doesn't require topdown metrics event must be closely after slots event. Thus it's a overkill to move topdown metrics event closely after slots event in events regrouping and furtherly cause the above issue.
Thus don't move topdown metrics events forward if they are already in a group.
--- tools/perf/arch/x86/util/evlist.c | 62 ++++++++++++++++++++++++++++++- 1 file changed, 61 insertions(+), 1 deletion(-)
tools/perf/arch/x86/util/evlist.c +61 -1
@@ -75,6 +75,61 @@ int arch_evlist__add_default_attrs(struct evlist *evlist,int arch_evlist__cmp (const struct evsel *lhs,const struct evsel *rhs) { +/* +* Currently the following topdown events sequence are supported to +* move and regroup correctly. +* +* a. all events in a group +* perf stat -e "{instructions,topdown-retiring,slots}" -C0 sleep 1 +* WARNING: events were regrouped to match PMUs +* Performance counter stats for 'CPU(s) 0': +* 15,066,240 slots +* 1,899,760 instructions +* 2,126,998 topdown-retiring +* b. all events not in a group +* perf stat -e "instructions,topdown-retiring,slots" -C0 sleep 1 +* WARNING: events were regrouped to match PMUs +* Performance counter stats for 'CPU(s) 0': +* 2,045,561 instructions +* 17,108,370 slots +* 2,281,116 topdown-retiring +* c. slots event in a group but topdown metrics events outside the group +* perf stat -e "{instructions,slots},topdown-retiring" -C0 sleep 1 +* WARNING: events were regrouped to match PMUs +* Performance counter stats for 'CPU(s) 0': +* 20,323,878 slots +* 2,634,884 instructions +* 3,028,656 topdown-retiring +* d. slots event and topdown metrics events in two groups +* perf stat -e "{instructions,slots},{topdown-retiring}" -C0 sleep 1 +* WARNING: events were regrouped to match PMUs +* Performance counter stats for 'CPU(s) 0': +* 26,319,024 slots +* 2,427,791 instructions +* 2,683,508 topdown-retiring +* +* If slots event and topdown metrics events are not in same group, the +* topdown metrics events must be first event after the slots event group, +* otherwise topdown metrics events can't be regrouped correctly, e.g. +* +* a. perf stat -e "{instructions,slots},cycles,topdown-retiring" -C0 sleep 1 +* WARNING: events were regrouped to match PMUs +* Performance counter stats for 'CPU(s) 0': +* 17,923,134 slots +* 2,154,855 instructions +* 3,015,058 cycles +* <not supported> topdown-retiring +* +* if slots event and topdown metrics events are in two groups, the group which +* has topdown metrics events must contain only the topdown metrics event, +* otherwise topdown metrics event can't be regrouped correctly as well, e.g. +* +* a. perf stat -e "{instructions,slots},{topdown-retiring,cycles}" -C0 sleep 1 +* WARNING: events were regrouped to match PMUs +* Error: +* The sys_perf_event_open() syscall returned with 22 (Invalid argument) for +* event (topdown-retiring) +*/ if (topdown_sys_has_perf_metrics () && (arch_evsel__must_be_in_group (lhs) ||arch_evsel__must_be_in_group (rhs))) {/* Ensure the topdown slots comes first. */
@@ -85,7 +140,12 @@ int arch_evlist__cmp(const struct evsel *lhs, const struct evsel *rhs)/* Followed by topdown events. */ if (arch_is_topdown_metrics (lhs) && !arch_is_topdown_metrics (rhs))return -1 ; -if (!arch_is_topdown_metrics (lhs) &&arch_is_topdown_metrics (rhs)) +/* +* Move topdown events forward only when topdown events +* are not in same group with previous event. +*/ +if (!arch_is_topdown_metrics (lhs) &&arch_is_topdown_metrics (rhs) && + lhs->core.leader != rhs->core.leader)return 1 ; }
Signature · 1 line
2.40.1
PATCH v5 4/6 perf tests: Add leader sampling test in record tests
In reply to Dapeng MiAdd leader sampling test to validate event counts are captured into record and the count value is consistent.
--- tools/perf/tests/shell/record.sh | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
tools/perf/tests/shell/record.sh +28 -0
@@ -17,6 +17,7 @@ skip_test_missing_symbol ${testsym} err=0 perfdata=$(mktemp /tmp/__perf_test.perf.data.XXXXX) +script_output=$(mktemp /tmp/__perf_test.perf.data.XXXXX.script) testprog="perf test -w thloop" cpu_pmu_dir="/sys/bus/event_source/devices/cpu*" br_cntr_file="/caps/branch_counter_nr"
@@ -228,6 +229,32 @@ test_cgroup() { echo "Cgroup sampling test [Success]" } +test_leader_sampling() { + echo "Basic leader sampling test" + if ! perf record -o "${perfdata}" -e "{branches,branches}:Su" perf test -w brstack 2> /dev/null + then + echo "Leader sampling [Failed record]" + err=1 + return + fi + index=0 + perf script -i "${perfdata}" > $script_output + while IFS= read -r line + do + # Check if the two branches counts are equal in each record + branches=$(echo $line | awk '{for(i=1;i<=NF;i++) if($i=="branches:") print $(i-1)}') + if [ $(($index%2)) -ne 0 ] && [ ${branches}x != ${prev_branches}x ] + then + echo "Leader sampling [Failed inconsistent branches count]" + err=1 + return + fi + index=$(($index+1)) + prev_branches=$branches + done < $script_output + echo "Basic leader sampling test [Success]" +} + # raise the limit of file descriptors to minimum if [[ $default_fd_limit -lt $min_fd_limit ]]; then ulimit -Sn $min_fd_limit
@@ -239,6 +266,7 @@ test_system_wide test_workload test_branch_counter test_cgroup +test_leader_sampling # restore the default value ulimit -Sn $default_fd_limit
Signature · 1 line
2.40.1
PATCH v5 5/6 perf tests: Add topdown events counting and sampling tests
In reply to Dapeng MiAdd counting and leader sampling tests to verify topdown events including raw format can be reordered correctly.
--- tools/perf/tests/shell/record.sh | 17 +++++++++++++++++ tools/perf/tests/shell/stat.sh | 6 ++++++ 2 files changed, 23 insertions(+)
tools/perf/tests/shell/record.sh +17 -0
@@ -255,6 +255,22 @@ test_leader_sampling() { echo "Basic leader sampling test [Success]" } +test_topdown_leader_sampling() { + echo "Topdown leader sampling test" + if ! perf stat -e "{slots,topdown-retiring}" true 2> /dev/null + then + echo "Topdown leader sampling [Skipped event parsing failed]" + return + fi + if ! perf record -o "${perfdata}" -e "{instructions,slots,topdown-retiring}:S" true 2> /dev/null + then + echo "Topdown leader sampling [Failed topdown events not reordered correctly]" + err=1 + return + fi + echo "Topdown leader sampling test [Success]" +} + # raise the limit of file descriptors to minimum if [[ $default_fd_limit -lt $min_fd_limit ]]; then ulimit -Sn $min_fd_limit
@@ -267,6 +283,7 @@ test_workload test_branch_counter test_cgroup test_leader_sampling +test_topdown_leader_sampling # restore the default value ulimit -Sn $default_fd_limit
tools/perf/tests/shell/stat.sh +6 -0
@@ -79,6 +79,12 @@ test_topdown_groups() { err=1 return fi + if perf stat -e '{instructions,r400,r8000}' true 2>&1 | grep -E -q "<not supported>" + then + echo "Topdown event group test [Failed raw format slots not reordered first]" + err=1 + return + fi echo "Topdown event group test [Success]" }
Signature · 1 line
2.40.1
PATCH v5 6/6 perf tests: Add more topdown events regroup tests
In reply to Dapeng MiAdd more test cases to cover all supported topdown events regroup cases.
--- tools/perf/tests/shell/stat.sh | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)
tools/perf/tests/shell/stat.sh +20 -2
@@ -73,9 +73,27 @@ test_topdown_groups() { err=1 return fi - if perf stat -e '{topdown-retiring,slots}' true 2>&1 | grep -E -q "<not supported>" + if perf stat -e 'instructions,topdown-retiring,slots' true 2>&1 | grep -E -q "<not supported>" then - echo "Topdown event group test [Failed slots not reordered first]" + echo "Topdown event group test [Failed slots not reordered first in no-group case]" + err=1 + return + fi + if perf stat -e '{instructions,topdown-retiring,slots}' true 2>&1 | grep -E -q "<not supported>" + then + echo "Topdown event group test [Failed slots not reordered first in single group case]" + err=1 + return + fi + if perf stat -e '{instructions,slots},topdown-retiring' true 2>&1 | grep -E -q "<not supported>" + then + echo "Topdown event group test [Failed topdown metrics event not move into slots group]" + err=1 + return + fi + if perf stat -e '{instructions,slots},{topdown-retiring}' true 2>&1 | grep -E -q "<not supported>" + then + echo "Topdown event group test [Failed topdown metrics group not merge into slots group]" err=1 return fi
Signature · 1 line
2.40.1
Quoted 6 lines · Dapeng Mi
Changes:
v5 -> v6:
* no function change.
* rebase patchset to latest code of perf-tool-next tree.
* Add Kan's reviewed-by tag.
Quoted 7 lines
History:
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/[...]
Applied to perf-tools-next, thanks!
Best regards,
Namhyung
Ian Rogers
reviewerQuoted 15 lines · Namhyung Kim
Quoted 12 lines · Dapeng Mi
Changes:
v5 -> v6:
* no function change.
* rebase patchset to latest code of perf-tool-next tree.
* Add Kan's reviewed-by tag.History:
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
Quoted 3 lines
Quoted 1 line
[...]
Applied to perf-tools-next, thanks!
I disagreed with an early patch set and the issue wasn't resolved. Specifically:
``` /* Followed by topdown events. */ if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; - if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) + /* + * Move topdown events forward only when topdown events + * are not in same group with previous event. + */ + if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && + lhs->core.leader != rhs->core.leader) return 1; ```
Is a broken comparator as the lhs then rhs behavior varies from the rhs then lhs behavior. The qsort implementation can randomly order the events.
Please drop/revert.
Thanks,
Ian
Quoted 2 lines
Best regards,
Namhyung
Ian Rogers
reviewerQuoted 37 lines · Ian Rogers
Quoted 15 lines · Namhyung Kim
Quoted 12 lines · Dapeng Mi
Changes:
v5 -> v6:
* no function change.
* rebase patchset to latest code of perf-tool-next tree.
* Add Kan's reviewed-by tag.History:
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/Quoted 3 lines
Quoted 1 line
[...]
Applied to perf-tools-next, thanks!
I disagreed with an early patch set and the issue wasn't resolved. Specifically:
``` /* Followed by topdown events. */ if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; - if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) + /* + * Move topdown events forward only when topdown events + * are not in same group with previous event. + */ + if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && + lhs->core.leader != rhs->core.leader) return 1; ```
Quoted 4 lines
Is a broken comparator as the lhs then rhs behavior varies from the rhs then lhs behavior. The qsort implementation can randomly order the events.
Please drop/revert.
Tihs is still in the tree:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/commit/?h=perf-tools-next&id=3b5edc0421e2598a0ae7f0adcd592017f37e3cdf
It should be invariant that for all comparators that:
a < b == b > a
This code breaks this property and so event sorting is broken.
Thanks,
Ian
Namhyung Kim
maintainerQuoted 36 lines · Ian Rogers
Quoted 15 lines · Namhyung Kim
Quoted 12 lines · Dapeng Mi
Changes:
v5 -> v6:
* no function change.
* rebase patchset to latest code of perf-tool-next tree.
* Add Kan's reviewed-by tag.History:
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/Quoted 3 lines
Quoted 1 line
[...]
Applied to perf-tools-next, thanks!
I disagreed with an early patch set and the issue wasn't resolved. Specifically:
``` /* Followed by topdown events. */ if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; - if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) + /* + * Move topdown events forward only when topdown events + * are not in same group with previous event. + */ + if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && + lhs->core.leader != rhs->core.leader) return 1; ```
Quoted 4 lines
Is a broken comparator as the lhs then rhs behavior varies from the rhs then lhs behavior. The qsort implementation can randomly order the events.
Please drop/revert.
Can you please provide an example when it's broken? I'm not sure how it can produce new errors, but it seems to fix a specific problem. Do you have a new test failure after this change?
Thanks,
Namhyung
Ian Rogers
reviewerQuoted 43 lines · Namhyung Kim
Quoted 36 lines · Ian Rogers
Quoted 15 lines · Namhyung Kim
Quoted 12 lines · Dapeng Mi
Changes:
v5 -> v6:
* no function change.
* rebase patchset to latest code of perf-tool-next tree.
* Add Kan's reviewed-by tag.History:
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/Quoted 3 lines
Quoted 1 line
[...]
Applied to perf-tools-next, thanks!
I disagreed with an early patch set and the issue wasn't resolved. Specifically:
``` /* Followed by topdown events. */ if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; - if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) + /* + * Move topdown events forward only when topdown events + * are not in same group with previous event. + */ + if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && + lhs->core.leader != rhs->core.leader) return 1; ```Quoted 4 lines
Is a broken comparator as the lhs then rhs behavior varies from the rhs then lhs behavior. The qsort implementation can randomly order the events.
Please drop/revert.
Quoted 3 lines
Can you please provide an example when it's broken? I'm not sure how it can produce new errors, but it seems to fix a specific problem. Do you have a new test failure after this change?
It may work with a particular sort routine in a particular library today, the issue is that if the sort routine were say changed from:
if (cmp(a, b) < 0)
to:
if (cmp(b, a) > 0)
the sort would vary with this change even though such a change in the sorting code is a no-op. It is fundamental algorithms that this code is broken, I'm not going to spend the time finding a list of instructions and a sort routine to demonstrate this.
Thanks,
Ian
Quoted 53 lines · Ian Rogers
Quoted 43 lines · Namhyung Kim
Quoted 36 lines · Ian Rogers
Quoted 15 lines · Namhyung Kim
Quoted 12 lines · Dapeng Mi
Changes:
v5 -> v6:
* no function change.
* rebase patchset to latest code of perf-tool-next tree.
* Add Kan's reviewed-by tag.History:
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/Quoted 3 lines
Quoted 1 line
[...]
Applied to perf-tools-next, thanks!
I disagreed with an early patch set and the issue wasn't resolved. Specifically:
``` /* Followed by topdown events. */ if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; - if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) + /* + * Move topdown events forward only when topdown events + * are not in same group with previous event. + */ + if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && + lhs->core.leader != rhs->core.leader) return 1; ```Quoted 4 lines
Is a broken comparator as the lhs then rhs behavior varies from the rhs then lhs behavior. The qsort implementation can randomly order the events.
Please drop/revert.Quoted 3 lines
Can you please provide an example when it's broken? I'm not sure how it can produce new errors, but it seems to fix a specific problem. Do you have a new test failure after this change?
It may work with a particular sort routine in a particular library today, the issue is that if the sort routine were say changed from:
if (
cmp(a, b)< 0)
Quoted 8 lines
to:
if (
cmp(b, a)> 0)the sort would vary with this change even though such a change in the sorting code is a no-op. It is fundamental algorithms that this code is broken, I'm not going to spend the time finding a list of instructions and a sort routine to demonstrate this.
I'm not sure I fully understand. But just want to mention that the change only impacts the Topdown perf metric group, which is only available for the ICL and later p-core. Nothing will change if no perf metrics topdown events are used. Very limited impact.
I like the patch set is because it provides examples and tests to cover the possible combination of the slots event and topdown metrics events. So we will have a clear expectation for different combinations caused by the perf metrics mess.
If the algorithms cannot be changed, can you please give some suggestions, especially for the sample read failure?
Thanks,
Kan
Ian Rogers
reviewerQuoted 67 lines · Kan
Quoted 53 lines · Ian Rogers
Quoted 43 lines · Namhyung Kim
Quoted 36 lines · Ian Rogers
Quoted 15 lines · Namhyung Kim
Quoted 12 lines · Dapeng Mi
Changes:
v5 -> v6:
* no function change.
* rebase patchset to latest code of perf-tool-next tree.
* Add Kan's reviewed-by tag.History:
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/Quoted 3 lines
Quoted 1 line
[...]
Applied to perf-tools-next, thanks!
I disagreed with an early patch set and the issue wasn't resolved. Specifically:
``` /* Followed by topdown events. */ if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; - if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) + /* + * Move topdown events forward only when topdown events + * are not in same group with previous event. + */ + if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && + lhs->core.leader != rhs->core.leader) return 1; ```Quoted 4 lines
Is a broken comparator as the lhs then rhs behavior varies from the rhs then lhs behavior. The qsort implementation can randomly order the events.
Please drop/revert.Quoted 3 lines
Can you please provide an example when it's broken? I'm not sure how it can produce new errors, but it seems to fix a specific problem. Do you have a new test failure after this change?
It may work with a particular sort routine in a particular library today, the issue is that if the sort routine were say changed from:
if (
cmp(a, b)< 0)Quoted 8 lines
to:
if (
cmp(b, a)> 0)the sort would vary with this change even though such a change in the sorting code is a no-op. It is fundamental algorithms that this code is broken, I'm not going to spend the time finding a list of instructions and a sort routine to demonstrate this.
Quoted 4 lines
I'm not sure I fully understand. But just want to mention that the change only impacts the Topdown perf metric group, which is only available for the ICL and later p-core. Nothing will change if no perf metrics topdown events are used. Very limited impact.
How is breaking every ICL and later Intel model (excluding atoms) limited impact?
Quoted 4 lines
I like the patch set is because it provides examples and tests to cover the possible combination of the slots event and topdown metrics events. So we will have a clear expectation for different combinations caused by the perf metrics mess.
Tests are good. I have no problem with the change, possibly it is inefficient, except that hacking a sort comparator to not be symmetric is broken.
Quoted 2 lines
If the algorithms cannot be changed, can you please give some suggestions, especially for the sample read failure?
So this is symmetric:
``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) return 1; ``` That is were lhs and rhs swapped then you'd get the expected comparison order. ``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ``` Is symmetric as well. ``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ```
(what this patch does) is not symmetric as the group leader impacts the greater-than case but not the less-than case.
It is not uncommon to see in a sort function:
```
if (cmp(a, b) <= 0) {
assert(cmp(b,a) >= 0 && "check for unstable/broken compare functions");
```and we could add this here:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/lib/list_sort.c?h=perf-tools-next#n22
Try searching up "Comparison method violates its general contract" which is what Java throws when its TimSort implementation detects cases like this. If you break the comparator you can get the sort into an infinite loop - maybe that's enough of an indication that the code is broken and you don't need the exception. As is common in kernel code we're missing guard rails in list_sort, were the comparator passed to qsort then expectations on breakage are a roll of the dice.
I believe when I originally gave this feedback I didn't think the fix was to do it in the comparator and maybe an additional pass over the list was going to be necessary. Basically a sort needs to be a sort and it can't kind of be a sort depending on the order of the comparison, this is just incurring tech debt and when a sort tweak happens everything will break. As the usual chump cleaning up this tech debt I'd prefer it wasn't introduced.
Fwiw, I refuse to carry this change in:
https://github.com/googleprodkernel/linux-perf/commits/google_tools_master/
and if the tests break as a consequence then the appropriate thing is to revert those too. Linus/upstream maintainers follow a different plan and that's their business, I can't build off of this code.
It is unclear to me why we get patches that have been pointed out to be broken rebased/resent repeatedly without the broken-ness corrected. For example, the abuse of aggregation for metrics in perf script. At least point to the disagreement in the commit message, call me an idiot, and let the maintainer make an informed or uninformed choice. I am frequently an idiot and I apologize for all those cases, to the best of my knowledge this isn't one of them.
Thanks,
Ian
Namhyung Kim
maintainerQuoted 68 lines · Ian Rogers
Quoted 67 lines · Kan
Quoted 61 lines · Ian Rogers
Quoted 43 lines · Namhyung Kim
Quoted 36 lines · Ian Rogers
Quoted 15 lines · Namhyung Kim
Quoted 12 lines · Dapeng Mi
Changes:
v5 -> v6:
* no function change.
* rebase patchset to latest code of perf-tool-next tree.
* Add Kan's reviewed-by tag.History:
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/Quoted 3 lines
Quoted 1 line
[...]
Applied to perf-tools-next, thanks!
I disagreed with an early patch set and the issue wasn't resolved. Specifically:
``` /* Followed by topdown events. */ if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; - if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) + /* + * Move topdown events forward only when topdown events + * are not in same group with previous event. + */ + if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && + lhs->core.leader != rhs->core.leader) return 1; ```Quoted 4 lines
Is a broken comparator as the lhs then rhs behavior varies from the rhs then lhs behavior. The qsort implementation can randomly order the events.
Please drop/revert.Quoted 3 lines
Can you please provide an example when it's broken? I'm not sure how it can produce new errors, but it seems to fix a specific problem. Do you have a new test failure after this change?
It may work with a particular sort routine in a particular library today, the issue is that if the sort routine were say changed from:
if (
cmp(a, b)< 0)to:
if (
cmp(b, a)> 0)the sort would vary with this change even though such a change in the sorting code is a no-op. It is fundamental algorithms that this code is broken, I'm not going to spend the time finding a list of instructions and a sort routine to demonstrate this.
Quoted 7 lines
Quoted 4 lines
I'm not sure I fully understand. But just want to mention that the change only impacts the Topdown perf metric group, which is only available for the ICL and later p-core. Nothing will change if no perf metrics topdown events are used. Very limited impact.
How is breaking every ICL and later Intel model (excluding atoms) limited impact?
I guess he meant it's only for Topdown metric groups with an incorrect order on those models.
Quoted 9 lines
Quoted 5 lines
I like the patch set is because it provides examples and tests to cover the possible combination of the slots event and topdown metrics events. So we will have a clear expectation for different combinations caused by the perf metrics mess.
Tests are good. I have no problem with the change, possibly it is inefficient, except that hacking a sort comparator to not be symmetric is broken.
Ok.
Quoted 28 lines
Quoted 3 lines
If the algorithms cannot be changed, can you please give some suggestions, especially for the sample read failure?
So this is symmetric:
``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) return 1; ``` That is were lhs and rhs swapped then you'd get the expected comparison order. ``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ``` Is symmetric as well. ``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ```
Quoted 8 lines
(what this patch does) is not symmetric as the group leader impacts the greater-than case but not the less-than case.
It is not uncommon to see in a sort function:
``` if (cmp(a, b) <= 0) { assert(cmp(b,a) >= 0 && "check for unstable/broken compare functions"); ```
I see. So are you proposing this?
tools/perf/arch/x86/util/evlist.c +2 -1
@@ -70,7 +70,8 @@ int arch_evlist__cmp(const struct evsel *lhs, const struct evsel *rhs)if (arch_is_topdown_slots (rhs))return 1 ;/* Followed by topdown events. */ -if (arch_is_topdown_metrics (lhs) && !arch_is_topdown_metrics (rhs)) +if (arch_is_topdown_metrics (lhs) && !arch_is_topdown_metrics (rhs) && + lhs->core.leader != rhs->core.leader)return -1 ;/* * Move topdown events forward only when topdown events
Dapeng and Kan, can you verify if it's ok? My quick tests look ok.
Quoted 10 lines
and we could add this here:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/lib/list_sort.c?h=perf-tools-next#n22
Try searching up "Comparison method violates its general contract" which is what Java throws when its TimSort implementation detects cases like this. If you break the comparator you can get the sort into an infinite loop - maybe that's enough of an indication that the code is broken and you don't need the exception. As is common in kernel code we're missing guard rails inlist_sort, were the comparator passed to qsort then expectations on breakage are a roll of the dice.
Quoted 7 lines
I believe when I originally gave this feedback I didn't think the fix was to do it in the comparator and maybe an additional pass over the list was going to be necessary. Basically a sort needs to be a sort and it can't kind of be a sort depending on the order of the comparison, this is just incurring tech debt and when a sort tweak happens everything will break. As the usual chump cleaning up this tech debt I'd prefer it wasn't introduced.
Sorry, I think I didn't follow the previous thread and missed your feedback.
Quoted 7 lines
Fwiw, I refuse to carry this change in:
https://github.com/googleprodkernel/linux-perf/commits/google_tools_master/
and if the tests break as a consequence then the appropriate thing is to revert those too. Linus/upstream maintainers follow a different plan and that's their business, I can't build off of this code.
Quoted 7 lines
It is unclear to me why we get patches that have been pointed out to be broken rebased/resent repeatedly without the broken-ness corrected. For example, the abuse of aggregation for metrics in perf script. At least point to the disagreement in the commit message, call me an idiot, and let the maintainer make an informed or uninformed choice. I am frequently an idiot and I apologize for all those cases, to the best of my knowledge this isn't one of them.
Yeah I understand your point and it sounds reasonable. Thanks for your feedback. I don't think I'll revert the change though, we can add the fix on top instead.
Thanks,
Namhyung
Liang, Kan
Tested-byQuoted 51 lines · Namhyung Kim
Quoted 26 lines
Quoted 2 lines
If the algorithms cannot be changed, can you please give some suggestions, especially for the sample read failure?
So this is symmetric:
``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) return 1; ``` That is were lhs and rhs swapped then you'd get the expected comparison order. ``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ``` Is symmetric as well. ``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ```Quoted 8 lines
(what this patch does) is not symmetric as the group leader impacts the greater-than case but not the less-than case.
It is not uncommon to see in a sort function:
``` if (cmp(a, b) <= 0) { assert(cmp(b,a) >= 0 && "check for unstable/broken compare functions"); ```I see. So are you proposing this?
tools/perf/arch/x86/util/evlist.c +2
-1@@ -70,7 +70,8 @@ int arch_evlist__cmp(const struct evsel *lhs, const struct evsel *rhs)if (arch_is_topdown_slots (rhs))return 1 ;/* Followed by topdown events. */ -if (arch_is_topdown_metrics (lhs) && !arch_is_topdown_metrics (rhs)) +if (arch_is_topdown_metrics (lhs) && !arch_is_topdown_metrics (rhs) && + lhs->core.leader != rhs->core.leader)return -1 ;/* * Move topdown events forward only when topdown events
Quoted 1 line
Dapeng and Kan, can you verify if it's ok? My quick tests look ok.
I verified the above change. It works well.
Thanks,
Kan
Ian Rogers
reviewerQuoted 55 lines · Kan
Quoted 51 lines · Namhyung Kim
Quoted 26 lines
Quoted 2 lines
If the algorithms cannot be changed, can you please give some suggestions, especially for the sample read failure?
So this is symmetric:
``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) return 1; ``` That is were lhs and rhs swapped then you'd get the expected comparison order. ``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ``` Is symmetric as well. ``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ```Quoted 8 lines
(what this patch does) is not symmetric as the group leader impacts the greater-than case but not the less-than case.
It is not uncommon to see in a sort function:
``` if (cmp(a, b) <= 0) { assert(cmp(b,a) >= 0 && "check for unstable/broken compare functions"); ```I see. So are you proposing this?
tools/perf/arch/x86/util/evlist.c +2
-1@@ -70,7 +70,8 @@ int arch_evlist__cmp(const struct evsel *lhs, const struct evsel *rhs)if (arch_is_topdown_slots (rhs))return 1 ;/* Followed by topdown events. */ -if (arch_is_topdown_metrics (lhs) && !arch_is_topdown_metrics (rhs)) +if (arch_is_topdown_metrics (lhs) && !arch_is_topdown_metrics (rhs) && + lhs->core.leader != rhs->core.leader)return -1 ;/* * Move topdown events forward only when topdown events
Quoted 5 lines
Quoted 1 line
Dapeng and Kan, can you verify if it's ok? My quick tests look ok.
I verified the above change. It works well.
Tested-by Kan Liang <[email protected]>
Dapeng's comment should cover replace the comment /* Followed by topdown events. */ but there are other things amiss. I'm thinking of something like: "slots,cycles,{instructions,topdown-be-bound}" the topdown-be-bound should get sorted and grouped with slots, but cycles and instructions have no reason to be reordered, so do we end up with slots, instructions and topdown-be-bound being grouped with cycles sitting ungrouped in the middle of the evlist? I believe there are assumptions that grouped evsels are adjacent in the evlist, not least in:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2106
Does cycles instructions end up being broken out of a group in this case? Which feels like the case the code was trying to avoid.
Thanks,
Ian
Namhyung Kim
maintainerQuoted 56 lines · Ian Rogers
Quoted 55 lines · Kan
Quoted 51 lines · Namhyung Kim
Quoted 26 lines
Quoted 2 lines
If the algorithms cannot be changed, can you please give some suggestions, especially for the sample read failure?
So this is symmetric:
``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) return 1; ``` That is were lhs and rhs swapped then you'd get the expected comparison order. ``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ``` Is symmetric as well. ``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ```Quoted 8 lines
(what this patch does) is not symmetric as the group leader impacts the greater-than case but not the less-than case.
It is not uncommon to see in a sort function:
``` if (cmp(a, b) <= 0) { assert(cmp(b,a) >= 0 && "check for unstable/broken compare functions"); ```I see. So are you proposing this?
tools/perf/arch/x86/util/evlist.c +2
-1@@ -70,7 +70,8 @@ int arch_evlist__cmp(const struct evsel *lhs, const struct evsel *rhs)if (arch_is_topdown_slots (rhs))return 1 ;/* Followed by topdown events. */ -if (arch_is_topdown_metrics (lhs) && !arch_is_topdown_metrics (rhs)) +if (arch_is_topdown_metrics (lhs) && !arch_is_topdown_metrics (rhs) && + lhs->core.leader != rhs->core.leader)return -1 ;/* * Move topdown events forward only when topdown events
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Dapeng and Kan, can you verify if it's ok? My quick tests look ok.
I verified the above change. It works well.
Tested-by Kan Liang <[email protected]>
Thanks for the check!
Quoted 13 lines
Dapeng's comment should cover replace the comment /* Followed by topdown events. */ but there are other things amiss. I'm thinking of something like: "slots,cycles,{instructions,topdown-be-bound}" the topdown-be-bound should get sorted and grouped with slots, but cycles and instructions have no reason to be reordered, so do we end up with slots, instructions and topdown-be-bound being grouped with cycles sitting ungrouped in the middle of the evlist? I believe there are assumptions that grouped evsels are adjacent in the evlist, not least in:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2106
Does cycles instructions end up being broken out of a group in this case? Which feels like the case the code was trying to avoid.
I got this:
$ sudo ./perf record -a -e "slots,cycles,{instructions,topdown-be-bound}" true
Error:
The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (topdown-be-bound).
"dmesg | grep -i perf" may provide additional information.Thanks,
Namhyung
Quoted 12 lines · Namhyung Kim
Quoted 4 lines
Dapeng's comment should cover replace the comment /* Followed by topdown events. */ but there are other things amiss. I'm thinking of something like: "slots,cycles,{instructions,topdown-be-bound}" the topdown-be-bound should get sorted and grouped with slots, but cycles
Quoted 8 lines
and instructions have no reason to be reordered, so do we end up with slots, instructions and topdown-be-bound being grouped with cycles sitting ungrouped in the middle of the evlist? I believe there are assumptions that grouped evsels are adjacent in the evlist, not least in:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2106
Does cycles instructions end up being broken out of a group in this case? Which feels like the case the code was trying to avoid.
Quoted 6 lines
I got this:
$ sudo ./perf record -a -e "slots,cycles,{instructions,topdown-be-bound}" true Error: The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (topdown-be-bound). "dmesg | grep -i perf" may provide additional information.
To be honest, I think the "slots,cycles,{instructions,topdown-be-bound}" is a meaningless case. Why a user wants to group instructions and topdown events, but leave the slots out of the group?
There could be hundreds of different combinations caused by the perf metrics mess. I don't think the re-ordering code should/can fix all of them.
For the case which the re-ordering cannot cover (like above), an error out is acceptable. So the end user can update their command to a more meaningful format, either {slots,cycles,instructions,topdown-be-bound} or {slots,topdown-be-bound},cycles,instructions still works.
I think what the patch set really fixed is the failure of sample read with perf metrics. Without the patch set, it never works no matter how you change the order of the events.
A better ordering is just a nice to have feature. If perf cannot provides a perfect re-ordering, I think an error out is also OK.
Thanks,
Kan
Ian Rogers
reviewerQuoted 23 lines · Kan
Quoted 12 lines · Namhyung Kim
Quoted 4 lines
Dapeng's comment should cover replace the comment /* Followed by topdown events. */ but there are other things amiss. I'm thinking of something like: "slots,cycles,{instructions,topdown-be-bound}" the topdown-be-bound should get sorted and grouped with slots, but cycles
Quoted 8 lines
and instructions have no reason to be reordered, so do we end up with slots, instructions and topdown-be-bound being grouped with cycles sitting ungrouped in the middle of the evlist? I believe there are assumptions that grouped evsels are adjacent in the evlist, not least in:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2106
Does cycles instructions end up being broken out of a group in this case? Which feels like the case the code was trying to avoid.Quoted 6 lines
I got this:
$ sudo ./perf record -a -e "slots,cycles,{instructions,topdown-be-bound}" true Error: The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (topdown-be-bound). "dmesg | grep -i perf" may provide additional information.
Quoted 5 lines
To be honest, I think the "slots,cycles,{instructions,topdown-be-bound}" is a meaningless case. Why a user wants to group instructions and topdown events, but leave the slots out of the group?
There could be hundreds of different combinations caused by the perf metrics mess. I don't think the re-ordering code should/can fix all of them.
I'm happy with better code and things don't need to be perfect. Can we fix the comments though? It'd be nice to also include that some things are going to be broken. I can imagine groups with topdown events but without slots, for example we group events in metrics and in tma_retiring we add "0 * tma_info_thread_slots" to the metric so that we get a slots event. If the multiply were optimized away as redundant then we'd have a topdown group without slots, we could pick up slots and other events from other metrics.
Quoted 4 lines
For the case which the re-ordering cannot cover (like above), an error out is acceptable. So the end user can update their command to a more meaningful format, either {slots,cycles,instructions,topdown-be-bound} or {slots,topdown-be-bound},cycles,instructions still works.
Perhaps we can add an arch error path that could help more for topdown events given they are a particular pain to open.
Quoted 5 lines
I think what the patch set really fixed is the failure of sample read with perf metrics. Without the patch set, it never works no matter how you change the order of the events.
A better ordering is just a nice to have feature. If perf cannot provides a perfect re-ordering, I think an error out is also OK.
Agreed, we don't need to fix everything and focussing on the common use cases makes sense.
Thanks,
Ian
Namhyung Kim
maintainerQuoted 30 lines · Ian Rogers
Quoted 23 lines · Kan
Quoted 12 lines · Namhyung Kim
Quoted 4 lines
Dapeng's comment should cover replace the comment /* Followed by topdown events. */ but there are other things amiss. I'm thinking of something like: "slots,cycles,{instructions,topdown-be-bound}" the topdown-be-bound should get sorted and grouped with slots, but cycles
Quoted 8 lines
and instructions have no reason to be reordered, so do we end up with slots, instructions and topdown-be-bound being grouped with cycles sitting ungrouped in the middle of the evlist? I believe there are assumptions that grouped evsels are adjacent in the evlist, not least in:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2106
Does cycles instructions end up being broken out of a group in this case? Which feels like the case the code was trying to avoid.Quoted 6 lines
I got this:
$ sudo ./perf record -a -e "slots,cycles,{instructions,topdown-be-bound}" true Error: The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (topdown-be-bound). "dmesg | grep -i perf" may provide additional information.Quoted 5 lines
To be honest, I think the "slots,cycles,{instructions,topdown-be-bound}" is a meaningless case. Why a user wants to group instructions and topdown events, but leave the slots out of the group?
There could be hundreds of different combinations caused by the perf metrics mess. I don't think the re-ordering code should/can fix all of them.
Quoted 3 lines
I'm happy with better code and things don't need to be perfect. Can we fix the comments though? It'd be nice to also include that some things are going to be broken. I can imagine groups with topdown events but
Can you please propose something?
Thanks,
Namhyung
Quoted 5 lines
without slots, for example we group events in metrics and in
tma_retiringwe add "0 *tma_info_thread_slots" to the metric so that we get a slots event. If the multiply were optimized away as redundant then we'd have a topdown group without slots, we could pick up slots and other events from other metrics.
Quoted 49 lines
-----Original Message-----
From: Ian Rogers <[email protected]>
Sent: Friday, October 4, 2024 7:36 AM
To: Liang, Kan <[email protected]>
Cc: Namhyung Kim <[email protected]>; Peter Zijlstra <[email protected]>; Ingo Molnar <[email protected]>; Arnaldo Carvalho de Melo <[email protected]>; Hunter, Adrian <[email protected]>; Alexander Shishkin <[email protected]>; Dapeng Mi <[email protected]>; [email protected]; linux- [email protected]; Yongwei Ma <[email protected]>; Mi, Dapeng1 <[email protected]>
Subject: Re: [Patch v5 0/6] Bug fixes on topdown events reorderingQuoted 18 lines · Kan
Quoted 14 lines · Namhyung Kim
Quoted 14 lines
Dapeng's comment should cover replace the comment /* Followed by topdown events. */ but there are other things amiss. I'm thinking of something like: "slots,cycles,{instructions,topdown-be-bound}" the topdown-be-bound should get sorted and grouped with slots, but cycles and instructions have no reason to be reordered, so do we end up with slots, instructions and topdown-be-bound being grouped with cycles sitting ungrouped in the middle of the evlist? I believe there are assumptions that grouped evsels are adjacent in the evlist, not least
in:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-nex
t.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2106 Does cycles instructions end up being broken out of a group in this case? Which feels like the case the code was trying to avoid.Quoted 3 lines
Quoted 3 lines
I got this:
$ sudo ./perf record -a -e "slots,cycles,{instructions,topdown-be-bound}"true
Quoted 2 lines
Quoted 2 lines
Error:
Thesys_perf_event_open()syscall returned with 22 (Invalid argument) forevent (topdown-be-bound).
Quoted 7 lines
Quoted 1 line
"dmesg | grep -i perf" may provide additional information.
To be honest, I think the "slots,cycles,{instructions,topdown-be-bound}" is a meaningless case. Why a user wants to group instructions and topdown events, but leave the slots out of the group?
There could be hundreds of different combinations caused by the perf metrics mess. I don't think the re-ordering code should/can fix all of them.
Quoted 7 lines
I'm happy with better code and things don't need to be perfect. Can we fix the comments though? It'd be nice to also include that some things are going to be broken. I can imagine groups with topdown events but without slots, for example we group events in metrics and in
tma_retiringwe add "0 *tma_info_thread_slots" to the metric so that we get a slots event. If the multiply were optimized away as redundant then we'd have a topdown group without slots, we could pick up slots and other events from other metrics.
I don't think this patch would break any current regroup case. It just blocks to move topdown metrics event if they are already in same group with slot events. We can see same error for this event combination "slots,cycles,{instructions,topdown-be-bound}" in the original code.
As Kan said, there could be hundreds of topdown metrics combinations, it's unrealistic to cover all these combinations just using sorting, and even it can be done, the comparator would become much complicated and hard to maintain.
I think we'd better just maintain several commonly used regroup cases, it would be fine to raise an error for these unsupported combinations as long as error message is clear enough.
Ian, I don't fully understand your words, could you please give an example? Thanks.
Quoted 20 lines
Quoted 5 lines
For the case which the re-ordering cannot cover (like above), an error out is acceptable. So the end user can update their command to a more meaningful format, either {slots,cycles,instructions,topdown-be-bound} or {slots,topdown-be-bound},cycles,instructions still works.
Perhaps we can add an arch error path that could help more for topdown events given they are a particular pain to open.
Quoted 5 lines
I think what the patch set really fixed is the failure of sample read with perf metrics. Without the patch set, it never works no matter how you change the order of the events.
A better ordering is just a nice to have feature. If perf cannot provides a perfect re-ordering, I think an error out is also OK.Agreed, we don't need to fix everything and focussing on the common use cases makes sense.
Thanks,
Ian
Ian Rogers
reviewerQuoted 58 lines · On Mon, Oct 7, 2024 at 7:52 PM Mi, Dapeng1 <[email protected]> wrote:
Quoted 50 lines
-----Original Message-----
From: Ian Rogers <[email protected]>
Sent: Friday, October 4, 2024 7:36 AM
To: Liang, Kan <[email protected]>
Cc: Namhyung Kim <[email protected]>; Peter Zijlstra <[email protected]>; Ingo Molnar <[email protected]>; Arnaldo Carvalho de Melo <[email protected]>; Hunter, Adrian <[email protected]>; Alexander Shishkin <[email protected]>; Dapeng Mi <[email protected]>; [email protected]; linux- [email protected]; Yongwei Ma <[email protected]>; Mi, Dapeng1 <[email protected]>
Subject: Re: [Patch v5 0/6] Bug fixes on topdown events reorderingQuoted 18 lines · Kan
Quoted 14 lines · Namhyung Kim
Quoted 14 lines
Dapeng's comment should cover replace the comment /* Followed by topdown events. */ but there are other things amiss. I'm thinking of something like: "slots,cycles,{instructions,topdown-be-bound}" the topdown-be-bound should get sorted and grouped with slots, but cycles and instructions have no reason to be reordered, so do we end up with slots, instructions and topdown-be-bound being grouped with cycles sitting ungrouped in the middle of the evlist? I believe there are assumptions that grouped evsels are adjacent in the evlist, not least
in:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-nex
t.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2106 Does cycles instructions end up being broken out of a group in this case? Which feels like the case the code was trying to avoid.Quoted 3 lines
Quoted 3 lines
I got this:
$ sudo ./perf record -a -e "slots,cycles,{instructions,topdown-be-bound}"true
Quoted 2 lines
Quoted 2 lines
Error:
Thesys_perf_event_open()syscall returned with 22 (Invalid argument) forevent (topdown-be-bound).
Quoted 7 lines
Quoted 1 line
"dmesg | grep -i perf" may provide additional information.
To be honest, I think the "slots,cycles,{instructions,topdown-be-bound}" is a meaningless case. Why a user wants to group instructions and topdown events, but leave the slots out of the group?
There could be hundreds of different combinations caused by the perf metrics mess. I don't think the re-ordering code should/can fix all of them.Quoted 7 lines
I'm happy with better code and things don't need to be perfect. Can we fix the comments though? It'd be nice to also include that some things are going to be broken. I can imagine groups with topdown events but without slots, for example we group events in metrics and in
tma_retiringwe add "0 *tma_info_thread_slots" to the metric so that we get a slots event. If the multiply were optimized away as redundant then we'd have a topdown group without slots, we could pick up slots and other events from other metrics.
Quoted 7 lines
I don't think this patch would break any current regroup case. It just blocks to move topdown metrics event if they are already in same group with slot events. We can see same error for this event combination "slots,cycles,{instructions,topdown-be-bound}" in the original code.
As Kan said, there could be hundreds of topdown metrics combinations, it's unrealistic to cover all these combinations just using sorting, and even it can be done, the comparator would become much complicated and hard to maintain.
I think we'd better just maintain several commonly used regroup cases, it would be fine to raise an error for these unsupported combinations as long as error message is clear enough.
Ian, I don't fully understand your words, could you please give an example? Thanks.
So in the comparator I think most people won't understand the list of cases that are supported and those that are not supported:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/arch/x86/util/evlist.c?h=perf-tools-next#n35
The groups usually come from metrics and to work around issues there are constraints on those that may or may not group events. This could yield situations that don't work given the cases you list, but I don't think current metrics will violate this. We do test metrics individually but not together as part of perf test.
Thanks,
Ian
Quoted 20 lines
Quoted 20 lines
Quoted 5 lines
For the case which the re-ordering cannot cover (like above), an error out is acceptable. So the end user can update their command to a more meaningful format, either {slots,cycles,instructions,topdown-be-bound} or {slots,topdown-be-bound},cycles,instructions still works.
Perhaps we can add an arch error path that could help more for topdown events given they are a particular pain to open.
Quoted 5 lines
I think what the patch set really fixed is the failure of sample read with perf metrics. Without the patch set, it never works no matter how you change the order of the events.
A better ordering is just a nice to have feature. If perf cannot provides a perfect re-ordering, I think an error out is also OK.Agreed, we don't need to fix everything and focussing on the common use cases makes sense.
Thanks,
Ian
Quoted 80 lines
-----Original Message-----
From: Ian Rogers <[email protected]>
Sent: Friday, October 4, 2024 5:26 AM
To: Liang, Kan <[email protected]>
Cc: Namhyung Kim <[email protected]>; Peter Zijlstra <[email protected]>; Ingo Molnar <[email protected]>; Arnaldo Carvalho de Melo <[email protected]>; Hunter, Adrian <[email protected]>; Alexander Shishkin <[email protected]>; Dapeng Mi <[email protected]>; [email protected]; linux- [email protected]; Yongwei Ma <[email protected]>; Mi, Dapeng1 <[email protected]>
Subject: Re: [Patch v5 0/6] Bug fixes on topdown events reorderingQuoted 6 lines · Kan
Quoted 2 lines · Namhyung Kim
Quoted 2 lines
Quoted 2 lines
If the algorithms cannot be changed, can you please give some suggestions, especially for the sample read failure?
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
So this is symmetric:
``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) return 1; ```That is were lhs and rhs swapped then you'd get the expected comparison
order.
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ``` Is symmetric as well. ``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ```(what this patch does) is not symmetric as the group leader impacts the greater-than case but not the less-than case.
It is not uncommon to see in a sort function:
```
if (cmp(a, b)<= 0) {
assert(cmp(b,a) >= 0 && "check for unstable/broken compare
functions"); ```Quoted 8 lines
tools/perf/arch/x86/util/evlist.c int arch_evlist__cmp(const struct evsel *lhs, const
Quoted 8 lines
I see. So are you proposing this?
tools/perf/arch/x86/util/evlist.c +0
-0@@ -70,7 +70,8 @@ int arch_evlist__cmp(const struct evsel *lhs, conststruct evsel *rhs)
Quoted 4 lines
tools/perf/arch/x86/util/evlist.c int arch_evlist__cmp(const struct evsel *lhs, const struct evsel *rhs)
Quoted 4 lines
tools/perf/arch/x86/util/evlist.c +0
-1if (arch_is_topdown_slots (rhs))return 1 ;/* Followed by topdown events. */ - if (arch_is_topdown_metrics(lhs)&& !arch_is_topdown_metrics(rhs))
Quoted 1 line
tools/perf/arch/x86/util/evlist.c int arch_evlist__cmp(const struct evsel *lhs, const · line 73
Quoted 1 line
tools/perf/arch/x86/util/evlist.c +1
-0+ if (arch_is_topdown_metrics(lhs)&& !arch_is_topdown_metrics(rhs) &&
Quoted 6 lines
Quoted 6 lines
tools/perf/arch/x86/util/evlist.c +1
-0+ lhs->core.leader != rhs->core.leader)return -1 ;/* * Move topdown events forward only when topdown events
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Dapeng and Kan, can you verify if it's ok? My quick tests look ok.
I verified the above change. It works well.
Tested-by Kan Liang <[email protected]>Dapeng's comment should cover replace the comment /* Followed by topdown events. */ but there are other things amiss. I'm thinking of something like:
Thanks. I would change the comments.
Quoted 13 lines
"slots,cycles,{instructions,topdown-be-bound}" the topdown-be-bound should get sorted and grouped with slots, but cycles and instructions have no reason to be reordered, so do we end up with slots, instructions and topdown-be-bound being grouped with cycles sitting ungrouped in the middle of the evlist? I believe there are assumptions that grouped evsels are adjacent in the evlist, not least in:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2106 Does cycles instructions end up being broken out of a group in this case? Which feels like the case the code was trying to avoid.Thanks,
Ian
Quoted 74 lines
-----Original Message-----
From: Liang, Kan <[email protected]>
Sent: Friday, October 4, 2024 3:45 AM
To: Namhyung Kim <[email protected]>; Ian Rogers <[email protected]>
Cc: Peter Zijlstra <[email protected]>; Ingo Molnar <[email protected]>; Arnaldo Carvalho de Melo <[email protected]>; Hunter, Adrian <[email protected]>; Alexander Shishkin <[email protected]>; Dapeng Mi <[email protected]>; [email protected]; linux- [email protected]; Yongwei Ma <[email protected]>; Mi, Dapeng1 <[email protected]>
Subject: Re: [Patch v5 0/6] Bug fixes on topdown events reorderingQuoted 10 lines · Namhyung Kim
Quoted 10 lines
Quoted 2 lines
If the algorithms cannot be changed, can you please give some suggestions, especially for the sample read failure?
So this is symmetric:
``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs)) return 1; ```That is were lhs and rhs swapped then you'd get the expected comparison
order.
tools/perf/arch/x86/util/evlist.c int arch_evlist__cmp(const struct evsel *lhs, const struct
Quoted 32 lines
Quoted 16 lines
``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ``` Is symmetric as well. ``` if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)) return -1; if (!arch_is_topdown_metrics(lhs) && arch_is_topdown_metrics(rhs) && lhs->core.leader != rhs->core.leader) return 1; ```Quoted 8 lines
(what this patch does) is not symmetric as the group leader impacts the greater-than case but not the less-than case.
It is not uncommon to see in a sort function:
```
if (cmp(a, b)<= 0) {
assert(cmp(b,a) >= 0 && "check for unstable/broken compare
functions"); ```I see. So are you proposing this?
tools/perf/arch/x86/util/evlist.c +0
-0@@ -70,7 +70,8 @@ int arch_evlist__cmp(const struct evsel *lhs, const structevsel *rhs)
tools/perf/arch/x86/util/evlist.c int arch_evlist__cmp(const struct evsel *lhs, const struct evsel *rhs)
Quoted 5 lines
tools/perf/arch/x86/util/evlist.c +1
-1if (arch_is_topdown_slots (rhs))return 1 ;/* Followed by topdown events. */ -if (arch_is_topdown_metrics (lhs) && !arch_is_topdown_metrics (rhs)) + if (arch_is_topdown_metrics(lhs) && !arch_is_topdown_metrics(rhs)&&
Quoted 6 lines
tools/perf/arch/x86/util/evlist.c +1
-0+ lhs->core.leader != rhs->core.leader)return -1 ;/* * Move topdown events forward only when topdown events
Quoted 5 lines
Quoted 1 line
Dapeng and Kan, can you verify if it's ok? My quick tests look ok.
I verified the above change. It works well.
Tested-by Kan Liang <[email protected]>
Since the linux.intel.com mail server is down, I have to use the intel.com mail server (outlook client) to respond this mail. The mail format may vary. Sorry for this.
Thanks Kan for testing this.
Ian, thanks for pointing that the comparators are not symmetrical. I agree it would lead to an inconsistent sorting results if changing comparing condition from "cmp(a, b) < 0" to "cmp(b, a) > 0" or vice versa, but limit to some certain sort library like perf-tool, the sorting result should be still fixed, right?
Anyway, I would provide a new patch to make the comparators are symmetrical. Thanks.
Quoted 3 lines
Thanks,
Kan