Event reordering regression for software events
Hi Ian,
The current post 6.3 linux-perf-next causes some fairly drastic reordering in output for some event combinations. I bisected it down to your patch:
commit 347c2f0a0988c59c402148054ef54648853fa669
Author: Ian Rogers <[email protected]> Date: Sat Mar 11 18:15:40 2023 -0800
perf parse-events: Sort and group parsed events
This change is intended to be a no-op for most current cases, the default sort order is the order the events were parsed. Where it varies is in how groups are handled. Previously an uncore and core event that are grouped would most often cause the group to be removed:
I currently only have a large test case, but for the test case [1] below most of the "dummy"s and some other software events end up in a row and also in the wrong place, instead of the expected order.
good:
...
986476553;ns;duration_time;986476553;100.00;;
0;;dummy;983952237;100.00;;
0;;cs:u;983952237;100.00;;
16532;;minor-faults:u;983952237;100.00;;
0;;major-faults:u;983952237;100.00;;
0;;migrations:u;983952237;100.00;;
3532509496;;cycles:u;23999092;2.00;;
....
bad:
1010119571;ns;duration_time;1010119571;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;cs:u [software];1007748753;100.00;; 16496;;minor-faults:u [software];1007748753;100.00;; 0;;major-faults:u [software];1007748753;100.00;; 0;;migrations:u [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 3603193382;;cycles:u;23996241;2.00;; 2277091922;;cpu/event=0x0,umask=0x3/u;23996241;2.00;; 4182126406;;cpu/event=0xc2,umask=0x2/u;23996241;2.00;; 4364677170;;cpu/event=0xc0,umask=0x0/u;23996241;2.00;;
Unfortunately this totally breaks toplev. It needs to have the dummies in the right location.`
Another problem is that it also now adds [software] to lots of software events, which of course also breaks any parsing tools. I haven't bisected that too, but it needs fixing too.
I must say using perf recently is frustrating with all the regressions! Please be a bit more careful with compatibility.
-Andi
[1]
$PERF stat -x\; --no-merge -o xxx -e '{cpu/event=0xc0,umask=0x0/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x0,umask=0x3/u,cpu/event=0x3c,umask=0x2/u,cpu/event=0x3c,umask=0x1/u,cpu/event=0xb1,umask=0x1/u,cpu/event=0xb1,umask=0x2,cmask=1/u},msr/tsc/,duration_time,dummy,uncore_imc/event=0x4,umask=0x3/,uncore_imc/event=0x4,umask=0xc/,cs:u,minor-faults:u,major-faults:u,migrations:u,{cycles:u,cpu/event=0x0,umask=0x3/u,cpu/event=0xc2,umask=0x2/u,cpu/event=0xc0,umask=0x0/u,cpu/event=0xc4,umask=0x40/u,cpu/event=0xb1,umask=0x1/u,cpu/event=0xe,umask=0x1/u},{cpu/event=0xc2,umask=0x2/u,cpu/event=0xc2,umask=0x2,cmask=1/u,cpu/event=0x3c,umask=0x0/ku,cpu/event=0xc0,umask=0x0/ku},{cpu/event=0x79,umask=0x8/u,cpu/event=0xa8,umask=0x1/u,cpu/event=0x79,umask=0x4/u,cpu/event=0x79,umask=0x30/u,cpu/event=0xc0,umask=0x0/u,cpu/event=0x3c,umask=0x0/u},{cpu/event=0x10,umask=0x20/u,cpu/event=0x10,umask=0x80/u,cpu/event=0x10,umask=0x10/u,cpu/event=0x10,umask=0x40/u},{cpu/event=0x11,umask=0x2/u,cpu/event=0x11,umask=0x1/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x3c,umask=0x2/u,cpu/event=0x3c,umask=0x1/u},{cpu/event=0x10,umask=0x20/u,cpu/event=0x10,umask=0x80/u,cpu/event=0x10,umask=0x10/u,cpu/event=0x10,umask=0x40/u,cpu/event=0x3c,umask=0x0/u},{cpu/event=0x3c,umask=0x0/ku,cpu/event=0x3c,umask=0x1/u,cpu/event=0xc2,umask=0x2/u,cpu/event=0xd,umask=0x3,cmask=1/u},{cpu/event=0x9c,umask=0x1/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x3c,umask=0x2/u,cpu/event=0x3c,umask=0x1/u,cpu/event=0xc2,umask=0x2/u},{cpu/event=0xe,umask=0x1/u,cpu/event=0xc2,umask=0x2/u,cpu/event=0xd,umask=0x3,cmask=1/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x3c,umask=0x2/u},{cpu/event=0x9c,umask=0x1/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x3c,umask=0x2/u,cpu/event=0x3c,umask=0x1/u,cpu/event=0xe,umask=0x1/u},{cpu/event=0x3c,umask=0x0/u,cpu/event=0x9c,umask=0x1,cmask=4/u,cpu/event=0x3c,umask=0x2/u,cpu/event=0x3c,umask=0x1/u,cpu/event=0x9c,umask=0x1/u,cpu/event=0xc0,umask=0x0/u},{cpu/event=0xc2,umask=0x2/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x3c,umask=0x2/u,cpu/event=0x3c,umask=0x1/u,cpu/event=0xd,umask=0x3,cmask=1/u},{cpu/event=0xc2,umask=0x2/u,cpu/event=0xe,umask=0x1/u,cpu/event=0x79,umask=0x30/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x3c,umask=0x2/u},{cpu/event=0xe,umask=0x1/u,cpu/event=0xc2,umask=0x2/u,cpu/event=0xd,umask=0x3,cmask=1/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x3c,umask=0x2/u},{cpu/event=0xc5,umask=0x0/u,cpu/event=0xc3,umask=0x1,edge=1,cmask=1/u,cpu/event=0x3c,umask=0x1/u},dummy,{cpu/event=0xe,umask=0x1/u,cpu/event=0xc2,umask=0x2/u,cpu/event=0xd,umask=0x3,cmask=1/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x3c,umask=0x2/u},{cpu/event=0x9c,umask=0x1/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x3c,umask=0x2/u,cpu/event=0x3c,umask=0x1/u,cpu/event=0xe,umask=0x1/u},dummy,{cpu/event=0xa3,umask=0x6,cmask=6/u,cpu/event=0xa2,umask=0x8/u,cpu/event=0xa3,umask=0x4,cmask=4/u,cpu/event=0xb1,umask=0x1,cmask=1/u},{cpu/event=0xb1,umask=0x1,cmask=3/u,cpu/event=0xb1,umask=0x1,cmask=2/u,cpu/event=0xc0,umask=0x0/u,cpu/event=0x5e,umask=0x1/u,cpu/event=0x9c,umask=0x1,cmask=4/u},{cpu/event=0x9c,umask=0x1/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x3c,umask=0x2/u,cpu/event=0x3c,umask=0x1/u,cpu/event=0xe,umask=0x1/u},{cpu/event=0xc2,umask=0x2/u,cpu/event=0xd,umask=0x3,cmask=1/u},dummy,dummy,emulation-faults,dummy,{cpu/event=0xa3,umask=0x6,cmask=6/u,cpu/event=0xa2,umask=0x8/u,cpu/event=0xa3,umask=0x4,cmask=4/u,cpu/event=0xb1,umask=0x1,cmask=1/u},{cpu/event=0xb1,umask=0x1,cmask=3/u,cpu/event=0xb1,umask=0x1,cmask=2/u,cpu/event=0xc0,umask=0x0/u,cpu/event=0x5e,umask=0x1/u,cpu/event=0x9c,umask=0x1,cmask=4/u},{cpu/event=0x79,umask=0x30,edge=1,cmask=1/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x87,umask=0x1/u,cpu/event=0xab,umask=0x2/u,cpu/event=0xa2,umask=0x8/u},{cpu/event=0x85,umask=0x10/u,cpu/event=0x85,umask=0x4/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x8,umask=0x10/u,cpu/event=0x8,umask=0x4/u},{cpu/event=0xc5,umask=0x0/u,cpu/event=0xc3,umask=0x1,edge=1,cmask=1/u,cpu/event=0xe6,umask=0x1f/u,cpu/event=0x3c,umask=0x0/u},{cpu/event=0xd1,umask=0x4/u,cpu/event=0xd1,umask=0x20/u,cpu/event=0xa3,umask=0x5,cmask=5/u,cpu/event=0x3c,umask=0x0/u},{cpu/event=0x14,umask=0x1/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x3c,umask=0x2/u,cpu/event=0x3c,umask=0x1/u,cpu/event=0x11,umask=0x2/u},{cpu/event=0xc2,umask=0x2/u,cpu/event=0xe,umask=0x1/u,cpu/event=0x79,umask=0x30/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x3c,umask=0x2/u},{cpu/event=0xc2,umask=0x2/u,cpu/event=0x10,umask=0x1/u,cpu/event=0xb1,umask=0x1/u,cpu/event=0x10,umask=0x20/u},{cpu/event=0x10,umask=0x80/u,cpu/event=0x10,umask=0x10/u,cpu/event=0x10,umask=0x40/u,cpu/event=0x11,umask=0x1/u},{cpu/event=0x3c,umask=0x0/u,cpu/event=0x9c,umask=0x1,cmask=4/u,cpu/event=0x3c,umask=0x2/u,cpu/event=0x3c,umask=0x1/u},dummy,{cpu/event=0xa3,umask=0x4,cmask=4/u,cpu/event=0xb1,umask=0x1,cmask=1/u,cpu/event=0xb1,umask=0x1,cmask=3/u,cpu/event=0xb1,umask=0x1,cmask=2/u},{cpu/event=0xc0,umask=0x0/u,cpu/event=0x5e,umask=0x1/u,cpu/event=0xa2,umask=0x8/u,cpu/event=0xa3,umask=0x6,cmask=6/u,cpu/event=0x3c,umask=0x0/u,cpu/event=0x60,umask=0x8,cmask=6/u},{cpu/event=0x3c,umask=0x0/u,cpu/event=0x60,umask=0x8,cmask=1/u,cpu/event=0x60,umask=0x8,cmask=6/u},{cpu/event=0xc2,umask=0x2/u,cpu/event=0x10,umask=0x1/u,cpu/event=0xb1,umask=0x1/u},{cpu/event=0x10,umask=0x20/u,cpu/event=0x10,umask=0x80/u,cpu/event=0xb1,umask=0x1/u,cpu/event=0x10,umask=0x10/u},emulation-faults,{cpu/event=0x10,umask=0x10/u,cpu/event=0x10,umask=0x40/u,cpu/event=0x11,umask=0x1/u,cpu/event=0x11,umask=0x2/u},emulation-faults,{cpu/event=0x11,umask=0x2/u,cpu/event=0x11,umask=0x1/u,cpu/event=0xb1,umask=0x1/u}'
Em Thu, Jul 13, 2023 at 01:37:00PM -0700, Andi Kleen escreveu:
Quoted 59 lines
Hi Ian,
The current post 6.3 linux-perf-next causes some fairly drastic reordering in output for some event combinations. I bisected it down to your patch:
commit
347c2f0a0988c59c402148054ef54648853fa669Author: Ian Rogers <[email protected]> Date: Sat Mar 11 18:15:40 2023 -0800perf parse-events: Sort and group parsed events
This change is intended to be a no-op for most current cases, the default sort order is the order the events were parsed. Where it varies is in how groups are handled. Previously an uncore and core event that are grouped would most often cause the group to be removed:
I currently only have a large test case, but for the test case [1] below most of the "dummy"s and some other software events end up in a row and also in the wrong place, instead of the expected order.
good:
...
986476553;ns;duration_time;986476553;100.00;;
0;;dummy;983952237;100.00;;
0;;cs:u;983952237;100.00;;
16532;;minor-faults:u;983952237;100.00;;
0;;major-faults:u;983952237;100.00;;
0;;migrations:u;983952237;100.00;;
3532509496;;cycles:u;23999092;2.00;;
....bad:
1010119571;ns;duration_time;1010119571;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;cs:u [software];1007748753;100.00;; 16496;;minor-faults:u [software];1007748753;100.00;; 0;;major-faults:u [software];1007748753;100.00;; 0;;migrations:u [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 3603193382;;cycles:u;23996241;2.00;; 2277091922;;cpu/event=0x0,umask=0x3/u;23996241;2.00;; 4182126406;;cpu/event=0xc2,umask=0x2/u;23996241;2.00;; 4364677170;;cpu/event=0xc0,umask=0x0/u;23996241;2.00;;
Unfortunately this totally breaks toplev. It needs to have the dummies in the right location.`
Quoted 6 lines
Another problem is that it also now adds [software] to lots of software events, which of course also breaks any parsing tools. I haven't bisected that too, but it needs fixing too.
I must say using perf recently is frustrating with all the regressions! Please be a bit more careful with compatibility.
That is really unfortunate and indeed perf passed thru a lot of changes trying to properly support hybrid systems and verify reference counting correctness, a necessary step to resume attempts at being multithreaded.
We have seen a growing number of 'perf test' entries being submitted, with writing shell scripts that check if the output from various command lines produce expected results.
It would really be great to have new entries that exercise what is expected by toplev so that we really are careful with compatibility.
- Arnaldo
Quoted 19 lines · Arnaldo Carvalho de Melo
Em Thu, Jul 13, 2023 at 01:37:00PM -0700, Andi Kleen escreveu:
Quoted 17 lines
Hi Ian,
The current post 6.3 linux-perf-next causes some fairly drastic reordering in output for some event combinations. I bisected it down to your patch:
commit
347c2f0a0988c59c402148054ef54648853fa669Author: Ian Rogers <[email protected]> Date: Sat Mar 11 18:15:40 2023 -0800perf parse-events: Sort and group parsed events
This change is intended to be a no-op for most current cases, the default sort order is the order the events were parsed. Where it varies is in how groups are handled. Previously an uncore and core event that are grouped would most often cause the group to be removed:
So this is a no-op for most current cases where event groups aren't used. The default order for events is the order they were inserted in. Unfortunately Intel added perf metric, aka topdown, events where we need to add or regroup events and also multiple vendors did BIG.little/hybrid. Previously we'd sort and regroup uncore events like:
perf stat -e '{uncore_imc_free_running/data_read/,uncore_imc_free_running/data_write/}'
...
to be say two groups of:
{uncore_imc_free_running_0/data_read/,uncore_imc_free_running_1/data_write/} {uncore_imc_free_running_1/data_read/,uncore_imc_free_running_1/data_write/}
Rather than how they appear after parsing of:
{uncore_imc_free_running_0/data_read/,uncore_imc_free_running_1/data_read/,uncore_imc_free_running_0/data_write/,uncore_imc_free_running_1/data_write/}
Presumably toplev already supports this. With hybrid, what is true for uncore PMUs is now true for core PMUs, so:
perf stat -e '{instructions,cycles}' ...
becomes:
{cpu_atom/instructions/,cpu_atom/cycles/}
{cpu_core/instructions/,cpu_core/cycles/}
Perf metric events require being in a group with the slots event first. There are assumptions in the perf code that sibling events are adjacent in the evlist. To fix and maintain all the requirements on the evlist we've gotten to where we are now. I've tried to make the code as simple and transparent as possible, but still fulfilling the requirements coming from perf metric events and hybrid. The logic is in parse_events__sort_events_and_fix_groups:
where the compare function evlist__cmp and evsel__compute_group_pmu_name are key. The "group" PMU name is what attempts to place all events that belong in the same group together, which include software and aux events.
Quoted 39 lines
Quoted 39 lines
I currently only have a large test case, but for the test case [1] below most of the "dummy"s and some other software events end up in a row and also in the wrong place, instead of the expected order.
good:
...
986476553;ns;duration_time;986476553;100.00;;
0;;dummy;983952237;100.00;;
0;;cs:u;983952237;100.00;;
16532;;minor-faults:u;983952237;100.00;;
0;;major-faults:u;983952237;100.00;;
0;;migrations:u;983952237;100.00;;
3532509496;;cycles:u;23999092;2.00;;
....bad:
1010119571;ns;duration_time;1010119571;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;cs:u [software];1007748753;100.00;; 16496;;minor-faults:u [software];1007748753;100.00;; 0;;major-faults:u [software];1007748753;100.00;; 0;;migrations:u [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 3603193382;;cycles:u;23996241;2.00;; 2277091922;;cpu/event=0x0,umask=0x3/u;23996241;2.00;; 4182126406;;cpu/event=0xc2,umask=0x2/u;23996241;2.00;; 4364677170;;cpu/event=0xc0,umask=0x0/u;23996241;2.00;;
Quoted 6 lines
Quoted 6 lines
Unfortunately this totally breaks toplev. It needs to have the dummies in the right location.`
Another problem is that it also now adds [software] to lots of software events, which of course also breaks any parsing tools. I haven't bisected that too, but it needs fixing too.
There's nothing to bisect, the behavior has changed. We can revert the behavior, break Intel hybrid and Intel perf metric events, but that would seem unfortunate. Tbh, the mistake here is the assumptions that toplev is making.
Quoted 8 lines
Quoted 3 lines
I must say using perf recently is frustrating with all the regressions! Please be a bit more careful with compatibility.
That is really unfortunate and indeed perf passed thru a lot of changes trying to properly support hybrid systems and verify reference counting correctness, a necessary step to resume attempts at being multithreaded.
Quoted 6 lines
We have seen a growing number of 'perf test' entries being submitted, with writing shell scripts that check if the output from various command lines produce expected results.
It would really be great to have new entries that exercise what is expected by toplev so that we really are careful with compatibility.
We are careful with compatibility, in fact we've written a number of tests to ensure the json and CSV output doesn't regress. Kan recently added some tests for text output, but relying on it is likely a bad idea. I've heard >1 proposal to do nesting of metrics and the like, Kan recently removed events and put metric names in the text output instead, even hiding events that have been programmed.. basically relying on text output is wrong and toplev is broken in this regard.
Fixing hybrid and perf metric events weren't problems of my invention, nor were they done with a casual disregard for perf users and no care about breaking them. That said, testing can always be better so why not add toplev's tests to tools/perf/tests/shell ? This way I can only break things if I ignore a test regression.
Thanks,
Ian
Quoted 2 lines
- Arnaldo
Quoted 16 lines · Ian Rogers
Quoted 14 lines · Arnaldo Carvalho de Melo
Em Thu, Jul 13, 2023 at 01:37:00PM -0700, Andi Kleen escreveu:
Quoted 12 lines
Hi Ian,
The current post 6.3 linux-perf-next causes some fairly drastic reordering in output for some event combinations. I bisected it down to your patch:
commit
347c2f0a0988c59c402148054ef54648853fa669Author: Ian Rogers <[email protected]> Date: Sat Mar 11 18:15:40 2023 -0800perf parse-events: Sort and group parsed events
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
This change is intended to be a no-op for most current cases, the default sort order is the order the events were parsed. Where it varies is in how groups are handled. Previously an uncore and core event that are grouped would most often cause the group to be removed:
So this is a no-op for most current cases where event groups aren't used. The default order for events is the order they were inserted in.
This doesn't seem to be what happens because the patch reorders top-level events outside group. (like all the top level "dummy"ies)
But this commit broke things that are completely unrelated to hybrid and uncore, just plain software events not being in a group. There's no reason to reorder those.
Basically anything you do for hybrid and topdown shouldn't affect anything else. That's the basic rule.
Quoted 7 lines
Perf metric events require being in a group with the slots event first. There are assumptions in the perf code that sibling events are adjacent in the evlist. To fix and maintain all the requirements on the evlist we've gotten to where we are now. I've tried to make the code as simple and transparent as possible, but still fulfilling the requirements coming from perf metric events and hybrid. The logic is in
parse_events__sort_events_and_fix_groups:
This random reordering approach is just not viable. I explained this before. If you have a tool which generates the same event in multiple instances (for which there are many valid reasons related to multiplexing accuracy), then it absolutely cannot handle random reordering. The tool doesn't know which instance of its event is where if it's not in the same position.
If you find you need reordering for your purposes and you know you don't rely on ordering make it some kind of opt-in. But forcing it on all perf users doesn't work.
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
wrong place, instead of the expected order.
good:
...
986476553;ns;duration_time;986476553;100.00;;
0;;dummy;983952237;100.00;;
0;;cs:u;983952237;100.00;;
16532;;minor-faults:u;983952237;100.00;;
0;;major-faults:u;983952237;100.00;;
0;;migrations:u;983952237;100.00;;
3532509496;;cycles:u;23999092;2.00;;
....bad:
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
1010119571;ns;duration_time;1010119571;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;cs:u [software];1007748753;100.00;; 16496;;minor-faults:u [software];1007748753;100.00;; 0;;major-faults:u [software];1007748753;100.00;; 0;;migrations:u [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;;
Quoted 8 lines
Quoted 8 lines
0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;;
Please explain how this insane order can make sense even with hybrid or metrics.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
3603193382;;cycles:u;23996241;2.00;; 2277091922;;cpu/event=0x0,umask=0x3/u;23996241;2.00;; 4182126406;;cpu/event=0xc2,umask=0x2/u;23996241;2.00;; 4364677170;;cpu/event=0xc0,umask=0x0/u;23996241;2.00;;
Unfortunately this totally breaks toplev. It needs to have the dummies in the right location.`
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Another problem is that it also now adds [software] to lots of software events, which of course also breaks any parsing tools. I haven't bisected that too, but it needs fixing too.
There's nothing to bisect, the behavior has changed. We can revert the behavior, break Intel hybrid and Intel perf metric events, but that
I've been using perf metrics for many years without problems without this patch.
I know hybrid has some issues that need to be fixed, but that absolutely cannot break basic properties like having a parseable output format.
Quoted 2 lines
would seem unfortunate. Tbh, the mistake here is the assumptions that toplev is making.
It's not an random assumption, it's the only way to parse perf output uniquely.
Please make it an option. But this cannot stay this way unconditionally if perf stat is supposed to stay a tool which can be used from other programs.
Quoted 11 lines
Quoted 8 lines
correctness, a necessary step to resume attempts at being multithreaded.
We have seen a growing number of 'perf test' entries being submitted, with writing shell scripts that check if the output from various command lines produce expected results.
It would really be great to have new entries that exercise what is expected by toplev so that we really are careful with compatibility.
We are careful with compatibility, in fact we've written a number of tests to ensure the json and CSV output doesn't regress. Kan recently
The tests are super basic and don't seem to cover any even moderately complex cases. Yes it's great that you wrote tests, but given the poor coverage you really cannot rely on just a few basic unit tests here.
Given that, and the wide usage of perf and the existence of Hyrum's law the only safe way is to avoid changing default behaviors. If you want some behavior change make it opt-in.
Quoted 3 lines
Kan recently removed events and put metric names in the text output instead, even hiding events that have been programmed.. basically relying on text output is wrong and toplev is broken in this regard.
I missed your explanation how toplev is supposed to parse the output then instead?
Quoted 5 lines
Fixing hybrid and perf metric events weren't problems of my invention, nor were they done with a casual disregard for perf users and no care about breaking them. That said, testing can always be better so why not add toplev's tests to
tools/perf/tests/shell? This way I can only break things if I ignore a test regression.
toplev generates the perf command lines dynamically and they are highly dependent on the current system. I don't see a simple way to make independent tests, unless you want a big case statement for existing systems, or add most of toplev.
I guess could add some script that is the equivalent of
git clone https://github.com/andikleen/pmu-tools
cd pmu-tools
PERF=/point/to/new/perf/binary ./tl-tester
if that helps? The toplev test suite has reasonably good coverage.
-Andi
Quoted 2 lines
It would really be great to have new entries that exercise what is expected by toplev so that we really are careful with compatibility.
It is unfortunately difficult to do, see my reply to Ian.
But if you have automated checking tests perhays you could run my test suite too?
git clone https://github.com/andikleen/pmu-tools
cd pmu-tools
PERF=/new/perf ./tl-tester
-Andi
Quoted 31 lines · Andi Kleen
Quoted 16 lines · Ian Rogers
Quoted 14 lines · Arnaldo Carvalho de Melo
Em Thu, Jul 13, 2023 at 01:37:00PM -0700, Andi Kleen escreveu:
Quoted 12 lines
Hi Ian,
The current post 6.3 linux-perf-next causes some fairly drastic reordering in output for some event combinations. I bisected it down to your patch:
commit
347c2f0a0988c59c402148054ef54648853fa669Author: Ian Rogers <[email protected]> Date: Sat Mar 11 18:15:40 2023 -0800perf parse-events: Sort and group parsed events
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
This change is intended to be a no-op for most current cases, the default sort order is the order the events were parsed. Where it varies is in how groups are handled. Previously an uncore and core event that are grouped would most often cause the group to be removed:
So this is a no-op for most current cases where event groups aren't used. The default order for events is the order they were inserted in.
This doesn't seem to be what happens because the patch reorders top-level events outside group. (like all the top level "dummy"ies)
Quoted 6 lines
But this commit broke things that are completely unrelated to hybrid and uncore, just plain software events not being in a group. There's no reason to reorder those.
Basically anything you do for hybrid and topdown shouldn't affect anything else. That's the basic rule.
It is not clear that modifying the order of events in the evlist qualifies as a breakage, and it is necessary because of complex kernel APIs that particularly in the case of perf metric events, Intel were responsible for.
Quoted 15 lines
Quoted 7 lines
Perf metric events require being in a group with the slots event first. There are assumptions in the perf code that sibling events are adjacent in the evlist. To fix and maintain all the requirements on the evlist we've gotten to where we are now. I've tried to make the code as simple and transparent as possible, but still fulfilling the requirements coming from perf metric events and hybrid. The logic is in
parse_events__sort_events_and_fix_groups:This random reordering approach is just not viable. I explained this before. If you have a tool which generates the same event in multiple instances (for which there are many valid reasons related to multiplexing accuracy), then it absolutely cannot handle random reordering. The tool doesn't know which instance of its event is where if it's not in the same position.
Quoted 3 lines
If you find you need reordering for your purposes and you know you don't rely on ordering make it some kind of opt-in. But forcing it on all perf users doesn't work.
As I keep explaining, the necessity has come from Intel.
Quoted 33 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
wrong place, instead of the expected order.
good:
...
986476553;ns;duration_time;986476553;100.00;;
0;;dummy;983952237;100.00;;
0;;cs:u;983952237;100.00;;
16532;;minor-faults:u;983952237;100.00;;
0;;major-faults:u;983952237;100.00;;
0;;migrations:u;983952237;100.00;;
3532509496;;cycles:u;23999092;2.00;;
....bad:
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
1010119571;ns;duration_time;1010119571;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;cs:u [software];1007748753;100.00;; 16496;;minor-faults:u [software];1007748753;100.00;; 0;;major-faults:u [software];1007748753;100.00;; 0;;migrations:u [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;;
Quoted 8 lines
Quoted 8 lines
0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;;
Quoted 2 lines
Please explain how this insane order can make sense even with hybrid or metrics.
Tbh, I'm not sure what you are doing. linux-next is on 6.5 but you mention 6.3. You don't list a platform for testing on. I'm not even sure if this is a sincere bug report or just a rant that things have changed. I've hopefully motivated and explained why things changed, pointing to the exact location in the code that the changes in the evlist occur.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 12 lines
3603193382;;cycles:u;23996241;2.00;; 2277091922;;cpu/event=0x0,umask=0x3/u;23996241;2.00;; 4182126406;;cpu/event=0xc2,umask=0x2/u;23996241;2.00;; 4364677170;;cpu/event=0xc0,umask=0x0/u;23996241;2.00;;
Unfortunately this totally breaks toplev. It needs to have the dummies in the right location.`
Another problem is that it also now adds [software] to lots of software events, which of course also breaks any parsing tools. I haven't bisected that too, but it needs fixing too.
Quoted 8 lines
Quoted 2 lines
There's nothing to bisect, the behavior has changed. We can revert the behavior, break Intel hybrid and Intel perf metric events, but that
I've been using perf metrics for many years without problems without this patch.
I know hybrid has some issues that need to be fixed, but that absolutely cannot break basic properties like having a parseable output format.
I think the last count I had hybrid was broken in about a dozen different regards in Linux perf 6.4. It was a substantial undertaking fixing in for 6.5, where fixes were making the tool not segv, produce meaningful counts), etc. "some issues" is a complete misclassification, the perf tool on a hybrid system is barely usable in Linux 6.4. Running perf test is sufficient to confirm this.
Quoted 7 lines
Quoted 2 lines
would seem unfortunate. Tbh, the mistake here is the assumptions that toplev is making.
It's not an random assumption, it's the only way to parse perf output uniquely.
Please make it an option. But this cannot stay this way unconditionally if perf stat is supposed to stay a tool which can be used from other programs.
You should be using the json or CSV outputs. These are intended for tool consumption, but the formats could be better. The text output, as argued to me by your colleagues, is for human consumption and so should read well in preference to legacy compatibility.
Quoted 12 lines
Quoted 11 lines
Quoted 8 lines
correctness, a necessary step to resume attempts at being multithreaded.
We have seen a growing number of 'perf test' entries being submitted, with writing shell scripts that check if the output from various command lines produce expected results.
It would really be great to have new entries that exercise what is expected by toplev so that we really are careful with compatibility.
We are careful with compatibility, in fact we've written a number of tests to ensure the json and CSV output doesn't regress. Kan recently
Quoted 8 lines
The tests are super basic and don't seem to cover any even moderately complex cases. Yes it's great that you wrote tests, but given the poor coverage you really cannot rely on just a few basic unit tests here.
Given that, and the wide usage of perf and the existence of Hyrum's law the only safe way is to avoid changing default behaviors. If you want some behavior change make it opt-in.
Sure, can we reverse decisions relating to the perf metric events API and its grouping requirements? No, so we're stuck with an imperfect world where the tool has to do the best it can.
For hybrid the problem of regrouping events still isn't properly solved. Consider:
perf stat -e '{cycles,faults}' ...
On a hybrid system we get the cycles event for the cpu_atom and cpu_core PMUs, but faults as a software event will exist once and be grouped on only 1 of the PMUs. This behavior is clearly silly, faults should measure both on cpu_atom and cpu_core PMUs, as it would appear on a non-hybrid system, and we should fix this. But what you're arguing is that we can't and shouldn't as Hyrum's law tells us that someone is depending on the 1 PMU behavior. I don't believe this, or if they are depending on the behavior they are silly and should explicitly list the groups and PMUs. We need to make the tool behave properly for increasingly complex kernel PMU event requirements. Changing the evlist to make things work for the complex kernel event requirements isn't the same as saying we're willfully trying to break users, we're trying to make users succeed even as the underlying CPUs, PMUs, etc. become more complex.
This issue is often made substantially more complex as fixing Intel's kernel PMU driver is pushed back upon - for example, we can't rely on weak groups, or "perf stat -e 'slots,branches,slots' -I 1000" has the branches event multiplexing for no reason.
Quoted 6 lines
Quoted 3 lines
Kan recently removed events and put metric names in the text output instead, even hiding events that have been programmed.. basically relying on text output is wrong and toplev is broken in this regard.
I missed your explanation how toplev is supposed to parse the output then instead?
Use the json or CSV output formats, json should be more stable as values are labelled.
Quoted 10 lines
Quoted 5 lines
Fixing hybrid and perf metric events weren't problems of my invention, nor were they done with a casual disregard for perf users and no care about breaking them. That said, testing can always be better so why not add toplev's tests to
tools/perf/tests/shell? This way I can only break things if I ignore a test regression.toplev generates the perf command lines dynamically and they are highly dependent on the current system. I don't see a simple way to make independent tests, unless you want a big case statement for existing systems, or add most of toplev.
Some minimal tests could be:
- tests that repeat examples from man pages
- tests for bugs past and present
- some form of maximal stress test
I don't know toplev and don't have time to set this up.
Thanks,
Ian
Quoted 10 lines
I guess could add some script that is the equivalent of
git clone https://github.com/andikleen/pmu-tools
cd pmu-tools
PERF=/point/to/new/perf/binary ./tl-testerif that helps? The toplev test suite has reasonably good coverage.
-Andi
Quoted 36 lines · Ian Rogers
Quoted 35 lines · Andi Kleen
Quoted 16 lines · Ian Rogers
Quoted 14 lines · Arnaldo Carvalho de Melo
Em Thu, Jul 13, 2023 at 01:37:00PM -0700, Andi Kleen escreveu:
Quoted 12 lines
Hi Ian,
The current post 6.3 linux-perf-next causes some fairly drastic reordering in output for some event combinations. I bisected it down to your patch:
commit
347c2f0a0988c59c402148054ef54648853fa669Author: Ian Rogers <[email protected]> Date: Sat Mar 11 18:15:40 2023 -0800perf parse-events: Sort and group parsed events
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
This change is intended to be a no-op for most current cases, the default sort order is the order the events were parsed. Where it varies is in how groups are handled. Previously an uncore and core event that are grouped would most often cause the group to be removed:
So this is a no-op for most current cases where event groups aren't used. The default order for events is the order they were inserted in.
This doesn't seem to be what happens because the patch reorders top-level events outside group. (like all the top level "dummy"ies)
But this commit broke things that are completely unrelated to hybrid and uncore, just plain software events not being in a group. There's no reason to reorder those.
Quoted 5 lines
Quoted 2 lines
Basically anything you do for hybrid and topdown shouldn't affect anything else. That's the basic rule.
It is not clear that modifying the order of events in the evlist qualifies as a breakage,
I don't know what to say here.
Quoted 3 lines
and it is necessary because of complex kernel
APIs that particularly in the case of perf metric events, Intel were responsible for.
But my example has no perf metrics.
Also taking a step back:
You're saying reordering is needed for perf metrics because slots has to come first in the group.
But the expectation here was always that the user would specify the groups this way, otherwise it doesn't work. Do you disagree with that?
So why does the perf tool need to fix it? Just pass it to the kernel and it won't schedule and the user has to fix it.
Perhaps perf can give warnings to make it clearer.
I understand that the metrics code has to make sure to not generate such groups, but that's a different problem that shouldn't affect how non metrics events are handled.
Also I reviewed the commit in question and at least the example doesn't even have perf metrics ?!? It's trying to fix some uncore issue.
So it seems what we're talking about here has nothing to do with perf metrics.
How about replacing the sorting commit with a warning and ask the user to do the reordering? Would that work for you?
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
wrong place, instead of the expected order.
good:
...
986476553;ns;duration_time;986476553;100.00;;
0;;dummy;983952237;100.00;;
0;;cs:u;983952237;100.00;;
16532;;minor-faults:u;983952237;100.00;;
0;;major-faults:u;983952237;100.00;;
0;;migrations:u;983952237;100.00;;
3532509496;;cycles:u;23999092;2.00;;
....bad:
1010119571;ns;duration_time;1010119571;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;cs:u [software];1007748753;100.00;; 16496;;minor-faults:u [software];1007748753;100.00;; 0;;major-faults:u [software];1007748753;100.00;; 0;;migrations:u [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;;
Quoted 5 lines
Quoted 2 lines
Please explain how this insane order can make sense even with hybrid or metrics.
Tbh, I'm not sure what you are doing. linux-next is on 6.5 but you mention 6.3. You don't list a platform for testing on. I'm not even
6.3 was my original reference point, but since I bisected the issue it's not 6.3 but exactly just this one commit that changes behavior.
The platform is SKX.
Quoted 10 lines
Quoted 8 lines
Quoted 3 lines
would seem unfortunate. Tbh, the mistake here is the assumptions that toplev is making.
It's not an random assumption, it's the only way to parse perf output uniquely.
Please make it an option. But this cannot stay this way unconditionally if perf stat is supposed to stay a tool which can be used from other programs.
You should be using the json or CSV outputs. These are intended for
This is the CSV output that is reordered! (see the example you just quoted)
I haven't tested JSON, but I assume that is reordered too.
Quoted 3 lines
For hybrid the problem of regrouping events still isn't properly solved. Consider:
Yes I understand that hybrid has issues. But the first step to solving them is to not break existing platforms.
Quoted 10 lines
perf stat -e '{cycles,faults}' ...
On a hybrid system we get the cycles event for the
cpu_atomandcpu_corePMUs, but faults as a software event will exist once and be grouped on only 1 of the PMUs. This behavior is clearly silly, faults should measure both oncpu_atomandcpu_corePMUs, as it would appear on a non-hybrid system, and we should fix this. But what you're arguing is that we can't and shouldn't as Hyrum's law tells us that someone is depending on the 1 PMU behavior.
TBH i don't have a strong opinion on keeping hybrid the same As you say it is somewhat broken, so I guess there are less existing users and some change there is acceptable, as long as it doesn't affect the more mature platforms.
As for your example it seems somewhat obscure and I would probably consider it user error. The non PMU qualified support was really only intended to keep existing perf record/stat default working, for everything else it should be discouraged. Trying to fix every corner case
here automatically is unlikely to be productive.
So perhaps just print a warning in this case?
Quoted 2 lines
explicitly list the groups and PMUs. We need to make the tool behave properly for increasingly complex kernel PMU event requirements.
I agree, but reordering the complete event list with a huge impact on non hybrid is not the way to do it.
Quoted 3 lines
This issue is often made substantially more complex as fixing Intel's kernel PMU driver is pushed back upon - for example, we can't rely on weak groups,
I thought they still work most of the time?
-Andi
Quoted 38 lines · Andi Kleen
Quoted 36 lines · Ian Rogers
Quoted 35 lines · Andi Kleen
Quoted 16 lines · Ian Rogers
Quoted 14 lines · Arnaldo Carvalho de Melo
Em Thu, Jul 13, 2023 at 01:37:00PM -0700, Andi Kleen escreveu:
Quoted 12 lines
Hi Ian,
The current post 6.3 linux-perf-next causes some fairly drastic reordering in output for some event combinations. I bisected it down to your patch:
commit
347c2f0a0988c59c402148054ef54648853fa669Author: Ian Rogers <[email protected]> Date: Sat Mar 11 18:15:40 2023 -0800perf parse-events: Sort and group parsed events
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
This change is intended to be a no-op for most current cases, the default sort order is the order the events were parsed. Where it varies is in how groups are handled. Previously an uncore and core event that are grouped would most often cause the group to be removed:
So this is a no-op for most current cases where event groups aren't used. The default order for events is the order they were inserted in.
This doesn't seem to be what happens because the patch reorders top-level events outside group. (like all the top level "dummy"ies)
But this commit broke things that are completely unrelated to hybrid and uncore, just plain software events not being in a group. There's no reason to reorder those.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Basically anything you do for hybrid and topdown shouldn't affect anything else. That's the basic rule.
It is not clear that modifying the order of events in the evlist qualifies as a breakage,
I don't know what to say here.
We do wildcard expansion of the PMUs, which leads to the evlist order being unstable. Fwiw this will change with:
https://lore.kernel.org/lkml/[email protected]/
Quoted 6 lines
Quoted 3 lines
and it is necessary because of complex kernel
APIs that particularly in the case of perf metric events, Intel were responsible for.But my example has no perf metrics.
Quoted 8 lines
Also taking a step back:
You're saying reordering is needed for perf metrics because slots has to come first in the group.
But the expectation here was always that the user would specify the groups this way, otherwise it doesn't work. Do you disagree with that?
Yes I disagree. We can't always force perf metric events in a group because weak groups don't always work. When we place the perf metric events outside of a group they are broken. So now we don't group the events and fix the topdown events to be in a group, in arch specific code. It is a complete pain brought about by the Intel core PMU driver.
Quoted 2 lines
So why does the perf tool need to fix it? Just pass it to the kernel and it won't schedule and the user has to fix it.
Because in some cases, and the cases I have happened to care about, the groups of events come from metrics.
Quoted 1 line
Perhaps perf can give warnings to make it clearer.
We do warn when we reorder the events.
Quoted 3 lines
I understand that the metrics code has to make sure to not generate such groups, but that's a different problem that shouldn't affect how non metrics events are handled.
But things like wildcard group expansion and regrouping is a metric problem too. So why would there be a parse events for metrics and a separate one for events? Doing it this way would make metric group testing significantly more hard too, as we can't just try the events without the metric.
Quoted 2 lines
Also I reviewed the commit in question and at least the example doesn't even have perf metrics ?!? It's trying to fix some uncore issue.
The uncore issue has been long standing, however, the fix used to be in a separate "fix uncore" function. Then there was a similar "fix hybrid" function. What is different now is that there is a single fix function that isn't hard coded to Intel PMU names, etc.
Quoted 5 lines
So it seems what we're talking about here has nothing to do with perf metrics.
How about replacing the sorting commit with a warning and ask the user to do the reordering? Would that work for you?
No.
Quoted 52 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
wrong place, instead of the expected order.
good:
...
986476553;ns;duration_time;986476553;100.00;;
0;;dummy;983952237;100.00;;
0;;cs:u;983952237;100.00;;
16532;;minor-faults:u;983952237;100.00;;
0;;major-faults:u;983952237;100.00;;
0;;migrations:u;983952237;100.00;;
3532509496;;cycles:u;23999092;2.00;;
....bad:
1010119571;ns;duration_time;1010119571;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;cs:u [software];1007748753;100.00;; 16496;;minor-faults:u [software];1007748753;100.00;; 0;;major-faults:u [software];1007748753;100.00;; 0;;migrations:u [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;dummy [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;; 0;;emulation-faults [software];1007748753;100.00;;
Quoted 5 lines
Quoted 2 lines
Please explain how this insane order can make sense even with hybrid or metrics.
Tbh, I'm not sure what you are doing. linux-next is on 6.5 but you mention 6.3. You don't list a platform for testing on. I'm not even
6.3 was my original reference point, but since I bisected the issue it's not 6.3 but exactly just this one commit that changes behavior.
The platform is SKX.
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
would seem unfortunate. Tbh, the mistake here is the assumptions that toplev is making.
It's not an random assumption, it's the only way to parse perf output uniquely.
Please make it an option. But this cannot stay this way unconditionally if perf stat is supposed to stay a tool which can be used from other programs.
Quoted 6 lines
Quoted 1 line
You should be using the json or CSV outputs. These are intended for
This is the CSV output that is reordered! (see the example you just quoted)
I haven't tested JSON, but I assume that is reordered too.
The output of the JSON is a dictionary and inherently unordered.
Quoted 7 lines
Quoted 3 lines
For hybrid the problem of regrouping events still isn't properly solved. Consider:
Yes I understand that hybrid has issues. But the first step to solving them is to not break existing platforms.
This is a false assertion. Coverage testing is up about 10%, we have more tests and more tests passing on more platforms. Objectively the code base has improved. On hybrid the behavior is night and day - crashes vs having the complete TopdownL1 working on both PMUs, testing parity with non-hybrid. We also fixed all address and leak sanitizer bugs. Perf in Linux 6.5 even without the awesome new features like lock contention analysis, is the best perf tool we've ever had.
Quoted 22 lines
Quoted 10 lines
perf stat -e '{cycles,faults}' ...
On a hybrid system we get the cycles event for the
cpu_atomandcpu_corePMUs, but faults as a software event will exist once and be grouped on only 1 of the PMUs. This behavior is clearly silly, faults should measure both oncpu_atomandcpu_corePMUs, as it would appear on a non-hybrid system, and we should fix this. But what you're arguing is that we can't and shouldn't as Hyrum's law tells us that someone is depending on the 1 PMU behavior.TBH i don't have a strong opinion on keeping hybrid the same As you say it is somewhat broken, so I guess there are less existing users and some change there is acceptable, as long as it doesn't affect the more mature platforms.
As for your example it seems somewhat obscure and I would probably consider it user error. The non PMU qualified support was really only intended to keep existing perf record/stat default working, for everything else it should be discouraged. Trying to fix every corner case
here automatically is unlikely to be productive.
Quoted 7 lines
So perhaps just print a warning in this case?
Quoted 2 lines
explicitly list the groups and PMUs. We need to make the tool behave properly for increasingly complex kernel PMU event requirements.
I agree, but reordering the complete event list with a huge impact on non hybrid is not the way to do it.
Does SKX have uncore? Yes, so you need to reorder events and this change didn't start it.
Quoted 6 lines
Quoted 4 lines
This issue is often made substantially more complex as fixing Intel's kernel PMU driver is pushed back upon - for example, we can't rely on weak groups,
I thought they still work most of the time?
$ for i in `ls tools/perf/pmu-events/arch/x86/*/*-metrics.json`; do echo -n "$i "; grep NO_GROUP_EVENTS $i|wc -l; done tools/perf/pmu-events/arch/x86/alderlake/adl-metrics.json 21 tools/perf/pmu-events/arch/x86/alderlaken/adln-metrics.json 2 tools/perf/pmu-events/arch/x86/broadwell/bdw-metrics.json 20 tools/perf/pmu-events/arch/x86/broadwellde/bdwde-metrics.json 20 tools/perf/pmu-events/arch/x86/broadwellx/bdx-metrics.json 23 tools/perf/pmu-events/arch/x86/cascadelakex/clx-metrics.json 32 tools/perf/pmu-events/arch/x86/elkhartlake/ehl-metrics.json 0 tools/perf/pmu-events/arch/x86/haswell/hsw-metrics.json 20 tools/perf/pmu-events/arch/x86/haswellx/hsx-metrics.json 23 tools/perf/pmu-events/arch/x86/icelake/icl-metrics.json 19 tools/perf/pmu-events/arch/x86/icelakex/icx-metrics.json 19 tools/perf/pmu-events/arch/x86/ivybridge/ivb-metrics.json 20 tools/perf/pmu-events/arch/x86/ivytown/ivt-metrics.json 23 tools/perf/pmu-events/arch/x86/jaketown/jkt-metrics.json 8 tools/perf/pmu-events/arch/x86/rocketlake/rkl-metrics.json 19 tools/perf/pmu-events/arch/x86/sandybridge/snb-metrics.json 8 tools/perf/pmu-events/arch/x86/sapphirerapids/spr-metrics.json 19 tools/perf/pmu-events/arch/x86/skylake/skl-metrics.json 30 tools/perf/pmu-events/arch/x86/skylakex/skx-metrics.json 31 tools/perf/pmu-events/arch/x86/tigerlake/tgl-metrics.json 19
NO_GROUP_EVENTS is used to avoid a group being used, an important use being avoiding PMU driver bugs.
Thanks,
Ian
Quoted 1 line
-Andi
Quoted 6 lines · Ian Rogers
Yes I disagree. We can't always force perf metric events in a group because weak groups don't always work. When we place the perf metric events outside of a group they are broken. So now we don't group the events and fix the topdown events to be in a group, in arch specific code. It is a complete pain brought about by the Intel core PMU driver.
Was that the old topdown slots have to be first issue, or something else?
If it's so painful I guess could revisit the kernel part, although I suspect anything that the kernel could do the perf tool could do too.
Quoted 6 lines
Quoted 3 lines
So why does the perf tool need to fix it? Just pass it to the kernel and it won't schedule and the user has to fix it.
Because in some cases, and the cases I have happened to care about, the groups of events come from metrics.
That's fine, but can't you just fix it for the metrics only.
Also even for metrics if you freely reorder them they might become unparseable. But the events inside the metrics should be freely reorderable (well technically someone could have a dependency on that too, but at least in my mind that's much less likely than their dependency just being on the metric)
So what it really needs is a sort only insight the metric.
That shouldn't be too hard to implement, right?
Also I guess we could add some test cases on avoiding reordering, although I'm optimistic that once it is established that reordering should be minimized/avoided it is unlikely to be something that breask frequently.
Would it be possible to revert the global sort patch until that is sorted out? Otherwise I'm afraid I'll end up with perf releases that are absolutely incompatible to toplev, and I'll get problems with my user base who often cannot easily update perf.
Arnaldo, can you please comment on this, what is your policy on regressions here?
Quoted 5 lines
But things like wildcard group expansion and regrouping is a metric problem too. So why would there be a parse events for metrics and a separate one for events? Doing it this way would make metric group testing significantly more hard too, as we can't just try the events without the metric.
Hmm, perhaps add some kind of debug flag to handle this case? As long as it's not default I don't have any issues with that.
Quoted 4 lines
Quoted 2 lines
How about replacing the sorting commit with a warning and ask the user to do the reordering? Would that work for you?
No.
Well I hope we can get to some compromise here. Right now this arbitary unpredictable reordering is a show stopper for me, and I suspect it will be for any other perf stat users that do some kind of post processing.
As long as we find a solution like above that works for the metrics it should be ok right?
Quoted 11 lines
Quoted 11 lines
The platform is SKX.
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
would seem unfortunate. Tbh, the mistake here is the assumptions that toplev is making.
It's not an random assumption, it's the only way to parse perf output uniquely.
Please make it an option. But this cannot stay this way unconditionally if perf stat is supposed to stay a tool which can be used from other programs.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
You should be using the json or CSV outputs. These are intended for
This is the CSV output that is reordered! (see the example you just quoted)
I haven't tested JSON, but I assume that is reordered too.
The output of the JSON is a dictionary and inherently unordered.
I haven't tried to use it, but if you have something like
perf stat --json -e '{cycles,branches},{cycles,cache-misses}' how would you distinguish the two different cycles without order?
Quoted 6 lines
crashes vs having the complete TopdownL1 working on both PMUs, testing parity with non-hybrid. We also fixed all address and leak sanitizer bugs.
Perf in Linux 6.5 even without the awesome new features like lock contention analysis, is the best perf tool we've ever had.
Sure and without arbitrary reordering and unparseable output it would be even better ...
Quoted 2 lines
Does SKX have uncore? Yes, so you need to reorder events and this change didn't start it.
Yes of course it has an uncore.
You only need to reorder if you create broken groups. At least my tool doesn't generate them.
Quoted 3 lines
NO_GROUP_EVENTSis used to avoid a group being used, an important use being avoiding PMU driver bugs.
Hmm, not sure what that is refering to, but ok.
-Andi
Em Sun, Jul 16, 2023 at 07:00:44PM -0700, Andi Kleen escreveu:
Quoted 18 lines · Ian Rogers
Quoted 6 lines
Yes I disagree. We can't always force perf metric events in a group because weak groups don't always work. When we place the perf metric events outside of a group they are broken. So now we don't group the events and fix the topdown events to be in a group, in arch specific code. It is a complete pain brought about by the Intel core PMU driver.
Was that the old topdown slots have to be first issue, or something else?
If it's so painful I guess could revisit the kernel part, although I suspect anything that the kernel could do the perf tool could do too.
Quoted 4 lines
Quoted 3 lines
So why does the perf tool need to fix it? Just pass it to the kernel and it won't schedule and the user has to fix it.
Quoted 7 lines
Quoted 2 lines
Because in some cases, and the cases I have happened to care about, the groups of events come from metrics.
That's fine, but can't you just fix it for the metrics only.
Also even for metrics if you freely reorder them they might become unparseable. But the events inside the metrics should be freely
Don't we have a JSON output? My expectation is that after the work that Ian has been doing we will settle down, but even then downstreamers should try to use JSON as input as that would avoid the problems we are having?
You mentioned that it was difficult to add entries to 'perf test' but that we could try running toplev's regression test, right? I can add it to my set of tests but perhaps the best thing would be to wire it up to one of the CIs out there?
- Arnaldo
Quoted 93 lines
reorderable (well technically someone could have a dependency on that too, but at least in my mind that's much less likely than their dependency just being on the metric)
So what it really needs is a sort only insight the metric.
That shouldn't be too hard to implement, right?
Also I guess we could add some test cases on avoiding reordering, although I'm optimistic that once it is established that reordering should be minimized/avoided it is unlikely to be something that breask frequently.
Would it be possible to revert the global sort patch until that is sorted out? Otherwise I'm afraid I'll end up with perf releases that are absolutely incompatible to toplev, and I'll get problems with my user base who often cannot easily update perf.
Arnaldo, can you please comment on this, what is your policy on regressions here?
Quoted 5 lines
But things like wildcard group expansion and regrouping is a metric problem too. So why would there be a parse events for metrics and a separate one for events? Doing it this way would make metric group testing significantly more hard too, as we can't just try the events without the metric.
Hmm, perhaps add some kind of debug flag to handle this case? As long as it's not default I don't have any issues with that.
Quoted 4 lines
Quoted 2 lines
How about replacing the sorting commit with a warning and ask the user to do the reordering? Would that work for you?
No.
Well I hope we can get to some compromise here. Right now this arbitary unpredictable reordering is a show stopper for me, and I suspect it will be for any other perf stat users that do some kind of post processing.
As long as we find a solution like above that works for the metrics it should be ok right?
Quoted 11 lines
Quoted 11 lines
The platform is SKX.
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
would seem unfortunate. Tbh, the mistake here is the assumptions that toplev is making.
It's not an random assumption, it's the only way to parse perf output uniquely.
Please make it an option. But this cannot stay this way unconditionally if perf stat is supposed to stay a tool which can be used from other programs.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
You should be using the json or CSV outputs. These are intended for
This is the CSV output that is reordered! (see the example you just quoted)
I haven't tested JSON, but I assume that is reordered too.
The output of the JSON is a dictionary and inherently unordered.
I haven't tried to use it, but if you have something like
perf stat --json -e '{cycles,branches},{cycles,cache-misses}' how would you distinguish the two different cycles without order?Quoted 6 lines
crashes vs having the complete TopdownL1 working on both PMUs, testing parity with non-hybrid. We also fixed all address and leak sanitizer bugs.
Perf in Linux 6.5 even without the awesome new features like lock contention analysis, is the best perf tool we've ever had.
Sure and without arbitrary reordering and unparseable output it would be even better ...
Quoted 2 lines
Does SKX have uncore? Yes, so you need to reorder events and this change didn't start it.
Yes of course it has an uncore.
You only need to reorder if you create broken groups. At least my tool doesn't generate them.
Quoted 3 lines
NO_GROUP_EVENTSis used to avoid a group being used, an important use being avoiding PMU driver bugs.Hmm, not sure what that is refering to, but ok.
-Andi
Signature · 1 line
- Arnaldo
Quoted 11 lines · Andi Kleen
Quoted 6 lines · Ian Rogers
Yes I disagree. We can't always force perf metric events in a group because weak groups don't always work. When we place the perf metric events outside of a group they are broken. So now we don't group the events and fix the topdown events to be in a group, in arch specific code. It is a complete pain brought about by the Intel core PMU driver.
Was that the old topdown slots have to be first issue, or something else?
Yes, the current ordering is defined here:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/arch/x86/util/evlist.c?h=perf-tools-next#n79
Quoted 18 lines
If it's so painful I guess could revisit the kernel part, although I suspect anything that the kernel could do the perf tool could do too.
Quoted 6 lines
Quoted 3 lines
So why does the perf tool need to fix it? Just pass it to the kernel and it won't schedule and the user has to fix it.
Because in some cases, and the cases I have happened to care about, the groups of events come from metrics.
That's fine, but can't you just fix it for the metrics only.
Also even for metrics if you freely reorder them they might become unparseable. But the events inside the metrics should be freely reorderable (well technically someone could have a dependency on that too, but at least in my mind that's much less likely than their dependency just being on the metric)
Quoted 3 lines
So what it really needs is a sort only insight the metric.
That shouldn't be too hard to implement, right?
I'll describe why this isn't a fix below.
Quoted 9 lines
Also I guess we could add some test cases on avoiding reordering, although I'm optimistic that once it is established that reordering should be minimized/avoided it is unlikely to be something that breask frequently.
Would it be possible to revert the global sort patch until that is sorted out? Otherwise I'm afraid I'll end up with perf releases that are absolutely incompatible to toplev, and I'll get problems with my user base who often cannot easily update perf.
This would break the majority of metrics from Icelake forward.
Quoted 17 lines
Arnaldo, can you please comment on this, what is your policy on regressions here?
Quoted 5 lines
But things like wildcard group expansion and regrouping is a metric problem too. So why would there be a parse events for metrics and a separate one for events? Doing it this way would make metric group testing significantly more hard too, as we can't just try the events without the metric.
Hmm, perhaps add some kind of debug flag to handle this case? As long as it's not default I don't have any issues with that.
Quoted 4 lines
Quoted 2 lines
How about replacing the sorting commit with a warning and ask the user to do the reordering? Would that work for you?
No.
Quoted 7 lines
Well I hope we can get to some compromise here. Right now this arbitary unpredictable reordering is a show stopper for me, and I suspect it will be for any other perf stat users that do some kind of post processing.
As long as we find a solution like above that works for the metrics it should be ok right?
So you've resisted working through what your issue with the sorting is. To call the sorting arbitrary is of course wrong, but you keep wanting to say everything the perf tool is doing is wrong. Moving forward...
What is happening in your example is straightforward. The sorting
code's goal is to keep events in their place but what to do about:
perf stat -e 'cycles,{uncore_imc_free_running/data_read/,uncore_imc_free_running/data_write/}'
where there are 2 uncore_imc_free_running PMUs? After parsing the
evlist looks like:
cycles,{uncore_imc_free_running_0/data_read/,uncore_imc_free_running_1/data_read/,uncore_imc_free_running_0/data_write/,uncore_imc_free_running_1/data_write/} We need to turn 1 group into 2, we need to reorder by the PMUs, this leads to:
cycles,{uncore_imc_free_running_0/data_read/,uncore_imc_free_running_0/data_write/},{uncore_imc_free_running_1/data_read/,uncore_imc_free_running_1/data_write/} There is no regression here, it has always been perf's behavior to
reorder events and create groups globally.
When sorting groups the code uses the position in the original list of
the group's leader:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2124
For events within a group it sorts them by group PMU name (which is
normally just the PMU name), if the PMU names are the same it sorts
the events by their original position in the list possibly corrected
for perf metric events.
Okay, so now imagine the lack of grouping brought about by perf metric (topdown) events and broken Intel PMUs that fail to break weak groups. For a hypothetical example we create a list of events like:
perf stat -e 'topdown-fe-bound,cycles,instructions,slots' As with the group case we need to reorder the events to have slots and topdown-fe-bound together and grouped. The code achieves:
{slots,topdown-fe-bound},cycles,instructions
As grouped events need to be adjacent in the evlist, we need to sort all the events without a group to be together and place the perf metric events first. They are then placed in a group.
So that we best respect the order of ungrouped and grouped events, we place the ungrouped events as if they are a group with a leader with the index of the first ungrouped event.
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2101
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2170
So this then comes to your example where you have ungrouped events, grouped events in a giant list. Well on Icelake+ we could have perf metric events in those ungrouped events, this means we need to sort the ungrouped events together, potentially moving perf metric events to the front and grouping them.
Hey hey hey, you say. I'm on SkylakeX and don't have perf metric events. Why should the tool be trying to sort events to fix issues on a PMU I don't have?
The problem is that Icelake+ architectures all have this issue and I keep trying to make the code simpler and do the same thing for everybody. I got rid of hardcoded "cpu_atom" and "cpu_core", the separate hybrid event parsing, ... I'm trying to make it that perf isn't a concatenated group of architecture specific PMU drivers and there is commonality in how everything is done.
Well you say, I still want my ungrouped events to be scattered around my list and you better not move them as that is the world's largest regression, is crazy arbitrary rubbish and please go dump all your changes in the garbage.
I disagree, the sorting is doing its best to only fix what is
necessary and otherwise to rely on the command line. That said, it can
only fix:
perf stat -e 'slots,{uncore_imc_free_running_0/data_read/,uncore_imc_free_running_1/data_read/},topdown-fe-bound' If it is able to move slots and topdown-fe-bound to be adjacent.
So how can I make you happy:
1) throw everything away.. well that would be a challenge, break stuff and I don't want to fix the resultant mess.
2) we can sort only for metrics. This leads to metric event parsing and event parsing, duplication, testing and other issues. I'm trying to make metrics sit on top of event parsing and not fork them. This is more true after the hybrid changes, where things like setting up a default cycles event is done through event parsing to avoid duplication of the wildcard logic. Plus we still need event sorting and grouping logic without metrics for at least uncore events.
3) we can sort only on Icelake+ architectures. Well isn't that the majority of what Intel cares about now-a-days? It also leads to having 2 code paths and testing problems. It also won't work as uncore events won't be sorted, regrouped and fixed.
4) we add a special toplev flag that disables sorting and grouping. Well that will break sorting and grouping of uncore events and so I don't think this will make you happy.
5) we add a special toplev flag that on detecting a core PMUs causes sorting and regrouping to be disabled. This isn't much better than 4 and still unlikely to make you happy.
So what can we do?
1) change how perf metric (topdown) events are implemented in the PMU. This strategy works best with a time machine and we're duty bound, in that case, to fix other problems in history.
2) stop toplev being a PITA? Wouldn't that be nice.
3) we do some kind of ungrouped event sorting hack. Rather than make every ungrouped event look like having a common leader index, we only do that for events that need to be forced into a group (perf metric events). This should mean ungrouped events are sorted wrt groups in the place they are originally in the list. This should fix the pre-Icelake problems but on Icelake+ we can still get the "arbitrary" reordering issue you so despise for perf metric events.
I'm going to give (3) a go and mail it today.
Quoted 28 lines
Quoted 11 lines
Quoted 11 lines
The platform is SKX.
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
would seem unfortunate. Tbh, the mistake here is the assumptions that toplev is making.
It's not an random assumption, it's the only way to parse perf output uniquely.
Please make it an option. But this cannot stay this way unconditionally if perf stat is supposed to stay a tool which can be used from other programs.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
You should be using the json or CSV outputs. These are intended for
This is the CSV output that is reordered! (see the example you just quoted)
I haven't tested JSON, but I assume that is reordered too.
The output of the JSON is a dictionary and inherently unordered.
I haven't tried to use it, but if you have something like
perf stat --json -e '{cycles,branches},{cycles,cache-misses}' how would you distinguish the two different cycles without order?Quoted 3 lines
crashes vs having the complete TopdownL1 working on both PMUs, testing parity with non-hybrid. We also fixed all address and leak sanitizer bugs.
Quoted 5 lines
Quoted 2 lines
Perf in Linux 6.5 even without the awesome new features like lock contention analysis, is the best perf tool we've ever had.
Sure and without arbitrary reordering and unparseable output it would be even better ...
Ugh.. am I being motivated here to fix your toplev problem?
Quoted 7 lines
Quoted 2 lines
Does SKX have uncore? Yes, so you need to reorder events and this change didn't start it.
Yes of course it has an uncore.
You only need to reorder if you create broken groups. At least my tool doesn't generate them.
Zip-a-Dee-Doo-Dah.
Quoted 5 lines
Quoted 3 lines
NO_GROUP_EVENTSis used to avoid a group being used, an important use being avoiding PMU driver bugs.Hmm, not sure what that is refering to, but ok.
You can take the metrics and remove the NO_GROUP_EVENTS. The failure mode is that the weak group fails to be broken. I believe the issues to be some mix of errata, naivety in the fake PMU driver when scheduling onto counters and not appreciating that not all counters are created equal, possible bugs, etc. The issues may not just be on the core PMU, it depends on the metric.
Given I'm fixing things for you, can you please fix:
$ perf stat -e 'slots,branches,slots' -a sleep 1 to not multiplex on Icelake+.
This can be achieved with:
```
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -2607,7 +2607,7 @@ static int group_can_go_on(struct perf_event
*event, int can_add_hw)
* Otherwise, try to add it if all previous groups were able
* to go on.
*/
- return can_add_hw;
+ return 1;
}
static void add_event_to_ctx(struct perf_event *event,
```That means we'll consider adding every hardware event just as we consider adding every software event. A more complex fix would be to make can_add_hw some kind of bitmap that indicates different counter types are available, and to only stop searching only when all counter types are exhausted. I'm perfectly happy getting rid of the can_add_hw optimization and not worrying about the potential extra overhead. That overhead is theoretic and under user control. The unnecessary multiplexing is just silly and unavoidable.
Thanks,
Ian
Quoted 1 line
-Andi
Quoted 177 lines · Ian Rogers
Quoted 11 lines · Andi Kleen
Quoted 6 lines · Ian Rogers
Yes I disagree. We can't always force perf metric events in a group because weak groups don't always work. When we place the perf metric events outside of a group they are broken. So now we don't group the events and fix the topdown events to be in a group, in arch specific code. It is a complete pain brought about by the Intel core PMU driver.
Was that the old topdown slots have to be first issue, or something else?
Yes, the current ordering is defined here:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/arch/x86/util/evlist.c?h=perf-tools-next#n79Quoted 18 lines
If it's so painful I guess could revisit the kernel part, although I suspect anything that the kernel could do the perf tool could do too.
Quoted 6 lines
Quoted 3 lines
So why does the perf tool need to fix it? Just pass it to the kernel and it won't schedule and the user has to fix it.
Because in some cases, and the cases I have happened to care about, the groups of events come from metrics.
That's fine, but can't you just fix it for the metrics only.
Also even for metrics if you freely reorder them they might become unparseable. But the events inside the metrics should be freely reorderable (well technically someone could have a dependency on that too, but at least in my mind that's much less likely than their dependency just being on the metric)
Quoted 3 lines
So what it really needs is a sort only insight the metric.
That shouldn't be too hard to implement, right?
I'll describe why this isn't a fix below.
Quoted 9 lines
Also I guess we could add some test cases on avoiding reordering, although I'm optimistic that once it is established that reordering should be minimized/avoided it is unlikely to be something that breask frequently.
Would it be possible to revert the global sort patch until that is sorted out? Otherwise I'm afraid I'll end up with perf releases that are absolutely incompatible to toplev, and I'll get problems with my user base who often cannot easily update perf.
This would break the majority of metrics from Icelake forward.
Quoted 17 lines
Arnaldo, can you please comment on this, what is your policy on regressions here?
Quoted 5 lines
But things like wildcard group expansion and regrouping is a metric problem too. So why would there be a parse events for metrics and a separate one for events? Doing it this way would make metric group testing significantly more hard too, as we can't just try the events without the metric.
Hmm, perhaps add some kind of debug flag to handle this case? As long as it's not default I don't have any issues with that.
Quoted 4 lines
Quoted 2 lines
How about replacing the sorting commit with a warning and ask the user to do the reordering? Would that work for you?
No.
Quoted 7 lines
Well I hope we can get to some compromise here. Right now this arbitary unpredictable reordering is a show stopper for me, and I suspect it will be for any other perf stat users that do some kind of post processing.
As long as we find a solution like above that works for the metrics it should be ok right?
So you've resisted working through what your issue with the sorting is. To call the sorting arbitrary is of course wrong, but you keep wanting to say everything the perf tool is doing is wrong. Moving forward...
What is happening in your example is straightforward. The sorting
code's goal is to keep events in their place but what to do about:
perf stat -e 'cycles,{uncore_imc_free_running/data_read/,uncore_imc_free_running/data_write/}'
where there are 2uncore_imc_free_runningPMUs? After parsing the
evlist looks like:
cycles,{uncore_imc_free_running_0/data_read/,uncore_imc_free_running_1/data_read/,uncore_imc_free_running_0/data_write/,uncore_imc_free_running_1/data_write/} We need to turn 1 group into 2, we need to reorder by the PMUs, this leads to:
cycles,{uncore_imc_free_running_0/data_read/,uncore_imc_free_running_0/data_write/},{uncore_imc_free_running_1/data_read/,uncore_imc_free_running_1/data_write/} There is no regression here, it has always been perf's behavior to
reorder events and create groups globally.
When sorting groups the code uses the position in the original list of
the group's leader:
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2124
For events within a group it sorts them by group PMU name (which is
normally just the PMU name), if the PMU names are the same it sorts
the events by their original position in the list possibly corrected
for perf metric events.Okay, so now imagine the lack of grouping brought about by perf metric (topdown) events and broken Intel PMUs that fail to break weak groups. For a hypothetical example we create a list of events like:
perf stat -e 'topdown-fe-bound,cycles,instructions,slots' As with the group case we need to reorder the events to have slots and topdown-fe-bound together and grouped. The code achieves:
{slots,topdown-fe-bound},cycles,instructions
As grouped events need to be adjacent in the evlist, we need to sort all the events without a group to be together and place the perf metric events first. They are then placed in a group.
So that we best respect the order of ungrouped and grouped events, we place the ungrouped events as if they are a group with a leader with the index of the first ungrouped event.
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2101
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/parse-events.c?h=perf-tools-next#n2170So this then comes to your example where you have ungrouped events, grouped events in a giant list. Well on Icelake+ we could have perf metric events in those ungrouped events, this means we need to sort the ungrouped events together, potentially moving perf metric events to the front and grouping them.
Hey hey hey, you say. I'm on SkylakeX and don't have perf metric events. Why should the tool be trying to sort events to fix issues on a PMU I don't have?
The problem is that Icelake+ architectures all have this issue and I keep trying to make the code simpler and do the same thing for everybody. I got rid of hardcoded "cpu_atom" and "cpu_core", the separate hybrid event parsing, ... I'm trying to make it that perf isn't a concatenated group of architecture specific PMU drivers and there is commonality in how everything is done.
Well you say, I still want my ungrouped events to be scattered around my list and you better not move them as that is the world's largest regression, is crazy arbitrary rubbish and please go dump all your changes in the garbage.I disagree, the sorting is doing its best to only fix what is
necessary and otherwise to rely on the command line. That said, it can
only fix:
perf stat -e 'slots,{uncore_imc_free_running_0/data_read/,uncore_imc_free_running_1/data_read/},topdown-fe-bound' If it is able to move slots and topdown-fe-bound to be adjacent.So how can I make you happy:
1) throw everything away.. well that would be a challenge, break stuff and I don't want to fix the resultant mess.
2) we can sort only for metrics. This leads to metric event parsing and event parsing, duplication, testing and other issues. I'm trying to make metrics sit on top of event parsing and not fork them. This is more true after the hybrid changes, where things like setting up a default cycles event is done through event parsing to avoid duplication of the wildcard logic. Plus we still need event sorting and grouping logic without metrics for at least uncore events.
3) we can sort only on Icelake+ architectures. Well isn't that the majority of what Intel cares about now-a-days? It also leads to having 2 code paths and testing problems. It also won't work as uncore events won't be sorted, regrouped and fixed.
4) we add a special toplev flag that disables sorting and grouping. Well that will break sorting and grouping of uncore events and so I don't think this will make you happy.
5) we add a special toplev flag that on detecting a core PMUs causes sorting and regrouping to be disabled. This isn't much better than 4 and still unlikely to make you happy.So what can we do?
1) change how perf metric (topdown) events are implemented in the PMU. This strategy works best with a time machine and we're duty bound, in that case, to fix other problems in history.
2) stop toplev being a PITA? Wouldn't that be nice.
3) we do some kind of ungrouped event sorting hack. Rather than make every ungrouped event look like having a common leader index, we only do that for events that need to be forced into a group (perf metric events). This should mean ungrouped events are sorted wrt groups in the place they are originally in the list. This should fix the pre-Icelake problems but on Icelake+ we can still get the "arbitrary" reordering issue you so despise for perf metric events.
Quoted 1 line
I'm going to give (3) a go and mail it today.
Testing:
https://lore.kernel.org/lkml/[email protected]/
with the given events on SKX shows no event re-ordering compared to say perf 6.1.
The event list contains wild carded events for:
uncore_imc/event=0x4,umask=0x3/,uncore_imc/event=0x4,umask=0xc/
where the order is arbitrary and I see:
uncore_imc_5/event=0x4,umask=0x3/
uncore_imc_3/event=0x4,umask=0x3/
uncore_imc_1/event=0x4,umask=0x3/
uncore_imc_4/event=0x4,umask=0x3/
uncore_imc_2/event=0x4,umask=0x3/
uncore_imc_0/event=0x4,umask=0x3/
uncore_imc_5/event=0x4,umask=0xc/
uncore_imc_3/event=0x4,umask=0xc/
uncore_imc_1/event=0x4,umask=0xc/
uncore_imc_4/event=0x4,umask=0xc/
uncore_imc_2/event=0x4,umask=0xc/
uncore_imc_0/event=0x4,umask=0xc/
This order will be broken by:
https://lore.kernel.org/lkml/[email protected]/
as the order will be numerically ascending.
As described, the sorting will still apply to perf metric events. toplev will probably not be impacted as the perf metric events would need to already be sorted to work on older versions of perf.
Thanks,
Ian
Quoted 91 lines
Quoted 28 lines
Quoted 11 lines
Quoted 11 lines
The platform is SKX.
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
would seem unfortunate. Tbh, the mistake here is the assumptions that toplev is making.
It's not an random assumption, it's the only way to parse perf output uniquely.
Please make it an option. But this cannot stay this way unconditionally if perf stat is supposed to stay a tool which can be used from other programs.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
You should be using the json or CSV outputs. These are intended for
This is the CSV output that is reordered! (see the example you just quoted)
I haven't tested JSON, but I assume that is reordered too.
The output of the JSON is a dictionary and inherently unordered.
I haven't tried to use it, but if you have something like
perf stat --json -e '{cycles,branches},{cycles,cache-misses}' how would you distinguish the two different cycles without order?Quoted 3 lines
crashes vs having the complete TopdownL1 working on both PMUs, testing parity with non-hybrid. We also fixed all address and leak sanitizer bugs.
Quoted 5 lines
Quoted 2 lines
Perf in Linux 6.5 even without the awesome new features like lock contention analysis, is the best perf tool we've ever had.
Sure and without arbitrary reordering and unparseable output it would be even better ...
Ugh.. am I being motivated here to fix your toplev problem?
Quoted 7 lines
Quoted 2 lines
Does SKX have uncore? Yes, so you need to reorder events and this change didn't start it.
Yes of course it has an uncore.
You only need to reorder if you create broken groups. At least my tool doesn't generate them.
Zip-a-Dee-Doo-Dah.
Quoted 5 lines
Quoted 3 lines
NO_GROUP_EVENTSis used to avoid a group being used, an important use being avoiding PMU driver bugs.Hmm, not sure what that is refering to, but ok.
You can take the metrics and remove the
NO_GROUP_EVENTS. The failure mode is that the weak group fails to be broken. I believe the issues to be some mix of errata, naivety in the fake PMU driver when scheduling onto counters and not appreciating that not all counters are created equal, possible bugs, etc. The issues may not just be on the core PMU, it depends on the metric.Given I'm fixing things for you, can you please fix:
$ perf stat -e 'slots,branches,slots' -a sleep 1 to not multiplex on Icelake+.This can be achieved with:
```kernel/events/core.c +1
-1@@ -2607,7 +2607,7 @@ static int group_can_go_on(struct perf_event*event, int can_add_hw) * Otherwise, try to add it if all previous groups were able * to go on. */ -return can_add_hw; +return 1 ; }static void add_event_to_ctx(struct
perf_event*event,
```
That means we'll consider adding every hardware event just as we consider adding every software event. A more complex fix would be to makecan_add_hwsome kind of bitmap that indicates different counter types are available, and to only stop searching only when all counter types are exhausted. I'm perfectly happy getting rid of thecan_add_hwoptimization and not worrying about the potential extra overhead. That overhead is theoretic and under user control. The unnecessary multiplexing is just silly and unavoidable.Thanks,
IanQuoted 1 line
-Andi
<sorry for the late reply I was on vacation>
Quoted 4 lines · Arnaldo Carvalho de Melo
Don't we have a JSON output? My expectation is that after the work that Ian has been doing we will settle down, but even then downstreamers should try to use JSON as input as that would avoid the problems we are having?
In classic perf CSV was our machine readable output, and that is what got broken here. I hope nobody is advocating for abolishing CSV output as a machine readable format.
For JSON it got broken too because of:
Quoted 6 lines
Quoted 3 lines
I haven't tried to use it, but if you have something like
perf stat --json -e '{cycles,branches},{cycles,cache-misses}' how would you distinguish it without order?You mentioned that it was difficult to add entries to 'perf test' but
I can add some tests for basic reordering not occurring. That should be possible.
Quoted 3 lines
that we could try running toplev's regression test, right? I can add it to my set of tests but perhaps the best thing would be to wire it up to one of the CIs out there?
Yes that would be a good idea. Note that there are sometimes problems which are on the toplev side, not perf side, but I guess can sort that out when that happens.
-And