perf test attr: Fix the re-enabled test

From: Arnaldo Carvalho de Melo <[email protected]>

Hello all,

This series fixes the "Perf attribute expectations test" (perf test 82, formerly "Setup struct perf_event_attr").

Patches 1-2 are Michael Petlan's prerequisite fixes that make the shell wrapper propagate attr.py's return value and correct the perf_event_attr size expectation after struct config4 was added. They are included at the start of the series so that it applies cleanly on the current perf-tools-next (and for the review bots to verify the new patches); the new fixes are patches 3-6.

With the test reporting failures again, several stale expectations surfaced and are addressed here:

- test-record-graph-dwarf still expected PERF_SAMPLE_ADDR and PERF_SAMPLE_DATA_SRC for --call-graph dwarf. Those were dropped by bdd051e249141c79 ("perf record: Split --data-mmap option"), which decoupled data MMAP recording from the sample address bits.

- The record and stat group tests expected cache-misses and instructions to be the legacy PERF_TYPE_HARDWARE events, but since b1c5efbfd92eb84d ("perf parse-events: Remove hard coded legacy hardware and cache parsing") they resolve to the core PMU sysfs/JSON events, so request the legacy encodings explicitly with the legacy-hardware-config term.

- The perf stat default metrics open their events in groups and the raw encodings of instructions, branch-misses, cpu-cycles and stalled-cycles-frontend/backend differ between Intel and AMD, so base-stat now accepts the standalone and group variants and test-stat-default/detailed-{1,2,3} gained the per-vendor raw encodings.

- attr.py no longer requires a recorded event to be in a group when the matching expectation does not declare one, allowing the stat tests to verify event attributes without encoding the metric grouping. Expectations that declare a group are still verified in both directions.

The test passes with 'perf test -vv attribute' on x86_64. The AMD encodings come from the amd_pmu_event_map tables in arch/x86/events/amd/core.c.

Arnaldo

Arnaldo Carvalho de Melo (4):
perf test attr: Fix record dwarf sample_type expectation perf test attr: Fix legacy event encodings in group tests perf test attr: Fix default stat metrics expectations perf test attr: Relax group checking for ungrouped expectations

Michael Petlan (2):
perf test attr: Propagate the return value from the test to the
wrapper
perf test attr: Fix wrong size expectation for events

 tools/perf/tests/shell/attr.sh                |  1 +
 tools/perf/tests/shell/attr/README            | 10 +++---
 tools/perf/tests/shell/attr/base-record       |  2 +-
 tools/perf/tests/shell/attr/base-stat         |  8 ++---
 tools/perf/tests/shell/attr/system-wide-dummy |  2 +-
 .../tests/shell/attr/test-record-dummy-C0     |  2 +-
 .../tests/shell/attr/test-record-graph-dwarf  |  2 +-
 .../shell/attr/test-record-group-sampling     |  4 +--
 .../shell/attr/test-record-group-sampling1    |  4 +--
 .../shell/attr/test-record-group-sampling2    |  4 +--
 .../perf/tests/shell/attr/test-record-group1  |  2 +-
 .../perf/tests/shell/attr/test-record-group2  |  2 +-
 .../perf/tests/shell/attr/test-record-group3  |  2 +-
 tools/perf/tests/shell/attr/test-stat-default | 32 +++++++++++++++++--
 .../tests/shell/attr/test-stat-detailed-1     | 32 +++++++++++++++++--
 .../tests/shell/attr/test-stat-detailed-2     | 32 +++++++++++++++++--
 .../tests/shell/attr/test-stat-detailed-3     | 32 +++++++++++++++++--
 tools/perf/tests/shell/attr/test-stat-group1  |  2 +-
 tools/perf/tests/shell/lib/attr.py            | 11 +++++--
 19 files changed, 152 insertions(+), 34 deletions(-)

2.55.0

From: Michael Petlan <[email protected]>

Currently it does not matter what the python script actually returns, the test always passes, as the $err variable is always 0.

Fix that.

---
 tools/perf/tests/shell/attr.sh | 1 +
 1 file changed, 1 insertion(+)

index 5a4e43b2471de811..307e1535795a34dc 100755
@@ -18,5 +18,6 @@ trap trap_cleanup EXIT TERM INT
 shelldir=$(dirname "$0")
 perf_path=$(which perf)
 python "${shelldir}"/lib/attr.py -d "${shelldir}"/attr -v -p "$perf_path"
+err=$?
 cleanup
 exit $err

2.55.0

From: Michael Petlan <[email protected]>

Although config4 has been added into struct perf_event_attr, the size expectation for the struct hasn't been adjusted. This hasn't been noticed, since the testcase's return value has been ignored after rewriting the testcase to python until previous commit.

Fix that.

---
 tools/perf/tests/shell/attr/base-record          | 2 +-
 tools/perf/tests/shell/attr/base-stat            | 2 +-
 tools/perf/tests/shell/attr/system-wide-dummy    | 2 +-
 tools/perf/tests/shell/attr/test-record-dummy-C0 | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

index b44e4e6e444386af..a4597d4c0cd12550 100644
@@ -5,7 +5,7 @@ group_fd=-1
 flags=0|8
 cpu=*
 type=0|1
-size=136
+size=176
 config=0|1
 sample_period=*
 sample_type=263

index fccd8ec4d1b0225c..67d6eea8947034e1 100644
@@ -5,7 +5,7 @@ group_fd=-1
 flags=0|8
 cpu=*
 type=0
-size=136
+size=176
 config=0
 sample_period=0
 sample_type=65536

index a1e1d6a263bf14e7..b062ad7f3c1c61dc 100644
@@ -7,7 +7,7 @@ cpu=*
 pid=-1
 flags=8
 type=1
-size=136
+size=176
 config=9
 sample_period=1
 # PERF_SAMPLE_IP | PERF_SAMPLE_TID | PERF_SAMPLE_TIME |

index 91499405fff4d32b..54adb835d203f2b1 100644
@@ -10,7 +10,7 @@ cpu=0
 pid=-1
 flags=8
 type=1
-size=136
+size=176
 config=9
 sample_period=4000
 # PERF_SAMPLE_IP | PERF_SAMPLE_TID | PERF_SAMPLE_TIME |

2.55.0

From: Arnaldo Carvalho de Melo <[email protected]>

Since bdd051e249141c79 ("perf record: Split --data-mmap option"), record_opts__parse_callchain() sets record_data_mmap instead of record->sample_address for the DWARF unwind mode, so -d/--data and its implied PERF_SAMPLE_DATA_SRC are not enabled implicitly anymore.

As a result, --call-graph dwarf no longer sets PERF_SAMPLE_ADDR and PERF_SAMPLE_DATA_SRC, making the test fail:

expected sample_type=45359, got 12583

where 45359 (0xb12f) is:

IP|TID|TIME|ADDR|CALLCHAIN|PERIOD|REGS_USER|STACK_USER|DATA_SRC

and 12583 (0x3127) is:

IP|TID|TIME|CALLCHAIN|PERIOD|REGS_USER|STACK_USER

i.e. the values differ exactly by PERF_SAMPLE_ADDR (0x8) and PERF_SAMPLE_DATA_SRC (0x8000). The mmap_data=1 expectation is still correct as DWARF now enables record_data_mmap.

This went unnoticed because tests/attr.sh ignored attr.py's return value until 04a0a19ede01e771 ("perf test attr: Propagate the return value from the test to the wrapper").

---
 tools/perf/tests/shell/attr/test-record-graph-dwarf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index ae92061d611ded68..fbe0faa8cdb6a8f4 100644
@@ -4,7 +4,7 @@ args    = --no-bpf-event --call-graph dwarf -- kill >/dev/null 2>&1
 ret     = 1
 
 [event:base-record]
-sample_type=45359
+sample_type=12583
 exclude_callchain_user=1
 sample_stack_user=8192
 # TODO different for each arch, no support for that now

2.55.0

From: Arnaldo Carvalho de Melo <[email protected]>

Since b1c5efbfd92eb84d ("perf parse-events: Remove hard coded legacy hardware and cache parsing"), event names are resolved with sysfs/JSON priority over the legacy encodings, so bare cache-misses and instructions now become the core PMU sysfs/JSON events (on x86 type 4, configs 0x412e and 0xc0) rather than the legacy PERF_TYPE_HARDWARE/config=3 and config=1 events the tests expect, e.g.:

expected config=3, got 0
expected disabled=0, got 1
expected mmap=0, got 1
expected comm=0, got 1
expected freq=0, got 1
expected enable_on_exec=0, got 1
expected task=0, got 1
FAILED '.../test-record-group-sampling1' - match failure

The shown diff is against the cycles group leader, as no result event matches the expected cache-misses member anymore.

Request the legacy encodings explicitly with the legacy-hardware-config term, introduced for this purpose, keeping the original attribute expectations unchanged, in:

test-record-group1
test-record-group2
test-record-group3
test-record-group-sampling
test-record-group-sampling1
test-record-group-sampling2
test-stat-group1

This went unnoticed because tests/attr.sh ignored attr.py's return value until 04a0a19ede01e771 ("perf test attr: Propagate the return value from the test to the wrapper").

---
 tools/perf/tests/shell/attr/README                     | 10 +++++-----
 tools/perf/tests/shell/attr/test-record-group-sampling |  4 ++--
 .../perf/tests/shell/attr/test-record-group-sampling1  |  4 ++--
 .../perf/tests/shell/attr/test-record-group-sampling2  |  4 ++--
 tools/perf/tests/shell/attr/test-record-group1         |  2 +-
 tools/perf/tests/shell/attr/test-record-group2         |  2 +-
 tools/perf/tests/shell/attr/test-record-group3         |  2 +-
 tools/perf/tests/shell/attr/test-stat-group1           |  2 +-
 8 files changed, 15 insertions(+), 15 deletions(-)

index 67c4ca76b85d5611..c7b49cdf87297177 100644
@@ -49,10 +49,10 @@ Following tests are defined (with perf commands):
   perf record --call-graph dwarf kill		(test-record-graph-dwarf)
   perf record --call-graph fp kill              (test-record-graph-fp)
   perf record --call-graph fp kill              (test-record-graph-fp-aarch64)
-  perf record -e '{cycles,instructions}' kill   (test-record-group1)
-  perf record -e '{cycles/period=1/,instructions/period=2/}:S' kill (test-record-group2)
-  perf record -e '{cycles,cache-misses}:S' kill (test-record-group-sampling1)
-  perf record -c 10000 -e '{cycles,cache-misses}:S' kill (test-record-group-sampling2)
+  perf record -e '{cycles,default_core/legacy-hardware-config=1/}' kill (test-record-group1)
+  perf record -e '{cycles/period=1234000/,default_core/legacy-hardware-config=1,period=6789000/}:S' kill (test-record-group2)
+  perf record -e '{cycles,default_core/legacy-hardware-config=3/}:S' kill (test-record-group-sampling1)
+  perf record -c 10000 -e '{cycles,default_core/legacy-hardware-config=3/}:S' kill (test-record-group-sampling2)
   perf record -D kill                           (test-record-no-delay)
   perf record -i kill                           (test-record-no-inherit)
   perf record -n kill                           (test-record-no-samples)

@@ -67,5 +67,5 @@ Following tests are defined (with perf commands):
   perf stat -d kill                             (test-stat-detailed-1)
   perf stat -dd kill                            (test-stat-detailed-2)
   perf stat -ddd kill                           (test-stat-detailed-3)
-  perf stat -e '{cycles,instructions}' kill     (test-stat-group1)
+  perf stat -e '{cycles,default_core/legacy-hardware-config=1/}' kill (test-stat-group1)
   perf stat -i -e cycles kill                   (test-stat-no-inherit)

index 86a940d7895d7404..766aadef2f1f9ffd 100644
@@ -1,6 +1,6 @@
 [config]
 command = record
-args    = --no-bpf-event -e '{cycles,cache-misses}:S' kill >/dev/null 2>&1
+args    = --no-bpf-event -e '{cycles,default_core/legacy-hardware-config=3/}:S' kill >/dev/null 2>&1
 ret     = 1
 kernel_until = 6.12
 

@@ -15,7 +15,7 @@ inherit=0
 fd=2
 group_fd=1
 
-# cache-misses
+# cache-misses, explicitly requested with the legacy hardware encoding
 type=0
 config=3
 

index 4748ab7bf68437c7..5be9a1a2a4ff0751 100644
@@ -1,6 +1,6 @@
 [config]
 command = record
-args    = --no-bpf-event -e '{cycles,cache-misses}:S' kill >/dev/null 2>&1
+args    = --no-bpf-event -e '{cycles,default_core/legacy-hardware-config=3/}:S' kill >/dev/null 2>&1
 ret     = 1
 kernel_since = 6.12
 

@@ -30,7 +30,7 @@ inherit=1
 fd=2
 group_fd=1
 
-# cache-misses
+# cache-misses, explicitly requested with the legacy hardware encoding
 type=0
 config=3
 

index e0432244a0eb7f6c..c05225167ec3464b 100644
@@ -1,6 +1,6 @@
 [config]
 command = record
-args    = --no-bpf-event -c 10000 -e '{cycles,cache-misses}:S' kill >/dev/null 2>&1
+args    = --no-bpf-event -c 10000 -e '{cycles,default_core/legacy-hardware-config=3/}:S' kill >/dev/null 2>&1
 ret     = 1
 kernel_since = 6.12
 

@@ -36,7 +36,7 @@ write_backward=0
 fd=2
 group_fd=1
 
-# cache-misses
+# cache-misses, explicitly requested with the legacy hardware encoding
 type=0
 config=3
 

index eeb1db392bc9ceac..0437ea4939e88865 100644
@@ -1,6 +1,6 @@
 [config]
 command = record
-args    = --no-bpf-event -e '{cycles,instructions}' kill >/dev/null 2>&1
+args    = --no-bpf-event -e '{cycles,default_core/legacy-hardware-config=1/}' kill >/dev/null 2>&1
 ret     = 1
 
 [event-1:base-record]

index 891d41a7bddf785a..ca5bbf441968fd8c 100644
@@ -1,6 +1,6 @@
 [config]
 command = record
-args    = --no-bpf-event -e '{cycles/period=1234000/,instructions/period=6789000/}:S' kill >/dev/null 2>&1
+args    = --no-bpf-event -e '{cycles/period=1234000/,default_core/legacy-hardware-config=1,period=6789000/}:S' kill >/dev/null 2>&1
 ret     = 1
 kernel_until = 6.12
 

index 249be884959eaaba..8191b3af936de5aa 100644
@@ -1,6 +1,6 @@
 [config]
 command = record
-args    = --no-bpf-event -e '{cycles/period=1234000/,instructions/period=6789000/}:S' kill >/dev/null 2>&1
+args    = --no-bpf-event -e '{cycles/period=1234000/,default_core/legacy-hardware-config=1,period=6789000/}:S' kill >/dev/null 2>&1
 ret     = 1
 kernel_since = 6.12
 

index 1746751123dcb2c4..fa2ca74e4cf29ad5 100644
@@ -1,6 +1,6 @@
 [config]
 command = stat
-args    = -e '{cycles,instructions}' kill >/dev/null 2>&1
+args    = -e '{cycles,default_core/legacy-hardware-config=1/}' kill >/dev/null 2>&1
 ret     = 1
 
 [event-1:base-stat]

2.55.0

From: Arnaldo Carvalho de Melo <[email protected]>

Since c7adeb0974f18da4 ("perf jevents: Add set of common metrics based on default ones") and 2dfc0cab3da4a205 ("perf stat: Add detail -d,-dd,-ddd metrics") the perf stat default metrics open their events in groups, so group leaders and members have read_format=15 and members are enabled without enable_on_exec, while standalone events keep read_format=3 and are disabled until exec.

Also, since b1c5efbfd92eb84d ("perf parse-events: Remove hard coded legacy hardware and cache parsing") bare hardware events resolve to the core PMU sysfs/JSON events (type 4), so add the raw encodings seen on Intel and AMD for instructions, branch-misses, cpu-cycles and stalled-cycles-frontend/backend to test-stat-default and test-stat-detailed-1/2/3.

Update base-stat to accept both the standalone and group variants of read_format, disabled and enable_on_exec.

---
 tools/perf/tests/shell/attr/base-stat         |  6 ++--
 tools/perf/tests/shell/attr/test-stat-default | 32 +++++++++++++++++--
 .../tests/shell/attr/test-stat-detailed-1     | 32 +++++++++++++++++--
 .../tests/shell/attr/test-stat-detailed-2     | 32 +++++++++++++++++--
 .../tests/shell/attr/test-stat-detailed-3     | 32 +++++++++++++++++--
 5 files changed, 123 insertions(+), 11 deletions(-)

index 67d6eea8947034e1..dda41a626cbca64e 100644
@@ -9,8 +9,8 @@ size=176
 config=0
 sample_period=0
 sample_type=65536
-read_format=3
-disabled=1
+read_format=3|15
+disabled=0|1
 inherit=1
 pinned=0
 exclusive=0

@@ -22,7 +22,7 @@ mmap=0
 comm=0
 freq=0
 inherit_stat=0
-enable_on_exec=1
+enable_on_exec=0|1
 task=0
 watermark=0
 precise_ip=0

index 8dd27c1fb661fb77..e79bc56389fda180 100644
@@ -179,11 +179,11 @@ type=4
 config=17039629
 optional=1
 
-# PERF_TYPE_RAW / CPU_CLK_UNHALTED.THREAD
+# PERF_TYPE_RAW / cpu-cycles (core PMU sysfs event; Intel 0x3c, AMD 0x76)
 [event22:base-stat]
 fd=22
 type=4
-config=60
+config=60|118
 optional=1
 
 # PERF_TYPE_RAW / INT_MISC.RECOVERY_CYCLES_ANY

@@ -233,4 +233,32 @@ optional=1
 fd=29
 type=4
 config=4269
+optional=1
+
+# PERF_TYPE_RAW / instructions (core PMU sysfs event)
+[event30:base-stat]
+fd=30
+type=4
+config=192
+optional=1
+
+# PERF_TYPE_RAW / branch-misses (core PMU sysfs event; Intel 0xc5, AMD 0xc3)
+[event31:base-stat]
+fd=31
+type=4
+config=197|195
+optional=1
+
+# PERF_TYPE_RAW / stalled-cycles-frontend (core PMU sysfs event; AMD 0xd0/0x287/0xa9)
+[event32:base-stat]
+fd=32
+type=4
+config=208|647|169
+optional=1
+
+# PERF_TYPE_RAW / stalled-cycles-backend (core PMU sysfs event; AMD 0xd1/0x187)
+[event33:base-stat]
+fd=33
+type=4
+config=209|391
 optional=1
\ No newline at end of file

index 12a2ebf4e64a0077..cbd1041cd6c0430c 100644
@@ -181,11 +181,11 @@ type=4
 config=17039629
 optional=1
 
-# PERF_TYPE_RAW / CPU_CLK_UNHALTED.THREAD
+# PERF_TYPE_RAW / cpu-cycles (core PMU sysfs event; Intel 0x3c, AMD 0x76)
 [event22:base-stat]
 fd=22
 type=4
-config=60
+config=60|118
 optional=1
 
 # PERF_TYPE_RAW / INT_MISC.RECOVERY_CYCLES_ANY

@@ -275,4 +275,32 @@ optional=1
 fd=33
 type=4
 config=4269
+optional=1
+
+# PERF_TYPE_RAW / instructions (core PMU sysfs event)
+[event34:base-stat]
+fd=34
+type=4
+config=192
+optional=1
+
+# PERF_TYPE_RAW / branch-misses (core PMU sysfs event; Intel 0xc5, AMD 0xc3)
+[event35:base-stat]
+fd=35
+type=4
+config=197|195
+optional=1
+
+# PERF_TYPE_RAW / stalled-cycles-frontend (core PMU sysfs event; AMD 0xd0/0x287/0xa9)
+[event36:base-stat]
+fd=36
+type=4
+config=208|647|169
+optional=1
+
+# PERF_TYPE_RAW / stalled-cycles-backend (core PMU sysfs event; AMD 0xd1/0x187)
+[event37:base-stat]
+fd=37
+type=4
+config=209|391
 optional=1
\ No newline at end of file

index 66ea25b7d38f8d59..c47b22b4ec30259f 100644
@@ -181,11 +181,11 @@ type=4
 config=17039629
 optional=1
 
-# PERF_TYPE_RAW / CPU_CLK_UNHALTED.THREAD
+# PERF_TYPE_RAW / cpu-cycles (core PMU sysfs event; Intel 0x3c, AMD 0x76)
 [event22:base-stat]
 fd=22
 type=4
-config=60
+config=60|118
 optional=1
 
 # PERF_TYPE_RAW / INT_MISC.RECOVERY_CYCLES_ANY

@@ -335,4 +335,32 @@ optional=1
 fd=39
 type=4
 config=4269
+optional=1
+
+# PERF_TYPE_RAW / instructions (core PMU sysfs event)
+[event40:base-stat]
+fd=40
+type=4
+config=192
+optional=1
+
+# PERF_TYPE_RAW / branch-misses (core PMU sysfs event; Intel 0xc5, AMD 0xc3)
+[event41:base-stat]
+fd=41
+type=4
+config=197|195
+optional=1
+
+# PERF_TYPE_RAW / stalled-cycles-frontend (core PMU sysfs event; AMD 0xd0/0x287/0xa9)
+[event42:base-stat]
+fd=42
+type=4
+config=208|647|169
+optional=1
+
+# PERF_TYPE_RAW / stalled-cycles-backend (core PMU sysfs event; AMD 0xd1/0x187)
+[event43:base-stat]
+fd=43
+type=4
+config=209|391
 optional=1
\ No newline at end of file

index 4a27bbfb9f87a538..a6251005097a9639 100644
@@ -181,11 +181,11 @@ type=4
 config=17039629
 optional=1
 
-# PERF_TYPE_RAW / CPU_CLK_UNHALTED.THREAD
+# PERF_TYPE_RAW / cpu-cycles (core PMU sysfs event; Intel 0x3c, AMD 0x76)
 [event22:base-stat]
 fd=22
 type=4
-config=60
+config=60|118
 optional=1
 
 # PERF_TYPE_RAW / INT_MISC.RECOVERY_CYCLES_ANY

@@ -355,4 +355,32 @@ optional=1
 fd=41
 type=4
 config=4269
+optional=1
+
+# PERF_TYPE_RAW / instructions (core PMU sysfs event)
+[event42:base-stat]
+fd=42
+type=4
+config=192
+optional=1
+
+# PERF_TYPE_RAW / branch-misses (core PMU sysfs event; Intel 0xc5, AMD 0xc3)
+[event43:base-stat]
+fd=43
+type=4
+config=197|195
+optional=1
+
+# PERF_TYPE_RAW / stalled-cycles-frontend (core PMU sysfs event; AMD 0xd0/0x287/0xa9)
+[event44:base-stat]
+fd=44
+type=4
+config=208|647|169
+optional=1
+
+# PERF_TYPE_RAW / stalled-cycles-backend (core PMU sysfs event; AMD 0xd1/0x187)
+[event45:base-stat]
+fd=45
+type=4
+config=209|391
 optional=1
\ No newline at end of file

2.55.0

From: Arnaldo Carvalho de Melo <[email protected]>

When matching the recorded events back against the expectations, attr.py requires every event opened in a group to match an expectation that declares the same group. The perf stat default metrics open the events of each metric in groups, but the test expectations only verify the event attributes, so test-stat-default fails with:

FAILED '.../test-stat-default' - group failure

Only enforce the group of a recorded event when the expectation it matches declares one. Expectations that ask for a group are still verified in both directions.

---
 tools/perf/tests/shell/lib/attr.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

index 68f06df9565a0811..3319abfd451c4e4f 100644
@@ -309,7 +309,7 @@ class Test(object):
             else:
                 raise Unsup(self)
 
-    def compare(self, expect, result):
+    def compare(self, expect, result, strict_groups=True):
         match = {}
 
         log.debug("  compare")

@@ -344,7 +344,10 @@ class Test(object):
             match[exp_name] = exp_list
 
         # For each defined group in the expected events
-        # check we match the same group in the result.
+        # check we match the same group in the result. When matching
+        # the result back against the expectations, events opened in a
+        # group may match an expectation that doesn't request a group,
+        # as such expectations only verify the event attributes.
         for exp_name, exp_event in expect.items():
             group = exp_event.group
 

@@ -353,6 +356,8 @@ class Test(object):
 
             for res_name in match[exp_name]:
                 res_group = result[res_name].group
+                if res_group == '' and not strict_groups:
+                    continue
                 if res_group not in match[group]:
                     raise Fail(self, 'group failure')
 

@@ -391,7 +396,7 @@ class Test(object):
 
             # do the expectation - results matching - both ways
             self.compare(self.expect, self.result)
-            self.compare(self.result, self.expect)
+            self.compare(self.result, self.expect, strict_groups=False)
 
         finally:
             # cleanup

2.55.0

On my machine, the current tmp.perf-tools-next which contains these changes fails like this.

  $ tools/perf/perf test -v 82
  ---- start ----
  test child forked, pid 931585
  /home/namhyung/project/linux/tools/perf/tests/shell/attr.sh: line 20: python: command not found
  ---- end(-1) ----
   82: Perf attribute expectations test                                                 : FAILED!

=== Test Summary ===
Passed main tests : 0

  Passed subtests   : 0
  Skipped tests     : 0
  Failed tests      : 1

List of failed tests:
82: Perf attribute expectations test

The below patch fixes the problem.

Thanks,
Namhyung

index 307e1535795a34dc..ca3ca4a193971ec1 100755
@@ -4,6 +4,10 @@
 
 err=0
 
+shelldir=$(dirname "$0")
+# shellcheck source=lib/setup_python.sh
+. "${shelldir}"/lib/setup_python.sh
+
 cleanup() {
   trap - EXIT TERM INT
 }

@@ -17,7 +21,7 @@ trap trap_cleanup EXIT TERM INT
 
 shelldir=$(dirname "$0")
 perf_path=$(which perf)
-python "${shelldir}"/lib/attr.py -d "${shelldir}"/attr -v -p "$perf_path"
+$PYTHON "${shelldir}"/lib/attr.py -d "${shelldir}"/attr -v -p "$perf_path"
 err=$?
 cleanup
 exit $err

Hmm.. even with the fix, it failed like this.

Thanks,
Namhyung

  $ tools/perf/perf test -v 82
  ---- start ----
  ...
  expected mmap=1, got 0
  expected comm=1, got 0
  expected freq=1, got 0
  expected enable_on_exec=1, got 0
  expected task=1, got 0
  expected sample_id_all=1, got 0
  expected branch_sample_type=16, got 0
  FAILED '/home/namhyung/project/linux/tools/perf/tests/shell/attr/test-record-branch-filter-any_call' - match failure
  ---- end(-1) ----
   82: Perf attribute expectations test                                                                      : FAILED!

=== Test Summary ===
Passed main tests : 0

  Passed subtests   : 0
  Skipped tests     : 0
  Failed tests      : 1

List of failed tests:
82: Perf attribute expectations test

I'll address Sashiko's reviews and test this in more machines, Intel hybrid, AMD and arm64.

Will also incorporate your fix, thanks for testing it!

- Arnaldo.

- Arnaldo

still without your patch:

root@number:~# perf test attribute
 82: Perf attribute expectations t: Ok

=== Test Summary ===
Passed main tests : 1

 Passed subtests   : 0
 Skipped tests     : 0
 Failed tests      : 0
 root@number:~# grep "model name" -m1 /proc/cpuinfo
 model name      : AMD Ryzen 9 9950X3D 16-Core Processor
 root@number:~# uname -a
 Linux number 7.2.7-200.fc44.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Sep 21
 18:25:57 UTC 2026 x86_64 GNU/Linux
 root@number:~#

I reproduced the build failure in an ARM64 Debian Trixie machine, applying your patch to try again.

- Arnaldo

For the change to use PYTHON:

I think we can fix the attribute test expectations separately.

Thanks,
Ian

I fixed then already, will submit tomorrow morning:

⬢ [acme@toolbx perf-tools-next]$ git log --oneline -22
b8d6660fb3333fe9 (HEAD -> perf/alta-test-evsel-fix-and-mlock) perf tests: Mark the shell tests not mapping perf ring buffers
745ef567f0fe5705 perf tests: Mark the C tests mapping perf ring buffers
ebb8034d6627621f perf test: Take turns using the perf mlock budget in the mmaping tests 433379c85c0affc5 perf mutex: Add a robust, process-shared mutex initializer
52277c8f4bc4a5f8 perf tests: Introduce a 'needs_mlock' suite flag
7be6c60f58acca20 perf test attr: State, spelled out in a line, why a test is skipped

2a6b500277a921d1 perf test attr: Ignore the group read_format bit the PMU count can remove                                                              392722445f2b0a15 perf test attr: Ignore the dummy tracking event and the terms it carries                                                               b4631e5b762de4a8 perf test attr: Accept either sample id bit perf asks for  
b2a70402b56b34e5 perf test attr: Introduce mask_bits(), generalizing hide_core_pmu_type()                                                               f6711283e99de095 perf evsel: Don't dump probe events to PERF_TEST_ATTR
13b60625f92619aa perf test attr: Ignore the core PMU type perf puts in 'config'                                                                         394bfd10f1923f0d perf evsel: Mark the event as supported when retrying an open

50a76fcdb7321c57 perf test attr: Relax group checking for ungrouped expectations
0887d3cc78ca9b69 perf test attr: Fix default stat metrics expectations 048fe0dc4cc566f0 perf test attr: Fix legacy event encodings in group tests
f1f3c53b46c5da4e perf test attr: Allow tests to require a core PMU 75929770e7f9a907 perf test attr: Fix record dwarf sample_type expectation d4c243e294dee22c perf test attr: Fix wrong size expectation for events
16e6b715c7ab376b perf test attr: Propagate the return value from the test to the wrapper 74128ffa6d71a0ef perf test attr: Source setup_python.sh instead of calling python 1dc462fc21490767 (perf-tools-next.korg/perf-tools-next, five/perf-tools-next, acme/perf-tools-next) perf test: Add a test for perf timechart --tui
⬢ [acme@toolbx perf-tools-next]$

- Arnaldo