Skip to contentSkip to conversation

perf tests: mmap: skip user counter tests when runtime PMU access is unavailable

PATCH In review Qiao Zhao 3 participants 11 messages Jan 20 – Jan 27 1 message not in the archive perf events

lore
Share & tools
lore

Qiao Zhao

author
#0 lore raw

PATCH 1/1 perf tests: mmap: skip user counter tests when runtime PMU access is unavailable

On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.

[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]

Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.

---
 tools/perf/tests/mmap-basic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/tests/mmap-basic.c +2 -2
index 3313c236104e..4881a641749b 100644
@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled)
 		}
 
 		if (!rdpmc_supported && pc->cap_user_rdpmc) {
-			pr_err("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n",
+			pr_debug("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n",
 				pmu->name, pc->cap_user_rdpmc);
-			ret = TEST_FAIL;
+			ret = TEST_SKIP;
 			goto cleanup;
 		}
 
Signature · 1 line
2.49.0
author
#1 lore raw

Hi Kim, Ian,

Do you mind reviewing this change?

Quoted 47 lines · Qiao Zhao

On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.

[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]

Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.

---
 tools/perf/tests/mmap-basic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/tests/mmap-basic.c +2 -3
index 3313c236104e..4881a641749b 100644
@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled)
                }
 
                if (!rdpmc_supported && pc->cap_user_rdpmc) {
-                       pr_err("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n",
+                       pr_debug("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n",
                                pmu->name, pc->cap_user_rdpmc);
-                       ret = TEST_FAIL;
+                       ret = TEST_SKIP;
                        goto cleanup;
                }
 
--

2.49.0

reviewer
#2 lore raw
Quoted 41 lines · Qiao Zhao

Hi Kim, Ian,

Do you mind reviewing this change?

Quoted 35 lines · Qiao Zhao

On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.

[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]

Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.

---
 tools/perf/tests/mmap-basic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/tests/mmap-basic.c +0 -0
index 3313c236104e..4881a641749b 100644
@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled)
                }
 
Quoted 6 lines

tools/perf/tests/mmap-basic.c static int test_stat_user_read(u64 event, enum user_read_state enabled) · line 313

Quoted 6 lines
tools/perf/tests/mmap-basic.c +2 -2
                if (!rdpmc_supported && pc->cap_user_rdpmc) {
-                       pr_err("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n",
+                       pr_debug("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n",
                                pmu->name, pc->cap_user_rdpmc);
-                       ret = TEST_FAIL;
+                       ret = TEST_SKIP;

Hi Qiao,

It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...

So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243

```
#ifdef __aarch64__
.config1 = 0x2, /* Request user access */
#endif
```

So 1 question is why the perf_event_open is succeeding with a request user access bit set in the config in a system where this is blocked?

I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324

So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the cap_uesr_rdpmc is available. So is this an ARM PMU driver bug? I wonder if the setting of cap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615
should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753

So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?

Thanks,
Ian

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines
Quoted 6 lines
tools/perf/tests/mmap-basic.c +0 -1
                        goto cleanup;
                }
 
--

2.49.0

author
#3 lore raw

Hi Ian,

Thanks for the follow-up questions. I've gathered some additional data from the affected ARM systems.
Environment:
- Architecture: ARM (aarch64)
- CPU: Ampere(R) Altra(R) Processor Q80-30
- Tested on both bare-metal and ARM KVM guests
- Observed behavior is identical on bare-metal and KVM
- This is not a heterogeneous / big.LITTLE system
- perf version: 6.19.rc6.g053966c344db

rdpmc sysfs:
On these systems, there is no rdpmc attribute exposed under the PMU devices:

# grep . /sys/bus/event_source/devices/*/rdpmc
grep: /sys/bus/event_source/devices/*/rdpmc: No such file or directory

So the “restricted PMU access” I referred to is not due to toggling a visible rdpmc sysfs knob.
From a user-space perspective, there is simply no rdpmc control file available.

PMU devices:
The core PMU present is:

# grep . /sys/bus/event_source/devices/*/type | grep armv8
/sys/bus/event_source/devices/armv8_pmuv3_0/type:10
(Other devices are DSU/CMN/DMC PMUs.)

Virtualization:
The same rdpmc behavior is observed when running under ARM KVM, so this does not appear to be virtualization-specific.

Observed inconsistency:
Even though perf_event_open() succeeds with .config1=0x2 on aarch64, and the mmap page report
cap_user_rdpmc=1, user-space rdpmc access fails at runtime, which is what causes the test to treat
rdpmc as unsupported.

At this point, I agree the key issue seems to be the mismatch between the reported capability (cap_user_rdpmc)
and the actual runtime behavior, rather than perf_event_open() failing. I don't yet have enough insight to say
whether this is an ARM PMU driver capability-reporting issue or an incorrect test assumption, but hopefully the
above data helps narrow it down.

Please let me know if more details would be useful.

Thanks,
Qiao

Quoted 104 lines · Ian Rogers
Quoted 41 lines · Qiao Zhao

Hi Kim, Ian,

Do you mind reviewing this change?

Quoted 35 lines · Qiao Zhao

On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.

[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]

Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.

---
 tools/perf/tests/mmap-basic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/tests/mmap-basic.c +0 -0
index 3313c236104e..4881a641749b 100644
@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled)
                }
 
Quoted 6 lines

tools/perf/tests/mmap-basic.c static int test_stat_user_read(u64 event, enum user_read_state enabled) · line 313

Quoted 6 lines
tools/perf/tests/mmap-basic.c +2 -2
                if (!rdpmc_supported && pc->cap_user_rdpmc) {
-                       pr_err("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n",
+                       pr_debug("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n",
                                pmu->name, pc->cap_user_rdpmc);
-                       ret = TEST_FAIL;
+                       ret = TEST_SKIP;

Hi Qiao,

It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...

So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243

```
#ifdef __aarch64__
.config1 = 0x2, /* Request user access */
#endif
```

So 1 question is why the perf_event_open is succeeding with a request user access bit set in the config in a system where this is blocked?

I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324

So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the cap_uesr_rdpmc is available. So is this an ARM PMU driver bug? I wonder if the setting of cap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615
should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753

So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?

Thanks,
Ian

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines
Quoted 6 lines
tools/perf/tests/mmap-basic.c +0 -1
                        goto cleanup;
                }
 
--

2.49.0

reviewer
#4 lore raw
Quoted 47 lines · Qiao Zhao

Hi Ian,

Thanks for the follow-up questions. I've gathered some additional data from the affected ARM systems.
Environment:
- Architecture: ARM (aarch64)
- CPU: Ampere(R) Altra(R) Processor Q80-30
- Tested on both bare-metal and ARM KVM guests
- Observed behavior is identical on bare-metal and KVM
- This is not a heterogeneous / big.LITTLE system
- perf version: 6.19.rc6.g053966c344db

rdpmc sysfs:
On these systems, there is no rdpmc attribute exposed under the PMU devices:

# grep . /sys/bus/event_source/devices/*/rdpmc
grep: /sys/bus/event_source/devices/*/rdpmc: No such file or directory

So the “restricted PMU access” I referred to is not due to toggling a visible rdpmc sysfs knob.
From a user-space perspective, there is simply no rdpmc control file available.

PMU devices:
The core PMU present is:

# grep . /sys/bus/event_source/devices/*/type | grep armv8
/sys/bus/event_source/devices/armv8_pmuv3_0/type:10
(Other devices are DSU/CMN/DMC PMUs.)

Virtualization:
The same rdpmc behavior is observed when running under ARM KVM, so this does not appear to be virtualization-specific.

Observed inconsistency:
Even though perf_event_open() succeeds with .config1=0x2 on aarch64, and the mmap page report
cap_user_rdpmc=1, user-space rdpmc access fails at runtime, which is what causes the test to treat
rdpmc as unsupported.

At this point, I agree the key issue seems to be the mismatch between the reported capability (cap_user_rdpmc)
and the actual runtime behavior, rather than perf_event_open() failing. I don't yet have enough insight to say
whether this is an ARM PMU driver capability-reporting issue or an incorrect test assumption, but hopefully the
above data helps narrow it down.

Quoted 1 line

Please let me know if more details would be useful.

That's pretty much every detail I could think to gather :-) I looked on an Ampere machine and also noticed the rdpmc file is missing from the armv8_pmuv3_0 device. Perhaps James and Leo know about this?

In the test we care about the values of rdpmc_supported and pc->cap_user_rdpmc. The rdpmc_supported value comes from:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n298

```
if (saved_user_read_state == USER_READ_UNKNOWN)
rdpmc_supported = pc->cap_user_rdpmc && pc->index;
else
rdpmc_supported = (enabled == USER_READ_ENABLED);
```

In the test in the commit message I believe enabled is USER_READ_ENABLED and as the rdpmc file is missing I think saved_user_read_state is USER_READ_UNKNOWN. That confuses me then as "if (!rdpmc_supported && pc->cap_user_rdpmc) {" being true means that rdpmc_supported should be false.

Could you do me a favor and step through the test in gdb just to see why rdpmc_supported is false?

Thanks,
Ian

Quoted 109 lines

Thanks,
Qiao

Quoted 104 lines · Ian Rogers
Quoted 41 lines · Qiao Zhao

Hi Kim, Ian,

Do you mind reviewing this change?

Quoted 35 lines · Qiao Zhao

On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.

[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]

Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.

---
 tools/perf/tests/mmap-basic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/tests/mmap-basic.c +0 -0
index 3313c236104e..4881a641749b 100644
@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled)
                }
 
Quoted 6 lines

tools/perf/tests/mmap-basic.c static int test_stat_user_read(u64 event, enum user_read_state enabled) · line 313

Quoted 6 lines
tools/perf/tests/mmap-basic.c +2 -2
                if (!rdpmc_supported && pc->cap_user_rdpmc) {
-                       pr_err("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n",
+                       pr_debug("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n",
                                pmu->name, pc->cap_user_rdpmc);
-                       ret = TEST_FAIL;
+                       ret = TEST_SKIP;

Hi Qiao,

It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...

So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243

```
#ifdef __aarch64__
.config1 = 0x2, /* Request user access */
#endif
```

So 1 question is why the perf_event_open is succeeding with a request user access bit set in the config in a system where this is blocked?

I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324

So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the cap_uesr_rdpmc is available. So is this an ARM PMU driver bug? I wonder if the setting of cap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615
should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753

So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?

Thanks,
Ian

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines
Quoted 6 lines
tools/perf/tests/mmap-basic.c +0 -1
                        goto cleanup;
                }
 
--

2.49.0

A message that is not in the archive (sent off-list or not yet received): CAATMXfnSp_O7+MPpVY3kXLTTJLLZos_TZzCmUaPimmyeyJnX-w@mail.gmail.com

reviewer
#6 lore raw
Quoted 12 lines · Qiao Zhao

Hi Ian,

I stepped through the test in gdb as you suggested. The control flow matches your reasoning, but there are ARM-specific details that explain why rdpmc_supported ends up false.

Here's what I observed in test_stat_user_read():
- Execution enters the saved_user_read_state == USER_READ_UNKNOWN branch
- In the failing cases, enabled is indeed USER_READ_ENABLED
- Execution reaches the line:

   rdpmc_supported = pc->cap_user_rdpmc && pc->index;
Quoted 7 lines

However, on this ARM system(Ampere, armv8_pmuv3_0), the perf_event_mmap_page itself is not accessible from user space at runtime.

Concretely, in gdb:
- pc is a non-NULL pointer (e.g. 0xfffff7ff3000)
- but attempts to read pc->cap_user_rdpmc or pc->index results in:
“Cannot access memory at address …”

Thanks for the detailed explanation again! I'm confused what, "Cannot access memory at address ..." means. Is the program generating a segfault, but then how is the value of pc->cap_user_rdpmc read? If the mmap fails then the test should fail:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n281

Thanks,
Ian

Quoted 266 lines

So although the kernel reports cap_user_rdpmc = 1 in the shared page layout, the page backing "pc" is not actually readable by user space on this system. As a result, the expression pc->cap_user_rdpmc && pc->index effectively evaluates as false at runtime, and the test reports rdpmc as unsupported.

This explains why the condition:
if (!rdpmc_supported && pc->cap_user_rdpmc)
is reached, even though saved_user_read_state is USER_READ_UNKNOWN and enabled is USER_READ_ENABLED.

In short, the mismatch seems to be between the reported rdpmc capability and the actual accessibility of the mmap page on ARM, rather than a control-flow issue in the test itself.

Please let me know if it would be useful for me to capture this with additional instrumentation.

For reference, here are some gdb debug steps and outputs:

```
---- end ----
124.1: Read samples using the mmap interface                         : Ok
--- start ---
User space counter reading 1
User space counter reading for PMU armv8_pmuv3_0

Breakpoint 1, test_stat_user_read (event=1, enabled=USER_READ_ENABLED) at tests/mmap-basic.c:298
298            if (saved_user_read_state == USER_READ_UNKNOWN)
Missing rpms, try: dnf --enablerepo='*debug*' install python3-libs-debuginfo-3.12.12-2.el10_1.aarch64
(gdb) continue
Continuing.
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1]
---- end ----
124.2: User space counter reading of instructions                    : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMU armv8_pmuv3_0

Breakpoint 1, test_stat_user_read (event=0, enabled=USER_READ_ENABLED) at tests/mmap-basic.c:298
298            if (saved_user_read_state == USER_READ_UNKNOWN)
(gdb) n
299                rdpmc_supported = pc->cap_user_rdpmc && pc->index;
(gdb) n
311                pr_err("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n",
(gdb) info locals
evsel = 0x930170
err = 0
rdpmc_supported = <optimized out>
counts = {{{val = 0, ena = 0, run = 0, id = 0, lost = 0}, values = {0, 0, 0, 0, 0}}}
saved_user_read_state = USER_READ_UNKNOWN
attr = {type = 0, size = 0, config = 0, {sample_period = 0, sample_freq = 0}, sample_type = 0, read_format = 0, disabled = 0, inherit = 0, pinned = 0, exclusive = 0, exclude_user = 0, exclude_kernel = 0,
  exclude_hv = 0, exclude_idle = 0, mmap = 0, comm = 0, freq = 0, inherit_stat = 0, enable_on_exec = 0, task = 0, watermark = 0, precise_ip = 0, mmap_data = 0, sample_id_all = 0, exclude_host = 0,
  exclude_guest = 0, exclude_callchain_kernel = 0, exclude_callchain_user = 0, mmap2 = 0, comm_exec = 0, use_clockid = 0, context_switch = 0, write_backward = 0, namespaces = 0, ksymbol = 0, bpf_event = 0,
  aux_output = 0, cgroup = 0, text_poke = 0, build_id = 0, inherit_thread = 0, remove_on_exec = 0, sigtrap = 0, defer_callchain = 0, defer_output = 0, __reserved_1 = 0, {wakeup_events = 0,
    wakeup_watermark = 0}, bp_type = 0, {bp_addr = 2, kprobe_func = 2, uprobe_path = 2, config1 = 2}, {bp_len = 0, kprobe_addr = 0, probe_offset = 0, config2 = 0}, branch_sample_type = 0, sample_regs_user = 0,
  sample_stack_user = 0, clockid = 0, sample_regs_intr = 0, aux_watermark = 0, sample_max_stack = 0, __reserved_2 = 0, aux_sample_size = 0, {aux_action = 0, {aux_start_paused = 0, aux_pause = 0, aux_resume = 0,
      __reserved_3 = 0}}, sig_data = 0, config3 = 0, config4 = 0}
pc = 0xfffff7ff3000
mapped = true
opened = true
pmu = 0x923b50
threads = 0x92c1d0
ret = <optimized out>
cleanup = <optimized out>
(gdb) print saved_user_read_state
$1 = USER_READ_UNKNOWN
(gdb) print enabled
$2 = USER_READ_ENABLED
(gdb) print pc->cap_user_rdpmc
Cannot access memory at address 0xfffff7ff3000
(gdb) print pc->index
Cannot access memory at address 0xfffff7ff300c
(gdb) print rdpmc_supported
$3 = <optimized out>
(gdb) info args
event = 0
enabled = USER_READ_ENABLED
```

Thanks,
- Qiao

Quoted 186 lines · Ian Rogers
Quoted 47 lines · Qiao Zhao

Hi Ian,

Thanks for the follow-up questions. I've gathered some additional data from the affected ARM systems.
Environment:
- Architecture: ARM (aarch64)
- CPU: Ampere(R) Altra(R) Processor Q80-30
- Tested on both bare-metal and ARM KVM guests
- Observed behavior is identical on bare-metal and KVM
- This is not a heterogeneous / big.LITTLE system
- perf version: 6.19.rc6.g053966c344db

rdpmc sysfs:
On these systems, there is no rdpmc attribute exposed under the PMU devices:

# grep . /sys/bus/event_source/devices/*/rdpmc
grep: /sys/bus/event_source/devices/*/rdpmc: No such file or directory

So the “restricted PMU access” I referred to is not due to toggling a visible rdpmc sysfs knob.
From a user-space perspective, there is simply no rdpmc control file available.

PMU devices:
The core PMU present is:

# grep . /sys/bus/event_source/devices/*/type | grep armv8
/sys/bus/event_source/devices/armv8_pmuv3_0/type:10
(Other devices are DSU/CMN/DMC PMUs.)

Virtualization:
The same rdpmc behavior is observed when running under ARM KVM, so this does not appear to be virtualization-specific.

Observed inconsistency:
Even though perf_event_open() succeeds with .config1=0x2 on aarch64, and the mmap page report
cap_user_rdpmc=1, user-space rdpmc access fails at runtime, which is what causes the test to treat
rdpmc as unsupported.

At this point, I agree the key issue seems to be the mismatch between the reported capability (cap_user_rdpmc)
and the actual runtime behavior, rather than perf_event_open() failing. I don't yet have enough insight to say
whether this is an ARM PMU driver capability-reporting issue or an incorrect test assumption, but hopefully the
above data helps narrow it down.

Quoted 1 line

Please let me know if more details would be useful.

That's pretty much every detail I could think to gather :-) I looked on an Ampere machine and also noticed the rdpmc file is missing from the armv8_pmuv3_0 device. Perhaps James and Leo know about this?

In the test we care about the values of rdpmc_supported and pc->cap_user_rdpmc. The rdpmc_supported value comes from:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n298

```
if (saved_user_read_state == USER_READ_UNKNOWN)
rdpmc_supported = pc->cap_user_rdpmc && pc->index;
else
rdpmc_supported = (enabled == USER_READ_ENABLED);
```

In the test in the commit message I believe enabled is USER_READ_ENABLED and as the rdpmc file is missing I think saved_user_read_state is USER_READ_UNKNOWN. That confuses me then as "if (!rdpmc_supported && pc->cap_user_rdpmc) {" being true means that rdpmc_supported should be false.

Could you do me a favor and step through the test in gdb just to see why rdpmc_supported is false?

Thanks,
Ian

Quoted 109 lines

Thanks,
Qiao

Quoted 104 lines · Ian Rogers
Quoted 41 lines · Qiao Zhao

Hi Kim, Ian,

Do you mind reviewing this change?

Quoted 35 lines · Qiao Zhao

On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.

[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]

Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.

---
 tools/perf/tests/mmap-basic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/tests/mmap-basic.c +0 -0
index 3313c236104e..4881a641749b 100644
@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled)
                }
 
Quoted 6 lines

tools/perf/tests/mmap-basic.c static int test_stat_user_read(u64 event, enum user_read_state enabled) · line 313

Quoted 6 lines
tools/perf/tests/mmap-basic.c +2 -2
                if (!rdpmc_supported && pc->cap_user_rdpmc) {
-                       pr_err("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n",
+                       pr_debug("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n",
                                pmu->name, pc->cap_user_rdpmc);
-                       ret = TEST_FAIL;
+                       ret = TEST_SKIP;

Hi Qiao,

It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...

So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243

```
#ifdef __aarch64__
.config1 = 0x2, /* Request user access */
#endif
```

So 1 question is why the perf_event_open is succeeding with a request user access bit set in the config in a system where this is blocked?

I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324

So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the cap_uesr_rdpmc is available. So is this an ARM PMU driver bug? I wonder if the setting of cap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615
should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753

So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?

Thanks,
Ian

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines
Quoted 6 lines
tools/perf/tests/mmap-basic.c +0 -1
                        goto cleanup;
                }
 
--

2.49.0

reviewer
#7 lore raw
In reply to Ian Rogers
Quoted 61 lines · Ian Rogers
Quoted 41 lines · Qiao Zhao

Hi Kim, Ian,

Do you mind reviewing this change?

Quoted 35 lines · Qiao Zhao

On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.

[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]

Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.

---
  tools/perf/tests/mmap-basic.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/tests/mmap-basic.c +0 -0
index 3313c236104e..4881a641749b 100644
@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled)
                 }
 
Quoted 6 lines

tools/perf/tests/mmap-basic.c static int test_stat_user_read(u64 event, enum user_read_state enabled) · line 313

Quoted 6 lines
tools/perf/tests/mmap-basic.c +2 -2
                 if (!rdpmc_supported && pc->cap_user_rdpmc) {
-                       pr_err("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n",
+                       pr_debug("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n",
                                 pmu->name, pc->cap_user_rdpmc);
-                       ret = TEST_FAIL;
+                       ret = TEST_SKIP;

Hi Qiao,

It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...

So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243

Quoted 7 lines
```
#ifdef __aarch64__
.config1 = 0x2, /* Request user access */
#endif
```

So 1 question is why the perf_event_open is succeeding with a request user access bit set in the config in a system where this is blocked?

That's because sysctl_perf_user_access isn't checked when opening the event, only when scheduling it in. I don't think that's a bug, but just how it is. I suppose one benefit is that it allows you to add or revoke the permission at run time without recreating events.

The canonical rdpmc example in perf_event.h does say the index should be checked before reading the event:

* index = pc->index;
* count = pc->offset;
* if (pc->cap_user_rdpmc && index) {
* width = pc->pmc_width;
* pmc = rdpmc(index - 1);
* }

The version of this test before commit 588d22b40480 ("perf test: Expand user space event reading (rdpmc) tests") treated "supported" and "enabled" as things that could exist separately and skipped if the system has /proc/sys/kernel/perf_user_access = 0:

  $ echo 0 > /proc/sys/kernel/perf_user_access
  $ perf test -vv "user space counter"
  ...
  userspace counter access not enabled
  4.2: User space counter reading of instructions  : Skip (permissions)
  ..

Setting /proc/sys/kernel/perf_user_access makes that version of the test and the current version of the test pass. That also aligns with what's currently in the libperf test tools/lib/perf/tests/test-evsel.c which also checks both supported and enabled separately for a skip.

So it looks like commit 588d22b40480 has added the assumption that "supported" must equal "enabled" if rdmpc was asked for, but that's probably not right.

I think Qiao's change is ok, except it could probably go a bit further because we don't need to check pc->cap_user_rdpmc if !rdpmc_supported. Maybe the original separate "is supported" (pc->cap_user_rdpmc) and "is enabled" (pc->index) checks are clearer. Each can result in a skip on their own.

Thanks
James

Quoted 34 lines

I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324

So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the cap_uesr_rdpmc is available. So is this an ARM PMU driver bug? I wonder if the setting of cap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615
should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753

So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?

Thanks,
Ian

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines
Quoted 6 lines
tools/perf/tests/mmap-basic.c +0 -1
                         goto cleanup;
                 }
 
--

2.49.0

reviewer
#8 lore raw
Quoted 95 lines · James Clark
Quoted 64 lines · Ian Rogers
Quoted 42 lines · Qiao Zhao

Hi Kim, Ian,

Do you mind reviewing this change?

Quoted 36 lines · Qiao Zhao

On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.

[Test Log]
124.2: User space counter reading of instructions                    : FAILED! --- start ---
User space counter reading 0
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles                          : FAILED! --- start ---
User space counter reading 1
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]

Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.

---
  tools/perf/tests/mmap-basic.c | 4 ++--

  1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/tests/mmap-basic.c +0 -0
index 3313c236104e..4881a641749b 100644
@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum 
 user_read_state enabled)
                  }
 
Quoted 8 lines

tools/perf/tests/mmap-basic.c static int test_stat_user_read(u64 event, enum user_read_state enabled) · line 313

Quoted 8 lines
tools/perf/tests/mmap-basic.c +2 -2
                  if (!rdpmc_supported && pc->cap_user_rdpmc) {
-                       pr_err("User space counter reading for PMU 
 %s [Failed unexpected unsupported counter access %d]\n",
+                       pr_debug("User space counter reading for PMU 
 %s is not supported (cap_user_rdpmc=%d), skipping!\n",
                                  pmu->name, pc->cap_user_rdpmc);
-                       ret = TEST_FAIL;
+                       ret = TEST_SKIP;

Hi Qiao,

It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...

So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools- next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243

Quoted 7 lines
```
#ifdef __aarch64__
.config1 = 0x2, /* Request user access */
#endif
```

So 1 question is why the perf_event_open is succeeding with a request user access bit set in the config in a system where this is blocked?

That's because sysctl_perf_user_access isn't checked when opening the event, only when scheduling it in. I don't think that's a bug, but just how it is. I suppose one benefit is that it allows you to add or revoke the permission at run time without recreating events.

The canonical rdpmc example in perf_event.h does say the index should be checked before reading the event:

  * index = pc->index;
  * count = pc->offset;
  * if (pc->cap_user_rdpmc && index) {
  *   width = pc->pmc_width;
  *   pmc = rdpmc(index - 1);
  * }

The version of this test before commit 588d22b40480 ("perf test: Expand user space event reading (rdpmc) tests") treated "supported" and "enabled" as things that could exist separately and skipped if the system has /proc/sys/kernel/perf_user_access = 0:

   $ echo 0 > /proc/sys/kernel/perf_user_access
   $ perf test -vv "user space counter"
   ...
   userspace counter access not enabled
   4.2: User space counter reading of instructions  : Skip (permissions)
   ..
Quoted 9 lines

Setting /proc/sys/kernel/perf_user_access makes that version of the test and the current version of the test pass. That also aligns with what's currently in the libperf test tools/lib/perf/tests/test-evsel.c which also checks both supported and enabled separately for a skip.

So it looks like commit 588d22b40480 has added the assumption that "supported" must equal "enabled" if rdmpc was asked for, but that's probably not right.

I think Qiao's change is ok, except it could probably go a bit further because we don't need to check pc->cap_user_rdpmc if !rdpmc_supported. Maybe the original separate "is supported" (pc->cap_user_rdpmc) and "is enabled" (pc->index) checks are clearer. Each can result in a skip on their own.

Thanks
James

Actually if we check /proc/sys/kernel/perf_user_access in the test we can keep it strict and assume that "supported" must equal "enabled" as long as perf_user_access is set. That might be better and keep the test useful. Otherwise it can start skipping if the feature breaks.

Quoted 35 lines
Quoted 34 lines

I'm not sure on the support for the rdpmc file on ARM but it looks to
be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools- next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324

So in the test where you've changed it we've determined that rdpmc
shouldn't be supported but the shared page is saying the
cap_uesr_rdpmc is available. So is this an ARM PMU driver bug?
I wonder if the setting of cap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools- next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615 should also be testing whether support has been disabled through the
rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools- next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753

So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?

Thanks,
Ian

[1] https://lore.kernel.org/lkml/24bc287a-a3e0-45f6- [email protected]/

Quoted 7 lines
Quoted 6 lines

                         goto cleanup;
                 }

Signature · 1 line
2.49.0
reviewer
#9 lore raw
Quoted 7 lines · James Clark
Quoted 6 lines · James Clark
Quoted 5 lines · Ian Rogers
Quoted 4 lines · Qiao Zhao

Hi Kim, Ian,

Do you mind reviewing this change?

[snip]

Quoted 63 lines
Quoted 63 lines
Quoted 13 lines

Hi Qiao,

It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...

So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243

Quoted 7 lines
```
#ifdef __aarch64__
.config1 = 0x2, /* Request user access */
#endif
```

So 1 question is why the perf_event_open is succeeding with a request user access bit set in the config in a system where this is blocked?

That's because sysctl_perf_user_access isn't checked when opening the event, only when scheduling it in. I don't think that's a bug, but just how it is. I suppose one benefit is that it allows you to add or revoke the permission at run time without recreating events.

The canonical rdpmc example in perf_event.h does say the index should be checked before reading the event:

* index = pc->index;
* count = pc->offset;
* if (pc->cap_user_rdpmc && index) {
* width = pc->pmc_width;
* pmc = rdpmc(index - 1);
* }

The version of this test before commit 588d22b40480 ("perf test: Expand user space event reading (rdpmc) tests") treated "supported" and "enabled" as things that could exist separately and skipped if the system has /proc/sys/kernel/perf_user_access = 0:

   $ echo 0 > /proc/sys/kernel/perf_user_access
   $ perf test -vv "user space counter"
   ...
   userspace counter access not enabled
   4.2: User space counter reading of instructions  : Skip (permissions)
   ..

Setting /proc/sys/kernel/perf_user_access makes that version of the test and the current version of the test pass. That also aligns with what's currently in the libperf test tools/lib/perf/tests/test-evsel.c which also checks both supported and enabled separately for a skip.

So it looks like commit 588d22b40480 has added the assumption that "supported" must equal "enabled" if rdmpc was asked for, but that's probably not right.

I think Qiao's change is ok, except it could probably go a bit further because we don't need to check pc->cap_user_rdpmc if !rdpmc_supported. Maybe the original separate "is supported" (pc->cap_user_rdpmc) and "is enabled" (pc->index) checks are clearer. Each can result in a skip on their own.

Quoted 8 lines
Quoted 3 lines

Thanks
James

Actually if we check /proc/sys/kernel/perf_user_access in the test we can keep it strict and assume that "supported" must equal "enabled" as long as perf_user_access is set. That might be better and keep the test useful. Otherwise it can start skipping if the feature breaks.

Thanks James, is /proc/sys/kernel/perf_user_access ARM specific? It seems very similar to the rdpmc file. I don't immediately see it on the first x86 I tested on. Checking both the rdpmc and /proc/sys/kernel/perf_user_access sounds like a good solution to the problem to me.

Thanks,
Ian

Quoted 30 lines
Quoted 30 lines
Quoted 30 lines

I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324

So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the cap_uesr_rdpmc is available. So is this an ARM PMU driver bug? I wonder if the setting of cap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615 should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753

So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?

Thanks,
Ian

[1] https://lore.kernel.org/lkml/24bc287a-a3e0-45f6-
[email protected]/

reviewer
#10 lore raw
Quoted 81 lines · Ian Rogers
Quoted 7 lines · James Clark
Quoted 6 lines · James Clark
Quoted 5 lines · Ian Rogers
Quoted 4 lines · Qiao Zhao

Hi Kim, Ian,

Do you mind reviewing this change?

[snip]

Quoted 63 lines
Quoted 63 lines
Quoted 13 lines

Hi Qiao,

It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...

So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243

Quoted 7 lines
```
#ifdef __aarch64__
.config1 = 0x2, /* Request user access */
#endif
```

So 1 question is why the perf_event_open is succeeding with a request user access bit set in the config in a system where this is blocked?

That's because sysctl_perf_user_access isn't checked when opening the event, only when scheduling it in. I don't think that's a bug, but just how it is. I suppose one benefit is that it allows you to add or revoke the permission at run time without recreating events.

The canonical rdpmc example in perf_event.h does say the index should be checked before reading the event:

* index = pc->index;
* count = pc->offset;
* if (pc->cap_user_rdpmc && index) {
* width = pc->pmc_width;
* pmc = rdpmc(index - 1);
* }

The version of this test before commit 588d22b40480 ("perf test: Expand user space event reading (rdpmc) tests") treated "supported" and "enabled" as things that could exist separately and skipped if the system has /proc/sys/kernel/perf_user_access = 0:

    $ echo 0 > /proc/sys/kernel/perf_user_access
    $ perf test -vv "user space counter"
    ...
    userspace counter access not enabled
    4.2: User space counter reading of instructions  : Skip (permissions)
    ..

Setting /proc/sys/kernel/perf_user_access makes that version of the test and the current version of the test pass. That also aligns with what's currently in the libperf test tools/lib/perf/tests/test-evsel.c which also checks both supported and enabled separately for a skip.

So it looks like commit 588d22b40480 has added the assumption that "supported" must equal "enabled" if rdmpc was asked for, but that's probably not right.

I think Qiao's change is ok, except it could probably go a bit further because we don't need to check pc->cap_user_rdpmc if !rdpmc_supported. Maybe the original separate "is supported" (pc->cap_user_rdpmc) and "is enabled" (pc->index) checks are clearer. Each can result in a skip on their own.

Quoted 8 lines
Quoted 3 lines

Thanks
James

Actually if we check /proc/sys/kernel/perf_user_access in the test we can keep it strict and assume that "supported" must equal "enabled" as long as perf_user_access is set. That might be better and keep the test useful. Otherwise it can start skipping if the feature breaks.

Quoted 9 lines

Thanks James, is /proc/sys/kernel/perf_user_access ARM specific? It seems very similar to the rdpmc file. I don't immediately see it on the first x86 I tested on. Checking both the rdpmc and /proc/sys/kernel/perf_user_access sounds like a good solution to the problem to me.

Thanks,
Ian

Arm and now riscv a well, so if we #ifdef it, it should be for those platforms. The discussions about why to do it that way instead of using the /sys/devices/cpu/rdpmc file like on x86 seem to be suggesting that multiple PMU types would make this less good than a global setting which you might want to change for security. Although there isn't much detail about pros and cons of consistency vs usability and then setting just appeared on version after that. The x86 /rdpmc semantics are default on as well which wasn't desired so maybe that was another reason to diverge.

Either way it's there now.

Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines

I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324

So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the cap_uesr_rdpmc is available. So is this an ARM PMU driver bug? I wonder if the setting of cap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615 should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753

So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?

Thanks,
Ian

[1] https://lore.kernel.org/lkml/24bc287a-a3e0-45f6-
[email protected]/

reviewer
#11 lore raw
Quoted 101 lines · James Clark
Quoted 81 lines · Ian Rogers
Quoted 7 lines · James Clark
Quoted 6 lines · James Clark
Quoted 5 lines · Ian Rogers
Quoted 4 lines · Qiao Zhao

Hi Kim, Ian,

Do you mind reviewing this change?

[snip]

Quoted 63 lines
Quoted 63 lines
Quoted 13 lines

Hi Qiao,

It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...

So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243

Quoted 7 lines
```
#ifdef __aarch64__
.config1 = 0x2, /* Request user access */
#endif
```

So 1 question is why the perf_event_open is succeeding with a request user access bit set in the config in a system where this is blocked?

That's because sysctl_perf_user_access isn't checked when opening the event, only when scheduling it in. I don't think that's a bug, but just how it is. I suppose one benefit is that it allows you to add or revoke the permission at run time without recreating events.

The canonical rdpmc example in perf_event.h does say the index should be checked before reading the event:

* index = pc->index;
* count = pc->offset;
* if (pc->cap_user_rdpmc && index) {
* width = pc->pmc_width;
* pmc = rdpmc(index - 1);
* }

The version of this test before commit 588d22b40480 ("perf test: Expand user space event reading (rdpmc) tests") treated "supported" and "enabled" as things that could exist separately and skipped if the system has /proc/sys/kernel/perf_user_access = 0:

    $ echo 0 > /proc/sys/kernel/perf_user_access
    $ perf test -vv "user space counter"
    ...
    userspace counter access not enabled
    4.2: User space counter reading of instructions  : Skip (permissions)
    ..

Setting /proc/sys/kernel/perf_user_access makes that version of the test and the current version of the test pass. That also aligns with what's currently in the libperf test tools/lib/perf/tests/test-evsel.c which also checks both supported and enabled separately for a skip.

So it looks like commit 588d22b40480 has added the assumption that "supported" must equal "enabled" if rdmpc was asked for, but that's probably not right.

I think Qiao's change is ok, except it could probably go a bit further because we don't need to check pc->cap_user_rdpmc if !rdpmc_supported. Maybe the original separate "is supported" (pc->cap_user_rdpmc) and "is enabled" (pc->index) checks are clearer. Each can result in a skip on their own.

Quoted 8 lines
Quoted 3 lines

Thanks
James

Actually if we check /proc/sys/kernel/perf_user_access in the test we can keep it strict and assume that "supported" must equal "enabled" as long as perf_user_access is set. That might be better and keep the test useful. Otherwise it can start skipping if the feature breaks.

Quoted 9 lines

Thanks James, is /proc/sys/kernel/perf_user_access ARM specific? It seems very similar to the rdpmc file. I don't immediately see it on the first x86 I tested on. Checking both the rdpmc and /proc/sys/kernel/perf_user_access sounds like a good solution to the problem to me.

Thanks,
Ian

Arm and now riscv a well, so if we #ifdef it, it should be for those platforms. The discussions about why to do it that way instead of using the /sys/devices/cpu/rdpmc file like on x86 seem to be suggesting that multiple PMU types would make this less good than a global setting which you might want to change for security. Although there isn't much detail about pros and cons of consistency vs usability and then setting just appeared on version after that. The x86 /rdpmc semantics are default on as well which wasn't desired so maybe that was another reason to diverge.

Quoted 1 line

Either way it's there now.

I wonder if we can try to read both files, no need for ifdefs. Looking at the man page, I see references to rdpmc but not to perf_user_access:
https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/man/man2/perf_event_open.2#n3394

The rdpmc file path isn't using the event_source version which is a bug, ie:
/sys/devices/cpu/rdpmc
should be:
/sys/bus/event_source/devices/cpu/rdpmc
but that's only true on non-hybrid x86 where the core PMU is cpu. Actually some paths are fixed:
https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/man/man2/perf_event_open.2#n3669

Thanks,
Ian

Quoted 31 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines

I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324

So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the cap_uesr_rdpmc is available. So is this an ARM PMU driver bug? I wonder if the setting of cap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615 should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753

So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?

Thanks,
Ian

[1] https://lore.kernel.org/lkml/24bc287a-a3e0-45f6-
[email protected]/