perf tests: mmap: skip user counter tests when runtime PMU access is unavailable
PATCH 1/1 perf tests: mmap: skip user counter tests when runtime PMU access is unavailable
On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.
[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMU armv8_pmuv3_0
User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]
Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.
--- tools/perf/tests/mmap-basic.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/perf/tests/mmap-basic.c +2 -2
@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled) }if (!rdpmc_supported && pc->cap_user_rdpmc) { -pr_err ("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n" , +pr_debug ("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n" , pmu->name, pc->cap_user_rdpmc); - ret =TEST_FAIL ; + ret =TEST_SKIP ; goto cleanup; }
Signature · 1 line
2.49.0
Hi Kim, Ian,
Do you mind reviewing this change?
Quoted 47 lines · Qiao Zhao
On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.
[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.
Signed-off-by Qiao Zhao <[email protected]>--- tools/perf/tests/mmap-basic.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)tools/perf/tests/mmap-basic.c +2
-3@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled) }if (!rdpmc_supported && pc->cap_user_rdpmc) { -pr_err ("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n" , +pr_debug ("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n" , pmu->name, pc->cap_user_rdpmc); - ret =TEST_FAIL ; + ret =TEST_SKIP ; goto cleanup; } --2.49.0
Ian Rogers
reviewerQuoted 41 lines · Qiao Zhao
Hi Kim, Ian,
Do you mind reviewing this change?
Quoted 35 lines · Qiao Zhao
On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.
[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.
Signed-off-by Qiao Zhao <[email protected]>--- tools/perf/tests/mmap-basic.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)tools/perf/tests/mmap-basic.c +0
-0@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled) }
Quoted 6 lines
tools/perf/tests/mmap-basic.c static int test_stat_user_read(u64 event, enum user_read_state enabled) · line 313
Quoted 6 lines
tools/perf/tests/mmap-basic.c +2
-2if (!rdpmc_supported && pc->cap_user_rdpmc) { -pr_err ("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n" , +pr_debug ("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n" , pmu->name, pc->cap_user_rdpmc); - ret =TEST_FAIL ; + ret =TEST_SKIP ;
Hi Qiao,
It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...
So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243
``` #ifdef __aarch64__ .config1 = 0x2, /* Request user access */ #endif ```
So 1 question is why the perf_event_open is succeeding with a request user access bit set in the config in a system where this is blocked?
I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324
So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the cap_uesr_rdpmc is available. So is this an ARM PMU driver bug? I wonder if the setting of cap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615
should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753
So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?
Thanks,
Ian
[1] https://lore.kernel.org/lkml/[email protected]/
Quoted 7 lines
Quoted 6 lines
tools/perf/tests/mmap-basic.c +0
-1goto cleanup; } --2.49.0
Hi Ian,
Thanks for the follow-up questions. I've gathered some additional data from the affected ARM systems.
Environment:
- Architecture: ARM (aarch64)
- CPU: Ampere(R) Altra(R) Processor Q80-30
- Tested on both bare-metal and ARM KVM guests
- Observed behavior is identical on bare-metal and KVM
- This is not a heterogeneous / big.LITTLE system
- perf version: 6.19.rc6.g053966c344db
rdpmc sysfs:
On these systems, there is no rdpmc attribute exposed under the PMU devices:
# grep . /sys/bus/event_source/devices/*/rdpmc grep: /sys/bus/event_source/devices/*/rdpmc: No such file or directory
So the “restricted PMU access” I referred to is not due to toggling a visible rdpmc sysfs knob.
From a user-space perspective, there is simply no rdpmc control file available.
PMU devices:
The core PMU present is:
# grep . /sys/bus/event_source/devices/*/type | grep armv8 /sys/bus/event_source/devices/armv8_pmuv3_0/type:10 (Other devices are DSU/CMN/DMC PMUs.)
Virtualization:
The same rdpmc behavior is observed when running under ARM KVM, so this does not appear to be virtualization-specific.
Observed inconsistency:
Even though perf_event_open() succeeds with .config1=0x2 on aarch64, and the mmap page report
cap_user_rdpmc=1, user-space rdpmc access fails at runtime, which is what causes the test to treat
rdpmc as unsupported.
At this point, I agree the key issue seems to be the mismatch between the reported capability (cap_user_rdpmc)
and the actual runtime behavior, rather than perf_event_open() failing. I don't yet have enough insight to say
whether this is an ARM PMU driver capability-reporting issue or an incorrect test assumption, but hopefully the
above data helps narrow it down.
Please let me know if more details would be useful.
Thanks,
Qiao
Quoted 104 lines · Ian Rogers
Quoted 41 lines · Qiao Zhao
Hi Kim, Ian,
Do you mind reviewing this change?
Quoted 35 lines · Qiao Zhao
On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.
[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.
Signed-off-by Qiao Zhao <[email protected]>--- tools/perf/tests/mmap-basic.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)tools/perf/tests/mmap-basic.c +0
-0@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled) }Quoted 6 lines
tools/perf/tests/mmap-basic.c static int test_stat_user_read(u64 event, enum user_read_state enabled) · line 313
Quoted 6 lines
tools/perf/tests/mmap-basic.c +2
-2if (!rdpmc_supported && pc->cap_user_rdpmc) { -pr_err ("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n" , +pr_debug ("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n" , pmu->name, pc->cap_user_rdpmc); - ret =TEST_FAIL ; + ret =TEST_SKIP ;Hi Qiao,
It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...
So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243``` #ifdef __aarch64__ .config1 = 0x2, /* Request user access */ #endif ```So 1 question is why the
perf_event_openis succeeding with a request user access bit set in the config in a system where this is blocked?I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the
cap_uesr_rdpmcis available. So is this an ARM PMU driver bug? I wonder if the setting ofcap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615
should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?
Thanks,
Ian[1] https://lore.kernel.org/lkml/[email protected]/
Quoted 7 lines
Quoted 6 lines
tools/perf/tests/mmap-basic.c +0
-1goto cleanup; } --2.49.0
Ian Rogers
reviewerQuoted 47 lines · Qiao Zhao
Hi Ian,
Thanks for the follow-up questions. I've gathered some additional data from the affected ARM systems.
Environment:
- Architecture: ARM (aarch64)
- CPU: Ampere(R) Altra(R) Processor Q80-30
- Tested on both bare-metal and ARM KVM guests
- Observed behavior is identical on bare-metal and KVM
- This is not a heterogeneous / big.LITTLE system
- perf version: 6.19.rc6.g053966c344dbrdpmc sysfs:
On these systems, there is no rdpmc attribute exposed under the PMU devices:# grep . /sys/bus/event_source/devices/*/rdpmc grep: /sys/bus/event_source/devices/*/rdpmc: No such file or directorySo the “restricted PMU access” I referred to is not due to toggling a visible rdpmc sysfs knob.
From a user-space perspective, there is simply no rdpmc control file available.PMU devices:
The core PMU present is:# grep . /sys/bus/event_source/devices/*/type | grep armv8 /sys/bus/event_source/devices/armv8_pmuv3_0/type:10 (Other devices are DSU/CMN/DMC PMUs.)Virtualization:
The same rdpmc behavior is observed when running under ARM KVM, so this does not appear to be virtualization-specific.Observed inconsistency:
Even thoughperf_event_open()succeeds with .config1=0x2 on aarch64, and the mmap page report
cap_user_rdpmc=1, user-space rdpmc access fails at runtime, which is what causes the test to treat
rdpmc as unsupported.At this point, I agree the key issue seems to be the mismatch between the reported capability (
cap_user_rdpmc)
and the actual runtime behavior, rather thanperf_event_open()failing. I don't yet have enough insight to say
whether this is an ARM PMU driver capability-reporting issue or an incorrect test assumption, but hopefully the
above data helps narrow it down.
Quoted 1 line
Please let me know if more details would be useful.
That's pretty much every detail I could think to gather :-) I looked on an Ampere machine and also noticed the rdpmc file is missing from the armv8_pmuv3_0 device. Perhaps James and Leo know about this?
In the test we care about the values of rdpmc_supported and pc->cap_user_rdpmc. The rdpmc_supported value comes from:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n298
``` if (saved_user_read_state == USER_READ_UNKNOWN) rdpmc_supported = pc->cap_user_rdpmc && pc->index; else rdpmc_supported = (enabled == USER_READ_ENABLED); ```
In the test in the commit message I believe enabled is USER_READ_ENABLED and as the rdpmc file is missing I think saved_user_read_state is USER_READ_UNKNOWN. That confuses me then as "if (!rdpmc_supported && pc->cap_user_rdpmc) {" being true means that rdpmc_supported should be false.
Could you do me a favor and step through the test in gdb just to see why rdpmc_supported is false?
Thanks,
Ian
Quoted 109 lines
Thanks,
QiaoQuoted 104 lines · Ian Rogers
Quoted 41 lines · Qiao Zhao
Hi Kim, Ian,
Do you mind reviewing this change?
Quoted 35 lines · Qiao Zhao
On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.
[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.
Signed-off-by Qiao Zhao <[email protected]>--- tools/perf/tests/mmap-basic.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)tools/perf/tests/mmap-basic.c +0
-0@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled) }Quoted 6 lines
tools/perf/tests/mmap-basic.c static int test_stat_user_read(u64 event, enum user_read_state enabled) · line 313
Quoted 6 lines
tools/perf/tests/mmap-basic.c +2
-2if (!rdpmc_supported && pc->cap_user_rdpmc) { -pr_err ("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n" , +pr_debug ("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n" , pmu->name, pc->cap_user_rdpmc); - ret =TEST_FAIL ; + ret =TEST_SKIP ;Hi Qiao,
It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...
So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243``` #ifdef __aarch64__ .config1 = 0x2, /* Request user access */ #endif ```So 1 question is why the
perf_event_openis succeeding with a request user access bit set in the config in a system where this is blocked?I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the
cap_uesr_rdpmcis available. So is this an ARM PMU driver bug? I wonder if the setting ofcap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615
should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?
Thanks,
Ian[1] https://lore.kernel.org/lkml/[email protected]/
Quoted 7 lines
Quoted 6 lines
tools/perf/tests/mmap-basic.c +0
-1goto cleanup; } --2.49.0
A message that is not in the archive (sent off-list or not yet received): CAATMXfnSp_O7+MPpVY3kXLTTJLLZos_TZzCmUaPimmyeyJnX-w@mail.gmail.com
Ian Rogers
reviewerQuoted 12 lines · Qiao Zhao
Hi Ian,
I stepped through the test in gdb as you suggested. The control flow matches your reasoning, but there are ARM-specific details that explain why
rdpmc_supportedends up false.Here's what I observed in
test_stat_user_read():
- Execution enters thesaved_user_read_state==USER_READ_UNKNOWNbranch
- In the failing cases, enabled is indeedUSER_READ_ENABLED
- Execution reaches the line:rdpmc_supported = pc->cap_user_rdpmc && pc->index;
Quoted 7 lines
However, on this ARM
system(Ampere, armv8_pmuv3_0), theperf_event_mmap_pageitself is not accessible from user space at runtime.Concretely, in gdb:
- pc is a non-NULL pointer (e.g.0xfffff7ff3000)
- but attempts to readpc->cap_user_rdpmcorpc->indexresults in:
“Cannot access memory at address …”
Thanks for the detailed explanation again! I'm confused what, "Cannot access memory at address ..." means. Is the program generating a segfault, but then how is the value of pc->cap_user_rdpmc read? If the mmap fails then the test should fail:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n281
Thanks,
Ian
Quoted 266 lines
So although the kernel reports
cap_user_rdpmc= 1 in the shared page layout, the page backing "pc" is not actually readable by user space on this system. As a result, the expressionpc->cap_user_rdpmc&&pc->indexeffectively evaluates as false at runtime, and the test reports rdpmc as unsupported.This explains why the condition:
if (!rdpmc_supported &&pc->cap_user_rdpmc)
is reached, even thoughsaved_user_read_stateisUSER_READ_UNKNOWNand enabled isUSER_READ_ENABLED.In short, the mismatch seems to be between the reported rdpmc capability and the actual accessibility of the mmap page on ARM, rather than a control-flow issue in the test itself.
Please let me know if it would be useful for me to capture this with additional instrumentation.
For reference, here are some gdb debug steps and outputs:
``` ---- end ---- 124.1: Read samples using the mmap interface : Ok --- start --- User space counter reading 1 User space counter reading for PMU armv8_pmuv3_0 Breakpoint 1, test_stat_user_read (event=1, enabled=USER_READ_ENABLED) at tests/mmap-basic.c:298 298 if (saved_user_read_state == USER_READ_UNKNOWN) Missing rpms, try: dnf --enablerepo='*debug*' install python3-libs-debuginfo-3.12.12-2.el10_1.aarch64 (gdb) continue Continuing. User space counter reading for PMU armv8_pmuv3_0 [Failed unexpected unsupported counter access 1] ---- end ---- 124.2: User space counter reading of instructions : FAILED! --- start --- User space counter reading 0 User space counter reading for PMU armv8_pmuv3_0 Breakpoint 1, test_stat_user_read (event=0, enabled=USER_READ_ENABLED) at tests/mmap-basic.c:298 298 if (saved_user_read_state == USER_READ_UNKNOWN) (gdb) n 299 rdpmc_supported = pc->cap_user_rdpmc && pc->index; (gdb) n 311 pr_err("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n", (gdb) info locals evsel = 0x930170 err = 0 rdpmc_supported = <optimized out> counts = {{{val = 0, ena = 0, run = 0, id = 0, lost = 0}, values = {0, 0, 0, 0, 0}}} saved_user_read_state = USER_READ_UNKNOWN attr = {type = 0, size = 0, config = 0, {sample_period = 0, sample_freq = 0}, sample_type = 0, read_format = 0, disabled = 0, inherit = 0, pinned = 0, exclusive = 0, exclude_user = 0, exclude_kernel = 0, exclude_hv = 0, exclude_idle = 0, mmap = 0, comm = 0, freq = 0, inherit_stat = 0, enable_on_exec = 0, task = 0, watermark = 0, precise_ip = 0, mmap_data = 0, sample_id_all = 0, exclude_host = 0, exclude_guest = 0, exclude_callchain_kernel = 0, exclude_callchain_user = 0, mmap2 = 0, comm_exec = 0, use_clockid = 0, context_switch = 0, write_backward = 0, namespaces = 0, ksymbol = 0, bpf_event = 0, aux_output = 0, cgroup = 0, text_poke = 0, build_id = 0, inherit_thread = 0, remove_on_exec = 0, sigtrap = 0, defer_callchain = 0, defer_output = 0, __reserved_1 = 0, {wakeup_events = 0, wakeup_watermark = 0}, bp_type = 0, {bp_addr = 2, kprobe_func = 2, uprobe_path = 2, config1 = 2}, {bp_len = 0, kprobe_addr = 0, probe_offset = 0, config2 = 0}, branch_sample_type = 0, sample_regs_user = 0, sample_stack_user = 0, clockid = 0, sample_regs_intr = 0, aux_watermark = 0, sample_max_stack = 0, __reserved_2 = 0, aux_sample_size = 0, {aux_action = 0, {aux_start_paused = 0, aux_pause = 0, aux_resume = 0, __reserved_3 = 0}}, sig_data = 0, config3 = 0, config4 = 0} pc = 0xfffff7ff3000 mapped = true opened = true pmu = 0x923b50 threads = 0x92c1d0 ret = <optimized out> cleanup = <optimized out> (gdb) print saved_user_read_state $1 = USER_READ_UNKNOWN (gdb) print enabled $2 = USER_READ_ENABLED (gdb) print pc->cap_user_rdpmc Cannot access memory at address 0xfffff7ff3000 (gdb) print pc->index Cannot access memory at address 0xfffff7ff300c (gdb) print rdpmc_supported $3 = <optimized out> (gdb) info args event = 0 enabled = USER_READ_ENABLED ```Thanks,
- QiaoQuoted 186 lines · Ian Rogers
Quoted 47 lines · Qiao Zhao
Hi Ian,
Thanks for the follow-up questions. I've gathered some additional data from the affected ARM systems.
Environment:
- Architecture: ARM (aarch64)
- CPU: Ampere(R) Altra(R) Processor Q80-30
- Tested on both bare-metal and ARM KVM guests
- Observed behavior is identical on bare-metal and KVM
- This is not a heterogeneous / big.LITTLE system
- perf version: 6.19.rc6.g053966c344dbrdpmc sysfs:
On these systems, there is no rdpmc attribute exposed under the PMU devices:# grep . /sys/bus/event_source/devices/*/rdpmc grep: /sys/bus/event_source/devices/*/rdpmc: No such file or directorySo the “restricted PMU access” I referred to is not due to toggling a visible rdpmc sysfs knob.
From a user-space perspective, there is simply no rdpmc control file available.PMU devices:
The core PMU present is:# grep . /sys/bus/event_source/devices/*/type | grep armv8 /sys/bus/event_source/devices/armv8_pmuv3_0/type:10 (Other devices are DSU/CMN/DMC PMUs.)Virtualization:
The same rdpmc behavior is observed when running under ARM KVM, so this does not appear to be virtualization-specific.Observed inconsistency:
Even thoughperf_event_open()succeeds with .config1=0x2 on aarch64, and the mmap page report
cap_user_rdpmc=1, user-space rdpmc access fails at runtime, which is what causes the test to treat
rdpmc as unsupported.At this point, I agree the key issue seems to be the mismatch between the reported capability (
cap_user_rdpmc)
and the actual runtime behavior, rather thanperf_event_open()failing. I don't yet have enough insight to say
whether this is an ARM PMU driver capability-reporting issue or an incorrect test assumption, but hopefully the
above data helps narrow it down.Quoted 1 line
Please let me know if more details would be useful.
That's pretty much every detail I could think to gather :-) I looked on an Ampere machine and also noticed the rdpmc file is missing from the
armv8_pmuv3_0device. Perhaps James and Leo know about this?In the test we care about the values of
rdpmc_supportedandpc->cap_user_rdpmc. Therdpmc_supportedvalue comes from:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n298``` if (saved_user_read_state == USER_READ_UNKNOWN) rdpmc_supported = pc->cap_user_rdpmc && pc->index; else rdpmc_supported = (enabled == USER_READ_ENABLED); ```In the test in the commit message I believe enabled is
USER_READ_ENABLEDand as the rdpmc file is missing I thinksaved_user_read_stateisUSER_READ_UNKNOWN. That confuses me then as "if (!rdpmc_supported &&pc->cap_user_rdpmc) {" being true means thatrdpmc_supportedshould be false.Could you do me a favor and step through the test in gdb just to see why
rdpmc_supportedis false?Thanks,
IanQuoted 109 lines
Thanks,
QiaoQuoted 104 lines · Ian Rogers
Quoted 41 lines · Qiao Zhao
Hi Kim, Ian,
Do you mind reviewing this change?
Quoted 35 lines · Qiao Zhao
On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.
[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.
Signed-off-by Qiao Zhao <[email protected]>--- tools/perf/tests/mmap-basic.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)tools/perf/tests/mmap-basic.c +0
-0@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled) }Quoted 6 lines
tools/perf/tests/mmap-basic.c static int test_stat_user_read(u64 event, enum user_read_state enabled) · line 313
Quoted 6 lines
tools/perf/tests/mmap-basic.c +2
-2if (!rdpmc_supported && pc->cap_user_rdpmc) { -pr_err ("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n" , +pr_debug ("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n" , pmu->name, pc->cap_user_rdpmc); - ret =TEST_FAIL ; + ret =TEST_SKIP ;Hi Qiao,
It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...
So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243``` #ifdef __aarch64__ .config1 = 0x2, /* Request user access */ #endif ```So 1 question is why the
perf_event_openis succeeding with a request user access bit set in the config in a system where this is blocked?I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the
cap_uesr_rdpmcis available. So is this an ARM PMU driver bug? I wonder if the setting ofcap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615
should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?
Thanks,
Ian[1] https://lore.kernel.org/lkml/[email protected]/
Quoted 7 lines
Quoted 6 lines
tools/perf/tests/mmap-basic.c +0
-1goto cleanup; } --2.49.0
James Clark
reviewerQuoted 61 lines · Ian Rogers
Quoted 41 lines · Qiao Zhao
Hi Kim, Ian,
Do you mind reviewing this change?
Quoted 35 lines · Qiao Zhao
On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.
[Test Log]
124.2: User space counter reading of instructions : FAILED!
--- start ---
User space counter reading 0
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED!
--- start ---
User space counter reading 1
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.
Signed-off-by Qiao Zhao <[email protected]>--- tools/perf/tests/mmap-basic.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)tools/perf/tests/mmap-basic.c +0
-0@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled) }Quoted 6 lines
tools/perf/tests/mmap-basic.c static int test_stat_user_read(u64 event, enum user_read_state enabled) · line 313
Quoted 6 lines
tools/perf/tests/mmap-basic.c +2
-2if (!rdpmc_supported && pc->cap_user_rdpmc) { -pr_err ("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n" , +pr_debug ("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n" , pmu->name, pc->cap_user_rdpmc); - ret =TEST_FAIL ; + ret =TEST_SKIP ;Hi Qiao,
It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...
So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243
Quoted 7 lines
``` #ifdef __aarch64__ .config1 = 0x2, /* Request user access */ #endif ```So 1 question is why the
perf_event_openis succeeding with a request user access bit set in the config in a system where this is blocked?
That's because sysctl_perf_user_access isn't checked when opening the event, only when scheduling it in. I don't think that's a bug, but just how it is. I suppose one benefit is that it allows you to add or revoke the permission at run time without recreating events.
The canonical rdpmc example in perf_event.h does say the index should be checked before reading the event:
* index = pc->index;
* count = pc->offset;
* if (pc->cap_user_rdpmc && index) {
* width = pc->pmc_width;
* pmc = rdpmc(index - 1);
* }
The version of this test before commit 588d22b40480 ("perf test: Expand user space event reading (rdpmc) tests") treated "supported" and "enabled" as things that could exist separately and skipped if the system has /proc/sys/kernel/perf_user_access = 0:
$ echo 0 > /proc/sys/kernel/perf_user_access $ perf test -vv "user space counter" ... userspace counter access not enabled 4.2: User space counter reading of instructions : Skip (permissions) ..
Setting /proc/sys/kernel/perf_user_access makes that version of the test and the current version of the test pass. That also aligns with what's currently in the libperf test tools/lib/perf/tests/test-evsel.c which also checks both supported and enabled separately for a skip.
So it looks like commit 588d22b40480 has added the assumption that "supported" must equal "enabled" if rdmpc was asked for, but that's probably not right.
I think Qiao's change is ok, except it could probably go a bit further because we don't need to check pc->cap_user_rdpmc if !rdpmc_supported. Maybe the original separate "is supported" (pc->cap_user_rdpmc) and "is enabled" (pc->index) checks are clearer. Each can result in a skip on their own.
Thanks
James
Quoted 34 lines
I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the
cap_uesr_rdpmcis available. So is this an ARM PMU driver bug? I wonder if the setting ofcap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615
should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?
Thanks,
Ian[1] https://lore.kernel.org/lkml/[email protected]/
Quoted 7 lines
Quoted 6 lines
tools/perf/tests/mmap-basic.c +0
-1goto cleanup; } --2.49.0
James Clark
reviewerQuoted 95 lines · James Clark
Quoted 64 lines · Ian Rogers
Quoted 42 lines · Qiao Zhao
Hi Kim, Ian,
Do you mind reviewing this change?
Quoted 36 lines · Qiao Zhao
On some platforms (e.g. ARM systems with restricted PMU access), user space PMU counters may be reported as supported but blocked at runtime. Failing the mmap interface tests in this case is incorrect.
[Test Log]
124.2: User space counter reading of instructions : FAILED! --- start ---
User space counter reading 0
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
124.3: User space counter reading of cycles : FAILED! --- start ---
User space counter reading 1
User space counter reading for PMUarmv8_pmuv3_0
User space counter reading for PMUarmv8_pmuv3_0[Failed unexpected unsupported counter access 1] ---- end ----
[/Test Log]Skip the user counter mmap tests when runtime PMU access is not available, while preserving failure for genuinely inconsistent states.
Signed-off-by Qiao Zhao <[email protected]>---
tools/perf/tests/mmap-basic.c| 4 ++--1 file changed, 2 insertions(+), 2 deletions(-)tools/perf/tests/mmap-basic.c +0
-0@@ -308,9 +308,9 @@ static int test_stat_user_read(u64 event, enum user_read_state enabled) }Quoted 8 lines
tools/perf/tests/mmap-basic.c static int test_stat_user_read(u64 event, enum user_read_state enabled) · line 313
Quoted 8 lines
tools/perf/tests/mmap-basic.c +2
-2if (!rdpmc_supported && pc->cap_user_rdpmc) { - pr_err("User space counter reading for PMU %s [Failed unexpected unsupported counter access %d]\n", + pr_debug("User space counter reading for PMU %s is not supported (cap_user_rdpmc=%d), skipping!\n", pmu->name, pc->cap_user_rdpmc); - ret = TEST_FAIL; + ret = TEST_SKIP;Hi Qiao,
It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...
So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools- next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243Quoted 7 lines
``` #ifdef __aarch64__ .config1 = 0x2, /* Request user access */ #endif ```So 1 question is why the
perf_event_openis succeeding with a request user access bit set in the config in a system where this is blocked?That's because
sysctl_perf_user_accessisn't checked when opening the event, only when scheduling it in. I don't think that's a bug, but just how it is. I suppose one benefit is that it allows you to add or revoke the permission at run time without recreating events.The canonical rdpmc example in
perf_event.hdoes say the index should be checked before reading the event:* index =
pc->index;
* count =pc->offset;
* if (pc->cap_user_rdpmc&& index) {
* width =pc->pmc_width;
* pmc = rdpmc(index - 1);
* }The version of this test before commit
588d22b40480("perf test: Expand user space event reading (rdpmc) tests") treated "supported" and "enabled" as things that could exist separately and skipped if the system has/proc/sys/kernel/perf_user_access= 0:$ echo 0 > /proc/sys/kernel/perf_user_access $ perf test -vv "user space counter" ... userspace counter access not enabled 4.2: User space counter reading of instructions : Skip (permissions) ..
Quoted 9 lines
Setting
/proc/sys/kernel/perf_user_accessmakes that version of the test and the current version of the test pass. That also aligns with what's currently in the libperf testtools/lib/perf/tests/test-evsel.cwhich also checks both supported and enabled separately for a skip.So it looks like commit
588d22b40480has added the assumption that "supported" must equal "enabled" if rdmpc was asked for, but that's probably not right.I think Qiao's change is ok, except it could probably go a bit further because we don't need to check
pc->cap_user_rdpmcif !rdpmc_supported. Maybe the original separate "is supported" (pc->cap_user_rdpmc) and "is enabled" (pc->index) checks are clearer. Each can result in a skip on their own.Thanks
James
Actually if we check /proc/sys/kernel/perf_user_access in the test we can keep it strict and assume that "supported" must equal "enabled" as long as perf_user_access is set. That might be better and keep the test useful. Otherwise it can start skipping if the feature breaks.
Quoted 35 lines
Quoted 34 lines
I'm not sure on the support for the rdpmc file on ARM but it looks to
be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools- next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324So in the test where you've changed it we've determined that rdpmc
shouldn't be supported but the shared page is saying thecap_uesr_rdpmcis available. So is this an ARM PMU driver bug?
I wonder if the setting ofcap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools- next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615 should also be testing whether support has been disabled through the
rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools- next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?
Thanks,
Ian[1] https://lore.kernel.org/lkml/24bc287a-a3e0-45f6- [email protected]/
Quoted 7 lines
Quoted 6 lines
goto cleanup;
}Signature · 1 line
2.49.0
Ian Rogers
reviewerQuoted 7 lines · James Clark
Quoted 6 lines · James Clark
Quoted 5 lines · Ian Rogers
Quoted 4 lines · Qiao Zhao
Hi Kim, Ian,
Do you mind reviewing this change?
[snip]
Quoted 63 lines
Quoted 63 lines
Quoted 13 lines
Hi Qiao,
It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...
So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243Quoted 7 lines
``` #ifdef __aarch64__ .config1 = 0x2, /* Request user access */ #endif ```So 1 question is why the
perf_event_openis succeeding with a request user access bit set in the config in a system where this is blocked?That's because
sysctl_perf_user_accessisn't checked when opening the event, only when scheduling it in. I don't think that's a bug, but just how it is. I suppose one benefit is that it allows you to add or revoke the permission at run time without recreating events.The canonical rdpmc example in
perf_event.hdoes say the index should be checked before reading the event:* index =
pc->index;
* count =pc->offset;
* if (pc->cap_user_rdpmc&& index) {
* width =pc->pmc_width;
* pmc = rdpmc(index - 1);
* }The version of this test before commit
588d22b40480("perf test: Expand user space event reading (rdpmc) tests") treated "supported" and "enabled" as things that could exist separately and skipped if the system has/proc/sys/kernel/perf_user_access= 0:$ echo 0 > /proc/sys/kernel/perf_user_access $ perf test -vv "user space counter" ... userspace counter access not enabled 4.2: User space counter reading of instructions : Skip (permissions) ..Setting
/proc/sys/kernel/perf_user_accessmakes that version of the test and the current version of the test pass. That also aligns with what's currently in the libperf testtools/lib/perf/tests/test-evsel.cwhich also checks both supported and enabled separately for a skip.So it looks like commit
588d22b40480has added the assumption that "supported" must equal "enabled" if rdmpc was asked for, but that's probably not right.I think Qiao's change is ok, except it could probably go a bit further because we don't need to check
pc->cap_user_rdpmcif !rdpmc_supported. Maybe the original separate "is supported" (pc->cap_user_rdpmc) and "is enabled" (pc->index) checks are clearer. Each can result in a skip on their own.
Quoted 8 lines
Quoted 3 lines
Thanks
JamesActually if we check
/proc/sys/kernel/perf_user_accessin the test we can keep it strict and assume that "supported" must equal "enabled" as long asperf_user_accessis set. That might be better and keep the test useful. Otherwise it can start skipping if the feature breaks.
Thanks James, is /proc/sys/kernel/perf_user_access ARM specific? It seems very similar to the rdpmc file. I don't immediately see it on the first x86 I tested on. Checking both the rdpmc and /proc/sys/kernel/perf_user_access sounds like a good solution to the problem to me.
Thanks,
Ian
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the
cap_uesr_rdpmcis available. So is this an ARM PMU driver bug? I wonder if the setting ofcap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615 should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?
Thanks,
Ian[1] https://lore.kernel.org/lkml/24bc287a-a3e0-45f6-
[email protected]/
James Clark
reviewerQuoted 81 lines · Ian Rogers
Quoted 7 lines · James Clark
Quoted 6 lines · James Clark
Quoted 5 lines · Ian Rogers
Quoted 4 lines · Qiao Zhao
Hi Kim, Ian,
Do you mind reviewing this change?
[snip]
Quoted 63 lines
Quoted 63 lines
Quoted 13 lines
Hi Qiao,
It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...
So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243Quoted 7 lines
``` #ifdef __aarch64__ .config1 = 0x2, /* Request user access */ #endif ```So 1 question is why the
perf_event_openis succeeding with a request user access bit set in the config in a system where this is blocked?That's because
sysctl_perf_user_accessisn't checked when opening the event, only when scheduling it in. I don't think that's a bug, but just how it is. I suppose one benefit is that it allows you to add or revoke the permission at run time without recreating events.The canonical rdpmc example in
perf_event.hdoes say the index should be checked before reading the event:* index =
pc->index;
* count =pc->offset;
* if (pc->cap_user_rdpmc&& index) {
* width =pc->pmc_width;
* pmc = rdpmc(index - 1);
* }The version of this test before commit
588d22b40480("perf test: Expand user space event reading (rdpmc) tests") treated "supported" and "enabled" as things that could exist separately and skipped if the system has/proc/sys/kernel/perf_user_access= 0:$ echo 0 > /proc/sys/kernel/perf_user_access $ perf test -vv "user space counter" ... userspace counter access not enabled 4.2: User space counter reading of instructions : Skip (permissions) ..Setting
/proc/sys/kernel/perf_user_accessmakes that version of the test and the current version of the test pass. That also aligns with what's currently in the libperf testtools/lib/perf/tests/test-evsel.cwhich also checks both supported and enabled separately for a skip.So it looks like commit
588d22b40480has added the assumption that "supported" must equal "enabled" if rdmpc was asked for, but that's probably not right.I think Qiao's change is ok, except it could probably go a bit further because we don't need to check
pc->cap_user_rdpmcif !rdpmc_supported. Maybe the original separate "is supported" (pc->cap_user_rdpmc) and "is enabled" (pc->index) checks are clearer. Each can result in a skip on their own.Quoted 8 lines
Quoted 3 lines
Thanks
JamesActually if we check
/proc/sys/kernel/perf_user_accessin the test we can keep it strict and assume that "supported" must equal "enabled" as long asperf_user_accessis set. That might be better and keep the test useful. Otherwise it can start skipping if the feature breaks.
Quoted 9 lines
Thanks James, is
/proc/sys/kernel/perf_user_accessARM specific? It seems very similar to the rdpmc file. I don't immediately see it on the first x86 I tested on. Checking both the rdpmc and/proc/sys/kernel/perf_user_accesssounds like a good solution to the problem to me.Thanks,
Ian
Arm and now riscv a well, so if we #ifdef it, it should be for those platforms. The discussions about why to do it that way instead of using the /sys/devices/cpu/rdpmc file like on x86 seem to be suggesting that multiple PMU types would make this less good than a global setting which you might want to change for security. Although there isn't much detail about pros and cons of consistency vs usability and then setting just appeared on version after that. The x86 /rdpmc semantics are default on as well which wasn't desired so maybe that was another reason to diverge.
Either way it's there now.
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the
cap_uesr_rdpmcis available. So is this an ARM PMU driver bug? I wonder if the setting ofcap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615 should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?
Thanks,
Ian[1] https://lore.kernel.org/lkml/24bc287a-a3e0-45f6-
[email protected]/
Ian Rogers
reviewerQuoted 101 lines · James Clark
Quoted 81 lines · Ian Rogers
Quoted 7 lines · James Clark
Quoted 6 lines · James Clark
Quoted 5 lines · Ian Rogers
Quoted 4 lines · Qiao Zhao
Hi Kim, Ian,
Do you mind reviewing this change?
[snip]
Quoted 63 lines
Quoted 63 lines
Quoted 13 lines
Hi Qiao,
It sounds like the ARM behavior is divergent from the Intel behavior, but that said Intel hybrid rdpmc is known to be problematic. There's a fix for part of that which will hopefully land [1]. One weird thing on Intel hybrid is that the rdpmc permission file within the PMU is kind of "shared" between all core PMUs, so disabling rdpmc on the e-core PMU disables it on the p-core. In one iteration of this test tried to incorporate support for this. Anyway, ARM...
So in the test we're doing:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/tools/perf/tests/mmap-basic.c?h=perf-tools-next#n243Quoted 7 lines
``` #ifdef __aarch64__ .config1 = 0x2, /* Request user access */ #endif ```So 1 question is why the
perf_event_openis succeeding with a request user access bit set in the config in a system where this is blocked?That's because
sysctl_perf_user_accessisn't checked when opening the event, only when scheduling it in. I don't think that's a bug, but just how it is. I suppose one benefit is that it allows you to add or revoke the permission at run time without recreating events.The canonical rdpmc example in
perf_event.hdoes say the index should be checked before reading the event:* index =
pc->index;
* count =pc->offset;
* if (pc->cap_user_rdpmc&& index) {
* width =pc->pmc_width;
* pmc = rdpmc(index - 1);
* }The version of this test before commit
588d22b40480("perf test: Expand user space event reading (rdpmc) tests") treated "supported" and "enabled" as things that could exist separately and skipped if the system has/proc/sys/kernel/perf_user_access= 0:$ echo 0 > /proc/sys/kernel/perf_user_access $ perf test -vv "user space counter" ... userspace counter access not enabled 4.2: User space counter reading of instructions : Skip (permissions) ..Setting
/proc/sys/kernel/perf_user_accessmakes that version of the test and the current version of the test pass. That also aligns with what's currently in the libperf testtools/lib/perf/tests/test-evsel.cwhich also checks both supported and enabled separately for a skip.So it looks like commit
588d22b40480has added the assumption that "supported" must equal "enabled" if rdmpc was asked for, but that's probably not right.I think Qiao's change is ok, except it could probably go a bit further because we don't need to check
pc->cap_user_rdpmcif !rdpmc_supported. Maybe the original separate "is supported" (pc->cap_user_rdpmc) and "is enabled" (pc->index) checks are clearer. Each can result in a skip on their own.Quoted 8 lines
Quoted 3 lines
Thanks
JamesActually if we check
/proc/sys/kernel/perf_user_accessin the test we can keep it strict and assume that "supported" must equal "enabled" as long asperf_user_accessis set. That might be better and keep the test useful. Otherwise it can start skipping if the feature breaks.Quoted 9 lines
Thanks James, is
/proc/sys/kernel/perf_user_accessARM specific? It seems very similar to the rdpmc file. I don't immediately see it on the first x86 I tested on. Checking both the rdpmc and/proc/sys/kernel/perf_user_accesssounds like a good solution to the problem to me.Thanks,
IanArm and now riscv a well, so if we #ifdef it, it should be for those platforms. The discussions about why to do it that way instead of using the
/sys/devices/cpu/rdpmcfile like on x86 seem to be suggesting that multiple PMU types would make this less good than a global setting which you might want to change for security. Although there isn't much detail about pros and cons of consistency vs usability and then setting just appeared on version after that. The x86 /rdpmc semantics are default on as well which wasn't desired so maybe that was another reason to diverge.
Quoted 1 line
Either way it's there now.
I wonder if we can try to read both files, no need for ifdefs. Looking at the man page, I see references to rdpmc but not to perf_user_access:
https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/man/man2/perf_event_open.2#n3394
The rdpmc file path isn't using the event_source version which is a bug, ie:/sys/devices/cpu/rdpmc
should be:/sys/bus/event_source/devices/cpu/rdpmc
but that's only true on non-hybrid x86 where the core PMU is cpu. Actually some paths are fixed:
https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/man/man2/perf_event_open.2#n3669
Thanks,
Ian
Quoted 31 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
I'm not sure on the support for the rdpmc file on ARM but it looks to be wired up:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n324So in the test where you've changed it we've determined that rdpmc shouldn't be supported but the shared page is saying the
cap_uesr_rdpmcis available. So is this an ARM PMU driver bug? I wonder if the setting ofcap_user_read:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/drivers/perf/arm_pmuv3.c?h=perf-tools-next#n1615 should also be testing whether support has been disabled through the rdpmc attribute? Hmm.. but x86 doesn't do that:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-
next.git/tree/arch/x86/events/core.c?h=perf-tools-next#n2753So given that ARM and x86 look in sync, x86 doesn't have issues I'm aware of, and it certainly looks like a failure inconsistency that the test thinks that user rdpmc support is disabled but the page says otherwise. I could use more details to understand this, like what's the contents of the rdpmc file? In your commit message, is the restricted PMU access the rdpmc file or a different capability? Could this be a weird hybrid/BIG.little issue like on x86?
Thanks,
Ian[1] https://lore.kernel.org/lkml/24bc287a-a3e0-45f6-
[email protected]/