Skip to contentSkip to conversation

arm64: Enable access to pmu registers by user-space

PATCH Superseded Raphael Gault 4 participants 14 messages Aug 16, 2019 – Aug 20, 2019

Subsystems (5)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover arm64: Enable access to pmu registers by user-space

Hi,

Changes since v2:
* Rebased on linux-next/master again (next-20190814)
* Use linux/compiler.h header as suggested by Arnaldo

The perf user-space tool relies on the PMU to monitor events. It offers an abstraction layer over the hardware counters since the underlying implementation is cpu-dependent. We want to allow userspace tools to have access to the registers storing the hardware counters' values directly. This targets specifically self-monitoring tasks in order to reduce the overhead by directly accessing the registers without having to go through the kernel.
In order to do this we need to setup the pmu so that it exposes its registers to userspace access.

The first patch add a test to the perf tool so that we can test that the access to the registers works correctly from userspace.

The second patch add a capability in the arm64 cpufeatures framework in order to detect when we are running on a heterogeneous system.

The third patch focuses on the armv8 pmuv3 PMU support and makes sure that the access to the pmu registers is enable and that the userspace have access to the relevent information in order to use them.

The fourth patch put in place callbacks to enable access to the hardware counters from userspace when a compatible event is opened using the perf API.

The fifth patch adds a short documentation about PMU counters direct access from userspace.

Raphael Gault (5):
perf: arm64: Add test to check userspace access to hardware counters.
arm64: cpufeature: Add feature to detect heterogeneous systems
arm64: pmu: Add function implementation to update event index in
userpage.
arm64: perf: Enable pmu counter direct access for perf event on armv8
Documentation: arm64: Document PMU counters access from userspace

 .../arm64/pmu_counter_user_access.txt         |  42 +++
 arch/arm64/include/asm/cpucaps.h              |   3 +-
 arch/arm64/include/asm/mmu.h                  |   6 +
 arch/arm64/include/asm/mmu_context.h          |   2 +
 arch/arm64/include/asm/perf_event.h           |  14 +
 arch/arm64/kernel/cpufeature.c                |  20 ++
 arch/arm64/kernel/perf_event.c                |  23 ++
 drivers/perf/arm_pmu.c                        |  38 +++
 include/linux/perf/arm_pmu.h                  |   2 +
 tools/perf/arch/arm64/include/arch-tests.h    |   7 +
 tools/perf/arch/arm64/tests/Build             |   1 +
 tools/perf/arch/arm64/tests/arch-tests.c      |   4 +
 tools/perf/arch/arm64/tests/user-events.c     | 254 ++++++++++++++++++
 13 files changed, 415 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/arm64/pmu_counter_user_access.txt
 create mode 100644 tools/perf/arch/arm64/tests/user-events.c
Signature · 1 line
2.17.1
#1 lore raw

PATCH v3 1/5 perf: arm64: Add test to check userspace access to hardware counters.

This test relies on the fact that the PMU registers are accessible from userspace. It then uses the perf_event_mmap_page to retrieve the counter index and access the underlying register.

This test uses sched_setaffinity(2) in order to run on all CPU and thus check the behaviour of the PMU of all cpus in a big.LITTLE environment.

---
 tools/perf/arch/arm64/include/arch-tests.h |   7 +
 tools/perf/arch/arm64/tests/Build          |   1 +
 tools/perf/arch/arm64/tests/arch-tests.c   |   4 +
 tools/perf/arch/arm64/tests/user-events.c  | 254 +++++++++++++++++++++
 4 files changed, 266 insertions(+)
 create mode 100644 tools/perf/arch/arm64/tests/user-events.c
tools/perf/arch/arm64/include/arch-tests.h +7 -0
index 90ec4c8cb880..6a8483de1015 100644
@@ -2,11 +2,18 @@
 #ifndef ARCH_TESTS_H
 #define ARCH_TESTS_H
 
+#include <linux/compiler.h>
+
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
 struct thread;
 struct perf_sample;
+int test__arch_unwind_sample(struct perf_sample *sample,
+			     struct thread *thread);
 #endif
 
 extern struct test arch_tests[];
+int test__rd_pmevcntr(struct test *test __maybe_unused,
+		      int subtest __maybe_unused);
+
 
 #endif
tools/perf/arch/arm64/tests/Build +1 -0
index a61c06bdb757..3f9a20c17fc6 100644
@@ -1,4 +1,5 @@
 perf-y += regs_load.o
 perf-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
 
+perf-y += user-events.o
 perf-y += arch-tests.o
tools/perf/arch/arm64/tests/arch-tests.c +4 -0
index 5b1543c98022..57df9b89dede 100644
@@ -10,6 +10,10 @@ struct test arch_tests[] = {
 		.func = test__dwarf_unwind,
 	},
 #endif
+	{
+		.desc = "User counter access",
+		.func = test__rd_pmevcntr,
+	},
 	{
 		.func = NULL,
 	},
tools/perf/arch/arm64/tests/user-events.c new file +254 -0

254 lines not on this page · raw message

Signature · 1 line
2.17.1
#2 lore raw

PATCH v3 2/5 arm64: cpufeature: Add feature to detect heterogeneous systems

In reply to Raphael Gault

This feature is required in order to enable PMU counters direct access from userspace only when the system is homogeneous. This feature checks the model of each CPU brought online and compares it to the boot CPU. If it differs then it is heterogeneous.

---
 arch/arm64/include/asm/cpucaps.h |  3 ++-
 arch/arm64/kernel/cpufeature.c   | 20 ++++++++++++++++++++
 arch/arm64/kernel/perf_event.c   |  1 +
 3 files changed, 23 insertions(+), 1 deletion(-)
arch/arm64/include/asm/cpucaps.h +2 -1
index f19fe4b9acc4..040370af38ad 100644
@@ -52,7 +52,8 @@
 #define ARM64_HAS_IRQ_PRIO_MASKING		42
 #define ARM64_HAS_DCPODP			43
 #define ARM64_WORKAROUND_1463225		44
+#define ARM64_HAS_HETEROGENEOUS_PMU		45
 
-#define ARM64_NCAPS				45
+#define ARM64_NCAPS				46
 
 #endif /* __ASM_CPUCAPS_H */
arch/arm64/kernel/cpufeature.c +20 -0
index 9323bcc40a58..bbdd809f12a6 100644
@@ -1260,6 +1260,15 @@ static bool can_use_gic_priorities(const struct arm64_cpu_capabilities *entry,
 }
 #endif
 
+static bool has_heterogeneous_pmu(const struct arm64_cpu_capabilities *entry,
+				     int scope)
+{
+	u32 model = read_cpuid_id() & MIDR_CPU_MODEL_MASK;
+	struct cpuinfo_arm64 *boot = &per_cpu(cpu_data, 0);
+
+	return  (boot->reg_midr & MIDR_CPU_MODEL_MASK) != model;
+}
+
 static const struct arm64_cpu_capabilities arm64_features[] = {
 	{
 		.desc = "GIC system register CPU interface",
@@ -1560,6 +1569,16 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.min_field_value = 1,
 	},
 #endif
+	{
+		/*
+		 * Detect whether the system is heterogeneous or
+		 * homogeneous
+		 */
+		.desc = "Detect whether we have heterogeneous CPUs",
+		.capability = ARM64_HAS_HETEROGENEOUS_PMU,
+		.type = ARM64_CPUCAP_SCOPE_LOCAL_CPU | ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU,
+		.matches = has_heterogeneous_pmu,
+	},
 	{},
 };
 
@@ -1727,6 +1746,7 @@ static void __init setup_elf_hwcaps(const struct arm64_cpu_capabilities *hwcaps)
 			cap_set_elf_hwcap(hwcaps);
 }
 
+
 static void update_cpu_capabilities(u16 scope_mask)
 {
 	int i;
arch/arm64/kernel/perf_event.c +1 -0
index 2d3bdebdf6df..a0b4f1bca491 100644
@@ -19,6 +19,7 @@
 #include <linux/of.h>
 #include <linux/perf/arm_pmu.h>
 #include <linux/platform_device.h>
+#include <linux/smp.h>
 
 /* ARMv8 Cortex-A53 specific event types. */
 #define ARMV8_A53_PERFCTR_PREF_LINEFILL				0xC2
Signature · 1 line
2.17.1
maintainer
#3 lore raw

Re: [PATCH v3 2/5] arm64: cpufeature: Add feature to detect heterogeneous systems

Hi Raphael,

Quoted 4 lines · Raphael Gault

This feature is required in order to enable PMU counters direct access from userspace only when the system is homogeneous. This feature checks the model of each CPU brought online and compares it to the boot CPU. If it differs then it is heterogeneous.

I t would be worth noting that this patch prevents heterogeneous CPUs being brought online late if the system was uniform at boot time.

Quoted 28 lines
---
 arch/arm64/include/asm/cpucaps.h |  3 ++-
 arch/arm64/kernel/cpufeature.c   | 20 ++++++++++++++++++++
 arch/arm64/kernel/perf_event.c   |  1 +
 3 files changed, 23 insertions(+), 1 deletion(-)
arch/arm64/include/asm/cpucaps.h +2 -1
index f19fe4b9acc4..040370af38ad 100644
@@ -52,7 +52,8 @@
 #define ARM64_HAS_IRQ_PRIO_MASKING		42
 #define ARM64_HAS_DCPODP			43
 #define ARM64_WORKAROUND_1463225		44
+#define ARM64_HAS_HETEROGENEOUS_PMU		45
 
-#define ARM64_NCAPS				45
+#define ARM64_NCAPS				46
 
 #endif /* __ASM_CPUCAPS_H */
arch/arm64/kernel/cpufeature.c +0 -0
index 9323bcc40a58..bbdd809f12a6 100644
@@ -1260,6 +1260,15 @@ static bool can_use_gic_priorities(const struct arm64_cpu_capabilities *entry,
 }

arch/arm64/kernel/cpufeature.c static bool can_use_gic_priorities(const struct arm64_cpu_capabilities *entry, · line 1270

Quoted 10 lines
arch/arm64/kernel/cpufeature.c +8 -0
 #endif
 
+static bool has_heterogeneous_pmu(const struct arm64_cpu_capabilities *entry,
+				     int scope)
+{
+	u32 model = read_cpuid_id() & MIDR_CPU_MODEL_MASK;
+	struct cpuinfo_arm64 *boot = &per_cpu(cpu_data, 0);
+
+	return  (boot->reg_midr & MIDR_CPU_MODEL_MASK) != model;
+}

We should use boot_cpu_data rather than &per_cpu(cpu_data, 0) here. We can make that __ro_after_init, and declare it in arch/arm64/includ/asm/smp.h.

That caters for CPU0 being hotplugged off and then a different physical CPU being hotplugged on in its place.

Quoted 5 lines
arch/arm64/kernel/cpufeature.c +1 -0
+
 static const struct arm64_cpu_capabilities arm64_features[] = {
 	{
 		.desc = "GIC system register CPU interface",

arch/arm64/kernel/cpufeature.c static const struct arm64_cpu_capabilities arm64_features[] = { · line 1577

Quoted 9 lines
arch/arm64/kernel/cpufeature.c +6 -0
@@ -1560,6 +1569,16 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.min_field_value = 1,
 	},
 #endif
+	{
+		/*
+		 * Detect whether the system is heterogeneous or
+		 * homogeneous
+		 */
+		.desc = "Detect whether we have heterogeneous CPUs",

The desc gets printed in dmesg with a prefix, e.g.

[    0.058267][    T1] CPU features: detected: Privileged Access Never
[    0.058340][    T1] CPU features: detected: LSE atomic instructions
[    0.058416][    T1] CPU features: detected: RAS Extension Support
[    0.058489][    T1] CPU features: detected: CRC32 instructions

... so this should only say "Heterogeneous CPUs".

arch/arm64/kernel/cpufeature.c static void __init setup_elf_hwcaps(const struct arm64_cpu_capabilities *hwcaps) · line 1749

Quoted 12 lines
arch/arm64/kernel/cpufeature.c +5 -0
+		.capability = ARM64_HAS_HETEROGENEOUS_PMU,
+		.type = ARM64_CPUCAP_SCOPE_LOCAL_CPU | ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU,
+		.matches = has_heterogeneous_pmu,
+	},
 	{},
 };
 
@@ -1727,6 +1746,7 @@ static void __init setup_elf_hwcaps(const struct arm64_cpu_capabilities *hwcaps)
 			cap_set_elf_hwcap(hwcaps);
 }
 
+

This whitespace addition can go.

arch/arm64/kernel/perf_event.c line 22

Quoted 12 lines
arch/arm64/kernel/cpufeature.c +0 -0
 static void update_cpu_capabilities(u16 scope_mask)
 {
 	int i;
arch/arm64/kernel/perf_event.c +1 -0
index 2d3bdebdf6df..a0b4f1bca491 100644
@@ -19,6 +19,7 @@
 #include <linux/of.h>
 #include <linux/perf/arm_pmu.h>
 #include <linux/platform_device.h>
+#include <linux/smp.h>

I think this should be added in a separate patch.

It looks like this is a missing include that we need today for smp_processor_id(), so please spin that as a preparatory patch (with my Acked-by).

Thanks,
Mark.

Quoted 6 lines
arch/arm64/kernel/perf_event.c +0 -0
 /* ARMv8 Cortex-A53 specific event types. */
 #define ARMV8_A53_PERFCTR_PREF_LINEFILL				0xC2
Signature · 1 line
2.17.1
maintainer
#4 lore raw

Re: [PATCH v3 2/5] arm64: cpufeature: Add feature to detect heterogeneous systems

Quoted 10 lines · Mark Rutland

Hi Raphael,

Quoted 4 lines · Raphael Gault

This feature is required in order to enable PMU counters direct access from userspace only when the system is homogeneous. This feature checks the model of each CPU brought online and compares it to the boot CPU. If it differs then it is heterogeneous.

It would be worth noting that this patch prevents heterogeneous CPUs being brought online late if the system was uniform at boot time.

Looking again, I think I'd misunderstood how ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU was dealt with, but we do have a problem in this area.

[...]

Quoted 5 lines

arch/arm64/kernel/cpufeature.c static const struct arm64_cpu_capabilities arm64_features[] = { · line 1581

Quoted 5 lines
arch/arm64/kernel/cpufeature.c +4 -0
+		.capability = ARM64_HAS_HETEROGENEOUS_PMU,
+		.type = ARM64_CPUCAP_SCOPE_LOCAL_CPU | ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU,
+		.matches = has_heterogeneous_pmu,
+	},

I had a quick chat with Will, and we concluded that we must permit late onlining of heterogeneous CPUs here as people are likely to rely on late CPU onlining on some heterogeneous systems.

I think the above permits that, but that also means that we need some support code to fail gracefully in that case (e.g. without sending a SIGILL to unaware userspace code).

That means that we'll need the counter emulation code that you had in previous versions of this patch (e.g. to handle potential UNDEFs when a new CPU has fewer counters than the previously online CPUs).

Further, I think the context switch (and event index) code needs to take this cap into account, and disable direct access once the system becomes heterogeneous.

Thanks,
Mark.

#5 lore raw

Re: [PATCH v3 2/5] arm64: cpufeature: Add feature to detect heterogeneous systems

Hi Mark,

Thank you for your comments.

Quoted 28 lines · Mark Rutland
Quoted 10 lines · Mark Rutland

Hi Raphael,

Quoted 4 lines · Raphael Gault

This feature is required in order to enable PMU counters direct access from userspace only when the system is homogeneous. This feature checks the model of each CPU brought online and compares it to the boot CPU. If it differs then it is heterogeneous.

It would be worth noting that this patch prevents heterogeneous CPUs being brought online late if the system was uniform at boot time.

Looking again, I think I'd misunderstood how ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU was dealt with, but we do have a problem in this area.

[...]

Quoted 5 lines

arch/arm64/kernel/cpufeature.c static const struct arm64_cpu_capabilities arm64_features[] = { · line 1581

Quoted 5 lines
arch/arm64/kernel/cpufeature.c +4 -0
+		.capability = ARM64_HAS_HETEROGENEOUS_PMU,
+		.type = ARM64_CPUCAP_SCOPE_LOCAL_CPU | ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU,
+		.matches = has_heterogeneous_pmu,
+	},

I had a quick chat with Will, and we concluded that we must permit late onlining of heterogeneous CPUs here as people are likely to rely on late CPU onlining on some heterogeneous systems.

Quoted 3 lines

I think the above permits that, but that also means that we need some support code to fail gracefully in that case (e.g. without sending a SIGILL to unaware userspace code).

I understand, however, I understood that ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU did not allow later CPU to be heterogeneous if the capability wasn't already enabled. Thus if as you say we need to allow the system to switch from homogeneous to heterogeneous, then I should change the type of this capability.

Quoted 7 lines

That means that we'll need the counter emulation code that you had in previous versions of this patch (e.g. to handle potential UNDEFs when a new CPU has fewer counters than the previously online CPUs).

Further, I think the context switch (and event index) code needs to take this cap into account, and disable direct access once the system becomes heterogeneous.

That is a good point indeed.

Thanks,

Signature · 1 line
Raphael Gault
maintainer
#6 lore raw

Re: [PATCH v3 2/5] arm64: cpufeature: Add feature to detect heterogeneous systems

Quoted 33 lines · Raphael Gault

Hi Mark,

Thank you for your comments.

Quoted 28 lines · Mark Rutland
Quoted 10 lines · Mark Rutland

Hi Raphael,

Quoted 4 lines · Raphael Gault

This feature is required in order to enable PMU counters direct access from userspace only when the system is homogeneous. This feature checks the model of each CPU brought online and compares it to the boot CPU. If it differs then it is heterogeneous.

It would be worth noting that this patch prevents heterogeneous CPUs being brought online late if the system was uniform at boot time.

Looking again, I think I'd misunderstood how ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU was dealt with, but we do have a problem in this area.

[...]

Quoted 5 lines

arch/arm64/kernel/cpufeature.c static const struct arm64_cpu_capabilities arm64_features[] = { · line 1581

Quoted 5 lines
arch/arm64/kernel/cpufeature.c +4 -0
+		.capability = ARM64_HAS_HETEROGENEOUS_PMU,
+		.type = ARM64_CPUCAP_SCOPE_LOCAL_CPU | ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU,
+		.matches = has_heterogeneous_pmu,
+	},

I had a quick chat with Will, and we concluded that we must permit late onlining of heterogeneous CPUs here as people are likely to rely on late CPU onlining on some heterogeneous systems.

Quoted 7 lines
Quoted 3 lines

I think the above permits that, but that also means that we need some support code to fail gracefully in that case (e.g. without sending a SIGILL to unaware userspace code).

I understand, however, I understood that ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU did not allow later CPU to be heterogeneous if the capability wasn't already enabled.

Yes, I think that you're right. IIUC the absence of ARM64_CPUCAP_PERMITTED_FOR_LATE_CPU is what prevents that from happening.

Quoted 3 lines

Thus if as you say we need to allow the system to switch from homogeneous to heterogeneous, then I should change the type of this capability.

I'm afraid so!

I believe we need both ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU and ARM64_CPUCAP_PERMITTED_FOR_LATE_CPU, so I guess we should be using ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE.

Does that sound right to you? ... or have I confused myself again?

Thanks,
Mark.

Quoted 14 lines
Quoted 7 lines

That means that we'll need the counter emulation code that you had in previous versions of this patch (e.g. to handle potential UNDEFs when a new CPU has fewer counters than the previously online CPUs).

Further, I think the context switch (and event index) code needs to take this cap into account, and disable direct access once the system becomes heterogeneous.

That is a good point indeed.

Thanks,

Signature · 1 line
Raphael Gault
#7 lore raw

PATCH v3 3/5 arm64: pmu: Add function implementation to update event index in userpage.

In reply to Raphael Gault

In order to be able to access the counter directly for userspace, we need to provide the index of the counter using the userpage. We thus need to override the event_idx function to retrieve and convert the perf_event index to armv8 hardware index.

Since the arm_pmu driver can be used by any implementation, even if not armv8, two components play a role into making sure the behaviour is correct and consistent with the PMU capabilities:

* the ARMPMU_EL0_RD_CNTR flag which denotes the capability to access counter from userspace.
* the event_idx call back, which is implemented and initialized by the PMU implementation: if no callback is provided, the default behaviour applies, returning 0 as index value.

---
 arch/arm64/kernel/perf_event.c | 22 ++++++++++++++++++++++
 include/linux/perf/arm_pmu.h   |  2 ++
 2 files changed, 24 insertions(+)
arch/arm64/kernel/perf_event.c +22 -0
index a0b4f1bca491..9fe3f6909513 100644
@@ -818,6 +818,22 @@ static void armv8pmu_clear_event_idx(struct pmu_hw_events *cpuc,
 		clear_bit(idx - 1, cpuc->used_mask);
 }
 
+static int armv8pmu_access_event_idx(struct perf_event *event)
+{
+	if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
+		return 0;
+
+	/*
+	 * We remap the cycle counter index to 32 to
+	 * match the offset applied to the rest of
+	 * the counter indeces.
+	 */
+	if (event->hw.idx == ARMV8_IDX_CYCLE_COUNTER)
+		return 32;
+
+	return event->hw.idx;
+}
+
 /*
  * Add an event filter to a given event.
  */
@@ -911,6 +927,9 @@ static int __armv8_pmuv3_map_event(struct perf_event *event,
 	if (armv8pmu_event_is_64bit(event))
 		event->hw.flags |= ARMPMU_EVT_64BIT;
 
+	if (!cpus_have_const_cap(ARM64_HAS_HETEROGENEOUS_PMU))
+		event->hw.flags |= ARMPMU_EL0_RD_CNTR;
+
 	/* Only expose micro/arch events supported by this PMU */
 	if ((hw_event_id > 0) && (hw_event_id < ARMV8_PMUV3_MAX_COMMON_EVENTS)
 	    && test_bit(hw_event_id, armpmu->pmceid_bitmap)) {
@@ -1031,6 +1050,8 @@ static int armv8_pmu_init(struct arm_pmu *cpu_pmu)
 	cpu_pmu->set_event_filter	= armv8pmu_set_event_filter;
 	cpu_pmu->filter_match		= armv8pmu_filter_match;
 
+	cpu_pmu->pmu.event_idx		= armv8pmu_access_event_idx;
+
 	return 0;
 }
 
@@ -1209,6 +1230,7 @@ void arch_perf_update_userpage(struct perf_event *event,
 	 */
 	freq = arch_timer_get_rate();
 	userpg->cap_user_time = 1;
+	userpg->cap_user_rdpmc = !!(event->hw.flags & ARMPMU_EL0_RD_CNTR);
 
 	clocks_calc_mult_shift(&userpg->time_mult, &shift, freq,
 			NSEC_PER_SEC, 0);
include/linux/perf/arm_pmu.h +2 -0
index 71f525a35ac2..1106a9ac00fd 100644
@@ -26,6 +26,8 @@
  */
 /* Event uses a 64bit counter */
 #define ARMPMU_EVT_64BIT		1
+/* Allow access to hardware counter from userspace */
+#define ARMPMU_EL0_RD_CNTR		2
 
 #define HW_OP_UNSUPPORTED		0xFFFF
 #define C(_x)				PERF_COUNT_HW_CACHE_##_x
Signature · 1 line
2.17.1
maintainer
#8 lore raw

Re: [PATCH v3 3/5] arm64: pmu: Add function implementation to update event index in userpage.

Quoted 28 lines · Raphael Gault

In order to be able to access the counter directly for userspace, we need to provide the index of the counter using the userpage. We thus need to override the event_idx function to retrieve and convert the perf_event index to armv8 hardware index.

Since the arm_pmu driver can be used by any implementation, even if not armv8, two components play a role into making sure the behaviour is correct and consistent with the PMU capabilities:

* the ARMPMU_EL0_RD_CNTR flag which denotes the capability to access counter from userspace.
* the event_idx call back, which is implemented and initialized by the PMU implementation: if no callback is provided, the default behaviour applies, returning 0 as index value.

---
 arch/arm64/kernel/perf_event.c | 22 ++++++++++++++++++++++
 include/linux/perf/arm_pmu.h   |  2 ++
 2 files changed, 24 insertions(+)
arch/arm64/kernel/perf_event.c +0 -0
index a0b4f1bca491..9fe3f6909513 100644
@@ -818,6 +818,22 @@ static void armv8pmu_clear_event_idx(struct pmu_hw_events *cpuc,
 		clear_bit(idx - 1, cpuc->used_mask);
 }

arch/arm64/kernel/perf_event.c static void armv8pmu_clear_event_idx(struct pmu_hw_events *cpuc, · line 829

Quoted 10 lines
arch/arm64/kernel/perf_event.c +9 -0
 
+static int armv8pmu_access_event_idx(struct perf_event *event)
+{
+	if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
+		return 0;
+
+	/*
+	 * We remap the cycle counter index to 32 to
+	 * match the offset applied to the rest of
+	 * the counter indeces.

Typo: s/indeces/indices/

Quoted 30 lines
arch/arm64/kernel/perf_event.c +12 -0
+	 */
+	if (event->hw.idx == ARMV8_IDX_CYCLE_COUNTER)
+		return 32;
+
+	return event->hw.idx;
+}
+
 /*
  * Add an event filter to a given event.
  */
@@ -911,6 +927,9 @@ static int __armv8_pmuv3_map_event(struct perf_event *event,
 	if (armv8pmu_event_is_64bit(event))
 		event->hw.flags |= ARMPMU_EVT_64BIT;
 
+	if (!cpus_have_const_cap(ARM64_HAS_HETEROGENEOUS_PMU))
+		event->hw.flags |= ARMPMU_EL0_RD_CNTR;
+
 	/* Only expose micro/arch events supported by this PMU */
 	if ((hw_event_id > 0) && (hw_event_id < ARMV8_PMUV3_MAX_COMMON_EVENTS)
 	    && test_bit(hw_event_id, armpmu->pmceid_bitmap)) {
@@ -1031,6 +1050,8 @@ static int armv8_pmu_init(struct arm_pmu *cpu_pmu)
 	cpu_pmu->set_event_filter	= armv8pmu_set_event_filter;
 	cpu_pmu->filter_match		= armv8pmu_filter_match;
 
+	cpu_pmu->pmu.event_idx		= armv8pmu_access_event_idx;
+
 	return 0;
 }
 

arch/arm64/kernel/perf_event.c void arch_perf_update_userpage(struct perf_event *event, · line 1233

Quoted 4 lines
arch/arm64/kernel/perf_event.c +1 -0
@@ -1209,6 +1230,7 @@ void arch_perf_update_userpage(struct perf_event *event,
 	 */
 	freq = arch_timer_get_rate();
 	userpg->cap_user_time = 1;
+	userpg->cap_user_rdpmc = !!(event->hw.flags & ARMPMU_EL0_RD_CNTR);

For bisectability, we should only expose this to userspace once we have the code to enable/disable it, so the code exposing the index and setting up the user page cap needs to be added after the context switch code.

Thanks,
Mark.

Quoted 19 lines
arch/arm64/kernel/perf_event.c +0 -0
 	clocks_calc_mult_shift(&userpg->time_mult, &shift, freq,
 			NSEC_PER_SEC, 0);
include/linux/perf/arm_pmu.h +2 -0
index 71f525a35ac2..1106a9ac00fd 100644
@@ -26,6 +26,8 @@
  */
 /* Event uses a 64bit counter */
 #define ARMPMU_EVT_64BIT		1
+/* Allow access to hardware counter from userspace */
+#define ARMPMU_EL0_RD_CNTR		2
 
 #define HW_OP_UNSUPPORTED		0xFFFF
 #define C(_x)				PERF_COUNT_HW_CACHE_##_x
Signature · 1 line
2.17.1
#9 lore raw

PATCH v3 4/5 arm64: perf: Enable pmu counter direct access for perf event on armv8

In reply to Raphael Gault

Keep track of event opened with direct access to the hardware counters and modify permissions while they are open.

The strategy used here is the same which x86 uses: everytime an event is mapped, the permissions are set if required. The atomic field added in the mm_context helps keep track of the different event opened and de-activate the permissions when all are unmapped.
We also need to update the permissions in the context switch code so that tasks keep the right permissions.

---
 arch/arm64/include/asm/mmu.h         |  6 +++++
 arch/arm64/include/asm/mmu_context.h |  2 ++
 arch/arm64/include/asm/perf_event.h  | 14 ++++++++++
 drivers/perf/arm_pmu.c               | 38 ++++++++++++++++++++++++++++
 4 files changed, 60 insertions(+)
arch/arm64/include/asm/mmu.h +6 -0
index fd6161336653..88ed4466bd06 100644
@@ -18,6 +18,12 @@
 
 typedef struct {
 	atomic64_t	id;
+
+	/*
+	 * non-zero if userspace have access to hardware
+	 * counters directly.
+	 */
+	atomic_t	pmu_direct_access;
 	void		*vdso;
 	unsigned long	flags;
 } mm_context_t;
arch/arm64/include/asm/mmu_context.h +2 -0
index 7ed0adb187a8..6e66ff940494 100644
@@ -21,6 +21,7 @@
 #include <asm-generic/mm_hooks.h>
 #include <asm/cputype.h>
 #include <asm/pgtable.h>
+#include <asm/perf_event.h>
 #include <asm/sysreg.h>
 #include <asm/tlbflush.h>
 
@@ -224,6 +225,7 @@ static inline void __switch_mm(struct mm_struct *next)
 	}
 
 	check_and_switch_context(next, cpu);
+	perf_switch_user_access(next);
 }
 
 static inline void
arch/arm64/include/asm/perf_event.h +14 -0
index 2bdbc79bbd01..ba58fa726631 100644
@@ -8,6 +8,7 @@
 
 #include <asm/stack_pointer.h>
 #include <asm/ptrace.h>
+#include <linux/mm_types.h>
 
 #define	ARMV8_PMU_MAX_COUNTERS	32
 #define	ARMV8_PMU_COUNTER_MASK	(ARMV8_PMU_MAX_COUNTERS - 1)
@@ -223,4 +224,17 @@ extern unsigned long perf_misc_flags(struct pt_regs *regs);
 	(regs)->pstate = PSR_MODE_EL1h;	\
 }
 
+static inline void perf_switch_user_access(struct mm_struct *mm)
+{
+	if (!IS_ENABLED(CONFIG_PERF_EVENTS))
+		return;
+
+	if (atomic_read(&mm->context.pmu_direct_access)) {
+		write_sysreg(ARMV8_PMU_USERENR_ER|ARMV8_PMU_USERENR_CR,
+			     pmuserenr_el0);
+	} else {
+		write_sysreg(0, pmuserenr_el0);
+	}
+}
+
 #endif
drivers/perf/arm_pmu.c +38 -0
index df352b334ea7..3a48cc9f17af 100644
@@ -25,6 +25,7 @@
 #include <linux/irqdesc.h>
 
 #include <asm/irq_regs.h>
+#include <asm/mmu_context.h>
 
 static DEFINE_PER_CPU(struct arm_pmu *, cpu_armpmu);
 static DEFINE_PER_CPU(int, cpu_irq);
@@ -778,6 +779,41 @@ static void cpu_pmu_destroy(struct arm_pmu *cpu_pmu)
 					    &cpu_pmu->node);
 }
 
+static void refresh_pmuserenr(void *mm)
+{
+	perf_switch_user_access(mm);
+}
+
+static void armpmu_event_mapped(struct perf_event *event, struct mm_struct *mm)
+{
+	if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
+		return;
+
+	/*
+	 * This function relies on not being called concurrently in two
+	 * tasks in the same mm.  Otherwise one task could observe
+	 * pmu_direct_access > 1 and return all the way back to
+	 * userspace with user access disabled while another task is still
+	 * doing on_each_cpu_mask() to enable user access.
+	 *
+	 * For now, this can't happen because all callers hold mmap_sem
+	 * for write.  If this changes, we'll need a different solution.
+	 */
+	lockdep_assert_held_write(&mm->mmap_sem);
+
+	if (atomic_inc_return(&mm->context.pmu_direct_access) == 1)
+		on_each_cpu(refresh_pmuserenr, mm, 1);
+}
+
+static void armpmu_event_unmapped(struct perf_event *event, struct mm_struct *mm)
+{
+	if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
+		return;
+
+	if (atomic_dec_and_test(&mm->context.pmu_direct_access))
+		on_each_cpu_mask(mm_cpumask(mm), refresh_pmuserenr, NULL, 1);
+}
+
 static struct arm_pmu *__armpmu_alloc(gfp_t flags)
 {
 	struct arm_pmu *pmu;
@@ -799,6 +835,8 @@ static struct arm_pmu *__armpmu_alloc(gfp_t flags)
 		.pmu_enable	= armpmu_enable,
 		.pmu_disable	= armpmu_disable,
 		.event_init	= armpmu_event_init,
+		.event_mapped	= armpmu_event_mapped,
+		.event_unmapped	= armpmu_event_unmapped,
 		.add		= armpmu_add,
 		.del		= armpmu_del,
 		.start		= armpmu_start,
Signature · 1 line
2.17.1
#10 lore raw

Re: [PATCH v3 4/5] arm64: perf: Enable pmu counter direct access for perf event on armv8

Thank you for the patch! Yet something to improve:

Hi Raphael,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on linus/master]
[cannot apply to v5.3-rc4 next-20190816]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url: https://github.com/0day-ci/linux/commits/Raphael-Gault/perf-arm64-Add-test-to-check-userspace-access-to-hardware-counters/20190818-182238
config: arm-omap2plus_defconfig (attached as .config)
compiler: arm-linux-gnueabi-gcc (GCC) 7.4.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.4.0 make.cross ARCH=arm

If you fix the issue, kindly add following tag

All errors (new ones prefixed by >>):

drivers/perf/arm_pmu.c: In function 'refresh_pmuserenr':

>> drivers/perf/arm_pmu.c:784:2: error: implicit declaration of function 'perf_switch_user_access'; did you mean 'perf_fetch_caller_regs'? [-Werror=implicit-function-declaration]
     perf_switch_user_access(mm);
     ^~~~~~~~~~~~~~~~~~~~~~~
     perf_fetch_caller_regs
   drivers/perf/arm_pmu.c: In function 'armpmu_event_mapped':
>> drivers/perf/arm_pmu.c:804:36: error: 'mm_context_t {aka struct <anonymous>}' has no member named 'pmu_direct_access'
     if (atomic_inc_return(&mm->context.pmu_direct_access) == 1)
                                       ^
   drivers/perf/arm_pmu.c: In function 'armpmu_event_unmapped':
   drivers/perf/arm_pmu.c:813:38: error: 'mm_context_t {aka struct <anonymous>}' has no member named 'pmu_direct_access'
     if (atomic_dec_and_test(&mm->context.pmu_direct_access))
                                         ^

cc1: some warnings being treated as errors

vim +784 drivers/perf/arm_pmu.c

781
782 static void refresh_pmuserenr(void *mm)
783 {

 > 784		perf_switch_user_access(mm);
   785	}
   786	
   787	static void armpmu_event_mapped(struct perf_event *event, struct mm_struct *mm)
   788	{
   789		if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
   790			return;
   791	
   792		/*
   793		 * This function relies on not being called concurrently in two
   794		 * tasks in the same mm.  Otherwise one task could observe
   795		 * pmu_direct_access > 1 and return all the way back to
   796		 * userspace with user access disabled while another task is still
   797		 * doing on_each_cpu_mask() to enable user access.
   798		 *
   799		 * For now, this can't happen because all callers hold mmap_sem
   800		 * for write.  If this changes, we'll need a different solution.
   801		 */
   802		lockdep_assert_held_write(&mm->mmap_sem);
   803	
 > 804		if (atomic_inc_return(&mm->context.pmu_direct_access) == 1)

805 on_each_cpu(refresh_pmuserenr, mm, 1);
806 }
807

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#11 lore raw

Re: [PATCH v3 4/5] arm64: perf: Enable pmu counter direct access for perf event on armv8

Hi,

Quoted 7 lines · kbuild test robot

Hi Raphael,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on linus/master]
[cannot apply to v5.3-rc4 next-20190816]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

This patchset was based on linux-next/master and note linus

Thanks,

Signature · 1 line
Raphael Gault
#12 lore raw

Re: [kbuild-all] [PATCH v3 4/5] arm64: perf: Enable pmu counter direct access for perf event on armv8

Quoted 5 lines · Raphael Gault

Hi,

Quoted 2 lines · kbuild test robot

Hi Raphael,

Quoted 7 lines
Quoted 5 lines

Thank you for the patch! Yet something to improve:

[auto build test ERROR on linus/master]
[cannot apply to v5.3-rc4 next-20190816]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

This patchset was based on linux-next/master and note linus

thanks for the input, we will look for how to find better base to test.

Quoted 9 lines

Thanks,

Signature · 5 lines
Raphael Gault
_______________________________________________
kbuild-all mailing list
[email protected]
https://lists.01.org/mailman/listinfo/kbuild-all
#13 lore raw

PATCH v3 5/5 Documentation: arm64: Document PMU counters access from userspace

In reply to Raphael Gault

Add a documentation file to describe the access to the pmu hardware counters from userspace

---
 .../arm64/pmu_counter_user_access.txt         | 42 +++++++++++++++++++
 1 file changed, 42 insertions(+)
 create mode 100644 Documentation/arm64/pmu_counter_user_access.txt
Documentation/arm64/pmu_counter_user_access.txt new file +42 -0
new file mode 100644
index 000000000000..6788b1107381
@@ -0,0 +1,42 @@
+Access to PMU hardware counter from userspace
+=============================================
+
+Overview
+--------
+The perf user-space tool relies on the PMU to monitor events. It offers an
+abstraction layer over the hardware counters since the underlying
+implementation is cpu-dependent.
+Arm64 allows userspace tools to have access to the registers storing the
+hardware counters' values directly.
+
+This targets specifically self-monitoring tasks in order to reduce the overhead
+by directly accessing the registers without having to go through the kernel.
+
+How-to
+------
+The focus is set on the armv8 pmuv3 which makes sure that the access to the pmu
+registers is enable and that the userspace have access to the relevent
+information in order to use them.
+
+In order to have access to the hardware counter it is necessary to open the event
+using the perf tool interface: the sys_perf_event_open syscall returns a fd which
+can subsequently be used with the mmap syscall in order to retrieve a page of memory
+containing information about the event.
+The PMU driver uses this page to expose to the user the hardware counter's
+index. Using this index enables the user to access the PMU registers using the
+`mrs` instruction.
+
+Have a look `at tools/perf/arch/arm64/tests/user-events.c` for an example. It can be
+run using the perf tool to check that the access to the registers works
+correctly from userspace:
+
+./perf test -v
+
+About chained events
+--------------------
+When the user requests for an event to be counted on 64 bits, two hardware
+counters are used and need to be combined to retrieve the correct value:
+
+val = read_counter(idx);
+if ((event.attr.config1 & 0x1))
+	val = (val << 32) | read_counter(idx - 1);
Signature · 1 line
2.17.1