arm64: Enable access to pmu registers by user-space
Subsystems (5)
v2 cover arm64: Enable access to pmu registers by user-space
The perf user-space tool relies on the PMU to monitor events. It offers an abstraction layer over the hardware counters since the underlying implementation is cpu-dependent. We want to allow userspace tools to have access to the registers storing the hardware counters' values directly. This targets specifically self-monitoring tasks in order to reduce the overhead by directly accessing the registers without having to go through the kernel.
In order to do this we need to setup the pmu so that it exposes its registers to userspace access.
The first patch add a test to the perf tool so that we can test that the access to the registers works correctly from userspace.
The second patch add a capability in the arm64 cpufeatures framework in order to detect when we are running on a heterogeneous system.
The third patch focuses on the armv8 pmuv3 PMU support and makes sure that the access to the pmu registers is enable and that the userspace have access to the relevent information in order to use them.
The fourth patch put in place callbacks to enable access to the hardware counters from userspace when a compatible event is opened using the perf API.
The fifth patch adds a short documentation about PMU counters direct access from userspace.
**Changes since v1**
* Rebased on linux-next/master
* Do not include RSEQs materials (test and utilities) since we want to enable direct access to counters only on homogeneous systems.
* Do not include the hook defitinions for the same reason as above.
* Add a cpu feature/capability to detect heterogeneous systems.
Raphael Gault (5):
perf: arm64: Add test to check userspace access to hardware counters.
arm64: cpufeature: Add feature to detect heterogeneous systems
arm64: pmu: Add function implementation to update event index in
userpage.
arm64: perf: Enable pmu counter direct access for perf event on armv8
Documentation: arm64: Document PMU counters access from userspace
.../arm64/pmu_counter_user_access.txt | 42 +++ arch/arm64/include/asm/cpucaps.h | 3 +- arch/arm64/include/asm/mmu.h | 6 + arch/arm64/include/asm/mmu_context.h | 2 + arch/arm64/include/asm/perf_event.h | 14 + arch/arm64/kernel/cpufeature.c | 20 ++ arch/arm64/kernel/perf_event.c | 23 ++ drivers/perf/arm_pmu.c | 38 +++ include/linux/perf/arm_pmu.h | 2 + tools/perf/arch/arm64/include/arch-tests.h | 6 + tools/perf/arch/arm64/tests/Build | 1 + tools/perf/arch/arm64/tests/arch-tests.c | 4 + tools/perf/arch/arm64/tests/user-events.c | 255 ++++++++++++++++++ 13 files changed, 415 insertions(+), 1 deletion(-) create mode 100644 Documentation/arm64/pmu_counter_user_access.txt create mode 100644 tools/perf/arch/arm64/tests/user-events.c
Signature · 1 line
2.17.1
PATCH v2 1/5 perf: arm64: Add test to check userspace access to hardware counters.
This test relies on the fact that the PMU registers are accessible from userspace. It then uses the perf_event_mmap_page to retrieve the counter index and access the underlying register.
This test uses sched_setaffinity(2) in order to run on all CPU and thus check the behaviour of the PMU of all cpus in a big.LITTLE environment.
--- tools/perf/arch/arm64/include/arch-tests.h | 6 + tools/perf/arch/arm64/tests/Build | 1 + tools/perf/arch/arm64/tests/arch-tests.c | 4 + tools/perf/arch/arm64/tests/user-events.c | 255 +++++++++++++++++++++ 4 files changed, 266 insertions(+) create mode 100644 tools/perf/arch/arm64/tests/user-events.c
tools/perf/arch/arm64/include/arch-tests.h +6 -0
@@ -2,11 +2,17 @@#ifndef ARCH_TESTS_H #define ARCH_TESTS_H +#define __maybe_unused __attribute__((unused))#ifdef HAVE_DWARF_UNWIND_SUPPORT struct thread ;struct perf_sample ; +int test__arch_unwind_sample (struct perf_sample *sample, +struct thread *thread);#endif extern struct test arch_tests[]; +int test__rd_pmevcntr (struct test *test__maybe_unused , +int subtest__maybe_unused ); +#endif
tools/perf/arch/arm64/tests/Build +1 -0
@@ -1,4 +1,5 @@ perf-y += regs_load.o perf-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o +perf-y += user-events.o perf-y += arch-tests.o
tools/perf/arch/arm64/tests/arch-tests.c +4 -0
@@ -10,6 +10,10 @@ struct test arch_tests[] = { .func = test__dwarf_unwind, },#endif + { + .desc = "User counter access", + .func = test__rd_pmevcntr, + }, { .func = NULL, },
tools/perf/arch/arm64/tests/user-events.c new file +255 -0
255 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v2 1/5] perf: arm64: Add test to check userspace access to hardware counters.
Em Fri, Jul 05, 2019 at 09:55:37AM +0100, Raphael Gault escreveu:
Quoted 21 lines
This test relies on the fact that the PMU registers are accessible from userspace. It then uses the
perf_event_mmap_pageto retrieve the counter index and access the underlying register.This test uses
sched_setaffinity(2)in order to run on all CPU and thus check the behaviour of the PMU of all cpus in a big.LITTLE environment.Signed-off-by Raphael Gault <[email protected]>--- tools/perf/arch/arm64/include/arch-tests.h | 6 + tools/perf/arch/arm64/tests/Build | 1 + tools/perf/arch/arm64/tests/arch-tests.c | 4 + tools/perf/arch/arm64/tests/user-events.c | 255 +++++++++++++++++++++ 4 files changed, 266 insertions(+) create mode 100644 tools/perf/arch/arm64/tests/user-events.ctools/perf/arch/arm64/include/arch-tests.h +0
-0
tools/perf/arch/arm64/include/arch-tests.h line 5
Quoted 4 lines
tools/perf/arch/arm64/include/arch-tests.h +1
-0@@ -2,11 +2,17 @@#ifndef ARCH_TESTS_H #define ARCH_TESTS_H +#define __maybe_unused __attribute__((unused))
What is wrong with using:
#include <linux/compiler.h>
?
[acme@quaco perf]$ find tools/perf/ -name "*.[ch]" | xargs grep __maybe_unused | wc -l 1115
[acme@quaco perf]$ grep __maybe_unused tools/include/linux/compiler.h #ifndef __maybe_unused
# define __maybe_unused __attribute__((unused)) [acme@quaco perf]$
Also please don't break strings in multiple lines just to comply with the 80 column limit. That is ok when you have multiple lines ending with a newline, but otherwise just makes it look ugly.
- Arnaldo
Re: [PATCH v2 1/5] perf: arm64: Add test to check userspace access to hardware counters.
Hi Arnaldo,
Quoted 38 lines · Arnaldo Carvalho de Melo
Em Fri, Jul 05, 2019 at 09:55:37AM +0100, Raphael Gault escreveu:
Quoted 21 lines
This test relies on the fact that the PMU registers are accessible from userspace. It then uses the
perf_event_mmap_pageto retrieve the counter index and access the underlying register.This test uses
sched_setaffinity(2)in order to run on all CPU and thus check the behaviour of the PMU of all cpus in a big.LITTLE environment.Signed-off-by Raphael Gault <[email protected]>--- tools/perf/arch/arm64/include/arch-tests.h | 6 + tools/perf/arch/arm64/tests/Build | 1 + tools/perf/arch/arm64/tests/arch-tests.c | 4 + tools/perf/arch/arm64/tests/user-events.c | 255 +++++++++++++++++++++ 4 files changed, 266 insertions(+)create mode 100644
tools/perf/arch/arm64/tests/user-events.ctools/perf/arch/arm64/include/arch-tests.h +0
-0tools/perf/arch/arm64/include/arch-tests.h line 5
Quoted 4 lines
tools/perf/arch/arm64/include/arch-tests.h +1
-0@@ -2,11 +2,17 @@#ifndef ARCH_TESTS_H #define ARCH_TESTS_H +#define __maybe_unused __attribute__((unused))What is wrong with using:
#include <
linux/compiler.h>?
[acme@quaco perf]$ find
tools/perf/-name "*.[ch]" | xargs grep__maybe_unused| wc -l 1115
[acme@quaco perf]$ grep__maybe_unusedtools/include/linux/compiler.h#ifndef__maybe_unused
Quoted 7 lines
# define __maybe_unused __attribute__((unused)) [acme@quaco perf]$Also please don't break strings in multiple lines just to comply with the 80 column limit. That is ok when you have multiple lines ending with a newline, but otherwise just makes it look ugly.
You're right, I shall correct those points.
Thanks,
Signature · 1 line
Raphael Gault
PATCH v2 2/5 arm64: cpufeature: Add feature to detect heterogeneous systems
In reply to Raphael GaultThis feature is required in order to enable PMU counters direct access from userspace only when the system is homogeneous. This feature checks the model of each CPU brought online and compares it to the boot CPU. If it differs then it is heterogeneous.
--- arch/arm64/include/asm/cpucaps.h | 3 ++- arch/arm64/kernel/cpufeature.c | 20 ++++++++++++++++++++ arch/arm64/kernel/perf_event.c | 1 + 3 files changed, 23 insertions(+), 1 deletion(-)
arch/arm64/include/asm/cpucaps.h +2 -1
@@ -52,7 +52,8 @@#define ARM64_HAS_IRQ_PRIO_MASKING 42#define ARM64_HAS_DCPODP 43#define ARM64_WORKAROUND_1463225 44 +#define ARM64_HAS_HETEROGENEOUS_PMU 45 -#define ARM64_NCAPS 45 +#define ARM64_NCAPS 46#endif /* __ASM_CPUCAPS_H */
arch/arm64/kernel/cpufeature.c +20 -0
@@ -1248,6 +1248,15 @@ static bool can_use_gic_priorities(const struct arm64_cpu_capabilities *entry, }#endif +static bool has_heterogeneous_pmu (const struct arm64_cpu_capabilities *entry, +int scope) +{ +u32 model =read_cpuid_id () &MIDR_CPU_MODEL_MASK ; +struct cpuinfo_arm64 *boot = &per_cpu (cpu_data,0 ); + +return (boot->reg_midr &MIDR_CPU_MODEL_MASK ) != model; +} +static const struct arm64_cpu_capabilities arm64_features[] = { { .desc ="GIC system register CPU interface" ,
@@ -1548,6 +1557,16 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .min_field_value = 1, },#endif + { +/* +* Detect whether the system is heterogeneous or +* homogeneous +*/ + .desc = "Detect whether we have heterogeneous CPUs", + .capability = ARM64_HAS_HETEROGENEOUS_PMU, + .type = ARM64_CPUCAP_SCOPE_LOCAL_CPU | ARM64_CPUCAP_OPTIONAL_FOR_LATE_CPU, + .matches = has_heterogeneous_pmu, + }, {}, };
@@ -1715,6 +1734,7 @@ static void __init setup_elf_hwcaps(const struct arm64_cpu_capabilities *hwcaps)cap_set_elf_hwcap (hwcaps); } +static void update_cpu_capabilities (u16 scope_mask) {int i;
arch/arm64/kernel/perf_event.c +1 -0
@@ -19,6 +19,7 @@#include <linux/of.h> #include <linux/perf/arm_pmu.h> #include <linux/platform_device.h> +#include <linux/smp.h> /* ARMv8 Cortex-A53 specific event types. */ #define ARMV8_A53_PERFCTR_PREF_LINEFILL 0xC2
Signature · 1 line
2.17.1
PATCH v2 3/5 arm64: pmu: Add function implementation to update event index in userpage.
In reply to Raphael GaultIn order to be able to access the counter directly for userspace, we need to provide the index of the counter using the userpage. We thus need to override the event_idx function to retrieve and convert the perf_event index to armv8 hardware index.
Since the arm_pmu driver can be used by any implementation, even if not armv8, two components play a role into making sure the behaviour is correct and consistent with the PMU capabilities:
* the ARMPMU_EL0_RD_CNTR flag which denotes the capability to access counter from userspace.
* the event_idx call back, which is implemented and initialized by the PMU implementation: if no callback is provided, the default behaviour applies, returning 0 as index value.
--- arch/arm64/kernel/perf_event.c | 22 ++++++++++++++++++++++ include/linux/perf/arm_pmu.h | 2 ++ 2 files changed, 24 insertions(+)
arch/arm64/kernel/perf_event.c +22 -0
@@ -819,6 +819,22 @@ static void armv8pmu_clear_event_idx(struct pmu_hw_events *cpuc,clear_bit (idx -1 , cpuc->used_mask); } +static int armv8pmu_access_event_idx (struct perf_event *event) +{ +if (!(event->hw.flags &ARMPMU_EL0_RD_CNTR )) +return 0 ; + +/* +* We remap the cycle counter index to 32 to +* match the offset applied to the rest of +* the counter indeces. +*/ +if (event->hw.idx ==ARMV8_IDX_CYCLE_COUNTER ) +return 32 ; + +return event->hw.idx; +} +/* * Add an event filter to a given event. */
@@ -912,6 +928,9 @@ static int __armv8_pmuv3_map_event(struct perf_event *event,if (armv8pmu_event_is_64bit (event)) event->hw.flags |=ARMPMU_EVT_64BIT ; +if (!cpus_have_const_cap (ARM64_HAS_HETEROGENEOUS_PMU )) + event->hw.flags |=ARMPMU_EL0_RD_CNTR ; +/* Only expose micro/arch events supported by this PMU */ if ((hw_event_id >0 ) && (hw_event_id <ARMV8_PMUV3_MAX_COMMON_EVENTS ) &&test_bit (hw_event_id, armpmu->pmceid_bitmap)) {
@@ -1032,6 +1051,8 @@ static int armv8_pmu_init(struct arm_pmu *cpu_pmu) cpu_pmu->set_event_filter = armv8pmu_set_event_filter; cpu_pmu->filter_match = armv8pmu_filter_match; + cpu_pmu->pmu.event_idx = armv8pmu_access_event_idx; +return 0 ; }
@@ -1210,6 +1231,7 @@ void arch_perf_update_userpage(struct perf_event *event,*/ freq =arch_timer_get_rate (); userpg->cap_user_time =1 ; + userpg->cap_user_rdpmc = !!(event->hw.flags &ARMPMU_EL0_RD_CNTR );clocks_calc_mult_shift (&userpg->time_mult, &shift, freq,NSEC_PER_SEC ,0 );
include/linux/perf/arm_pmu.h +2 -0
@@ -26,6 +26,8 @@*/ /* Event uses a 64bit counter */ #define ARMPMU_EVT_64BIT 1 +/* Allow access to hardware counter from userspace */ +#define ARMPMU_EL0_RD_CNTR 2#define HW_OP_UNSUPPORTED 0xFFFF#define C (_x) PERF_COUNT_HW_CACHE_##_x
Signature · 1 line
2.17.1
PATCH v2 4/5 arm64: perf: Enable pmu counter direct access for perf event on armv8
In reply to Raphael GaultKeep track of event opened with direct access to the hardware counters and modify permissions while they are open.
The strategy used here is the same which x86 uses: everytime an event is mapped, the permissions are set if required. The atomic field added in the mm_context helps keep track of the different event opened and de-activate the permissions when all are unmapped.
We also need to update the permissions in the context switch code so that tasks keep the right permissions.
--- arch/arm64/include/asm/mmu.h | 6 +++++ arch/arm64/include/asm/mmu_context.h | 2 ++ arch/arm64/include/asm/perf_event.h | 14 ++++++++++ drivers/perf/arm_pmu.c | 38 ++++++++++++++++++++++++++++ 4 files changed, 60 insertions(+)
arch/arm64/include/asm/mmu.h +6 -0
@@ -18,6 +18,12 @@typedef struct {atomic64_t id; + +/* +* non-zero if userspace have access to hardware +* counters directly. +*/ +atomic_t pmu_direct_access;void *vdso;unsigned long flags; }mm_context_t ;
arch/arm64/include/asm/mmu_context.h +2 -0
@@ -21,6 +21,7 @@#include <asm-generic/mm_hooks.h> #include <asm/cputype.h> #include <asm/pgtable.h> +#include <asm/perf_event.h> #include <asm/sysreg.h> #include <asm/tlbflush.h>
@@ -224,6 +225,7 @@ static inline void __switch_mm(struct mm_struct *next) }check_and_switch_context (next, cpu); +perf_switch_user_access (next); } static inline void
arch/arm64/include/asm/perf_event.h +14 -0
@@ -8,6 +8,7 @@#include <asm/stack_pointer.h> #include <asm/ptrace.h> +#include <linux/mm_types.h> #define ARMV8_PMU_MAX_COUNTERS 32#define ARMV8_PMU_COUNTER_MASK (ARMV8_PMU_MAX_COUNTERS - 1)
@@ -223,4 +224,17 @@ extern unsigned long perf_misc_flags(struct pt_regs *regs); (regs)->pstate = PSR_MODE_EL1h; \ } +static inline void perf_switch_user_access (struct mm_struct *mm) +{ +if (!IS_ENABLED (CONFIG_PERF_EVENTS )) +return ; + +if (atomic_read (&mm->context.pmu_direct_access)) { +write_sysreg (ARMV8_PMU_USERENR_ER |ARMV8_PMU_USERENR_CR , + pmuserenr_el0); + }else { +write_sysreg (0 , pmuserenr_el0); + } +} +#endif
drivers/perf/arm_pmu.c +38 -0
@@ -25,6 +25,7 @@#include <linux/irqdesc.h> #include <asm/irq_regs.h> +#include <asm/mmu_context.h> static DEFINE_PER_CPU(struct arm_pmu *, cpu_armpmu); static DEFINE_PER_CPU(int, cpu_irq);
@@ -778,6 +779,41 @@ static void cpu_pmu_destroy(struct arm_pmu *cpu_pmu) &cpu_pmu->node); } +static void refresh_pmuserenr (void *mm) +{ +perf_switch_user_access (mm); +} + +static void armpmu_event_mapped (struct perf_event *event,struct mm_struct *mm) +{ +if (!(event->hw.flags &ARMPMU_EL0_RD_CNTR )) +return ; + +/* +* This function relies on not being called concurrently in two +* tasks in the same mm. Otherwise one task could observe +* pmu_direct_access > 1 and return all the way back to +* userspace with user access disabled while another task is still +* doing on_each_cpu_mask() to enable user access. +* +* For now, this can't happen because all callers hold mmap_sem +* for write. If this changes, we'll need a different solution. +*/ +lockdep_assert_held_write (&mm->mmap_sem); + +if (atomic_inc_return (&mm->context.pmu_direct_access) ==1 ) +on_each_cpu (refresh_pmuserenr, mm,1 ); +} + +static void armpmu_event_unmapped (struct perf_event *event,struct mm_struct *mm) +{ +if (!(event->hw.flags &ARMPMU_EL0_RD_CNTR )) +return ; + +if (atomic_dec_and_test (&mm->context.pmu_direct_access)) +on_each_cpu_mask (mm_cpumask (mm), refresh_pmuserenr,NULL ,1 ); +} +static struct arm_pmu *__armpmu_alloc (gfp_t flags) {struct arm_pmu *pmu;
@@ -799,6 +835,8 @@ static struct arm_pmu *__armpmu_alloc(gfp_t flags) .pmu_enable = armpmu_enable, .pmu_disable = armpmu_disable, .event_init = armpmu_event_init, + .event_mapped = armpmu_event_mapped, + .event_unmapped = armpmu_event_unmapped, .add = armpmu_add, .del = armpmu_del, .start = armpmu_start,
Signature · 1 line
2.17.1
PATCH v2 5/5 Documentation: arm64: Document PMU counters access from userspace
In reply to Raphael GaultAdd a documentation file to describe the access to the pmu hardware counters from userspace
--- .../arm64/pmu_counter_user_access.txt | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 Documentation/arm64/pmu_counter_user_access.txt
Documentation/arm64/pmu_counter_user_access.txt new file +42 -0
@@ -0,0 +1,42 @@ +Access to PMU hardware counter from userspace +============================================= + +Overview +-------- +The perf user-space tool relies on the PMU to monitor events. It offers an +abstraction layer over the hardware counters since the underlying +implementation is cpu-dependent. +Arm64 allows userspace tools to have access to the registers storing the +hardware counters' values directly. + +This targets specifically self-monitoring tasks in order to reduce the overhead +by directly accessing the registers without having to go through the kernel. + +How-to +------ +The focus is set on the armv8 pmuv3 which makes sure that the access to the pmu +registers is enable and that the userspace have access to the relevent +information in order to use them. + +In order to have access to the hardware counter it is necessary to open the event +using the perf tool interface: the sys_perf_event_open syscall returns a fd which +can subsequently be used with the mmap syscall in order to retrieve a page of memory +containing information about the event. +The PMU driver uses this page to expose to the user the hardware counter's +index. Using this index enables the user to access the PMU registers using the +`mrs` instruction. + +Have a look `at tools/perf/arch/arm64/tests/user-events.c` for an example. It can be +run using the perf tool to check that the access to the registers works +correctly from userspace: + +./perf test -v + +About chained events +-------------------- +When the user requests for an event to be counted on 64 bits, two hardware +counters are used and need to be combined to retrieve the correct value: + +val = read_counter(idx); +if ((event.attr.config1 & 0x1)) + val = (val << 32) | read_counter(idx - 1);
Signature · 1 line
2.17.1
Hi,
Any further comments on this patchset ?
Cheers,
Quoted 60 lines · Raphael Gault
The perf user-space tool relies on the PMU to monitor events. It offers an abstraction layer over the hardware counters since the underlying implementation is cpu-dependent. We want to allow userspace tools to have access to the registers storing the hardware counters' values directly. This targets specifically self-monitoring tasks in order to reduce the overhead by directly accessing the registers without having to go through the kernel.
In order to do this we need to setup the pmu so that it exposes its registers to userspace access.The first patch add a test to the perf tool so that we can test that the access to the registers works correctly from userspace.
The second patch add a capability in the arm64 cpufeatures framework in order to detect when we are running on a heterogeneous system.
The third patch focuses on the armv8 pmuv3 PMU support and makes sure that the access to the pmu registers is enable and that the userspace have access to the relevent information in order to use them.
The fourth patch put in place callbacks to enable access to the hardware counters from userspace when a compatible event is opened using the perf API.
The fifth patch adds a short documentation about PMU counters direct access from userspace.
**Changes since v1**
* Rebased on linux-next/master
* Do not include RSEQs materials (test and utilities) since we want to enable direct access to counters only on homogeneous systems.
* Do not include the hook defitinions for the same reason as above.
* Add a cpu feature/capability to detect heterogeneous systems.Raphael Gault (5):
perf: arm64: Add test to check userspace access to hardware counters.
arm64: cpufeature: Add feature to detect heterogeneous systems
arm64: pmu: Add function implementation to update event index in
userpage.
arm64: perf: Enable pmu counter direct access for perf event on armv8
Documentation: arm64: Document PMU counters access from userspace.../arm64/pmu_counter_user_access.txt | 42 +++ arch/arm64/include/asm/cpucaps.h | 3 +- arch/arm64/include/asm/mmu.h | 6 + arch/arm64/include/asm/mmu_context.h | 2 + arch/arm64/include/asm/perf_event.h | 14 + arch/arm64/kernel/cpufeature.c | 20 ++ arch/arm64/kernel/perf_event.c | 23 ++ drivers/perf/arm_pmu.c | 38 +++ include/linux/perf/arm_pmu.h | 2 + tools/perf/arch/arm64/include/arch-tests.h | 6 + tools/perf/arch/arm64/tests/Build | 1 + tools/perf/arch/arm64/tests/arch-tests.c | 4 + tools/perf/arch/arm64/tests/user-events.c | 255 ++++++++++++++++++ 13 files changed, 415 insertions(+), 1 deletion(-)create mode 100644
Documentation/arm64/pmu_counter_user_access.txtcreate mode 100644tools/perf/arch/arm64/tests/user-events.c
Signature · 1 line
Raphael Gault