Skip to contentSkip to conversation

arm64: Enable access to pmu registers by user-space

RFC Superseded Raphael Gault 5 participants 17 messages May 28, 2019 – May 29, 2019

Subsystems (5)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover arm64: Enable access to pmu registers by user-space

The perf user-space tool relies on the PMU to monitor events. It offers an abstraction layer over the hardware counters since the underlying implementation is cpu-dependent. We want to allow userspace tools to have access to the registers storing the hardware counters' values directly. This targets specifically self-monitoring tasks in order to reduce the overhead by directly accessing the registers without having to go through the kernel.
In order to do this we need to setup the pmu so that it exposes its registers to userspace access.

The first patch enables the tests for arm64 architecture in the perf tool to be compiled systematically.

The second patch add a test to the perf tool so that we can test that the access to the registers works correctly from userspace.

The third patch adds another test similar to the first one but this time using rseq as mechanism to make sure of the data correctness.

The fourth patch focuses on the armv8 pmuv3 PMU support and makes sure that the access to the pmu registers is enable and that the userspace have access to the relevent information in order to use them.

The fifth patch adds a hook to handle faulting access to the pmu registers. This is necessary in order to have a coherent behaviour on big.LITTLE environment.

The sixth patch put in place callbacks to enable access to the hardware counters from userspace when a compatible event is opened using the perf API.

RFC: In my opinion there is no need to save pmselr_el0 when context switching like we do for pmuserenr_el0 since whether it's the seqlock mechanism or the restartable sequences, the user should notice right away that the value held in pmxevcntr_el0 is incorrect when the task has been rescheduled. However, I still wanted to discuss this point on the list to see if that's indeed not necessary to save it.

Changes since V1: Add a test using rseq

Raphael Gault (7):
perf: arm64: Compile tests unconditionally
perf: arm64: Add test to check userspace access to hardware counters.
perf: arm64: Use rseq to test userspace access to pmu counters
arm64: pmu: Add function implementation to update event index in
userpage.
arm64: pmu: Add hook to handle pmu-related undefined instructions
arm64: perf: Enable pmu counter direct access for perf event on armv8
Documentation: arm64: Document PMU counters access from userspace

 .../arm64/pmu_counter_user_access.txt         |  42 +++
 arch/arm64/include/asm/mmu.h                  |   6 +
 arch/arm64/include/asm/mmu_context.h          |   2 +
 arch/arm64/include/asm/perf_event.h           |  14 +
 arch/arm64/kernel/cpufeature.c                |   4 +-
 arch/arm64/kernel/perf_event.c                |  62 +++++
 drivers/perf/arm_pmu.c                        |  38 +++
 include/linux/perf/arm_pmu.h                  |   2 +
 tools/perf/arch/arm64/Build                   |   2 +-
 tools/perf/arch/arm64/include/arch-tests.h    |   9 +
 tools/perf/arch/arm64/include/rseq-arm64.h    | 220 +++++++++++++++
 tools/perf/arch/arm64/tests/Build             |   4 +-
 tools/perf/arch/arm64/tests/arch-tests.c      |  10 +
 tools/perf/arch/arm64/tests/rseq-pmu-events.c | 219 +++++++++++++++
 tools/perf/arch/arm64/tests/user-events.c     | 255 ++++++++++++++++++
 15 files changed, 885 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/arm64/pmu_counter_user_access.txt
 create mode 100644 tools/perf/arch/arm64/include/rseq-arm64.h
 create mode 100644 tools/perf/arch/arm64/tests/rseq-pmu-events.c
 create mode 100644 tools/perf/arch/arm64/tests/user-events.c
Signature · 1 line
2.17.1
#1 lore raw

RFC v2 1/7 perf: arm64: Compile tests unconditionally

In order to subsequently add more tests for the arm64 architecture we compile the tests target for arm64 systematically.

---
 tools/perf/arch/arm64/Build       | 2 +-
 tools/perf/arch/arm64/tests/Build | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
tools/perf/arch/arm64/Build +1 -1
index 36222e64bbf7..a7dd46a5b678 100644
@@ -1,2 +1,2 @@
 perf-y += util/
-perf-$(CONFIG_DWARF_UNWIND) += tests/
+perf-y += tests/
tools/perf/arch/arm64/tests/Build +1 -1
index 41707fea74b3..a61c06bdb757 100644
@@ -1,4 +1,4 @@
 perf-y += regs_load.o
-perf-y += dwarf-unwind.o
+perf-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
 
 perf-y += arch-tests.o
Signature · 1 line
2.17.1
#2 lore raw

Re: [RFC 1/7] perf: arm64: Compile tests unconditionally

Em Tue, May 28, 2019 at 04:03:14PM +0100, Raphael Gault escreveu:

Quoted 2 lines

In order to subsequently add more tests for the arm64 architecture we compile the tests target for arm64 systematically.

Humm, the subject doesn't match the description? I.e. it _was_ unconditionally built, now it is only built if CONFIG_DWARF_UNWIND is set to 'y'.

- Arnaldo

Quoted 26 lines
---
 tools/perf/arch/arm64/Build       | 2 +-
 tools/perf/arch/arm64/tests/Build | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
tools/perf/arch/arm64/Build +1 -1
index 36222e64bbf7..a7dd46a5b678 100644
@@ -1,2 +1,2 @@
 perf-y += util/
-perf-$(CONFIG_DWARF_UNWIND) += tests/
+perf-y += tests/
tools/perf/arch/arm64/tests/Build +1 -1
index 41707fea74b3..a61c06bdb757 100644
@@ -1,4 +1,4 @@
 perf-y += regs_load.o
-perf-y += dwarf-unwind.o
+perf-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
 
 perf-y += arch-tests.o
Signature · 1 line
2.17.1
Signature · 1 line
- Arnaldo
maintainer
#3 lore raw

Re: [RFC 1/7] perf: arm64: Compile tests unconditionally

Quoted 7 lines · Arnaldo Carvalho de Melo

Em Tue, May 28, 2019 at 04:03:14PM +0100, Raphael Gault escreveu:

Quoted 2 lines

In order to subsequently add more tests for the arm64 architecture we compile the tests target for arm64 systematically.

Humm, the subject doesn't match the description? I.e. it _was_ unconditionally built, now it is only built if CONFIG_DWARF_UNWIND is set to 'y'.

Perhaps it's hard to read, but we haven't introduced new conditionality.

Previously we'd only build the tests directory if CONFIG_DWARF_UNWIND was selected, so tests/dwarf-unwind.o was always dependent on that.

Now we always try to build the tests directory, and move the CONFIG_DWARF_UNWIND guard specifically to tests/dwarf-unwind.o.

Thanks,
Mark.

Quoted 33 lines

- Arnaldo

Quoted 26 lines
---
 tools/perf/arch/arm64/Build       | 2 +-
 tools/perf/arch/arm64/tests/Build | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
tools/perf/arch/arm64/Build +1 -1
index 36222e64bbf7..a7dd46a5b678 100644
@@ -1,2 +1,2 @@
 perf-y += util/
-perf-$(CONFIG_DWARF_UNWIND) += tests/
+perf-y += tests/
tools/perf/arch/arm64/tests/Build +1 -1
index 41707fea74b3..a61c06bdb757 100644
@@ -1,4 +1,4 @@
 perf-y += regs_load.o
-perf-y += dwarf-unwind.o
+perf-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
 
 perf-y += arch-tests.o
Signature · 1 line
2.17.1
Signature · 1 line
- Arnaldo
#4 lore raw

RFC v2 2/7 perf: arm64: Add test to check userspace access to hardware counters.

In reply to Raphael Gault

This test relies on the fact that the PMU registers are accessible from userspace. It then uses the perf_event_mmap_page to retrieve the counter index and access the underlying register.

This test uses sched_setaffinity(2) in order to run on all CPU and thus check the behaviour of the PMU of all cpus in a big.LITTLE environment.

---
 tools/perf/arch/arm64/include/arch-tests.h |   6 +
 tools/perf/arch/arm64/tests/Build          |   1 +
 tools/perf/arch/arm64/tests/arch-tests.c   |   4 +
 tools/perf/arch/arm64/tests/user-events.c  | 255 +++++++++++++++++++++
 4 files changed, 266 insertions(+)
 create mode 100644 tools/perf/arch/arm64/tests/user-events.c
tools/perf/arch/arm64/include/arch-tests.h +6 -0
index 90ec4c8cb880..a9b17ae0560b 100644
@@ -2,11 +2,17 @@
 #ifndef ARCH_TESTS_H
 #define ARCH_TESTS_H
 
+#define __maybe_unused	__attribute__((unused))
 #ifdef HAVE_DWARF_UNWIND_SUPPORT
 struct thread;
 struct perf_sample;
+int test__arch_unwind_sample(struct perf_sample *sample,
+			     struct thread *thread);
 #endif
 
 extern struct test arch_tests[];
+int test__rd_pmevcntr(struct test *test __maybe_unused,
+		      int subtest __maybe_unused);
+
 
 #endif
tools/perf/arch/arm64/tests/Build +1 -0
index a61c06bdb757..3f9a20c17fc6 100644
@@ -1,4 +1,5 @@
 perf-y += regs_load.o
 perf-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
 
+perf-y += user-events.o
 perf-y += arch-tests.o
tools/perf/arch/arm64/tests/arch-tests.c +4 -0
index 5b1543c98022..57df9b89dede 100644
@@ -10,6 +10,10 @@ struct test arch_tests[] = {
 		.func = test__dwarf_unwind,
 	},
 #endif
+	{
+		.desc = "User counter access",
+		.func = test__rd_pmevcntr,
+	},
 	{
 		.func = NULL,
 	},
tools/perf/arch/arm64/tests/user-events.c new file +255 -0

255 lines not on this page · raw message

Signature · 1 line
2.17.1
#5 lore raw

RFC v2 3/7 perf: arm64: Use rseq to test userspace access to pmu counters

In reply to Raphael Gault

Add an extra test to check userspace access to pmu hardware counters. This test doesn't rely on the seqlock as a synchronisation mechanism but instead uses the restartable sequences to make sure that the thread is not interrupted when reading the index of the counter and the associated pmu register.

In addition to reading the pmu counters, this test is run several time in order to measure the ratio of failures:
I ran this test on the Juno development platform, which is big.LITTLE with 4 Cortex A53 and 2 Cortex A57. The results vary quite a lot (running it with 100 tests is not so long and I did it several times). I ran it once with 10000 iterations:
runs: 10000, abort: 62.53%, zero: 34.93%, success: 2.54%

---
 tools/perf/arch/arm64/include/arch-tests.h    |   5 +-
 tools/perf/arch/arm64/include/rseq-arm64.h    | 220 ++++++++++++++++++
 tools/perf/arch/arm64/tests/Build             |   1 +
 tools/perf/arch/arm64/tests/arch-tests.c      |   6 +
 tools/perf/arch/arm64/tests/rseq-pmu-events.c | 219 +++++++++++++++++
 5 files changed, 450 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/arch/arm64/include/rseq-arm64.h
 create mode 100644 tools/perf/arch/arm64/tests/rseq-pmu-events.c
tools/perf/arch/arm64/include/arch-tests.h +4 -1
index a9b17ae0560b..4164762b43c6 100644
@@ -13,6 +13,9 @@ int test__arch_unwind_sample(struct perf_sample *sample,
 extern struct test arch_tests[];
 int test__rd_pmevcntr(struct test *test __maybe_unused,
 		      int subtest __maybe_unused);
-
+#ifdef CONFIG_RSEQ
+int rseq__rd_pmevcntr(struct test *test __maybe_unused,
+		      int subtest __maybe_unused);
+#endif
 
 #endif
tools/perf/arch/arm64/include/rseq-arm64.h new file +220 -0

220 lines not on this page · raw message

tools/perf/arch/arm64/tests/Build +1 -0
index 3f9a20c17fc6..b53823603066 100644
@@ -1,5 +1,6 @@
 perf-y += regs_load.o
 perf-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
+perf-$(CONFIG_RSEQ) += rseq-pmu-events.o
 
 perf-y += user-events.o
 perf-y += arch-tests.o
tools/perf/arch/arm64/tests/arch-tests.c +6 -0
index 57df9b89dede..5f08bdc41d89 100644
@@ -14,6 +14,12 @@ struct test arch_tests[] = {
 		.desc = "User counter access",
 		.func = test__rd_pmevcntr,
 	},
+#ifdef CONFIG_RSEQ
+	{
+		.desc = "User counter access with rseq",
+		.func = rseq__rd_pmevcntr,
+	},
+#endif
 	{
 		.func = NULL,
 	},
tools/perf/arch/arm64/tests/rseq-pmu-events.c new file +219 -0

219 lines not on this page · raw message

Signature · 1 line
2.17.1
#6 lore raw

RFC v2 4/7 arm64: pmu: Add function implementation to update event index in userpage.

In reply to Raphael Gault

In order to be able to access the counter directly for userspace, we need to provide the index of the counter using the userpage. We thus need to override the event_idx function to retrieve and convert the perf_event index to armv8 hardware index.

Since the arm_pmu driver can be used by any implementation, even if not armv8, two components play a role into making sure the behaviour is correct and consistent with the PMU capabilities:

* the ARMPMU_EL0_RD_CNTR flag which denotes the capability to access counter from userspace.
* the event_idx call back, which is implemented and initialized by the PMU implementation: if no callback is provided, the default behaviour applies, returning 0 as index value.

---
 arch/arm64/kernel/perf_event.c | 21 +++++++++++++++++++++
 include/linux/perf/arm_pmu.h   |  2 ++
 2 files changed, 23 insertions(+)
arch/arm64/kernel/perf_event.c +21 -0
index 6164d389eed6..3dc1265540df 100644
@@ -809,6 +809,22 @@ static void armv8pmu_clear_event_idx(struct pmu_hw_events *cpuc,
 		clear_bit(idx - 1, cpuc->used_mask);
 }
 
+static int armv8pmu_access_event_idx(struct perf_event *event)
+{
+	if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
+		return 0;
+
+	/*
+	 * We remap the cycle counter index to 32 to
+	 * match the offset applied to the rest of
+	 * the counter indeces.
+	 */
+	if (event->hw.idx == ARMV8_IDX_CYCLE_COUNTER)
+		return 32;
+
+	return event->hw.idx;
+}
+
 /*
  * Add an event filter to a given event.
  */
@@ -890,6 +906,8 @@ static int __armv8_pmuv3_map_event(struct perf_event *event,
 	if (armv8pmu_event_is_64bit(event))
 		event->hw.flags |= ARMPMU_EVT_64BIT;
 
+	event->hw.flags |= ARMPMU_EL0_RD_CNTR;
+
 	/* Only expose micro/arch events supported by this PMU */
 	if ((hw_event_id > 0) && (hw_event_id < ARMV8_PMUV3_MAX_COMMON_EVENTS)
 	    && test_bit(hw_event_id, armpmu->pmceid_bitmap)) {
@@ -1010,6 +1028,8 @@ static int armv8_pmu_init(struct arm_pmu *cpu_pmu)
 	cpu_pmu->set_event_filter	= armv8pmu_set_event_filter;
 	cpu_pmu->filter_match		= armv8pmu_filter_match;
 
+	cpu_pmu->pmu.event_idx		= armv8pmu_access_event_idx;
+
 	return 0;
 }
 
@@ -1188,6 +1208,7 @@ void arch_perf_update_userpage(struct perf_event *event,
 	 */
 	freq = arch_timer_get_rate();
 	userpg->cap_user_time = 1;
+	userpg->cap_user_rdpmc = !!(event->hw.flags & ARMPMU_EL0_RD_CNTR);
 
 	clocks_calc_mult_shift(&userpg->time_mult, &shift, freq,
 			NSEC_PER_SEC, 0);
include/linux/perf/arm_pmu.h +2 -0
index 4641e850b204..3bef390c1069 100644
@@ -30,6 +30,8 @@
  */
 /* Event uses a 64bit counter */
 #define ARMPMU_EVT_64BIT		1
+/* Allow access to hardware counter from userspace */
+#define ARMPMU_EL0_RD_CNTR		2
 
 #define HW_OP_UNSUPPORTED		0xFFFF
 #define C(_x)				PERF_COUNT_HW_CACHE_##_x
Signature · 1 line
2.17.1
maintainer
#7 lore raw

Re: [RFC 4/7] arm64: pmu: Add function implementation to update event index in userpage.

Quoted 4 lines · Raphael Gault
arch/arm64/kernel/perf_event.c +4 -0
+static int armv8pmu_access_event_idx(struct perf_event *event)
+{
+	if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
+		return 0;

arch/arm64/kernel/perf_event.c static void armv8pmu_clear_event_idx(struct pmu_hw_events *cpuc, · line 825

Quoted 10 lines
arch/arm64/kernel/perf_event.c +10 -0
+
+	/*
+	 * We remap the cycle counter index to 32 to
+	 * match the offset applied to the rest of
+	 * the counter indeces.
+	 */
+	if (event->hw.idx == ARMV8_IDX_CYCLE_COUNTER)
+		return 32;
+
+	return event->hw.idx;

Is there a guarantee event->hw.idx is never 0? Or should you, just like x86, use +1 here?

Quoted 1 line

+}

#8 lore raw

Re: [RFC 4/7] arm64: pmu: Add function implementation to update event index in userpage.

Hi Peter,

Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Raphael Gault
arch/arm64/kernel/perf_event.c +9 -0
+static int armv8pmu_access_event_idx(struct perf_event *event)
+{
+	if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
+		return 0;
+
+	/*
+	 * We remap the cycle counter index to 32 to
+	 * match the offset applied to the rest of
+	 * the counter indeces.
Quoted 9 lines

arch/arm64/kernel/perf_event.c static void armv8pmu_clear_event_idx(struct pmu_hw_events *cpuc, · line 825

Quoted 5 lines
arch/arm64/kernel/perf_event.c +5 -0
+	 */
+	if (event->hw.idx == ARMV8_IDX_CYCLE_COUNTER)
+		return 32;
+
+	return event->hw.idx;

Is there a guarantee event->hw.idx is never 0? Or should you, just like x86, use +1 here?

You are right, I should use +1 here. Thanks for pointing that out.

Quoted 1 line
Quoted 1 line

+}

Thanks,

Signature · 1 line
Raphael Gault
#9 lore raw

Re: [RFC 4/7] arm64: pmu: Add function implementation to update event index in userpage.

Quoted 23 lines · Raphael Gault

Hi Peter,

Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Raphael Gault
arch/arm64/kernel/perf_event.c +9 -0
+static int armv8pmu_access_event_idx(struct perf_event *event)
+{
+    if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
+        return 0;
+
+    /*
+     * We remap the cycle counter index to 32 to
+     * match the offset applied to the rest of
+     * the counter indeces.
Quoted 9 lines
Quoted 5 lines

+     */
+    if (event->hw.idx == ARMV8_IDX_CYCLE_COUNTER) +        return 32;
+
+    return event->hw.idx;

Is there a guarantee event->hw.idx is never 0? Or should you, just like x86, use +1 here?

Quoted 1 line

You are right, I should use +1 here. Thanks for pointing that out.

Isn't that already the case though, since we reserve index 0 for the cycle counter? I'm looking at ARMV8_IDX_TO_COUNTER() here...

Robin.

#10 lore raw

Re: [RFC 4/7] arm64: pmu: Add function implementation to update event index in userpage.

Hi Robin, Hi Peter,

Quoted 24 lines · Robin Murphy
Quoted 23 lines · Raphael Gault

Hi Peter,

Quoted 19 lines · Peter Zijlstra
Quoted 4 lines · Raphael Gault
arch/arm64/kernel/perf_event.c +4 -0
+static int armv8pmu_access_event_idx(struct perf_event *event)
+{
+    if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
+        return 0;

arch/arm64/kernel/perf_event.c static void armv8pmu_clear_event_idx(struct pmu_hw_events *cpuc, · line 825

Quoted 10 lines
arch/arm64/kernel/perf_event.c +10 -0
+
+    /*
+     * We remap the cycle counter index to 32 to
+     * match the offset applied to the rest of
+     * the counter indeces.
+     */
+    if (event->hw.idx == ARMV8_IDX_CYCLE_COUNTER)
+        return 32;
+
+    return event->hw.idx;

Is there a guarantee event->hw.idx is never 0? Or should you, just like x86, use +1 here?

Quoted 4 lines
Quoted 1 line

You are right, I should use +1 here. Thanks for pointing that out.

Isn't that already the case though, since we reserve index 0 for the cycle counter? I'm looking at ARMV8_IDX_TO_COUNTER() here...

Well the current behaviour is correct and takes care of the zero case with the ARMV8_IDX_CYCLE_COUNTER check. But using ARMV8_IDX_TO_COUNTER() and add 1 would also work. However this seems indeed redundant with the current value held in event->hw.idx.

Quoted 1 line

Robin.

Signature · 1 line
Raphael Gault
maintainer
#11 lore raw

Re: [RFC 4/7] arm64: pmu: Add function implementation to update event index in userpage.

Quoted 29 lines · Raphael Gault

Hi Robin, Hi Peter,

Quoted 26 lines · Robin Murphy
Quoted 24 lines · Raphael Gault

Hi Peter,

Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Raphael Gault
arch/arm64/kernel/perf_event.c +9 -0
+static int armv8pmu_access_event_idx(struct perf_event *event)
+{
+    if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
+        return 0;
+
+    /*
+     * We remap the cycle counter index to 32 to
+     * match the offset applied to the rest of
+     * the counter indeces.
Quoted 9 lines
Quoted 5 lines

+     */
+    if (event->hw.idx == ARMV8_IDX_CYCLE_COUNTER) +        return 32;
+
+    return event->hw.idx;

Is there a guarantee event->hw.idx is never 0? Or should you, just like x86, use +1 here?

You are right, I should use +1 here. Thanks for pointing that out.

Quoted 8 lines
Quoted 3 lines

Isn't that already the case though, since we reserve index 0 for the cycle counter? I'm looking at ARMV8_IDX_TO_COUNTER() here...

Well the current behaviour is correct and takes care of the zero case with the ARMV8_IDX_CYCLE_COUNTER check. But using ARMV8_IDX_TO_COUNTER() and add 1 would also work. However this seems indeed redundant with the current value held in event->hw.idx.

Note that whatever you pick now will become ABI. Also note that the comment/pseudo-code in perf_event_mmap_page suggests to use idx-1 for the actual hardware access.

#12 lore raw

Re: [RFC 4/7] arm64: pmu: Add function implementation to update event index in userpage.

Hi Peter,

Quoted 34 lines · Peter Zijlstra
Quoted 33 lines · Raphael Gault

Hi Robin, Hi Peter,

Quoted 29 lines · Robin Murphy
Quoted 23 lines · Raphael Gault

Hi Peter,

Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Raphael Gault
arch/arm64/kernel/perf_event.c +9 -0
+static int armv8pmu_access_event_idx(struct perf_event *event)
+{
+    if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
+        return 0;
+
+    /*
+     * We remap the cycle counter index to 32 to
+     * match the offset applied to the rest of
+     * the counter indeces.
Quoted 9 lines
Quoted 5 lines

+     */
+    if (event->hw.idx == ARMV8_IDX_CYCLE_COUNTER) +        return 32;
+
+    return event->hw.idx;

Is there a guarantee event->hw.idx is never 0? Or should you, just like x86, use +1 here?

Quoted 1 line

You are right, I should use +1 here. Thanks for pointing that out.

Isn't that already the case though, since we reserve index 0 for the cycle counter? I'm looking at ARMV8_IDX_TO_COUNTER() here...

Quoted 9 lines
Quoted 4 lines

Well the current behaviour is correct and takes care of the zero case with the ARMV8_IDX_CYCLE_COUNTER check. But using ARMV8_IDX_TO_COUNTER() and add 1 would also work. However this seems indeed redundant with the current value held in event->hw.idx.

Note that whatever you pick now will become ABI. Also note that the comment/pseudo-code in perf_event_mmap_page suggests to use idx-1 for the actual hardware access.

Indeed that's true. As for the pseudo-code in perf_event_mmap_page. It is compatible with what I do here. The two approach are only different in form but it is in both case necessary to subtract 1 on the returned value in order to access the correct hardware counter.

Thank you,

Signature · 1 line
Raphael Gault
#13 lore raw

RFC v2 5/7 arm64: pmu: Add hook to handle pmu-related undefined instructions

In reply to Raphael Gault

In order to prevent the userspace processes which are trying to access the registers from the pmu registers on a big.LITTLE environment we introduce a hook to handle undefined instructions.

The goal here is to prevent the process to be interrupted by a signal when the error is caused by the task being scheduled while accessing a counter, causing the counter access to be invalid. As we are not able to know efficiently the number of counters available physically on both pmu in that context we consider that any faulting access to a counter which is architecturally correct should not cause a SIGILL signal if the permissions are set accordingly.

This commit also modifies the mask of the mrs_hook declared in arch/arm64/kernel/cpufeatures.c which emulates only feature register access. This is necessary because this hook's mask was too large and thus masking any mrs instruction, even if not related to the emulated registers which made the pmu emulation inefficient.

---
 arch/arm64/kernel/cpufeature.c |  4 ++--
 arch/arm64/kernel/perf_event.c | 41 ++++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 2 deletions(-)
arch/arm64/kernel/cpufeature.c +2 -2
index 2b807f129e60..daa7b31f2c73 100644
@@ -2166,8 +2166,8 @@ static int emulate_mrs(struct pt_regs *regs, u32 insn)
 }
 
 static struct undef_hook mrs_hook = {
-	.instr_mask = 0xfff00000,
-	.instr_val  = 0xd5300000,
+	.instr_mask = 0xffff0000,
+	.instr_val  = 0xd5380000,
 	.pstate_mask = PSR_AA32_MODE_MASK,
 	.pstate_val = PSR_MODE_EL0t,
 	.fn = emulate_mrs,
arch/arm64/kernel/perf_event.c +41 -0

56 lines not on this page · raw message

Signature · 1 line
2.17.1
maintainer
#14 lore raw

Re: [RFC 5/7] arm64: pmu: Add hook to handle pmu-related undefined instructions

arch/arm64/kernel/perf_event.c static int armv8pmu_probe_pmu(struct arm_pmu *cpu_pmu) · line 1026

Quoted 13 lines · Raphael Gault
arch/arm64/kernel/perf_event.c +13 -0
+static int emulate_pmu(struct pt_regs *regs, u32 insn)
+{
+	u32 sys_reg, rt;
+	u32 pmuserenr;
+
+	sys_reg = (u32)aarch64_insn_decode_immediate(AARCH64_INSN_IMM_16, insn) << 5;
+	rt = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RT, insn);
+	pmuserenr = read_sysreg(pmuserenr_el0);
+
+	if ((pmuserenr & (ARMV8_PMU_USERENR_ER|ARMV8_PMU_USERENR_CR)) !=
+	    (ARMV8_PMU_USERENR_ER|ARMV8_PMU_USERENR_CR))
+		return -EINVAL;
+

I would really prefer there to be a comment here that explain how the '0' value works. Maybe something like:

/*
* Userspace is expected to only use this in the context of the
* scheme described in the struct perf_event_mmap_page comments. *
* Given that context, we can only get here if we got migrated
* between getting the register index and doing the MSR read.
* This in turn implies we'll fail the sequence and retry, so
* any value returned is 'good', all we need is to be non-fatal. */

arch/arm64/kernel/perf_event.c static int armv8pmu_probe_pmu(struct arm_pmu *cpu_pmu) · line 1027

Quoted 1 line
arch/arm64/kernel/perf_event.c +1 -0
+	pt_regs_write_reg(regs, rt, 0);

And given the above, we don't even need to do this, we can simply preserve whatever garbage was in the register and return to userspace.

The only thing we really need is for the trap to be non-fatal.

Quoted 24 lines
arch/arm64/kernel/perf_event.c +24 -0
+
+	arm64_skip_faulting_instruction(regs, 4);
+	return 0;
+}
+
+/*
+ * This hook will only be triggered by mrs
+ * instructions on PMU registers. This is mandatory
+ * in order to have a consistent behaviour even on
+ * big.LITTLE systems.
+ */
+static struct undef_hook pmu_hook = {
+	.instr_mask = 0xffff8800,
+	.instr_val  = 0xd53b8800,
+	.fn = emulate_pmu,
+};
+
+static int __init enable_pmu_emulation(void)
+{
+	register_undef_hook(&pmu_hook);
+	return 0;
+}
+
+core_initcall(enable_pmu_emulation);
#15 lore raw

RFC v2 6/7 arm64: perf: Enable pmu counter direct access for perf event on armv8

In reply to Raphael Gault

Keep track of event opened with direct access to the hardware counters and modify permissions while they are open.

The strategy used here is the same which x86 uses: everytime an event is mapped, the permissions are set if required. The atomic field added in the mm_context helps keep track of the different event opened and de-activate the permissions when all are unmapped.
We also need to update the permissions in the context switch code so that tasks keep the right permissions.

---
 arch/arm64/include/asm/mmu.h         |  6 +++++
 arch/arm64/include/asm/mmu_context.h |  2 ++
 arch/arm64/include/asm/perf_event.h  | 14 ++++++++++
 drivers/perf/arm_pmu.c               | 38 ++++++++++++++++++++++++++++
 4 files changed, 60 insertions(+)
arch/arm64/include/asm/mmu.h +6 -0
index 67ef25d037ea..9de4cf0b17c7 100644
@@ -29,6 +29,12 @@
 
 typedef struct {
 	atomic64_t	id;
+
+	/*
+	 * non-zero if userspace have access to hardware
+	 * counters directly.
+	 */
+	atomic_t	pmu_direct_access;
 	void		*vdso;
 	unsigned long	flags;
 } mm_context_t;
arch/arm64/include/asm/mmu_context.h +2 -0
index 2da3e478fd8f..33494af613d8 100644
@@ -32,6 +32,7 @@
 #include <asm-generic/mm_hooks.h>
 #include <asm/cputype.h>
 #include <asm/pgtable.h>
+#include <asm/perf_event.h>
 #include <asm/sysreg.h>
 #include <asm/tlbflush.h>
 
@@ -235,6 +236,7 @@ static inline void __switch_mm(struct mm_struct *next)
 	}
 
 	check_and_switch_context(next, cpu);
+	perf_switch_user_access(next);
 }
 
 static inline void
arch/arm64/include/asm/perf_event.h +14 -0
index c593761ba61c..32a6d604bb3b 100644
@@ -19,6 +19,7 @@
 
 #include <asm/stack_pointer.h>
 #include <asm/ptrace.h>
+#include <linux/mm_types.h>
 
 #define	ARMV8_PMU_MAX_COUNTERS	32
 #define	ARMV8_PMU_COUNTER_MASK	(ARMV8_PMU_MAX_COUNTERS - 1)
@@ -234,4 +235,17 @@ extern unsigned long perf_misc_flags(struct pt_regs *regs);
 	(regs)->pstate = PSR_MODE_EL1h;	\
 }
 
+static inline void perf_switch_user_access(struct mm_struct *mm)
+{
+	if (!IS_ENABLED(CONFIG_PERF_EVENTS))
+		return;
+
+	if (atomic_read(&mm->context.pmu_direct_access)) {
+		write_sysreg(ARMV8_PMU_USERENR_ER|ARMV8_PMU_USERENR_CR,
+			     pmuserenr_el0);
+	} else {
+		write_sysreg(0, pmuserenr_el0);
+	}
+}
+
 #endif
drivers/perf/arm_pmu.c +38 -0
index eec75b97e7ea..0e5588cd2f39 100644
@@ -24,6 +24,7 @@
 #include <linux/irqdesc.h>
 
 #include <asm/irq_regs.h>
+#include <asm/mmu_context.h>
 
 static DEFINE_PER_CPU(struct arm_pmu *, cpu_armpmu);
 static DEFINE_PER_CPU(int, cpu_irq);
@@ -777,6 +778,41 @@ static void cpu_pmu_destroy(struct arm_pmu *cpu_pmu)
 					    &cpu_pmu->node);
 }
 
+static void refresh_pmuserenr(void *mm)
+{
+	perf_switch_user_access(mm);
+}
+
+static void armpmu_event_mapped(struct perf_event *event, struct mm_struct *mm)
+{
+	if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
+		return;
+
+	/*
+	 * This function relies on not being called concurrently in two
+	 * tasks in the same mm.  Otherwise one task could observe
+	 * pmu_direct_access > 1 and return all the way back to
+	 * userspace with user access disabled while another task is still
+	 * doing on_each_cpu_mask() to enable user access.
+	 *
+	 * For now, this can't happen because all callers hold mmap_sem
+	 * for write.  If this changes, we'll need a different solution.
+	 */
+	lockdep_assert_held_exclusive(&mm->mmap_sem);
+
+	if (atomic_inc_return(&mm->context.pmu_direct_access) == 1)
+		on_each_cpu(refresh_pmuserenr, mm, 1);
+}
+
+static void armpmu_event_unmapped(struct perf_event *event, struct mm_struct *mm)
+{
+	if (!(event->hw.flags & ARMPMU_EL0_RD_CNTR))
+		return;
+
+	if (atomic_dec_and_test(&mm->context.pmu_direct_access))
+		on_each_cpu_mask(mm_cpumask(mm), refresh_pmuserenr, NULL, 1);
+}
+
 static struct arm_pmu *__armpmu_alloc(gfp_t flags)
 {
 	struct arm_pmu *pmu;
@@ -798,6 +834,8 @@ static struct arm_pmu *__armpmu_alloc(gfp_t flags)
 		.pmu_enable	= armpmu_enable,
 		.pmu_disable	= armpmu_disable,
 		.event_init	= armpmu_event_init,
+		.event_mapped	= armpmu_event_mapped,
+		.event_unmapped	= armpmu_event_unmapped,
 		.add		= armpmu_add,
 		.del		= armpmu_del,
 		.start		= armpmu_start,
Signature · 1 line
2.17.1
#16 lore raw

RFC v2 7/7 Documentation: arm64: Document PMU counters access from userspace

In reply to Raphael Gault

Add a documentation file to describe the access to the pmu hardware counters from userspace

---
 .../arm64/pmu_counter_user_access.txt         | 42 +++++++++++++++++++
 1 file changed, 42 insertions(+)
 create mode 100644 Documentation/arm64/pmu_counter_user_access.txt
Documentation/arm64/pmu_counter_user_access.txt new file +42 -0
new file mode 100644
index 000000000000..6788b1107381
@@ -0,0 +1,42 @@
+Access to PMU hardware counter from userspace
+=============================================
+
+Overview
+--------
+The perf user-space tool relies on the PMU to monitor events. It offers an
+abstraction layer over the hardware counters since the underlying
+implementation is cpu-dependent.
+Arm64 allows userspace tools to have access to the registers storing the
+hardware counters' values directly.
+
+This targets specifically self-monitoring tasks in order to reduce the overhead
+by directly accessing the registers without having to go through the kernel.
+
+How-to
+------
+The focus is set on the armv8 pmuv3 which makes sure that the access to the pmu
+registers is enable and that the userspace have access to the relevent
+information in order to use them.
+
+In order to have access to the hardware counter it is necessary to open the event
+using the perf tool interface: the sys_perf_event_open syscall returns a fd which
+can subsequently be used with the mmap syscall in order to retrieve a page of memory
+containing information about the event.
+The PMU driver uses this page to expose to the user the hardware counter's
+index. Using this index enables the user to access the PMU registers using the
+`mrs` instruction.
+
+Have a look `at tools/perf/arch/arm64/tests/user-events.c` for an example. It can be
+run using the perf tool to check that the access to the registers works
+correctly from userspace:
+
+./perf test -v
+
+About chained events
+--------------------
+When the user requests for an event to be counted on 64 bits, two hardware
+counters are used and need to be combined to retrieve the correct value:
+
+val = read_counter(idx);
+if ((event.attr.config1 & 0x1))
+	val = (val << 32) | read_counter(idx - 1);
Signature · 1 line
2.17.1