Skip to contentSkip to conversation

drivers/perf: CPU PMU driver for Apple M1

PATCH Superseded Marc Zyngier 4 participants 16 messages Nov 13, 2021 – Nov 29, 2021

Subsystems (5)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

cover drivers/perf: CPU PMU driver for Apple M1

The M1 SoC embeds a per-CPU PMU that has a very different programming interface compared to the architected PMUv3 that is normally present on standard implementations.

This small series adds a driver for this HW by leveraging the arm_pmu infrastructure, resulting in a rather simple driver.

Of course, we know next to nothing about the actual events this PMU counts, aside from CPU cycles and instructions. Everything else is undocumented.

My hope is that this driver will help people to explore the event space and propose possible interpretations for these events using reproducible test cases.

Marc Zyngier (8):
dt-bindings: arm-pmu: Document Apple PMU compatible strings
dt-bindings: apple,aic: Add CPU PMU per-cpu pseudo-interrupts irqchip/apple-aic: Add cpumasks for E and P cores irqchip/apple-aic: Wire PMU interrupts irqchip/apple-aic: Move PMU-specific registers to their own include
file
arm64: apple: t8301: Add PMU nodes
drivers/perf: arm_pmu: Handle 47 bit counters drivers/perf: Add Apple icestorm/firestorm CPU PMU driver

 .../devicetree/bindings/arm/pmu.yaml          |   2 +
 .../interrupt-controller/apple,aic.yaml       |   2 +
 arch/arm64/boot/dts/apple/t8103.dtsi          |  12 +
 arch/arm64/include/asm/apple_m1_pmu.h         |  64 ++
 drivers/irqchip/irq-apple-aic.c               |  59 +-
 drivers/perf/Kconfig                          |   7 +
 drivers/perf/Makefile                         |   1 +
 drivers/perf/apple_m1_cpu_pmu.c               | 632 ++++++++++++++++++
 drivers/perf/arm_pmu.c                        |   2 +
 .../interrupt-controller/apple-aic.h          |   2 +
 include/linux/perf/arm_pmu.h                  |   2 +
 11 files changed, 763 insertions(+), 22 deletions(-)
 create mode 100644 arch/arm64/include/asm/apple_m1_pmu.h
 create mode 100644 drivers/perf/apple_m1_cpu_pmu.c
Signature · 1 line
2.30.2
#1 lore raw

PATCH 1/8 dt-bindings: arm-pmu: Document Apple PMU compatible strings

As we are about to add support fur the Apple PMUs, document the compatible strings associated with the two micro-architectures present in the Apple M1.

---
 Documentation/devicetree/bindings/arm/pmu.yaml | 2 ++
 1 file changed, 2 insertions(+)
Documentation/devicetree/bindings/arm/pmu.yaml +2 -0
index e17ac049e890..1a6986b4e552 100644
@@ -20,6 +20,8 @@ properties:
     items:
       - enum:
           - apm,potenza-pmu
+          - apple,firestorm-pmu
+          - apple,icestorm-pmu
           - arm,armv8-pmuv3 # Only for s/w models
           - arm,arm1136-pmu
           - arm,arm1176-pmu
Signature · 1 line
2.30.2
maintainer Acked-by on v1 1/8
#2 lore raw

Re: [PATCH 1/8] dt-bindings: arm-pmu: Document Apple PMU compatible strings

Quoted 8 lines · Marc Zyngier

As we are about to add support fur the Apple PMUs, document the compatible strings associated with the two micro-architectures present in the Apple M1.

---
 Documentation/devicetree/bindings/arm/pmu.yaml | 2 ++
 1 file changed, 2 insertions(+)
#3 lore raw

PATCH 2/8 dt-bindings: apple,aic: Add CPU PMU per-cpu pseudo-interrupts

In reply to Marc Zyngier

Advertise the two pseudo-interrupts that tied to the two PMU flavours present in the Apple M1 SoC.

We choose the expose two different pseudo-interrupts to the OS as the e-core PMU is obviously different from the p-core one, effectively presenting two different devices.

---
 .../devicetree/bindings/interrupt-controller/apple,aic.yaml     | 2 ++
 include/dt-bindings/interrupt-controller/apple-aic.h            | 2 ++
 2 files changed, 4 insertions(+)
Documentation/devicetree/bindings/interrupt-controller/apple,aic.yaml +2 -0
index cf6c091a07b1..b95e41816953 100644
@@ -56,6 +56,8 @@ properties:
           - 1: virtual HV timer
           - 2: physical guest timer
           - 3: virtual guest timer
+          - 4: 'efficient' CPU PMU
+          - 5: 'performance' CPU PMU
 
       The 3rd cell contains the interrupt flags. This is normally
       IRQ_TYPE_LEVEL_HIGH (4).
include/dt-bindings/interrupt-controller/apple-aic.h +2 -0
index 604f2bb30ac0..bf3aac0e5491 100644
@@ -11,5 +11,7 @@
 #define AIC_TMR_HV_VIRT		1
 #define AIC_TMR_GUEST_PHYS	2
 #define AIC_TMR_GUEST_VIRT	3
+#define AIC_CPU_PMU_E		4
+#define AIC_CPU_PMU_P		5
 
 #endif
Signature · 1 line
2.30.2
maintainer Acked-by on v1 2/8
#4 lore raw

Re: [PATCH 2/8] dt-bindings: apple,aic: Add CPU PMU per-cpu pseudo-interrupts

Quoted 7 lines · Marc Zyngier

Advertise the two pseudo-interrupts that tied to the two PMU flavours present in the Apple M1 SoC.

We choose the expose two different pseudo-interrupts to the OS as the e-core PMU is obviously different from the p-core one, effectively presenting two different devices.

Quoted 6 lines
---
 .../devicetree/bindings/interrupt-controller/apple,aic.yaml     | 2 ++
 include/dt-bindings/interrupt-controller/apple-aic.h            | 2 ++
 2 files changed, 4 insertions(+)
#5 lore raw

PATCH 3/8 irqchip/apple-aic: Add cpumasks for E and P cores

In reply to Marc Zyngier

In order to be able to tell the core IRQ code about the affinity of the PMU interrupt in later patches, compute the cpumasks of the P and E cores at boot time.

This relies on the affinity scheme used by the vendor, which seems to work for the couple of SoCs that are out in the wild.

---
 drivers/irqchip/irq-apple-aic.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
drivers/irqchip/irq-apple-aic.c +14 -0
index 3759dc36cc8f..30ca80ccda8b 100644
@@ -177,6 +177,8 @@ struct aic_irq_chip {
 	void __iomem *base;
 	struct irq_domain *hw_domain;
 	struct irq_domain *ipi_domain;
+	struct cpumask ecore_mask;
+	struct cpumask pcore_mask;
 	int nr_hw;
 	int ipi_hwirq;
 };
@@ -200,6 +202,11 @@ static void aic_ic_write(struct aic_irq_chip *ic, u32 reg, u32 val)
 	writel_relaxed(val, ic->base + reg);
 }
 
+static bool __is_pcore(u64 mpidr)
+{
+	return MPIDR_AFFINITY_LEVEL(mpidr, 2) == 1;
+}
+
 /*
  * IRQ irqchip
  */
@@ -833,6 +840,13 @@ static int __init aic_of_ic_init(struct device_node *node, struct device_node *p
 		return -ENODEV;
 	}
 
+	for_each_possible_cpu(i) {
+		if (__is_pcore(cpu_logical_map(i)))
+			cpumask_set_cpu(i, &irqc->pcore_mask);
+		else
+			cpumask_set_cpu(i, &irqc->ecore_mask);
+	}
+
 	set_handle_irq(aic_handle_irq);
 	set_handle_fiq(aic_handle_fiq);
 
Signature · 1 line
2.30.2
#6 lore raw

PATCH 4/8 irqchip/apple-aic: Wire PMU interrupts

In reply to Marc Zyngier

Add the necessary code to configure and P and E-core PMU interrupts with their respective affinities. When such an interrupt fires, map it onto the right pseudo-interrupt.

---
 drivers/irqchip/irq-apple-aic.c | 34 +++++++++++++++++++++------------
 1 file changed, 22 insertions(+), 12 deletions(-)
drivers/irqchip/irq-apple-aic.c +22 -12
index 30ca80ccda8b..23f5f10e974e 100644
@@ -155,7 +155,7 @@
 #define SYS_IMP_APL_UPMSR_EL1		sys_reg(3, 7, 15, 6, 4)
 #define UPMSR_IACT			BIT(0)
 
-#define AIC_NR_FIQ		4
+#define AIC_NR_FIQ		6
 #define AIC_NR_SWIPI		32
 
 /*
@@ -207,6 +207,11 @@ static bool __is_pcore(u64 mpidr)
 	return MPIDR_AFFINITY_LEVEL(mpidr, 2) == 1;
 }
 
+static bool is_pcore(void)
+{
+	return __is_pcore(read_cpuid_mpidr());
+}
+
 /*
  * IRQ irqchip
  */
@@ -420,16 +425,10 @@ static void __exception_irq_entry aic_handle_fiq(struct pt_regs *regs)
 						  aic_irqc->nr_hw + AIC_TMR_EL02_VIRT);
 	}
 
-	if ((read_sysreg_s(SYS_IMP_APL_PMCR0_EL1) & (PMCR0_IMODE | PMCR0_IACT)) ==
-			(FIELD_PREP(PMCR0_IMODE, PMCR0_IMODE_FIQ) | PMCR0_IACT)) {
-		/*
-		 * Not supported yet, let's figure out how to handle this when
-		 * we implement these proprietary performance counters. For now,
-		 * just mask it and move on.
-		 */
-		pr_err_ratelimited("PMC FIQ fired. Masking.\n");
-		sysreg_clear_set_s(SYS_IMP_APL_PMCR0_EL1, PMCR0_IMODE | PMCR0_IACT,
-				   FIELD_PREP(PMCR0_IMODE, PMCR0_IMODE_OFF));
+	if (read_sysreg_s(SYS_IMP_APL_PMCR0_EL1) & PMCR0_IACT) {
+		int irq = is_pcore() ? AIC_CPU_PMU_P : AIC_CPU_PMU_E;
+		generic_handle_domain_irq(aic_irqc->hw_domain,
+					  aic_irqc->nr_hw + irq);
 	}
 
 	if (FIELD_GET(UPMCR0_IMODE, read_sysreg_s(SYS_IMP_APL_UPMCR0_EL1)) == UPMCR0_IMODE_FIQ &&
@@ -469,7 +468,18 @@ static int aic_irq_domain_map(struct irq_domain *id, unsigned int irq,
 				    handle_fasteoi_irq, NULL, NULL);
 		irqd_set_single_target(irq_desc_get_irq_data(irq_to_desc(irq)));
 	} else {
-		irq_set_percpu_devid(irq);
+		switch (hw - ic->nr_hw) {
+		case AIC_CPU_PMU_P:
+			irq_set_percpu_devid_partition(irq, &ic->pcore_mask);
+			break;
+		case AIC_CPU_PMU_E:
+			irq_set_percpu_devid_partition(irq, &ic->ecore_mask);
+			break;
+		default:
+			irq_set_percpu_devid(irq);
+			break;
+		}
+
 		irq_domain_set_info(id, irq, hw, &fiq_chip, id->host_data,
 				    handle_percpu_devid_irq, NULL, NULL);
 	}
Signature · 1 line
2.30.2
#7 lore raw

PATCH 5/8 irqchip/apple-aic: Move PMU-specific registers to their own include file

In reply to Marc Zyngier

As we are about to have a PMU driver, move the PMU bits from the AIC driver into a common include file.

---
 arch/arm64/include/asm/apple_m1_pmu.h | 19 +++++++++++++++++++
 drivers/irqchip/irq-apple-aic.c       | 11 +----------
 2 files changed, 20 insertions(+), 10 deletions(-)
 create mode 100644 arch/arm64/include/asm/apple_m1_pmu.h
arch/arm64/include/asm/apple_m1_pmu.h new file +19 -0
new file mode 100644
index 000000000000..b848af7faadc
@@ -0,0 +1,19 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#ifndef __ASM_APPLE_M1_PMU_h
+#define __ASM_APPLE_M1_PMU_h
+
+#include <linux/bits.h>
+#include <asm/sysreg.h>
+
+/* Core PMC control register */
+#define SYS_IMP_APL_PMCR0_EL1	sys_reg(3, 1, 15, 0, 0)
+#define PMCR0_IMODE		GENMASK(10, 8)
+#define PMCR0_IMODE_OFF		0
+#define PMCR0_IMODE_PMI		1
+#define PMCR0_IMODE_AIC		2
+#define PMCR0_IMODE_HALT	3
+#define PMCR0_IMODE_FIQ		4
+#define PMCR0_IACT		BIT(11)
+
+#endif /* __ASM_APPLE_M1_PMU_h */
drivers/irqchip/irq-apple-aic.c +1 -10
index 23f5f10e974e..9663166fd97f 100644
@@ -55,6 +55,7 @@
 #include <linux/limits.h>
 #include <linux/of_address.h>
 #include <linux/slab.h>
+#include <asm/apple_m1_pmu.h>
 #include <asm/exception.h>
 #include <asm/sysreg.h>
 #include <asm/virt.h>
@@ -109,16 +110,6 @@
  * Note: sysreg-based IPIs are not supported yet.
  */
 
-/* Core PMC control register */
-#define SYS_IMP_APL_PMCR0_EL1		sys_reg(3, 1, 15, 0, 0)
-#define PMCR0_IMODE			GENMASK(10, 8)
-#define PMCR0_IMODE_OFF			0
-#define PMCR0_IMODE_PMI			1
-#define PMCR0_IMODE_AIC			2
-#define PMCR0_IMODE_HALT		3
-#define PMCR0_IMODE_FIQ			4
-#define PMCR0_IACT			BIT(11)
-
 /* IPI request registers */
 #define SYS_IMP_APL_IPI_RR_LOCAL_EL1	sys_reg(3, 5, 15, 0, 0)
 #define SYS_IMP_APL_IPI_RR_GLOBAL_EL1	sys_reg(3, 5, 15, 0, 1)
Signature · 1 line
2.30.2
#8 lore raw

PATCH 6/8 arm64: apple: t8301: Add PMU nodes

In reply to Marc Zyngier

Advertise the two PMU nodes for the t8103 SoC.

---
 arch/arm64/boot/dts/apple/t8103.dtsi | 12 ++++++++++++
 1 file changed, 12 insertions(+)
arch/arm64/boot/dts/apple/t8103.dtsi +12 -0
index fc8b2bb06ffe..d2e9afde3729 100644
@@ -96,6 +96,18 @@ timer {
 			     <AIC_FIQ AIC_TMR_HV_VIRT IRQ_TYPE_LEVEL_HIGH>;
 	};
 
+	pmu-e {
+		compatible = "apple,icestorm-pmu";
+		interrupt-parent = <&aic>;
+		interrupts = <AIC_FIQ AIC_CPU_PMU_E IRQ_TYPE_LEVEL_HIGH>;
+	};
+
+	pmu-p {
+		compatible = "apple,firestorm-pmu";
+		interrupt-parent = <&aic>;
+		interrupts = <AIC_FIQ AIC_CPU_PMU_P IRQ_TYPE_LEVEL_HIGH>;
+	};
+
 	clk24: clock-24m {
 		compatible = "fixed-clock";
 		#clock-cells = <0>;
Signature · 1 line
2.30.2
#9 lore raw

PATCH 7/8 drivers/perf: arm_pmu: Handle 47 bit counters

In reply to Marc Zyngier

The current ARM PMU framework can only deal with 32 or 64bit counters. Teach it about a 47bit flavour.

Yes, this is odd.

---
 drivers/perf/arm_pmu.c       | 2 ++
 include/linux/perf/arm_pmu.h | 2 ++
 2 files changed, 4 insertions(+)
drivers/perf/arm_pmu.c +2 -0
index 295cc7952d0e..0a9ed1a061ac 100644
@@ -109,6 +109,8 @@ static inline u64 arm_pmu_event_max_period(struct perf_event *event)
 {
 	if (event->hw.flags & ARMPMU_EVT_64BIT)
 		return GENMASK_ULL(63, 0);
+	else if (event->hw.flags & ARMPMU_EVT_47BIT)
+		return GENMASK_ULL(46, 0);
 	else
 		return GENMASK_ULL(31, 0);
 }
include/linux/perf/arm_pmu.h +2 -0
index 2512e2f9cd4e..0407a38b470a 100644
@@ -26,6 +26,8 @@
  */
 /* Event uses a 64bit counter */
 #define ARMPMU_EVT_64BIT		1
+/* Event uses a 47bit counter */
+#define ARMPMU_EVT_47BIT		2
 
 #define HW_OP_UNSUPPORTED		0xFFFF
 #define C(_x)				PERF_COUNT_HW_CACHE_##_x
Signature · 1 line
2.30.2
#10 lore raw

PATCH 8/8 drivers/perf: Add Apple icestorm/firestorm CPU PMU driver

In reply to Marc Zyngier

Add a new, weird and wonderful driver for the equally weird Apple PMU HW. Although the PMU itself is functional, we don't know much about the events yet, so this can be considered as yet another random number generator...

Nonetheless, it can reliably count at least cycles and instructions in the usually wonky big-little way. For anything else, it of course supports raw event numbers.

---
 arch/arm64/include/asm/apple_m1_pmu.h |  45 ++
 drivers/perf/Kconfig                  |   7 +
 drivers/perf/Makefile                 |   1 +
 drivers/perf/apple_m1_cpu_pmu.c       | 632 ++++++++++++++++++++++++++
 4 files changed, 685 insertions(+)
 create mode 100644 drivers/perf/apple_m1_cpu_pmu.c
arch/arm64/include/asm/apple_m1_pmu.h +45 -0
index b848af7faadc..99483b19b99f 100644
@@ -6,8 +6,21 @@
 #include <linux/bits.h>
 #include <asm/sysreg.h>
 
+/* Counters */
+#define SYS_IMP_APL_PMC0_EL1	sys_reg(3, 2, 15, 0, 0)
+#define SYS_IMP_APL_PMC1_EL1	sys_reg(3, 2, 15, 1, 0)
+#define SYS_IMP_APL_PMC2_EL1	sys_reg(3, 2, 15, 2, 0)
+#define SYS_IMP_APL_PMC3_EL1	sys_reg(3, 2, 15, 3, 0)
+#define SYS_IMP_APL_PMC4_EL1	sys_reg(3, 2, 15, 4, 0)
+#define SYS_IMP_APL_PMC5_EL1	sys_reg(3, 2, 15, 5, 0)
+#define SYS_IMP_APL_PMC6_EL1	sys_reg(3, 2, 15, 6, 0)
+#define SYS_IMP_APL_PMC7_EL1	sys_reg(3, 2, 15, 7, 0)
+#define SYS_IMP_APL_PMC8_EL1	sys_reg(3, 2, 15, 9, 0)
+#define SYS_IMP_APL_PMC9_EL1	sys_reg(3, 2, 15, 10, 0)
+
 /* Core PMC control register */
 #define SYS_IMP_APL_PMCR0_EL1	sys_reg(3, 1, 15, 0, 0)
+#define PMCR0_CNT_ENABLE_0_7	GENMASK(7, 0)
 #define PMCR0_IMODE		GENMASK(10, 8)
 #define PMCR0_IMODE_OFF		0
 #define PMCR0_IMODE_PMI		1
@@ -15,5 +28,37 @@
 #define PMCR0_IMODE_HALT	3
 #define PMCR0_IMODE_FIQ		4
 #define PMCR0_IACT		BIT(11)
+#define PMCR0_PMI_ENABLE_0_7	GENMASK(19, 12)
+#define PMCR0_STOP_CNT_ON_PMI	BIT(20)
+#define PMCR0_CNT_GLOB_L2C_EVT	BIT(21)
+#define PMCR0_DEFER_PMI_TO_ERET	BIT(22)
+#define PMCR0_ALLOW_CNT_EN_EL0	BIT(30)
+#define PMCR0_CNT_ENABLE_8_9	GENMASK(33, 32)
+#define PMCR0_PMI_ENABLE_8_9	GENMASK(45, 44)
+
+#define SYS_IMP_APL_PMCR1_EL1	sys_reg(3, 1, 15, 1, 0)
+#define PMCR1_COUNT_A64_EL0_0_7	GENMASK(15, 8)
+#define PMCR1_COUNT_A64_EL1_0_7	GENMASK(23, 16)
+#define PMCR1_COUNT_A64_EL0_8_9	GENMASK(41, 40)
+#define PMCR1_COUNT_A64_EL1_8_9	GENMASK(49, 48)
+
+#define SYS_IMP_APL_PMCR2_EL1	sys_reg(3, 1, 15, 2, 0)
+#define SYS_IMP_APL_PMCR3_EL1	sys_reg(3, 1, 15, 3, 0)
+#define SYS_IMP_APL_PMCR4_EL1	sys_reg(3, 1, 15, 4, 0)
+
+#define SYS_IMP_APL_PMESR0_EL1	sys_reg(3, 1, 15, 5, 0)
+#define PMESR0_EVT_CNT_2	GENMASK(7, 0)
+#define PMESR0_EVT_CNT_3	GENMASK(15, 8)
+#define PMESR0_EVT_CNT_4	GENMASK(23, 16)
+#define PMESR0_EVT_CNT_5	GENMASK(31, 24)
+
+#define SYS_IMP_APL_PMESR1_EL1	sys_reg(3, 1, 15, 6, 0)
+#define PMESR1_EVT_CNT_6	GENMASK(7, 0)
+#define PMESR1_EVT_CNT_7	GENMASK(15, 8)
+#define PMESR1_EVT_CNT_8	GENMASK(23, 16)
+#define PMESR1_EVT_CNT_9	GENMASK(31, 24)
+
+#define SYS_IMP_APL_PMSR_EL1	sys_reg(3, 1, 15, 13, 0)
+#define PMSR_OVERFLOW		GENMASK(9, 0)
 
 #endif /* __ASM_APPLE_M1_PMU_h */
drivers/perf/Kconfig +7 -0
index 4374af292e6d..a6af7bcb82ef 100644
@@ -139,6 +139,13 @@ config ARM_DMC620_PMU
 	  Support for PMU events monitoring on the ARM DMC-620 memory
 	  controller.
 
+config APPLE_M1_CPU_PMU
+	bool "Apple M1 CPU PMU support"
+	depends on ARM_PMU && ARCH_APPLE
+	help
+	  Provides support for the non-architectural CPU PMUs present on
+	  the Apple M1 SoCs and derivatives.
+
 source "drivers/perf/hisilicon/Kconfig"
 
 endmenu
drivers/perf/Makefile +1 -0
index 5260b116c7da..1c8cffc8c326 100644
@@ -14,3 +14,4 @@ obj-$(CONFIG_THUNDERX2_PMU) += thunderx2_pmu.o
 obj-$(CONFIG_XGENE_PMU) += xgene_pmu.o
 obj-$(CONFIG_ARM_SPE_PMU) += arm_spe_pmu.o
 obj-$(CONFIG_ARM_DMC620_PMU) += arm_dmc620_pmu.o
+obj-$(CONFIG_APPLE_M1_CPU_PMU) += apple_m1_cpu_pmu.o
drivers/perf/apple_m1_cpu_pmu.c new file +632 -0

632 lines not on this page · raw message

Signature · 1 line
2.30.2
#11 lore raw

Re: [PATCH 8/8] drivers/perf: Add Apple icestorm/firestorm CPU PMU driver

Cc'ing Dougall who has worked with the CPU performance counters extensively and might be able to shine light on the interpretations.

Quoted 750 lines · Marc Zyngier

Add a new, weird and wonderful driver for the equally weird Apple PMU HW. Although the PMU itself is functional, we don't know much about the events yet, so this can be considered as yet another random number generator...

Nonetheless, it can reliably count at least cycles and instructions in the usually wonky big-little way. For anything else, it of course supports raw event numbers.

---
 arch/arm64/include/asm/apple_m1_pmu.h |  45 ++
 drivers/perf/Kconfig                  |   7 +
 drivers/perf/Makefile                 |   1 +
 drivers/perf/apple_m1_cpu_pmu.c       | 632 ++++++++++++++++++++++++++
 4 files changed, 685 insertions(+)
 create mode 100644 drivers/perf/apple_m1_cpu_pmu.c
arch/arm64/include/asm/apple_m1_pmu.h +45 -0
index b848af7faadc..99483b19b99f 100644
@@ -6,8 +6,21 @@
 #include <linux/bits.h>
 #include <asm/sysreg.h>
 
+/* Counters */
+#define SYS_IMP_APL_PMC0_EL1	sys_reg(3, 2, 15, 0, 0)
+#define SYS_IMP_APL_PMC1_EL1	sys_reg(3, 2, 15, 1, 0)
+#define SYS_IMP_APL_PMC2_EL1	sys_reg(3, 2, 15, 2, 0)
+#define SYS_IMP_APL_PMC3_EL1	sys_reg(3, 2, 15, 3, 0)
+#define SYS_IMP_APL_PMC4_EL1	sys_reg(3, 2, 15, 4, 0)
+#define SYS_IMP_APL_PMC5_EL1	sys_reg(3, 2, 15, 5, 0)
+#define SYS_IMP_APL_PMC6_EL1	sys_reg(3, 2, 15, 6, 0)
+#define SYS_IMP_APL_PMC7_EL1	sys_reg(3, 2, 15, 7, 0)
+#define SYS_IMP_APL_PMC8_EL1	sys_reg(3, 2, 15, 9, 0)
+#define SYS_IMP_APL_PMC9_EL1	sys_reg(3, 2, 15, 10, 0)
+
 /* Core PMC control register */
 #define SYS_IMP_APL_PMCR0_EL1	sys_reg(3, 1, 15, 0, 0)
+#define PMCR0_CNT_ENABLE_0_7	GENMASK(7, 0)
 #define PMCR0_IMODE		GENMASK(10, 8)
 #define PMCR0_IMODE_OFF		0
 #define PMCR0_IMODE_PMI		1
@@ -15,5 +28,37 @@
 #define PMCR0_IMODE_HALT	3
 #define PMCR0_IMODE_FIQ		4
 #define PMCR0_IACT		BIT(11)
+#define PMCR0_PMI_ENABLE_0_7	GENMASK(19, 12)
+#define PMCR0_STOP_CNT_ON_PMI	BIT(20)
+#define PMCR0_CNT_GLOB_L2C_EVT	BIT(21)
+#define PMCR0_DEFER_PMI_TO_ERET	BIT(22)
+#define PMCR0_ALLOW_CNT_EN_EL0	BIT(30)
+#define PMCR0_CNT_ENABLE_8_9	GENMASK(33, 32)
+#define PMCR0_PMI_ENABLE_8_9	GENMASK(45, 44)
+
+#define SYS_IMP_APL_PMCR1_EL1	sys_reg(3, 1, 15, 1, 0)
+#define PMCR1_COUNT_A64_EL0_0_7	GENMASK(15, 8)
+#define PMCR1_COUNT_A64_EL1_0_7	GENMASK(23, 16)
+#define PMCR1_COUNT_A64_EL0_8_9	GENMASK(41, 40)
+#define PMCR1_COUNT_A64_EL1_8_9	GENMASK(49, 48)
+
+#define SYS_IMP_APL_PMCR2_EL1	sys_reg(3, 1, 15, 2, 0)
+#define SYS_IMP_APL_PMCR3_EL1	sys_reg(3, 1, 15, 3, 0)
+#define SYS_IMP_APL_PMCR4_EL1	sys_reg(3, 1, 15, 4, 0)
+
+#define SYS_IMP_APL_PMESR0_EL1	sys_reg(3, 1, 15, 5, 0)
+#define PMESR0_EVT_CNT_2	GENMASK(7, 0)
+#define PMESR0_EVT_CNT_3	GENMASK(15, 8)
+#define PMESR0_EVT_CNT_4	GENMASK(23, 16)
+#define PMESR0_EVT_CNT_5	GENMASK(31, 24)
+
+#define SYS_IMP_APL_PMESR1_EL1	sys_reg(3, 1, 15, 6, 0)
+#define PMESR1_EVT_CNT_6	GENMASK(7, 0)
+#define PMESR1_EVT_CNT_7	GENMASK(15, 8)
+#define PMESR1_EVT_CNT_8	GENMASK(23, 16)
+#define PMESR1_EVT_CNT_9	GENMASK(31, 24)
+
+#define SYS_IMP_APL_PMSR_EL1	sys_reg(3, 1, 15, 13, 0)
+#define PMSR_OVERFLOW		GENMASK(9, 0)
 
 #endif /* __ASM_APPLE_M1_PMU_h */
drivers/perf/Kconfig +7 -0
index 4374af292e6d..a6af7bcb82ef 100644
@@ -139,6 +139,13 @@ config ARM_DMC620_PMU
 	  Support for PMU events monitoring on the ARM DMC-620 memory
 	  controller.
 
+config APPLE_M1_CPU_PMU
+	bool "Apple M1 CPU PMU support"
+	depends on ARM_PMU && ARCH_APPLE
+	help
+	  Provides support for the non-architectural CPU PMUs present on
+	  the Apple M1 SoCs and derivatives.
+
 source "drivers/perf/hisilicon/Kconfig"
 
 endmenu
drivers/perf/Makefile +1 -0
index 5260b116c7da..1c8cffc8c326 100644
@@ -14,3 +14,4 @@ obj-$(CONFIG_THUNDERX2_PMU) += thunderx2_pmu.o
 obj-$(CONFIG_XGENE_PMU) += xgene_pmu.o
 obj-$(CONFIG_ARM_SPE_PMU) += arm_spe_pmu.o
 obj-$(CONFIG_ARM_DMC620_PMU) += arm_dmc620_pmu.o
+obj-$(CONFIG_APPLE_M1_CPU_PMU) += apple_m1_cpu_pmu.o
drivers/perf/apple_m1_cpu_pmu.c new file +632 -0
new file mode 100644
index 000000000000..bc991fc892eb
@@ -0,0 +1,632 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * CPU PMU driver for the Apple M1 and derivatives
+ *
+ * Copyright (C) 2021 Google LLC
+ *
+ * Author: Marc Zyngier <[email protected]>
+ *
+ * Most of the information used in this driver was provided by the
+ * Asahi Linux project. The rest was experimentally discovered.
+ */
+
+#include <linux/of.h>
+#include <linux/perf/arm_pmu.h>
+#include <linux/platform_device.h>
+
+#include <asm/apple_m1_pmu.h>
+#include <asm/irq_regs.h>
+#include <asm/perf_event.h>
+
+#define M1_PMU_NR_COUNTERS		10
+
+#define M1_PMU_CFG_EVENT		GENMASK(7, 0)
+
+#define ANY_BUT_0_1			GENMASK(9, 2)
+#define ONLY_2_TO_7			GENMASK(7, 2)
+#define ONLY_2_4_6			(BIT(2) | BIT(4) | BIT(6))
+#define ONLY_5_6_7			GENMASK(7, 5)
+
+/*
+ * Description of the events we actually know about, as well as those with
+ * a specific counter affinity. Yes, this is a grand total of two known
+ * counters, and the rest is anybody's guess.
+ *
+ * Not all counters can count all events. Counters #0 and #1 are wired to
+ * count cycles and instructions respectively, and some events have
+ * bizarre mappings (every other counter, or even *one* counter). These
+ * restrictins equally apply to both P and E cores.
+ *
+ * It is worth noting that the PMUs attached to P and E cores are likely
+ * to be different because the underlying uarches are different. At the
+ * moment, we don't really need to distinguish between the two because we
+ * know next to nothing about the events themselves, and we already have
+ * per cpu-type PMU abstractions.
+ *
+ * If we eventually find out that the events are different across
+ * implementations, we'll have to introduce per cpu-type tables.
+ */
+enum m1_pmu_events {
+	M1_PMU_PERFCTR_UNKNOWN_01	= 0x01,
+	M1_PMU_PERFCTR_CPU_CYCLES	= 0x02,
+	M1_PMU_PERFCTR_INSTRUCTIONS	= 0x8c,
+	M1_PMU_PERFCTR_UNKNOWN_8d	= 0x8d,
+	M1_PMU_PERFCTR_UNKNOWN_8e	= 0x8e,
+	M1_PMU_PERFCTR_UNKNOWN_8f	= 0x8f,
+	M1_PMU_PERFCTR_UNKNOWN_90	= 0x90,
+	M1_PMU_PERFCTR_UNKNOWN_93	= 0x93,
+	M1_PMU_PERFCTR_UNKNOWN_94	= 0x94,
+	M1_PMU_PERFCTR_UNKNOWN_95	= 0x95,
+	M1_PMU_PERFCTR_UNKNOWN_96	= 0x96,
+	M1_PMU_PERFCTR_UNKNOWN_97	= 0x97,
+	M1_PMU_PERFCTR_UNKNOWN_98	= 0x98,
+	M1_PMU_PERFCTR_UNKNOWN_99	= 0x99,
+	M1_PMU_PERFCTR_UNKNOWN_9a	= 0x9a,
+	M1_PMU_PERFCTR_UNKNOWN_9b	= 0x9b,
+	M1_PMU_PERFCTR_UNKNOWN_9c	= 0x9c,
+	M1_PMU_PERFCTR_UNKNOWN_9f	= 0x9f,
+	M1_PMU_PERFCTR_UNKNOWN_bf	= 0xbf,
+	M1_PMU_PERFCTR_UNKNOWN_c0	= 0xc0,
+	M1_PMU_PERFCTR_UNKNOWN_c1	= 0xc1,
+	M1_PMU_PERFCTR_UNKNOWN_c4	= 0xc4,
+	M1_PMU_PERFCTR_UNKNOWN_c5	= 0xc5,
+	M1_PMU_PERFCTR_UNKNOWN_c6	= 0xc6,
+	M1_PMU_PERFCTR_UNKNOWN_c8	= 0xc8,
+	M1_PMU_PERFCTR_UNKNOWN_ca	= 0xca,
+	M1_PMU_PERFCTR_UNKNOWN_cb	= 0xcb,
+	M1_PMU_PERFCTR_UNKNOWN_f5	= 0xf5,
+	M1_PMU_PERFCTR_UNKNOWN_f6	= 0xf6,
+	M1_PMU_PERFCTR_UNKNOWN_f7	= 0xf7,
+	M1_PMU_PERFCTR_UNKNOWN_f8	= 0xf8,
+	M1_PMU_PERFCTR_UNKNOWN_fd	= 0xfd,
+	M1_PMU_PERFCTR_LAST		= M1_PMU_CFG_EVENT,
+
+	/*
+	 * From this point onwards, these are not actual HW events,
+	 * but attributes that get stored in hw->config_base.
+	 */
+	M1_PMU_CFG_COUNT_USER		= BIT(8),
+	M1_PMU_CFG_COUNT_KERNEL		= BIT(9),
+};
+
+/*
+ * Per-event affinity table. Most events can be installed on counter
+ * 2-9, but there are a numbre of exceptions. Note that this table
+ * has been created experimentally, and I wouldn't be surprised if more
+ * counters had strange affinities.
+ */
+static const u16 m1_pmu_event_affinity[M1_PMU_PERFCTR_LAST + 1] = {
+	[0 ... M1_PMU_PERFCTR_LAST]	= ANY_BUT_0_1,
+	[M1_PMU_PERFCTR_UNKNOWN_01]	= BIT(7),
+	[M1_PMU_PERFCTR_CPU_CYCLES]	= ANY_BUT_0_1 | BIT(0),
+	[M1_PMU_PERFCTR_INSTRUCTIONS]	= BIT(7) | BIT(1),
+	[M1_PMU_PERFCTR_UNKNOWN_8d]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_8e]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_8f]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_90]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_93]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_94]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_95]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_96]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_97]	= BIT(7),
+	[M1_PMU_PERFCTR_UNKNOWN_98]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_99]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_9a]	= BIT(7),
+	[M1_PMU_PERFCTR_UNKNOWN_9b]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_9c]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_9f]	= BIT(7),
+	[M1_PMU_PERFCTR_UNKNOWN_bf]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_c0]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_c1]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_c4]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_c5]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_c6]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_c8]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_ca]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_cb]	= ONLY_5_6_7,
+	[M1_PMU_PERFCTR_UNKNOWN_f5]	= ONLY_2_4_6,
+	[M1_PMU_PERFCTR_UNKNOWN_f6]	= ONLY_2_4_6,
+	[M1_PMU_PERFCTR_UNKNOWN_f7]	= ONLY_2_4_6,
+	[M1_PMU_PERFCTR_UNKNOWN_f8]	= ONLY_2_TO_7,
+	[M1_PMU_PERFCTR_UNKNOWN_fd]	= ONLY_2_4_6,
+};
+
+static const unsigned m1_pmu_perf_map[PERF_COUNT_HW_MAX] = {
+	PERF_MAP_ALL_UNSUPPORTED,
+	[PERF_COUNT_HW_CPU_CYCLES]	= M1_PMU_PERFCTR_CPU_CYCLES,
+	[PERF_COUNT_HW_INSTRUCTIONS]	= M1_PMU_PERFCTR_INSTRUCTIONS,
+	/* No idea about the rest yet */
+};
+
+/* sysfs definitions */
+static ssize_t m1_pmu_events_sysfs_show(struct device *dev,
+					struct device_attribute *attr,
+					char *page)
+{
+	struct perf_pmu_events_attr *pmu_attr;
+
+	pmu_attr = container_of(attr, struct perf_pmu_events_attr, attr);
+
+	return sprintf(page, "event=0x%04llx\n", pmu_attr->id);
+}
+
+#define M1_PMU_EVENT_ATTR(name, config)					\
+	PMU_EVENT_ATTR_ID(name, m1_pmu_events_sysfs_show, config)
+
+static struct attribute *m1_pmu_event_attrs[] = {
+	M1_PMU_EVENT_ATTR(cycles, M1_PMU_PERFCTR_CPU_CYCLES),
+	M1_PMU_EVENT_ATTR(instructions, M1_PMU_PERFCTR_INSTRUCTIONS),
+	NULL,
+};
+
+static const struct attribute_group m1_pmu_events_attr_group = {
+	.name = "events",
+	.attrs = m1_pmu_event_attrs,
+};
+
+PMU_FORMAT_ATTR(event, "config:0-7");
+
+static struct attribute *m1_pmu_format_attrs[] = {
+	&format_attr_event.attr,
+	NULL,
+};
+
+static const struct attribute_group m1_pmu_format_attr_group = {
+	.name = "format",
+	.attrs = m1_pmu_format_attrs,
+};
+
+/* Low level accessors. No synchronisation. */
+#define PMU_READ_COUNTER(_idx)						\
+	case _idx:	return read_sysreg_s(SYS_IMP_APL_PMC## _idx ##_EL1)
+
+#define PMU_WRITE_COUNTER(_val, _idx)					\
+	case _idx:							\
+		write_sysreg_s(_val, SYS_IMP_APL_PMC## _idx ##_EL1);	\
+		return
+
+static u64 m1_pmu_read_hw_counter(unsigned int index)
+{
+	switch (index) {
+		PMU_READ_COUNTER(0);
+		PMU_READ_COUNTER(1);
+		PMU_READ_COUNTER(2);
+		PMU_READ_COUNTER(3);
+		PMU_READ_COUNTER(4);
+		PMU_READ_COUNTER(5);
+		PMU_READ_COUNTER(6);
+		PMU_READ_COUNTER(7);
+		PMU_READ_COUNTER(8);
+		PMU_READ_COUNTER(9);
+	}
+
+	BUG();
+}
+
+static void m1_pmu_write_hw_counter(u64 val, unsigned int index)
+{
+	switch (index) {
+		PMU_WRITE_COUNTER(val, 0);
+		PMU_WRITE_COUNTER(val, 1);
+		PMU_WRITE_COUNTER(val, 2);
+		PMU_WRITE_COUNTER(val, 3);
+		PMU_WRITE_COUNTER(val, 4);
+		PMU_WRITE_COUNTER(val, 5);
+		PMU_WRITE_COUNTER(val, 6);
+		PMU_WRITE_COUNTER(val, 7);
+		PMU_WRITE_COUNTER(val, 8);
+		PMU_WRITE_COUNTER(val, 9);
+	}
+
+	BUG();
+}
+
+#define get_bit_offset(index, mask)	(__ffs(mask) + (index))
+
+static void __m1_pmu_enable_counter(unsigned int index, bool en)
+{
+	u64 val, bit;
+
+	switch (index) {
+	case 0 ... 7:
+		bit = BIT(get_bit_offset(index, PMCR0_CNT_ENABLE_0_7));
+		break;
+	case 8 ... 9:
+		bit = BIT(get_bit_offset(index - 8, PMCR0_CNT_ENABLE_8_9));
+		break;
+	default:
+		BUG();
+	}
+
+	val = read_sysreg_s(SYS_IMP_APL_PMCR0_EL1);
+
+	if (en)
+		val |= bit;
+	else
+		val &= ~bit;
+
+	write_sysreg_s(val, SYS_IMP_APL_PMCR0_EL1);
+}
+
+static void m1_pmu_enable_counter(unsigned int index)
+{
+	__m1_pmu_enable_counter(index, true);
+}
+
+static void m1_pmu_disable_counter(unsigned int index)
+{
+	__m1_pmu_enable_counter(index, false);
+}
+
+static void __m1_pmu_enable_counter_interrupt(unsigned int index, bool en)
+{
+	u64 val, bit;
+
+	switch (index) {
+	case 0 ... 7:
+		bit = BIT(get_bit_offset(index, PMCR0_PMI_ENABLE_0_7));
+		break;
+	case 8 ... 9:
+		bit = BIT(get_bit_offset(index - 8, PMCR0_PMI_ENABLE_8_9));
+		break;
+	default:
+		BUG();
+	}
+
+	val = read_sysreg_s(SYS_IMP_APL_PMCR0_EL1);
+
+	if (en)
+		val |= bit;
+	else
+		val &= ~bit;
+
+	write_sysreg_s(val, SYS_IMP_APL_PMCR0_EL1);
+}
+
+static void m1_pmu_enable_counter_interrupt(unsigned int index)
+{
+	__m1_pmu_enable_counter_interrupt(index, true);
+}
+
+static void m1_pmu_disable_counter_interrupt(unsigned int index)
+{
+	__m1_pmu_enable_counter_interrupt(index, false);
+}
+
+static void m1_pmu_configure_counter(unsigned int index, u8 event,
+				     bool user, bool kernel)
+{
+	u64 val, user_bit, kernel_bit;
+	int shift;
+
+	switch (index) {
+	case 0 ... 7:
+		user_bit = BIT(get_bit_offset(index, PMCR1_COUNT_A64_EL0_0_7));
+		kernel_bit = BIT(get_bit_offset(index, PMCR1_COUNT_A64_EL1_0_7));
+		break;
+	case 8 ... 9:
+		user_bit = BIT(get_bit_offset(index - 8, PMCR1_COUNT_A64_EL0_8_9));
+		kernel_bit = BIT(get_bit_offset(index - 8, PMCR1_COUNT_A64_EL1_8_9));
+		break;
+	default:
+		BUG();
+	}
+
+	val = read_sysreg_s(SYS_IMP_APL_PMCR1_EL1);
+
+	if (user)
+		val |= user_bit;
+	else
+		val &= ~user_bit;
+
+	if (kernel)
+		val |= kernel_bit;
+	else
+		val &= ~kernel_bit;
+
+	write_sysreg_s(val, SYS_IMP_APL_PMCR1_EL1);
+
+	switch (index) {
+	case 0 ... 1:
+		/* 0 and 1 have fixed events */
+		break;
+	case 2 ... 5:
+		shift = (index - 2) * 8;
+		val = read_sysreg_s(SYS_IMP_APL_PMESR0_EL1);
+		val &= ~((u64)0xff << shift);
+		val |= (u64)event << shift;
+		write_sysreg_s(val, SYS_IMP_APL_PMESR0_EL1);
+		break;
+	case 6 ... 9:
+		shift = (index - 6) * 8;
+		val = read_sysreg_s(SYS_IMP_APL_PMESR1_EL1);
+		val &= ~((u64)0xff << shift);
+		val |= (u64)event << shift;
+		write_sysreg_s(val, SYS_IMP_APL_PMESR1_EL1);
+		break;
+	}
+}
+
+/* arm_pmu backend */
+static void m1_pmu_enable_event(struct perf_event *event)
+{
+	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
+	struct pmu_hw_events *cpuc = this_cpu_ptr(cpu_pmu->hw_events);
+	unsigned long flags;
+	bool user, kernel;
+	u8 evt;
+
+	evt = event->hw.config_base & M1_PMU_CFG_EVENT;
+	user = event->hw.config_base & M1_PMU_CFG_COUNT_USER;
+	kernel = event->hw.config_base & M1_PMU_CFG_COUNT_KERNEL;
+
+	raw_spin_lock_irqsave(&cpuc->pmu_lock, flags);
+
+	m1_pmu_disable_counter_interrupt(event->hw.idx);
+	m1_pmu_disable_counter(event->hw.idx);
+	isb();
+
+	m1_pmu_configure_counter(event->hw.idx, evt, user, kernel);
+	m1_pmu_enable_counter(event->hw.idx);
+	m1_pmu_enable_counter_interrupt(event->hw.idx);
+	isb();
+
+	raw_spin_unlock_irqrestore(&cpuc->pmu_lock, flags);
+}
+
+static void __m1_pmu_disable_event(struct perf_event *event)
+{
+	m1_pmu_disable_counter_interrupt(event->hw.idx);
+	m1_pmu_disable_counter(event->hw.idx);
+	isb();
+}
+
+static void m1_pmu_disable_event(struct perf_event *event)
+{
+	struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
+	struct pmu_hw_events *cpuc = this_cpu_ptr(cpu_pmu->hw_events);
+	unsigned long flags;
+
+	raw_spin_lock_irqsave(&cpuc->pmu_lock, flags);
+
+	__m1_pmu_disable_event(event);
+
+	raw_spin_unlock_irqrestore(&cpuc->pmu_lock, flags);
+}
+
+static irqreturn_t m1_pmu_handle_irq(struct arm_pmu *cpu_pmu)
+{
+	struct pmu_hw_events *cpuc = this_cpu_ptr(cpu_pmu->hw_events);
+	irqreturn_t ret = IRQ_HANDLED;
+	struct pt_regs *regs;
+	u64 overflow, state;
+	unsigned long flags;
+	int idx;
+
+	raw_spin_lock_irqsave(&cpuc->pmu_lock, flags);
+	state = read_sysreg_s(SYS_IMP_APL_PMCR0_EL1);
+	overflow = read_sysreg_s(SYS_IMP_APL_PMSR_EL1);
+	if (!overflow) {
+		ret = IRQ_NONE;
+		goto out;
+	}
+
+	regs = get_irq_regs();
+
+	for (idx = 0; idx < cpu_pmu->num_events; idx++) {
+		struct perf_event *event = cpuc->events[idx];
+		struct perf_sample_data data;
+
+		if (!event)
+			continue;
+
+		armpmu_event_update(event);
+		perf_sample_data_init(&data, 0, event->hw.last_period);
+		if (!armpmu_event_set_period(event))
+			continue;
+
+		if (perf_event_overflow(event, &data, regs))
+			__m1_pmu_disable_event(event);
+	}
+
+out:
+	state &= ~PMCR0_IACT;
+	write_sysreg_s(state, SYS_IMP_APL_PMCR0_EL1);
+	isb();
+
+	raw_spin_unlock_irqrestore(&cpuc->pmu_lock, flags);
+
+	return ret;
+}
+
+static u64 m1_pmu_read_counter(struct perf_event *event)
+{
+	return m1_pmu_read_hw_counter(event->hw.idx);
+}
+
+static void m1_pmu_write_counter(struct perf_event *event, u64 value)
+{
+	m1_pmu_write_hw_counter(value, event->hw.idx);
+	isb();
+}
+
+static int m1_pmu_get_event_idx(struct pmu_hw_events *cpuc,
+				struct perf_event *event)
+{
+	unsigned long evtype = event->hw.config_base & M1_PMU_CFG_EVENT;
+	unsigned long affinity = m1_pmu_event_affinity[evtype];
+	int idx;
+
+	/*
+	 * Place the event on the first free counter that can count
+	 * this event.
+	 *
+	 * We could do a better job if we had a view of all the events
+	 * counting on the PMU at any given time, and by placing the
+	 * most constraint events first.
+	 */
+	for_each_set_bit(idx, &affinity, M1_PMU_NR_COUNTERS) {
+		if (!test_and_set_bit(idx, cpuc->used_mask))
+			return idx;
+	}
+
+	return -EAGAIN;
+}
+
+static void m1_pmu_clear_event_idx(struct pmu_hw_events *cpuc,
+				   struct perf_event *event)
+{
+	clear_bit(event->hw.idx, cpuc->used_mask);
+}
+
+static void m1_pmu_start(struct arm_pmu *cpu_pmu)
+{
+	struct pmu_hw_events *cpuc = this_cpu_ptr(cpu_pmu->hw_events);
+	unsigned long flags;
+	u64 val;
+
+	raw_spin_lock_irqsave(&cpuc->pmu_lock, flags);
+
+	val = read_sysreg_s(SYS_IMP_APL_PMCR0_EL1);
+	val &= ~PMCR0_IMODE;
+	val |= FIELD_PREP(PMCR0_IMODE, PMCR0_IMODE_FIQ);
+	val |= PMCR0_STOP_CNT_ON_PMI;
+
+	write_sysreg_s(val, SYS_IMP_APL_PMCR0_EL1);
+	isb();
+
+	raw_spin_unlock_irqrestore(&cpuc->pmu_lock, flags);
+}
+
+static void __m1_pmu_stop(void)
+{
+	u64 val;
+
+	val = read_sysreg_s(SYS_IMP_APL_PMCR0_EL1);
+	val &= ~PMCR0_IMODE;
+	val |= FIELD_PREP(PMCR0_IMODE, PMCR0_IMODE_OFF);
+	write_sysreg_s(val, SYS_IMP_APL_PMCR0_EL1);
+	isb();
+}
+
+static void m1_pmu_stop(struct arm_pmu *cpu_pmu)
+{
+	struct pmu_hw_events *cpuc = this_cpu_ptr(cpu_pmu->hw_events);
+	unsigned long flags;
+
+	raw_spin_lock_irqsave(&cpuc->pmu_lock, flags);
+
+	__m1_pmu_stop();
+
+	raw_spin_unlock_irqrestore(&cpuc->pmu_lock, flags);
+}
+
+static int m1_pmu_map_event(struct perf_event *event)
+{
+	/*
+	 * Although the counters are 48bit wide, bit 47 is what
+	 * triggers the overflow interrupt. Advertise the counters
+	 * being 47bit wide to mimick the behaviour of the ARM PMU.
+	 */
+	event->hw.flags |= ARMPMU_EVT_47BIT;
+	return armpmu_map_event(event, &m1_pmu_perf_map, NULL, M1_PMU_CFG_EVENT);
+}
+
+static void m1_pmu_reset(void *info)
+{
+	int i;
+
+	__m1_pmu_stop();
+
+	for (i = 0; i < M1_PMU_NR_COUNTERS; i++) {
+		m1_pmu_disable_counter(i);
+		m1_pmu_disable_counter_interrupt(i);
+		m1_pmu_write_hw_counter(0, i);
+	}
+
+	isb();
+}
+
+static int m1_pmu_set_event_filter(struct hw_perf_event *event,
+				   struct perf_event_attr *attr)
+{
+	unsigned long config_base = 0;
+
+	if (!attr->exclude_kernel)
+		config_base |= M1_PMU_CFG_COUNT_KERNEL;
+	if (!attr->exclude_user)
+		config_base |= M1_PMU_CFG_COUNT_USER;
+
+	event->config_base = config_base;
+
+	return 0;
+}
+
+static int m1_pmu_init(struct arm_pmu *cpu_pmu)
+{
+	cpu_pmu->handle_irq	  = m1_pmu_handle_irq;
+	cpu_pmu->enable		  = m1_pmu_enable_event;
+	cpu_pmu->disable	  = m1_pmu_disable_event;
+	cpu_pmu->read_counter	  = m1_pmu_read_counter;
+	cpu_pmu->write_counter	  = m1_pmu_write_counter;
+	cpu_pmu->get_event_idx	  = m1_pmu_get_event_idx;
+	cpu_pmu->clear_event_idx  = m1_pmu_clear_event_idx;
+	cpu_pmu->start		  = m1_pmu_start;
+	cpu_pmu->stop		  = m1_pmu_stop;
+	cpu_pmu->map_event	  = m1_pmu_map_event;
+	cpu_pmu->reset		  = m1_pmu_reset;
+	cpu_pmu->set_event_filter = m1_pmu_set_event_filter;
+
+	cpu_pmu->num_events	  = M1_PMU_NR_COUNTERS;
+	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_EVENTS] = &m1_pmu_events_attr_group;
+	cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] = &m1_pmu_format_attr_group;
+	return 0;
+}
+
+/* Device driver gunk */
+static int m1_pmu_ice_init(struct arm_pmu *cpu_pmu)
+{
+	cpu_pmu->name = "apple_icestorm_pmu";
+	return m1_pmu_init(cpu_pmu);
+}
+
+static int m1_pmu_fire_init(struct arm_pmu *cpu_pmu)
+{
+	cpu_pmu->name = "apple_firestorm_pmu";
+	return m1_pmu_init(cpu_pmu);
+}
+
+static const struct of_device_id m1_pmu_of_device_ids[] = {
+	{ .compatible = "apple,icestorm-pmu",	.data = m1_pmu_ice_init, },
+	{ .compatible = "apple,firestorm-pmu",	.data = m1_pmu_fire_init, },
+	{ },
+};
+MODULE_DEVICE_TABLE(of, m1_pmu_of_device_ids);
+
+static int m1_pmu_device_probe(struct platform_device *pdev)
+{
+	int ret;
+
+	ret = arm_pmu_device_probe(pdev, m1_pmu_of_device_ids, NULL);
+	if (!ret) {
+		/*
+		 * If probe succeeds, taint the kernel as this is all
+		 * undocumented, implementation defined black magic.
+		 */
+		add_taint(TAINT_CPU_OUT_OF_SPEC, LOCKDEP_STILL_OK);
+	}
+
+	return ret;
+}
+
+static struct platform_driver m1_pmu_driver = {
+	.driver		= {
+		.name			= "apple-m1-cpu-pmu",
+		.of_match_table		= m1_pmu_of_device_ids,
+		.suppress_bind_attrs	= true,
+	},
+	.probe		= m1_pmu_device_probe,
+};
+
+module_platform_driver(m1_pmu_driver);
+MODULE_LICENSE("GPL v2");
Signature · 1 line
2.30.2
#12 lore raw

Re: [PATCH 8/8] drivers/perf: Add Apple icestorm/firestorm CPU PMU driver

Apple distributes names (and descriptions and affinity masks) for 55 of the events with macOS in the file /usr/share/kpep/a14.plist (exposed to users in Instruments.app). Many of those 55 events were added in macOS 12, so it's good to check the latest version. I use the command "plutil -convert json -o - /usr/share/kpep/a14.plist" to get these as JSON.

There are many more events that I have discovered experimentally, but this work is unusually hard to verify, so I'd be inclined to stick with what's documented.

However, I have observed a few oddities that might be of interest.

The counter 0x9B (INST_LDST) works on PMCs 5, 6 and 7, but gives different results for paired AMX instructions on PMC 7 (7 counts instructions, while 5 and 6 count pairs as one). Apple addresses this by restricting the affinity mask to PMC 7. This is also seen on undocumented counter 0x96, which counts integer stores. (For context, microarchitecturally non-load-store AMX operations appear as stores, as they just need to be posted to the AMX coprocessor on commit. Consecutive non-load-store AMX operations can be paired (fused), such that they issue as one uop, which is where this anomaly can be seen.)

Undocumented counters 0xF1 through 0xFF appear to be operation counters, meaning their result depends on events selected on other counters. There are three threshold registers (PMTRHLD2, PMTRHLD4, PMTRHLD6) which can specify a threshold (in number of cycles) for the operation counter on the PMC with the same number. There is also a mapping register (PMMAP), which contains a 3-bit field for each counter from PMC2 to PMC7, each specifying a PMC index which can be used as an input to the operation. Binary operations only use PMC2/4/6 and use PMC(n+1) as their other input. These operation counters may also behave differently depending on the value currently in the corresponding PMC (specifically counters F9/FA which implement shortest/longest run of non-zero counts).

This is complicated, and it's not exposed to the user by macOS, so I wouldn't worry about supporting it for now. Despite all this, the events and features on the P and E cores seem to be the same, so I don't expect a need to distinguish between them in the future.

(I've been meaning to write all this up properly, but haven't got around to it, sorry!)

Dougall

Quoted 3 lines · Alyssa Rosenzweig

Cc'ing Dougall who has worked with the CPU performance counters extensively and might be able to shine light on the interpretations.

#13 lore raw

Re: [PATCH 8/8] drivers/perf: Add Apple icestorm/firestorm CPU PMU driver

Quoted 7 lines · Dougall

Apple distributes names (and descriptions and affinity masks) for 55 of the events with macOS in the file /usr/share/kpep/a14.plist (exposed to users in Instruments.app). Many of those 55 events were added in macOS 12, so it's good to check the latest version. I use the command "plutil -convert json -o - /usr/share/kpep/a14.plist" to get these as JSON.

As it appears, the perf tool can ingest an event description from a json file, and none of it has to be in the kernel itself.

So if someone was to provide a tool to convert the macOS file into something that perf can understand, it would be great, and wouldn't require any distribution of otherwise tainted material (distribute the tool, and not the data).

Quoted 7 lines

There are many more events that I have discovered experimentally, but this work is unusually hard to verify, so I'd be inclined to stick with what's documented.

However, I have observed a few oddities that might be of interest.

Quoted 10 lines

The counter 0x9B (INST_LDST) works on PMCs 5, 6 and 7, but gives different results for paired AMX instructions on PMC 7 (7 counts instructions, while 5 and 6 count pairs as one). Apple addresses this by restricting the affinity mask to PMC 7. This is also seen on undocumented counter 0x96, which counts integer stores. (For context, microarchitecturally non-load-store AMX operations appear as stores, as they just need to be posted to the AMX coprocessor on commit. Consecutive non-load-store AMX operations can be paired (fused), such that they issue as one uop, which is where this anomaly can be seen.)

Interesting. I guess we're unlikely to see any AMX support anytime soon on Linux, unless we can make it fit the architected SME model (and even that would be pretty controversial).

Quoted 4 lines

Undocumented counters 0xF1 through 0xFF appear to be operation counters, meaning their result depends on events selected on other counters. There are three threshold registers (PMTRHLD2, PMTRHLD4, PMTRHLD6) which can specify a threshold (in number of cycles) for

Quoted 8 lines

the operation counter on the PMC with the same number. There is also a mapping register (PMMAP), which contains a 3-bit field for each counter from PMC2 to PMC7, each specifying a PMC index which can be used as an input to the operation. Binary operations only use PMC2/4/6 and use PMC(n+1) as their other input. These operation counters may also behave differently depending on the value currently in the corresponding PMC (specifically counters F9/FA which implement shortest/longest run of non-zero counts).

Weeee... I'm sure there are super interesting uses for this, but I'd rather have something simple for a start. Thanks for the heads up though, this is extremely interesting!

Quoted 2 lines

This is complicated, and it's not exposed to the user by macOS, so I wouldn't worry about supporting it for now.

We're in strong agreement here.

Quoted 3 lines

Despite all this, the
events and features on the P and E cores seem to be the same, so I don't expect a need to distinguish between them in the future.

That'd be the first big-little implementation to have consistent events across the board. Amazing! :D

Quoted 2 lines

(I've been meaning to write all this up properly, but haven't got around to it, sorry!)

No worries, and thanks for taking the time to write this email!

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#14 lore raw

Re: [PATCH 8/8] drivers/perf: Add Apple icestorm/firestorm CPU PMU driver

In reply to Marc Zyngier

arch/arm64/include/asm/apple_m1_pmu.h line 17

Quoted 9 lines
arch/arm64/include/asm/apple_m1_pmu.h +9 -0
+/* Counters */
+#define SYS_IMP_APL_PMC0_EL1	sys_reg(3, 2, 15, 0, 0)
+#define SYS_IMP_APL_PMC1_EL1	sys_reg(3, 2, 15, 1, 0)
+#define SYS_IMP_APL_PMC2_EL1	sys_reg(3, 2, 15, 2, 0)
+#define SYS_IMP_APL_PMC3_EL1	sys_reg(3, 2, 15, 3, 0)
+#define SYS_IMP_APL_PMC4_EL1	sys_reg(3, 2, 15, 4, 0)
+#define SYS_IMP_APL_PMC5_EL1	sys_reg(3, 2, 15, 5, 0)
+#define SYS_IMP_APL_PMC6_EL1	sys_reg(3, 2, 15, 6, 0)
+#define SYS_IMP_APL_PMC7_EL1	sys_reg(3, 2, 15, 7, 0)

--gap--

arch/arm64/include/asm/apple_m1_pmu.h line 19

Quoted 2 lines
arch/arm64/include/asm/apple_m1_pmu.h +2 -0
+#define SYS_IMP_APL_PMC8_EL1	sys_reg(3, 2, 15, 9, 0)
+#define SYS_IMP_APL_PMC9_EL1	sys_reg(3, 2, 15, 10, 0)

Do we know what the gap is?

Quoted 9 lines

+/*
+ * Description of the events we actually know about, as well as those with
+ * a specific counter affinity. Yes, this is a grand total of two known
+ * counters, and the rest is anybody's guess.
+ *
+ * Not all counters can count all events. Counters #0 and #1 are wired to
+ * count cycles and instructions respectively, and some events have
+ * bizarre mappings (every other counter, or even *one* counter). These
+ * restrictins equally apply to both P and E cores.

restrictions

Quoted 34 lines
drivers/perf/apple_m1_cpu_pmu.c new file +34 -0
+/* Low level accessors. No synchronisation. */
+#define PMU_READ_COUNTER(_idx)						\
+	case _idx:	return read_sysreg_s(SYS_IMP_APL_PMC## _idx ##_EL1)
+
+#define PMU_WRITE_COUNTER(_val, _idx)					\
+	case _idx:							\
+		write_sysreg_s(_val, SYS_IMP_APL_PMC## _idx ##_EL1);	\
+		return
+
+static u64 m1_pmu_read_hw_counter(unsigned int index)
+{
+	switch (index) {
+		PMU_READ_COUNTER(0);
+		PMU_READ_COUNTER(1);
+		PMU_READ_COUNTER(2);
+		PMU_READ_COUNTER(3);
+		PMU_READ_COUNTER(4);
+		PMU_READ_COUNTER(5);
+		PMU_READ_COUNTER(6);
+		PMU_READ_COUNTER(7);
+		PMU_READ_COUNTER(8);
+		PMU_READ_COUNTER(9);
+	}
+
+	BUG();
+}
+
+static void m1_pmu_write_hw_counter(u64 val, unsigned int index)
+{
+	switch (index) {
+		PMU_WRITE_COUNTER(val, 0);
+		PMU_WRITE_COUNTER(val, 1);
+		PMU_WRITE_COUNTER(val, 2);
+		PMU_WRITE_COUNTER(val, 3);

drivers/perf/apple_m1_cpu_pmu.c line 222

Quoted 10 lines
drivers/perf/apple_m1_cpu_pmu.c new file +10 -0
+		PMU_WRITE_COUNTER(val, 4);
+		PMU_WRITE_COUNTER(val, 5);
+		PMU_WRITE_COUNTER(val, 6);
+		PMU_WRITE_COUNTER(val, 7);
+		PMU_WRITE_COUNTER(val, 8);
+		PMU_WRITE_COUNTER(val, 9);
+	}
+
+	BUG();
+}

Probbaly cleaner to use a single switch and no macros, registers become greppable and the code is shorter too. Caveat: didn't check if it compiles.

	static inline u64 m1_pmu_hw_counter(unsigned int index)
	{
		switch (index) {
		case 0: return SYS_IMP_APL_PMC0_EL1;
		case 1: return SYS_IMP_APL_PMC1_EL1;
		case 2: return SYS_IMP_APL_PMC2_EL1;
		case 3: return SYS_IMP_APL_PMC3_EL1;
		case 4: return SYS_IMP_APL_PMC4_EL1;
		case 5: return SYS_IMP_APL_PMC5_EL1;
		case 6: return SYS_IMP_APL_PMC6_EL1;
		case 7: return SYS_IMP_APL_PMC7_EL1;
		case 8: return SYS_IMP_APL_PMC8_EL1;
		case 9: return SYS_IMP_APL_PMC9_EL1;
		}

		BUG();
	}

	static u64 m1_pmu_read_hw_counter(unsigned int index) {
		return read_sysreg_s(m1_pmu_hw_counter(index));
	}


	static void m1_pmu_write_hw_counter(u64 val, unsigned int index)
	{
		write_sysreg_s(val, m1_pmu_hw_counter(index));
	}
Quoted 24 lines
drivers/perf/apple_m1_cpu_pmu.c new file +24 -0
+static void __m1_pmu_enable_counter(unsigned int index, bool en)
+{
+	u64 val, bit;
+
+	switch (index) {
+	case 0 ... 7:
+		bit = BIT(get_bit_offset(index, PMCR0_CNT_ENABLE_0_7));
+		break;
+	case 8 ... 9:
+		bit = BIT(get_bit_offset(index - 8, PMCR0_CNT_ENABLE_8_9));
+		break;
+	default:
+		BUG();
+	}
+
+	val = read_sysreg_s(SYS_IMP_APL_PMCR0_EL1);
+
+	if (en)
+		val |= bit;
+	else
+		val &= ~bit;
+
+	write_sysreg_s(val, SYS_IMP_APL_PMCR0_EL1);
+}

...

Quoted 14 lines
drivers/perf/apple_m1_cpu_pmu.c new file +14 -0
+static void __m1_pmu_enable_counter_interrupt(unsigned int index, bool en)
+{
+	u64 val, bit;
+
+	switch (index) {
+	case 0 ... 7:
+		bit = BIT(get_bit_offset(index, PMCR0_PMI_ENABLE_0_7));
+		break;
+	case 8 ... 9:
+		bit = BIT(get_bit_offset(index - 8, PMCR0_PMI_ENABLE_8_9));
+		break;
+	default:
+		BUG();
+	}

drivers/perf/apple_m1_cpu_pmu.c line 284

Quoted 10 lines
drivers/perf/apple_m1_cpu_pmu.c new file +10 -0
+
+	val = read_sysreg_s(SYS_IMP_APL_PMCR0_EL1);
+
+	if (en)
+		val |= bit;
+	else
+		val &= ~bit;
+
+	write_sysreg_s(val, SYS_IMP_APL_PMCR0_EL1);
+}

These two helper functions have basically the same logic -- maybe worth combining?

Quoted 3 lines
drivers/perf/apple_m1_cpu_pmu.c new file +3 -0
+static void m1_pmu_configure_counter(unsigned int index, u8 event,
+				     bool user, bool kernel)
+{

....

Quoted 10 lines
drivers/perf/apple_m1_cpu_pmu.c new file +10 -0
+	switch (index) {
+	case 0 ... 1:
+		/* 0 and 1 have fixed events */
+		break;
+	case 2 ... 5:
+		shift = (index - 2) * 8;
+		val = read_sysreg_s(SYS_IMP_APL_PMESR0_EL1);
+		val &= ~((u64)0xff << shift);
+		val |= (u64)event << shift;
+		write_sysreg_s(val, SYS_IMP_APL_PMESR0_EL1);

drivers/perf/apple_m1_cpu_pmu.c line 348

Quoted 10 lines
drivers/perf/apple_m1_cpu_pmu.c new file +10 -0
+		break;
+	case 6 ... 9:
+		shift = (index - 6) * 8;
+		val = read_sysreg_s(SYS_IMP_APL_PMESR1_EL1);
+		val &= ~((u64)0xff << shift);
+		val |= (u64)event << shift;
+		write_sysreg_s(val, SYS_IMP_APL_PMESR1_EL1);
+		break;
+	}
+}

I'd love an explanation what's happening here.

drivers/perf/apple_m1_cpu_pmu.c line 467

Quoted 8 lines
drivers/perf/apple_m1_cpu_pmu.c new file +8 -0
+	/*
+	 * Place the event on the first free counter that can count
+	 * this event.
+	 *
+	 * We could do a better job if we had a view of all the events
+	 * counting on the PMU at any given time, and by placing the
+	 * most constraint events first.
+	 */

constraining

Quoted 5 lines
drivers/perf/apple_m1_cpu_pmu.c new file +5 -0
+static int m1_pmu_device_probe(struct platform_device *pdev)
+{
+	int ret;
+
+	ret = arm_pmu_device_probe(pdev, m1_pmu_of_device_ids, NULL);

drivers/perf/apple_m1_cpu_pmu.c line 620

Quoted 10 lines
drivers/perf/apple_m1_cpu_pmu.c new file +10 -0
+	if (!ret) {
+		/*
+		 * If probe succeeds, taint the kernel as this is all
+		 * undocumented, implementation defined black magic.
+		 */
+		add_taint(TAINT_CPU_OUT_OF_SPEC, LOCKDEP_STILL_OK);
+	}
+
+	return ret;
+}

What are the implications of this taint? You could say that about every driver we've written for the M1, but...

#15 lore raw

Re: [PATCH 8/8] drivers/perf: Add Apple icestorm/firestorm CPU PMU driver

Quoted 11 lines · Alyssa Rosenzweig

Review

Quoted 10 lines
(quoted diff) +9 -0
+/* Counters */
+#define SYS_IMP_APL_PMC0_EL1	sys_reg(3, 2, 15, 0, 0)
+#define SYS_IMP_APL_PMC1_EL1	sys_reg(3, 2, 15, 1, 0)
+#define SYS_IMP_APL_PMC2_EL1	sys_reg(3, 2, 15, 2, 0)
+#define SYS_IMP_APL_PMC3_EL1	sys_reg(3, 2, 15, 3, 0)
+#define SYS_IMP_APL_PMC4_EL1	sys_reg(3, 2, 15, 4, 0)
+#define SYS_IMP_APL_PMC5_EL1	sys_reg(3, 2, 15, 5, 0)
+#define SYS_IMP_APL_PMC6_EL1	sys_reg(3, 2, 15, 6, 0)
+#define SYS_IMP_APL_PMC7_EL1	sys_reg(3, 2, 15, 7, 0)

--gap--

Quoted 4 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+#define SYS_IMP_APL_PMC8_EL1	sys_reg(3, 2, 15, 9, 0)
+#define SYS_IMP_APL_PMC9_EL1	sys_reg(3, 2, 15, 10, 0)

Do we know what the gap is?

If it exists, it is an IMPDEF register.

Quoted 53 lines
Quoted 10 lines

+/*
+ * Description of the events we actually know about, as well as those with
+ * a specific counter affinity. Yes, this is a grand total of two known
+ * counters, and the rest is anybody's guess.
+ *
+ * Not all counters can count all events. Counters #0 and #1 are wired to
+ * count cycles and instructions respectively, and some events have
+ * bizarre mappings (every other counter, or even *one* counter). These
+ * restrictins equally apply to both P and E cores.

restrictions

Quoted 40 lines
(quoted diff) +40 -0
+/* Low level accessors. No synchronisation. */
+#define PMU_READ_COUNTER(_idx)						\
+	case _idx:	return read_sysreg_s(SYS_IMP_APL_PMC## _idx ##_EL1)
+
+#define PMU_WRITE_COUNTER(_val, _idx)					\
+	case _idx:							\
+		write_sysreg_s(_val, SYS_IMP_APL_PMC## _idx ##_EL1);	\
+		return
+
+static u64 m1_pmu_read_hw_counter(unsigned int index)
+{
+	switch (index) {
+		PMU_READ_COUNTER(0);
+		PMU_READ_COUNTER(1);
+		PMU_READ_COUNTER(2);
+		PMU_READ_COUNTER(3);
+		PMU_READ_COUNTER(4);
+		PMU_READ_COUNTER(5);
+		PMU_READ_COUNTER(6);
+		PMU_READ_COUNTER(7);
+		PMU_READ_COUNTER(8);
+		PMU_READ_COUNTER(9);
+	}
+
+	BUG();
+}
+
+static void m1_pmu_write_hw_counter(u64 val, unsigned int index)
+{
+	switch (index) {
+		PMU_WRITE_COUNTER(val, 0);
+		PMU_WRITE_COUNTER(val, 1);
+		PMU_WRITE_COUNTER(val, 2);
+		PMU_WRITE_COUNTER(val, 3);
+		PMU_WRITE_COUNTER(val, 4);
+		PMU_WRITE_COUNTER(val, 5);
+		PMU_WRITE_COUNTER(val, 6);
+		PMU_WRITE_COUNTER(val, 7);
+		PMU_WRITE_COUNTER(val, 8);
+		PMU_WRITE_COUNTER(val, 9);
Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+	}
+
+	BUG();
+}

Probbaly cleaner to use a single switch and no macros, registers become greppable and the code is shorter too. Caveat: didn't check if it compiles.

I can tell you haven't!

Quoted 13 lines
	static inline u64 m1_pmu_hw_counter(unsigned int index)
	{
		switch (index) {
		case 0: return SYS_IMP_APL_PMC0_EL1;
		case 1: return SYS_IMP_APL_PMC1_EL1;
		case 2: return SYS_IMP_APL_PMC2_EL1;
		case 3: return SYS_IMP_APL_PMC3_EL1;
		case 4: return SYS_IMP_APL_PMC4_EL1;
		case 5: return SYS_IMP_APL_PMC5_EL1;
		case 6: return SYS_IMP_APL_PMC6_EL1;
		case 7: return SYS_IMP_APL_PMC7_EL1;
		case 8: return SYS_IMP_APL_PMC8_EL1;
Quoted 8 lines
		case 9: return SYS_IMP_APL_PMC9_EL1;
		}

		BUG();
	}

	static u64 m1_pmu_read_hw_counter(unsigned int index) {
		return read_sysreg_s(m1_pmu_hw_counter(index));

read/write_sysreg_s() cannot take a variable as a parameter. They rely on direct macro expansion to generate the inline asm. See the various examples all over the code base where we have similar constructs for this exact reason.

Quoted 52 lines
	}


	static void m1_pmu_write_hw_counter(u64 val, unsigned int index)
	{
		write_sysreg_s(val, m1_pmu_hw_counter(index));
	}
Quoted 24 lines
(quoted diff) +24 -0
+static void __m1_pmu_enable_counter(unsigned int index, bool en)
+{
+	u64 val, bit;
+
+	switch (index) {
+	case 0 ... 7:
+		bit = BIT(get_bit_offset(index, PMCR0_CNT_ENABLE_0_7));
+		break;
+	case 8 ... 9:
+		bit = BIT(get_bit_offset(index - 8, PMCR0_CNT_ENABLE_8_9));
+		break;
+	default:
+		BUG();
+	}
+
+	val = read_sysreg_s(SYS_IMP_APL_PMCR0_EL1);
+
+	if (en)
+		val |= bit;
+	else
+		val &= ~bit;
+
+	write_sysreg_s(val, SYS_IMP_APL_PMCR0_EL1);
+}

...

Quoted 19 lines
(quoted diff) +19 -0
+static void __m1_pmu_enable_counter_interrupt(unsigned int index, bool en)
+{
+	u64 val, bit;
+
+	switch (index) {
+	case 0 ... 7:
+		bit = BIT(get_bit_offset(index, PMCR0_PMI_ENABLE_0_7));
+		break;
+	case 8 ... 9:
+		bit = BIT(get_bit_offset(index - 8, PMCR0_PMI_ENABLE_8_9));
+		break;
+	default:
+		BUG();
+	}
+
+	val = read_sysreg_s(SYS_IMP_APL_PMCR0_EL1);
+
+	if (en)
+		val |= bit;
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+	else
+		val &= ~bit;
+
+	write_sysreg_s(val, SYS_IMP_APL_PMCR0_EL1);
+}

These two helper functions have basically the same logic -- maybe worth combining?

I've tried, and I find the result pretty unconvincing, see below.

drivers/perf/apple_m1_cpu_pmu.c +25 -22
index bc991fc892eb..40f9e14b12da 100644
@@ -223,29 +223,38 @@ static void m1_pmu_write_hw_counter(u64 val, unsigned int index)
 
 #define get_bit_offset(index, mask)	(__ffs(mask) + (index))
 
-static void __m1_pmu_enable_counter(unsigned int index, bool en)
+static void __m1_pmu_write_pmcr0(unsigned int index, bool en, u64 mask)
 {
 	u64 val, bit;
 
+	bit = BIT(get_bit_offset(index, mask));
+	val = read_sysreg_s(SYS_IMP_APL_PMCR0_EL1);
+
+	if (en)
+		val |= bit;
+	else
+		val &= ~bit;
+
+	write_sysreg_s(val, SYS_IMP_APL_PMCR0_EL1);
+}
+
+static void __m1_pmu_enable_counter(unsigned int index, bool en)
+{
+	u64 mask;
+
 	switch (index) {
 	case 0 ... 7:
-		bit = BIT(get_bit_offset(index, PMCR0_CNT_ENABLE_0_7));
+		mask = PMCR0_CNT_ENABLE_0_7;
 		break;
 	case 8 ... 9:
-		bit = BIT(get_bit_offset(index - 8, PMCR0_CNT_ENABLE_8_9));
+		mask = PMCR0_CNT_ENABLE_8_9;
+		index -= 8;
 		break;
 	default:
 		BUG();
 	}
 
-	val = read_sysreg_s(SYS_IMP_APL_PMCR0_EL1);
-
-	if (en)
-		val |= bit;
-	else
-		val &= ~bit;
-
-	write_sysreg_s(val, SYS_IMP_APL_PMCR0_EL1);
+	__m1_pmu_write_pmcr0(index, en, mask);
 }
 
 static void m1_pmu_enable_counter(unsigned int index)
@@ -260,27 +269,21 @@ static void m1_pmu_disable_counter(unsigned int index)
 
 static void __m1_pmu_enable_counter_interrupt(unsigned int index, bool en)
 {
-	u64 val, bit;
+	u64 mask;
 
 	switch (index) {
 	case 0 ... 7:
-		bit = BIT(get_bit_offset(index, PMCR0_PMI_ENABLE_0_7));
+		mask =  PMCR0_PMI_ENABLE_0_7;
 		break;
 	case 8 ... 9:
-		bit = BIT(get_bit_offset(index - 8, PMCR0_PMI_ENABLE_8_9));
+		mask = PMCR0_PMI_ENABLE_8_9;
+		index -= 8;
 		break;
 	default:
 		BUG();
 	}
 
-	val = read_sysreg_s(SYS_IMP_APL_PMCR0_EL1);
-
-	if (en)
-		val |= bit;
-	else
-		val &= ~bit;
-
-	write_sysreg_s(val, SYS_IMP_APL_PMCR0_EL1);
+	__m1_pmu_write_pmcr0(index, en, mask);
 }
 
 static void m1_pmu_enable_counter_interrupt(unsigned int index)
Quoted 19 lines
Quoted 4 lines
(quoted diff) +3 -0
+static void m1_pmu_configure_counter(unsigned int index, u8 event,
+				     bool user, bool kernel)
+{

....

Quoted 14 lines
(quoted diff) +14 -0
+	switch (index) {
+	case 0 ... 1:
+		/* 0 and 1 have fixed events */
+		break;
+	case 2 ... 5:
+		shift = (index - 2) * 8;
+		val = read_sysreg_s(SYS_IMP_APL_PMESR0_EL1);
+		val &= ~((u64)0xff << shift);
+		val |= (u64)event << shift;
+		write_sysreg_s(val, SYS_IMP_APL_PMESR0_EL1);
+		break;
+	case 6 ... 9:
+		shift = (index - 6) * 8;
+		val = read_sysreg_s(SYS_IMP_APL_PMESR1_EL1);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+		val &= ~((u64)0xff << shift);
+		val |= (u64)event << shift;
+		write_sysreg_s(val, SYS_IMP_APL_PMESR1_EL1);
+		break;
+	}
+}

I'd love an explanation what's happening here.

PMESR{0,1} contain the event numbers for counters 2-5, resp 6-9, and we happily shove an event number there. Is that the sort of information you are after?

Quoted 22 lines

Review

Quoted 9 lines
(quoted diff) +8 -0
+	/*
+	 * Place the event on the first free counter that can count
+	 * this event.
+	 *
+	 * We could do a better job if we had a view of all the events
+	 * counting on the PMU at any given time, and by placing the
+	 * most constraint events first.
+	 */

constraining

Quoted 10 lines
(quoted diff) +10 -0
+static int m1_pmu_device_probe(struct platform_device *pdev)
+{
+	int ret;
+
+	ret = arm_pmu_device_probe(pdev, m1_pmu_of_device_ids, NULL);
+	if (!ret) {
+		/*
+		 * If probe succeeds, taint the kernel as this is all
+		 * undocumented, implementation defined black magic.
+		 */
Quoted 8 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+		add_taint(TAINT_CPU_OUT_OF_SPEC, LOCKDEP_STILL_OK);
+	}
+
+	return ret;
+}

What are the implications of this taint? You could say that about every driver we've written for the M1, but...

This taint status appears on the kernel crash, telling people the kernel is using CPU features that are not documented and about which we can only hope that they work as we think they work. The same thing takes place with KVM and the use of unadvertised GICv3 CPU registers.

Ideally, this would happen at early boot because of the fixed VHE crap, but this is more effort that it deserves.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.