Skip to contentSkip to conversation

arm_mpam: Add MPAM-Fb firmware support

PATCH Superseded Andre Przywara 5 participants 34 messages Jul 29 – Jul 30

Subsystems (2)

v5 of 12 Superseded Latest is v12 →

lore
Share & tools
lore
#0 lore raw

v5 cover arm_mpam: Add MPAM-Fb firmware support

Hi,

this is version 5 of the MPAM-Fb code, for firmware based MSC accesses. Many thanks to all the diligent reviewers, uncovering bugs and less pretty code. Big change this time is the move to a threaded IRQ handler, to be able to handle error interrupts despite MPAM-Fb MSC accesses requiring process context. Another notable change is that the MPAM-Fb request routine now tears down the whole of MPAM upon noticing an error condition, as we cannot really recover easily at this point. This might be refined and partially lifted in the future.

Find a changelog below. Based on v7.2-rc1.

=======================
The Arm MPAM specification defines Memory System Components (MSCs), which are devices that are programmed through an MMIO register frame. In some occasions this turned out to be too limiting: the MSC might be located behind a separate bus system (for instance inside an on-board controller), it might be mapped secure-only, or in a different processor socket without direct MMIO mapping. Also the MMIO access might be too slow or it would need to be filtered or otherwise access controlled. Finally there might be bugs in the MSC integration, which require a mediating firmware to be accessible.

To accommodate all those different use cases, the MPAM-Fb specification
[1] describes an alternative way to access MSCs. Accesses to an MSC would be wrapped in a message and communicated to the system using a shared-memory/mailbox system mostly mimicking the Arm SCMI spec. For ACPI systems, this would be abstracted through an ACPI PCC channel, which provides the shared-memory region and the mailbox trigger. We can lean on existing ACPI parsing code to register with these two subsystems, but cannot rely on the existing SCMI code in the kernel. This means we somewhat need to open code a very simplified SCMI handler, which just provides enough functionality for the very basic subset of SCMI that the MPAM-Fb spec requires.

The first six patches rework all MSC access wrappers to propagate error information. Pure MMIO based MSC accesses would never fail, but the MPAM-Fb access can go wrong in multiple ways. The patches have been split up purely for reviewing reasons, if the number is a problem, we could as well squash them. Please note that until the very last patch of this series any MSC accesses would always only return 0, it's only the final enablement of MPAM-Fb that could possibly introduce errors. Hence all former patches can add error handling gradually, those code paths wouldn't be triggered before patch 10/10.
Patch 7/10 solves a nasty problem: At the moment we protect stateful MSC register accesses (mon_sel) through a spinlock. Unfortunately the mailbox subsystem and the slow nature of the communication through this channel forbid MPAM-Fb access in atomic context. So this patch keeps using a spinlock for MMIO based accesses, but reverts to a mutex otherwise. We just deny taking the lock for MPAM-Fb in atomic context, ideally we wouldn't need that (no need to IPI another core when the MSC access does not need to be local to one particular core), or we simply deny that part of the functionality (access through perf).
Patch 8/10 adds the code to redirect MSC accesses through the PCC shmem/mailbox system.
Patch 9/10 reworks the error interrupt handler to use a threaded IRQ, to be able to do MPAM-Fb MSC accesses inside (which might sleep). The final patch 10/10 then adds the code to detect and store the PCC channel information from the ACPI tables, and eventually enables MPAM-Fb accesses.

This would enable systems where some MSCs are not accessible via MMIO to use those components anyway.

Please have a look and test!

Cheers,
Andre

[1] https://developer.arm.com/documentation/den0144/latest

Changes in v5:
- do not initialise _high variables early
- fix whitespace issues
- add Jonathan's Reviewed-by: tags
- init only either the spinlock OR the mutex for the mon_sel_lock
- drop warning about mapped_hwpage_sz for MPAM-Fb in mpam_msc_zero_mbwu_l()
- change IRQ handler to threaded handler, to allow MPAM-Fb MSC accesses
- disallow per-CPU IRQs (PPIs) for MPAM-Fb
- trigger MPAM disabling on MPAM-Fb protocol errors
- fix MPAM_PROTOCOL_VERSION_CMD name
- use PCC driver defined symbol for PCC_CHAN_FLAGS_IRQ
- add comment about usage of PCC's .signature field
- check for illegal MPAM-Fb command before sending request
- drop CPU accessibility check for MPAM-Fb based MSC accesses
- move MPAM-Fb protocol message structs inside build functions
- add comment about largest message size
- drop extra msc->mpam_fb_msc_id, in favour using pdev->id
- use already defined local dev variable instead of &pdev->dev
- use FIELD_GET() macro for MPAM-Fb protocol check
- drop devm_mutex_init() for pcc_chan_lock

Changes in v4:
- unify error check patterns
- use scoped_guard(mutex) and ACQUIRE to simplify error handling
- use struct kref for PCC channel refcount
- drop unused mailbox rx_callback function
- increase size of props[] array to hold new msc_id property
- set mailbox TX timeout based on ACPI MPAM table field
- use devm_mutex_init() to allow automatic cleanup
- avoid overwriting error value for arm,not-ready-us property read
- prune MPAM-Fb protocol token to fit into bitmask inside register
- check return value for AIDR read
- use two separate MPAM-Fb protocol payload structs
- prune header list for mpam_fb.c
- drop "nrdy handled as an error" change
- fix message size field in MPAM-Fb protocol shmem area
- drop message size return from MPAM-Fb message build functions
- bring back IRQ flag in MPAM-Fb protocol header

Changes in v3:
- drop inner/outer mon_sel lock patch, replace with simpler version
- harmonise code patterns in error propagation changes
- drop mon_sel lock before erroring out in mpam_ris_hw_probe_csu_nrdy()
- refactor mpam_msc_read_mbwu_l() to return an error
- drop special NRDY handling in __ris_msmon_read()
- add IRQ numbers in error interrupt handler to help pinpoint failure
- refactor MPAM-Fb message generation to accommodate more than read/write
- check MPAM-Fb protocol version at probe time
- drop mpam_fb.h header, merge into mpam_internal.h
- translate MPAM-Fb error code in Linux codes where needed
- clear IRQ flag bit in MPAM-Fb protocol header
- drop unneeded endianness conversions when crafting MPAM-Fb message
- use C struct to model MPAM-Fb message payload

Changes in v2:
- add patches to add error propagation to MSC access wrappers
- drop former patch 1/5 (not needed)
- drop lock in mpam_reprogram_msc(), to avoid double lock
- add support for multiple MSCs per PCC channel
- adjust SCMI protocol code to use a PCC subtype 3 channel
- let PCC code handle the PCC channel negotiation (due to subtype 3)
- drop SCMI names in shmem offsets, and use existing PCC type 3 struct
- adjust shmem field offsets to match MPAM-Fb spec, not pure SCMI
- prevent MPAM-Fb calls inside atomic smp_call_function_any() payload
- skip all MSC accesses inside the IRQ handler when using MPAM-Fb

Andre Przywara (10):
arm_mpam: let low level MSC accessors return an error
arm_mpam: propagate MSC access errors for hw_probe functions
arm_mpam: propagate MSC access errors for MBWU counters
arm_mpam: propagate MSC access errors for msmon helpers
arm_mpam: propagate MSC access errors for __ris_msmon_read()
arm_mpam: propagate MSC access errors for state saving function
arm_mpam: prepare mon_sel locking for MPAM-Fb
arm_mpam: add MPAM-Fb MSC firmware access support
arm_mpam: change error IRQ to use a threaded IRQ handler
arm_mpam: detect and enable MPAM-Fb PCC support

 drivers/resctrl/Makefile        |   2 +-
 drivers/resctrl/mpam_devices.c  | 674 ++++++++++++++++++++++++--------
 drivers/resctrl/mpam_fb.c       | 214 ++++++++++
 drivers/resctrl/mpam_internal.h |  68 +++-
 include/linux/arm_mpam.h        |   2 +-
 5 files changed, 781 insertions(+), 179 deletions(-)
 create mode 100644 drivers/resctrl/mpam_fb.c
Signature · 1 line
2.43.0
#1 lore raw

PATCH v5 1/10 arm_mpam: let low level MSC accessors return an error

The upcoming MPAM-Fb support does not use MMIO primitives to access an MSC, but employs a shared-memory/doorbell based firmware protocol. Its complexity means that it must be able to handle errors, whereas we always assume an MMIO based MSC access succeeds today.

Change the __mpam_read_reg() low level accessor function to return the requested data through a pointer, and return an error code instead. Also change the __mpam_write_reg() accessor function to return an error code. At the moment this is always 0, so all error handling paths are not exercised at the moment. This will change later when alternative MSC access methods like MPAM-Fb are activated.

Change all direct users of those MSC wrappers to comply with the new prototypes, though the errors are not propagated all the way up yet.

---
 drivers/resctrl/mpam_devices.c | 208 +++++++++++++++++++++------------
 1 file changed, 134 insertions(+), 74 deletions(-)
drivers/resctrl/mpam_devices.c +134 -74
index b69f99488111..04d504031752 100644
@@ -177,59 +177,68 @@ static void mpam_assert_partid_sizes_fixed(void)
 		WARN_ON_ONCE(!partid_max_published);
 }
 
-static u32 __mpam_read_reg(struct mpam_msc *msc, u16 reg)
+static int __mpam_read_reg(struct mpam_msc *msc, u16 reg, u32 *res)
 {
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
-	return readl_relaxed(msc->mapped_hwpage + reg);
+	*res = readl_relaxed(msc->mapped_hwpage + reg);
+
+	return 0;
 }
 
-static inline u32 _mpam_read_partsel_reg(struct mpam_msc *msc, u16 reg)
+static inline int _mpam_read_partsel_reg(struct mpam_msc *msc, u16 reg,
+					 u32 *res)
 {
 	lockdep_assert_held_once(&msc->part_sel_lock);
-	return __mpam_read_reg(msc, reg);
+	return __mpam_read_reg(msc, reg, res);
 }
 
-#define mpam_read_partsel_reg(msc, reg) _mpam_read_partsel_reg(msc, MPAMF_##reg)
+#define mpam_read_partsel_reg(msc, reg, res) _mpam_read_partsel_reg(msc, MPAMF_##reg, res)
 
-static void __mpam_write_reg(struct mpam_msc *msc, u16 reg, u32 val)
+static int __mpam_write_reg(struct mpam_msc *msc, u16 reg, u32 val)
 {
 	WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz);
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
 	writel_relaxed(val, msc->mapped_hwpage + reg);
+
+	return 0;
 }
 
-static inline void _mpam_write_partsel_reg(struct mpam_msc *msc, u16 reg, u32 val)
+static inline int _mpam_write_partsel_reg(struct mpam_msc *msc, u16 reg, u32 val)
 {
 	lockdep_assert_held_once(&msc->part_sel_lock);
-	__mpam_write_reg(msc, reg, val);
+	return __mpam_write_reg(msc, reg, val);
 }
 
 #define mpam_write_partsel_reg(msc, reg, val)  _mpam_write_partsel_reg(msc, MPAMCFG_##reg, val)
 
-static inline u32 _mpam_read_monsel_reg(struct mpam_msc *msc, u16 reg)
+static inline int _mpam_read_monsel_reg(struct mpam_msc *msc, u16 reg,
+					u32 *res)
 {
 	mpam_mon_sel_lock_held(msc);
-	return __mpam_read_reg(msc, reg);
+	return __mpam_read_reg(msc, reg, res);
 }
 
-#define mpam_read_monsel_reg(msc, reg) _mpam_read_monsel_reg(msc, MSMON_##reg)
+#define mpam_read_monsel_reg(msc, reg, res) _mpam_read_monsel_reg(msc, MSMON_##reg, res)
 
-static inline void _mpam_write_monsel_reg(struct mpam_msc *msc, u16 reg, u32 val)
+static inline int _mpam_write_monsel_reg(struct mpam_msc *msc, u16 reg, u32 val)
 {
 	mpam_mon_sel_lock_held(msc);
-	__mpam_write_reg(msc, reg, val);
+	return __mpam_write_reg(msc, reg, val);
 }
 
 #define mpam_write_monsel_reg(msc, reg, val)   _mpam_write_monsel_reg(msc, MSMON_##reg, val)
 
 static bool mpam_msc_check_aidr(struct mpam_msc *msc)
 {
-	u32 aidr = __mpam_read_reg(msc, MPAMF_AIDR);
-	u32 major = FIELD_GET(MPAMF_AIDR_ARCH_MAJOR_REV, aidr);
-	u32 minor = FIELD_GET(MPAMF_AIDR_ARCH_MINOR_REV, aidr);
+	u32 aidr, major, minor;
 
+	if (__mpam_read_reg(msc, MPAMF_AIDR, &aidr))
+		return false;
+
+	major = FIELD_GET(MPAMF_AIDR_ARCH_MAJOR_REV, aidr);
+	minor = FIELD_GET(MPAMF_AIDR_ARCH_MINOR_REV, aidr);
 	/*
 	 * v0.0 and >v2.x aren't supported, but anything else should be backward
 	 * compatible to v0.1 or v1.0.
@@ -242,25 +251,41 @@ static bool mpam_msc_check_aidr(struct mpam_msc *msc)
 	return true;
 }
 
-static u64 mpam_msc_read_idr(struct mpam_msc *msc)
+static int mpam_msc_read_idr(struct mpam_msc *msc, u64 *res)
 {
-	u64 idr_high = 0, idr_low;
+	u32 idr_high, idr_low;
+	int ret;
 
 	lockdep_assert_held(&msc->part_sel_lock);
 
-	idr_low = mpam_read_partsel_reg(msc, IDR);
-	if (FIELD_GET(MPAMF_IDR_EXT, idr_low))
-		idr_high = mpam_read_partsel_reg(msc, IDR + 4);
+	ret = mpam_read_partsel_reg(msc, IDR, &idr_low);
+	if (ret)
+		return ret;
+
+	if (FIELD_GET(MPAMF_IDR_EXT, idr_low)) {
+		ret = mpam_read_partsel_reg(msc, IDR + 4, &idr_high);
+		if (ret)
+			return ret;
+	} else {
+		idr_high = 0;
+	}
+
+	*res = ((u64)idr_high << 32) | idr_low;
 
-	return (idr_high << 32) | idr_low;
+	return 0;
 }
 
-static void mpam_msc_clear_esr(struct mpam_msc *msc)
+static int mpam_msc_clear_esr(struct mpam_msc *msc)
 {
-	u64 esr_low = __mpam_read_reg(msc, MPAMF_ESR);
+	u32 esr_low;
+	int ret;
+
+	ret = __mpam_read_reg(msc, MPAMF_ESR, &esr_low);
+	if (ret)
+		return ret;
 
 	if (!esr_low)
-		return;
+		return 0;
 
 	/*
 	 * Clearing the high/low bits of MPAMF_ESR can not be atomic.
@@ -268,44 +293,59 @@ static void mpam_msc_clear_esr(struct mpam_msc *msc)
 	 * lower half prevent hardware from updating either half of the
 	 * register.
 	 */
-	if (msc->has_extd_esr)
-		__mpam_write_reg(msc, MPAMF_ESR + 4, 0);
-	__mpam_write_reg(msc, MPAMF_ESR, 0);
+	if (msc->has_extd_esr) {
+		ret = __mpam_write_reg(msc, MPAMF_ESR + 4, 0);
+		if (ret)
+			return ret;
+	}
+
+	return __mpam_write_reg(msc, MPAMF_ESR, 0);
 }
 
-static u64 mpam_msc_read_esr(struct mpam_msc *msc)
+static int mpam_msc_read_esr(struct mpam_msc *msc, u64 *res)
 {
-	u64 esr_high = 0, esr_low;
+	u32 esr_high, esr_low;
+	int ret;
 
-	esr_low = __mpam_read_reg(msc, MPAMF_ESR);
-	if (msc->has_extd_esr)
-		esr_high = __mpam_read_reg(msc, MPAMF_ESR + 4);
+	ret = __mpam_read_reg(msc, MPAMF_ESR, &esr_low);
+	if (ret)
+		return ret;
 
-	return (esr_high << 32) | esr_low;
+	if (msc->has_extd_esr) {
+		ret = __mpam_read_reg(msc, MPAMF_ESR + 4, &esr_high);
+		if (ret)
+			return ret;
+	} else {
+		esr_high = 0;
+	}
+
+	*res = ((u64)esr_high << 32) | esr_low;
+
+	return 0;
 }
 
-static void __mpam_part_sel_raw(u32 partsel, struct mpam_msc *msc)
+static int __mpam_part_sel_raw(u32 partsel, struct mpam_msc *msc)
 {
 	lockdep_assert_held(&msc->part_sel_lock);
 
-	mpam_write_partsel_reg(msc, PART_SEL, partsel);
+	return mpam_write_partsel_reg(msc, PART_SEL, partsel);
 }
 
-static void __mpam_part_sel(u8 ris_idx, u16 partid, struct mpam_msc *msc)
+static int __mpam_part_sel(u8 ris_idx, u16 partid, struct mpam_msc *msc)
 {
 	u32 partsel = FIELD_PREP(MPAMCFG_PART_SEL_RIS, ris_idx) |
 		      FIELD_PREP(MPAMCFG_PART_SEL_PARTID_SEL, partid);
 
-	__mpam_part_sel_raw(partsel, msc);
+	return __mpam_part_sel_raw(partsel, msc);
 }
 
-static void __mpam_intpart_sel(u8 ris_idx, u16 intpartid, struct mpam_msc *msc)
+static int __mpam_intpart_sel(u8 ris_idx, u16 intpartid, struct mpam_msc *msc)
 {
 	u32 partsel = FIELD_PREP(MPAMCFG_PART_SEL_RIS, ris_idx) |
 		      FIELD_PREP(MPAMCFG_PART_SEL_PARTID_SEL, intpartid) |
 		      MPAMCFG_PART_SEL_INTERNAL;
 
-	__mpam_part_sel_raw(partsel, msc);
+	return __mpam_part_sel_raw(partsel, msc);
 }
 
 int mpam_register_requestor(u16 partid_max, u8 pmg_max)
@@ -774,13 +814,13 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris)
 	mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
 
 	_mpam_write_monsel_reg(msc, MSMON_CSU, MSMON___NRDY);
-	now = _mpam_read_monsel_reg(msc, MSMON_CSU);
+	_mpam_read_monsel_reg(msc, MSMON_CSU, &now);
 	can_set = now & MSMON___NRDY;
 
 	_mpam_write_monsel_reg(msc, MSMON_CSU, 0);
 	/* Configuration change to try and coax hardware into setting nrdy */
 	mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0x1);
-	now = _mpam_read_monsel_reg(msc, MSMON_CSU);
+	_mpam_read_monsel_reg(msc, MSMON_CSU, &now);
 	can_clear = !(now & MSMON___NRDY);
 	mpam_mon_sel_unlock(msc);
 
@@ -800,7 +840,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 
 	/* Cache Capacity Partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_CCAP_PART, ris->idr)) {
-		u32 ccap_features = mpam_read_partsel_reg(msc, CCAP_IDR);
+		u32 ccap_features;
+
+		mpam_read_partsel_reg(msc, CCAP_IDR, &ccap_features);
 
 		props->cmax_wd = FIELD_GET(MPAMF_CCAP_IDR_CMAX_WD, ccap_features);
 		if (props->cmax_wd &&
@@ -823,7 +865,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 
 	/* Cache Portion partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_CPOR_PART, ris->idr)) {
-		u32 cpor_features = mpam_read_partsel_reg(msc, CPOR_IDR);
+		u32 cpor_features;
+
+		mpam_read_partsel_reg(msc, CPOR_IDR, &cpor_features);
 
 		props->cpbm_wd = FIELD_GET(MPAMF_CPOR_IDR_CPBM_WD, cpor_features);
 		if (props->cpbm_wd)
@@ -832,7 +876,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 
 	/* Memory bandwidth partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_MBW_PART, ris->idr)) {
-		u32 mbw_features = mpam_read_partsel_reg(msc, MBW_IDR);
+		u32 mbw_features;
+
+		mpam_read_partsel_reg(msc, MBW_IDR, &mbw_features);
 
 		/* portion bitmap resolution */
 		props->mbw_pbm_bits = FIELD_GET(MPAMF_MBW_IDR_BWPBM_WD, mbw_features);
@@ -860,7 +906,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 
 	/* Priority partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_PRI_PART, ris->idr)) {
-		u32 pri_features = mpam_read_partsel_reg(msc, PRI_IDR);
+		u32 pri_features;
+
+		mpam_read_partsel_reg(msc, PRI_IDR, &pri_features);
 
 		props->intpri_wd = FIELD_GET(MPAMF_PRI_IDR_INTPRI_WD, pri_features);
 		if (props->intpri_wd && FIELD_GET(MPAMF_PRI_IDR_HAS_INTPRI, pri_features)) {
@@ -879,7 +927,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 
 	/* Performance Monitoring */
 	if (FIELD_GET(MPAMF_IDR_HAS_MSMON, ris->idr)) {
-		u32 msmon_features = mpam_read_partsel_reg(msc, MSMON_IDR);
+		u32 msmon_features;
+
+		mpam_read_partsel_reg(msc, MSMON_IDR, &msmon_features);
 
 		/*
 		 * If the firmware max-nrdy-us property is missing, the
@@ -892,7 +942,8 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 		if (FIELD_GET(MPAMF_MSMON_IDR_MSMON_CSU, msmon_features)) {
 			u32 csumonidr;
 
-			csumonidr = mpam_read_partsel_reg(msc, CSUMON_IDR);
+			mpam_read_partsel_reg(msc, CSUMON_IDR, &csumonidr);
+
 			props->num_csu_mon = FIELD_GET(MPAMF_CSUMON_IDR_NUM_MON, csumonidr);
 			if (props->num_csu_mon) {
 				bool hw_managed;
@@ -915,7 +966,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 		}
 		if (FIELD_GET(MPAMF_MSMON_IDR_MSMON_MBWU, msmon_features)) {
 			bool has_long;
-			u32 mbwumon_idr = mpam_read_partsel_reg(msc, MBWUMON_IDR);
+			u32 mbwumon_idr;
+
+			mpam_read_partsel_reg(msc, MBWUMON_IDR, &mbwumon_idr);
 
 			props->num_mbwu_mon = FIELD_GET(MPAMF_MBWUMON_IDR_NUM_MON, mbwumon_idr);
 			if (props->num_mbwu_mon) {
@@ -945,8 +998,11 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	 */
 	if (FIELD_GET(MPAMF_IDR_HAS_PARTID_NRW, ris->idr) &&
 	    class->type != MPAM_CLASS_UNKNOWN) {
-		u32 nrwidr = mpam_read_partsel_reg(msc, PARTID_NRW_IDR);
-		u16 partid_max = FIELD_GET(MPAMF_PARTID_NRW_IDR_INTPARTID_MAX, nrwidr);
+		u16 partid_max;
+		u32 nrwidr;
+
+		mpam_read_partsel_reg(msc, PARTID_NRW_IDR, &nrwidr);
+		partid_max = FIELD_GET(MPAMF_PARTID_NRW_IDR_INTPARTID_MAX, nrwidr);
 
 		mpam_set_feature(mpam_feat_partid_nrw, props);
 		msc->partid_max = min(msc->partid_max, partid_max);
@@ -970,8 +1026,9 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 
 	/* Grab an IDR value to find out how many RIS there are */
 	mutex_lock(&msc->part_sel_lock);
-	idr = mpam_msc_read_idr(msc);
-	msc->iidr = mpam_read_partsel_reg(msc, IIDR);
+	mpam_msc_read_idr(msc, &idr);
+	mpam_read_partsel_reg(msc, IIDR, &msc->iidr);
+
 	mutex_unlock(&msc->part_sel_lock);
 
 	mpam_enable_quirks(msc);
@@ -985,7 +1042,7 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 	for (ris_idx = 0; ris_idx <= msc->ris_max; ris_idx++) {
 		mutex_lock(&msc->part_sel_lock);
 		__mpam_part_sel(ris_idx, 0, msc);
-		idr = mpam_msc_read_idr(msc);
+		mpam_msc_read_idr(msc, &idr);
 		mutex_unlock(&msc->part_sel_lock);
 
 		partid_max = FIELD_GET(MPAMF_IDR_PARTID_MAX, idr);
@@ -1039,24 +1096,24 @@ static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)
 {
 	int retry = 3;
 	u32 mbwu_l_low;
-	u64 mbwu_l_high1, mbwu_l_high2;
+	u32 mbwu_l_high1, mbwu_l_high2;
 
 	mpam_mon_sel_lock_held(msc);
 
 	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
-	mbwu_l_high2 = __mpam_read_reg(msc, MSMON_MBWU_L + 4);
+	__mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
 	do {
 		mbwu_l_high1 = mbwu_l_high2;
-		mbwu_l_low = __mpam_read_reg(msc, MSMON_MBWU_L);
-		mbwu_l_high2 = __mpam_read_reg(msc, MSMON_MBWU_L + 4);
+		__mpam_read_reg(msc, MSMON_MBWU_L, &mbwu_l_low);
+		__mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
 
 		retry--;
 	} while (mbwu_l_high1 != mbwu_l_high2 && retry > 0);
 
 	if (mbwu_l_high1 == mbwu_l_high2)
-		return (mbwu_l_high1 << 32) | mbwu_l_low;
+		return ((u64)mbwu_l_high1 << 32) | mbwu_l_low;
 
 	pr_warn("Failed to read a stable value\n");
 	return MSMON___L_NRDY;
@@ -1120,14 +1177,14 @@ static void read_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val,
 
 	switch (m->type) {
 	case mpam_feat_msmon_csu:
-		*ctl_val = mpam_read_monsel_reg(msc, CFG_CSU_CTL);
-		*flt_val = mpam_read_monsel_reg(msc, CFG_CSU_FLT);
+		mpam_read_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
+		mpam_read_monsel_reg(msc, CFG_CSU_FLT, flt_val);
 		break;
 	case mpam_feat_msmon_mbwu_31counter:
 	case mpam_feat_msmon_mbwu_44counter:
 	case mpam_feat_msmon_mbwu_63counter:
-		*ctl_val = mpam_read_monsel_reg(msc, CFG_MBWU_CTL);
-		*flt_val = mpam_read_monsel_reg(msc, CFG_MBWU_FLT);
+		mpam_read_monsel_reg(msc, CFG_MBWU_CTL, ctl_val);
+		mpam_read_monsel_reg(msc, CFG_MBWU_FLT, flt_val);
 		break;
 	default:
 		pr_warn("Unexpected monitor type %d\n", m->type);
@@ -1206,6 +1263,7 @@ static u64 mpam_msmon_overflow_val(enum mpam_device_features type,
 static void __ris_msmon_read(void *arg)
 {
 	u64 now;
+	u32 now32;
 	bool nrdy = false;
 	bool config_mismatch;
 	bool overflow = false;
@@ -1268,9 +1326,9 @@ static void __ris_msmon_read(void *arg)
 
 	switch (m->type) {
 	case mpam_feat_msmon_csu:
-		now = mpam_read_monsel_reg(msc, CSU);
-		nrdy = now & MSMON___NRDY;
-		now = FIELD_GET(MSMON___VALUE, now);
+		mpam_read_monsel_reg(msc, CSU, &now32);
+		nrdy = now32 & MSMON___NRDY;
+		now = FIELD_GET(MSMON___VALUE, now32);
 
 		if (mpam_has_quirk(IGNORE_CSU_NRDY, msc) && m->waited_timeout)
 			nrdy = false;
@@ -1288,9 +1346,9 @@ static void __ris_msmon_read(void *arg)
 			else
 				now = FIELD_GET(MSMON___L_VALUE, now);
 		} else {
-			now = mpam_read_monsel_reg(msc, MBWU);
-			nrdy = now & MSMON___NRDY;
-			now = FIELD_GET(MSMON___VALUE, now);
+			mpam_read_monsel_reg(msc, MBWU, &now32);
+			nrdy = now32 & MSMON___NRDY;
+			now = FIELD_GET(MSMON___VALUE, now32);
 		}
 
 		if (mpam_has_quirk(T241_MBW_COUNTER_SCALE_64, msc) &&
@@ -1685,16 +1743,18 @@ static int mpam_save_mbwu_state(void *arg)
 		mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) |
 			  FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx);
 		mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
-
-		cur_flt = mpam_read_monsel_reg(msc, CFG_MBWU_FLT);
-		cur_ctl = mpam_read_monsel_reg(msc, CFG_MBWU_CTL);
+		mpam_read_monsel_reg(msc, CFG_MBWU_FLT, &cur_flt);
+		mpam_read_monsel_reg(msc, CFG_MBWU_CTL, &cur_ctl);
 		mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
 
 		if (mpam_ris_has_mbwu_long_counter(ris)) {
 			val = mpam_msc_read_mbwu_l(msc);
 			mpam_msc_zero_mbwu_l(msc);
 		} else {
-			val = mpam_read_monsel_reg(msc, MBWU);
+			u32 val32;
+
+			mpam_read_monsel_reg(msc, MBWU, &val32);
+			val = val32;
 			mpam_write_monsel_reg(msc, MBWU, 0);
 		}
 
@@ -2465,7 +2525,7 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
 					   &msc->accessibility)))
 		return IRQ_NONE;
 
-	reg = mpam_msc_read_esr(msc);
+	mpam_msc_read_esr(msc, &reg);
 
 	errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
 	if (!errcode)
Signature · 1 line
2.43.0
maintainer Reviewed-by on v5 1/10
#2 lore raw

Re: [PATCH v5 01/10] arm_mpam: let low level MSC accessors return an error

Hi Andre,

Quoted 12 lines · Andre Przywara

The upcoming MPAM-Fb support does not use MMIO primitives to access an MSC, but employs a shared-memory/doorbell based firmware protocol. Its complexity means that it must be able to handle errors, whereas we always assume an MMIO based MSC access succeeds today.

Change the __mpam_read_reg() low level accessor function to return the requested data through a pointer, and return an error code instead. Also change the __mpam_write_reg() accessor function to return an error code. At the moment this is always 0, so all error handling paths are not exercised at the moment. This will change later when alternative MSC access methods like MPAM-Fb are activated.

Quoted 5 lines

Change all direct users of those MSC wrappers to comply with the new prototypes, though the errors are not propagated all the way up yet.

Thanks,

Ben

#3 lore raw

PATCH v5 2/10 arm_mpam: propagate MSC access errors for hw_probe functions

In reply to Andre Przywara

Allow the functions probing for MSC hardware and features to return an error, and propagate read and write errors from the lower level up. This uses some "scoped cleanup" functions like scoped_guard() and ACQUIRE() to avoid the complexity of error handling when a lock has been taken. Since the mon_sel_lock is a bit special (even more so in an upcoming patch), we define a new GUARD type for it.

---
 drivers/resctrl/mpam_devices.c  | 126 +++++++++++++++++++++-----------
 drivers/resctrl/mpam_internal.h |   4 +
 2 files changed, 87 insertions(+), 43 deletions(-)
drivers/resctrl/mpam_devices.c +83 -43
index 04d504031752..017e5bf067bc 100644
@@ -798,38 +798,46 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris)
 	bool can_set, can_clear;
 	struct mpam_msc *msc = ris->vmsc->msc;
 
-	if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc)))
+	ACQUIRE(mon_sel_lock, guard)(msc);
+	if (ACQUIRE_ERR(mon_sel_lock, &guard))
 		return false;
 
 	mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, 0) |
 		  FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx);
-	mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
+	if (mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel))
+		return false;
 
 	/* Hardware might ignore nrdy if it's not enabled */
 	ctl_val = MSMON_CFG_CSU_CTL_TYPE_CSU;
 	ctl_val |= MSMON_CFG_x_CTL_MATCH_PARTID;
 	ctl_val |= MSMON_CFG_x_CTL_MATCH_PMG;
 	ctl_val |= MSMON_CFG_x_CTL_EN;
-	mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0);
-	mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
+	if (mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0))
+		return false;
+	if (mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val))
+		return false;
 
-	_mpam_write_monsel_reg(msc, MSMON_CSU, MSMON___NRDY);
-	_mpam_read_monsel_reg(msc, MSMON_CSU, &now);
+	if (_mpam_write_monsel_reg(msc, MSMON_CSU, MSMON___NRDY))
+		return false;
+	if (_mpam_read_monsel_reg(msc, MSMON_CSU, &now))
+		return false;
 	can_set = now & MSMON___NRDY;
 
-	_mpam_write_monsel_reg(msc, MSMON_CSU, 0);
+	if (_mpam_write_monsel_reg(msc, MSMON_CSU, 0))
+		return false;
 	/* Configuration change to try and coax hardware into setting nrdy */
-	mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0x1);
-	_mpam_read_monsel_reg(msc, MSMON_CSU, &now);
+	if (mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0x1))
+		return false;
+	if (_mpam_read_monsel_reg(msc, MSMON_CSU, &now))
+		return false;
 	can_clear = !(now & MSMON___NRDY);
-	mpam_mon_sel_unlock(msc);
 
 	return (!can_set || !can_clear);
 }
 
-static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
+static int mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 {
-	int err;
+	int fw_has_nrdy_err, err;
 	struct mpam_msc *msc = ris->vmsc->msc;
 	struct device *dev = &msc->pdev->dev;
 	struct mpam_props *props = &ris->props;
@@ -842,7 +850,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	if (FIELD_GET(MPAMF_IDR_HAS_CCAP_PART, ris->idr)) {
 		u32 ccap_features;
 
-		mpam_read_partsel_reg(msc, CCAP_IDR, &ccap_features);
+		err = mpam_read_partsel_reg(msc, CCAP_IDR, &ccap_features);
+		if (err)
+			return err;
 
 		props->cmax_wd = FIELD_GET(MPAMF_CCAP_IDR_CMAX_WD, ccap_features);
 		if (props->cmax_wd &&
@@ -867,7 +877,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	if (FIELD_GET(MPAMF_IDR_HAS_CPOR_PART, ris->idr)) {
 		u32 cpor_features;
 
-		mpam_read_partsel_reg(msc, CPOR_IDR, &cpor_features);
+		err = mpam_read_partsel_reg(msc, CPOR_IDR, &cpor_features);
+		if (err)
+			return err;
 
 		props->cpbm_wd = FIELD_GET(MPAMF_CPOR_IDR_CPBM_WD, cpor_features);
 		if (props->cpbm_wd)
@@ -877,8 +889,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	/* Memory bandwidth partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_MBW_PART, ris->idr)) {
 		u32 mbw_features;
-
-		mpam_read_partsel_reg(msc, MBW_IDR, &mbw_features);
+		err = mpam_read_partsel_reg(msc, MBW_IDR, &mbw_features);
+		if (err)
+			return err;
 
 		/* portion bitmap resolution */
 		props->mbw_pbm_bits = FIELD_GET(MPAMF_MBW_IDR_BWPBM_WD, mbw_features);
@@ -907,8 +920,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	/* Priority partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_PRI_PART, ris->idr)) {
 		u32 pri_features;
-
-		mpam_read_partsel_reg(msc, PRI_IDR, &pri_features);
+		err = mpam_read_partsel_reg(msc, PRI_IDR, &pri_features);
+		if (err)
+			return err;
 
 		props->intpri_wd = FIELD_GET(MPAMF_PRI_IDR_INTPRI_WD, pri_features);
 		if (props->intpri_wd && FIELD_GET(MPAMF_PRI_IDR_HAS_INTPRI, pri_features)) {
@@ -929,20 +943,24 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	if (FIELD_GET(MPAMF_IDR_HAS_MSMON, ris->idr)) {
 		u32 msmon_features;
 
-		mpam_read_partsel_reg(msc, MSMON_IDR, &msmon_features);
+		err = mpam_read_partsel_reg(msc, MSMON_IDR, &msmon_features);
+		if (err)
+			return err;
 
 		/*
 		 * If the firmware max-nrdy-us property is missing, the
 		 * CSU counters can't be used. Should we wait forever?
 		 */
-		err = device_property_read_u32(&msc->pdev->dev,
-					       "arm,not-ready-us",
-					       &msc->nrdy_usec);
+		fw_has_nrdy_err = device_property_read_u32(&msc->pdev->dev,
+							   "arm,not-ready-us",
+							   &msc->nrdy_usec);
 
 		if (FIELD_GET(MPAMF_MSMON_IDR_MSMON_CSU, msmon_features)) {
 			u32 csumonidr;
 
-			mpam_read_partsel_reg(msc, CSUMON_IDR, &csumonidr);
+			err = mpam_read_partsel_reg(msc, CSUMON_IDR, &csumonidr);
+			if (err)
+				return err;
 
 			props->num_csu_mon = FIELD_GET(MPAMF_CSUMON_IDR_NUM_MON, csumonidr);
 			if (props->num_csu_mon) {
@@ -960,7 +978,7 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 				 * Accept the missing firmware property if NRDY appears
 				 * un-implemented.
 				 */
-				if (err && hw_managed)
+				if (fw_has_nrdy_err && hw_managed)
 					dev_err_once(dev, "Counters are not usable because not-ready timeout was not provided by firmware.");
 			}
 		}
@@ -968,7 +986,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 			bool has_long;
 			u32 mbwumon_idr;
 
-			mpam_read_partsel_reg(msc, MBWUMON_IDR, &mbwumon_idr);
+			err = mpam_read_partsel_reg(msc, MBWUMON_IDR, &mbwumon_idr);
+			if (err)
+				return err;
 
 			props->num_mbwu_mon = FIELD_GET(MPAMF_MBWUMON_IDR_NUM_MON, mbwumon_idr);
 			if (props->num_mbwu_mon) {
@@ -1001,16 +1021,22 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 		u16 partid_max;
 		u32 nrwidr;
 
-		mpam_read_partsel_reg(msc, PARTID_NRW_IDR, &nrwidr);
+		err = mpam_read_partsel_reg(msc, PARTID_NRW_IDR, &nrwidr);
+		if (err)
+			return err;
+
 		partid_max = FIELD_GET(MPAMF_PARTID_NRW_IDR_INTPARTID_MAX, nrwidr);
 
 		mpam_set_feature(mpam_feat_partid_nrw, props);
 		msc->partid_max = min(msc->partid_max, partid_max);
 	}
+
+	return 0;
 }
 
 static int mpam_msc_hw_probe(struct mpam_msc *msc)
 {
+	int ret;
 	u64 idr;
 	u16 partid_max;
 	u8 ris_idx, pmg_max;
@@ -1025,11 +1051,15 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 	}
 
 	/* Grab an IDR value to find out how many RIS there are */
-	mutex_lock(&msc->part_sel_lock);
-	mpam_msc_read_idr(msc, &idr);
-	mpam_read_partsel_reg(msc, IIDR, &msc->iidr);
+	scoped_guard(mutex, &msc->part_sel_lock) {
+		ret = mpam_msc_read_idr(msc, &idr);
+		if (ret)
+			return ret;
 
-	mutex_unlock(&msc->part_sel_lock);
+		ret = mpam_read_partsel_reg(msc, IIDR, &msc->iidr);
+		if (ret)
+			return ret;
+	}
 
 	mpam_enable_quirks(msc);
 
@@ -1040,10 +1070,15 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 	msc->pmg_max = FIELD_GET(MPAMF_IDR_PMG_MAX, idr);
 
 	for (ris_idx = 0; ris_idx <= msc->ris_max; ris_idx++) {
-		mutex_lock(&msc->part_sel_lock);
-		__mpam_part_sel(ris_idx, 0, msc);
-		mpam_msc_read_idr(msc, &idr);
-		mutex_unlock(&msc->part_sel_lock);
+		scoped_guard(mutex, &msc->part_sel_lock) {
+			ret = __mpam_part_sel(ris_idx, 0, msc);
+			if (ret)
+				return ret;
+
+			ret = mpam_msc_read_idr(msc, &idr);
+			if (ret)
+				return ret;
+		}
 
 		partid_max = FIELD_GET(MPAMF_IDR_PARTID_MAX, idr);
 		pmg_max = FIELD_GET(MPAMF_IDR_PMG_MAX, idr);
@@ -1051,17 +1086,22 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 		msc->pmg_max = min(msc->pmg_max, pmg_max);
 		msc->has_extd_esr = FIELD_GET(MPAMF_IDR_HAS_EXTD_ESR, idr);
 
-		mutex_lock(&mpam_list_lock);
-		ris = mpam_get_or_create_ris(msc, ris_idx);
-		mutex_unlock(&mpam_list_lock);
-		if (IS_ERR(ris))
-			return PTR_ERR(ris);
+		scoped_guard(mutex, &mpam_list_lock) {
+			ris = mpam_get_or_create_ris(msc, ris_idx);
+			if (IS_ERR(ris))
+				return PTR_ERR(ris);
+		}
 		ris->idr = idr;
 
-		mutex_lock(&msc->part_sel_lock);
-		__mpam_part_sel(ris_idx, 0, msc);
-		mpam_ris_hw_probe(ris);
-		mutex_unlock(&msc->part_sel_lock);
+		scoped_guard(mutex, &msc->part_sel_lock) {
+			ret = __mpam_part_sel(ris_idx, 0, msc);
+			if (ret)
+				return ret;
+
+			ret = mpam_ris_hw_probe(ris);
+			if (ret)
+				return ret;
+		}
 	}
 
 	/* Clear any stale errors */
drivers/resctrl/mpam_internal.h +4 -0
index 04d1a59f02af..57f2b3dbf7d0 100644
@@ -162,6 +162,10 @@ static inline void mpam_mon_sel_lock_init(struct mpam_msc *msc)
 	raw_spin_lock_init(&msc->_mon_sel_lock);
 }
 
+DEFINE_GUARD(mon_sel, struct mpam_msc *,
+	     mpam_mon_sel_lock(_T), mpam_mon_sel_unlock(_T));
+DEFINE_GUARD_COND(mon_sel, _lock, mpam_mon_sel_lock(_T), _RET);
+
 /* Bits for mpam features bitmaps */
 enum mpam_device_features {
 	mpam_feat_cpor_part,
Signature · 1 line
2.43.0
maintainer Reviewed-by on v5 2/10
#4 lore raw

Re: [PATCH v5 02/10] arm_mpam: propagate MSC access errors for hw_probe functions

Hi Andre,

Quoted 9 lines · Andre Przywara

Allow the functions probing for MSC hardware and features to return an error, and propagate read and write errors from the lower level up. This uses some "scoped cleanup" functions like scoped_guard() and ACQUIRE() to avoid the complexity of error handling when a lock has been taken. Since the mon_sel_lock is a bit special (even more so in an upcoming patch), we define a new GUARD type for it.

Thanks,

Ben

#5 lore raw

PATCH v5 3/10 arm_mpam: propagate MSC access errors for MBWU counters

In reply to Andre Przywara

Allow the mpam_msc_read_mbwu_l() function to return an error, and propagate errors from the lower level up. This also changes the users of this function.

---
 drivers/resctrl/mpam_devices.c | 45 +++++++++++++++++++++++-----------
 1 file changed, 31 insertions(+), 14 deletions(-)
drivers/resctrl/mpam_devices.c +31 -14
index 017e5bf067bc..32ad1864f70f 100644
@@ -1132,8 +1132,9 @@ static bool mpam_ris_has_mbwu_long_counter(struct mpam_msc_ris *ris)
 		mpam_has_feature(mpam_feat_msmon_mbwu_44counter, &ris->props));
 }
 
-static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)
+static int mpam_msc_read_mbwu_l(struct mpam_msc *msc, u64 *res)
 {
+	int ret;
 	int retry = 3;
 	u32 mbwu_l_low;
 	u32 mbwu_l_high1, mbwu_l_high2;
@@ -1143,20 +1144,30 @@ static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)
 	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
-	__mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+	ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+	if (ret)
+		return ret;
+
 	do {
 		mbwu_l_high1 = mbwu_l_high2;
-		__mpam_read_reg(msc, MSMON_MBWU_L, &mbwu_l_low);
-		__mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+		ret = __mpam_read_reg(msc, MSMON_MBWU_L, &mbwu_l_low);
+		if (ret)
+			return ret;
+		ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+		if (ret)
+			return ret;
 
 		retry--;
 	} while (mbwu_l_high1 != mbwu_l_high2 && retry > 0);
 
-	if (mbwu_l_high1 == mbwu_l_high2)
-		return ((u64)mbwu_l_high1 << 32) | mbwu_l_low;
+	if (mbwu_l_high1 == mbwu_l_high2) {
+		*res = ((u64)mbwu_l_high1 << 32) | mbwu_l_low;
+	} else {
+		pr_warn("Failed to read a stable value\n");
+		*res = MSMON___L_NRDY;
+	}
 
-	pr_warn("Failed to read a stable value\n");
-	return MSMON___L_NRDY;
+	return 0;
 }
 
 static void mpam_msc_zero_mbwu_l(struct mpam_msc *msc)
@@ -1315,10 +1326,12 @@ static void __ris_msmon_read(void *arg)
 	struct mpam_msc *msc = m->ris->vmsc->msc;
 	u32 mon_sel, ctl_val, flt_val, cur_ctl, cur_flt;
 
-	if (!mpam_mon_sel_lock(msc)) {
+	ACQUIRE(mon_sel_lock, guard)(msc);
+	if (ACQUIRE_ERR(mon_sel_lock, &guard)) {
 		m->err = -EIO;
 		return;
 	}
+
 	mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, ctx->mon) |
 		  FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx);
 	mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
@@ -1378,7 +1391,9 @@ static void __ris_msmon_read(void *arg)
 	case mpam_feat_msmon_mbwu_44counter:
 	case mpam_feat_msmon_mbwu_63counter:
 		if (m->type != mpam_feat_msmon_mbwu_31counter) {
-			now = mpam_msc_read_mbwu_l(msc);
+			m->err = mpam_msc_read_mbwu_l(msc, &now);
+			if (m->err)
+				return;
 			nrdy = now & MSMON___L_NRDY;
 
 			if (m->type == mpam_feat_msmon_mbwu_63counter)
@@ -1412,7 +1427,6 @@ static void __ris_msmon_read(void *arg)
 	default:
 		m->err = -EINVAL;
 	}
-	mpam_mon_sel_unlock(msc);
 
 	if (nrdy)
 		m->err = -EBUSY;
@@ -1767,6 +1781,7 @@ static int mpam_save_mbwu_state(void *arg)
 {
 	int i;
 	u64 val;
+	int ret;
 	struct mon_cfg *cfg;
 	u32 cur_flt, cur_ctl, mon_sel;
 	struct mpam_msc_ris *ris = arg;
@@ -1777,7 +1792,8 @@ static int mpam_save_mbwu_state(void *arg)
 		mbwu_state = &ris->mbwu_state[i];
 		cfg = &mbwu_state->cfg;
 
-		if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc)))
+		ACQUIRE(mon_sel_lock, guard)(msc);
+		if (ACQUIRE_ERR(mon_sel_lock, &guard))
 			return -EIO;
 
 		mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) |
@@ -1788,7 +1804,9 @@ static int mpam_save_mbwu_state(void *arg)
 		mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
 
 		if (mpam_ris_has_mbwu_long_counter(ris)) {
-			val = mpam_msc_read_mbwu_l(msc);
+			ret = mpam_msc_read_mbwu_l(msc, &val);
+			if (ret)
+				return ret;
 			mpam_msc_zero_mbwu_l(msc);
 		} else {
 			u32 val32;
@@ -1804,7 +1822,6 @@ static int mpam_save_mbwu_state(void *arg)
 		cfg->partid = FIELD_GET(MSMON_CFG_x_FLT_PARTID, cur_flt);
 		mbwu_state->correction += val;
 		mbwu_state->enabled = FIELD_GET(MSMON_CFG_x_CTL_EN, cur_ctl);
-		mpam_mon_sel_unlock(msc);
 	}
 
 	return 0;
Signature · 1 line
2.43.0
#6 lore raw

Re: [PATCH v5 03/10] arm_mpam: propagate MSC access errors for MBWU counters

Quoted 19 lines · Andre Przywara
drivers/resctrl/mpam_devices.c +3 -1
@@ -1767,6 +1781,7 @@ static int mpam_save_mbwu_state(void *arg)
  {
  	int i;
  	u64 val;
+	int ret;
  	struct mon_cfg *cfg;
  	u32 cur_flt, cur_ctl, mon_sel;
  	struct mpam_msc_ris *ris = arg;
@@ -1777,7 +1792,8 @@ static int mpam_save_mbwu_state(void *arg)
  		mbwu_state = &ris->mbwu_state[i];
  		cfg = &mbwu_state->cfg;
 
-		if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc)))
+		ACQUIRE(mon_sel_lock, guard)(msc);
+		if (ACQUIRE_ERR(mon_sel_lock, &guard))
  			return -EIO;
 
  		mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) |

drivers/resctrl/mpam_devices.c static int mpam_save_mbwu_state(void *arg) · line 1812

Quoted 10 lines
drivers/resctrl/mpam_devices.c +3 -1
@@ -1788,7 +1804,9 @@ static int mpam_save_mbwu_state(void *arg)
  		mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
 
  		if (mpam_ris_has_mbwu_long_counter(ris)) {
-			val = mpam_msc_read_mbwu_l(msc);
+			ret = mpam_msc_read_mbwu_l(msc, &val);
+			if (ret)
+				return ret;
  			mpam_msc_zero_mbwu_l(msc);
  		} else {
  			u32 val32;

I think checking for (val & MSMON____L_NRDRY) is still required before updating the saved state. mpam_msc_read_mbw_l() can return 0 while setting val to MSMON___L_NRDY when it cannot obtain a stable value. The state saving path later adds val to mbwu_state->correction, so the sentinel would corrupt the saved correction. Additionally, as Ben noted during the v3 review, hardware may also set the NRDY bit.

Quoted 8 lines
drivers/resctrl/mpam_devices.c +0 -1
@@ -1804,7 +1822,6 @@ static int mpam_save_mbwu_state(void *arg)
  		cfg->partid = FIELD_GET(MSMON_CFG_x_FLT_PARTID, cur_flt);
  		mbwu_state->correction += val;
  		mbwu_state->enabled = FIELD_GET(MSMON_CFG_x_CTL_EN, cur_ctl);
-		mpam_mon_sel_unlock(msc);
  	}
 
  	return 0;
maintainer
#7 lore raw

Re: [PATCH v5 03/10] arm_mpam: propagate MSC access errors for MBWU counters

Hi Lee,

Quoted 25 lines · Lee Trager
Quoted 23 lines · Andre Przywara
drivers/resctrl/mpam_devices.c +3 -0
@@ -1767,6 +1781,7 @@ static int mpam_save_mbwu_state(void *arg)
   {
       int i;
       u64 val;
+    int ret;
       struct mon_cfg *cfg;
       u32 cur_flt, cur_ctl, mon_sel;
       struct mpam_msc_ris *ris = arg;
@@ -1777,7 +1792,8 @@ static int mpam_save_mbwu_state(void *arg)
           mbwu_state = &ris->mbwu_state[i];
           cfg = &mbwu_state->cfg;
   -        if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc)))
+        ACQUIRE(mon_sel_lock, guard)(msc);
+        if (ACQUIRE_ERR(mon_sel_lock, &guard))
               return -EIO;

            mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) |

drivers/resctrl/mpam_devices.c +3 -1
@@ -1788,7 +1804,9 @@ static int mpam_save_mbwu_state(void *arg)
           mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
             if (mpam_ris_has_mbwu_long_counter(ris)) {
-            val = mpam_msc_read_mbwu_l(msc);
+            ret = mpam_msc_read_mbwu_l(msc, &val);
+            if (ret)
+                return ret;
Quoted 8 lines

drivers/resctrl/mpam_devices.c static int mpam_save_mbwu_state(void *arg) · line 1812

Quoted 3 lines
drivers/resctrl/mpam_devices.c +0 -0
               mpam_msc_zero_mbwu_l(msc);
           } else {
               u32 val32;

I think checking for (val & MSMON____L_NRDRY) is still required before updating the saved state. mpam_msc_read_mbw_l() can return 0 while setting val to MSMON___L_NRDY when it cannot obtain a stable value. The state saving path later adds val to mbwu_state->correction, so the sentinel would corrupt the saved correction. Additionally, as Ben noted during the v3 review, hardware may also set the NRDY bit.

This is an existing issue which I address in this patch:
https://lore.kernel.org/linux-arm-kernel/[email protected]/

Please can you check and see if that looks good to you.

Thanks,

Ben

Quoted 7 lines
Quoted 7 lines
drivers/resctrl/mpam_devices.c +0 -1
@@ -1804,7 +1822,6 @@ static int mpam_save_mbwu_state(void *arg)
           cfg->partid = FIELD_GET(MSMON_CFG_x_FLT_PARTID, cur_flt);
           mbwu_state->correction += val;
           mbwu_state->enabled = FIELD_GET(MSMON_CFG_x_CTL_EN, cur_ctl);
-        mpam_mon_sel_unlock(msc);
       }
         return 0;
maintainer Reviewed-by on v5 3/10
#8 lore raw

Re: [PATCH v5 03/10] arm_mpam: propagate MSC access errors for MBWU counters

In reply to Andre Przywara

Hi Andre,

Quoted 6 lines · Andre Przywara

Allow the mpam_msc_read_mbwu_l() function to return an error, and propagate errors from the lower level up. This also changes the users of this function.

Thanks,

Ben

#9 lore raw

PATCH v5 4/10 arm_mpam: propagate MSC access errors for msmon helpers

In reply to Andre Przywara

Allow the helper functions for msmon accesses to return an error, and propagate read and write errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 72 ++++++++++++++++++++++------------
 1 file changed, 47 insertions(+), 25 deletions(-)
drivers/resctrl/mpam_devices.c +47 -25
index 32ad1864f70f..9d9f0c635527 100644
@@ -1170,15 +1170,20 @@ static int mpam_msc_read_mbwu_l(struct mpam_msc *msc, u64 *res)
 	return 0;
 }
 
-static void mpam_msc_zero_mbwu_l(struct mpam_msc *msc)
+static int mpam_msc_zero_mbwu_l(struct mpam_msc *msc)
 {
+	int ret;
+
 	mpam_mon_sel_lock_held(msc);
 
 	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
-	__mpam_write_reg(msc, MSMON_MBWU_L, 0);
-	__mpam_write_reg(msc, MSMON_MBWU_L + 4, 0);
+	ret = __mpam_write_reg(msc, MSMON_MBWU_L, 0);
+	if (ret)
+		return ret;
+
+	return __mpam_write_reg(msc, MSMON_MBWU_L + 4, 0);
 }
 
 static void gen_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val,
@@ -1221,24 +1226,28 @@ static void gen_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val,
 	}
 }
 
-static void read_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val,
-				    u32 *flt_val)
+static int read_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val,
+				   u32 *flt_val)
 {
 	struct mpam_msc *msc = m->ris->vmsc->msc;
+	int ret;
 
 	switch (m->type) {
 	case mpam_feat_msmon_csu:
-		mpam_read_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
-		mpam_read_monsel_reg(msc, CFG_CSU_FLT, flt_val);
-		break;
+		ret = mpam_read_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
+		if (ret)
+			return ret;
+		return mpam_read_monsel_reg(msc, CFG_CSU_FLT, flt_val);
 	case mpam_feat_msmon_mbwu_31counter:
 	case mpam_feat_msmon_mbwu_44counter:
 	case mpam_feat_msmon_mbwu_63counter:
-		mpam_read_monsel_reg(msc, CFG_MBWU_CTL, ctl_val);
-		mpam_read_monsel_reg(msc, CFG_MBWU_FLT, flt_val);
-		break;
+		ret = mpam_read_monsel_reg(msc, CFG_MBWU_CTL, ctl_val);
+		if (ret)
+			return ret;
+		return mpam_read_monsel_reg(msc, CFG_MBWU_FLT, flt_val);
 	default:
 		pr_warn("Unexpected monitor type %d\n", m->type);
+		return -EINVAL;
 	}
 }
 
@@ -1251,10 +1260,11 @@ static inline void clean_msmon_ctl_val(u32 *cur_ctl)
 		*cur_ctl &= ~MSMON_CFG_MBWU_CTL_OFLOW_STATUS_L;
 }
 
-static void write_msmon_ctl_flt_vals(struct mon_read *m, u32 ctl_val,
-				     u32 flt_val)
+static int write_msmon_ctl_flt_vals(struct mon_read *m, u32 ctl_val,
+				    u32 flt_val)
 {
 	struct mpam_msc *msc = m->ris->vmsc->msc;
+	int ret;
 
 	/*
 	 * Write the ctl_val with the enable bit cleared, reset the counter,
@@ -1262,25 +1272,37 @@ static void write_msmon_ctl_flt_vals(struct mon_read *m, u32 ctl_val,
 	 */
 	switch (m->type) {
 	case mpam_feat_msmon_csu:
-		mpam_write_monsel_reg(msc, CFG_CSU_FLT, flt_val);
-		mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
-		mpam_write_monsel_reg(msc, CSU, 0);
-		mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val | MSMON_CFG_x_CTL_EN);
-		break;
+		ret = mpam_write_monsel_reg(msc, CFG_CSU_FLT, flt_val);
+		if (ret)
+			return ret;
+		ret = mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
+		if (ret)
+			return ret;
+		ret = mpam_write_monsel_reg(msc, CSU, 0);
+		if (ret)
+			return ret;
+		return mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val | MSMON_CFG_x_CTL_EN);
 	case mpam_feat_msmon_mbwu_31counter:
 	case mpam_feat_msmon_mbwu_44counter:
 	case mpam_feat_msmon_mbwu_63counter:
-		mpam_write_monsel_reg(msc, CFG_MBWU_FLT, flt_val);
-		mpam_write_monsel_reg(msc, CFG_MBWU_CTL, ctl_val);
-		mpam_write_monsel_reg(msc, CFG_MBWU_CTL, ctl_val | MSMON_CFG_x_CTL_EN);
+		ret = mpam_write_monsel_reg(msc, CFG_MBWU_FLT, flt_val);
+		if (ret)
+			return ret;
+		ret = mpam_write_monsel_reg(msc, CFG_MBWU_CTL, ctl_val);
+		if (ret)
+			return ret;
+		ret = mpam_write_monsel_reg(msc, CFG_MBWU_CTL,
+					    ctl_val | MSMON_CFG_x_CTL_EN);
+		if (ret)
+			return ret;
 		/* Counting monitors require NRDY to be reset by software */
 		if (m->type == mpam_feat_msmon_mbwu_31counter)
-			mpam_write_monsel_reg(msc, MBWU, 0);
-		else
-			mpam_msc_zero_mbwu_l(m->ris->vmsc->msc);
-		break;
+			return mpam_write_monsel_reg(msc, MBWU, 0);
+
+		return mpam_msc_zero_mbwu_l(m->ris->vmsc->msc);
 	default:
 		pr_warn("Unexpected monitor type %d\n", m->type);
+		return -EINVAL;
 	}
 }
 
Signature · 1 line
2.43.0
#11 lore raw

PATCH v5 5/10 arm_mpam: propagate MSC access errors for __ris_msmon_read()

In reply to Andre Przywara

Allow the function for RIS accesses to return an error, and propagate read and write errors from the lower level up.
This function is a bit special, since it can be called via an SMP helper, so the prototype must stay as void. So propagate any errors using the "err" field in the struct passed in as the sole argument to that function.

---
 drivers/resctrl/mpam_devices.c | 36 +++++++++++++++++++++++++---------
 1 file changed, 27 insertions(+), 9 deletions(-)
drivers/resctrl/mpam_devices.c +27 -9
index 9d9f0c635527..44eafef1e7ab 100644
@@ -1333,6 +1333,10 @@ static u64 mpam_msmon_overflow_val(enum mpam_device_features type,
 	return overflow_val;
 }
 
+/*
+ * This function might be called via smp_call_function_any(), so propagate
+ * errors inside the arg struct.
+ */
 static void __ris_msmon_read(void *arg)
 {
 	u64 now;
@@ -1356,7 +1360,9 @@ static void __ris_msmon_read(void *arg)
 
 	mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, ctx->mon) |
 		  FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx);
-	mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
+	m->err = mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
+	if (m->err)
+		return;
 
 	switch (m->type) {
 	case mpam_feat_msmon_mbwu_31counter:
@@ -1376,7 +1382,9 @@ static void __ris_msmon_read(void *arg)
 	 * Read the existing configuration to avoid re-writing the same values.
 	 * This saves waiting for 'nrdy' on subsequent reads.
 	 */
-	read_msmon_ctl_flt_vals(m, &cur_ctl, &cur_flt);
+	m->err = read_msmon_ctl_flt_vals(m, &cur_ctl, &cur_flt);
+	if (m->err)
+		return;
 
 	if (mpam_feat_msmon_mbwu_31counter == m->type)
 		overflow = cur_ctl & MSMON_CFG_x_CTL_OFLOW_STATUS;
@@ -1390,18 +1398,24 @@ static void __ris_msmon_read(void *arg)
 			  cur_ctl != (ctl_val | MSMON_CFG_x_CTL_EN);
 
 	if (config_mismatch || reset_on_next_read) {
-		write_msmon_ctl_flt_vals(m, ctl_val, flt_val);
+		m->err = write_msmon_ctl_flt_vals(m, ctl_val, flt_val);
+		if (m->err)
+			return;
 		overflow = false;
 	} else if (overflow) {
-		mpam_write_monsel_reg(msc, CFG_MBWU_CTL,
-				      cur_ctl &
-				      ~(MSMON_CFG_x_CTL_OFLOW_STATUS |
-					MSMON_CFG_MBWU_CTL_OFLOW_STATUS_L));
+		m->err = mpam_write_monsel_reg(msc, CFG_MBWU_CTL,
+					    cur_ctl &
+					    ~(MSMON_CFG_x_CTL_OFLOW_STATUS |
+					    MSMON_CFG_MBWU_CTL_OFLOW_STATUS_L));
+		if (m->err)
+			return;
 	}
 
 	switch (m->type) {
 	case mpam_feat_msmon_csu:
-		mpam_read_monsel_reg(msc, CSU, &now32);
+		m->err = mpam_read_monsel_reg(msc, CSU, &now32);
+		if (m->err)
+			return;
 		nrdy = now32 & MSMON___NRDY;
 		now = FIELD_GET(MSMON___VALUE, now32);
 
@@ -1423,7 +1437,9 @@ static void __ris_msmon_read(void *arg)
 			else
 				now = FIELD_GET(MSMON___L_VALUE, now);
 		} else {
-			mpam_read_monsel_reg(msc, MBWU, &now32);
+			m->err = mpam_read_monsel_reg(msc, MBWU, &now32);
+			if (m->err)
+				return;
 			nrdy = now32 & MSMON___NRDY;
 			now = FIELD_GET(MSMON___VALUE, now32);
 		}
@@ -1792,6 +1808,8 @@ static int mpam_restore_mbwu_state(void *_ris)
 			mwbu_arg.val = &val;
 
 			__ris_msmon_read(&mwbu_arg);
+			if (mwbu_arg.err)
+				return mwbu_arg.err;
 		}
 	}
 
Signature · 1 line
2.43.0
maintainer
#12 lore raw

Re: [PATCH v5 05/10] arm_mpam: propagate MSC access errors for __ris_msmon_read()

Hi Andre,

Quoted 93 lines · Andre Przywara

Allow the function for RIS accesses to return an error, and propagate read and write errors from the lower level up.
This function is a bit special, since it can be called via an SMP helper, so the prototype must stay as void. So propagate any errors using the "err" field in the struct passed in as the sole argument to that function.

---
 drivers/resctrl/mpam_devices.c | 36 +++++++++++++++++++++++++---------
 1 file changed, 27 insertions(+), 9 deletions(-)
drivers/resctrl/mpam_devices.c +25 -9
index 9d9f0c635527..44eafef1e7ab 100644
@@ -1333,6 +1333,10 @@ static u64 mpam_msmon_overflow_val(enum mpam_device_features type,
 	return overflow_val;
 }
 
+/*
+ * This function might be called via smp_call_function_any(), so propagate
+ * errors inside the arg struct.
+ */
 static void __ris_msmon_read(void *arg)
 {
 	u64 now;
@@ -1356,7 +1360,9 @@ static void __ris_msmon_read(void *arg)
 
 	mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, ctx->mon) |
 		  FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx);
-	mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
+	m->err = mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
+	if (m->err)
+		return;
 
 	switch (m->type) {
 	case mpam_feat_msmon_mbwu_31counter:
@@ -1376,7 +1382,9 @@ static void __ris_msmon_read(void *arg)
 	 * Read the existing configuration to avoid re-writing the same values.
 	 * This saves waiting for 'nrdy' on subsequent reads.
 	 */
-	read_msmon_ctl_flt_vals(m, &cur_ctl, &cur_flt);
+	m->err = read_msmon_ctl_flt_vals(m, &cur_ctl, &cur_flt);
+	if (m->err)
+		return;
 
 	if (mpam_feat_msmon_mbwu_31counter == m->type)
 		overflow = cur_ctl & MSMON_CFG_x_CTL_OFLOW_STATUS;
@@ -1390,18 +1398,24 @@ static void __ris_msmon_read(void *arg)
 			  cur_ctl != (ctl_val | MSMON_CFG_x_CTL_EN);
 
 	if (config_mismatch || reset_on_next_read) {
-		write_msmon_ctl_flt_vals(m, ctl_val, flt_val);
+		m->err = write_msmon_ctl_flt_vals(m, ctl_val, flt_val);
+		if (m->err)
+			return;
 		overflow = false;
 	} else if (overflow) {
-		mpam_write_monsel_reg(msc, CFG_MBWU_CTL,
-				      cur_ctl &
-				      ~(MSMON_CFG_x_CTL_OFLOW_STATUS |
-					MSMON_CFG_MBWU_CTL_OFLOW_STATUS_L));
+		m->err = mpam_write_monsel_reg(msc, CFG_MBWU_CTL,
+					    cur_ctl &
+					    ~(MSMON_CFG_x_CTL_OFLOW_STATUS |
+					    MSMON_CFG_MBWU_CTL_OFLOW_STATUS_L));
+		if (m->err)
+			return;
 	}
 
 	switch (m->type) {
 	case mpam_feat_msmon_csu:
-		mpam_read_monsel_reg(msc, CSU, &now32);
+		m->err = mpam_read_monsel_reg(msc, CSU, &now32);
+		if (m->err)
+			return;
 		nrdy = now32 & MSMON___NRDY;
 		now = FIELD_GET(MSMON___VALUE, now32);
 
@@ -1423,7 +1437,9 @@ static void __ris_msmon_read(void *arg)
 			else
 				now = FIELD_GET(MSMON___L_VALUE, now);
 		} else {
-			mpam_read_monsel_reg(msc, MBWU, &now32);
+			m->err = mpam_read_monsel_reg(msc, MBWU, &now32);
+			if (m->err)
+				return;
 			nrdy = now32 & MSMON___NRDY;
 			now = FIELD_GET(MSMON___VALUE, now32);
 		}

drivers/resctrl/mpam_devices.c static int mpam_restore_mbwu_state(void *_ris) · line 1812

Quoted 5 lines
drivers/resctrl/mpam_devices.c +2 -0
@@ -1792,6 +1808,8 @@ static int mpam_restore_mbwu_state(void *_ris)
 			mwbu_arg.val = &val;
 
 			__ris_msmon_read(&mwbu_arg);
+			if (mwbu_arg.err)
+				return mwbu_arg.err;

As sashiko points out [1], this is a change of behaviour as mbwu_arg.err could already be non-zero previous to this patch. In particular, -EBUSY when NRDY is set. When this is the case we should continue restoring the other counters rather than changing to bailing out early.

As -EBUSY is only used in the NRDY case you could change the condition to be:

if (mbwu_arg.err && mbwu_arg.err != -EBUSY)

         return mwbu_arg.err;

[1] https://sashiko.dev/#/patchset/20260729134124.2506269-1-andre.przywara%40arm.com

Thanks,

Ben

Quoted 3 lines
drivers/resctrl/mpam_devices.c +0 -0
 		}
 	}
 
#13 lore raw

PATCH v5 6/10 arm_mpam: propagate MSC access errors for state saving function

In reply to Andre Przywara

Allow the mpam_save_mbwu_state() function to return an error, and propagate read and write errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 29 ++++++++++++++++++++++-------
 1 file changed, 22 insertions(+), 7 deletions(-)
drivers/resctrl/mpam_devices.c +22 -7
index 44eafef1e7ab..9b6d266b4fa6 100644
@@ -1838,22 +1838,37 @@ static int mpam_save_mbwu_state(void *arg)
 
 		mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) |
 			  FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx);
-		mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
-		mpam_read_monsel_reg(msc, CFG_MBWU_FLT, &cur_flt);
-		mpam_read_monsel_reg(msc, CFG_MBWU_CTL, &cur_ctl);
-		mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
+		ret = mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
+		if (ret)
+			return ret;
+		ret = mpam_read_monsel_reg(msc, CFG_MBWU_FLT, &cur_flt);
+		if (ret)
+			return ret;
+		ret = mpam_read_monsel_reg(msc, CFG_MBWU_CTL, &cur_ctl);
+		if (ret)
+			return ret;
+		ret = mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
+		if (ret)
+			return ret;
 
 		if (mpam_ris_has_mbwu_long_counter(ris)) {
 			ret = mpam_msc_read_mbwu_l(msc, &val);
 			if (ret)
 				return ret;
-			mpam_msc_zero_mbwu_l(msc);
+			ret = mpam_msc_zero_mbwu_l(msc);
+			if (ret)
+				return ret;
 		} else {
 			u32 val32;
 
-			mpam_read_monsel_reg(msc, MBWU, &val32);
+			ret = mpam_read_monsel_reg(msc, MBWU, &val32);
+			if (ret)
+				return ret;
+
 			val = val32;
-			mpam_write_monsel_reg(msc, MBWU, 0);
+			ret = mpam_write_monsel_reg(msc, MBWU, 0);
+			if (ret)
+				return ret;
 		}
 
 		cfg->mon = i;
Signature · 1 line
2.43.0
#15 lore raw

PATCH v5 7/10 arm_mpam: prepare mon_sel locking for MPAM-Fb

In reply to Andre Przywara

The MSC MON_SEL register needs to be accessed from hardirq for the overflow interrupt, and when taking an IPI to access these registers on platforms where MSCs are not accesible from every CPU. This makes an irqsave spinlock the obvious lock to protect these registers. On systems with MPAM-Fb mailbox MSC access it must be able to sleep, meaning a mutex must be used. So MPAM-Fb platforms cannot support an overflow interrupt easily. Clearly these two methods can't exist for one MSC at the same time.

Change the mon_sel locking wrapper function to only use a spinlock when the MSC is accessed directly via MMIO. In case of MPAM-Fb, we use a mutex, but only if we are in a sleepable context. If that's not the case, we return an error. This should not happen, as MPAM-Fb by design does not require an MSC access to happen from a specific CPU, so there is no need for any IPIs or preemption disabling to satisfy CPU constraints. And since overflow interrupts are not supported at the moment anyway, we also wouldn't meet the other case.
Bailing out early is already happening in rare occasions today.

---
 drivers/resctrl/mpam_devices.c  |  5 ++++-
 drivers/resctrl/mpam_internal.h | 38 +++++++++++++++++++++++++++------
 2 files changed, 35 insertions(+), 8 deletions(-)
drivers/resctrl/mpam_devices.c +4 -1
index 9b6d266b4fa6..f6910ab3bbc2 100644
@@ -2230,7 +2230,10 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
 	if (err)
 		return ERR_PTR(err);
 
-	mpam_mon_sel_lock_init(msc);
+	err = mpam_mon_sel_lock_init(dev, msc);
+	if (err)
+		return ERR_PTR(err);
+
 	msc->id = pdev->id;
 	msc->pdev = pdev;
 	INIT_LIST_HEAD_RCU(&msc->all_msc_list);
drivers/resctrl/mpam_internal.h +31 -7
index 57f2b3dbf7d0..2b81b6b0bf4e 100644
@@ -126,6 +126,7 @@ struct mpam_msc {
 	 */
 	raw_spinlock_t		_mon_sel_lock;
 	unsigned long		_mon_sel_flags;
+	struct mutex		mon_sel_mutex;
 
 	void __iomem		*mapped_hwpage;
 	size_t			mapped_hwpage_sz;
@@ -139,27 +140,50 @@ struct mpam_msc {
 /* Returning false here means accesses to mon_sel must fail and report an error. */
 static inline bool __must_check mpam_mon_sel_lock(struct mpam_msc *msc)
 {
-	/* Locking will require updating to support a firmware backed interface */
-	if (WARN_ON_ONCE(msc->iface != MPAM_IFACE_MMIO))
+	if (msc->iface == MPAM_IFACE_MMIO) {
+		raw_spin_lock_irqsave(&msc->_mon_sel_lock, msc->_mon_sel_flags);
+
+		return true;
+	}
+
+	if (!preemptible())
 		return false;
 
-	raw_spin_lock_irqsave(&msc->_mon_sel_lock, msc->_mon_sel_flags);
+	mutex_lock(&msc->mon_sel_mutex);
+
 	return true;
 }
 
 static inline void mpam_mon_sel_unlock(struct mpam_msc *msc)
 {
-	raw_spin_unlock_irqrestore(&msc->_mon_sel_lock, msc->_mon_sel_flags);
+	if (msc->iface == MPAM_IFACE_MMIO) {
+		raw_spin_unlock_irqrestore(&msc->_mon_sel_lock,
+					   msc->_mon_sel_flags);
+
+		return;
+	}
+
+	mutex_unlock(&msc->mon_sel_mutex);
 }
 
 static inline void mpam_mon_sel_lock_held(struct mpam_msc *msc)
 {
-	lockdep_assert_held_once(&msc->_mon_sel_lock);
+	if (msc->iface == MPAM_IFACE_MMIO)
+		lockdep_assert_held_once(&msc->_mon_sel_lock);
+	else
+		lockdep_assert_held_once(&msc->mon_sel_mutex);
 }
 
-static inline void mpam_mon_sel_lock_init(struct mpam_msc *msc)
+static inline int mpam_mon_sel_lock_init(struct device *dev,
+					 struct mpam_msc *msc)
 {
-	raw_spin_lock_init(&msc->_mon_sel_lock);
+	if (msc->iface == MPAM_IFACE_MMIO) {
+		raw_spin_lock_init(&msc->_mon_sel_lock);
+
+		return 0;
+	}
+
+	return devm_mutex_init(dev, &msc->mon_sel_mutex);
 }
 
 DEFINE_GUARD(mon_sel, struct mpam_msc *,
Signature · 1 line
2.43.0
#16 lore raw

PATCH v5 8/10 arm_mpam: add MPAM-Fb MSC firmware access support

In reply to Andre Przywara

The Arm MPAM Firmware-backed (Fb) Profile document[1] describes an alternative way of accessing the "Memory System Components" (MSC) in an MPAM enabled system.

Normally the MSCs are MMIO mapped, but in some implementations this might not be possible (MSC located outside of the local socket, MSC mapped secure-only) or desirable (direct MMIO access too slow or needs to be mediated through a control processor). MPAM-fb standardises a protocol to abstract MSC accesses, building on the SCMI protocol.

Add functions that do an MSC read or write access by redirecting the request through a firmware interface. For now this done via an ACPI PCC shared memory and mailbox combination.

Since the protocol used is only a small subset of the full SCMI spec, and the SCMI protocol has no full ACPI support anyway, open-code the (simple) SCMI message generation, for just the fields we need.

[1] https://developer.arm.com/documentation/den0144/latest

---
 drivers/resctrl/Makefile        |   2 +-
 drivers/resctrl/mpam_devices.c  |  52 ++++++--
 drivers/resctrl/mpam_fb.c       | 214 ++++++++++++++++++++++++++++++++
 drivers/resctrl/mpam_internal.h |  22 ++++
 include/linux/arm_mpam.h        |   2 +-
 5 files changed, 279 insertions(+), 13 deletions(-)
 create mode 100644 drivers/resctrl/mpam_fb.c
drivers/resctrl/Makefile +1 -1
index 4f6d0e81f9b8..097c036724e9 100644
@@ -1,5 +1,5 @@
 obj-$(CONFIG_ARM64_MPAM_DRIVER)			+= mpam.o
-mpam-y						+= mpam_devices.o
+mpam-y						+= mpam_devices.o mpam_fb.o
 mpam-$(CONFIG_ARM64_MPAM_RESCTRL_FS)		+= mpam_resctrl.o
 
 ccflags-$(CONFIG_ARM64_MPAM_DRIVER_DEBUG)	+= -DDEBUG
drivers/resctrl/mpam_devices.c +41 -11

97 lines not on this page · raw message

drivers/resctrl/mpam_fb.c new file +214 -0

214 lines not on this page · raw message

drivers/resctrl/mpam_internal.h +22 -0
index 2b81b6b0bf4e..a2193e7df57c 100644
@@ -11,6 +11,7 @@
 #include <linux/io.h>
 #include <linux/jump_label.h>
 #include <linux/llist.h>
+#include <linux/mailbox_client.h>
 #include <linux/mutex.h>
 #include <linux/resctrl.h>
 #include <linux/spinlock.h>
@@ -57,6 +58,15 @@ struct mpam_garbage {
 	struct platform_device	*pdev;
 };
 
+struct mpam_pcc_chan {
+	struct list_head	pcc_chans;
+	struct mbox_client	pcc_cl;
+	struct pcc_mbox_chan	*pcc_chan;
+	struct mutex		pcc_chan_lock; /* only one message at a time */
+	struct kref		refcount;
+	int			subspace_id;
+};
+
 struct mpam_msc {
 	/* member of mpam_all_msc */
 	struct list_head	all_msc_list;
@@ -66,6 +76,8 @@ struct mpam_msc {
 
 	/* Not modified after mpam_is_enabled() becomes true */
 	enum mpam_msc_iface	iface;
+	struct mpam_pcc_chan	*pcc_chan;
+	int			mpam_fb_msc_id;	/* in its own name space */
 	u32			nrdy_usec;
 	cpumask_t		accessibility;
 	bool			has_extd_esr;
@@ -484,6 +496,9 @@ extern u8 mpam_pmg_max;
 void mpam_enable(struct work_struct *work);
 void mpam_disable(struct work_struct *work);
 
+/* helper function to call from outside mpam_devices.c */
+void mpam_fb_disable_mpam(int err);
+
 /* Reset all the RIS in a class under cpus_read_lock() */
 void mpam_reset_class_locked(struct mpam_class *class);
 
@@ -511,6 +526,13 @@ static inline void mpam_resctrl_offline_cpu(unsigned int cpu) { }
 static inline void mpam_resctrl_teardown_class(struct mpam_class *class) { }
 #endif /* CONFIG_RESCTRL_FS */
 
+/* MPAM-Fb Firmware-backed protocol wrappers */
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result);
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value);
+int mpam_fb_get_protocol_version(struct mpam_msc *msc);
+
+#define MPAM_FB_PROT_HEADER_LEN		sizeof(u32)
+
 /*
  * MPAM MSCs have the following register layout. See:
  * Arm Memory System Resource Partitioning and Monitoring (MPAM) System
include/linux/arm_mpam.h +1 -1
index f92a36187a52..002f56e15362 100644
@@ -12,7 +12,7 @@ struct mpam_msc;
 
 enum mpam_msc_iface {
 	MPAM_IFACE_MMIO,	/* a real MPAM MSC */
-	MPAM_IFACE_PCC,		/* a fake MPAM MSC */
+	MPAM_IFACE_PCC,		/* using the MPAM-Fb firmware redirection */
 };
 
 enum mpam_class_types {
Signature · 1 line
2.43.0
maintainer
#17 lore raw

Re: [PATCH v5 08/10] arm_mpam: add MPAM-Fb MSC firmware access support

Hi Andre,

Quoted 21 lines · Andre Przywara

The Arm MPAM Firmware-backed (Fb) Profile document[1] describes an alternative way of accessing the "Memory System Components" (MSC) in an MPAM enabled system.

Normally the MSCs are MMIO mapped, but in some implementations this might not be possible (MSC located outside of the local socket, MSC mapped secure-only) or desirable (direct MMIO access too slow or needs to be mediated through a control processor). MPAM-fb standardises a protocol to abstract MSC accesses, building on the SCMI protocol.

Add functions that do an MSC read or write access by redirecting the request through a firmware interface. For now this done via an ACPI PCC shared memory and mailbox combination.

Since the protocol used is only a small subset of the full SCMI spec, and the SCMI protocol has no full ACPI support anyway, open-code the (simple) SCMI message generation, for just the fields we need.

[1] https://developer.arm.com/documentation/den0144/latest

[...]

Quoted 29 lines
drivers/resctrl/mpam_internal.h +10 -0
index 2b81b6b0bf4e..a2193e7df57c 100644
@@ -11,6 +11,7 @@
 #include <linux/io.h>
 #include <linux/jump_label.h>
 #include <linux/llist.h>
+#include <linux/mailbox_client.h>
 #include <linux/mutex.h>
 #include <linux/resctrl.h>
 #include <linux/spinlock.h>
@@ -57,6 +58,15 @@ struct mpam_garbage {
 	struct platform_device	*pdev;
 };
 
+struct mpam_pcc_chan {
+	struct list_head	pcc_chans;
+	struct mbox_client	pcc_cl;
+	struct pcc_mbox_chan	*pcc_chan;
+	struct mutex		pcc_chan_lock; /* only one message at a time */
+	struct kref		refcount;
+	int			subspace_id;
+};
+
 struct mpam_msc {
 	/* member of mpam_all_msc */
 	struct list_head	all_msc_list;

drivers/resctrl/mpam_internal.h struct mpam_msc { · line 80

Quoted 5 lines
drivers/resctrl/mpam_internal.h +2 -0
@@ -66,6 +76,8 @@ struct mpam_msc {
 
 	/* Not modified after mpam_is_enabled() becomes true */
 	enum mpam_msc_iface	iface;
+	struct mpam_pcc_chan	*pcc_chan;
+	int			mpam_fb_msc_id;	/* in its own name space */

This gets removed in patch 10 now. I guess you just edited the wrong patch.

Thanks,

Ben

#18 lore raw

Re: [PATCH v5 08/10] arm_mpam: add MPAM-Fb MSC firmware access support

Hi Ben,

thanks for having a look!

Quoted 57 lines · Ben Horgan

Hi Andre,

Quoted 21 lines · Andre Przywara

The Arm MPAM Firmware-backed (Fb) Profile document[1] describes an alternative way of accessing the "Memory System Components" (MSC) in an MPAM enabled system.

Normally the MSCs are MMIO mapped, but in some implementations this might not be possible (MSC located outside of the local socket, MSC mapped secure-only) or desirable (direct MMIO access too slow or needs to be mediated through a control processor). MPAM-fb standardises a protocol to abstract MSC accesses, building on the SCMI protocol.

Add functions that do an MSC read or write access by redirecting the request through a firmware interface. For now this done via an ACPI PCC shared memory and mailbox combination.

Since the protocol used is only a small subset of the full SCMI spec, and the SCMI protocol has no full ACPI support anyway, open-code the (simple) SCMI message generation, for just the fields we need.

[1] https://developer.arm.com/documentation/den0144/latest

[...]

Quoted 30 lines
drivers/resctrl/mpam_internal.h +10 -0
index 2b81b6b0bf4e..a2193e7df57c 100644
@@ -11,6 +11,7 @@
  #include <linux/io.h>
  #include <linux/jump_label.h>
  #include <linux/llist.h>
+#include <linux/mailbox_client.h>
  #include <linux/mutex.h>
  #include <linux/resctrl.h>
  #include <linux/spinlock.h>
@@ -57,6 +58,15 @@ struct mpam_garbage {
  	struct platform_device	*pdev;
  };
 
+struct mpam_pcc_chan {
+	struct list_head	pcc_chans;
+	struct mbox_client	pcc_cl;
+	struct pcc_mbox_chan	*pcc_chan;
+	struct mutex		pcc_chan_lock; /* only one message at a time */
+	struct kref		refcount;
+	int			subspace_id;
+};
+
  struct mpam_msc {
  	/* member of mpam_all_msc */
  	struct list_head	all_msc_list;
@@ -66,6 +76,8 @@ struct mpam_msc {
 
Quoted 6 lines

drivers/resctrl/mpam_internal.h struct mpam_msc { · line 80

Quoted 4 lines
drivers/resctrl/mpam_internal.h +2 -0
  	/* Not modified after mpam_is_enabled() becomes true */
  	enum mpam_msc_iface	iface;
+	struct mpam_pcc_chan	*pcc_chan;
+	int			mpam_fb_msc_id;	/* in its own name space */

This gets removed in patch 10 now. I guess you just edited the wrong patch.

Argh, you are right, I squashed the fix into 10/10, but indeed the member is already introduced here. Good catch!

Cheers,
Andre

maintainer
#19 lore raw

Re: [PATCH v5 08/10] arm_mpam: add MPAM-Fb MSC firmware access support

In reply to Andre Przywara

Hi Andre,

Quoted 59 lines · Andre Przywara

The Arm MPAM Firmware-backed (Fb) Profile document[1] describes an alternative way of accessing the "Memory System Components" (MSC) in an MPAM enabled system.

Normally the MSCs are MMIO mapped, but in some implementations this might not be possible (MSC located outside of the local socket, MSC mapped secure-only) or desirable (direct MMIO access too slow or needs to be mediated through a control processor). MPAM-fb standardises a protocol to abstract MSC accesses, building on the SCMI protocol.

Add functions that do an MSC read or write access by redirecting the request through a firmware interface. For now this done via an ACPI PCC shared memory and mailbox combination.

Since the protocol used is only a small subset of the full SCMI spec, and the SCMI protocol has no full ACPI support anyway, open-code the (simple) SCMI message generation, for just the fields we need.

[1] https://developer.arm.com/documentation/den0144/latest

---
 drivers/resctrl/Makefile        |   2 +-
 drivers/resctrl/mpam_devices.c  |  52 ++++++--
 drivers/resctrl/mpam_fb.c       | 214 ++++++++++++++++++++++++++++++++
 drivers/resctrl/mpam_internal.h |  22 ++++
 include/linux/arm_mpam.h        |   2 +-
 5 files changed, 279 insertions(+), 13 deletions(-)
 create mode 100644 drivers/resctrl/mpam_fb.c
drivers/resctrl/Makefile +1 -1
index 4f6d0e81f9b8..097c036724e9 100644
@@ -1,5 +1,5 @@
 obj-$(CONFIG_ARM64_MPAM_DRIVER)			+= mpam.o
-mpam-y						+= mpam_devices.o
+mpam-y						+= mpam_devices.o mpam_fb.o
 mpam-$(CONFIG_ARM64_MPAM_RESCTRL_FS)		+= mpam_resctrl.o
 
 ccflags-$(CONFIG_ARM64_MPAM_DRIVER_DEBUG)	+= -DDEBUG
drivers/resctrl/mpam_devices.c +10 -0
index f6910ab3bbc2..abe1e628928f 100644
@@ -83,6 +83,16 @@ static DECLARE_WORK(mpam_broken_work, &mpam_disable);
 /* When mpam is disabled, the printed reason to aid debugging */
 static char *mpam_disable_reason;
 
+void mpam_fb_disable_mpam(int err)
+{
+	static char mpam_fb_reason[32];
+
+	snprintf(mpam_fb_reason, sizeof(mpam_fb_reason), "MPAM-Fb error %d",
+		 err);
+	mpam_disable_reason = mpam_fb_reason;
+	schedule_work(&mpam_broken_work);
+}
+

[...]

Quoted 71 lines
drivers/resctrl/mpam_devices.c +71 -0
+static int mpam_fb_send_request(struct mpam_pcc_chan *pcc_chan, u32 msc_id,
+				u16 reg, u32 *result, int mpam_fb_command)
+{
+	unsigned int token = atomic_inc_return(&mpam_fb_token);
+	struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem;
+	struct pcc_mbox_chan *chan;
+	void __iomem *payload_ofs;
+	u32 status;
+	int ret;
+
+	if (!pcc_chan)
+		return -ENODEV;
+
+	chan = pcc_chan->pcc_chan;
+
+	/* prune token to fit into the 10 bits inside the command register */
+	token = FIELD_GET(MPAM_MSC_TOKEN_MASK,
+			  FIELD_PREP(MPAM_MSC_TOKEN_MASK, token));
+
+	guard(mutex)(&pcc_chan->pcc_chan_lock);
+
+	switch (mpam_fb_command) {
+	case MPAM_MSC_WRITE_CMD:
+		mpam_fb_build_write_message(msc_id, reg, *result,
+					    token, chan->shmem);
+		break;
+	case MPAM_MSC_READ_CMD:
+		mpam_fb_build_read_message(msc_id, reg, token, chan->shmem);
+		break;
+	case MPAM_PROTOCOL_VERSION_CMD:
+		mpam_fb_build_version_message(token, chan->shmem);
+		break;
+	default:
+		dev_err(pcc_chan->pcc_cl.dev, "unsupported MPAM-Fb command %d\n",
+			mpam_fb_command);
+		ret = -EINVAL;
+		goto out_err;
+	}
+
+	ret = mbox_send_message(chan->mchan, NULL);
+	if (ret < 0)
+		goto out_err;
+
+	pcc_shmem = chan->shmem;
+	payload_ofs = chan->shmem + sizeof(*pcc_shmem);
+	status = readl(&pcc_shmem->command);
+	if (FIELD_GET(MPAM_MSC_TOKEN_MASK, status) != token) {
+		ret = -ETIMEDOUT;
+
+		goto out_err;
+	}
+
+	ret = readl(payload_ofs + 0x0);
+	if (ret < 0) {
+		switch (ret) {
+		case MPAM_FB_ERR_NOT_SUPPORTED:
+			ret = -EOPNOTSUPP;
+			break;
+		case MPAM_FB_ERR_INVALID_PARAMETERS:
+			ret = -EINVAL;
+			break;
+		case MPAM_FB_ERR_NOT_FOUND:
+			ret = -ENOENT;
+			break;
+		case MPAM_FB_ERR_OUT_OF_RANGE:
+			ret = -ERANGE;
+			break;
+		default:
+			ret = -EINVAL;
+		}
+

drivers/resctrl/mpam_devices.c

Quoted 10 lines
drivers/resctrl/mpam_devices.c +10 -0
+		goto out_err;
+	}
+
+	if (mpam_fb_command != MPAM_MSC_WRITE_CMD)
+		*result = readl(payload_ofs + 0x4);
+
+	return 0;
+
+out_err:
+	mpam_fb_disable_mpam(ret);

Given that the 'ret' here is just used to create the log message we could provide more information just by indicating if we timed out waiting for firmware response or by giving the mpam_fb error code.

Thanks,

Ben

#20 lore raw

Re: [PATCH v5 08/10] arm_mpam: add MPAM-Fb MSC firmware access support

Hi Ben,

Quoted 141 lines · Ben Horgan

Hi Andre,

Quoted 59 lines · Andre Przywara

The Arm MPAM Firmware-backed (Fb) Profile document[1] describes an alternative way of accessing the "Memory System Components" (MSC) in an MPAM enabled system.

Normally the MSCs are MMIO mapped, but in some implementations this might not be possible (MSC located outside of the local socket, MSC mapped secure-only) or desirable (direct MMIO access too slow or needs to be mediated through a control processor). MPAM-fb standardises a protocol to abstract MSC accesses, building on the SCMI protocol.

Add functions that do an MSC read or write access by redirecting the request through a firmware interface. For now this done via an ACPI PCC shared memory and mailbox combination.

Since the protocol used is only a small subset of the full SCMI spec, and the SCMI protocol has no full ACPI support anyway, open-code the (simple) SCMI message generation, for just the fields we need.

[1] https://developer.arm.com/documentation/den0144/latest

---
  drivers/resctrl/Makefile        |   2 +-
  drivers/resctrl/mpam_devices.c  |  52 ++++++--
  drivers/resctrl/mpam_fb.c       | 214 ++++++++++++++++++++++++++++++++
  drivers/resctrl/mpam_internal.h |  22 ++++
  include/linux/arm_mpam.h        |   2 +-
  5 files changed, 279 insertions(+), 13 deletions(-)

create mode 100644 drivers/resctrl/mpam_fb.c

drivers/resctrl/Makefile +1 -1
index 4f6d0e81f9b8..097c036724e9 100644
@@ -1,5 +1,5 @@
  obj-$(CONFIG_ARM64_MPAM_DRIVER)			+= mpam.o
-mpam-y						+= mpam_devices.o
+mpam-y						+= mpam_devices.o mpam_fb.o
  mpam-$(CONFIG_ARM64_MPAM_RESCTRL_FS)		+= mpam_resctrl.o
 
  ccflags-$(CONFIG_ARM64_MPAM_DRIVER_DEBUG)	+= -DDEBUG
drivers/resctrl/mpam_devices.c +10 -0
index f6910ab3bbc2..abe1e628928f 100644
@@ -83,6 +83,16 @@ static DECLARE_WORK(mpam_broken_work, &mpam_disable);
  /* When mpam is disabled, the printed reason to aid debugging */
  static char *mpam_disable_reason;
 
+void mpam_fb_disable_mpam(int err)
+{
+	static char mpam_fb_reason[32];
+
+	snprintf(mpam_fb_reason, sizeof(mpam_fb_reason), "MPAM-Fb error %d",
+		 err);
+	mpam_disable_reason = mpam_fb_reason;
+	schedule_work(&mpam_broken_work);
+}
+

[...]

Quoted 76 lines
drivers/resctrl/mpam_devices.c +76 -0
+static int mpam_fb_send_request(struct mpam_pcc_chan *pcc_chan, u32 msc_id,
+				u16 reg, u32 *result, int mpam_fb_command)
+{
+	unsigned int token = atomic_inc_return(&mpam_fb_token);
+	struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem;
+	struct pcc_mbox_chan *chan;
+	void __iomem *payload_ofs;
+	u32 status;
+	int ret;
+
+	if (!pcc_chan)
+		return -ENODEV;
+
+	chan = pcc_chan->pcc_chan;
+
+	/* prune token to fit into the 10 bits inside the command register */
+	token = FIELD_GET(MPAM_MSC_TOKEN_MASK,
+			  FIELD_PREP(MPAM_MSC_TOKEN_MASK, token));
+
+	guard(mutex)(&pcc_chan->pcc_chan_lock);
+
+	switch (mpam_fb_command) {
+	case MPAM_MSC_WRITE_CMD:
+		mpam_fb_build_write_message(msc_id, reg, *result,
+					    token, chan->shmem);
+		break;
+	case MPAM_MSC_READ_CMD:
+		mpam_fb_build_read_message(msc_id, reg, token, chan->shmem);
+		break;
+	case MPAM_PROTOCOL_VERSION_CMD:
+		mpam_fb_build_version_message(token, chan->shmem);
+		break;
+	default:
+		dev_err(pcc_chan->pcc_cl.dev, "unsupported MPAM-Fb command %d\n",
+			mpam_fb_command);
+		ret = -EINVAL;
+		goto out_err;
+	}
+
+	ret = mbox_send_message(chan->mchan, NULL);
+	if (ret < 0)
+		goto out_err;
+
+	pcc_shmem = chan->shmem;
+	payload_ofs = chan->shmem + sizeof(*pcc_shmem);
+	status = readl(&pcc_shmem->command);
+	if (FIELD_GET(MPAM_MSC_TOKEN_MASK, status) != token) {
+		ret = -ETIMEDOUT;
+
+		goto out_err;
+	}
+
+	ret = readl(payload_ofs + 0x0);
+	if (ret < 0) {
+		switch (ret) {
+		case MPAM_FB_ERR_NOT_SUPPORTED:
+			ret = -EOPNOTSUPP;
+			break;
+		case MPAM_FB_ERR_INVALID_PARAMETERS:
+			ret = -EINVAL;
+			break;
+		case MPAM_FB_ERR_NOT_FOUND:
+			ret = -ENOENT;
+			break;
+		case MPAM_FB_ERR_OUT_OF_RANGE:
+			ret = -ERANGE;
+			break;
+		default:
+			ret = -EINVAL;
+		}
+
+		goto out_err;
+	}
+
+	if (mpam_fb_command != MPAM_MSC_WRITE_CMD)
+		*result = readl(payload_ofs + 0x4);
Quoted 8 lines

drivers/resctrl/mpam_devices.c

Quoted 5 lines
drivers/resctrl/mpam_devices.c +5 -0
+
+	return 0;
+
+out_err:
+	mpam_fb_disable_mpam(ret);

Given that the 'ret' here is just used to create the log message we could provide more information just by indicating if we timed out waiting for firmware response or by giving the mpam_fb error code.

I see what you mean. It gets a bit nasty in the details, since mpam_disable_reason is just a pointer, so putting arbitrary strings of unknown length into that message is a bit involved. I changed that now to also pass the untranslated error value here, so both numbers are printed in the logs. If we still feel like it, we can improve on this later, I think.

Cheers,
Andre

#21 lore raw

Re: [PATCH v5 08/10] arm_mpam: add MPAM-Fb MSC firmware access support

In reply to Andre Przywara

Hi Andre,

Quoted 317 lines · Andre Przywara

The Arm MPAM Firmware-backed (Fb) Profile document[1] describes an alternative way of accessing the "Memory System Components" (MSC) in an MPAM enabled system.

Normally the MSCs are MMIO mapped, but in some implementations this might not be possible (MSC located outside of the local socket, MSC mapped secure-only) or desirable (direct MMIO access too slow or needs to be mediated through a control processor). MPAM-fb standardises a protocol to abstract MSC accesses, building on the SCMI protocol.

Add functions that do an MSC read or write access by redirecting the request through a firmware interface. For now this done via an ACPI PCC shared memory and mailbox combination.

Since the protocol used is only a small subset of the full SCMI spec, and the SCMI protocol has no full ACPI support anyway, open-code the (simple) SCMI message generation, for just the fields we need.

[1] https://developer.arm.com/documentation/den0144/latest

---
  drivers/resctrl/Makefile        |   2 +-
  drivers/resctrl/mpam_devices.c  |  52 ++++++--
  drivers/resctrl/mpam_fb.c       | 214 ++++++++++++++++++++++++++++++++
  drivers/resctrl/mpam_internal.h |  22 ++++
  include/linux/arm_mpam.h        |   2 +-
  5 files changed, 279 insertions(+), 13 deletions(-)

create mode 100644 drivers/resctrl/mpam_fb.c

drivers/resctrl/Makefile +1 -1
index 4f6d0e81f9b8..097c036724e9 100644
@@ -1,5 +1,5 @@
  obj-$(CONFIG_ARM64_MPAM_DRIVER)			+= mpam.o
-mpam-y						+= mpam_devices.o
+mpam-y						+= mpam_devices.o mpam_fb.o
  mpam-$(CONFIG_ARM64_MPAM_RESCTRL_FS)		+= mpam_resctrl.o
 
  ccflags-$(CONFIG_ARM64_MPAM_DRIVER_DEBUG)	+= -DDEBUG
drivers/resctrl/mpam_devices.c +41 -11
index f6910ab3bbc2..abe1e628928f 100644
@@ -83,6 +83,16 @@ static DECLARE_WORK(mpam_broken_work, &mpam_disable);
  /* When mpam is disabled, the printed reason to aid debugging */
  static char *mpam_disable_reason;
 
+void mpam_fb_disable_mpam(int err)
+{
+	static char mpam_fb_reason[32];
+
+	snprintf(mpam_fb_reason, sizeof(mpam_fb_reason), "MPAM-Fb error %d",
+		 err);
+	mpam_disable_reason = mpam_fb_reason;
+	schedule_work(&mpam_broken_work);
+}
+
  /*
   * Whether resctrl has been setup. Used by cpuhp in preference to
   * mpam_is_enabled(). The disable call after an error interrupt makes
@@ -179,8 +189,11 @@ static void mpam_assert_partid_sizes_fixed(void)
 
  static int __mpam_read_reg(struct mpam_msc *msc, u16 reg, u32 *res)
  {
-	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
+	if (msc->iface == MPAM_IFACE_PCC)
+		return mpam_fb_send_read_request(msc, reg, res);
+
+	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
  	*res = readl_relaxed(msc->mapped_hwpage + reg);
 
  	return 0;
@@ -197,9 +210,12 @@ static inline int _mpam_read_partsel_reg(struct mpam_msc *msc, u16 reg,
 
  static int __mpam_write_reg(struct mpam_msc *msc, u16 reg, u32 val)
  {
-	WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz);
-	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
+	if (msc->iface == MPAM_IFACE_PCC)
+		return mpam_fb_send_write_request(msc, reg, val);
+
+	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
+	WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz);
  	writel_relaxed(val, msc->mapped_hwpage + reg);
 
  	return 0;
@@ -1141,8 +1157,11 @@ static int mpam_msc_read_mbwu_l(struct mpam_msc *msc, u64 *res)
 
  	mpam_mon_sel_lock_held(msc);
 
-	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
-	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
+	if (msc->iface == MPAM_IFACE_MMIO) {
+		WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
+		WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(),
+			     &msc->accessibility));
+	}
 
  	ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
  	if (ret)
@@ -1176,8 +1195,11 @@ static int mpam_msc_zero_mbwu_l(struct mpam_msc *msc)
 
  	mpam_mon_sel_lock_held(msc);
 
-	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
-	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
+	if (msc->iface == MPAM_IFACE_MMIO) {
+		WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
+		WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(),
+			     &msc->accessibility));
+	}
 
  	ret = __mpam_write_reg(msc, MSMON_MBWU_L, 0);
  	if (ret)
@@ -1490,11 +1512,16 @@ static int _msmon_read(struct mpam_component *comp, struct mon_read *arg)
  					 srcu_read_lock_held(&mpam_srcu)) {
  			arg->ris = ris;
 
-			err = smp_call_function_any(&msc->accessibility,
-						    __ris_msmon_read, arg,
-						    true);
-			if (!err && arg->err)
+			if (msc->iface == MPAM_IFACE_MMIO) {
+				err = smp_call_function_any(&msc->accessibility,
+							    __ris_msmon_read,
+							    arg, true);
+				if (!err)
+					err = arg->err;
+			} else {
+				__ris_msmon_read(arg);
  				err = arg->err;
+			}
 
  			/*
  			 * Save one error to be returned to the caller, but
@@ -1921,6 +1948,9 @@ static int mpam_get_msc_preferred_cpu(struct mpam_msc *msc)
 
  static int mpam_touch_msc(struct mpam_msc *msc, int (*fn)(void *a), void *arg)
  {
+	if (msc->iface != MPAM_IFACE_MMIO)
+		return fn(arg);
+
  	lockdep_assert_irqs_enabled();
  	lockdep_assert_cpus_held();
  	WARN_ON_ONCE(!srcu_read_lock_held((&mpam_srcu)));
drivers/resctrl/mpam_fb.c new file +162 -0
new file mode 100644
index 000000000000..dbe984241b0d
@@ -0,0 +1,214 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2024-2026 Arm Ltd.
+
+#include <linux/arm_mpam.h>
+#include <linux/cleanup.h>
+#include <linux/errno.h>
+#include <linux/mailbox_client.h>
+#include <linux/mutex.h>
+#include <linux/types.h>
+
+#include <acpi/pcc.h>
+#include <asm/mpam.h>
+
+#include "mpam_internal.h"
+
+#define MPAM_FB_PROTOCOL_ID		0x1a
+
+#define MPAM_PROTOCOL_VERSION_CMD	0x0
+#define MPAM_MSC_ATTRIBUTES_CMD		0x3
+#define MPAM_MSC_READ_CMD		0x4
+#define MPAM_MSC_WRITE_CMD		0x5
+
+#define MPAM_FB_ERR_SUCCESS		 0
+#define MPAM_FB_ERR_NOT_SUPPORTED	-1
+#define MPAM_FB_ERR_INVALID_PARAMETERS	-2
+#define MPAM_FB_ERR_DENIED		-3
+#define MPAM_FB_ERR_NOT_FOUND		-4
+#define MPAM_FB_ERR_OUT_OF_RANGE	-5
+#define MPAM_FB_ERR_BUSY		-6
+#define MPAM_FB_ERR_COMMS_ERROR		-7
+#define MPAM_FB_ERR_GENERIC_ERROR	-8
+#define MPAM_FB_ERR_HW_ERROR		-9
+#define MPAM_FB_ERR_PROTOCOL_ERROR	-10
+#define MPAM_FB_ERR_IN_USE		-11
+
+#define MPAM_MSC_PROT_ID_MASK	GENMASK(17, 10)
+#define MPAM_MSC_TOKEN_MASK	GENMASK(27, 18)
+
+static atomic_t mpam_fb_token = ATOMIC_INIT(0);
+
+static void mpam_fb_build_version_message(unsigned int token,
+					  void __iomem *msg_buf)
+{
+	struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem = msg_buf;
+
+	/* .signature is filled by the platform */
+	writel_relaxed(PCC_CMD_COMPLETION_NOTIFY, &pcc_shmem->flags);
+	writel_relaxed(MPAM_FB_PROT_HEADER_LEN, &pcc_shmem->length);
+	writel_relaxed(MPAM_PROTOCOL_VERSION_CMD |
+		       FIELD_PREP(MPAM_MSC_TOKEN_MASK, token) |
+		       FIELD_PREP(MPAM_MSC_PROT_ID_MASK, MPAM_FB_PROTOCOL_ID),
+		       &pcc_shmem->command);
+}
+
+static void mpam_fb_build_read_message(int msc_id, int reg, unsigned int token,
+				       void __iomem *msg_buf)
+{
+	struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem = msg_buf;
+	struct mpam_fb_read_payload {
+		u32 msc_id;
+		u32 flags;
+		u32 reg_offset;
+	} __packed __iomem *payload = msg_buf + sizeof(*pcc_shmem);
+	int msg_size = MPAM_FB_PROT_HEADER_LEN + sizeof(*payload);
+
+	/* .signature is filled by the platform */
+	writel_relaxed(PCC_CMD_COMPLETION_NOTIFY, &pcc_shmem->flags);
+	writel_relaxed(msg_size, &pcc_shmem->length);
+	writel_relaxed(MPAM_MSC_READ_CMD |
+		       FIELD_PREP(MPAM_MSC_TOKEN_MASK, token) |
+		       FIELD_PREP(MPAM_MSC_PROT_ID_MASK, MPAM_FB_PROTOCOL_ID),
+		       &pcc_shmem->command);
+
+	writel_relaxed(msc_id, &payload->msc_id);
+	writel_relaxed(0, &payload->flags);
+	writel_relaxed(reg, &payload->reg_offset);
+}
+
+static void mpam_fb_build_write_message(int msc_id, int reg, u32 val,
+					unsigned int token,
+					void __iomem *msg_buf)
+{
+	struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem = msg_buf;
+	struct mpam_fb_write_payload {
+		u32 msc_id;
+		u32 flags;
+		u32 reg_offset;
+		u32 value;
+	} __packed __iomem *payload = msg_buf + sizeof(*pcc_shmem);
+	int msg_size = MPAM_FB_PROT_HEADER_LEN + sizeof(*payload);
+
+	/* .signature is filled by the platform */
+	writel_relaxed(PCC_CMD_COMPLETION_NOTIFY, &pcc_shmem->flags);
+	writel_relaxed(msg_size, &pcc_shmem->length);
+	writel_relaxed(MPAM_MSC_WRITE_CMD |
+		       FIELD_PREP(MPAM_MSC_TOKEN_MASK, token) |
+		       FIELD_PREP(MPAM_MSC_PROT_ID_MASK, MPAM_FB_PROTOCOL_ID),
+		       &pcc_shmem->command);
+
+	writel_relaxed(msc_id, &payload->msc_id);
+	writel_relaxed(0, &payload->flags);
+	writel_relaxed(reg, &payload->reg_offset);
+	writel_relaxed(val, &payload->value);
+}
+
+static int mpam_fb_send_request(struct mpam_pcc_chan *pcc_chan, u32 msc_id,
+				u16 reg, u32 *result, int mpam_fb_command)
+{
+	unsigned int token = atomic_inc_return(&mpam_fb_token);
+	struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem;
+	struct pcc_mbox_chan *chan;
+	void __iomem *payload_ofs;
+	u32 status;
+	int ret;
+
+	if (!pcc_chan)
+		return -ENODEV;
+
+	chan = pcc_chan->pcc_chan;
+
+	/* prune token to fit into the 10 bits inside the command register */
+	token = FIELD_GET(MPAM_MSC_TOKEN_MASK,
+			  FIELD_PREP(MPAM_MSC_TOKEN_MASK, token));
+
+	guard(mutex)(&pcc_chan->pcc_chan_lock);
+
+	switch (mpam_fb_command) {
+	case MPAM_MSC_WRITE_CMD:
+		mpam_fb_build_write_message(msc_id, reg, *result,
+					    token, chan->shmem);
+		break;
+	case MPAM_MSC_READ_CMD:
+		mpam_fb_build_read_message(msc_id, reg, token, chan->shmem);
+		break;
+	case MPAM_PROTOCOL_VERSION_CMD:
+		mpam_fb_build_version_message(token, chan->shmem);
+		break;
+	default:
+		dev_err(pcc_chan->pcc_cl.dev, "unsupported MPAM-Fb command %d\n",
+			mpam_fb_command);
+		ret = -EINVAL;
+		goto out_err;
+	}
+
+	ret = mbox_send_message(chan->mchan, NULL);
+	if (ret < 0)
+		goto out_err;
+
+	pcc_shmem = chan->shmem;
+	payload_ofs = chan->shmem + sizeof(*pcc_shmem);
+	status = readl(&pcc_shmem->command);
+	if (FIELD_GET(MPAM_MSC_TOKEN_MASK, status) != token) {
+		ret = -ETIMEDOUT;
+
+		goto out_err;
+	}
+
+	ret = readl(payload_ofs + 0x0);
+	if (ret < 0) {
+		switch (ret) {
+		case MPAM_FB_ERR_NOT_SUPPORTED:
+			ret = -EOPNOTSUPP;

drivers/resctrl/mpam_fb.c line 172

Quoted 10 lines
drivers/resctrl/mpam_fb.c new file +10 -0
+			break;
+		case MPAM_FB_ERR_INVALID_PARAMETERS:
+			ret = -EINVAL;
+			break;
+		case MPAM_FB_ERR_NOT_FOUND:
+			ret = -ENOENT;
+			break;
+		case MPAM_FB_ERR_OUT_OF_RANGE:
+			ret = -ERANGE;
+			break;

While testing v4 on a QEMU setup with a fake PCC-backed MSC, I added a small error injection mechanism to verify the firmware response status code translations in mpam_fb_send_request(). I injected each defined MPAM_FB_ERR_* code and observed the following.

drivers/resctrl/mpam_fb.c line 156

Quoted 6 lines
drivers/resctrl/mpam_fb.c new file +6 -0
+		default:
+			ret = -EINVAL;
+		}
+
+		goto out_err;
+	}

MPAM_FB_ERR_BUSY (-6) falls through to this default and gets -EINVAL.
Would -EAGAIN be more appropriate here? Callers could then maybe add a short retry loop inside mpam_fb_send_request() itself, or in the probe path, convert -EAGAIN to -EPROBE_DEFER so the driver core retries probe automatically.

The other unhandled codes also collapse to -EINVAL, like EPROTO, EBUSY, I guess that can come later.

Quoted 102 lines
drivers/resctrl/mpam_fb.c new file +36 -0
+
+	if (mpam_fb_command != MPAM_MSC_WRITE_CMD)
+		*result = readl(payload_ofs + 0x4);
+
+	return 0;
+
+out_err:
+	mpam_fb_disable_mpam(ret);
+
+	return ret;
+}
+
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result)
+{
+	return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+				    reg, result, MPAM_MSC_READ_CMD);
+}
+
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value)
+{
+	return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+				    reg, &value, MPAM_MSC_WRITE_CMD);
+}
+
+int mpam_fb_get_protocol_version(struct mpam_msc *msc)
+{
+	u32 version;
+	int ret;
+
+	ret = mpam_fb_send_request(msc->pcc_chan, 0,
+				   0, &version, MPAM_PROTOCOL_VERSION_CMD);
+	if (ret)
+		return ret;
+
+	return version;
+}
drivers/resctrl/mpam_internal.h +22 -0
index 2b81b6b0bf4e..a2193e7df57c 100644
@@ -11,6 +11,7 @@
  #include <linux/io.h>
  #include <linux/jump_label.h>
  #include <linux/llist.h>
+#include <linux/mailbox_client.h>
  #include <linux/mutex.h>
  #include <linux/resctrl.h>
  #include <linux/spinlock.h>
@@ -57,6 +58,15 @@ struct mpam_garbage {
  	struct platform_device	*pdev;
  };
 
+struct mpam_pcc_chan {
+	struct list_head	pcc_chans;
+	struct mbox_client	pcc_cl;
+	struct pcc_mbox_chan	*pcc_chan;
+	struct mutex		pcc_chan_lock; /* only one message at a time */
+	struct kref		refcount;
+	int			subspace_id;
+};
+
  struct mpam_msc {
  	/* member of mpam_all_msc */
  	struct list_head	all_msc_list;
@@ -66,6 +76,8 @@ struct mpam_msc {
 
  	/* Not modified after mpam_is_enabled() becomes true */
  	enum mpam_msc_iface	iface;
+	struct mpam_pcc_chan	*pcc_chan;
+	int			mpam_fb_msc_id;	/* in its own name space */
  	u32			nrdy_usec;
  	cpumask_t		accessibility;
  	bool			has_extd_esr;
@@ -484,6 +496,9 @@ extern u8 mpam_pmg_max;
  void mpam_enable(struct work_struct *work);
  void mpam_disable(struct work_struct *work);
 
+/* helper function to call from outside mpam_devices.c */
+void mpam_fb_disable_mpam(int err);
+
  /* Reset all the RIS in a class under cpus_read_lock() */
  void mpam_reset_class_locked(struct mpam_class *class);
 
@@ -511,6 +526,13 @@ static inline void mpam_resctrl_offline_cpu(unsigned int cpu) { }
  static inline void mpam_resctrl_teardown_class(struct mpam_class *class) { }
  #endif /* CONFIG_RESCTRL_FS */
 
+/* MPAM-Fb Firmware-backed protocol wrappers */
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result);
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value);
+int mpam_fb_get_protocol_version(struct mpam_msc *msc);
+
+#define MPAM_FB_PROT_HEADER_LEN		sizeof(u32)
+
  /*
   * MPAM MSCs have the following register layout. See:
   * Arm Memory System Resource Partitioning and Monitoring (MPAM) System
include/linux/arm_mpam.h +0 -0
index f92a36187a52..002f56e15362 100644

include/linux/arm_mpam.h struct mpam_msc; · line 18

Quoted 8 lines
include/linux/arm_mpam.h +1 -1
@@ -12,7 +12,7 @@ struct mpam_msc;
 
  enum mpam_msc_iface {
  	MPAM_IFACE_MMIO,	/* a real MPAM MSC */
-	MPAM_IFACE_PCC,		/* a fake MPAM MSC */
+	MPAM_IFACE_PCC,		/* using the MPAM-Fb firmware redirection */
  };
 
  enum mpam_class_types {

Best Regards,
Srivathsa

#22 lore raw

Re: [PATCH v5 08/10] arm_mpam: add MPAM-Fb MSC firmware access support

Hey Srivathsa,

Quoted 47 lines · Srivathsa L Rao

Hi Andre,

Quoted 44 lines · Andre Przywara

The Arm MPAM Firmware-backed (Fb) Profile document[1] describes an alternative way of accessing the "Memory System Components" (MSC) in an MPAM enabled system.

Normally the MSCs are MMIO mapped, but in some implementations this might not be possible (MSC located outside of the local socket, MSC mapped secure-only) or desirable (direct MMIO access too slow or needs to be mediated through a control processor). MPAM-fb standardises a protocol to abstract MSC accesses, building on the SCMI protocol.

Add functions that do an MSC read or write access by redirecting the request through a firmware interface. For now this done via an ACPI PCC shared memory and mailbox combination.

Since the protocol used is only a small subset of the full SCMI spec, and the SCMI protocol has no full ACPI support anyway, open-code the (simple) SCMI message generation, for just the fields we need.

[1] https://developer.arm.com/documentation/den0144/latest

---
  drivers/resctrl/Makefile        |   2 +- drivers/resctrl/mpam_devices.c  |  52 ++++++-- drivers/resctrl/mpam_fb.c       | 214 ++++++++++++++++++++++++++++++++ drivers/resctrl/mpam_internal.h |  22 ++++ include/linux/arm_mpam.h        |   2 +-

  5 files changed, 279 insertions(+), 13 deletions(-)

  create mode 100644 drivers/resctrl/mpam_fb.c

drivers/resctrl/Makefile +0 -0
index 4f6d0e81f9b8..097c036724e9 100644
@@ -1,5 +1,5 @@

  obj-$(CONFIG_ARM64_MPAM_DRIVER)            += mpam.o

drivers/resctrl/Makefile +1 -1
-mpam-y                        += mpam_devices.o
+mpam-y                        += mpam_devices.o mpam_fb.o

  mpam-$(CONFIG_ARM64_MPAM_RESCTRL_FS)        += mpam_resctrl.o ccflags-$(CONFIG_ARM64_MPAM_DRIVER_DEBUG)    += -DDEBUG

drivers/resctrl/mpam_devices.c +0 -0
index f6910ab3bbc2..abe1e628928f 100644

[ ... ]

Quoted 67 lines
Quoted 67 lines

+static int mpam_fb_send_request(struct mpam_pcc_chan *pcc_chan, u32 msc_id,
+                u16 reg, u32 *result, int mpam_fb_command)

drivers/resctrl/mpam_devices.c +14 -0
+{
+    unsigned int token = atomic_inc_return(&mpam_fb_token);
+    struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem;
+    struct pcc_mbox_chan *chan;
+    void __iomem *payload_ofs;
+    u32 status;
+    int ret;
+
+    if (!pcc_chan)
+        return -ENODEV;
+
+    chan = pcc_chan->pcc_chan;
+
+    /* prune token to fit into the 10 bits inside the command 

register */
+    token = FIELD_GET(MPAM_MSC_TOKEN_MASK,
+              FIELD_PREP(MPAM_MSC_TOKEN_MASK, token));

drivers/resctrl/mpam_devices.c +16 -0
+
+    guard(mutex)(&pcc_chan->pcc_chan_lock);
+
+    switch (mpam_fb_command) {
+    case MPAM_MSC_WRITE_CMD:
+        mpam_fb_build_write_message(msc_id, reg, *result,
+                        token, chan->shmem);
+        break;
+    case MPAM_MSC_READ_CMD:
+        mpam_fb_build_read_message(msc_id, reg, token, chan->shmem);
+        break;
+    case MPAM_PROTOCOL_VERSION_CMD:
+        mpam_fb_build_version_message(token, chan->shmem);
+        break;
+    default:
+        dev_err(pcc_chan->pcc_cl.dev, "unsupported MPAM-Fb command 

%d\n",
+            mpam_fb_command);
+        ret = -EINVAL;
+        goto out_err;
+    }

drivers/resctrl/mpam_devices.c +26 -0
+
+    ret = mbox_send_message(chan->mchan, NULL);
+    if (ret < 0)
+        goto out_err;
+
+    pcc_shmem = chan->shmem;
+    payload_ofs = chan->shmem + sizeof(*pcc_shmem);
+    status = readl(&pcc_shmem->command);
+    if (FIELD_GET(MPAM_MSC_TOKEN_MASK, status) != token) {
+        ret = -ETIMEDOUT;
+
+        goto out_err;
+    }
+
+    ret = readl(payload_ofs + 0x0);
+    if (ret < 0) {
+        switch (ret) {
+        case MPAM_FB_ERR_NOT_SUPPORTED:
+            ret = -EOPNOTSUPP;
+            break;
+        case MPAM_FB_ERR_INVALID_PARAMETERS:
+            ret = -EINVAL;
+            break;
+        case MPAM_FB_ERR_NOT_FOUND:
+            ret = -ENOENT;
+            break;
Quoted 8 lines
Quoted 3 lines

+        case MPAM_FB_ERR_OUT_OF_RANGE:
+            ret = -ERANGE;
+            break;

While testing v4 on a QEMU setup with a fake PCC-backed MSC, I added a small error injection mechanism to verify the firmware response status code translations in mpam_fb_send_request(). I injected each defined MPAM_FB_ERR_* code and observed the following.

Ah, very nice, thanks for doing this!

Quoted 9 lines
Quoted 6 lines

+        default:
+            ret = -EINVAL;
+        }
+
+        goto out_err;
+    }

MPAM_FB_ERR_BUSY (-6) falls through to this default and gets -EINVAL. Would -EAGAIN be more appropriate here?  Callers could then maybe add a

Oh, sure, I somehow missed that, even though it's one of the more obvious mappings and probably even the most useful one.

Thanks for catching that, added.

Cheers,
Andre

Quoted 114 lines

short retry loop inside mpam_fb_send_request() itself, or in the probe path, convert -EAGAIN to -EPROBE_DEFER so the driver core retries probe automatically.

The other unhandled codes also collapse to -EINVAL, like EPROTO, EBUSY, I guess that can come later.

Quoted 102 lines
drivers/resctrl/mpam_devices.c +13 -0
+
+    if (mpam_fb_command != MPAM_MSC_WRITE_CMD)
+        *result = readl(payload_ofs + 0x4);
+
+    return 0;
+
+out_err:
+    mpam_fb_disable_mpam(ret);
+
+    return ret;
+}
+
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 

*result)

drivers/resctrl/mpam_devices.c +23 -0
+{
+    return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+                    reg, result, MPAM_MSC_READ_CMD);
+}
+
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value)
+{
+    return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+                    reg, &value, MPAM_MSC_WRITE_CMD);
+}
+
+int mpam_fb_get_protocol_version(struct mpam_msc *msc)
+{
+    u32 version;
+    int ret;
+
+    ret = mpam_fb_send_request(msc->pcc_chan, 0,
+                   0, &version, MPAM_PROTOCOL_VERSION_CMD);
+    if (ret)
+        return ret;
+
+    return version;
+}
drivers/resctrl/mpam_internal.h +22 -0
index 2b81b6b0bf4e..a2193e7df57c 100644
@@ -11,6 +11,7 @@
   #include <linux/io.h>
   #include <linux/jump_label.h>
   #include <linux/llist.h>
+#include <linux/mailbox_client.h>
   #include <linux/mutex.h>
   #include <linux/resctrl.h>
   #include <linux/spinlock.h>
@@ -57,6 +58,15 @@ struct mpam_garbage {
       struct platform_device    *pdev;
   };
+struct mpam_pcc_chan {
+    struct list_head    pcc_chans;
+    struct mbox_client    pcc_cl;
+    struct pcc_mbox_chan    *pcc_chan;
+    struct mutex        pcc_chan_lock; /* only one message at a time */
+    struct kref        refcount;
+    int            subspace_id;
+};
+
   struct mpam_msc {
       /* member of mpam_all_msc */
       struct list_head    all_msc_list;
@@ -66,6 +76,8 @@ struct mpam_msc {
       /* Not modified after mpam_is_enabled() becomes true */
       enum mpam_msc_iface    iface;
+    struct mpam_pcc_chan    *pcc_chan;
+    int            mpam_fb_msc_id;    /* in its own name space */
       u32            nrdy_usec;
       cpumask_t        accessibility;
       bool            has_extd_esr;
@@ -484,6 +496,9 @@ extern u8 mpam_pmg_max;
   void mpam_enable(struct work_struct *work);
   void mpam_disable(struct work_struct *work);
+/* helper function to call from outside mpam_devices.c */
+void mpam_fb_disable_mpam(int err);
+
   /* Reset all the RIS in a class under cpus_read_lock() */
   void mpam_reset_class_locked(struct mpam_class *class);
@@ -511,6 +526,13 @@ static inline void 
 mpam_resctrl_offline_cpu(unsigned int cpu) { }
   static inline void mpam_resctrl_teardown_class(struct mpam_class *class) { }
   #endif /* CONFIG_RESCTRL_FS */
+/* MPAM-Fb Firmware-backed protocol wrappers */
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 
 *result);
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 
 value);
+int mpam_fb_get_protocol_version(struct mpam_msc *msc);
+
+#define MPAM_FB_PROT_HEADER_LEN        sizeof(u32)
+
   /*

   * MPAM MSCs have the following register layout. See:
   * Arm Memory System Resource Partitioning and Monitoring (MPAM) System

include/linux/arm_mpam.h +0 -0
index f92a36187a52..002f56e15362 100644
@@ -12,7 +12,7 @@ struct mpam_msc;
   enum mpam_msc_iface {
Quoted 5 lines

      MPAM_IFACE_MMIO,    /* a real MPAM MSC */
-    MPAM_IFACE_PCC,        /* a fake MPAM MSC */
+    MPAM_IFACE_PCC,        /* using the MPAM-Fb firmware redirection */ };
  enum mpam_class_types {

Best Regards,
Srivathsa

maintainer
#23 lore raw

Re: [PATCH v5 08/10] arm_mpam: add MPAM-Fb MSC firmware access support

In reply to Andre Przywara

Hi Andre,

Quoted 270 lines · Andre Przywara

The Arm MPAM Firmware-backed (Fb) Profile document[1] describes an alternative way of accessing the "Memory System Components" (MSC) in an MPAM enabled system.

Normally the MSCs are MMIO mapped, but in some implementations this might not be possible (MSC located outside of the local socket, MSC mapped secure-only) or desirable (direct MMIO access too slow or needs to be mediated through a control processor). MPAM-fb standardises a protocol to abstract MSC accesses, building on the SCMI protocol.

Add functions that do an MSC read or write access by redirecting the request through a firmware interface. For now this done via an ACPI PCC shared memory and mailbox combination.

Since the protocol used is only a small subset of the full SCMI spec, and the SCMI protocol has no full ACPI support anyway, open-code the (simple) SCMI message generation, for just the fields we need.

[1] https://developer.arm.com/documentation/den0144/latest

---
 drivers/resctrl/Makefile        |   2 +-
 drivers/resctrl/mpam_devices.c  |  52 ++++++--
 drivers/resctrl/mpam_fb.c       | 214 ++++++++++++++++++++++++++++++++
 drivers/resctrl/mpam_internal.h |  22 ++++
 include/linux/arm_mpam.h        |   2 +-
 5 files changed, 279 insertions(+), 13 deletions(-)
 create mode 100644 drivers/resctrl/mpam_fb.c
drivers/resctrl/Makefile +1 -1
index 4f6d0e81f9b8..097c036724e9 100644
@@ -1,5 +1,5 @@
 obj-$(CONFIG_ARM64_MPAM_DRIVER)			+= mpam.o
-mpam-y						+= mpam_devices.o
+mpam-y						+= mpam_devices.o mpam_fb.o
 mpam-$(CONFIG_ARM64_MPAM_RESCTRL_FS)		+= mpam_resctrl.o
 
 ccflags-$(CONFIG_ARM64_MPAM_DRIVER_DEBUG)	+= -DDEBUG
drivers/resctrl/mpam_devices.c +41 -11
index f6910ab3bbc2..abe1e628928f 100644
@@ -83,6 +83,16 @@ static DECLARE_WORK(mpam_broken_work, &mpam_disable);
 /* When mpam is disabled, the printed reason to aid debugging */
 static char *mpam_disable_reason;
 
+void mpam_fb_disable_mpam(int err)
+{
+	static char mpam_fb_reason[32];
+
+	snprintf(mpam_fb_reason, sizeof(mpam_fb_reason), "MPAM-Fb error %d",
+		 err);
+	mpam_disable_reason = mpam_fb_reason;
+	schedule_work(&mpam_broken_work);
+}
+
 /*
  * Whether resctrl has been setup. Used by cpuhp in preference to
  * mpam_is_enabled(). The disable call after an error interrupt makes
@@ -179,8 +189,11 @@ static void mpam_assert_partid_sizes_fixed(void)
 
 static int __mpam_read_reg(struct mpam_msc *msc, u16 reg, u32 *res)
 {
-	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
+	if (msc->iface == MPAM_IFACE_PCC)
+		return mpam_fb_send_read_request(msc, reg, res);
+
+	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 	*res = readl_relaxed(msc->mapped_hwpage + reg);
 
 	return 0;
@@ -197,9 +210,12 @@ static inline int _mpam_read_partsel_reg(struct mpam_msc *msc, u16 reg,
 
 static int __mpam_write_reg(struct mpam_msc *msc, u16 reg, u32 val)
 {
-	WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz);
-	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
+	if (msc->iface == MPAM_IFACE_PCC)
+		return mpam_fb_send_write_request(msc, reg, val);
+
+	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
+	WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz);
 	writel_relaxed(val, msc->mapped_hwpage + reg);
 
 	return 0;
@@ -1141,8 +1157,11 @@ static int mpam_msc_read_mbwu_l(struct mpam_msc *msc, u64 *res)
 
 	mpam_mon_sel_lock_held(msc);
 
-	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
-	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
+	if (msc->iface == MPAM_IFACE_MMIO) {
+		WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
+		WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(),
+			     &msc->accessibility));
+	}
 
 	ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
 	if (ret)
@@ -1176,8 +1195,11 @@ static int mpam_msc_zero_mbwu_l(struct mpam_msc *msc)
 
 	mpam_mon_sel_lock_held(msc);
 
-	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
-	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
+	if (msc->iface == MPAM_IFACE_MMIO) {
+		WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
+		WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(),
+			     &msc->accessibility));
+	}
 
 	ret = __mpam_write_reg(msc, MSMON_MBWU_L, 0);
 	if (ret)
@@ -1490,11 +1512,16 @@ static int _msmon_read(struct mpam_component *comp, struct mon_read *arg)
 					 srcu_read_lock_held(&mpam_srcu)) {
 			arg->ris = ris;
 
-			err = smp_call_function_any(&msc->accessibility,
-						    __ris_msmon_read, arg,
-						    true);
-			if (!err && arg->err)
+			if (msc->iface == MPAM_IFACE_MMIO) {
+				err = smp_call_function_any(&msc->accessibility,
+							    __ris_msmon_read,
+							    arg, true);
+				if (!err)
+					err = arg->err;
+			} else {
+				__ris_msmon_read(arg);
 				err = arg->err;
+			}
 
 			/*
 			 * Save one error to be returned to the caller, but
@@ -1921,6 +1948,9 @@ static int mpam_get_msc_preferred_cpu(struct mpam_msc *msc)
 
 static int mpam_touch_msc(struct mpam_msc *msc, int (*fn)(void *a), void *arg)
 {
+	if (msc->iface != MPAM_IFACE_MMIO)
+		return fn(arg);
+
 	lockdep_assert_irqs_enabled();
 	lockdep_assert_cpus_held();
 	WARN_ON_ONCE(!srcu_read_lock_held((&mpam_srcu)));
drivers/resctrl/mpam_fb.c new file +115 -0
new file mode 100644
index 000000000000..dbe984241b0d
@@ -0,0 +1,214 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2024-2026 Arm Ltd.
+
+#include <linux/arm_mpam.h>
+#include <linux/cleanup.h>
+#include <linux/errno.h>
+#include <linux/mailbox_client.h>
+#include <linux/mutex.h>
+#include <linux/types.h>
+
+#include <acpi/pcc.h>
+#include <asm/mpam.h>
+
+#include "mpam_internal.h"
+
+#define MPAM_FB_PROTOCOL_ID		0x1a
+
+#define MPAM_PROTOCOL_VERSION_CMD	0x0
+#define MPAM_MSC_ATTRIBUTES_CMD		0x3
+#define MPAM_MSC_READ_CMD		0x4
+#define MPAM_MSC_WRITE_CMD		0x5
+
+#define MPAM_FB_ERR_SUCCESS		 0
+#define MPAM_FB_ERR_NOT_SUPPORTED	-1
+#define MPAM_FB_ERR_INVALID_PARAMETERS	-2
+#define MPAM_FB_ERR_DENIED		-3
+#define MPAM_FB_ERR_NOT_FOUND		-4
+#define MPAM_FB_ERR_OUT_OF_RANGE	-5
+#define MPAM_FB_ERR_BUSY		-6
+#define MPAM_FB_ERR_COMMS_ERROR		-7
+#define MPAM_FB_ERR_GENERIC_ERROR	-8
+#define MPAM_FB_ERR_HW_ERROR		-9
+#define MPAM_FB_ERR_PROTOCOL_ERROR	-10
+#define MPAM_FB_ERR_IN_USE		-11
+
+#define MPAM_MSC_PROT_ID_MASK	GENMASK(17, 10)
+#define MPAM_MSC_TOKEN_MASK	GENMASK(27, 18)
+
+static atomic_t mpam_fb_token = ATOMIC_INIT(0);
+
+static void mpam_fb_build_version_message(unsigned int token,
+					  void __iomem *msg_buf)
+{
+	struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem = msg_buf;
+
+	/* .signature is filled by the platform */
+	writel_relaxed(PCC_CMD_COMPLETION_NOTIFY, &pcc_shmem->flags);
+	writel_relaxed(MPAM_FB_PROT_HEADER_LEN, &pcc_shmem->length);
+	writel_relaxed(MPAM_PROTOCOL_VERSION_CMD |
+		       FIELD_PREP(MPAM_MSC_TOKEN_MASK, token) |
+		       FIELD_PREP(MPAM_MSC_PROT_ID_MASK, MPAM_FB_PROTOCOL_ID),
+		       &pcc_shmem->command);
+}
+
+static void mpam_fb_build_read_message(int msc_id, int reg, unsigned int token,
+				       void __iomem *msg_buf)
+{
+	struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem = msg_buf;
+	struct mpam_fb_read_payload {
+		u32 msc_id;
+		u32 flags;
+		u32 reg_offset;
+	} __packed __iomem *payload = msg_buf + sizeof(*pcc_shmem);
+	int msg_size = MPAM_FB_PROT_HEADER_LEN + sizeof(*payload);
+
+	/* .signature is filled by the platform */
+	writel_relaxed(PCC_CMD_COMPLETION_NOTIFY, &pcc_shmem->flags);
+	writel_relaxed(msg_size, &pcc_shmem->length);
+	writel_relaxed(MPAM_MSC_READ_CMD |
+		       FIELD_PREP(MPAM_MSC_TOKEN_MASK, token) |
+		       FIELD_PREP(MPAM_MSC_PROT_ID_MASK, MPAM_FB_PROTOCOL_ID),
+		       &pcc_shmem->command);
+
+	writel_relaxed(msc_id, &payload->msc_id);
+	writel_relaxed(0, &payload->flags);
+	writel_relaxed(reg, &payload->reg_offset);
+}
+
+static void mpam_fb_build_write_message(int msc_id, int reg, u32 val,
+					unsigned int token,
+					void __iomem *msg_buf)
+{
+	struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem = msg_buf;
+	struct mpam_fb_write_payload {
+		u32 msc_id;
+		u32 flags;
+		u32 reg_offset;
+		u32 value;
+	} __packed __iomem *payload = msg_buf + sizeof(*pcc_shmem);
+	int msg_size = MPAM_FB_PROT_HEADER_LEN + sizeof(*payload);
+
+	/* .signature is filled by the platform */
+	writel_relaxed(PCC_CMD_COMPLETION_NOTIFY, &pcc_shmem->flags);
+	writel_relaxed(msg_size, &pcc_shmem->length);
+	writel_relaxed(MPAM_MSC_WRITE_CMD |
+		       FIELD_PREP(MPAM_MSC_TOKEN_MASK, token) |
+		       FIELD_PREP(MPAM_MSC_PROT_ID_MASK, MPAM_FB_PROTOCOL_ID),
+		       &pcc_shmem->command);
+
+	writel_relaxed(msc_id, &payload->msc_id);
+	writel_relaxed(0, &payload->flags);
+	writel_relaxed(reg, &payload->reg_offset);
+	writel_relaxed(val, &payload->value);
+}
+
+static int mpam_fb_send_request(struct mpam_pcc_chan *pcc_chan, u32 msc_id,
+				u16 reg, u32 *result, int mpam_fb_command)
+{
+	unsigned int token = atomic_inc_return(&mpam_fb_token);
+	struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem;
+	struct pcc_mbox_chan *chan;
+	void __iomem *payload_ofs;
+	u32 status;
+	int ret;
+

drivers/resctrl/mpam_fb.c line 125

Quoted 10 lines
drivers/resctrl/mpam_fb.c new file +10 -0
+	if (!pcc_chan)
+		return -ENODEV;
+
+	chan = pcc_chan->pcc_chan;
+
+	/* prune token to fit into the 10 bits inside the command register */
+	token = FIELD_GET(MPAM_MSC_TOKEN_MASK,
+			  FIELD_PREP(MPAM_MSC_TOKEN_MASK, token));
+
+	guard(mutex)(&pcc_chan->pcc_chan_lock);

As sashiko pointed out (strangely on its comments for patch 10) it's best to use mutex_lock() and mutex_unlock() in this function now that we have lots of gotos here. As per the guidance in include/linux/cleanup.h.

Thanks,

Ben

Quoted 163 lines
drivers/resctrl/mpam_fb.c new file +89 -0
+
+	switch (mpam_fb_command) {
+	case MPAM_MSC_WRITE_CMD:
+		mpam_fb_build_write_message(msc_id, reg, *result,
+					    token, chan->shmem);
+		break;
+	case MPAM_MSC_READ_CMD:
+		mpam_fb_build_read_message(msc_id, reg, token, chan->shmem);
+		break;
+	case MPAM_PROTOCOL_VERSION_CMD:
+		mpam_fb_build_version_message(token, chan->shmem);
+		break;
+	default:
+		dev_err(pcc_chan->pcc_cl.dev, "unsupported MPAM-Fb command %d\n",
+			mpam_fb_command);
+		ret = -EINVAL;
+		goto out_err;
+	}
+
+	ret = mbox_send_message(chan->mchan, NULL);
+	if (ret < 0)
+		goto out_err;
+
+	pcc_shmem = chan->shmem;
+	payload_ofs = chan->shmem + sizeof(*pcc_shmem);
+	status = readl(&pcc_shmem->command);
+	if (FIELD_GET(MPAM_MSC_TOKEN_MASK, status) != token) {
+		ret = -ETIMEDOUT;
+
+		goto out_err;
+	}
+
+	ret = readl(payload_ofs + 0x0);
+	if (ret < 0) {
+		switch (ret) {
+		case MPAM_FB_ERR_NOT_SUPPORTED:
+			ret = -EOPNOTSUPP;
+			break;
+		case MPAM_FB_ERR_INVALID_PARAMETERS:
+			ret = -EINVAL;
+			break;
+		case MPAM_FB_ERR_NOT_FOUND:
+			ret = -ENOENT;
+			break;
+		case MPAM_FB_ERR_OUT_OF_RANGE:
+			ret = -ERANGE;
+			break;
+		default:
+			ret = -EINVAL;
+		}
+
+		goto out_err;
+	}
+
+	if (mpam_fb_command != MPAM_MSC_WRITE_CMD)
+		*result = readl(payload_ofs + 0x4);
+
+	return 0;
+
+out_err:
+	mpam_fb_disable_mpam(ret);
+
+	return ret;
+}
+
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result)
+{
+	return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+				    reg, result, MPAM_MSC_READ_CMD);
+}
+
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value)
+{
+	return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+				    reg, &value, MPAM_MSC_WRITE_CMD);
+}
+
+int mpam_fb_get_protocol_version(struct mpam_msc *msc)
+{
+	u32 version;
+	int ret;
+
+	ret = mpam_fb_send_request(msc->pcc_chan, 0,
+				   0, &version, MPAM_PROTOCOL_VERSION_CMD);
+	if (ret)
+		return ret;
+
+	return version;
+}
drivers/resctrl/mpam_internal.h +22 -0
index 2b81b6b0bf4e..a2193e7df57c 100644
@@ -11,6 +11,7 @@
 #include <linux/io.h>
 #include <linux/jump_label.h>
 #include <linux/llist.h>
+#include <linux/mailbox_client.h>
 #include <linux/mutex.h>
 #include <linux/resctrl.h>
 #include <linux/spinlock.h>
@@ -57,6 +58,15 @@ struct mpam_garbage {
 	struct platform_device	*pdev;
 };
 
+struct mpam_pcc_chan {
+	struct list_head	pcc_chans;
+	struct mbox_client	pcc_cl;
+	struct pcc_mbox_chan	*pcc_chan;
+	struct mutex		pcc_chan_lock; /* only one message at a time */
+	struct kref		refcount;
+	int			subspace_id;
+};
+
 struct mpam_msc {
 	/* member of mpam_all_msc */
 	struct list_head	all_msc_list;
@@ -66,6 +76,8 @@ struct mpam_msc {
 
 	/* Not modified after mpam_is_enabled() becomes true */
 	enum mpam_msc_iface	iface;
+	struct mpam_pcc_chan	*pcc_chan;
+	int			mpam_fb_msc_id;	/* in its own name space */
 	u32			nrdy_usec;
 	cpumask_t		accessibility;
 	bool			has_extd_esr;
@@ -484,6 +496,9 @@ extern u8 mpam_pmg_max;
 void mpam_enable(struct work_struct *work);
 void mpam_disable(struct work_struct *work);
 
+/* helper function to call from outside mpam_devices.c */
+void mpam_fb_disable_mpam(int err);
+
 /* Reset all the RIS in a class under cpus_read_lock() */
 void mpam_reset_class_locked(struct mpam_class *class);
 
@@ -511,6 +526,13 @@ static inline void mpam_resctrl_offline_cpu(unsigned int cpu) { }
 static inline void mpam_resctrl_teardown_class(struct mpam_class *class) { }
 #endif /* CONFIG_RESCTRL_FS */
 
+/* MPAM-Fb Firmware-backed protocol wrappers */
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result);
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value);
+int mpam_fb_get_protocol_version(struct mpam_msc *msc);
+
+#define MPAM_FB_PROT_HEADER_LEN		sizeof(u32)
+
 /*
  * MPAM MSCs have the following register layout. See:
  * Arm Memory System Resource Partitioning and Monitoring (MPAM) System
include/linux/arm_mpam.h +1 -1
index f92a36187a52..002f56e15362 100644
@@ -12,7 +12,7 @@ struct mpam_msc;
 
 enum mpam_msc_iface {
 	MPAM_IFACE_MMIO,	/* a real MPAM MSC */
-	MPAM_IFACE_PCC,		/* a fake MPAM MSC */
+	MPAM_IFACE_PCC,		/* using the MPAM-Fb firmware redirection */
 };
 
 enum mpam_class_types {
#24 lore raw

Re: [PATCH v5 08/10] arm_mpam: add MPAM-Fb MSC firmware access support

Hi,

Quoted 33 lines · Ben Horgan

Hi Andre,

Quoted 30 lines · Andre Przywara

The Arm MPAM Firmware-backed (Fb) Profile document[1] describes an alternative way of accessing the "Memory System Components" (MSC) in an MPAM enabled system.

Normally the MSCs are MMIO mapped, but in some implementations this might not be possible (MSC located outside of the local socket, MSC mapped secure-only) or desirable (direct MMIO access too slow or needs to be mediated through a control processor). MPAM-fb standardises a protocol to abstract MSC accesses, building on the SCMI protocol.

Add functions that do an MSC read or write access by redirecting the request through a firmware interface. For now this done via an ACPI PCC shared memory and mailbox combination.

Since the protocol used is only a small subset of the full SCMI spec, and the SCMI protocol has no full ACPI support anyway, open-code the (simple) SCMI message generation, for just the fields we need.

[1] https://developer.arm.com/documentation/den0144/latest

---
  drivers/resctrl/Makefile        |   2 +-
  drivers/resctrl/mpam_devices.c  |  52 ++++++--
  drivers/resctrl/mpam_fb.c       | 214 ++++++++++++++++++++++++++++++++
  drivers/resctrl/mpam_internal.h |  22 ++++
  include/linux/arm_mpam.h        |   2 +-
  5 files changed, 279 insertions(+), 13 deletions(-)

create mode 100644 drivers/resctrl/mpam_fb.c

[ ... ]

Quoted 5 lines
Quoted 5 lines
drivers/resctrl/mpam_fb.c new file +0 -0
new file mode 100644
index 000000000000..dbe984241b0d

[ ... ]

Quoted 16 lines
Quoted 16 lines
drivers/resctrl/mpam_fb.c new file +16 -0
+static int mpam_fb_send_request(struct mpam_pcc_chan *pcc_chan, u32 msc_id,
+				u16 reg, u32 *result, int mpam_fb_command)
+{
+	unsigned int token = atomic_inc_return(&mpam_fb_token);
+	struct acpi_pcct_ext_pcc_shared_memory __iomem *pcc_shmem;
+	struct pcc_mbox_chan *chan;
+	void __iomem *payload_ofs;
+	u32 status;
+	int ret;
+
+	if (!pcc_chan)
+		return -ENODEV;
+
+	chan = pcc_chan->pcc_chan;
+
+	/* prune token to fit into the 10 bits inside the command register */
Quoted 8 lines

drivers/resctrl/mpam_fb.c line 125

Quoted 4 lines
drivers/resctrl/mpam_fb.c new file +4 -0
+	token = FIELD_GET(MPAM_MSC_TOKEN_MASK,
+			  FIELD_PREP(MPAM_MSC_TOKEN_MASK, token));
+
+	guard(mutex)(&pcc_chan->pcc_chan_lock);

As sashiko pointed out (strangely on its comments for patch 10) it's best to use mutex_lock() and mutex_unlock() in this function now that we have lots of gotos here. As per the guidance in include/linux/cleanup.h.

Ah, indeed, I didn't make it to the very end of this comment ;-) Fixed.

Cheers,
Andre

Quoted 164 lines
Quoted 163 lines
drivers/resctrl/mpam_fb.c new file +89 -0
+
+	switch (mpam_fb_command) {
+	case MPAM_MSC_WRITE_CMD:
+		mpam_fb_build_write_message(msc_id, reg, *result,
+					    token, chan->shmem);
+		break;
+	case MPAM_MSC_READ_CMD:
+		mpam_fb_build_read_message(msc_id, reg, token, chan->shmem);
+		break;
+	case MPAM_PROTOCOL_VERSION_CMD:
+		mpam_fb_build_version_message(token, chan->shmem);
+		break;
+	default:
+		dev_err(pcc_chan->pcc_cl.dev, "unsupported MPAM-Fb command %d\n",
+			mpam_fb_command);
+		ret = -EINVAL;
+		goto out_err;
+	}
+
+	ret = mbox_send_message(chan->mchan, NULL);
+	if (ret < 0)
+		goto out_err;
+
+	pcc_shmem = chan->shmem;
+	payload_ofs = chan->shmem + sizeof(*pcc_shmem);
+	status = readl(&pcc_shmem->command);
+	if (FIELD_GET(MPAM_MSC_TOKEN_MASK, status) != token) {
+		ret = -ETIMEDOUT;
+
+		goto out_err;
+	}
+
+	ret = readl(payload_ofs + 0x0);
+	if (ret < 0) {
+		switch (ret) {
+		case MPAM_FB_ERR_NOT_SUPPORTED:
+			ret = -EOPNOTSUPP;
+			break;
+		case MPAM_FB_ERR_INVALID_PARAMETERS:
+			ret = -EINVAL;
+			break;
+		case MPAM_FB_ERR_NOT_FOUND:
+			ret = -ENOENT;
+			break;
+		case MPAM_FB_ERR_OUT_OF_RANGE:
+			ret = -ERANGE;
+			break;
+		default:
+			ret = -EINVAL;
+		}
+
+		goto out_err;
+	}
+
+	if (mpam_fb_command != MPAM_MSC_WRITE_CMD)
+		*result = readl(payload_ofs + 0x4);
+
+	return 0;
+
+out_err:
+	mpam_fb_disable_mpam(ret);
+
+	return ret;
+}
+
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result)
+{
+	return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+				    reg, result, MPAM_MSC_READ_CMD);
+}
+
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value)
+{
+	return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+				    reg, &value, MPAM_MSC_WRITE_CMD);
+}
+
+int mpam_fb_get_protocol_version(struct mpam_msc *msc)
+{
+	u32 version;
+	int ret;
+
+	ret = mpam_fb_send_request(msc->pcc_chan, 0,
+				   0, &version, MPAM_PROTOCOL_VERSION_CMD);
+	if (ret)
+		return ret;
+
+	return version;
+}
drivers/resctrl/mpam_internal.h +22 -0
index 2b81b6b0bf4e..a2193e7df57c 100644
@@ -11,6 +11,7 @@
  #include <linux/io.h>
  #include <linux/jump_label.h>
  #include <linux/llist.h>
+#include <linux/mailbox_client.h>
  #include <linux/mutex.h>
  #include <linux/resctrl.h>
  #include <linux/spinlock.h>
@@ -57,6 +58,15 @@ struct mpam_garbage {
  	struct platform_device	*pdev;
  };
 
+struct mpam_pcc_chan {
+	struct list_head	pcc_chans;
+	struct mbox_client	pcc_cl;
+	struct pcc_mbox_chan	*pcc_chan;
+	struct mutex		pcc_chan_lock; /* only one message at a time */
+	struct kref		refcount;
+	int			subspace_id;
+};
+
  struct mpam_msc {
  	/* member of mpam_all_msc */
  	struct list_head	all_msc_list;
@@ -66,6 +76,8 @@ struct mpam_msc {
 
  	/* Not modified after mpam_is_enabled() becomes true */
  	enum mpam_msc_iface	iface;
+	struct mpam_pcc_chan	*pcc_chan;
+	int			mpam_fb_msc_id;	/* in its own name space */
  	u32			nrdy_usec;
  	cpumask_t		accessibility;
  	bool			has_extd_esr;
@@ -484,6 +496,9 @@ extern u8 mpam_pmg_max;
  void mpam_enable(struct work_struct *work);
  void mpam_disable(struct work_struct *work);
 
+/* helper function to call from outside mpam_devices.c */
+void mpam_fb_disable_mpam(int err);
+
  /* Reset all the RIS in a class under cpus_read_lock() */
  void mpam_reset_class_locked(struct mpam_class *class);
 
@@ -511,6 +526,13 @@ static inline void mpam_resctrl_offline_cpu(unsigned int cpu) { }
  static inline void mpam_resctrl_teardown_class(struct mpam_class *class) { }
  #endif /* CONFIG_RESCTRL_FS */
 
+/* MPAM-Fb Firmware-backed protocol wrappers */
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result);
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value);
+int mpam_fb_get_protocol_version(struct mpam_msc *msc);
+
+#define MPAM_FB_PROT_HEADER_LEN		sizeof(u32)
+
  /*
   * MPAM MSCs have the following register layout. See:
   * Arm Memory System Resource Partitioning and Monitoring (MPAM) System
include/linux/arm_mpam.h +1 -1
index f92a36187a52..002f56e15362 100644
@@ -12,7 +12,7 @@ struct mpam_msc;
 
  enum mpam_msc_iface {
  	MPAM_IFACE_MMIO,	/* a real MPAM MSC */
-	MPAM_IFACE_PCC,		/* a fake MPAM MSC */
+	MPAM_IFACE_PCC,		/* using the MPAM-Fb firmware redirection */
  };
 
  enum mpam_class_types {
#25 lore raw

PATCH v5 9/10 arm_mpam: change error IRQ to use a threaded IRQ handler

In reply to Andre Przywara

When an MPAM MSC gets into an error condition, it can trigger an error IRQ. We cannot really do much about those errors, but we at least query and log the error, then disable MPAM functionality.

This error report relies on reading the MSC's error status register (ESR) in the IRQ handler, which is not possible for MPAM-Fb based MSC accesses, since they involve mailbox routines that might sleep. The same is true for clearing the interrupt at the source, which requires an MSC access as well.

Change the error IRQ to use a threaded IRQ handler, with an empty hard IRQ routine, and doing all the MSC accesses (to access the status and disable the IRQ line) in the threaded part. The change is minimal, we just check for the first MSC access error and bail out early. Also forbid per-CPU interrupts (PPIs) for MPAM-Fb, as we cannot use a threaded IRQ here.

---
 drivers/resctrl/mpam_devices.c | 42 ++++++++++++++++++++++------------
 1 file changed, 28 insertions(+), 14 deletions(-)
drivers/resctrl/mpam_devices.c +28 -14
index abe1e628928f..e535603d2c7d 100644
@@ -2659,9 +2659,11 @@ static int mpam_disable_msc_ecr(void *_msc)
 	return 0;
 }
 
+/* threaded IRQ handler for shared IRQs, to allow MPAM-Fb accesses to sleep */
 static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
 {
 	u64 reg;
+	int ret;
 	u16 partid;
 	u8 errcode, pmg, ris;
 
@@ -2670,13 +2672,19 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
 					   &msc->accessibility)))
 		return IRQ_NONE;
 
-	mpam_msc_read_esr(msc, &reg);
+	ret = mpam_msc_read_esr(msc, &reg);
+	if (ret) {
+		pr_err_ratelimited("unknown error irq from msc:%u\n", msc->id);
+
+		/* Try out best here ... */
+		goto out_disable;
+	}
 
 	errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
 	if (!errcode)
 		return IRQ_NONE;
 
-	/* Clear level triggered irq */
+	/* Clear level triggered irq. Ignore errors, we need to proceed. */
 	mpam_msc_clear_esr(msc);
 
 	partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg);
@@ -2687,19 +2695,19 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
 			   msc->id, mpam_errcode_names[errcode], partid, pmg,
 			   ris);
 
-	/* Disable this interrupt. */
+out_disable:
+	/* Disable this interrupt. Ignore errors, we need to proceed anyway. */
 	mpam_disable_msc_ecr(msc);
 
-	/* Are we racing with the thread disabling MPAM? */
-	if (!mpam_is_enabled())
-		return IRQ_HANDLED;
-
 	/*
-	 * Schedule the teardown work. Don't use a threaded IRQ as we can't
-	 * unregister the interrupt from the threaded part of the handler.
+	 * Schedule the teardown work. We have to defer it as we can't
+	 * unregister the interrupt from the threaded part of a handler.
+	 * Check whether we are racing with the thread disabling MPAM.
 	 */
-	mpam_disable_reason = "hardware error interrupt";
-	schedule_work(&mpam_broken_work);
+	if (mpam_is_enabled()) {
+		mpam_disable_reason = "hardware error interrupt";
+		schedule_work(&mpam_broken_work);
+	}
 
 	return IRQ_HANDLED;
 }
@@ -2735,6 +2743,11 @@ static int mpam_register_irqs(void)
 		/* The MPAM spec says the interrupt can be SPI, PPI or LPI */
 		/* We anticipate sharing the interrupt with other MSCs */
 		if (irq_is_percpu(irq)) {
+			if (msc->iface != MPAM_IFACE_MMIO) {
+				dev_err(&msc->pdev->dev,
+					"Only MMIO MSCs can use per-CPU interrupts\n");
+				return -EINVAL;
+			}
 			err = request_percpu_irq(irq, &mpam_ppi_handler,
 						 "mpam:msc:error",
 						 msc->error_dev_id);
@@ -2746,9 +2759,10 @@ static int mpam_register_irqs(void)
 					       &_enable_percpu_irq, &irq,
 					       true);
 		} else {
-			err = devm_request_irq(&msc->pdev->dev, irq,
-					       &mpam_spi_handler, IRQF_SHARED,
-					       "mpam:msc:error", msc);
+			err = devm_request_threaded_irq(&msc->pdev->dev, irq,
+							NULL, &mpam_spi_handler,
+							IRQF_SHARED | IRQF_ONESHOT,
+							"mpam:msc:error", msc);
 			if (err)
 				return err;
 		}
Signature · 1 line
2.43.0
maintainer
#26 lore raw

Re: [PATCH v5 09/10] arm_mpam: change error IRQ to use a threaded IRQ handler

Hi Andre,

Quoted 27 lines · Andre Przywara

When an MPAM MSC gets into an error condition, it can trigger an error IRQ. We cannot really do much about those errors, but we at least query and log the error, then disable MPAM functionality.

This error report relies on reading the MSC's error status register (ESR) in the IRQ handler, which is not possible for MPAM-Fb based MSC accesses, since they involve mailbox routines that might sleep. The same is true for clearing the interrupt at the source, which requires an MSC access as well.

Change the error IRQ to use a threaded IRQ handler, with an empty hard IRQ routine, and doing all the MSC accesses (to access the status and disable the IRQ line) in the threaded part. The change is minimal, we just check for the first MSC access error and bail out early. Also forbid per-CPU interrupts (PPIs) for MPAM-Fb, as we cannot use a threaded IRQ here.

---
 drivers/resctrl/mpam_devices.c | 42 ++++++++++++++++++++++------------
 1 file changed, 28 insertions(+), 14 deletions(-)
drivers/resctrl/mpam_devices.c +0 -0
index abe1e628928f..e535603d2c7d 100644

drivers/resctrl/mpam_devices.c static int mpam_disable_msc_ecr(void *_msc) · line 2662

Quoted 4 lines
drivers/resctrl/mpam_devices.c +1 -0
@@ -2659,9 +2659,11 @@ static int mpam_disable_msc_ecr(void *_msc)
 	return 0;
 }
 
+/* threaded IRQ handler for shared IRQs, to allow MPAM-Fb accesses to sleep */

This comment seems incomplete. The same handler is used for the PPI hardirq.

drivers/resctrl/mpam_devices.c static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc) · line 2675

Quoted 13 lines
drivers/resctrl/mpam_devices.c +2 -1
 static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
 {
 	u64 reg;
+	int ret;
 	u16 partid;
 	u8 errcode, pmg, ris;
 
@@ -2670,13 +2672,19 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
 					   &msc->accessibility)))
 		return IRQ_NONE;
 
-	mpam_msc_read_esr(msc, &reg);
+	ret = mpam_msc_read_esr(msc, &reg);

This is also called for MMIO accesses and the checks in the reads and writes call smp_processor_id() which will give warnings if we are preemptible. There is also a similar to check higher up in this handler. We need to choose in what context we call things based on whether we are using MMIO accesses or MPAM-Fb.

Thanks,

Ben

Quoted 71 lines
drivers/resctrl/mpam_devices.c +25 -13
+	if (ret) {
+		pr_err_ratelimited("unknown error irq from msc:%u\n", msc->id);
+
+		/* Try out best here ... */
+		goto out_disable;
+	}
 
 	errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
 	if (!errcode)
 		return IRQ_NONE;
 
-	/* Clear level triggered irq */
+	/* Clear level triggered irq. Ignore errors, we need to proceed. */
 	mpam_msc_clear_esr(msc);
 
 	partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg);
@@ -2687,19 +2695,19 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
 			   msc->id, mpam_errcode_names[errcode], partid, pmg,
 			   ris);
 
-	/* Disable this interrupt. */
+out_disable:
+	/* Disable this interrupt. Ignore errors, we need to proceed anyway. */
 	mpam_disable_msc_ecr(msc);
 
-	/* Are we racing with the thread disabling MPAM? */
-	if (!mpam_is_enabled())
-		return IRQ_HANDLED;
-
 	/*
-	 * Schedule the teardown work. Don't use a threaded IRQ as we can't
-	 * unregister the interrupt from the threaded part of the handler.
+	 * Schedule the teardown work. We have to defer it as we can't
+	 * unregister the interrupt from the threaded part of a handler.
+	 * Check whether we are racing with the thread disabling MPAM.
 	 */
-	mpam_disable_reason = "hardware error interrupt";
-	schedule_work(&mpam_broken_work);
+	if (mpam_is_enabled()) {
+		mpam_disable_reason = "hardware error interrupt";
+		schedule_work(&mpam_broken_work);
+	}
 
 	return IRQ_HANDLED;
 }
@@ -2735,6 +2743,11 @@ static int mpam_register_irqs(void)
 		/* The MPAM spec says the interrupt can be SPI, PPI or LPI */
 		/* We anticipate sharing the interrupt with other MSCs */
 		if (irq_is_percpu(irq)) {
+			if (msc->iface != MPAM_IFACE_MMIO) {
+				dev_err(&msc->pdev->dev,
+					"Only MMIO MSCs can use per-CPU interrupts\n");
+				return -EINVAL;
+			}
 			err = request_percpu_irq(irq, &mpam_ppi_handler,
 						 "mpam:msc:error",
 						 msc->error_dev_id);
@@ -2746,9 +2759,10 @@ static int mpam_register_irqs(void)
 					       &_enable_percpu_irq, &irq,
 					       true);
 		} else {
-			err = devm_request_irq(&msc->pdev->dev, irq,
-					       &mpam_spi_handler, IRQF_SHARED,
-					       "mpam:msc:error", msc);
+			err = devm_request_threaded_irq(&msc->pdev->dev, irq,
+							NULL, &mpam_spi_handler,
+							IRQF_SHARED | IRQF_ONESHOT,
+							"mpam:msc:error", msc);
 			if (err)
 				return err;
 		}
#27 lore raw

Re: [PATCH v5 09/10] arm_mpam: change error IRQ to use a threaded IRQ handler

Hi Ben,

Quoted 33 lines · Ben Horgan

Hi Andre,

Quoted 30 lines · Andre Przywara

When an MPAM MSC gets into an error condition, it can trigger an error IRQ. We cannot really do much about those errors, but we at least query and log the error, then disable MPAM functionality.

This error report relies on reading the MSC's error status register (ESR) in the IRQ handler, which is not possible for MPAM-Fb based MSC accesses, since they involve mailbox routines that might sleep. The same is true for clearing the interrupt at the source, which requires an MSC access as well.

Change the error IRQ to use a threaded IRQ handler, with an empty hard IRQ routine, and doing all the MSC accesses (to access the status and disable the IRQ line) in the threaded part. The change is minimal, we just check for the first MSC access error and bail out early. Also forbid per-CPU interrupts (PPIs) for MPAM-Fb, as we cannot use a threaded IRQ here.

---
  drivers/resctrl/mpam_devices.c | 42 ++++++++++++++++++++++------------
  1 file changed, 28 insertions(+), 14 deletions(-)
drivers/resctrl/mpam_devices.c +0 -0
index abe1e628928f..e535603d2c7d 100644
@@ -2659,9 +2659,11 @@ static int mpam_disable_msc_ecr(void *_msc)
  	return 0;
  }
 
Quoted 3 lines

drivers/resctrl/mpam_devices.c static int mpam_disable_msc_ecr(void *_msc) · line 2662

Quoted 1 line
drivers/resctrl/mpam_devices.c +1 -0
+/* threaded IRQ handler for shared IRQs, to allow MPAM-Fb accesses to sleep */

This comment seems incomplete. The same handler is used for the PPI hardirq.

Yes, I failed to convey that it's only a threaded handler for shared IRQs. Reworded that to cover both cases.

Quoted 11 lines
Quoted 11 lines
drivers/resctrl/mpam_devices.c +1 -0
  static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
  {
  	u64 reg;
+	int ret;
  	u16 partid;
  	u8 errcode, pmg, ris;
 
@@ -2670,13 +2672,19 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
  					   &msc->accessibility)))
  		return IRQ_NONE;
 
Quoted 7 lines

drivers/resctrl/mpam_devices.c static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc) · line 2675

Quoted 2 lines
drivers/resctrl/mpam_devices.c +1 -1
-	mpam_msc_read_esr(msc, &reg);
+	ret = mpam_msc_read_esr(msc, &reg);

This is also called for MMIO accesses and the checks in the reads and writes call smp_processor_id() which will give warnings if we are preemptible. There is also a similar to check higher up in this handler. We need to choose in what context we call things based on whether we are using MMIO accesses or MPAM-Fb.

Alright, after some off-line discussion we settled for keeping a hard-IRQ handler for MMIO, and just use a threaded handler for MPAM-Fb. Should solve this problem elegantly.

Thanks for pointing this out!

Cheers,
Andre

Quoted 77 lines

Thanks,

Ben

Quoted 71 lines
drivers/resctrl/mpam_devices.c +25 -13
+	if (ret) {
+		pr_err_ratelimited("unknown error irq from msc:%u\n", msc->id);
+
+		/* Try out best here ... */
+		goto out_disable;
+	}
 
  	errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
  	if (!errcode)
  		return IRQ_NONE;
 
-	/* Clear level triggered irq */
+	/* Clear level triggered irq. Ignore errors, we need to proceed. */
  	mpam_msc_clear_esr(msc);
 
  	partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg);
@@ -2687,19 +2695,19 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
  			   msc->id, mpam_errcode_names[errcode], partid, pmg,
  			   ris);
 
-	/* Disable this interrupt. */
+out_disable:
+	/* Disable this interrupt. Ignore errors, we need to proceed anyway. */
  	mpam_disable_msc_ecr(msc);
 
-	/* Are we racing with the thread disabling MPAM? */
-	if (!mpam_is_enabled())
-		return IRQ_HANDLED;
-
  	/*
-	 * Schedule the teardown work. Don't use a threaded IRQ as we can't
-	 * unregister the interrupt from the threaded part of the handler.
+	 * Schedule the teardown work. We have to defer it as we can't
+	 * unregister the interrupt from the threaded part of a handler.
+	 * Check whether we are racing with the thread disabling MPAM.
  	 */
-	mpam_disable_reason = "hardware error interrupt";
-	schedule_work(&mpam_broken_work);
+	if (mpam_is_enabled()) {
+		mpam_disable_reason = "hardware error interrupt";
+		schedule_work(&mpam_broken_work);
+	}
 
  	return IRQ_HANDLED;
  }
@@ -2735,6 +2743,11 @@ static int mpam_register_irqs(void)
  		/* The MPAM spec says the interrupt can be SPI, PPI or LPI */
  		/* We anticipate sharing the interrupt with other MSCs */
  		if (irq_is_percpu(irq)) {
+			if (msc->iface != MPAM_IFACE_MMIO) {
+				dev_err(&msc->pdev->dev,
+					"Only MMIO MSCs can use per-CPU interrupts\n");
+				return -EINVAL;
+			}
  			err = request_percpu_irq(irq, &mpam_ppi_handler,
  						 "mpam:msc:error",
  						 msc->error_dev_id);
@@ -2746,9 +2759,10 @@ static int mpam_register_irqs(void)
  					       &_enable_percpu_irq, &irq,
  					       true);
  		} else {
-			err = devm_request_irq(&msc->pdev->dev, irq,
-					       &mpam_spi_handler, IRQF_SHARED,
-					       "mpam:msc:error", msc);
+			err = devm_request_threaded_irq(&msc->pdev->dev, irq,
+							NULL, &mpam_spi_handler,
+							IRQF_SHARED | IRQF_ONESHOT,
+							"mpam:msc:error", msc);
  			if (err)
  				return err;
  		}
#28 lore raw

PATCH v5 10/10 arm_mpam: detect and enable MPAM-Fb PCC support

In reply to Andre Przywara

The Arm MPAM-Fb specification [1] describes a protocol to access MSC registers through a firmware interface. This requires a shared memory region to hold the message, and a mailbox to trigger the access. For ACPI this is wrapped as a PCC channel, described using existing ACPI abstractions.

Add code to parse those PCC table descriptions associated with an MSC, and store the parsed information in the MSC struct.
There can be multiple PCC channels, and each channel can serve multiple MSCs, so we need to keep track of the channel usage, using a list and a refcount.

This will be used by the MPAM-Fb access wrapper code.

[1] https://developer.arm.com/documentation/den0144/latest

---
 drivers/resctrl/mpam_devices.c  | 119 +++++++++++++++++++++++++++++++-
 drivers/resctrl/mpam_fb.c       |   4 +-
 drivers/resctrl/mpam_internal.h |   6 +-
 3 files changed, 124 insertions(+), 5 deletions(-)
drivers/resctrl/mpam_devices.c +117 -2
index e535603d2c7d..e360051f4aa7 100644
@@ -19,14 +19,19 @@
 #include <linux/irqdesc.h>
 #include <linux/list.h>
 #include <linux/lockdep.h>
+#include <linux/mailbox_client.h>
 #include <linux/mutex.h>
 #include <linux/platform_device.h>
 #include <linux/printk.h>
+#include <linux/property.h>
 #include <linux/srcu.h>
 #include <linux/spinlock.h>
 #include <linux/types.h>
 #include <linux/workqueue.h>
 
+#include <acpi/pcc.h>
+#include <acpi/acpi_io.h>
+
 #include "mpam_internal.h"
 
 /* Values for the T241 errata workaround */
@@ -49,6 +54,84 @@ static LIST_HEAD(mpam_all_msc);
 
 struct srcu_struct mpam_srcu;
 
+/* PCC channels might be serving multiple MSCs, so keep a refcounted list. */
+static DEFINE_MUTEX(pcc_chan_list_lock);
+static LIST_HEAD(pcc_chan_list);
+
+static void mpam_pcc_chan_release(struct kref *ref)
+{
+	struct mpam_pcc_chan *cur = container_of(ref, struct mpam_pcc_chan,
+						 refcount);
+
+	pcc_mbox_free_channel(cur->pcc_chan);
+	list_del(&cur->pcc_chans);
+	mutex_destroy(&cur->pcc_chan_lock);
+	kfree(cur);
+}
+
+static struct mpam_pcc_chan *mpam_pcc_chan_get(struct device *dev,
+					       int subspace_id)
+{
+	struct mpam_pcc_chan *cur;
+
+	guard(mutex)(&pcc_chan_list_lock);
+
+	list_for_each_entry(cur, &pcc_chan_list, pcc_chans) {
+		if (cur->subspace_id == subspace_id) {
+			kref_get(&cur->refcount);
+
+			return cur;
+		}
+	}
+
+	cur = kzalloc_obj(*cur);
+	if (!cur)
+		return ERR_PTR(-ENOMEM);
+
+	cur->pcc_cl.dev = dev;
+	cur->pcc_cl.tx_block = true;
+
+	cur->pcc_chan = pcc_mbox_request_channel(&cur->pcc_cl, subspace_id);
+	if (IS_ERR(cur->pcc_chan)) {
+		long err = PTR_ERR(cur->pcc_chan);
+
+		kfree(cur);
+		return ERR_PTR(err);
+	}
+
+	/* Timeout based on the "nominal latency" from the PCC ACPI table. */
+	cur->pcc_cl.tx_tout = cur->pcc_chan->latency * 5;
+
+	mutex_init(&cur->pcc_chan_lock);
+
+	cur->subspace_id = subspace_id;
+	kref_init(&cur->refcount);
+
+	list_add_tail(&cur->pcc_chans, &pcc_chan_list);
+
+	return cur;
+}
+
+static int mpam_pcc_chan_put(struct mpam_pcc_chan *pcc_chan)
+{
+	struct mpam_pcc_chan *cur, *tmp;
+
+	if (!pcc_chan)
+		return 0;
+
+	guard(mutex)(&pcc_chan_list_lock);
+
+	list_for_each_entry_safe(cur, tmp, &pcc_chan_list, pcc_chans) {
+		if (cur == pcc_chan) {
+			kref_put(&cur->refcount, mpam_pcc_chan_release);
+
+			return 0;
+		}
+	}
+
+	return -ENOENT;
+}
+
 /*
  * Number of MSCs that have been probed. Once all MSCs have been probed MPAM
  * can be enabled.
@@ -2221,6 +2304,8 @@ static void mpam_msc_drv_remove(struct platform_device *pdev)
 {
 	struct mpam_msc *msc = platform_get_drvdata(pdev);
 
+	mpam_pcc_chan_put(msc->pcc_chan);
+
 	mutex_lock(&mpam_list_lock);
 	mpam_msc_destroy(msc);
 	mutex_unlock(&mpam_list_lock);
@@ -2231,7 +2316,7 @@ static void mpam_msc_drv_remove(struct platform_device *pdev)
 static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
 {
 	int err;
-	u32 tmp;
+	u32 pcc_subspace_id;
 	struct mpam_msc *msc;
 	struct resource *msc_res;
 	struct device *dev = &pdev->dev;
@@ -2279,7 +2364,7 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
 	if (err)
 		return ERR_PTR(err);
 
-	if (device_property_read_u32(&pdev->dev, "pcc-channel", &tmp))
+	if (device_property_read_u32(dev, "pcc-channel", &pcc_subspace_id))
 		msc->iface = MPAM_IFACE_MMIO;
 	else
 		msc->iface = MPAM_IFACE_PCC;
@@ -2295,6 +2380,36 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
 		}
 		msc->mapped_hwpage_sz = msc_res->end - msc_res->start;
 		msc->mapped_hwpage = io;
+	} else if (msc->iface == MPAM_IFACE_PCC) {
+		int ret;
+
+		msc->pcc_chan = mpam_pcc_chan_get(dev, pcc_subspace_id);
+		if (IS_ERR(msc->pcc_chan)) {
+			pr_err("Failed to request MSC PCC channel\n");
+			return ERR_CAST(msc->pcc_chan);
+		}
+
+		if (msc->pcc_chan->pcc_chan->shmem_size < MPAM_FB_MAX_MSG_SIZE) {
+			pr_err("MPAM-Fb PCC channel size too small.\n");
+			mpam_pcc_chan_put(msc->pcc_chan);
+			return ERR_PTR(-ENOMEM);
+		}
+		ret = mpam_fb_get_protocol_version(msc);
+		if (ret < 0) {
+			pr_err("Cannot query MPAM-Fb protocol version.\n");
+			mpam_pcc_chan_put(msc->pcc_chan);
+
+			/* ret is an MPAM-Fb error code, return a Linux one. */
+			return ERR_PTR(-EIO);
+		}
+		if (FIELD_GET(MPAM_FB_VERSION_MAJOR_MASK, ret) != 1) {
+			pr_err("Incompatible MPAM-Fb protocol version %ld.%ld\n",
+			       FIELD_GET(MPAM_FB_VERSION_MAJOR_MASK, ret),
+			       FIELD_GET(MPAM_FB_VERSION_MINOR_MASK, ret));
+			mpam_pcc_chan_put(msc->pcc_chan);
+
+			return ERR_PTR(-EINVAL);
+		}
 	} else {
 		return ERR_PTR(-EINVAL);
 	}
drivers/resctrl/mpam_fb.c +2 -2
index dbe984241b0d..55c4778a8379 100644
@@ -190,13 +190,13 @@ static int mpam_fb_send_request(struct mpam_pcc_chan *pcc_chan, u32 msc_id,
 
 int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result)
 {
-	return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+	return mpam_fb_send_request(msc->pcc_chan, msc->id,
 				    reg, result, MPAM_MSC_READ_CMD);
 }
 
 int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value)
 {
-	return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+	return mpam_fb_send_request(msc->pcc_chan, msc->id,
 				    reg, &value, MPAM_MSC_WRITE_CMD);
 }
 
drivers/resctrl/mpam_internal.h +5 -1
index a2193e7df57c..e926b7feb72a 100644
@@ -77,7 +77,6 @@ struct mpam_msc {
 	/* Not modified after mpam_is_enabled() becomes true */
 	enum mpam_msc_iface	iface;
 	struct mpam_pcc_chan	*pcc_chan;
-	int			mpam_fb_msc_id;	/* in its own name space */
 	u32			nrdy_usec;
 	cpumask_t		accessibility;
 	bool			has_extd_esr;
@@ -532,6 +531,11 @@ int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value);
 int mpam_fb_get_protocol_version(struct mpam_msc *msc);
 
 #define MPAM_FB_PROT_HEADER_LEN		sizeof(u32)
+/* The longest message is MPAM_MSC_WRITE, with 4 parameters. */
+#define MPAM_FB_MAX_MSG_SIZE	(MPAM_FB_PROT_HEADER_LEN + 4 * sizeof(u32))
+
+#define MPAM_FB_VERSION_MAJOR_MASK	GENMASK(31, 16)
+#define MPAM_FB_VERSION_MINOR_MASK	GENMASK(15, 0)
 
 /*
  * MPAM MSCs have the following register layout. See:
Signature · 1 line
2.43.0
#29 lore raw

Re: [PATCH v5 10/10] arm_mpam: detect and enable MPAM-Fb PCC support

Quoted 22 lines · Andre Przywara
drivers/resctrl/mpam_devices.c +2 -2
@@ -2231,7 +2316,7 @@ static void mpam_msc_drv_remove(struct platform_device *pdev)
  static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
  {
  	int err;
-	u32 tmp;
+	u32 pcc_subspace_id;
  	struct mpam_msc *msc;
  	struct resource *msc_res;
  	struct device *dev = &pdev->dev;
@@ -2279,7 +2364,7 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
  	if (err)
  		return ERR_PTR(err);
 
-	if (device_property_read_u32(&pdev->dev, "pcc-channel", &tmp))
+	if (device_property_read_u32(dev, "pcc-channel", &pcc_subspace_id))
  		msc->iface = MPAM_IFACE_MMIO;
  	else
  		msc->iface = MPAM_IFACE_PCC;
@@ -2295,6 +2380,36 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
  		}
  		msc->mapped_hwpage_sz = msc_res->end - msc_res->start;
  		msc->mapped_hwpage = io;

drivers/resctrl/mpam_devices.c static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev) · line 2392

Quoted 10 lines
drivers/resctrl/mpam_devices.c +10 -0
+	} else if (msc->iface == MPAM_IFACE_PCC) {
+		int ret;
+
+		msc->pcc_chan = mpam_pcc_chan_get(dev, pcc_subspace_id);
+		if (IS_ERR(msc->pcc_chan)) {
+			pr_err("Failed to request MSC PCC channel\n");
+			return ERR_CAST(msc->pcc_chan);
+		}
+
+		if (msc->pcc_chan->pcc_chan->shmem_size < MPAM_FB_MAX_MSG_SIZE) {

I think the PCC shared-memory size undercounts the required region. From my understanding shmem_size is the length of the complete PCC shared-memory region advertised by the PCCT and mapped by the PCC driver. MPAM_FB_MAX_MSG_SIZE is the 4-byte protocol header stored in pcc_shmem->command + the 16-byte write payload.

However, mpam_fb_build_write_message() places the payload after the complete 16 byte struct acpi_pcct_ext_pcc_shared_memory. The largest request therefore occupies 32 bytes of shared memory region(16-byte PCC header+ 16-byte write payload). Thus a firmware region between 20 and 31 bytes currently passes the check but overruns when constructing a write request. I think this should be

if (msc->pcc_chan->pcc_chan->shmem_size <

    sizeof(struct acpi_pcct_ext_pcc_shared_memory) + MPAM_FB_MAX_MSG_SIZE - MPAM_FB_PROT_HEADER_LEN)

Quoted 25 lines
drivers/resctrl/mpam_internal.h +5 -1
index a2193e7df57c..e926b7feb72a 100644
@@ -77,7 +77,6 @@ struct mpam_msc {
  	/* Not modified after mpam_is_enabled() becomes true */
  	enum mpam_msc_iface	iface;
  	struct mpam_pcc_chan	*pcc_chan;
-	int			mpam_fb_msc_id;	/* in its own name space */
  	u32			nrdy_usec;
  	cpumask_t		accessibility;
  	bool			has_extd_esr;
@@ -532,6 +531,11 @@ int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value);
  int mpam_fb_get_protocol_version(struct mpam_msc *msc);
 
  #define MPAM_FB_PROT_HEADER_LEN		sizeof(u32)
+/* The longest message is MPAM_MSC_WRITE, with 4 parameters. */
+#define MPAM_FB_MAX_MSG_SIZE	(MPAM_FB_PROT_HEADER_LEN + 4 * sizeof(u32))
+
+#define MPAM_FB_VERSION_MAJOR_MASK	GENMASK(31, 16)
+#define MPAM_FB_VERSION_MINOR_MASK	GENMASK(15, 0)
 
  /*
   * MPAM MSCs have the following register layout. See:
#30 lore raw

Re: [PATCH v5 10/10] arm_mpam: detect and enable MPAM-Fb PCC support

Hi Lee,

Quoted 39 lines · Lee Trager
Quoted 25 lines · Andre Przywara
drivers/resctrl/mpam_devices.c +1 -1
@@ -2231,7 +2316,7 @@ static void mpam_msc_drv_remove(struct 
 platform_device *pdev)
   static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
   {
       int err;
-    u32 tmp;
+    u32 pcc_subspace_id;
       struct mpam_msc *msc;
       struct resource *msc_res;

      struct device *dev = &pdev->dev;

drivers/resctrl/mpam_devices.c +1 -1
@@ -2279,7 +2364,7 @@ static struct mpam_msc 
 *do_mpam_msc_drv_probe(struct platform_device *pdev)
       if (err)
           return ERR_PTR(err);
-    if (device_property_read_u32(&pdev->dev, "pcc-channel", &tmp))
+    if (device_property_read_u32(dev, "pcc-channel", &pcc_subspace_id))
           msc->iface = MPAM_IFACE_MMIO;

      else
          msc->iface = MPAM_IFACE_PCC;

drivers/resctrl/mpam_devices.c +1 -0
@@ -2295,6 +2380,36 @@ static struct mpam_msc 
 *do_mpam_msc_drv_probe(struct platform_device *pdev)
           }
           msc->mapped_hwpage_sz = msc_res->end - msc_res->start;
           msc->mapped_hwpage = io;
+    } else if (msc->iface == MPAM_IFACE_PCC) {

drivers/resctrl/mpam_devices.c static struct mpam_msc · line 2394

Quoted 10 lines
drivers/resctrl/mpam_devices.c +9 -0
+        int ret;
+
+        msc->pcc_chan = mpam_pcc_chan_get(dev, pcc_subspace_id);
+        if (IS_ERR(msc->pcc_chan)) {
+            pr_err("Failed to request MSC PCC channel\n");
+            return ERR_CAST(msc->pcc_chan);
+        }
+
+        if (msc->pcc_chan->pcc_chan->shmem_size < 
 MPAM_FB_MAX_MSG_SIZE) {

I think the PCC shared-memory size undercounts the required region. From my understanding shmem_size is the length of the complete PCC shared- memory region advertised by the PCCT and mapped by the PCC driver. MPAM_FB_MAX_MSG_SIZE is the 4-byte protocol header stored in pcc_shmem-

Quoted 3 lines
Quoted 1 line

command + the 16-byte write payload.

However, mpam_fb_build_write_message() places the payload after the complete 16 byte struct acpi_pcct_ext_pcc_shared_memory. The largest request therefore occupies 32 bytes of shared memory region(16-byte PCC header+ 16-byte write payload). Thus a firmware region between 20 and 31 bytes currently passes the check but overruns when constructing a write request. I think this should be

Ah, you are right, there are indeed multiple things at play here.

Quoted 3 lines

if (msc->pcc_chan->pcc_chan->shmem_size <>
    sizeof(struct acpi_pcct_ext_pcc_shared_memory) + MPAM_FB_MAX_MSG_SIZE - MPAM_FB_PROT_HEADER_LEN)

I simplified that by just dropping MPAM_FB_PROT_HEADER_LEN from the MPAM_FB_MAX_MSG_SIZE definition.

Thanks for having a thorough look!

Cheers,
Andre

Quoted 25 lines
Quoted 25 lines
drivers/resctrl/mpam_internal.h +5 -1
index a2193e7df57c..e926b7feb72a 100644
@@ -77,7 +77,6 @@ struct mpam_msc {
       /* Not modified after mpam_is_enabled() becomes true */
       enum mpam_msc_iface    iface;
       struct mpam_pcc_chan    *pcc_chan;
-    int            mpam_fb_msc_id;    /* in its own name space */
       u32            nrdy_usec;
       cpumask_t        accessibility;
       bool            has_extd_esr;
@@ -532,6 +531,11 @@ int mpam_fb_send_write_request(struct mpam_msc 
 *msc, u16 reg, u32 value);
   int mpam_fb_get_protocol_version(struct mpam_msc *msc);
   #define MPAM_FB_PROT_HEADER_LEN        sizeof(u32)
+/* The longest message is MPAM_MSC_WRITE, with 4 parameters. */
+#define MPAM_FB_MAX_MSG_SIZE    (MPAM_FB_PROT_HEADER_LEN + 4 * 
 sizeof(u32))
+
+#define MPAM_FB_VERSION_MAJOR_MASK    GENMASK(31, 16)
+#define MPAM_FB_VERSION_MINOR_MASK    GENMASK(15, 0)
   /*
    * MPAM MSCs have the following register layout. See:
#31 lore raw

Re: [PATCH v5 10/10] arm_mpam: detect and enable MPAM-Fb PCC support

In reply to Andre Przywara

Hi Andre ,

Quoted 41 lines · Andre Przywara

The Arm MPAM-Fb specification [1] describes a protocol to access MSC registers through a firmware interface. This requires a shared memory region to hold the message, and a mailbox to trigger the access. For ACPI this is wrapped as a PCC channel, described using existing ACPI abstractions.

Add code to parse those PCC table descriptions associated with an MSC, and store the parsed information in the MSC struct.
There can be multiple PCC channels, and each channel can serve multiple MSCs, so we need to keep track of the channel usage, using a list and a refcount.

This will be used by the MPAM-Fb access wrapper code.

[1] https://developer.arm.com/documentation/den0144/latest

---
  drivers/resctrl/mpam_devices.c  | 119 +++++++++++++++++++++++++++++++-
  drivers/resctrl/mpam_fb.c       |   4 +-
  drivers/resctrl/mpam_internal.h |   6 +-
  3 files changed, 124 insertions(+), 5 deletions(-)
drivers/resctrl/mpam_devices.c +2 -0
index e535603d2c7d..e360051f4aa7 100644
@@ -19,14 +19,19 @@
  #include <linux/irqdesc.h>
  #include <linux/list.h>
  #include <linux/lockdep.h>
+#include <linux/mailbox_client.h>
  #include <linux/mutex.h>
  #include <linux/platform_device.h>
  #include <linux/printk.h>
+#include <linux/property.h>
  #include <linux/srcu.h>
  #include <linux/spinlock.h>
  #include <linux/types.h>
  #include <linux/workqueue.h>
 

[...]

Quoted 31 lines

/*
* Number of MSCs that have been probed. Once all MSCs have been probed MPAM
* can be enabled.

drivers/resctrl/mpam_devices.c +4 -2
@@ -2221,6 +2304,8 @@ static void mpam_msc_drv_remove(struct platform_device *pdev)
  {
  	struct mpam_msc *msc = platform_get_drvdata(pdev);
 
+	mpam_pcc_chan_put(msc->pcc_chan);
+
  	mutex_lock(&mpam_list_lock);
  	mpam_msc_destroy(msc);
  	mutex_unlock(&mpam_list_lock);
@@ -2231,7 +2316,7 @@ static void mpam_msc_drv_remove(struct platform_device *pdev)
  static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
  {
  	int err;
-	u32 tmp;
+	u32 pcc_subspace_id;
  	struct mpam_msc *msc;
  	struct resource *msc_res;
  	struct device *dev = &pdev->dev;
@@ -2279,7 +2364,7 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
  	if (err)
  		return ERR_PTR(err);
 
-	if (device_property_read_u32(&pdev->dev, "pcc-channel", &tmp))
+	if (device_property_read_u32(dev, "pcc-channel", &pcc_subspace_id))
  		msc->iface = MPAM_IFACE_MMIO;
  	else
  		msc->iface = MPAM_IFACE_PCC;

drivers/resctrl/mpam_devices.c static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev) · line 2381

Quoted 2 lines
drivers/resctrl/mpam_devices.c +0 -0
@@ -2295,6 +2380,36 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
  		}
  		msc->mapped_hwpage_sz = msc_res->end - msc_res->start;

Ritwick pointed out in the v4 thread that mapped_hwpage_sz should use resource_size() rather than open-coding (end - start). That fix doesn't appear to have made it into v5.

Also I did verify the resource_size() in my setup by adding prints, it did work well.> msc->mapped_hwpage = io;

Quoted 67 lines
drivers/resctrl/mpam_devices.c +30 -0
+	} else if (msc->iface == MPAM_IFACE_PCC) {
+		int ret;
+
+		msc->pcc_chan = mpam_pcc_chan_get(dev, pcc_subspace_id);
+		if (IS_ERR(msc->pcc_chan)) {
+			pr_err("Failed to request MSC PCC channel\n");
+			return ERR_CAST(msc->pcc_chan);
+		}
+
+		if (msc->pcc_chan->pcc_chan->shmem_size < MPAM_FB_MAX_MSG_SIZE) {
+			pr_err("MPAM-Fb PCC channel size too small.\n");
+			mpam_pcc_chan_put(msc->pcc_chan);
+			return ERR_PTR(-ENOMEM);
+		}
+		ret = mpam_fb_get_protocol_version(msc);
+		if (ret < 0) {
+			pr_err("Cannot query MPAM-Fb protocol version.\n");
+			mpam_pcc_chan_put(msc->pcc_chan);
+
+			/* ret is an MPAM-Fb error code, return a Linux one. */
+			return ERR_PTR(-EIO);
+		}
+		if (FIELD_GET(MPAM_FB_VERSION_MAJOR_MASK, ret) != 1) {
+			pr_err("Incompatible MPAM-Fb protocol version %ld.%ld\n",
+			       FIELD_GET(MPAM_FB_VERSION_MAJOR_MASK, ret),
+			       FIELD_GET(MPAM_FB_VERSION_MINOR_MASK, ret));
+			mpam_pcc_chan_put(msc->pcc_chan);
+
+			return ERR_PTR(-EINVAL);
+		}
  	} else {
  		return ERR_PTR(-EINVAL);
  	}
drivers/resctrl/mpam_fb.c +2 -2
index dbe984241b0d..55c4778a8379 100644
@@ -190,13 +190,13 @@ static int mpam_fb_send_request(struct mpam_pcc_chan *pcc_chan, u32 msc_id,
 
  int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result)
  {
-	return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+	return mpam_fb_send_request(msc->pcc_chan, msc->id,
  				    reg, result, MPAM_MSC_READ_CMD);
  }
 
  int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value)
  {
-	return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+	return mpam_fb_send_request(msc->pcc_chan, msc->id,
  				    reg, &value, MPAM_MSC_WRITE_CMD);
  }
 
drivers/resctrl/mpam_internal.h +0 -1
index a2193e7df57c..e926b7feb72a 100644
@@ -77,7 +77,6 @@ struct mpam_msc {
  	/* Not modified after mpam_is_enabled() becomes true */
  	enum mpam_msc_iface	iface;
  	struct mpam_pcc_chan	*pcc_chan;
-	int			mpam_fb_msc_id;	/* in its own name space */
  	u32			nrdy_usec;
  	cpumask_t		accessibility;
  	bool			has_extd_esr;
@@ -532,6 +531,11 @@ int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value);
  int mpam_fb_get_protocol_version(struct mpam_msc *msc);

drivers/resctrl/mpam_internal.h int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value); · line 541

Quoted 10 lines
drivers/resctrl/mpam_internal.h +5 -0
 
  #define MPAM_FB_PROT_HEADER_LEN		sizeof(u32)
+/* The longest message is MPAM_MSC_WRITE, with 4 parameters. */
+#define MPAM_FB_MAX_MSG_SIZE	(MPAM_FB_PROT_HEADER_LEN + 4 * sizeof(u32))
+
+#define MPAM_FB_VERSION_MAJOR_MASK	GENMASK(31, 16)
+#define MPAM_FB_VERSION_MINOR_MASK	GENMASK(15, 0)
 
  /*
   * MPAM MSCs have the following register layout. See:

--
Best Regards,
Srivathsa

#32 lore raw

Re: [PATCH v5 10/10] arm_mpam: detect and enable MPAM-Fb PCC support

Hi Srivathsa,

many thanks for having a thorough look!

Quoted 77 lines · Srivathsa L Rao

Hi Andre ,

Quoted 41 lines · Andre Przywara

The Arm MPAM-Fb specification [1] describes a protocol to access MSC registers through a firmware interface. This requires a shared memory region to hold the message, and a mailbox to trigger the access. For ACPI this is wrapped as a PCC channel, described using existing ACPI abstractions.

Add code to parse those PCC table descriptions associated with an MSC, and store the parsed information in the MSC struct.
There can be multiple PCC channels, and each channel can serve multiple MSCs, so we need to keep track of the channel usage, using a list and a refcount.

This will be used by the MPAM-Fb access wrapper code.

[1] https://developer.arm.com/documentation/den0144/latest

---
  drivers/resctrl/mpam_devices.c  | 119 +++++++++++++++++++++++++++++++- drivers/resctrl/mpam_fb.c       |   4 +- drivers/resctrl/mpam_internal.h |   6 +-

  3 files changed, 124 insertions(+), 5 deletions(-)
drivers/resctrl/mpam_devices.c +2 -0
index e535603d2c7d..e360051f4aa7 100644
@@ -19,14 +19,19 @@
   #include <linux/irqdesc.h>
   #include <linux/list.h>
   #include <linux/lockdep.h>
+#include <linux/mailbox_client.h>
   #include <linux/mutex.h>
   #include <linux/platform_device.h>
   #include <linux/printk.h>
+#include <linux/property.h>
   #include <linux/srcu.h>
   #include <linux/spinlock.h>
   #include <linux/types.h>
   #include <linux/workqueue.h>
 

[...]

Quoted 31 lines

  /*
   * Number of MSCs that have been probed. Once all MSCs have been probed MPAM
   * can be enabled.

drivers/resctrl/mpam_devices.c +3 -1
@@ -2221,6 +2304,8 @@ static void mpam_msc_drv_remove(struct 
 platform_device *pdev)
   {
       struct mpam_msc *msc = platform_get_drvdata(pdev);
+    mpam_pcc_chan_put(msc->pcc_chan);
+
       mutex_lock(&mpam_list_lock);
       mpam_msc_destroy(msc);
       mutex_unlock(&mpam_list_lock);
@@ -2231,7 +2316,7 @@ static void mpam_msc_drv_remove(struct 
 platform_device *pdev)
   static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
   {
       int err;
-    u32 tmp;
+    u32 pcc_subspace_id;
       struct mpam_msc *msc;
       struct resource *msc_res;

      struct device *dev = &pdev->dev;

drivers/resctrl/mpam_devices.c +1 -1
@@ -2279,7 +2364,7 @@ static struct mpam_msc 
 *do_mpam_msc_drv_probe(struct platform_device *pdev)
       if (err)
           return ERR_PTR(err);
-    if (device_property_read_u32(&pdev->dev, "pcc-channel", &tmp))
+    if (device_property_read_u32(dev, "pcc-channel", &pcc_subspace_id))
           msc->iface = MPAM_IFACE_MMIO;

      else
          msc->iface = MPAM_IFACE_PCC;

Quoted 8 lines

drivers/resctrl/mpam_devices.c static struct mpam_msc · line 2382

Quoted 4 lines
drivers/resctrl/mpam_devices.c +0 -0
@@ -2295,6 +2380,36 @@ static struct mpam_msc 
 *do_mpam_msc_drv_probe(struct platform_device *pdev)
           }
           msc->mapped_hwpage_sz = msc_res->end - msc_res->start;

Ritwick pointed out in the v4 thread that mapped_hwpage_sz should use resource_size() rather than open-coding (end - start).  That fix doesn't appear to have made it into v5.

Yes, that's because that's an existing bug, so it deserves a separate fix (from him), with a Fixes: tag. I didn't want to complicate things by making it part of this series unnecessarily. He will send something ASAP. That being a fix it's less timing critical (as in: could be taken any time).

Cheers,
Andre

Quoted 84 lines

Also I did verify the resource_size() in my setup by adding prints, it did work well.>           msc->mapped_hwpage = io;

Quoted 76 lines
drivers/resctrl/mpam_devices.c +30 -0
+    } else if (msc->iface == MPAM_IFACE_PCC) {
+        int ret;
+
+        msc->pcc_chan = mpam_pcc_chan_get(dev, pcc_subspace_id);
+        if (IS_ERR(msc->pcc_chan)) {
+            pr_err("Failed to request MSC PCC channel\n");
+            return ERR_CAST(msc->pcc_chan);
+        }
+
+        if (msc->pcc_chan->pcc_chan->shmem_size < 
 MPAM_FB_MAX_MSG_SIZE) {
+            pr_err("MPAM-Fb PCC channel size too small.\n");
+            mpam_pcc_chan_put(msc->pcc_chan);
+            return ERR_PTR(-ENOMEM);
+        }
+        ret = mpam_fb_get_protocol_version(msc);
+        if (ret < 0) {
+            pr_err("Cannot query MPAM-Fb protocol version.\n");
+            mpam_pcc_chan_put(msc->pcc_chan);
+
+            /* ret is an MPAM-Fb error code, return a Linux one. */
+            return ERR_PTR(-EIO);
+        }
+        if (FIELD_GET(MPAM_FB_VERSION_MAJOR_MASK, ret) != 1) {
+            pr_err("Incompatible MPAM-Fb protocol version %ld.%ld\n",
+                   FIELD_GET(MPAM_FB_VERSION_MAJOR_MASK, ret),
+                   FIELD_GET(MPAM_FB_VERSION_MINOR_MASK, ret));
+            mpam_pcc_chan_put(msc->pcc_chan);
+
+            return ERR_PTR(-EINVAL);
+        }
       } else {
           return ERR_PTR(-EINVAL);

      }

drivers/resctrl/mpam_fb.c +2 -2
index dbe984241b0d..55c4778a8379 100644
@@ -190,13 +190,13 @@ static int mpam_fb_send_request(struct 
 mpam_pcc_chan *pcc_chan, u32 msc_id,
   int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result)
   {
-    return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+    return mpam_fb_send_request(msc->pcc_chan, msc->id,
                       reg, result, MPAM_MSC_READ_CMD);
   }
   int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value)
   {
-    return mpam_fb_send_request(msc->pcc_chan, msc->mpam_fb_msc_id,
+    return mpam_fb_send_request(msc->pcc_chan, msc->id,
                       reg, &value, MPAM_MSC_WRITE_CMD);
   }
drivers/resctrl/mpam_internal.h +5 -1
index a2193e7df57c..e926b7feb72a 100644
@@ -77,7 +77,6 @@ struct mpam_msc {
       /* Not modified after mpam_is_enabled() becomes true */
       enum mpam_msc_iface    iface;
       struct mpam_pcc_chan    *pcc_chan;
-    int            mpam_fb_msc_id;    /* in its own name space */
       u32            nrdy_usec;
       cpumask_t        accessibility;
       bool            has_extd_esr;
@@ -532,6 +531,11 @@ int mpam_fb_send_write_request(struct mpam_msc 
 *msc, u16 reg, u32 value);
   int mpam_fb_get_protocol_version(struct mpam_msc *msc);
   #define MPAM_FB_PROT_HEADER_LEN        sizeof(u32)
+/* The longest message is MPAM_MSC_WRITE, with 4 parameters. */
+#define MPAM_FB_MAX_MSG_SIZE    (MPAM_FB_PROT_HEADER_LEN + 4 * 
 sizeof(u32))
+
+#define MPAM_FB_VERSION_MAJOR_MASK    GENMASK(31, 16)
+#define MPAM_FB_VERSION_MINOR_MASK    GENMASK(15, 0)
   /*
    * MPAM MSCs have the following register layout. See:
Signature · 2 lines
Best Regards,
Srivathsa