Skip to contentSkip to conversation

arm_mpam: Add MPAM-Fb firmware support

PATCH Superseded Andre Przywara 2 participants 38 messages Jul 2 – Jul 10

Subsystems (3)

v2 of 12 Superseded Latest is v12 →

lore
Share & tools
lore
#0 lore raw

v2 cover arm_mpam: Add MPAM-Fb firmware support

Hi,

version 2 of the MPAM-Fb code, for firmware based MSC accesses. Major change is a rudimentary propagation of errors during the MSC accesses, this has been split up into 11 patches, coming first in this series. The patch to parse additional information from the ACPI table has been dropped (former patch 1/5), as it's actually not needed for MPAM-Fb, and the features parsed are not supported by the kernel yet. Quite some details have been fixed in the remaining patches, thanks to the diligent reviewers. This should address most of the comments, but I am pretty sure this won't be the final version ;-)
Find a changelog below. Based on v7.2-rc1.

=======================
The Arm MPAM specification defines Memory System Components (MSCs), which are devices that are programmed through an MMIO register frame. In some occasions this turned out to be too limiting: the MSC might be located behind a separate bus system (for instance inside an on-board controller), it might be mapped secure-only, or in a different processor socket without direct MMIO mapping. Also the MMIO access might be too slow or it would need to be filtered or otherwise access controlled. Finally there might be bugs in the MSC integration, which require a mediating firmware to be accessible.

To accommodate all those different use cases, the MPAM-Fb specification
[1] describes an alternative way to access MSCs. Accesses to an MSC would be wrapped in a message and communicated to the system using a shared-memory/mailbox system mostly mimicking the Arm SCMI spec. For ACPI systems, this would be abstracted through an ACPI PCC channel, which provides the shared-memory region and the mailbox trigger. We can lean on existing ACPI parsing code to register with these two subsystems, but cannot rely on the existing SCMI code in the kernel. This means we somewhat need to open code a very simplified SCMI handler, which just provides enough functionality for the very basic subset of SCMI that the MPAM-Fb spec requires.

The first 11 patches rework all MSC access wrappers to propagate error information. Pure MMIO based MSC accesses would never fail, but the MPAM-Fb access can go wrong in multiple ways. The patches have been split up purely for reviewing reasons, if the number is a problem, we could as well squash them.
Patch 12/15 solves a nasty problem: At the moment we protect stateful MSC register accesses (mon_sel) through a spinlock. Unfortunately the mailbox subsystem and the slow nature of the communication through this channel forbid MPAM-Fb access in atomic context. So this patch splits the lock in two: the "outer" one is a mutex, and only the inner one is a spinlock, which would need to be taken when programming MSCs inside an interrupt handler, for instance. We just deny the latter when using MPAM-Fb, ideally we wouldn't need that (no need to IPI another core when the MSC access does not need to be local to one particular core), or we simply deny that part of the functionality (access through perf). Patch 13/15 adds the code to redirect MSC accesses through the PCC shmem/mailbox system.
Patch 14/15 avoids the error IRQ handler to do an MSC access when using MPAM_Fb, since those accesses cannot run in atomic context. The final patch 15/15 then adds the code to detect and store the PCC channel information from the ACPI tables, and eventually enables MPAM-Fb accesses.

This would enable systems where some MSCs are not accessible via MMIO to use those components anyway.

Please have a look and test!

Cheers,
Andre

[1] https://developer.arm.com/documentation/den0144/latest

Changelog v1 .. v2:
- add patches to add error propagation to MSC access wrappers
- drop former patch 1/5 (not needed)
- drop lock in mpam_reprogram_msc(), to avoid double lock
- add support for multiple MSCs per PCC channel
- adjust SCMI protocol code to use a PCC subtype 3 channel
- let PCC code handle the PCC channel negotiation (due to subtype 3)
- drop SCMI names in shmem offsets, and use existing PCC type 3 struct
- adjust shmem field offsets to match MPAM-Fb spec, not pure SCMI
- prevent MPAM-Fb calls inside atomic smp_call_function_any() payload
- skip all MSC accesses inside the IRQ handler when using MPAM-Fb

Andre Przywara (14):
arm_mpam: let low level MSC read accessors return an error
arm_mpam: propagate MSC read errors for wrapper functions
arm_mpam: propagate MSC read errors for hw_probe functions
arm_mpam: propagate MSC read errors for mpam_msc_read_mbwu_l()
arm_mpam: propagate MSC read errors for msmon helpers
arm_mpam: propagate MSC read errors for __ris_msmon_read()
arm_mpam: propagate MSC read errors for state saving functions
arm_mpam: let low level MSC write accessors return an error
arm_mpam: propagate MSC write errors for ESR and part_sel wrappers
arm_mpam: propagate MSC write errors for hardware probe functions
arm_mpam: propagate MSC write errors for remaining MSC write users
arm_mpam: add MPAM-Fb MSC firmware access support
arm_mpam: prevent MPAM-Fb accesses inside IRQ handler
arm_mpam: detect and enable MPAM-Fb PCC support

James Morse (1):
arm_mpam: Split the locking around the mon_sel registers

 drivers/acpi/arm64/mpam.c       |   2 +
 drivers/resctrl/Makefile        |   2 +-
 drivers/resctrl/mpam_devices.c  | 630 ++++++++++++++++++++++++--------
 drivers/resctrl/mpam_fb.c       | 137 +++++++
 drivers/resctrl/mpam_fb.h       |  17 +
 drivers/resctrl/mpam_internal.h |  79 +++-
 include/linux/arm_mpam.h        |   2 +-
 7 files changed, 688 insertions(+), 181 deletions(-)
 create mode 100644 drivers/resctrl/mpam_fb.c
 create mode 100644 drivers/resctrl/mpam_fb.h

base-commit: dc59e4fea9d83f03bad6bddf3fa2e52491777482

Signature · 1 line
2.43.0
#1 lore raw

PATCH v2 1/15 arm_mpam: let low level MSC read accessors return an error

The upcoming MPAM-Fb support does not use MMIO primitives to access an MSC, but employs a shared-memory/doorbell based firmware protocol. Its complexity means that is must be able to handle errors, whereas we always assume an MSC access succeeds today.

Change the __mpam_read_reg() low level accessor function to return the requested data through a pointer, and return an error code instead. At the moment this is always 0, but this will change with alternative MSC access methods.
Change all users of those MSC read wrappers to comply with the new prototype, though at the moment without propagating any errors.

---
 drivers/resctrl/mpam_devices.c | 129 ++++++++++++++++++++-------------
 1 file changed, 78 insertions(+), 51 deletions(-)
drivers/resctrl/mpam_devices.c +78 -51
index b69f99488111..df14b4513382 100644
@@ -177,20 +177,23 @@ static void mpam_assert_partid_sizes_fixed(void)
 		WARN_ON_ONCE(!partid_max_published);
 }
 
-static u32 __mpam_read_reg(struct mpam_msc *msc, u16 reg)
+static int __mpam_read_reg(struct mpam_msc *msc, u16 reg, u32 *res)
 {
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
-	return readl_relaxed(msc->mapped_hwpage + reg);
+	*res = readl_relaxed(msc->mapped_hwpage + reg);
+
+	return 0;
 }
 
-static inline u32 _mpam_read_partsel_reg(struct mpam_msc *msc, u16 reg)
+static inline int _mpam_read_partsel_reg(struct mpam_msc *msc, u16 reg,
+					 u32 *res)
 {
 	lockdep_assert_held_once(&msc->part_sel_lock);
-	return __mpam_read_reg(msc, reg);
+	return __mpam_read_reg(msc, reg, res);
 }
 
-#define mpam_read_partsel_reg(msc, reg) _mpam_read_partsel_reg(msc, MPAMF_##reg)
+#define mpam_read_partsel_reg(msc, reg, res) _mpam_read_partsel_reg(msc, MPAMF_##reg, res)
 
 static void __mpam_write_reg(struct mpam_msc *msc, u16 reg, u32 val)
 {
@@ -208,13 +211,14 @@ static inline void _mpam_write_partsel_reg(struct mpam_msc *msc, u16 reg, u32 va
 
 #define mpam_write_partsel_reg(msc, reg, val)  _mpam_write_partsel_reg(msc, MPAMCFG_##reg, val)
 
-static inline u32 _mpam_read_monsel_reg(struct mpam_msc *msc, u16 reg)
+static inline int _mpam_read_monsel_reg(struct mpam_msc *msc, u16 reg,
+					u32 *res)
 {
 	mpam_mon_sel_lock_held(msc);
-	return __mpam_read_reg(msc, reg);
+	return __mpam_read_reg(msc, reg, res);
 }
 
-#define mpam_read_monsel_reg(msc, reg) _mpam_read_monsel_reg(msc, MSMON_##reg)
+#define mpam_read_monsel_reg(msc, reg, res) _mpam_read_monsel_reg(msc, MSMON_##reg, res)
 
 static inline void _mpam_write_monsel_reg(struct mpam_msc *msc, u16 reg, u32 val)
 {
@@ -226,10 +230,11 @@ static inline void _mpam_write_monsel_reg(struct mpam_msc *msc, u16 reg, u32 val
 
 static bool mpam_msc_check_aidr(struct mpam_msc *msc)
 {
-	u32 aidr = __mpam_read_reg(msc, MPAMF_AIDR);
-	u32 major = FIELD_GET(MPAMF_AIDR_ARCH_MAJOR_REV, aidr);
-	u32 minor = FIELD_GET(MPAMF_AIDR_ARCH_MINOR_REV, aidr);
+	u32 aidr, major, minor;
 
+	__mpam_read_reg(msc, MPAMF_AIDR, &aidr);
+	major = FIELD_GET(MPAMF_AIDR_ARCH_MAJOR_REV, aidr);
+	minor = FIELD_GET(MPAMF_AIDR_ARCH_MINOR_REV, aidr);
 	/*
 	 * v0.0 and >v2.x aren't supported, but anything else should be backward
 	 * compatible to v0.1 or v1.0.
@@ -244,20 +249,22 @@ static bool mpam_msc_check_aidr(struct mpam_msc *msc)
 
 static u64 mpam_msc_read_idr(struct mpam_msc *msc)
 {
-	u64 idr_high = 0, idr_low;
+	u32 idr_high = 0, idr_low;
 
 	lockdep_assert_held(&msc->part_sel_lock);
 
-	idr_low = mpam_read_partsel_reg(msc, IDR);
+	mpam_read_partsel_reg(msc, IDR, &idr_low);
 	if (FIELD_GET(MPAMF_IDR_EXT, idr_low))
-		idr_high = mpam_read_partsel_reg(msc, IDR + 4);
+		mpam_read_partsel_reg(msc, IDR + 4, &idr_high);
 
-	return (idr_high << 32) | idr_low;
+	return ((u64)idr_high << 32) | idr_low;
 }
 
 static void mpam_msc_clear_esr(struct mpam_msc *msc)
 {
-	u64 esr_low = __mpam_read_reg(msc, MPAMF_ESR);
+	u32 esr_low;
+
+	__mpam_read_reg(msc, MPAMF_ESR, &esr_low);
 
 	if (!esr_low)
 		return;
@@ -275,13 +282,13 @@ static void mpam_msc_clear_esr(struct mpam_msc *msc)
 
 static u64 mpam_msc_read_esr(struct mpam_msc *msc)
 {
-	u64 esr_high = 0, esr_low;
+	u32 esr_high = 0, esr_low;
 
-	esr_low = __mpam_read_reg(msc, MPAMF_ESR);
+	__mpam_read_reg(msc, MPAMF_ESR, &esr_low);
 	if (msc->has_extd_esr)
-		esr_high = __mpam_read_reg(msc, MPAMF_ESR + 4);
+		__mpam_read_reg(msc, MPAMF_ESR + 4, &esr_high);
 
-	return (esr_high << 32) | esr_low;
+	return ((u64)esr_high << 32) | esr_low;
 }
 
 static void __mpam_part_sel_raw(u32 partsel, struct mpam_msc *msc)
@@ -774,13 +781,13 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris)
 	mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
 
 	_mpam_write_monsel_reg(msc, MSMON_CSU, MSMON___NRDY);
-	now = _mpam_read_monsel_reg(msc, MSMON_CSU);
+	_mpam_read_monsel_reg(msc, MSMON_CSU, &now);
 	can_set = now & MSMON___NRDY;
 
 	_mpam_write_monsel_reg(msc, MSMON_CSU, 0);
 	/* Configuration change to try and coax hardware into setting nrdy */
 	mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0x1);
-	now = _mpam_read_monsel_reg(msc, MSMON_CSU);
+	_mpam_read_monsel_reg(msc, MSMON_CSU, &now);
 	can_clear = !(now & MSMON___NRDY);
 	mpam_mon_sel_unlock(msc);
 
@@ -800,7 +807,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 
 	/* Cache Capacity Partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_CCAP_PART, ris->idr)) {
-		u32 ccap_features = mpam_read_partsel_reg(msc, CCAP_IDR);
+		u32 ccap_features;
+
+		mpam_read_partsel_reg(msc, CCAP_IDR, &ccap_features);
 
 		props->cmax_wd = FIELD_GET(MPAMF_CCAP_IDR_CMAX_WD, ccap_features);
 		if (props->cmax_wd &&
@@ -823,7 +832,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 
 	/* Cache Portion partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_CPOR_PART, ris->idr)) {
-		u32 cpor_features = mpam_read_partsel_reg(msc, CPOR_IDR);
+		u32 cpor_features;
+
+		mpam_read_partsel_reg(msc, CPOR_IDR, &cpor_features);
 
 		props->cpbm_wd = FIELD_GET(MPAMF_CPOR_IDR_CPBM_WD, cpor_features);
 		if (props->cpbm_wd)
@@ -832,7 +843,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 
 	/* Memory bandwidth partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_MBW_PART, ris->idr)) {
-		u32 mbw_features = mpam_read_partsel_reg(msc, MBW_IDR);
+		u32 mbw_features;
+
+		mpam_read_partsel_reg(msc, MBW_IDR, &mbw_features);
 
 		/* portion bitmap resolution */
 		props->mbw_pbm_bits = FIELD_GET(MPAMF_MBW_IDR_BWPBM_WD, mbw_features);
@@ -860,7 +873,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 
 	/* Priority partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_PRI_PART, ris->idr)) {
-		u32 pri_features = mpam_read_partsel_reg(msc, PRI_IDR);
+		u32 pri_features;
+
+		mpam_read_partsel_reg(msc, PRI_IDR, &pri_features);
 
 		props->intpri_wd = FIELD_GET(MPAMF_PRI_IDR_INTPRI_WD, pri_features);
 		if (props->intpri_wd && FIELD_GET(MPAMF_PRI_IDR_HAS_INTPRI, pri_features)) {
@@ -879,7 +894,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 
 	/* Performance Monitoring */
 	if (FIELD_GET(MPAMF_IDR_HAS_MSMON, ris->idr)) {
-		u32 msmon_features = mpam_read_partsel_reg(msc, MSMON_IDR);
+		u32 msmon_features;
+
+		mpam_read_partsel_reg(msc, MSMON_IDR, &msmon_features);
 
 		/*
 		 * If the firmware max-nrdy-us property is missing, the
@@ -892,7 +909,8 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 		if (FIELD_GET(MPAMF_MSMON_IDR_MSMON_CSU, msmon_features)) {
 			u32 csumonidr;
 
-			csumonidr = mpam_read_partsel_reg(msc, CSUMON_IDR);
+			mpam_read_partsel_reg(msc, CSUMON_IDR, &csumonidr);
+
 			props->num_csu_mon = FIELD_GET(MPAMF_CSUMON_IDR_NUM_MON, csumonidr);
 			if (props->num_csu_mon) {
 				bool hw_managed;
@@ -915,7 +933,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 		}
 		if (FIELD_GET(MPAMF_MSMON_IDR_MSMON_MBWU, msmon_features)) {
 			bool has_long;
-			u32 mbwumon_idr = mpam_read_partsel_reg(msc, MBWUMON_IDR);
+			u32 mbwumon_idr;
+
+			mpam_read_partsel_reg(msc, MBWUMON_IDR, &mbwumon_idr);
 
 			props->num_mbwu_mon = FIELD_GET(MPAMF_MBWUMON_IDR_NUM_MON, mbwumon_idr);
 			if (props->num_mbwu_mon) {
@@ -945,8 +965,11 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	 */
 	if (FIELD_GET(MPAMF_IDR_HAS_PARTID_NRW, ris->idr) &&
 	    class->type != MPAM_CLASS_UNKNOWN) {
-		u32 nrwidr = mpam_read_partsel_reg(msc, PARTID_NRW_IDR);
-		u16 partid_max = FIELD_GET(MPAMF_PARTID_NRW_IDR_INTPARTID_MAX, nrwidr);
+		u16 partid_max;
+		u32 nrwidr;
+
+		mpam_read_partsel_reg(msc, PARTID_NRW_IDR, &nrwidr);
+		partid_max = FIELD_GET(MPAMF_PARTID_NRW_IDR_INTPARTID_MAX, nrwidr);
 
 		mpam_set_feature(mpam_feat_partid_nrw, props);
 		msc->partid_max = min(msc->partid_max, partid_max);
@@ -971,7 +994,8 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 	/* Grab an IDR value to find out how many RIS there are */
 	mutex_lock(&msc->part_sel_lock);
 	idr = mpam_msc_read_idr(msc);
-	msc->iidr = mpam_read_partsel_reg(msc, IIDR);
+	mpam_read_partsel_reg(msc, IIDR, &msc->iidr);
+
 	mutex_unlock(&msc->part_sel_lock);
 
 	mpam_enable_quirks(msc);
@@ -1039,24 +1063,24 @@ static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)
 {
 	int retry = 3;
 	u32 mbwu_l_low;
-	u64 mbwu_l_high1, mbwu_l_high2;
+	u32 mbwu_l_high1, mbwu_l_high2;
 
 	mpam_mon_sel_lock_held(msc);
 
 	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
-	mbwu_l_high2 = __mpam_read_reg(msc, MSMON_MBWU_L + 4);
+	__mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
 	do {
 		mbwu_l_high1 = mbwu_l_high2;
-		mbwu_l_low = __mpam_read_reg(msc, MSMON_MBWU_L);
-		mbwu_l_high2 = __mpam_read_reg(msc, MSMON_MBWU_L + 4);
+		__mpam_read_reg(msc, MSMON_MBWU_L, &mbwu_l_low);
+		__mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
 
 		retry--;
 	} while (mbwu_l_high1 != mbwu_l_high2 && retry > 0);
 
 	if (mbwu_l_high1 == mbwu_l_high2)
-		return (mbwu_l_high1 << 32) | mbwu_l_low;
+		return ((u64)mbwu_l_high1 << 32) | mbwu_l_low;
 
 	pr_warn("Failed to read a stable value\n");
 	return MSMON___L_NRDY;
@@ -1120,14 +1144,14 @@ static void read_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val,
 
 	switch (m->type) {
 	case mpam_feat_msmon_csu:
-		*ctl_val = mpam_read_monsel_reg(msc, CFG_CSU_CTL);
-		*flt_val = mpam_read_monsel_reg(msc, CFG_CSU_FLT);
+		mpam_read_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
+		mpam_read_monsel_reg(msc, CFG_CSU_FLT, flt_val);
 		break;
 	case mpam_feat_msmon_mbwu_31counter:
 	case mpam_feat_msmon_mbwu_44counter:
 	case mpam_feat_msmon_mbwu_63counter:
-		*ctl_val = mpam_read_monsel_reg(msc, CFG_MBWU_CTL);
-		*flt_val = mpam_read_monsel_reg(msc, CFG_MBWU_FLT);
+		mpam_read_monsel_reg(msc, CFG_MBWU_CTL, ctl_val);
+		mpam_read_monsel_reg(msc, CFG_MBWU_FLT, flt_val);
 		break;
 	default:
 		pr_warn("Unexpected monitor type %d\n", m->type);
@@ -1206,6 +1230,7 @@ static u64 mpam_msmon_overflow_val(enum mpam_device_features type,
 static void __ris_msmon_read(void *arg)
 {
 	u64 now;
+	u32 now32;
 	bool nrdy = false;
 	bool config_mismatch;
 	bool overflow = false;
@@ -1268,9 +1293,9 @@ static void __ris_msmon_read(void *arg)
 
 	switch (m->type) {
 	case mpam_feat_msmon_csu:
-		now = mpam_read_monsel_reg(msc, CSU);
-		nrdy = now & MSMON___NRDY;
-		now = FIELD_GET(MSMON___VALUE, now);
+		mpam_read_monsel_reg(msc, CSU, &now32);
+		nrdy = now32 & MSMON___NRDY;
+		now = FIELD_GET(MSMON___VALUE, now32);
 
 		if (mpam_has_quirk(IGNORE_CSU_NRDY, msc) && m->waited_timeout)
 			nrdy = false;
@@ -1288,9 +1313,9 @@ static void __ris_msmon_read(void *arg)
 			else
 				now = FIELD_GET(MSMON___L_VALUE, now);
 		} else {
-			now = mpam_read_monsel_reg(msc, MBWU);
-			nrdy = now & MSMON___NRDY;
-			now = FIELD_GET(MSMON___VALUE, now);
+			mpam_read_monsel_reg(msc, MBWU, &now32);
+			nrdy = now32 & MSMON___NRDY;
+			now = FIELD_GET(MSMON___VALUE, now32);
 		}
 
 		if (mpam_has_quirk(T241_MBW_COUNTER_SCALE_64, msc) &&
@@ -1685,16 +1710,18 @@ static int mpam_save_mbwu_state(void *arg)
 		mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) |
 			  FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx);
 		mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
-
-		cur_flt = mpam_read_monsel_reg(msc, CFG_MBWU_FLT);
-		cur_ctl = mpam_read_monsel_reg(msc, CFG_MBWU_CTL);
+		mpam_read_monsel_reg(msc, CFG_MBWU_FLT, &cur_flt);
+		mpam_read_monsel_reg(msc, CFG_MBWU_CTL, &cur_ctl);
 		mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
 
 		if (mpam_ris_has_mbwu_long_counter(ris)) {
 			val = mpam_msc_read_mbwu_l(msc);
 			mpam_msc_zero_mbwu_l(msc);
 		} else {
-			val = mpam_read_monsel_reg(msc, MBWU);
+			u32 val32;
+
+			mpam_read_monsel_reg(msc, MBWU, &val32);
+			val = val32;
 			mpam_write_monsel_reg(msc, MBWU, 0);
 		}
 
Signature · 1 line
2.43.0
#2 lore raw

PATCH v2 2/15 arm_mpam: propagate MSC read errors for wrapper functions

In reply to Andre Przywara

Allow the wrapper functions for IDR and ESR accesses to return an error, and propagate read errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 51 +++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 16 deletions(-)
drivers/resctrl/mpam_devices.c +35 -16
index df14b4513382..ce8738adb6ff 100644
@@ -247,27 +247,34 @@ static bool mpam_msc_check_aidr(struct mpam_msc *msc)
 	return true;
 }
 
-static u64 mpam_msc_read_idr(struct mpam_msc *msc)
+static int mpam_msc_read_idr(struct mpam_msc *msc, u64 *res)
 {
 	u32 idr_high = 0, idr_low;
+	int ret;
 
 	lockdep_assert_held(&msc->part_sel_lock);
 
-	mpam_read_partsel_reg(msc, IDR, &idr_low);
-	if (FIELD_GET(MPAMF_IDR_EXT, idr_low))
-		mpam_read_partsel_reg(msc, IDR + 4, &idr_high);
+	ret = mpam_read_partsel_reg(msc, IDR, &idr_low);
+	if (!ret && FIELD_GET(MPAMF_IDR_EXT, idr_low))
+		ret = mpam_read_partsel_reg(msc, IDR + 4, &idr_high);
 
-	return ((u64)idr_high << 32) | idr_low;
+	if (!ret)
+		*res = ((u64)idr_high << 32) | idr_low;
+
+	return ret;
 }
 
-static void mpam_msc_clear_esr(struct mpam_msc *msc)
+static int mpam_msc_clear_esr(struct mpam_msc *msc)
 {
 	u32 esr_low;
+	int ret;
 
-	__mpam_read_reg(msc, MPAMF_ESR, &esr_low);
+	ret = __mpam_read_reg(msc, MPAMF_ESR, &esr_low);
+	if (ret)
+		return ret;
 
 	if (!esr_low)
-		return;
+		return 0;
 
 	/*
 	 * Clearing the high/low bits of MPAMF_ESR can not be atomic.
@@ -277,18 +284,30 @@ static void mpam_msc_clear_esr(struct mpam_msc *msc)
 	 */
 	if (msc->has_extd_esr)
 		__mpam_write_reg(msc, MPAMF_ESR + 4, 0);
+
 	__mpam_write_reg(msc, MPAMF_ESR, 0);
+
+	return 0;
 }
 
-static u64 mpam_msc_read_esr(struct mpam_msc *msc)
+static int mpam_msc_read_esr(struct mpam_msc *msc, u64 *res)
 {
 	u32 esr_high = 0, esr_low;
+	int ret;
 
-	__mpam_read_reg(msc, MPAMF_ESR, &esr_low);
-	if (msc->has_extd_esr)
-		__mpam_read_reg(msc, MPAMF_ESR + 4, &esr_high);
+	ret = __mpam_read_reg(msc, MPAMF_ESR, &esr_low);
+	if (ret)
+		return ret;
+
+	if (msc->has_extd_esr) {
+		ret = __mpam_read_reg(msc, MPAMF_ESR + 4, &esr_high);
+		if (ret)
+			return ret;
+	}
 
-	return ((u64)esr_high << 32) | esr_low;
+	*res = ((u64)esr_high << 32) | esr_low;
+
+	return 0;
 }
 
 static void __mpam_part_sel_raw(u32 partsel, struct mpam_msc *msc)
@@ -993,7 +1012,7 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 
 	/* Grab an IDR value to find out how many RIS there are */
 	mutex_lock(&msc->part_sel_lock);
-	idr = mpam_msc_read_idr(msc);
+	mpam_msc_read_idr(msc, &idr);
 	mpam_read_partsel_reg(msc, IIDR, &msc->iidr);
 
 	mutex_unlock(&msc->part_sel_lock);
@@ -1009,7 +1028,7 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 	for (ris_idx = 0; ris_idx <= msc->ris_max; ris_idx++) {
 		mutex_lock(&msc->part_sel_lock);
 		__mpam_part_sel(ris_idx, 0, msc);
-		idr = mpam_msc_read_idr(msc);
+		mpam_msc_read_idr(msc, &idr);
 		mutex_unlock(&msc->part_sel_lock);
 
 		partid_max = FIELD_GET(MPAMF_IDR_PARTID_MAX, idr);
@@ -2492,7 +2511,7 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
 					   &msc->accessibility)))
 		return IRQ_NONE;
 
-	reg = mpam_msc_read_esr(msc);
+	mpam_msc_read_esr(msc, &reg);
 
 	errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
 	if (!errcode)
Signature · 1 line
2.43.0
maintainer
#3 lore raw

Re: [PATCH v2 02/15] arm_mpam: propagate MSC read errors for wrapper functions

Hi Andre,

Quoted 27 lines · Andre Przywara

Allow the wrapper functions for IDR and ESR accesses to return an error, and propagate read errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 51 +++++++++++++++++++++++-----------
 1 file changed, 35 insertions(+), 16 deletions(-)
drivers/resctrl/mpam_devices.c +2 -4
index df14b4513382..ce8738adb6ff 100644
@@ -247,27 +247,34 @@ static bool mpam_msc_check_aidr(struct mpam_msc *msc)
 	return true;
 }
 
-static u64 mpam_msc_read_idr(struct mpam_msc *msc)
+static int mpam_msc_read_idr(struct mpam_msc *msc, u64 *res)
 {
 	u32 idr_high = 0, idr_low;
+	int ret;
 
 	lockdep_assert_held(&msc->part_sel_lock);
 
-	mpam_read_partsel_reg(msc, IDR, &idr_low);
-	if (FIELD_GET(MPAMF_IDR_EXT, idr_low))
-		mpam_read_partsel_reg(msc, IDR + 4, &idr_high);

drivers/resctrl/mpam_devices.c static bool mpam_msc_check_aidr(struct mpam_msc *msc) · line 265

Quoted 10 lines
drivers/resctrl/mpam_devices.c +7 -1
+	ret = mpam_read_partsel_reg(msc, IDR, &idr_low);
+	if (!ret && FIELD_GET(MPAMF_IDR_EXT, idr_low))
+		ret = mpam_read_partsel_reg(msc, IDR + 4, &idr_high);
 
-	return ((u64)idr_high << 32) | idr_low;
+	if (!ret)
+		*res = ((u64)idr_high << 32) | idr_low;
+
+	return ret;
 }

Why does the pattern here, use of !ret rather than early return, differ from the one you've used in mpam_msc_read_esr()?

Thanks,

Ben

Quoted 81 lines
drivers/resctrl/mpam_devices.c +26 -11
-static void mpam_msc_clear_esr(struct mpam_msc *msc)
+static int mpam_msc_clear_esr(struct mpam_msc *msc)
 {
 	u32 esr_low;
+	int ret;
 
-	__mpam_read_reg(msc, MPAMF_ESR, &esr_low);
+	ret = __mpam_read_reg(msc, MPAMF_ESR, &esr_low);
+	if (ret)
+		return ret;
 
 	if (!esr_low)
-		return;
+		return 0;
 
 	/*
 	 * Clearing the high/low bits of MPAMF_ESR can not be atomic.
@@ -277,18 +284,30 @@ static void mpam_msc_clear_esr(struct mpam_msc *msc)
 	 */
 	if (msc->has_extd_esr)
 		__mpam_write_reg(msc, MPAMF_ESR + 4, 0);
+
 	__mpam_write_reg(msc, MPAMF_ESR, 0);
+
+	return 0;
 }
 
-static u64 mpam_msc_read_esr(struct mpam_msc *msc)
+static int mpam_msc_read_esr(struct mpam_msc *msc, u64 *res)
 {
 	u32 esr_high = 0, esr_low;
+	int ret;
 
-	__mpam_read_reg(msc, MPAMF_ESR, &esr_low);
-	if (msc->has_extd_esr)
-		__mpam_read_reg(msc, MPAMF_ESR + 4, &esr_high);
+	ret = __mpam_read_reg(msc, MPAMF_ESR, &esr_low);
+	if (ret)
+		return ret;
+
+	if (msc->has_extd_esr) {
+		ret = __mpam_read_reg(msc, MPAMF_ESR + 4, &esr_high);
+		if (ret)
+			return ret;
+	}
 
-	return ((u64)esr_high << 32) | esr_low;
+	*res = ((u64)esr_high << 32) | esr_low;
+
+	return 0;
 }
 
 static void __mpam_part_sel_raw(u32 partsel, struct mpam_msc *msc)
@@ -993,7 +1012,7 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 
 	/* Grab an IDR value to find out how many RIS there are */
 	mutex_lock(&msc->part_sel_lock);
-	idr = mpam_msc_read_idr(msc);
+	mpam_msc_read_idr(msc, &idr);
 	mpam_read_partsel_reg(msc, IIDR, &msc->iidr);
 
 	mutex_unlock(&msc->part_sel_lock);
@@ -1009,7 +1028,7 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 	for (ris_idx = 0; ris_idx <= msc->ris_max; ris_idx++) {
 		mutex_lock(&msc->part_sel_lock);
 		__mpam_part_sel(ris_idx, 0, msc);
-		idr = mpam_msc_read_idr(msc);
+		mpam_msc_read_idr(msc, &idr);
 		mutex_unlock(&msc->part_sel_lock);
 
 		partid_max = FIELD_GET(MPAMF_IDR_PARTID_MAX, idr);
@@ -2492,7 +2511,7 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
 					   &msc->accessibility)))
 		return IRQ_NONE;
 
-	reg = mpam_msc_read_esr(msc);
+	mpam_msc_read_esr(msc, &reg);
 
 	errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
 	if (!errcode)
#4 lore raw

Re: [PATCH v2 02/15] arm_mpam: propagate MSC read errors for wrapper functions

Hi,

Quoted 34 lines · Ben Horgan

Hi Andre,

Quoted 31 lines · Andre Przywara

Allow the wrapper functions for IDR and ESR accesses to return an error, and propagate read errors from the lower level up.

---
  drivers/resctrl/mpam_devices.c | 51 +++++++++++++++++++++++-----------
  1 file changed, 35 insertions(+), 16 deletions(-)
drivers/resctrl/mpam_devices.c +5 -4
index df14b4513382..ce8738adb6ff 100644
@@ -247,27 +247,34 @@ static bool mpam_msc_check_aidr(struct mpam_msc *msc)
  	return true;
  }
 
-static u64 mpam_msc_read_idr(struct mpam_msc *msc)
+static int mpam_msc_read_idr(struct mpam_msc *msc, u64 *res)
  {
  	u32 idr_high = 0, idr_low;
+	int ret;
 
  	lockdep_assert_held(&msc->part_sel_lock);
 
-	mpam_read_partsel_reg(msc, IDR, &idr_low);
-	if (FIELD_GET(MPAMF_IDR_EXT, idr_low))
-		mpam_read_partsel_reg(msc, IDR + 4, &idr_high);
+	ret = mpam_read_partsel_reg(msc, IDR, &idr_low);
+	if (!ret && FIELD_GET(MPAMF_IDR_EXT, idr_low))
+		ret = mpam_read_partsel_reg(msc, IDR + 4, &idr_high);
 
Quoted 9 lines

drivers/resctrl/mpam_devices.c static bool mpam_msc_check_aidr(struct mpam_msc *msc) · line 265

Quoted 6 lines
drivers/resctrl/mpam_devices.c +4 -1
-	return ((u64)idr_high << 32) | idr_low;
+	if (!ret)
+		*res = ((u64)idr_high << 32) | idr_low;
+
+	return ret;
  }

Why does the pattern here, use of !ret rather than early return, differ from the one you've used in mpam_msc_read_esr()?

No particular reason, I think I switched patterns, for instance depending on whether a lock was held or not, or when there are some side effect of that function, so probably just repeated the last exercise.

I changed it to return early here.

Cheers,
Andre

Quoted 83 lines
Quoted 82 lines
drivers/resctrl/mpam_devices.c +26 -11
-static void mpam_msc_clear_esr(struct mpam_msc *msc)
+static int mpam_msc_clear_esr(struct mpam_msc *msc)
  {
  	u32 esr_low;
+	int ret;
 
-	__mpam_read_reg(msc, MPAMF_ESR, &esr_low);
+	ret = __mpam_read_reg(msc, MPAMF_ESR, &esr_low);
+	if (ret)
+		return ret;
 
  	if (!esr_low)
-		return;
+		return 0;
 
  	/*
  	 * Clearing the high/low bits of MPAMF_ESR can not be atomic.
@@ -277,18 +284,30 @@ static void mpam_msc_clear_esr(struct mpam_msc *msc)
  	 */
  	if (msc->has_extd_esr)
  		__mpam_write_reg(msc, MPAMF_ESR + 4, 0);
+
  	__mpam_write_reg(msc, MPAMF_ESR, 0);
+
+	return 0;
  }
 
-static u64 mpam_msc_read_esr(struct mpam_msc *msc)
+static int mpam_msc_read_esr(struct mpam_msc *msc, u64 *res)
  {
  	u32 esr_high = 0, esr_low;
+	int ret;
 
-	__mpam_read_reg(msc, MPAMF_ESR, &esr_low);
-	if (msc->has_extd_esr)
-		__mpam_read_reg(msc, MPAMF_ESR + 4, &esr_high);
+	ret = __mpam_read_reg(msc, MPAMF_ESR, &esr_low);
+	if (ret)
+		return ret;
+
+	if (msc->has_extd_esr) {
+		ret = __mpam_read_reg(msc, MPAMF_ESR + 4, &esr_high);
+		if (ret)
+			return ret;
+	}
 
-	return ((u64)esr_high << 32) | esr_low;
+	*res = ((u64)esr_high << 32) | esr_low;
+
+	return 0;
  }
 
  static void __mpam_part_sel_raw(u32 partsel, struct mpam_msc *msc)
@@ -993,7 +1012,7 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 
  	/* Grab an IDR value to find out how many RIS there are */
  	mutex_lock(&msc->part_sel_lock);
-	idr = mpam_msc_read_idr(msc);
+	mpam_msc_read_idr(msc, &idr);
  	mpam_read_partsel_reg(msc, IIDR, &msc->iidr);
 
  	mutex_unlock(&msc->part_sel_lock);
@@ -1009,7 +1028,7 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
  	for (ris_idx = 0; ris_idx <= msc->ris_max; ris_idx++) {
  		mutex_lock(&msc->part_sel_lock);
  		__mpam_part_sel(ris_idx, 0, msc);
-		idr = mpam_msc_read_idr(msc);
+		mpam_msc_read_idr(msc, &idr);
  		mutex_unlock(&msc->part_sel_lock);
 
  		partid_max = FIELD_GET(MPAMF_IDR_PARTID_MAX, idr);
@@ -2492,7 +2511,7 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
  					   &msc->accessibility)))
  		return IRQ_NONE;
 
-	reg = mpam_msc_read_esr(msc);
+	mpam_msc_read_esr(msc, &reg);
 
  	errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
  	if (!errcode)
#5 lore raw

PATCH v2 3/15 arm_mpam: propagate MSC read errors for hw_probe functions

In reply to Andre Przywara

Allow the functions probing for MSC hardware and features to return an error, and propagate read errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 62 ++++++++++++++++++++++++----------
 1 file changed, 44 insertions(+), 18 deletions(-)
drivers/resctrl/mpam_devices.c +44 -18
index ce8738adb6ff..011d1e3544d7 100644
@@ -800,20 +800,22 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris)
 	mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
 
 	_mpam_write_monsel_reg(msc, MSMON_CSU, MSMON___NRDY);
-	_mpam_read_monsel_reg(msc, MSMON_CSU, &now);
+	if (_mpam_read_monsel_reg(msc, MSMON_CSU, &now))
+		return false;
 	can_set = now & MSMON___NRDY;
 
 	_mpam_write_monsel_reg(msc, MSMON_CSU, 0);
 	/* Configuration change to try and coax hardware into setting nrdy */
 	mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0x1);
-	_mpam_read_monsel_reg(msc, MSMON_CSU, &now);
+	if (_mpam_read_monsel_reg(msc, MSMON_CSU, &now))
+		return false;
 	can_clear = !(now & MSMON___NRDY);
 	mpam_mon_sel_unlock(msc);
 
 	return (!can_set || !can_clear);
 }
 
-static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
+static int mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 {
 	int err;
 	struct mpam_msc *msc = ris->vmsc->msc;
@@ -828,7 +830,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	if (FIELD_GET(MPAMF_IDR_HAS_CCAP_PART, ris->idr)) {
 		u32 ccap_features;
 
-		mpam_read_partsel_reg(msc, CCAP_IDR, &ccap_features);
+		err = mpam_read_partsel_reg(msc, CCAP_IDR, &ccap_features);
+		if (err)
+			return err;
 
 		props->cmax_wd = FIELD_GET(MPAMF_CCAP_IDR_CMAX_WD, ccap_features);
 		if (props->cmax_wd &&
@@ -853,7 +857,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	if (FIELD_GET(MPAMF_IDR_HAS_CPOR_PART, ris->idr)) {
 		u32 cpor_features;
 
-		mpam_read_partsel_reg(msc, CPOR_IDR, &cpor_features);
+		err = mpam_read_partsel_reg(msc, CPOR_IDR, &cpor_features);
+		if (err)
+			return err;
 
 		props->cpbm_wd = FIELD_GET(MPAMF_CPOR_IDR_CPBM_WD, cpor_features);
 		if (props->cpbm_wd)
@@ -863,8 +869,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	/* Memory bandwidth partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_MBW_PART, ris->idr)) {
 		u32 mbw_features;
-
-		mpam_read_partsel_reg(msc, MBW_IDR, &mbw_features);
+		err = mpam_read_partsel_reg(msc, MBW_IDR, &mbw_features);
+		if (err)
+			return err;
 
 		/* portion bitmap resolution */
 		props->mbw_pbm_bits = FIELD_GET(MPAMF_MBW_IDR_BWPBM_WD, mbw_features);
@@ -893,8 +900,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	/* Priority partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_PRI_PART, ris->idr)) {
 		u32 pri_features;
-
-		mpam_read_partsel_reg(msc, PRI_IDR, &pri_features);
+		err = mpam_read_partsel_reg(msc, PRI_IDR, &pri_features);
+		if (err)
+			return err;
 
 		props->intpri_wd = FIELD_GET(MPAMF_PRI_IDR_INTPRI_WD, pri_features);
 		if (props->intpri_wd && FIELD_GET(MPAMF_PRI_IDR_HAS_INTPRI, pri_features)) {
@@ -915,7 +923,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	if (FIELD_GET(MPAMF_IDR_HAS_MSMON, ris->idr)) {
 		u32 msmon_features;
 
-		mpam_read_partsel_reg(msc, MSMON_IDR, &msmon_features);
+		err = mpam_read_partsel_reg(msc, MSMON_IDR, &msmon_features);
+		if (err)
+			return err;
 
 		/*
 		 * If the firmware max-nrdy-us property is missing, the
@@ -928,7 +938,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 		if (FIELD_GET(MPAMF_MSMON_IDR_MSMON_CSU, msmon_features)) {
 			u32 csumonidr;
 
-			mpam_read_partsel_reg(msc, CSUMON_IDR, &csumonidr);
+			err = mpam_read_partsel_reg(msc, CSUMON_IDR, &csumonidr);
+			if (err)
+				return err;
 
 			props->num_csu_mon = FIELD_GET(MPAMF_CSUMON_IDR_NUM_MON, csumonidr);
 			if (props->num_csu_mon) {
@@ -954,7 +966,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 			bool has_long;
 			u32 mbwumon_idr;
 
-			mpam_read_partsel_reg(msc, MBWUMON_IDR, &mbwumon_idr);
+			err = mpam_read_partsel_reg(msc, MBWUMON_IDR, &mbwumon_idr);
+			if (err)
+				return err;
 
 			props->num_mbwu_mon = FIELD_GET(MPAMF_MBWUMON_IDR_NUM_MON, mbwumon_idr);
 			if (props->num_mbwu_mon) {
@@ -987,16 +1001,22 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 		u16 partid_max;
 		u32 nrwidr;
 
-		mpam_read_partsel_reg(msc, PARTID_NRW_IDR, &nrwidr);
+		err = mpam_read_partsel_reg(msc, PARTID_NRW_IDR, &nrwidr);
+		if (err)
+			return err;
+
 		partid_max = FIELD_GET(MPAMF_PARTID_NRW_IDR_INTPARTID_MAX, nrwidr);
 
 		mpam_set_feature(mpam_feat_partid_nrw, props);
 		msc->partid_max = min(msc->partid_max, partid_max);
 	}
+
+	return 0;
 }
 
 static int mpam_msc_hw_probe(struct mpam_msc *msc)
 {
+	int ret;
 	u64 idr;
 	u16 partid_max;
 	u8 ris_idx, pmg_max;
@@ -1012,10 +1032,12 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 
 	/* Grab an IDR value to find out how many RIS there are */
 	mutex_lock(&msc->part_sel_lock);
-	mpam_msc_read_idr(msc, &idr);
-	mpam_read_partsel_reg(msc, IIDR, &msc->iidr);
-
+	ret = mpam_msc_read_idr(msc, &idr);
+	if (!ret)
+		ret = mpam_read_partsel_reg(msc, IIDR, &msc->iidr);
 	mutex_unlock(&msc->part_sel_lock);
+	if (ret)
+		return ret;
 
 	mpam_enable_quirks(msc);
 
@@ -1028,8 +1050,10 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 	for (ris_idx = 0; ris_idx <= msc->ris_max; ris_idx++) {
 		mutex_lock(&msc->part_sel_lock);
 		__mpam_part_sel(ris_idx, 0, msc);
-		mpam_msc_read_idr(msc, &idr);
+		ret = mpam_msc_read_idr(msc, &idr);
 		mutex_unlock(&msc->part_sel_lock);
+		if (ret)
+			return ret;
 
 		partid_max = FIELD_GET(MPAMF_IDR_PARTID_MAX, idr);
 		pmg_max = FIELD_GET(MPAMF_IDR_PMG_MAX, idr);
@@ -1046,8 +1070,10 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 
 		mutex_lock(&msc->part_sel_lock);
 		__mpam_part_sel(ris_idx, 0, msc);
-		mpam_ris_hw_probe(ris);
+		ret = mpam_ris_hw_probe(ris);
 		mutex_unlock(&msc->part_sel_lock);
+		if (ret)
+			return ret;
 	}
 
 	/* Clear any stale errors */
Signature · 1 line
2.43.0
maintainer
#6 lore raw

Re: [PATCH v2 03/15] arm_mpam: propagate MSC read errors for hw_probe functions

Hi Andre,

Quoted 13 lines · Andre Przywara

Allow the functions probing for MSC hardware and features to return an error, and propagate read errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 62 ++++++++++++++++++++++++----------
 1 file changed, 44 insertions(+), 18 deletions(-)
drivers/resctrl/mpam_devices.c +0 -0
index ce8738adb6ff..011d1e3544d7 100644

drivers/resctrl/mpam_devices.c static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris) · line 804

Quoted 6 lines
drivers/resctrl/mpam_devices.c +2 -1
@@ -800,20 +800,22 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris)
 	mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
 
 	_mpam_write_monsel_reg(msc, MSMON_CSU, MSMON___NRDY);
-	_mpam_read_monsel_reg(msc, MSMON_CSU, &now);
+	if (_mpam_read_monsel_reg(msc, MSMON_CSU, &now))
+		return false;

This early return leaves the mon_sel lock held.

drivers/resctrl/mpam_devices.c static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris) · line 805

Quoted 1 line
drivers/resctrl/mpam_devices.c +0 -0
 	can_set = now & MSMON___NRDY;

As you're adding early returns to this function does it make sense to return early if can_set is false?

Thanks,

Ben

Quoted 161 lines
drivers/resctrl/mpam_devices.c +42 -17
 	_mpam_write_monsel_reg(msc, MSMON_CSU, 0);
 	/* Configuration change to try and coax hardware into setting nrdy */
 	mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0x1);
-	_mpam_read_monsel_reg(msc, MSMON_CSU, &now);
+	if (_mpam_read_monsel_reg(msc, MSMON_CSU, &now))
+		return false;
 	can_clear = !(now & MSMON___NRDY);
 	mpam_mon_sel_unlock(msc);
 
 	return (!can_set || !can_clear);
 }
 
-static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
+static int mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 {
 	int err;
 	struct mpam_msc *msc = ris->vmsc->msc;
@@ -828,7 +830,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	if (FIELD_GET(MPAMF_IDR_HAS_CCAP_PART, ris->idr)) {
 		u32 ccap_features;
 
-		mpam_read_partsel_reg(msc, CCAP_IDR, &ccap_features);
+		err = mpam_read_partsel_reg(msc, CCAP_IDR, &ccap_features);
+		if (err)
+			return err;
 
 		props->cmax_wd = FIELD_GET(MPAMF_CCAP_IDR_CMAX_WD, ccap_features);
 		if (props->cmax_wd &&
@@ -853,7 +857,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	if (FIELD_GET(MPAMF_IDR_HAS_CPOR_PART, ris->idr)) {
 		u32 cpor_features;
 
-		mpam_read_partsel_reg(msc, CPOR_IDR, &cpor_features);
+		err = mpam_read_partsel_reg(msc, CPOR_IDR, &cpor_features);
+		if (err)
+			return err;
 
 		props->cpbm_wd = FIELD_GET(MPAMF_CPOR_IDR_CPBM_WD, cpor_features);
 		if (props->cpbm_wd)
@@ -863,8 +869,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	/* Memory bandwidth partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_MBW_PART, ris->idr)) {
 		u32 mbw_features;
-
-		mpam_read_partsel_reg(msc, MBW_IDR, &mbw_features);
+		err = mpam_read_partsel_reg(msc, MBW_IDR, &mbw_features);
+		if (err)
+			return err;
 
 		/* portion bitmap resolution */
 		props->mbw_pbm_bits = FIELD_GET(MPAMF_MBW_IDR_BWPBM_WD, mbw_features);
@@ -893,8 +900,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	/* Priority partitioning */
 	if (FIELD_GET(MPAMF_IDR_HAS_PRI_PART, ris->idr)) {
 		u32 pri_features;
-
-		mpam_read_partsel_reg(msc, PRI_IDR, &pri_features);
+		err = mpam_read_partsel_reg(msc, PRI_IDR, &pri_features);
+		if (err)
+			return err;
 
 		props->intpri_wd = FIELD_GET(MPAMF_PRI_IDR_INTPRI_WD, pri_features);
 		if (props->intpri_wd && FIELD_GET(MPAMF_PRI_IDR_HAS_INTPRI, pri_features)) {
@@ -915,7 +923,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 	if (FIELD_GET(MPAMF_IDR_HAS_MSMON, ris->idr)) {
 		u32 msmon_features;
 
-		mpam_read_partsel_reg(msc, MSMON_IDR, &msmon_features);
+		err = mpam_read_partsel_reg(msc, MSMON_IDR, &msmon_features);
+		if (err)
+			return err;
 
 		/*
 		 * If the firmware max-nrdy-us property is missing, the
@@ -928,7 +938,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 		if (FIELD_GET(MPAMF_MSMON_IDR_MSMON_CSU, msmon_features)) {
 			u32 csumonidr;
 
-			mpam_read_partsel_reg(msc, CSUMON_IDR, &csumonidr);
+			err = mpam_read_partsel_reg(msc, CSUMON_IDR, &csumonidr);
+			if (err)
+				return err;
 
 			props->num_csu_mon = FIELD_GET(MPAMF_CSUMON_IDR_NUM_MON, csumonidr);
 			if (props->num_csu_mon) {
@@ -954,7 +966,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 			bool has_long;
 			u32 mbwumon_idr;
 
-			mpam_read_partsel_reg(msc, MBWUMON_IDR, &mbwumon_idr);
+			err = mpam_read_partsel_reg(msc, MBWUMON_IDR, &mbwumon_idr);
+			if (err)
+				return err;
 
 			props->num_mbwu_mon = FIELD_GET(MPAMF_MBWUMON_IDR_NUM_MON, mbwumon_idr);
 			if (props->num_mbwu_mon) {
@@ -987,16 +1001,22 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 		u16 partid_max;
 		u32 nrwidr;
 
-		mpam_read_partsel_reg(msc, PARTID_NRW_IDR, &nrwidr);
+		err = mpam_read_partsel_reg(msc, PARTID_NRW_IDR, &nrwidr);
+		if (err)
+			return err;
+
 		partid_max = FIELD_GET(MPAMF_PARTID_NRW_IDR_INTPARTID_MAX, nrwidr);
 
 		mpam_set_feature(mpam_feat_partid_nrw, props);
 		msc->partid_max = min(msc->partid_max, partid_max);
 	}
+
+	return 0;
 }
 
 static int mpam_msc_hw_probe(struct mpam_msc *msc)
 {
+	int ret;
 	u64 idr;
 	u16 partid_max;
 	u8 ris_idx, pmg_max;
@@ -1012,10 +1032,12 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 
 	/* Grab an IDR value to find out how many RIS there are */
 	mutex_lock(&msc->part_sel_lock);
-	mpam_msc_read_idr(msc, &idr);
-	mpam_read_partsel_reg(msc, IIDR, &msc->iidr);
-
+	ret = mpam_msc_read_idr(msc, &idr);
+	if (!ret)
+		ret = mpam_read_partsel_reg(msc, IIDR, &msc->iidr);
 	mutex_unlock(&msc->part_sel_lock);
+	if (ret)
+		return ret;
 
 	mpam_enable_quirks(msc);
 
@@ -1028,8 +1050,10 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 	for (ris_idx = 0; ris_idx <= msc->ris_max; ris_idx++) {
 		mutex_lock(&msc->part_sel_lock);
 		__mpam_part_sel(ris_idx, 0, msc);
-		mpam_msc_read_idr(msc, &idr);
+		ret = mpam_msc_read_idr(msc, &idr);
 		mutex_unlock(&msc->part_sel_lock);
+		if (ret)
+			return ret;
 
 		partid_max = FIELD_GET(MPAMF_IDR_PARTID_MAX, idr);
 		pmg_max = FIELD_GET(MPAMF_IDR_PMG_MAX, idr);
@@ -1046,8 +1070,10 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 
 		mutex_lock(&msc->part_sel_lock);
 		__mpam_part_sel(ris_idx, 0, msc);
-		mpam_ris_hw_probe(ris);
+		ret = mpam_ris_hw_probe(ris);
 		mutex_unlock(&msc->part_sel_lock);
+		if (ret)
+			return ret;
 	}
 
 	/* Clear any stale errors */
#7 lore raw

Re: [PATCH v2 03/15] arm_mpam: propagate MSC read errors for hw_probe functions

Hi Ben,

thanks for having a thorough look!

Quoted 18 lines · Ben Horgan

Hi Andre,

Quoted 15 lines · Andre Przywara

Allow the functions probing for MSC hardware and features to return an error, and propagate read errors from the lower level up.

---
  drivers/resctrl/mpam_devices.c | 62 ++++++++++++++++++++++++----------
  1 file changed, 44 insertions(+), 18 deletions(-)
drivers/resctrl/mpam_devices.c +0 -0
index ce8738adb6ff..011d1e3544d7 100644
@@ -800,20 +800,22 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris)
  	mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
 
Quoted 6 lines

drivers/resctrl/mpam_devices.c static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris) · line 804

Quoted 4 lines
drivers/resctrl/mpam_devices.c +2 -1
  	_mpam_write_monsel_reg(msc, MSMON_CSU, MSMON___NRDY);
-	_mpam_read_monsel_reg(msc, MSMON_CSU, &now);
+	if (_mpam_read_monsel_reg(msc, MSMON_CSU, &now))
+		return false;

This early return leaves the mon_sel lock held.

Ouch, well spotted. I think this "if (WARN_ON_ONCE(...))" wrapper distracted me from the lock... Changed it to use "goto out_unlock;"

Quoted 5 lines

drivers/resctrl/mpam_devices.c static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris) · line 805

Quoted 2 lines
drivers/resctrl/mpam_devices.c +0 -0
  	can_set = now & MSMON___NRDY;

As you're adding early returns to this function does it make sense to return early if can_set is false?

Well, this could have been done before as well, but it executes the remaining accesses nevertheless. I don't feel like judging whether this has a side effect, so just mimicked the current behaviour. I feel like this should be a separate patch, if we need it.

Thanks,
Andre

Quoted 162 lines
Quoted 161 lines
drivers/resctrl/mpam_devices.c +42 -17
  	_mpam_write_monsel_reg(msc, MSMON_CSU, 0);
  	/* Configuration change to try and coax hardware into setting nrdy */
  	mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0x1);
-	_mpam_read_monsel_reg(msc, MSMON_CSU, &now);
+	if (_mpam_read_monsel_reg(msc, MSMON_CSU, &now))
+		return false;
  	can_clear = !(now & MSMON___NRDY);
  	mpam_mon_sel_unlock(msc);
 
  	return (!can_set || !can_clear);
  }
 
-static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
+static int mpam_ris_hw_probe(struct mpam_msc_ris *ris)
  {
  	int err;
  	struct mpam_msc *msc = ris->vmsc->msc;
@@ -828,7 +830,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
  	if (FIELD_GET(MPAMF_IDR_HAS_CCAP_PART, ris->idr)) {
  		u32 ccap_features;
 
-		mpam_read_partsel_reg(msc, CCAP_IDR, &ccap_features);
+		err = mpam_read_partsel_reg(msc, CCAP_IDR, &ccap_features);
+		if (err)
+			return err;
 
  		props->cmax_wd = FIELD_GET(MPAMF_CCAP_IDR_CMAX_WD, ccap_features);
  		if (props->cmax_wd &&
@@ -853,7 +857,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
  	if (FIELD_GET(MPAMF_IDR_HAS_CPOR_PART, ris->idr)) {
  		u32 cpor_features;
 
-		mpam_read_partsel_reg(msc, CPOR_IDR, &cpor_features);
+		err = mpam_read_partsel_reg(msc, CPOR_IDR, &cpor_features);
+		if (err)
+			return err;
 
  		props->cpbm_wd = FIELD_GET(MPAMF_CPOR_IDR_CPBM_WD, cpor_features);
  		if (props->cpbm_wd)
@@ -863,8 +869,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
  	/* Memory bandwidth partitioning */
  	if (FIELD_GET(MPAMF_IDR_HAS_MBW_PART, ris->idr)) {
  		u32 mbw_features;
-
-		mpam_read_partsel_reg(msc, MBW_IDR, &mbw_features);
+		err = mpam_read_partsel_reg(msc, MBW_IDR, &mbw_features);
+		if (err)
+			return err;
 
  		/* portion bitmap resolution */
  		props->mbw_pbm_bits = FIELD_GET(MPAMF_MBW_IDR_BWPBM_WD, mbw_features);
@@ -893,8 +900,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
  	/* Priority partitioning */
  	if (FIELD_GET(MPAMF_IDR_HAS_PRI_PART, ris->idr)) {
  		u32 pri_features;
-
-		mpam_read_partsel_reg(msc, PRI_IDR, &pri_features);
+		err = mpam_read_partsel_reg(msc, PRI_IDR, &pri_features);
+		if (err)
+			return err;
 
  		props->intpri_wd = FIELD_GET(MPAMF_PRI_IDR_INTPRI_WD, pri_features);
  		if (props->intpri_wd && FIELD_GET(MPAMF_PRI_IDR_HAS_INTPRI, pri_features)) {
@@ -915,7 +923,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
  	if (FIELD_GET(MPAMF_IDR_HAS_MSMON, ris->idr)) {
  		u32 msmon_features;
 
-		mpam_read_partsel_reg(msc, MSMON_IDR, &msmon_features);
+		err = mpam_read_partsel_reg(msc, MSMON_IDR, &msmon_features);
+		if (err)
+			return err;
 
  		/*
  		 * If the firmware max-nrdy-us property is missing, the
@@ -928,7 +938,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
  		if (FIELD_GET(MPAMF_MSMON_IDR_MSMON_CSU, msmon_features)) {
  			u32 csumonidr;
 
-			mpam_read_partsel_reg(msc, CSUMON_IDR, &csumonidr);
+			err = mpam_read_partsel_reg(msc, CSUMON_IDR, &csumonidr);
+			if (err)
+				return err;
 
  			props->num_csu_mon = FIELD_GET(MPAMF_CSUMON_IDR_NUM_MON, csumonidr);
  			if (props->num_csu_mon) {
@@ -954,7 +966,9 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
  			bool has_long;
  			u32 mbwumon_idr;
 
-			mpam_read_partsel_reg(msc, MBWUMON_IDR, &mbwumon_idr);
+			err = mpam_read_partsel_reg(msc, MBWUMON_IDR, &mbwumon_idr);
+			if (err)
+				return err;
 
  			props->num_mbwu_mon = FIELD_GET(MPAMF_MBWUMON_IDR_NUM_MON, mbwumon_idr);
  			if (props->num_mbwu_mon) {
@@ -987,16 +1001,22 @@ static void mpam_ris_hw_probe(struct mpam_msc_ris *ris)
  		u16 partid_max;
  		u32 nrwidr;
 
-		mpam_read_partsel_reg(msc, PARTID_NRW_IDR, &nrwidr);
+		err = mpam_read_partsel_reg(msc, PARTID_NRW_IDR, &nrwidr);
+		if (err)
+			return err;
+
  		partid_max = FIELD_GET(MPAMF_PARTID_NRW_IDR_INTPARTID_MAX, nrwidr);
 
  		mpam_set_feature(mpam_feat_partid_nrw, props);
  		msc->partid_max = min(msc->partid_max, partid_max);
  	}
+
+	return 0;
  }
 
  static int mpam_msc_hw_probe(struct mpam_msc *msc)
  {
+	int ret;
  	u64 idr;
  	u16 partid_max;
  	u8 ris_idx, pmg_max;
@@ -1012,10 +1032,12 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 
  	/* Grab an IDR value to find out how many RIS there are */
  	mutex_lock(&msc->part_sel_lock);
-	mpam_msc_read_idr(msc, &idr);
-	mpam_read_partsel_reg(msc, IIDR, &msc->iidr);
-
+	ret = mpam_msc_read_idr(msc, &idr);
+	if (!ret)
+		ret = mpam_read_partsel_reg(msc, IIDR, &msc->iidr);
  	mutex_unlock(&msc->part_sel_lock);
+	if (ret)
+		return ret;
 
  	mpam_enable_quirks(msc);
 
@@ -1028,8 +1050,10 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
  	for (ris_idx = 0; ris_idx <= msc->ris_max; ris_idx++) {
  		mutex_lock(&msc->part_sel_lock);
  		__mpam_part_sel(ris_idx, 0, msc);
-		mpam_msc_read_idr(msc, &idr);
+		ret = mpam_msc_read_idr(msc, &idr);
  		mutex_unlock(&msc->part_sel_lock);
+		if (ret)
+			return ret;
 
  		partid_max = FIELD_GET(MPAMF_IDR_PARTID_MAX, idr);
  		pmg_max = FIELD_GET(MPAMF_IDR_PMG_MAX, idr);
@@ -1046,8 +1070,10 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 
  		mutex_lock(&msc->part_sel_lock);
  		__mpam_part_sel(ris_idx, 0, msc);
-		mpam_ris_hw_probe(ris);
+		ret = mpam_ris_hw_probe(ris);
  		mutex_unlock(&msc->part_sel_lock);
+		if (ret)
+			return ret;
  	}
 
  	/* Clear any stale errors */
#8 lore raw

PATCH v2 4/15 arm_mpam: propagate MSC read errors for mpam_msc_read_mbwu_l()

In reply to Andre Przywara

Allow the mpam_msc_read_mbwu_l() function to return an error, and propagate read errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
drivers/resctrl/mpam_devices.c +10 -3
index 011d1e3544d7..1d06902fb970 100644
@@ -1106,6 +1106,7 @@ static bool mpam_ris_has_mbwu_long_counter(struct mpam_msc_ris *ris)
 
 static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)
 {
+	int ret;
 	int retry = 3;
 	u32 mbwu_l_low;
 	u32 mbwu_l_high1, mbwu_l_high2;
@@ -1115,11 +1116,17 @@ static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)
 	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
-	__mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+	ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+	if (ret)
+		return MSMON___L_NRDY;
+
 	do {
 		mbwu_l_high1 = mbwu_l_high2;
-		__mpam_read_reg(msc, MSMON_MBWU_L, &mbwu_l_low);
-		__mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+		ret = __mpam_read_reg(msc, MSMON_MBWU_L, &mbwu_l_low);
+		if (!ret)
+			ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+		if (ret)
+			return MSMON___L_NRDY;
 
 		retry--;
 	} while (mbwu_l_high1 != mbwu_l_high2 && retry > 0);
Signature · 1 line
2.43.0
maintainer
#9 lore raw

Re: [PATCH v2 04/15] arm_mpam: propagate MSC read errors for mpam_msc_read_mbwu_l()

Hi Andre,

Quoted 13 lines · Andre Przywara

Allow the mpam_msc_read_mbwu_l() function to return an error, and propagate read errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
drivers/resctrl/mpam_devices.c +0 -0
index 011d1e3544d7..1d06902fb970 100644

drivers/resctrl/mpam_devices.c static bool mpam_ris_has_mbwu_long_counter(struct mpam_msc_ris *ris) · line 1107

Quoted 2 lines
drivers/resctrl/mpam_devices.c +0 -0
@@ -1106,6 +1106,7 @@ static bool mpam_ris_has_mbwu_long_counter(struct mpam_msc_ris *ris)
 
 static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)

Now that your changing the other helpers to return an error and pass a pointer to hold the register value it would seem more consistent for this call to follow the same pattern.

Thanks,

Ben

Quoted 26 lines
drivers/resctrl/mpam_devices.c +10 -3
 {
+	int ret;
 	int retry = 3;
 	u32 mbwu_l_low;
 	u32 mbwu_l_high1, mbwu_l_high2;
@@ -1115,11 +1116,17 @@ static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)
 	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
-	__mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+	ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+	if (ret)
+		return MSMON___L_NRDY;
+
 	do {
 		mbwu_l_high1 = mbwu_l_high2;
-		__mpam_read_reg(msc, MSMON_MBWU_L, &mbwu_l_low);
-		__mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+		ret = __mpam_read_reg(msc, MSMON_MBWU_L, &mbwu_l_low);
+		if (!ret)
+			ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+		if (ret)
+			return MSMON___L_NRDY;
 
 		retry--;
 	} while (mbwu_l_high1 != mbwu_l_high2 && retry > 0);
#10 lore raw

Re: [PATCH v2 04/15] arm_mpam: propagate MSC read errors for mpam_msc_read_mbwu_l()

Hi Ben,

Quoted 17 lines · Ben Horgan

Hi Andre,

Quoted 14 lines · Andre Przywara

Allow the mpam_msc_read_mbwu_l() function to return an error, and propagate read errors from the lower level up.

---
  drivers/resctrl/mpam_devices.c | 13 ++++++++++---
  1 file changed, 10 insertions(+), 3 deletions(-)
drivers/resctrl/mpam_devices.c +0 -0
index 011d1e3544d7..1d06902fb970 100644
@@ -1106,6 +1106,7 @@ static bool mpam_ris_has_mbwu_long_counter(struct mpam_msc_ris *ris)
 
Quoted 5 lines

drivers/resctrl/mpam_devices.c static bool mpam_ris_has_mbwu_long_counter(struct mpam_msc_ris *ris) · line 1107

Quoted 1 line
drivers/resctrl/mpam_devices.c +0 -0
  static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)

Now that your changing the other helpers to return an error and pass a pointer to hold the register value it would seem more consistent for this call to follow the same pattern.

Ah, I wasn't entirely sure whether this MSMON___L_NRDY is just some error flag (abuse), since NRDY is some kind of architecture feature. If that's indeed here the case, I will be changing the prototype and adjust the function.

Cheers,
Andre

Quoted 31 lines

Thanks,

Ben

Quoted 26 lines
drivers/resctrl/mpam_devices.c +10 -3
  {
+	int ret;
  	int retry = 3;
  	u32 mbwu_l_low;
  	u32 mbwu_l_high1, mbwu_l_high2;
@@ -1115,11 +1116,17 @@ static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)
  	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
  	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
-	__mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+	ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+	if (ret)
+		return MSMON___L_NRDY;
+
  	do {
  		mbwu_l_high1 = mbwu_l_high2;
-		__mpam_read_reg(msc, MSMON_MBWU_L, &mbwu_l_low);
-		__mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+		ret = __mpam_read_reg(msc, MSMON_MBWU_L, &mbwu_l_low);
+		if (!ret)
+			ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+		if (ret)
+			return MSMON___L_NRDY;
 
  		retry--;
  	} while (mbwu_l_high1 != mbwu_l_high2 && retry > 0);
maintainer
#11 lore raw

Re: [PATCH v2 04/15] arm_mpam: propagate MSC read errors for mpam_msc_read_mbwu_l()

Hi Andre,

Quoted 27 lines · Andre Przywara

Hi Ben,

Quoted 11 lines · Ben Horgan

Hi Andre,

Quoted 8 lines · Andre Przywara

Allow the mpam_msc_read_mbwu_l() function to return an error, and propagate read errors from the lower level up.

---
  drivers/resctrl/mpam_devices.c | 13 ++++++++++---

  1 file changed, 10 insertions(+), 3 deletions(-)
Quoted 12 lines

drivers/resctrl/mpam_devices.c static bool · line 1107

Quoted 8 lines
drivers/resctrl/ renamed from drivers/resctrl/mpam_devices.c +0 -0

mpam_devices.c
index 011d1e3544d7..1d06902fb970 100644

drivers/resctrl/mpam_devices.c +0 -0
@@ -1106,6 +1106,7 @@ static bool
 mpam_ris_has_mbwu_long_counter(struct mpam_msc_ris *ris)
     static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)

Now that your changing the other helpers to return an error and pass a pointer to hold the register value it would seem more consistent for this call to follow the same pattern.

Quoted 2 lines

Ah, I wasn't entirely sure whether this MSMON___L_NRDY is just some error flag (abuse), since NRDY is some kind of architecture feature.

Yeah, as only 32 bit reads are supported it's abused to report the case where we can't get a stable reading of the upper bits. Architecturally, NRDY is (optionally) used to say we haven't yet got a stable value after a configuration.

Thanks,

Ben

Quoted 40 lines

If that's indeed here the case, I will be changing the prototype and adjust the function.

Cheers,
Andre

Quoted 33 lines

Thanks,

Ben

Quoted 28 lines
drivers/resctrl/mpam_devices.c +1 -0
   {
+    int ret;
       int retry = 3;
       u32 mbwu_l_low;
       u32 mbwu_l_high1, mbwu_l_high2;
@@ -1115,11 +1116,17 @@ static u64 mpam_msc_read_mbwu_l(struct
 mpam_msc *msc)

      WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc-

Quoted 1 line

mapped_hwpage_sz);

      WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc-

Quoted 1 line

accessibility));

  -    __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2); +    ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2); +    if (ret)
+        return MSMON___L_NRDY;
+
      do {
          mbwu_l_high1 = mbwu_l_high2;
-        __mpam_read_reg(msc, MSMON_MBWU_L, &mbwu_l_low); -        __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2); +        ret = __mpam_read_reg(msc, MSMON_MBWU_L, &mbwu_l_low); +        if (!ret)
+            ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
+        if (ret)
+            return MSMON___L_NRDY;
            retry--;
      } while (mbwu_l_high1 != mbwu_l_high2 && retry > 0);

#12 lore raw

PATCH v2 5/15 arm_mpam: propagate MSC read errors for msmon helpers

In reply to Andre Przywara

Allow the helper functions for msmon accesses to return an error, and propagate read errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
drivers/resctrl/mpam_devices.c +12 -6
index 1d06902fb970..b259fe20a614 100644
@@ -1189,25 +1189,31 @@ static void gen_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val,
 	}
 }
 
-static void read_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val,
-				    u32 *flt_val)
+static int read_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val,
+				   u32 *flt_val)
 {
 	struct mpam_msc *msc = m->ris->vmsc->msc;
+	int ret;
 
 	switch (m->type) {
 	case mpam_feat_msmon_csu:
-		mpam_read_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
-		mpam_read_monsel_reg(msc, CFG_CSU_FLT, flt_val);
+		ret = mpam_read_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
+		if (!ret)
+			ret = mpam_read_monsel_reg(msc, CFG_CSU_FLT, flt_val);
 		break;
 	case mpam_feat_msmon_mbwu_31counter:
 	case mpam_feat_msmon_mbwu_44counter:
 	case mpam_feat_msmon_mbwu_63counter:
-		mpam_read_monsel_reg(msc, CFG_MBWU_CTL, ctl_val);
-		mpam_read_monsel_reg(msc, CFG_MBWU_FLT, flt_val);
+		ret = mpam_read_monsel_reg(msc, CFG_MBWU_CTL, ctl_val);
+		if (!ret)
+			ret = mpam_read_monsel_reg(msc, CFG_MBWU_FLT, flt_val);
 		break;
 	default:
 		pr_warn("Unexpected monitor type %d\n", m->type);
+		return -EINVAL;
 	}
+
+	return ret;
 }
 
 /* Remove values set by the hardware to prevent apparent mismatches. */
Signature · 1 line
2.43.0
#13 lore raw

PATCH v2 6/15 arm_mpam: propagate MSC read errors for __ris_msmon_read()

In reply to Andre Przywara

Allow the function for RIS accesses to return an error, and propagate read errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 35 +++++++++++++++++++++++++++-------
 1 file changed, 28 insertions(+), 7 deletions(-)
drivers/resctrl/mpam_devices.c +28 -7
index b259fe20a614..d18c7be86aaa 100644
@@ -1285,9 +1285,14 @@ static u64 mpam_msmon_overflow_val(enum mpam_device_features type,
 	return overflow_val;
 }
 
+/*
+ * This function might be called via smp_call_function_any(), so propagate
+ * errors inside the arg struct.
+ */
 static void __ris_msmon_read(void *arg)
 {
 	u64 now;
+	int ret;
 	u32 now32;
 	bool nrdy = false;
 	bool config_mismatch;
@@ -1326,7 +1331,9 @@ static void __ris_msmon_read(void *arg)
 	 * Read the existing configuration to avoid re-writing the same values.
 	 * This saves waiting for 'nrdy' on subsequent reads.
 	 */
-	read_msmon_ctl_flt_vals(m, &cur_ctl, &cur_flt);
+	ret = read_msmon_ctl_flt_vals(m, &cur_ctl, &cur_flt);
+	if (ret)
+		goto out_unlock;
 
 	if (mpam_feat_msmon_mbwu_31counter == m->type)
 		overflow = cur_ctl & MSMON_CFG_x_CTL_OFLOW_STATUS;
@@ -1351,7 +1358,9 @@ static void __ris_msmon_read(void *arg)
 
 	switch (m->type) {
 	case mpam_feat_msmon_csu:
-		mpam_read_monsel_reg(msc, CSU, &now32);
+		ret = mpam_read_monsel_reg(msc, CSU, &now32);
+		if (ret)
+			goto out_unlock;
 		nrdy = now32 & MSMON___NRDY;
 		now = FIELD_GET(MSMON___VALUE, now32);
 
@@ -1371,7 +1380,9 @@ static void __ris_msmon_read(void *arg)
 			else
 				now = FIELD_GET(MSMON___L_VALUE, now);
 		} else {
-			mpam_read_monsel_reg(msc, MBWU, &now32);
+			ret = mpam_read_monsel_reg(msc, MBWU, &now32);
+			if (ret)
+				goto out_unlock;
 			nrdy = now32 & MSMON___NRDY;
 			now = FIELD_GET(MSMON___VALUE, now32);
 		}
@@ -1402,10 +1413,15 @@ static void __ris_msmon_read(void *arg)
 	if (nrdy)
 		m->err = -EBUSY;
 
-	if (m->err)
-		return;
+	if (!m->err)
+		*m->val += now;
+
+	return;
 
-	*m->val += now;
+out_unlock:
+	mpam_mon_sel_unlock(msc);
+
+	m->err = ret;
 }
 
 static int _msmon_read(struct mpam_component *comp, struct mon_read *arg)
@@ -1729,6 +1745,7 @@ static int mpam_restore_mbwu_state(void *_ris)
 {
 	int i;
 	u64 val;
+	int ret = 0;
 	struct mon_read mwbu_arg;
 	struct mpam_msc_ris *ris = _ris;
 	struct mpam_class *class = ris->vmsc->comp->class;
@@ -1741,10 +1758,14 @@ static int mpam_restore_mbwu_state(void *_ris)
 			mwbu_arg.val = &val;
 
 			__ris_msmon_read(&mwbu_arg);
+			if (mwbu_arg.err) {
+				ret = mwbu_arg.err;
+				break;
+			}
 		}
 	}
 
-	return 0;
+	return ret;
 }
 
 /* Call with MSC cfg_lock held */
Signature · 1 line
2.43.0
maintainer
#14 lore raw

Re: [PATCH v2 06/15] arm_mpam: propagate MSC read errors for __ris_msmon_read()

Hi Andre,

Quoted 61 lines · Andre Przywara

Allow the function for RIS accesses to return an error, and propagate read errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 35 +++++++++++++++++++++++++++-------
 1 file changed, 28 insertions(+), 7 deletions(-)
drivers/resctrl/mpam_devices.c +14 -3
index b259fe20a614..d18c7be86aaa 100644
@@ -1285,9 +1285,14 @@ static u64 mpam_msmon_overflow_val(enum mpam_device_features type,
 	return overflow_val;
 }
 
+/*
+ * This function might be called via smp_call_function_any(), so propagate
+ * errors inside the arg struct.
+ */
 static void __ris_msmon_read(void *arg)
 {
 	u64 now;
+	int ret;
 	u32 now32;
 	bool nrdy = false;
 	bool config_mismatch;
@@ -1326,7 +1331,9 @@ static void __ris_msmon_read(void *arg)
 	 * Read the existing configuration to avoid re-writing the same values.
 	 * This saves waiting for 'nrdy' on subsequent reads.
 	 */
-	read_msmon_ctl_flt_vals(m, &cur_ctl, &cur_flt);
+	ret = read_msmon_ctl_flt_vals(m, &cur_ctl, &cur_flt);
+	if (ret)
+		goto out_unlock;
 
 	if (mpam_feat_msmon_mbwu_31counter == m->type)
 		overflow = cur_ctl & MSMON_CFG_x_CTL_OFLOW_STATUS;
@@ -1351,7 +1358,9 @@ static void __ris_msmon_read(void *arg)
 
 	switch (m->type) {
 	case mpam_feat_msmon_csu:
-		mpam_read_monsel_reg(msc, CSU, &now32);
+		ret = mpam_read_monsel_reg(msc, CSU, &now32);
+		if (ret)
+			goto out_unlock;
 		nrdy = now32 & MSMON___NRDY;
 		now = FIELD_GET(MSMON___VALUE, now32);
 
@@ -1371,7 +1380,9 @@ static void __ris_msmon_read(void *arg)
 			else
 				now = FIELD_GET(MSMON___L_VALUE, now);
 		} else {
-			mpam_read_monsel_reg(msc, MBWU, &now32);
+			ret = mpam_read_monsel_reg(msc, MBWU, &now32);
+			if (ret)
+				goto out_unlock;
 			nrdy = now32 & MSMON___NRDY;
 			now = FIELD_GET(MSMON___VALUE, now32);
 		}

drivers/resctrl/mpam_devices.c static void __ris_msmon_read(void *arg) · line 1417

Quoted 7 lines
drivers/resctrl/mpam_devices.c +2 -2
@@ -1402,10 +1413,15 @@ static void __ris_msmon_read(void *arg)
 	if (nrdy)
 		m->err = -EBUSY;
 
-	if (m->err)
-		return;
+	if (!m->err)
+		*m->val += now;

Why does this change? Can't you keep this the 'if' statement as before and more the 'return' after the '*m->val += now;'?

Thanks,

Ben

Quoted 35 lines
drivers/resctrl/mpam_devices.c +12 -2
+
+	return;
 
-	*m->val += now;
+out_unlock:
+	mpam_mon_sel_unlock(msc);
+
+	m->err = ret;
 }
 
 static int _msmon_read(struct mpam_component *comp, struct mon_read *arg)
@@ -1729,6 +1745,7 @@ static int mpam_restore_mbwu_state(void *_ris)
 {
 	int i;
 	u64 val;
+	int ret = 0;
 	struct mon_read mwbu_arg;
 	struct mpam_msc_ris *ris = _ris;
 	struct mpam_class *class = ris->vmsc->comp->class;
@@ -1741,10 +1758,14 @@ static int mpam_restore_mbwu_state(void *_ris)
 			mwbu_arg.val = &val;
 
 			__ris_msmon_read(&mwbu_arg);
+			if (mwbu_arg.err) {
+				ret = mwbu_arg.err;
+				break;
+			}
 		}
 	}
 
-	return 0;
+	return ret;
 }
 
 /* Call with MSC cfg_lock held */
#15 lore raw

Re: [PATCH v2 06/15] arm_mpam: propagate MSC read errors for __ris_msmon_read()

Hi Ben,

Quoted 67 lines · Ben Horgan

Hi Andre,

Quoted 64 lines · Andre Przywara

Allow the function for RIS accesses to return an error, and propagate read errors from the lower level up.

---
  drivers/resctrl/mpam_devices.c | 35 +++++++++++++++++++++++++++-------
  1 file changed, 28 insertions(+), 7 deletions(-)
drivers/resctrl/mpam_devices.c +14 -3
index b259fe20a614..d18c7be86aaa 100644
@@ -1285,9 +1285,14 @@ static u64 mpam_msmon_overflow_val(enum mpam_device_features type,
  	return overflow_val;
  }
 
+/*
+ * This function might be called via smp_call_function_any(), so propagate
+ * errors inside the arg struct.
+ */
  static void __ris_msmon_read(void *arg)
  {
  	u64 now;
+	int ret;
  	u32 now32;
  	bool nrdy = false;
  	bool config_mismatch;
@@ -1326,7 +1331,9 @@ static void __ris_msmon_read(void *arg)
  	 * Read the existing configuration to avoid re-writing the same values.
  	 * This saves waiting for 'nrdy' on subsequent reads.
  	 */
-	read_msmon_ctl_flt_vals(m, &cur_ctl, &cur_flt);
+	ret = read_msmon_ctl_flt_vals(m, &cur_ctl, &cur_flt);
+	if (ret)
+		goto out_unlock;
 
  	if (mpam_feat_msmon_mbwu_31counter == m->type)
  		overflow = cur_ctl & MSMON_CFG_x_CTL_OFLOW_STATUS;
@@ -1351,7 +1358,9 @@ static void __ris_msmon_read(void *arg)
 
  	switch (m->type) {
  	case mpam_feat_msmon_csu:
-		mpam_read_monsel_reg(msc, CSU, &now32);
+		ret = mpam_read_monsel_reg(msc, CSU, &now32);
+		if (ret)
+			goto out_unlock;
  		nrdy = now32 & MSMON___NRDY;
  		now = FIELD_GET(MSMON___VALUE, now32);
 
@@ -1371,7 +1380,9 @@ static void __ris_msmon_read(void *arg)
  			else
  				now = FIELD_GET(MSMON___L_VALUE, now);
  		} else {
-			mpam_read_monsel_reg(msc, MBWU, &now32);
+			ret = mpam_read_monsel_reg(msc, MBWU, &now32);
+			if (ret)
+				goto out_unlock;
  			nrdy = now32 & MSMON___NRDY;
  			now = FIELD_GET(MSMON___VALUE, now32);
  		}
@@ -1402,10 +1413,15 @@ static void __ris_msmon_read(void *arg)
  	if (nrdy)
  		m->err = -EBUSY;
 
Quoted 7 lines

drivers/resctrl/mpam_devices.c static void __ris_msmon_read(void *arg) · line 1417

Quoted 4 lines
drivers/resctrl/mpam_devices.c +2 -2
-	if (m->err)
-		return;
+	if (!m->err)
+		*m->val += now;

Why does this change? Can't you keep this the 'if' statement as before and more the 'return' after the '*m->val += now;'?

Because I need to explicitly return now anyway, since the error path below would just unlock again otherwise. So it would look like:

	if (m->err)
		return;

	*m->val += now;

	return;
out_unlock:
	mpam_mon_sel_unlock(msc);

...

which seemed like one return would be enough, and just conditionally do the assignment instead. I'd prefer to keep it like in this patch, but let me know if you disagree and I would change it to the one sketched above.

Cheers,
Andre

Quoted 41 lines

Thanks,

Ben

Quoted 35 lines
drivers/resctrl/mpam_devices.c +12 -2
+
+	return;
 
-	*m->val += now;
+out_unlock:
+	mpam_mon_sel_unlock(msc);
+
+	m->err = ret;
  }
 
  static int _msmon_read(struct mpam_component *comp, struct mon_read *arg)
@@ -1729,6 +1745,7 @@ static int mpam_restore_mbwu_state(void *_ris)
  {
  	int i;
  	u64 val;
+	int ret = 0;
  	struct mon_read mwbu_arg;
  	struct mpam_msc_ris *ris = _ris;
  	struct mpam_class *class = ris->vmsc->comp->class;
@@ -1741,10 +1758,14 @@ static int mpam_restore_mbwu_state(void *_ris)
  			mwbu_arg.val = &val;
 
  			__ris_msmon_read(&mwbu_arg);
+			if (mwbu_arg.err) {
+				ret = mwbu_arg.err;
+				break;
+			}
  		}
  	}
 
-	return 0;
+	return ret;
  }
 
  /* Call with MSC cfg_lock held */
#16 lore raw

PATCH v2 7/15 arm_mpam: propagate MSC read errors for state saving functions

In reply to Andre Przywara

Allow the mpam_save_mbwu_state() function to return an error, and propagate read errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 47 +++++++++++++++++++++-------------
 1 file changed, 29 insertions(+), 18 deletions(-)
drivers/resctrl/mpam_devices.c +29 -18
index d18c7be86aaa..c50ca0e4f426 100644
@@ -1773,6 +1773,7 @@ static int mpam_save_mbwu_state(void *arg)
 {
 	int i;
 	u64 val;
+	int ret;
 	struct mon_cfg *cfg;
 	u32 cur_flt, cur_ctl, mon_sel;
 	struct mpam_msc_ris *ris = arg;
@@ -1789,31 +1790,41 @@ static int mpam_save_mbwu_state(void *arg)
 		mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) |
 			  FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx);
 		mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
-		mpam_read_monsel_reg(msc, CFG_MBWU_FLT, &cur_flt);
-		mpam_read_monsel_reg(msc, CFG_MBWU_CTL, &cur_ctl);
-		mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
+		ret = mpam_read_monsel_reg(msc, CFG_MBWU_FLT, &cur_flt);
+		if (!ret)
+			ret = mpam_read_monsel_reg(msc, CFG_MBWU_CTL, &cur_ctl);
+		if (!ret)
+			mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
 
-		if (mpam_ris_has_mbwu_long_counter(ris)) {
-			val = mpam_msc_read_mbwu_l(msc);
-			mpam_msc_zero_mbwu_l(msc);
-		} else {
-			u32 val32;
+		if (!ret) {
+			if (mpam_ris_has_mbwu_long_counter(ris)) {
+				val = mpam_msc_read_mbwu_l(msc);
+				mpam_msc_zero_mbwu_l(msc);
+			} else {
+				u32 val32;
 
-			mpam_read_monsel_reg(msc, MBWU, &val32);
-			val = val32;
-			mpam_write_monsel_reg(msc, MBWU, 0);
+				ret = mpam_read_monsel_reg(msc, MBWU, &val32);
+				if (!ret) {
+					val = val32;
+					mpam_write_monsel_reg(msc, MBWU, 0);
+				}
+			}
 		}
 
-		cfg->mon = i;
-		cfg->pmg = FIELD_GET(MSMON_CFG_x_FLT_PMG, cur_flt);
-		cfg->match_pmg = FIELD_GET(MSMON_CFG_x_CTL_MATCH_PMG, cur_ctl);
-		cfg->partid = FIELD_GET(MSMON_CFG_x_FLT_PARTID, cur_flt);
-		mbwu_state->correction += val;
-		mbwu_state->enabled = FIELD_GET(MSMON_CFG_x_CTL_EN, cur_ctl);
+		if (!ret && val != MSMON___L_NRDY) {
+			cfg->mon = i;
+			cfg->pmg = FIELD_GET(MSMON_CFG_x_FLT_PMG, cur_flt);
+			cfg->match_pmg = FIELD_GET(MSMON_CFG_x_CTL_MATCH_PMG, cur_ctl);
+			cfg->partid = FIELD_GET(MSMON_CFG_x_FLT_PARTID, cur_flt);
+			mbwu_state->correction += val;
+			mbwu_state->enabled = FIELD_GET(MSMON_CFG_x_CTL_EN, cur_ctl);
+		}
 		mpam_mon_sel_unlock(msc);
+		if (ret)
+			break;
 	}
 
-	return 0;
+	return ret;
 }
 
 /*
Signature · 1 line
2.43.0
maintainer
#17 lore raw

Re: [PATCH v2 07/15] arm_mpam: propagate MSC read errors for state saving functions

Hi Andre,

Quoted 63 lines · Andre Przywara

Allow the mpam_save_mbwu_state() function to return an error, and propagate read errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 47 +++++++++++++++++++++-------------
 1 file changed, 29 insertions(+), 18 deletions(-)
drivers/resctrl/mpam_devices.c +19 -17
index d18c7be86aaa..c50ca0e4f426 100644
@@ -1773,6 +1773,7 @@ static int mpam_save_mbwu_state(void *arg)
 {
 	int i;
 	u64 val;
+	int ret;
 	struct mon_cfg *cfg;
 	u32 cur_flt, cur_ctl, mon_sel;
 	struct mpam_msc_ris *ris = arg;
@@ -1789,31 +1790,41 @@ static int mpam_save_mbwu_state(void *arg)
 		mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) |
 			  FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx);
 		mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
-		mpam_read_monsel_reg(msc, CFG_MBWU_FLT, &cur_flt);
-		mpam_read_monsel_reg(msc, CFG_MBWU_CTL, &cur_ctl);
-		mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
+		ret = mpam_read_monsel_reg(msc, CFG_MBWU_FLT, &cur_flt);
+		if (!ret)
+			ret = mpam_read_monsel_reg(msc, CFG_MBWU_CTL, &cur_ctl);
+		if (!ret)
+			mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
 
-		if (mpam_ris_has_mbwu_long_counter(ris)) {
-			val = mpam_msc_read_mbwu_l(msc);
-			mpam_msc_zero_mbwu_l(msc);
-		} else {
-			u32 val32;
+		if (!ret) {
+			if (mpam_ris_has_mbwu_long_counter(ris)) {
+				val = mpam_msc_read_mbwu_l(msc);
+				mpam_msc_zero_mbwu_l(msc);
+			} else {
+				u32 val32;
 
-			mpam_read_monsel_reg(msc, MBWU, &val32);
-			val = val32;
-			mpam_write_monsel_reg(msc, MBWU, 0);
+				ret = mpam_read_monsel_reg(msc, MBWU, &val32);
+				if (!ret) {
+					val = val32;
+					mpam_write_monsel_reg(msc, MBWU, 0);
+				}
+			}
 		}
 
-		cfg->mon = i;
-		cfg->pmg = FIELD_GET(MSMON_CFG_x_FLT_PMG, cur_flt);
-		cfg->match_pmg = FIELD_GET(MSMON_CFG_x_CTL_MATCH_PMG, cur_ctl);
-		cfg->partid = FIELD_GET(MSMON_CFG_x_FLT_PARTID, cur_flt);
-		mbwu_state->correction += val;
-		mbwu_state->enabled = FIELD_GET(MSMON_CFG_x_CTL_EN, cur_ctl);
+		if (!ret && val != MSMON___L_NRDY) {

drivers/resctrl/mpam_devices.c static int mpam_save_mbwu_state(void *arg) · line 1824

Quoted 10 lines
drivers/resctrl/mpam_devices.c +9 -0
+			cfg->mon = i;
+			cfg->pmg = FIELD_GET(MSMON_CFG_x_FLT_PMG, cur_flt);
+			cfg->match_pmg = FIELD_GET(MSMON_CFG_x_CTL_MATCH_PMG, cur_ctl);
+			cfg->partid = FIELD_GET(MSMON_CFG_x_FLT_PARTID, cur_flt);
+			mbwu_state->correction += val;
+			mbwu_state->enabled = FIELD_GET(MSMON_CFG_x_CTL_EN, cur_ctl);
+		}
 		mpam_mon_sel_unlock(msc);
+		if (ret)
+			break;

There is a lot of if(!ret) in this loop. Does it not end up cleaner to add an out_unlock label after the loop and jump to that in the failure cases?

Thanks,

Ben

Quoted 7 lines
drivers/resctrl/mpam_devices.c +1 -1
 	}
 
-	return 0;
+	return ret;
 }
 
 /*
#18 lore raw

Re: [PATCH v2 07/15] arm_mpam: propagate MSC read errors for state saving functions

Hi,

Quoted 72 lines · Ben Horgan

Hi Andre,

Quoted 69 lines · Andre Przywara

Allow the mpam_save_mbwu_state() function to return an error, and propagate read errors from the lower level up.

---
  drivers/resctrl/mpam_devices.c | 47 +++++++++++++++++++++-------------
  1 file changed, 29 insertions(+), 18 deletions(-)
drivers/resctrl/mpam_devices.c +25 -17
index d18c7be86aaa..c50ca0e4f426 100644
@@ -1773,6 +1773,7 @@ static int mpam_save_mbwu_state(void *arg)
  {
  	int i;
  	u64 val;
+	int ret;
  	struct mon_cfg *cfg;
  	u32 cur_flt, cur_ctl, mon_sel;
  	struct mpam_msc_ris *ris = arg;
@@ -1789,31 +1790,41 @@ static int mpam_save_mbwu_state(void *arg)
  		mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) |
  			  FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx);
  		mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
-		mpam_read_monsel_reg(msc, CFG_MBWU_FLT, &cur_flt);
-		mpam_read_monsel_reg(msc, CFG_MBWU_CTL, &cur_ctl);
-		mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
+		ret = mpam_read_monsel_reg(msc, CFG_MBWU_FLT, &cur_flt);
+		if (!ret)
+			ret = mpam_read_monsel_reg(msc, CFG_MBWU_CTL, &cur_ctl);
+		if (!ret)
+			mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
 
-		if (mpam_ris_has_mbwu_long_counter(ris)) {
-			val = mpam_msc_read_mbwu_l(msc);
-			mpam_msc_zero_mbwu_l(msc);
-		} else {
-			u32 val32;
+		if (!ret) {
+			if (mpam_ris_has_mbwu_long_counter(ris)) {
+				val = mpam_msc_read_mbwu_l(msc);
+				mpam_msc_zero_mbwu_l(msc);
+			} else {
+				u32 val32;
 
-			mpam_read_monsel_reg(msc, MBWU, &val32);
-			val = val32;
-			mpam_write_monsel_reg(msc, MBWU, 0);
+				ret = mpam_read_monsel_reg(msc, MBWU, &val32);
+				if (!ret) {
+					val = val32;
+					mpam_write_monsel_reg(msc, MBWU, 0);
+				}
+			}
  		}
 
-		cfg->mon = i;
-		cfg->pmg = FIELD_GET(MSMON_CFG_x_FLT_PMG, cur_flt);
-		cfg->match_pmg = FIELD_GET(MSMON_CFG_x_CTL_MATCH_PMG, cur_ctl);
-		cfg->partid = FIELD_GET(MSMON_CFG_x_FLT_PARTID, cur_flt);
-		mbwu_state->correction += val;
-		mbwu_state->enabled = FIELD_GET(MSMON_CFG_x_CTL_EN, cur_ctl);
+		if (!ret && val != MSMON___L_NRDY) {
+			cfg->mon = i;
+			cfg->pmg = FIELD_GET(MSMON_CFG_x_FLT_PMG, cur_flt);
+			cfg->match_pmg = FIELD_GET(MSMON_CFG_x_CTL_MATCH_PMG, cur_ctl);
+			cfg->partid = FIELD_GET(MSMON_CFG_x_FLT_PARTID, cur_flt);
+			mbwu_state->correction += val;
+			mbwu_state->enabled = FIELD_GET(MSMON_CFG_x_CTL_EN, cur_ctl);
Quoted 8 lines

drivers/resctrl/mpam_devices.c static int mpam_save_mbwu_state(void *arg) · line 1824

Quoted 4 lines
drivers/resctrl/mpam_devices.c +3 -0
+		}
  		mpam_mon_sel_unlock(msc);
+		if (ret)
+			break;

There is a lot of if(!ret) in this loop. Does it not end up cleaner to add an out_unlock label after the loop and jump to that in the failure cases?

Yeah, I changed it, I guess the diff looks better. Not entirely sure it's cleaner, there is now this little bitter taste of a global unlock where the current lock/unlock pair is *inside* the loop. Feel free to have a look and comment on an eventual v3 post.

Cheers,
Andre

Quoted 13 lines

Thanks,

Ben

Quoted 7 lines
drivers/resctrl/mpam_devices.c +1 -1
  	}
 
-	return 0;
+	return ret;
  }
 
  /*
#19 lore raw

PATCH v2 8/15 arm_mpam: let low level MSC write accessors return an error

In reply to Andre Przywara

The upcoming MPAM-Fb support does not use MMIO primitives to access an MSC, but employs a shared-memory/doorbell based firmware protocol. Its complexity means that is must be able to handle errors, whereas we always assume an MSC access succeeds today.

Change the __mpam_write_reg() low level accessor function to return an error code. At the moment this is always 0, but this will change with alternative MSC access methods.

Also change some low level wrappers to propagate the error.

---
 drivers/resctrl/mpam_devices.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
drivers/resctrl/mpam_devices.c +7 -5
index c50ca0e4f426..a7adc75a079c 100644
@@ -195,18 +195,20 @@ static inline int _mpam_read_partsel_reg(struct mpam_msc *msc, u16 reg,
 
 #define mpam_read_partsel_reg(msc, reg, res) _mpam_read_partsel_reg(msc, MPAMF_##reg, res)
 
-static void __mpam_write_reg(struct mpam_msc *msc, u16 reg, u32 val)
+static int __mpam_write_reg(struct mpam_msc *msc, u16 reg, u32 val)
 {
 	WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz);
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
 	writel_relaxed(val, msc->mapped_hwpage + reg);
+
+	return 0;
 }
 
-static inline void _mpam_write_partsel_reg(struct mpam_msc *msc, u16 reg, u32 val)
+static inline int _mpam_write_partsel_reg(struct mpam_msc *msc, u16 reg, u32 val)
 {
 	lockdep_assert_held_once(&msc->part_sel_lock);
-	__mpam_write_reg(msc, reg, val);
+	return __mpam_write_reg(msc, reg, val);
 }
 
 #define mpam_write_partsel_reg(msc, reg, val)  _mpam_write_partsel_reg(msc, MPAMCFG_##reg, val)
@@ -220,10 +222,10 @@ static inline int _mpam_read_monsel_reg(struct mpam_msc *msc, u16 reg,
 
 #define mpam_read_monsel_reg(msc, reg, res) _mpam_read_monsel_reg(msc, MSMON_##reg, res)
 
-static inline void _mpam_write_monsel_reg(struct mpam_msc *msc, u16 reg, u32 val)
+static inline int _mpam_write_monsel_reg(struct mpam_msc *msc, u16 reg, u32 val)
 {
 	mpam_mon_sel_lock_held(msc);
-	__mpam_write_reg(msc, reg, val);
+	return __mpam_write_reg(msc, reg, val);
 }
 
 #define mpam_write_monsel_reg(msc, reg, val)   _mpam_write_monsel_reg(msc, MSMON_##reg, val)
Signature · 1 line
2.43.0
#20 lore raw

PATCH v2 9/15 arm_mpam: propagate MSC write errors for ESR and part_sel wrappers

In reply to Andre Przywara

Allow the wrapper functions for part_sel and ESR accesses to return an error, and propagate write errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 23 ++++++++++++-----------
 1 file changed, 12 insertions(+), 11 deletions(-)
drivers/resctrl/mpam_devices.c +12 -11
index a7adc75a079c..23a5deb290b3 100644
@@ -284,12 +284,13 @@ static int mpam_msc_clear_esr(struct mpam_msc *msc)
 	 * lower half prevent hardware from updating either half of the
 	 * register.
 	 */
-	if (msc->has_extd_esr)
-		__mpam_write_reg(msc, MPAMF_ESR + 4, 0);
-
-	__mpam_write_reg(msc, MPAMF_ESR, 0);
+	if (msc->has_extd_esr) {
+		ret = __mpam_write_reg(msc, MPAMF_ESR + 4, 0);
+		if (ret)
+			return ret;
+	}
 
-	return 0;
+	return __mpam_write_reg(msc, MPAMF_ESR, 0);
 }
 
 static int mpam_msc_read_esr(struct mpam_msc *msc, u64 *res)
@@ -312,28 +313,28 @@ static int mpam_msc_read_esr(struct mpam_msc *msc, u64 *res)
 	return 0;
 }
 
-static void __mpam_part_sel_raw(u32 partsel, struct mpam_msc *msc)
+static int __mpam_part_sel_raw(u32 partsel, struct mpam_msc *msc)
 {
 	lockdep_assert_held(&msc->part_sel_lock);
 
-	mpam_write_partsel_reg(msc, PART_SEL, partsel);
+	return mpam_write_partsel_reg(msc, PART_SEL, partsel);
 }
 
-static void __mpam_part_sel(u8 ris_idx, u16 partid, struct mpam_msc *msc)
+static int __mpam_part_sel(u8 ris_idx, u16 partid, struct mpam_msc *msc)
 {
 	u32 partsel = FIELD_PREP(MPAMCFG_PART_SEL_RIS, ris_idx) |
 		      FIELD_PREP(MPAMCFG_PART_SEL_PARTID_SEL, partid);
 
-	__mpam_part_sel_raw(partsel, msc);
+	return __mpam_part_sel_raw(partsel, msc);
 }
 
-static void __mpam_intpart_sel(u8 ris_idx, u16 intpartid, struct mpam_msc *msc)
+static int __mpam_intpart_sel(u8 ris_idx, u16 intpartid, struct mpam_msc *msc)
 {
 	u32 partsel = FIELD_PREP(MPAMCFG_PART_SEL_RIS, ris_idx) |
 		      FIELD_PREP(MPAMCFG_PART_SEL_PARTID_SEL, intpartid) |
 		      MPAMCFG_PART_SEL_INTERNAL;
 
-	__mpam_part_sel_raw(partsel, msc);
+	return __mpam_part_sel_raw(partsel, msc);
 }
 
 int mpam_register_requestor(u16 partid_max, u8 pmg_max)
Signature · 1 line
2.43.0
#21 lore raw

PATCH v2 10/15 arm_mpam: propagate MSC write errors for hardware probe functions

In reply to Andre Przywara

Allow the MSC write accesses in the hardware probe functions to return an error, and propagate write errors from the lower level up.

---
 drivers/resctrl/mpam_devices.c | 29 +++++++++++++++++++----------
 1 file changed, 19 insertions(+), 10 deletions(-)
drivers/resctrl/mpam_devices.c +19 -10
index 23a5deb290b3..f3558d248c38 100644
@@ -792,24 +792,30 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris)
 
 	mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, 0) |
 		  FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx);
-	mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
+	if (mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel))
+		return false;
 
 	/* Hardware might ignore nrdy if it's not enabled */
 	ctl_val = MSMON_CFG_CSU_CTL_TYPE_CSU;
 	ctl_val |= MSMON_CFG_x_CTL_MATCH_PARTID;
 	ctl_val |= MSMON_CFG_x_CTL_MATCH_PMG;
 	ctl_val |= MSMON_CFG_x_CTL_EN;
-	mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0);
-	mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
+	if (mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0))
+		return false;
+	if (mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val))
+		return false;
 
-	_mpam_write_monsel_reg(msc, MSMON_CSU, MSMON___NRDY);
+	if (_mpam_write_monsel_reg(msc, MSMON_CSU, MSMON___NRDY))
+		return false;
 	if (_mpam_read_monsel_reg(msc, MSMON_CSU, &now))
 		return false;
 	can_set = now & MSMON___NRDY;
 
-	_mpam_write_monsel_reg(msc, MSMON_CSU, 0);
+	if (_mpam_write_monsel_reg(msc, MSMON_CSU, 0))
+		return false;
 	/* Configuration change to try and coax hardware into setting nrdy */
-	mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0x1);
+	if (mpam_write_monsel_reg(msc, CFG_CSU_FLT, 0x1))
+		return false;
 	if (_mpam_read_monsel_reg(msc, MSMON_CSU, &now))
 		return false;
 	can_clear = !(now & MSMON___NRDY);
@@ -1052,8 +1058,9 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 
 	for (ris_idx = 0; ris_idx <= msc->ris_max; ris_idx++) {
 		mutex_lock(&msc->part_sel_lock);
-		__mpam_part_sel(ris_idx, 0, msc);
-		ret = mpam_msc_read_idr(msc, &idr);
+		ret = __mpam_part_sel(ris_idx, 0, msc);
+		if (!ret)
+			ret = mpam_msc_read_idr(msc, &idr);
 		mutex_unlock(&msc->part_sel_lock);
 		if (ret)
 			return ret;
@@ -1072,9 +1079,11 @@ static int mpam_msc_hw_probe(struct mpam_msc *msc)
 		ris->idr = idr;
 
 		mutex_lock(&msc->part_sel_lock);
-		__mpam_part_sel(ris_idx, 0, msc);
-		ret = mpam_ris_hw_probe(ris);
+		ret = __mpam_part_sel(ris_idx, 0, msc);
+		if (!ret)
+			ret = mpam_ris_hw_probe(ris);
 		mutex_unlock(&msc->part_sel_lock);
+
 		if (ret)
 			return ret;
 	}
Signature · 1 line
2.43.0
#22 lore raw

PATCH v2 11/15 arm_mpam: propagate MSC write errors for remaining MSC write users

In reply to Andre Przywara

Allow the remaining MSC device functions to return an error, and propagate write errors from the lower level up.

propagate write errors up for mpam_save_mbwu_state()

---
 drivers/resctrl/mpam_devices.c | 76 ++++++++++++++++++++++------------
 1 file changed, 49 insertions(+), 27 deletions(-)
drivers/resctrl/mpam_devices.c +49 -27
index f3558d248c38..addc25e78345 100644
@@ -1150,15 +1150,20 @@ static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)
 	return MSMON___L_NRDY;
 }
 
-static void mpam_msc_zero_mbwu_l(struct mpam_msc *msc)
+static int mpam_msc_zero_mbwu_l(struct mpam_msc *msc)
 {
+	int ret;
+
 	mpam_mon_sel_lock_held(msc);
 
 	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
-	__mpam_write_reg(msc, MSMON_MBWU_L, 0);
-	__mpam_write_reg(msc, MSMON_MBWU_L + 4, 0);
+	ret = __mpam_write_reg(msc, MSMON_MBWU_L, 0);
+	if (!ret)
+		ret = __mpam_write_reg(msc, MSMON_MBWU_L + 4, 0);
+
+	return ret;
 }
 
 static void gen_msmon_ctl_flt_vals(struct mon_read *m, u32 *ctl_val,
@@ -1237,10 +1242,11 @@ static inline void clean_msmon_ctl_val(u32 *cur_ctl)
 		*cur_ctl &= ~MSMON_CFG_MBWU_CTL_OFLOW_STATUS_L;
 }
 
-static void write_msmon_ctl_flt_vals(struct mon_read *m, u32 ctl_val,
-				     u32 flt_val)
+static int write_msmon_ctl_flt_vals(struct mon_read *m, u32 ctl_val,
+				    u32 flt_val)
 {
 	struct mpam_msc *msc = m->ris->vmsc->msc;
+	int ret;
 
 	/*
 	 * Write the ctl_val with the enable bit cleared, reset the counter,
@@ -1248,26 +1254,37 @@ static void write_msmon_ctl_flt_vals(struct mon_read *m, u32 ctl_val,
 	 */
 	switch (m->type) {
 	case mpam_feat_msmon_csu:
-		mpam_write_monsel_reg(msc, CFG_CSU_FLT, flt_val);
-		mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
-		mpam_write_monsel_reg(msc, CSU, 0);
-		mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val | MSMON_CFG_x_CTL_EN);
+		ret = mpam_write_monsel_reg(msc, CFG_CSU_FLT, flt_val);
+		if (!ret)
+			ret = mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val);
+		if (!ret)
+			ret = mpam_write_monsel_reg(msc, CSU, 0);
+		if (!ret)
+			ret = mpam_write_monsel_reg(msc, CFG_CSU_CTL, ctl_val | MSMON_CFG_x_CTL_EN);
 		break;
 	case mpam_feat_msmon_mbwu_31counter:
 	case mpam_feat_msmon_mbwu_44counter:
 	case mpam_feat_msmon_mbwu_63counter:
-		mpam_write_monsel_reg(msc, CFG_MBWU_FLT, flt_val);
-		mpam_write_monsel_reg(msc, CFG_MBWU_CTL, ctl_val);
-		mpam_write_monsel_reg(msc, CFG_MBWU_CTL, ctl_val | MSMON_CFG_x_CTL_EN);
+		ret = mpam_write_monsel_reg(msc, CFG_MBWU_FLT, flt_val);
+		if (!ret)
+			ret = mpam_write_monsel_reg(msc, CFG_MBWU_CTL, ctl_val);
+		if (!ret)
+			ret = mpam_write_monsel_reg(msc, CFG_MBWU_CTL,
+						    ctl_val | MSMON_CFG_x_CTL_EN);
 		/* Counting monitors require NRDY to be reset by software */
-		if (m->type == mpam_feat_msmon_mbwu_31counter)
-			mpam_write_monsel_reg(msc, MBWU, 0);
-		else
-			mpam_msc_zero_mbwu_l(m->ris->vmsc->msc);
+		if (!ret) {
+			if (m->type == mpam_feat_msmon_mbwu_31counter)
+				ret = mpam_write_monsel_reg(msc, MBWU, 0);
+			else
+				ret = mpam_msc_zero_mbwu_l(m->ris->vmsc->msc);
+		}
 		break;
 	default:
 		pr_warn("Unexpected monitor type %d\n", m->type);
+		return -EINVAL;
 	}
+
+	return ret;
 }
 
 static u64 __mpam_msmon_overflow_val(enum mpam_device_features type)
@@ -1323,7 +1340,9 @@ static void __ris_msmon_read(void *arg)
 	}
 	mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, ctx->mon) |
 		  FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx);
-	mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
+	ret = mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
+	if (ret)
+		goto out_unlock;
 
 	switch (m->type) {
 	case mpam_feat_msmon_mbwu_31counter:
@@ -1359,14 +1378,16 @@ static void __ris_msmon_read(void *arg)
 			  cur_ctl != (ctl_val | MSMON_CFG_x_CTL_EN);
 
 	if (config_mismatch || reset_on_next_read) {
-		write_msmon_ctl_flt_vals(m, ctl_val, flt_val);
+		ret = write_msmon_ctl_flt_vals(m, ctl_val, flt_val);
 		overflow = false;
 	} else if (overflow) {
-		mpam_write_monsel_reg(msc, CFG_MBWU_CTL,
-				      cur_ctl &
-				      ~(MSMON_CFG_x_CTL_OFLOW_STATUS |
-					MSMON_CFG_MBWU_CTL_OFLOW_STATUS_L));
+		ret = mpam_write_monsel_reg(msc, CFG_MBWU_CTL,
+					    cur_ctl &
+					    ~(MSMON_CFG_x_CTL_OFLOW_STATUS |
+					    MSMON_CFG_MBWU_CTL_OFLOW_STATUS_L));
 	}
+	if (ret)
+		goto out_unlock;
 
 	switch (m->type) {
 	case mpam_feat_msmon_csu:
@@ -1801,24 +1822,25 @@ static int mpam_save_mbwu_state(void *arg)
 
 		mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) |
 			  FIELD_PREP(MSMON_CFG_MON_SEL_RIS, ris->ris_idx);
-		mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
-		ret = mpam_read_monsel_reg(msc, CFG_MBWU_FLT, &cur_flt);
+		ret = mpam_write_monsel_reg(msc, CFG_MON_SEL, mon_sel);
+		if (!ret)
+			ret = mpam_read_monsel_reg(msc, CFG_MBWU_FLT, &cur_flt);
 		if (!ret)
 			ret = mpam_read_monsel_reg(msc, CFG_MBWU_CTL, &cur_ctl);
 		if (!ret)
-			mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
+			ret = mpam_write_monsel_reg(msc, CFG_MBWU_CTL, 0);
 
 		if (!ret) {
 			if (mpam_ris_has_mbwu_long_counter(ris)) {
 				val = mpam_msc_read_mbwu_l(msc);
-				mpam_msc_zero_mbwu_l(msc);
+				ret = mpam_msc_zero_mbwu_l(msc);
 			} else {
 				u32 val32;
 
 				ret = mpam_read_monsel_reg(msc, MBWU, &val32);
 				if (!ret) {
 					val = val32;
-					mpam_write_monsel_reg(msc, MBWU, 0);
+					ret = mpam_write_monsel_reg(msc, MBWU, 0);
 				}
 			}
 		}
Signature · 1 line
2.43.0
#23 lore raw

PATCH v2 12/15 arm_mpam: Split the locking around the mon_sel registers

In reply to Andre Przywara

From: James Morse <[email protected]>

The MSC MON_SEL register needs to be accessed from hardirq for the overflow interrupt, and when taking an IPI to access these registers on platforms where MSC are not accesible from every CPU. This makes an irqsave spinlock the obvious lock to protect these registers. On systems with SCMI mailboxes it must be able to sleep, meaning a mutex must be used. The SCMI platforms can't support an overflow interrupt.

Clearly these two can't exist for one MSC at the same time.

Split the existing helper into a raw spinlock and a mutex, named inner and outer. The outer lock must be taken in an a pre-emptible context before the inner lock can be taken. On systems with SCMI mailboxes where the MON_SEL accesses must sleep - the inner lock will fail to be taken if the caller is unable to sleep.
This will allow callers to fail without having to explicitly check the interface type of each MSC.

[Andre: remove redundant outer lock in mpam_reprogram_msc()]
---
 drivers/resctrl/mpam_devices.c  | 56 +++++++++++++++++++--------
 drivers/resctrl/mpam_internal.h | 67 ++++++++++++++++++++++++---------
 2 files changed, 90 insertions(+), 33 deletions(-)
drivers/resctrl/mpam_devices.c +41 -15
index addc25e78345..824bc6c97851 100644
@@ -787,7 +787,7 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris)
 	bool can_set, can_clear;
 	struct mpam_msc *msc = ris->vmsc->msc;
 
-	if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc)))
+	if (WARN_ON_ONCE(!mpam_mon_sel_inner_lock(msc)))
 		return false;
 
 	mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, 0) |
@@ -819,7 +819,7 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris)
 	if (_mpam_read_monsel_reg(msc, MSMON_CSU, &now))
 		return false;
 	can_clear = !(now & MSMON___NRDY);
-	mpam_mon_sel_unlock(msc);
+	mpam_mon_sel_inner_unlock(msc);
 
 	return (!can_set || !can_clear);
 }
@@ -961,7 +961,9 @@ static int mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 					mpam_set_feature(mpam_feat_msmon_csu_xcl, props);
 
 				/* Is NRDY hardware managed? */
+				mpam_mon_sel_outer_lock(msc);
 				hw_managed = mpam_ris_hw_probe_csu_nrdy(ris);
+				mpam_mon_sel_outer_unlock(msc);
 
 				/*
 				 * Accept the missing firmware property if NRDY appears
@@ -1334,7 +1336,7 @@ static void __ris_msmon_read(void *arg)
 	struct mpam_msc *msc = m->ris->vmsc->msc;
 	u32 mon_sel, ctl_val, flt_val, cur_ctl, cur_flt;
 
-	if (!mpam_mon_sel_lock(msc)) {
+	if (!mpam_mon_sel_inner_lock(msc)) {
 		m->err = -EIO;
 		return;
 	}
@@ -1441,7 +1443,7 @@ static void __ris_msmon_read(void *arg)
 	default:
 		m->err = -EINVAL;
 	}
-	mpam_mon_sel_unlock(msc);
+	mpam_mon_sel_inner_unlock(msc);
 
 	if (nrdy)
 		m->err = -EBUSY;
@@ -1452,7 +1454,7 @@ static void __ris_msmon_read(void *arg)
 	return;
 
 out_unlock:
-	mpam_mon_sel_unlock(msc);
+	mpam_mon_sel_inner_unlock(msc);
 
 	m->err = ret;
 }
@@ -1468,6 +1470,7 @@ static int _msmon_read(struct mpam_component *comp, struct mon_read *arg)
 		struct mpam_msc *msc = vmsc->msc;
 		struct mpam_msc_ris *ris;
 
+		mpam_mon_sel_outer_lock(msc);
 		list_for_each_entry_srcu(ris, &vmsc->ris, vmsc_list,
 					 srcu_read_lock_held(&mpam_srcu)) {
 			arg->ris = ris;
@@ -1486,6 +1489,7 @@ static int _msmon_read(struct mpam_component *comp, struct mon_read *arg)
 			if (err)
 				any_err = err;
 		}
+		mpam_mon_sel_outer_unlock(msc);
 	}
 
 	return any_err;
@@ -1568,18 +1572,20 @@ void mpam_msmon_reset_mbwu(struct mpam_component *comp, struct mon_cfg *ctx)
 			continue;
 
 		msc = vmsc->msc;
+		mpam_mon_sel_outer_lock(msc);
 		list_for_each_entry_srcu(ris, &vmsc->ris, vmsc_list,
 					 srcu_read_lock_held(&mpam_srcu)) {
 			if (!mpam_has_feature(mpam_feat_msmon_mbwu, &ris->props))
 				continue;
 
-			if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc)))
+			if (WARN_ON_ONCE(!mpam_mon_sel_inner_lock(msc)))
 				continue;
 
 			ris->mbwu_state[ctx->mon].correction = 0;
 			ris->mbwu_state[ctx->mon].reset_on_next_read = true;
-			mpam_mon_sel_unlock(msc);
+			mpam_mon_sel_inner_unlock(msc);
 		}
+		mpam_mon_sel_outer_unlock(msc);
 	}
 }
 
@@ -1781,8 +1787,11 @@ static int mpam_restore_mbwu_state(void *_ris)
 	int ret = 0;
 	struct mon_read mwbu_arg;
 	struct mpam_msc_ris *ris = _ris;
+	struct mpam_msc *msc = ris->vmsc->msc;
 	struct mpam_class *class = ris->vmsc->comp->class;
 
+	mpam_mon_sel_outer_lock(msc);
+
 	for (i = 0; i < ris->props.num_mbwu_mon; i++) {
 		if (ris->mbwu_state[i].enabled) {
 			mwbu_arg.ris = ris;
@@ -1798,10 +1807,12 @@ static int mpam_restore_mbwu_state(void *_ris)
 		}
 	}
 
+	mpam_mon_sel_outer_unlock(msc);
+
 	return ret;
 }
 
-/* Call with MSC cfg_lock held */
+/* Call with MSC cfg_lock and outer mon_sel lock held */
 static int mpam_save_mbwu_state(void *arg)
 {
 	int i;
@@ -1817,7 +1828,7 @@ static int mpam_save_mbwu_state(void *arg)
 		mbwu_state = &ris->mbwu_state[i];
 		cfg = &mbwu_state->cfg;
 
-		if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc)))
+		if (WARN_ON_ONCE(!mpam_mon_sel_inner_lock(msc)))
 			return -EIO;
 
 		mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) |
@@ -1853,7 +1864,9 @@ static int mpam_save_mbwu_state(void *arg)
 			mbwu_state->correction += val;
 			mbwu_state->enabled = FIELD_GET(MSMON_CFG_x_CTL_EN, cur_ctl);
 		}
-		mpam_mon_sel_unlock(msc);
+
+		mpam_mon_sel_inner_unlock(msc);
+
 		if (ret)
 			break;
 	}
@@ -2050,6 +2063,7 @@ static int mpam_cpu_offline(unsigned int cpu)
 			struct mpam_msc_ris *ris;
 
 			mutex_lock(&msc->cfg_lock);
+			mpam_mon_sel_outer_lock(msc);
 			list_for_each_entry_srcu(ris, &msc->ris, msc_list,
 						 srcu_read_lock_held(&mpam_srcu)) {
 				mpam_touch_msc(msc, &mpam_reset_ris, ris);
@@ -2063,6 +2077,7 @@ static int mpam_cpu_offline(unsigned int cpu)
 				if (mpam_is_enabled())
 					mpam_touch_msc(msc, &mpam_save_mbwu_state, ris);
 			}
+			mpam_mon_sel_outer_unlock(msc);
 			mutex_unlock(&msc->cfg_lock);
 		}
 	}
@@ -2756,11 +2771,13 @@ static void __destroy_component_cfg(struct mpam_component *comp)
 	list_for_each_entry(vmsc, &comp->vmsc, comp_list) {
 		msc = vmsc->msc;
 
-		if (mpam_mon_sel_lock(msc)) {
+		mpam_mon_sel_outer_lock(msc);
+		if (mpam_mon_sel_inner_lock(msc)) {
 			list_for_each_entry(ris, &vmsc->ris, vmsc_list)
 				add_to_garbage(ris->mbwu_state);
-			mpam_mon_sel_unlock(msc);
+			mpam_mon_sel_inner_unlock(msc);
 		}
+		mpam_mon_sel_outer_unlock(msc);
 	}
 }
 
@@ -2807,6 +2824,7 @@ static int __allocate_component_cfg(struct mpam_component *comp)
 	mpam_reset_component_cfg(comp);
 
 	list_for_each_entry(vmsc, &comp->vmsc, comp_list) {
+		int err = 0;
 		struct mpam_msc *msc;
 		struct mpam_msc_ris *ris;
 		struct msmon_mbwu_state *mbwu_state;
@@ -2815,6 +2833,7 @@ static int __allocate_component_cfg(struct mpam_component *comp)
 			continue;
 
 		msc = vmsc->msc;
+		mpam_mon_sel_outer_lock(msc);
 		list_for_each_entry(ris, &vmsc->ris, vmsc_list) {
 			if (!ris->props.num_mbwu_mon)
 				continue;
@@ -2823,16 +2842,21 @@ static int __allocate_component_cfg(struct mpam_component *comp)
 						  ris->props.num_mbwu_mon);
 			if (!mbwu_state) {
 				__destroy_component_cfg(comp);
-				return -ENOMEM;
+				err = -ENOMEM;
+				break;
 			}
 
 			init_garbage(&mbwu_state[0].garbage);
 
-			if (mpam_mon_sel_lock(msc)) {
+			if (mpam_mon_sel_inner_lock(msc)) {
 				ris->mbwu_state = mbwu_state;
-				mpam_mon_sel_unlock(msc);
+				mpam_mon_sel_inner_unlock(msc);
 			}
 		}
+		mpam_mon_sel_outer_unlock(msc);
+
+		if (err)
+			return err;
 	}
 
 	return 0;
@@ -3085,12 +3109,14 @@ int mpam_apply_config(struct mpam_component *comp, u16 partid,
 		msc = vmsc->msc;
 
 		mutex_lock(&msc->cfg_lock);
+		mpam_mon_sel_outer_lock(msc);
 		list_for_each_entry_srcu(ris, &vmsc->ris, vmsc_list,
 					 srcu_read_lock_held(&mpam_srcu)) {
 			arg.ris = ris;
 			mpam_touch_msc(msc, __write_config, &arg);
 			ris->in_reset_state = false;
 		}
+		mpam_mon_sel_outer_unlock(msc);
 		mutex_unlock(&msc->cfg_lock);
 	}
 
drivers/resctrl/mpam_internal.h +49 -18

94 lines not on this page · raw message

Signature · 1 line
2.43.0
maintainer
#24 lore raw

Re: [PATCH v2 12/15] arm_mpam: Split the locking around the mon_sel registers

Hi Andre,

Quoted 4 lines · Andre Przywara

From: James Morse <[email protected]>

The MSC MON_SEL register needs to be accessed from hardirq for the overflow interrupt,

There is no current support for overflow interrupts.

and when taking an IPI to access these registers on platforms

Quoted 1 line

where MSC are not accesible from every CPU.

This is the case for the registers requiring the part_sel mutex.

This makes an irqsave

Quoted 2 lines

spinlock the obvious lock to protect these registers. On systems with SCMI mailboxes it must be able to sleep, meaning a mutex must be used.

What problems do we hit if we convert the mon_sel lock to a mutex?

Thanks,

Ben

The

Quoted 352 lines

SCMI platforms can't support an overflow interrupt.

Clearly these two can't exist for one MSC at the same time.

Split the existing helper into a raw spinlock and a mutex, named inner and outer. The outer lock must be taken in an a pre-emptible context before the inner lock can be taken. On systems with SCMI mailboxes where the MON_SEL accesses must sleep - the inner lock will fail to be taken if the caller is unable to sleep.
This will allow callers to fail without having to explicitly check the interface type of each MSC.

[Andre: remove redundant outer lock in mpam_reprogram_msc()]
---
 drivers/resctrl/mpam_devices.c  | 56 +++++++++++++++++++--------
 drivers/resctrl/mpam_internal.h | 67 ++++++++++++++++++++++++---------
 2 files changed, 90 insertions(+), 33 deletions(-)
drivers/resctrl/mpam_devices.c +41 -15
index addc25e78345..824bc6c97851 100644
@@ -787,7 +787,7 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris)
 	bool can_set, can_clear;
 	struct mpam_msc *msc = ris->vmsc->msc;
 
-	if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc)))
+	if (WARN_ON_ONCE(!mpam_mon_sel_inner_lock(msc)))
 		return false;
 
 	mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, 0) |
@@ -819,7 +819,7 @@ static bool mpam_ris_hw_probe_csu_nrdy(struct mpam_msc_ris *ris)
 	if (_mpam_read_monsel_reg(msc, MSMON_CSU, &now))
 		return false;
 	can_clear = !(now & MSMON___NRDY);
-	mpam_mon_sel_unlock(msc);
+	mpam_mon_sel_inner_unlock(msc);
 
 	return (!can_set || !can_clear);
 }
@@ -961,7 +961,9 @@ static int mpam_ris_hw_probe(struct mpam_msc_ris *ris)
 					mpam_set_feature(mpam_feat_msmon_csu_xcl, props);
 
 				/* Is NRDY hardware managed? */
+				mpam_mon_sel_outer_lock(msc);
 				hw_managed = mpam_ris_hw_probe_csu_nrdy(ris);
+				mpam_mon_sel_outer_unlock(msc);
 
 				/*
 				 * Accept the missing firmware property if NRDY appears
@@ -1334,7 +1336,7 @@ static void __ris_msmon_read(void *arg)
 	struct mpam_msc *msc = m->ris->vmsc->msc;
 	u32 mon_sel, ctl_val, flt_val, cur_ctl, cur_flt;
 
-	if (!mpam_mon_sel_lock(msc)) {
+	if (!mpam_mon_sel_inner_lock(msc)) {
 		m->err = -EIO;
 		return;
 	}
@@ -1441,7 +1443,7 @@ static void __ris_msmon_read(void *arg)
 	default:
 		m->err = -EINVAL;
 	}
-	mpam_mon_sel_unlock(msc);
+	mpam_mon_sel_inner_unlock(msc);
 
 	if (nrdy)
 		m->err = -EBUSY;
@@ -1452,7 +1454,7 @@ static void __ris_msmon_read(void *arg)
 	return;
 
 out_unlock:
-	mpam_mon_sel_unlock(msc);
+	mpam_mon_sel_inner_unlock(msc);
 
 	m->err = ret;
 }
@@ -1468,6 +1470,7 @@ static int _msmon_read(struct mpam_component *comp, struct mon_read *arg)
 		struct mpam_msc *msc = vmsc->msc;
 		struct mpam_msc_ris *ris;
 
+		mpam_mon_sel_outer_lock(msc);
 		list_for_each_entry_srcu(ris, &vmsc->ris, vmsc_list,
 					 srcu_read_lock_held(&mpam_srcu)) {
 			arg->ris = ris;
@@ -1486,6 +1489,7 @@ static int _msmon_read(struct mpam_component *comp, struct mon_read *arg)
 			if (err)
 				any_err = err;
 		}
+		mpam_mon_sel_outer_unlock(msc);
 	}
 
 	return any_err;
@@ -1568,18 +1572,20 @@ void mpam_msmon_reset_mbwu(struct mpam_component *comp, struct mon_cfg *ctx)
 			continue;
 
 		msc = vmsc->msc;
+		mpam_mon_sel_outer_lock(msc);
 		list_for_each_entry_srcu(ris, &vmsc->ris, vmsc_list,
 					 srcu_read_lock_held(&mpam_srcu)) {
 			if (!mpam_has_feature(mpam_feat_msmon_mbwu, &ris->props))
 				continue;
 
-			if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc)))
+			if (WARN_ON_ONCE(!mpam_mon_sel_inner_lock(msc)))
 				continue;
 
 			ris->mbwu_state[ctx->mon].correction = 0;
 			ris->mbwu_state[ctx->mon].reset_on_next_read = true;
-			mpam_mon_sel_unlock(msc);
+			mpam_mon_sel_inner_unlock(msc);
 		}
+		mpam_mon_sel_outer_unlock(msc);
 	}
 }
 
@@ -1781,8 +1787,11 @@ static int mpam_restore_mbwu_state(void *_ris)
 	int ret = 0;
 	struct mon_read mwbu_arg;
 	struct mpam_msc_ris *ris = _ris;
+	struct mpam_msc *msc = ris->vmsc->msc;
 	struct mpam_class *class = ris->vmsc->comp->class;
 
+	mpam_mon_sel_outer_lock(msc);
+
 	for (i = 0; i < ris->props.num_mbwu_mon; i++) {
 		if (ris->mbwu_state[i].enabled) {
 			mwbu_arg.ris = ris;
@@ -1798,10 +1807,12 @@ static int mpam_restore_mbwu_state(void *_ris)
 		}
 	}
 
+	mpam_mon_sel_outer_unlock(msc);
+
 	return ret;
 }
 
-/* Call with MSC cfg_lock held */
+/* Call with MSC cfg_lock and outer mon_sel lock held */
 static int mpam_save_mbwu_state(void *arg)
 {
 	int i;
@@ -1817,7 +1828,7 @@ static int mpam_save_mbwu_state(void *arg)
 		mbwu_state = &ris->mbwu_state[i];
 		cfg = &mbwu_state->cfg;
 
-		if (WARN_ON_ONCE(!mpam_mon_sel_lock(msc)))
+		if (WARN_ON_ONCE(!mpam_mon_sel_inner_lock(msc)))
 			return -EIO;
 
 		mon_sel = FIELD_PREP(MSMON_CFG_MON_SEL_MON_SEL, i) |
@@ -1853,7 +1864,9 @@ static int mpam_save_mbwu_state(void *arg)
 			mbwu_state->correction += val;
 			mbwu_state->enabled = FIELD_GET(MSMON_CFG_x_CTL_EN, cur_ctl);
 		}
-		mpam_mon_sel_unlock(msc);
+
+		mpam_mon_sel_inner_unlock(msc);
+
 		if (ret)
 			break;
 	}
@@ -2050,6 +2063,7 @@ static int mpam_cpu_offline(unsigned int cpu)
 			struct mpam_msc_ris *ris;
 
 			mutex_lock(&msc->cfg_lock);
+			mpam_mon_sel_outer_lock(msc);
 			list_for_each_entry_srcu(ris, &msc->ris, msc_list,
 						 srcu_read_lock_held(&mpam_srcu)) {
 				mpam_touch_msc(msc, &mpam_reset_ris, ris);
@@ -2063,6 +2077,7 @@ static int mpam_cpu_offline(unsigned int cpu)
 				if (mpam_is_enabled())
 					mpam_touch_msc(msc, &mpam_save_mbwu_state, ris);
 			}
+			mpam_mon_sel_outer_unlock(msc);
 			mutex_unlock(&msc->cfg_lock);
 		}
 	}
@@ -2756,11 +2771,13 @@ static void __destroy_component_cfg(struct mpam_component *comp)
 	list_for_each_entry(vmsc, &comp->vmsc, comp_list) {
 		msc = vmsc->msc;
 
-		if (mpam_mon_sel_lock(msc)) {
+		mpam_mon_sel_outer_lock(msc);
+		if (mpam_mon_sel_inner_lock(msc)) {
 			list_for_each_entry(ris, &vmsc->ris, vmsc_list)
 				add_to_garbage(ris->mbwu_state);
-			mpam_mon_sel_unlock(msc);
+			mpam_mon_sel_inner_unlock(msc);
 		}
+		mpam_mon_sel_outer_unlock(msc);
 	}
 }
 
@@ -2807,6 +2824,7 @@ static int __allocate_component_cfg(struct mpam_component *comp)
 	mpam_reset_component_cfg(comp);
 
 	list_for_each_entry(vmsc, &comp->vmsc, comp_list) {
+		int err = 0;
 		struct mpam_msc *msc;
 		struct mpam_msc_ris *ris;
 		struct msmon_mbwu_state *mbwu_state;
@@ -2815,6 +2833,7 @@ static int __allocate_component_cfg(struct mpam_component *comp)
 			continue;
 
 		msc = vmsc->msc;
+		mpam_mon_sel_outer_lock(msc);
 		list_for_each_entry(ris, &vmsc->ris, vmsc_list) {
 			if (!ris->props.num_mbwu_mon)
 				continue;
@@ -2823,16 +2842,21 @@ static int __allocate_component_cfg(struct mpam_component *comp)
 						  ris->props.num_mbwu_mon);
 			if (!mbwu_state) {
 				__destroy_component_cfg(comp);
-				return -ENOMEM;
+				err = -ENOMEM;
+				break;
 			}
 
 			init_garbage(&mbwu_state[0].garbage);
 
-			if (mpam_mon_sel_lock(msc)) {
+			if (mpam_mon_sel_inner_lock(msc)) {
 				ris->mbwu_state = mbwu_state;
-				mpam_mon_sel_unlock(msc);
+				mpam_mon_sel_inner_unlock(msc);
 			}
 		}
+		mpam_mon_sel_outer_unlock(msc);
+
+		if (err)
+			return err;
 	}
 
 	return 0;
@@ -3085,12 +3109,14 @@ int mpam_apply_config(struct mpam_component *comp, u16 partid,
 		msc = vmsc->msc;
 
 		mutex_lock(&msc->cfg_lock);
+		mpam_mon_sel_outer_lock(msc);
 		list_for_each_entry_srcu(ris, &vmsc->ris, vmsc_list,
 					 srcu_read_lock_held(&mpam_srcu)) {
 			arg.ris = ris;
 			mpam_touch_msc(msc, __write_config, &arg);
 			ris->in_reset_state = false;
 		}
+		mpam_mon_sel_outer_unlock(msc);
 		mutex_unlock(&msc->cfg_lock);
 	}
 
drivers/resctrl/mpam_internal.h +49 -18
index 04d1a59f02af..4616f1283f1a 100644
@@ -116,16 +116,20 @@ struct mpam_msc {
 	/*
 	 * mon_sel_lock protects access to the MSC hardware registers that are
 	 * affected by MPAMCFG_MON_SEL, and the mbwu_state.
-	 * Access to mon_sel is needed from both process and interrupt contexts,
-	 * but is complicated by firmware-backed platforms that can't make any
-	 * access unless they can sleep.
-	 * Always use the mpam_mon_sel_lock() helpers.
-	 * Accesses to mon_sel need to be able to fail if they occur in the wrong
-	 * context.
+	 * Both the 'inner' and 'outer' must be taken.
+	 * For real MMIO MSC, the outer lock is unnecessary - but keeps the
+	 * code common with:
+	 * Firmware backed MSC need to sleep when accessing the MSC, which
+	 * means some code-paths will always fail. For these MSC the outer
+	 * lock is providing the protection, and the inner lock fails to
+	 * be taken if the task is unable to sleep.
+	 *
 	 * If needed, take msc->probe_lock first.
 	 */
-	raw_spinlock_t		_mon_sel_lock;
-	unsigned long		_mon_sel_flags;
+	struct mutex		outer_mon_sel_lock;
+	bool			outer_lock_held;
+	raw_spinlock_t		inner_mon_sel_lock;
+	unsigned long		inner_mon_sel_flags;
 
 	void __iomem		*mapped_hwpage;
 	size_t			mapped_hwpage_sz;
@@ -137,29 +141,56 @@ struct mpam_msc {
 };
 
 /* Returning false here means accesses to mon_sel must fail and report an error. */
-static inline bool __must_check mpam_mon_sel_lock(struct mpam_msc *msc)
+static inline bool __must_check mpam_mon_sel_inner_lock(struct mpam_msc *msc)
 {
-	/* Locking will require updating to support a firmware backed interface */
-	if (WARN_ON_ONCE(msc->iface != MPAM_IFACE_MMIO))
-		return false;
+	/*
+	 * The outer lock may be taken by a CPU that then issues an IPI to run
+	 * a helper that takes the inner lock. lockdep can't help us here.
+	 */
+	WARN_ON_ONCE(!READ_ONCE(msc->outer_lock_held));
+
+	if (msc->iface == MPAM_IFACE_MMIO) {
+		raw_spin_lock_irqsave(&msc->inner_mon_sel_lock, msc->inner_mon_sel_flags);
+		return true;
+	}
+
+	/* Accesses must fail if we are not pre-emptible */
+	return !!preemptible();
+}
 
-	raw_spin_lock_irqsave(&msc->_mon_sel_lock, msc->_mon_sel_flags);
-	return true;
+static inline void mpam_mon_sel_inner_unlock(struct mpam_msc *msc)
+{
+	WARN_ON_ONCE(!READ_ONCE(msc->outer_lock_held));
+
+	if (msc->iface == MPAM_IFACE_MMIO)
+		raw_spin_unlock_irqrestore(&msc->inner_mon_sel_lock, msc->inner_mon_sel_flags);
+}
+
+static inline void mpam_mon_sel_outer_lock(struct mpam_msc *msc)
+{
+	mutex_lock(&msc->outer_mon_sel_lock);
+	msc->outer_lock_held = true;
 }
 
-static inline void mpam_mon_sel_unlock(struct mpam_msc *msc)
+static inline void mpam_mon_sel_outer_unlock(struct mpam_msc *msc)
 {
-	raw_spin_unlock_irqrestore(&msc->_mon_sel_lock, msc->_mon_sel_flags);
+	msc->outer_lock_held = false;
+	mutex_unlock(&msc->outer_mon_sel_lock);
 }
 
 static inline void mpam_mon_sel_lock_held(struct mpam_msc *msc)
 {
-	lockdep_assert_held_once(&msc->_mon_sel_lock);
+	WARN_ON_ONCE(!READ_ONCE(msc->outer_lock_held));
+	if (msc->iface == MPAM_IFACE_MMIO)
+		lockdep_assert_held_once(&msc->inner_mon_sel_lock);
+	else
+		lockdep_assert_preemption_enabled();
 }
 
 static inline void mpam_mon_sel_lock_init(struct mpam_msc *msc)
 {
-	raw_spin_lock_init(&msc->_mon_sel_lock);
+	raw_spin_lock_init(&msc->inner_mon_sel_lock);
+	mutex_init(&msc->outer_mon_sel_lock);
 }
 
 /* Bits for mpam features bitmaps */
#25 lore raw

PATCH v2 13/15 arm_mpam: add MPAM-Fb MSC firmware access support

In reply to Andre Przywara

The Arm MPAM Firmware-backed (Fb) Profile document[1] describes an alternative way of accessing the "Memory System Components" (MSC) in an MPAM enabled system.
Normally the MSCs are MMIO mapped, but in some implementations this might not be possible (MSC located outside of the local socket, MSC mapped secure-only) or desirable (direct MMIO access too slow or needs to be mediated through a control processor). MPAM-fb standardises a protocol to abstract MSC accesses, building on the SCMI protocol.

Add functions that do an MSC read or write access by redirecting the request through a firmware interface. For now this done via an ACPI PCC shared memory and mailbox combination.

Since the protocol used is only a small subset of the full SCMI spec, and the SCMI protocol has no full ACPI support anyway, open-code the SCMI message generation and handshake, for just the fields we need.

[1] https://developer.arm.com/documentation/den0144/latest

---
 drivers/resctrl/Makefile        |   2 +-
 drivers/resctrl/mpam_devices.c  |  27 +++++--
 drivers/resctrl/mpam_fb.c       | 137 ++++++++++++++++++++++++++++++++
 drivers/resctrl/mpam_fb.h       |  17 ++++
 drivers/resctrl/mpam_internal.h |  12 +++
 include/linux/arm_mpam.h        |   2 +-
 6 files changed, 190 insertions(+), 7 deletions(-)
 create mode 100644 drivers/resctrl/mpam_fb.c
 create mode 100644 drivers/resctrl/mpam_fb.h
drivers/resctrl/Makefile +1 -1
index 4f6d0e81f9b8..097c036724e9 100644
@@ -1,5 +1,5 @@
 obj-$(CONFIG_ARM64_MPAM_DRIVER)			+= mpam.o
-mpam-y						+= mpam_devices.o
+mpam-y						+= mpam_devices.o mpam_fb.o
 mpam-$(CONFIG_ARM64_MPAM_RESCTRL_FS)		+= mpam_resctrl.o
 
 ccflags-$(CONFIG_ARM64_MPAM_DRIVER_DEBUG)	+= -DDEBUG
drivers/resctrl/mpam_devices.c +22 -5
index 824bc6c97851..b858ff389bff 100644
@@ -28,6 +28,7 @@
 #include <linux/workqueue.h>
 
 #include "mpam_internal.h"
+#include "mpam_fb.h"
 
 /* Values for the T241 errata workaround */
 #define T241_CHIPS_MAX			4
@@ -181,6 +182,9 @@ static int __mpam_read_reg(struct mpam_msc *msc, u16 reg, u32 *res)
 {
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
+	if (msc->iface == MPAM_IFACE_PCC)
+		return mpam_fb_send_read_request(msc, reg, res);
+
 	*res = readl_relaxed(msc->mapped_hwpage + reg);
 
 	return 0;
@@ -197,9 +201,12 @@ static inline int _mpam_read_partsel_reg(struct mpam_msc *msc, u16 reg,
 
 static int __mpam_write_reg(struct mpam_msc *msc, u16 reg, u32 val)
 {
-	WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz);
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
+	if (msc->iface == MPAM_IFACE_PCC)
+		return mpam_fb_send_write_request(msc, reg, val);
+
+	WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz);
 	writel_relaxed(val, msc->mapped_hwpage + reg);
 
 	return 0;
@@ -1127,7 +1134,8 @@ static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)
 
 	mpam_mon_sel_lock_held(msc);
 
-	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
+	if (msc->iface == MPAM_IFACE_MMIO)
+		WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
 	ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
@@ -1475,9 +1483,15 @@ static int _msmon_read(struct mpam_component *comp, struct mon_read *arg)
 					 srcu_read_lock_held(&mpam_srcu)) {
 			arg->ris = ris;
 
-			err = smp_call_function_any(&msc->accessibility,
-						    __ris_msmon_read, arg,
-						    true);
+			if (msc->iface == MPAM_IFACE_MMIO) {
+				err = smp_call_function_any(&msc->accessibility,
+							    __ris_msmon_read,
+							    arg, true);
+			} else {
+				__ris_msmon_read(arg);
+				err = 0;
+			}
+
 			if (!err && arg->err)
 				err = arg->err;
 
@@ -1913,6 +1927,9 @@ static int mpam_get_msc_preferred_cpu(struct mpam_msc *msc)
 
 static int mpam_touch_msc(struct mpam_msc *msc, int (*fn)(void *a), void *arg)
 {
+	if (msc->iface != MPAM_IFACE_MMIO)
+		return fn(arg);
+
 	lockdep_assert_irqs_enabled();
 	lockdep_assert_cpus_held();
 	WARN_ON_ONCE(!srcu_read_lock_held((&mpam_srcu)));
drivers/resctrl/mpam_fb.c new file +137 -0

137 lines not on this page · raw message

drivers/resctrl/mpam_fb.h new file +17 -0
new file mode 100644
index 000000000000..45cd572a28ad
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+// Copyright (C) 2024-2025 Arm Ltd.
+
+#ifndef MPAM_FB_H_
+#define MPAM_FB_H_
+
+#include <linux/types.h>
+#include "mpam_internal.h"
+
+#define PCC_TYPE3_MSG_PAYLOAD_OFS	0x10
+#define MPAM_WRITE_MSG_SIZE	(PCC_TYPE3_MSG_PAYLOAD_OFS + 4 * sizeof(u32))
+#define MPAM_FB_MAX_MSG_SIZE	MPAM_WRITE_MSG_SIZE
+
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result);
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value);
+
+#endif
drivers/resctrl/mpam_internal.h +12 -0
index 4616f1283f1a..9e7778534143 100644
@@ -11,6 +11,7 @@
 #include <linux/io.h>
 #include <linux/jump_label.h>
 #include <linux/llist.h>
+#include <linux/mailbox_client.h>
 #include <linux/mutex.h>
 #include <linux/resctrl.h>
 #include <linux/spinlock.h>
@@ -57,6 +58,15 @@ struct mpam_garbage {
 	struct platform_device	*pdev;
 };
 
+struct mpam_pcc_chan {
+	struct list_head	pcc_chans;
+	struct mbox_client	pcc_cl;
+	struct pcc_mbox_chan	*pcc_chan;
+	struct mutex		pcc_chan_lock; /* only one message at a time */
+	int			subspace_id;
+	int			refcount;
+};
+
 struct mpam_msc {
 	/* member of mpam_all_msc */
 	struct list_head	all_msc_list;
@@ -66,6 +76,8 @@ struct mpam_msc {
 
 	/* Not modified after mpam_is_enabled() becomes true */
 	enum mpam_msc_iface	iface;
+	struct mpam_pcc_chan	*pcc_chan;
+	int			mpam_fb_msc_id;	/* in its own name space */
 	u32			nrdy_usec;
 	cpumask_t		accessibility;
 	bool			has_extd_esr;
include/linux/arm_mpam.h +1 -1
index f92a36187a52..002f56e15362 100644
@@ -12,7 +12,7 @@ struct mpam_msc;
 
 enum mpam_msc_iface {
 	MPAM_IFACE_MMIO,	/* a real MPAM MSC */
-	MPAM_IFACE_PCC,		/* a fake MPAM MSC */
+	MPAM_IFACE_PCC,		/* using the MPAM-Fb firmware redirection */
 };
 
 enum mpam_class_types {
Signature · 1 line
2.43.0
maintainer
#26 lore raw

Re: [PATCH v2 13/15] arm_mpam: add MPAM-Fb MSC firmware access support

Hi Andre,

Quoted 151 lines · Andre Przywara

The Arm MPAM Firmware-backed (Fb) Profile document[1] describes an alternative way of accessing the "Memory System Components" (MSC) in an MPAM enabled system.
Normally the MSCs are MMIO mapped, but in some implementations this might not be possible (MSC located outside of the local socket, MSC mapped secure-only) or desirable (direct MMIO access too slow or needs to be mediated through a control processor). MPAM-fb standardises a protocol to abstract MSC accesses, building on the SCMI protocol.

Add functions that do an MSC read or write access by redirecting the request through a firmware interface. For now this done via an ACPI PCC shared memory and mailbox combination.

Since the protocol used is only a small subset of the full SCMI spec, and the SCMI protocol has no full ACPI support anyway, open-code the SCMI message generation and handshake, for just the fields we need.

[1] https://developer.arm.com/documentation/den0144/latest

---
 drivers/resctrl/Makefile        |   2 +-
 drivers/resctrl/mpam_devices.c  |  27 +++++--
 drivers/resctrl/mpam_fb.c       | 137 ++++++++++++++++++++++++++++++++
 drivers/resctrl/mpam_fb.h       |  17 ++++
 drivers/resctrl/mpam_internal.h |  12 +++
 include/linux/arm_mpam.h        |   2 +-
 6 files changed, 190 insertions(+), 7 deletions(-)
 create mode 100644 drivers/resctrl/mpam_fb.c
 create mode 100644 drivers/resctrl/mpam_fb.h
drivers/resctrl/Makefile +1 -1
index 4f6d0e81f9b8..097c036724e9 100644
@@ -1,5 +1,5 @@
 obj-$(CONFIG_ARM64_MPAM_DRIVER)			+= mpam.o
-mpam-y						+= mpam_devices.o
+mpam-y						+= mpam_devices.o mpam_fb.o
 mpam-$(CONFIG_ARM64_MPAM_RESCTRL_FS)		+= mpam_resctrl.o
 
 ccflags-$(CONFIG_ARM64_MPAM_DRIVER_DEBUG)	+= -DDEBUG
drivers/resctrl/mpam_devices.c +22 -5
index 824bc6c97851..b858ff389bff 100644
@@ -28,6 +28,7 @@
 #include <linux/workqueue.h>
 
 #include "mpam_internal.h"
+#include "mpam_fb.h"
 
 /* Values for the T241 errata workaround */
 #define T241_CHIPS_MAX			4
@@ -181,6 +182,9 @@ static int __mpam_read_reg(struct mpam_msc *msc, u16 reg, u32 *res)
 {
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
+	if (msc->iface == MPAM_IFACE_PCC)
+		return mpam_fb_send_read_request(msc, reg, res);
+
 	*res = readl_relaxed(msc->mapped_hwpage + reg);
 
 	return 0;
@@ -197,9 +201,12 @@ static inline int _mpam_read_partsel_reg(struct mpam_msc *msc, u16 reg,
 
 static int __mpam_write_reg(struct mpam_msc *msc, u16 reg, u32 val)
 {
-	WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz);
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
+	if (msc->iface == MPAM_IFACE_PCC)
+		return mpam_fb_send_write_request(msc, reg, val);
+
+	WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz);
 	writel_relaxed(val, msc->mapped_hwpage + reg);
 
 	return 0;
@@ -1127,7 +1134,8 @@ static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)
 
 	mpam_mon_sel_lock_held(msc);
 
-	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
+	if (msc->iface == MPAM_IFACE_MMIO)
+		WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
 	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
 	ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
@@ -1475,9 +1483,15 @@ static int _msmon_read(struct mpam_component *comp, struct mon_read *arg)
 					 srcu_read_lock_held(&mpam_srcu)) {
 			arg->ris = ris;
 
-			err = smp_call_function_any(&msc->accessibility,
-						    __ris_msmon_read, arg,
-						    true);
+			if (msc->iface == MPAM_IFACE_MMIO) {
+				err = smp_call_function_any(&msc->accessibility,
+							    __ris_msmon_read,
+							    arg, true);
+			} else {
+				__ris_msmon_read(arg);
+				err = 0;
+			}
+
 			if (!err && arg->err)
 				err = arg->err;
 
@@ -1913,6 +1927,9 @@ static int mpam_get_msc_preferred_cpu(struct mpam_msc *msc)
 
 static int mpam_touch_msc(struct mpam_msc *msc, int (*fn)(void *a), void *arg)
 {
+	if (msc->iface != MPAM_IFACE_MMIO)
+		return fn(arg);
+
 	lockdep_assert_irqs_enabled();
 	lockdep_assert_cpus_held();
 	WARN_ON_ONCE(!srcu_read_lock_held((&mpam_srcu)));
drivers/resctrl/mpam_fb.c new file +28 -0
new file mode 100644
index 000000000000..687be6f8a152
@@ -0,0 +1,137 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2024 Arm Ltd.
+
+#include <linux/arm_mpam.h>
+#include <linux/cleanup.h>
+#include <linux/device.h>
+#include <linux/errno.h>
+#include <linux/gfp.h>
+#include <linux/list.h>
+#include <linux/mailbox_client.h>
+#include <linux/mutex.h>
+#include <linux/of.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+#include <linux/printk.h>
+#include <linux/processor.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+
+#include <acpi/pcc.h>
+
+#include <asm/mpam.h>
+
+#include "mpam_fb.h"
+
+#define MPAM_FB_PROTOCOL_ID	0x1a
+#define MPAM_MSC_ATTRIBUTES_CMD	0x3

drivers/resctrl/mpam_fb.c line 36

Quoted 10 lines
drivers/resctrl/mpam_fb.c new file +8 -0
+#define MPAM_MSC_READ_CMD	0x4
+#define MPAM_MSC_WRITE_CMD	0x5
+
+#define MPAM_MSC_PROT_ID_MASK	GENMASK(17, 10)
+#define MPAM_MSC_TOKEN_MASK	GENMASK(27, 18)
+
+#define PCC_CHAN_FLAGS_IRQ	BIT(0)
 
 
+#define MPAM_READ_MSG_SIZE	(PCC_TYPE3_MSG_PAYLOAD_OFS + 3 * sizeof(u32))

How about packed structs for the msgs to avoid the magic numbers? Something like:

struct mpam_fb_read_msg {
	uint32_t read;
	uint32_t flags;
	uint32_t offset;
} __packed;

drivers/resctrl/mpam_fb.c line 45

Quoted 9 lines
drivers/resctrl/mpam_fb.c new file +9 -0
+#define MPAM_WRITE_MSG_SIZE	(PCC_TYPE3_MSG_PAYLOAD_OFS + 4 * sizeof(u32))
+
+static atomic_t mpam_fb_token = ATOMIC_INIT(0);
+
+static int mpam_fb_build_read_message(int msc_id, int reg, unsigned int token,
+				      void __iomem *msg_buf)
+{
+	struct acpi_pcct_ext_pcc_shared_memory *pcc_shmem = msg_buf;
+	void __iomem *payload_ofs = msg_buf + sizeof(*pcc_shmem);

What about pcc_shmem->signature?

drivers/resctrl/mpam_fb.c

Quoted 2 lines
drivers/resctrl/mpam_fb.c new file +2 -0
+
+	writel_relaxed(PCC_CHAN_FLAGS_IRQ, &pcc_shmem->flags);

Based on the acpi spec, this can be ignored if a platform interrupt is not set in PCCT global flags. Do we rely on the interrupt?

drivers/resctrl/mpam_fb.c line 54

Quoted 7 lines
drivers/resctrl/mpam_fb.c new file +7 -0
+	writel_relaxed(MPAM_READ_MSG_SIZE, &pcc_shmem->length);
+	writel_relaxed(MPAM_MSC_READ_CMD |
+		       FIELD_PREP(MPAM_MSC_TOKEN_MASK, token) |
+		       FIELD_PREP(MPAM_MSC_PROT_ID_MASK, MPAM_FB_PROTOCOL_ID),
+		       &pcc_shmem->command);
+
+	writel_relaxed(cpu_to_le32(msc_id), payload_ofs + 0x0);

Why cpu_to_le32() here? writel_relaxed() looks to already do this?

drivers/resctrl/mpam_fb.c

Quoted 2 lines
drivers/resctrl/mpam_fb.c new file +2 -0
+	writel_relaxed(0, payload_ofs + 0x4);
+	writel_relaxed(cpu_to_le32(reg), payload_ofs + 0x8);

Does memcpy_toio() work here to reduce the number of repeated calls to writel_relaxed()?

Quoted 55 lines
drivers/resctrl/mpam_fb.c new file +55 -0
+
+	return MPAM_READ_MSG_SIZE;
+}
+
+static int mpam_fb_build_write_message(int msc_id, int reg, u32 val,
+				       unsigned int token,
+				       void __iomem *msg_buf)
+{
+	struct acpi_pcct_ext_pcc_shared_memory *pcc_shmem = msg_buf;
+	void __iomem *payload_ofs = msg_buf + sizeof(*pcc_shmem);
+
+	writel_relaxed(MPAM_WRITE_MSG_SIZE, &pcc_shmem->length);
+	writel_relaxed(MPAM_MSC_WRITE_CMD |
+		       FIELD_PREP(MPAM_MSC_TOKEN_MASK, token) |
+		       FIELD_PREP(MPAM_MSC_PROT_ID_MASK, MPAM_FB_PROTOCOL_ID),
+		       &pcc_shmem->command);
+
+	writel_relaxed(cpu_to_le32(msc_id), payload_ofs + 0x0);
+	writel_relaxed(0, payload_ofs + 0x4);
+	writel_relaxed(cpu_to_le32(reg), payload_ofs + 0x8);
+	writel_relaxed(cpu_to_le32(val), payload_ofs + 0xc);
+
+	return MPAM_WRITE_MSG_SIZE;
+}
+
+static int mpam_fb_send_request(struct mpam_msc *msc, u16 reg, u32 *result,
+				bool is_write)
+{
+	unsigned int token = atomic_inc_return(&mpam_fb_token);
+	struct acpi_pcct_ext_pcc_shared_memory *pcc_shmem;
+	struct mpam_pcc_chan *pcc_chan = msc->pcc_chan;
+	struct pcc_mbox_chan *chan;
+	void __iomem *payload_ofs;
+	u32 status;
+	int ret;
+
+	if (!pcc_chan)
+		return -ENODEV;
+
+	chan = pcc_chan->pcc_chan;
+
+	guard(mutex)(&pcc_chan->pcc_chan_lock);
+
+	if (is_write)
+		ret = mpam_fb_build_write_message(msc->mpam_fb_msc_id, reg,
+						  *result, token, chan->shmem);
+	else
+		ret = mpam_fb_build_read_message(msc->mpam_fb_msc_id, reg,
+						 token, chan->shmem);
+	if (ret < 0)
+		return ret;
+
+	ret = mbox_send_message(chan->mchan, NULL);
+	if (ret < 0)
+		return ret;

drivers/resctrl/mpam_fb.c line 121

Quoted 10 lines
drivers/resctrl/mpam_fb.c new file +10 -0
+
+	pcc_shmem = chan->shmem;
+	payload_ofs = chan->shmem + sizeof(*pcc_shmem);
+	status = readl(&pcc_shmem->command);
+	if (FIELD_GET(MPAM_MSC_TOKEN_MASK, status) != token)
+		return -ETIMEDOUT;
+
+	ret = readl(payload_ofs + 0x0);
+	if (ret < 0)
+		return ret;

This needs translating from an MPAM-FB error code to a kernel error code.

Quoted 24 lines
drivers/resctrl/mpam_fb.c new file +16 -0
+
+	if (!is_write)
+		*result = readl(payload_ofs + 0x4);
+
+	return 0;
+}
+
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result)
+{
+	return mpam_fb_send_request(msc, reg, result, false);
+}
+
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value)
+{
+	return mpam_fb_send_request(msc, reg, &value, true);
+}
drivers/resctrl/mpam_fb.h new file +2 -0
new file mode 100644
index 000000000000..45cd572a28ad
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+// Copyright (C) 2024-2025 Arm Ltd.

drivers/resctrl/mpam_fb.h line 12

Quoted 10 lines
drivers/resctrl/mpam_fb.h new file +10 -0
+
+#ifndef MPAM_FB_H_
+#define MPAM_FB_H_
+
+#include <linux/types.h>
+#include "mpam_internal.h"
+
+#define PCC_TYPE3_MSG_PAYLOAD_OFS	0x10
+#define MPAM_WRITE_MSG_SIZE	(PCC_TYPE3_MSG_PAYLOAD_OFS + 4 * sizeof(u32))
+#define MPAM_FB_MAX_MSG_SIZE	MPAM_WRITE_MSG_SIZE

Duplicates of those in mpam_fb.c

drivers/resctrl/mpam_fb.h line 15

Quoted 3 lines
drivers/resctrl/mpam_fb.h new file +3 -0
+
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result);
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value);

These 2 declarations can just go in mpam_internal.h and get rid of the extra header.

Thanks,

Ben

Quoted 52 lines
drivers/resctrl/mpam_fb.h new file +2 -0
+
+#endif
drivers/resctrl/mpam_internal.h +12 -0
index 4616f1283f1a..9e7778534143 100644
@@ -11,6 +11,7 @@
 #include <linux/io.h>
 #include <linux/jump_label.h>
 #include <linux/llist.h>
+#include <linux/mailbox_client.h>
 #include <linux/mutex.h>
 #include <linux/resctrl.h>
 #include <linux/spinlock.h>
@@ -57,6 +58,15 @@ struct mpam_garbage {
 	struct platform_device	*pdev;
 };
 
+struct mpam_pcc_chan {
+	struct list_head	pcc_chans;
+	struct mbox_client	pcc_cl;
+	struct pcc_mbox_chan	*pcc_chan;
+	struct mutex		pcc_chan_lock; /* only one message at a time */
+	int			subspace_id;
+	int			refcount;
+};
+
 struct mpam_msc {
 	/* member of mpam_all_msc */
 	struct list_head	all_msc_list;
@@ -66,6 +76,8 @@ struct mpam_msc {
 
 	/* Not modified after mpam_is_enabled() becomes true */
 	enum mpam_msc_iface	iface;
+	struct mpam_pcc_chan	*pcc_chan;
+	int			mpam_fb_msc_id;	/* in its own name space */
 	u32			nrdy_usec;
 	cpumask_t		accessibility;
 	bool			has_extd_esr;
include/linux/arm_mpam.h +1 -1
index f92a36187a52..002f56e15362 100644
@@ -12,7 +12,7 @@ struct mpam_msc;
 
 enum mpam_msc_iface {
 	MPAM_IFACE_MMIO,	/* a real MPAM MSC */
-	MPAM_IFACE_PCC,		/* a fake MPAM MSC */
+	MPAM_IFACE_PCC,		/* using the MPAM-Fb firmware redirection */
 };
 
 enum mpam_class_types {
#27 lore raw

Re: [PATCH v2 13/15] arm_mpam: add MPAM-Fb MSC firmware access support

Hi,

Quoted 165 lines · Ben Horgan

Hi Andre,

Quoted 151 lines · Andre Przywara

The Arm MPAM Firmware-backed (Fb) Profile document[1] describes an alternative way of accessing the "Memory System Components" (MSC) in an MPAM enabled system.
Normally the MSCs are MMIO mapped, but in some implementations this might not be possible (MSC located outside of the local socket, MSC mapped secure-only) or desirable (direct MMIO access too slow or needs to be mediated through a control processor). MPAM-fb standardises a protocol to abstract MSC accesses, building on the SCMI protocol.

Add functions that do an MSC read or write access by redirecting the request through a firmware interface. For now this done via an ACPI PCC shared memory and mailbox combination.

Since the protocol used is only a small subset of the full SCMI spec, and the SCMI protocol has no full ACPI support anyway, open-code the SCMI message generation and handshake, for just the fields we need.

[1] https://developer.arm.com/documentation/den0144/latest

---
  drivers/resctrl/Makefile        |   2 +-
  drivers/resctrl/mpam_devices.c  |  27 +++++--
  drivers/resctrl/mpam_fb.c       | 137 ++++++++++++++++++++++++++++++++
  drivers/resctrl/mpam_fb.h       |  17 ++++
  drivers/resctrl/mpam_internal.h |  12 +++
  include/linux/arm_mpam.h        |   2 +-
  6 files changed, 190 insertions(+), 7 deletions(-)

create mode 100644 drivers/resctrl/mpam_fb.c
create mode 100644 drivers/resctrl/mpam_fb.h

drivers/resctrl/Makefile +1 -1
index 4f6d0e81f9b8..097c036724e9 100644
@@ -1,5 +1,5 @@
  obj-$(CONFIG_ARM64_MPAM_DRIVER)			+= mpam.o
-mpam-y						+= mpam_devices.o
+mpam-y						+= mpam_devices.o mpam_fb.o
  mpam-$(CONFIG_ARM64_MPAM_RESCTRL_FS)		+= mpam_resctrl.o
 
  ccflags-$(CONFIG_ARM64_MPAM_DRIVER_DEBUG)	+= -DDEBUG
drivers/resctrl/mpam_devices.c +22 -5
index 824bc6c97851..b858ff389bff 100644
@@ -28,6 +28,7 @@
  #include <linux/workqueue.h>
 
  #include "mpam_internal.h"
+#include "mpam_fb.h"
 
  /* Values for the T241 errata workaround */
  #define T241_CHIPS_MAX			4
@@ -181,6 +182,9 @@ static int __mpam_read_reg(struct mpam_msc *msc, u16 reg, u32 *res)
  {
  	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
+	if (msc->iface == MPAM_IFACE_PCC)
+		return mpam_fb_send_read_request(msc, reg, res);
+
  	*res = readl_relaxed(msc->mapped_hwpage + reg);
 
  	return 0;
@@ -197,9 +201,12 @@ static inline int _mpam_read_partsel_reg(struct mpam_msc *msc, u16 reg,
 
  static int __mpam_write_reg(struct mpam_msc *msc, u16 reg, u32 val)
  {
-	WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz);
  	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
+	if (msc->iface == MPAM_IFACE_PCC)
+		return mpam_fb_send_write_request(msc, reg, val);
+
+	WARN_ON_ONCE(reg + sizeof(u32) > msc->mapped_hwpage_sz);
  	writel_relaxed(val, msc->mapped_hwpage + reg);
 
  	return 0;
@@ -1127,7 +1134,8 @@ static u64 mpam_msc_read_mbwu_l(struct mpam_msc *msc)
 
  	mpam_mon_sel_lock_held(msc);
 
-	WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
+	if (msc->iface == MPAM_IFACE_MMIO)
+		WARN_ON_ONCE((MSMON_MBWU_L + sizeof(u64)) > msc->mapped_hwpage_sz);
  	WARN_ON_ONCE(!cpumask_test_cpu(smp_processor_id(), &msc->accessibility));
 
  	ret = __mpam_read_reg(msc, MSMON_MBWU_L + 4, &mbwu_l_high2);
@@ -1475,9 +1483,15 @@ static int _msmon_read(struct mpam_component *comp, struct mon_read *arg)
  					 srcu_read_lock_held(&mpam_srcu)) {
  			arg->ris = ris;
 
-			err = smp_call_function_any(&msc->accessibility,
-						    __ris_msmon_read, arg,
-						    true);
+			if (msc->iface == MPAM_IFACE_MMIO) {
+				err = smp_call_function_any(&msc->accessibility,
+							    __ris_msmon_read,
+							    arg, true);
+			} else {
+				__ris_msmon_read(arg);
+				err = 0;
+			}
+
  			if (!err && arg->err)
  				err = arg->err;
 
@@ -1913,6 +1927,9 @@ static int mpam_get_msc_preferred_cpu(struct mpam_msc *msc)
 
  static int mpam_touch_msc(struct mpam_msc *msc, int (*fn)(void *a), void *arg)
  {
+	if (msc->iface != MPAM_IFACE_MMIO)
+		return fn(arg);
+
  	lockdep_assert_irqs_enabled();
  	lockdep_assert_cpus_held();
  	WARN_ON_ONCE(!srcu_read_lock_held((&mpam_srcu)));
drivers/resctrl/mpam_fb.c new file +28 -0
new file mode 100644
index 000000000000..687be6f8a152
@@ -0,0 +1,137 @@
+// SPDX-License-Identifier: GPL-2.0
+// Copyright (C) 2024 Arm Ltd.
+
+#include <linux/arm_mpam.h>
+#include <linux/cleanup.h>
+#include <linux/device.h>
+#include <linux/errno.h>
+#include <linux/gfp.h>
+#include <linux/list.h>
+#include <linux/mailbox_client.h>
+#include <linux/mutex.h>
+#include <linux/of.h>
+#include <linux/of_platform.h>
+#include <linux/platform_device.h>
+#include <linux/printk.h>
+#include <linux/processor.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+
+#include <acpi/pcc.h>
+
+#include <asm/mpam.h>
+
+#include "mpam_fb.h"
+
+#define MPAM_FB_PROTOCOL_ID	0x1a
+#define MPAM_MSC_ATTRIBUTES_CMD	0x3

drivers/resctrl/mpam_fb.c line 36

Quoted 10 lines
drivers/resctrl/mpam_fb.c new file +8 -0
+#define MPAM_MSC_READ_CMD	0x4
+#define MPAM_MSC_WRITE_CMD	0x5
+
+#define MPAM_MSC_PROT_ID_MASK	GENMASK(17, 10)
+#define MPAM_MSC_TOKEN_MASK	GENMASK(27, 18)
+
+#define PCC_CHAN_FLAGS_IRQ	BIT(0)
 
 
+#define MPAM_READ_MSG_SIZE	(PCC_TYPE3_MSG_PAYLOAD_OFS + 3 * sizeof(u32))
Quoted 7 lines

How about packed structs for the msgs to avoid the magic numbers? Something like:

struct mpam_fb_read_msg {
	uint32_t read;
	uint32_t flags;
	uint32_t offset;
} __packed;

Well, the "magic numbers" are actually the reference, in the specification, so hiding them away in a C struct is not necessarily the best solution. But since we use a struct already for the generic PCC header, I will follow suit here.

Quoted 11 lines

drivers/resctrl/mpam_fb.c line 45

Quoted 9 lines
drivers/resctrl/mpam_fb.c new file +8 -0
+#define MPAM_WRITE_MSG_SIZE	(PCC_TYPE3_MSG_PAYLOAD_OFS + 4 * sizeof(u32))>> +
+static atomic_t mpam_fb_token = ATOMIC_INIT(0);
+
+static int mpam_fb_build_read_message(int msc_id, int reg, unsigned int token,
+				      void __iomem *msg_buf)
+{
+	struct acpi_pcct_ext_pcc_shared_memory *pcc_shmem = msg_buf;
+	void __iomem *payload_ofs = msg_buf + sizeof(*pcc_shmem);

What about pcc_shmem->signature?

Do you mean why it's not populated? According to the ACPI spec v6.6, 14.3 "Extended PCC Subspace Shared Memory Region":
"The signature is populated by the platform and is verified by OSPM."

Quoted 5 lines

drivers/resctrl/mpam_fb.c

Quoted 2 lines
drivers/resctrl/mpam_fb.c new file +2 -0
+
+	writel_relaxed(PCC_CHAN_FLAGS_IRQ, &pcc_shmem->flags);

Based on the acpi spec, this can be ignored if a platform interrupt is not set in PCCT global flags. Do we rely on the interrupt?

No, we actually leave that to PCC to decide, so we should probably not set it here. I will clear it.

Quoted 10 lines

drivers/resctrl/mpam_fb.c line 54

Quoted 8 lines
drivers/resctrl/mpam_fb.c new file +7 -0
+	writel_relaxed(MPAM_READ_MSG_SIZE, &pcc_shmem->length);
+	writel_relaxed(MPAM_MSC_READ_CMD |
+		       FIELD_PREP(MPAM_MSC_TOKEN_MASK, token) |
+		       FIELD_PREP(MPAM_MSC_PROT_ID_MASK, MPAM_FB_PROTOCOL_ID),
+		       &pcc_shmem->command);
+
+	writel_relaxed(cpu_to_le32(msc_id), payload_ofs + 0x0);

Why cpu_to_le32() here? writel_relaxed() looks to already do this?

Ah yeah, of course, thanks for spotting!

Quoted 6 lines

drivers/resctrl/mpam_fb.c

Quoted 3 lines
drivers/resctrl/mpam_fb.c new file +2 -0
+	writel_relaxed(0, payload_ofs + 0x4);
+	writel_relaxed(cpu_to_le32(reg), payload_ofs + 0x8);

Does memcpy_toio() work here to reduce the number of repeated calls to writel_relaxed()?

Not sure it makes too much sense to first fill a struct with the value, and then memcpy_toio()ing the data? With using the struct members as targets it's more readable already, I think, so I'd prefer to stick to the writel's.

Quoted 60 lines
Quoted 60 lines
drivers/resctrl/mpam_fb.c new file +59 -0
+
+	return MPAM_READ_MSG_SIZE;
+}
+
+static int mpam_fb_build_write_message(int msc_id, int reg, u32 val,
+				       unsigned int token,
+				       void __iomem *msg_buf)
+{
+	struct acpi_pcct_ext_pcc_shared_memory *pcc_shmem = msg_buf;
+	void __iomem *payload_ofs = msg_buf + sizeof(*pcc_shmem);
+
+	writel_relaxed(MPAM_WRITE_MSG_SIZE, &pcc_shmem->length);
+	writel_relaxed(MPAM_MSC_WRITE_CMD |
+		       FIELD_PREP(MPAM_MSC_TOKEN_MASK, token) |
+		       FIELD_PREP(MPAM_MSC_PROT_ID_MASK, MPAM_FB_PROTOCOL_ID),
+		       &pcc_shmem->command);
+
+	writel_relaxed(cpu_to_le32(msc_id), payload_ofs + 0x0);
+	writel_relaxed(0, payload_ofs + 0x4);
+	writel_relaxed(cpu_to_le32(reg), payload_ofs + 0x8);
+	writel_relaxed(cpu_to_le32(val), payload_ofs + 0xc);
+
+	return MPAM_WRITE_MSG_SIZE;
+}
+
+static int mpam_fb_send_request(struct mpam_msc *msc, u16 reg, u32 *result,
+				bool is_write)
+{
+	unsigned int token = atomic_inc_return(&mpam_fb_token);
+	struct acpi_pcct_ext_pcc_shared_memory *pcc_shmem;
+	struct mpam_pcc_chan *pcc_chan = msc->pcc_chan;
+	struct pcc_mbox_chan *chan;
+	void __iomem *payload_ofs;
+	u32 status;
+	int ret;
+
+	if (!pcc_chan)
+		return -ENODEV;
+
+	chan = pcc_chan->pcc_chan;
+
+	guard(mutex)(&pcc_chan->pcc_chan_lock);
+
+	if (is_write)
+		ret = mpam_fb_build_write_message(msc->mpam_fb_msc_id, reg,
+						  *result, token, chan->shmem);
+	else
+		ret = mpam_fb_build_read_message(msc->mpam_fb_msc_id, reg,
+						 token, chan->shmem);
+	if (ret < 0)
+		return ret;
+
+	ret = mbox_send_message(chan->mchan, NULL);
+	if (ret < 0)
+		return ret;
+
+	pcc_shmem = chan->shmem;
+	payload_ofs = chan->shmem + sizeof(*pcc_shmem);
+	status = readl(&pcc_shmem->command);
Quoted 8 lines

drivers/resctrl/mpam_fb.c line 121

Quoted 6 lines
drivers/resctrl/mpam_fb.c new file +6 -0
+	if (FIELD_GET(MPAM_MSC_TOKEN_MASK, status) != token)
+		return -ETIMEDOUT;
+
+	ret = readl(payload_ofs + 0x0);
+	if (ret < 0)
+		return ret;

This needs translating from an MPAM-FB error code to a kernel error code.

Ah, right.

Quoted 28 lines
Quoted 28 lines
drivers/resctrl/mpam_fb.c new file +16 -0
+
+	if (!is_write)
+		*result = readl(payload_ofs + 0x4);
+
+	return 0;
+}
+
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result)
+{
+	return mpam_fb_send_request(msc, reg, result, false);
+}
+
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value)
+{
+	return mpam_fb_send_request(msc, reg, &value, true);
+}
drivers/resctrl/mpam_fb.h new file +6 -0
new file mode 100644
index 000000000000..45cd572a28ad
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+// Copyright (C) 2024-2025 Arm Ltd.
+
+#ifndef MPAM_FB_H_
+#define MPAM_FB_H_
+
Quoted 8 lines

drivers/resctrl/mpam_fb.h line 12

Quoted 6 lines
drivers/resctrl/mpam_fb.h new file +6 -0
+#include <linux/types.h>
+#include "mpam_internal.h"
+
+#define PCC_TYPE3_MSG_PAYLOAD_OFS	0x10
+#define MPAM_WRITE_MSG_SIZE	(PCC_TYPE3_MSG_PAYLOAD_OFS + 4 * sizeof(u32))
+#define MPAM_FB_MAX_MSG_SIZE	MPAM_WRITE_MSG_SIZE

Duplicates of those in mpam_fb.c

Yes, on purpose. We need that value in mpam_devices.c, just for a safey check, but I didn't want to break abstraction by allowing MPAM-Fb specifics to bleed into the generic MPAM code.

Quoted 6 lines

drivers/resctrl/mpam_fb.h line 15

Quoted 3 lines
drivers/resctrl/mpam_fb.h new file +3 -0
+
+int mpam_fb_send_read_request(struct mpam_msc *msc, u16 reg, u32 *result);
+int mpam_fb_send_write_request(struct mpam_msc *msc, u16 reg, u32 value);

These 2 declarations can just go in mpam_internal.h and get rid of the extra header.

Indeed, good idea!

Cheers,
Andre

Quoted 58 lines

Thanks,

Ben

Quoted 52 lines
drivers/resctrl/mpam_fb.h new file +2 -0
+
+#endif
drivers/resctrl/mpam_internal.h +12 -0
index 4616f1283f1a..9e7778534143 100644
@@ -11,6 +11,7 @@
  #include <linux/io.h>
  #include <linux/jump_label.h>
  #include <linux/llist.h>
+#include <linux/mailbox_client.h>
  #include <linux/mutex.h>
  #include <linux/resctrl.h>
  #include <linux/spinlock.h>
@@ -57,6 +58,15 @@ struct mpam_garbage {
  	struct platform_device	*pdev;
  };
 
+struct mpam_pcc_chan {
+	struct list_head	pcc_chans;
+	struct mbox_client	pcc_cl;
+	struct pcc_mbox_chan	*pcc_chan;
+	struct mutex		pcc_chan_lock; /* only one message at a time */
+	int			subspace_id;
+	int			refcount;
+};
+
  struct mpam_msc {
  	/* member of mpam_all_msc */
  	struct list_head	all_msc_list;
@@ -66,6 +76,8 @@ struct mpam_msc {
 
  	/* Not modified after mpam_is_enabled() becomes true */
  	enum mpam_msc_iface	iface;
+	struct mpam_pcc_chan	*pcc_chan;
+	int			mpam_fb_msc_id;	/* in its own name space */
  	u32			nrdy_usec;
  	cpumask_t		accessibility;
  	bool			has_extd_esr;
include/linux/arm_mpam.h +1 -1
index f92a36187a52..002f56e15362 100644
@@ -12,7 +12,7 @@ struct mpam_msc;
 
  enum mpam_msc_iface {
  	MPAM_IFACE_MMIO,	/* a real MPAM MSC */
-	MPAM_IFACE_PCC,		/* a fake MPAM MSC */
+	MPAM_IFACE_PCC,		/* using the MPAM-Fb firmware redirection */
  };
 
  enum mpam_class_types {
#28 lore raw

PATCH v2 14/15 arm_mpam: prevent MPAM-Fb accesses inside IRQ handler

In reply to Andre Przywara

When an MPAM MSC gets into an error condition, it can trigger an error IRQ. We cannot really do much about those errors, but we at least query and log the error, then disable MPAM functionality.

This error report relies on reading the MSC's error status register (ESR) in the IRQ handler, which is not possible for MPAM-Fb based MSC accesses, since they involve mailbox routines that might sleep. The same is true for clearing the interrupt at the source, which requires MSC access.

For simplicity just skip the ESR read when the MSC is not using direct MMIO accesses, and just ignore the pending interrupts. We will wrap up MPAM functionality regardless, knowing the exact error value will not change that.

---
 drivers/resctrl/mpam_devices.c | 35 +++++++++++++++++++---------------
 1 file changed, 20 insertions(+), 15 deletions(-)
drivers/resctrl/mpam_devices.c +20 -15
index b858ff389bff..4a088e6cd235 100644
@@ -2639,7 +2639,7 @@ static int mpam_disable_msc_ecr(void *_msc)
 
 static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
 {
-	u64 reg;
+	u64 reg = 0;
 	u16 partid;
 	u8 errcode, pmg, ris;
 
@@ -2648,25 +2648,30 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
 					   &msc->accessibility)))
 		return IRQ_NONE;
 
-	mpam_msc_read_esr(msc, &reg);
+	/* MPAM-Fb MSC accesses cannot be done in atomic context. */
+	if (msc->iface == MPAM_IFACE_MMIO) {
+		mpam_msc_read_esr(msc, &reg);
 
-	errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
-	if (!errcode)
-		return IRQ_NONE;
+		errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
+		if (!errcode)
+			return IRQ_NONE;
 
-	/* Clear level triggered irq */
-	mpam_msc_clear_esr(msc);
+		/* Clear level triggered irq */
+		mpam_msc_clear_esr(msc);
 
-	partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg);
-	pmg = FIELD_GET(MPAMF_ESR_PMG, reg);
-	ris = FIELD_GET(MPAMF_ESR_RIS, reg);
+		partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg);
+		pmg = FIELD_GET(MPAMF_ESR_PMG, reg);
+		ris = FIELD_GET(MPAMF_ESR_RIS, reg);
 
-	pr_err_ratelimited("error irq from msc:%u '%s', partid:%u, pmg: %u, ris: %u\n",
-			   msc->id, mpam_errcode_names[errcode], partid, pmg,
-			   ris);
+		pr_err_ratelimited("error irq from msc:%u '%s', partid:%u, pmg: %u, ris: %u\n",
+				   msc->id, mpam_errcode_names[errcode], partid,
+				   pmg, ris);
 
-	/* Disable this interrupt. */
-	mpam_disable_msc_ecr(msc);
+		/* Disable this interrupt. */
+		mpam_disable_msc_ecr(msc);
+	} else {
+		pr_err_ratelimited("unknown error irq from msc:%u\n", msc->id);
+	}
 
 	/* Are we racing with the thread disabling MPAM? */
 	if (!mpam_is_enabled())
Signature · 1 line
2.43.0
maintainer
#29 lore raw

Re: [PATCH v2 14/15] arm_mpam: prevent MPAM-Fb accesses inside IRQ handler

Hi Andre,

Quoted 62 lines · Andre Przywara

When an MPAM MSC gets into an error condition, it can trigger an error IRQ. We cannot really do much about those errors, but we at least query and log the error, then disable MPAM functionality.

This error report relies on reading the MSC's error status register (ESR) in the IRQ handler, which is not possible for MPAM-Fb based MSC accesses, since they involve mailbox routines that might sleep. The same is true for clearing the interrupt at the source, which requires MSC access.

For simplicity just skip the ESR read when the MSC is not using direct MMIO accesses, and just ignore the pending interrupts. We will wrap up MPAM functionality regardless, knowing the exact error value will not change that.

---
 drivers/resctrl/mpam_devices.c | 35 +++++++++++++++++++---------------
 1 file changed, 20 insertions(+), 15 deletions(-)
drivers/resctrl/mpam_devices.c +12 -11
index b858ff389bff..4a088e6cd235 100644
@@ -2639,7 +2639,7 @@ static int mpam_disable_msc_ecr(void *_msc)
 
 static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
 {
-	u64 reg;
+	u64 reg = 0;
 	u16 partid;
 	u8 errcode, pmg, ris;
 
@@ -2648,25 +2648,30 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
 					   &msc->accessibility)))
 		return IRQ_NONE;
 
-	mpam_msc_read_esr(msc, &reg);
+	/* MPAM-Fb MSC accesses cannot be done in atomic context. */
+	if (msc->iface == MPAM_IFACE_MMIO) {
+		mpam_msc_read_esr(msc, &reg);
 
-	errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
-	if (!errcode)
-		return IRQ_NONE;
+		errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
+		if (!errcode)
+			return IRQ_NONE;
 
-	/* Clear level triggered irq */
-	mpam_msc_clear_esr(msc);
+		/* Clear level triggered irq */
+		mpam_msc_clear_esr(msc);
 
-	partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg);
-	pmg = FIELD_GET(MPAMF_ESR_PMG, reg);
-	ris = FIELD_GET(MPAMF_ESR_RIS, reg);
+		partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg);
+		pmg = FIELD_GET(MPAMF_ESR_PMG, reg);
+		ris = FIELD_GET(MPAMF_ESR_RIS, reg);
 
-	pr_err_ratelimited("error irq from msc:%u '%s', partid:%u, pmg: %u, ris: %u\n",

drivers/resctrl/mpam_devices.c static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc) · line 2671

Quoted 10 lines
drivers/resctrl/mpam_devices.c +5 -4
-			   msc->id, mpam_errcode_names[errcode], partid, pmg,
-			   ris);
+		pr_err_ratelimited("error irq from msc:%u '%s', partid:%u, pmg: %u, ris: %u\n",
+				   msc->id, mpam_errcode_names[errcode], partid,
+				   pmg, ris);
 
-	/* Disable this interrupt. */
-	mpam_disable_msc_ecr(msc);
+		/* Disable this interrupt. */
+		mpam_disable_msc_ecr(msc);

As an error interrupt is final can we just disable the IRQ? Is it useful? I see there is a function disable_irq_no_sync().

drivers/resctrl/mpam_devices.c static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc) · line 2673

Quoted 2 lines
drivers/resctrl/mpam_devices.c +2 -0
+	} else {
+		pr_err_ratelimited("unknown error irq from msc:%u\n", msc->id);

Should we report by irq number?
As MSC may share interrupts we don't know which MSC caused the error irq at this point. On MMIO platforms we read the ESR to establish this.

Thanks,

Ben

Quoted 4 lines
drivers/resctrl/mpam_devices.c +1 -0
+	}
 
 	/* Are we racing with the thread disabling MPAM? */
 	if (!mpam_is_enabled())
#30 lore raw

Re: [PATCH v2 14/15] arm_mpam: prevent MPAM-Fb accesses inside IRQ handler

Hi,

Quoted 71 lines · Ben Horgan

Hi Andre,

Quoted 68 lines · Andre Przywara

When an MPAM MSC gets into an error condition, it can trigger an error IRQ. We cannot really do much about those errors, but we at least query and log the error, then disable MPAM functionality.

This error report relies on reading the MSC's error status register (ESR) in the IRQ handler, which is not possible for MPAM-Fb based MSC accesses, since they involve mailbox routines that might sleep. The same is true for clearing the interrupt at the source, which requires MSC access.

For simplicity just skip the ESR read when the MSC is not using direct MMIO accesses, and just ignore the pending interrupts. We will wrap up MPAM functionality regardless, knowing the exact error value will not change that.

---
  drivers/resctrl/mpam_devices.c | 35 +++++++++++++++++++---------------
  1 file changed, 20 insertions(+), 15 deletions(-)
drivers/resctrl/mpam_devices.c +15 -13
index b858ff389bff..4a088e6cd235 100644
@@ -2639,7 +2639,7 @@ static int mpam_disable_msc_ecr(void *_msc)
 
  static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
  {
-	u64 reg;
+	u64 reg = 0;
  	u16 partid;
  	u8 errcode, pmg, ris;
 
@@ -2648,25 +2648,30 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
  					   &msc->accessibility)))
  		return IRQ_NONE;
 
-	mpam_msc_read_esr(msc, &reg);
+	/* MPAM-Fb MSC accesses cannot be done in atomic context. */
+	if (msc->iface == MPAM_IFACE_MMIO) {
+		mpam_msc_read_esr(msc, &reg);
 
-	errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
-	if (!errcode)
-		return IRQ_NONE;
+		errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
+		if (!errcode)
+			return IRQ_NONE;
 
-	/* Clear level triggered irq */
-	mpam_msc_clear_esr(msc);
+		/* Clear level triggered irq */
+		mpam_msc_clear_esr(msc);
 
-	partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg);
-	pmg = FIELD_GET(MPAMF_ESR_PMG, reg);
-	ris = FIELD_GET(MPAMF_ESR_RIS, reg);
+		partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg);
+		pmg = FIELD_GET(MPAMF_ESR_PMG, reg);
+		ris = FIELD_GET(MPAMF_ESR_RIS, reg);
 
-	pr_err_ratelimited("error irq from msc:%u '%s', partid:%u, pmg: %u, ris: %u\n",
-			   msc->id, mpam_errcode_names[errcode], partid, pmg,
-			   ris);
+		pr_err_ratelimited("error irq from msc:%u '%s', partid:%u, pmg: %u, ris: %u\n",
+				   msc->id, mpam_errcode_names[errcode], partid,
+				   pmg, ris);
 
Quoted 6 lines

drivers/resctrl/mpam_devices.c static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc) · line 2671

Quoted 4 lines
drivers/resctrl/mpam_devices.c +2 -2
-	/* Disable this interrupt. */
-	mpam_disable_msc_ecr(msc);
+		/* Disable this interrupt. */
+		mpam_disable_msc_ecr(msc);

As an error interrupt is final can we just disable the IRQ?

Doing that should be covered by mpam_unregister_irqs() as part of the mpam_broken_work, shouldn't it? Or do you want to do it earlier?

Quoted 2 lines

Is it
useful? I see there is a function disable_irq_no_sync().

If we want to do it earlier, the _nosync variant sounds promising, although the comment talks about it being nested, so I guess it would need to be balanced? Which might be tricky here, since I guess the IRQ would be disabled again in mpam_unregister_irqs()?

Quoted 6 lines

drivers/resctrl/mpam_devices.c static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc) · line 2673

Quoted 2 lines
drivers/resctrl/mpam_devices.c +2 -0
+	} else {
+		pr_err_ratelimited("unknown error irq from msc:%u\n", msc->id);

Should we report by irq number?
As MSC may share interrupts we don't know which MSC caused the error irq at this point. On MMIO platforms we read the ESR to establish this.

I see what you mean, though I am not sure if the user would be able to make sense of any interrupt number? I would put it in anyway, more information doesn't hurt.

Cheers,
Andre

Quoted 9 lines

Thanks,

Ben

Quoted 4 lines
drivers/resctrl/mpam_devices.c +1 -0
+	}
 
  	/* Are we racing with the thread disabling MPAM? */
  	if (!mpam_is_enabled())
maintainer
#31 lore raw

Re: [PATCH v2 14/15] arm_mpam: prevent MPAM-Fb accesses inside IRQ handler

Hi Andre,

Quoted 77 lines · Andre Przywara

Hi,

Quoted 65 lines · Ben Horgan

Hi Andre,

Quoted 62 lines · Andre Przywara

When an MPAM MSC gets into an error condition, it can trigger an error IRQ. We cannot really do much about those errors, but we at least query and log the error, then disable MPAM functionality.

This error report relies on reading the MSC's error status register (ESR) in the IRQ handler, which is not possible for MPAM-Fb based MSC accesses, since they involve mailbox routines that might sleep. The same is true for clearing the interrupt at the source, which requires MSC access.

For simplicity just skip the ESR read when the MSC is not using direct MMIO accesses, and just ignore the pending interrupts. We will wrap up MPAM functionality regardless, knowing the exact error value will not change that.

---
  drivers/resctrl/mpam_devices.c | 35 +++++++++++++++++++---------------

  1 file changed, 20 insertions(+), 15 deletions(-)
drivers/resctrl/ renamed from drivers/resctrl/mpam_devices.c +0 -0

mpam_devices.c
index b858ff389bff..4a088e6cd235 100644

drivers/resctrl/mpam_devices.c +1 -1
@@ -2639,7 +2639,7 @@ static int mpam_disable_msc_ecr(void *_msc)
     static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
   {
-    u64 reg;
+    u64 reg = 0;
       u16 partid;
       u8 errcode, pmg, ris;

  @@ -2648,25 +2648,30 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
                         &msc->accessibility)))
          return IRQ_NONE;
  -    mpam_msc_read_esr(msc, &reg);
+    /* MPAM-Fb MSC accesses cannot be done in atomic context. */ +    if (msc->iface == MPAM_IFACE_MMIO) {
+        mpam_msc_read_esr(msc, &reg);
  -    errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
-    if (!errcode)
-        return IRQ_NONE;
+        errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
+        if (!errcode)
+            return IRQ_NONE;
  -    /* Clear level triggered irq */
-    mpam_msc_clear_esr(msc);
+        /* Clear level triggered irq */
+        mpam_msc_clear_esr(msc);
  -    partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg);
-    pmg = FIELD_GET(MPAMF_ESR_PMG, reg);
-    ris = FIELD_GET(MPAMF_ESR_RIS, reg);
+        partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg); +        pmg = FIELD_GET(MPAMF_ESR_PMG, reg);
+        ris = FIELD_GET(MPAMF_ESR_RIS, reg);
  -    pr_err_ratelimited("error irq from msc:%u '%s', partid:%u,
pmg: %u, ris: %u\n",
-               msc->id, mpam_errcode_names[errcode], partid, pmg, -               ris);
+        pr_err_ratelimited("error irq from msc:%u '%s', partid:%u,
pmg: %u, ris: %u\n",

Quoted 8 lines
Quoted 6 lines

+                   msc->id, mpam_errcode_names[errcode], partid, +                   pmg, ris);
  -    /* Disable this interrupt. */
-    mpam_disable_msc_ecr(msc);
+        /* Disable this interrupt. */
+        mpam_disable_msc_ecr(msc);

As an error interrupt is final can we just disable the IRQ?

Quoted 2 lines

Doing that should be covered by mpam_unregister_irqs() as part of the mpam_broken_work, shouldn't it? Or do you want to do it earlier?

I think leaving it to mpam_broken_work risks having the hard irq handler run again and again once we get an error interrupt. I'm just looking at the code today but I think this is analogous to using IRQF_ONESHOT to mask the interrupt until the threaded handler is finished.

Quoted 8 lines
Quoted 3 lines

Is it
useful? I see there is a function disable_irq_no_sync().

If we want to do it earlier, the _nosync variant sounds promising, although the comment talks about it being nested, so I guess it would need to be balanced? Which might be tricky here, since I guess the IRQ would be disabled again in mpam_unregister_irqs()?

mpam_unregister_irqs() disables the interrupt by clearing the msc enable in mpam_disable_msc_ecr() rather than disabling by irq. I don't see how this would cause a problem. Or does free_percpu_irq() or devm_free_irq() do something incompatible?

Quoted 24 lines
Quoted 7 lines
Quoted 3 lines

+    } else {
+        pr_err_ratelimited("unknown error irq from msc:%u\n", msc->id);

Should we report by irq number?
As MSC may share interrupts we don't know which MSC caused the error irq at this point. On MMIO platforms we read the ESR to establish this.

I see what you mean, though I am not sure if the user would be able to make sense of any interrupt number? I would put it in anyway, more information doesn't hurt.

Cheers,
Andre

Quoted 8 lines

Thanks,

Ben

Quoted 3 lines

+    }
        /* Are we racing with the thread disabling MPAM? */ if (!mpam_is_enabled())

#32 lore raw

Re: [PATCH v2 14/15] arm_mpam: prevent MPAM-Fb accesses inside IRQ handler

Hi,

Quoted 80 lines · Ben Horgan

Hi Andre,

Quoted 77 lines · Andre Przywara

Hi,

Quoted 73 lines · Ben Horgan

Hi Andre,

Quoted 60 lines · Andre Przywara

When an MPAM MSC gets into an error condition, it can trigger an error IRQ. We cannot really do much about those errors, but we at least query and log the error, then disable MPAM functionality.

This error report relies on reading the MSC's error status register (ESR) in the IRQ handler, which is not possible for MPAM-Fb based MSC accesses, since they involve mailbox routines that might sleep. The same is true for clearing the interrupt at the source, which requires MSC access.

For simplicity just skip the ESR read when the MSC is not using direct MMIO accesses, and just ignore the pending interrupts. We will wrap up MPAM functionality regardless, knowing the exact error value will not change that.

---
   drivers/resctrl/mpam_devices.c | 35 +++++++++++++++++++---------------
   1 file changed, 20 insertions(+), 15 deletions(-)
drivers/resctrl/ renamed from drivers/resctrl/mpam_devices.c +0 -0

mpam_devices.c
index b858ff389bff..4a088e6cd235 100644

drivers/resctrl/mpam_devices.c +1 -1
@@ -2639,7 +2639,7 @@ static int mpam_disable_msc_ecr(void *_msc)
     static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
   {
-    u64 reg;
+    u64 reg = 0;
       u16 partid;
       u8 errcode, pmg, ris;
   @@ -2648,25 +2648,30 @@ static irqreturn_t __mpam_irq_handler(int
 irq, struct mpam_msc *msc)

                         &msc->accessibility)))
          return IRQ_NONE;
  -    mpam_msc_read_esr(msc, &reg);
+    /* MPAM-Fb MSC accesses cannot be done in atomic context. */ +    if (msc->iface == MPAM_IFACE_MMIO) {
+        mpam_msc_read_esr(msc, &reg);
  -    errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
-    if (!errcode)
-        return IRQ_NONE;
+        errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
+        if (!errcode)
+            return IRQ_NONE;
  -    /* Clear level triggered irq */
-    mpam_msc_clear_esr(msc);
+        /* Clear level triggered irq */
+        mpam_msc_clear_esr(msc);
  -    partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg);
-    pmg = FIELD_GET(MPAMF_ESR_PMG, reg);
-    ris = FIELD_GET(MPAMF_ESR_RIS, reg);
+        partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg); +        pmg = FIELD_GET(MPAMF_ESR_PMG, reg);
+        ris = FIELD_GET(MPAMF_ESR_RIS, reg);
  -    pr_err_ratelimited("error irq from msc:%u '%s', partid:%u,
pmg: %u, ris: %u\n",
-               msc->id, mpam_errcode_names[errcode], partid, pmg, -               ris);

Quoted 8 lines

+        pr_err_ratelimited("error irq from msc:%u '%s', partid:%u,
pmg: %u, ris: %u\n",
+                   msc->id, mpam_errcode_names[errcode], partid, +                   pmg, ris);
  -    /* Disable this interrupt. */
-    mpam_disable_msc_ecr(msc);
+        /* Disable this interrupt. */
+        mpam_disable_msc_ecr(msc);

As an error interrupt is final can we just disable the IRQ?

Quoted 8 lines
Quoted 2 lines

Doing that should be covered by mpam_unregister_irqs() as part of the mpam_broken_work, shouldn't it? Or do you want to do it earlier?

I think leaving it to mpam_broken_work risks having the hard irq handler run again and again once we get an error interrupt. I'm just looking at the code today but I think this is analogous to using IRQF_ONESHOT to mask the interrupt until the threaded handler is finished.

Maybe, but masking and disabling are not the same thing, and ONESHOT most probably takes care of any imbalances. Whereas when we just disable the IRQ here on some occasions, without telling anyone, it may leave it in a weird state?

Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

Is it
useful? I see there is a function disable_irq_no_sync().

If we want to do it earlier, the _nosync variant sounds promising, although the comment talks about it being nested, so I guess it would need to be balanced? Which might be tricky here, since I guess the IRQ would be disabled again in mpam_unregister_irqs()?

Quoted 4 lines

mpam_unregister_irqs() disables the interrupt by clearing the msc enable in mpam_disable_msc_ecr() rather than disabling by irq. I don't see how this would cause a problem. Or does free_percpu_irq() or devm_free_irq() do something incompatible?

According to the description of free_irq(), it does disable the IRQ when no user is left. I'd assume it does this via the same mechanism we are trying to use, which would cause an imbalance?

Cheers,
Andre

Quoted 25 lines
Quoted 24 lines
Quoted 7 lines
Quoted 3 lines

+    } else {
+        pr_err_ratelimited("unknown error irq from msc:%u\n", msc->id);

Should we report by irq number?
As MSC may share interrupts we don't know which MSC caused the error irq at this point. On MMIO platforms we read the ESR to establish this.

I see what you mean, though I am not sure if the user would be able to make sense of any interrupt number? I would put it in anyway, more information doesn't hurt.

Cheers,
Andre

Quoted 8 lines

Thanks,

Ben

Quoted 3 lines

+    }
        /* Are we racing with the thread disabling MPAM? */ if (!mpam_is_enabled())

maintainer
#33 lore raw

Re: [PATCH v2 14/15] arm_mpam: prevent MPAM-Fb accesses inside IRQ handler

Hi Andre,

Quoted 95 lines · Andre Przywara

Hi,

Quoted 83 lines · Ben Horgan

Hi Andre,

Quoted 80 lines · Andre Przywara

Hi,

Quoted 76 lines · Ben Horgan

Hi Andre,

Quoted 63 lines · Andre Przywara

When an MPAM MSC gets into an error condition, it can trigger an error IRQ. We cannot really do much about those errors, but we at least query
and log the error, then disable MPAM functionality.

This error report relies on reading the MSC's error status register (ESR) in the IRQ handler, which is not possible for MPAM-Fb based MSC accesses, since they involve mailbox routines that might sleep. The same is true for clearing the interrupt at the source, which requires MSC access.

For simplicity just skip the ESR read when the MSC is not using direct MMIO accesses, and just ignore the pending interrupts. We will wrap up MPAM functionality regardless, knowing the exact error value will not change that.

---
   drivers/resctrl/mpam_devices.c | 35 ++++++++++++++++++
+---------------

   1 file changed, 20 insertions(+), 15 deletions(-)
drivers/resctrl/ renamed from drivers/resctrl/mpam_devices.c +0 -0

mpam_devices.c
index b858ff389bff..4a088e6cd235 100644

drivers/resctrl/mpam_devices.c +0 -0
@@ -2639,7 +2639,7 @@ static int mpam_disable_msc_ecr(void *_msc)

     static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
   {
-    u64 reg;
+    u64 reg = 0;
       u16 partid;
       u8 errcode, pmg, ris;
   @@ -2648,25 +2648,30 @@ static irqreturn_t __mpam_irq_handler(int irq, struct mpam_msc *msc)
                          &msc->accessibility)))
           return IRQ_NONE;
   -    mpam_msc_read_esr(msc, &reg);
+    /* MPAM-Fb MSC accesses cannot be done in atomic context. */ +    if (msc->iface == MPAM_IFACE_MMIO) {
+        mpam_msc_read_esr(msc, &reg);
   -    errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
-    if (!errcode)
-        return IRQ_NONE;
+        errcode = FIELD_GET(MPAMF_ESR_ERRCODE, reg);
+        if (!errcode)
+            return IRQ_NONE;
   -    /* Clear level triggered irq */
-    mpam_msc_clear_esr(msc);
+        /* Clear level triggered irq */
+        mpam_msc_clear_esr(msc);
   -    partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg);
-    pmg = FIELD_GET(MPAMF_ESR_PMG, reg);
-    ris = FIELD_GET(MPAMF_ESR_RIS, reg);
+        partid = FIELD_GET(MPAMF_ESR_PARTID_MON, reg);
+        pmg = FIELD_GET(MPAMF_ESR_PMG, reg);
+        ris = FIELD_GET(MPAMF_ESR_RIS, reg);
   -    pr_err_ratelimited("error irq from msc:%u '%s', partid:%u,
pmg: %u, ris: %u\n",
-               msc->id, mpam_errcode_names[errcode], partid, pmg, -               ris);

Quoted 8 lines

+        pr_err_ratelimited("error irq from msc:%u '%s', partid:%u,
pmg: %u, ris: %u\n",
+                   msc->id, mpam_errcode_names[errcode], partid, +                   pmg, ris);
   -    /* Disable this interrupt. */
-    mpam_disable_msc_ecr(msc);
+        /* Disable this interrupt. */
+        mpam_disable_msc_ecr(msc);

As an error interrupt is final can we just disable the IRQ?

Quoted 8 lines
Quoted 2 lines

Doing that should be covered by mpam_unregister_irqs() as part of the mpam_broken_work, shouldn't it? Or do you want to do it earlier?

I think leaving it to mpam_broken_work risks having the hard irq handler run again and again once we get an error interrupt. I'm just looking at the code today but I think this is analogous to using IRQF_ONESHOT to mask the interrupt until the threaded handler is finished.

Quoted 4 lines

Maybe, but masking and disabling are not the same thing, and ONESHOT most probably takes care of any imbalances. Whereas when we just disable the IRQ here on some occasions, without telling anyone, it may leave it in a weird state?

Ok, that could be problematic.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines

Is it
useful? I see there is a function disable_irq_no_sync().

If we want to do it earlier, the _nosync variant sounds promising, although the comment talks about it being nested, so I guess it would need to be balanced? Which might be tricky here, since I guess the IRQ would be disabled again in mpam_unregister_irqs()?

Quoted 8 lines
Quoted 4 lines

mpam_unregister_irqs() disables the interrupt by clearing the msc enable in mpam_disable_msc_ecr() rather than disabling by irq. I don't see how this would cause a problem. Or does free_percpu_irq() or devm_free_irq() do something incompatible?

According to the description of free_irq(), it does disable the IRQ when no user is left. I'd assume it does this via the same mechanism we are trying to use, which would cause an imbalance?

Can we get around this by using a ONESHOT threaded interrupt and just using the threaded part to do mpam_disable_msc_ecr() and queue the mpam_broken_work?

Thanks,

Ben

Quoted 32 lines

Cheers,
Andre

Quoted 27 lines
Quoted 26 lines
Quoted 9 lines
Quoted 4 lines

+    } else {
+        pr_err_ratelimited("unknown error irq from msc:%u\n", msc-

Quoted 1 line

id);

Should we report by irq number?
As MSC may share interrupts we don't know which MSC caused the error irq
at this point. On MMIO platforms we read the ESR to establish this.

I see what you mean, though I am not sure if the user would be able to make sense of any interrupt number? I would put it in anyway, more information doesn't hurt.

Cheers,
Andre

Quoted 8 lines

Thanks,

Ben

Quoted 3 lines

+    }
         /* Are we racing with the thread disabling MPAM? */ if (!mpam_is_enabled())

#34 lore raw

PATCH v2 15/15 arm_mpam: detect and enable MPAM-Fb PCC support

In reply to Andre Przywara

The Arm MPAM-Fb specification [1] describes a protocol to access MSC registers through a firmware interface. This requires a shared memory region to hold the message, and a mailbox to trigger the access. For ACPI this is wrapped as a PCC channel, described using existing ACPI abstractions.

Add code to parse those PCC table descriptions associated with an MSC, and store the parsed information in the MSC struct.
There can be multiple PCC channels, and each channel can serve multiple MSCs, so we need to keep track of the channel usage, using a list and a refcount.
This will be used by the MPAM-Fb access wrapper code.

[1] https://developer.arm.com/documentation/den0144/latest

---
 drivers/acpi/arm64/mpam.c      |   2 +
 drivers/resctrl/mpam_devices.c | 113 ++++++++++++++++++++++++++++++++-
 2 files changed, 113 insertions(+), 2 deletions(-)
drivers/acpi/arm64/mpam.c +2 -0
index 84963a20c3e7..64bc84bb2029 100644
@@ -256,6 +256,8 @@ static struct platform_device * __init acpi_mpam_parse_msc(struct acpi_mpam_msc_
 	} else if (iface == MPAM_IFACE_PCC) {
 		props[next_prop++] = PROPERTY_ENTRY_U32("pcc-channel",
 							tbl_msc->base_address);
+		props[next_prop++] = PROPERTY_ENTRY_U32("msc-id",
+							tbl_msc->identifier);
 	}
 
 	acpi_mpam_parse_irqs(pdev, tbl_msc, res, &next_res);
drivers/resctrl/mpam_devices.c +111 -2

155 lines not on this page · raw message

Signature · 1 line
2.43.0
maintainer
#35 lore raw

Re: [PATCH v2 15/15] arm_mpam: detect and enable MPAM-Fb PCC support

Hi Andre,

Quoted 187 lines · Andre Przywara

The Arm MPAM-Fb specification [1] describes a protocol to access MSC registers through a firmware interface. This requires a shared memory region to hold the message, and a mailbox to trigger the access. For ACPI this is wrapped as a PCC channel, described using existing ACPI abstractions.

Add code to parse those PCC table descriptions associated with an MSC, and store the parsed information in the MSC struct.
There can be multiple PCC channels, and each channel can serve multiple MSCs, so we need to keep track of the channel usage, using a list and a refcount.
This will be used by the MPAM-Fb access wrapper code.

[1] https://developer.arm.com/documentation/den0144/latest

---
 drivers/acpi/arm64/mpam.c      |   2 +
 drivers/resctrl/mpam_devices.c | 113 ++++++++++++++++++++++++++++++++-
 2 files changed, 113 insertions(+), 2 deletions(-)
drivers/acpi/arm64/mpam.c +2 -0
index 84963a20c3e7..64bc84bb2029 100644
@@ -256,6 +256,8 @@ static struct platform_device * __init acpi_mpam_parse_msc(struct acpi_mpam_msc_
 	} else if (iface == MPAM_IFACE_PCC) {
 		props[next_prop++] = PROPERTY_ENTRY_U32("pcc-channel",
 							tbl_msc->base_address);
+		props[next_prop++] = PROPERTY_ENTRY_U32("msc-id",
+							tbl_msc->identifier);
 	}
 
 	acpi_mpam_parse_irqs(pdev, tbl_msc, res, &next_res);
drivers/resctrl/mpam_devices.c +101 -2
index 4a088e6cd235..35214520bfd4 100644
@@ -19,6 +19,7 @@
 #include <linux/irqdesc.h>
 #include <linux/list.h>
 #include <linux/lockdep.h>
+#include <linux/mailbox_client.h>
 #include <linux/mutex.h>
 #include <linux/platform_device.h>
 #include <linux/printk.h>
@@ -27,6 +28,9 @@
 #include <linux/types.h>
 #include <linux/workqueue.h>
 
+#include <acpi/pcc.h>
+#include <acpi/acpi_io.h>
+
 #include "mpam_internal.h"
 #include "mpam_fb.h"
 
@@ -50,6 +54,88 @@ static LIST_HEAD(mpam_all_msc);
 
 struct srcu_struct mpam_srcu;
 
+/* PCC channels might be serving multiple MSCs, so keep a refcounted list. */
+static DEFINE_MUTEX(pcc_chan_list_lock);
+static LIST_HEAD(pcc_chan_list);
+
+static void mpam_pcc_rx_callback(struct mbox_client *cl, void *msg)
+{
+	/* TODO: wake up tasks blocked on this MSC's PCC channel */
+}
+
+static struct mpam_pcc_chan *mpam_pcc_chan_get(struct device *dev,
+					       int subspace_id)
+{
+	struct mpam_pcc_chan *cur;
+
+	mutex_lock(&pcc_chan_list_lock);
+
+	list_for_each_entry(cur, &pcc_chan_list, pcc_chans) {
+		if (cur->subspace_id == subspace_id) {
+			cur->refcount++;
+			mutex_unlock(&pcc_chan_list_lock);
+
+			return cur;
+		}
+	}
+
+	cur = kzalloc_obj(*cur);
+	if (!cur) {
+		mutex_unlock(&pcc_chan_list_lock);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	cur->pcc_cl.dev = dev;
+	cur->pcc_cl.rx_callback = mpam_pcc_rx_callback;
+	cur->pcc_cl.tx_block = true;
+	cur->pcc_cl.tx_tout = 1000; /* 1s */
+
+	cur->pcc_chan = pcc_mbox_request_channel(&cur->pcc_cl, subspace_id);
+	if (IS_ERR(cur->pcc_chan)) {
+		long err = PTR_ERR(cur->pcc_chan);
+
+		kfree(cur);
+		mutex_unlock(&pcc_chan_list_lock);
+		return ERR_PTR(err);
+	}
+
+	mutex_init(&cur->pcc_chan_lock);
+	cur->subspace_id = subspace_id;
+	cur->refcount = 1;
+
+	list_add_tail(&cur->pcc_chans, &pcc_chan_list);
+
+	mutex_unlock(&pcc_chan_list_lock);
+
+	return cur;
+}
+
+static int mpam_pcc_chan_put(struct mpam_pcc_chan *pcc_chan)
+{
+	struct mpam_pcc_chan *cur, *tmp;
+
+	if (!pcc_chan)
+		return 0;
+
+	mutex_lock(&pcc_chan_list_lock);
+
+	list_for_each_entry_safe(cur, tmp, &pcc_chan_list, pcc_chans) {
+		if (cur == pcc_chan) {
+			if (!--cur->refcount) {
+				pcc_mbox_free_channel(cur->pcc_chan);
+				list_del(&pcc_chan->pcc_chans);
+				kfree(cur);
+			}
+			mutex_unlock(&pcc_chan_list_lock);
+			return 0;
+		}
+	}
+
+	mutex_unlock(&pcc_chan_list_lock);
+
+	return -ENOENT;
+}
+
 /*
  * Number of MSCs that have been probed. Once all MSCs have been probed MPAM
  * can be enabled.
@@ -2202,6 +2288,8 @@ static void mpam_msc_drv_remove(struct platform_device *pdev)
 {
 	struct mpam_msc *msc = platform_get_drvdata(pdev);
 
+	mpam_pcc_chan_put(msc->pcc_chan);
+
 	mutex_lock(&mpam_list_lock);
 	mpam_msc_destroy(msc);
 	mutex_unlock(&mpam_list_lock);
@@ -2212,7 +2300,7 @@ static void mpam_msc_drv_remove(struct platform_device *pdev)
 static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
 {
 	int err;
-	u32 tmp;
+	u32 pcc_subspace_id;
 	struct mpam_msc *msc;
 	struct resource *msc_res;
 	struct device *dev = &pdev->dev;
@@ -2257,7 +2345,8 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
 	if (err)
 		return ERR_PTR(err);
 
-	if (device_property_read_u32(&pdev->dev, "pcc-channel", &tmp))
+	if (device_property_read_u32(&pdev->dev, "pcc-channel",
+				     &pcc_subspace_id))
 		msc->iface = MPAM_IFACE_MMIO;
 	else
 		msc->iface = MPAM_IFACE_PCC;
@@ -2273,6 +2362,26 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
 		}
 		msc->mapped_hwpage_sz = msc_res->end - msc_res->start;
 		msc->mapped_hwpage = io;
+	} else if (msc->iface == MPAM_IFACE_PCC) {
+		u32 msc_id;
+
+		if (device_property_read_u32(&pdev->dev, "msc-id", &msc_id)) {
+			pr_err("missing MPAM-Fb MSC identifier\n");
+			return ERR_PTR(-EINVAL);
+		}
+		msc->mpam_fb_msc_id = msc_id;
+
+		msc->pcc_chan = mpam_pcc_chan_get(&pdev->dev, pcc_subspace_id);

drivers/resctrl/mpam_devices.c static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev) · line 2384

Quoted 10 lines
drivers/resctrl/mpam_devices.c +10 -0
+		if (IS_ERR(msc->pcc_chan)) {
+			pr_err("Failed to request MSC PCC channel\n");
+			return (void *)msc->pcc_chan;
+		}
+
+		if (msc->pcc_chan->pcc_chan->shmem_size < MPAM_FB_MAX_MSG_SIZE) {
+			pr_err("MPAM-Fb PCC channel size too small.\n");
+			mpam_pcc_chan_put(msc->pcc_chan);
+			return ERR_PTR(-ENOMEM);
+		}

I think we need a version check here just in case a MPAM-Fb 2.x is released.

Thanks,

Ben

Quoted 3 lines
drivers/resctrl/mpam_devices.c +0 -0
 	} else {
 		return ERR_PTR(-EINVAL);
 	}
maintainer
#36 lore raw

Re: [PATCH v2 15/15] arm_mpam: detect and enable MPAM-Fb PCC support

Hi Andre,

Quoted 24 lines · Ben Horgan

Hi Andre,

Quoted 21 lines · Andre Przywara

The Arm MPAM-Fb specification [1] describes a protocol to access MSC registers through a firmware interface. This requires a shared memory region to hold the message, and a mailbox to trigger the access. For ACPI this is wrapped as a PCC channel, described using existing ACPI abstractions.

Add code to parse those PCC table descriptions associated with an MSC, and store the parsed information in the MSC struct.
There can be multiple PCC channels, and each channel can serve multiple MSCs, so we need to keep track of the channel usage, using a list and a refcount.
This will be used by the MPAM-Fb access wrapper code.

[1] https://developer.arm.com/documentation/den0144/latest

---
 drivers/acpi/arm64/mpam.c      |   2 +
 drivers/resctrl/mpam_devices.c | 113 ++++++++++++++++++++++++++++++++-
 2 files changed, 113 insertions(+), 2 deletions(-)
Quoted 10 lines

drivers/acpi/arm64/mpam.c static struct platform_device * __init acpi_mpam_parse_msc(struct acpi_mpam_msc_ · line 260

Quoted 10 lines
drivers/acpi/arm64/mpam.c +2 -0
index 84963a20c3e7..64bc84bb2029 100644
@@ -256,6 +256,8 @@ static struct platform_device * __init acpi_mpam_parse_msc(struct acpi_mpam_msc_
 	} else if (iface == MPAM_IFACE_PCC) {
 		props[next_prop++] = PROPERTY_ENTRY_U32("pcc-channel",
 							tbl_msc->base_address);
+		props[next_prop++] = PROPERTY_ENTRY_U32("msc-id",
+							tbl_msc->identifier);

I expect it's easier if changes in this file are a separate patch Just to help bring this to the attention of the ACPI/arm64 maintainers.

Thanks,

Ben

Quoted 177 lines
Quoted 156 lines
drivers/acpi/arm64/mpam.c +0 -0
 	}
 
 	acpi_mpam_parse_irqs(pdev, tbl_msc, res, &next_res);
drivers/resctrl/mpam_devices.c +101 -2
index 4a088e6cd235..35214520bfd4 100644
@@ -19,6 +19,7 @@
 #include <linux/irqdesc.h>
 #include <linux/list.h>
 #include <linux/lockdep.h>
+#include <linux/mailbox_client.h>
 #include <linux/mutex.h>
 #include <linux/platform_device.h>
 #include <linux/printk.h>
@@ -27,6 +28,9 @@
 #include <linux/types.h>
 #include <linux/workqueue.h>
 
+#include <acpi/pcc.h>
+#include <acpi/acpi_io.h>
+
 #include "mpam_internal.h"
 #include "mpam_fb.h"
 
@@ -50,6 +54,88 @@ static LIST_HEAD(mpam_all_msc);
 
 struct srcu_struct mpam_srcu;
 
+/* PCC channels might be serving multiple MSCs, so keep a refcounted list. */
+static DEFINE_MUTEX(pcc_chan_list_lock);
+static LIST_HEAD(pcc_chan_list);
+
+static void mpam_pcc_rx_callback(struct mbox_client *cl, void *msg)
+{
+	/* TODO: wake up tasks blocked on this MSC's PCC channel */
+}
+
+static struct mpam_pcc_chan *mpam_pcc_chan_get(struct device *dev,
+					       int subspace_id)
+{
+	struct mpam_pcc_chan *cur;
+
+	mutex_lock(&pcc_chan_list_lock);
+
+	list_for_each_entry(cur, &pcc_chan_list, pcc_chans) {
+		if (cur->subspace_id == subspace_id) {
+			cur->refcount++;
+			mutex_unlock(&pcc_chan_list_lock);
+
+			return cur;
+		}
+	}
+
+	cur = kzalloc_obj(*cur);
+	if (!cur) {
+		mutex_unlock(&pcc_chan_list_lock);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	cur->pcc_cl.dev = dev;
+	cur->pcc_cl.rx_callback = mpam_pcc_rx_callback;
+	cur->pcc_cl.tx_block = true;
+	cur->pcc_cl.tx_tout = 1000; /* 1s */
+
+	cur->pcc_chan = pcc_mbox_request_channel(&cur->pcc_cl, subspace_id);
+	if (IS_ERR(cur->pcc_chan)) {
+		long err = PTR_ERR(cur->pcc_chan);
+
+		kfree(cur);
+		mutex_unlock(&pcc_chan_list_lock);
+		return ERR_PTR(err);
+	}
+
+	mutex_init(&cur->pcc_chan_lock);
+	cur->subspace_id = subspace_id;
+	cur->refcount = 1;
+
+	list_add_tail(&cur->pcc_chans, &pcc_chan_list);
+
+	mutex_unlock(&pcc_chan_list_lock);
+
+	return cur;
+}
+
+static int mpam_pcc_chan_put(struct mpam_pcc_chan *pcc_chan)
+{
+	struct mpam_pcc_chan *cur, *tmp;
+
+	if (!pcc_chan)
+		return 0;
+
+	mutex_lock(&pcc_chan_list_lock);
+
+	list_for_each_entry_safe(cur, tmp, &pcc_chan_list, pcc_chans) {
+		if (cur == pcc_chan) {
+			if (!--cur->refcount) {
+				pcc_mbox_free_channel(cur->pcc_chan);
+				list_del(&pcc_chan->pcc_chans);
+				kfree(cur);
+			}
+			mutex_unlock(&pcc_chan_list_lock);
+			return 0;
+		}
+	}
+
+	mutex_unlock(&pcc_chan_list_lock);
+
+	return -ENOENT;
+}
+
 /*
  * Number of MSCs that have been probed. Once all MSCs have been probed MPAM
  * can be enabled.
@@ -2202,6 +2288,8 @@ static void mpam_msc_drv_remove(struct platform_device *pdev)
 {
 	struct mpam_msc *msc = platform_get_drvdata(pdev);
 
+	mpam_pcc_chan_put(msc->pcc_chan);
+
 	mutex_lock(&mpam_list_lock);
 	mpam_msc_destroy(msc);
 	mutex_unlock(&mpam_list_lock);
@@ -2212,7 +2300,7 @@ static void mpam_msc_drv_remove(struct platform_device *pdev)
 static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
 {
 	int err;
-	u32 tmp;
+	u32 pcc_subspace_id;
 	struct mpam_msc *msc;
 	struct resource *msc_res;
 	struct device *dev = &pdev->dev;
@@ -2257,7 +2345,8 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
 	if (err)
 		return ERR_PTR(err);
 
-	if (device_property_read_u32(&pdev->dev, "pcc-channel", &tmp))
+	if (device_property_read_u32(&pdev->dev, "pcc-channel",
+				     &pcc_subspace_id))
 		msc->iface = MPAM_IFACE_MMIO;
 	else
 		msc->iface = MPAM_IFACE_PCC;
@@ -2273,6 +2362,26 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
 		}
 		msc->mapped_hwpage_sz = msc_res->end - msc_res->start;
 		msc->mapped_hwpage = io;
+	} else if (msc->iface == MPAM_IFACE_PCC) {
+		u32 msc_id;
+
+		if (device_property_read_u32(&pdev->dev, "msc-id", &msc_id)) {
+			pr_err("missing MPAM-Fb MSC identifier\n");
+			return ERR_PTR(-EINVAL);
+		}
+		msc->mpam_fb_msc_id = msc_id;
+
+		msc->pcc_chan = mpam_pcc_chan_get(&pdev->dev, pcc_subspace_id);

drivers/resctrl/mpam_devices.c static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev) · line 2384

Quoted 10 lines
drivers/resctrl/mpam_devices.c +10 -0
+		if (IS_ERR(msc->pcc_chan)) {
+			pr_err("Failed to request MSC PCC channel\n");
+			return (void *)msc->pcc_chan;
+		}
+
+		if (msc->pcc_chan->pcc_chan->shmem_size < MPAM_FB_MAX_MSG_SIZE) {
+			pr_err("MPAM-Fb PCC channel size too small.\n");
+			mpam_pcc_chan_put(msc->pcc_chan);
+			return ERR_PTR(-ENOMEM);
+		}

I think we need a version check here just in case a MPAM-Fb 2.x is released.

Thanks,

Ben

Quoted 3 lines
drivers/resctrl/mpam_devices.c +0 -0
 	} else {
 		return ERR_PTR(-EINVAL);
 	}
#37 lore raw

Re: [PATCH v2 15/15] arm_mpam: detect and enable MPAM-Fb PCC support

In reply to Ben Horgan

Hi,

Quoted 194 lines · Ben Horgan

Hi Andre,

Quoted 191 lines · Andre Przywara

The Arm MPAM-Fb specification [1] describes a protocol to access MSC registers through a firmware interface. This requires a shared memory region to hold the message, and a mailbox to trigger the access. For ACPI this is wrapped as a PCC channel, described using existing ACPI abstractions.

Add code to parse those PCC table descriptions associated with an MSC, and store the parsed information in the MSC struct.
There can be multiple PCC channels, and each channel can serve multiple MSCs, so we need to keep track of the channel usage, using a list and a refcount.
This will be used by the MPAM-Fb access wrapper code.

[1] https://developer.arm.com/documentation/den0144/latest

---
  drivers/acpi/arm64/mpam.c      |   2 +
  drivers/resctrl/mpam_devices.c | 113 ++++++++++++++++++++++++++++++++-
  2 files changed, 113 insertions(+), 2 deletions(-)
drivers/acpi/arm64/mpam.c +2 -0
index 84963a20c3e7..64bc84bb2029 100644
@@ -256,6 +256,8 @@ static struct platform_device * __init acpi_mpam_parse_msc(struct acpi_mpam_msc_
  	} else if (iface == MPAM_IFACE_PCC) {
  		props[next_prop++] = PROPERTY_ENTRY_U32("pcc-channel",
  							tbl_msc->base_address);
+		props[next_prop++] = PROPERTY_ENTRY_U32("msc-id",
+							tbl_msc->identifier);
  	}
 
  	acpi_mpam_parse_irqs(pdev, tbl_msc, res, &next_res);
drivers/resctrl/mpam_devices.c +105 -2
index 4a088e6cd235..35214520bfd4 100644
@@ -19,6 +19,7 @@
  #include <linux/irqdesc.h>
  #include <linux/list.h>
  #include <linux/lockdep.h>
+#include <linux/mailbox_client.h>
  #include <linux/mutex.h>
  #include <linux/platform_device.h>
  #include <linux/printk.h>
@@ -27,6 +28,9 @@
  #include <linux/types.h>
  #include <linux/workqueue.h>
 
+#include <acpi/pcc.h>
+#include <acpi/acpi_io.h>
+
  #include "mpam_internal.h"
  #include "mpam_fb.h"
 
@@ -50,6 +54,88 @@ static LIST_HEAD(mpam_all_msc);
 
  struct srcu_struct mpam_srcu;
 
+/* PCC channels might be serving multiple MSCs, so keep a refcounted list. */
+static DEFINE_MUTEX(pcc_chan_list_lock);
+static LIST_HEAD(pcc_chan_list);
+
+static void mpam_pcc_rx_callback(struct mbox_client *cl, void *msg)
+{
+	/* TODO: wake up tasks blocked on this MSC's PCC channel */
+}
+
+static struct mpam_pcc_chan *mpam_pcc_chan_get(struct device *dev,
+					       int subspace_id)
+{
+	struct mpam_pcc_chan *cur;
+
+	mutex_lock(&pcc_chan_list_lock);
+
+	list_for_each_entry(cur, &pcc_chan_list, pcc_chans) {
+		if (cur->subspace_id == subspace_id) {
+			cur->refcount++;
+			mutex_unlock(&pcc_chan_list_lock);
+
+			return cur;
+		}
+	}
+
+	cur = kzalloc_obj(*cur);
+	if (!cur) {
+		mutex_unlock(&pcc_chan_list_lock);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	cur->pcc_cl.dev = dev;
+	cur->pcc_cl.rx_callback = mpam_pcc_rx_callback;
+	cur->pcc_cl.tx_block = true;
+	cur->pcc_cl.tx_tout = 1000; /* 1s */
+
+	cur->pcc_chan = pcc_mbox_request_channel(&cur->pcc_cl, subspace_id);
+	if (IS_ERR(cur->pcc_chan)) {
+		long err = PTR_ERR(cur->pcc_chan);
+
+		kfree(cur);
+		mutex_unlock(&pcc_chan_list_lock);
+		return ERR_PTR(err);
+	}
+
+	mutex_init(&cur->pcc_chan_lock);
+	cur->subspace_id = subspace_id;
+	cur->refcount = 1;
+
+	list_add_tail(&cur->pcc_chans, &pcc_chan_list);
+
+	mutex_unlock(&pcc_chan_list_lock);
+
+	return cur;
+}
+
+static int mpam_pcc_chan_put(struct mpam_pcc_chan *pcc_chan)
+{
+	struct mpam_pcc_chan *cur, *tmp;
+
+	if (!pcc_chan)
+		return 0;
+
+	mutex_lock(&pcc_chan_list_lock);
+
+	list_for_each_entry_safe(cur, tmp, &pcc_chan_list, pcc_chans) {
+		if (cur == pcc_chan) {
+			if (!--cur->refcount) {
+				pcc_mbox_free_channel(cur->pcc_chan);
+				list_del(&pcc_chan->pcc_chans);
+				kfree(cur);
+			}
+			mutex_unlock(&pcc_chan_list_lock);
+			return 0;
+		}
+	}
+
+	mutex_unlock(&pcc_chan_list_lock);
+
+	return -ENOENT;
+}
+
  /*
   * Number of MSCs that have been probed. Once all MSCs have been probed MPAM
   * can be enabled.
@@ -2202,6 +2288,8 @@ static void mpam_msc_drv_remove(struct platform_device *pdev)
  {
  	struct mpam_msc *msc = platform_get_drvdata(pdev);
 
+	mpam_pcc_chan_put(msc->pcc_chan);
+
  	mutex_lock(&mpam_list_lock);
  	mpam_msc_destroy(msc);
  	mutex_unlock(&mpam_list_lock);
@@ -2212,7 +2300,7 @@ static void mpam_msc_drv_remove(struct platform_device *pdev)
  static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
  {
  	int err;
-	u32 tmp;
+	u32 pcc_subspace_id;
  	struct mpam_msc *msc;
  	struct resource *msc_res;
  	struct device *dev = &pdev->dev;
@@ -2257,7 +2345,8 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
  	if (err)
  		return ERR_PTR(err);
 
-	if (device_property_read_u32(&pdev->dev, "pcc-channel", &tmp))
+	if (device_property_read_u32(&pdev->dev, "pcc-channel",
+				     &pcc_subspace_id))
  		msc->iface = MPAM_IFACE_MMIO;
  	else
  		msc->iface = MPAM_IFACE_PCC;
@@ -2273,6 +2362,26 @@ static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev)
  		}
  		msc->mapped_hwpage_sz = msc_res->end - msc_res->start;
  		msc->mapped_hwpage = io;
+	} else if (msc->iface == MPAM_IFACE_PCC) {
+		u32 msc_id;
+
+		if (device_property_read_u32(&pdev->dev, "msc-id", &msc_id)) {
+			pr_err("missing MPAM-Fb MSC identifier\n");
+			return ERR_PTR(-EINVAL);
+		}
+		msc->mpam_fb_msc_id = msc_id;
+
+		msc->pcc_chan = mpam_pcc_chan_get(&pdev->dev, pcc_subspace_id);
+		if (IS_ERR(msc->pcc_chan)) {
+			pr_err("Failed to request MSC PCC channel\n");
+			return (void *)msc->pcc_chan;
+		}
Quoted 8 lines

drivers/resctrl/mpam_devices.c static struct mpam_msc *do_mpam_msc_drv_probe(struct platform_device *pdev) · line 2384

Quoted 6 lines
drivers/resctrl/mpam_devices.c +6 -0
+
+		if (msc->pcc_chan->pcc_chan->shmem_size < MPAM_FB_MAX_MSG_SIZE) {
+			pr_err("MPAM-Fb PCC channel size too small.\n");
+			mpam_pcc_chan_put(msc->pcc_chan);
+			return ERR_PTR(-ENOMEM);
+		}

I think we need a version check here just in case a MPAM-Fb 2.x is released.

OK, that was slightly more involved, since that requires an MPAM-Fb message to learn the protocol version, but I did it now. Includes refactoring of mpam_fb_send_request() to support more than two message types.

Cheers,
Andre