Skip to contentSkip to conversation

Introduce MMIO-based CMT access for Enhanced RDT

PATCH Superseded Chen Yu 2 participants 27 messages Jul 1 – Jul 14

Subsystems (2)

v5 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Chen Yu

author
#0 lore raw

v5 cover Introduce MMIO-based CMT access for Enhanced RDT

v4: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v1: https://lore.kernel.org/lkml/[email protected]/

Intel Enhanced Resource Director Technology (ERDT) extends the existing RDT framework with two major capabilities:

1. MMIO-based access to monitoring and allocation registers, replacing the legacy MSR-based interface.
2. Region-aware RDT for fine-grained control over different tiers of

     memory (e.g., CXL.mem, DDR).

This is described in the Intel RDT Architecture Specification:
https://cdrdv2-public.intel.com/789566/356688-intel-rdt-arch-spec.pdf

This patch set focuses on the first part: enabling MMIO-based access for Cache Monitoring Technology (CMT), while CAT/MBM/MBA are still using MSR. The platform advertises the MMIO register layout through the ACPI ERDT (Enhanced Resource Director Technology) table, which contains sub-tables describing per-domain register regions for monitoring and allocation.

With ERDT, L3 cache occupancy counters are read via MMIO rather than MSR, allowing the reads to be performed from any CPU without requiring cross-CPU IPIs. This series parses the relevant ACPI sub-tables (RMDD, CMRC), prepares the resctrl monitor infrastructure for MMIO-based reads, and adds initial support for reading L3 occupancy via the CMRC interface.

kselftest of CMT and L3_CAT has passed with minor adjustment at
https://lore.kernel.org/lkml/[email protected]/.

V5 has undergone internal Sashiko review.

Changes from V4 to V5:
There are some major changes since v4:
- (biggest change) Eliminate the xarray for runtime lookups; embed struct erdt_domain_info directly in rdt_hw_l3_mon_domain and assign during l3_mon_domain_setup().
- Separate CPUID and ACPI enumeration cleanly. Do not use CPUID feature flags to gate MMIO-based monitoring. Use ACPI table presence (e.g., CMRC table) to determine event enablement.
- Use ACPI RMDD's own "Max RMID" field for MMIO access instead of relying on CPUID's max RMID (which applies to MSR).
- Enforce the SNC constraint in code rather than burying it behind a comment WARN. Disable mon_capable in rdt_get_l3_mon_config() when ERDT is enabled and snc_nodes_per_l3_cache > 1.
- Split non-resctrl changes (topology.c, apic.h) into a separate preparatory patch prefixed with x86/topology.
- Move the "depends on X86" to "X86_64" adjustment to a separate patch with explicit justification in its changelog.

Misc in V5:
- arch_priv is not used to distinguish whether a monitor is MSR-based or ERDT backed, since a unified erdt_cpu_has() is introduced to check whether a specific RDT feature is backed by ERDT and its sub-tables. [PATCH 9/10]
- I have removed Thomas's Reviewed-by tag from [PATCH 10/10], as the patch was updated to rely on erdt_cpu_has().

Thanks Tony, Reinette, Thomas, Hongyu for your time to look at this patch set.

Changes from V3 to V4:
- Remove the redundant table length check in subtbl_valid() (Thomas Gleixner)
- Reuse subtbl_valid() for all the table iteration (Thomas Gleixner)
- Refine the commit log of [PATCH 5/6] to state that this change is a preparation for [PATCH 6/6] rather than fixing an existing issue (Thomas Gleixner, Reinette Chatre, Tony Luck)
- Fix if CACD lists all CPUs in the LLC domain (sashiko)
- Deal with a corner case that if there is no valid RMDD tables, the erdt_enabled_flag should remain false.(sashiko)
- Add Thomas's Reviewed-by and Hongyu's Tested-by.

Changes from V2 to V3:
- Wrap __resctrl_arch_late_init() to avoid the goto logic. (Thomas Gleixner)
- Make the variables in struct erdt_domain_info tabular format (Thomas Gleixner)
- Remove tail comments (Thomas Gleixner)
- Make the name of erdt_enabled() and variable in it consistent and comprehensible. (Thomas Gleixner)
- Use topo_lookup_cpuid() to search the CPU id according to the x2apic id (Thomas Gleixner)
- Fix kernel doc comment format (Thomas Gleixner)
- Use brackets for multiple lines "if" case. (Thomas Gleixner)
- Let the parameter for cacd_init() to fully utilize 100 characters. (Thomas Gleixner)
- Variables are reordered in reverse fir-tree.(Thomas Gleixner)
- Added a named constant and use it in the rmdd->flags check. (Thomas Gleixner)
- Introduce helper functions to make the code readable when iterating the RMDD tables. (Thomas Gleixner)
- Make the macros tabular format. (Thomas Gleixner)

Changes from V1 to V2:
- Add #include <linux/cleanup.h> to follow the "include-what-you-use" best practice (Tony Luck)
- Fix 3 issues reported by:
https://sashiko.dev/#/patchset/cover.1779872016.git.yu.c.chen%40intel.com
Remove the variable of cacd in struct erdt_domain_info as it will never be used after initialization.
Invoke erdt_exit() to avoid resource leak if rdt_alloc_capable and rdt_mon_capable are both false.
Adjust the comments suggested by sashiko.

Anil S Keshavamurthy (1):
x86/resctrl: Parse ACPI ERDT table and save CACD cpumask for RMDD
domains

Chen Yu (8):
x86/resctrl: Require 64-bit x86 for resctrl support x86/topology: Export topo_lookup_cpuid() for resctrl use x86/resctrl: Attach ACPI ERDT information to L3 mon domain on CPU
online
x86/resctrl: Parse ACPI CMRC table
x86/resctrl: Replace "msr" in monitoring data identifiers x86/resctrl: Refactor the monitor read function x86/resctrl: Introduce helpers to read L3 occupancy via MMIO x86/resctrl: Enable read L3 occupancy via MMIO

Tony Luck (1):
fs/resctrl: Do not invoke smp_processor_id() in preemptible context

 arch/x86/Kconfig                       |   4 +-
 arch/x86/include/asm/apic.h            |   1 +
 arch/x86/include/asm/resctrl.h         |   6 +
 arch/x86/kernel/cpu/resctrl/Makefile   |   1 +
 arch/x86/kernel/cpu/resctrl/core.c     |  64 +++-
 arch/x86/kernel/cpu/resctrl/erdt.c     | 439 +++++++++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/internal.h |  40 ++-
 arch/x86/kernel/cpu/resctrl/monitor.c  |  82 +++--
 arch/x86/kernel/cpu/topology.c         |   2 +-
 fs/resctrl/monitor.c                   |  48 ++-
 include/linux/resctrl.h                |   1 +
 11 files changed, 643 insertions(+), 45 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/resctrl/erdt.c
Signature · 1 line
2.45.2
author
#1 lore raw

PATCH v5 1/10 x86/resctrl: Require 64-bit x86 for resctrl support

All known hardware that supports Intel RDT or AMD QoS is 64-bit. Future enhancements like MMIO-based Enhanced RDT (ERDT) monitoring require readq()/writeq() which are only available on 64-bit. There is also no realistic use case for users to build a 32-bit kernel on a server and enable resctrl.

Drop 32-bit support by changing the X86_CPU_RESCTRL dependency from X86 to X86_64.

Since X86_CPU_RESCTRL now implies X86_64, remove the redundant X86_64 dependency from X86_CPU_RESCTRL_INTEL_AET.

---
v4->v5:
New patch. Split from previous patch in v4.

---
 arch/x86/Kconfig | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/Kconfig +2 -2
index fa4f0079614c..b2c2b10f8a57 100644
@@ -514,7 +514,7 @@ config X86_MPPARSE
 
 config X86_CPU_RESCTRL
 	bool "x86 CPU resource control support"
-	depends on X86 && (CPU_SUP_INTEL || CPU_SUP_AMD)
+	depends on X86_64 && (CPU_SUP_INTEL || CPU_SUP_AMD)
 	depends on MISC_FILESYSTEMS
 	select ARCH_HAS_CPU_RESCTRL
 	select RESCTRL_FS
@@ -537,7 +537,7 @@ config X86_CPU_RESCTRL
 
 config X86_CPU_RESCTRL_INTEL_AET
 	bool "Intel Application Energy Telemetry"
-	depends on X86_64 && X86_CPU_RESCTRL && CPU_SUP_INTEL && INTEL_PMT_TELEMETRY=y && INTEL_TPMI=y
+	depends on X86_CPU_RESCTRL && CPU_SUP_INTEL && INTEL_PMT_TELEMETRY=y && INTEL_TPMI=y
 	help
 	  Enable per-RMID telemetry events in resctrl.
 
Signature · 1 line
2.45.2
author
#2 lore raw

PATCH v5 2/10 x86/topology: Export topo_lookup_cpuid() for resctrl use

In reply to Chen Yu

Make topo_lookup_cpuid() accessible outside of topology.c and add its declaration to asm/apic.h. This is needed by Enhanced RDT (ERDT) to resolve x2APIC IDs from ACPI tables to logical CPU numbers.

No functional change expected.

---
v4->v5:
New patch.

---
 arch/x86/include/asm/apic.h    | 1 +
 arch/x86/kernel/cpu/topology.c | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/asm/apic.h +1 -0
index 9cd493d467d4..bb84651b14bd 100644
@@ -54,6 +54,7 @@ static inline void x86_32_probe_apic(void) { }
 #endif
 
 extern u32 cpuid_to_apicid[];
+int topo_lookup_cpuid(u32 apic_id);
 
 #define CPU_ACPIID_INVALID	U32_MAX
 
arch/x86/kernel/cpu/topology.c +1 -1
index 4913b64ec592..bcee70fb9277 100644
@@ -92,7 +92,7 @@ static inline u32 topo_apicid(u32 apicid, enum x86_topology_domains dom)
 	return apicid & (UINT_MAX << x86_topo_system.dom_shifts[dom - 1]);
 }
 
-static int topo_lookup_cpuid(u32 apic_id)
+int topo_lookup_cpuid(u32 apic_id)
 {
 	int i;
 
Signature · 1 line
2.45.2
maintainer
#3 lore raw

Re: [PATCH v5 02/10] x86/topology: Export topo_lookup_cpuid() for resctrl use

Hi Chenyu,

Since this targets a different subsystem, could you please re-order the series to have this as first patch to make it obvious that it does not depend on anything in this series.

Quoted 3 lines · Chen Yu

Make topo_lookup_cpuid() accessible outside of topology.c and add its declaration to asm/apic.h. This is needed by Enhanced RDT (ERDT) to resolve x2APIC IDs from ACPI tables to logical CPU numbers.

The first sentence of this changelog describes verbatim what can clearly be seen from the patch self without any context or problem description about why this change is needed.
tip changelogs always need to start with *why* the patch is needed before describing the change self. For more details about changelog requirements, please see "Changelog" in Documentation/process/maintainer-tip.rst

Quoted 5 lines

No functional change expected.

Also please follow "Ordering of commit tags" found in Documentation/process/maintainer-tip.rst for all patches in this series.

Reinette

author
#4 lore raw

Re: [PATCH v5 02/10] x86/topology: Export topo_lookup_cpuid() for resctrl use

Hi Reinette,

Quoted 6 lines · Reinette Chatre

Hi Chenyu,

Since this targets a different subsystem, could you please re-order the series to have this as first patch to make it obvious that it does not depend on anything in this series.

OK, will do.

Quoted 5 lines · Chen Yu
Quoted 3 lines

Make topo_lookup_cpuid() accessible outside of topology.c and add its declaration to asm/apic.h. This is needed by Enhanced RDT (ERDT) to resolve x2APIC IDs from ACPI tables to logical CPU numbers.

Quoted 7 lines

The first sentence of this changelog describes verbatim what can clearly be seen from the patch self without any context or problem description about why this change is needed.
tip changelogs always need to start with *why* the patch is needed before describing the change self. For more details about changelog requirements, please see "Changelog" in Documentation/process/maintainer-tip.rst

OK, let me revisit the commit log of this patch set to revise it.

Quoted 9 lines
Quoted 5 lines

No functional change expected.

Also please follow "Ordering of commit tags" found in Documentation/process/maintainer-tip.rst for all patches in this series.

Thank you for pointing that out; I wasn't aware. Let me rearrange the order of this patch set.

thanks,
Chenyu

author
#5 lore raw

PATCH v5 3/10 x86/resctrl: Parse ACPI ERDT table and save CACD cpumask for RMDD domains

In reply to Chen Yu

From: Anil S Keshavamurthy <[email protected]>

Parse the ERDT (Enhanced RDT) ACPI table so enhanced RDT features can consume firmware-provided domain information.

The ERDT may contain these sub-tables:

- Resource Management Domain Description Structure (RMDD)
- CPU Agent Collection Description Structure (CACD)
- Cache Monitoring Registers for CPU Agents Description Structure

    (CMRC)

There is one ERDT per platform. Each RMDD describes one resource management domain (RMD), also known as an L3 domain, and carries MMIO base information for later monitoring support.

Add basic ERDT table parsing and retain the relevant sub-table information for later processing.

Handle RMDD specially. For each RMDD, parse the associated CACD, map its x2APIC IDs to logical CPUs, and save the resulting CPU mask. This mask associates each ERDT domain with the CPUs that belong to it and is used later when attaching ERDT data to resctrl monitoring domains.

---
v4->v5:
Listed the major changes below:
Eliminate the xarray for runtime lookups, embed struct erdt_domain_info directly in rdt_hw_l3_mon_domain and assign during l3_mon_domain_setup().

Removing 32-bit support is a significant change, move it to a separate patch.

Use the "maximum RMID" exposed via ACPI tables instead of relying on CPUID maximum RMID. Introduce a helper for this.

Split non-resctrl changes (topology.c, apic.h) into a separate preparatory patch with x86/topology prefix.

---
 arch/x86/kernel/cpu/resctrl/Makefile   |   1 +
 arch/x86/kernel/cpu/resctrl/core.c     |  14 +-
 arch/x86/kernel/cpu/resctrl/erdt.c     | 271 +++++++++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/internal.h |  24 +++
 4 files changed, 309 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/kernel/cpu/resctrl/erdt.c
arch/x86/kernel/cpu/resctrl/Makefile +1 -0
index 273ddfa30836..2216ee084832 100644
@@ -2,6 +2,7 @@
 obj-$(CONFIG_X86_CPU_RESCTRL)		+= core.o rdtgroup.o monitor.o
 obj-$(CONFIG_X86_CPU_RESCTRL)		+= ctrlmondata.o
 obj-$(CONFIG_X86_CPU_RESCTRL_INTEL_AET)	+= intel_aet.o
+obj-$(CONFIG_X86_CPU_RESCTRL)		+= erdt.o
 obj-$(CONFIG_RESCTRL_FS_PSEUDO_LOCK)	+= pseudo_lock.o
 
 # To allow define_trace.h's recursive include:
arch/x86/kernel/cpu/resctrl/core.c +13 -1
index 9c01d2562b7a..23925bcd71d7 100644
@@ -1013,6 +1013,7 @@ static __init void check_quirks(void)
 
 static __init bool get_rdt_resources(void)
 {
+	erdt_init();
 	rdt_alloc_capable = get_rdt_alloc_resources();
 	rdt_mon_capable = get_rdt_mon_resources();
 
@@ -1114,7 +1115,7 @@ void resctrl_cpu_detect(struct cpuinfo_x86 *c)
 	}
 }
 
-static int __init resctrl_arch_late_init(void)
+static int __init __resctrl_arch_late_init(void)
 {
 	struct rdt_resource *r;
 	int state, ret, i;
@@ -1157,6 +1158,15 @@ static int __init resctrl_arch_late_init(void)
 	return 0;
 }
 
+static int __init resctrl_arch_late_init(void)
+{
+	int ret = __resctrl_arch_late_init();
+
+	if (ret)
+		erdt_exit();
+	return ret;
+}
+
 late_initcall(resctrl_arch_late_init);
 
 static void __exit resctrl_arch_exit(void)
@@ -1166,6 +1176,8 @@ static void __exit resctrl_arch_exit(void)
 	cpuhp_remove_state(rdt_online);
 
 	resctrl_exit();
+
+	erdt_exit();
 }
 
 __exitcall(resctrl_arch_exit);
arch/x86/kernel/cpu/resctrl/erdt.c new file +271 -0

271 lines not on this page · raw message

arch/x86/kernel/cpu/resctrl/internal.h +24 -0
index e3cfa0c10e92..299d7222f693 100644
@@ -21,6 +21,26 @@
 
 #define RMID_VAL_UNAVAIL		BIT_ULL(62)
 
+/*
+ * Index into erdt_domain_info::base[] for each MMIO region.
+ * @ERDT_MMIO_RMDD_CREG: RMDD control register base address
+ * @ERDT_MMIO_CMRC_BASE: CMRC monitoring register base address
+ */
+enum erdt_mmio_type {
+	ERDT_MMIO_RMDD_CREG,
+	ERDT_MMIO_CMRC_BASE,
+	ERDT_MMIO_LAST = ERDT_MMIO_CMRC_BASE
+};
+
+#define ERDT_MMIO_NUM_TYPES	(ERDT_MMIO_LAST + 1)
+
+struct erdt_domain_info {
+	void __iomem		*base[ERDT_MMIO_NUM_TYPES];
+	cpumask_var_t		cpu_mask;
+	int			max_rmid;
+	struct list_head	list;
+};
+
 /*
  * With the above fields in use 62 bits remain in MSR_IA32_QM_CTR for
  * data to be returned. The counter width is discovered from the hardware
@@ -253,4 +273,8 @@ static inline void intel_aet_mon_domain_setup(int cpu, int id, struct rdt_resour
 static inline bool intel_handle_aet_option(bool force_off, char *tok) { return false; }
 #endif
 
+int erdt_get_max_rmid(int cpu);
+int erdt_init(void);
+void erdt_exit(void);
+
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
Signature · 1 line
2.45.2
maintainer
#6 lore raw

Re: [PATCH v5 03/10] x86/resctrl: Parse ACPI ERDT table and save CACD cpumask for RMDD domains

Hi Chenyu,

Quoted 11 lines · Chen Yu

From: Anil S Keshavamurthy <[email protected]>

Parse the ERDT (Enhanced RDT) ACPI table so enhanced RDT features can consume firmware-provided domain information.

The ERDT may contain these sub-tables:

- Resource Management Domain Description Structure (RMDD)
- CPU Agent Collection Description Structure (CACD)
- Cache Monitoring Registers for CPU Agents Description Structure

    (CMRC)

How should "The ERDT may contain these sub-tables" be interpreted here? This sounds like some high level partial description of ERDT that is not specific to this patch.

Quoted 2 lines

There is one ERDT per platform. Each RMDD describes one resource

"one ERDT" -> "one ERDT table"?

Quoted 17 lines

management domain (RMD), also known as an L3 domain, and carries MMIO base information for later monitoring support.

Add basic ERDT table parsing and retain the relevant sub-table information for later processing.

Handle RMDD specially. For each RMDD, parse the associated CACD, map its x2APIC IDs to logical CPUs, and save the resulting CPU mask. This mask associates each ERDT domain with the CPUs that belong to it and is used later when attaching ERDT data to resctrl monitoring domains.

---

...

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +3 -0
new file mode 100644
index 000000000000..6405df9be817
@@ -0,0 +1,271 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Enhanced Resource Director Technology (ERDT)

arch/x86/kernel/cpu/resctrl/erdt.c line 13

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +10 -0
+ *
+ * Copyright (C) 2026 Intel Corporation
+ *
+ */
+
+#define pr_fmt(fmt)     "resctrl: " fmt
+
+#include <linux/acpi.h>
+#include <linux/cleanup.h>
+#include <linux/cpu.h>

Which parts of cpu.h are used?

arch/x86/kernel/cpu/resctrl/erdt.c line 18

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +5 -0
+#include <linux/err.h>
+#include <linux/overflow.h>
+#include <linux/resctrl.h>
+#include <linux/sizes.h>
+#include <linux/xarray.h>

xarray is no longer needed?

arch/x86/kernel/cpu/resctrl/erdt.c line 26

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +8 -0
+
+#include <asm/apic.h>
+
+#include "internal.h"
+
+static LIST_HEAD(domain_info_list);
+
+static bool __erdt_enabled;

Is double underscore needed?
Could you please add a comment above the variable to describe what it means when "erdt is enabled"?

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +7 -0
+
+#define ERDT_VALID_VERSION		1
+#define RMDD_FLAG_CPU_L3_DOMAIN		BIT(0)
+
+/* Bitmask of valid sub-tables found in the first RMDD, used to ensure all RMDDs match. */
+static u32 valid_subtbl_mask;
+

arch/x86/kernel/cpu/resctrl/erdt.c line 43

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +10 -0
+int erdt_get_max_rmid(int cpu)
+{
+	struct erdt_domain_info *d;
+	struct list_head *pos;
+
+	if (!__erdt_enabled)
+		return 0;
+
+	list_for_each(pos, &domain_info_list) {
+		d = container_of(pos, struct erdt_domain_info, list);

(list_for_each_entry()?)

arch/x86/kernel/cpu/resctrl/erdt.c line 50

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +7 -0
+
+		if (cpumask_test_cpu(cpu, d->cpu_mask))
+			return d->max_rmid;
+	}
+
+	return -1;
+}

Using a CPU as parameter to determine the maximum RMID supported by ERDT is unexpected. Looking ahead at how this function is used I find only one usage:

	rdt_get_l3_mon_config() {
		...
		/*
		 * Currently assume all CPU domains share the same maximum RMID
		 * value from the RMDD table, use CPU0 domain's value.
		 */
		int erdt_max_rmid = erdt_get_max_rmid(0);

From this usage I do not see any reason why to do any CPU matching ... erdt_get_max_rmid() could just return the max_rmid of any domain ... but that does not look right either since the comment states an assumption that is never enforced in the code.

Could this be made more robust by replacing the "per-erdt-domain max_rmid" with one global "ERDT max RMID" to which all RMDD's max RMID is compared to *ensure* they are all the same. If there is no such guarantee then I expect this will be the minimum among all RMDD? All seems to point to there only being one global value that is determined during ERDT enumeration and then this helper can just return that value directly?

arch/x86/kernel/cpu/resctrl/erdt.c line 57

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +7 -0
+
+static void __iomem *erdt_ioremap(phys_addr_t base, u32 num_pages, const char *desc)
+{
+	void __iomem *addr;
+	size_t size;
+
+	if (check_mul_overflow((size_t)num_pages, (size_t)SZ_4K, &size))

I do not think check_mul_overflow() requires size_t type, does it?

arch/x86/kernel/cpu/resctrl/erdt.c line 64

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +7 -0
+		return NULL;
+
+	addr = ioremap(base, size);
+	if (!addr) {
+		pr_err("ERDT: Failed to map %s at phys addr %pa (size: %u pages)\n",
+		       desc, &base, num_pages);
+	}

(unnecessary braces)

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +7 -0
+	return addr;
+}
+
+static void erdt_iounmap_domain(struct erdt_domain_info *domain)
+{
+	for (int i = 0; i < ERDT_MMIO_NUM_TYPES; i++) {
+		if (domain->base[i]) {

arch/x86/kernel/cpu/resctrl/erdt.c line 81

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +10 -0
+			iounmap(domain->base[i]);
+			domain->base[i] = NULL;
+		}
+	}
+}
+
+static void cleanup_one_domain(struct erdt_domain_info *d)
+{
+	erdt_iounmap_domain(d);
+	free_cpumask_var(d->cpu_mask);

free_cpumask_var() does not look right ... it is for stack usage, no? (more later)

Quoted 51 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +51 -0
+	kfree(d);
+}
+
+/*
+ * Save CACD information for this RMDD:
+ * convert the X2APIC to CPU and save them in a mask.
+ */
+static __init int cacd_init(struct acpi_subtbl_hdr_16 *subtbl,
+			    struct erdt_domain_info *domain_info)
+{
+	struct acpi_erdt_cacd *cacd = (struct acpi_erdt_cacd *)subtbl;
+	int num_ids, cpu;
+
+	if (cacd->header.length < struct_size(cacd, X2APICIDS, 1)) {
+		pr_warn(FW_BUG "Invalid x2apicid CACD table\n");
+		return -EIO;
+	}
+
+	num_ids = (cacd->header.length - sizeof(*cacd)) / sizeof(cacd->X2APICIDS[0]);
+
+	for (int i = 0; i < num_ids; i++) {
+		cpu = topo_lookup_cpuid(cacd->X2APICIDS[i]);
+		if (cpu < 0) {
+			pr_warn(FW_BUG "Unknown x2apicid 0x%x\n", cacd->X2APICIDS[i]);
+			return -EIO;
+		}
+
+		cpumask_set_cpu(cpu, domain_info->cpu_mask);
+	}
+
+	return 0;
+}
+
+static inline struct acpi_subtbl_hdr_16 *rmdd_subtbl(struct acpi_erdt_rmdd *rmdd)
+{
+	return (void *)rmdd + sizeof(*rmdd);
+}
+
+static inline struct acpi_subtbl_hdr_16 *next_subtbl(struct acpi_subtbl_hdr_16 *subtbl)
+{
+	return (void *)subtbl + subtbl->length;
+}
+
+static inline bool subtbl_valid(void *end, struct acpi_subtbl_hdr_16 *subtbl)
+{
+	/* Ensure the header is within bounds before dereferencing it. */
+	if ((void *)subtbl + sizeof(*subtbl) > end)
+		return false;
+
+	/* A sub-table must be at least as large as its header. */
+	if (subtbl->length < sizeof(*subtbl))

arch/x86/kernel/cpu/resctrl/erdt.c line 142

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +10 -0
+		return false;
+
+	/* The entire sub-table (including body) must fit within the parent. */
+	if ((void *)subtbl + subtbl->length > end)
+		return false;
+
+	return true;
+}
+
+static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr)

nit: what is motivation for the "entry" term? this is only occurance of the word "entry" in this patch while RMDD is more frequently referred to as "table" or "sub-table".

arch/x86/kernel/cpu/resctrl/erdt.c line 150

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +8 -0
+{
+	struct erdt_domain_info *domain_info;
+	struct acpi_subtbl_hdr_16 *subtbl;
+	struct acpi_erdt_rmdd *rmdd;
+	u32 subtbl_mask = 0;
+
+	if (rmdd_hdr->length < sizeof(*rmdd)) {
+		pr_warn(FW_BUG "Invalid RMDD length %u\n", rmdd_hdr->length);

Please include unit, similar to equivalent ERDT message.

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +5 -0
+		return false;
+	}
+
+	rmdd = (struct acpi_erdt_rmdd *)rmdd_hdr;
+

arch/x86/kernel/cpu/resctrl/erdt.c line 165

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +10 -0
+	/* Quietly ignore non-CPU-based L3 domains */
+	if (!(rmdd->flags & RMDD_FLAG_CPU_L3_DOMAIN))
+		return true;
+
+	domain_info = kzalloc_obj(*domain_info, GFP_KERNEL);
+	if (!domain_info)
+		return false;
+
+	if (!zalloc_cpumask_var(&domain_info->cpu_mask, GFP_KERNEL))
+		goto cleanup;

Similar to free_cpumask_var() this does not look right since the cpu_mask is not on stack here ... (more later)

arch/x86/kernel/cpu/resctrl/erdt.c line 176

Quoted 11 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +11 -0
+
+	domain_info->base[ERDT_MMIO_RMDD_CREG] =
+		erdt_ioremap(rmdd->creg_base, rmdd->creg_size, "RMDD ctrl base");
+	if (!domain_info->base[ERDT_MMIO_RMDD_CREG])
+		goto cleanup;
+
+	for (subtbl = rmdd_subtbl(rmdd);
+	     subtbl_valid((void *)rmdd + rmdd->header.length, subtbl);
+	     subtbl = next_subtbl(subtbl)) {
+		switch (subtbl->type) {
+		case ACPI_ERDT_TYPE_CACD:

It is quite subtle that there could be multiple CACD tables. Could this be highlighted with a small comment? Something like:

			/* An RMDD table has one or more CACD sub-table(s) */
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +10 -0
+			if (cacd_init(subtbl, domain_info))
+				goto cleanup;
+
+			subtbl_mask |= BIT(ACPI_ERDT_TYPE_CACD);
+			break;
+		default:
+			break;
+		}
+	}
+

arch/x86/kernel/cpu/resctrl/erdt.c line 196

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +10 -0
+	if (!subtbl_mask)
+		goto cleanup;
+
+	/*
+	 * Require all RMDDs to support same set of sub-tables
+	 */
+	if (!valid_subtbl_mask) {
+		valid_subtbl_mask = subtbl_mask;
+	} else if (subtbl_mask != valid_subtbl_mask) {
+		pr_warn(FW_BUG "RMDD sub-table set does not match the first RMDD\n");

Would it be useful to print the domain ID to help diagnistics?

arch/x86/kernel/cpu/resctrl/erdt.c line 200

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +4 -0
+		goto cleanup;
+	}
+
+	if (!rmdd->max_rmid || rmdd->max_rmid > INT_MAX) {

rmdd->max_rmid is a u32 so INT_MAX test is not clear to me here

arch/x86/kernel/cpu/resctrl/erdt.c line 201

Quoted 1 line
arch/x86/kernel/cpu/resctrl/erdt.c new file +1 -0
+		pr_warn(FW_BUG "Unreasonable RMDD max_rmid %u\n", rmdd->max_rmid);

Is there a limit in the spec?

arch/x86/kernel/cpu/resctrl/erdt.c line 204

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +3 -0
+		goto cleanup;
+	}
+	domain_info->max_rmid = rmdd->max_rmid;

As mentioned before, instead of a per-domain RMID, could there be a global ERDT RMID that is initialized by first ERDT domain and updated/compared with every following ERDT domain?

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +6 -0
+
+	list_add(&domain_info->list, &domain_info_list);
+
+	return true;
+
+cleanup:

arch/x86/kernel/cpu/resctrl/erdt.c line 220

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +10 -0
+	cleanup_one_domain(domain_info);
+	return false;
+}
+
+void erdt_exit(void)
+{
+	struct erdt_domain_info *d;
+	struct list_head *pos, *n;
+
+	list_for_each_safe(pos, n, &domain_info_list) {

list_for_each_entry_safe()?

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +6 -0
+		d = container_of(pos, struct erdt_domain_info, list);
+		list_del(pos);
+		cleanup_one_domain(d);
+	}
+	__erdt_enabled = false;
+	valid_subtbl_mask = 0;

arch/x86/kernel/cpu/resctrl/erdt.c line 236

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +10 -0
+}
+
+static __init int enumerate_erdt_table(struct acpi_table_header *table_hdr)
+{
+	struct acpi_table_erdt *erdt = (struct acpi_table_erdt *)table_hdr;
+	struct acpi_subtbl_hdr_16 *subtbl;
+	void *table_end;
+
+	if (erdt->header.revision != ERDT_VALID_VERSION) {
+		pr_info("Unsupported ERDT table revision %d\n", erdt->header.revision);

Would it be helpful to print what the ERDT table revision is to help diagnostics?

Quoted 46 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +35 -0
+		return -EINVAL;
+	}
+
+	if (erdt->header.length < sizeof(*erdt)) {
+		pr_warn(FW_BUG "ERDT: Invalid table length %u bytes\n", erdt->header.length);
+		return -EINVAL;
+	}
+
+	subtbl = (void *)erdt + sizeof(struct acpi_table_erdt);
+	table_end = (void *)erdt + erdt->header.length;
+
+	while (subtbl_valid(table_end, subtbl)) {
+		if (subtbl->type == ACPI_ERDT_TYPE_RMDD &&
+		    !parse_rmdd_entry(subtbl))
+			goto cleanup;
+
+		subtbl = next_subtbl(subtbl);
+	}
+
+	if (list_empty(&domain_info_list))
+		goto cleanup;
+
+	__erdt_enabled = true;
+
+	return 0;
+
+cleanup:
+	erdt_exit();
+	return -EINVAL;
+}
+
+int __init erdt_init(void)
+{
+	return acpi_table_parse(ACPI_SIG_ERDT, enumerate_erdt_table);
+}
arch/x86/kernel/cpu/resctrl/internal.h +3 -0
index e3cfa0c10e92..299d7222f693 100644
@@ -21,6 +21,26 @@
 
 #define RMID_VAL_UNAVAIL		BIT_ULL(62)
 
+/*
+ * Index into erdt_domain_info::base[] for each MMIO region.
+ * @ERDT_MMIO_RMDD_CREG: RMDD control register base address

arch/x86/kernel/cpu/resctrl/internal.h line 36

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/internal.h +10 -0
+ * @ERDT_MMIO_CMRC_BASE: CMRC monitoring register base address
+ */
+enum erdt_mmio_type {
+	ERDT_MMIO_RMDD_CREG,
+	ERDT_MMIO_CMRC_BASE,
+	ERDT_MMIO_LAST = ERDT_MMIO_CMRC_BASE
+};
+
+#define ERDT_MMIO_NUM_TYPES	(ERDT_MMIO_LAST + 1)
+

Please add documentation here that describes each member of struct erdt_domain_info.

arch/x86/kernel/cpu/resctrl/internal.h line 39

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/internal.h +3 -0
+struct erdt_domain_info {
+	void __iomem		*base[ERDT_MMIO_NUM_TYPES];
+	cpumask_var_t		cpu_mask;

Should this be struct cpumask instead? Compare with struct rdt_domain_hdr. When evaluating cpumask_var_t, please read the detailed comments above its definition in include/linux/cpumask_types.h - specifically note the start:
*cpumask_var_t: struct cpumask for stack usage*

arch/x86/kernel/cpu/resctrl/internal.h line 41

Quoted 2 lines
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
+	int			max_rmid;
+	struct list_head	list;

Could you please rename this to be "node" or "entry" to make it obvious that this is an entry of a list? This is not consistent in resctrl but really helps when reading the code.

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
+};
+
 /*
  * With the above fields in use 62 bits remain in MSR_IA32_QM_CTR for
  * data to be returned. The counter width is discovered from the hardware

arch/x86/kernel/cpu/resctrl/internal.h static inline void intel_aet_mon_domain_setup(int cpu, int id, struct rdt_resour · line 280

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/internal.h +4 -0
@@ -253,4 +273,8 @@ static inline void intel_aet_mon_domain_setup(int cpu, int id, struct rdt_resour
 static inline bool intel_handle_aet_option(bool force_off, char *tok) { return false; }
 #endif
 
+int erdt_get_max_rmid(int cpu);
+int erdt_init(void);
+void erdt_exit(void);
+
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */

Reinette

author
#7 lore raw

Re: [PATCH v5 03/10] x86/resctrl: Parse ACPI ERDT table and save CACD cpumask for RMDD domains

Hi Chenyu,

Quoted 10 lines · Reinette Chatre

Hi Chenyu,

Quoted 7 lines · Chen Yu

From: Anil S Keshavamurthy <[email protected]>

Parse the ERDT (Enhanced RDT) ACPI table so enhanced RDT features can consume firmware-provided domain information.

The ERDT may contain these sub-tables:

Quoted 9 lines
Quoted 4 lines

- Resource Management Domain Description Structure (RMDD)
- CPU Agent Collection Description Structure (CACD)
- Cache Monitoring Registers for CPU Agents Description Structure

     (CMRC)

How should "The ERDT may contain these sub-tables" be interpreted here? This sounds like some high level partial description of ERDT that is not specific to this patch.

I planned to add some background context on ERDT. Let me remove unrelated content and change the sentence to:
Parse the RMDD subtables within the ERDT ACPI table and their nested CACD entries to construct per-domain CPU masks.

Quoted 5 lines
Quoted 2 lines

There is one ERDT per platform. Each RMDD describes one resource

"one ERDT" -> "one ERDT table"?

OK.

Quoted 13 lines
Quoted 13 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +7 -0
new file mode 100644
index 000000000000..6405df9be817
@@ -0,0 +1,271 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Enhanced Resource Director Technology (ERDT)
+ *
+ * Copyright (C) 2026 Intel Corporation
+ *
+ */
Quoted 9 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 13

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +6 -0
+
+#define pr_fmt(fmt)     "resctrl: " fmt
+
+#include <linux/acpi.h>
+#include <linux/cleanup.h>
+#include <linux/cpu.h>

Which parts of cpu.h are used?

It was used in previous version but not needed in this version. Additionally, cleanup.h and err.h can be removed as well, I will drop them.

Quoted 8 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 18

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +5 -0
+#include <linux/err.h>
+#include <linux/overflow.h>
+#include <linux/resctrl.h>
+#include <linux/sizes.h>
+#include <linux/xarray.h>

xarray is no longer needed?

Not needed, will remove it.

Quoted 10 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 26

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +8 -0
+
+#include <asm/apic.h>
+
+#include "internal.h"
+
+static LIST_HEAD(domain_info_list);
+
+static bool __erdt_enabled;

Is double underscore needed?

The double underscore was used to prevent name collisions with the existing erdt_enabled(). Let me switch it to a single underscore instead.

Quoted 3 lines

Could you please add a comment above the variable to describe what it means when "erdt is enabled"?

OK, let me add this:

/*
* Set when the ERDT ACPI table has been successfully parsed and at least
* one valid RMDD domain with a CACD cpumask exists. Cleared on teardown. */

Quoted 11 lines
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +11 -0
+
+#define ERDT_VALID_VERSION		1
+#define RMDD_FLAG_CPU_L3_DOMAIN		BIT(0)
+
+/* Bitmask of valid sub-tables found in the first RMDD, used to ensure all RMDDs match. */
+static u32 valid_subtbl_mask;
+
+int erdt_get_max_rmid(int cpu)
+{
+	struct erdt_domain_info *d;
+	struct list_head *pos;
Quoted 9 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 43

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +6 -0
+
+	if (!__erdt_enabled)
+		return 0;
+
+	list_for_each(pos, &domain_info_list) {
+		d = container_of(pos, struct erdt_domain_info, list);

(list_for_each_entry()?)

OK, will do.

Quoted 22 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 50

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +7 -0
+
+		if (cpumask_test_cpu(cpu, d->cpu_mask))
+			return d->max_rmid;
+	}
+
+	return -1;
+}

Using a CPU as parameter to determine the maximum RMID supported by ERDT is unexpected. Looking ahead at how this function is used I find only one usage:

	rdt_get_l3_mon_config() {
		...
		/*
		 * Currently assume all CPU domains share the same maximum RMID
		 * value from the RMDD table, use CPU0 domain's value.
		 */
		int erdt_max_rmid = erdt_get_max_rmid(0);

From this usage I do not see any reason why to do any CPU matching ... erdt_get_max_rmid()
could just return the max_rmid of any domain ... but that does not look right either since the comment states an assumption that is never enforced in the code.

Quoted 6 lines

Could this be made more robust by replacing the "per-erdt-domain max_rmid" with one global "ERDT max RMID" to which all RMDD's max RMID is compared to *ensure* they are all the same. If there is no such guarantee then I expect this will be the minimum among all RMDD? All seems to point to there only being one global value that is determined during ERDT enumeration and then this helper can just return that value directly?

Got it, let me switch to one global max_rmid = min(all CPU domains's non-zero max_rmid ). IO-domain RMID checks can be deferred for future support.

Quoted 10 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 57

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +7 -0
+
+static void __iomem *erdt_ioremap(phys_addr_t base, u32 num_pages, const char *desc)
+{
+	void __iomem *addr;
+	size_t size;
+
+	if (check_mul_overflow((size_t)num_pages, (size_t)SZ_4K, &size))

I do not think check_mul_overflow() requires size_t type, does it?

No need to stick to the size_t convention - there is no risk of overflow, I’ll remove it.

Quoted 10 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 64

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +7 -0
+		return NULL;
+
+	addr = ioremap(base, size);
+	if (!addr) {
+		pr_err("ERDT: Failed to map %s at phys addr %pa (size: %u pages)\n",
+		       desc, &base, num_pages);
+	}

(unnecessary braces)

Will remove it.

Quoted 11 lines
Quoted 11 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +11 -0
+	return addr;
+}
+
+static void erdt_iounmap_domain(struct erdt_domain_info *domain)
+{
+	for (int i = 0; i < ERDT_MMIO_NUM_TYPES; i++) {
+		if (domain->base[i]) {
+			iounmap(domain->base[i]);
+			domain->base[i] = NULL;
+		}
+	}
Quoted 9 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 81

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +6 -0
+}
+
+static void cleanup_one_domain(struct erdt_domain_info *d)
+{
+	erdt_iounmap_domain(d);
+	free_cpumask_var(d->cpu_mask);

free_cpumask_var() does not look right ... it is for stack usage, no? (more later)

If I understand correctly, cpumask_var_t can be used for both stack contexts and dynamic allocation? Depending on whether OFFSTACK is defined, it may either be a dynamically allocated pointer or an inline single-element struct cpumask array. That said, since erdt_domain_info itself is dynamically allocated, embedding cpu_mask inside erdt_domain_info would be clearer. Let me change it.

Quoted 6 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 142

Quoted 1 line
arch/x86/kernel/cpu/resctrl/erdt.c new file +1 -0
+static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr)

nit: what is motivation for the "entry" term? this is only occurance of the word "entry" in this patch while RMDD is more frequently referred to as "table" or "sub-table".

I previously considered the RMDD table an entry within the parent ERDT table. I’ll change the function name to parse_rmdd_table().

Quoted 11 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 150

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +8 -0
+{
+	struct erdt_domain_info *domain_info;
+	struct acpi_subtbl_hdr_16 *subtbl;
+	struct acpi_erdt_rmdd *rmdd;
+	u32 subtbl_mask = 0;
+
+	if (rmdd_hdr->length < sizeof(*rmdd)) {
+		pr_warn(FW_BUG "Invalid RMDD length %u\n", rmdd_hdr->length);

Please include unit, similar to equivalent ERDT message.

OK, will do.

Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +10 -0
+		return false;
+	}
+
+	rmdd = (struct acpi_erdt_rmdd *)rmdd_hdr;
+
+	/* Quietly ignore non-CPU-based L3 domains */
+	if (!(rmdd->flags & RMDD_FLAG_CPU_L3_DOMAIN))
+		return true;
+
+	domain_info = kzalloc_obj(*domain_info, GFP_KERNEL);
Quoted 9 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 165

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +5 -0
+	if (!domain_info)
+		return false;
+
+	if (!zalloc_cpumask_var(&domain_info->cpu_mask, GFP_KERNEL))
+		goto cleanup;

Similar to free_cpumask_var() this does not look right since the cpu_mask is not on stack here ... (more later)

Ok, let me switch to struct cpumask instead.

Quoted 14 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 176

Quoted 11 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +11 -0
+
+	domain_info->base[ERDT_MMIO_RMDD_CREG] =
+		erdt_ioremap(rmdd->creg_base, rmdd->creg_size, "RMDD ctrl base");
+	if (!domain_info->base[ERDT_MMIO_RMDD_CREG])
+		goto cleanup;
+
+	for (subtbl = rmdd_subtbl(rmdd);
+	     subtbl_valid((void *)rmdd + rmdd->header.length, subtbl);
+	     subtbl = next_subtbl(subtbl)) {
+		switch (subtbl->type) {
+		case ACPI_ERDT_TYPE_CACD:

It is quite subtle that there could be multiple CACD tables. Could this be highlighted with a small comment? Something like:

Quoted 2 lines
			/* An RMDD table has one or more CACD sub-table(s) */

OK, will do.

Quoted 14 lines
Quoted 14 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +14 -0
+			if (cacd_init(subtbl, domain_info))
+				goto cleanup;
+
+			subtbl_mask |= BIT(ACPI_ERDT_TYPE_CACD);
+			break;
+		default:
+			break;
+		}
+	}
+
+	if (!subtbl_mask)
+		goto cleanup;
+
+	/*
Quoted 9 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 196

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +6 -0
+	 * Require all RMDDs to support same set of sub-tables
+	 */
+	if (!valid_subtbl_mask) {
+		valid_subtbl_mask = subtbl_mask;
+	} else if (subtbl_mask != valid_subtbl_mask) {
+		pr_warn(FW_BUG "RMDD sub-table set does not match the first RMDD\n");

Would it be useful to print the domain ID to help diagnistics?

OK, let print the ID of the first RMDD and the "broken" RMDD.

Quoted 7 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 200

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +4 -0
+		goto cleanup;
+	}
+
+	if (!rmdd->max_rmid || rmdd->max_rmid > INT_MAX) {

rmdd->max_rmid is a u32 so INT_MAX test is not clear to me here

I overlooked this, let me remove the INT_MAX check.

Quoted 4 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 201

Quoted 1 line
arch/x86/kernel/cpu/resctrl/erdt.c new file +1 -0
+		pr_warn(FW_BUG "Unreasonable RMDD max_rmid %u\n", rmdd->max_rmid);

Is there a limit in the spec?

The spec defines it as a 4-byte value, so a u32 here should not overflow.

Quoted 8 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 204

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +3 -0
+		goto cleanup;
+	}
+	domain_info->max_rmid = rmdd->max_rmid;

As mentioned before, instead of a per-domain RMID, could there be a global ERDT RMID that is initialized by first ERDT domain and updated/compared with every following ERDT domain?

OK, will do.

Quoted 10 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 220

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +7 -0
+
+void erdt_exit(void)
+{
+	struct erdt_domain_info *d;
+	struct list_head *pos, *n;
+
+	list_for_each_safe(pos, n, &domain_info_list) {

list_for_each_entry_safe()?

OK.

Quoted 10 lines
Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +10 -0
+		d = container_of(pos, struct erdt_domain_info, list);
+		list_del(pos);
+		cleanup_one_domain(d);
+	}
+	__erdt_enabled = false;
+	valid_subtbl_mask = 0;
+}
+
+static __init int enumerate_erdt_table(struct acpi_table_header *table_hdr)
+{
Quoted 9 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 236

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +6 -0
+	struct acpi_table_erdt *erdt = (struct acpi_table_erdt *)table_hdr;
+	struct acpi_subtbl_hdr_16 *subtbl;
+	void *table_end;
+
+	if (erdt->header.revision != ERDT_VALID_VERSION) {
+		pr_info("Unsupported ERDT table revision %d\n", erdt->header.revision);

Would it be helpful to print what the ERDT table revision is to help diagnostics?

Yes, it will print the revision, do you mean print the expected revision?

pr_info("Unsupported ERDT table revision %d (expected %d)\n",

        erdt->header.revision, ERDT_VALID_VERSION);
Quoted 3 lines

Please add documentation here that describes each member of struct erdt_domain_info.

OK, will do.

Quoted 9 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 170

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +3 -0
+struct erdt_domain_info {
+	void __iomem		*base[ERDT_MMIO_NUM_TYPES];
+	cpumask_var_t		cpu_mask;

Should this be struct cpumask instead? Compare with struct rdt_domain_hdr. When evaluating cpumask_var_t, please read the detailed comments above its definition in include/linux/cpumask_types.h - specifically note the start:
*cpumask_var_t: struct cpumask for stack usage*

Yes, let me switch to struct cpumask instead.

Quoted 7 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 172

Quoted 2 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +2 -0
+	int			max_rmid;
+	struct list_head	list;

Could you please rename this to be "node" or "entry" to make it obvious that this is an entry of a list? This is not consistent in resctrl but really helps when reading the code.

OK, let me rename it to "entry".

thanks,
Chenyu

maintainer
#8 lore raw

Re: [PATCH v5 03/10] x86/resctrl: Parse ACPI ERDT table and save CACD cpumask for RMDD domains

Hi Chenyu,

Quoted 23 lines · Yu C

Hi Chenyu,

Quoted 10 lines · Reinette Chatre

Hi Chenyu,

Quoted 7 lines · Chen Yu

From: Anil S Keshavamurthy <[email protected]>

Parse the ERDT (Enhanced RDT) ACPI table so enhanced RDT features can consume firmware-provided domain information.

The ERDT may contain these sub-tables:

Quoted 9 lines
Quoted 4 lines

   - Resource Management Domain Description Structure (RMDD)
   - CPU Agent Collection Description Structure (CACD)
   - Cache Monitoring Registers for CPU Agents Description Structure (CMRC)

How should "The ERDT may contain these sub-tables" be interpreted here? This sounds like some high level partial description of ERDT that is not specific to this patch.

Quoted 4 lines

I planned to add some background context on ERDT. Let me remove unrelated content and change the sentence to:
Parse the RMDD subtables within the ERDT ACPI table and their nested CACD entries to construct per-domain CPU masks.

Thank you. Making this patch specific will also help to identify unrelated changes, for example the introduction of ERDT_MMIO_CMRC_BASE that would be more appropriate for the patch that uses it.

...

Quoted 11 lines
Quoted 10 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 26

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +8 -0
+
+#include <asm/apic.h>
+
+#include "internal.h"
+
+static LIST_HEAD(domain_info_list);
+
+static bool __erdt_enabled;

Is double underscore needed?

Quoted 2 lines

The double underscore was used to prevent name collisions with the existing erdt_enabled(). Let me switch it to a single underscore instead.

You say "existing" but I am not able to find erdt_enabled() in this series. Is it something that will be needed later?

Quoted 10 lines
Quoted 4 lines

Could you please add a comment above the variable to describe what it means when "erdt is enabled"?

OK, let me add this:

/*
 * Set when the ERDT ACPI table has been successfully parsed and at least
 * one valid RMDD domain with a CACD cpumask exists. Cleared on teardown.

To make this easier to understand, please aim to have comments that do not describe code verbatim but instead what it means. Consider, for example, something like:
"True when the ERDT ACPI table describes at least one domain with at least one CPU."

Quoted 20 lines
Quoted 10 lines
Quoted 10 lines

+        d = container_of(pos, struct erdt_domain_info, list); +        list_del(pos);
+        cleanup_one_domain(d);
+    }
+    __erdt_enabled = false;
+    valid_subtbl_mask = 0;

arch/x86/kernel/cpu/resctrl/erdt.c new file +4 -0
+}
+
+static __init int enumerate_erdt_table(struct acpi_table_header *table_hdr)
+{
Quoted 9 lines
Quoted 6 lines

+    struct acpi_table_erdt *erdt = (struct acpi_table_erdt *)table_hdr; +    struct acpi_subtbl_hdr_16 *subtbl;
+    void *table_end;
+
+    if (erdt->header.revision != ERDT_VALID_VERSION) {
+        pr_info("Unsupported ERDT table revision %d\n", erdt->header.revision);

Would it be helpful to print what the ERDT table revision is to help diagnostics?

Quoted 4 lines

Yes, it will print the revision, do you mean print the expected revision?

pr_info("Unsupported ERDT table revision %d (expected %d)\n", erdt->header.revision, ERDT_VALID_VERSION);

oh, apologies, I misread what you had originally. Even so, your new proposal looks good to me.
As a sidenote, please use most accurate conversion when printing, for example %u for unsigned instead of %d.

Reinette

author
#9 lore raw

Re: [PATCH v5 03/10] x86/resctrl: Parse ACPI ERDT table and save CACD cpumask for RMDD domains

Hi Reinette,

Quoted 2 lines · Reinette Chatre

Hi Chenyu,

[ ... ]

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines

The ERDT may contain these sub-tables:

   - Resource Management Domain Description Structure (RMDD)
   - CPU Agent Collection Description Structure (CACD)
   - Cache Monitoring Registers for CPU Agents Description Structure (CMRC)

How should "The ERDT may contain these sub-tables" be interpreted here? This sounds like some high level partial description of ERDT that is not specific to this patch.

Quoted 9 lines
Quoted 4 lines

I planned to add some background context on ERDT. Let me remove unrelated content and change the sentence to:
Parse the RMDD subtables within the ERDT ACPI table and their nested CACD entries to construct per-domain CPU masks.

Thank you. Making this patch specific will also help to identify unrelated changes, for example the introduction of ERDT_MMIO_CMRC_BASE that would be more appropriate for the patch that uses it.

OK, will also move the ERDT_MMIO_CMRC_BASE to the patch where it is used.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

arch/x86/kernel/cpu/resctrl/erdt.c line 26

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/erdt.c new file +8 -0
+
+#include <asm/apic.h>
+
+#include "internal.h"
+
+static LIST_HEAD(domain_info_list);
+
+static bool __erdt_enabled;

Is double underscore needed?

Quoted 6 lines
Quoted 2 lines

The double underscore was used to prevent name collisions with the existing erdt_enabled(). Let me switch it to a single underscore instead.

You say "existing" but I am not able to find erdt_enabled() in this series. Is it something that will be needed later?

Ah, erdt_enabled() was removed in current version. Let me change it to static bool erdt_enabled;

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

Could you please add a comment above the variable to describe what it means when "erdt is enabled"?

OK, let me add this:

Quoted 9 lines
Quoted 3 lines

/*
 * Set when the ERDT ACPI table has been successfully parsed and at least
 * one valid RMDD domain with a CACD cpumask exists. Cleared on teardown.

To make this easier to understand, please aim to have comments that do not describe code verbatim but instead what it means. Consider, for example, something like:
"True when the ERDT ACPI table describes at least one domain with at least one CPU."

Got it.

Quoted 22 lines
Quoted 22 lines
Quoted 10 lines
Quoted 10 lines

+        d = container_of(pos, struct erdt_domain_info, list); +        list_del(pos);
+        cleanup_one_domain(d);
+    }
+    __erdt_enabled = false;
+    valid_subtbl_mask = 0;

arch/x86/kernel/cpu/resctrl/erdt.c new file +4 -0
+}
+
+static __init int enumerate_erdt_table(struct acpi_table_header *table_hdr)
+{
Quoted 9 lines
Quoted 6 lines

+    struct acpi_table_erdt *erdt = (struct acpi_table_erdt *)table_hdr; +    struct acpi_subtbl_hdr_16 *subtbl;
+    void *table_end;
+
+    if (erdt->header.revision != ERDT_VALID_VERSION) {
+        pr_info("Unsupported ERDT table revision %d\n", erdt->header.revision);

Would it be helpful to print what the ERDT table revision is to help diagnostics?

Yes, it will print the revision, do you mean print the expected revision?

Quoted 8 lines
Quoted 2 lines

pr_info("Unsupported ERDT table revision %d (expected %d)\n",
        erdt->header.revision, ERDT_VALID_VERSION);

oh, apologies, I misread what you had originally. Even so, your new proposal looks good to me.
As a sidenote, please use most accurate conversion when printing, for example %u for unsigned instead of %d.

OK, will do.

thanks,
Chenyu

author
#10 lore raw

PATCH v5 4/10 x86/resctrl: Attach ACPI ERDT information to L3 mon domain on CPU online

In reply to Chen Yu

After the rdt_hw_l3_mon_domain has been created during CPU online, attach the pre-parsed ACPI ERDT table information to the rdt_hw_l3_mon_domain to facilitate monitor data read via the ERDT and its sub-tables information.

During attachment, a sanity check is triggered to verify whether the CPU mask reported by silicon (CPUID leaf 4) matches the information exposed by firmware (CACD table). If inconsistent, this CPU will be cleared from the legitimate domain mask.

---
v4->v5:
new patch.

---
 arch/x86/kernel/cpu/resctrl/core.c     | 17 +++++++++
 arch/x86/kernel/cpu/resctrl/erdt.c     | 49 ++++++++++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/internal.h |  4 +++
 3 files changed, 70 insertions(+)
arch/x86/kernel/cpu/resctrl/core.c +17 -0
index 23925bcd71d7..2e95586ebe45 100644
@@ -549,6 +549,19 @@ static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct
 	d->ci_id = ci->id;
 	cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
 
+	/*
+	 * Verify whether the CPU domain information matches the ACPI data.
+	 * Skip adding the newly created domain to the list if there is a mismatch.
+	 * ACPI information should be assigned to the domain prior to its insertion
+	 * into the list, in case others might iterate the list in parallel.
+	 */
+	if (erdt_l3_mon_domain_setup(cpu, &d->hdr)) {
+		pr_warn("CPU%d has inconsistent domain information, do not add this new domain\n", cpu);
+		cpumask_clear_cpu(cpu, &d->hdr.cpu_mask);
+		l3_mon_domain_free(hw_dom);
+		return;
+	}
+
 	arch_mon_domain_online(r, d);
 
 	if (l3_mon_domain_mbm_alloc(r->mon.num_rmid, hw_dom)) {
@@ -591,6 +604,10 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 			resctrl_arch_mbm_cntr_assign_set_one(r);
 		if (!hdr)
 			l3_mon_domain_setup(cpu, id, r, add_pos);
+		else if (erdt_l3_mon_domain_setup(cpu, hdr)) {
+			pr_warn("CPU%d has inconsistent domain information, remove it from the domain\n", cpu);
+			cpumask_clear_cpu(cpu, &hdr->cpu_mask);
+		}
 		break;
 	case RDT_RESOURCE_PERF_PKG:
 		if (!hdr)
arch/x86/kernel/cpu/resctrl/erdt.c +49 -0

55 lines not on this page · raw message

arch/x86/kernel/cpu/resctrl/internal.h +4 -0
index 299d7222f693..7d9100b7648f 100644
@@ -92,14 +92,18 @@ struct rdt_hw_ctrl_domain {
  * @arch_mbm_states:	Per-event pointer to the MBM event's saved state.
  *			An MBM event's state is an array of struct arch_mbm_state
  *			indexed by RMID on x86.
+ * @d_info:		ERDT table information of this domain(read-only)
  *
  * Members of this structure are accessed via helpers that provide abstraction.
  */
 struct rdt_hw_l3_mon_domain {
 	struct rdt_l3_mon_domain	d_resctrl;
 	struct arch_mbm_state		*arch_mbm_states[QOS_NUM_L3_MBM_EVENTS];
+	struct erdt_domain_info		*d_info;
 };
 
+int erdt_l3_mon_domain_setup(int cpu, struct rdt_domain_hdr *hdr);
+
 static inline struct rdt_hw_ctrl_domain *resctrl_to_arch_ctrl_dom(struct rdt_ctrl_domain *r)
 {
 	return container_of(r, struct rdt_hw_ctrl_domain, d_resctrl);
Signature · 1 line
2.45.2
maintainer
#11 lore raw

Re: [PATCH v5 04/10] x86/resctrl: Attach ACPI ERDT information to L3 mon domain on CPU online

Hi Chenyu,

Quoted 4 lines · Chen Yu

After the rdt_hw_l3_mon_domain has been created during CPU online, attach the pre-parsed ACPI ERDT table information to the rdt_hw_l3_mon_domain to facilitate monitor data read via the ERDT and its sub-tables information.

Please follow tip guidance for changelog by starting with context before jumping to what the patch does.

Quoted 2 lines

During attachment, a sanity check is triggered to verify whether the

Please write in imperative tone.

Quoted 22 lines

CPU mask reported by silicon (CPUID leaf 4) matches the information exposed by firmware (CACD table). If inconsistent, this CPU will be cleared from the legitimate domain mask.

---
v4->v5:
new patch.

---
 arch/x86/kernel/cpu/resctrl/core.c     | 17 +++++++++
 arch/x86/kernel/cpu/resctrl/erdt.c     | 49 ++++++++++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/internal.h |  4 +++
 3 files changed, 70 insertions(+)
arch/x86/kernel/cpu/resctrl/core.c +0 -0
index 23925bcd71d7..2e95586ebe45 100644
@@ -549,6 +549,19 @@ static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct
 	d->ci_id = ci->id;

arch/x86/kernel/cpu/resctrl/core.c static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct · line 559

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c +8 -0
 	cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
 
+	/*
+	 * Verify whether the CPU domain information matches the ACPI data.
+	 * Skip adding the newly created domain to the list if there is a mismatch.
+	 * ACPI information should be assigned to the domain prior to its insertion
+	 * into the list, in case others might iterate the list in parallel.
+	 */
+	if (erdt_l3_mon_domain_setup(cpu, &d->hdr)) {
+		pr_warn("CPU%d has inconsistent domain information, do not add this new domain\n", cpu);

erdt_l3_mon_domain_setup() already contains a pr_warn() when there is an error, is this pr_warn() needed?

arch/x86/kernel/cpu/resctrl/core.c static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct · line 563

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/core.c +4 -0
+		cpumask_clear_cpu(cpu, &d->hdr.cpu_mask);
+		l3_mon_domain_free(hw_dom);
+		return;
+	}

Since the rdt_hw_l3_mon_domain was just allocated and the current CPU is the *only* CPU assigned to it, erdt_l3_mon_domain_setup() should never fail? I do not think any CPU checking is necessary here - when the flow reaches l3_mon_domain_setup() it can just assign the ERDT domain directly, no?

arch/x86/kernel/cpu/resctrl/core.c static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) · line 609

Quoted 11 lines
arch/x86/kernel/cpu/resctrl/core.c +4 -0
+
 	arch_mon_domain_online(r, d);
 
 	if (l3_mon_domain_mbm_alloc(r->mon.num_rmid, hw_dom)) {
@@ -591,6 +604,10 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 			resctrl_arch_mbm_cntr_assign_set_one(r);
 		if (!hdr)
 			l3_mon_domain_setup(cpu, id, r, add_pos);
+		else if (erdt_l3_mon_domain_setup(cpu, hdr)) {
+			pr_warn("CPU%d has inconsistent domain information, remove it from the domain\n", cpu);
+			cpumask_clear_cpu(cpu, &hdr->cpu_mask);

I do not think it is necessary to add the CPU to hdr->cpu_mask for the ERDT test to be performed (more below). It thus looks to me as though this ERDT domain assignment should be split - the error check is only needed if there is a header and if that finds a problem it can just bail early (before adding CPU to hdr->cpu_mask). Something like:

		hdr = resctrl_find_domain(&r->mon_domains, id, &add_pos);
		if (hdr) {
			if (/* This CPU's ERDT domain a NOT subset of hdr->cpu_mask */)
				bail;
			cpumask_set_cpu(cpu, &hdr->cpu_mask);
		}

		switch (r->rid) {
		case RDT_RESOURCE_L3:
			...
			if (!hdr)
				/* l3_mon_domain_setup() initializes hw_dom->d_info without CPU checking */
				l3_mon_domain_setup(cpu, id, r, add_pos);
			
			...
		}
Quoted 16 lines
arch/x86/kernel/cpu/resctrl/core.c +1 -0
+		}
 		break;
 	case RDT_RESOURCE_PERF_PKG:
 		if (!hdr)
arch/x86/kernel/cpu/resctrl/erdt.c +4 -0
index 6405df9be817..6c1df7e43eab 100644
@@ -212,6 +212,55 @@ static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr)
 	return false;
 }
 
+/*
+ * Associate ERDT table information with this domain.
+ */
+int erdt_l3_mon_domain_setup(int cpu, struct rdt_domain_hdr *hdr)

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 228

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c +10 -0
+{
+	struct rdt_hw_l3_mon_domain *hw_dom;
+	struct erdt_domain_info *d;
+	struct list_head *pos;
+
+	if (!__erdt_enabled)
+		return 0;
+
+	/*
+	 * Find the erdt_domain_info that contains this CPU,

Please use entire line length available.

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 231

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/erdt.c +3 -0
+	 * compare erdt_domain_info's cpumask with the cpumask
+	 * exposed by hw_dom (derived from CPUID leaf 4).
+	 * If yes, assign the erdt_domain_info in the hw_dom,

What does "If yes" refer to?

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 234

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/erdt.c +3 -0
+	 * otherwise this CPU should be isolated from resctrl.
+	 * For example, the hw_dom reports CPU{0,1} are in
+	 * l3 domain0, CPU{2,3} belongs to domain1. Meanwhile

Please use consistent terms (l3 -> L3, cpumask -> cpu_mask, etc.)

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 238

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/erdt.c +4 -0
+	 * erdt_domain_info reports that CPU{0,2} are in domain0,
+	 * CPU{1,3} are in domain1. So when it comes to CPU1,
+	 * a mismatch is detected, we should remove CPU1 from
+	 * resctrl.

CPU1 and CPU2?

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 241

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/erdt.c +3 -0
+	 */
+	list_for_each(pos, &domain_info_list) {
+		d = container_of(pos, struct erdt_domain_info, list);

list_for_each_entry()

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 244

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/erdt.c +3 -0
+
+		if (cpumask_test_cpu(cpu, d->cpu_mask)) {
+			if (!cpumask_subset(&hdr->cpu_mask, d->cpu_mask)) {

Consider the above two tests, the first, cpumask_test_cpu() already checks that CPU in parameter belongs to d->cpu_mask, it is thus not required for the same CPU to be a member of hdr->cpu_mask in the cpumask_subset() that follows? The callers currently add the CPU to hdr->cpu_mask before calling erdt_l3_mon_domain_setup() and then remove the CPU on failure ... that thus looks unnecessary? Caller can just add the CPU to hdr->cpu_mask on success?

Quoted 27 lines
arch/x86/kernel/cpu/resctrl/erdt.c +19 -0
+				pr_warn(FW_BUG "Mismatch detected, CPU%d in L3 domain(%*pbl) and CACD domain(%*pbl)\n",
+					cpu, cpumask_pr_args(&hdr->cpu_mask), cpumask_pr_args(d->cpu_mask));
+
+				return -EIO;
+			}
+
+			hw_dom = resctrl_to_arch_mon_dom(container_of(hdr, struct rdt_l3_mon_domain, hdr));
+			/* No mismatch, assign the ERDT information to hw_dom */
+			if (!hw_dom->d_info)
+				hw_dom->d_info = d;
+
+			return 0;
+		}
+	}
+
+	pr_warn(FW_BUG "Cannot find CACD domain for CPU%d\n", cpu);
+	return -ENOENT;
+}
+
 void erdt_exit(void)
 {
 	struct erdt_domain_info *d;
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
index 299d7222f693..7d9100b7648f 100644

arch/x86/kernel/cpu/resctrl/internal.h struct rdt_hw_ctrl_domain { · line 95

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
@@ -92,14 +92,18 @@ struct rdt_hw_ctrl_domain {
  * @arch_mbm_states:	Per-event pointer to the MBM event's saved state.
  *			An MBM event's state is an array of struct arch_mbm_state
  *			indexed by RMID on x86.
+ * @d_info:		ERDT table information of this domain(read-only)

read-only in comments can be made explicit with a const below?

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
  *
  * Members of this structure are accessed via helpers that provide abstraction.
  */
 struct rdt_hw_l3_mon_domain {

arch/x86/kernel/cpu/resctrl/internal.h struct rdt_hw_ctrl_domain { · line 109

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/internal.h +3 -0
 	struct rdt_l3_mon_domain	d_resctrl;
 	struct arch_mbm_state		*arch_mbm_states[QOS_NUM_L3_MBM_EVENTS];
+	struct erdt_domain_info		*d_info;
 };
 
+int erdt_l3_mon_domain_setup(int cpu, struct rdt_domain_hdr *hdr);
+
 static inline struct rdt_hw_ctrl_domain *resctrl_to_arch_ctrl_dom(struct rdt_ctrl_domain *r)
 {
 	return container_of(r, struct rdt_hw_ctrl_domain, d_resctrl);

Reinette

author
#12 lore raw

Re: [PATCH v5 04/10] x86/resctrl: Attach ACPI ERDT information to L3 mon domain on CPU online

Hi Reinette,

Quoted 11 lines · Reinette Chatre

Hi Chenyu,

Quoted 4 lines · Chen Yu

After the rdt_hw_l3_mon_domain has been created during CPU online, attach the pre-parsed ACPI ERDT table information to the rdt_hw_l3_mon_domain to facilitate monitor data read via the ERDT and its sub-tables information.

Please follow tip guidance for changelog by starting with context before jumping to what the patch does.

Okay, will update the changelog and all other patches accordingly.

Quoted 5 lines
Quoted 2 lines

During attachment, a sanity check is triggered to verify whether the

Please write in imperative tone.

OK.

Quoted 4 lines
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/core.c +0 -0
@@ -549,6 +549,19 @@ static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct
  	d->ci_id = ci->id;
  	cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
 
Quoted 12 lines

arch/x86/kernel/cpu/resctrl/core.c static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct · line 559

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/core.c +8 -0
+	/*
+	 * Verify whether the CPU domain information matches the ACPI data.
+	 * Skip adding the newly created domain to the list if there is a mismatch.
+	 * ACPI information should be assigned to the domain prior to its insertion
+	 * into the list, in case others might iterate the list in parallel.
+	 */
+	if (erdt_l3_mon_domain_setup(cpu, &d->hdr)) {
+		pr_warn("CPU%d has inconsistent domain information, do not add this new domain\n", cpu);

erdt_l3_mon_domain_setup() already contains a pr_warn() when there is an error, is this pr_warn() needed?

erdt_l3_mon_domain_setup() was used to detect mismatches, while l3_mon_domain_setup() would take corrective action upon a mismatch (CPU removal) - so they print slightly differently. After switching to your proposed approach below, all mismatch logging can be centralized within erdt.c

Quoted 10 lines

arch/x86/kernel/cpu/resctrl/core.c static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct · line 563

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/core.c +4 -0
+		cpumask_clear_cpu(cpu, &d->hdr.cpu_mask);
+		l3_mon_domain_free(hw_dom);
+		return;
+	}

Since the rdt_hw_l3_mon_domain was just allocated and the current CPU is the *only* CPU assigned to it, erdt_l3_mon_domain_setup() should never fail? I do not think any CPU checking is necessary here - when the flow reaches l3_mon_domain_setup() it can just assign the ERDT domain directly, no?

You are right; this cpumask check is unnecessary for this newly created rdt_hw_l3_mon_domain case. erdt_l3_mon_domain_setup() was originally implemented to handle both the cpumask validation and ERDT domain assignment, and it was intended for use in both the !hdr and hdr paths.
As a result, the !hdr path (for newly allocated rdt_hw_l3_mon_domain) carries unnecessary logic for cpumask mismatch detection.

Quoted 18 lines

arch/x86/kernel/cpu/resctrl/core.c static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) · line 609

Quoted 12 lines
arch/x86/kernel/cpu/resctrl/core.c +4 -0
+
  	arch_mon_domain_online(r, d);
 
  	if (l3_mon_domain_mbm_alloc(r->mon.num_rmid, hw_dom)) {
@@ -591,6 +604,10 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
  			resctrl_arch_mbm_cntr_assign_set_one(r);
  		if (!hdr)
  			l3_mon_domain_setup(cpu, id, r, add_pos);
+		else if (erdt_l3_mon_domain_setup(cpu, hdr)) {
+			pr_warn("CPU%d has inconsistent domain information, remove it from the domain\n", cpu);
+			cpumask_clear_cpu(cpu, &hdr->cpu_mask);

I do not think it is necessary to add the CPU to hdr->cpu_mask for the ERDT test to be performed (more below). It thus looks to me as though this ERDT domain assignment should be split - the error check is only needed if there is a header and if that finds a problem it can just bail early (before adding CPU to hdr->cpu_mask). Something like:

Quoted 17 lines
		hdr = resctrl_find_domain(&r->mon_domains, id, &add_pos);
		if (hdr) {
			if (/* This CPU's ERDT domain a NOT subset of hdr->cpu_mask */)
				bail;
			cpumask_set_cpu(cpu, &hdr->cpu_mask);
		}

		switch (r->rid) {
		case RDT_RESOURCE_L3:
			...
			if (!hdr)
				/* l3_mon_domain_setup() initializes hw_dom->d_info without CPU checking */
				l3_mon_domain_setup(cpu, id, r, add_pos);
			
			...
		}

Yes, this version is clearer. I will switch over to this approach.

Quoted 12 lines
Quoted 12 lines
arch/x86/kernel/cpu/resctrl/core.c +1 -0
+		}
  		break;
  	case RDT_RESOURCE_PERF_PKG:
  		if (!hdr)
arch/x86/kernel/cpu/resctrl/erdt.c +0 -0
index 6405df9be817..6c1df7e43eab 100644
@@ -212,6 +212,55 @@ static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr)
  	return false;
  }
 
Quoted 17 lines
Quoted 4 lines
arch/x86/kernel/cpu/resctrl/erdt.c +4 -0
+/*
+ * Associate ERDT table information with this domain.
+ */
+int erdt_l3_mon_domain_setup(int cpu, struct rdt_domain_hdr *hdr)

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 228

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c +10 -0
+{
+	struct rdt_hw_l3_mon_domain *hw_dom;
+	struct erdt_domain_info *d;
+	struct list_head *pos;
+
+	if (!__erdt_enabled)
+		return 0;
+
+	/*
+	 * Find the erdt_domain_info that contains this CPU,

Please use entire line length available.

OK, will do.

Quoted 6 lines

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 231

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/erdt.c +3 -0
+	 * compare erdt_domain_info's cpumask with the cpumask
+	 * exposed by hw_dom (derived from CPUID leaf 4).
+	 * If yes, assign the erdt_domain_info in the hw_dom,

What does "If yes" refer to?

My intention was to state that the cpumask of hw_dom is a subset of the cpumask exposed by ERDT. Let me revise the expression.

Quoted 6 lines

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 234

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/erdt.c +3 -0
+	 * otherwise this CPU should be isolated from resctrl.
+	 * For example, the hw_dom reports CPU{0,1} are in
+	 * l3 domain0, CPU{2,3} belongs to domain1. Meanwhile

Please use consistent terms (l3 -> L3, cpumask -> cpu_mask, etc.)

OK, will do.

Quoted 7 lines

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 238

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/erdt.c +4 -0
+	 * erdt_domain_info reports that CPU{0,2} are in domain0,
+	 * CPU{1,3} are in domain1. So when it comes to CPU1,
+	 * a mismatch is detected, we should remove CPU1 from
+	 * resctrl.

CPU1 and CPU2?

I think it should be CPU1 and CPU3 that will be removed from resctrl - because the first CPU of the hw_dom->cpumask will not find a mismatch per your description previously.

Quoted 6 lines

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 241

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/erdt.c +3 -0
+	 */
+	list_for_each(pos, &domain_info_list) {
+		d = container_of(pos, struct erdt_domain_info, list);

list_for_each_entry()

OK, will do.

Quoted 11 lines

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 244

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/erdt.c +3 -0
+
+		if (cpumask_test_cpu(cpu, d->cpu_mask)) {
+			if (!cpumask_subset(&hdr->cpu_mask, d->cpu_mask)) {

Consider the above two tests, the first, cpumask_test_cpu() already checks that CPU in parameter belongs to d->cpu_mask, it is thus not required for the same CPU to be a member of hdr->cpu_mask in the cpumask_subset() that follows? The callers currently add the CPU to hdr->cpu_mask before calling erdt_l3_mon_domain_setup() and then remove the CPU on failure ... that thus looks unnecessary? Caller can just add the CPU to hdr->cpu_mask on success?

Yes, let me replace this "set then clear on error" by "checking cpu sanity before setting it to hdr->cpu_mask"

Quoted 10 lines

arch/x86/kernel/cpu/resctrl/internal.h struct rdt_hw_ctrl_domain { · line 95

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
@@ -92,14 +92,18 @@ struct rdt_hw_ctrl_domain {
   * @arch_mbm_states:	Per-event pointer to the MBM event's saved state.
   *			An MBM event's state is an array of struct arch_mbm_state
   *			indexed by RMID on x86.
+ * @d_info:		ERDT table information of this domain(read-only)

read-only in comments can be made explicit with a const below?

Ok, will change it to
const struct erdt_domain_info *d_info;
as after hw_dom->d_info = d, the content of the pointed-to erdt_domain_info is never modified through d_info.

thanks,
Chenyu

maintainer
#13 lore raw

Re: [PATCH v5 04/10] x86/resctrl: Attach ACPI ERDT information to L3 mon domain on CPU online

Hi Chenyu,

Quoted 2 lines · Yu C
Quoted 1 line · Reinette Chatre

On 7/1/26 6:45 AM, Chen Yu wrote:

...

Quoted 14 lines
Quoted 6 lines
Quoted 3 lines

+     * otherwise this CPU should be isolated from resctrl. +     * For example, the hw_dom reports CPU{0,1} are in +     * l3 domain0, CPU{2,3} belongs to domain1. Meanwhile

Please use consistent terms (l3 -> L3, cpumask -> cpu_mask, etc.)

OK, will do.

Quoted 5 lines
Quoted 4 lines

+     * erdt_domain_info reports that CPU{0,2} are in domain0, +     * CPU{1,3} are in domain1. So when it comes to CPU1, +     * a mismatch is detected, we should remove CPU1 from +     * resctrl.

Quoted 6 lines
Quoted 2 lines

CPU1 and CPU2?

I think it should be CPU1 and CPU3 that will be removed from resctrl - because the first CPU of the hw_dom->cpumask will not find a mismatch per your description previously.

I expected that only the CPUs CPUID and ACPI disagree on will be removed? In the example CPUID and ACPI agree on which domain CPU0 and CPU3 belongs to while they disagree on which domain CPU1 and CPU2 belongs to.

Reinette

author
#14 lore raw

Re: [PATCH v5 04/10] x86/resctrl: Attach ACPI ERDT information to L3 mon domain on CPU online

Hi Reinette,

Quoted 1 line · Reinette Chatre

Hi Chenyu,

[ ... ]

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

+     * erdt_domain_info reports that CPU{0,2} are in domain0, +     * CPU{1,3} are in domain1. So when it comes to CPU1, +     * a mismatch is detected, we should remove CPU1 from +     * resctrl.

CPU1 and CPU2?

Quoted 8 lines
Quoted 3 lines

I think it should be CPU1 and CPU3 that will be removed from resctrl - because the first CPU of the hw_dom->cpumask will not find a mismatch per your description previously.

I expected that only the CPUs CPUID and ACPI disagree on will be removed? In the example CPUID and ACPI agree on which domain CPU0 and CPU3 belongs to while they disagree on which domain CPU1 and CPU2 belongs to.

Got it. After switching to "check-before-add-to-cpumask", CPU1 and CPU2 will be filtered out. I'll adjust the code.

thanks,
Chenyu

author
#15 lore raw

PATCH v5 5/10 x86/resctrl: Parse ACPI CMRC table

In reply to Chen Yu

The CMRC (Cache Monitoring Registers for CPU Agents Description) sub-table of ERDT describes the MMIO registers used to read cache monitoring counters (e.g. LLC occupancy) for an RMD.

Parse each CMRC sub-table, ioremap its register window, and save the CMRC pointer in the corresponding ERDT domain entry so that later monitoring code can read the counters via MMIO.

---
v4->v5:
No change.

---
 arch/x86/kernel/cpu/resctrl/erdt.c     | 44 ++++++++++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/internal.h |  1 +
 2 files changed, 45 insertions(+)
arch/x86/kernel/cpu/resctrl/erdt.c +44 -0
index 6c1df7e43eab..a5754d64fcc1 100644
@@ -26,6 +26,7 @@ static LIST_HEAD(domain_info_list);
 static bool __erdt_enabled;
 
 #define ERDT_VALID_VERSION		1
+#define CMRC_SUPPORTED_INDEX_FN		1
 #define RMDD_FLAG_CPU_L3_DOMAIN		BIT(0)
 
 /* Bitmask of valid sub-tables found in the first RMDD, used to ensure all RMDDs match. */
@@ -79,6 +80,7 @@ static void cleanup_one_domain(struct erdt_domain_info *d)
 {
 	erdt_iounmap_domain(d);
 	free_cpumask_var(d->cpu_mask);
+	kfree(d->cmrc);
 	kfree(d);
 }
 
@@ -112,6 +114,41 @@ static __init int cacd_init(struct acpi_subtbl_hdr_16 *subtbl,
 	return 0;
 }
 
+static __init int cmrc_init(struct acpi_subtbl_hdr_16 *subtbl,
+			    struct erdt_domain_info *domain_info)
+{
+	struct acpi_erdt_cmrc *cmrc = (struct acpi_erdt_cmrc *)subtbl;
+
+	if (subtbl->length < sizeof(*cmrc)) {
+		pr_warn(FW_BUG "Truncated CMRC subtable\n");
+		return -EIO;
+	}
+
+	if (cmrc->index_fn != CMRC_SUPPORTED_INDEX_FN) {
+		pr_info("Unsupported CMRC index function %d\n", cmrc->index_fn);
+		return -EIO;
+	}
+
+	if (!cmrc->clump_size) {
+		pr_warn(FW_BUG "CMRC clump_size is zero\n");
+		return -EIO;
+	}
+
+	domain_info->base[ERDT_MMIO_CMRC_BASE] =
+		erdt_ioremap(cmrc->cmt_reg_base, cmrc->cmt_reg_size, "CMRC base");
+	if (!domain_info->base[ERDT_MMIO_CMRC_BASE])
+		return -EIO;
+
+	domain_info->cmrc = kmemdup(cmrc, subtbl->length, GFP_KERNEL);
+	if (!domain_info->cmrc) {
+		iounmap(domain_info->base[ERDT_MMIO_CMRC_BASE]);
+		domain_info->base[ERDT_MMIO_CMRC_BASE] = NULL;
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+
 static inline struct acpi_subtbl_hdr_16 *rmdd_subtbl(struct acpi_erdt_rmdd *rmdd)
 {
 	return (void *)rmdd + sizeof(*rmdd);
@@ -178,6 +215,13 @@ static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr)
 				goto cleanup;
 
 			subtbl_mask |= BIT(ACPI_ERDT_TYPE_CACD);
+			break;
+		case ACPI_ERDT_TYPE_CMRC:
+			/* TBD: Only 1 CMRR per domain is allowed? */
+			if (!(subtbl_mask & BIT(ACPI_ERDT_TYPE_CMRC)) &&
+			    !cmrc_init(subtbl, domain_info))
+				subtbl_mask |= BIT(ACPI_ERDT_TYPE_CMRC);
+
 			break;
 		default:
 			break;
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
index 7d9100b7648f..3dab05c38003 100644
@@ -36,6 +36,7 @@ enum erdt_mmio_type {
 
 struct erdt_domain_info {
 	void __iomem		*base[ERDT_MMIO_NUM_TYPES];
+	struct acpi_erdt_cmrc	*cmrc;
 	cpumask_var_t		cpu_mask;
 	int			max_rmid;
 	struct list_head	list;
Signature · 1 line
2.45.2
maintainer
#16 lore raw

Re: [PATCH v5 05/10] x86/resctrl: Parse ACPI CMRC table

Hi Chenyu,

Quoted 3 lines · Chen Yu

The CMRC (Cache Monitoring Registers for CPU Agents Description) sub-table of ERDT describes the MMIO registers used to read cache monitoring counters (e.g. LLC occupancy) for an RMD.

Please use the entire line length available (please check all changelogs). Some context for this request:
https://lore.kernel.org/lkml/20250916105447.GCaMlB976WLxHHeNMD@fat_crate.local/

Quoted 4 lines

Parse each CMRC sub-table, ioremap its register window, and save the CMRC pointer in the corresponding ERDT domain entry so that later monitoring code can read the counters via MMIO.

Is the "save the CMRC pointer ..." referring to how resctrl makes a copy of the CMRC table? If so, that is very different from "saving a pointer".

Quoted 34 lines

---
v4->v5:
No change.

---
 arch/x86/kernel/cpu/resctrl/erdt.c     | 44 ++++++++++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/internal.h |  1 +
 2 files changed, 45 insertions(+)
arch/x86/kernel/cpu/resctrl/erdt.c +2 -0
index 6c1df7e43eab..a5754d64fcc1 100644
@@ -26,6 +26,7 @@ static LIST_HEAD(domain_info_list);
 static bool __erdt_enabled;
 
 #define ERDT_VALID_VERSION		1
+#define CMRC_SUPPORTED_INDEX_FN		1
 #define RMDD_FLAG_CPU_L3_DOMAIN		BIT(0)
 
 /* Bitmask of valid sub-tables found in the first RMDD, used to ensure all RMDDs match. */
@@ -79,6 +80,7 @@ static void cleanup_one_domain(struct erdt_domain_info *d)
 {
 	erdt_iounmap_domain(d);
 	free_cpumask_var(d->cpu_mask);
+	kfree(d->cmrc);
 	kfree(d);
 }
 

arch/x86/kernel/cpu/resctrl/erdt.c static __init int cacd_init(struct acpi_subtbl_hdr_16 *subtbl, · line 122

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/erdt.c +6 -0
@@ -112,6 +114,41 @@ static __init int cacd_init(struct acpi_subtbl_hdr_16 *subtbl,
 	return 0;
 }
 
+static __init int cmrc_init(struct acpi_subtbl_hdr_16 *subtbl,
+			    struct erdt_domain_info *domain_info)
+{
+	struct acpi_erdt_cmrc *cmrc = (struct acpi_erdt_cmrc *)subtbl;
+
+	if (subtbl->length < sizeof(*cmrc)) {

I think it will be helpful if the ERDT parsing code uses consistent patterns. Here, for example, now that a pointer of accurate type is available via cmrc, can cmrc be referenced instead to be consistent with cacd_init()?

Quoted 33 lines
arch/x86/kernel/cpu/resctrl/erdt.c +29 -0
+		pr_warn(FW_BUG "Truncated CMRC subtable\n");
+		return -EIO;
+	}
+
+	if (cmrc->index_fn != CMRC_SUPPORTED_INDEX_FN) {
+		pr_info("Unsupported CMRC index function %d\n", cmrc->index_fn);
+		return -EIO;
+	}
+
+	if (!cmrc->clump_size) {
+		pr_warn(FW_BUG "CMRC clump_size is zero\n");
+		return -EIO;
+	}
+
+	domain_info->base[ERDT_MMIO_CMRC_BASE] =
+		erdt_ioremap(cmrc->cmt_reg_base, cmrc->cmt_reg_size, "CMRC base");
+	if (!domain_info->base[ERDT_MMIO_CMRC_BASE])
+		return -EIO;
+
+	domain_info->cmrc = kmemdup(cmrc, subtbl->length, GFP_KERNEL);
+	if (!domain_info->cmrc) {
+		iounmap(domain_info->base[ERDT_MMIO_CMRC_BASE]);
+		domain_info->base[ERDT_MMIO_CMRC_BASE] = NULL;
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+
 static inline struct acpi_subtbl_hdr_16 *rmdd_subtbl(struct acpi_erdt_rmdd *rmdd)
 {
 	return (void *)rmdd + sizeof(*rmdd);

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 220

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/erdt.c +3 -0
@@ -178,6 +215,13 @@ static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr)
 				goto cleanup;
 
 			subtbl_mask |= BIT(ACPI_ERDT_TYPE_CACD);
+			break;
+		case ACPI_ERDT_TYPE_CMRC:
+			/* TBD: Only 1 CMRR per domain is allowed? */

Who are you asking this question to?

Quoted 12 lines
arch/x86/kernel/cpu/resctrl/erdt.c +4 -0
+			if (!(subtbl_mask & BIT(ACPI_ERDT_TYPE_CMRC)) &&
+			    !cmrc_init(subtbl, domain_info))
+				subtbl_mask |= BIT(ACPI_ERDT_TYPE_CMRC);
+
 			break;
 		default:
 			break;
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
index 7d9100b7648f..3dab05c38003 100644

arch/x86/kernel/cpu/resctrl/internal.h enum erdt_mmio_type { · line 42

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
@@ -36,6 +36,7 @@ enum erdt_mmio_type {
 
 struct erdt_domain_info {
 	void __iomem		*base[ERDT_MMIO_NUM_TYPES];
+	struct acpi_erdt_cmrc	*cmrc;
 	cpumask_var_t		cpu_mask;
 	int			max_rmid;
 	struct list_head	list;

Reinette

author
#17 lore raw

Re: [PATCH v5 05/10] x86/resctrl: Parse ACPI CMRC table

Hi Reinette,

Quoted 11 lines · Reinette Chatre

Hi Chenyu,

Quoted 3 lines · Chen Yu

The CMRC (Cache Monitoring Registers for CPU Agents Description) sub-table of ERDT describes the MMIO registers used to read cache monitoring counters (e.g. LLC occupancy) for an RMD.

Please use the entire line length available (please check all changelogs). Some context for this request:
https://lore.kernel.org/lkml/20250916105447.GCaMlB976WLxHHeNMD@fat_crate.local/

OK, will use 80 characters per line.

Quoted 9 lines
Quoted 4 lines

Parse each CMRC sub-table, ioremap its register window, and save the CMRC pointer in the corresponding ERDT domain entry so that later monitoring code can read the counters via MMIO.

Is the "save the CMRC pointer ..." referring to how resctrl makes a copy of the CMRC table? If so, that is very different from "saving a pointer".

It is a copy of the original table content. Let me adjust this description.

[ ... ]

Quoted 11 lines

arch/x86/kernel/cpu/resctrl/erdt.c static __init int cacd_init(struct acpi_subtbl_hdr_16 *subtbl, · line 122

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/erdt.c +6 -0
+static __init int cmrc_init(struct acpi_subtbl_hdr_16 *subtbl,
+			    struct erdt_domain_info *domain_info)
+{
+	struct acpi_erdt_cmrc *cmrc = (struct acpi_erdt_cmrc *)subtbl;
+
+	if (subtbl->length < sizeof(*cmrc)) {

I think it will be helpful if the ERDT parsing code uses consistent patterns. Here, for example, now that a pointer of accurate type is available via cmrc, can cmrc be referenced instead to be consistent with cacd_init()?

OK, I will use cmrc->length to keep consistency.

Quoted 7 lines
Quoted 7 lines
arch/x86/kernel/cpu/resctrl/erdt.c +1 -0
+
  static inline struct acpi_subtbl_hdr_16 *rmdd_subtbl(struct acpi_erdt_rmdd *rmdd)
  {
  	return (void *)rmdd + sizeof(*rmdd);
@@ -178,6 +215,13 @@ static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr)
  				goto cleanup;
 
Quoted 7 lines

arch/x86/kernel/cpu/resctrl/erdt.c static __init bool parse_rmdd_entry(struct acpi_subtbl_hdr_16 *rmdd_hdr) · line 220

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/erdt.c +3 -0
  			subtbl_mask |= BIT(ACPI_ERDT_TYPE_CACD);
+			break;
+		case ACPI_ERDT_TYPE_CMRC:
+			/* TBD: Only 1 CMRR per domain is allowed? */

Who are you asking this question to?

The current code supports 1 CMRC per domain. The spec says there are "at least one instance of this structure for each RDT domain" But it is unclear to me how to route to different CMRC. I'll check with the arch team to figure out.

thanks,
Chenyu

author
#18 lore raw

PATCH v5 6/10 x86/resctrl: Replace "msr" in monitoring data identifiers

In reply to Chen Yu

Monitoring counter values are currently obtained via MSR and many identifiers involved in processing these values contain "msr" as part of their names, for example "prev_msr" and "msr_val".

The ERDT ACPI table describes MMIO registers for monitoring data access. Rename "msr"-based identifiers to be interface-agnostic to support their upcoming use for MMIO-read values.

No functional change.

---
v4->v5:
Revise commit log per Reinette's suggestion.

---
 arch/x86/kernel/cpu/resctrl/internal.h |  8 +++----
 arch/x86/kernel/cpu/resctrl/monitor.c  | 30 +++++++++++++-------------
 2 files changed, 19 insertions(+), 19 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +4 -4
index 3dab05c38003..6eb0fdea6b63 100644
@@ -52,13 +52,13 @@ struct erdt_domain_info {
 /**
  * struct arch_mbm_state - values used to compute resctrl_arch_rmid_read()s
  *			   return value.
- * @chunks:	Total data moved (multiply by rdt_group.mon_scale to get bytes)
- * @prev_msr:	Value of IA32_QM_CTR last time it was read for the RMID used to
- *		find this struct.
+ * @chunks:		Total data moved (multiply by rdt_group.mon_scale to get bytes)
+ * @prev_mon_val:	Previous monitor counter value for the RMID used to
+ *			find this struct.
  */
 struct arch_mbm_state {
 	u64	chunks;
-	u64	prev_msr;
+	u64	prev_mon_val;
 };
 
 /* Setting bit 0 in L3_QOS_EXT_CFG enables the ABMC feature. */
arch/x86/kernel/cpu/resctrl/monitor.c +15 -15
index 03ee6102ab07..893080e6fb9e 100644
@@ -187,7 +187,7 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_l3_mon_domain *d
 
 		prmid = logical_rmid_to_physical_rmid(cpu, rmid);
 		/* Record any initial, non-zero count value. */
-		__rmid_read_phys(prmid, eventid, &am->prev_msr);
+		__rmid_read_phys(prmid, eventid, &am->prev_mon_val);
 	}
 }
 
@@ -210,16 +210,16 @@ void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_l3_mon_domai
 	}
 }
 
-static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width)
+static u64 mbm_overflow_count(u64 prev_val, u64 cur_val, unsigned int width)
 {
 	u64 shift = 64 - width, chunks;
 
-	chunks = (cur_msr << shift) - (prev_msr << shift);
+	chunks = (cur_val << shift) - (prev_val << shift);
 	return chunks >> shift;
 }
 
 static u64 get_corrected_val(struct rdt_resource *r, struct rdt_l3_mon_domain *d,
-			     u32 rmid, enum resctrl_event_id eventid, u64 msr_val)
+			     u32 rmid, enum resctrl_event_id eventid, u64 mon_val)
 {
 	struct rdt_hw_l3_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
@@ -228,12 +228,12 @@ static u64 get_corrected_val(struct rdt_resource *r, struct rdt_l3_mon_domain *d
 
 	am = get_arch_mbm_state(hw_dom, rmid, eventid);
 	if (am) {
-		am->chunks += mbm_overflow_count(am->prev_msr, msr_val,
+		am->chunks += mbm_overflow_count(am->prev_mon_val, mon_val,
 						 hw_res->mbm_width);
 		chunks = get_corrected_mbm_count(rmid, am->chunks);
-		am->prev_msr = msr_val;
+		am->prev_mon_val = mon_val;
 	} else {
-		chunks = msr_val;
+		chunks = mon_val;
 	}
 
 	return chunks * hw_res->mon_scale;
@@ -246,7 +246,7 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 	struct rdt_hw_l3_mon_domain *hw_dom;
 	struct rdt_l3_mon_domain *d;
 	struct arch_mbm_state *am;
-	u64 msr_val;
+	u64 mon_val;
 	u32 prmid;
 	int cpu;
 	int ret;
@@ -263,14 +263,14 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 	hw_dom = resctrl_to_arch_mon_dom(d);
 	cpu = cpumask_any(&hdr->cpu_mask);
 	prmid = logical_rmid_to_physical_rmid(cpu, rmid);
-	ret = __rmid_read_phys(prmid, eventid, &msr_val);
+	ret = __rmid_read_phys(prmid, eventid, &mon_val);
 
 	if (!ret) {
-		*val = get_corrected_val(r, d, rmid, eventid, msr_val);
+		*val = get_corrected_val(r, d, rmid, eventid, mon_val);
 	} else if (ret == -EINVAL) {
 		am = get_arch_mbm_state(hw_dom, rmid, eventid);
 		if (am)
-			am->prev_msr = 0;
+			am->prev_mon_val = 0;
 	}
 
 	return ret;
@@ -325,7 +325,7 @@ void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d
 		memset(am, 0, sizeof(*am));
 
 		/* Record any initial, non-zero count value. */
-		__cntr_id_read(cntr_id, &am->prev_msr);
+		__cntr_id_read(cntr_id, &am->prev_mon_val);
 	}
 }
 
@@ -333,14 +333,14 @@ int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_l3_mon_domain *d,
 			   u32 unused, u32 rmid, int cntr_id,
 			   enum resctrl_event_id eventid, u64 *val)
 {
-	u64 msr_val;
+	u64 mon_val;
 	int ret;
 
-	ret = __cntr_id_read(cntr_id, &msr_val);
+	ret = __cntr_id_read(cntr_id, &mon_val);
 	if (ret)
 		return ret;
 
-	*val = get_corrected_val(r, d, rmid, eventid, msr_val);
+	*val = get_corrected_val(r, d, rmid, eventid, mon_val);
 
 	return 0;
 }
Signature · 1 line
2.45.2
maintainer
#19 lore raw

Re: [PATCH v5 06/10] x86/resctrl: Replace "msr" in monitoring data identifiers

Hi Chenyu,

Quoted 7 lines · Chen Yu

Monitoring counter values are currently obtained via MSR and many identifiers involved in processing these values contain "msr" as part of their names, for example "prev_msr" and "msr_val".

The ERDT ACPI table describes MMIO registers for monitoring data access. Rename "msr"-based identifiers to be interface-agnostic to support their upcoming use for MMIO-read values.

Are any of the identifiers renamed by this patch actually used by the MMIO reading code introduced by this series?

Quoted 7 lines

No functional change.

---

...

arch/x86/kernel/cpu/resctrl/internal.h struct erdt_domain_info { · line 55

Quoted 12 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -3
index 3dab05c38003..6eb0fdea6b63 100644
@@ -52,13 +52,13 @@ struct erdt_domain_info {
 /**
  * struct arch_mbm_state - values used to compute resctrl_arch_rmid_read()s
  *			   return value.
- * @chunks:	Total data moved (multiply by rdt_group.mon_scale to get bytes)
- * @prev_msr:	Value of IA32_QM_CTR last time it was read for the RMID used to
- *		find this struct.
+ * @chunks:		Total data moved (multiply by rdt_group.mon_scale to get bytes)

One renaming that does seem warranted is mon_scale that is a per-resource value that currently claims to "cqm counter * mon_scale = occupancy in bytes" but this implementation does not use it and instead uses a value private to ERDT.

Reinette

author
#20 lore raw

PATCH v5 7/10 x86/resctrl: Refactor the monitor read function

In reply to Chen Yu

Split the monitor read helper into an L3 read path and an AET (Intel Application Energy Telemetry) read path. This makes the two distinct monitoring sources easier to extend independently and prepares the L3 path for ERDT-based MMIO reads added in a later patch.

No functional change.

---
v4->v5:
No change.

---
 arch/x86/kernel/cpu/resctrl/monitor.c | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)
arch/x86/kernel/cpu/resctrl/monitor.c +19 -8
index 893080e6fb9e..f4f4c9015ceb 100644
@@ -239,9 +239,9 @@ static u64 get_corrected_val(struct rdt_resource *r, struct rdt_l3_mon_domain *d
 	return chunks * hw_res->mon_scale;
 }
 
-int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
-			   u32 unused, u32 rmid, enum resctrl_event_id eventid,
-			   void *arch_priv, u64 *val, void *ignored)
+static int arch_l3_read_event(struct rdt_domain_hdr *hdr, u32 rmid,
+			      enum resctrl_event_id eventid, u64 *val,
+			      struct rdt_resource *r)
 {
 	struct rdt_hw_l3_mon_domain *hw_dom;
 	struct rdt_l3_mon_domain *d;
@@ -251,11 +251,6 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 	int cpu;
 	int ret;
 
-	resctrl_arch_rmid_read_context_check();
-
-	if (r->rid == RDT_RESOURCE_PERF_PKG)
-		return intel_aet_read_event(hdr->id, rmid, arch_priv, val);
-
 	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		return -EINVAL;
 
@@ -276,6 +271,22 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 	return ret;
 }
 
+int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
+			   u32 unused, u32 rmid, enum resctrl_event_id eventid,
+			   void *arch_priv, u64 *val, void *ignored)
+{
+	resctrl_arch_rmid_read_context_check();
+
+	switch (r->rid) {
+	case RDT_RESOURCE_L3:
+		return arch_l3_read_event(hdr, rmid, eventid, val, r);
+	case RDT_RESOURCE_PERF_PKG:
+		return intel_aet_read_event(hdr->id, rmid, arch_priv, val);
+	default:
+		return -EINVAL;
+	}
+}
+
 static int __cntr_id_read(u32 cntr_id, u64 *val)
 {
 	u64 msr_val;
Signature · 1 line
2.45.2
author
#21 lore raw

PATCH v5 8/10 fs/resctrl: Do not invoke smp_processor_id() in preemptible context

In reply to Chen Yu

From: Tony Luck <[email protected]>

Currently, mon_evt::any_cpu is false for all events associated with RDT_RESOURCE_L3. The MMIO-based CMT will set any_cpu to true in a follow-up patch. This change will trigger a warning when calling smp_processor_id() inside __l3_mon_event_count(), because __l3_mon_event_count() may be invoked from arbitrary CPUs in task context. Since the calling context is preemptible here, smp_processor_id() will emit a debug warning.

To prepare for MMIO-based CMT reads, skip the current CPU lookup when an event's any_cpu flag is set. Events with this flag do not require execution on a specific CPU.

---
v4->v5:
No change.

---
 fs/resctrl/monitor.c | 42 ++++++++++++++++++++++++++++++++----------
 1 file changed, 32 insertions(+), 10 deletions(-)
fs/resctrl/monitor.c +32 -10
index 0e6a389a16bf..6a7c86a72c51 100644
@@ -417,9 +417,37 @@ static void mbm_cntr_free(struct rdt_l3_mon_domain *d, int cntr_id)
 	memset(&d->cntr_cfg[cntr_id], 0, sizeof(*d->cntr_cfg));
 }
 
+/**
+ * cpu_on_correct_domain() - Check if current CPU is in the correct
+ *			      domain for the event.
+ * @rr: The rmid_read structure containing event and domain information.
+ *
+ * Context: Preemptible process context when @rr->evt->any_cpu is set.
+ *          Non-migratable process context (via smp_call_on_cpu()) or
+ *          non-preemptible context (via smp_call_function_any()) when
+ *          the event must be read on a specific CPU.
+ * Return: true if the current CPU can read this event, false otherwise.
+ */
+static bool cpu_on_correct_domain(struct rmid_read *rr)
+{
+	int cpu;
+
+	/* Any CPU is OK for this event */
+	if (rr->evt->any_cpu)
+		return true;
+
+	cpu = smp_processor_id();
+
+	/* Single domain. Must be on a CPU in that domain. */
+	if (rr->hdr)
+		return cpumask_test_cpu(cpu, &rr->hdr->cpu_mask);
+
+	/* Summing domains that share a cache, must be on a CPU for that cache. */
+	return cpumask_test_cpu(cpu, &rr->ci->shared_cpu_map);
+}
+
 static int __l3_mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 {
-	int cpu = smp_processor_id();
 	u32 closid = rdtgrp->closid;
 	u32 rmid = rdtgrp->mon.rmid;
 	struct rdt_l3_mon_domain *d;
@@ -452,9 +480,6 @@ static int __l3_mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 		return 0;
 	}
 
-	/* Reading a single domain, must be on a CPU in that domain. */
-	if (!cpumask_test_cpu(cpu, &d->hdr.cpu_mask))
-		return -EINVAL;
 	if (rr->is_mbm_cntr)
 		rr->err = resctrl_arch_cntr_read(rr->r, d, closid, rmid, cntr_id,
 						 rr->evt->evtid, &tval);
@@ -472,7 +497,6 @@ static int __l3_mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 
 static int __l3_mon_event_count_sum(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 {
-	int cpu = smp_processor_id();
 	u32 closid = rdtgrp->closid;
 	u32 rmid = rdtgrp->mon.rmid;
 	struct rdt_l3_mon_domain *d;
@@ -490,10 +514,6 @@ static int __l3_mon_event_count_sum(struct rdtgroup *rdtgrp, struct rmid_read *r
 		return -EINVAL;
 	}
 
-	/* Summing domains that share a cache, must be on a CPU for that cache. */
-	if (!cpumask_test_cpu(cpu, &rr->ci->shared_cpu_map))
-		return -EINVAL;
-
 	/*
 	 * Legacy files must report the sum of an event across all
 	 * domains that share the same L3 cache instance.
@@ -524,7 +544,9 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 {
 	switch (rr->r->rid) {
 	case RDT_RESOURCE_L3:
-		WARN_ON_ONCE(rr->evt->any_cpu);
+		if (!cpu_on_correct_domain(rr))
+			return -EINVAL;
+
 		if (rr->hdr)
 			return __l3_mon_event_count(rdtgrp, rr);
 		else
Signature · 1 line
2.45.2
maintainer
#22 lore raw

Re: [PATCH v5 08/10] fs/resctrl: Do not invoke smp_processor_id() in preemptible context

Hi Chenyu,

Quoted 5 lines · Chen Yu

From: Tony Luck <[email protected]>

Currently, mon_evt::any_cpu is false for all events associated with RDT_RESOURCE_L3. The MMIO-based CMT will set any_cpu to true in a follow-up patch. This change will trigger a warning when calling

Please drop the "in a follow-up patch" since that becomes irrelevant after merge. Also please rewrite this paragraph to not just describe code but what this actually means at a high level.

Quoted 8 lines

smp_processor_id() inside __l3_mon_event_count(), because __l3_mon_event_count() may be invoked from arbitrary CPUs in task context. Since the calling context is preemptible here, smp_processor_id() will emit a debug warning.

To prepare for MMIO-based CMT reads, skip the current CPU lookup when an event's any_cpu flag is set. Events with this flag do not require execution on a specific CPU.

Please note there is an upcoming change that will impact this change. Nothing major, the change should be clear.
https://lore.kernel.org/lkml/6b3c66a49788828bd8c04a6911bd74c91ccd56f3.1782857711.git.reinette.chatre@intel.com/

Reinette

author
#23 lore raw

PATCH v5 9/10 x86/resctrl: Introduce helpers to read L3 occupancy via MMIO

In reply to Chen Yu

Introduce erdt_cpu_has() to verify if a specific RDT feature is backed by an ERDT table. erdt_cpu_has() is derived from rdt_cpu_has(), which not only considers firmware (ERDT table and its sub-tables) support for an event, but also considers userspace input like "rdt=!cmt". erdt_cpu_has() expects the same input parameters as rdt_cpu_has().

Introduce erdt_mon_read(), a helper that retrieves monitoring data for a given RMID and event ID from an ERDT domain. erdt_mon_read() leverages erdt_cpu_has() to check whether the system supports the corresponding ACPI tables, such as ERDT and CMRC (Cache Monitoring Registers for CPU Agents Description). It invokes the low-level MMIO read callbacks (introduced later) if supported.

---
v4->v5:
A new patch. It extracts the logic that checks whether a specific RDT feature is backed by ERDT. The original plan was to leverage arch_priv for ERDT-specific operations, but MMIO-based CMT shares interleaved logic with MSR-based CMT, making them difficult to decouple from one another. A helper such as erdt_cpu_has() will therefore simplify differentiation.

---
 arch/x86/include/asm/resctrl.h         |  6 +++++
 arch/x86/kernel/cpu/resctrl/core.c     | 33 ++++++++++++++++++++++++--
 arch/x86/kernel/cpu/resctrl/erdt.c     | 12 ++++++++++
 arch/x86/kernel/cpu/resctrl/internal.h |  3 +++
 arch/x86/kernel/cpu/resctrl/monitor.c  | 25 +++++++++++++++++--
 fs/resctrl/monitor.c                   |  6 +++++
 include/linux/resctrl.h                |  1 +
 7 files changed, 82 insertions(+), 4 deletions(-)
arch/x86/include/asm/resctrl.h +6 -0
index 575f8408a9e7..0fd4bf85f628 100644
@@ -49,6 +49,8 @@ DECLARE_STATIC_KEY_FALSE(rdt_enable_key);
 DECLARE_STATIC_KEY_FALSE(rdt_alloc_enable_key);
 DECLARE_STATIC_KEY_FALSE(rdt_mon_enable_key);
 
+bool erdt_cpu_has(int flag);
+
 static inline bool resctrl_arch_alloc_capable(void)
 {
 	return rdt_alloc_capable;
@@ -131,6 +133,10 @@ static inline unsigned int resctrl_arch_round_mon_val(unsigned int val)
 {
 	unsigned int scale = boot_cpu_data.x86_cache_occ_scale;
 
+	/* ERDT itself factors and rounds the data within erdt.c */
+	if (erdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC))
+		return val;
+
 	/* h/w works in units of "boot_cpu_data.x86_cache_occ_scale" */
 	val /= scale;
 	return val * scale;
arch/x86/kernel/cpu/resctrl/core.c +31 -2
index 2e95586ebe45..5932cf813cb4 100644
@@ -895,6 +895,29 @@ bool rdt_cpu_has(int flag)
 	return ret;
 }
 
+bool erdt_cpu_has(int flag)
+{
+	struct rdt_options *o;
+	bool ret;
+
+	ret = erdt_support_features(flag);
+
+	if (!ret)
+		return ret;
+
+	for (o = rdt_options; o < &rdt_options[NUM_RDT_OPTIONS]; o++) {
+		if (flag == o->flag) {
+			if (o->force_off)
+				ret = false;
+			if (o->force_on)
+				ret = true;
+			break;
+		}
+	}
+
+	return ret;
+}
+
 bool resctrl_arch_is_evt_configurable(enum resctrl_event_id evt)
 {
 	if (!rdt_cpu_has(X86_FEATURE_BMEC))
@@ -982,7 +1005,10 @@ static __init bool get_rdt_mon_resources(void)
 	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
 	bool ret = false;
 
-	if (rdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC)) {
+	if (erdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC)) {
+		resctrl_enable_mon_event(QOS_L3_OCCUP_EVENT_ID, true, 0, NULL);
+		ret = true;
+	} else if (rdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC)) {
 		resctrl_enable_mon_event(QOS_L3_OCCUP_EVENT_ID, false, 0, NULL);
 		ret = true;
 	}
@@ -1000,7 +1026,10 @@ static __init bool get_rdt_mon_resources(void)
 	if (!ret)
 		return false;
 
-	return !rdt_get_l3_mon_config(r);
+	if (rdt_get_l3_mon_config(r))
+		return false;
+
+	return r->mon_capable;
 }
 
 static __init void __check_quirks_intel(void)
arch/x86/kernel/cpu/resctrl/erdt.c +12 -0
index a5754d64fcc1..1114ad4e3b42 100644
@@ -18,6 +18,7 @@
 #include <linux/xarray.h>
 
 #include <asm/apic.h>
+#include <asm/cpufeatures.h>
 
 #include "internal.h"
 
@@ -27,11 +28,17 @@ static bool __erdt_enabled;
 
 #define ERDT_VALID_VERSION		1
 #define CMRC_SUPPORTED_INDEX_FN		1
+#define UNAVAILABLE_COUNTER		BIT_ULL(63)
 #define RMDD_FLAG_CPU_L3_DOMAIN		BIT(0)
 
 /* Bitmask of valid sub-tables found in the first RMDD, used to ensure all RMDDs match. */
 static u32 valid_subtbl_mask;
 
+bool erdt_support_features(int flag)
+{
+	return false;
+}
+
 int erdt_get_max_rmid(int cpu)
 {
 	struct erdt_domain_info *d;
@@ -50,6 +57,11 @@ int erdt_get_max_rmid(int cpu)
 	return -1;
 }
 
+int erdt_mon_read(struct rdt_domain_hdr *hdr, int ev_id, int rmid, u64 *val)
+{
+	return -EIO;
+}
+
 static void __iomem *erdt_ioremap(phys_addr_t base, u32 num_pages, const char *desc)
 {
 	void __iomem *addr;
arch/x86/kernel/cpu/resctrl/internal.h +3 -0
index 6eb0fdea6b63..ecb44f82581e 100644
@@ -278,8 +278,11 @@ static inline void intel_aet_mon_domain_setup(int cpu, int id, struct rdt_resour
 static inline bool intel_handle_aet_option(bool force_off, char *tok) { return false; }
 #endif
 
+bool erdt_support_features(int flag);
+bool erdt_cpu_has(int flag);
 int erdt_get_max_rmid(int cpu);
 int erdt_init(void);
 void erdt_exit(void);
+int erdt_mon_read(struct rdt_domain_hdr *hdr, int ev_id, int rmid, u64 *val);
 
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/monitor.c +23 -2
index f4f4c9015ceb..e6d7037f000b 100644
@@ -279,6 +279,10 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 
 	switch (r->rid) {
 	case RDT_RESOURCE_L3:
+		if (eventid == QOS_L3_OCCUP_EVENT_ID &&
+		    erdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC))
+			return erdt_mon_read(hdr, eventid, rmid, val);
+
 		return arch_l3_read_event(hdr, rmid, eventid, val, r);
 	case RDT_RESOURCE_PERF_PKG:
 		return intel_aet_read_event(hdr->id, rmid, arch_priv, val);
@@ -423,6 +427,11 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r)
 {
 	unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+	/*
+	 * Currently assume all CPU domains share the same maximum RMID
+	 * value from the RMDD table, use CPU0 domain's value.
+	 */
+	int erdt_max_rmid = erdt_get_max_rmid(0);
 	unsigned int threshold;
 	u32 eax, ebx, ecx, edx;
 
@@ -430,7 +439,8 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r)
 
 	resctrl_rmid_realloc_limit = boot_cpu_data.x86_cache_size * 1024;
 	hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale / snc_nodes_per_l3_cache;
-	r->mon.num_rmid = (boot_cpu_data.x86_cache_max_rmid + 1) / snc_nodes_per_l3_cache;
+	r->mon.num_rmid = (erdt_max_rmid > 0) ? erdt_max_rmid + 1 :
+			   (boot_cpu_data.x86_cache_max_rmid + 1) / snc_nodes_per_l3_cache;
 	hw_res->mbm_width = MBM_CNTR_WIDTH_BASE;
 
 	if (mbm_offset > 0 && mbm_offset <= MBM_CNTR_WIDTH_OFFSET_MAX)
@@ -477,7 +487,18 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r)
 		hw_res->mbm_cntr_assign_enabled = true;
 	}
 
-	r->mon_capable = true;
+	/*
+	 * If the platform has ERDT but the SNC is enabled,
+	 * this monitor should not be enabled.
+	 */
+	if (erdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC) &&
+	    snc_nodes_per_l3_cache > 1) {
+		WARN_ONCE(1, "ERDT is enabled but SNC%d is enabled, monitors for resource[%s] should be disabled\n",
+			  snc_nodes_per_l3_cache, r->name);
+		resctrl_disable_mon_event(QOS_L3_OCCUP_EVENT_ID);
+	} else {
+		r->mon_capable = true;
+	}
 
 	return 0;
 }
fs/resctrl/monitor.c +6 -0
index 6a7c86a72c51..2cf03e4cf72a 100644
@@ -1034,6 +1034,12 @@ bool resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu,
 	return true;
 }
 
+void resctrl_disable_mon_event(enum resctrl_event_id eventid)
+{
+	if (mon_event_all[eventid].enabled)
+		mon_event_all[eventid].enabled = false;
+}
+
 bool resctrl_is_mon_event_enabled(enum resctrl_event_id eventid)
 {
 	return eventid >= QOS_FIRST_EVENT && eventid < QOS_NUM_EVENTS &&
include/linux/resctrl.h +1 -0
index 73ff522448a0..dfde025432ab 100644
@@ -420,6 +420,7 @@ int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid);
 
 bool resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu,
 			      unsigned int binary_bits, void *arch_priv);
+void resctrl_disable_mon_event(enum resctrl_event_id eventid);
 
 bool resctrl_is_mon_event_enabled(enum resctrl_event_id eventid);
 
Signature · 1 line
2.45.2
maintainer
#24 lore raw

Re: [PATCH v5 09/10] x86/resctrl: Introduce helpers to read L3 occupancy via MMIO

Hi Chenyu,

Quoted 5 lines · Chen Yu

Introduce erdt_cpu_has() to verify if a specific RDT feature is backed by an ERDT table. erdt_cpu_has() is derived from rdt_cpu_has(), which not only considers firmware (ERDT table and its sub-tables) support for an event, but also considers userspace input like "rdt=!cmt". erdt_cpu_has() expects the same input parameters as

parameters -> parameter
although this can be more helpful with something like:
"Similar to rdt_cpu_has(), erdt_cpu_has() expects an x86 feature flag as parameter."

Quoted 9 lines

rdt_cpu_has().

Introduce erdt_mon_read(), a helper that retrieves monitoring data for a given RMID and event ID from an ERDT domain. erdt_mon_read() leverages erdt_cpu_has() to check whether the system supports the corresponding ACPI tables, such as ERDT and CMRC (Cache Monitoring Registers for CPU Agents Description). It invokes the low-level MMIO read callbacks (introduced later) if supported.

Since this patch contains two logical changes, can it be split?

Quoted 3 lines

---

...

Quoted 14 lines
arch/x86/include/asm/resctrl.h +2 -0
index 575f8408a9e7..0fd4bf85f628 100644
@@ -49,6 +49,8 @@ DECLARE_STATIC_KEY_FALSE(rdt_enable_key);
 DECLARE_STATIC_KEY_FALSE(rdt_alloc_enable_key);
 DECLARE_STATIC_KEY_FALSE(rdt_mon_enable_key);
 
+bool erdt_cpu_has(int flag);
+
 static inline bool resctrl_arch_alloc_capable(void)
 {
 	return rdt_alloc_capable;

arch/x86/include/asm/resctrl.h static inline unsigned int resctrl_arch_round_mon_val(unsigned int val) · line 139

Quoted 7 lines
arch/x86/include/asm/resctrl.h +4 -0
@@ -131,6 +133,10 @@ static inline unsigned int resctrl_arch_round_mon_val(unsigned int val)
 {
 	unsigned int scale = boot_cpu_data.x86_cache_occ_scale;
 
+	/* ERDT itself factors and rounds the data within erdt.c */
+	if (erdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC))
+		return val;
+

This does not seem to take all usages of resctrl_arch_round_mon_val() into account. This function is also used to determine a valid (user controlled) threshold for the limbo handler because it needs to be ensured that the threshold is a value that the hardware can actually measure.

Speaking of the limbo handler ... I did not notice any mention of limbo handler or any changes to it. From what I can tell this implementation keeps one limbo handler per domain even though occupancy counts can be read from any CPU. Is this intended?

Quoted 8 lines
arch/x86/include/asm/resctrl.h +0 -0
 	/* h/w works in units of "boot_cpu_data.x86_cache_occ_scale" */
 	val /= scale;
 	return val * scale;
arch/x86/kernel/cpu/resctrl/core.c +0 -0
index 2e95586ebe45..5932cf813cb4 100644

arch/x86/kernel/cpu/resctrl/core.c bool rdt_cpu_has(int flag) · line 903

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/core.c +6 -0
@@ -895,6 +895,29 @@ bool rdt_cpu_has(int flag)
 	return ret;
 }
 
+bool erdt_cpu_has(int flag)
+{
+	struct rdt_options *o;
+	bool ret;
+
+	ret = erdt_support_features(flag);

erdt_support_features() -> erdt_support_feature() or maybe just "erdt_supports()" ? (only one feature can be checked at a time?)

Quoted 33 lines
arch/x86/kernel/cpu/resctrl/core.c +21 -1
+
+	if (!ret)
+		return ret;
+
+	for (o = rdt_options; o < &rdt_options[NUM_RDT_OPTIONS]; o++) {
+		if (flag == o->flag) {
+			if (o->force_off)
+				ret = false;
+			if (o->force_on)
+				ret = true;
+			break;
+		}
+	}
+
+	return ret;
+}
+
 bool resctrl_arch_is_evt_configurable(enum resctrl_event_id evt)
 {
 	if (!rdt_cpu_has(X86_FEATURE_BMEC))
@@ -982,7 +1005,10 @@ static __init bool get_rdt_mon_resources(void)
 	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
 	bool ret = false;
 
-	if (rdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC)) {
+	if (erdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC)) {
+		resctrl_enable_mon_event(QOS_L3_OCCUP_EVENT_ID, true, 0, NULL);
+		ret = true;
+	} else if (rdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC)) {
 		resctrl_enable_mon_event(QOS_L3_OCCUP_EVENT_ID, false, 0, NULL);
 		ret = true;
 	}

arch/x86/kernel/cpu/resctrl/core.c static __init bool get_rdt_mon_resources(void) · line 1032

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/core.c +4 -1
@@ -1000,7 +1026,10 @@ static __init bool get_rdt_mon_resources(void)
 	if (!ret)
 		return false;
 
-	return !rdt_get_l3_mon_config(r);
+	if (rdt_get_l3_mon_config(r))
+		return false;
+
+	return r->mon_capable;

It is not clear to me why this change is necessary.

arch/x86/kernel/cpu/resctrl/erdt.c line 22

Quoted 13 lines
arch/x86/kernel/cpu/resctrl/core.c +0 -0
 }
 
 static __init void __check_quirks_intel(void)
arch/x86/kernel/cpu/resctrl/erdt.c +1 -0
index a5754d64fcc1..1114ad4e3b42 100644
@@ -18,6 +18,7 @@
 #include <linux/xarray.h>
 
 #include <asm/apic.h>
+#include <asm/cpufeatures.h>
 

This include is not used in this change?

arch/x86/kernel/cpu/resctrl/erdt.c static bool __erdt_enabled; · line 31

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/erdt.c +1 -0
 #include "internal.h"
 
@@ -27,11 +28,17 @@ static bool __erdt_enabled;
 
 #define ERDT_VALID_VERSION		1
 #define CMRC_SUPPORTED_INDEX_FN		1
+#define UNAVAILABLE_COUNTER		BIT_ULL(63)

Not used in this patch.

arch/x86/kernel/cpu/resctrl/erdt.c static bool __erdt_enabled; · line 40

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/erdt.c +4 -0
 #define RMDD_FLAG_CPU_L3_DOMAIN		BIT(0)
 
 /* Bitmask of valid sub-tables found in the first RMDD, used to ensure all RMDDs match. */
 static u32 valid_subtbl_mask;
 
+bool erdt_support_features(int flag)
+{
+	return false;
+}

Why not provide implementation here?

arch/x86/kernel/cpu/resctrl/erdt.c int erdt_get_max_rmid(int cpu) · line 61

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c +3 -0
+
 int erdt_get_max_rmid(int cpu)
 {
 	struct erdt_domain_info *d;
@@ -50,6 +57,11 @@ int erdt_get_max_rmid(int cpu)
 	return -1;
 }
 
+int erdt_mon_read(struct rdt_domain_hdr *hdr, int ev_id, int rmid, u64 *val)
+{

Please use accurate types ... rmid is u32, evt_id is enum resctrl_event_id ...

Quoted 50 lines
arch/x86/kernel/cpu/resctrl/erdt.c +3 -0
+	return -EIO;
+}
+
 static void __iomem *erdt_ioremap(phys_addr_t base, u32 num_pages, const char *desc)
 {
 	void __iomem *addr;
arch/x86/kernel/cpu/resctrl/internal.h +3 -0
index 6eb0fdea6b63..ecb44f82581e 100644
@@ -278,8 +278,11 @@ static inline void intel_aet_mon_domain_setup(int cpu, int id, struct rdt_resour
 static inline bool intel_handle_aet_option(bool force_off, char *tok) { return false; }
 #endif
 
+bool erdt_support_features(int flag);
+bool erdt_cpu_has(int flag);
 int erdt_get_max_rmid(int cpu);
 int erdt_init(void);
 void erdt_exit(void);
+int erdt_mon_read(struct rdt_domain_hdr *hdr, int ev_id, int rmid, u64 *val);
 
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/monitor.c +9 -0
index f4f4c9015ceb..e6d7037f000b 100644
@@ -279,6 +279,10 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 
 	switch (r->rid) {
 	case RDT_RESOURCE_L3:
+		if (eventid == QOS_L3_OCCUP_EVENT_ID &&
+		    erdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC))
+			return erdt_mon_read(hdr, eventid, rmid, val);
+
 		return arch_l3_read_event(hdr, rmid, eventid, val, r);
 	case RDT_RESOURCE_PERF_PKG:
 		return intel_aet_read_event(hdr->id, rmid, arch_priv, val);
@@ -423,6 +427,11 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r)
 {
 	unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
+	/*
+	 * Currently assume all CPU domains share the same maximum RMID
+	 * value from the RMDD table, use CPU0 domain's value.
+	 */
+	int erdt_max_rmid = erdt_get_max_rmid(0);
 	unsigned int threshold;
 	u32 eax, ebx, ecx, edx;
 

arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_l3_mon_config(struct rdt_resource *r) · line 441

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/monitor.c +0 -0
@@ -430,7 +439,8 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r)
 
 	resctrl_rmid_realloc_limit = boot_cpu_data.x86_cache_size * 1024;
 	hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale / snc_nodes_per_l3_cache;

This implementation still mixes CPUID and ERDT enumeration. Above, for example, a system that supports ERDT cace occupancy counts is expected to provide another scale value via CPUID? Could you please clearly separate ERDT and CPUID enumeration.

arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_l3_mon_config(struct rdt_resource *r) · line 443

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/monitor.c +2 -1
-	r->mon.num_rmid = (boot_cpu_data.x86_cache_max_rmid + 1) / snc_nodes_per_l3_cache;
+	r->mon.num_rmid = (erdt_max_rmid > 0) ? erdt_max_rmid + 1 :
+			   (boot_cpu_data.x86_cache_max_rmid + 1) / snc_nodes_per_l3_cache;

Here things are also intertwined. Above essentially means: "if cache occupancy counters are read via ERDT then use max RMID from CMRC and let it *override* the RMID from CPUID that is appropriate for memory bandwidth monitoring"!

I find the code like above to become increasingly obfuscated. This patch starts with a promise that "erdt_cpu_has() indicates support for ERDT feature" ... but then above generic code uses the more subtle "erdt_max_rmid > 0" as a subtle check of ERDT support. Having the CPUID and ERDT support be so intertwined with subtle checks just added for convenience makes the code difficult to understand and I am concerned it will be error prone when folks try to build on top of it.

arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_l3_mon_config(struct rdt_resource *r) · line 492

Quoted 11 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3 -1
 	hw_res->mbm_width = MBM_CNTR_WIDTH_BASE;
 
 	if (mbm_offset > 0 && mbm_offset <= MBM_CNTR_WIDTH_OFFSET_MAX)
@@ -477,7 +487,18 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r)
 		hw_res->mbm_cntr_assign_enabled = true;
 	}
 
-	r->mon_capable = true;
+	/*
+	 * If the platform has ERDT but the SNC is enabled,
+	 * this monitor should not be enabled.

(line length, please check entire series)

arch/x86/kernel/cpu/resctrl/monitor.c int __init rdt_get_l3_mon_config(struct rdt_resource *r) · line 498

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/monitor.c +6 -0
+	 */
+	if (erdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC) &&
+	    snc_nodes_per_l3_cache > 1) {
+		WARN_ONCE(1, "ERDT is enabled but SNC%d is enabled, monitors for resource[%s] should be disabled\n",
+			  snc_nodes_per_l3_cache, r->name);
+		resctrl_disable_mon_event(QOS_L3_OCCUP_EVENT_ID);

This seems backwards. I think this highlights how ERDT needs to be handled separately and not try to wedge into CPUID enabling. I am surprised about this behavior. Isn't SNC and RDT supported via MSR? Would this not prevent users from using cache occupancy on an SNC enabled system when it could be supported? Is a WARN actually justified here? Could SNC capability not be checked early _before_ enabling ERDT cace occupancy event and if SNC is active then the MSR cache occupancy can be used instead?

Quoted 11 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3 -0
+	} else {
+		r->mon_capable = true;
+	}
 
 	return 0;
 }
fs/resctrl/monitor.c +0 -0
index 6a7c86a72c51..2cf03e4cf72a 100644

fs/resctrl/monitor.c bool resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu, · line 1042

Quoted 9 lines
fs/resctrl/monitor.c +6 -0
@@ -1034,6 +1034,12 @@ bool resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu,
 	return true;
 }
 
+void resctrl_disable_mon_event(enum resctrl_event_id eventid)
+{
+	if (mon_event_all[eventid].enabled)
+		mon_event_all[eventid].enabled = false;
+}
+

Please see https://lore.kernel.org/lkml/[email protected]/

Quoted 8 lines
fs/resctrl/monitor.c +0 -0
 bool resctrl_is_mon_event_enabled(enum resctrl_event_id eventid)
 {
 	return eventid >= QOS_FIRST_EVENT && eventid < QOS_NUM_EVENTS &&
include/linux/resctrl.h +0 -0
index 73ff522448a0..dfde025432ab 100644

include/linux/resctrl.h int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid); · line 426

Quoted 7 lines
include/linux/resctrl.h +1 -0
@@ -420,6 +420,7 @@ int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid);
 
 bool resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu,
 			      unsigned int binary_bits, void *arch_priv);
+void resctrl_disable_mon_event(enum resctrl_event_id eventid);
 
 bool resctrl_is_mon_event_enabled(enum resctrl_event_id eventid);
 

Reinette

author
#25 lore raw

PATCH v5 10/10 x86/resctrl: Enable read L3 occupancy via MMIO

In reply to Chen Yu

Implement the erdt_mon_read().

Use the CMRC (Cache Monitoring Registers for CPU Agents Description) ACPI sub-table to read LLC occupancy counters for each RMID via MMIO when ERDT is enabled. This CMRC information is stored in the rdt_hw_l3_mon_domain, which could be accessed directly.

---
v4->v5:
Split into a smaller patch; no functional code changes in the low level implementation, but read from the CMRC information stored per domain .

---
 arch/x86/kernel/cpu/resctrl/erdt.c | 63 ++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)
arch/x86/kernel/cpu/resctrl/erdt.c +63 -0
index 1114ad4e3b42..f897c7e247bb 100644
@@ -36,6 +36,9 @@ static u32 valid_subtbl_mask;
 
 bool erdt_support_features(int flag)
 {
+	if (flag == X86_FEATURE_CQM_OCCUP_LLC)
+		return valid_subtbl_mask & BIT(ACPI_ERDT_TYPE_CMRC);
+
 	return false;
 }
 
@@ -57,8 +60,68 @@ int erdt_get_max_rmid(int cpu)
 	return -1;
 }
 
+static void __iomem *cmrc_index_function_1(struct erdt_domain_info *d,
+					   struct acpi_erdt_cmrc *cmrc, int rmid)
+{
+	u16 clump_size, stride_size;
+	void __iomem *vaddr;
+
+	clump_size = cmrc->clump_size;
+	stride_size = cmrc->clump_stride;
+
+	/*
+	 * MMIO_ADDRESS_for_RMID# = CMRC Base +
+	 *   (RMID / ClumpSize) * Stride +
+	 *   (RMID % ClumpSize) * 8
+	 */
+	vaddr = d->base[ERDT_MMIO_CMRC_BASE] +
+		(rmid / clump_size) * stride_size +
+		(rmid % clump_size) * 8;
+
+	return vaddr;
+}
+
+static int erdt_read_l3_occupancy(struct erdt_domain_info *d, int rmid, u64 *val)
+{
+	struct acpi_erdt_cmrc *cmrc;
+	void __iomem *vaddr;
+	u64 l3_cmt_count;
+	u32 offset;
+
+	cmrc = d->cmrc;
+	if (!cmrc)
+		return -EIO;
+
+	offset = (rmid / cmrc->clump_size) * cmrc->clump_stride +
+		 (rmid % cmrc->clump_size) * 8;
+	/* Overflow of cmt_reg_size * SZ_4K already validated in erdt_ioremap(). */
+	if (offset + sizeof(u64) > (u32)cmrc->cmt_reg_size * SZ_4K)
+		return -EINVAL;
+
+	vaddr = cmrc_index_function_1(d, cmrc, rmid);
+
+	l3_cmt_count = readq(vaddr);
+	if (l3_cmt_count & UNAVAILABLE_COUNTER)
+		return -EINVAL;
+
+	*val = l3_cmt_count * cmrc->up_scale;
+
+	return 0;
+}
+
 int erdt_mon_read(struct rdt_domain_hdr *hdr, int ev_id, int rmid, u64 *val)
 {
+	struct rdt_hw_l3_mon_domain *hw_dom;
+	struct erdt_domain_info *d;
+
+	hw_dom = resctrl_to_arch_mon_dom(container_of(hdr, struct rdt_l3_mon_domain, hdr));
+	d = hw_dom->d_info;
+	if (!d)
+		return -EIO;
+
+	if (ev_id == QOS_L3_OCCUP_EVENT_ID)
+		return erdt_read_l3_occupancy(d, rmid, val);
+
 	return -EIO;
 }
 
Signature · 1 line
2.45.2
maintainer
#26 lore raw

Re: [PATCH v5 10/10] x86/resctrl: Enable read L3 occupancy via MMIO

Hi Chenyu,

Quoted 59 lines · Chen Yu

Implement the erdt_mon_read().

Use the CMRC (Cache Monitoring Registers for CPU Agents Description) ACPI sub-table to read LLC occupancy counters for each RMID via MMIO when ERDT is enabled. This CMRC information is stored in the rdt_hw_l3_mon_domain, which could be accessed directly.

---
v4->v5:
Split into a smaller patch; no functional code changes in the low level implementation, but read from the CMRC information stored per domain .

---
 arch/x86/kernel/cpu/resctrl/erdt.c | 63 ++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)
arch/x86/kernel/cpu/resctrl/erdt.c +27 -0
index 1114ad4e3b42..f897c7e247bb 100644
@@ -36,6 +36,9 @@ static u32 valid_subtbl_mask;
 
 bool erdt_support_features(int flag)
 {
+	if (flag == X86_FEATURE_CQM_OCCUP_LLC)
+		return valid_subtbl_mask & BIT(ACPI_ERDT_TYPE_CMRC);
+
 	return false;
 }
 
@@ -57,8 +60,68 @@ int erdt_get_max_rmid(int cpu)
 	return -1;
 }
 
+static void __iomem *cmrc_index_function_1(struct erdt_domain_info *d,
+					   struct acpi_erdt_cmrc *cmrc, int rmid)
+{
+	u16 clump_size, stride_size;
+	void __iomem *vaddr;
+
+	clump_size = cmrc->clump_size;
+	stride_size = cmrc->clump_stride;
+
+	/*
+	 * MMIO_ADDRESS_for_RMID# = CMRC Base +
+	 *   (RMID / ClumpSize) * Stride +
+	 *   (RMID % ClumpSize) * 8
+	 */
+	vaddr = d->base[ERDT_MMIO_CMRC_BASE] +
+		(rmid / clump_size) * stride_size +
+		(rmid % clump_size) * 8;
+
+	return vaddr;
+}
+
+static int erdt_read_l3_occupancy(struct erdt_domain_info *d, int rmid, u64 *val)
+{
+	struct acpi_erdt_cmrc *cmrc;

arch/x86/kernel/cpu/resctrl/erdt.c int erdt_get_max_rmid(int cpu) · line 96

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c +10 -0
+	void __iomem *vaddr;
+	u64 l3_cmt_count;
+	u32 offset;
+
+	cmrc = d->cmrc;
+	if (!cmrc)
+		return -EIO;
+
+	offset = (rmid / cmrc->clump_size) * cmrc->clump_stride +
+		 (rmid % cmrc->clump_size) * 8;

This is the same function open coded in cmrc_index_function_1() - can it just be coded once and then re-used?

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/erdt.c +5 -0
+	/* Overflow of cmt_reg_size * SZ_4K already validated in erdt_ioremap(). */
+	if (offset + sizeof(u64) > (u32)cmrc->cmt_reg_size * SZ_4K)
+		return -EINVAL;
+
+	vaddr = cmrc_index_function_1(d, cmrc, rmid);

arch/x86/kernel/cpu/resctrl/erdt.c int erdt_get_max_rmid(int cpu) · line 111

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/erdt.c +10 -0
+
+	l3_cmt_count = readq(vaddr);
+	if (l3_cmt_count & UNAVAILABLE_COUNTER)
+		return -EINVAL;
+
+	*val = l3_cmt_count * cmrc->up_scale;
+
+	return 0;
+}
+

Reinette