Introduce MMIO-based CMT access for Enhanced RDT
Subsystems (3)
v10 cover Introduce MMIO-based CMT access for Enhanced RDT
v9: https://lore.kernel.org/lkml/[email protected]/
v8: https://lore.kernel.org/lkml/[email protected]/
v7: https://lore.kernel.org/lkml/[email protected]/
v6: https://lore.kernel.org/lkml/[email protected]/
v5: https://lore.kernel.org/lkml/[email protected]/
v4: https://lore.kernel.org/lkml/[email protected]/
v3: https://lore.kernel.org/lkml/[email protected]/
v2: https://lore.kernel.org/lkml/[email protected]/
v1: https://lore.kernel.org/lkml/[email protected]/
Intel Enhanced Resource Director Technology (ERDT) extends the existing RDT framework with two major capabilities:
1. MMIO-based access to monitoring and allocation registers, replacing the legacy MSR-based interface.
2. Region-aware RDT for fine-grained control over different tiers of
memory (e.g., CXL.mem, DDR).
This is described in the Intel RDT Architecture Specification:
https://cdrdv2-public.intel.com/789566/356688-intel-rdt-arch-spec.pdf
This patch set focuses on the first part: enabling MMIO-based access for Cache Monitoring Technology (CMT), while CAT/MBM/MBA are still using MSR. The platform advertises the MMIO register layout through the ACPI ERDT (Enhanced Resource Director Technology) table, which contains sub-tables describing per-domain register regions for monitoring and allocation.
With ERDT, L3 cache occupancy counters are read via MMIO rather than MSR, allowing the reads to be performed from any CPU without requiring cross-CPU IPIs. This series parses the relevant ACPI sub-tables (RMDD, CACD, CMRC), prepares the resctrl monitor infrastructure for MMIO-based reads, and adds initial support for reading L3 occupancy via the CMRC interface. ERDT is not compatible with Sub-NUMA Cluster (SNC) mode and stays disabled when SNC is enabled.
Thanks Tony, Reinette, Thomas, Dave, Boris, Peter, Hongyu for your time to look at this patch set.
Internal sashiko has not reported any issues on the v10. The change history in the section "Changes since v9" in each patch is generated by AI.
Major changes from V9 to V10:
- Reorder the series (Reinette Chatre):
- Move "arm,x86,fs/resctrl: Move resctrl_arch_round_mon_val() out of line" (renamed from "x86,fs/resctrl: ...") to patch 02 so that MPAM maintainers only need to review that patch.
- Replace v9 "x86/resctrl: Do not enable ERDT when Sub-NUMA Cluster is enabled" with the new [PATCH 04/12] "x86/resctrl: Detect Sub-NUMA Cluster mode before enumerating resources", and move the ERDT/SNC check into [PATCH 05/12].
- Replace v9 "x86/resctrl: Add a global ERDT monitor scale" with the new [PATCH 11/12] "x86/resctrl: Add the ERDT CMT scale", placed after the CMRC parsing and erdt_cpu_has() patches. The scale is renamed to erdt_cmt_scale/erdt_get_cmt_scale() and is reset in erdt_exit().
- [behavior change] Do not fail the CPU online operation when ERDT and CPUID disagree on a CPU's L3 domain. Keep the CPU online and in the default group, but out of all resctrl domains. erdt_try_bind_cpu() is renamed to erdt_try_bind_domain(). (Reinette Chatre)
- Skip an RMDD that does not list any CPU known to the kernel, e.g. when all of them are excluded by nr_cpus=, and map its control registers only after it is known to have CPUs. (Reinette Chatre)
- Limit RMIDs by the ERDT max_rmid whenever ERDT is enabled, not only when LLC occupancy is read via MMIO, because RMDD max_rmid applies to all RMIDs. (Reinette Chatre)
- [bug fix] Fix the truncating u32 cast in the CMRC register size check. (Reinette Chatre)
Changes from V8 to V9:
- Do not enable ERDT when Sub-NUMA Cluster (SNC) is enabled.
New patch [PATCH 04/12]. (Reinette Chatre, Tony Luck, sashiko)
- CPUID remains the architectural enumeration mechanism: erdt_cpu_has() now requires boot_cpu_has(flag) in addition to ERDT firmware support. (Reinette Chatre, Tony Luck)
- [bug fix] Attach the ERDT information to the L3 monitoring domain before the domain is published on the RCU protected domain list.
(Reinette Chatre, sashiko)
- Do not discard the whole ERDT table when a CACD x2APIC ID has no logical CPU, e.g. (Reinette Chatre, sashiko)
Changes from V7 to V8:
- Rebased on latest x86/cache branch
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
- git commit log revisement
- pick Tony's x86_64 dependency patch from AET patch set and replaced mine.
- No other code logic changes
Changes from V6 to V7:
- Let get_rdt_resources() clean up its own state on failure and drop the asymmetric __resctrl_arch_late_init() wrapper. (Reinette Chatre)
- Make erdt_max_rmid and num_ids unsigned and use a plain min(), match erdt_ioremap() argument types to ioremap(), and reset erdt_max_rmid in erdt_exit(). (Reinette Chatre)
- Run erdt_cpu_valid() once in resctrl_arch_online_cpu(), and match an ERDT domain to a resctrl monitoring domain by domain id rather than by cpumask. Warn on an invalid dom_id and on a duplicate ERDT-to-resctrl domain mapping. (Reinette Chatre)
- Fix the silent truncation of the 64-bit CMRC up_scale: reject an out-of-range value with a FW_BUG warning and fail closed instead of using max_t(int, ...). (Reinette Chatre, Tony Luck)
- Keep the original resctrl_arch_rmid_read() parameter order (resource first) and drop the unused closid/arch_priv arguments. (Reinette Chatre)
- Keep a single erdt_cpu_has() declaration in asm/resctrl.h, required by the inline resctrl_arch_round_mon_val(). (Reinette Chatre)
- Use switch() in erdt_support(), clamp max_rmid with min_t() so MSR-based MBM events are not handed the ERDT RMID, and divide the ERDT scale by snc_nodes_per_l3_cache under SNC. (Reinette Chatre)
Changes from V5 to V6:
- Reorder the series so that the x86/topology change, which touches a different subsystem, comes first. (Reinette Chatre)
- Drop the v5 "x86/resctrl: Replace 'msr' in monitoring data identifiers" patch. None of the renamed identifiers are used by the MMIO code. (Reinette Chatre)
- Split the domain setup into erdt_cpu_valid() and erdt_l3_mon_domain_setup(). Validation now happens in domain_add_cpu_mon() before the CPU is added to the domain cpumask, and attaching ERDT data to a freshly created domain (Reinette Chatre)
- Drop the SNC special case and the new resctrl_disable_mon_event(). (Reinette Chatre)
- resctrl_arch_round_mon_val() now rounds to the ERDT scale instead of returning the value unchanged (Reinette Chatre)
- Make erdt_get_max_rmid() a global value instead of a per-CPU lookup. (Reinette Chatre)
- Use struct cpumask instead of cpumask_var_t, list_for_each_entry() instead of open coded container_of(), and document every member of struct erdt_domain_info. (Reinette Chatre)
Changes from V4 to V5:
There are some major changes since v4:
- (biggest change) Eliminate the xarray for runtime lookups; embed struct erdt_domain_info directly in rdt_hw_l3_mon_domain and assign during l3_mon_domain_setup().
- Separate CPUID and ACPI enumeration cleanly. Do not use CPUID feature flags to gate MMIO-based monitoring. Use ACPI table presence (e.g., CMRC table) to determine event enablement.
- Use ACPI RMDD's own "Max RMID" field for MMIO access instead of relying on CPUID's max RMID (which applies to MSR).
- Enforce the SNC constraint in code rather than burying it behind a comment WARN. Disable mon_capable in rdt_get_l3_mon_config() when ERDT is enabled and snc_nodes_per_l3_cache > 1.
- Split non-resctrl changes (topology.c, apic.h) into a separate preparatory patch prefixed with x86/topology.
- Move the "depends on X86" to "X86_64" adjustment to a separate patch with explicit justification in its changelog.
Changes from V3 to V4:
- Remove the redundant table length check in subtbl_valid() (Thomas Gleixner)
- Reuse subtbl_valid() for all the table iteration (Thomas Gleixner)
- Refine the commit log of [PATCH 5/6] to state that this change is a preparation for [PATCH 6/6] rather than fixing an existing issue (Thomas Gleixner, Reinette Chatre, Tony Luck)
- Fix if CACD lists all CPUs in the LLC domain (sashiko)
- Deal with a corner case that if there is no valid RMDD tables, the erdt_enabled_flag should remain false.(sashiko)
- Add Thomas's Reviewed-by and Hongyu's Tested-by.
Changes from V2 to V3:
- Wrap __resctrl_arch_late_init() to avoid the goto logic. (Thomas Gleixner)
- Make the variables in struct erdt_domain_info tabular format (Thomas Gleixner)
- Remove tail comments (Thomas Gleixner)
- Make the name of erdt_enabled() and variable in it consistent and comprehensible. (Thomas Gleixner)
- Use topo_lookup_cpuid() to search the CPU id according to the x2apic id (Thomas Gleixner)
- Fix kernel doc comment format (Thomas Gleixner)
- Use brackets for multiple lines "if" case. (Thomas Gleixner)
- Let the parameter for cacd_init() to fully utilize 100 characters. (Thomas Gleixner)
- Variables are reordered in reverse fir-tree.(Thomas Gleixner)
- Added a named constant and use it in the rmdd->flags check. (Thomas Gleixner)
- Introduce helper functions to make the code readable when iterating the RMDD tables. (Thomas Gleixner)
- Make the macros tabular format. (Thomas Gleixner)
Changes from V1 to V2:
- Add #include <linux/cleanup.h> to follow the "include-what-you-use" best practice (Tony Luck)
- Fix 3 issues reported by:
https://sashiko.dev/#/patchset/cover.1779872016.git.yu.c.chen%40intel.com
Remove the variable of cacd in struct erdt_domain_info as it will never be used after initialization.
Invoke erdt_exit() to avoid resource leak if rdt_alloc_capable and rdt_mon_capable are both false.
Adjust the comments suggested by sashiko.
Chen Yu (10):
x86/topology: Export topo_lookup_cpuid() for resctrl use arm,x86,fs/resctrl: Move resctrl_arch_round_mon_val() out of line x86/resctrl: Detect Sub-NUMA Cluster mode before enumerating resources x86/resctrl: Parse ACPI ERDT table and save CACD cpumask per RMD x86/resctrl: Attach ACPI ERDT information to L3 mon domain on CPU
online
x86/resctrl: Parse ACPI CMRC table
x86/resctrl: Refactor the monitor read function x86/resctrl: Introduce erdt_cpu_has() and erdt_support() x86/resctrl: Add the ERDT CMT scale
x86/resctrl: Add MMIO-based LLC occupancy monitoring support
Tony Luck (2):
x86/resctrl: Require 64-bit x86 for resctrl support fs/resctrl: Do not invoke smp_processor_id() in preemptible context
arch/x86/Kconfig | 4 +- arch/x86/include/asm/apic.h | 1 + arch/x86/include/asm/resctrl.h | 9 - arch/x86/kernel/cpu/resctrl/Makefile | 2 +- arch/x86/kernel/cpu/resctrl/core.c | 50 ++- arch/x86/kernel/cpu/resctrl/erdt.c | 559 +++++++++++++++++++++++++ arch/x86/kernel/cpu/resctrl/internal.h | 51 ++- arch/x86/kernel/cpu/resctrl/monitor.c | 54 ++- arch/x86/kernel/cpu/topology.c | 2 +- drivers/resctrl/mpam_resctrl.c | 5 + fs/resctrl/monitor.c | 47 ++- include/linux/arm_mpam.h | 5 - include/linux/resctrl.h | 1 + 13 files changed, 737 insertions(+), 53 deletions(-) create mode 100644 arch/x86/kernel/cpu/resctrl/erdt.c
Signature · 1 line
2.25.1
PATCH v10 1/12 x86/topology: Export topo_lookup_cpuid() for resctrl use
Enhanced RDT (ERDT) enumerates its monitoring and control resources through ACPI tables that identify each CPU by its x2APIC ID. To associate this firmware-provided data with the kernel's per-CPU structures, ERDT must translate those x2APIC IDs into logical CPU numbers.
topo_lookup_cpuid() already performs this translation, but it is static to topology.c and therefore not reachable from resctrl.
Export topo_lookup_cpuid() and add its declaration to asm/apic.h that already exposes the inverse mapping via cpuid_to_apicid[], so that ERDT can resolve x2APIC IDs from ACPI tables to logical CPU numbers.
No functional change intended.
---
Changes since v9:
- No code change.
arch/x86/include/asm/apic.h | 1 + arch/x86/kernel/cpu/topology.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
arch/x86/include/asm/apic.h +1 -0
@@ -54,6 +54,7 @@ static inline void x86_32_probe_apic(void) { }#endif extern u32 cpuid_to_apicid[]; +int topo_lookup_cpuid (u32 apic_id);#define CPU_ACPIID_INVALID U32_MAX
arch/x86/kernel/cpu/topology.c +1 -1
@@ -92,7 +92,7 @@ static inline u32 topo_apicid(u32 apicid, enum x86_topology_domains dom)return apicid & (UINT_MAX << x86_topo_system.dom_shifts[dom -1 ]); } -static int topo_lookup_cpuid (u32 apic_id) +int topo_lookup_cpuid (u32 apic_id) {int i;
Signature · 1 line
2.25.1
PATCH v10 2/12 arm,x86,fs/resctrl: Move resctrl_arch_round_mon_val() out of line
In reply to Chen Yuresctrl_arch_round_mon_val() is an inline function in the x86 and MPAM architecture headers. It is only called during resctrl initialization and when user space writes "max_threshold_occupancy", neither of which is a hot path that benefits from inlining.
Upcoming ERDT support needs resctrl_arch_round_mon_val() to consult ERDT-specific state. Keeping it inline would force ERDT prototypes into the x86 asm header, fragmenting the ERDT code into a header where it does not belong.
Move resctrl_arch_round_mon_val() into monitor.c and declare it in include/linux/resctrl.h alongside the other resctrl_arch_*() calls. Move the MPAM stub from arm_mpam.h into mpam_resctrl.c accordingly.
No functional change.
---
Changes since v9:
- Move this patch from the middle of the series to patch 2, right after the x86/topology change, so that MPAM maintainers only need to review this patch. (Reinette Chatre)
- Change the subject prefix from "x86,fs/resctrl:" to "arm,x86,fs/resctrl:" to draw the attention of MPAM maintainers. (Reinette Chatre)
- Changelog: say "x86 and MPAM architecture headers" and "x86 asm header", and state that keeping the function inline would place ERDT code in a header where it does not belong. (Reinette Chatre)
- Since the patch is now ahead of the SNC change, the new function is placed after snc_get_config() instead of after the dropped resctrl_arch_update_snc(). No code change.
- Add Ben Horgan's Acked-by.
arch/x86/include/asm/resctrl.h | 9 --------- arch/x86/kernel/cpu/resctrl/monitor.c | 9 +++++++++ drivers/resctrl/mpam_resctrl.c | 5 +++++ include/linux/arm_mpam.h | 5 ----- include/linux/resctrl.h | 1 + 5 files changed, 15 insertions(+), 14 deletions(-)
arch/x86/include/asm/resctrl.h +0 -9
@@ -130,15 +130,6 @@ static inline void __resctrl_sched_in(struct task_struct *tsk) } } -static inline unsigned int resctrl_arch_round_mon_val (unsigned int val) -{ -unsigned int scale = boot_cpu_data.x86_cache_occ_scale; - -/* h/w works in units of "boot_cpu_data.x86_cache_occ_scale" */ - val /= scale; -return val * scale; -} -static inline void resctrl_arch_set_cpu_default_closid_rmid (int cpu,u32 closid,u32 rmid) {
arch/x86/kernel/cpu/resctrl/monitor.c +9 -0
@@ -416,6 +416,15 @@ static __init int snc_get_config(void)return ret; } +unsigned int resctrl_arch_round_mon_val (unsigned int val) +{ +unsigned int scale = boot_cpu_data.x86_cache_occ_scale; + +/* h/w works in units of "boot_cpu_data.x86_cache_occ_scale" */ + val /= scale; +return val * scale; +} +int __init rdt_get_l3_mon_config (struct rdt_resource *r) {unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;
drivers/resctrl/mpam_resctrl.c +5 -0
@@ -143,6 +143,11 @@ void resctrl_arch_pre_mount(void) { } +unsigned int resctrl_arch_round_mon_val (unsigned int val) +{ +return val; +} +bool resctrl_arch_get_cdp_enabled (enum resctrl_res_level rid) {return mpam_resctrl_controls[rid].cdp_enabled;
include/linux/arm_mpam.h +0 -5
@@ -76,11 +76,6 @@ static inline void resctrl_arch_disable_mon(void) { }static inline void resctrl_arch_enable_alloc (void ) { }static inline void resctrl_arch_disable_alloc (void ) { } -static inline unsigned int resctrl_arch_round_mon_val (unsigned int val) -{ -return val; -} -/** * mpam_register_requestor() - Register a requestor with the MPAM driver * @partid_max: The maximum PARTID value the requestor can generate.
include/linux/resctrl.h +1 -0
@@ -418,6 +418,7 @@ static inline u32 resctrl_get_default_ctrl(struct rdt_resource *r)u32 resctrl_arch_get_num_closid (struct rdt_resource *r);u32 resctrl_arch_system_num_rmid_idx (void );int resctrl_arch_update_domains (struct rdt_resource *r,u32 closid); +unsigned int resctrl_arch_round_mon_val (unsigned int val);bool resctrl_enable_mon_event (enum resctrl_event_id eventid,bool any_cpu,unsigned int binary_bits,void *arch_priv);
Signature · 1 line
2.25.1
PATCH v10 3/12 x86/resctrl: Require 64-bit x86 for resctrl support
In reply to Chen YuFrom: Tony Luck <[email protected]>
Intel Application Energy Telemetry (AET) provides event counts in 64-bit registers in MMIO space. For accurate results these should be read with a single 64-bit load operation from the CPU.
In preparation for the removal of the separate CONFIG_X86_CPU_RESCTRL_INTEL_AET configuration option, move the X86_64 dependency to X86_CPU_RESCTRL.
This is in line with Dave Hansen's proposal[1] for an addition to Linux documentation that details reduced 32-bit x86 support.
---
Changes since v9:
- This patch is picked up from the AET series so that both series carry identical versions, which is why its changelog refers to AET rather than ERDT. (Reinette Chatre)
- Add Reinette's Reviewed-by.
arch/x86/Kconfig | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/Kconfig +2 -2
@@ -515,7 +515,7 @@ config X86_MPPARSE config X86_CPU_RESCTRL bool "x86 CPU resource control support" - depends on X86 && (CPU_SUP_INTEL || CPU_SUP_AMD) + depends on X86_64 && (CPU_SUP_INTEL || CPU_SUP_AMD) depends on MISC_FILESYSTEMS select ARCH_HAS_CPU_RESCTRL select RESCTRL_FS
@@ -538,7 +538,7 @@ config X86_CPU_RESCTRL config X86_CPU_RESCTRL_INTEL_AET bool "Intel Application Energy Telemetry" - depends on X86_64 && X86_CPU_RESCTRL && CPU_SUP_INTEL && INTEL_PMT_TELEMETRY=y && INTEL_TPMI=y + depends on X86_CPU_RESCTRL && CPU_SUP_INTEL && INTEL_PMT_TELEMETRY=y && INTEL_TPMI=y help Enable per-RMID telemetry events in resctrl.
Signature · 1 line
2.25.1
PATCH v10 4/12 x86/resctrl: Detect Sub-NUMA Cluster mode before enumerating resources
In reply to Chen YuThe Sub-NUMA Cluster (SNC) configuration is currently detected only while setting up L3 monitoring, so it is unknown to code that runs earlier during resctrl initialization. Upcoming Enhanced RDT (ERDT) support needs to know whether SNC is enabled while parsing its ACPI table, before the monitoring resources are set up.
Detect SNC once at the start of resource enumeration so that its configuration is available to all of resctrl initialization. SNC is now also detected on systems without L3 monitoring, which is harmless because only monitoring depends on the SNC configuration.
---
Changes since v9:
- New patch, split out of v9 "x86/resctrl: Do not enable ERDT when Sub-NUMA Cluster is enabled". It only moves SNC detection to the start of resource enumeration. The ERDT/SNC check itself moves into "x86/resctrl: Parse ACPI ERDT table and save CACD cpumask per RMD". (Reinette Chatre)
- Drop the resctrl_arch_update_snc() helper: the resctrl_arch_ prefix is reserved for the functions declared in include/linux/resctrl.h. Move the definition of snc_nodes_per_l3_cache to core.c, declare it in internal.h, and open code its initialization in get_rdt_resources(). Existing users in monitor.c are unchanged. (Reinette Chatre)
- Describe the change at a high level in the changelog instead of narrating the code. (Reinette Chatre)
arch/x86/kernel/cpu/resctrl/core.c | 3 +++ arch/x86/kernel/cpu/resctrl/internal.h | 3 +++ arch/x86/kernel/cpu/resctrl/monitor.c | 6 +----- 3 files changed, 7 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +3 -0
@@ -51,6 +51,8 @@ DEFINE_PER_CPU(struct resctrl_pqr_state, pqr_state);*/ bool rdt_alloc_capable; +int snc_nodes_per_l3_cache =1 ; +static void mba_wrmsr_intel (struct msr_param *m);static void cat_wrmsr (struct msr_param *m);static void mba_wrmsr_amd (struct msr_param *m);
@@ -1016,6 +1018,7 @@ static __init void check_quirks(void)static __init bool get_rdt_resources (void ) { + snc_nodes_per_l3_cache =snc_get_config (); rdt_alloc_capable =get_rdt_alloc_resources (); rdt_mon_capable =get_rdt_mon_resources ();
arch/x86/kernel/cpu/resctrl/internal.h +3 -0
@@ -151,6 +151,8 @@ static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *rextern struct rdt_hw_resource rdt_resources_all[]; +extern int snc_nodes_per_l3_cache; +void arch_mon_domain_online (struct rdt_resource *r,struct rdt_l3_mon_domain *d);/* CPUID.(EAX=10H, ECX=ResID=1).EAX */
@@ -225,6 +227,7 @@ union l3_qos_abmc_cfg {void rdt_ctrl_update (void *arg);int rdt_get_l3_mon_config (struct rdt_resource *r); +__init int snc_get_config (void );bool rdt_cpu_has (int flag);
arch/x86/kernel/cpu/resctrl/monitor.c +1 -5
@@ -34,8 +34,6 @@ bool rdt_mon_capable;#define CF (cf) ((unsigned long)(1048576 * (cf) + 0.5)) -static int snc_nodes_per_l3_cache =1 ; -/* * The correction factor table is documented in Documentation/filesystems/resctrl.rst. * If rmid > rmid threshold, MBM total and local values should be multiplied
@@ -384,7 +382,7 @@ static const struct x86_cpu_id snc_cpu_ids[] __initconst = { {} }; -static __init int snc_get_config (void ) +__init int snc_get_config (void ) {int ret;
@@ -432,8 +430,6 @@ int __init rdt_get_l3_mon_config(struct rdt_resource *r)unsigned int threshold;u32 eax, ebx, ecx, edx; - snc_nodes_per_l3_cache =snc_get_config (); - resctrl_rmid_realloc_limit = boot_cpu_data.x86_cache_size *1024 ; hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale / snc_nodes_per_l3_cache; r->mon.num_rmid = (boot_cpu_data.x86_cache_max_rmid +1 ) / snc_nodes_per_l3_cache;
Signature · 1 line
2.25.1
PATCH v10 5/12 x86/resctrl: Parse ACPI ERDT table and save CACD cpumask per RMD
In reply to Chen YuEnhanced RDT (ERDT) exposes per-domain MMIO registers for RDT monitoring and control. A single per-platform ERDT ACPI table describes them.
Each Resource Management Domain Description (RMDD) sub-table within the ERDT table represents one resource management domain (RMD). When RMDD_FLAG_CPU_L3_DOMAIN is set, it identifies the domain as a CPU-scoped L3 domain. The set of CPUs in the domain is listed by x2APIC ID in the CPU Agent Collection Description (CACD) sub-table.
The hierarchy of these ACPI tables is illustrated below:
+---------------------------------------------------------------+ | ERDT (one instance per platform) | | | | +-------------------------------------------------------+ | | | RMDD #0 | | | | max_rmid | | | | | | | | +-----------------------------------------------+ | | | | | CACD | | | | | | x2APIC ID[0], x2APIC ID[1], ... | | | | | +-----------------------------------------------+ | | | +-------------------------------------------------------+ | | +-------------------------------------------------------+ | | | RMDD #1 | | | | ... | | | +-------------------------------------------------------+ | | ... | | +-------------------------------------------------------+ | | | RMDD #N-1 | | | +-------------------------------------------------------+ | +---------------------------------------------------------------+
Parse the RMDD sub-tables in preparation for attaching each ERDT domain to a resctrl L3 domain. For each CPU-based L3 RMDD, map its control registers and record the CPUs listed in its CACD sub-tables. Record the largest RMID that is valid on every RMD. Require all RMDDs to have the same set of sub-table types.
ERDT is not compatible with Sub-NUMA Cluster (SNC) mode: ERDT describes CPU monitoring resources only per L3 cache, while SNC divides an L3 cache into NUMA node scoped monitoring domains. Leave ERDT disabled when SNC is enabled so that resctrl keeps using the MSR interface.
---
Changes since v9:
- Fold in the ERDT/SNC check from v9 "x86/resctrl: Do not enable ERDT when Sub-NUMA Cluster is enabled" and note in the changelog that ERDT and SNC are not compatible. (Reinette Chatre)
- Make the SNC check the first check in enumerate_erdt_table() so that no parsing is done when SNC is enabled. (Reinette Chatre)
- Drop "CMT" from the SNC comment and message because the restriction is not specific to CMT, drop the comment sentence about where snc_nodes_per_l3_cache is computed, and print "ERDT not supported with Sub-NUMA Cluster enabled". (Reinette Chatre)
- Skip an RMDD whose CACD sub-tables do not list any CPU known to the kernel, e.g. when all of them are excluded by nr_cpus=, instead of adding an ERDT domain without CPUs. (Reinette Chatre)
- Report a CPU-based L3 RMDD without a CACD sub-table as a firmware bug.
- Map the RMDD control registers only after the RMDD is known to have CPUs. (Reinette Chatre)
- Changelog: drop the sentence about the CMRC sub-table and CMT counters, which are unrelated to this patch, drop domain_id from the diagram because this patch does not use it, and shorten the description of the change. (Reinette Chatre)
- Add erdt.o to the existing ctrlmondata.o line in the Makefile. (Reinette Chatre)
- Simplify the erdt_max_rmid comment to "Largest RMID that is valid in every ERDT domain." (Reinette Chatre)
- Use kernel-doc format for enum erdt_mmio_type, and add braces around the multi-line if body in erdt_ioremap().
arch/x86/kernel/cpu/resctrl/Makefile | 2 +- arch/x86/kernel/cpu/resctrl/core.c | 16 +- arch/x86/kernel/cpu/resctrl/erdt.c | 274 +++++++++++++++++++++++++ arch/x86/kernel/cpu/resctrl/internal.h | 30 +++ 4 files changed, 319 insertions(+), 3 deletions(-) create mode 100644 arch/x86/kernel/cpu/resctrl/erdt.c
arch/x86/kernel/cpu/resctrl/Makefile +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_X86_CPU_RESCTRL) += core.o rdtgroup.o monitor.o -obj-$(CONFIG_X86_CPU_RESCTRL) += ctrlmondata.o +obj-$(CONFIG_X86_CPU_RESCTRL) += ctrlmondata.o erdt.o obj-$(CONFIG_X86_CPU_RESCTRL_INTEL_AET) += intel_aet.o obj-$(CONFIG_RESCTRL_FS_PSEUDO_LOCK) += pseudo_lock.o
arch/x86/kernel/cpu/resctrl/core.c +14 -2
@@ -1018,11 +1018,18 @@ static __init void check_quirks(void)static __init bool get_rdt_resources (void ) { +bool ret; + snc_nodes_per_l3_cache =snc_get_config (); +erdt_init (); rdt_alloc_capable =get_rdt_alloc_resources (); rdt_mon_capable =get_rdt_mon_resources (); -return (rdt_mon_capable || rdt_alloc_capable); + ret = (rdt_mon_capable || rdt_alloc_capable); +if (!ret) +erdt_exit (); + +return ret; }static __init void rdt_init_res_defs_intel (void )
@@ -1144,12 +1151,15 @@ static int __init resctrl_arch_late_init(void)"x86/resctrl/cat:online:" , resctrl_arch_online_cpu, resctrl_arch_offline_cpu); -if (state <0 ) +if (state <0 ) { +erdt_exit ();return state; + } ret =resctrl_init ();if (ret) {cpuhp_remove_state (state); +erdt_exit ();return ret; } rdt_online = state;
@@ -1172,6 +1182,8 @@ static void __exit resctrl_arch_exit(void)cpuhp_remove_state (rdt_online);resctrl_exit (); + +erdt_exit (); }__exitcall (resctrl_arch_exit);
arch/x86/kernel/cpu/resctrl/erdt.c new file +274 -0
274 lines not on this page · raw message
arch/x86/kernel/cpu/resctrl/internal.h +30 -0
@@ -21,6 +21,33 @@#define RMID_VAL_UNAVAIL BIT_ULL(62) +/** +* enum erdt_mmio_type - Index into erdt_domain_info::base[] for each MMIO region +* @ERDT_MMIO_RMDD_CREG: RMDD control register base address +* @ERDT_MMIO_LAST: Last MMIO region type +*/ +enum erdt_mmio_type { +ERDT_MMIO_RMDD_CREG , +ERDT_MMIO_LAST =ERDT_MMIO_RMDD_CREG +}; + +#define ERDT_MMIO_NUM_TYPES (ERDT_MMIO_LAST + 1) + +/** +* struct erdt_domain_info - L3 domain ERDT information +* @base: Array of ioremapped MMIO region base addresses, indexed by ERDT_MMIO_* +* @cpu_mask: CPUs belonging to this resource management domain +* @dom_id: L3 cache ID shared by all CPUs in this domain +* (ERDT_DOMAIN_ID_UNSET if unset) +* @entry: Links into the global domain_info_list +*/ +struct erdt_domain_info { +void __iomem *base[ERDT_MMIO_NUM_TYPES ]; +struct cpumask cpu_mask; +int dom_id; +struct list_head entry; +}; +/* * With the above fields in use 62 bits remain in MSR_IA32_QM_CTR for * data to be returned. The counter width is discovered from the hardware
@@ -256,4 +283,7 @@ static inline void intel_aet_mon_domain_setup(int cpu, int id, struct rdt_resourstatic inline bool intel_handle_aet_option (bool force_off,char *tok) {return false; }#endif +void erdt_init (void ); +void erdt_exit (void ); +#endif /* _ASM_X86_RESCTRL_INTERNAL_H */
Signature · 1 line
2.25.1
PATCH v10 6/12 x86/resctrl: Attach ACPI ERDT information to L3 mon domain on CPU online
In reply to Chen YuEach ERDT domain provides MMIO registers to read the monitoring counters of the CPUs it covers. A monitor read is issued against a resctrl L3 monitoring domain, so the read path needs the matching ERDT domain to find those MMIO registers.
ERDT and CPUID enumerate CPU-to-L3-domain membership independently. CPUID leaf 4 describes the L3 cache topology, while the firmware CACD sub-table lists the CPUs of each ERDT domain. An L3 monitoring domain can only be associated with an ERDT domain if both views agree on the L3 domain of each of its CPUs.
ERDT is not enabled in Sub-NUMA Cluster mode, so the L3 monitoring domains have L3 cache scope and their IDs are L3 cache IDs.
When a CPU comes online, check that its ERDT domain and its CPUID L3 cache agree, and bind the ERDT domain to the CPU's L3 cache ID. On a mismatch, keep the CPU online but out of all resctrl domains. This matches how resctrl handles a failed domain creation: the CPU still joins the default group so tasks can run on it with valid CLOSID/RMID values, but it is not added to any domain's CPU mask.
When an L3 monitoring domain is created, associate it with the ERDT domain bound to its L3 cache ID.
---
Changes since v9:
- Do not fail the CPU online operation when ERDT and CPUID disagree on the CPU's L3 domain. Keep the CPU online and in the default group, but out of all resctrl domains, which matches how resctrl handles a failed domain creation. (Reinette Chatre)
- Rename erdt_try_bind_cpu() to erdt_try_bind_domain(). (Reinette Chatre)
- Describe what erdt_try_bind_domain() returns, instead of how the caller uses the result, and replace "covered by resctrl" with "added to any resctrl domain". (Reinette Chatre)
- Document in the erdt_l3_mon_domain_setup() comment why hw_dom->d_info always points to a valid ERDT domain while ERDT is enabled. (Reinette Chatre)
- Put the domain_list_lock comment about the ERDT binding in its own paragraph, and say that it serializes binding an ERDT domain to an L3 cache ID (erdt_domain_info::dom_id), not modifying domain_info_list. (Reinette Chatre)
- Rewrite the changelog to give the context before describing the change, use imperative mood, and make clear what "it" refers to. (Reinette Chatre)
- Use kernel-doc format for both function comments, with "Context:
Caller must hold domain_list_lock.", and reword the "Can't find L3 id" warning.
arch/x86/kernel/cpu/resctrl/core.c | 14 ++- arch/x86/kernel/cpu/resctrl/erdt.c | 123 +++++++++++++++++++++++++ arch/x86/kernel/cpu/resctrl/internal.h | 6 ++ 3 files changed, 141 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +12 -2
@@ -33,9 +33,12 @@* The rdt_resource's domain list is updated when this happens. Readers of * the domain list must either take cpus_read_lock(), or rely on an RCU * read-side critical section, to avoid observing concurrent modification. -* All writers take this mutex: +* All writers take this mutex. +* +* This mutex also serializes binding an ERDT domain to an L3 cache ID +* (erdt_domain_info::dom_id) when a CPU comes online. */ -static DEFINE_MUTEX (domain_list_lock); +DEFINE_MUTEX (domain_list_lock);/* * The cached resctrl_pqr_state is strictly per CPU and can never be
@@ -560,6 +563,8 @@ static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, structreturn ; } +erdt_l3_mon_domain_setup (&d->hdr); + err =resctrl_online_mon_domain (r, &d->hdr);if (err) {l3_mon_domain_free (hw_dom);
@@ -744,8 +749,13 @@ static int resctrl_arch_online_cpu(unsigned int cpu)struct rdt_resource *r;mutex_lock (&domain_list_lock); +/* Keep a CPU out of all domains if its ERDT and CPUID L3 domains conflict. */ +if (!erdt_try_bind_domain (cpu)) + goto unlock; +for_each_capable_rdt_resource (r)domain_add_cpu (cpu, r); +unlock:mutex_unlock (&domain_list_lock);clear_closid_rmid (cpu);
arch/x86/kernel/cpu/resctrl/erdt.c +123 -0
129 lines not on this page · raw message
arch/x86/kernel/cpu/resctrl/internal.h +6 -0
@@ -99,12 +99,14 @@ struct rdt_hw_ctrl_domain {* @arch_mbm_states: Per-event pointer to the MBM event's saved state. * An MBM event's state is an array of struct arch_mbm_state * indexed by RMID on x86. +* @d_info: ERDT table information of this domain * * Members of this structure are accessed via helpers that provide abstraction. */ struct rdt_hw_l3_mon_domain {struct rdt_l3_mon_domain d_resctrl;struct arch_mbm_state *arch_mbm_states[QOS_NUM_L3_MBM_EVENTS ]; +const struct erdt_domain_info *d_info; };static inline struct rdt_hw_ctrl_domain *resctrl_to_arch_ctrl_dom (struct rdt_ctrl_domain *r)
@@ -180,6 +182,8 @@ extern struct rdt_hw_resource rdt_resources_all[];extern int snc_nodes_per_l3_cache; +extern struct mutex domain_list_lock; +void arch_mon_domain_online (struct rdt_resource *r,struct rdt_l3_mon_domain *d);/* CPUID.(EAX=10H, ECX=ResID=1).EAX */
@@ -285,5 +289,7 @@ static inline bool intel_handle_aet_option(bool force_off, char *tok) { return fvoid erdt_init (void );void erdt_exit (void ); +bool erdt_try_bind_domain (int cpu); +void erdt_l3_mon_domain_setup (struct rdt_domain_hdr *hdr);#endif /* _ASM_X86_RESCTRL_INTERNAL_H */
Signature · 1 line
2.25.1
PATCH v10 7/12 x86/resctrl: Parse ACPI CMRC table
In reply to Chen YuThe CMRC (Cache Monitoring Registers for CPU Agents Description) sub-table of ERDT describes the MMIO registers used to read the LLC occupancy counters of an ERDT domain.
Reading the LLC occupancy of an RMID via MMIO, rather than via the IA32_QM_EVTSEL/IA32_QM_CTR MSRs, requires the following data from the CMRC sub-table:
- Where the counters are: the base address and size of the domain's monitoring register block.
- How the counters are laid out: the index function, clump size and clump stride, which locate the counter of a given RMID within that block.
- How to interpret a counter value: the flags, which indicate whether a counter can report that its data is unavailable, and up_scale, which converts a counter value to bytes.
The LLC occupancy read path needs all of this data on every read. The limbo handler also needs up_scale to round its occupancy threshold.
Parse each CMRC sub-table, validate the data above, ioremap the monitoring register block and save a copy of the CMRC sub-table in the corresponding ERDT domain entry.
---
Changes since v9:
- Rebased on the kernel-doc enum erdt_mmio_type from "x86/resctrl:
Parse ACPI ERDT table and save CACD cpumask per RMD". No code change.
arch/x86/kernel/cpu/resctrl/erdt.c | 53 ++++++++++++++++++++++++++ arch/x86/kernel/cpu/resctrl/internal.h | 6 ++- 2 files changed, 58 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/erdt.c +53 -0
@@ -23,6 +23,7 @@ static LIST_HEAD(domain_info_list);static bool erdt_enabled;#define ERDT_VALID_REVISION 1 +#define CMRC_SUPPORTED_INDEX_FN 1#define RMDD_FLAG_CPU_L3_DOMAIN BIT(0)#define ERDT_DOMAIN_ID_UNSET -1
@@ -65,6 +66,7 @@ static void erdt_iounmap_domain(struct erdt_domain_info *domain)static void cleanup_one_domain (struct erdt_domain_info *d) {erdt_iounmap_domain (d); +kfree (d->cmrc);kfree (d); }
@@ -98,6 +100,47 @@ static __init bool cacd_init(struct acpi_subtbl_hdr_16 *subtbl,return true; } +static __init bool cmrc_init (struct acpi_subtbl_hdr_16 *subtbl, +struct erdt_domain_info *domain_info) +{ +struct acpi_erdt_cmrc *cmrc = (struct acpi_erdt_cmrc *)subtbl; + +if (cmrc->header.length <sizeof (*cmrc)) { +pr_warn (FW_BUG "Truncated CMRC sub-table\n" ); +return false; + } + +if (cmrc->index_fn !=CMRC_SUPPORTED_INDEX_FN ) { +pr_info ("Unsupported CMRC index function %u\n" , cmrc->index_fn); +return false; + } + +if (!cmrc->clump_size) { +pr_warn (FW_BUG "CMRC clump_size is zero\n" ); +return false; + } + +/* resctrl scales monitoring values with an unsigned int. */ +if (cmrc->up_scale >UINT_MAX ) { +pr_warn (FW_BUG "Insane CMRC up_scale value 0x%llx\n" , cmrc->up_scale); +return false; + } + + domain_info->base[ERDT_MMIO_CMRC_BASE ] = +erdt_ioremap (cmrc->cmt_reg_base, cmrc->cmt_reg_size,"CMRC base" ); +if (!domain_info->base[ERDT_MMIO_CMRC_BASE ]) +return false; + + domain_info->cmrc =kmemdup (cmrc, cmrc->header.length,GFP_KERNEL ); +if (!domain_info->cmrc) { +iounmap (domain_info->base[ERDT_MMIO_CMRC_BASE ]); + domain_info->base[ERDT_MMIO_CMRC_BASE ] =NULL ; +return false; + } + +return true; +} +static inline struct acpi_subtbl_hdr_16 *next_subtbl (struct acpi_subtbl_hdr_16 *subtbl) {return (void *)subtbl + subtbl->length;
@@ -154,6 +197,16 @@ static __init bool parse_rmdd_table(struct acpi_subtbl_hdr_16 *rmdd_hdr) subtbl_mask |=BIT (ACPI_ERDT_TYPE_CACD );break ; +/* An RMDD table supports at most one CMRC sub-table */ +case ACPI_ERDT_TYPE_CMRC : +if (subtbl_mask &BIT (ACPI_ERDT_TYPE_CMRC )) +break ; + +if (!cmrc_init (subtbl, domain_info)) + goto cleanup; + + subtbl_mask |=BIT (ACPI_ERDT_TYPE_CMRC ); +break ;default :break ; }
arch/x86/kernel/cpu/resctrl/internal.h +5 -1
@@ -24,11 +24,13 @@/** * enum erdt_mmio_type - Index into erdt_domain_info::base[] for each MMIO region * @ERDT_MMIO_RMDD_CREG: RMDD control register base address +* @ERDT_MMIO_CMRC_BASE: CMRC monitoring register base address * @ERDT_MMIO_LAST: Last MMIO region type */ enum erdt_mmio_type {ERDT_MMIO_RMDD_CREG , -ERDT_MMIO_LAST =ERDT_MMIO_RMDD_CREG +ERDT_MMIO_CMRC_BASE , +ERDT_MMIO_LAST =ERDT_MMIO_CMRC_BASE };#define ERDT_MMIO_NUM_TYPES (ERDT_MMIO_LAST + 1)
@@ -36,6 +38,7 @@ enum erdt_mmio_type {/** * struct erdt_domain_info - L3 domain ERDT information * @base: Array of ioremapped MMIO region base addresses, indexed by ERDT_MMIO_* +* @cmrc: Copy of the ACPI CMRC sub-table for this domain * @cpu_mask: CPUs belonging to this resource management domain * @dom_id: L3 cache ID shared by all CPUs in this domain * (ERDT_DOMAIN_ID_UNSET if unset)
@@ -43,6 +46,7 @@ enum erdt_mmio_type {*/ struct erdt_domain_info {void __iomem *base[ERDT_MMIO_NUM_TYPES ]; +struct acpi_erdt_cmrc *cmrc;struct cpumask cpu_mask;int dom_id;struct list_head entry;
Signature · 1 line
2.25.1
PATCH v10 8/12 x86/resctrl: Refactor the monitor read function
In reply to Chen YuSplit the monitor read helper into an L3 read path and an AET (Intel Application Energy Telemetry) read path. This makes the two distinct monitoring sources easier to extend independently and prepares the L3 path for ERDT-based MMIO reads.
No functional change.
---
Changes since v9:
- No change.
arch/x86/kernel/cpu/resctrl/monitor.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-)
arch/x86/kernel/cpu/resctrl/monitor.c +18 -8
@@ -237,9 +237,8 @@ static u64 get_corrected_val(struct rdt_resource *r, struct rdt_l3_mon_domain *dreturn chunks * hw_res->mon_scale; } -int resctrl_arch_rmid_read (struct rdt_resource *r,struct rdt_domain_hdr *hdr, -u32 unused,u32 rmid,enum resctrl_event_id eventid, -void *arch_priv,u64 *val,void *ignored) +static int arch_l3_read_event (struct rdt_resource *r,struct rdt_domain_hdr *hdr, +u32 rmid,enum resctrl_event_id eventid,u64 *val) {struct rdt_hw_l3_mon_domain *hw_dom;struct rdt_l3_mon_domain *d;
@@ -249,11 +248,6 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,int cpu;int ret; -resctrl_arch_rmid_read_context_check (); - -if (r->rid ==RDT_RESOURCE_PERF_PKG ) -return intel_aet_read_event (hdr->id, rmid, arch_priv, val); -if (!domain_header_is_valid (hdr,RESCTRL_MON_DOMAIN ,RDT_RESOURCE_L3 ))return -EINVAL ;
@@ -279,6 +273,22 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,return ret; } +int resctrl_arch_rmid_read (struct rdt_resource *r,struct rdt_domain_hdr *hdr, +u32 unused,u32 rmid,enum resctrl_event_id eventid, +void *arch_priv,u64 *val,void *ignored) +{ +resctrl_arch_rmid_read_context_check (); + +switch (r->rid) { +case RDT_RESOURCE_L3 : +return arch_l3_read_event (r, hdr, rmid, eventid, val); +case RDT_RESOURCE_PERF_PKG : +return intel_aet_read_event (hdr->id, rmid, arch_priv, val); +default : +return -EINVAL ; + } +} +static int __cntr_id_read (u32 cntr_id,u64 *val) {struct msr msr_val = {
Signature · 1 line
2.25.1
PATCH v10 9/12 fs/resctrl: Do not invoke smp_processor_id() in preemptible context
In reply to Chen YuFrom: Tony Luck <[email protected]>
An event flagged with mon_evt::any_cpu is read directly in preemptible task context instead of being dispatched to a CPU of the monitoring domain. Only Intel AET events set this flag, so __mon_event_count() rejects an any_cpu L3 event with WARN_ON_ONCE(). The L3 read paths can therefore call smp_processor_id() unconditionally to check that the current CPU belongs to the domain being read.
MMIO-based LLC occupancy counters can be read from any CPU, so the L3 read paths have to tolerate an any_cpu event. Move that check into a helper which returns early for any_cpu events, avoiding smp_processor_id() in preemptible context.
---
Changes since v9:
- No code change.
fs/resctrl/monitor.c | 47 +++++++++++++++++++++++++++++++------------- 1 file changed, 33 insertions(+), 14 deletions(-)
fs/resctrl/monitor.c +33 -14
@@ -422,9 +422,36 @@ static void mbm_cntr_free(struct rdt_l3_mon_domain *d, int cntr_id)memset (&d->cntr_cfg[cntr_id],0 ,sizeof (*d->cntr_cfg)); } +/** +* cpu_can_read_event() - Check if the current CPU can read the event. +* @rr: The rmid_read structure containing event and domain information. +* +* Context: Preemptible process context when @rr->evt->any_cpu is set. +* Non-migratable process context (via smp_call_on_cpu()) or +* non-preemptible context (via smp_call_function_any()) when +* the event must be read on a specific CPU. +* Return: true if the current CPU can read this event, false otherwise. +*/ +static bool cpu_can_read_event (struct rmid_read *rr) +{ +int cpu; + +/* Any CPU is OK for this event */ +if (rr->evt->any_cpu) +return true; + + cpu =smp_processor_id (); + +/* Single domain. Must be on a CPU in that domain. */ +if (rr->hdr) +return cpumask_test_cpu (cpu, &rr->hdr->cpu_mask); + +/* Summing domains that share a cache, must be on a CPU for that cache. */ +return cpumask_test_cpu (cpu, &rr->ci->shared_cpu_map); +} +static int __l3_mon_event_count (struct rdtgroup *rdtgrp,struct rmid_read *rr) { -int cpu =smp_processor_id ();u32 closid = rdtgrp->closid;u32 rmid = rdtgrp->mon.rmid;struct rdt_l3_mon_domain *d;
@@ -457,11 +484,6 @@ static int __l3_mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)return 0 ; } -/* Reading a single domain, must be on a CPU in that domain. */ -if (!cpumask_test_cpu (cpu, &d->hdr.cpu_mask)) { - rr->err = -EIO ; -return -EINVAL ; - }if (rr->is_mbm_cntr) rr->err =resctrl_arch_cntr_read (rr->r, d, closid, rmid, cntr_id, rr->evt->evtid, &tval);
@@ -479,7 +501,6 @@ static int __l3_mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)static int __l3_mon_event_count_sum (struct rdtgroup *rdtgrp,struct rmid_read *rr) { -int cpu =smp_processor_id ();u32 closid = rdtgrp->closid;u32 rmid = rdtgrp->mon.rmid;struct rdt_l3_mon_domain *d;
@@ -497,12 +518,6 @@ static int __l3_mon_event_count_sum(struct rdtgroup *rdtgrp, struct rmid_read *rreturn -EINVAL ; } -/* Summing domains that share a cache, must be on a CPU for that cache. */ -if (!cpumask_test_cpu (cpu, &rr->ci->shared_cpu_map)) { - rr->err = -EIO ; -return -EINVAL ; - } -/* * Legacy files must report the sum of an event across all * domains that share the same L3 cache instance.
@@ -538,7 +553,11 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) {switch (rr->r->rid) {case RDT_RESOURCE_L3 : -WARN_ON_ONCE (rr->evt->any_cpu); +if (!cpu_can_read_event (rr)) { + rr->err = -EIO ; +return -EINVAL ; + } +if (rr->hdr)return __l3_mon_event_count (rdtgrp, rr);else
Signature · 1 line
2.25.1
PATCH v10 10/12 x86/resctrl: Introduce erdt_cpu_has() and erdt_support()
In reply to Chen YuIntroduce erdt_cpu_has() to verify if a specific RDT feature is backed by an ERDT table. A feature is only reported as ERDT backed when the CPU enumerates it via CPUID and firmware describes it in the ERDT table (and its sub-tables). erdt_cpu_has() also considers userspace input like "rdt=!cmt". Similar to rdt_cpu_has(), erdt_cpu_has() expects an x86 feature flag as parameter.
Introduce erdt_support() as a helper called by erdt_cpu_has() to check whether the ERDT firmware supports a given feature.
erdt_support() deliberately claims no feature yet and returns false. Features should be added to erdt_support() as their MMIO implementations are introduced.
---
Changes since v9:
- Drop the erdt_get_scale() context: the ERDT scale is now introduced after this patch by "x86/resctrl: Add the ERDT CMT scale". No code change. (Reinette Chatre)
arch/x86/kernel/cpu/resctrl/core.c | 13 +++++++++++-- arch/x86/kernel/cpu/resctrl/erdt.c | 5 +++++ arch/x86/kernel/cpu/resctrl/internal.h | 2 ++ 3 files changed, 18 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +11 -2
@@ -873,9 +873,8 @@ static int __init set_rdt_options(char *str) }__setup ("rdt" , set_rdt_options); -bool rdt_cpu_has (int flag) +static bool apply_rdt_options (int flag,bool ret) { -bool ret =boot_cpu_has (flag);struct rdt_options *o;if (!ret)
@@ -893,6 +892,16 @@ bool rdt_cpu_has(int flag)return ret; } +bool rdt_cpu_has (int flag) +{ +return apply_rdt_options (flag,boot_cpu_has (flag)); +} + +bool erdt_cpu_has (int flag) +{ +return apply_rdt_options (flag,boot_cpu_has (flag) &&erdt_support (flag)); +} +bool resctrl_arch_is_evt_configurable (enum resctrl_event_id evt) {if (!rdt_cpu_has (X86_FEATURE_BMEC ))
arch/x86/kernel/cpu/resctrl/erdt.c +5 -0
@@ -36,6 +36,11 @@ static u16 first_rmdd_domain_id;/* Largest RMID that is valid in every ERDT domain. */ static unsigned int erdt_max_rmid; +bool erdt_support (int flag) +{ +return false; +} +static void __iomem *erdt_ioremap (resource_size_t base,u32 num_pages,const char *desc) {void __iomem *addr;
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
@@ -291,6 +291,8 @@ static inline void intel_aet_mon_domain_setup(int cpu, int id, struct rdt_resourstatic inline bool intel_handle_aet_option (bool force_off,char *tok) {return false; }#endif +bool erdt_cpu_has (int flag); +bool erdt_support (int flag);void erdt_init (void );void erdt_exit (void );bool erdt_try_bind_domain (int cpu);
Signature · 1 line
2.25.1
PATCH v10 11/12 x86/resctrl: Add the ERDT CMT scale
In reply to Chen YuThe limbo handler polls the LLC occupancy of limbo RMIDs and makes an RMID available for reuse once its occupancy drops below resctrl_rmid_realloc_threshold. User space sets this threshold in bytes by writing to the "max_threshold_occupancy" file. The occupancy counter only has the granularity of its scale, the number of bytes one counter unit represents, so resctrl_arch_round_mon_val() rounds the written byte value down to a multiple of that scale before it is stored in resctrl_rmid_realloc_threshold.
resctrl_arch_round_mon_val() uses the CPUID-provided scale. LLC occupancy read via ERDT is instead scaled by the CMRC up_scale of each ERDT domain. resctrl_rmid_realloc_threshold is a single global value, so one scale has to be derived from the per-domain values.
Add erdt_cmt_scale to hold the largest CMRC up_scale among all ERDT domains. Use the largest because rounding is a floor: a larger scale yields a slightly lower threshold, i.e. an RMID has to drop to a slightly lower occupancy before it is reused.
---
Changes since v9:
- New patch, replacing v9 "x86/resctrl: Add a global ERDT monitor scale". Place it after the CMRC parsing and erdt_cpu_has() patches so that the scale is added in one coherent change instead of as a placeholder that stays 0 until the last patch. (Reinette Chatre)
- Rename erdt_scale/erdt_get_scale() to erdt_cmt_scale and erdt_get_cmt_scale() so that the names say this is the CMT scale and not a global monitor scale. Move the erdt_cpu_has() check from the last patch into erdt_get_cmt_scale(). (Reinette Chatre)
- Compute erdt_cmt_scale from the CMRC up_scale of each RMDD that is accepted (moved here from the last patch), and reset it in erdt_exit(). (Reinette Chatre)
- Squash in the fix for the stale "h/w works in units of boot_cpu_data.x86_cache_occ_scale" comment from the last patch. (Reinette Chatre)
- Changelog: say "limbo RMIDs" instead of "stale RMIDs", distinguish the "max_threshold_occupancy" file from the resctrl_rmid_realloc_threshold variable, and state that the threshold is in bytes. (Reinette Chatre)
arch/x86/kernel/cpu/resctrl/erdt.c | 23 +++++++++++++++++++++++ arch/x86/kernel/cpu/resctrl/internal.h | 1 + arch/x86/kernel/cpu/resctrl/monitor.c | 7 +++++-- 3 files changed, 29 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/erdt.c +23 -0
@@ -36,11 +36,30 @@ static u16 first_rmdd_domain_id;/* Largest RMID that is valid in every ERDT domain. */ static unsigned int erdt_max_rmid; +/* +* Used only to round the global resctrl_rmid_realloc_threshold for the limbo +* handler. resctrl_rmid_realloc_threshold is exposed to user space as a single +* global value, so ERDT uses one scale derived from the per-domain +* cmrc->up_scale values. Keep this single threshold conservative across +* domains because the rounding is a floor: a larger scale yields a slightly +* lower threshold, i.e. an RMID has to drop to a slightly lower occupancy +* before it is reused. +*/ +static unsigned int erdt_cmt_scale; +bool erdt_support (int flag) {return false; } +unsigned int erdt_get_cmt_scale (void ) +{ +if (!erdt_cpu_has (X86_FEATURE_CQM_OCCUP_LLC )) +return 0 ; + +return erdt_cmt_scale; +} +static void __iomem *erdt_ioremap (resource_size_t base,u32 num_pages,const char *desc) {void __iomem *addr;
@@ -256,6 +275,9 @@ static __init bool parse_rmdd_table(struct acpi_subtbl_hdr_16 *rmdd_hdr)else erdt_max_rmid =min (erdt_max_rmid, rmdd->max_rmid); +if (domain_info->cmrc) + erdt_cmt_scale =max (erdt_cmt_scale, domain_info->cmrc->up_scale); +list_add (&domain_info->entry, &domain_info_list);return true;
@@ -400,6 +422,7 @@ void erdt_exit(void) valid_subtbl_mask =0 ; first_rmdd_domain_id =0 ; erdt_max_rmid =0 ; + erdt_cmt_scale =0 ; }static __init int enumerate_erdt_table (struct acpi_table_header *table_hdr)
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
@@ -293,6 +293,7 @@ static inline bool intel_handle_aet_option(bool force_off, char *tok) { return fbool erdt_cpu_has (int flag);bool erdt_support (int flag); +unsigned int erdt_get_cmt_scale (void );void erdt_init (void );void erdt_exit (void );bool erdt_try_bind_domain (int cpu);
arch/x86/kernel/cpu/resctrl/monitor.c +5 -2
@@ -426,9 +426,12 @@ __init int snc_get_config(void)unsigned int resctrl_arch_round_mon_val (unsigned int val) { -unsigned int scale = boot_cpu_data.x86_cache_occ_scale; +unsigned int scale =erdt_get_cmt_scale (); -/* h/w works in units of "boot_cpu_data.x86_cache_occ_scale" */ +if (!scale) + scale = boot_cpu_data.x86_cache_occ_scale; + +/* h/w works in units of the occupancy scale */ val /= scale;return val * scale; }
Signature · 1 line
2.25.1
PATCH v10 12/12 x86/resctrl: Add MMIO-based LLC occupancy monitoring support
In reply to Chen YuAdd erdt_mon_read() to read LLC occupancy via MMIO and use it when the platform supports ERDT. Fall back to the MSR-based path if ERDT is unavailable on this platform. Mark the LLC occupancy event as readable from any CPU when it is read via MMIO.
When LLC occupancy is read via MMIO, limit the number of RMIDs to what every ERDT domain supports. resctrl uses a single number of RMIDs for all monitoring events, so this limit applies to the MSR-based MBM events as well. When LLC occupancy is not read via MMIO, e.g. with rdt=!cmt or without CMRC, no event is read via ERDT and the number of RMIDs is determined by CPUID alone.
Keep the per domain limbo handler even though the occupancy counters can now be read from any CPU via MMIO. A single global handler would still have to check the limbo RMIDs of every L3 domain, so it would do the same amount of work. The only saving would be running one limbo worker instead of one per L3 domain, which does not justify maintaining two limbo handler models.
---
Changes since v9:
- Make erdt_get_max_rmid() depend on erdt_enabled instead of erdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC): RMDD max_rmid limits all RMIDs, not only those used for LLC occupancy. Say so in the changelog. (Reinette Chatre)
- Move the erdt_cmt_scale update, the erdt_get_cmt_scale() check, and the resctrl_arch_round_mon_val() comment fix to "x86/resctrl: Add the ERDT CMT scale". (Reinette Chatre)
- Fix the CMRC register size check: cast cmrc->cmt_reg_size to u64 instead of u32 so that the multiplication by SZ_4K is not truncated. (Reinette Chatre)
- Use WARN_ON_ONCE() when an L3 monitoring domain has no ERDT domain attached. This should not happen.
- Do not shadow the erdt_max_rmid global in rdt_get_l3_mon_config():
rename the local to erdt_rmid_limit, make max_rmid unsigned and initialize it from CPUID, use min(), and keep reverse fir tree order. (Reinette Chatre)
- Document that hw_res->mon_scale is not used when LLC occupancy is read via ERDT.
- Changelog: start the limbo handler paragraph with "Keep the per domain limbo handler" and explain the "nr_domains - 1" saving in plain English. (Reinette Chatre)
arch/x86/kernel/cpu/resctrl/core.c | 4 +- arch/x86/kernel/cpu/resctrl/erdt.c | 83 +++++++++++++++++++++++++- arch/x86/kernel/cpu/resctrl/internal.h | 5 +- arch/x86/kernel/cpu/resctrl/monitor.c | 12 +++- 4 files changed, 100 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +3 -1
@@ -990,7 +990,9 @@ static __init bool get_rdt_mon_resources(void)bool ret = false;if (rdt_cpu_has (X86_FEATURE_CQM_OCCUP_LLC )) { -resctrl_enable_mon_event (QOS_L3_OCCUP_EVENT_ID , false,0 ,NULL ); +resctrl_enable_mon_event (QOS_L3_OCCUP_EVENT_ID , +erdt_cpu_has (X86_FEATURE_CQM_OCCUP_LLC ), +0 ,NULL ); ret = true; }if (rdt_cpu_has (X86_FEATURE_CQM_MBM_TOTAL )) {
arch/x86/kernel/cpu/resctrl/erdt.c +82 -1
101 lines not on this page · raw message
arch/x86/kernel/cpu/resctrl/internal.h +4 -1
@@ -155,7 +155,8 @@ struct msr_param {* which has been corrected for features like CDP. * @msr_base: Base MSR address for CBMs * @msr_update: Function pointer to update QOS MSRs -* @mon_scale: cqm counter * mon_scale = occupancy in bytes +* @mon_scale: cqm counter * mon_scale = occupancy in bytes. +* Not used when LLC occupancy is read via ERDT. * @mbm_width: Monitor width, to detect and correct for overflow. * @cdp_enabled: CDP state of this resource * @mbm_cntr_assign_enabled: ABMC feature is enabled
@@ -294,6 +295,8 @@ static inline bool intel_handle_aet_option(bool force_off, char *tok) { return fbool erdt_cpu_has (int flag);bool erdt_support (int flag);unsigned int erdt_get_cmt_scale (void ); +unsigned int erdt_get_cmt_max_rmid (void ); +int erdt_mon_read (struct rdt_domain_hdr *hdr,enum resctrl_event_id evtid,u32 rmid,u64 *val);void erdt_init (void );void erdt_exit (void );bool erdt_try_bind_domain (int cpu);
arch/x86/kernel/cpu/resctrl/monitor.c +11 -1
@@ -281,6 +281,10 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,switch (r->rid) {case RDT_RESOURCE_L3 : +if (eventid ==QOS_L3_OCCUP_EVENT_ID && +erdt_cpu_has (X86_FEATURE_CQM_OCCUP_LLC )) +return erdt_mon_read (hdr, eventid, rmid, val); +return arch_l3_read_event (r, hdr, rmid, eventid, val);case RDT_RESOURCE_PERF_PKG :return intel_aet_read_event (hdr->id, rmid, arch_priv, val);
@@ -439,13 +443,19 @@ unsigned int resctrl_arch_round_mon_val(unsigned int val)int __init rdt_get_l3_mon_config (struct rdt_resource *r) {unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset; +unsigned int max_rmid = boot_cpu_data.x86_cache_max_rmid;struct rdt_hw_resource *hw_res =resctrl_to_arch_res (r); +unsigned int erdt_rmid_limit;unsigned int threshold;u32 eax, ebx, ecx, edx; + erdt_rmid_limit =erdt_get_cmt_max_rmid (); +if (erdt_rmid_limit) + max_rmid =min (max_rmid, erdt_rmid_limit); + resctrl_rmid_realloc_limit = boot_cpu_data.x86_cache_size *1024 ; hw_res->mon_scale = boot_cpu_data.x86_cache_occ_scale / snc_nodes_per_l3_cache; - r->mon.num_rmid = (boot_cpu_data.x86_cache_max_rmid +1 ) / snc_nodes_per_l3_cache; + r->mon.num_rmid = (max_rmid +1 ) / snc_nodes_per_l3_cache; hw_res->mbm_width =MBM_CNTR_WIDTH_BASE ;if (mbm_offset >0 && mbm_offset <=MBM_CNTR_WIDTH_OFFSET_MAX )
Signature · 1 line
2.25.1