Skip to contentSkip to conversation

x86,fs/resctrl telemetry monitoring

PATCH Superseded Tony Luck 2 participants 64 messages Oct 13, 2025 – Oct 28, 2025

Subsystems (3)

v12 of 17 Superseded Latest is v17 →

lore
Share & tools
lore

Tony Luck

author maintainer
#0 lore raw

v12 cover x86,fs/resctrl telemetry monitoring

Patches based on tip tree x86/urgent branch

Top commit of x86/urgent at time of my snapshot was:
15292f1b4c55 ("x86/resctrl: Fix miscount of bandwidth event when reactivating previously unavailable RMID")

Series available here:
git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git rdt-aet-v12

Changes since v11 was posted here:

My usual convention, patches from v11 named "XX/28", patches in this series when they have a different number as "00YY".

--- [PATCH v11 00/28] x86,fs/resctrl telemetry monitoring --- "IA32_PQR_ASSOC.RMID" -> "MSR_IA32_PQR_ASSOC.RMID

--- [PATCH v11 01/28] x86,fs/resctrl: Improve domain type checking --- Replace commit message with version suggested by Reinette

--- [PATCH v11 02/28] x86/resctrl: Move L3 initialization into new helper --- Added Reinette RB tag

--- [PATCH v11 03/28] x86,fs/resctrl: Refactor domain_remove_cpu_mon() --- Fix Subject. s/fs,x86/x86/
"L3 processing" -> "L3 domain processing"
"general actions" -> "general domain actions"
Move L3 specific local variable declarations into "case RDT_RESOURCE_L3:"

--- [PATCH v11 04/28] x86/resctrl: Clean up domain_remove_cpu_ctrl() --- Reformat commit comment to use more page width.

--- [PATCH v11 05/28] x86,fs/resctrl: Refactor domain create/remove using --- Replace commit message with version supplied by Reinette.
Fix mon_event_read(). If hdr is NULL, then "d" must be set to NULL. Defer SNC related changes in mon_add_all_files() and mkdir_mondata_subdir() to later patch.
Limit change in resctrl_offline_mon_domain() to check for valid RDT_RESOURCE_L3 header (defer until adding RDT_RESOURCE_PERF_PKG). Move superfluous "err = 0" from resctrl_online_mon_domain() to patch 0022 where it is needed after "goto mkdir;" is added.

--- [PATCH v11 06/28] x86,fs/resctrl: Use struct rdt_domain_hdr when --- Handle "hdr == NULL" case in __mon_event_count(). Note that hdr is NULL for the SNC "sum domains" case. That's orthogonal to both rr->is_mbm_cntr and rr->first. Duplicated code will be cleaned up in later patch.

Update commit message with explanation of use of rmid_read structure.

--- [PATCH v11 07/28] x86,fs/resctrl: Rename struct rdt_mon_domain and --- Added Reinette RB tag

--- [PATCH v11 08/28] x86,fs/resctrl: Rename some L3 specific functions --- Pulled in kerneldoc changes that fix Return: for two of the renamed functions from patch 31.
Added Reinette RB tag

--- [PATCH v11 09/28] fs/resctrl: Make event details accessible to --- "more context" -> "more context than the event id" "mon_evt" -> "struct mon_evt"
Added Reinette RB tag

--- [PATCH v11 10/28] x86,fs/resctrl: Handle events that can be read from --- Replace 2nd paragraph of commit message with Reinette re-write. Drop 3rd paragraph (just describes code change).
Replace 4th paragraph of commit message with Reinette re-write.

--- [PATCH v11 11/28] x86,fs/resctrl: Support binary fixed point event --- Unchanged.

--- [PATCH v11 12/28] x86,fs/resctrl: Add an architectural hook called --- Unchanged.

--- [PATCH v11 13/28] x86,fs/resctrl: Add and initialize rdt_resource for --- Merged in the kerneldoc changes for mon_get_kn_priv(...do_sum) and mon_data::sum from v11 patch 31
Added some WARN calls to confirm implicitly operating on RDT_RESOURCE_L3.

--- [PATCH v11 14/28] x86/resctrl: Discover hardware telemetry events --- Modify get_pmt_feature() to handle the case where intel_pmt_get_regions_by_feature() returns a list of telemetry regions with more than one guid known to resctrl by getting a separate copy for each guid.
"IA32_PQR_ASSOC.RMID" -> "MSR_IA32_PQR_ASSOC.RMID"

--- [PATCH v11 15/28] x86,fs/resctrl: Fill in details of events for guid --- Unchanged.

--- [PATCH v11 16/28] x86,fs/resctrl: Add architectural event pointer --- Replace commit message with suggested text by Reinette.

--- [PATCH v11 17/28] x86/resctrl: Find and enable usable telemetry --- Merge in mark_telem_region_unusable() function from patch 19.

--- PATCH v11 18/31] fs/resctrl: Refactor L3 specific parts of __mon_event_count() --- Updated Subject.
Add checks for !rr->hdr before using in container_of().
Split out the L3 dependent code into a helper function.
Added Suggested-by; Reinette tag.

--- PATCH v11 19/31] x86/resctrl: Read telemetry events --- Changes based on earlier updates to this series:
1) intel_aet_read_event() now checks for correct guid since more than one might be enabled because of changes in patch 11.
2) Add RDT_RESOURCE_PERF_PKG case to __mon_event_count()

--- PATCH v11 20/31] fs/resctrl: Refactor Sub-NUMA Cluster (SNC) in ---
NEW: 0020: Split mkdir_mondata_subdir() into SNC/non-SNC helpers
NEW: 0021: Split rmdir_mondata_subdir_allrdtgrp() into SNC/non-SNC helpers

--- PATCH v11 21/31] x86/resctrl: Handle domain creation/deletion for --- Now 0022:
Move declaration of struct rdt_perf_pkg_mon_domain *pkgd inside case RDT_RESOURCE_PERF_PKG in domain_remove_cpu_mon()
Initialize rdt_domain_hdr::rid using RDT_RESOURCE_PERF_PKG Drop the L3 checks in rdtgroup_mondata_show() and mon_get_kn_priv() that are unrelated to the subject of this patch.

--- PATCH v11 22/31] x86/resctrl: Add energy/perf choices to rdt boot --- Now 0023: No change.

--- PATCH v11 23/31] x86/resctrl: Handle number of RMIDs supported by --- Now 0024
Subject: "telemetry resources" -> "RDT_RESOURCE_PERF_PKG"
"the IA32_PQR_ASSOC MSR" -> "MSR_IA32_PQR_ASSOC"
Clarify under-resources event groups by adding "for any of the event group's telemetry regions"
Replace text from "The request to disable..." with Reinette suggested version.
Don't run skip_telem_region() tests multiple times. Just mark regions unusable once.
Update comment about adjustment of e->num_rmids.

--- PATCH v11 24/31] fs/resctrl: Move allocation/free of --- 0025: Replace commit comment with Reinette version.

--- PATCH v11 25/31] fs,x86/resctrl: Compute number of RMIDs as minimum --- 0026: s/fs,x86/x86,fs/ in Subject:
Added Reinette RB tag

--- PATCH v11 26/31] fs/resctrl: Move RMID initialization to first mount --- 0027:
Move initialization of idx_limit in setup_rmid_lru_list() after checks for resctrl_arch_mon_capable() and rmid_ptrs.
Added comment that rmid_ptrs[] may still be accessed by the limbo handler after resctrl is unmounted.

--- PATCH v11 27/31] x86/resctrl: Enable RDT_RESOURCE_PERF_PKG --- 0028:
Move setting of r->mon_capable and rdt_mon_capable from enable_events() to resctrl_arch_pre_mount() inside cpus_read_lock() to avoid double add of CPUs if a hot add event happens during telemetry enumeration. Drop the test for r->mon_capable in resctrl_arch_pre_mount() Replace commit message with Reinette version.

--- PATCH v11 28/31] fs/resctrl: Provide interface to create --- 0029: No change

--- PATCH v11 29/31] x86/resctrl: Add debugfs files to show telemetry --- 0030: Move initialization of debugs to resctrl_arch_pre_mount() [after the resource is marked mon_capable because of change in 0028 Change type of pkg argument to make_status_files() to u8 and sprintf() with %u

--- PATCH v11 30/31] x86,fs/resctrl: Update Documentation for package --- 0031:
Subject: "Documentation" -> "documentation" and "package events" -> "telemetry events"

Added details of how the "num_rmids" file for telemetry events is based on supported counters in each type of telemetry event. Tied it to the "rdt=" boot option.

Replaced "set" terminology with separate descriptions of mon_L3 and mon_PERF_PKG directories.

"these two counters" -> ""core energy" and "activity""

Increased line length of added lines to match rest of file.

--- PATCH v11 31/31] fs/resctrl: Some kerneldoc updates --- Dropped. Kerneldoc changes merged into patches 0008 and 0013

Background
----------
On Intel systems that support per-RMID telemetry monitoring each logical processor keeps a local count for various events. When the MSR_IA32_PQR_ASSOC.RMID value for the logical processor changes (or when a two millisecond counter expires) these event counts are transmitted to an event aggregator on the same package as the processor together with the current RMID value. The event counters are reset to zero to begin counting again.

Each aggregator takes the incoming event counts and adds them to cumulative counts for each event for each RMID. Note that there can be multiple aggregators on each package with no architectural association between logical processors and an aggregator.

All of these aggregated counters can be read by an operating system from the MMIO space of the Out Of Band Management Service Module (OOBMSM) device(s) on a system. Any counter can be read from any logical processor.

Intel publishes details for each processor generation showing which events are counted by each logical processor and the offsets for each accumulated counter value within the MMIO space in XML files here:
https://github.com/intel/Intel-PMT.

For example there are two energy related telemetry events for the Clearwater Forest family of processors and the MMIO space looks like this:

Offset  RMID    Event
------  ----    -----
0x0000  0       core_energy
0x0008  0       activity
0x0010  1       core_energy
0x0018  1       activity
...
0x23F0  575     core_energy
0x23F8  575     activity

In addition the XML file provides the units (Joules for core_energy, Farads for activity) and the type of data (fixed-point binary with bit 63 used to indicate the data is valid, and the low 18 bits as a binary fraction).

Finally, each XML file provides a 32-bit unique id (or guid) that is used as an index to find the correct XML description file for each telemetry implementation.

The INTEL_PMT_TELEMETRY driver provides intel_pmt_get_regions_by_feature() to enumerate the aggregator instances (also referred to as "telemetry regions" in this series) on a platform. It provides:

1) guid - so resctrl can determine which events are supported
2) MMIO base address of counters
3) package id

Resctrl accumulates counts from all aggregators on a package in order to provide a consistent user interface across processor generations.

Directory structure for the telemetry events looks like this:

$ tree /sys/fs/resctrl/mon_data/
/sys/fs/resctrl/mon_data/
mon_data
├── mon_PERF_PKG_00
│   ├── activity
│   └── core_energy
└── mon_PERF_PKG_01
    ├── activity
    └── core_energy

Reading the "core_energy" file from some resctrl mon_data directory shows the cumulative energy (in Joules) used by all tasks that ran with the RMID associated with that directory on a given package. Note that "core_energy" reports only energy consumed by CPU cores (data processing units, L1/L2 caches, etc.). It does not include energy used in the "uncore" (L3 cache, on package devices, etc.), or used by memory or I/O devices.

Tony Luck (31):
x86,fs/resctrl: Improve domain type checking x86/resctrl: Move L3 initialization into new helper function x86/resctrl: Refactor domain_remove_cpu_mon() ready for new domain
types
x86/resctrl: Clean up domain_remove_cpu_ctrl() x86,fs/resctrl: Refactor domain create/remove using struct
rdt_domain_hdr
x86,fs/resctrl: Use struct rdt_domain_hdr when reading counters x86,fs/resctrl: Rename struct rdt_mon_domain and rdt_hw_mon_domain x86,fs/resctrl: Rename some L3 specific functions fs/resctrl: Make event details accessible to functions when reading
events
x86,fs/resctrl: Handle events that can be read from any CPU x86,fs/resctrl: Support binary fixed point event counters x86,fs/resctrl: Add an architectural hook called for each mount x86,fs/resctrl: Add and initialize rdt_resource for package scope
monitor
x86/resctrl: Discover hardware telemetry events x86,fs/resctrl: Fill in details of events for guid 0x26696143 and

    0x26557651

x86,fs/resctrl: Add architectural event pointer x86/resctrl: Find and enable usable telemetry events fs/resctrl: Split L3 dependent parts out of __mon_event_count() x86/resctrl: Read telemetry events
fs/resctrl: Refactor mkdir_mondata_subdir()
fs/resctrl: Refactor rmdir_mondata_subdir_allrdtgrp() x86,fs/resctrl: Handle domain creation/deletion for
RDT_RESOURCE_PERF_PKG
x86/resctrl: Add energy/perf choices to rdt boot option x86/resctrl: Handle number of RMIDs supported by RDT_RESOURCE_PERF_PKG fs/resctrl: Move allocation/free of closid_num_dirty_rmid[] x86,fs/resctrl: Compute number of RMIDs as minimum across resources fs/resctrl: Move RMID initialization to first mount x86/resctrl: Enable RDT_RESOURCE_PERF_PKG
fs/resctrl: Provide interface to create architecture specific debugfs
area
x86/resctrl: Add debugfs files to show telemetry aggregator status x86,fs/resctrl: Update documentation for telemetry events

 .../admin-guide/kernel-parameters.txt         |   2 +-
 Documentation/filesystems/resctrl.rst         | 102 ++++-
 include/linux/resctrl.h                       |  69 ++-
 include/linux/resctrl_types.h                 |  11 +
 arch/x86/kernel/cpu/resctrl/internal.h        |  52 ++-
 fs/resctrl/internal.h                         |  68 ++-
 arch/x86/kernel/cpu/resctrl/core.c            | 275 +++++++++---
 arch/x86/kernel/cpu/resctrl/intel_aet.c       | 423 ++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/monitor.c         |  55 ++-
 fs/resctrl/ctrlmondata.c                      | 121 ++++-
 fs/resctrl/monitor.c                          | 356 +++++++++------
 fs/resctrl/rdtgroup.c                         | 261 ++++++++---
 arch/x86/Kconfig                              |  13 +
 arch/x86/kernel/cpu/resctrl/Makefile          |   1 +
 14 files changed, 1433 insertions(+), 376 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/resctrl/intel_aet.c

base-commit: 15292f1b4c55a3a7c940dbcb6cb8793871ed3d92

Signature · 1 line
2.51.0
author maintainer
#1 lore raw

PATCH v12 1/31 x86,fs/resctrl: Improve domain type checking

Every resctrl resource has a list of domain structures. struct rdt_ctrl_domain and struct rdt_mon_domain both begin with struct rdt_domain_hdr with rdt_domain_hdr::type used in validity checks before accessing the domain of a particular type.

Add the resource id to struct rdt_domain_hdr in preparation for a new monitoring domain structure that will be associated with a new monitoring resource. Improve existing domain validity checks with a new helper domain_header_is_valid() that checks both domain type and resource id. domain_header_is_valid() should be used before every call to container_of() that accesses a domain structure.

---
 include/linux/resctrl.h            |  9 +++++++++
 arch/x86/kernel/cpu/resctrl/core.c | 10 ++++++----
 fs/resctrl/ctrlmondata.c           |  2 +-
 3 files changed, 16 insertions(+), 5 deletions(-)
include/linux/resctrl.h +9 -0
index a7d92718b653..dfc91c5e8483 100644
@@ -131,15 +131,24 @@ enum resctrl_domain_type {
  * @list:		all instances of this resource
  * @id:			unique id for this instance
  * @type:		type of this instance
+ * @rid:		resource id for this instance
  * @cpu_mask:		which CPUs share this resource
  */
 struct rdt_domain_hdr {
 	struct list_head		list;
 	int				id;
 	enum resctrl_domain_type	type;
+	enum resctrl_res_level		rid;
 	struct cpumask			cpu_mask;
 };
 
+static inline bool domain_header_is_valid(struct rdt_domain_hdr *hdr,
+					  enum resctrl_domain_type type,
+					  enum resctrl_res_level rid)
+{
+	return !WARN_ON_ONCE(hdr->type != type || hdr->rid != rid);
+}
+
 /**
  * struct rdt_ctrl_domain - group of CPUs sharing a resctrl control resource
  * @hdr:		common header for different domain types
arch/x86/kernel/cpu/resctrl/core.c +6 -4
index 06ca5a30140c..8be2619db2e7 100644
@@ -459,7 +459,7 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r)
 
 	hdr = resctrl_find_domain(&r->ctrl_domains, id, &add_pos);
 	if (hdr) {
-		if (WARN_ON_ONCE(hdr->type != RESCTRL_CTRL_DOMAIN))
+		if (!domain_header_is_valid(hdr, RESCTRL_CTRL_DOMAIN, r->rid))
 			return;
 		d = container_of(hdr, struct rdt_ctrl_domain, hdr);
 
@@ -476,6 +476,7 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r)
 	d = &hw_dom->d_resctrl;
 	d->hdr.id = id;
 	d->hdr.type = RESCTRL_CTRL_DOMAIN;
+	d->hdr.rid = r->rid;
 	cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
 
 	rdt_domain_reconfigure_cdp(r);
@@ -515,7 +516,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 
 	hdr = resctrl_find_domain(&r->mon_domains, id, &add_pos);
 	if (hdr) {
-		if (WARN_ON_ONCE(hdr->type != RESCTRL_MON_DOMAIN))
+		if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, r->rid))
 			return;
 		d = container_of(hdr, struct rdt_mon_domain, hdr);
 
@@ -533,6 +534,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 	d = &hw_dom->d_resctrl;
 	d->hdr.id = id;
 	d->hdr.type = RESCTRL_MON_DOMAIN;
+	d->hdr.rid = r->rid;
 	ci = get_cpu_cacheinfo_level(cpu, RESCTRL_L3_CACHE);
 	if (!ci) {
 		pr_warn_once("Can't find L3 cache for CPU:%d resource %s\n", cpu, r->name);
@@ -593,7 +595,7 @@ static void domain_remove_cpu_ctrl(int cpu, struct rdt_resource *r)
 		return;
 	}
 
-	if (WARN_ON_ONCE(hdr->type != RESCTRL_CTRL_DOMAIN))
+	if (!domain_header_is_valid(hdr, RESCTRL_CTRL_DOMAIN, r->rid))
 		return;
 
 	d = container_of(hdr, struct rdt_ctrl_domain, hdr);
@@ -639,7 +641,7 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
 		return;
 	}
 
-	if (WARN_ON_ONCE(hdr->type != RESCTRL_MON_DOMAIN))
+	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, r->rid))
 		return;
 
 	d = container_of(hdr, struct rdt_mon_domain, hdr);
fs/resctrl/ctrlmondata.c +1 -1
index 0d0ef54fc4de..f248eaf50d3c 100644
@@ -649,7 +649,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 		 * the resource to find the domain with "domid".
 		 */
 		hdr = resctrl_find_domain(&r->mon_domains, domid, NULL);
-		if (!hdr || WARN_ON_ONCE(hdr->type != RESCTRL_MON_DOMAIN)) {
+		if (!hdr || !domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, resid)) {
 			ret = -ENOENT;
 			goto out;
 		}
Signature · 1 line
2.51.0
maintainer Reviewed-by on v12 1/31
#2 lore raw

Re: [PATCH v12 01/31] x86,fs/resctrl: Improve domain type checking

Hi Tony,

Quoted 5 lines · Tony Luck

Every resctrl resource has a list of domain structures. struct rdt_ctrl_domain and struct rdt_mon_domain both begin with struct rdt_domain_hdr with rdt_domain_hdr::type used in validity checks before accessing the domain of a particular type.

Quoted 8 lines

Add the resource id to struct rdt_domain_hdr in preparation for a new monitoring domain structure that will be associated with a new monitoring resource. Improve existing domain validity checks with a new helper domain_header_is_valid() that checks both domain type and resource id. domain_header_is_valid() should be used before every call to container_of() that accesses a domain structure.

---

Reinette

author maintainer
#3 lore raw

PATCH v12 2/31 x86/resctrl: Move L3 initialization into new helper function

In reply to Tony Luck

Carve out the resource monitoring domain init code into a separate helper in order to be able to initialize new types of monitoring domains besides the usual L3 ones.

---
 arch/x86/kernel/cpu/resctrl/core.c | 64 ++++++++++++++++--------------
 1 file changed, 34 insertions(+), 30 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +34 -30
index 8be2619db2e7..d422ae3b7ed6 100644
@@ -496,37 +496,13 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r)
 	}
 }
 
-static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
+static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct list_head *add_pos)
 {
-	int id = get_domain_id_from_scope(cpu, r->mon_scope);
-	struct list_head *add_pos = NULL;
 	struct rdt_hw_mon_domain *hw_dom;
-	struct rdt_domain_hdr *hdr;
 	struct rdt_mon_domain *d;
 	struct cacheinfo *ci;
 	int err;
 
-	lockdep_assert_held(&domain_list_lock);
-
-	if (id < 0) {
-		pr_warn_once("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n",
-			     cpu, r->mon_scope, r->name);
-		return;
-	}
-
-	hdr = resctrl_find_domain(&r->mon_domains, id, &add_pos);
-	if (hdr) {
-		if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, r->rid))
-			return;
-		d = container_of(hdr, struct rdt_mon_domain, hdr);
-
-		cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
-		/* Update the mbm_assign_mode state for the CPU if supported */
-		if (r->mon.mbm_cntr_assignable)
-			resctrl_arch_mbm_cntr_assign_set_one(r);
-		return;
-	}
-
 	hw_dom = kzalloc_node(sizeof(*hw_dom), GFP_KERNEL, cpu_to_node(cpu));
 	if (!hw_dom)
 		return;
@@ -534,7 +510,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 	d = &hw_dom->d_resctrl;
 	d->hdr.id = id;
 	d->hdr.type = RESCTRL_MON_DOMAIN;
-	d->hdr.rid = r->rid;
+	d->hdr.rid = RDT_RESOURCE_L3;
 	ci = get_cpu_cacheinfo_level(cpu, RESCTRL_L3_CACHE);
 	if (!ci) {
 		pr_warn_once("Can't find L3 cache for CPU:%d resource %s\n", cpu, r->name);
@@ -544,10 +520,6 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 	d->ci_id = ci->id;
 	cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
 
-	/* Update the mbm_assign_mode state for the CPU if supported */
-	if (r->mon.mbm_cntr_assignable)
-		resctrl_arch_mbm_cntr_assign_set_one(r);
-
 	arch_mon_domain_online(r, d);
 
 	if (arch_domain_mbm_alloc(r->mon.num_rmid, hw_dom)) {
@@ -565,6 +537,38 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 	}
 }
 
+static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
+{
+	int id = get_domain_id_from_scope(cpu, r->mon_scope);
+	struct list_head *add_pos = NULL;
+	struct rdt_domain_hdr *hdr;
+
+	lockdep_assert_held(&domain_list_lock);
+
+	if (id < 0) {
+		pr_warn_once("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n",
+			     cpu, r->mon_scope, r->name);
+		return;
+	}
+
+	hdr = resctrl_find_domain(&r->mon_domains, id, &add_pos);
+	if (hdr)
+		cpumask_set_cpu(cpu, &hdr->cpu_mask);
+
+	switch (r->rid) {
+	case RDT_RESOURCE_L3:
+		/* Update the mbm_assign_mode state for the CPU if supported */
+		if (r->mon.mbm_cntr_assignable)
+			resctrl_arch_mbm_cntr_assign_set_one(r);
+		if (!hdr)
+			l3_mon_domain_setup(cpu, id, r, add_pos);
+		break;
+	default:
+		pr_warn_once("Unknown resource rid=%d\n", r->rid);
+		break;
+	}
+}
+
 static void domain_add_cpu(int cpu, struct rdt_resource *r)
 {
 	if (r->alloc_capable)
Signature · 1 line
2.51.0
author maintainer
#4 lore raw

PATCH v12 3/31 x86/resctrl: Refactor domain_remove_cpu_mon() ready for new domain types

In reply to Tony Luck

New telemetry events will be associated with a new package scoped resource with new domain structures.

Refactor domain_remove_cpu_mon() so all the L3 domain processing is separate from general domain actions of clearing the CPU bit in the mask.

---
 arch/x86/kernel/cpu/resctrl/core.c | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +17 -10
index d422ae3b7ed6..c7bfaa391e9f 100644
@@ -626,9 +626,7 @@ static void domain_remove_cpu_ctrl(int cpu, struct rdt_resource *r)
 static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
 {
 	int id = get_domain_id_from_scope(cpu, r->mon_scope);
-	struct rdt_hw_mon_domain *hw_dom;
 	struct rdt_domain_hdr *hdr;
-	struct rdt_mon_domain *d;
 
 	lockdep_assert_held(&domain_list_lock);
 
@@ -645,20 +643,29 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
 		return;
 	}
 
-	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, r->rid))
+	cpumask_clear_cpu(cpu, &hdr->cpu_mask);
+	if (!cpumask_empty(&hdr->cpu_mask))
 		return;
 
-	d = container_of(hdr, struct rdt_mon_domain, hdr);
-	hw_dom = resctrl_to_arch_mon_dom(d);
+	switch (r->rid) {
+	case RDT_RESOURCE_L3: {
+		struct rdt_hw_mon_domain *hw_dom;
+		struct rdt_mon_domain *d;
 
-	cpumask_clear_cpu(cpu, &d->hdr.cpu_mask);
-	if (cpumask_empty(&d->hdr.cpu_mask)) {
+		if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+			return;
+
+		d = container_of(hdr, struct rdt_mon_domain, hdr);
+		hw_dom = resctrl_to_arch_mon_dom(d);
 		resctrl_offline_mon_domain(r, d);
-		list_del_rcu(&d->hdr.list);
+		list_del_rcu(&hdr->list);
 		synchronize_rcu();
 		mon_domain_free(hw_dom);
-
-		return;
+		break;
+	}
+	default:
+		pr_warn_once("Unknown resource rid=%d\n", r->rid);
+		break;
 	}
 }
 
Signature · 1 line
2.51.0
maintainer Reviewed-by on v12 3/31
#5 lore raw

Re: [PATCH v12 03/31] x86/resctrl: Refactor domain_remove_cpu_mon() ready for new domain types

Hi Tony,

Quoted 8 lines · Tony Luck

New telemetry events will be associated with a new package scoped resource with new domain structures.

Refactor domain_remove_cpu_mon() so all the L3 domain processing is separate from general domain actions of clearing the CPU bit in the mask.

---

Reinette

author maintainer
#6 lore raw

PATCH v12 4/31 x86/resctrl: Clean up domain_remove_cpu_ctrl()

In reply to Tony Luck

For symmetry with domain_remove_cpu_mon() refactor domain_remove_cpu_ctrl() to take an early return when removing a CPU does not empty the domain.

---
 arch/x86/kernel/cpu/resctrl/core.c | 29 ++++++++++++++---------------
 1 file changed, 14 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +14 -15
index c7bfaa391e9f..b28663757dcf 100644
@@ -599,28 +599,27 @@ static void domain_remove_cpu_ctrl(int cpu, struct rdt_resource *r)
 		return;
 	}
 
+	cpumask_clear_cpu(cpu, &hdr->cpu_mask);
+	if (!cpumask_empty(&hdr->cpu_mask))
+		return;
+
 	if (!domain_header_is_valid(hdr, RESCTRL_CTRL_DOMAIN, r->rid))
 		return;
 
 	d = container_of(hdr, struct rdt_ctrl_domain, hdr);
 	hw_dom = resctrl_to_arch_ctrl_dom(d);
 
-	cpumask_clear_cpu(cpu, &d->hdr.cpu_mask);
-	if (cpumask_empty(&d->hdr.cpu_mask)) {
-		resctrl_offline_ctrl_domain(r, d);
-		list_del_rcu(&d->hdr.list);
-		synchronize_rcu();
-
-		/*
-		 * rdt_ctrl_domain "d" is going to be freed below, so clear
-		 * its pointer from pseudo_lock_region struct.
-		 */
-		if (d->plr)
-			d->plr->d = NULL;
-		ctrl_domain_free(hw_dom);
+	resctrl_offline_ctrl_domain(r, d);
+	list_del_rcu(&hdr->list);
+	synchronize_rcu();
 
-		return;
-	}
+	/*
+	 * rdt_ctrl_domain "d" is going to be freed below, so clear
+	 * its pointer from pseudo_lock_region struct.
+	 */
+	if (d->plr)
+		d->plr->d = NULL;
+	ctrl_domain_free(hw_dom);
 }
 
 static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
Signature · 1 line
2.51.0
author maintainer
#7 lore raw

PATCH v12 5/31 x86,fs/resctrl: Refactor domain create/remove using struct rdt_domain_hdr

In reply to Tony Luck

Up until now, all monitoring events were associated with the L3 resource and it made sense to use the L3 specific "struct rdt_mon_domain *" argument to functions operating on domains.

Telemetry events will be tied to a new resource with its instances represented by a new domain structure that, just like struct rdt_mon_domain, starts with the generic struct rdt_domain_hdr.

Prepare to support domains belonging to different resources by changing the calling convention of functions operating on domains. Pass the generic header and use that to find the domain specific structure where needed.

---
 include/linux/resctrl.h            |  4 +-
 fs/resctrl/internal.h              |  2 +-
 arch/x86/kernel/cpu/resctrl/core.c |  4 +-
 fs/resctrl/ctrlmondata.c           | 17 +++++---
 fs/resctrl/rdtgroup.c              | 65 +++++++++++++++++++++---------
 5 files changed, 63 insertions(+), 29 deletions(-)
include/linux/resctrl.h +2 -2
index dfc91c5e8483..0b55809af5d7 100644
@@ -504,9 +504,9 @@ int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d,
 u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d,
 			    u32 closid, enum resctrl_conf_type type);
 int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d);
-int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d);
+int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr);
 void resctrl_offline_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d);
-void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d);
+void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr);
 void resctrl_online_cpu(unsigned int cpu);
 void resctrl_offline_cpu(unsigned int cpu);
 
fs/resctrl/internal.h +1 -1
index cf1fd82dc5a9..22fdb3a9b6f4 100644
@@ -362,7 +362,7 @@ void mon_event_count(void *info);
 int rdtgroup_mondata_show(struct seq_file *m, void *arg);
 
 void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
-		    struct rdt_mon_domain *d, struct rdtgroup *rdtgrp,
+		    struct rdt_domain_hdr *hdr, struct rdtgroup *rdtgrp,
 		    cpumask_t *cpumask, int evtid, int first);
 
 int resctrl_mon_resource_init(void);
arch/x86/kernel/cpu/resctrl/core.c +2 -2
index b28663757dcf..44495bb915d5 100644
@@ -529,7 +529,7 @@ static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct
 
 	list_add_tail_rcu(&d->hdr.list, add_pos);
 
-	err = resctrl_online_mon_domain(r, d);
+	err = resctrl_online_mon_domain(r, &d->hdr);
 	if (err) {
 		list_del_rcu(&d->hdr.list);
 		synchronize_rcu();
@@ -656,7 +656,7 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
 
 		d = container_of(hdr, struct rdt_mon_domain, hdr);
 		hw_dom = resctrl_to_arch_mon_dom(d);
-		resctrl_offline_mon_domain(r, d);
+		resctrl_offline_mon_domain(r, hdr);
 		list_del_rcu(&hdr->list);
 		synchronize_rcu();
 		mon_domain_free(hw_dom);
fs/resctrl/ctrlmondata.c +12 -5
index f248eaf50d3c..8c07a9810706 100644
@@ -547,11 +547,18 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,
 }
 
 void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
-		    struct rdt_mon_domain *d, struct rdtgroup *rdtgrp,
+		    struct rdt_domain_hdr *hdr, struct rdtgroup *rdtgrp,
 		    cpumask_t *cpumask, int evtid, int first)
 {
+	struct rdt_mon_domain *d = NULL;
 	int cpu;
 
+	if (hdr) {
+		if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+			return;
+		d = container_of(hdr, struct rdt_mon_domain, hdr);
+	}
+
 	/* When picking a CPU from cpu_mask, ensure it can't race with cpuhp */
 	lockdep_assert_cpus_held();
 
@@ -598,7 +605,6 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	enum resctrl_event_id evtid;
 	struct rdt_domain_hdr *hdr;
 	struct rmid_read rr = {0};
-	struct rdt_mon_domain *d;
 	struct rdtgroup *rdtgrp;
 	int domid, cpu, ret = 0;
 	struct rdt_resource *r;
@@ -623,6 +629,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	r = resctrl_arch_get_resource(resid);
 
 	if (md->sum) {
+		struct rdt_mon_domain *d;
+
 		/*
 		 * This file requires summing across all domains that share
 		 * the L3 cache id that was provided in the "domid" field of the
@@ -649,12 +657,11 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 		 * the resource to find the domain with "domid".
 		 */
 		hdr = resctrl_find_domain(&r->mon_domains, domid, NULL);
-		if (!hdr || !domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, resid)) {
+		if (!hdr) {
 			ret = -ENOENT;
 			goto out;
 		}
-		d = container_of(hdr, struct rdt_mon_domain, hdr);
-		mon_event_read(&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false);
+		mon_event_read(&rr, r, hdr, rdtgrp, &hdr->cpu_mask, evtid, false);
 	}
 
 checkresult:
fs/resctrl/rdtgroup.c +46 -19

192 lines not on this page · raw message

Signature · 1 line
2.51.0
maintainer
#8 lore raw

Re: [PATCH v12 05/31] x86,fs/resctrl: Refactor domain create/remove using struct rdt_domain_hdr

Hi Tony,

Quoted 8 lines · Tony Luck

Up until now, all monitoring events were associated with the L3 resource and it made sense to use the L3 specific "struct rdt_mon_domain *" argument to functions operating on domains.

Telemetry events will be tied to a new resource with its instances represented by a new domain structure that, just like struct rdt_mon_domain, starts with the generic struct rdt_domain_hdr.

Quoted 6 lines

Prepare to support domains belonging to different resources by changing the calling convention of functions operating on domains. Pass the generic header and use that to find the domain specific structure where needed.

---

...> diff --git a/fs/resctrl/ctrlmondata.c b/fs/resctrl/ctrlmondata.c

Quoted 13 lines

index f248eaf50d3c..8c07a9810706 100644

fs/resctrl/ctrlmondata.c +2 -1
@@ -547,11 +547,18 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,
 }
 
 void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
-		    struct rdt_mon_domain *d, struct rdtgroup *rdtgrp,
+		    struct rdt_domain_hdr *hdr, struct rdtgroup *rdtgrp,
 		    cpumask_t *cpumask, int evtid, int first)
 {
+	struct rdt_mon_domain *d = NULL;
 	int cpu;

fs/resctrl/ctrlmondata.c struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id, · line 564

Quoted 10 lines
fs/resctrl/ctrlmondata.c +6 -0
 
+	if (hdr) {
+		if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+			return;
+		d = container_of(hdr, struct rdt_mon_domain, hdr);
+	}
+
 	/* When picking a CPU from cpu_mask, ensure it can't race with cpuhp */
 	lockdep_assert_cpus_held();
 

Please keep lockdep_assert_cpus_held() at beginning of the function.

Quoted 9 lines
fs/resctrl/ctrlmondata.c +0 -1
@@ -598,7 +605,6 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	enum resctrl_event_id evtid;
 	struct rdt_domain_hdr *hdr;
 	struct rmid_read rr = {0};
-	struct rdt_mon_domain *d;
 	struct rdtgroup *rdtgrp;
 	int domid, cpu, ret = 0;
 	struct rdt_resource *r;

fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 636

Quoted 8 lines
fs/resctrl/ctrlmondata.c +2 -0
@@ -623,6 +629,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	r = resctrl_arch_get_resource(resid);
 
 	if (md->sum) {
+		struct rdt_mon_domain *d;
+
 		/*
 		 * This file requires summing across all domains that share
 		 * the L3 cache id that was provided in the "domid" field of the

This hunk is unlike any other in this patch. To me the pattern of changes found in this patch is that functions modified used to obtain struct rdt_mon_domain as parameter and after this change these functions obtain struct rdt_domain_hdr as parameter and need to use the container_of() to obtain the needed struct rdt_mon_domain. This hunk stands out in that it just blindly assumes that the domain obtained by traversing rdt_resource::mon_domains is an L3 domain.

How is this hunk relevant to this patch?

From what I can tell the above two hunks need to be separated into a new patch and merged with the unrelated changes added to patch #13 after it was reviewed.

Reinette

author maintainer
#9 lore raw

PATCH v12 6/31 x86,fs/resctrl: Use struct rdt_domain_hdr when reading counters

In reply to Tony Luck

struct rmid_read contains data passed around to read event counts. Use the generic domain header struct rdt_domain_hdr in struct rmid_read in order to support other telemetry events' domains besides an L3 one. Adjust the code interacting with it to the new struct layout.

---
 include/linux/resctrl.h               |  8 +++---
 fs/resctrl/internal.h                 | 18 +++++++------
 arch/x86/kernel/cpu/resctrl/monitor.c | 20 +++++++++++---
 fs/resctrl/ctrlmondata.c              |  9 +------
 fs/resctrl/monitor.c                  | 38 ++++++++++++++++++---------
 5 files changed, 56 insertions(+), 37 deletions(-)
include/linux/resctrl.h +4 -4
index 0b55809af5d7..0fef3045cac3 100644
@@ -514,7 +514,7 @@ void resctrl_offline_cpu(unsigned int cpu);
  * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid
  *			      for this resource and domain.
  * @r:			resource that the counter should be read from.
- * @d:			domain that the counter should be read from.
+ * @hdr:		Header of domain that the counter should be read from.
  * @closid:		closid that matches the rmid. Depending on the architecture, the
  *			counter may match traffic of both @closid and @rmid, or @rmid
  *			only.
@@ -535,7 +535,7 @@ void resctrl_offline_cpu(unsigned int cpu);
  * Return:
  * 0 on success, or -EIO, -EINVAL etc on error.
  */
-int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
+int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 			   u32 closid, u32 rmid, enum resctrl_event_id eventid,
 			   u64 *val, void *arch_mon_ctx);
 
@@ -630,7 +630,7 @@ void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
  *			      assigned to the RMID, event pair for this resource
  *			      and domain.
  * @r:		Resource that the counter should be read from.
- * @d:		Domain that the counter should be read from.
+ * @hdr:	Header of domain that the counter should be read from.
  * @closid:	CLOSID that matches the RMID.
  * @rmid:	The RMID to which @cntr_id is assigned.
  * @cntr_id:	The counter to read.
@@ -644,7 +644,7 @@ void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
  * Return:
  * 0 on success, or -EIO, -EINVAL etc on error.
  */
-int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_mon_domain *d,
+int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 			   u32 closid, u32 rmid, int cntr_id,
 			   enum resctrl_event_id eventid, u64 *val);
 
fs/resctrl/internal.h +10 -8
index 22fdb3a9b6f4..698ed84fd073 100644
@@ -106,24 +106,26 @@ struct mon_data {
  *	   resource group then its event count is summed with the count from all
  *	   its child resource groups.
  * @r:	   Resource describing the properties of the event being read.
- * @d:	   Domain that the counter should be read from. If NULL then sum all
- *	   domains in @r sharing L3 @ci.id
+ * @hdr:   Header of domain that the counter should be read from. If NULL then
+ *	   sum all domains in @r sharing L3 @ci.id
  * @evtid: Which monitor event to read.
  * @first: Initialize MBM counter when true.
- * @ci:    Cacheinfo for L3. Only set when @d is NULL. Used when summing domains.
+ * @ci:    Cacheinfo for L3. Only set when @hdr is NULL. Used when summing
+ *	   domains.
  * @is_mbm_cntr: true if "mbm_event" counter assignment mode is enabled and it
  *	   is an MBM event.
  * @err:   Error encountered when reading counter.
- * @val:   Returned value of event counter. If @rgrp is a parent resource group,
- *	   @val includes the sum of event counts from its child resource groups.
- *	   If @d is NULL, @val includes the sum of all domains in @r sharing @ci.id,
- *	   (summed across child resource groups if @rgrp is a parent resource group).
+ * @val:   Returned value of event counter. If @rgrp is a parent resource
+ *	   group, @val includes the sum of event counts from its child
+ *	   resource groups.  If @hdr is NULL, @val includes the sum of all
+ *	   domains in @r sharing @ci.id, (summed across child resource groups
+ *	   if @rgrp is a parent resource group).
  * @arch_mon_ctx: Hardware monitor allocated for this read request (MPAM only).
  */
 struct rmid_read {
 	struct rdtgroup		*rgrp;
 	struct rdt_resource	*r;
-	struct rdt_mon_domain	*d;
+	struct rdt_domain_hdr	*hdr;
 	enum resctrl_event_id	evtid;
 	bool			first;
 	struct cacheinfo	*ci;
arch/x86/kernel/cpu/resctrl/monitor.c +16 -4
index 2cd25a0d4637..1f47844b5246 100644
@@ -238,19 +238,26 @@ static u64 get_corrected_val(struct rdt_resource *r, struct rdt_mon_domain *d,
 	return chunks * hw_res->mon_scale;
 }
 
-int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
+int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 			   u32 unused, u32 rmid, enum resctrl_event_id eventid,
 			   u64 *val, void *ignored)
 {
-	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
-	int cpu = cpumask_any(&d->hdr.cpu_mask);
+	struct rdt_hw_mon_domain *hw_dom;
 	struct arch_mbm_state *am;
+	struct rdt_mon_domain *d;
 	u64 msr_val;
 	u32 prmid;
+	int cpu;
 	int ret;
 
 	resctrl_arch_rmid_read_context_check();
 
+	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+		return -EINVAL;
+
+	d = container_of(hdr, struct rdt_mon_domain, hdr);
+	hw_dom = resctrl_to_arch_mon_dom(d);
+	cpu = cpumask_any(&hdr->cpu_mask);
 	prmid = logical_rmid_to_physical_rmid(cpu, rmid);
 	ret = __rmid_read_phys(prmid, eventid, &msr_val);
 
@@ -318,13 +325,18 @@ void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
 	}
 }
 
-int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_mon_domain *d,
+int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 			   u32 unused, u32 rmid, int cntr_id,
 			   enum resctrl_event_id eventid, u64 *val)
 {
+	struct rdt_mon_domain *d;
 	u64 msr_val;
 	int ret;
 
+	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+		return -EINVAL;
+
+	d = container_of(hdr, struct rdt_mon_domain, hdr);
 	ret = __cntr_id_read(cntr_id, &msr_val);
 	if (ret)
 		return ret;
fs/resctrl/ctrlmondata.c +1 -8
index 8c07a9810706..7b9fc5d3bdc8 100644
@@ -550,15 +550,8 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 		    struct rdt_domain_hdr *hdr, struct rdtgroup *rdtgrp,
 		    cpumask_t *cpumask, int evtid, int first)
 {
-	struct rdt_mon_domain *d = NULL;
 	int cpu;
 
-	if (hdr) {
-		if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
-			return;
-		d = container_of(hdr, struct rdt_mon_domain, hdr);
-	}
-
 	/* When picking a CPU from cpu_mask, ensure it can't race with cpuhp */
 	lockdep_assert_cpus_held();
 
@@ -568,7 +561,7 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 	rr->rgrp = rdtgrp;
 	rr->evtid = evtid;
 	rr->r = r;
-	rr->d = d;
+	rr->hdr = hdr;
 	rr->first = first;
 	if (resctrl_arch_mbm_cntr_assign_enabled(r) &&
 	    resctrl_is_mbm_event(evtid)) {
fs/resctrl/monitor.c +25 -13
index 4076336fbba6..521f78f42f07 100644
@@ -159,7 +159,7 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free)
 			break;
 
 		entry = __rmid_entry(idx);
-		if (resctrl_arch_rmid_read(r, d, entry->closid, entry->rmid,
+		if (resctrl_arch_rmid_read(r, &d->hdr, entry->closid, entry->rmid,
 					   QOS_L3_OCCUP_EVENT_ID, &val,
 					   arch_mon_ctx)) {
 			rmid_dirty = true;
@@ -425,7 +425,11 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	u64 tval = 0;
 
 	if (rr->is_mbm_cntr) {
-		cntr_id = mbm_cntr_get(rr->r, rr->d, rdtgrp, rr->evtid);
+		if (!rr->hdr || !domain_header_is_valid(rr->hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+			return -EINVAL;
+		d = container_of(rr->hdr, struct rdt_mon_domain, hdr);
+
+		cntr_id = mbm_cntr_get(rr->r, d, rdtgrp, rr->evtid);
 		if (cntr_id < 0) {
 			rr->err = -ENOENT;
 			return -EINVAL;
@@ -433,25 +437,29 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	}
 
 	if (rr->first) {
+		if (!rr->hdr || !domain_header_is_valid(rr->hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+			return -EINVAL;
+		d = container_of(rr->hdr, struct rdt_mon_domain, hdr);
+
 		if (rr->is_mbm_cntr)
-			resctrl_arch_reset_cntr(rr->r, rr->d, closid, rmid, cntr_id, rr->evtid);
+			resctrl_arch_reset_cntr(rr->r, d, closid, rmid, cntr_id, rr->evtid);
 		else
-			resctrl_arch_reset_rmid(rr->r, rr->d, closid, rmid, rr->evtid);
-		m = get_mbm_state(rr->d, closid, rmid, rr->evtid);
+			resctrl_arch_reset_rmid(rr->r, d, closid, rmid, rr->evtid);
+		m = get_mbm_state(d, closid, rmid, rr->evtid);
 		if (m)
 			memset(m, 0, sizeof(struct mbm_state));
 		return 0;
 	}
 
-	if (rr->d) {
+	if (rr->hdr) {
 		/* Reading a single domain, must be on a CPU in that domain. */
-		if (!cpumask_test_cpu(cpu, &rr->d->hdr.cpu_mask))
+		if (!cpumask_test_cpu(cpu, &rr->hdr->cpu_mask))
 			return -EINVAL;
 		if (rr->is_mbm_cntr)
-			rr->err = resctrl_arch_cntr_read(rr->r, rr->d, closid, rmid, cntr_id,
+			rr->err = resctrl_arch_cntr_read(rr->r, rr->hdr, closid, rmid, cntr_id,
 							 rr->evtid, &tval);
 		else
-			rr->err = resctrl_arch_rmid_read(rr->r, rr->d, closid, rmid,
+			rr->err = resctrl_arch_rmid_read(rr->r, rr->hdr, closid, rmid,
 							 rr->evtid, &tval, rr->arch_mon_ctx);
 		if (rr->err)
 			return rr->err;
@@ -477,10 +485,10 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 		if (d->ci_id != rr->ci->id)
 			continue;
 		if (rr->is_mbm_cntr)
-			err = resctrl_arch_cntr_read(rr->r, d, closid, rmid, cntr_id,
+			err = resctrl_arch_cntr_read(rr->r, &d->hdr, closid, rmid, cntr_id,
 						     rr->evtid, &tval);
 		else
-			err = resctrl_arch_rmid_read(rr->r, d, closid, rmid,
+			err = resctrl_arch_rmid_read(rr->r, &d->hdr, closid, rmid,
 						     rr->evtid, &tval, rr->arch_mon_ctx);
 		if (!err) {
 			rr->val += tval;
@@ -511,9 +519,13 @@ static void mbm_bw_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	u64 cur_bw, bytes, cur_bytes;
 	u32 closid = rdtgrp->closid;
 	u32 rmid = rdtgrp->mon.rmid;
+	struct rdt_mon_domain *d;
 	struct mbm_state *m;
 
-	m = get_mbm_state(rr->d, closid, rmid, rr->evtid);
+	if (!domain_header_is_valid(rr->hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+		return;
+	d = container_of(rr->hdr, struct rdt_mon_domain, hdr);
+	m = get_mbm_state(d, closid, rmid, rr->evtid);
 	if (WARN_ON_ONCE(!m))
 		return;
 
@@ -686,7 +698,7 @@ static void mbm_update_one_event(struct rdt_resource *r, struct rdt_mon_domain *
 	struct rmid_read rr = {0};
 
 	rr.r = r;
-	rr.d = d;
+	rr.hdr = &d->hdr;
 	rr.evtid = evtid;
 	if (resctrl_arch_mbm_cntr_assign_enabled(r)) {
 		rr.is_mbm_cntr = true;
Signature · 1 line
2.51.0
maintainer
#10 lore raw

Re: [PATCH v12 06/31] x86,fs/resctrl: Use struct rdt_domain_hdr when reading counters

Hi Tony,

Quoted 3 lines · Tony Luck

struct rmid_read contains data passed around to read event counts. Use the generic domain header struct rdt_domain_hdr in struct rmid_read in order to support other telemetry events' domains besides an L3 one. Adjust the code

"telemetry events" -> "monitoring events"?

Quoted 1 line

interacting with it to the new struct layout.

How does this justify the changes to resctrl_arch_rmid_read() and resctrl_arch_cntr_read()? If these functions really needed to be changed in support of the change to struct rmid_read then resctrl_arch_reset_cntr() and resctrl_arch_reset_rmid() would need to be changed also, no? All four of these functions are called in the same way before this change but this patch inconsistently changes the calling convention of only two of them without any motivation.

Seems like the resctrl_arch_rmid_read() change is sneaked in to support later reading of telemetry events while the change to resctrl_arch_cntr_read() is a remnant of a previous version of code in support of telemetry events?

Quoted 24 lines
---
 include/linux/resctrl.h               |  8 +++---
 fs/resctrl/internal.h                 | 18 +++++++------
 arch/x86/kernel/cpu/resctrl/monitor.c | 20 +++++++++++---
 fs/resctrl/ctrlmondata.c              |  9 +------
 fs/resctrl/monitor.c                  | 38 ++++++++++++++++++---------
 5 files changed, 56 insertions(+), 37 deletions(-)
include/linux/resctrl.h +1 -1
index 0b55809af5d7..0fef3045cac3 100644
@@ -514,7 +514,7 @@ void resctrl_offline_cpu(unsigned int cpu);
  * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid
  *			      for this resource and domain.
  * @r:			resource that the counter should be read from.
- * @d:			domain that the counter should be read from.
+ * @hdr:		Header of domain that the counter should be read from.
  * @closid:		closid that matches the rmid. Depending on the architecture, the
  *			counter may match traffic of both @closid and @rmid, or @rmid
  *			only.

include/linux/resctrl.h void resctrl_offline_cpu(unsigned int cpu); · line 541

Quoted 8 lines
include/linux/resctrl.h +1 -1
@@ -535,7 +535,7 @@ void resctrl_offline_cpu(unsigned int cpu);
  * Return:
  * 0 on success, or -EIO, -EINVAL etc on error.
  */
-int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
+int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 			   u32 closid, u32 rmid, enum resctrl_event_id eventid,
 			   u64 *val, void *arch_mon_ctx);
 

This change is not related to a change to struct rmid_read.

Quoted 10 lines
include/linux/resctrl.h +1 -1
@@ -630,7 +630,7 @@ void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
  *			      assigned to the RMID, event pair for this resource
  *			      and domain.
  * @r:		Resource that the counter should be read from.
- * @d:		Domain that the counter should be read from.
+ * @hdr:	Header of domain that the counter should be read from.
  * @closid:	CLOSID that matches the RMID.
  * @rmid:	The RMID to which @cntr_id is assigned.
  * @cntr_id:	The counter to read.

include/linux/resctrl.h void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d, · line 650

Quoted 8 lines
include/linux/resctrl.h +1 -1
@@ -644,7 +644,7 @@ void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
  * Return:
  * 0 on success, or -EIO, -EINVAL etc on error.
  */
-int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_mon_domain *d,
+int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 			   u32 closid, u32 rmid, int cntr_id,
 			   enum resctrl_event_id eventid, u64 *val);
 

Same with this change.

...

Quoted 14 lines
fs/resctrl/monitor.c +1 -1
index 4076336fbba6..521f78f42f07 100644
@@ -159,7 +159,7 @@ void __check_limbo(struct rdt_mon_domain *d, bool force_free)
 			break;
 
 		entry = __rmid_entry(idx);
-		if (resctrl_arch_rmid_read(r, d, entry->closid, entry->rmid,
+		if (resctrl_arch_rmid_read(r, &d->hdr, entry->closid, entry->rmid,
 					   QOS_L3_OCCUP_EVENT_ID, &val,
 					   arch_mon_ctx)) {
 			rmid_dirty = true;

fs/resctrl/monitor.c static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) · line 429

Quoted 6 lines
fs/resctrl/monitor.c +2 -1
@@ -425,7 +425,11 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	u64 tval = 0;
 
 	if (rr->is_mbm_cntr) {
-		cntr_id = mbm_cntr_get(rr->r, rr->d, rdtgrp, rr->evtid);
+		if (!rr->hdr || !domain_header_is_valid(rr->hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+			return -EINVAL;

I do not think returning an error directly is a pattern that should continue. This error is dropped by caller while rmid_read::err is what continues on. This can be something like:

			rr->err = -EIO;
			return -EINVAL;

fs/resctrl/monitor.c static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) · line 441

Quoted 12 lines
fs/resctrl/monitor.c +5 -0
+		d = container_of(rr->hdr, struct rdt_mon_domain, hdr);
+
+		cntr_id = mbm_cntr_get(rr->r, d, rdtgrp, rr->evtid);
 		if (cntr_id < 0) {
 			rr->err = -ENOENT;
 			return -EINVAL;
@@ -433,25 +437,29 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	}
 
 	if (rr->first) {
+		if (!rr->hdr || !domain_header_is_valid(rr->hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+			return -EINVAL;

Same here .

Quoted 60 lines
fs/resctrl/monitor.c +16 -10
+		d = container_of(rr->hdr, struct rdt_mon_domain, hdr);
+
 		if (rr->is_mbm_cntr)
-			resctrl_arch_reset_cntr(rr->r, rr->d, closid, rmid, cntr_id, rr->evtid);
+			resctrl_arch_reset_cntr(rr->r, d, closid, rmid, cntr_id, rr->evtid);
 		else
-			resctrl_arch_reset_rmid(rr->r, rr->d, closid, rmid, rr->evtid);
-		m = get_mbm_state(rr->d, closid, rmid, rr->evtid);
+			resctrl_arch_reset_rmid(rr->r, d, closid, rmid, rr->evtid);
+		m = get_mbm_state(d, closid, rmid, rr->evtid);
 		if (m)
 			memset(m, 0, sizeof(struct mbm_state));
 		return 0;
 	}
 
-	if (rr->d) {
+	if (rr->hdr) {
 		/* Reading a single domain, must be on a CPU in that domain. */
-		if (!cpumask_test_cpu(cpu, &rr->d->hdr.cpu_mask))
+		if (!cpumask_test_cpu(cpu, &rr->hdr->cpu_mask))
 			return -EINVAL;
 		if (rr->is_mbm_cntr)
-			rr->err = resctrl_arch_cntr_read(rr->r, rr->d, closid, rmid, cntr_id,
+			rr->err = resctrl_arch_cntr_read(rr->r, rr->hdr, closid, rmid, cntr_id,
 							 rr->evtid, &tval);
 		else
-			rr->err = resctrl_arch_rmid_read(rr->r, rr->d, closid, rmid,
+			rr->err = resctrl_arch_rmid_read(rr->r, rr->hdr, closid, rmid,
 							 rr->evtid, &tval, rr->arch_mon_ctx);
 		if (rr->err)
 			return rr->err;
@@ -477,10 +485,10 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 		if (d->ci_id != rr->ci->id)
 			continue;
 		if (rr->is_mbm_cntr)
-			err = resctrl_arch_cntr_read(rr->r, d, closid, rmid, cntr_id,
+			err = resctrl_arch_cntr_read(rr->r, &d->hdr, closid, rmid, cntr_id,
 						     rr->evtid, &tval);
 		else
-			err = resctrl_arch_rmid_read(rr->r, d, closid, rmid,
+			err = resctrl_arch_rmid_read(rr->r, &d->hdr, closid, rmid,
 						     rr->evtid, &tval, rr->arch_mon_ctx);
 		if (!err) {
 			rr->val += tval;
@@ -511,9 +519,13 @@ static void mbm_bw_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	u64 cur_bw, bytes, cur_bytes;
 	u32 closid = rdtgrp->closid;
 	u32 rmid = rdtgrp->mon.rmid;
+	struct rdt_mon_domain *d;
 	struct mbm_state *m;
 
-	m = get_mbm_state(rr->d, closid, rmid, rr->evtid);
+	if (!domain_header_is_valid(rr->hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+		return;
+	d = container_of(rr->hdr, struct rdt_mon_domain, hdr);
+	m = get_mbm_state(d, closid, rmid, rr->evtid);
 	if (WARN_ON_ONCE(!m))
 		return;
 

fs/resctrl/monitor.c static void mbm_update_one_event(struct rdt_resource *r, struct rdt_mon_domain * · line 704

Quoted 8 lines
fs/resctrl/monitor.c +1 -1
@@ -686,7 +698,7 @@ static void mbm_update_one_event(struct rdt_resource *r, struct rdt_mon_domain *
 	struct rmid_read rr = {0};
 
 	rr.r = r;
-	rr.d = d;
+	rr.hdr = &d->hdr;
 	rr.evtid = evtid;
 	if (resctrl_arch_mbm_cntr_assign_enabled(r)) {
 		rr.is_mbm_cntr = true;

Reinette

author maintainer
#11 lore raw

Re: [PATCH v12 06/31] x86,fs/resctrl: Use struct rdt_domain_hdr when reading counters

Quoted 11 lines · Reinette Chatre

Hi Tony,

Quoted 3 lines · Tony Luck

struct rmid_read contains data passed around to read event counts. Use the generic domain header struct rdt_domain_hdr in struct rmid_read in order to support other telemetry events' domains besides an L3 one. Adjust the code

"telemetry events" -> "monitoring events"?

Quoted 1 line

interacting with it to the new struct layout.

Quoted 9 lines

How does this justify the changes to resctrl_arch_rmid_read() and resctrl_arch_cntr_read()? If these functions really needed to be changed in support of the change to struct rmid_read then resctrl_arch_reset_cntr() and resctrl_arch_reset_rmid() would need to be changed also, no? All four of these functions are called in the same way before this change but this patch inconsistently changes the calling convention of only two of them without any motivation. Seems like the resctrl_arch_rmid_read() change is sneaked in to support later reading of telemetry events while the change to resctrl_arch_cntr_read() is a remnant of a previous version of code in support of telemetry events?

These functions got caught up in this change, but it probably isn't needed as they are L3 specific. I'll see how things look if I pull the refactor of __mon_event_count() earlier so they can avoid this unneeded churn.

-Tony

author maintainer
#12 lore raw

PATCH v12 7/31 x86,fs/resctrl: Rename struct rdt_mon_domain and rdt_hw_mon_domain

In reply to Tony Luck

The upcoming telemetry event monitoring are not tied to the L3 resource and will have a new domain structures.

Rename the L3 resource specific domain data structures to include "l3_" in their names to avoid confusion between the different resource specific domain structures:
rdt_mon_domain -> rdt_l3_mon_domain
rdt_hw_mon_domain -> rdt_hw_l3_mon_domain

No functional change.

---
 include/linux/resctrl.h                | 20 ++++----
 arch/x86/kernel/cpu/resctrl/internal.h | 16 +++---
 fs/resctrl/internal.h                  |  8 +--
 arch/x86/kernel/cpu/resctrl/core.c     | 14 +++---
 arch/x86/kernel/cpu/resctrl/monitor.c  | 38 +++++++-------
 fs/resctrl/ctrlmondata.c               |  2 +-
 fs/resctrl/monitor.c                   | 70 +++++++++++++-------------
 fs/resctrl/rdtgroup.c                  | 40 +++++++--------
 8 files changed, 104 insertions(+), 104 deletions(-)
include/linux/resctrl.h +10 -10
index 0fef3045cac3..66569662efee 100644
@@ -178,7 +178,7 @@ struct mbm_cntr_cfg {
 };
 
 /**
- * struct rdt_mon_domain - group of CPUs sharing a resctrl monitor resource
+ * struct rdt_l3_mon_domain - group of CPUs sharing a resctrl monitor resource
  * @hdr:		common header for different domain types
  * @ci_id:		cache info id for this domain
  * @rmid_busy_llc:	bitmap of which limbo RMIDs are above threshold
@@ -192,7 +192,7 @@ struct mbm_cntr_cfg {
  * @cntr_cfg:		array of assignable counters' configuration (indexed
  *			by counter ID)
  */
-struct rdt_mon_domain {
+struct rdt_l3_mon_domain {
 	struct rdt_domain_hdr		hdr;
 	unsigned int			ci_id;
 	unsigned long			*rmid_busy_llc;
@@ -364,10 +364,10 @@ struct resctrl_cpu_defaults {
 };
 
 struct resctrl_mon_config_info {
-	struct rdt_resource	*r;
-	struct rdt_mon_domain	*d;
-	u32			evtid;
-	u32			mon_config;
+	struct rdt_resource		*r;
+	struct rdt_l3_mon_domain	*d;
+	u32				evtid;
+	u32				mon_config;
 };
 
 /**
@@ -582,7 +582,7 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,
  *
  * This can be called from any CPU.
  */
-void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
+void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_l3_mon_domain *d,
 			     u32 closid, u32 rmid,
 			     enum resctrl_event_id eventid);
 
@@ -595,7 +595,7 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
  *
  * This can be called from any CPU.
  */
-void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *d);
+void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_l3_mon_domain *d);
 
 /**
  * resctrl_arch_reset_all_ctrls() - Reset the control for each CLOSID to its
@@ -621,7 +621,7 @@ void resctrl_arch_reset_all_ctrls(struct rdt_resource *r);
  *
  * This can be called from any CPU.
  */
-void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d,
 			      enum resctrl_event_id evtid, u32 rmid, u32 closid,
 			      u32 cntr_id, bool assign);
 
@@ -659,7 +659,7 @@ int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
  *
  * This can be called from any CPU.
  */
-void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d,
 			     u32 closid, u32 rmid, int cntr_id,
 			     enum resctrl_event_id eventid);
 
arch/x86/kernel/cpu/resctrl/internal.h +8 -8
index 9f4c2f0aaf5c..6eca3d522fcc 100644
@@ -60,17 +60,17 @@ struct rdt_hw_ctrl_domain {
 };
 
 /**
- * struct rdt_hw_mon_domain - Arch private attributes of a set of CPUs that share
- *			      a resource for a monitor function
- * @d_resctrl:	Properties exposed to the resctrl file system
+ * struct rdt_hw_l3_mon_domain - Arch private attributes of a set of CPUs that share
+ *				 a resource for a monitor function
+ * @d_resctrl:		Properties exposed to the resctrl file system
  * @arch_mbm_states:	Per-event pointer to the MBM event's saved state.
  *			An MBM event's state is an array of struct arch_mbm_state
  *			indexed by RMID on x86.
  *
  * Members of this structure are accessed via helpers that provide abstraction.
  */
-struct rdt_hw_mon_domain {
-	struct rdt_mon_domain		d_resctrl;
+struct rdt_hw_l3_mon_domain {
+	struct rdt_l3_mon_domain	d_resctrl;
 	struct arch_mbm_state		*arch_mbm_states[QOS_NUM_L3_MBM_EVENTS];
 };
 
@@ -79,9 +79,9 @@ static inline struct rdt_hw_ctrl_domain *resctrl_to_arch_ctrl_dom(struct rdt_ctr
 	return container_of(r, struct rdt_hw_ctrl_domain, d_resctrl);
 }
 
-static inline struct rdt_hw_mon_domain *resctrl_to_arch_mon_dom(struct rdt_mon_domain *r)
+static inline struct rdt_hw_l3_mon_domain *resctrl_to_arch_mon_dom(struct rdt_l3_mon_domain *r)
 {
-	return container_of(r, struct rdt_hw_mon_domain, d_resctrl);
+	return container_of(r, struct rdt_hw_l3_mon_domain, d_resctrl);
 }
 
 /**
@@ -135,7 +135,7 @@ static inline struct rdt_hw_resource *resctrl_to_arch_res(struct rdt_resource *r
 
 extern struct rdt_hw_resource rdt_resources_all[];
 
-void arch_mon_domain_online(struct rdt_resource *r, struct rdt_mon_domain *d);
+void arch_mon_domain_online(struct rdt_resource *r, struct rdt_l3_mon_domain *d);
 
 /* CPUID.(EAX=10H, ECX=ResID=1).EAX */
 union cpuid_0x10_1_eax {
fs/resctrl/internal.h +4 -4
index 698ed84fd073..d9e291d94926 100644
@@ -369,7 +369,7 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 
 int resctrl_mon_resource_init(void);
 
-void mbm_setup_overflow_handler(struct rdt_mon_domain *dom,
+void mbm_setup_overflow_handler(struct rdt_l3_mon_domain *dom,
 				unsigned long delay_ms,
 				int exclude_cpu);
 
@@ -377,14 +377,14 @@ void mbm_handle_overflow(struct work_struct *work);
 
 bool is_mba_sc(struct rdt_resource *r);
 
-void cqm_setup_limbo_handler(struct rdt_mon_domain *dom, unsigned long delay_ms,
+void cqm_setup_limbo_handler(struct rdt_l3_mon_domain *dom, unsigned long delay_ms,
 			     int exclude_cpu);
 
 void cqm_handle_limbo(struct work_struct *work);
 
-bool has_busy_rmid(struct rdt_mon_domain *d);
+bool has_busy_rmid(struct rdt_l3_mon_domain *d);
 
-void __check_limbo(struct rdt_mon_domain *d, bool force_free);
+void __check_limbo(struct rdt_l3_mon_domain *d, bool force_free);
 
 void resctrl_file_fflags_init(const char *config, unsigned long fflags);
 
arch/x86/kernel/cpu/resctrl/core.c +7 -7
index 44495bb915d5..8137c1442139 100644
@@ -363,7 +363,7 @@ static void ctrl_domain_free(struct rdt_hw_ctrl_domain *hw_dom)
 	kfree(hw_dom);
 }
 
-static void mon_domain_free(struct rdt_hw_mon_domain *hw_dom)
+static void mon_domain_free(struct rdt_hw_l3_mon_domain *hw_dom)
 {
 	int idx;
 
@@ -400,7 +400,7 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_ctrl_domain *
  * @num_rmid:	The size of the MBM counter array
  * @hw_dom:	The domain that owns the allocated arrays
  */
-static int arch_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_mon_domain *hw_dom)
+static int arch_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_l3_mon_domain *hw_dom)
 {
 	size_t tsize = sizeof(*hw_dom->arch_mbm_states[0]);
 	enum resctrl_event_id eventid;
@@ -498,8 +498,8 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r)
 
 static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct list_head *add_pos)
 {
-	struct rdt_hw_mon_domain *hw_dom;
-	struct rdt_mon_domain *d;
+	struct rdt_hw_l3_mon_domain *hw_dom;
+	struct rdt_l3_mon_domain *d;
 	struct cacheinfo *ci;
 	int err;
 
@@ -648,13 +648,13 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
 
 	switch (r->rid) {
 	case RDT_RESOURCE_L3: {
-		struct rdt_hw_mon_domain *hw_dom;
-		struct rdt_mon_domain *d;
+		struct rdt_hw_l3_mon_domain *hw_dom;
+		struct rdt_l3_mon_domain *d;
 
 		if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 			return;
 
-		d = container_of(hdr, struct rdt_mon_domain, hdr);
+		d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
 		hw_dom = resctrl_to_arch_mon_dom(d);
 		resctrl_offline_mon_domain(r, hdr);
 		list_del_rcu(&hdr->list);
arch/x86/kernel/cpu/resctrl/monitor.c +19 -19
index 1f47844b5246..3980c59cf5dc 100644
@@ -109,7 +109,7 @@ static inline u64 get_corrected_mbm_count(u32 rmid, unsigned long val)
  *
  * In RMID sharing mode there are fewer "logical RMID" values available
  * to accumulate data ("physical RMIDs" are divided evenly between SNC
- * nodes that share an L3 cache). Linux creates an rdt_mon_domain for
+ * nodes that share an L3 cache). Linux creates an rdt_l3_mon_domain for
  * each SNC node.
  *
  * The value loaded into IA32_PQR_ASSOC is the "logical RMID".
@@ -157,7 +157,7 @@ static int __rmid_read_phys(u32 prmid, enum resctrl_event_id eventid, u64 *val)
 	return 0;
 }
 
-static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_dom,
+static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_l3_mon_domain *hw_dom,
 						 u32 rmid,
 						 enum resctrl_event_id eventid)
 {
@@ -171,11 +171,11 @@ static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_do
 	return state ? &state[rmid] : NULL;
 }
 
-void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
+void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_l3_mon_domain *d,
 			     u32 unused, u32 rmid,
 			     enum resctrl_event_id eventid)
 {
-	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
+	struct rdt_hw_l3_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
 	int cpu = cpumask_any(&d->hdr.cpu_mask);
 	struct arch_mbm_state *am;
 	u32 prmid;
@@ -194,9 +194,9 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
  * Assumes that hardware counters are also reset and thus that there is
  * no need to record initial non-zero counts.
  */
-void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *d)
+void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_l3_mon_domain *d)
 {
-	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
+	struct rdt_hw_l3_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
 	enum resctrl_event_id eventid;
 	int idx;
 
@@ -217,10 +217,10 @@ static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width)
 	return chunks >> shift;
 }
 
-static u64 get_corrected_val(struct rdt_resource *r, struct rdt_mon_domain *d,
+static u64 get_corrected_val(struct rdt_resource *r, struct rdt_l3_mon_domain *d,
 			     u32 rmid, enum resctrl_event_id eventid, u64 msr_val)
 {
-	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
+	struct rdt_hw_l3_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
 	struct arch_mbm_state *am;
 	u64 chunks;
@@ -242,9 +242,9 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 			   u32 unused, u32 rmid, enum resctrl_event_id eventid,
 			   u64 *val, void *ignored)
 {
-	struct rdt_hw_mon_domain *hw_dom;
+	struct rdt_hw_l3_mon_domain *hw_dom;
 	struct arch_mbm_state *am;
-	struct rdt_mon_domain *d;
+	struct rdt_l3_mon_domain *d;
 	u64 msr_val;
 	u32 prmid;
 	int cpu;
@@ -255,7 +255,7 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		return -EINVAL;
 
-	d = container_of(hdr, struct rdt_mon_domain, hdr);
+	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
 	hw_dom = resctrl_to_arch_mon_dom(d);
 	cpu = cpumask_any(&hdr->cpu_mask);
 	prmid = logical_rmid_to_physical_rmid(cpu, rmid);
@@ -309,11 +309,11 @@ static int __cntr_id_read(u32 cntr_id, u64 *val)
 	return 0;
 }
 
-void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d,
 			     u32 unused, u32 rmid, int cntr_id,
 			     enum resctrl_event_id eventid)
 {
-	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
+	struct rdt_hw_l3_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
 	struct arch_mbm_state *am;
 
 	am = get_arch_mbm_state(hw_dom, rmid, eventid);
@@ -329,14 +329,14 @@ int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 			   u32 unused, u32 rmid, int cntr_id,
 			   enum resctrl_event_id eventid, u64 *val)
 {
-	struct rdt_mon_domain *d;
+	struct rdt_l3_mon_domain *d;
 	u64 msr_val;
 	int ret;
 
 	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		return -EINVAL;
 
-	d = container_of(hdr, struct rdt_mon_domain, hdr);
+	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
 	ret = __cntr_id_read(cntr_id, &msr_val);
 	if (ret)
 		return ret;
@@ -360,7 +360,7 @@ int resctrl_arch_cntr_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
  * must adjust RMID counter numbers based on SNC node. See
  * logical_rmid_to_physical_rmid() for code that does this.
  */
-void arch_mon_domain_online(struct rdt_resource *r, struct rdt_mon_domain *d)
+void arch_mon_domain_online(struct rdt_resource *r, struct rdt_l3_mon_domain *d)
 {
 	if (snc_nodes_per_l3_cache > 1)
 		msr_clear_bit(MSR_RMID_SNC_CONFIG, 0);
@@ -512,7 +512,7 @@ static void resctrl_abmc_set_one_amd(void *arg)
  */
 static void _resctrl_abmc_enable(struct rdt_resource *r, bool enable)
 {
-	struct rdt_mon_domain *d;
+	struct rdt_l3_mon_domain *d;
 
 	lockdep_assert_cpus_held();
 
@@ -551,11 +551,11 @@ static void resctrl_abmc_config_one_amd(void *info)
 /*
  * Send an IPI to the domain to assign the counter to RMID, event pair.
  */
-void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_mon_domain *d,
+void resctrl_arch_config_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d,
 			      enum resctrl_event_id evtid, u32 rmid, u32 closid,
 			      u32 cntr_id, bool assign)
 {
-	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
+	struct rdt_hw_l3_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
 	union l3_qos_abmc_cfg abmc_cfg = { 0 };
 	struct arch_mbm_state *am;
 
fs/resctrl/ctrlmondata.c +1 -1
index 7b9fc5d3bdc8..c95f8eb8e731 100644
@@ -622,7 +622,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	r = resctrl_arch_get_resource(resid);
 
 	if (md->sum) {
-		struct rdt_mon_domain *d;
+		struct rdt_l3_mon_domain *d;
 
 		/*
 		 * This file requires summing across all domains that share
fs/resctrl/monitor.c +35 -35

276 lines not on this page · raw message

fs/resctrl/rdtgroup.c +20 -20
index f5a65c48bcab..f9d35eb86dcc 100644
@@ -1618,7 +1618,7 @@ static void mondata_config_read(struct resctrl_mon_config_info *mon_info)
 static int mbm_config_show(struct seq_file *s, struct rdt_resource *r, u32 evtid)
 {
 	struct resctrl_mon_config_info mon_info;
-	struct rdt_mon_domain *dom;
+	struct rdt_l3_mon_domain *dom;
 	bool sep = false;
 
 	cpus_read_lock();
@@ -1666,7 +1666,7 @@ static int mbm_local_bytes_config_show(struct kernfs_open_file *of,
 }
 
 static void mbm_config_write_domain(struct rdt_resource *r,
-				    struct rdt_mon_domain *d, u32 evtid, u32 val)
+				    struct rdt_l3_mon_domain *d, u32 evtid, u32 val)
 {
 	struct resctrl_mon_config_info mon_info = {0};
 
@@ -1707,8 +1707,8 @@ static void mbm_config_write_domain(struct rdt_resource *r,
 static int mon_config_write(struct rdt_resource *r, char *tok, u32 evtid)
 {
 	char *dom_str = NULL, *id_str;
+	struct rdt_l3_mon_domain *d;
 	unsigned long dom_id, val;
-	struct rdt_mon_domain *d;
 
 	/* Walking r->domains, ensure it can't race with cpuhp */
 	lockdep_assert_cpus_held();
@@ -2716,7 +2716,7 @@ static int rdt_get_tree(struct fs_context *fc)
 {
 	struct rdt_fs_context *ctx = rdt_fc2context(fc);
 	unsigned long flags = RFTYPE_CTRL_BASE;
-	struct rdt_mon_domain *dom;
+	struct rdt_l3_mon_domain *dom;
 	struct rdt_resource *r;
 	int ret;
 
@@ -3167,7 +3167,7 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,
 					   struct rdt_domain_hdr *hdr)
 {
 	struct rdtgroup *prgrp, *crgrp;
-	struct rdt_mon_domain *d;
+	struct rdt_l3_mon_domain *d;
 	char subname[32];
 	bool snc_mode;
 	char name[32];
@@ -3175,7 +3175,7 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,
 	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		return;
 
-	d = container_of(hdr, struct rdt_mon_domain, hdr);
+	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
 	snc_mode = r->mon_scope == RESCTRL_L3_NODE;
 	sprintf(name, "mon_%s_%02d", r->name, snc_mode ? d->ci_id : d->hdr.id);
 	if (snc_mode)
@@ -3194,7 +3194,7 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_domain_hdr *hdr,
 			     bool do_sum)
 {
 	struct rmid_read rr = {0};
-	struct rdt_mon_domain *d;
+	struct rdt_l3_mon_domain *d;
 	struct mon_data *priv;
 	struct mon_evt *mevt;
 	int ret, domid;
@@ -3202,7 +3202,7 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_domain_hdr *hdr,
 	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		return -EINVAL;
 
-	d = container_of(hdr, struct rdt_mon_domain, hdr);
+	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
 	for_each_mon_event(mevt) {
 		if (mevt->rid != r->rid || !mevt->enabled)
 			continue;
@@ -3227,7 +3227,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,
 				struct rdt_resource *r, struct rdtgroup *prgrp)
 {
 	struct kernfs_node *kn, *ckn;
-	struct rdt_mon_domain *d;
+	struct rdt_l3_mon_domain *d;
 	char name[32];
 	bool snc_mode;
 	int ret = 0;
@@ -3237,7 +3237,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,
 	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		return -EINVAL;
 
-	d = container_of(hdr, struct rdt_mon_domain, hdr);
+	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
 	snc_mode = r->mon_scope == RESCTRL_L3_NODE;
 	sprintf(name, "mon_%s_%02d", r->name, snc_mode ? d->ci_id : d->hdr.id);
 	kn = kernfs_find_and_get(parent_kn, name);
@@ -4180,7 +4180,7 @@ static void rdtgroup_setup_default(void)
 	mutex_unlock(&rdtgroup_mutex);
 }
 
-static void domain_destroy_mon_state(struct rdt_mon_domain *d)
+static void domain_destroy_mon_state(struct rdt_l3_mon_domain *d)
 {
 	int idx;
 
@@ -4204,14 +4204,14 @@ void resctrl_offline_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain
 
 void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr)
 {
-	struct rdt_mon_domain *d;
+	struct rdt_l3_mon_domain *d;
 
 	mutex_lock(&rdtgroup_mutex);
 
 	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		goto out_unlock;
 
-	d = container_of(hdr, struct rdt_mon_domain, hdr);
+	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
 
 	/*
 	 * If resctrl is mounted, remove all the
@@ -4253,7 +4253,7 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h
  *
  * Returns 0 for success, or -ENOMEM.
  */
-static int domain_setup_mon_state(struct rdt_resource *r, struct rdt_mon_domain *d)
+static int domain_setup_mon_state(struct rdt_resource *r, struct rdt_l3_mon_domain *d)
 {
 	u32 idx_limit = resctrl_arch_system_num_rmid_idx();
 	size_t tsize = sizeof(*d->mbm_states[0]);
@@ -4311,7 +4311,7 @@ int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d
 
 int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr)
 {
-	struct rdt_mon_domain *d;
+	struct rdt_l3_mon_domain *d;
 	int err = -EINVAL;
 
 	mutex_lock(&rdtgroup_mutex);
@@ -4319,7 +4319,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr
 	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		goto out_unlock;
 
-	d = container_of(hdr, struct rdt_mon_domain, hdr);
+	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
 	err = domain_setup_mon_state(r, d);
 	if (err)
 		goto out_unlock;
@@ -4366,10 +4366,10 @@ static void clear_childcpus(struct rdtgroup *r, unsigned int cpu)
 	}
 }
 
-static struct rdt_mon_domain *get_mon_domain_from_cpu(int cpu,
-						      struct rdt_resource *r)
+static struct rdt_l3_mon_domain *get_mon_domain_from_cpu(int cpu,
+							 struct rdt_resource *r)
 {
-	struct rdt_mon_domain *d;
+	struct rdt_l3_mon_domain *d;
 
 	lockdep_assert_cpus_held();
 
@@ -4385,7 +4385,7 @@ static struct rdt_mon_domain *get_mon_domain_from_cpu(int cpu,
 void resctrl_offline_cpu(unsigned int cpu)
 {
 	struct rdt_resource *l3 = resctrl_arch_get_resource(RDT_RESOURCE_L3);
-	struct rdt_mon_domain *d;
+	struct rdt_l3_mon_domain *d;
 	struct rdtgroup *rdtgrp;
 
 	mutex_lock(&rdtgroup_mutex);
Signature · 1 line
2.51.0
maintainer
#13 lore raw

Re: [PATCH v12 07/31] x86,fs/resctrl: Rename struct rdt_mon_domain and rdt_hw_mon_domain

Hi Tony,

Quoted 9 lines · Tony Luck

The upcoming telemetry event monitoring are not tied to the L3 resource and will have a new domain structures.

Rename the L3 resource specific domain data structures to include "l3_" in their names to avoid confusion between the different resource specific domain structures:
rdt_mon_domain -> rdt_l3_mon_domain
rdt_hw_mon_domain -> rdt_hw_l3_mon_domain

Quoted 4 lines

No functional change.

Some style hiccups with the changes added since the RB tag:

...

arch/x86/kernel/cpu/resctrl/monitor.c int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr, · line 247

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/monitor.c +2 -2
@@ -242,9 +242,9 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 			   u32 unused, u32 rmid, enum resctrl_event_id eventid,
 			   u64 *val, void *ignored)
 {
-	struct rdt_hw_mon_domain *hw_dom;
+	struct rdt_hw_l3_mon_domain *hw_dom;
 	struct arch_mbm_state *am;
-	struct rdt_mon_domain *d;
+	struct rdt_l3_mon_domain *d;

This is a new hunk and it needs to change to maintain reverse fir order.

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/monitor.c +0 -0
 	u64 msr_val;
 	u32 prmid;
 	int cpu;

...

fs/resctrl/rdtgroup.c static int mon_add_all_files(struct kernfs_node *kn, struct rdt_domain_hdr *hdr, · line 3200

Quoted 9 lines
fs/resctrl/rdtgroup.c +1 -1
@@ -3194,7 +3194,7 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_domain_hdr *hdr,
 			     bool do_sum)
 {
 	struct rmid_read rr = {0};
-	struct rdt_mon_domain *d;
+	struct rdt_l3_mon_domain *d;
 	struct mon_data *priv;
 	struct mon_evt *mevt;
 	int ret, domid;

Another new hunk that needs to change to maintain reverse fir order.

Reinette

author maintainer
#14 lore raw

PATCH v12 8/31 x86,fs/resctrl: Rename some L3 specific functions

In reply to Tony Luck

With the arrival of monitor events tied to new domains associated with a different resource it would be clearer if the L3 resource specific functions are more accurately named.

Rename three groups of functions:

Functions that allocate/free architecture per-RMID MBM state information:
arch_domain_mbm_alloc() -> l3_mon_domain_mbm_alloc()
mon_domain_free() -> l3_mon_domain_free()

Functions that allocate/free filesystem per-RMID MBM state information:
domain_setup_mon_state() -> domain_setup_l3_mon_state()
domain_destroy_mon_state() -> domain_destroy_l3_mon_state()

Initialization/exit:
rdt_get_mon_l3_config() -> rdt_get_l3_mon_config()
resctrl_mon_resource_init() -> resctrl_l3_mon_resource_init()
resctrl_mon_resource_exit() -> resctrl_l3_mon_resource_exit()

Fixed kerneldoc "Return:" for l3_mon_domain_mbm_alloc(), resctrl_l3_mon_resource_init(), and domain_setup_l3_mon_state().

---
 arch/x86/kernel/cpu/resctrl/internal.h |  2 +-
 fs/resctrl/internal.h                  |  6 +++---
 arch/x86/kernel/cpu/resctrl/core.c     | 20 +++++++++++---------
 arch/x86/kernel/cpu/resctrl/monitor.c  |  2 +-
 fs/resctrl/monitor.c                   |  8 ++++----
 fs/resctrl/rdtgroup.c                  | 24 ++++++++++++------------
 6 files changed, 32 insertions(+), 30 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +1 -1
index 6eca3d522fcc..14fadcff0d2b 100644
@@ -208,7 +208,7 @@ union l3_qos_abmc_cfg {
 
 void rdt_ctrl_update(void *arg);
 
-int rdt_get_mon_l3_config(struct rdt_resource *r);
+int rdt_get_l3_mon_config(struct rdt_resource *r);
 
 bool rdt_cpu_has(int flag);
 
fs/resctrl/internal.h +3 -3
index d9e291d94926..88b4489b68e1 100644
@@ -357,7 +357,9 @@ int alloc_rmid(u32 closid);
 
 void free_rmid(u32 closid, u32 rmid);
 
-void resctrl_mon_resource_exit(void);
+int resctrl_l3_mon_resource_init(void);
+
+void resctrl_l3_mon_resource_exit(void);
 
 void mon_event_count(void *info);
 
@@ -367,8 +369,6 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 		    struct rdt_domain_hdr *hdr, struct rdtgroup *rdtgrp,
 		    cpumask_t *cpumask, int evtid, int first);
 
-int resctrl_mon_resource_init(void);
-
 void mbm_setup_overflow_handler(struct rdt_l3_mon_domain *dom,
 				unsigned long delay_ms,
 				int exclude_cpu);
arch/x86/kernel/cpu/resctrl/core.c +11 -9
index 8137c1442139..a9fa506f09fa 100644
@@ -363,7 +363,7 @@ static void ctrl_domain_free(struct rdt_hw_ctrl_domain *hw_dom)
 	kfree(hw_dom);
 }
 
-static void mon_domain_free(struct rdt_hw_l3_mon_domain *hw_dom)
+static void l3_mon_domain_free(struct rdt_hw_l3_mon_domain *hw_dom)
 {
 	int idx;
 
@@ -396,11 +396,13 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_ctrl_domain *
 }
 
 /**
- * arch_domain_mbm_alloc() - Allocate arch private storage for the MBM counters
+ * l3_mon_domain_mbm_alloc() - Allocate arch private storage for the MBM counters
  * @num_rmid:	The size of the MBM counter array
  * @hw_dom:	The domain that owns the allocated arrays
+ *
+ * Return:	%0 for success; Error code otherwise.
  */
-static int arch_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_l3_mon_domain *hw_dom)
+static int l3_mon_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_l3_mon_domain *hw_dom)
 {
 	size_t tsize = sizeof(*hw_dom->arch_mbm_states[0]);
 	enum resctrl_event_id eventid;
@@ -514,7 +516,7 @@ static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct
 	ci = get_cpu_cacheinfo_level(cpu, RESCTRL_L3_CACHE);
 	if (!ci) {
 		pr_warn_once("Can't find L3 cache for CPU:%d resource %s\n", cpu, r->name);
-		mon_domain_free(hw_dom);
+		l3_mon_domain_free(hw_dom);
 		return;
 	}
 	d->ci_id = ci->id;
@@ -522,8 +524,8 @@ static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct
 
 	arch_mon_domain_online(r, d);
 
-	if (arch_domain_mbm_alloc(r->mon.num_rmid, hw_dom)) {
-		mon_domain_free(hw_dom);
+	if (l3_mon_domain_mbm_alloc(r->mon.num_rmid, hw_dom)) {
+		l3_mon_domain_free(hw_dom);
 		return;
 	}
 
@@ -533,7 +535,7 @@ static void l3_mon_domain_setup(int cpu, int id, struct rdt_resource *r, struct
 	if (err) {
 		list_del_rcu(&d->hdr.list);
 		synchronize_rcu();
-		mon_domain_free(hw_dom);
+		l3_mon_domain_free(hw_dom);
 	}
 }
 
@@ -659,7 +661,7 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
 		resctrl_offline_mon_domain(r, hdr);
 		list_del_rcu(&hdr->list);
 		synchronize_rcu();
-		mon_domain_free(hw_dom);
+		l3_mon_domain_free(hw_dom);
 		break;
 	}
 	default:
@@ -908,7 +910,7 @@ static __init bool get_rdt_mon_resources(void)
 	if (!ret)
 		return false;
 
-	return !rdt_get_mon_l3_config(r);
+	return !rdt_get_l3_mon_config(r);
 }
 
 static __init void __check_quirks_intel(void)
arch/x86/kernel/cpu/resctrl/monitor.c +1 -1
index 3980c59cf5dc..8f9a2ec3bd04 100644
@@ -429,7 +429,7 @@ static __init int snc_get_config(void)
 	return ret;
 }
 
-int __init rdt_get_mon_l3_config(struct rdt_resource *r)
+int __init rdt_get_l3_mon_config(struct rdt_resource *r)
 {
 	unsigned int mbm_offset = boot_cpu_data.x86_cache_mbm_width_offset;
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);
fs/resctrl/monitor.c +4 -4
index 69b6aa31b986..26e2ecbddb5c 100644
@@ -1754,7 +1754,7 @@ ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf,
 }
 
 /**
- * resctrl_mon_resource_init() - Initialise global monitoring structures.
+ * resctrl_l3_mon_resource_init() - Initialise global monitoring structures.
  *
  * Allocate and initialise global monitor resources that do not belong to a
  * specific domain. i.e. the rmid_ptrs[] used for the limbo and free lists.
@@ -1763,9 +1763,9 @@ ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf,
  * Resctrl's cpuhp callbacks may be called before this point to bring a domain
  * online.
  *
- * Returns 0 for success, or -ENOMEM.
+ * Return: %0 for success; -ENOMEM otherwise.
  */
-int resctrl_mon_resource_init(void)
+int resctrl_l3_mon_resource_init(void)
 {
 	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
 	int ret;
@@ -1817,7 +1817,7 @@ int resctrl_mon_resource_init(void)
 	return 0;
 }
 
-void resctrl_mon_resource_exit(void)
+void resctrl_l3_mon_resource_exit(void)
 {
 	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
 
fs/resctrl/rdtgroup.c +12 -12
index f9d35eb86dcc..3845ed277355 100644
@@ -4180,7 +4180,7 @@ static void rdtgroup_setup_default(void)
 	mutex_unlock(&rdtgroup_mutex);
 }
 
-static void domain_destroy_mon_state(struct rdt_l3_mon_domain *d)
+static void domain_destroy_l3_mon_state(struct rdt_l3_mon_domain *d)
 {
 	int idx;
 
@@ -4235,13 +4235,13 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h
 		cancel_delayed_work(&d->cqm_limbo);
 	}
 
-	domain_destroy_mon_state(d);
+	domain_destroy_l3_mon_state(d);
 out_unlock:
 	mutex_unlock(&rdtgroup_mutex);
 }
 
 /**
- * domain_setup_mon_state() -  Initialise domain monitoring structures.
+ * domain_setup_l3_mon_state() -  Initialise domain monitoring structures.
  * @r:	The resource for the newly online domain.
  * @d:	The newly online domain.
  *
@@ -4249,11 +4249,11 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h
  * Called when the first CPU of a domain comes online, regardless of whether
  * the filesystem is mounted.
  * During boot this may be called before global allocations have been made by
- * resctrl_mon_resource_init().
+ * resctrl_l3_mon_resource_init().
  *
- * Returns 0 for success, or -ENOMEM.
+ * Return: %0 for success; -ENOMEM otherwise.
  */
-static int domain_setup_mon_state(struct rdt_resource *r, struct rdt_l3_mon_domain *d)
+static int domain_setup_l3_mon_state(struct rdt_resource *r, struct rdt_l3_mon_domain *d)
 {
 	u32 idx_limit = resctrl_arch_system_num_rmid_idx();
 	size_t tsize = sizeof(*d->mbm_states[0]);
@@ -4320,7 +4320,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr
 		goto out_unlock;
 
 	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
-	err = domain_setup_mon_state(r, d);
+	err = domain_setup_l3_mon_state(r, d);
 	if (err)
 		goto out_unlock;
 
@@ -4435,13 +4435,13 @@ int resctrl_init(void)
 
 	thread_throttle_mode_init();
 
-	ret = resctrl_mon_resource_init();
+	ret = resctrl_l3_mon_resource_init();
 	if (ret)
 		return ret;
 
 	ret = sysfs_create_mount_point(fs_kobj, "resctrl");
 	if (ret) {
-		resctrl_mon_resource_exit();
+		resctrl_l3_mon_resource_exit();
 		return ret;
 	}
 
@@ -4476,7 +4476,7 @@ int resctrl_init(void)
 
 cleanup_mountpoint:
 	sysfs_remove_mount_point(fs_kobj, "resctrl");
-	resctrl_mon_resource_exit();
+	resctrl_l3_mon_resource_exit();
 
 	return ret;
 }
@@ -4512,7 +4512,7 @@ static bool resctrl_online_domains_exist(void)
  * When called by the architecture code, all CPUs and resctrl domains must be
  * offline. This ensures the limbo and overflow handlers are not scheduled to
  * run, meaning the data structures they access can be freed by
- * resctrl_mon_resource_exit().
+ * resctrl_l3_mon_resource_exit().
  *
  * After resctrl_exit() returns, the architecture code should return an
  * error from all resctrl_arch_ functions that can do this.
@@ -4539,5 +4539,5 @@ void resctrl_exit(void)
 	 * it can be used to umount resctrl.
 	 */
 
-	resctrl_mon_resource_exit();
+	resctrl_l3_mon_resource_exit();
 }
Signature · 1 line
2.51.0
maintainer
#15 lore raw

Re: [PATCH v12 08/31] x86,fs/resctrl: Rename some L3 specific functions

Hi Tony,

Quoted 14 lines · Tony Luck

With the arrival of monitor events tied to new domains associated with a different resource it would be clearer if the L3 resource specific functions are more accurately named.

Rename three groups of functions:

Functions that allocate/free architecture per-RMID MBM state information:
arch_domain_mbm_alloc() -> l3_mon_domain_mbm_alloc()
mon_domain_free() -> l3_mon_domain_free()

Functions that allocate/free filesystem per-RMID MBM state information:
domain_setup_mon_state() -> domain_setup_l3_mon_state()
domain_destroy_mon_state() -> domain_destroy_l3_mon_state()

Quoted 6 lines

Initialization/exit:
rdt_get_mon_l3_config() -> rdt_get_l3_mon_config()
resctrl_mon_resource_init() -> resctrl_l3_mon_resource_init()
resctrl_mon_resource_exit() -> resctrl_l3_mon_resource_exit()

Fixed kerneldoc "Return:" for l3_mon_domain_mbm_alloc(),

Please always use imperative tone, be clear what problem is being fixed, and avoid descriptions that can be seen from the patch self.

Consider, for example:
"Ensure kernel-doc descriptions of these functions' return values are
present and correctly formatted."

arch/x86/kernel/cpu/resctrl/core.c static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_ctrl_domain * · line 403

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c +3 -1
@@ -396,11 +396,13 @@ static int domain_setup_ctrlval(struct rdt_resource *r, struct rdt_ctrl_domain *
 }
 
 /**
- * arch_domain_mbm_alloc() - Allocate arch private storage for the MBM counters
+ * l3_mon_domain_mbm_alloc() - Allocate arch private storage for the MBM counters
  * @num_rmid:	The size of the MBM counter array
  * @hw_dom:	The domain that owns the allocated arrays
+ *
+ * Return:	%0 for success; Error code otherwise.

While using % to format a constant is correct kernel-doc syntax it is not consistent with how resctrl formats these statements. Please rather keep the addition consistent with rest of resctrl. Since only -ENOMEM is possible this can also be made consistent with other changes in this regard:
Return: 0 for success, or -ENOMEM.

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/core.c +1 -1
  */
-static int arch_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_l3_mon_domain *hw_dom)
+static int l3_mon_domain_mbm_alloc(u32 num_rmid, struct rdt_hw_l3_mon_domain *hw_dom)
 {
 	size_t tsize = sizeof(*hw_dom->arch_mbm_states[0]);
 	enum resctrl_event_id eventid;

...

Quoted 14 lines
fs/resctrl/monitor.c +1 -1
index 69b6aa31b986..26e2ecbddb5c 100644
@@ -1754,7 +1754,7 @@ ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf,
 }
 
 /**
- * resctrl_mon_resource_init() - Initialise global monitoring structures.
+ * resctrl_l3_mon_resource_init() - Initialise global monitoring structures.
  *
  * Allocate and initialise global monitor resources that do not belong to a
  * specific domain. i.e. the rmid_ptrs[] used for the limbo and free lists.

fs/resctrl/monitor.c ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf, · line 1767

Quoted 6 lines
fs/resctrl/monitor.c +1 -1
@@ -1763,9 +1763,9 @@ ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf,
  * Resctrl's cpuhp callbacks may be called before this point to bring a domain
  * online.
  *
- * Returns 0 for success, or -ENOMEM.
+ * Return: %0 for success; -ENOMEM otherwise.
  */

The only actual fix is to add a ":", no? The rest are style changes that (a) diverge from the rest of resctrl and (b) create inconsistency with itself since % is used for prefix of the 0 constant but not as prefix for -ENOMEM. What is motivation for the style changes?

Quoted 5 lines
fs/resctrl/monitor.c +1 -1
-int resctrl_mon_resource_init(void)
+int resctrl_l3_mon_resource_init(void)
 {
 	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
 	int ret;

...

fs/resctrl/monitor.c void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h · line 4254

Quoted 9 lines
fs/resctrl/monitor.c +2 -2
@@ -4249,11 +4249,11 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h
  * Called when the first CPU of a domain comes online, regardless of whether
  * the filesystem is mounted.
  * During boot this may be called before global allocations have been made by
- * resctrl_mon_resource_init().
+ * resctrl_l3_mon_resource_init().
  *
- * Returns 0 for success, or -ENOMEM.
+ * Return: %0 for success; -ENOMEM otherwise.

same here.

fs/resctrl/monitor.c void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h · line 4259

Quoted 6 lines
fs/resctrl/monitor.c +1 -1
  */
-static int domain_setup_mon_state(struct rdt_resource *r, struct rdt_l3_mon_domain *d)
+static int domain_setup_l3_mon_state(struct rdt_resource *r, struct rdt_l3_mon_domain *d)
 {
 	u32 idx_limit = resctrl_arch_system_num_rmid_idx();
 	size_t tsize = sizeof(*d->mbm_states[0]);

Reinette

author maintainer
#16 lore raw

PATCH v12 9/31 fs/resctrl: Make event details accessible to functions when reading events

In reply to Tony Luck

Reading monitoring event data from MMIO requires more context than the event id to be able to read the correct memory location. struct mon_evt is the appropriate place for this event specific context.

Prepare for addition of extra fields to struct mon_evt by changing the calling conventions to pass a pointer to the mon_evt structure instead of just the event id.

---
 fs/resctrl/internal.h    | 10 +++++-----
 fs/resctrl/ctrlmondata.c | 18 +++++++++---------
 fs/resctrl/monitor.c     | 24 ++++++++++++------------
 fs/resctrl/rdtgroup.c    |  6 +++---
 4 files changed, 29 insertions(+), 29 deletions(-)
fs/resctrl/internal.h +5 -5
index 88b4489b68e1..12a2ab7e3c9b 100644
@@ -81,7 +81,7 @@ extern struct mon_evt mon_event_all[QOS_NUM_EVENTS];
  * struct mon_data - Monitoring details for each event file.
  * @list:            Member of the global @mon_data_kn_priv_list list.
  * @rid:             Resource id associated with the event file.
- * @evtid:           Event id associated with the event file.
+ * @evt:             Event structure associated with the event file.
  * @sum:             Set when event must be summed across multiple
  *                   domains.
  * @domid:           When @sum is zero this is the domain to which
@@ -95,7 +95,7 @@ extern struct mon_evt mon_event_all[QOS_NUM_EVENTS];
 struct mon_data {
 	struct list_head	list;
 	enum resctrl_res_level	rid;
-	enum resctrl_event_id	evtid;
+	struct mon_evt		*evt;
 	int			domid;
 	bool			sum;
 };
@@ -108,7 +108,7 @@ struct mon_data {
  * @r:	   Resource describing the properties of the event being read.
  * @hdr:   Header of domain that the counter should be read from. If NULL then
  *	   sum all domains in @r sharing L3 @ci.id
- * @evtid: Which monitor event to read.
+ * @evt:   Which monitor event to read.
  * @first: Initialize MBM counter when true.
  * @ci:    Cacheinfo for L3. Only set when @hdr is NULL. Used when summing
  *	   domains.
@@ -126,7 +126,7 @@ struct rmid_read {
 	struct rdtgroup		*rgrp;
 	struct rdt_resource	*r;
 	struct rdt_domain_hdr	*hdr;
-	enum resctrl_event_id	evtid;
+	struct mon_evt		*evt;
 	bool			first;
 	struct cacheinfo	*ci;
 	bool			is_mbm_cntr;
@@ -367,7 +367,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg);
 
 void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 		    struct rdt_domain_hdr *hdr, struct rdtgroup *rdtgrp,
-		    cpumask_t *cpumask, int evtid, int first);
+		    cpumask_t *cpumask, struct mon_evt *evt, int first);
 
 void mbm_setup_overflow_handler(struct rdt_l3_mon_domain *dom,
 				unsigned long delay_ms,
fs/resctrl/ctrlmondata.c +9 -9
index c95f8eb8e731..77602563cb1f 100644
@@ -548,7 +548,7 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,
 
 void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 		    struct rdt_domain_hdr *hdr, struct rdtgroup *rdtgrp,
-		    cpumask_t *cpumask, int evtid, int first)
+		    cpumask_t *cpumask, struct mon_evt *evt, int first)
 {
 	int cpu;
 
@@ -559,15 +559,15 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 	 * Setup the parameters to pass to mon_event_count() to read the data.
 	 */
 	rr->rgrp = rdtgrp;
-	rr->evtid = evtid;
+	rr->evt = evt;
 	rr->r = r;
 	rr->hdr = hdr;
 	rr->first = first;
 	if (resctrl_arch_mbm_cntr_assign_enabled(r) &&
-	    resctrl_is_mbm_event(evtid)) {
+	    resctrl_is_mbm_event(evt->evtid)) {
 		rr->is_mbm_cntr = true;
 	} else {
-		rr->arch_mon_ctx = resctrl_arch_mon_ctx_alloc(r, evtid);
+		rr->arch_mon_ctx = resctrl_arch_mon_ctx_alloc(r, evt->evtid);
 		if (IS_ERR(rr->arch_mon_ctx)) {
 			rr->err = -EINVAL;
 			return;
@@ -588,20 +588,20 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 		smp_call_on_cpu(cpu, smp_mon_event_count, rr, false);
 
 	if (rr->arch_mon_ctx)
-		resctrl_arch_mon_ctx_free(r, evtid, rr->arch_mon_ctx);
+		resctrl_arch_mon_ctx_free(r, evt->evtid, rr->arch_mon_ctx);
 }
 
 int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 {
 	struct kernfs_open_file *of = m->private;
 	enum resctrl_res_level resid;
-	enum resctrl_event_id evtid;
 	struct rdt_domain_hdr *hdr;
 	struct rmid_read rr = {0};
 	struct rdtgroup *rdtgrp;
 	int domid, cpu, ret = 0;
 	struct rdt_resource *r;
 	struct cacheinfo *ci;
+	struct mon_evt *evt;
 	struct mon_data *md;
 
 	rdtgrp = rdtgroup_kn_lock_live(of->kn);
@@ -618,7 +618,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 
 	resid = md->rid;
 	domid = md->domid;
-	evtid = md->evtid;
+	evt = md->evt;
 	r = resctrl_arch_get_resource(resid);
 
 	if (md->sum) {
@@ -638,7 +638,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 					continue;
 				rr.ci = ci;
 				mon_event_read(&rr, r, NULL, rdtgrp,
-					       &ci->shared_cpu_map, evtid, false);
+					       &ci->shared_cpu_map, evt, false);
 				goto checkresult;
 			}
 		}
@@ -654,7 +654,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 			ret = -ENOENT;
 			goto out;
 		}
-		mon_event_read(&rr, r, hdr, rdtgrp, &hdr->cpu_mask, evtid, false);
+		mon_event_read(&rr, r, hdr, rdtgrp, &hdr->cpu_mask, evt, false);
 	}
 
 checkresult:
fs/resctrl/monitor.c +12 -12
index 26e2ecbddb5c..c7f0af8a239e 100644
@@ -429,7 +429,7 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 			return -EINVAL;
 		d = container_of(rr->hdr, struct rdt_l3_mon_domain, hdr);
 
-		cntr_id = mbm_cntr_get(rr->r, d, rdtgrp, rr->evtid);
+		cntr_id = mbm_cntr_get(rr->r, d, rdtgrp, rr->evt->evtid);
 		if (cntr_id < 0) {
 			rr->err = -ENOENT;
 			return -EINVAL;
@@ -442,10 +442,10 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 		d = container_of(rr->hdr, struct rdt_l3_mon_domain, hdr);
 
 		if (rr->is_mbm_cntr)
-			resctrl_arch_reset_cntr(rr->r, d, closid, rmid, cntr_id, rr->evtid);
+			resctrl_arch_reset_cntr(rr->r, d, closid, rmid, cntr_id, rr->evt->evtid);
 		else
-			resctrl_arch_reset_rmid(rr->r, d, closid, rmid, rr->evtid);
-		m = get_mbm_state(d, closid, rmid, rr->evtid);
+			resctrl_arch_reset_rmid(rr->r, d, closid, rmid, rr->evt->evtid);
+		m = get_mbm_state(d, closid, rmid, rr->evt->evtid);
 		if (m)
 			memset(m, 0, sizeof(struct mbm_state));
 		return 0;
@@ -457,10 +457,10 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 			return -EINVAL;
 		if (rr->is_mbm_cntr)
 			rr->err = resctrl_arch_cntr_read(rr->r, rr->hdr, closid, rmid, cntr_id,
-							 rr->evtid, &tval);
+							 rr->evt->evtid, &tval);
 		else
 			rr->err = resctrl_arch_rmid_read(rr->r, rr->hdr, closid, rmid,
-							 rr->evtid, &tval, rr->arch_mon_ctx);
+							 rr->evt->evtid, &tval, rr->arch_mon_ctx);
 		if (rr->err)
 			return rr->err;
 
@@ -486,10 +486,10 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 			continue;
 		if (rr->is_mbm_cntr)
 			err = resctrl_arch_cntr_read(rr->r, &d->hdr, closid, rmid, cntr_id,
-						     rr->evtid, &tval);
+						     rr->evt->evtid, &tval);
 		else
 			err = resctrl_arch_rmid_read(rr->r, &d->hdr, closid, rmid,
-						     rr->evtid, &tval, rr->arch_mon_ctx);
+						     rr->evt->evtid, &tval, rr->arch_mon_ctx);
 		if (!err) {
 			rr->val += tval;
 			ret = 0;
@@ -525,7 +525,7 @@ static void mbm_bw_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	if (!domain_header_is_valid(rr->hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		return;
 	d = container_of(rr->hdr, struct rdt_l3_mon_domain, hdr);
-	m = get_mbm_state(d, closid, rmid, rr->evtid);
+	m = get_mbm_state(d, closid, rmid, rr->evt->evtid);
 	if (WARN_ON_ONCE(!m))
 		return;
 
@@ -699,11 +699,11 @@ static void mbm_update_one_event(struct rdt_resource *r, struct rdt_l3_mon_domai
 
 	rr.r = r;
 	rr.hdr = &d->hdr;
-	rr.evtid = evtid;
+	rr.evt = &mon_event_all[evtid];
 	if (resctrl_arch_mbm_cntr_assign_enabled(r)) {
 		rr.is_mbm_cntr = true;
 	} else {
-		rr.arch_mon_ctx = resctrl_arch_mon_ctx_alloc(rr.r, rr.evtid);
+		rr.arch_mon_ctx = resctrl_arch_mon_ctx_alloc(rr.r, evtid);
 		if (IS_ERR(rr.arch_mon_ctx)) {
 			pr_warn_ratelimited("Failed to allocate monitor context: %ld",
 					    PTR_ERR(rr.arch_mon_ctx));
@@ -721,7 +721,7 @@ static void mbm_update_one_event(struct rdt_resource *r, struct rdt_l3_mon_domai
 		mbm_bw_count(rdtgrp, &rr);
 
 	if (rr.arch_mon_ctx)
-		resctrl_arch_mon_ctx_free(rr.r, rr.evtid, rr.arch_mon_ctx);
+		resctrl_arch_mon_ctx_free(rr.r, evtid, rr.arch_mon_ctx);
 }
 
 static void mbm_update(struct rdt_resource *r, struct rdt_l3_mon_domain *d,
fs/resctrl/rdtgroup.c +3 -3
index 3845ed277355..6112e1b83b87 100644
@@ -3038,7 +3038,7 @@ static struct mon_data *mon_get_kn_priv(enum resctrl_res_level rid, int domid,
 
 	list_for_each_entry(priv, &mon_data_kn_priv_list, list) {
 		if (priv->rid == rid && priv->domid == domid &&
-		    priv->sum == do_sum && priv->evtid == mevt->evtid)
+		    priv->sum == do_sum && priv->evt == mevt)
 			return priv;
 	}
 
@@ -3049,7 +3049,7 @@ static struct mon_data *mon_get_kn_priv(enum resctrl_res_level rid, int domid,
 	priv->rid = rid;
 	priv->domid = domid;
 	priv->sum = do_sum;
-	priv->evtid = mevt->evtid;
+	priv->evt = mevt;
 	list_add_tail(&priv->list, &mon_data_kn_priv_list);
 
 	return priv;
@@ -3216,7 +3216,7 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_domain_hdr *hdr,
 			return ret;
 
 		if (!do_sum && resctrl_is_mbm_event(mevt->evtid))
-			mon_event_read(&rr, r, hdr, prgrp, &hdr->cpu_mask, mevt->evtid, true);
+			mon_event_read(&rr, r, hdr, prgrp, &hdr->cpu_mask, mevt, true);
 	}
 
 	return 0;
Signature · 1 line
2.51.0
author maintainer
#17 lore raw

PATCH v12 10/31 x86,fs/resctrl: Handle events that can be read from any CPU

In reply to Tony Luck

resctrl assumes that monitor events can only be read from a CPU in the cpumask_t set of each domain. This is true for x86 events accessed with an MSR interface, but may not be true for other access methods such as MMIO.

Introduce and use flag mon_evt::any_cpu, settable by architecture, that indicates there are no restrictions on which CPU can read that event.

Refactor the CPU checking to avoid always calling smp_processor_id() now that events can be read from preemptible context.

---
 include/linux/resctrl.h            |  2 +-
 fs/resctrl/internal.h              |  2 ++
 arch/x86/kernel/cpu/resctrl/core.c |  6 ++---
 fs/resctrl/ctrlmondata.c           |  6 +++++
 fs/resctrl/monitor.c               | 43 ++++++++++++++++++++++--------
 5 files changed, 44 insertions(+), 15 deletions(-)
include/linux/resctrl.h +1 -1
index 66569662efee..22edd8d131d8 100644
@@ -409,7 +409,7 @@ u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
 u32 resctrl_arch_system_num_rmid_idx(void);
 int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid);
 
-void resctrl_enable_mon_event(enum resctrl_event_id eventid);
+void resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu);
 
 bool resctrl_is_mon_event_enabled(enum resctrl_event_id eventid);
 
fs/resctrl/internal.h +2 -0
index 12a2ab7e3c9b..40b76eaa33d0 100644
@@ -61,6 +61,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
  *			READS_TO_REMOTE_MEM) being tracked by @evtid.
  *			Only valid if @evtid is an MBM event.
  * @configurable:	true if the event is configurable
+ * @any_cpu:		true if the event can be read from any CPU
  * @enabled:		true if the event is enabled
  */
 struct mon_evt {
@@ -69,6 +70,7 @@ struct mon_evt {
 	char			*name;
 	u32			evt_cfg;
 	bool			configurable;
+	bool			any_cpu;
 	bool			enabled;
 };
 
arch/x86/kernel/cpu/resctrl/core.c +3 -3
index a9fa506f09fa..1ee379014ef8 100644
@@ -893,15 +893,15 @@ static __init bool get_rdt_mon_resources(void)
 	bool ret = false;
 
 	if (rdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC)) {
-		resctrl_enable_mon_event(QOS_L3_OCCUP_EVENT_ID);
+		resctrl_enable_mon_event(QOS_L3_OCCUP_EVENT_ID, false);
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_CQM_MBM_TOTAL)) {
-		resctrl_enable_mon_event(QOS_L3_MBM_TOTAL_EVENT_ID);
+		resctrl_enable_mon_event(QOS_L3_MBM_TOTAL_EVENT_ID, false);
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_CQM_MBM_LOCAL)) {
-		resctrl_enable_mon_event(QOS_L3_MBM_LOCAL_EVENT_ID);
+		resctrl_enable_mon_event(QOS_L3_MBM_LOCAL_EVENT_ID, false);
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_ABMC))
fs/resctrl/ctrlmondata.c +6 -0
index 77602563cb1f..fbf55e61445c 100644
@@ -574,6 +574,11 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 		}
 	}
 
+	if (evt->any_cpu) {
+		mon_event_count(rr);
+		goto out_ctx_free;
+	}
+
 	cpu = cpumask_any_housekeeping(cpumask, RESCTRL_PICK_ANY_CPU);
 
 	/*
@@ -587,6 +592,7 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 	else
 		smp_call_on_cpu(cpu, smp_mon_event_count, rr, false);
 
+out_ctx_free:
 	if (rr->arch_mon_ctx)
 		resctrl_arch_mon_ctx_free(r, evt->evtid, rr->arch_mon_ctx);
 }
fs/resctrl/monitor.c +32 -11

88 lines not on this page · raw message

Signature · 1 line
2.51.0
maintainer
#18 lore raw

Re: [PATCH v12 10/31] x86,fs/resctrl: Handle events that can be read from any CPU

Hi Tony,

fs/resctrl/monitor.c static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) · line 453

Quoted 7 lines · Tony Luck
fs/resctrl/monitor.c +3 -0
@@ -424,6 +448,9 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	int err, ret;
 	u64 tval = 0;
 
+	if (!cpu_on_correct_domain(rr))
+		return -EINVAL;
+

To ensure error propagates, could you please change to:

	if (!cpu_on_correct_domain(rr)) {
		rr->err = -EIO;
		return -EINVAL;
	}

Reinette

author maintainer
#19 lore raw

PATCH v12 11/31 x86,fs/resctrl: Support binary fixed point event counters

In reply to Tony Luck

resctrl assumes that all monitor events can be displayed as unsigned decimal integers.

Hardware architecture counters may provide some telemetry events with greater precision where the event is not a simple count, but is a measurement of some sort (e.g. Joules for energy consumed).

Add a new argument to resctrl_enable_mon_event() for architecture code to inform the file system that the value for a counter is a fixed-point value with a specific number of binary places.
Only allow architecture to use floating point format on events that the file system has marked with mon_evt::is_floating_point.

Display fixed point values with values rounded to an appropriate number of decimal places for the precision of the number of binary places provided. Add one extra decimal place for every three additional binary places, except for low precision binary values where exact representation is possible:

1 binary place is 0.0 or 0.5 => 1 decimal place
2 binary places is 0.0, 0.25, 0.5, 0.75 => 2 decimal places
3 binary places is 0.0, 0.125, etc. => 3 decimal places

---
 include/linux/resctrl.h            |  3 +-
 fs/resctrl/internal.h              |  8 +++
 arch/x86/kernel/cpu/resctrl/core.c |  6 +--
 fs/resctrl/ctrlmondata.c           | 84 ++++++++++++++++++++++++++++++
 fs/resctrl/monitor.c               | 14 +++--
 5 files changed, 107 insertions(+), 8 deletions(-)
include/linux/resctrl.h +2 -1
index 22edd8d131d8..de66928e9430 100644
@@ -409,7 +409,8 @@ u32 resctrl_arch_get_num_closid(struct rdt_resource *r);
 u32 resctrl_arch_system_num_rmid_idx(void);
 int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid);
 
-void resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu);
+void resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu,
+			      unsigned int binary_bits);
 
 bool resctrl_is_mon_event_enabled(enum resctrl_event_id eventid);
 
fs/resctrl/internal.h +8 -0
index 40b76eaa33d0..f5189b6771a0 100644
@@ -62,6 +62,9 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
  *			Only valid if @evtid is an MBM event.
  * @configurable:	true if the event is configurable
  * @any_cpu:		true if the event can be read from any CPU
+ * @is_floating_point:	event values are displayed in floating point format
+ * @binary_bits:	number of fixed-point binary bits from architecture,
+ *			only valid if @is_floating_point is true
  * @enabled:		true if the event is enabled
  */
 struct mon_evt {
@@ -71,6 +74,8 @@ struct mon_evt {
 	u32			evt_cfg;
 	bool			configurable;
 	bool			any_cpu;
+	bool			is_floating_point;
+	unsigned int		binary_bits;
 	bool			enabled;
 };
 
@@ -79,6 +84,9 @@ extern struct mon_evt mon_event_all[QOS_NUM_EVENTS];
 #define for_each_mon_event(mevt) for (mevt = &mon_event_all[QOS_FIRST_EVENT];	\
 				      mevt < &mon_event_all[QOS_NUM_EVENTS]; mevt++)
 
+/* Limit for mon_evt::binary_bits */
+#define MAX_BINARY_BITS	27
+
 /**
  * struct mon_data - Monitoring details for each event file.
  * @list:            Member of the global @mon_data_kn_priv_list list.
arch/x86/kernel/cpu/resctrl/core.c +3 -3
index 1ee379014ef8..7670ce7cdcf1 100644
@@ -893,15 +893,15 @@ static __init bool get_rdt_mon_resources(void)
 	bool ret = false;
 
 	if (rdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC)) {
-		resctrl_enable_mon_event(QOS_L3_OCCUP_EVENT_ID, false);
+		resctrl_enable_mon_event(QOS_L3_OCCUP_EVENT_ID, false, 0);
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_CQM_MBM_TOTAL)) {
-		resctrl_enable_mon_event(QOS_L3_MBM_TOTAL_EVENT_ID, false);
+		resctrl_enable_mon_event(QOS_L3_MBM_TOTAL_EVENT_ID, false, 0);
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_CQM_MBM_LOCAL)) {
-		resctrl_enable_mon_event(QOS_L3_MBM_LOCAL_EVENT_ID, false);
+		resctrl_enable_mon_event(QOS_L3_MBM_LOCAL_EVENT_ID, false, 0);
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_ABMC))
fs/resctrl/ctrlmondata.c +84 -0

102 lines not on this page · raw message

fs/resctrl/monitor.c +10 -4
index 7b7162c4f3f1..255d4bad24cb 100644
@@ -981,16 +981,22 @@ struct mon_evt mon_event_all[QOS_NUM_EVENTS] = {
 	},
 };
 
-void resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu)
+void resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu, unsigned int binary_bits)
 {
-	if (WARN_ON_ONCE(eventid < QOS_FIRST_EVENT || eventid >= QOS_NUM_EVENTS))
+	if (WARN_ON_ONCE(eventid < QOS_FIRST_EVENT || eventid >= QOS_NUM_EVENTS ||
+			 binary_bits > MAX_BINARY_BITS))
 		return;
 	if (mon_event_all[eventid].enabled) {
 		pr_warn("Duplicate enable for event %d\n", eventid);
 		return;
 	}
+	if (binary_bits && !mon_event_all[eventid].is_floating_point) {
+		pr_warn("Event %d may not be floating point\n", eventid);
+		return;
+	}
 
 	mon_event_all[eventid].any_cpu = any_cpu;
+	mon_event_all[eventid].binary_bits = binary_bits;
 	mon_event_all[eventid].enabled = true;
 }
 
@@ -1816,9 +1822,9 @@ int resctrl_l3_mon_resource_init(void)
 
 	if (r->mon.mbm_cntr_assignable) {
 		if (!resctrl_is_mon_event_enabled(QOS_L3_MBM_TOTAL_EVENT_ID))
-			resctrl_enable_mon_event(QOS_L3_MBM_TOTAL_EVENT_ID, false);
+			resctrl_enable_mon_event(QOS_L3_MBM_TOTAL_EVENT_ID, false, 0);
 		if (!resctrl_is_mon_event_enabled(QOS_L3_MBM_LOCAL_EVENT_ID))
-			resctrl_enable_mon_event(QOS_L3_MBM_LOCAL_EVENT_ID, false);
+			resctrl_enable_mon_event(QOS_L3_MBM_LOCAL_EVENT_ID, false, 0);
 		mon_event_all[QOS_L3_MBM_TOTAL_EVENT_ID].evt_cfg = r->mon.mbm_cfg_mask;
 		mon_event_all[QOS_L3_MBM_LOCAL_EVENT_ID].evt_cfg = r->mon.mbm_cfg_mask &
 								   (READS_TO_LOCAL_MEM |
Signature · 1 line
2.51.0
author maintainer
#20 lore raw

PATCH v12 12/31 x86,fs/resctrl: Add an architectural hook called for each mount

In reply to Tony Luck

Enumeration of Intel telemetry events is an asynchronous process involving several mutually dependent drivers added as auxiliary devices during the device_initcall() phase of Linux boot. The process finishes after the probe functions of these drivers completes. But this happens after resctrl_arch_late_init() is executed.

Tracing the enumeration process shows that it does complete a full seven seconds before the earliest possible mount of the resctrl file system (when included in /etc/fstab for automatic mount by systemd).

Add a hook at the beginning of the mount code that will be used to check for telemetry events and initialize if any are found.

Call the hook on every attempted mount. Expectations are that most actions (like enumeration) will only need to be performed on the first call.

resctrl filesystem calls the hook with no locks held. Architecture code is responsible for any required locking.

---
 include/linux/resctrl.h            | 6 ++++++
 arch/x86/kernel/cpu/resctrl/core.c | 9 +++++++++
 fs/resctrl/rdtgroup.c              | 2 ++
 3 files changed, 17 insertions(+)
include/linux/resctrl.h +6 -0
index de66928e9430..6350064ac8be 100644
@@ -511,6 +511,12 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h
 void resctrl_online_cpu(unsigned int cpu);
 void resctrl_offline_cpu(unsigned int cpu);
 
+/*
+ * Architecture hook called at beginning of each file system mount attempt.
+ * No locks are held.
+ */
+void resctrl_arch_pre_mount(void);
+
 /**
  * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid
  *			      for this resource and domain.
arch/x86/kernel/cpu/resctrl/core.c +9 -0
index 7670ce7cdcf1..5b6af3b48de3 100644
@@ -721,6 +721,15 @@ static int resctrl_arch_offline_cpu(unsigned int cpu)
 	return 0;
 }
 
+void resctrl_arch_pre_mount(void)
+{
+	static atomic_t only_once = ATOMIC_INIT(0);
+	int old = 0;
+
+	if (!atomic_try_cmpxchg(&only_once, &old, 1))
+		return;
+}
+
 enum {
 	RDT_FLAG_CMT,
 	RDT_FLAG_MBM_TOTAL,
fs/resctrl/rdtgroup.c +2 -0
index 6112e1b83b87..f9bfbdeecf94 100644
@@ -2720,6 +2720,8 @@ static int rdt_get_tree(struct fs_context *fc)
 	struct rdt_resource *r;
 	int ret;
 
+	resctrl_arch_pre_mount();
+
 	cpus_read_lock();
 	mutex_lock(&rdtgroup_mutex);
 	/*
Signature · 1 line
2.51.0
author maintainer
#21 lore raw

PATCH v12 13/31 x86,fs/resctrl: Add and initialize rdt_resource for package scope monitor

In reply to Tony Luck

Add a new PERF_PKG resource and introduce package level scope for monitoring telemetry events so that CPU hot plug notifiers can build domains at the package granularity.

Use the physical package ID available via topology_physical_package_id() to identify the monitoring domains with package level scope. This enables user space to use:
/sys/devices/system/cpu/cpuX/topology/physical_package_id
to identify the monitoring domain a CPU is associated with.

Now there is a new monitoring resource, add a WARN to places that implicitly assume RDT_RESOURCE_L3.

Update some kerneldoc to point out L3 dependencies.

---
 include/linux/resctrl.h            |  2 ++
 fs/resctrl/internal.h              |  6 ++++--
 arch/x86/kernel/cpu/resctrl/core.c | 10 ++++++++++
 fs/resctrl/ctrlmondata.c           |  3 +++
 fs/resctrl/monitor.c               |  3 +++
 fs/resctrl/rdtgroup.c              |  5 ++++-
 6 files changed, 26 insertions(+), 3 deletions(-)
include/linux/resctrl.h +2 -0
index 6350064ac8be..ff67224b80c8 100644
@@ -53,6 +53,7 @@ enum resctrl_res_level {
 	RDT_RESOURCE_L2,
 	RDT_RESOURCE_MBA,
 	RDT_RESOURCE_SMBA,
+	RDT_RESOURCE_PERF_PKG,
 
 	/* Must be the last */
 	RDT_NUM_RESOURCES,
@@ -267,6 +268,7 @@ enum resctrl_scope {
 	RESCTRL_L2_CACHE = 2,
 	RESCTRL_L3_CACHE = 3,
 	RESCTRL_L3_NODE,
+	RESCTRL_PACKAGE,
 };
 
 /**
fs/resctrl/internal.h +4 -2
index f5189b6771a0..39bdaf45fa2a 100644
@@ -92,8 +92,8 @@ extern struct mon_evt mon_event_all[QOS_NUM_EVENTS];
  * @list:            Member of the global @mon_data_kn_priv_list list.
  * @rid:             Resource id associated with the event file.
  * @evt:             Event structure associated with the event file.
- * @sum:             Set when event must be summed across multiple
- *                   domains.
+ * @sum:             Set for RDT_RESOURCE_L3 when event must be summed
+ *                   across multiple domains.
  * @domid:           When @sum is zero this is the domain to which
  *                   the event file belongs. When @sum is one this
  *                   is the id of the L3 cache that all domains to be
@@ -255,6 +255,8 @@ struct rdtgroup {
 
 #define RFTYPE_ASSIGN_CONFIG		BIT(11)
 
+#define RFTYPE_RES_PERF_PKG		BIT(11)
+
 #define RFTYPE_CTRL_INFO		(RFTYPE_INFO | RFTYPE_CTRL)
 
 #define RFTYPE_MON_INFO			(RFTYPE_INFO | RFTYPE_MON)
arch/x86/kernel/cpu/resctrl/core.c +10 -0
index 5b6af3b48de3..57a328fdde59 100644
@@ -100,6 +100,14 @@ struct rdt_hw_resource rdt_resources_all[RDT_NUM_RESOURCES] = {
 			.schema_fmt		= RESCTRL_SCHEMA_RANGE,
 		},
 	},
+	[RDT_RESOURCE_PERF_PKG] =
+	{
+		.r_resctrl = {
+			.name			= "PERF_PKG",
+			.mon_scope		= RESCTRL_PACKAGE,
+			.mon_domains		= mon_domain_init(RDT_RESOURCE_PERF_PKG),
+		},
+	},
 };
 
 u32 resctrl_arch_system_num_rmid_idx(void)
@@ -435,6 +443,8 @@ static int get_domain_id_from_scope(int cpu, enum resctrl_scope scope)
 		return get_cpu_cacheinfo_id(cpu, scope);
 	case RESCTRL_L3_NODE:
 		return cpu_to_node(cpu);
+	case RESCTRL_PACKAGE:
+		return topology_physical_package_id(cpu);
 	default:
 		break;
 	}
fs/resctrl/ctrlmondata.c +3 -0
index ae43e09fa5e5..d681b71e6eca 100644
@@ -712,6 +712,9 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	if (md->sum) {
 		struct rdt_l3_mon_domain *d;
 
+		if (WARN_ON_ONCE(resid != RDT_RESOURCE_L3))
+			return -EINVAL;
+
 		/*
 		 * This file requires summing across all domains that share
 		 * the L3 cache id that was provided in the "domid" field of the
fs/resctrl/monitor.c +3 -0
index 255d4bad24cb..4c984dc6784e 100644
@@ -501,6 +501,9 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	 * all domains fail for any reason.
 	 */
 	ret = -EINVAL;
+	if (WARN_ON_ONCE(rr->r->rid != RDT_RESOURCE_L3))
+		return ret;
+
 	list_for_each_entry(d, &rr->r->mon_domains, hdr.list) {
 		if (d->ci_id != rr->ci->id)
 			continue;
fs/resctrl/rdtgroup.c +4 -1
index f9bfbdeecf94..c9d2cc1fd8bf 100644
@@ -2330,6 +2330,8 @@ static unsigned long fflags_from_resource(struct rdt_resource *r)
 	case RDT_RESOURCE_MBA:
 	case RDT_RESOURCE_SMBA:
 		return RFTYPE_RES_MB;
+	case RDT_RESOURCE_PERF_PKG:
+		return RFTYPE_RES_PERF_PKG;
 	}
 
 	return WARN_ON_ONCE(1);
@@ -3028,7 +3030,8 @@ static void rmdir_all_sub(void)
  * @rid:    The resource id for the event file being created.
  * @domid:  The domain id for the event file being created.
  * @mevt:   The type of event file being created.
- * @do_sum: Whether SNC summing monitors are being created.
+ * @do_sum: Whether SNC summing monitors are being created. Only set
+ *	    when @rid == RDT_RESOURCE_L3.
  */
 static struct mon_data *mon_get_kn_priv(enum resctrl_res_level rid, int domid,
 					struct mon_evt *mevt,
Signature · 1 line
2.51.0
maintainer
#22 lore raw

Re: [PATCH v12 13/31] x86,fs/resctrl: Add and initialize rdt_resource for package scope monitor

Hi Tony,

Quoted 10 lines · Tony Luck

Add a new PERF_PKG resource and introduce package level scope for monitoring telemetry events so that CPU hot plug notifiers can build domains at the package granularity.

Use the physical package ID available via topology_physical_package_id() to identify the monitoring domains with package level scope. This enables user space to use:
/sys/devices/system/cpu/cpuX/topology/physical_package_id
to identify the monitoring domain a CPU is associated with.

Quoted 7 lines

Now there is a new monitoring resource, add a WARN to places that implicitly assume RDT_RESOURCE_L3.

Update some kerneldoc to point out L3 dependencies.

I do not believe that the changes added since my RB tag fit with this patch.

As mentioned in patch #5 I find that (all but one of) the new changes added to this patch can be merged with those stray hunks found in patch #5 to form one logical change.

Below is a high level of what I have in mind presented as a draft changelog for that new patch:

The feature to sum event data across multiple domains supports systems with Sub-NUMA Cluster (SNC) mode enabled. The top-level monitoring files in each "mon_L3_XX" directory provide the sum of data across all SNC nodes sharing an L3 cache instance while the "mon_sub_L3_YY" sub-directories provide the event data of the individual nodes.

SNC is only associated with the L3 resource and domains and as a result the flow handling the sum of event data implicitly assumes it is working with the L3 resource and domains.

Reading of telemetry events do not require to sum event data so this
feature can remain dedicated to SNC and keep the implicit assumption of working with the L3 resource and domains.

Add a WARN to where the implicit assumption of working with the L3 resource
is made and add comments on how the structure controlling the event sum feature is used.

Quoted 6 lines

---

fs/resctrl/internal.h +0 -0
index f5189b6771a0..39bdaf45fa2a 100644

fs/resctrl/internal.h extern struct mon_evt mon_event_all[QOS_NUM_EVENTS]; · line 99

Quoted 10 lines
fs/resctrl/internal.h +2 -2
@@ -92,8 +92,8 @@ extern struct mon_evt mon_event_all[QOS_NUM_EVENTS];
  * @list:            Member of the global @mon_data_kn_priv_list list.
  * @rid:             Resource id associated with the event file.
  * @evt:             Event structure associated with the event file.
- * @sum:             Set when event must be summed across multiple
- *                   domains.
+ * @sum:             Set for RDT_RESOURCE_L3 when event must be summed
+ *                   across multiple domains.
  * @domid:           When @sum is zero this is the domain to which
  *                   the event file belongs. When @sum is one this
  *                   is the id of the L3 cache that all domains to be

above can move to the new patch

...

Quoted 5 lines
fs/resctrl/ctrlmondata.c +0 -0
index ae43e09fa5e5..d681b71e6eca 100644

fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 720

Quoted 9 lines
fs/resctrl/ctrlmondata.c +3 -0
@@ -712,6 +712,9 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	if (md->sum) {
 		struct rdt_l3_mon_domain *d;
 
+		if (WARN_ON_ONCE(resid != RDT_RESOURCE_L3))
+			return -EINVAL;
+
 		/*
 		 * This file requires summing across all domains that share
 		 * the L3 cache id that was provided in the "domid" field of the

above can move to the new patch ... it can form a new hunk with the hunk from patch #5 like:

if (md->sum) {
+ struct rdt_l3_mon_domain *d;
+
+ if (WARN_ON_ONCE(resid != RDT_RESOURCE_L3))
+ FIXME
+
/*
* This file requires summing across all domains that share
* the L3 cache id that was provided in the "domid" field of the

I added the FIXME because this should not do a direct return here while holding CPU hotplug lock as well as rdtgroup mutex.

fs/resctrl/monitor.c static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) · line 506

Quoted 11 lines
fs/resctrl/monitor.c +3 -0
index 255d4bad24cb..4c984dc6784e 100644
@@ -501,6 +501,9 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	 * all domains fail for any reason.
 	 */
 	ret = -EINVAL;
+	if (WARN_ON_ONCE(rr->r->rid != RDT_RESOURCE_L3))
+		return ret;
+

This looks redundant to me when considering the WARN added to rdtgroup_mondata_show() ... and it is removed in patch #18 anyway.

Quoted 3 lines
fs/resctrl/monitor.c +0 -0
 	list_for_each_entry(d, &rr->r->mon_domains, hdr.list) {
 		if (d->ci_id != rr->ci->id)
 			continue;

...

fs/resctrl/monitor.c static void rmdir_all_sub(void) · line 3037

Quoted 10 lines
fs/resctrl/monitor.c +2 -1
@@ -3028,7 +3030,8 @@ static void rmdir_all_sub(void)
  * @rid:    The resource id for the event file being created.
  * @domid:  The domain id for the event file being created.
  * @mevt:   The type of event file being created.
- * @do_sum: Whether SNC summing monitors are being created.
+ * @do_sum: Whether SNC summing monitors are being created. Only set
+ *	    when @rid == RDT_RESOURCE_L3.
  */
 static struct mon_data *mon_get_kn_priv(enum resctrl_res_level rid, int domid,
 					struct mon_evt *mevt,

above can move to new patch

Reinette

author maintainer
#23 lore raw

PATCH v12 14/31 x86/resctrl: Discover hardware telemetry events

In reply to Tony Luck

Each CPU collects data for telemetry events that it sends to the nearest telemetry event aggregator either when the value of MSR_IA32_PQR_ASSOC.RMID changes, or when a two millisecond timer expires.

The telemetry event aggregators maintain per-RMID per-event counts of the total seen for all the CPUs. There may be more than one set of telemetry event aggregators per package.

There are separate sets of aggregators for each type of event, but all aggregators for a given type are symmetric keeping counts for the same set of events for the CPUs that provide data to them.

Each telemetry event aggregator is responsible for a specific group of events. E.g. on the Intel Clearwater Forest CPU there are two types of aggregators. One type tracks a pair of energy related events. The other type tracks a subset of "perf" type events.

The event counts are made available to Linux in a region of MMIO space for each aggregator. All details about the layout of counters in each aggregator MMIO region are described in XML files published by Intel and made available in a GitHub repository [1].

The key to matching a specific telemetry aggregator to the XML file that describes the MMIO layout is a 32-bit value. The Linux telemetry subsystem refers to this as a "guid" while the XML files call it a "uniqueid".

Each XML file provides the following information:
1) Which telemetry events are included in the group.
2) The order in which the event counters appear for each RMID.
3) The value type of each event counter (integer or fixed-point).
4) The number of RMIDs supported.
5) Which additional aggregator status registers are included.
6) The total size of the MMIO region for an aggregator.

The INTEL_PMT_TELEMETRY driver enumerates support for telemetry events. This driver provides intel_pmt_get_regions_by_feature() to list all available telemetry event aggregators. The list includes the "guid", the base address in MMIO space for the region where the event counters are exposed, and the package id where the all the CPUs that report to this aggregator are located.

Add a new Kconfig option CONFIG_X86_CPU_RESCTRL_INTEL_AET for the Intel specific parts of telemetry code. This depends on the INTEL_PMT_TELEMETRY and INTEL_TPMI drivers being built-in to the kernel for enumeration of telemetry features.

Use INTEL_PMT_TELEMETRY's intel_pmt_get_regions_by_feature() with each per-RMID telemetry feature id to obtain a private copy of struct pmt_feature_group that contains all discovered/enumerated telemetry aggregator data for all event groups (known and unknown to resctrl) of that feature id. Further processing on this structure will enable all supported events in resctrl. Return the structure to INTEL_PMT_TELEMETRY at resctrl exit time.

---
 arch/x86/kernel/cpu/resctrl/internal.h  |   8 ++
 arch/x86/kernel/cpu/resctrl/core.c      |   5 +
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 140 ++++++++++++++++++++++++
 arch/x86/Kconfig                        |  13 +++
 arch/x86/kernel/cpu/resctrl/Makefile    |   1 +
 5 files changed, 167 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/resctrl/intel_aet.c
arch/x86/kernel/cpu/resctrl/internal.h +8 -0
index 14fadcff0d2b..886261a82b81 100644
@@ -217,4 +217,12 @@ void __init intel_rdt_mbm_apply_quirk(void);
 void rdt_domain_reconfigure_cdp(struct rdt_resource *r);
 void resctrl_arch_mbm_cntr_assign_set_one(struct rdt_resource *r);
 
+#ifdef CONFIG_X86_CPU_RESCTRL_INTEL_AET
+bool intel_aet_get_events(void);
+void __exit intel_aet_exit(void);
+#else
+static inline bool intel_aet_get_events(void) { return false; }
+static inline void __exit intel_aet_exit(void) { }
+#endif
+
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/core.c +5 -0
index 57a328fdde59..9451bafde923 100644
@@ -738,6 +738,9 @@ void resctrl_arch_pre_mount(void)
 
 	if (!atomic_try_cmpxchg(&only_once, &old, 1))
 		return;
+
+	if (!intel_aet_get_events())
+		return;
 }
 
 enum {
@@ -1095,6 +1098,8 @@ late_initcall(resctrl_arch_late_init);
 
 static void __exit resctrl_arch_exit(void)
 {
+	intel_aet_exit();
+
 	cpuhp_remove_state(rdt_online);
 
 	resctrl_exit();
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +140 -0

140 lines not on this page · raw message

arch/x86/Kconfig +13 -0
index fa3b616af03a..da5775056ec8 100644
@@ -536,6 +536,19 @@ config X86_CPU_RESCTRL
 
 	  Say N if unsure.
 
+config X86_CPU_RESCTRL_INTEL_AET
+	bool "Intel Application Energy Telemetry"
+	depends on X86_CPU_RESCTRL && CPU_SUP_INTEL && INTEL_PMT_TELEMETRY=y && INTEL_TPMI=y
+	help
+	  Enable per-RMID telemetry events in resctrl.
+
+	  Intel feature that collects per-RMID execution data
+	  about energy consumption, measure of frequency independent
+	  activity and other performance metrics. Data is aggregated
+	  per package.
+
+	  Say N if unsure.
+
 config X86_FRED
 	bool "Flexible Return and Event Delivery"
 	depends on X86_64
arch/x86/kernel/cpu/resctrl/Makefile +1 -0
index d8a04b195da2..273ddfa30836 100644
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_X86_CPU_RESCTRL)		+= core.o rdtgroup.o monitor.o
 obj-$(CONFIG_X86_CPU_RESCTRL)		+= ctrlmondata.o
+obj-$(CONFIG_X86_CPU_RESCTRL_INTEL_AET)	+= intel_aet.o
 obj-$(CONFIG_RESCTRL_FS_PSEUDO_LOCK)	+= pseudo_lock.o
 
 # To allow define_trace.h's recursive include:
Signature · 1 line
2.51.0
maintainer
#24 lore raw

Re: [PATCH v12 14/31] x86/resctrl: Discover hardware telemetry events

Hi Tony,

Quoted 11 lines · Tony Luck

Each CPU collects data for telemetry events that it sends to the nearest telemetry event aggregator either when the value of MSR_IA32_PQR_ASSOC.RMID changes, or when a two millisecond timer expires.

The telemetry event aggregators maintain per-RMID per-event counts of the total seen for all the CPUs. There may be more than one set of telemetry event aggregators per package.

There are separate sets of aggregators for each type of event, but all aggregators for a given type are symmetric keeping counts for the same set of events for the CPUs that provide data to them.

Above still seems to reflect previous implementation that only supported one guid per type of event. If I understand correctly this implementation supports the scenario where there may be aggregators of one type that support different guid and thus are not symmetric.

Quoted 3 lines

Each telemetry event aggregator is responsible for a specific group of events. E.g. on the Intel Clearwater Forest CPU there are two types of

Could you please define "event group"? It is first used in this patch but I cannot seem to find definition. Would be helpful to distinguish it from a "feature group".

Quoted 11 lines

aggregators. One type tracks a pair of energy related events. The other type tracks a subset of "perf" type events.

The event counts are made available to Linux in a region of MMIO space for each aggregator. All details about the layout of counters in each aggregator MMIO region are described in XML files published by Intel and made available in a GitHub repository [1].

The key to matching a specific telemetry aggregator to the XML file that describes the MMIO layout is a 32-bit value. The Linux telemetry subsystem refers to this as a "guid" while the XML files call it a "uniqueid".

https://github.com/intel/Intel-PMT mentions:
Using guid and size information, the associated Intel PMT XML files set can be identified using repository metadata at xml/pmt.xml.

From the above quote it seems as though size should also be used as key to match aggregator to the XML file? Also, looking at
https://github.com/intel/Intel-PMT/blob/main/xml/pmt.xml the file does use "guid" so there does not seem to be a need to mention that XML files call it "uniqueid".

Quoted 16 lines

Each XML file provides the following information:
1) Which telemetry events are included in the group.
2) The order in which the event counters appear for each RMID.
3) The value type of each event counter (integer or fixed-point).
4) The number of RMIDs supported.
5) Which additional aggregator status registers are included.
6) The total size of the MMIO region for an aggregator.

The INTEL_PMT_TELEMETRY driver enumerates support for telemetry events. This driver provides intel_pmt_get_regions_by_feature() to list all available telemetry event aggregators. The list includes the "guid", the base address in MMIO space for the region where the event counters are exposed, and the package id where the all the CPUs that report to this aggregator are located.

Quoted 7 lines

Add a new Kconfig option CONFIG_X86_CPU_RESCTRL_INTEL_AET for the Intel specific parts of telemetry code. This depends on the INTEL_PMT_TELEMETRY and INTEL_TPMI drivers being built-in to the kernel for enumeration of telemetry features.

Use INTEL_PMT_TELEMETRY's intel_pmt_get_regions_by_feature() with each per-RMID telemetry feature id to obtain a private copy of

"feature id" is not defined or introduced until here. Perhaps it should be "event type" instead to remain consistent with rest of changelog?

Quoted 5 lines

struct pmt_feature_group that contains all discovered/enumerated telemetry aggregator data for all event groups (known and unknown to resctrl) of that feature id. Further processing on this structure will enable all supported events in resctrl. Return the structure to INTEL_PMT_TELEMETRY at resctrl exit time.

This still documents the previous design.

Quoted 77 lines
---
 arch/x86/kernel/cpu/resctrl/internal.h  |   8 ++
 arch/x86/kernel/cpu/resctrl/core.c      |   5 +
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 140 ++++++++++++++++++++++++
 arch/x86/Kconfig                        |  13 +++
 arch/x86/kernel/cpu/resctrl/Makefile    |   1 +
 5 files changed, 167 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/resctrl/intel_aet.c
arch/x86/kernel/cpu/resctrl/internal.h +8 -0
index 14fadcff0d2b..886261a82b81 100644
@@ -217,4 +217,12 @@ void __init intel_rdt_mbm_apply_quirk(void);
 void rdt_domain_reconfigure_cdp(struct rdt_resource *r);
 void resctrl_arch_mbm_cntr_assign_set_one(struct rdt_resource *r);
 
+#ifdef CONFIG_X86_CPU_RESCTRL_INTEL_AET
+bool intel_aet_get_events(void);
+void __exit intel_aet_exit(void);
+#else
+static inline bool intel_aet_get_events(void) { return false; }
+static inline void __exit intel_aet_exit(void) { }
+#endif
+
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/core.c +5 -0
index 57a328fdde59..9451bafde923 100644
@@ -738,6 +738,9 @@ void resctrl_arch_pre_mount(void)
 
 	if (!atomic_try_cmpxchg(&only_once, &old, 1))
 		return;
+
+	if (!intel_aet_get_events())
+		return;
 }
 
 enum {
@@ -1095,6 +1098,8 @@ late_initcall(resctrl_arch_late_init);
 
 static void __exit resctrl_arch_exit(void)
 {
+	intel_aet_exit();
+
 	cpuhp_remove_state(rdt_online);
 
 	resctrl_exit();
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +19 -0
new file mode 100644
index 000000000000..67e479bdbc93
@@ -0,0 +1,140 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Resource Director Technology(RDT)
+ * - Intel Application Energy Telemetry
+ *
+ * Copyright (C) 2025 Intel Corporation
+ *
+ * Author:
+ *    Tony Luck <[email protected]>
+ */
+
+#define pr_fmt(fmt)   "resctrl: " fmt
+
+#include <linux/array_size.h>
+#include <linux/cleanup.h>
+#include <linux/cpu.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/intel_pmt_features.h>

arch/x86/kernel/cpu/resctrl/intel_aet.c line 29

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +10 -0
+#include <linux/intel_vsec.h>
+#include <linux/overflow.h>
+#include <linux/resctrl.h>
+#include <linux/stddef.h>
+#include <linux/types.h>
+
+#include "internal.h"
+
+/**
+ * struct event_group - All information about a group of telemetry events.

Please place the definition of "event group" in this description also. At this time it looks as though descriptions have not taken into account the design change made in this version.

Quoted 45 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +45 -0
+ * @pfg:		Points to the aggregated telemetry space information
+ *			returned by the intel_pmt_get_regions_by_feature()
+ *			call to the INTEL_PMT_TELEMETRY driver that contains
+ *			data for all telemetry regions of a specific type.
+ *			Valid if the system supports the event group.
+ *			NULL otherwise.
+ * @guid:		Unique number per XML description file.
+ */
+struct event_group {
+	/* Data fields for additional structures to manage this group. */
+	struct pmt_feature_group	*pfg;
+
+	/* Remaining fields initialized from XML file. */
+	u32				guid;
+};
+
+/*
+ * Link: https://github.com/intel/Intel-PMT
+ * File: xml/CWF/OOBMSM/RMID-ENERGY/cwf_aggregator.xml
+ */
+static struct event_group energy_0x26696143 = {
+	.guid		= 0x26696143,
+};
+
+/*
+ * Link: https://github.com/intel/Intel-PMT
+ * File: xml/CWF/OOBMSM/RMID-PERF/cwf_aggregator.xml
+ */
+static struct event_group perf_0x26557651 = {
+	.guid		= 0x26557651,
+};
+
+static struct event_group *known_energy_event_groups[] = {
+	&energy_0x26696143,
+};
+
+static struct event_group *known_perf_event_groups[] = {
+	&perf_0x26557651,
+};
+
+#define for_each_enabled_event_group(_peg, _grp)			\
+	for (_peg = (_grp); _peg < &_grp[ARRAY_SIZE(_grp)]; _peg++)	\
+		if ((*_peg)->pfg)
+
+/* Stub for now */

arch/x86/kernel/cpu/resctrl/intel_aet.c line 84

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +10 -0
+static bool enable_events(struct event_group *e, struct pmt_feature_group *p)
+{
+	return false;
+}
+
+/*
+ * Make a request to the INTEL_PMT_TELEMETRY driver for a copy of the
+ * pmt_feature_group for a specific feature. If there is one, the returned
+ * structure has an array of telemetry_region structures, each element of
+ * the array describes one telemetry aggregator.

Above describes the previous design?

arch/x86/kernel/cpu/resctrl/intel_aet.c line 85

Quoted 1 line
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +1 -0
+ * A single pmt_feature_group may include multiple different guids.

"A single pmt_feature_group may include multiple different guids." -> "The telemetry aggregators may have different guids."?

arch/x86/kernel/cpu/resctrl/intel_aet.c line 86

Quoted 1 line
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +1 -0
+ * Try to use every telemetry aggregator with a known guid.

Can "Try to use" be replaced with more specific description of this implementation?

arch/x86/kernel/cpu/resctrl/intel_aet.c line 98

Quoted 12 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +12 -0
+ */
+static bool get_pmt_feature(enum pmt_feature_id feature, struct event_group **evgs,
+			    unsigned int num_evg)
+{
+	struct pmt_feature_group *p;
+	struct event_group **peg;
+	bool ret = false;
+
+	for (peg = evgs; peg < &evgs[num_evg]; peg++) {
+		p = intel_pmt_get_regions_by_feature(feature);
+		if (IS_ERR_OR_NULL(p))
+			return false;

This appears to assume that intel_pmt_get_regions_by_feature() can only fail if the feature is invalid/unavailable? I do not think resctrl should make assumptions about what could cause a failure. It is theoretically possible that one call may succeed and the second call with the same feature ID fails. This scenario is not handled here and will result in a leak.

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +5 -0
+		if (enable_events(*peg, p)) {
+			(*peg)->pfg = p;
+			ret = true;
+		} else {
+			intel_pmt_put_feature_group(p);

arch/x86/kernel/cpu/resctrl/intel_aet.c line 113

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +10 -0
+		}
+	}
+
+	return ret;
+}
+
+/*
+ * Ask INTEL_PMT_TELEMETRY driver for all the RMID based telemetry groups
+ * that it supports.
+ */

Reinette

author maintainer
#25 lore raw

PATCH v12 15/31 x86,fs/resctrl: Fill in details of events for guid 0x26696143 and 0x26557651

In reply to Tony Luck

The Intel Clearwater Forest CPU supports two RMID-based PMT feature groups documented in the xml/CWF/OOBMSM/RMID-ENERGY/cwf_aggregator.xml and xml/CWF/OOBMSM/RMID-PERF/cwf_aggregator.xml files in the Intel PMT GIT repository [1].

The counter offsets in MMIO space are arranged in groups for each RMID.

E.g the "energy" counters for guid 0x26696143 are arranged like this:

        MMIO offset:0x0000 Counter for RMID 0 PMT_EVENT_ENERGY
        MMIO offset:0x0008 Counter for RMID 0 PMT_EVENT_ACTIVITY
        MMIO offset:0x0010 Counter for RMID 1 PMT_EVENT_ENERGY
        MMIO offset:0x0018 Counter for RMID 1 PMT_EVENT_ACTIVITY
        ...
        MMIO offset:0x23F0 Counter for RMID 575 PMT_EVENT_ENERGY
        MMIO offset:0x23F8 Counter for RMID 575 PMT_EVENT_ACTIVITY

After all counters there are three status registers that provide indications of how many times an aggregator was unable to process event counts, the time stamp for the most recent loss of data, and the time stamp of the most recent successful update.

	MMIO offset:0x2400 AGG_DATA_LOSS_COUNT
	MMIO offset:0x2408 AGG_DATA_LOSS_TIMESTAMP
	MMIO offset:0x2410 LAST_UPDATE_TIMESTAMP

Define these events in the file system code and add the events to the event_group structures.

PMT_EVENT_ENERGY and PMT_EVENT_ACTIVITY are produced in fixed point format. File system code must output as floating point values.

---
 include/linux/resctrl_types.h           | 11 +++++++
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 43 +++++++++++++++++++++++++
 fs/resctrl/monitor.c                    | 35 +++++++++++---------
 3 files changed, 74 insertions(+), 15 deletions(-)
include/linux/resctrl_types.h +11 -0
index acfe07860b34..a5f56faa18d2 100644
@@ -50,6 +50,17 @@ enum resctrl_event_id {
 	QOS_L3_MBM_TOTAL_EVENT_ID	= 0x02,
 	QOS_L3_MBM_LOCAL_EVENT_ID	= 0x03,
 
+	/* Intel Telemetry Events */
+	PMT_EVENT_ENERGY,
+	PMT_EVENT_ACTIVITY,
+	PMT_EVENT_STALLS_LLC_HIT,
+	PMT_EVENT_C1_RES,
+	PMT_EVENT_UNHALTED_CORE_CYCLES,
+	PMT_EVENT_STALLS_LLC_MISS,
+	PMT_EVENT_AUTO_C6_RES,
+	PMT_EVENT_UNHALTED_REF_CYCLES,
+	PMT_EVENT_UOPS_RETIRED,
+
 	/* Must be the last */
 	QOS_NUM_EVENTS,
 };
arch/x86/kernel/cpu/resctrl/intel_aet.c +43 -0

86 lines not on this page · raw message

fs/resctrl/monitor.c +20 -15
index 4c984dc6784e..6ee7011cb948 100644
@@ -961,27 +961,32 @@ static void dom_data_exit(struct rdt_resource *r)
 	mutex_unlock(&rdtgroup_mutex);
 }
 
+#define MON_EVENT(_eventid, _name, _res, _fp)	\
+	[_eventid] = {				\
+	.name			= _name,	\
+	.evtid			= _eventid,	\
+	.rid			= _res,		\
+	.is_floating_point	= _fp,		\
+}
+
 /*
  * All available events. Architecture code marks the ones that
  * are supported by a system using resctrl_enable_mon_event()
  * to set .enabled.
  */
 struct mon_evt mon_event_all[QOS_NUM_EVENTS] = {
-	[QOS_L3_OCCUP_EVENT_ID] = {
-		.name	= "llc_occupancy",
-		.evtid	= QOS_L3_OCCUP_EVENT_ID,
-		.rid	= RDT_RESOURCE_L3,
-	},
-	[QOS_L3_MBM_TOTAL_EVENT_ID] = {
-		.name	= "mbm_total_bytes",
-		.evtid	= QOS_L3_MBM_TOTAL_EVENT_ID,
-		.rid	= RDT_RESOURCE_L3,
-	},
-	[QOS_L3_MBM_LOCAL_EVENT_ID] = {
-		.name	= "mbm_local_bytes",
-		.evtid	= QOS_L3_MBM_LOCAL_EVENT_ID,
-		.rid	= RDT_RESOURCE_L3,
-	},
+	MON_EVENT(QOS_L3_OCCUP_EVENT_ID,		"llc_occupancy",	RDT_RESOURCE_L3,	false),
+	MON_EVENT(QOS_L3_MBM_TOTAL_EVENT_ID,		"mbm_total_bytes",	RDT_RESOURCE_L3,	false),
+	MON_EVENT(QOS_L3_MBM_LOCAL_EVENT_ID,		"mbm_local_bytes",	RDT_RESOURCE_L3,	false),
+	MON_EVENT(PMT_EVENT_ENERGY,			"core_energy",		RDT_RESOURCE_PERF_PKG,	true),
+	MON_EVENT(PMT_EVENT_ACTIVITY,			"activity",		RDT_RESOURCE_PERF_PKG,	true),
+	MON_EVENT(PMT_EVENT_STALLS_LLC_HIT,		"stalls_llc_hit",	RDT_RESOURCE_PERF_PKG,	false),
+	MON_EVENT(PMT_EVENT_C1_RES,			"c1_res",		RDT_RESOURCE_PERF_PKG,	false),
+	MON_EVENT(PMT_EVENT_UNHALTED_CORE_CYCLES,	"unhalted_core_cycles",	RDT_RESOURCE_PERF_PKG,	false),
+	MON_EVENT(PMT_EVENT_STALLS_LLC_MISS,		"stalls_llc_miss",	RDT_RESOURCE_PERF_PKG,	false),
+	MON_EVENT(PMT_EVENT_AUTO_C6_RES,		"c6_res",		RDT_RESOURCE_PERF_PKG,	false),
+	MON_EVENT(PMT_EVENT_UNHALTED_REF_CYCLES,	"unhalted_ref_cycles",	RDT_RESOURCE_PERF_PKG,	false),
+	MON_EVENT(PMT_EVENT_UOPS_RETIRED,		"uops_retired",		RDT_RESOURCE_PERF_PKG,	false),
 };
 
 void resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu, unsigned int binary_bits)
Signature · 1 line
2.51.0
maintainer
#26 lore raw

Re: [PATCH v12 15/31] x86,fs/resctrl: Fill in details of events for guid 0x26696143 and 0x26557651

Hi Tony,

Quoted 2 lines · Tony Luck

The Intel Clearwater Forest CPU supports two RMID-based PMT feature groups documented in the xml/CWF/OOBMSM/RMID-ENERGY/cwf_aggregator.xml

Considering the new design I understand "feature group" (thanks to struct pmt_feature_group) as all aggregators of a particular type (type being either energy or perf) where there may be aggregators with different guid, each with possibly different events. In comparison I understand "event group" (thanks to struct event_group) as all aggregators of a particular type (energy or perf) that have the *same* guid and thus the same events. Does this match the intent?

Quoted 20 lines

and xml/CWF/OOBMSM/RMID-PERF/cwf_aggregator.xml files in the Intel PMT GIT repository [1].

The counter offsets in MMIO space are arranged in groups for each RMID.

E.g the "energy" counters for guid 0x26696143 are arranged like this:

        MMIO offset:0x0000 Counter for RMID 0 PMT_EVENT_ENERGY
        MMIO offset:0x0008 Counter for RMID 0 PMT_EVENT_ACTIVITY
        MMIO offset:0x0010 Counter for RMID 1 PMT_EVENT_ENERGY
        MMIO offset:0x0018 Counter for RMID 1 PMT_EVENT_ACTIVITY
        ...
        MMIO offset:0x23F0 Counter for RMID 575 PMT_EVENT_ENERGY
        MMIO offset:0x23F8 Counter for RMID 575 PMT_EVENT_ACTIVITY

After all counters there are three status registers that provide indications of how many times an aggregator was unable to process event counts, the time stamp for the most recent loss of data, and the time stamp of the most recent successful update.

Quoted 6 lines
	MMIO offset:0x2400 AGG_DATA_LOSS_COUNT
	MMIO offset:0x2408 AGG_DATA_LOSS_TIMESTAMP
	MMIO offset:0x2410 LAST_UPDATE_TIMESTAMP

Define these events in the file system code and add the events to the event_group structures.

Could you please correct earlier definition to help make this transition from "feature group" to "event group" clear?

Quoted 7 lines

PMT_EVENT_ENERGY and PMT_EVENT_ACTIVITY are produced in fixed point format. File system code must output as floating point values.

---

Reinette

author maintainer
#27 lore raw

PATCH v12 16/31 x86,fs/resctrl: Add architectural event pointer

In reply to Tony Luck

The resctrl file system layer passes the domain, RMID, and event id to the architecture to fetch an event counter.

Fetching a telemetry event counter requires additional information that is private to the architecture, for example, the offset into MMIO space from where the counter should be read.

Add mon_evt::arch_priv that architecture can use for any private data related to the event. resctrl filesystem initializes mon_evt::arch_priv when the architecture enables the event and passes it back to architecture when needing to fetch an event counter.

---
 include/linux/resctrl.h               |  7 +++++--
 fs/resctrl/internal.h                 |  4 ++++
 arch/x86/kernel/cpu/resctrl/core.c    |  6 +++---
 arch/x86/kernel/cpu/resctrl/monitor.c |  2 +-
 fs/resctrl/monitor.c                  | 18 ++++++++++++------
 5 files changed, 25 insertions(+), 12 deletions(-)
include/linux/resctrl.h +5 -2
index ff67224b80c8..111c8f1dc77e 100644
@@ -412,7 +412,7 @@ u32 resctrl_arch_system_num_rmid_idx(void);
 int resctrl_arch_update_domains(struct rdt_resource *r, u32 closid);
 
 void resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu,
-			      unsigned int binary_bits);
+			      unsigned int binary_bits, void *arch_priv);
 
 bool resctrl_is_mon_event_enabled(enum resctrl_event_id eventid);
 
@@ -529,6 +529,9 @@ void resctrl_arch_pre_mount(void);
  *			only.
  * @rmid:		rmid of the counter to read.
  * @eventid:		eventid to read, e.g. L3 occupancy.
+ * @arch_priv:		Architecture private data for this event.
+ *			The @arch_priv provided by the architecture via
+ *			resctrl_enable_mon_event().
  * @val:		result of the counter read in bytes.
  * @arch_mon_ctx:	An architecture specific value from
  *			resctrl_arch_mon_ctx_alloc(), for MPAM this identifies
@@ -546,7 +549,7 @@ void resctrl_arch_pre_mount(void);
  */
 int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 			   u32 closid, u32 rmid, enum resctrl_event_id eventid,
-			   u64 *val, void *arch_mon_ctx);
+			   void *arch_priv, u64 *val, void *arch_mon_ctx);
 
 /**
  * resctrl_arch_rmid_read_context_check()  - warn about invalid contexts
fs/resctrl/internal.h +4 -0
index 39bdaf45fa2a..46fd648a2961 100644
@@ -66,6 +66,9 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
  * @binary_bits:	number of fixed-point binary bits from architecture,
  *			only valid if @is_floating_point is true
  * @enabled:		true if the event is enabled
+ * @arch_priv:		Architecture private data for this event.
+ *			The @arch_priv provided by the architecture via
+ *			resctrl_enable_mon_event().
  */
 struct mon_evt {
 	enum resctrl_event_id	evtid;
@@ -77,6 +80,7 @@ struct mon_evt {
 	bool			is_floating_point;
 	unsigned int		binary_bits;
 	bool			enabled;
+	void			*arch_priv;
 };
 
 extern struct mon_evt mon_event_all[QOS_NUM_EVENTS];
arch/x86/kernel/cpu/resctrl/core.c +3 -3
index 9451bafde923..6584461d4cb3 100644
@@ -915,15 +915,15 @@ static __init bool get_rdt_mon_resources(void)
 	bool ret = false;
 
 	if (rdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC)) {
-		resctrl_enable_mon_event(QOS_L3_OCCUP_EVENT_ID, false, 0);
+		resctrl_enable_mon_event(QOS_L3_OCCUP_EVENT_ID, false, 0, NULL);
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_CQM_MBM_TOTAL)) {
-		resctrl_enable_mon_event(QOS_L3_MBM_TOTAL_EVENT_ID, false, 0);
+		resctrl_enable_mon_event(QOS_L3_MBM_TOTAL_EVENT_ID, false, 0, NULL);
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_CQM_MBM_LOCAL)) {
-		resctrl_enable_mon_event(QOS_L3_MBM_LOCAL_EVENT_ID, false, 0);
+		resctrl_enable_mon_event(QOS_L3_MBM_LOCAL_EVENT_ID, false, 0, NULL);
 		ret = true;
 	}
 	if (rdt_cpu_has(X86_FEATURE_ABMC))
arch/x86/kernel/cpu/resctrl/monitor.c +1 -1
index 8f9a2ec3bd04..5565a8752266 100644
@@ -240,7 +240,7 @@ static u64 get_corrected_val(struct rdt_resource *r, struct rdt_l3_mon_domain *d
 
 int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 			   u32 unused, u32 rmid, enum resctrl_event_id eventid,
-			   u64 *val, void *ignored)
+			   void *arch_priv, u64 *val, void *ignored)
 {
 	struct rdt_hw_l3_mon_domain *hw_dom;
 	struct arch_mbm_state *am;
fs/resctrl/monitor.c +12 -6
index 6ee7011cb948..895cf5b0f524 100644
@@ -137,9 +137,11 @@ void __check_limbo(struct rdt_l3_mon_domain *d, bool force_free)
 	struct rmid_entry *entry;
 	u32 idx, cur_idx = 1;
 	void *arch_mon_ctx;
+	void *arch_priv;
 	bool rmid_dirty;
 	u64 val = 0;
 
+	arch_priv = mon_event_all[QOS_L3_OCCUP_EVENT_ID].arch_priv;
 	arch_mon_ctx = resctrl_arch_mon_ctx_alloc(r, QOS_L3_OCCUP_EVENT_ID);
 	if (IS_ERR(arch_mon_ctx)) {
 		pr_warn_ratelimited("Failed to allocate monitor context: %ld",
@@ -160,7 +162,7 @@ void __check_limbo(struct rdt_l3_mon_domain *d, bool force_free)
 
 		entry = __rmid_entry(idx);
 		if (resctrl_arch_rmid_read(r, &d->hdr, entry->closid, entry->rmid,
-					   QOS_L3_OCCUP_EVENT_ID, &val,
+					   QOS_L3_OCCUP_EVENT_ID, arch_priv, &val,
 					   arch_mon_ctx)) {
 			rmid_dirty = true;
 		} else {
@@ -484,7 +486,8 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 							 rr->evt->evtid, &tval);
 		else
 			rr->err = resctrl_arch_rmid_read(rr->r, rr->hdr, closid, rmid,
-							 rr->evt->evtid, &tval, rr->arch_mon_ctx);
+							 rr->evt->evtid, rr->evt->arch_priv,
+							 &tval, rr->arch_mon_ctx);
 		if (rr->err)
 			return rr->err;
 
@@ -512,7 +515,8 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 						     rr->evt->evtid, &tval);
 		else
 			err = resctrl_arch_rmid_read(rr->r, &d->hdr, closid, rmid,
-						     rr->evt->evtid, &tval, rr->arch_mon_ctx);
+						     rr->evt->evtid, rr->evt->arch_priv,
+						     &tval, rr->arch_mon_ctx);
 		if (!err) {
 			rr->val += tval;
 			ret = 0;
@@ -989,7 +993,8 @@ struct mon_evt mon_event_all[QOS_NUM_EVENTS] = {
 	MON_EVENT(PMT_EVENT_UOPS_RETIRED,		"uops_retired",		RDT_RESOURCE_PERF_PKG,	false),
 };
 
-void resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu, unsigned int binary_bits)
+void resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu,
+			      unsigned int binary_bits, void *arch_priv)
 {
 	if (WARN_ON_ONCE(eventid < QOS_FIRST_EVENT || eventid >= QOS_NUM_EVENTS ||
 			 binary_bits > MAX_BINARY_BITS))
@@ -1005,6 +1010,7 @@ void resctrl_enable_mon_event(enum resctrl_event_id eventid, bool any_cpu, unsig
 
 	mon_event_all[eventid].any_cpu = any_cpu;
 	mon_event_all[eventid].binary_bits = binary_bits;
+	mon_event_all[eventid].arch_priv = arch_priv;
 	mon_event_all[eventid].enabled = true;
 }
 
@@ -1830,9 +1836,9 @@ int resctrl_l3_mon_resource_init(void)
 
 	if (r->mon.mbm_cntr_assignable) {
 		if (!resctrl_is_mon_event_enabled(QOS_L3_MBM_TOTAL_EVENT_ID))
-			resctrl_enable_mon_event(QOS_L3_MBM_TOTAL_EVENT_ID, false, 0);
+			resctrl_enable_mon_event(QOS_L3_MBM_TOTAL_EVENT_ID, false, 0, NULL);
 		if (!resctrl_is_mon_event_enabled(QOS_L3_MBM_LOCAL_EVENT_ID))
-			resctrl_enable_mon_event(QOS_L3_MBM_LOCAL_EVENT_ID, false, 0);
+			resctrl_enable_mon_event(QOS_L3_MBM_LOCAL_EVENT_ID, false, 0, NULL);
 		mon_event_all[QOS_L3_MBM_TOTAL_EVENT_ID].evt_cfg = r->mon.mbm_cfg_mask;
 		mon_event_all[QOS_L3_MBM_LOCAL_EVENT_ID].evt_cfg = r->mon.mbm_cfg_mask &
 								   (READS_TO_LOCAL_MEM |
Signature · 1 line
2.51.0
maintainer Reviewed-by on v12 16/31
#28 lore raw

Re: [PATCH v12 16/31] x86,fs/resctrl: Add architectural event pointer

Hi Tony,

Quoted 7 lines · Tony Luck

The resctrl file system layer passes the domain, RMID, and event id to the architecture to fetch an event counter.

Fetching a telemetry event counter requires additional information that is private to the architecture, for example, the offset into MMIO space from where the counter should be read.

Quoted 8 lines

Add mon_evt::arch_priv that architecture can use for any private data related to the event. resctrl filesystem initializes mon_evt::arch_priv when the architecture enables the event and passes it back to architecture when needing to fetch an event counter.

---

This will need to be rebased, but that should be straight forward.

Reinette

author maintainer
#29 lore raw

PATCH v12 17/31 x86/resctrl: Find and enable usable telemetry events

In reply to Tony Luck

The INTEL_PMT_TELEMETRY driver provides telemetry region structures of the types requested by resctrl.

Scan these structures to discover which pass sanity checks to derive a list of valid regions:

1) They have guid known to resctrl.
2) They have a valid package ID.
3) The enumerated size of the MMIO region matches the expected value from the XML description file.

Mark regions that fail any of these checks as unusable.

Enable events that usable telemetry regions are responsible for.

Note that it is architecturally possible that some telemetry events are only supported by a subset of the packages in the system. It is not expected that systems will ever do this. If they do the user will see event files in resctrl that always return "Unavailable".

---
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 53 ++++++++++++++++++++++++-
 1 file changed, 51 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/intel_aet.c +51 -2
index 036086404e28..a500b5d384fb 100644
@@ -20,9 +20,11 @@
 #include <linux/intel_pmt_features.h>
 #include <linux/intel_vsec.h>
 #include <linux/overflow.h>
+#include <linux/printk.h>
 #include <linux/resctrl.h>
 #include <linux/resctrl_types.h>
 #include <linux/stddef.h>
+#include <linux/topology.h>
 #include <linux/types.h>
 
 #include "internal.h"
@@ -114,12 +116,59 @@ static struct event_group *known_perf_event_groups[] = {
 	for (_peg = (_grp); _peg < &_grp[ARRAY_SIZE(_grp)]; _peg++)	\
 		if ((*_peg)->pfg)
 
-/* Stub for now */
-static bool enable_events(struct event_group *e, struct pmt_feature_group *p)
+/*
+ * Clear the address field of regions that did not pass the checks in
+ * skip_telem_region() so they will not be used by intel_aet_read_event().
+ * This is safe to do because intel_pmt_get_regions_by_feature() allocates
+ * a new pmt_feature_group structure to return to each caller and only makes
+ * use of the pmt_feature_group::kref field when intel_pmt_put_feature_group()
+ * returns the structure.
+ */
+static void mark_telem_region_unusable(struct telemetry_region *tr)
+{
+	tr->addr = NULL;
+}
+
+static bool skip_telem_region(struct telemetry_region *tr, struct event_group *e)
 {
+	if (tr->guid != e->guid)
+		return true;
+	if (tr->plat_info.package_id >= topology_max_packages()) {
+		pr_warn("Bad package %u in guid 0x%x\n", tr->plat_info.package_id,
+			tr->guid);
+		return true;
+	}
+	if (tr->size != e->mmio_size) {
+		pr_warn("MMIO space wrong size (%zu bytes) for guid 0x%x. Expected %zu bytes.\n",
+			tr->size, e->guid, e->mmio_size);
+		return true;
+	}
+
 	return false;
 }
 
+static bool enable_events(struct event_group *e, struct pmt_feature_group *p)
+{
+	bool usable_events = false;
+
+	for (int i = 0; i < p->count; i++) {
+		if (skip_telem_region(&p->regions[i], e)) {
+			mark_telem_region_unusable(&p->regions[i]);
+			continue;
+		}
+		usable_events = true;
+	}
+
+	if (!usable_events)
+		return false;
+
+	for (int j = 0; j < e->num_events; j++)
+		resctrl_enable_mon_event(e->evts[j].id, true,
+					 e->evts[j].bin_bits, &e->evts[j]);
+
+	return true;
+}
+
 /*
  * Make a request to the INTEL_PMT_TELEMETRY driver for a copy of the
  * pmt_feature_group for a specific feature. If there is one, the returned
Signature · 1 line
2.51.0
maintainer
#30 lore raw

Re: [PATCH v12 17/31] x86/resctrl: Find and enable usable telemetry events

Hi Tony,

Quoted 6 lines · Tony Luck

The INTEL_PMT_TELEMETRY driver provides telemetry region structures of the types requested by resctrl.

Scan these structures to discover which pass sanity checks to derive a list of valid regions:

Quoted 6 lines

1) They have guid known to resctrl.
2) They have a valid package ID.
3) The enumerated size of the MMIO region matches the expected value from the XML description file.

Mark regions that fail any of these checks as unusable.

Above needs an update to reflect new design that will even mark regions that have known guid as unusable. So far the changelogs and comments do not reflect the new design and will make this implementation harder to understand for folks not familiar with its evolution.

Reinette

author maintainer
#31 lore raw

PATCH v12 18/31 fs/resctrl: Split L3 dependent parts out of __mon_event_count()

In reply to Tony Luck

Almost all of the code in __mon_event_count() is specific to the RDT_RESOURCE_L3 resource.

Split it out into __l3_mon_event_count().

---
 fs/resctrl/monitor.c | 70 ++++++++++++++++++++++++--------------------
 1 file changed, 39 insertions(+), 31 deletions(-)
fs/resctrl/monitor.c +39 -31
index 895cf5b0f524..faaa2851c5c2 100644
@@ -415,32 +415,7 @@ static void mbm_cntr_free(struct rdt_l3_mon_domain *d, int cntr_id)
 	memset(&d->cntr_cfg[cntr_id], 0, sizeof(*d->cntr_cfg));
 }
 
-/*
- * Called from preemptible context via a direct call of mon_event_count() for
- * events that can be read on any CPU.
- * Called from preemptible but non-migratable process context (mon_event_count()
- * via smp_call_on_cpu()) OR non-preemptible context (mon_event_count() via
- * smp_call_function_any()) for events that need to be read on a specific CPU.
- */
-static bool cpu_on_correct_domain(struct rmid_read *rr)
-{
-	int cpu;
-
-	/* Any CPU is OK for this event */
-	if (rr->evt->any_cpu)
-		return true;
-
-	cpu = smp_processor_id();
-
-	/* Single domain. Must be on a CPU in that domain. */
-	if (rr->hdr)
-		return cpumask_test_cpu(cpu, &rr->hdr->cpu_mask);
-
-	/* Summing domains that share a cache, must be on a CPU for that cache. */
-	return cpumask_test_cpu(cpu, &rr->ci->shared_cpu_map);
-}
-
-static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
+static int __l3_mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 {
 	u32 closid = rdtgrp->closid;
 	u32 rmid = rdtgrp->mon.rmid;
@@ -450,9 +425,6 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	int err, ret;
 	u64 tval = 0;
 
-	if (!cpu_on_correct_domain(rr))
-		return -EINVAL;
-
 	if (rr->is_mbm_cntr) {
 		if (!rr->hdr || !domain_header_is_valid(rr->hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 			return -EINVAL;
@@ -504,8 +476,6 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	 * all domains fail for any reason.
 	 */
 	ret = -EINVAL;
-	if (WARN_ON_ONCE(rr->r->rid != RDT_RESOURCE_L3))
-		return ret;
 
 	list_for_each_entry(d, &rr->r->mon_domains, hdr.list) {
 		if (d->ci_id != rr->ci->id)
@@ -529,6 +499,44 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	return ret;
 }
 
+/*
+ * Called from preemptible context via a direct call of mon_event_count() for
+ * events that can be read on any CPU.
+ * Called from preemptible but non-migratable process context (mon_event_count()
+ * via smp_call_on_cpu()) OR non-preemptible context (mon_event_count() via
+ * smp_call_function_any()) for events that need to be read on a specific CPU.
+ */
+static bool cpu_on_correct_domain(struct rmid_read *rr)
+{
+	int cpu;
+
+	/* Any CPU is OK for this event */
+	if (rr->evt->any_cpu)
+		return true;
+
+	cpu = smp_processor_id();
+
+	/* Single domain. Must be on a CPU in that domain. */
+	if (rr->hdr)
+		return cpumask_test_cpu(cpu, &rr->hdr->cpu_mask);
+
+	/* Summing domains that share a cache, must be on a CPU for that cache. */
+	return cpumask_test_cpu(cpu, &rr->ci->shared_cpu_map);
+}
+
+static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
+{
+	if (!cpu_on_correct_domain(rr))
+		return -EINVAL;
+
+	switch (rr->r->rid) {
+	case RDT_RESOURCE_L3:
+		return __l3_mon_event_count(rdtgrp, rr);
+	default:
+		return -EINVAL;
+	}
+}
+
 /*
  * mbm_bw_count() - Update bw count from values previously read by
  *		    __mon_event_count().
Signature · 1 line
2.51.0
maintainer
#32 lore raw

Re: [PATCH v12 18/31] fs/resctrl: Split L3 dependent parts out of __mon_event_count()

Hi Tony,

Quoted 4 lines · Tony Luck

Almost all of the code in __mon_event_count() is specific to the RDT_RESOURCE_L3 resource.

Split it out into __l3_mon_event_count().

Missing a "why". We could perhaps write it similar to an earlier commit message:
Carve out the L3 resource specific event reading code into a separate helper to support reading event data from a new monitoring resource.

Quoted 29 lines

---

fs/resctrl/monitor.c +19 -0
@@ -529,6 +499,44 @@ static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 	return ret;
 }
 
+/*
+ * Called from preemptible context via a direct call of mon_event_count() for
+ * events that can be read on any CPU.
+ * Called from preemptible but non-migratable process context (mon_event_count()
+ * via smp_call_on_cpu()) OR non-preemptible context (mon_event_count() via
+ * smp_call_function_any()) for events that need to be read on a specific CPU.
+ */
+static bool cpu_on_correct_domain(struct rmid_read *rr)
+{
+	int cpu;
+
+	/* Any CPU is OK for this event */
+	if (rr->evt->any_cpu)
+		return true;
+
+	cpu = smp_processor_id();
+
+	/* Single domain. Must be on a CPU in that domain. */
+	if (rr->hdr)

fs/resctrl/monitor.c static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr) · line 530

Quoted 10 lines
fs/resctrl/monitor.c +10 -0
+		return cpumask_test_cpu(cpu, &rr->hdr->cpu_mask);
+
+	/* Summing domains that share a cache, must be on a CPU for that cache. */
+	return cpumask_test_cpu(cpu, &rr->ci->shared_cpu_map);
+}
+
+static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
+{
+	if (!cpu_on_correct_domain(rr))
+		return -EINVAL;

It is a bit subtle how cpu_on_correct_domain() contains L3 specific code. This may be ok if one rather thinks of it as a sanity check of struct rmid_read.

Reinette

author maintainer
#33 lore raw

PATCH v12 19/31 x86/resctrl: Read telemetry events

In reply to Tony Luck

Telemetry events are enabled during the first mount of the resctrl file system.

Mark telemetry regions that did not pass the sanity checks by clearing their MMIO address fields so that they will not be used when reading events.

Introduce intel_aet_read_event() to read telemetry events for resource RDT_RESOURCE_PERF_PKG. There may be multiple aggregators tracking each package, so scan all of them and add up all counters. Aggregators may return an invalid data indication if they have received no records for a given RMID. Return success to the user if one or more aggregators provide valid data.

Resctrl now uses readq() so depends on X86_64. Update Kconfig.

---
 arch/x86/kernel/cpu/resctrl/internal.h  |  7 ++++
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 48 +++++++++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/monitor.c   |  3 ++
 fs/resctrl/monitor.c                    | 14 ++++++++
 arch/x86/Kconfig                        |  2 +-
 5 files changed, 73 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/internal.h +7 -0
index 886261a82b81..97616c81682b 100644
@@ -220,9 +220,16 @@ void resctrl_arch_mbm_cntr_assign_set_one(struct rdt_resource *r);
 #ifdef CONFIG_X86_CPU_RESCTRL_INTEL_AET
 bool intel_aet_get_events(void);
 void __exit intel_aet_exit(void);
+int intel_aet_read_event(int domid, u32 rmid, enum resctrl_event_id evtid,
+			 void *arch_priv, u64 *val);
 #else
 static inline bool intel_aet_get_events(void) { return false; }
 static inline void __exit intel_aet_exit(void) { }
+static inline int intel_aet_read_event(int domid, u32 rmid, enum resctrl_event_id evtid,
+				       void *arch_priv, u64 *val)
+{
+	return -EINVAL;
+}
 #endif
 
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/intel_aet.c +48 -0

64 lines not on this page · raw message

arch/x86/kernel/cpu/resctrl/monitor.c +3 -0
index 5565a8752266..6804b83934e8 100644
@@ -252,6 +252,9 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 
 	resctrl_arch_rmid_read_context_check();
 
+	if (r->rid == RDT_RESOURCE_PERF_PKG)
+		return intel_aet_read_event(hdr->id, rmid, eventid, arch_priv, val);
+
 	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		return -EINVAL;
 
fs/resctrl/monitor.c +14 -0
index faaa2851c5c2..84a0cbe90748 100644
@@ -526,12 +526,26 @@ static bool cpu_on_correct_domain(struct rmid_read *rr)
 
 static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 {
+	u64 tval = 0;
+
 	if (!cpu_on_correct_domain(rr))
 		return -EINVAL;
 
 	switch (rr->r->rid) {
 	case RDT_RESOURCE_L3:
 		return __l3_mon_event_count(rdtgrp, rr);
+
+	case RDT_RESOURCE_PERF_PKG:
+		rr->err = resctrl_arch_rmid_read(rr->r, rr->hdr, rdtgrp->closid,
+						 rdtgrp->mon.rmid, rr->evt->evtid,
+						 rr->evt->arch_priv,
+						 &tval, rr->arch_mon_ctx);
+		if (rr->err)
+			return rr->err;
+
+		rr->val += tval;
+
+		return 0;
 	default:
 		return -EINVAL;
 	}
arch/x86/Kconfig +1 -1
index da5775056ec8..60ace4427ede 100644
@@ -538,7 +538,7 @@ config X86_CPU_RESCTRL
 
 config X86_CPU_RESCTRL_INTEL_AET
 	bool "Intel Application Energy Telemetry"
-	depends on X86_CPU_RESCTRL && CPU_SUP_INTEL && INTEL_PMT_TELEMETRY=y && INTEL_TPMI=y
+	depends on X86_64 && X86_CPU_RESCTRL && CPU_SUP_INTEL && INTEL_PMT_TELEMETRY=y && INTEL_TPMI=y
 	help
 	  Enable per-RMID telemetry events in resctrl.
 
Signature · 1 line
2.51.0
maintainer
#34 lore raw

Re: [PATCH v12 19/31] x86/resctrl: Read telemetry events

Hi Tony,

Quoted 2 lines · Tony Luck

Telemetry events are enabled during the first mount of the resctrl file system.

Not clear how above is relevant to this patch?

Quoted 4 lines

Mark telemetry regions that did not pass the sanity checks by clearing their MMIO address fields so that they will not be used when reading events.

Above is definitely not relevant.

Quoted 7 lines

Introduce intel_aet_read_event() to read telemetry events for resource RDT_RESOURCE_PERF_PKG. There may be multiple aggregators tracking each package, so scan all of them and add up all counters. Aggregators may return an invalid data indication if they have received no records for a given RMID. Return success to the user if one or more aggregators provide valid data.

"success" (via return code of 0) is always returned to the user. The difference is whether user will see event data, "Unavailable", or "Error".

Quoted 5 lines

Resctrl now uses readq() so depends on X86_64. Update Kconfig.

---

...

Quoted 33 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +29 -0
@@ -230,3 +234,47 @@ void __exit intel_aet_exit(void)
 		(*peg)->pfg = NULL;
 	}
 }
+
+#define DATA_VALID	BIT_ULL(63)
+#define DATA_BITS	GENMASK_ULL(62, 0)
+
+/*
+ * Read counter for an event on a domain (summing all aggregators
+ * on the domain). If an aggregator hasn't received any data for a
+ * specific RMID, the MMIO read indicates that data is not valid.
+ * Return success if at least one aggregator has valid data.
+ */
+int intel_aet_read_event(int domid, u32 rmid, enum resctrl_event_id eventid,
+			 void *arch_priv, u64 *val)
+{
+	struct pmt_event *pevt = arch_priv;
+	struct event_group *e;
+	bool valid = false;
+	u64 evtcount;
+	void *pevt0;
+	u32 idx;
+
+	pevt0 = pevt - pevt->idx;
+	e = container_of(pevt0, struct event_group, evts);
+	idx = rmid * e->num_events;
+	idx += pevt->idx;
+
+	if (idx * sizeof(u64) + sizeof(u64) > e->mmio_size) {
+		pr_warn_once("MMIO index %u out of range\n", idx);
+		return -EIO;
+	}

arch/x86/kernel/cpu/resctrl/intel_aet.c void __exit intel_aet_exit(void) · line 275

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +10 -0
+
+	for (int i = 0; i < e->pfg->count; i++) {
+		if (!e->pfg->regions[i].addr)
+			continue;
+		if (e->pfg->regions[i].plat_info.package_id != domid)
+			continue;
+		evtcount = readq(e->pfg->regions[i].addr + idx * sizeof(u64));
+		if (!(evtcount & DATA_VALID))
+			continue;
+		*val += evtcount & DATA_BITS;

I missed this before. This just blindly adds the event data to whichever variable the caller provides. Could you please instead use a local variable to do this addition and on success assign it to the provided val parameter?

Quoted 24 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +5 -0
+		valid = true;
+	}
+
+	return valid ? 0 : -EINVAL;
+}
arch/x86/kernel/cpu/resctrl/monitor.c +3 -0
index 5565a8752266..6804b83934e8 100644
@@ -252,6 +252,9 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_domain_hdr *hdr,
 
 	resctrl_arch_rmid_read_context_check();
 
+	if (r->rid == RDT_RESOURCE_PERF_PKG)
+		return intel_aet_read_event(hdr->id, rmid, eventid, arch_priv, val);
+
 	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		return -EINVAL;
 
fs/resctrl/monitor.c +0 -0
index faaa2851c5c2..84a0cbe90748 100644

fs/resctrl/monitor.c static bool cpu_on_correct_domain(struct rmid_read *rr) · line 529

Quoted 4 lines
fs/resctrl/monitor.c +1 -0
@@ -526,12 +526,26 @@ static bool cpu_on_correct_domain(struct rmid_read *rr)
 
 static int __mon_event_count(struct rdtgroup *rdtgrp, struct rmid_read *rr)
 {
+	u64 tval = 0;

please declare this variable local to the PERF_PKG block to help keep the different resource code separate.

Quoted 12 lines
fs/resctrl/monitor.c +6 -0
+
 	if (!cpu_on_correct_domain(rr))
 		return -EINVAL;
 
 	switch (rr->r->rid) {
 	case RDT_RESOURCE_L3:
 		return __l3_mon_event_count(rdtgrp, rr);
+
+	case RDT_RESOURCE_PERF_PKG:
+		rr->err = resctrl_arch_rmid_read(rr->r, rr->hdr, rdtgrp->closid,
+						 rdtgrp->mon.rmid, rr->evt->evtid,
+						 rr->evt->arch_priv,

fs/resctrl/monitor.c static bool cpu_on_correct_domain(struct rmid_read *rr) · line 551

Quoted 10 lines
fs/resctrl/monitor.c +7 -0
+						 &tval, rr->arch_mon_ctx);
+		if (rr->err)
+			return rr->err;
+
+		rr->val += tval;
+
+		return 0;
 	default:
 		return -EINVAL;
 	}

Reinette

author maintainer
#35 lore raw

PATCH v12 20/31 fs/resctrl: Refactor mkdir_mondata_subdir()

In reply to Tony Luck

mkdir_mondata_subdir() is unreasonably complicated because of the support for Sub-NUMA Cluster (SNC) mode.

Split out the SNC code into a helper function to make it easier to add support for a new telemetry resource.

---
 fs/resctrl/rdtgroup.c | 83 +++++++++++++++++++++++++++----------------
 1 file changed, 52 insertions(+), 31 deletions(-)
fs/resctrl/rdtgroup.c +52 -31
index c9d2cc1fd8bf..b7eaa4388768 100644
@@ -3196,22 +3196,16 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,
 
 static int mon_add_all_files(struct kernfs_node *kn, struct rdt_domain_hdr *hdr,
 			     struct rdt_resource *r, struct rdtgroup *prgrp,
-			     bool do_sum)
+			     int domid, bool do_sum)
 {
 	struct rmid_read rr = {0};
-	struct rdt_l3_mon_domain *d;
 	struct mon_data *priv;
 	struct mon_evt *mevt;
-	int ret, domid;
-
-	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
-		return -EINVAL;
+	int ret;
 
-	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
 	for_each_mon_event(mevt) {
 		if (mevt->rid != r->rid || !mevt->enabled)
 			continue;
-		domid = do_sum ? d->ci_id : d->hdr.id;
 		priv = mon_get_kn_priv(r->rid, domid, mevt, do_sum);
 		if (WARN_ON_ONCE(!priv))
 			return -EINVAL;
@@ -3227,24 +3221,20 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_domain_hdr *hdr,
 	return 0;
 }
 
-static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,
-				struct rdt_domain_hdr *hdr,
-				struct rdt_resource *r, struct rdtgroup *prgrp)
+static int mkdir_mondata_subdir_snc(struct kernfs_node *parent_kn,
+				    struct rdt_domain_hdr *hdr,
+				    struct rdt_resource *r, struct rdtgroup *prgrp)
 {
 	struct kernfs_node *kn, *ckn;
 	struct rdt_l3_mon_domain *d;
 	char name[32];
-	bool snc_mode;
 	int ret = 0;
 
-	lockdep_assert_held(&rdtgroup_mutex);
-
 	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		return -EINVAL;
 
 	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
-	snc_mode = r->mon_scope == RESCTRL_L3_NODE;
-	sprintf(name, "mon_%s_%02d", r->name, snc_mode ? d->ci_id : d->hdr.id);
+	sprintf(name, "mon_%s_%02d", r->name, d->ci_id);
 	kn = kernfs_find_and_get(parent_kn, name);
 	if (kn) {
 		/*
@@ -3260,27 +3250,58 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,
 		ret = rdtgroup_kn_set_ugid(kn);
 		if (ret)
 			goto out_destroy;
-		ret = mon_add_all_files(kn, hdr, r, prgrp, snc_mode);
+		ret = mon_add_all_files(kn, hdr, r, prgrp, d->ci_id, true);
 		if (ret)
 			goto out_destroy;
 	}
 
-	if (snc_mode) {
-		sprintf(name, "mon_sub_%s_%02d", r->name, hdr->id);
-		ckn = kernfs_create_dir(kn, name, parent_kn->mode, prgrp);
-		if (IS_ERR(ckn)) {
-			ret = -EINVAL;
-			goto out_destroy;
-		}
+	sprintf(name, "mon_sub_%s_%02d", r->name, hdr->id);
+	ckn = kernfs_create_dir(kn, name, parent_kn->mode, prgrp);
+	if (IS_ERR(ckn)) {
+		ret = -EINVAL;
+		goto out_destroy;
+	}
 
-		ret = rdtgroup_kn_set_ugid(ckn);
-		if (ret)
-			goto out_destroy;
+	ret = rdtgroup_kn_set_ugid(ckn);
+	if (ret)
+		goto out_destroy;
 
-		ret = mon_add_all_files(ckn, hdr, r, prgrp, false);
-		if (ret)
-			goto out_destroy;
-	}
+	ret = mon_add_all_files(ckn, hdr, r, prgrp, hdr->id, false);
+	if (ret)
+		goto out_destroy;
+
+	kernfs_activate(kn);
+	return 0;
+
+out_destroy:
+	kernfs_remove(kn);
+	return ret;
+}
+
+static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,
+				struct rdt_domain_hdr *hdr,
+				struct rdt_resource *r, struct rdtgroup *prgrp)
+{
+	struct kernfs_node *kn;
+	char name[32];
+	int ret = 0;
+
+	lockdep_assert_held(&rdtgroup_mutex);
+
+	if (r->mon_scope == RESCTRL_L3_NODE)
+		return mkdir_mondata_subdir_snc(parent_kn, hdr, r, prgrp);
+
+	sprintf(name, "mon_%s_%02d", r->name, hdr->id);
+	kn = kernfs_create_dir(parent_kn, name, parent_kn->mode, prgrp);
+	if (IS_ERR(kn))
+		return PTR_ERR(kn);
+
+	ret = rdtgroup_kn_set_ugid(kn);
+	if (ret)
+		goto out_destroy;
+	ret = mon_add_all_files(kn, hdr, r, prgrp, hdr->id, false);
+	if (ret)
+		goto out_destroy;
 
 	kernfs_activate(kn);
 	return 0;
Signature · 1 line
2.51.0
maintainer
#36 lore raw

Re: [PATCH v12 20/31] fs/resctrl: Refactor mkdir_mondata_subdir()

Hi Tony,

Quoted 2 lines · Tony Luck

mkdir_mondata_subdir() is unreasonably complicated because of the support for Sub-NUMA Cluster (SNC) mode.

"mkdir_mondata_subdir()" -> "Population of a monitor group's mon_data directory"? (to not describe code but what the code does)

Quoted 15 lines

Split out the SNC code into a helper function to make it easier to add support for a new telemetry resource.

---
 fs/resctrl/rdtgroup.c | 83 +++++++++++++++++++++++++++----------------
 1 file changed, 52 insertions(+), 31 deletions(-)
fs/resctrl/rdtgroup.c +0 -0
index c9d2cc1fd8bf..b7eaa4388768 100644

fs/resctrl/rdtgroup.c static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, · line 3199

Quoted 5 lines
fs/resctrl/rdtgroup.c +1 -1
@@ -3196,22 +3196,16 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,
 
 static int mon_add_all_files(struct kernfs_node *kn, struct rdt_domain_hdr *hdr,
 			     struct rdt_resource *r, struct rdtgroup *prgrp,
-			     bool do_sum)
+			     int domid, bool do_sum)

I think there is some redundant information in the parameters that can be simplified. It is already familiar that a domain/header of NULL implies that this is SNC/summing of domains. For example, mon_event_read() and rmid_read::hdr. I think it will make things simpler and consistent to do the same here where a hdr value of NULL implies sum of events. With that do_sum will no longer be needed.

Quoted 49 lines
fs/resctrl/rdtgroup.c +5 -15
 {
 	struct rmid_read rr = {0};
-	struct rdt_l3_mon_domain *d;
 	struct mon_data *priv;
 	struct mon_evt *mevt;
-	int ret, domid;
-
-	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
-		return -EINVAL;
+	int ret;
 
-	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
 	for_each_mon_event(mevt) {
 		if (mevt->rid != r->rid || !mevt->enabled)
 			continue;
-		domid = do_sum ? d->ci_id : d->hdr.id;
 		priv = mon_get_kn_priv(r->rid, domid, mevt, do_sum);
 		if (WARN_ON_ONCE(!priv))
 			return -EINVAL;
@@ -3227,24 +3221,20 @@ static int mon_add_all_files(struct kernfs_node *kn, struct rdt_domain_hdr *hdr,
 	return 0;
 }
 
-static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,
-				struct rdt_domain_hdr *hdr,
-				struct rdt_resource *r, struct rdtgroup *prgrp)
+static int mkdir_mondata_subdir_snc(struct kernfs_node *parent_kn,
+				    struct rdt_domain_hdr *hdr,
+				    struct rdt_resource *r, struct rdtgroup *prgrp)
 {
 	struct kernfs_node *kn, *ckn;
 	struct rdt_l3_mon_domain *d;
 	char name[32];
-	bool snc_mode;
 	int ret = 0;
 
-	lockdep_assert_held(&rdtgroup_mutex);
-
 	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		return -EINVAL;
 
 	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
-	snc_mode = r->mon_scope == RESCTRL_L3_NODE;
-	sprintf(name, "mon_%s_%02d", r->name, snc_mode ? d->ci_id : d->hdr.id);
+	sprintf(name, "mon_%s_%02d", r->name, d->ci_id);
 	kn = kernfs_find_and_get(parent_kn, name);
 	if (kn) {
 		/*

fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3253

Quoted 5 lines
fs/resctrl/rdtgroup.c +1 -1
@@ -3260,27 +3250,58 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,
 		ret = rdtgroup_kn_set_ugid(kn);
 		if (ret)
 			goto out_destroy;
-		ret = mon_add_all_files(kn, hdr, r, prgrp, snc_mode);
+		ret = mon_add_all_files(kn, hdr, r, prgrp, d->ci_id, true);

Continuing from the earlier mon_add_all_files() comment, passing a value for hdr here does not seem right since there is no single domain attached to these files being created. Now that this code is made to be more specific it can do so more clearly by passing NULL instead.

fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3259

Quoted 13 lines
fs/resctrl/rdtgroup.c +2 -7
 		if (ret)
 			goto out_destroy;
 	}
 
-	if (snc_mode) {
-		sprintf(name, "mon_sub_%s_%02d", r->name, hdr->id);
-		ckn = kernfs_create_dir(kn, name, parent_kn->mode, prgrp);
-		if (IS_ERR(ckn)) {
-			ret = -EINVAL;
-			goto out_destroy;
-		}
+	sprintf(name, "mon_sub_%s_%02d", r->name, hdr->id);
+	ckn = kernfs_create_dir(kn, name, parent_kn->mode, prgrp);

Noting here that kn was created earlier with mode of parent_kn->mode. It thus looks to me like above can also be written as:

	ckn = kernfs_create_dir(kn, name, kn->mode, prgrp);

The reason I mention this is that this patch adds a third copy of a very similar code snippet (kernfs_create_dir(), rdtgroup_kn_set_ugid(), mon_add_all_files()) that looks like a good candidate for a helper?

Quoted 29 lines
fs/resctrl/rdtgroup.c +20 -7
+	if (IS_ERR(ckn)) {
+		ret = -EINVAL;
+		goto out_destroy;
+	}
 
-		ret = rdtgroup_kn_set_ugid(ckn);
-		if (ret)
-			goto out_destroy;
+	ret = rdtgroup_kn_set_ugid(ckn);
+	if (ret)
+		goto out_destroy;
 
-		ret = mon_add_all_files(ckn, hdr, r, prgrp, false);
-		if (ret)
-			goto out_destroy;
-	}
+	ret = mon_add_all_files(ckn, hdr, r, prgrp, hdr->id, false);
+	if (ret)
+		goto out_destroy;
+
+	kernfs_activate(kn);
+	return 0;
+
+out_destroy:
+	kernfs_remove(kn);
+	return ret;
+}
+
+static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,

fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3291

Quoted 10 lines
fs/resctrl/rdtgroup.c +10 -0
+				struct rdt_domain_hdr *hdr,
+				struct rdt_resource *r, struct rdtgroup *prgrp)
+{
+	struct kernfs_node *kn;
+	char name[32];
+	int ret = 0;
+
+	lockdep_assert_held(&rdtgroup_mutex);
+
+	if (r->mon_scope == RESCTRL_L3_NODE)

What happened between the time you indicated that the above will receive a resource check and this patch?
https://lore.kernel.org/lkml/aOhd_A5L8PV0OYba@agluck-desk3/

Quoted 6 lines
fs/resctrl/rdtgroup.c +6 -0
+		return mkdir_mondata_subdir_snc(parent_kn, hdr, r, prgrp);
+
+	sprintf(name, "mon_%s_%02d", r->name, hdr->id);
+	kn = kernfs_create_dir(parent_kn, name, parent_kn->mode, prgrp);
+	if (IS_ERR(kn))
+		return PTR_ERR(kn);

fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3307

Quoted 10 lines
fs/resctrl/rdtgroup.c +7 -0
+
+	ret = rdtgroup_kn_set_ugid(kn);
+	if (ret)
+		goto out_destroy;
+	ret = mon_add_all_files(kn, hdr, r, prgrp, hdr->id, false);
+	if (ret)
+		goto out_destroy;
 
 	kernfs_activate(kn);
 	return 0;

Reinette

author maintainer
#37 lore raw

Re: [PATCH v12 20/31] fs/resctrl: Refactor mkdir_mondata_subdir()

Quoted 10 lines · Reinette Chatre

fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3259

Quoted 2 lines
fs/resctrl/rdtgroup.c +2 -0
+	sprintf(name, "mon_sub_%s_%02d", r->name, hdr->id);
+	ckn = kernfs_create_dir(kn, name, parent_kn->mode, prgrp);

Noting here that kn was created earlier with mode of parent_kn->mode. It thus looks to me like above can also be written as:

	ckn = kernfs_create_dir(kn, name, kn->mode, prgrp);

The reason I mention this is that this patch adds a third copy of a very similar code snippet (kernfs_create_dir(), rdtgroup_kn_set_ugid(), mon_add_all_files()) that looks like a good candidate for a helper?

I looked at this. But the helper needs a lot of arguments to cover these three cases. Something like:

static struct kernfs_node *make_and_fill_mondir(struct kernfs_node *parent_kn,
char *name, umode_t mode, struct rdtgroup *prgrp, struct rdt_domain_hdr *hdr, struct rdt_resource *r, int domid)
{
code here
}

A subset of this repeated pattern (just the kernfs_create_dir(), rdtgroup_kn_set_ugid()) happens on every(?) mkdir. Perhaps a better helper (for cleanup patch unrelated to this series) would build the rdtgroup_kn_set_ugid() into a new rdtgroup_add_dir() helper?

-Tony

maintainer
#38 lore raw

Re: [PATCH v12 20/31] fs/resctrl: Refactor mkdir_mondata_subdir()

Hi Tony,

Quoted 12 lines · Tony
Quoted 10 lines · Reinette Chatre

fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3259

Quoted 2 lines
fs/resctrl/rdtgroup.c +2 -0
+	sprintf(name, "mon_sub_%s_%02d", r->name, hdr->id);
+	ckn = kernfs_create_dir(kn, name, parent_kn->mode, prgrp);

Noting here that kn was created earlier with mode of parent_kn->mode. It thus looks to me like above can also be written as:

	ckn = kernfs_create_dir(kn, name, kn->mode, prgrp);

The reason I mention this is that this patch adds a third copy of a very similar code snippet (kernfs_create_dir(), rdtgroup_kn_set_ugid(), mon_add_all_files()) that looks like a good candidate for a helper?

Quoted 5 lines

I looked at this. But the helper needs a lot of arguments to cover these three cases. Something like:

static struct kernfs_node *make_and_fill_mondir(struct kernfs_node *parent_kn,
char *name, umode_t mode,

I aimed to preempt a response like this in the text you quoted that notes that a "mode" argument does not seem necessary. Are you hinting that mode is indeed required? If not, without "mode" there are six arguments which is just one more than mon_add_all_files() that will be called by it.
What is the threshold for there being too many arguments? This series does not seem to have trouble pushing an arch API call resctrl_arch_rmid_read() to eight arguments while there is a concern with the number of arguments of this static call that has fewer?

Quoted 8 lines

struct rdtgroup *prgrp, struct rdt_domain_hdr *hdr, struct rdt_resource *r, int domid)
{
code here
}

Quoted 4 lines

A subset of this repeated pattern (just the kernfs_create_dir(), rdtgroup_kn_set_ugid()) happens on every(?) mkdir. Perhaps a better helper (for cleanup patch unrelated to this series) would build the rdtgroup_kn_set_ugid() into a new rdtgroup_add_dir() helper?

rdtgroup_add_dir() sounds like a useful helper. It is not clear to me how its motivation is unique though. That is, why use motivation "rdtgroup_add_dir() is a helper that supports two calls that are often repeated: kernfs_create_dir() followed by rdtgroup_kn_set_ugid()." while, what will follow, two calls that are often repeated rdtgroup_add_dir() followed by mon_add_all_files() does not need a helper?

Reinette

author maintainer
#39 lore raw

Re: [PATCH v12 20/31] fs/resctrl: Refactor mkdir_mondata_subdir()

Quoted 21 lines · Reinette Chatre

Hi Tony,

Quoted 12 lines · Tony
Quoted 10 lines · Reinette Chatre

fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3259

Quoted 2 lines
fs/resctrl/rdtgroup.c +2 -0
+	sprintf(name, "mon_sub_%s_%02d", r->name, hdr->id);
+	ckn = kernfs_create_dir(kn, name, parent_kn->mode, prgrp);

Noting here that kn was created earlier with mode of parent_kn->mode. It thus looks to me like above can also be written as:

	ckn = kernfs_create_dir(kn, name, kn->mode, prgrp);

The reason I mention this is that this patch adds a third copy of a very similar code snippet (kernfs_create_dir(), rdtgroup_kn_set_ugid(), mon_add_all_files()) that looks like a good candidate for a helper?

Quoted 5 lines

I looked at this. But the helper needs a lot of arguments to cover these three cases. Something like:

static struct kernfs_node *make_and_fill_mondir(struct kernfs_node *parent_kn,
char *name, umode_t mode,

Quoted 8 lines

I aimed to preempt a response like this in the text you quoted that notes that a "mode" argument does not seem necessary. Are you hinting that mode is indeed required? If not, without "mode" there are six arguments which is just one more than mon_add_all_files() that will be called by it.
What is the threshold for there being too many arguments? This series does not seem to have trouble pushing an arch API call resctrl_arch_rmid_read() to eight arguments while there is a concern with the number of arguments of this static call that has fewer?

Ok. you have me convinced. Since this helper will be the only caller of mon_add_all_files(), would it be good to just add the extra args needed to this function and have it create the directory and add the files? It would need a new name to describe the added functions it performs. Maybe mon_add_domain_dir()? But better suggestions welcomed.

-Tony

maintainer
#40 lore raw

Re: [PATCH v12 20/31] fs/resctrl: Refactor mkdir_mondata_subdir()

Hi Tony,

Quoted 31 lines · Tony
Quoted 21 lines · Reinette Chatre

Hi Tony,

Quoted 12 lines · Tony
Quoted 10 lines · Reinette Chatre

fs/resctrl/rdtgroup.c static int mkdir_mondata_subdir(struct kernfs_node *parent_kn, · line 3259

Quoted 2 lines
fs/resctrl/rdtgroup.c +2 -0
+	sprintf(name, "mon_sub_%s_%02d", r->name, hdr->id);
+	ckn = kernfs_create_dir(kn, name, parent_kn->mode, prgrp);

Noting here that kn was created earlier with mode of parent_kn->mode. It thus looks to me like above can also be written as:

	ckn = kernfs_create_dir(kn, name, kn->mode, prgrp);

The reason I mention this is that this patch adds a third copy of a very similar code snippet (kernfs_create_dir(), rdtgroup_kn_set_ugid(), mon_add_all_files()) that looks like a good candidate for a helper?

Quoted 5 lines

I looked at this. But the helper needs a lot of arguments to cover these three cases. Something like:

static struct kernfs_node *make_and_fill_mondir(struct kernfs_node *parent_kn,
char *name, umode_t mode,

Quoted 8 lines

I aimed to preempt a response like this in the text you quoted that notes that a "mode" argument does not seem necessary. Are you hinting that mode is indeed required? If not, without "mode" there are six arguments which is just one more than mon_add_all_files() that will be called by it.
What is the threshold for there being too many arguments? This series does not seem to have trouble pushing an arch API call resctrl_arch_rmid_read() to eight arguments while there is a concern with the number of arguments of this static call that has fewer?

Quoted 5 lines

Ok. you have me convinced. Since this helper will be the only caller of mon_add_all_files(), would it be good to just add the extra args needed to this function and have it create the directory and add the files? It would need a new name to describe the added functions it performs. Maybe mon_add_domain_dir()? But better suggestions welcomed.

Modifying mon_add_all_files() sounds good. I assume the node activation (kernfs_active()) will still be done by the caller which would have this new function return a struct kernfs_node *? If so, I think it will make code easier to read if name implies that a new kn is created. Since the caller is already mkdir_mondata_subdir(), what do you think of _mkdir_mondata_subdir()?

Reinette

author maintainer
#41 lore raw

Re: [PATCH v12 20/31] fs/resctrl: Refactor mkdir_mondata_subdir()

Quoted 4 lines · Reinette Chatre

Modifying mon_add_all_files() sounds good. I assume the node activation (kernfs_active()) will still be done by the caller which would have this new function return a struct kernfs_node *? If so, I think it will make code easier to read if name implies that a new kn is created. Since the caller is already mkdir_mondata_subdir(), what do you think of _mkdir_mondata_subdir()?

Reinette,

This section of fs/resctrl/rdtgroup.c now looks like this. Call sites are now free of duplicated code. Thanks for the nudge to do this.

-Tony

---

/*
* Create a directory for a domain and populate it with monitor files.
*/
static struct kernfs_node *_mkdir_mondata_subdir(struct kernfs_node *parent_kn, char *name,
struct rdt_domain_hdr *hdr, struct rdt_resource *r, struct rdtgroup *prgrp, int domid)

{
	struct rmid_read rr = {0};
	struct kernfs_node *kn;
	struct mon_data *priv;
	struct mon_evt *mevt;
	int ret;

	kn = kernfs_create_dir(parent_kn, name, parent_kn->mode, prgrp);
	if (IS_ERR(kn))
		return kn;

	ret = rdtgroup_kn_set_ugid(kn);
	if (ret)
		goto out_destroy;

	for_each_mon_event(mevt) {
		if (mevt->rid != r->rid || !mevt->enabled)
			continue;
		priv = mon_get_kn_priv(r->rid, domid, mevt, !hdr);
		if (WARN_ON_ONCE(!priv)) {
			ret = -EINVAL;
			goto out_destroy;
		}

		ret = mon_addfile(kn, mevt->name, priv);
		if (ret)
			goto out_destroy;

		if (hdr && resctrl_is_mbm_event(mevt->evtid))
			mon_event_read(&rr, r, hdr, prgrp, &hdr->cpu_mask, mevt, true);
	}

	return kn;
out_destroy:
	kernfs_remove(kn);
	return ERR_PTR(ret);
}

static int mkdir_mondata_subdir_snc(struct kernfs_node *parent_kn,
struct rdt_domain_hdr *hdr,
struct rdt_resource *r, struct rdtgroup *prgrp)

{
	struct kernfs_node *ckn, *kn;
	struct rdt_l3_mon_domain *d;
	char name[32];

	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
		return -EINVAL;

	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
	sprintf(name, "mon_%s_%02d", r->name, d->ci_id);
	kn = kernfs_find_and_get(parent_kn, name);
	if (kn) {
		/*
		 * rdtgroup_mutex will prevent this directory from being
		 * removed. No need to keep this hold.
		 */
		kernfs_put(kn);
	} else {
		kn = _mkdir_mondata_subdir(parent_kn, name, NULL, r, prgrp, d->ci_id);
		if (IS_ERR(kn))
			return PTR_ERR(kn);
	}

	sprintf(name, "mon_sub_%s_%02d", r->name, hdr->id);
	ckn = _mkdir_mondata_subdir(kn, name, hdr, r, prgrp, hdr->id);
	if (IS_ERR(ckn)) {
		kernfs_remove(kn);
		return PTR_ERR(ckn);
	}

	kernfs_activate(kn);
	return 0;
}

static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,
struct rdt_domain_hdr *hdr,
struct rdt_resource *r, struct rdtgroup *prgrp)

{
	struct kernfs_node *kn;
	char name[32];

	lockdep_assert_held(&rdtgroup_mutex);

	if (r->rid == RDT_RESOURCE_L3 && r->mon_scope == RESCTRL_L3_NODE)
		return mkdir_mondata_subdir_snc(parent_kn, hdr, r, prgrp);

	sprintf(name, "mon_%s_%02d", r->name, hdr->id);
	kn = _mkdir_mondata_subdir(parent_kn, name, hdr, r, prgrp, hdr->id);
	if (IS_ERR(kn))
		return PTR_ERR(kn);

	kernfs_activate(kn);
	return 0;
}
maintainer
#42 lore raw

Re: [PATCH v12 20/31] fs/resctrl: Refactor mkdir_mondata_subdir()

Hi Tony,

Quoted 8 lines · Tony
Quoted 4 lines · Reinette Chatre

Modifying mon_add_all_files() sounds good. I assume the node activation (kernfs_active()) will still be done by the caller which would have this new function return a struct kernfs_node *? If so, I think it will make code easier to read if name implies that a new kn is created. Since the caller is already mkdir_mondata_subdir(), what do you think of _mkdir_mondata_subdir()?

Reinette,

Quoted 7 lines

This section of fs/resctrl/rdtgroup.c now looks like this. Call sites are now free of duplicated code. Thanks for the nudge to do this.

/*
* Create a directory for a domain and populate it with monitor files.

Could you please add a short snippet about the different meanings of @hdr? It does not have to be entirely kernel-doc style, just a note for the developer trying to understand this function. Something like: "Create summing monitors when @hdr is NULL. No need to initialize summing monitors."

Quoted 1 line

*/

...

I find this new version much easier to parse. Thank you very much.

Reinette

author maintainer
#43 lore raw

PATCH v12 21/31 fs/resctrl: Refactor rmdir_mondata_subdir_allrdtgrp()

In reply to Tony Luck

rmdir_mondata_subdir_allrdtgrp() is complicated because of the support for Sub-NUMA Cluster (SNC) mode.

Refactor into a helper function for the SNC case to make it easier to add support for a new telemetry resource.

---
 fs/resctrl/rdtgroup.c | 41 +++++++++++++++++++++++++++++------------
 1 file changed, 29 insertions(+), 12 deletions(-)
fs/resctrl/rdtgroup.c +29 -12
index b7eaa4388768..837279d97db4 100644
@@ -3162,6 +3162,29 @@ static void mon_rmdir_one_subdir(struct kernfs_node *pkn, char *name, char *subn
 		kernfs_remove_by_name(kn, subname);
 }
 
+static void rmdir_mondata_subdir_allrdtgrp_snc(struct rdt_resource *r,
+					       struct rdt_domain_hdr *hdr)
+{
+	struct rdtgroup *prgrp, *crgrp;
+	struct rdt_l3_mon_domain *d;
+	char subname[32];
+	char name[32];
+
+	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+		return;
+
+	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
+	sprintf(name, "mon_%s_%02d", r->name, d->ci_id);
+	sprintf(subname, "mon_sub_%s_%02d", r->name, hdr->id);
+
+	list_for_each_entry(prgrp, &rdt_all_groups, rdtgroup_list) {
+		mon_rmdir_one_subdir(prgrp->mon.mon_data_kn, name, subname);
+
+		list_for_each_entry(crgrp, &prgrp->mon.crdtgrp_list, mon.crdtgrp_list)
+			mon_rmdir_one_subdir(crgrp->mon.mon_data_kn, name, subname);
+	}
+}
+
 /*
  * Remove all subdirectories of mon_data of ctrl_mon groups
  * and monitor groups for the given domain.
@@ -3172,25 +3195,19 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,
 					   struct rdt_domain_hdr *hdr)
 {
 	struct rdtgroup *prgrp, *crgrp;
-	struct rdt_l3_mon_domain *d;
-	char subname[32];
-	bool snc_mode;
 	char name[32];
 
-	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+	if (r->mon_scope == RESCTRL_L3_NODE) {
+		rmdir_mondata_subdir_allrdtgrp_snc(r, hdr);
 		return;
+	}
 
-	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
-	snc_mode = r->mon_scope == RESCTRL_L3_NODE;
-	sprintf(name, "mon_%s_%02d", r->name, snc_mode ? d->ci_id : d->hdr.id);
-	if (snc_mode)
-		sprintf(subname, "mon_sub_%s_%02d", r->name, d->hdr.id);
-
+	sprintf(name, "mon_%s_%02d", r->name, hdr->id);
 	list_for_each_entry(prgrp, &rdt_all_groups, rdtgroup_list) {
-		mon_rmdir_one_subdir(prgrp->mon.mon_data_kn, name, subname);
+		kernfs_remove_by_name(prgrp->mon.mon_data_kn, name);
 
 		list_for_each_entry(crgrp, &prgrp->mon.crdtgrp_list, mon.crdtgrp_list)
-			mon_rmdir_one_subdir(crgrp->mon.mon_data_kn, name, subname);
+			kernfs_remove_by_name(crgrp->mon.mon_data_kn, name);
 	}
 }
 
Signature · 1 line
2.51.0
maintainer
#44 lore raw

Re: [PATCH v12 21/31] fs/resctrl: Refactor rmdir_mondata_subdir_allrdtgrp()

Hi Tony,

Quoted 2 lines · Tony Luck

rmdir_mondata_subdir_allrdtgrp() is complicated because of the support for Sub-NUMA Cluster (SNC) mode.

"rmdir_mondata_subdir_allrdtgrp()" -> "Clearing a monitor group's mon_data directory"?

Quoted 2 lines

Refactor into a helper function for the SNC case to make it easier to add support

"Refactor into a helper function for the SNC case" -> "Refactor the SNC case into a helper function"?

Quoted 43 lines

for a new telemetry resource.

---
 fs/resctrl/rdtgroup.c | 41 +++++++++++++++++++++++++++++------------
 1 file changed, 29 insertions(+), 12 deletions(-)
fs/resctrl/rdtgroup.c +23 -0
index b7eaa4388768..837279d97db4 100644
@@ -3162,6 +3162,29 @@ static void mon_rmdir_one_subdir(struct kernfs_node *pkn, char *name, char *subn
 		kernfs_remove_by_name(kn, subname);
 }
 
+static void rmdir_mondata_subdir_allrdtgrp_snc(struct rdt_resource *r,
+					       struct rdt_domain_hdr *hdr)
+{
+	struct rdtgroup *prgrp, *crgrp;
+	struct rdt_l3_mon_domain *d;
+	char subname[32];
+	char name[32];
+
+	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+		return;
+
+	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
+	sprintf(name, "mon_%s_%02d", r->name, d->ci_id);
+	sprintf(subname, "mon_sub_%s_%02d", r->name, hdr->id);
+
+	list_for_each_entry(prgrp, &rdt_all_groups, rdtgroup_list) {
+		mon_rmdir_one_subdir(prgrp->mon.mon_data_kn, name, subname);
+
+		list_for_each_entry(crgrp, &prgrp->mon.crdtgrp_list, mon.crdtgrp_list)
+			mon_rmdir_one_subdir(crgrp->mon.mon_data_kn, name, subname);
+	}
+}
+
 /*
  * Remove all subdirectories of mon_data of ctrl_mon groups
  * and monitor groups for the given domain.

fs/resctrl/rdtgroup.c static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, · line 3195

Quoted 1 line
fs/resctrl/rdtgroup.c +0 -0
@@ -3172,25 +3195,19 @@ static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,
 					   struct rdt_domain_hdr *hdr)

This function's comments still mentions the actions needed to support SNC. Can this be moved with the code?

fs/resctrl/rdtgroup.c static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, · line 3200

Quoted 9 lines
fs/resctrl/rdtgroup.c +1 -4
 {
 	struct rdtgroup *prgrp, *crgrp;
-	struct rdt_l3_mon_domain *d;
-	char subname[32];
-	bool snc_mode;
 	char name[32];
 
-	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+	if (r->mon_scope == RESCTRL_L3_NODE) {

https://lore.kernel.org/lkml/aOhd_A5L8PV0OYba@agluck-desk3/

Quoted 11 lines
fs/resctrl/rdtgroup.c +3 -6
+		rmdir_mondata_subdir_allrdtgrp_snc(r, hdr);
 		return;
+	}
 
-	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
-	snc_mode = r->mon_scope == RESCTRL_L3_NODE;
-	sprintf(name, "mon_%s_%02d", r->name, snc_mode ? d->ci_id : d->hdr.id);
-	if (snc_mode)
-		sprintf(subname, "mon_sub_%s_%02d", r->name, d->hdr.id);
-
+	sprintf(name, "mon_%s_%02d", r->name, hdr->id);

fs/resctrl/rdtgroup.c static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, · line 3213

Quoted 10 lines
fs/resctrl/rdtgroup.c +2 -2
 	list_for_each_entry(prgrp, &rdt_all_groups, rdtgroup_list) {
-		mon_rmdir_one_subdir(prgrp->mon.mon_data_kn, name, subname);
+		kernfs_remove_by_name(prgrp->mon.mon_data_kn, name);
 
 		list_for_each_entry(crgrp, &prgrp->mon.crdtgrp_list, mon.crdtgrp_list)
-			mon_rmdir_one_subdir(crgrp->mon.mon_data_kn, name, subname);
+			kernfs_remove_by_name(crgrp->mon.mon_data_kn, name);
 	}
 }
 

Reinette

author maintainer
#45 lore raw

PATCH v12 22/31 x86,fs/resctrl: Handle domain creation/deletion for RDT_RESOURCE_PERF_PKG

In reply to Tony Luck

The L3 resource has several requirements for domains. There are per-domain structures that hold the 64-bit values of counters, and elements to keep track of the overflow and limbo threads.

None of these are needed for the PERF_PKG resource. The hardware counters are wide enough that they do not wrap around for decades.

Define a new rdt_perf_pkg_mon_domain structure which just consists of the standard rdt_domain_hdr to keep track of domain id and CPU mask.

Update resctrl_online_mon_domain() for RDT_RESOURCE_PERF_PKG. The only action needed for this resource is to create and populate domain directories if a domain is added while resctrl is mounted.

Similarly resctrl_offline_mon_domain() only needs to remove domain directories.

---
 arch/x86/kernel/cpu/resctrl/internal.h  | 13 +++++++++++
 arch/x86/kernel/cpu/resctrl/core.c      | 17 +++++++++++++++
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 29 +++++++++++++++++++++++++
 fs/resctrl/rdtgroup.c                   | 17 ++++++++++-----
 4 files changed, 71 insertions(+), 5 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +13 -0
index 97616c81682b..b920f54f8736 100644
@@ -84,6 +84,14 @@ static inline struct rdt_hw_l3_mon_domain *resctrl_to_arch_mon_dom(struct rdt_l3
 	return container_of(r, struct rdt_hw_l3_mon_domain, d_resctrl);
 }
 
+/**
+ * struct rdt_perf_pkg_mon_domain - CPUs sharing an package scoped resctrl monitor resource
+ * @hdr:	common header for different domain types
+ */
+struct rdt_perf_pkg_mon_domain {
+	struct rdt_domain_hdr	hdr;
+};
+
 /**
  * struct msr_param - set a range of MSRs from a domain
  * @res:       The resource to use
@@ -222,6 +230,8 @@ bool intel_aet_get_events(void);
 void __exit intel_aet_exit(void);
 int intel_aet_read_event(int domid, u32 rmid, enum resctrl_event_id evtid,
 			 void *arch_priv, u64 *val);
+void intel_aet_mon_domain_setup(int cpu, int id, struct rdt_resource *r,
+				struct list_head *add_pos);
 #else
 static inline bool intel_aet_get_events(void) { return false; }
 static inline void __exit intel_aet_exit(void) { }
@@ -230,6 +240,9 @@ static inline int intel_aet_read_event(int domid, u32 rmid, enum resctrl_event_i
 {
 	return -EINVAL;
 }
+
+static inline void intel_aet_mon_domain_setup(int cpu, int id, struct rdt_resource *r,
+					      struct list_head *add_pos) { }
 #endif
 
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/core.c +17 -0
index 6584461d4cb3..7d15112c5868 100644
@@ -575,6 +575,10 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 		if (!hdr)
 			l3_mon_domain_setup(cpu, id, r, add_pos);
 		break;
+	case RDT_RESOURCE_PERF_PKG:
+		if (!hdr)
+			intel_aet_mon_domain_setup(cpu, id, r, add_pos);
+		break;
 	default:
 		pr_warn_once("Unknown resource rid=%d\n", r->rid);
 		break;
@@ -674,6 +678,19 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
 		l3_mon_domain_free(hw_dom);
 		break;
 	}
+	case RDT_RESOURCE_PERF_PKG: {
+		struct rdt_perf_pkg_mon_domain *pkgd;
+
+		if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_PERF_PKG))
+			return;
+
+		pkgd = container_of(hdr, struct rdt_perf_pkg_mon_domain, hdr);
+		resctrl_offline_mon_domain(r, hdr);
+		list_del_rcu(&hdr->list);
+		synchronize_rcu();
+		kfree(pkgd);
+		break;
+	}
 	default:
 		pr_warn_once("Unknown resource rid=%d\n", r->rid);
 		break;
arch/x86/kernel/cpu/resctrl/intel_aet.c +29 -0
index 003b6426c8bd..fa6d3456fb7a 100644
@@ -17,16 +17,21 @@
 #include <linux/compiler_types.h>
 #include <linux/container_of.h>
 #include <linux/cpu.h>
+#include <linux/cpumask.h>
 #include <linux/err.h>
 #include <linux/errno.h>
+#include <linux/gfp_types.h>
 #include <linux/init.h>
 #include <linux/intel_pmt_features.h>
 #include <linux/intel_vsec.h>
 #include <linux/io.h>
 #include <linux/overflow.h>
 #include <linux/printk.h>
+#include <linux/rculist.h>
+#include <linux/rcupdate.h>
 #include <linux/resctrl.h>
 #include <linux/resctrl_types.h>
+#include <linux/slab.h>
 #include <linux/stddef.h>
 #include <linux/topology.h>
 #include <linux/types.h>
@@ -278,3 +283,27 @@ int intel_aet_read_event(int domid, u32 rmid, enum resctrl_event_id eventid,
 
 	return valid ? 0 : -EINVAL;
 }
+
+void intel_aet_mon_domain_setup(int cpu, int id, struct rdt_resource *r,
+				struct list_head *add_pos)
+{
+	struct rdt_perf_pkg_mon_domain *d;
+	int err;
+
+	d = kzalloc_node(sizeof(*d), GFP_KERNEL, cpu_to_node(cpu));
+	if (!d)
+		return;
+
+	d->hdr.id = id;
+	d->hdr.type = RESCTRL_MON_DOMAIN;
+	d->hdr.rid = RDT_RESOURCE_PERF_PKG;
+	cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
+	list_add_tail_rcu(&d->hdr.list, add_pos);
+
+	err = resctrl_online_mon_domain(r, &d->hdr);
+	if (err) {
+		list_del_rcu(&d->hdr.list);
+		synchronize_rcu();
+		kfree(d);
+	}
+}
fs/resctrl/rdtgroup.c +12 -5
index 837279d97db4..2238a5536f4b 100644
@@ -4251,11 +4251,6 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h
 
 	mutex_lock(&rdtgroup_mutex);
 
-	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
-		goto out_unlock;
-
-	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
-
 	/*
 	 * If resctrl is mounted, remove all the
 	 * per domain monitor data directories.
@@ -4263,6 +4258,13 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h
 	if (resctrl_mounted && resctrl_arch_mon_capable())
 		rmdir_mondata_subdir_allrdtgrp(r, hdr);
 
+	if (r->rid != RDT_RESOURCE_L3)
+		goto out_unlock;
+
+	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
+		goto out_unlock;
+
+	d = container_of(hdr, struct rdt_l3_mon_domain, hdr);
 	if (resctrl_is_mbm_enabled())
 		cancel_delayed_work(&d->mbm_over);
 	if (resctrl_is_mon_event_enabled(QOS_L3_OCCUP_EVENT_ID) && has_busy_rmid(d)) {
@@ -4359,6 +4361,9 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr
 
 	mutex_lock(&rdtgroup_mutex);
 
+	if (r->rid != RDT_RESOURCE_L3)
+		goto mkdir;
+
 	if (!domain_header_is_valid(hdr, RESCTRL_MON_DOMAIN, RDT_RESOURCE_L3))
 		goto out_unlock;
 
@@ -4376,6 +4381,8 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr
 	if (resctrl_is_mon_event_enabled(QOS_L3_OCCUP_EVENT_ID))
 		INIT_DELAYED_WORK(&d->cqm_limbo, cqm_handle_limbo);
 
+mkdir:
+	err = 0;
 	/*
 	 * If the filesystem is not mounted then only the default resource group
 	 * exists. Creation of its directories is deferred until mount time
Signature · 1 line
2.51.0
maintainer Reviewed-by on v12 22/31
#46 lore raw

Re: [PATCH v12 22/31] x86,fs/resctrl: Handle domain creation/deletion for RDT_RESOURCE_PERF_PKG

Hi Tony,

Quoted 14 lines · Tony Luck

The L3 resource has several requirements for domains. There are per-domain structures that hold the 64-bit values of counters, and elements to keep track of the overflow and limbo threads.

None of these are needed for the PERF_PKG resource. The hardware counters are wide enough that they do not wrap around for decades.

Define a new rdt_perf_pkg_mon_domain structure which just consists of the standard rdt_domain_hdr to keep track of domain id and CPU mask.

Update resctrl_online_mon_domain() for RDT_RESOURCE_PERF_PKG. The only action needed for this resource is to create and populate domain directories if a domain is added while resctrl is mounted.

Quoted 5 lines

Similarly resctrl_offline_mon_domain() only needs to remove domain directories.

---

Reinette

author maintainer
#47 lore raw

PATCH v12 23/31 x86/resctrl: Add energy/perf choices to rdt boot option

In reply to Tony Luck

Legacy resctrl features are enumerated by X86_FEATURE_* flags. These may be overridden by quirks to disable features in the case of errata. Users can use kernel command line options to either disable a feature, or to force enable a feature that was disabled by a quirk.

Provide similar functionality for hardware features that do not have an X86_FEATURE_* flag. Unlike other features that are tied to X86_FEATURE_* flags, these must be queried by name. Add rdt_is_feature_enabled() to check whether quirks or kernel command line have disabled a feature.

Users may force a feature to be disabled. E.g. "rdt=!perf" will ensure that none of the perf telemetry events are enabled.

Resctrl architecture code may disable a feature that does not provide full functionality. Users may override that decision. E.g. "rdt=energy" will enable any available energy telemetry events even if they do not provide full functionality.

---
 .../admin-guide/kernel-parameters.txt         |  2 +-
 arch/x86/kernel/cpu/resctrl/internal.h        |  2 ++
 arch/x86/kernel/cpu/resctrl/core.c            | 29 +++++++++++++++++++
 3 files changed, 32 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +1 -1
index 6c42061ca20e..bb8f5d73ebf8 100644
@@ -6207,7 +6207,7 @@
 	rdt=		[HW,X86,RDT]
 			Turn on/off individual RDT features. List is:
 			cmt, mbmtotal, mbmlocal, l3cat, l3cdp, l2cat, l2cdp,
-			mba, smba, bmec, abmc.
+			mba, smba, bmec, abmc, energy, perf.
 			E.g. to turn on cmt and turn off mba use:
 				rdt=cmt,!mba
 
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
index b920f54f8736..e3710b9f993e 100644
@@ -225,6 +225,8 @@ void __init intel_rdt_mbm_apply_quirk(void);
 void rdt_domain_reconfigure_cdp(struct rdt_resource *r);
 void resctrl_arch_mbm_cntr_assign_set_one(struct rdt_resource *r);
 
+bool rdt_is_feature_enabled(char *name);
+
 #ifdef CONFIG_X86_CPU_RESCTRL_INTEL_AET
 bool intel_aet_get_events(void);
 void __exit intel_aet_exit(void);
arch/x86/kernel/cpu/resctrl/core.c +29 -0
index 7d15112c5868..9d14ebde9d89 100644
@@ -772,6 +772,8 @@ enum {
 	RDT_FLAG_SMBA,
 	RDT_FLAG_BMEC,
 	RDT_FLAG_ABMC,
+	RDT_FLAG_ENERGY,
+	RDT_FLAG_PERF,
 };
 
 #define RDT_OPT(idx, n, f)	\
@@ -798,6 +800,8 @@ static struct rdt_options rdt_options[]  __ro_after_init = {
 	RDT_OPT(RDT_FLAG_SMBA,	    "smba",	X86_FEATURE_SMBA),
 	RDT_OPT(RDT_FLAG_BMEC,	    "bmec",	X86_FEATURE_BMEC),
 	RDT_OPT(RDT_FLAG_ABMC,	    "abmc",	X86_FEATURE_ABMC),
+	RDT_OPT(RDT_FLAG_ENERGY,    "energy",	0),
+	RDT_OPT(RDT_FLAG_PERF,	    "perf",	0),
 };
 #define NUM_RDT_OPTIONS ARRAY_SIZE(rdt_options)
 
@@ -847,6 +851,31 @@ bool rdt_cpu_has(int flag)
 	return ret;
 }
 
+/*
+ * Hardware features that do not have X86_FEATURE_* bits. There is no
+ * "hardware does not support this at all" case. Assume that the caller
+ * has already determined that hardware support is present and just needs
+ * to check if the feature has been disabled by a quirk that has not been
+ * overridden by a command line option.
+ */
+bool rdt_is_feature_enabled(char *name)
+{
+	struct rdt_options *o;
+	bool ret = true;
+
+	for (o = rdt_options; o < &rdt_options[NUM_RDT_OPTIONS]; o++) {
+		if (!strcmp(name, o->name)) {
+			if (o->force_off)
+				ret = false;
+			if (o->force_on)
+				ret = true;
+			break;
+		}
+	}
+
+	return ret;
+}
+
 bool resctrl_arch_is_evt_configurable(enum resctrl_event_id evt)
 {
 	if (!rdt_cpu_has(X86_FEATURE_BMEC))
Signature · 1 line
2.51.0
maintainer Reviewed-by on v12 23/31
#48 lore raw

Re: [PATCH v12 23/31] x86/resctrl: Add energy/perf choices to rdt boot option

Hi Tony,

Quoted 13 lines · Tony Luck

Legacy resctrl features are enumerated by X86_FEATURE_* flags. These may be overridden by quirks to disable features in the case of errata. Users can use kernel command line options to either disable a feature, or to force enable a feature that was disabled by a quirk.

Provide similar functionality for hardware features that do not have an X86_FEATURE_* flag. Unlike other features that are tied to X86_FEATURE_* flags, these must be queried by name. Add rdt_is_feature_enabled() to check whether quirks or kernel command line have disabled a feature.

Users may force a feature to be disabled. E.g. "rdt=!perf" will ensure that none of the perf telemetry events are enabled.

Quoted 7 lines

Resctrl architecture code may disable a feature that does not provide full functionality. Users may override that decision. E.g. "rdt=energy" will enable any available energy telemetry events even if they do not provide full functionality.

---

Reinette

author maintainer
#49 lore raw

PATCH v12 24/31 x86/resctrl: Handle number of RMIDs supported by RDT_RESOURCE_PERF_PKG

In reply to Tony Luck

There are now three meanings for "number of RMIDs":

1) The number for legacy features enumerated by CPUID leaf 0xF. This is the maximum number of distinct values that can be loaded into MSR_IA32_PQR_ASSOC. Note that systems with Sub-NUMA Cluster mode enabled will force scaling down the CPUID enumerated value by the number of SNC nodes per L3-cache.

2) The number of registers in MMIO space for each event. This is enumerated in the XML files and is the value initialized into event_group::num_rmids.

3) The number of "hardware counters" (this isn't a strictly accurate description of how things work, but serves as a useful analogy that does describe the limitations) feeding to those MMIO registers. This is enumerated in telemetry_region::num_rmids returned from the call to intel_pmt_get_regions_by_feature()

Event groups with insufficient "hardware counters" to track all RMIDs are difficult for users to use, since the system may reassign "hardware counters" at any time. This means that users cannot reliably collect two consecutive event counts to compute the rate at which events are occurring.

Introduce rdt_set_feature_disabled() to mark any under-resourced event groups (those with telemetry_region::num_rmids < event_group::num_rmids for any of the event group's telemetry regions) as unusable. Note that the rdt_options[] structure must now be writable at run-time.

Limit an event group's number of possible monitor resource groups to the lowest number of "hardware counters" if the user explicitly requests to enable an under-resourced event group.

Scan all enabled event groups and assign the RDT_RESOURCE_PERF_PKG resource "num_rmids" value to the smallest of these values as this value will be used later to compare against the number of RMIDs supported by other resources to determine how many monitoring resource groups are supported.

N.B. Change type of rdt_resource::num_rmid to u32 to match type of event_group::num_rmids so that min(r->num_rmid, e->num_rmids) won't complain about mixing signed and unsigned types. Print r->num_rmid as unsigned value in rdt_num_rmids_show().

---
 include/linux/resctrl.h                 |  2 +-
 arch/x86/kernel/cpu/resctrl/internal.h  |  2 +
 arch/x86/kernel/cpu/resctrl/core.c      | 18 +++++++-
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 56 +++++++++++++++++++++++--
 fs/resctrl/rdtgroup.c                   |  2 +-
 5 files changed, 74 insertions(+), 6 deletions(-)
include/linux/resctrl.h +1 -1
index 111c8f1dc77e..c7b5e56d25bb 100644
@@ -292,7 +292,7 @@ enum resctrl_schema_fmt {
  *			events of monitor groups created via mkdir.
  */
 struct resctrl_mon {
-	int			num_rmid;
+	u32			num_rmid;
 	unsigned int		mbm_cfg_mask;
 	int			num_mbm_cntrs;
 	bool			mbm_cntr_assignable;
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
index e3710b9f993e..cea76f88422c 100644
@@ -227,6 +227,8 @@ void resctrl_arch_mbm_cntr_assign_set_one(struct rdt_resource *r);
 
 bool rdt_is_feature_enabled(char *name);
 
+void rdt_set_feature_disabled(char *name);
+
 #ifdef CONFIG_X86_CPU_RESCTRL_INTEL_AET
 bool intel_aet_get_events(void);
 void __exit intel_aet_exit(void);
arch/x86/kernel/cpu/resctrl/core.c +17 -1
index 9d14ebde9d89..cc3479429044 100644
@@ -788,7 +788,7 @@ struct rdt_options {
 	bool	force_off, force_on;
 };
 
-static struct rdt_options rdt_options[]  __ro_after_init = {
+static struct rdt_options rdt_options[] = {
 	RDT_OPT(RDT_FLAG_CMT,	    "cmt",	X86_FEATURE_CQM_OCCUP_LLC),
 	RDT_OPT(RDT_FLAG_MBM_TOTAL, "mbmtotal", X86_FEATURE_CQM_MBM_TOTAL),
 	RDT_OPT(RDT_FLAG_MBM_LOCAL, "mbmlocal", X86_FEATURE_CQM_MBM_LOCAL),
@@ -851,6 +851,22 @@ bool rdt_cpu_has(int flag)
 	return ret;
 }
 
+/*
+ * Can be called during feature enumeration if sanity check of
+ * a feature's parameters indicates problems with the feature.
+ */
+void rdt_set_feature_disabled(char *name)
+{
+	struct rdt_options *o;
+
+	for (o = rdt_options; o < &rdt_options[NUM_RDT_OPTIONS]; o++) {
+		if (!strcmp(name, o->name)) {
+			o->force_off = true;
+			return;
+		}
+	}
+}
+
 /*
  * Hardware features that do not have X86_FEATURE_* bits. There is no
  * "hardware does not support this at all" case. Assume that the caller
arch/x86/kernel/cpu/resctrl/intel_aet.c +53 -3

116 lines not on this page · raw message

fs/resctrl/rdtgroup.c +1 -1
index 2238a5536f4b..f18cc5b38315 100644
@@ -1135,7 +1135,7 @@ static int rdt_num_rmids_show(struct kernfs_open_file *of,
 {
 	struct rdt_resource *r = rdt_kn_parent_priv(of->kn);
 
-	seq_printf(seq, "%d\n", r->mon.num_rmid);
+	seq_printf(seq, "%u\n", r->mon.num_rmid);
 
 	return 0;
 }
Signature · 1 line
2.51.0
maintainer
#50 lore raw

Re: [PATCH v12 24/31] x86/resctrl: Handle number of RMIDs supported by RDT_RESOURCE_PERF_PKG

Hi Tony,

Quoted 24 lines · Tony Luck

There are now three meanings for "number of RMIDs":

1) The number for legacy features enumerated by CPUID leaf 0xF. This is the maximum number of distinct values that can be loaded into MSR_IA32_PQR_ASSOC. Note that systems with Sub-NUMA Cluster mode enabled will force scaling down the CPUID enumerated value by the number of SNC nodes per L3-cache.

2) The number of registers in MMIO space for each event. This is enumerated in the XML files and is the value initialized into event_group::num_rmids.

3) The number of "hardware counters" (this isn't a strictly accurate description of how things work, but serves as a useful analogy that does describe the limitations) feeding to those MMIO registers. This is enumerated in telemetry_region::num_rmids returned from the call to intel_pmt_get_regions_by_feature()

Event groups with insufficient "hardware counters" to track all RMIDs are difficult for users to use, since the system may reassign "hardware counters" at any time. This means that users cannot reliably collect two consecutive event counts to compute the rate at which events are occurring.

Quoted 8 lines

Introduce rdt_set_feature_disabled() to mark any under-resourced event groups (those with telemetry_region::num_rmids < event_group::num_rmids for any of the event group's telemetry regions) as unusable. Note that the rdt_options[] structure must now be writable at run-time.

Limit an event group's number of possible monitor resource groups to the lowest number of "hardware counters" if the user explicitly requests to enable an under-resourced event group.

How about:
Limit an under-resourced event group's number of possible monitor resource groups to the lowest number of "hardware counters" if the user explicitly requests to enable it.

Quoted 10 lines

Scan all enabled event groups and assign the RDT_RESOURCE_PERF_PKG resource "num_rmids" value to the smallest of these values as this value will be used later to compare against the number of RMIDs supported by other resources to determine how many monitoring resource groups are supported.

N.B. Change type of rdt_resource::num_rmid to u32 to match type of event_group::num_rmids so that min(r->num_rmid, e->num_rmids) won't complain about mixing signed and unsigned types. Print r->num_rmid as unsigned value in rdt_num_rmids_show().

"Print r->num_rmid ..." can be dropped since that is clear from the patch.

Quoted 3 lines

---

...

arch/x86/kernel/cpu/resctrl/intel_aet.c static bool skip_telem_region(struct telemetry_region *tr, struct event_group *e · line 171

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +1 -0
@@ -156,21 +168,59 @@ static bool skip_telem_region(struct telemetry_region *tr, struct event_group *e
 	return false;
 }
 
+/* Side effect: Detects unusable regions and marks them as unusable */

:/

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +8 -0
+static bool all_regions_have_sufficient_rmid(struct event_group *e, struct pmt_feature_group *p)
+{
+	struct telemetry_region *tr;
+	bool ret = true;
+
+	for (int i = 0; i < p->count; i++) {
+		tr = &p->regions[i];
+		if (skip_telem_region(tr, e)) {

arch/x86/kernel/cpu/resctrl/intel_aet.c static bool skip_telem_region(struct telemetry_region *tr, struct event_group *e · line 189

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +10 -0
+			mark_telem_region_unusable(tr);
+			continue;
+		}
+
+		if (tr->num_rmids < e->num_rmids)
+			ret = false;
+	}
+
+	return ret;
+}

This does not look right. Wouldn't this return "true" for all_regions_have_sufficient_rmid() when there are no usable regions? Trying to have one function do two things is not working well here.

This also seems awkward where the regions are marked as unusable here as a "side effect" but then later the caller attempts to track "usable_events" separately? This change does not look to be integrated well.

Why not just determine which regions are usable as a first step and from then on just interact with usable regions?

arch/x86/kernel/cpu/resctrl/intel_aet.c static bool skip_telem_region(struct telemetry_region *tr, struct event_group *e · line 195

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +2 -0
+
 static bool enable_events(struct event_group *e, struct pmt_feature_group *p)
 {
+	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG].r_resctrl;
 	bool usable_events = false;
 

This flow can start by determining and mark which regions are usable. Could be something like:

	if (!group_has_usable_regions(e, p))
		return false;

To reduce churn group_has_usable_regions() can be introduced in patch #17 to replace the open code of it in enable_events().

From this point the "feature group" is guaranteed to have at least one telemetry region usable by the associated "event group" and all interactions can be with just the usable regions within it. For example, all_regions_have_sufficient_rmid() can change to:

	static bool all_regions_have_sufficient_rmid(struct event_group *e, struct pmt_feature_group *p)
	{
		struct telemetry_region *tr;
		bool ret = true;

		for (int i = 0; i < p->count; i++) {
			if (!p->regions[i].addr)
				continue;
			tr = &p->regions[i];
			if (tr->num_rmids < e->num_rmids)
				ret = false;
		}

		return ret;
	}
Quoted 39 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +20 -3
+	/* Disable feature if insufficient RMIDs */
+	if (!all_regions_have_sufficient_rmid(e, p))
+		rdt_set_feature_disabled(e->name);
+
+	/* User can override above disable from kernel command line */
+	if (!rdt_is_feature_enabled(e->name))
+		return false;
+
 	for (int i = 0; i < p->count; i++) {
-		if (skip_telem_region(&p->regions[i], e)) {
-			mark_telem_region_unusable(&p->regions[i]);
+		if (!p->regions[i].addr)
 			continue;
-		}
+		/*
+		 * e->num_rmids only adjusted lower if user (via rdt= kernel
+		 * parameter) forces an event group with insufficient RMID
+		 * to be enabled.
+		 */
+		e->num_rmids = min(e->num_rmids, p->regions[i].num_rmids);
 		usable_events = true;
 	}
 
 	if (!usable_events)
 		return false;
 
+	if (r->mon.num_rmid)
+		r->mon.num_rmid = min(r->mon.num_rmid, e->num_rmids);
+	else
+		r->mon.num_rmid = e->num_rmids;
+
 	for (int j = 0; j < e->num_events; j++)
 		resctrl_enable_mon_event(e->evts[j].id, true,
 					 e->evts[j].bin_bits, &e->evts[j]);
fs/resctrl/rdtgroup.c +0 -0
index 2238a5536f4b..f18cc5b38315 100644

fs/resctrl/rdtgroup.c static int rdt_num_rmids_show(struct kernfs_open_file *of, · line 1141

Quoted 8 lines
fs/resctrl/rdtgroup.c +1 -1
@@ -1135,7 +1135,7 @@ static int rdt_num_rmids_show(struct kernfs_open_file *of,
 {
 	struct rdt_resource *r = rdt_kn_parent_priv(of->kn);
 
-	seq_printf(seq, "%d\n", r->mon.num_rmid);
+	seq_printf(seq, "%u\n", r->mon.num_rmid);
 
 	return 0;
 }

Reinette

author maintainer
#51 lore raw

PATCH v12 25/31 fs/resctrl: Move allocation/free of closid_num_dirty_rmid[]

In reply to Tony Luck

closid_num_dirty_rmid[] and rmid_ptrs[] are allocated together during resctrl initialization and freed together during resctrl exit.

Telemetry events are enumerated on resctrl mount so only at resctrl mount will the number of RMID supported by all monitoring resources and needed as size for rmid_ptrs[] be known.

Separate closid_num_dirty_rmid[] and rmid_ptrs[] allocation and free in preparation for rmid_ptrs[] to be allocated on resctrl mount.

Keep the rdtgroup_mutex protection around the allocation and free of closid_num_dirty_rmid[] as ARM needs this to guarantee memory ordering.

---
 fs/resctrl/monitor.c | 77 ++++++++++++++++++++++++++++----------------
 1 file changed, 49 insertions(+), 28 deletions(-)
fs/resctrl/monitor.c +49 -28
index 84a0cbe90748..2782e2e212f6 100644
@@ -910,36 +910,14 @@ void mbm_setup_overflow_handler(struct rdt_l3_mon_domain *dom, unsigned long del
 static int dom_data_init(struct rdt_resource *r)
 {
 	u32 idx_limit = resctrl_arch_system_num_rmid_idx();
-	u32 num_closid = resctrl_arch_get_num_closid(r);
 	struct rmid_entry *entry = NULL;
 	int err = 0, i;
 	u32 idx;
 
 	mutex_lock(&rdtgroup_mutex);
-	if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
-		u32 *tmp;
-
-		/*
-		 * If the architecture hasn't provided a sanitised value here,
-		 * this may result in larger arrays than necessary. Resctrl will
-		 * use a smaller system wide value based on the resources in
-		 * use.
-		 */
-		tmp = kcalloc(num_closid, sizeof(*tmp), GFP_KERNEL);
-		if (!tmp) {
-			err = -ENOMEM;
-			goto out_unlock;
-		}
-
-		closid_num_dirty_rmid = tmp;
-	}
 
 	rmid_ptrs = kcalloc(idx_limit, sizeof(struct rmid_entry), GFP_KERNEL);
 	if (!rmid_ptrs) {
-		if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
-			kfree(closid_num_dirty_rmid);
-			closid_num_dirty_rmid = NULL;
-		}
 		err = -ENOMEM;
 		goto out_unlock;
 	}
@@ -975,11 +953,6 @@ static void dom_data_exit(struct rdt_resource *r)
 	if (!r->mon_capable)
 		goto out_unlock;
 
-	if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
-		kfree(closid_num_dirty_rmid);
-		closid_num_dirty_rmid = NULL;
-	}
-
 	kfree(rmid_ptrs);
 	rmid_ptrs = NULL;
 
@@ -1816,6 +1789,43 @@ ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf,
 	return ret ?: nbytes;
 }
 
+static int closid_num_dirty_rmid_alloc(struct rdt_resource *r)
+{
+	if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
+		u32 num_closid = resctrl_arch_get_num_closid(r);
+		u32 *tmp;
+
+		/* For ARM memory ordering access to closid_num_dirty_rmid */
+		mutex_lock(&rdtgroup_mutex);
+
+		/*
+		 * If the architecture hasn't provided a sanitised value here,
+		 * this may result in larger arrays than necessary. Resctrl will
+		 * use a smaller system wide value based on the resources in
+		 * use.
+		 */
+		tmp = kcalloc(num_closid, sizeof(*tmp), GFP_KERNEL);
+		if (!tmp)
+			return -ENOMEM;
+
+		closid_num_dirty_rmid = tmp;
+
+		mutex_unlock(&rdtgroup_mutex);
+	}
+
+	return 0;
+}
+
+static void closid_num_dirty_rmid_free(void)
+{
+	if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
+		mutex_lock(&rdtgroup_mutex);
+		kfree(closid_num_dirty_rmid);
+		closid_num_dirty_rmid = NULL;
+		mutex_unlock(&rdtgroup_mutex);
+	}
+}
+
 /**
  * resctrl_l3_mon_resource_init() - Initialise global monitoring structures.
  *
@@ -1836,10 +1846,16 @@ int resctrl_l3_mon_resource_init(void)
 	if (!r->mon_capable)
 		return 0;
 
-	ret = dom_data_init(r);
+	ret = closid_num_dirty_rmid_alloc(r);
 	if (ret)
 		return ret;
 
+	ret = dom_data_init(r);
+	if (ret) {
+		closid_num_dirty_rmid_free();
+		return ret;
+	}
+
 	if (resctrl_arch_is_evt_configurable(QOS_L3_MBM_TOTAL_EVENT_ID)) {
 		mon_event_all[QOS_L3_MBM_TOTAL_EVENT_ID].configurable = true;
 		resctrl_file_fflags_init("mbm_total_bytes_config",
@@ -1884,5 +1900,10 @@ void resctrl_l3_mon_resource_exit(void)
 {
 	struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
 
+	if (!r->mon_capable)
+		return;
+
+	closid_num_dirty_rmid_free();
+
 	dom_data_exit(r);
 }
Signature · 1 line
2.51.0
maintainer
#52 lore raw

Re: [PATCH v12 25/31] fs/resctrl: Move allocation/free of closid_num_dirty_rmid[]

Hi Tony,

Quoted 8 lines · Tony Luck
fs/resctrl/monitor.c +8 -0
+static int closid_num_dirty_rmid_alloc(struct rdt_resource *r)
+{
+	if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
+		u32 num_closid = resctrl_arch_get_num_closid(r);
+		u32 *tmp;
+
+		/* For ARM memory ordering access to closid_num_dirty_rmid */
+		mutex_lock(&rdtgroup_mutex);

fs/resctrl/monitor.c ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf, · line 1809

Quoted 10 lines
fs/resctrl/monitor.c +10 -0
+
+		/*
+		 * If the architecture hasn't provided a sanitised value here,
+		 * this may result in larger arrays than necessary. Resctrl will
+		 * use a smaller system wide value based on the resources in
+		 * use.
+		 */
+		tmp = kcalloc(num_closid, sizeof(*tmp), GFP_KERNEL);
+		if (!tmp)
+			return -ENOMEM;

Drop rdtgroup_mutex before returning here.

fs/resctrl/monitor.c ssize_t mbm_L3_assignments_write(struct kernfs_open_file *of, char *buf, · line 1817

Quoted 8 lines
fs/resctrl/monitor.c +8 -0
+
+		closid_num_dirty_rmid = tmp;
+
+		mutex_unlock(&rdtgroup_mutex);
+	}
+
+	return 0;
+}

Reinette

author maintainer
#53 lore raw

PATCH v12 26/31 x86,fs/resctrl: Compute number of RMIDs as minimum across resources

In reply to Tony Luck

resctrl assumes that only the L3 resource supports monitor events, so it simply takes the rdt_resource::num_rmid from RDT_RESOURCE_L3 as the system's number of RMIDs.

The addition of telemetry events in a different resource breaks that assumption.

Compute the number of available RMIDs as the minimum value across all mon_capable resources (analogous to how the number of CLOSIDs is computed across alloc_capable resources).

Note that mount time enumeration of the telemetry resource means that this number can be reduced. If this happens, then some memory will be wasted as the allocations for rdt_l3_mon_domain::mbm_states[] and rdt_l3_mon_domain::rmid_busy_llc created during resctrl initialization will be larger than needed.

---
 arch/x86/kernel/cpu/resctrl/core.c | 15 +++++++++++++--
 fs/resctrl/rdtgroup.c              |  6 ++++++
 2 files changed, 19 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +13 -2
index cc3479429044..955a1904ba2a 100644
@@ -110,12 +110,23 @@ struct rdt_hw_resource rdt_resources_all[RDT_NUM_RESOURCES] = {
 	},
 };
 
+/**
+ * resctrl_arch_system_num_rmid_idx - Compute number of supported RMIDs
+ *				      (minimum across all mon_capable resource)
+ *
+ * Return: Number of supported RMIDs at time of call. Note that mount time
+ * enumeration of resources may reduce the number.
+ */
 u32 resctrl_arch_system_num_rmid_idx(void)
 {
-	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
+	u32 num_rmids = U32_MAX;
+	struct rdt_resource *r;
+
+	for_each_mon_capable_rdt_resource(r)
+		num_rmids = min(num_rmids, r->mon.num_rmid);
 
 	/* RMID are independent numbers for x86. num_rmid_idx == num_rmid */
-	return r->mon.num_rmid;
+	return num_rmids == U32_MAX ? 0 : num_rmids;
 }
 
 struct rdt_resource *resctrl_arch_get_resource(enum resctrl_res_level l)
fs/resctrl/rdtgroup.c +6 -0
index f18cc5b38315..c94536382316 100644
@@ -4296,6 +4296,12 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *h
  * During boot this may be called before global allocations have been made by
  * resctrl_l3_mon_resource_init().
  *
+ * Called during CPU online that may run as soon as CPU online callbacks
+ * are set up during resctrl initialization. The number of supported RMIDs
+ * may be reduced if additional mon_capable resources are enumerated
+ * at mount time. This means the rdt_l3_mon_domain::mbm_states[] and
+ * rdt_l3_mon_domain::rmid_busy_llc allocations may be larger than needed.
+ *
  * Return: %0 for success; -ENOMEM otherwise.
  */
 static int domain_setup_l3_mon_state(struct rdt_resource *r, struct rdt_l3_mon_domain *d)
Signature · 1 line
2.51.0
author maintainer
#54 lore raw

PATCH v12 27/31 fs/resctrl: Move RMID initialization to first mount

In reply to Tony Luck

L3 monitor features are enumerated during resctrl initialization and rmid_ptrs[] that tracks all RMIDs and depends on the number of supported RMIDs is allocated during this time.

Telemetry monitor features are enumerated during first resctrl mount and may support a different number of RMIDs compared to L3 monitor features.

Delay allocation and initialization of rmid_ptrs[] until first mount. Since the number of RMIDs cannot change on later mounts, keep the same set of rmid_ptrs[] until resctrl_exit(). This is required because the limbo handler keeps running after resctrl is unmounted and may likely need to access rmid_ptrs[] as it keeps tracking busy RMIDs after unmount.

Rename routines to match what they now do:
dom_data_init() -> setup_rmid_lru_list()
dom_data_exit() -> free_rmid_lru_list()

---
 fs/resctrl/internal.h |  4 ++++
 fs/resctrl/monitor.c  | 54 ++++++++++++++++++++-----------------------
 fs/resctrl/rdtgroup.c |  5 ++++
 3 files changed, 34 insertions(+), 29 deletions(-)
fs/resctrl/internal.h +4 -0
index 46fd648a2961..0dd89d3fa31a 100644
@@ -369,6 +369,10 @@ int closids_supported(void);
 
 void closid_free(int closid);
 
+int setup_rmid_lru_list(void);
+
+void free_rmid_lru_list(void);
+
 int alloc_rmid(u32 closid);
 
 void free_rmid(u32 closid, u32 rmid);
fs/resctrl/monitor.c +25 -29

100 lines not on this page · raw message

fs/resctrl/rdtgroup.c +5 -0
index c94536382316..c1c02b5bf5d7 100644
@@ -2734,6 +2734,10 @@ static int rdt_get_tree(struct fs_context *fc)
 		goto out;
 	}
 
+	ret = setup_rmid_lru_list();
+	if (ret)
+		goto out;
+
 	ret = rdtgroup_setup_root(ctx);
 	if (ret)
 		goto out;
@@ -4596,4 +4600,5 @@ void resctrl_exit(void)
 	 */
 
 	resctrl_l3_mon_resource_exit();
+	free_rmid_lru_list();
 }
Signature · 1 line
2.51.0
maintainer
#55 lore raw

Re: [PATCH v12 27/31] fs/resctrl: Move RMID initialization to first mount

Hi Tony,

Quoted 4 lines · Tony Luck

L3 monitor features are enumerated during resctrl initialization and rmid_ptrs[] that tracks all RMIDs and depends on the number of supported RMIDs is allocated during this time.

Quoted 8 lines

Telemetry monitor features are enumerated during first resctrl mount and may support a different number of RMIDs compared to L3 monitor features.

Delay allocation and initialization of rmid_ptrs[] until first mount. Since the number of RMIDs cannot change on later mounts, keep the same set of rmid_ptrs[] until resctrl_exit(). This is required because the limbo handler keeps running after resctrl is unmounted and may likely need to access rmid_ptrs[] as it keeps tracking busy RMIDs after unmount.

"may likely need" -> "needs"

Quoted 7 lines

Rename routines to match what they now do:
dom_data_init() -> setup_rmid_lru_list()
dom_data_exit() -> free_rmid_lru_list()

---

| Reviewed-by: Reinette Chatre <[email protected]>

Reinette

author maintainer
#56 lore raw

PATCH v12 28/31 x86/resctrl: Enable RDT_RESOURCE_PERF_PKG

In reply to Tony Luck

Since telemetry events are enumerated on resctrl mount the RDT_RESOURCE_PERF_PKG resource is not considered "monitoring capable" during early resctrl initialization. This means that the domain list for RDT_RESOURCE_PERF_PKG is not built when the CPU hot plug notifiers are registered and run for the first time right after resctrl initialization.

Mark the RDT_RESOURCE_PERF_PKG as "monitoring capable" upon successful telemetry event enumeration to ensure future CPU hotplug events include this resource and initialize its domain list for CPUs that are already online.

---
 arch/x86/kernel/cpu/resctrl/core.c      | 16 +++++++++++++++-
 arch/x86/kernel/cpu/resctrl/intel_aet.c |  2 ++
 2 files changed, 17 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/core.c +15 -1
index 955a1904ba2a..2d8c20819ec3 100644
@@ -761,14 +761,28 @@ static int resctrl_arch_offline_cpu(unsigned int cpu)
 
 void resctrl_arch_pre_mount(void)
 {
+	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG].r_resctrl;
 	static atomic_t only_once = ATOMIC_INIT(0);
-	int old = 0;
+	int cpu, old = 0;
 
 	if (!atomic_try_cmpxchg(&only_once, &old, 1))
 		return;
 
 	if (!intel_aet_get_events())
 		return;
+
+	/*
+	 * Late discovery of telemetry events means the domains for the
+	 * resource were not built. Do that now.
+	 */
+	cpus_read_lock();
+	mutex_lock(&domain_list_lock);
+	r->mon_capable = true;
+	rdt_mon_capable = true;
+	for_each_online_cpu(cpu)
+		domain_add_cpu_mon(cpu, r);
+	mutex_unlock(&domain_list_lock);
+	cpus_read_unlock();
 }
 
 enum {
arch/x86/kernel/cpu/resctrl/intel_aet.c +2 -0
index 03d159afcfe1..bc0a4d84a4e0 100644
@@ -225,6 +225,8 @@ static bool enable_events(struct event_group *e, struct pmt_feature_group *p)
 		resctrl_enable_mon_event(e->evts[j].id, true,
 					 e->evts[j].bin_bits, &e->evts[j]);
 
+	pr_info("%s %s monitoring detected\n", r->name, e->name);
+
 	return true;
 }
 
Signature · 1 line
2.51.0
maintainer
#57 lore raw

Re: [PATCH v12 28/31] x86/resctrl: Enable RDT_RESOURCE_PERF_PKG

Hi Tony,

arch/x86/kernel/cpu/resctrl/intel_aet.c static bool enable_events(struct event_group *e, struct pmt_feature_group *p) · line 232

Quoted 13 lines · Tony Luck
arch/x86/kernel/cpu/resctrl/intel_aet.c +2 -0
index 03d159afcfe1..bc0a4d84a4e0 100644
@@ -225,6 +225,8 @@ static bool enable_events(struct event_group *e, struct pmt_feature_group *p)
 		resctrl_enable_mon_event(e->evts[j].id, true,
 					 e->evts[j].bin_bits, &e->evts[j]);
 
+	pr_info("%s %s monitoring detected\n", r->name, e->name);
+
 	return true;
 }
 

Is this a stray hunk? It seems unrelated to what the changelog describes.

reinette

author maintainer
#58 lore raw

PATCH v12 29/31 fs/resctrl: Provide interface to create architecture specific debugfs area

In reply to Tony Luck

All files below /sys/fs/resctrl are considered user ABI.

This leaves no place for architectures to provide additional interfaces.

Add resctrl_debugfs_mon_info_arch_mkdir() which creates a directory in the debugfs file system for a monitoring resource. Naming follows the layout of the main resctrl hierarchy:

	/sys/kernel/debug/resctrl/info/{resource}_MON/{arch}

The {arch} last level directory name matches the output of the user level "uname -m" command.

Architecture code may use this directory for debug information, or for minor tuning of features. It must not be used for basic feature enabling as debugfs may not be configured/mounted on production systems.

---
 include/linux/resctrl.h | 10 ++++++++++
 fs/resctrl/rdtgroup.c   | 29 +++++++++++++++++++++++++++++
 2 files changed, 39 insertions(+)
include/linux/resctrl.h +10 -0
index c7b5e56d25bb..d4be0f54c7e8 100644
@@ -678,6 +678,16 @@ void resctrl_arch_reset_cntr(struct rdt_resource *r, struct rdt_l3_mon_domain *d
 extern unsigned int resctrl_rmid_realloc_threshold;
 extern unsigned int resctrl_rmid_realloc_limit;
 
+/**
+ * resctrl_debugfs_mon_info_arch_mkdir() - Create a debugfs info directory.
+ *					   Removed by resctrl_exit().
+ * @r:	Resource (must be mon_capable).
+ *
+ * Return: NULL if resource is not monitoring capable,
+ * dentry pointer on success, or ERR_PTR(-ERROR) on failure.
+ */
+struct dentry *resctrl_debugfs_mon_info_arch_mkdir(struct rdt_resource *r);
+
 int resctrl_init(void);
 void resctrl_exit(void);
 
fs/resctrl/rdtgroup.c +29 -0
index c1c02b5bf5d7..728f4936e5d5 100644
@@ -24,6 +24,7 @@
 #include <linux/sched/task.h>
 #include <linux/slab.h>
 #include <linux/user_namespace.h>
+#include <linux/utsname.h>
 
 #include <uapi/linux/magic.h>
 
@@ -75,6 +76,8 @@ static void rdtgroup_destroy_root(void);
 
 struct dentry *debugfs_resctrl;
 
+static struct dentry *debugfs_resctrl_info;
+
 /*
  * Memory bandwidth monitoring event to use for the default CTRL_MON group
  * and each new CTRL_MON group created by the user.  Only relevant when
@@ -4541,6 +4544,31 @@ int resctrl_init(void)
 	return ret;
 }
 
+/*
+ * Create /sys/kernel/debug/resctrl/info/{r->name}_MON/{arch} directory
+ * by request for architecture to use for debugging or minor tuning.
+ * Basic functionality of features must not be controlled by files
+ * added to this directory as debugfs may not be configured/mounted
+ * on production systems.
+ */
+struct dentry *resctrl_debugfs_mon_info_arch_mkdir(struct rdt_resource *r)
+{
+	struct dentry *moninfodir;
+	char name[32];
+
+	if (!r->mon_capable)
+		return NULL;
+
+	if (!debugfs_resctrl_info)
+		debugfs_resctrl_info = debugfs_create_dir("info", debugfs_resctrl);
+
+	sprintf(name, "%s_MON", r->name);
+
+	moninfodir = debugfs_create_dir(name, debugfs_resctrl_info);
+
+	return debugfs_create_dir(utsname()->machine, moninfodir);
+}
+
 static bool resctrl_online_domains_exist(void)
 {
 	struct rdt_resource *r;
@@ -4592,6 +4620,7 @@ void resctrl_exit(void)
 
 	debugfs_remove_recursive(debugfs_resctrl);
 	debugfs_resctrl = NULL;
+	debugfs_resctrl_info = NULL;
 	unregister_filesystem(&rdt_fs_type);
 
 	/*
Signature · 1 line
2.51.0
maintainer Reviewed-by on v12 29/31
#59 lore raw

Re: [PATCH v12 29/31] fs/resctrl: Provide interface to create architecture specific debugfs area

Hi Tony,

Quoted 14 lines · Tony Luck

All files below /sys/fs/resctrl are considered user ABI.

This leaves no place for architectures to provide additional interfaces.

Add resctrl_debugfs_mon_info_arch_mkdir() which creates a directory in the debugfs file system for a monitoring resource. Naming follows the layout of the main resctrl hierarchy:

	/sys/kernel/debug/resctrl/info/{resource}_MON/{arch}

The {arch} last level directory name matches the output of the user level "uname -m" command.

Quoted 8 lines

Architecture code may use this directory for debug information, or for minor tuning of features. It must not be used for basic feature enabling as debugfs may not be configured/mounted on production systems.

---

Reinette

author maintainer
#60 lore raw

PATCH v12 30/31 x86/resctrl: Add debugfs files to show telemetry aggregator status

In reply to Tony Luck

Each telemetry aggregator provides three status registers at the top end of MMIO space after all the per-RMID per-event counters:

data_loss_count: This counts the number of times that this aggregator failed to accumulate a counter value supplied by a CPU core.

data_loss_timestamp: This is a "timestamp" from a free running 25MHz uncore timer indicating when the most recent data loss occurred.

last_update_timestamp: Another 25MHz timestamp indicating when the most recent counter update was successfully applied.

Create files in /sys/kernel/debug/resctrl/info/PERF_PKG_MON/x86_64/ to display the value of each of these status registers for each aggregator in each enabled event group. The prefix for each file name describes the type of aggregator, which package it is located on, and an opaque instance number to provide a unique file name when there are multiple aggregators on a package.

The suffix is one of the three strings listed above. An example name is:

energy_pkg0_agg2_data_loss_count

These files are removed along with all other debugfs entries by the call to debugfs_remove_recursive() in resctrl_exit().

---
 arch/x86/kernel/cpu/resctrl/internal.h  |  2 +
 arch/x86/kernel/cpu/resctrl/core.c      |  2 +
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 62 +++++++++++++++++++++++++
 3 files changed, 66 insertions(+)
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
index cea76f88422c..8d4bdae735e4 100644
@@ -236,6 +236,7 @@ int intel_aet_read_event(int domid, u32 rmid, enum resctrl_event_id evtid,
 			 void *arch_priv, u64 *val);
 void intel_aet_mon_domain_setup(int cpu, int id, struct rdt_resource *r,
 				struct list_head *add_pos);
+void intel_aet_add_debugfs(void);
 #else
 static inline bool intel_aet_get_events(void) { return false; }
 static inline void __exit intel_aet_exit(void) { }
@@ -247,6 +248,7 @@ static inline int intel_aet_read_event(int domid, u32 rmid, enum resctrl_event_i
 
 static inline void intel_aet_mon_domain_setup(int cpu, int id, struct rdt_resource *r,
 					      struct list_head *add_pos) { }
+static inline void intel_aet_add_debugfs(void) { }
 #endif
 
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/core.c +2 -0
index 2d8c20819ec3..94ca7902420b 100644
@@ -783,6 +783,8 @@ void resctrl_arch_pre_mount(void)
 		domain_add_cpu_mon(cpu, r);
 	mutex_unlock(&domain_list_lock);
 	cpus_read_unlock();
+
+	intel_aet_add_debugfs();
 }
 
 enum {
arch/x86/kernel/cpu/resctrl/intel_aet.c +62 -0

85 lines not on this page · raw message

Signature · 1 line
2.51.0
maintainer
#61 lore raw

Re: [PATCH v12 30/31] x86/resctrl: Add debugfs files to show telemetry aggregator status

Hi Tony,

Quoted 21 lines · Tony Luck

Each telemetry aggregator provides three status registers at the top end of MMIO space after all the per-RMID per-event counters:

data_loss_count: This counts the number of times that this aggregator failed to accumulate a counter value supplied by a CPU core.

data_loss_timestamp: This is a "timestamp" from a free running 25MHz uncore timer indicating when the most recent data loss occurred.

last_update_timestamp: Another 25MHz timestamp indicating when the most recent counter update was successfully applied.

Create files in /sys/kernel/debug/resctrl/info/PERF_PKG_MON/x86_64/ to display the value of each of these status registers for each aggregator in each enabled event group. The prefix for each file name describes the type of aggregator, which package it is located on, and an opaque instance number to provide a unique file name when there are multiple aggregators on a package.

The suffix is one of the three strings listed above. An example name is:

Quoted 7 lines

energy_pkg0_agg2_data_loss_count

These files are removed along with all other debugfs entries by the call to debugfs_remove_recursive() in resctrl_exit().

---

(nit: please fix that style issue reported by checkpatch.pl for intel_aet_add_debugfs())

| Reviewed-by: Reinette Chatre <[email protected]>

Reinette

author maintainer
#62 lore raw

PATCH v12 31/31 x86,fs/resctrl: Update documentation for telemetry events

In reply to Tony Luck

Update resctrl filesystem documentation with the details about the resctrl files that support telemetry events.

---
 Documentation/filesystems/resctrl.rst | 102 +++++++++++++++++++++++---
 1 file changed, 90 insertions(+), 12 deletions(-)
Documentation/filesystems/resctrl.rst +90 -12
index b7f35b07876a..5e4a5d4bbe9b 100644
@@ -168,13 +168,12 @@ with respect to allocation:
 			bandwidth percentages are directly applied to
 			the threads running on the core
 
-If RDT monitoring is available there will be an "L3_MON" directory
+If L3 monitoring is available there will be an "L3_MON" directory
 with the following files:
 
 "num_rmids":
-		The number of RMIDs available. This is the
-		upper bound for how many "CTRL_MON" + "MON"
-		groups can be created.
+		The number of RMIDs supported by hardware for
+		L3 monitoring events.
 
 "mon_features":
 		Lists the monitoring events if
@@ -400,6 +399,24 @@ with the following files:
 		bytes) at which a previously used LLC_occupancy
 		counter can be considered for re-use.
 
+If telemetry monitoring is available there will be an "PERF_PKG_MON" directory
+with the following files:
+
+"num_rmids":
+		The number of RMIDs for telemetry monitoring events. By default,
+		resctrl will not enable telemetry events of a particular type
+		("perf" or "energy") if the number of RMIDs supported is lower
+		than the number of RMIDs supported by the system. The user can
+		force-enable each type of telemetry events with the "rdt=" boot
+		command line option, but this will reduce the number of "MON"
+		groups that can be created.
+
+"mon_features":
+		Lists the telemetry monitoring events that are enabled on this system.
+
+The upper bound for how many "CTRL_MON" + "MON" can be created
+is the smaller of the L3_MON and PERF_PKG_MON "num_rmids" values.
+
 Finally, in the top level of the "info" directory there is a file
 named "last_cmd_status". This is reset with every "command" issued
 via the file system (making new directories or writing to any of the
@@ -505,15 +522,40 @@ When control is enabled all CTRL_MON groups will also contain:
 When monitoring is enabled all MON groups will also contain:
 
 "mon_data":
-	This contains a set of files organized by L3 domain and by
-	RDT event. E.g. on a system with two L3 domains there will
-	be subdirectories "mon_L3_00" and "mon_L3_01".	Each of these
-	directories have one file per event (e.g. "llc_occupancy",
-	"mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
-	files provide a read out of the current value of the event for
-	all tasks in the group. In CTRL_MON groups these files provide
-	the sum for all tasks in the CTRL_MON group and all tasks in
+	This contains directories for each monitor domain.
+
+	If L3 monitoring is enabled, there will be a "mon_L3_XX" directory for
+	each instance of an L3 cache. Each directory contains files for the enabled
+	L3 events (e.g. "llc_occupancy", "mbm_total_bytes", and "mbm_local_bytes").
+
+	If telemetry monitoring is enabled, there will be a "mon_PERF_PKG_YY"
+	directory for each physical processor package. Each directory contains
+	files for the enabled telemetry events (e.g. "core_energy". "activity",
+	"uops_retired", etc.)
+
+	The info/`*`/mon_features files provide the full list of enabled
+	event/file names.
+
+	"core energy" reports a floating point number for the energy (in Joules)
+	consumed by cores (registers, arithmetic units, TLB and L1/L2 caches)
+	during execution of instructions summed across all logical CPUs on a
+	package for the current RMID.
+
+	"activity" also reports a floating point value (in Farads).  This provides
+	an estimate of work done independent of the frequency that the CPUs used
+	for execution.
+
+	Note that "core energy" and "activity" only measure energy/activity in the
+	"core" of the CPU (arithmetic units, TLB, L1 and L2 caches, etc.). They
+	do not include L3 cache, memory, I/O devices etc.
+
+	All other events report decimal integer values.
+
+	In a MON group these files provide a read out of the current value of
+	the event for all tasks in the group. In CTRL_MON groups these files
+	provide the sum for all tasks in the CTRL_MON group and all tasks in
 	MON groups. Please see example section for more details on usage.
+
 	On systems with Sub-NUMA Cluster (SNC) enabled there are extra
 	directories for each node (located within the "mon_L3_XX" directory
 	for the L3 cache they occupy). These are named "mon_sub_L3_YY"
@@ -1506,6 +1548,42 @@ Example with C::
     resctrl_release_lock(fd);
   }
 
+Debugfs
+=======
+In addition to the use of debugfs for tracing of pseudo-locking performance,
+architecture code may create debugfs directories associated with monitoring
+features for a specific resource.
+
+The full pathname for these is in the form:
+
+    /sys/kernel/debug/resctrl/info/{resource_name}_MON/{arch}/
+
+The presence, names, and format of these files may vary between architectures
+even if the same resource is present.
+
+PERF_PKG_MON/x86_64
+-------------------
+Three files are present per telemetry aggregator instance that show status.
+The prefix of each file name describes the type ("energy" or "perf") which
+processor package it belongs to, and the instance number of the aggregator.
+For example: "energy_pkg1_agg2".
+
+The suffix describes which data is reported in the file and
+is one of:
+
+data_loss_count:
+	This counts the number of times that this aggregator
+	failed to accumulate a counter value supplied by a CPU.
+
+data_loss_timestamp:
+	This is a "timestamp" from a free running 25MHz uncore
+	timer indicating when the most recent data loss occurred.
+
+last_update_timestamp:
+	Another 25MHz timestamp indicating when the
+	most recent counter update was successfully applied.
+
+
 Examples for RDT Monitoring along with allocation usage
 =======================================================
 Reading monitored data
Signature · 1 line
2.51.0
maintainer
#63 lore raw

Re: [PATCH v12 31/31] x86,fs/resctrl: Update documentation for telemetry events

Hi Tony,

Quoted 31 lines · Tony Luck

Update resctrl filesystem documentation with the details about the resctrl files that support telemetry events.

---
 Documentation/filesystems/resctrl.rst | 102 +++++++++++++++++++++++---
 1 file changed, 90 insertions(+), 12 deletions(-)
Documentation/filesystems/resctrl.rst +3 -4
index b7f35b07876a..5e4a5d4bbe9b 100644
@@ -168,13 +168,12 @@ with respect to allocation:
 			bandwidth percentages are directly applied to
 			the threads running on the core
 
-If RDT monitoring is available there will be an "L3_MON" directory
+If L3 monitoring is available there will be an "L3_MON" directory
 with the following files:
 
 "num_rmids":
-		The number of RMIDs available. This is the
-		upper bound for how many "CTRL_MON" + "MON"
-		groups can be created.
+		The number of RMIDs supported by hardware for
+		L3 monitoring events.
 
 "mon_features":
 		Lists the monitoring events if
@@ -400,6 +399,24 @@ with the following files:
 		bytes) at which a previously used LLC_occupancy

Documentation/filesystems/resctrl.rst with the following files: · line 409

Quoted 10 lines
Documentation/filesystems/resctrl.rst +8 -0
 		counter can be considered for re-use.
 
+If telemetry monitoring is available there will be an "PERF_PKG_MON" directory
+with the following files:
+
+"num_rmids":
+		The number of RMIDs for telemetry monitoring events. By default,
+		resctrl will not enable telemetry events of a particular type
+		("perf" or "energy") if the number of RMIDs supported is lower
+		than the number of RMIDs supported by the system. The user can

I do not know how to interpret: "if the number of RMIDs supported is lower than the number of RMIDs supported by the system"

Documentation/filesystems/resctrl.rst with the following files: · line 411

Quoted 2 lines
Documentation/filesystems/resctrl.rst +2 -0
+		force-enable each type of telemetry events with the "rdt=" boot
+		command line option, but this will reduce the number of "MON"

It is not obvious to me that number of MON groups will always be reduced ... for example, this will also depend on the L3 RMIDs. Perhaps it should be "this may reduce" to not create confusion.

Quoted 31 lines
Documentation/filesystems/resctrl.rst +16 -8
+		groups that can be created.
+
+"mon_features":
+		Lists the telemetry monitoring events that are enabled on this system.
+
+The upper bound for how many "CTRL_MON" + "MON" can be created
+is the smaller of the L3_MON and PERF_PKG_MON "num_rmids" values.
+
 Finally, in the top level of the "info" directory there is a file
 named "last_cmd_status". This is reset with every "command" issued
 via the file system (making new directories or writing to any of the
@@ -505,15 +522,40 @@ When control is enabled all CTRL_MON groups will also contain:
 When monitoring is enabled all MON groups will also contain:
 
 "mon_data":
-	This contains a set of files organized by L3 domain and by
-	RDT event. E.g. on a system with two L3 domains there will
-	be subdirectories "mon_L3_00" and "mon_L3_01".	Each of these
-	directories have one file per event (e.g. "llc_occupancy",
-	"mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
-	files provide a read out of the current value of the event for
-	all tasks in the group. In CTRL_MON groups these files provide
-	the sum for all tasks in the CTRL_MON group and all tasks in
+	This contains directories for each monitor domain.
+
+	If L3 monitoring is enabled, there will be a "mon_L3_XX" directory for
+	each instance of an L3 cache. Each directory contains files for the enabled
+	L3 events (e.g. "llc_occupancy", "mbm_total_bytes", and "mbm_local_bytes").
+
+	If telemetry monitoring is enabled, there will be a "mon_PERF_PKG_YY"
+	directory for each physical processor package. Each directory contains

Documentation/filesystems/resctrl.rst When control is enabled all CTRL_MON groups will also contain: · line 542

Quoted 10 lines
Documentation/filesystems/resctrl.rst +10 -0
+	files for the enabled telemetry events (e.g. "core_energy". "activity",
+	"uops_retired", etc.)
+
+	The info/`*`/mon_features files provide the full list of enabled
+	event/file names.
+
+	"core energy" reports a floating point number for the energy (in Joules)
+	consumed by cores (registers, arithmetic units, TLB and L1/L2 caches)
+	during execution of instructions summed across all logical CPUs on a
+	package for the current RMID.

"the current RMID" -> "for the current monitoring group"?

Quoted 20 lines
Documentation/filesystems/resctrl.rst +15 -0
+
+	"activity" also reports a floating point value (in Farads).  This provides
+	an estimate of work done independent of the frequency that the CPUs used
+	for execution.
+
+	Note that "core energy" and "activity" only measure energy/activity in the
+	"core" of the CPU (arithmetic units, TLB, L1 and L2 caches, etc.). They
+	do not include L3 cache, memory, I/O devices etc.
+
+	All other events report decimal integer values.
+
+	In a MON group these files provide a read out of the current value of
+	the event for all tasks in the group. In CTRL_MON groups these files
+	provide the sum for all tasks in the CTRL_MON group and all tasks in
 	MON groups. Please see example section for more details on usage.
+
 	On systems with Sub-NUMA Cluster (SNC) enabled there are extra
 	directories for each node (located within the "mon_L3_XX" directory
 	for the L3 cache they occupy). These are named "mon_sub_L3_YY"

Documentation/filesystems/resctrl.rst Example with C:: · line 1550

Quoted 3 lines
Documentation/filesystems/resctrl.rst +0 -0
@@ -1506,6 +1548,42 @@ Example with C::
     resctrl_release_lock(fd);
   }
 

Reinette