Skip to contentSkip to conversation

x86/resctrl telemetry monitoring

RFC PATCH Superseded Tony Luck 1 participant 18 messages Mar 3, 2025

Subsystems (5)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Tony Luck

author maintainer
#0 lore raw

cover x86/resctrl telemetry monitoring

Preamble

I'm trying here to provide a picture of the capabilities of telemetry monitoring with a possible resctrl implementation. I'm looking for high level comments on my general approach (especially as it may interact with monitoring features that other architectures may want to add in the future.

The first patch in the series just provides a fake copy of the enumeration interface that should show up in the OOBMSM driver in the near future. It allows building, and running, of this series on Intel (and perhaps AMD) systems that don't have h/w support.

Background

Telemetry features are being implemented in conjunction with the IA32_PQR_ASSOC.RMID value on each logical CPU. This is used to send counts for various events to a collector in a nearby OOMMSM device to be accumulated with counts for each <RMID, event> pair received from other CPUs. Cores send event counts when the RMID value changes, or after each 2ms elapsed time.

Each OOBMSM device may implement multiple event collectors with each servicing a subset of the logical CPUs on a package. In the initial hardware implementation, there are two categories of events:

1) Energy - Two counters
core_energy: This is an estimate of Joules consumed by each core. It is calculated based on the types of instructions executed, not from a power meter. This counter is useful to understand how much energy a workload is consuming.

activity: This measures "accumulated dynamic capacitance". Users who want to optimize energy consumption for a workload may use this rather than core_energy because it provides consistent results independent of any frequency or voltage changes that may occur during the runtime of the application (e.g. entry/exit from turbo mode).

2) Performance - Seven counters
These are similar events to those available via the Linux "perf" tool, but collected in a way with mush lower overhead (no need to collect data on every context switch).

stalls_llc_hit - Counts the total number of unhalted core clock cycles when the core is stalled due to a demand load miss which hit in the LLC

c1_res - Counts the total C1 residency across all cores. The underlying counter increments on 100MHz clock ticks

unhalted_core_cycles - Counts the total number of unhalted core clock cycles

stalls_llc_miss - Counts the total number of unhalted core clock cycles when the core is stalled due to a demand load miss which missed all the local caches

c6_res - Counts the total C6 residency. The underlying counter increments on crystal clock (25MHz) ticks

unhalted_ref_cycles - Counts the total number of unhalted reference clock (TSC) cycles

uops_retired - Counts the total number of uops retired

Enumeration

The only CPUID based enumeration for this feature is the legacy CPUID(eax=7,ecx=0).ebx{12} that indicates the presence of the IA32_PQR_ASSOC MSR and the RMID field within it.

The OOBMSM driver discovers which features are present via PCIe VSEC capabilities. Each feature is tagged with a unique identifier. These identifiers indicate which XML description file from
https://github.com/intel/Intel-PMT describes which event counters are available and their layout within the MMIO BAR space of the OOBMSM device.

Resctrl User Interface

Because there may be multiple OOBMSM collection agents per processor package, resctrl accumulates event counts from all agents on a package and presents a single value to users. This will provide a consistent user interface on future platforms that vary the number of collectors, or the mappings from logical CPUs to collectors.

Users will see the legacy monitoring files in the "L3" directories and the telemetry files in "PKG" directories (with each file providing the aggregated value from all OOBMSM collectors on that package).

$ tree /sys/fs/resctrl/mon_data/
/sys/fs/resctrl/mon_data/
├── mon_L3_00
│   ├── llc_occupancy
│   ├── mbm_local_bytes
│   └── mbm_total_bytes
├── mon_L3_01
│   ├── llc_occupancy
│   ├── mbm_local_bytes
│   └── mbm_total_bytes
├── mon_PKG_00
│   ├── activity
│   ├── c1_res
│   ├── c6_res
│   ├── core_energy
│   ├── stalls_llc_hit
│   ├── stalls_llc_miss
│   ├── unhalted_core_cycles
│   ├── unhalted_ref_cycles
│   └── uops_retired
└── mon_PKG_01
    ├── activity
    ├── c1_res
    ├── c6_res
    ├── core_energy
    ├── stalls_llc_hit
    ├── stalls_llc_miss
    ├── unhalted_core_cycles
    ├── unhalted_ref_cycles
    └── uops_retired

Resctrl Implementation

The OOBMSM driver exposes a function "intel_pmt_get_regions_by_feature()" that returns an array of structures describing the per-RMID groups it found from the VSEC enumeration. Linux looks at the unique identifiers for each group and enables resctrl for all groups with known unique identifiers.

The memory map for the counters for each <RMID, event> pair is described by the XML file. This is too unwieldy to use in the Linux kernel, so a simplified representation is built into the resctrl code. Note that the counters are in MMIO space instead of accessed using the IA32_QM_EVTSEL and IA32_QM_CTR MSRs. This means there is no need for cross-processor calls to read counters from a CPU in a specific domain. The counters can be read from any CPU.

High level description of code changes:

1) New scope RESCTRL_PACKAGE
2) New struct rdt_resource RDT_RESOURCE_INTEL_PMT
3) Refactor monitor code paths to split existing L3 paths from new ones. In some cases this ends up with:

        switch (r->rid) {
        case RDT_RESOURCE_L3:
                helper for L3
                break;
        case RDT_RESOURCE_INTEL_PMT:
                helper for PKG
                break;
        }

4) New source code file "intel_pmt.c" for the code to enumerate, configure, and report event counts.

With only one platform providing this feature, it's tricky to tell exactly where it is going to go. I've made the event definitions platform specific (based on the unique ID from the VSEC enumeration). It seems possible/likely that the list of events may change from generation to generation.

I've picked names for events based on the descriptions in the XML file.

Tony Luck (17):
x86/rectrl: Fake OOBMSM interface
x86/resctrl: Move L3 initialization out of domain_add_cpu_mon() x86/resctrl: Refactor domain_remove_cpu_mon() ready for new domain
types
x86/resctrl: Split L3 specific code out of rdtgroup_mondata_show() x86/resctrl: Change generic monitor functions to use struct
rdt_domain_hdr
x86/resctrl: Add and initialize rdt_resource for package scope core
monitor
x86/resctrl: Prepare for resource specific event ids x86/resctrl: Add initialization hook for Intel PMT events x86/resctrl: Add Intel PMT domain specific code x86/resctrl: Add detailed descriptions for Clearwater Forest events x86/resctrl: Allocate per-package structures for known events x86/resctrl: Add list of known events to RDT_RESOURCE_INTEL_PMT x86/resctrl: Build lookup table for package events x86/resctrl: Add code in rdtgroup_mondata_show() for core monitor
events
x86/resctrl: Add status files to info/PKG_MON x86/resctrl: Enable package event monitoring x86/resctrl: Update Documentation for package events

 Documentation/arch/x86/resctrl.rst            |  25 +-
 include/linux/resctrl.h                       |  11 +-
 .../cpu/resctrl/fake_intel_pmt_features.h     |  72 +++
 arch/x86/kernel/cpu/resctrl/internal.h        |  51 +-
 arch/x86/kernel/cpu/resctrl/core.c            | 105 ++--
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c     |  82 +--
 .../cpu/resctrl/fake_intel_pmt_features.c     |  64 +++
 arch/x86/kernel/cpu/resctrl/intel_pmt.c       | 486 ++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/rdtgroup.c        |  87 +++-
 arch/x86/Kconfig                              |   1 +
 arch/x86/kernel/cpu/resctrl/Makefile          |   2 +
 drivers/platform/x86/intel/pmt/Kconfig        |   3 +
 12 files changed, 882 insertions(+), 107 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/resctrl/fake_intel_pmt_features.h
 create mode 100644 arch/x86/kernel/cpu/resctrl/fake_intel_pmt_features.c
 create mode 100644 arch/x86/kernel/cpu/resctrl/intel_pmt.c

base-commit: 7eb172143d5508b4da468ed59ee857c6e5e01da6

Signature · 1 line
2.48.1
author maintainer
#1 lore raw

RFC PATCH 1/17 x86/rectrl: Fake OOBMSM interface

Real version is coming soon ... this is here so the remaining parts will build (and run ... assuming a 2 socket system that supports RDT monitoring ... only missing part is that the event counters just report fixed values).

Real version of this would just add the INTEL_PMT_RESCTRL Kconfig option with dependency checks on
INTEL_VSEC=y && INTEL_PMT_TELEMETRY=y && INTEL_PMT_DISCOVERY=y

Just for RFC discussion.

---
 .../cpu/resctrl/fake_intel_pmt_features.h     | 72 +++++++++++++++++++
 .../cpu/resctrl/fake_intel_pmt_features.c     | 64 +++++++++++++++++
 arch/x86/Kconfig                              |  1 +
 arch/x86/kernel/cpu/resctrl/Makefile          |  1 +
 drivers/platform/x86/intel/pmt/Kconfig        |  3 +
 5 files changed, 141 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/resctrl/fake_intel_pmt_features.h
 create mode 100644 arch/x86/kernel/cpu/resctrl/fake_intel_pmt_features.c
arch/x86/kernel/cpu/resctrl/fake_intel_pmt_features.h new file +72 -0
new file mode 100644
index 000000000000..52fdd04e5094
@@ -0,0 +1,72 @@
+
+/* Bits stolen from OOBMSM VSEC discovery code */
+
+enum pmt_feature_id {
+	FEATURE_INVALID			= 0x0,
+	FEATURE_PER_CORE_PERF_TELEM	= 0x1,
+	FEATURE_PER_CORE_ENV_TELEM	= 0x2,
+	FEATURE_PER_RMID_PERF_TELEM	= 0x3,
+	FEATURE_ACCEL_TELEM		= 0x4,
+	FEATURE_UNCORE_TELEM		= 0x5,
+	FEATURE_CRASH_LOG		= 0x6,
+	FEATURE_PETE_LOG		= 0x7,
+	FEATURE_TPMI_CTRL		= 0x8,
+	FEATURE_RESERVED		= 0x9,
+	FEATURE_TRACING			= 0xA,
+	FEATURE_PER_RMID_ENERGY_TELEM	= 0xB,
+	FEATURE_MAX			= 0xB,
+};
+
+/**
+ * struct oobmsm_plat_info - Platform information for a device instance
+ * @cdie_mask:       Mask of all compute dies in the partition
+ * @package_id:      CPU Package id
+ * @partition:       Package partition id when multiple VSEC PCI devices per package
+ * @segment:         PCI segment ID
+ * @bus_number:      PCI bus number
+ * @device_number:   PCI device number
+ * @function_number: PCI function number
+ *
+ * Structure to store platform data for a OOBMSM device instance.
+ */
+struct oobmsm_plat_info {
+	u16 cdie_mask;
+	u8 package_id;
+	u8 partition;
+	u8 segment;
+	u8 bus_number;
+	u8 device_number;
+	u8 function_number;
+};
+
+enum oobmsm_supplier_type {
+	OOBMSM_SUP_PLAT_INFO,
+	OOBMSM_SUP_DISC_INFO,
+	OOBMSM_SUP_S3M_SIMICS,
+	OOBMSM_SUP_TYPE_MAX
+};
+
+struct oobmsm_mapping_supplier {
+	struct device *supplier_dev[OOBMSM_SUP_TYPE_MAX];
+	struct oobmsm_plat_info plat_info;
+	unsigned long features;
+};
+
+struct telemetry_region {
+	struct oobmsm_plat_info	plat_info;
+	void __iomem		*addr;
+	size_t			size;
+	u32			guid;
+	u32			num_rmids;
+};
+
+struct pmt_feature_group {
+	enum pmt_feature_id	id;
+	int			count;
+	struct kref		kref;
+	struct telemetry_region	regions[];
+};
+
+struct pmt_feature_group *intel_pmt_get_regions_by_feature(enum pmt_feature_id id);
+
+void intel_pmt_put_feature_group(struct pmt_feature_group *feature_group);
arch/x86/kernel/cpu/resctrl/fake_intel_pmt_features.c new file +64 -0

64 lines not on this page · raw message

arch/x86/Kconfig +1 -0
index be2c311f5118..ec6a7b996e09 100644
@@ -505,6 +505,7 @@ config X86_CPU_RESCTRL
 	depends on X86 && (CPU_SUP_INTEL || CPU_SUP_AMD)
 	select KERNFS
 	select PROC_CPU_RESCTRL		if PROC_FS
+	select INTEL_PMT_RESCTRL
 	help
 	  Enable x86 CPU resource control support.
 
arch/x86/kernel/cpu/resctrl/Makefile +1 -0
index 4a06c37b9cf1..72621ea6cd6a 100644
@@ -1,4 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_X86_CPU_RESCTRL)	+= core.o rdtgroup.o monitor.o
 obj-$(CONFIG_X86_CPU_RESCTRL)	+= ctrlmondata.o pseudo_lock.o
+obj-$(CONFIG_INTEL_PMT_RESCTRL)	+= fake_intel_pmt_features.o
 CFLAGS_pseudo_lock.o = -I$(src)
drivers/platform/x86/intel/pmt/Kconfig +3 -0
index e916fc966221..4cbfd8a42252 100644
@@ -38,3 +38,6 @@ config INTEL_PMT_CRASHLOG
 
 	  To compile this driver as a module, choose M here: the module
 	  will be called intel_pmt_crashlog.
+
+config INTEL_PMT_RESCTRL
+	bool
Signature · 1 line
2.48.1
author maintainer
#2 lore raw

RFC PATCH 2/17 x86/resctrl: Move L3 initialization out of domain_add_cpu_mon()

In reply to Tony Luck

To prepare for additional types of monitoring domains, move all the L3 specific initialization into a helper function.

---
 arch/x86/kernel/cpu/resctrl/core.c | 52 +++++++++++++++++-------------
 1 file changed, 30 insertions(+), 22 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +30 -22
index 3d1735ed8d1f..dd694958df5a 100644
@@ -583,33 +583,12 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r)
 	}
 }
 
-static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
+static void setup_l3_mon_domain(int cpu, int id, struct rdt_resource *r, struct list_head *add_pos)
 {
-	int id = get_domain_id_from_scope(cpu, r->mon_scope);
-	struct list_head *add_pos = NULL;
 	struct rdt_hw_mon_domain *hw_dom;
-	struct rdt_domain_hdr *hdr;
 	struct rdt_mon_domain *d;
 	int err;
 
-	lockdep_assert_held(&domain_list_lock);
-
-	if (id < 0) {
-		pr_warn_once("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n",
-			     cpu, r->mon_scope, r->name);
-		return;
-	}
-
-	hdr = rdt_find_domain(&r->mon_domains, id, &add_pos);
-	if (hdr) {
-		if (WARN_ON_ONCE(hdr->type != RESCTRL_MON_DOMAIN))
-			return;
-		d = container_of(hdr, struct rdt_mon_domain, hdr);
-
-		cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
-		return;
-	}
-
 	hw_dom = kzalloc_node(sizeof(*hw_dom), GFP_KERNEL, cpu_to_node(cpu));
 	if (!hw_dom)
 		return;
@@ -642,6 +621,35 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 	}
 }
 
+static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
+{
+	int id = get_domain_id_from_scope(cpu, r->mon_scope);
+	struct list_head *add_pos = NULL;
+	struct rdt_domain_hdr *hdr;
+
+	lockdep_assert_held(&domain_list_lock);
+
+	if (id < 0) {
+		pr_warn_once("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n",
+			     cpu, r->mon_scope, r->name);
+		return;
+	}
+
+	hdr = rdt_find_domain(&r->mon_domains, id, &add_pos);
+	if (hdr) {
+		cpumask_set_cpu(cpu, &hdr->cpu_mask);
+		return;
+	}
+
+	switch (r->rid) {
+	case RDT_RESOURCE_L3:
+		setup_l3_mon_domain(cpu, id, r, add_pos);
+		break;
+	default:
+		WARN_ON_ONCE(1);
+	}
+}
+
 static void domain_add_cpu(int cpu, struct rdt_resource *r)
 {
 	if (r->alloc_capable)
Signature · 1 line
2.48.1
author maintainer
#3 lore raw

RFC PATCH 3/17 x86/resctrl: Refactor domain_remove_cpu_mon() ready for new domain types

In reply to Tony Luck

Different types of domains require different actions when the last CPU in the domain is removed.

Refactor to make it easy to add new actions for new types of domains.

---
 arch/x86/kernel/cpu/resctrl/core.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +8 -8
index dd694958df5a..4974cbb842c1 100644
@@ -726,20 +726,20 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
 		return;
 	}
 
-	if (WARN_ON_ONCE(hdr->type != RESCTRL_MON_DOMAIN))
+	cpumask_clear_cpu(cpu, &hdr->cpu_mask);
+	if (!cpumask_empty(&hdr->cpu_mask))
 		return;
 
-	d = container_of(hdr, struct rdt_mon_domain, hdr);
-	hw_dom = resctrl_to_arch_mon_dom(d);
+	switch (r->rid) {
+	case RDT_RESOURCE_L3:
+		d = container_of(hdr, struct rdt_mon_domain, hdr);
+		hw_dom = resctrl_to_arch_mon_dom(d);
 
-	cpumask_clear_cpu(cpu, &d->hdr.cpu_mask);
-	if (cpumask_empty(&d->hdr.cpu_mask)) {
 		resctrl_offline_mon_domain(r, d);
-		list_del_rcu(&d->hdr.list);
+		list_del_rcu(&hdr->list);
 		synchronize_rcu();
 		mon_domain_free(hw_dom);
-
-		return;
+		break;
 	}
 }
 
Signature · 1 line
2.48.1
author maintainer
#4 lore raw

RFC PATCH 4/17 x86/resctrl: Split L3 specific code out of rdtgroup_mondata_show()

In reply to Tony Luck

New monitor resources require different operations to read counters.

Move the L3 specific code to a helper function.

---
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 79 +++++++++++++----------
 1 file changed, 46 insertions(+), 33 deletions(-)
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +46 -33
index 536351159cc2..515a9bec05cd 100644
@@ -627,31 +627,14 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 	resctrl_arch_mon_ctx_free(r, evtid, rr->arch_mon_ctx);
 }
 
-int rdtgroup_mondata_show(struct seq_file *m, void *arg)
+static int rdtgroup_l3_data_show(struct seq_file *m, struct rdt_resource *r,
+				 int domid, struct rdtgroup *rdtgrp, u32 evtid,
+				 bool sum, struct rmid_read *rr)
 {
-	struct kernfs_open_file *of = m->private;
 	struct rdt_domain_hdr *hdr;
-	struct rmid_read rr = {0};
 	struct rdt_mon_domain *d;
-	u32 resid, evtid, domid;
-	struct rdtgroup *rdtgrp;
-	struct rdt_resource *r;
-	union mon_data_bits md;
-	int ret = 0;
-
-	rdtgrp = rdtgroup_kn_lock_live(of->kn);
-	if (!rdtgrp) {
-		ret = -ENOENT;
-		goto out;
-	}
 
-	md.priv = of->kn->priv;
-	resid = md.u.rid;
-	domid = md.u.domid;
-	evtid = md.u.evtid;
-	r = &rdt_resources_all[resid].r_resctrl;
-
-	if (md.u.sum) {
+	if (sum) {
 		/*
 		 * This file requires summing across all domains that share
 		 * the L3 cache id that was provided in the "domid" field of the
@@ -660,37 +643,67 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 		 */
 		list_for_each_entry(d, &r->mon_domains, hdr.list) {
 			if (d->ci->id == domid) {
-				rr.ci = d->ci;
-				mon_event_read(&rr, r, NULL, rdtgrp,
+				rr->ci = d->ci;
+				mon_event_read(rr, r, NULL, rdtgrp,
 					       &d->ci->shared_cpu_map, evtid, false);
 				goto checkresult;
 			}
 		}
-		ret = -ENOENT;
-		goto out;
+		return -ENOENT;
 	} else {
 		/*
 		 * This file provides data from a single domain. Search
 		 * the resource to find the domain with "domid".
 		 */
 		hdr = rdt_find_domain(&r->mon_domains, domid, NULL);
-		if (!hdr || WARN_ON_ONCE(hdr->type != RESCTRL_MON_DOMAIN)) {
-			ret = -ENOENT;
-			goto out;
-		}
+		if (!hdr || WARN_ON_ONCE(hdr->type != RESCTRL_MON_DOMAIN))
+			return -ENOENT;
 		d = container_of(hdr, struct rdt_mon_domain, hdr);
-		mon_event_read(&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false);
+		mon_event_read(rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false);
 	}
 
 checkresult:
 
-	if (rr.err == -EIO)
+	if (rr->err == -EIO)
 		seq_puts(m, "Error\n");
-	else if (rr.err == -EINVAL)
+	else if (rr->err == -EINVAL)
 		seq_puts(m, "Unavailable\n");
 	else
-		seq_printf(m, "%llu\n", rr.val);
+		seq_printf(m, "%llu\n", rr->val);
 
+	return 0;
+}
+
+int rdtgroup_mondata_show(struct seq_file *m, void *arg)
+{
+	struct kernfs_open_file *of = m->private;
+	struct rmid_read rr = {0};
+	u32 resid, evtid, domid;
+	struct rdtgroup *rdtgrp;
+	struct rdt_resource *r;
+	union mon_data_bits md;
+	int ret = 0;
+
+	rdtgrp = rdtgroup_kn_lock_live(of->kn);
+	if (!rdtgrp) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	md.priv = of->kn->priv;
+	resid = md.u.rid;
+	domid = md.u.domid;
+	evtid = md.u.evtid;
+	r = &rdt_resources_all[resid].r_resctrl;
+
+	switch (r->rid) {
+	case RDT_RESOURCE_L3:
+		ret = rdtgroup_l3_data_show(m, r, domid, rdtgrp, evtid, md.u.sum, &rr);
+		break;
+	default:
+		ret = -EINVAL;
+		break;
+	}
 out:
 	rdtgroup_kn_unlock(of->kn);
 	return ret;
Signature · 1 line
2.48.1
author maintainer
#5 lore raw

RFC PATCH 5/17 x86/resctrl: Change generic monitor functions to use struct rdt_domain_hdr

In reply to Tony Luck

Functions that don't need the internal details of the rdt_mon_domain can operate on just the rdt_domain_hdr.

---
 include/linux/resctrl.h                |  2 +-
 arch/x86/kernel/cpu/resctrl/core.c     |  2 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 60 +++++++++++++++++---------
 3 files changed, 42 insertions(+), 22 deletions(-)
include/linux/resctrl.h +1 -1
index d94abba1c716..a89f0a7e2724 100644
@@ -261,7 +261,7 @@ int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d,
 u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d,
 			    u32 closid, enum resctrl_conf_type type);
 int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d);
-int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d);
+int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr);
 void resctrl_offline_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d);
 void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d);
 void resctrl_online_cpu(unsigned int cpu);
arch/x86/kernel/cpu/resctrl/core.c +1 -1
index 4974cbb842c1..7eddc07d35ee 100644
@@ -613,7 +613,7 @@ static void setup_l3_mon_domain(int cpu, int id, struct rdt_resource *r, struct
 
 	list_add_tail_rcu(&d->hdr.list, add_pos);
 
-	err = resctrl_online_mon_domain(r, d);
+	err = resctrl_online_mon_domain(r, &d->hdr);
 	if (err) {
 		list_del_rcu(&d->hdr.list);
 		synchronize_rcu();
arch/x86/kernel/cpu/resctrl/rdtgroup.c +40 -20

169 lines not on this page · raw message

Signature · 1 line
2.48.1
author maintainer
#6 lore raw

RFC PATCH 6/17 x86/resctrl: Add and initialize rdt_resource for package scope core monitor

In reply to Tony Luck

New resource for monitoring core events reported at package level.

---
 include/linux/resctrl.h                |  1 +
 arch/x86/kernel/cpu/resctrl/internal.h |  2 ++
 arch/x86/kernel/cpu/resctrl/core.c     | 12 ++++++++++++
 3 files changed, 15 insertions(+)
include/linux/resctrl.h +1 -0
index a89f0a7e2724..9b64993a6cc8 100644
@@ -180,6 +180,7 @@ enum resctrl_scope {
 	RESCTRL_L2_CACHE = 2,
 	RESCTRL_L3_CACHE = 3,
 	RESCTRL_L3_NODE,
+	RESCTRL_PACKAGE,
 };
 
 /**
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
index 20c898f09b7e..b8f37170f6e5 100644
@@ -317,6 +317,7 @@ struct rdtgroup {
 #define RFTYPE_RES_CACHE		BIT(8)
 #define RFTYPE_RES_MB			BIT(9)
 #define RFTYPE_DEBUG			BIT(10)
+#define RFTYPE_RES_PKG			BIT(11)
 #define RFTYPE_CTRL_INFO		(RFTYPE_INFO | RFTYPE_CTRL)
 #define RFTYPE_MON_INFO			(RFTYPE_INFO | RFTYPE_MON)
 #define RFTYPE_TOP_INFO			(RFTYPE_INFO | RFTYPE_TOP)
@@ -517,6 +518,7 @@ enum resctrl_res_level {
 	RDT_RESOURCE_L2,
 	RDT_RESOURCE_MBA,
 	RDT_RESOURCE_SMBA,
+	RDT_RESOURCE_INTEL_PMT,
 
 	/* Must be the last */
 	RDT_NUM_RESOURCES,
arch/x86/kernel/cpu/resctrl/core.c +12 -0
index 7eddc07d35ee..75a815383427 100644
@@ -117,6 +117,16 @@ struct rdt_hw_resource rdt_resources_all[] = {
 			.fflags			= RFTYPE_RES_MB,
 		},
 	},
+	[RDT_RESOURCE_INTEL_PMT] =
+	{
+		.r_resctrl = {
+			.rid			= RDT_RESOURCE_INTEL_PMT,
+			.name			= "PKG",
+			.mon_scope		= RESCTRL_PACKAGE,
+			.mon_domains		= mon_domain_init(RDT_RESOURCE_INTEL_PMT),
+			.fflags			= RFTYPE_RES_PKG,
+		},
+	},
 };
 
 u32 resctrl_arch_system_num_rmid_idx(void)
@@ -521,6 +531,8 @@ static int get_domain_id_from_scope(int cpu, enum resctrl_scope scope)
 		return get_cpu_cacheinfo_id(cpu, scope);
 	case RESCTRL_L3_NODE:
 		return cpu_to_node(cpu);
+	case RESCTRL_PACKAGE:
+		return topology_physical_package_id(cpu);
 	default:
 		break;
 	}
Signature · 1 line
2.48.1
author maintainer
#7 lore raw

RFC PATCH 7/17 x86/resctrl: Prepare for resource specific event ids

In reply to Tony Luck

Event ids ought to be specific to each resource. Keep enum resctrl_event_id for event ids in the RDT_RESOURCE_L3. But change the type in generic areas to unsigned int so that new resources can have their own event ids.

---
 arch/x86/kernel/cpu/resctrl/internal.h | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +7 -7
index b8f37170f6e5..c8da6fac4720 100644
@@ -114,13 +114,13 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
 
 /**
  * struct mon_evt - Entry in the event list of a resource
- * @evtid:		event id
+ * @evtid:		per resource event id
  * @name:		name of the event
  * @configurable:	true if the event is configurable
  * @list:		entry in &rdt_resource->evt_list
  */
 struct mon_evt {
-	enum resctrl_event_id	evtid;
+	unsigned int		evtid;
 	char			*name;
 	bool			configurable;
 	struct list_head	list;
@@ -131,7 +131,7 @@ struct mon_evt {
  * @priv:              Used to store monitoring event data in @u
  *                     as kernfs private data.
  * @u.rid:             Resource id associated with the event file.
- * @u.evtid:           Event id associated with the event file.
+ * @u.evtid:           Per resource event id associated with the event file.
  * @u.sum:             Set when event must be summed across multiple
  *                     domains.
  * @u.domid:           When @u.sum is zero this is the domain to which
@@ -143,10 +143,10 @@ struct mon_evt {
 union mon_data_bits {
 	void *priv;
 	struct {
-		unsigned int rid		: 10;
-		enum resctrl_event_id evtid	: 7;
-		unsigned int sum		: 1;
-		unsigned int domid		: 14;
+		unsigned int rid	: 10;
+		unsigned int evtid	: 7;
+		unsigned int sum	: 1;
+		unsigned int domid	: 14;
 	} u;
 };
 
Signature · 1 line
2.48.1
author maintainer
#8 lore raw

RFC PATCH 8/17 x86/resctrl: Add initialization hook for Intel PMT events

In reply to Tony Luck

Call the OOBMSM discovery code to find out if there are any event groups that match unique identifiers understood by resctrl.

Note that initiialzation must happen in two phases because the OOBMSM VSEC discovery process is not complete at resctrl "lateinit()" initialization time. So there is an initial hook that assumes that Intel PMT will exist, called early so that package scoped domain groups are initialized.

At first mount the remainder of initialization is done. If there are no Intel PMT events, the package domain lists are removed.

Events for specific systems to be added by a separate patch.

---
 include/linux/resctrl.h                 |   8 ++
 arch/x86/kernel/cpu/resctrl/internal.h  |   7 +
 arch/x86/kernel/cpu/resctrl/core.c      |   9 +-
 arch/x86/kernel/cpu/resctrl/intel_pmt.c | 169 ++++++++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/rdtgroup.c  |   3 +
 arch/x86/kernel/cpu/resctrl/Makefile    |   1 +
 6 files changed, 194 insertions(+), 3 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/resctrl/intel_pmt.c
include/linux/resctrl.h +8 -0
index 9b64993a6cc8..6e463c65451b 100644
@@ -118,6 +118,14 @@ struct rdt_mon_domain {
 	int				cqm_work_cpu;
 };
 
+/**
+ * struct rdt_core_mon_domain - CPUs sharing an Intel-PMT-scoped resctrl monitor resource
+ * @hdr:		common header for different domain types
+ */
+struct rdt_core_mon_domain {
+	struct rdt_domain_hdr	hdr;
+};
+
 /**
  * struct resctrl_cache - Cache allocation related data
  * @cbm_len:		Length of the cache bit mask
arch/x86/kernel/cpu/resctrl/internal.h +7 -0
index c8da6fac4720..f530382d8871 100644
@@ -637,6 +637,13 @@ void closid_free(int closid);
 int alloc_rmid(u32 closid);
 void free_rmid(u32 closid, u32 rmid);
 int rdt_get_mon_l3_config(struct rdt_resource *r);
+#ifdef CONFIG_INTEL_PMT_RESCTRL
+int rdt_get_intel_pmt_mon_config(void);
+void rdt_get_intel_pmt_mount(void);
+#else
+static inline int rdt_get_intel_pmt_mon_config(void) { return 0; }
+static inline void rdt_get_intel_pmt_mount(void) { }
+#endif
 void __exit rdt_put_mon_l3_config(void);
 bool __init rdt_cpu_has(int flag);
 void mon_event_count(void *info);
arch/x86/kernel/cpu/resctrl/core.c +6 -3
index 75a815383427..c18d79e470d2 100644
@@ -972,6 +972,7 @@ static __init bool get_rdt_alloc_resources(void)
 static __init bool get_rdt_mon_resources(void)
 {
 	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
+	int ret1 = -EINVAL, ret2;
 
 	if (rdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC))
 		rdt_mon_features |= (1 << QOS_L3_OCCUP_EVENT_ID);
@@ -980,15 +981,17 @@ static __init bool get_rdt_mon_resources(void)
 	if (rdt_cpu_has(X86_FEATURE_CQM_MBM_LOCAL))
 		rdt_mon_features |= (1 << QOS_L3_MBM_LOCAL_EVENT_ID);
 
-	if (!rdt_mon_features)
-		return false;
+	if (rdt_mon_features)
+		ret1 = rdt_get_mon_l3_config(r);
+
+	ret2 = rdt_get_intel_pmt_mon_config();
 
 	if (is_mbm_local_enabled())
 		mba_mbps_default_event = QOS_L3_MBM_LOCAL_EVENT_ID;
 	else if (is_mbm_total_enabled())
 		mba_mbps_default_event = QOS_L3_MBM_TOTAL_EVENT_ID;
 
-	return !rdt_get_mon_l3_config(r);
+	return ret1 == 0 || ret2;
 }
 
 static __init void __check_quirks_intel(void)
arch/x86/kernel/cpu/resctrl/intel_pmt.c new file +169 -0

169 lines not on this page · raw message

arch/x86/kernel/cpu/resctrl/rdtgroup.c +3 -0
index d2f3718f16af..4259bded5b7b 100644
@@ -2645,6 +2645,9 @@ static int rdt_get_tree(struct fs_context *fc)
 
 	cpus_read_lock();
 	mutex_lock(&rdtgroup_mutex);
+
+	rdt_get_intel_pmt_mount();
+
 	/*
 	 * resctrl file system can only be mounted once.
 	 */
arch/x86/kernel/cpu/resctrl/Makefile +1 -0
index 72621ea6cd6a..fc79e767b263 100644
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_X86_CPU_RESCTRL)	+= core.o rdtgroup.o monitor.o
 obj-$(CONFIG_X86_CPU_RESCTRL)	+= ctrlmondata.o pseudo_lock.o
+obj-$(CONFIG_INTEL_PMT_RESCTRL)	+= intel_pmt.o
 obj-$(CONFIG_INTEL_PMT_RESCTRL)	+= fake_intel_pmt_features.o
 CFLAGS_pseudo_lock.o = -I$(src)
Signature · 1 line
2.48.1
author maintainer
#9 lore raw

RFC PATCH 9/17 x86/resctrl: Add Intel PMT domain specific code

In reply to Tony Luck

Domain specific initialization and tear down. Very simple as there are no domain specific elements that need to be allocated or initiailazed. Just the domain itself.

This does need the mkdir_mondata_subdir_allrdtgrp() and rmdir_mondata_subdir_allrdtgrp() functions. So change them from static to global so they can be called from other source files.

---
 arch/x86/kernel/cpu/resctrl/internal.h  |  7 +++++++
 arch/x86/kernel/cpu/resctrl/core.c      | 12 ++++++++++++
 arch/x86/kernel/cpu/resctrl/intel_pmt.c | 16 ++++++++++++++++
 arch/x86/kernel/cpu/resctrl/rdtgroup.c  |  6 ++----
 4 files changed, 37 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +7 -0
index f530382d8871..65bbe223f8a1 100644
@@ -640,9 +640,12 @@ int rdt_get_mon_l3_config(struct rdt_resource *r);
 #ifdef CONFIG_INTEL_PMT_RESCTRL
 int rdt_get_intel_pmt_mon_config(void);
 void rdt_get_intel_pmt_mount(void);
+void setup_intel_pmt_mon_domain(int cpu, int id, struct rdt_resource *r, struct list_head *add_pos);
 #else
 static inline int rdt_get_intel_pmt_mon_config(void) { return 0; }
 static inline void rdt_get_intel_pmt_mount(void) { }
+static inline void setup_intel_pmt_mon_domain(int cpu, int id, struct rdt_resource *r,
+					      struct list_head *add_pos) { }
 #endif
 void __exit rdt_put_mon_l3_config(void);
 bool __init rdt_cpu_has(int flag);
@@ -667,4 +670,8 @@ void resctrl_file_fflags_init(const char *config, unsigned long fflags);
 void rdt_staged_configs_clear(void);
 bool closid_allocated(unsigned int closid);
 int resctrl_find_cleanest_closid(void);
+void mkdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,
+				    struct rdt_domain_hdr *hdr);
+void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,
+				    struct rdt_domain_hdr *hdr);
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/core.c +12 -0
index c18d79e470d2..b6e6a25520f7 100644
@@ -657,6 +657,9 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 	case RDT_RESOURCE_L3:
 		setup_l3_mon_domain(cpu, id, r, add_pos);
 		break;
+	case RDT_RESOURCE_INTEL_PMT:
+		setup_intel_pmt_mon_domain(cpu, id, r, add_pos);
+		break;
 	default:
 		WARN_ON_ONCE(1);
 	}
@@ -752,6 +755,15 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
 		synchronize_rcu();
 		mon_domain_free(hw_dom);
 		break;
+	case RDT_RESOURCE_INTEL_PMT:
+		mutex_lock(&rdtgroup_mutex);
+		if (resctrl_mounted && resctrl_arch_mon_capable())
+			rmdir_mondata_subdir_allrdtgrp(r, hdr);
+		mutex_unlock(&rdtgroup_mutex);
+		list_del_rcu(&hdr->list);
+		synchronize_rcu();
+		kfree(container_of(hdr, struct rdt_core_mon_domain, hdr));
+		break;
 	}
 }
 
arch/x86/kernel/cpu/resctrl/intel_pmt.c +16 -0
index 44373052ca49..76ec42a67c48 100644
@@ -167,3 +167,19 @@ void rdt_get_intel_pmt_mount(void)
 		r->mon_capable = false;
 	}
 }
+
+void setup_intel_pmt_mon_domain(int cpu, int id, struct rdt_resource *r, struct list_head *add_pos)
+{
+	struct rdt_core_mon_domain *hw_dom;
+
+	hw_dom = kzalloc_node(sizeof(*hw_dom), GFP_KERNEL, cpu_to_node(cpu));
+	if (!hw_dom)
+		return;
+
+	hw_dom->hdr.id = id;
+	hw_dom->hdr.type = RESCTRL_MON_DOMAIN;
+	cpumask_set_cpu(cpu, &hw_dom->hdr.cpu_mask);
+	list_add_tail_rcu(&hw_dom->hdr.list, add_pos);
+	if (resctrl_mounted && resctrl_arch_mon_capable())
+		mkdir_mondata_subdir_allrdtgrp(r, &hw_dom->hdr);
+}
arch/x86/kernel/cpu/resctrl/rdtgroup.c +2 -4
index 4259bded5b7b..93da2e4f7fec 100644
@@ -3041,8 +3041,7 @@ static void mon_rmdir_one_subdir(struct kernfs_node *pkn, char *name, char *subn
  * Remove files and directories containing "sum" of domain data
  * when last domain being summed is removed.
  */
-static void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,
-					   struct rdt_domain_hdr *hdr)
+void rmdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, struct rdt_domain_hdr *hdr)
 {
 	struct rdtgroup *prgrp, *crgrp;
 	char subname[32];
@@ -3170,8 +3169,7 @@ static int mkdir_mondata_subdir(struct kernfs_node *parent_kn,
  * Add all subdirectories of mon_data for "ctrl_mon" groups
  * and "monitor" groups with given domain id.
  */
-static void mkdir_mondata_subdir_allrdtgrp(struct rdt_resource *r,
-					   struct rdt_domain_hdr *hdr)
+void mkdir_mondata_subdir_allrdtgrp(struct rdt_resource *r, struct rdt_domain_hdr *hdr)
 {
 	struct kernfs_node *parent_kn;
 	struct rdtgroup *prgrp, *crgrp;
Signature · 1 line
2.48.1
author maintainer
#10 lore raw

RFC PATCH 10/17 x86/resctrl: Add detailed descriptions for Clearwater Forest events

In reply to Tony Luck

There are two event groups one for energy reporting and another for "perf" events.

See the XML description files in https://github.com/intel/Intel-PMT
in the xml/CWF/OOBMSM/{RMID-ENERGY,RMID-PERF}/ for the detailed descriptions that were used to derive these descriptions.

---
 arch/x86/kernel/cpu/resctrl/intel_pmt.c | 43 +++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_pmt.c +43 -0
index 76ec42a67c48..78ab6d899ee2 100644
@@ -70,7 +70,50 @@ struct telem_entry {
 	struct pmt_event evts[];
 };
 
+#define CWF_NUM_RMIDS	576
+#define CWF_ENERGY_GUID	0x26696143
+#define CWF_PERF_GUID	0x26557651
+#define	CWF_ENERGY_STRIDE	0x10
+#define CWF_PERF_STRIDE	0x38
+
+static struct telem_entry cwf_energy = {
+	.name				= "energy",
+	.guid				= CWF_ENERGY_GUID,
+	.num_rmids			= CWF_NUM_RMIDS,
+	.stride				= CWF_ENERGY_STRIDE,
+	.overflow_counter_off		= CWF_NUM_RMIDS * CWF_ENERGY_STRIDE,
+	.last_overflow_tstamp_off	= CWF_NUM_RMIDS * CWF_ENERGY_STRIDE + 8,
+	.last_update_tstamp_off		= CWF_NUM_RMIDS * CWF_ENERGY_STRIDE + 16,
+	.evts				= {
+		EVT(PMT_EVENT_ENERGY, "core_energy", 0x0, EVT_U46_18),
+		EVT(PMT_EVENT_ACTIVITY, "activity", 0x8, EVT_U46_18),
+		{ }
+	}
+};
+
+static struct telem_entry cwf_perf = {
+	.name				= "perf",
+	.guid				= CWF_PERF_GUID,
+	.num_rmids			= CWF_NUM_RMIDS,
+	.stride				= CWF_PERF_STRIDE,
+	.overflow_counter_off		= CWF_NUM_RMIDS * CWF_PERF_STRIDE,
+	.last_overflow_tstamp_off	= CWF_NUM_RMIDS * CWF_PERF_STRIDE + 8,
+	.last_update_tstamp_off		= CWF_NUM_RMIDS * CWF_PERF_STRIDE + 16,
+	.evts				= {
+		EVT(PMT_EVENT_STALLS_LLC_HIT, "stalls_llc_hit", 0x0, EVT_U64),
+		EVT(PMT_EVENT_C1_RES, "c1_res", 0x8, EVT_U64),
+		EVT(PMT_EVENT_UNHALTED_CORE_CYCLES, "unhalted_core_cycles", 0x10, EVT_U64),
+		EVT(PMT_EVENT_STALLS_LLC_MISS, "stalls_llc_miss", 0x18, EVT_U64),
+		EVT(PMT_EVENT_AUTO_C6_RES, "c6_res", 0x20, EVT_U64),
+		EVT(PMT_EVENT_UNHALTED_REF_CYCLES, "unhalted_ref_cycles", 0x28, EVT_U64),
+		EVT(PMT_EVENT_UOPS_RETIRED, "uops_retired", 0x30, EVT_U64),
+		{ }
+	}
+};
+
 static struct telem_entry *telem_entry[] = {
+	&cwf_energy,
+	&cwf_perf,
 	NULL
 };
 
Signature · 1 line
2.48.1
author maintainer
#11 lore raw

RFC PATCH 11/17 x86/resctrl: Allocate per-package structures for known events

In reply to Tony Luck

Use the per-package counts of known events to allocate arrays to make a copy of just the known events.

Add hook into resctrl_exit() to cleanup.

---
 arch/x86/kernel/cpu/resctrl/internal.h  |  2 +
 arch/x86/kernel/cpu/resctrl/core.c      |  2 +
 arch/x86/kernel/cpu/resctrl/intel_pmt.c | 59 ++++++++++++++++++++++++-
 3 files changed, 62 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/internal.h +2 -0
index 65bbe223f8a1..24c4ab331c3c 100644
@@ -641,11 +641,13 @@ int rdt_get_mon_l3_config(struct rdt_resource *r);
 int rdt_get_intel_pmt_mon_config(void);
 void rdt_get_intel_pmt_mount(void);
 void setup_intel_pmt_mon_domain(int cpu, int id, struct rdt_resource *r, struct list_head *add_pos);
+void rdt_intel_pmt_exit(void);
 #else
 static inline int rdt_get_intel_pmt_mon_config(void) { return 0; }
 static inline void rdt_get_intel_pmt_mount(void) { }
 static inline void setup_intel_pmt_mon_domain(int cpu, int id, struct rdt_resource *r,
 					      struct list_head *add_pos) { }
+static inline void rdt_intel_pmt_exit(void) { };
 #endif
 void __exit rdt_put_mon_l3_config(void);
 bool __init rdt_cpu_has(int flag);
arch/x86/kernel/cpu/resctrl/core.c +2 -0
index b6e6a25520f7..da44d00acd98 100644
@@ -1170,6 +1170,8 @@ static void __exit resctrl_exit(void)
 
 	cpuhp_remove_state(rdt_online);
 
+	rdt_intel_pmt_exit();
+
 	rdtgroup_exit();
 
 	if (r->mon_capable)
arch/x86/kernel/cpu/resctrl/intel_pmt.c +58 -1

89 lines not on this page · raw message

Signature · 1 line
2.48.1
author maintainer
#12 lore raw

RFC PATCH 12/17 x86/resctrl: Add list of known events to RDT_RESOURCE_INTEL_PMT

In reply to Tony Luck

Core code uses this list to populate "mon_data" directories.

---
 arch/x86/kernel/cpu/resctrl/intel_pmt.c | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/resctrl/intel_pmt.c +23 -1
index 754748d858c6..05640e85d303 100644
@@ -13,6 +13,7 @@
 
 #include <linux/cpu.h>
 #include <linux/cleanup.h>
+#include <linux/minmax.h>
 #include <linux/slab.h>
 #include "fake_intel_pmt_features.h"
 #include <linux/intel_vsec.h>
@@ -254,14 +255,35 @@ void rdt_get_intel_pmt_mount(void)
 {
 	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_INTEL_PMT].r_resctrl;
 	struct rdt_core_mon_domain *d, *tmp;
+	struct telem_entry **tentry;
 	static int do_one_time;
+	struct mon_evt *evt;
+	bool ret = false;
 
 	if (do_one_time)
 		return;
 
 	do_one_time = 1;
 
-	if (!get_events()) {
+	if (!get_events())
+		goto done;
+
+	for (tentry = telem_entry; *tentry; tentry++) {
+		if (!(*tentry)->active)
+			continue;
+		for (int i = 0; (*tentry)->evts[i].evt.name; i++) {
+			evt = &(*tentry)->evts[i].evt;
+			list_add_tail(&evt->list, &r->evt_list);
+			ret = true;
+		}
+		if (!r->num_rmid)
+			r->num_rmid = (*tentry)->num_rmids;
+		else
+			r->num_rmid = min(r->num_rmid, (*tentry)->num_rmids);
+	}
+done:
+	if (!ret) {
+		r->mon_capable = false;
 		list_for_each_entry_safe(d, tmp, &r->mon_domains, hdr.list)
 			kfree(d);
 		r->mon_capable = false;
Signature · 1 line
2.48.1
author maintainer
#13 lore raw

RFC PATCH 13/17 x86/resctrl: Build lookup table for package events

In reply to Tony Luck

The resctrl filesystem saves the evt_type in the private data pointer of the kernfs_node of each file in the mon_data directories.

To print the values for each file the show() function will need to map from this evtid to:

num_rmids - to make sure data for this file exists guid - to pick the pmt_event(s) for each package stride/offset - to compute MMIO offset for this RMID/event

Build a lookup table for each event to save searching through lists and add macros for ease of use.

---
 arch/x86/kernel/cpu/resctrl/intel_pmt.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_pmt.c +15 -0
index 05640e85d303..c5557d7f5fbe 100644
@@ -38,6 +38,18 @@ enum pmt_event_id {
 	PMT_NUM_EVENTS
 };
 
+static struct evtinfo {
+	struct telem_entry	*telem_entry;
+	struct pmt_event	*pmt_event;
+} evtinfo[PMT_NUM_EVENTS];
+
+#define EVT_NUM_RMIDS(evtid)	(evtinfo[evtid].telem_entry->num_rmids)
+#define EVT_STRIDE(evtid)	(evtinfo[evtid].telem_entry->stride)
+#define EVT_GUID(evtid)		(evtinfo[evtid].telem_entry->guid)
+
+#define EVT_OFFSET(evtid)	(evtinfo[evtid].pmt_event->evt_offset)
+#define EVT_TYPE(evtid)		(evtinfo[evtid].pmt_event->evt_type)
+
 enum evt_type {
 	EVT_U64,
 	EVT_U46_18,
@@ -274,6 +286,9 @@ void rdt_get_intel_pmt_mount(void)
 		for (int i = 0; (*tentry)->evts[i].evt.name; i++) {
 			evt = &(*tentry)->evts[i].evt;
 			list_add_tail(&evt->list, &r->evt_list);
+
+			evtinfo[evt->evtid].telem_entry = *tentry;
+			evtinfo[evt->evtid].pmt_event = &(*tentry)->evts[i];
 			ret = true;
 		}
 		if (!r->num_rmid)
Signature · 1 line
2.48.1
author maintainer
#14 lore raw

RFC PATCH 14/17 x86/resctrl: Add code in rdtgroup_mondata_show() for core monitor events

In reply to Tony Luck

There may be multiple devices tracking each package, so scan all of them and add up counters.

Output format depends on the data type. Either a 63 bit integer, or a fixed point decimal.

---
 arch/x86/kernel/cpu/resctrl/internal.h    |  6 ++
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  3 +
 arch/x86/kernel/cpu/resctrl/intel_pmt.c   | 88 +++++++++++++++++++++++
 3 files changed, 97 insertions(+)
arch/x86/kernel/cpu/resctrl/internal.h +6 -0
index 24c4ab331c3c..068a47b015e7 100644
@@ -639,11 +639,17 @@ void free_rmid(u32 closid, u32 rmid);
 int rdt_get_mon_l3_config(struct rdt_resource *r);
 #ifdef CONFIG_INTEL_PMT_RESCTRL
 int rdt_get_intel_pmt_mon_config(void);
+int rdtgroup_intel_pmt_data_show(struct seq_file *m, struct rdt_resource *r,
+				 int domid, struct rdtgroup *rdtgrp, u32 evtid,
+				 struct rmid_read *rr);
 void rdt_get_intel_pmt_mount(void);
 void setup_intel_pmt_mon_domain(int cpu, int id, struct rdt_resource *r, struct list_head *add_pos);
 void rdt_intel_pmt_exit(void);
 #else
 static inline int rdt_get_intel_pmt_mon_config(void) { return 0; }
+static inline int rdtgroup_intel_pmt_data_show(struct seq_file *m, struct rdt_resource *r,
+					       int domid, struct rdtgroup *rdtgrp, u32 evtid,
+					       struct rmid_read *rr) { return 0; }
 static inline void rdt_get_intel_pmt_mount(void) { }
 static inline void setup_intel_pmt_mon_domain(int cpu, int id, struct rdt_resource *r,
 					      struct list_head *add_pos) { }
arch/x86/kernel/cpu/resctrl/ctrlmondata.c +3 -0
index 515a9bec05cd..0c3d407aca08 100644
@@ -700,6 +700,9 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	case RDT_RESOURCE_L3:
 		ret = rdtgroup_l3_data_show(m, r, domid, rdtgrp, evtid, md.u.sum, &rr);
 		break;
+	case RDT_RESOURCE_INTEL_PMT:
+		ret = rdtgroup_intel_pmt_data_show(m, r, domid, rdtgrp, evtid, &rr);
+		break;
 	default:
 		ret = -EINVAL;
 		break;
arch/x86/kernel/cpu/resctrl/intel_pmt.c +88 -0

91 lines not on this page · raw message

Signature · 1 line
2.48.1
author maintainer
#15 lore raw

RFC PATCH 15/17 x86/resctrl: Add status files to info/PKG_MON

In reply to Tony Luck

Each of the package event groups includes status about the number of missed events and timestamps (from a 25 MHz clock) on last time an update was missed and last time an update was processed.

Add a three info files to report per-aggregator values of these status values.

---
 arch/x86/kernel/cpu/resctrl/internal.h  | 13 ++++
 arch/x86/kernel/cpu/resctrl/intel_pmt.c | 92 ++++++++++++++++++++++---
 arch/x86/kernel/cpu/resctrl/rdtgroup.c  | 22 +++++-
 3 files changed, 117 insertions(+), 10 deletions(-)
arch/x86/kernel/cpu/resctrl/internal.h +13 -0
index 068a47b015e7..5f3656e938ed 100644
@@ -321,6 +321,7 @@ struct rdtgroup {
 #define RFTYPE_CTRL_INFO		(RFTYPE_INFO | RFTYPE_CTRL)
 #define RFTYPE_MON_INFO			(RFTYPE_INFO | RFTYPE_MON)
 #define RFTYPE_TOP_INFO			(RFTYPE_INFO | RFTYPE_TOP)
+#define RFTYPE_PKG_INFO			(RFTYPE_INFO | RFTYPE_RES_PKG)
 #define RFTYPE_CTRL_BASE		(RFTYPE_BASE | RFTYPE_CTRL)
 #define RFTYPE_MON_BASE			(RFTYPE_BASE | RFTYPE_MON)
 
@@ -642,6 +643,12 @@ int rdt_get_intel_pmt_mon_config(void);
 int rdtgroup_intel_pmt_data_show(struct seq_file *m, struct rdt_resource *r,
 				 int domid, struct rdtgroup *rdtgrp, u32 evtid,
 				 struct rmid_read *rr);
+int rdtgroup_last_update_show(struct kernfs_open_file *of,
+			      struct seq_file *seq, void *v);
+int rdtgroup_overflows_show(struct kernfs_open_file *of,
+			    struct seq_file *seq, void *v);
+int rdtgroup_overflow_timestamp_show(struct kernfs_open_file *of,
+				     struct seq_file *seq, void *v);
 void rdt_get_intel_pmt_mount(void);
 void setup_intel_pmt_mon_domain(int cpu, int id, struct rdt_resource *r, struct list_head *add_pos);
 void rdt_intel_pmt_exit(void);
@@ -650,6 +657,12 @@ static inline int rdt_get_intel_pmt_mon_config(void) { return 0; }
 static inline int rdtgroup_intel_pmt_data_show(struct seq_file *m, struct rdt_resource *r,
 					       int domid, struct rdtgroup *rdtgrp, u32 evtid,
 					       struct rmid_read *rr) { return 0; }
+static inline int rdtgroup_last_update_show(struct kernfs_open_file *of,
+					    struct seq_file *seq, void *v) { return 0; };
+static inline int rdtgroup_overflows_show(struct kernfs_open_file *of,
+					  struct seq_file *seq, void *v) { return 0; };
+static inline int rdtgroup_overflow_timestamp_show(struct kernfs_open_file *of,
+						   struct seq_file *seq, void *v) { return 0; };
 static inline void rdt_get_intel_pmt_mount(void) { }
 static inline void setup_intel_pmt_mon_domain(int cpu, int id, struct rdt_resource *r,
 					      struct list_head *add_pos) { }
arch/x86/kernel/cpu/resctrl/intel_pmt.c +83 -9

125 lines not on this page · raw message

arch/x86/kernel/cpu/resctrl/rdtgroup.c +21 -1
index 93da2e4f7fec..635c17042f7a 100644
@@ -1986,7 +1986,27 @@ static struct rftype res_common_files[] = {
 		.seq_show	= rdtgroup_closid_show,
 		.fflags		= RFTYPE_CTRL_BASE | RFTYPE_DEBUG,
 	},
-
+	{
+		.name		= "last_update",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= rdtgroup_last_update_show,
+		.fflags		= RFTYPE_PKG_INFO,
+	},
+	{
+		.name		= "overflows",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= rdtgroup_overflows_show,
+		.fflags		= RFTYPE_PKG_INFO,
+	},
+	{
+		.name		= "overflow_timestamp",
+		.mode		= 0444,
+		.kf_ops		= &rdtgroup_kf_single_ops,
+		.seq_show	= rdtgroup_overflow_timestamp_show,
+		.fflags		= RFTYPE_PKG_INFO,
+	},
 };
 
 static int rdtgroup_add_files(struct kernfs_node *kn, unsigned long fflags)
Signature · 1 line
2.48.1
author maintainer
#16 lore raw

RFC PATCH 16/17 x86/resctrl: Enable package event monitoring

In reply to Tony Luck

All the code to support this is in place. Unconditionally enable the package resource during early initialization so that domain lists are built.

Enumeration of the package events happens at first mount, if there are no package scoped events, then this resource is disabled.

TODO: Cleanup the domains.

---
 arch/x86/kernel/cpu/resctrl/intel_pmt.c | 2 ++
 1 file changed, 2 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_pmt.c +2 -0
index e89130387387..52ff5eb79b4e 100644
@@ -244,6 +244,8 @@ int rdt_get_intel_pmt_mon_config(void)
 
 	INIT_LIST_HEAD(&r->evt_list);
 
+	r->mon_capable = true;
+
 	return 1;
 }
 
Signature · 1 line
2.48.1
author maintainer
#17 lore raw

RFC PATCH 17/17 x86/resctrl: Update Documentation for package events

In reply to Tony Luck

Each "mon_data" directory is now divided between L3 events and package events.

---
 Documentation/arch/x86/resctrl.rst | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)
Documentation/arch/x86/resctrl.rst +16 -9
index 6768fc1fad16..0e915b7c55aa 100644
@@ -366,15 +366,22 @@ When control is enabled all CTRL_MON groups will also contain:
 When monitoring is enabled all MON groups will also contain:
 
 "mon_data":
-	This contains a set of files organized by L3 domain and by
-	RDT event. E.g. on a system with two L3 domains there will
-	be subdirectories "mon_L3_00" and "mon_L3_01".	Each of these
-	directories have one file per event (e.g. "llc_occupancy",
-	"mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
-	files provide a read out of the current value of the event for
-	all tasks in the group. In CTRL_MON groups these files provide
-	the sum for all tasks in the CTRL_MON group and all tasks in
-	MON groups. Please see example section for more details on usage.
+	This contains a set of directories, one for each instance
+	of an L3 cache, or of a processor package. The L3 cache
+	directories are named "mon_L3_00", "mon_L3_01" etc. The
+	package directories "mon_PKG_00", "mon_PKG_01" etc.
+
+	Within each directory there is one file per event. In
+	the L3 directories: "llc_occupancy", "mbm_total_bytes",
+	and "mbm_local_bytes".	In the PKG directories: "core_energy",
+	"activity", etc.
+
+	In a MON group these files provide a read out of the current
+	value of the event for all tasks in the group. In CTRL_MON groups
+	these files provide the sum for all tasks in the CTRL_MON group
+	and all tasks in MON groups. Please see example section for more
+	details on usage.
+
 	On systems with Sub-NUMA Cluster (SNC) enabled there are extra
 	directories for each node (located within the "mon_L3_XX" directory
 	for the L3 cache they occupy). These are named "mon_sub_L3_YY"
Signature · 1 line
2.48.1