Skip to contentSkip to conversation

x86/resctrl telemetry monitoring

PATCH Superseded Tony Luck 3 participants 67 messages Apr 7, 2025 – Apr 23, 2025

Subsystems (5)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Tony Luck

author maintainer
#0 lore raw

v3 cover x86/resctrl telemetry monitoring

Previous version here:
https://lore.kernel.org/all/[email protected]/

This series is based on James Morse's "fs/resctrl/" snapshot.

General changes since v2:
------------------------

Fixed several violations of the FS/ARCH interface. As far as possible the patches are now divided into patches for ARCH code v.s. patches for FS code.

All event names/numbers are defined by FS code.

Changed the names of new bits in FS code to PERF_PKG.

Changed names of detailed event descriptions from using "CWF" (short for Clearwater Forest) to names using the guid value. E.g. NUM_RMIDS_CWF -> NUM_RMIDS_0x26696143

Fixed typos from v2, and improved many commit messages and code comments.

Roadmap to patches in the series:
--------------------------------

First patch provides a simpler implementation of James' "Expand the width of dom_id by replacing mon_data_bits" patch. James will merge this into the next version of his resctrl fs series. So this patch will be dropped.

Second patch is a cleaned up version of something I posted before to improve handling multiple MBM events.
https://lore.kernel.org/all/[email protected]/
It turns out to be a useful base to add other monitor events.

Third patch is preparation for more monitor events changing how and when they are initialized. The "when" moves the initialization to first mount of resctrl ready for adding perf events (which are enumerated much later than normal resctrl initialization).

Fourth sets up the Kconfig elements for PERF events.

Fifth patch is the fake OOBMSM interface, providing the intel_pmt_get_regions_by_feature() and intel_pmt_put_feature_group() that will eventually be supplied directly by OOBMSM code.

Sixth patch is in response to Reinette's review of the V2 series where I had deleted some rdt_mon_domain type checking which could lead to problems with the additional domain type. I extended the type checking from simple "ctrl" vs. "mon" to also check the resource type.

7-9 refactor some x86 code to make adding a new rdt_resource easier

10 Implemented a suggestion from James on coding so that events that can be read from any CPU avoid a cross-processor interrupt while keeping the same calling sequence between FS and ARCH code. The attribute is available independently for each event.

11 Adds support for ARCH code to describe which format to display monitor values from choices listed in an FS enum. Two choices
implemented: 64-bit unsigned decimal number, and fixed-point with 18 binary digits displayed as floating point.

12 Add FS hook for ARCH code to set event counter attributes.

13 Add ARCH hook called by FS on each mount.

14-16 X86 enumeration of perf events using calls to OOBMSM

17 Add lookup table from FS event numbers to ARCH structures needed to process event read requests

18 Hook into resctrl_arch_rmid_read() for RDT_RESOURCE_PERF_PKG to read counters for its events. Code to read the events sums over all aggregators in a package.

19 Cross check three enumerated sources of number of RMIDS for perf events. This part needs further discussion on how to pass information to users. Existing FS/ARCH interfaces doesn't have obvious way to add additional info/ files.

20 Add new RDT_RESOURCE_PERF_PKG resource and package scope define

21 Domain creation/deletion for RDT_RESOURCE_PERF_PKG

22 Type define for resctrl files for RDT_RESOURCE_PERF_PKG

23 Add new events to typedef resctrl_event_id and all_events[].

24 Enable all enumerated telemetry events and the RDT_RESOURCE_PERF_PKG resource.

25 Add details for the telemetry events on Clearwater Forest systems.

26 Documentation.

Background

Telemetry features are being implemented in conjunction with the IA32_PQR_ASSOC.RMID value on each logical CPU. This is used to send counts for various events to a collector in a nearby OOMMSM device to be accumulated with counts for each <RMID, event> pair received from other CPUs. Cores send event counts when the RMID value changes, or after each 2ms elapsed time.

Each OOBMSM device may implement multiple event collectors with each servicing a subset of the logical CPUs on a package. In the initial hardware implementation, there are two categories of events:

The counters are arranged in groups in MMIO space of the OOBMSM device. E.g. for the energy counters the layout is:

Offset: Counter
0x00 core energy for RMID 0
0x08 core activity for RMID 0
0x10 core energy for RMID 1
0x18 core activity for RMID 1

1) Energy - Two counters
core_energy: This is an estimate of Joules consumed by each core. It is calculated based on the types of instructions executed, not from a power meter. This counter is useful to understand how much energy a workload is consuming.

activity: This measures "accumulated dynamic capacitance". Users who want to optimize energy consumption for a workload may use this rather than core_energy because it provides consistent results independent of any frequency or voltage changes that may occur during the runtime of the application (e.g. entry/exit from turbo mode).

2) Performance - Seven counters
These are similar events to those available via the Linux "perf" tool, but collected in a way with mush lower overhead (no need to collect data on every context switch).

stalls_llc_hit - Counts the total number of unhalted core clock cycles when the core is stalled due to a demand load miss which hit in the LLC

c1_res - Counts the total C1 residency across all cores. The underlying counter increments on 100MHz clock ticks

unhalted_core_cycles - Counts the total number of unhalted core clock cycles

stalls_llc_miss - Counts the total number of unhalted core clock cycles when the core is stalled due to a demand load miss which missed all the local caches

c6_res - Counts the total C6 residency. The underlying counter increments on crystal clock (25MHz) ticks

unhalted_ref_cycles - Counts the total number of unhalted reference clock (TSC) cycles

uops_retired - Counts the total number of uops retired

Enumeration

The only CPUID based enumeration for this feature is the legacy CPUID(eax=7,ecx=0).ebx{12} that indicates the presence of the IA32_PQR_ASSOC MSR and the RMID field within it.

The OOBMSM driver discovers which features are present via PCIe VSEC capabilities. Each feature is tagged with a unique identifier. These identifiers indicate which XML description file from
https://github.com/intel/Intel-PMT describes which event counters are available and their layout within the MMIO BAR space of the OOBMSM device.

Resctrl User Interface

Because there may be multiple OOBMSM collection agents per processor package, resctrl accumulates event counts from all agents on a package and presents a single value to users. This will provide a consistent user interface on future platforms that vary the number of collectors, or the mappings from logical CPUs to collectors.

Users will see the legacy monitoring files in the "L3" directories and the telemetry files in "PKG" directories (with each file providing the aggregated value from all OOBMSM collectors on that package).

$ tree /sys/fs/resctrl/mon_data/
/sys/fs/resctrl/mon_data/
├── mon_L3_00
│   ├── llc_occupancy
│   ├── mbm_local_bytes
│   └── mbm_total_bytes
├── mon_L3_01
│   ├── llc_occupancy
│   ├── mbm_local_bytes
│   └── mbm_total_bytes
├── mon_PKG_00
│   ├── activity
│   ├── c1_res
│   ├── c6_res
│   ├── core_energy
│   ├── stalls_llc_hit
│   ├── stalls_llc_miss
│   ├── unhalted_core_cycles
│   ├── unhalted_ref_cycles
│   └── uops_retired
└── mon_PKG_01
    ├── activity
    ├── c1_res
    ├── c6_res
    ├── core_energy
    ├── stalls_llc_hit
    ├── stalls_llc_miss
    ├── unhalted_core_cycles
    ├── unhalted_ref_cycles
    └── uops_retired

Resctrl Implementation

The OOBMSM driver exposes a function "intel_pmt_get_regions_by_feature()" that returns an array of structures describing the per-RMID groups it found from the VSEC enumeration. Linux looks at the unique identifiers for each group and enables resctrl for all groups with known unique identifiers.

The memory map for the counters for each <RMID, event> pair is described by the XML file. This is too unwieldy to use in the Linux kernel, so a simplified representation is built into the resctrl code. Note that the counters are in MMIO space instead of accessed using the IA32_QM_EVTSEL and IA32_QM_CTR MSRs. This means there is no need for cross-processor calls to read counters from a CPU in a specific domain. The counters can be read from any CPU.

High level description of code changes:

1) New scope RESCTRL_PACKAGE
2) New struct rdt_resource RDT_RESOURCE_INTEL_PMT
3) Refactor monitor code paths to split existing L3 paths from new ones. In some cases this ends up with:

        switch (r->rid) {
        case RDT_RESOURCE_L3:
                helper for L3
                break;
        case RDT_RESOURCE_INTEL_PMT:
                helper for PKG
                break;
        }

4) New source code file "intel_pmt.c" for the code to enumerate, configure, and report event counts.

With only one platform providing this feature, it's tricky to tell exactly where it is going to go. I've made the event definitions platform specific (based on the unique ID from the VSEC enumeration). It seems possible/likely that the list of events may change from generation to generation.

I've picked names for events based on the descriptions in the XML file.

Tony Luck (26):
fs/resctrl: Simplify allocation of mon_data structures fs-x86/resctrl: Prepare for more monitor events fs/resctrl: Change how events are initialized
fs/resctrl: Set up Kconfig options for telemetry events x86/rectrl: Fake OOBMSM interface
fs-x86/rectrl: Improve domain type checking
x86/resctrl: Move L3 initialization out of domain_add_cpu_mon() x86/resctrl: Refactor domain_remove_cpu_mon() ready for new domain
types
x86/resctrl: Change generic monitor functions to use struct
rdt_domain_hdr
fs/resctrl: Improve handling for events that can be read from any CPU fs/resctrl: Add support for additional monitor event display formats fs/resctrl: Add hook for architecture code to set monitor event
attributes
fs/resctrl: Add an architectural hook called for each mount x86/resctrl: Add first part of telemetry event enumeration x86/resctrl: Second stage of telemetry event enumeration x86/resctrl: Third phase of telemetry event enumeration x86/resctrl: Build a lookup table for each resctrl event id x86/resctrl: Add code to read core telemetry events x86/resctrl: Sanity check telemetry RMID values x86/resctrl: Add and initialize rdt_resource for package scope core
monitor
fs-x86/resctrl: Handle RDT_RESOURCE_PERF_PKG in domain create/delete fs/resctrl: Add type define for PERF_PKG files
fs/resctrl: Add new telemetry event id and structures x86/resctrl: Final steps to enable RDT_RESOURCE_PERF_PKG fs-x86/resctrl: Add detailed descriptions for Clearwater Forest events x86/resctrl: Update Documentation for package events

 Documentation/filesystems/resctrl.rst         |  38 +-
 include/linux/resctrl.h                       |  42 +-
 include/linux/resctrl_types.h                 |  36 +-
 arch/x86/include/asm/resctrl.h                |   8 +-
 .../cpu/resctrl/fake_intel_aet_features.h     |  73 ++++
 arch/x86/kernel/cpu/resctrl/internal.h        |  19 +-
 fs/resctrl/internal.h                         |  23 +-
 arch/x86/kernel/cpu/resctrl/core.c            | 199 ++++++---
 .../cpu/resctrl/fake_intel_aet_features.c     |  87 ++++
 arch/x86/kernel/cpu/resctrl/intel_aet.c       | 377 ++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/monitor.c         |  39 +-
 fs/resctrl/ctrlmondata.c                      |  76 ++--
 fs/resctrl/monitor.c                          | 164 ++++++--
 fs/resctrl/rdtgroup.c                         | 266 ++++++------
 arch/x86/Kconfig                              |   1 +
 arch/x86/kernel/cpu/resctrl/Makefile          |   2 +
 drivers/platform/x86/intel/pmt/Kconfig        |   6 +
 17 files changed, 1125 insertions(+), 331 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.h
 create mode 100644 arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c
 create mode 100644 arch/x86/kernel/cpu/resctrl/intel_aet.c

base-commit: 89679ca4122a5a6622530490ef4855dd39baa54e

Signature · 1 line
2.48.1
author maintainer
#1 lore raw

PATCH v3 1/26 fs/resctrl: Simplify allocation of mon_data structures

Instead of making a special case to allocate and attach these structures to kernfs files in the default control group, simply allocate a structure when a new combination of <rid, domain, mevt, do_sum> is needed and reuse existing structures when possible.

Free all structures when resctrl filesystem is unmounted.

Partial revert of commit fa563b5171e9 ("x86/resctrl: Expand the width of dom_id by replacing mon_data_bits")

---
 fs/resctrl/internal.h |   2 +
 fs/resctrl/rdtgroup.c | 138 ++++++++++++------------------------------
 2 files changed, 40 insertions(+), 100 deletions(-)
fs/resctrl/internal.h +2 -0
index ec3863d18f68..e5976bd52a35 100644
@@ -83,6 +83,7 @@ struct mon_evt {
 
 /**
  * struct mon_data - Monitoring details for each event file.
+ * @list:            List of all allocated structures.
  * @rid:             Resource id associated with the event file.
  * @evtid:           Event id associated with the event file.
  * @sum:             Set when event must be summed across multiple
@@ -96,6 +97,7 @@ struct mon_evt {
  * rdtgroup_mutex.
  */
 struct mon_data {
+	struct list_head list;
 	unsigned int rid;
 	enum resctrl_event_id evtid;
 	unsigned int sum;
fs/resctrl/rdtgroup.c +38 -100

197 lines not on this page · raw message

Signature · 1 line
2.48.1
maintainer
#2 lore raw

Re: [PATCH v3 01/26] fs/resctrl: Simplify allocation of mon_data structures

Hi Tony,

On 4/7/25 4:40 PM, Tony Luck wrote:
...

fs/resctrl/rdtgroup.c static struct file_system_type rdt_fs_type = { · line 2950

Quoted 6 lines
fs/resctrl/rdtgroup.c +3 -0
+	list_for_each_entry_safe(priv, tmp, &kn_priv_list, list) {
+		kfree(priv);
+		list_del(&priv->list);
 	}
 }

Did not look through this patch in detail considering its other version in the arch/fs split but this caught my eye. Order should be switched.

Reinette

author maintainer
#3 lore raw

PATCH v3 2/26 fs-x86/resctrl: Prepare for more monitor events

In reply to Tony Luck

There's a rule in computer programming that objects appear zero, once, or many times. So code accordingly.

There are two MBM events and resctrl is coded with a lot of

if (local)
do one thing
if (total)
do a different thing

Simplify the code by coding for many events using loops on which are enabled.

Make rdt_mon_features a bitmap to allow for expansion.

Move resctrl_is_mbm_event() to <asm/resctrl.h> as it gets used by core.c

---
 include/linux/resctrl.h                |  6 +--
 include/linux/resctrl_types.h          |  8 +++
 arch/x86/include/asm/resctrl.h         |  8 +--
 arch/x86/kernel/cpu/resctrl/internal.h |  6 +--
 fs/resctrl/internal.h                  |  4 ++
 arch/x86/kernel/cpu/resctrl/core.c     | 45 +++++++++--------
 arch/x86/kernel/cpu/resctrl/monitor.c  | 33 ++++++------
 fs/resctrl/ctrlmondata.c               | 41 ++++-----------
 fs/resctrl/monitor.c                   | 70 ++++++++++++++++----------
 fs/resctrl/rdtgroup.c                  | 47 ++++++++---------
 10 files changed, 133 insertions(+), 135 deletions(-)
include/linux/resctrl.h +2 -4
index 5c7c8bf2c47f..d6a926b6fc0e 100644
@@ -151,8 +151,7 @@ struct rdt_ctrl_domain {
  * @hdr:		common header for different domain types
  * @ci:			cache info for this domain
  * @rmid_busy_llc:	bitmap of which limbo RMIDs are above threshold
- * @mbm_total:		saved state for MBM total bandwidth
- * @mbm_local:		saved state for MBM local bandwidth
+ * @mbm_states:		saved state for each QOS MBM event
  * @mbm_over:		worker to periodically read MBM h/w counters
  * @cqm_limbo:		worker to periodically read CQM h/w counters
  * @mbm_work_cpu:	worker CPU for MBM h/w counters
@@ -162,8 +161,7 @@ struct rdt_mon_domain {
 	struct rdt_domain_hdr		hdr;
 	struct cacheinfo		*ci;
 	unsigned long			*rmid_busy_llc;
-	struct mbm_state		*mbm_total;
-	struct mbm_state		*mbm_local;
+	struct mbm_state		*mbm_states[QOS_NUM_MBM_EVENTS];
 	struct delayed_work		mbm_over;
 	struct delayed_work		cqm_limbo;
 	int				mbm_work_cpu;
include/linux/resctrl_types.h +8 -0
index a7faf2cd5406..898068a99ef7 100644
@@ -55,5 +55,13 @@ enum resctrl_event_id {
 };
 
 #define QOS_NUM_EVENTS		(QOS_L3_MBM_LOCAL_EVENT_ID + 1)
+#define QOS_NUM_MBM_EVENTS	(QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1)
+#define MBM_EVENT_IDX(evt)	((evt) - QOS_L3_MBM_TOTAL_EVENT_ID)
+
+static inline bool resctrl_is_mbm_event(int e)
+{
+	return (e >= QOS_L3_MBM_TOTAL_EVENT_ID &&
+		e <= QOS_L3_MBM_LOCAL_EVENT_ID);
+}
 
 #endif /* __LINUX_RESCTRL_TYPES_H */
arch/x86/include/asm/resctrl.h +4 -4
index 6eb7d5c94c7a..4346de48eeab 100644
@@ -42,7 +42,7 @@ DECLARE_PER_CPU(struct resctrl_pqr_state, pqr_state);
 
 extern bool rdt_alloc_capable;
 extern bool rdt_mon_capable;
-extern unsigned int rdt_mon_features;
+extern DECLARE_BITMAP(rdt_mon_features, QOS_NUM_EVENTS);
 
 DECLARE_STATIC_KEY_FALSE(rdt_enable_key);
 DECLARE_STATIC_KEY_FALSE(rdt_alloc_enable_key);
@@ -84,17 +84,17 @@ static inline void resctrl_arch_disable_mon(void)
 
 static inline bool resctrl_arch_is_llc_occupancy_enabled(void)
 {
-	return (rdt_mon_features & (1 << QOS_L3_OCCUP_EVENT_ID));
+	return test_bit(QOS_L3_OCCUP_EVENT_ID, rdt_mon_features);
 }
 
 static inline bool resctrl_arch_is_mbm_total_enabled(void)
 {
-	return (rdt_mon_features & (1 << QOS_L3_MBM_TOTAL_EVENT_ID));
+	return test_bit(QOS_L3_MBM_TOTAL_EVENT_ID, rdt_mon_features);
 }
 
 static inline bool resctrl_arch_is_mbm_local_enabled(void)
 {
-	return (rdt_mon_features & (1 << QOS_L3_MBM_LOCAL_EVENT_ID));
+	return test_bit(QOS_L3_MBM_LOCAL_EVENT_ID, rdt_mon_features);
 }
 
 /*
arch/x86/kernel/cpu/resctrl/internal.h +2 -4
index 521db28efb3f..45eabc7919c6 100644
@@ -59,15 +59,13 @@ struct rdt_hw_ctrl_domain {
  * struct rdt_hw_mon_domain - Arch private attributes of a set of CPUs that share
  *			      a resource for a monitor function
  * @d_resctrl:	Properties exposed to the resctrl file system
- * @arch_mbm_total:	arch private state for MBM total bandwidth
- * @arch_mbm_local:	arch private state for MBM local bandwidth
+ * @arch_mbm_states:	arch private state for each MBM event
  *
  * Members of this structure are accessed via helpers that provide abstraction.
  */
 struct rdt_hw_mon_domain {
 	struct rdt_mon_domain		d_resctrl;
-	struct arch_mbm_state		*arch_mbm_total;
-	struct arch_mbm_state		*arch_mbm_local;
+	struct arch_mbm_state		*arch_mbm_states[QOS_NUM_MBM_EVENTS];
 };
 
 static inline struct rdt_hw_ctrl_domain *resctrl_to_arch_ctrl_dom(struct rdt_ctrl_domain *r)
fs/resctrl/internal.h +4 -0
index e5976bd52a35..7a65ea02d442 100644
@@ -386,6 +386,10 @@ bool closid_allocated(unsigned int closid);
 
 int resctrl_find_cleanest_closid(void);
 
+int rdt_lookup_evtid_by_name(char *name);
+
+char *rdt_event_name(enum resctrl_event_id evt);
+
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
 
arch/x86/kernel/cpu/resctrl/core.c +23 -22

69 lines not on this page · raw message

arch/x86/kernel/cpu/resctrl/monitor.c +15 -18
index 163174cc0d3e..06623d51d006 100644
@@ -36,7 +36,7 @@ bool rdt_mon_capable;
 /*
  * Global to indicate which monitoring events are enabled.
  */
-unsigned int rdt_mon_features;
+DECLARE_BITMAP(rdt_mon_features, QOS_NUM_EVENTS);
 
 #define CF(cf)	((unsigned long)(1048576 * (cf) + 0.5))
 
@@ -168,19 +168,14 @@ static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_do
 						 u32 rmid,
 						 enum resctrl_event_id eventid)
 {
-	switch (eventid) {
-	case QOS_L3_OCCUP_EVENT_ID:
+	struct arch_mbm_state *state;
+
+	if (!resctrl_is_mbm_event(eventid))
 		return NULL;
-	case QOS_L3_MBM_TOTAL_EVENT_ID:
-		return &hw_dom->arch_mbm_total[rmid];
-	case QOS_L3_MBM_LOCAL_EVENT_ID:
-		return &hw_dom->arch_mbm_local[rmid];
-	}
 
-	/* Never expect to get here */
-	WARN_ON_ONCE(1);
+	state = hw_dom->arch_mbm_states[MBM_EVENT_IDX(eventid)];
 
-	return NULL;
+	return state ? &state[rmid] : NULL;
 }
 
 void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
@@ -209,14 +204,16 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
 void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *d)
 {
 	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
+	int evt, idx;
+
+	for_each_set_bit(evt, rdt_mon_features, QOS_NUM_EVENTS) {
+		idx = MBM_EVENT_IDX(evt);
+		if (!hw_dom->arch_mbm_states[idx])
+			continue;
+		memset(hw_dom->arch_mbm_states[idx], 0,
+		       sizeof(struct arch_mbm_state) * r->num_rmid);
+	}
 
-	if (resctrl_arch_is_mbm_total_enabled())
-		memset(hw_dom->arch_mbm_total, 0,
-		       sizeof(*hw_dom->arch_mbm_total) * r->num_rmid);
-
-	if (resctrl_arch_is_mbm_local_enabled())
-		memset(hw_dom->arch_mbm_local, 0,
-		       sizeof(*hw_dom->arch_mbm_local) * r->num_rmid);
 }
 
 static u64 mbm_overflow_count(u64 prev_msr, u64 cur_msr, unsigned int width)
fs/resctrl/ctrlmondata.c +9 -32

64 lines not on this page · raw message

fs/resctrl/monitor.c +44 -26

105 lines not on this page · raw message

fs/resctrl/rdtgroup.c +22 -25

74 lines not on this page · raw message

Signature · 1 line
2.48.1
maintainer
#4 lore raw

Re: [PATCH v3 02/26] fs-x86/resctrl: Prepare for more monitor events

Hi Tony,

Quoted 10 lines · Tony Luck

There's a rule in computer programming that objects appear zero, once, or many times. So code accordingly.

There are two MBM events and resctrl is coded with a lot of

if (local)
do one thing
if (total)
do a different thing

first change:

Quoted 2 lines

Simplify the code by coding for many events using loops on which are enabled.

Please elaborate on how the primary change is the change in data structure and that is what enables loops to be used.

second change:

Quoted 2 lines

Make rdt_mon_features a bitmap to allow for expansion.

... and then a third change: Introduce rdt_lookup_evtid_by_name() and rdt_event_name().

I recognize three logical changes. Could you please split this patch?

Quoted 2 lines

Move resctrl_is_mbm_event() to <asm/resctrl.h> as it gets used by core.c

What the patch actually does is move resctrl_is_mbm_event() to include/linux/resctrl_types.h that is in itself unexpected considering what resctrl_types.h is intended to be used for. See details in changelog of commit

f16adbaf9272 ("x86/resctrl: Move resctrl types to a separate header")

for details on how resctrl_types.h is intended to be used.

Quoted 15 lines
---
 include/linux/resctrl.h                |  6 +--
 include/linux/resctrl_types.h          |  8 +++
 arch/x86/include/asm/resctrl.h         |  8 +--
 arch/x86/kernel/cpu/resctrl/internal.h |  6 +--
 fs/resctrl/internal.h                  |  4 ++
 arch/x86/kernel/cpu/resctrl/core.c     | 45 +++++++++--------
 arch/x86/kernel/cpu/resctrl/monitor.c  | 33 ++++++------
 fs/resctrl/ctrlmondata.c               | 41 ++++-----------
 fs/resctrl/monitor.c                   | 70 ++++++++++++++++----------
 fs/resctrl/rdtgroup.c                  | 47 ++++++++---------
 10 files changed, 133 insertions(+), 135 deletions(-)

...

Quoted 6 lines
include/linux/resctrl_types.h +0 -0
index a7faf2cd5406..898068a99ef7 100644
@@ -55,5 +55,13 @@ enum resctrl_event_id {
 };

include/linux/resctrl_types.h enum resctrl_event_id { · line 65

Quoted 10 lines
include/linux/resctrl_types.h +8 -0
 
 #define QOS_NUM_EVENTS		(QOS_L3_MBM_LOCAL_EVENT_ID + 1)
+#define QOS_NUM_MBM_EVENTS	(QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1)
+#define MBM_EVENT_IDX(evt)	((evt) - QOS_L3_MBM_TOTAL_EVENT_ID)
+
+static inline bool resctrl_is_mbm_event(int e)
+{
+	return (e >= QOS_L3_MBM_TOTAL_EVENT_ID &&
+		e <= QOS_L3_MBM_LOCAL_EVENT_ID);
+}

include/linux/resctrl.h should be a better fit.

Quoted 2 lines
include/linux/resctrl_types.h +0 -0
 #endif /* __LINUX_RESCTRL_TYPES_H */

...

Quoted 8 lines

static int get_domain_id_from_scope(int cpu, enum resctrl_scope scope)

include/linux/resctrl_types.h +1 -1
@@ -864,13 +865,13 @@ static __init bool get_rdt_mon_resources(void)
 	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_L3].r_resctrl;
 
 	if (rdt_cpu_has(X86_FEATURE_CQM_OCCUP_LLC))
-		rdt_mon_features |= (1 << QOS_L3_OCCUP_EVENT_ID);
+		__set_bit(QOS_L3_OCCUP_EVENT_ID, rdt_mon_features);

include/linux/resctrl_types.h static __init bool get_rdt_mon_resources(void) · line 875

Quoted 10 lines
include/linux/resctrl_types.h +3 -3
 	if (rdt_cpu_has(X86_FEATURE_CQM_MBM_TOTAL))
-		rdt_mon_features |= (1 << QOS_L3_MBM_TOTAL_EVENT_ID);
+		__set_bit(QOS_L3_MBM_TOTAL_EVENT_ID, rdt_mon_features);
 	if (rdt_cpu_has(X86_FEATURE_CQM_MBM_LOCAL))
-		rdt_mon_features |= (1 << QOS_L3_MBM_LOCAL_EVENT_ID);
+		__set_bit(QOS_L3_MBM_LOCAL_EVENT_ID, rdt_mon_features);
 
-	if (!rdt_mon_features)
+	if (find_first_bit(rdt_mon_features, QOS_NUM_EVENTS) == QOS_NUM_EVENTS)
 		return false;

Could you please use bitmap_empty() instead? It does the same, but makes it obvious what is being tested for.

Quoted 41 lines
include/linux/resctrl_types.h +0 -0
 	return !rdt_get_mon_l3_config(r);
arch/x86/kernel/cpu/resctrl/monitor.c +6 -11
index 163174cc0d3e..06623d51d006 100644
@@ -36,7 +36,7 @@ bool rdt_mon_capable;
 /*
  * Global to indicate which monitoring events are enabled.
  */
-unsigned int rdt_mon_features;
+DECLARE_BITMAP(rdt_mon_features, QOS_NUM_EVENTS);
 
 #define CF(cf)	((unsigned long)(1048576 * (cf) + 0.5))
 
@@ -168,19 +168,14 @@ static struct arch_mbm_state *get_arch_mbm_state(struct rdt_hw_mon_domain *hw_do
 						 u32 rmid,
 						 enum resctrl_event_id eventid)
 {
-	switch (eventid) {
-	case QOS_L3_OCCUP_EVENT_ID:
+	struct arch_mbm_state *state;
+
+	if (!resctrl_is_mbm_event(eventid))
 		return NULL;
-	case QOS_L3_MBM_TOTAL_EVENT_ID:
-		return &hw_dom->arch_mbm_total[rmid];
-	case QOS_L3_MBM_LOCAL_EVENT_ID:
-		return &hw_dom->arch_mbm_local[rmid];
-	}
 
-	/* Never expect to get here */
-	WARN_ON_ONCE(1);
+	state = hw_dom->arch_mbm_states[MBM_EVENT_IDX(eventid)];
 
-	return NULL;
+	return state ? &state[rmid] : NULL;
 }
 
 void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,

arch/x86/kernel/cpu/resctrl/monitor.c void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d, · line 212

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/monitor.c +6 -0
@@ -209,14 +204,16 @@ void resctrl_arch_reset_rmid(struct rdt_resource *r, struct rdt_mon_domain *d,
 void resctrl_arch_reset_rmid_all(struct rdt_resource *r, struct rdt_mon_domain *d)
 {
 	struct rdt_hw_mon_domain *hw_dom = resctrl_to_arch_mon_dom(d);
+	int evt, idx;
+
+	for_each_set_bit(evt, rdt_mon_features, QOS_NUM_EVENTS) {
+		idx = MBM_EVENT_IDX(evt);
+		if (!hw_dom->arch_mbm_states[idx])
+			continue;

This does not look safe. Missing a resctrl_is_mbm_event() check?

Quoted 13 lines
arch/x86/kernel/cpu/resctrl/monitor.c +3 -7
+		memset(hw_dom->arch_mbm_states[idx], 0,
+		       sizeof(struct arch_mbm_state) * r->num_rmid);
+	}
 
-	if (resctrl_arch_is_mbm_total_enabled())
-		memset(hw_dom->arch_mbm_total, 0,
-		       sizeof(*hw_dom->arch_mbm_total) * r->num_rmid);
-
-	if (resctrl_arch_is_mbm_local_enabled())
-		memset(hw_dom->arch_mbm_local, 0,
-		       sizeof(*hw_dom->arch_mbm_local) * r->num_rmid);
 }
 

...

Quoted 27 lines
fs/resctrl/monitor.c +6 -7
index 3fe21dcf0fde..66e613906f3e 100644
@@ -347,15 +347,14 @@ static struct mbm_state *get_mbm_state(struct rdt_mon_domain *d, u32 closid,
 				       u32 rmid, enum resctrl_event_id evtid)
 {
 	u32 idx = resctrl_arch_rmid_idx_encode(closid, rmid);
+	struct mbm_state *states;
 
-	switch (evtid) {
-	case QOS_L3_MBM_TOTAL_EVENT_ID:
-		return &d->mbm_total[idx];
-	case QOS_L3_MBM_LOCAL_EVENT_ID:
-		return &d->mbm_local[idx];
-	default:
+	if (!resctrl_is_mbm_event(evtid))
 		return NULL;
-	}
+
+	states = d->mbm_states[MBM_EVENT_IDX(evtid)];
+
+	return states ? &states[idx] : NULL;
 }
 
 static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr)

fs/resctrl/monitor.c static void dom_data_exit(struct rdt_resource *r) · line 845

Quoted 7 lines
fs/resctrl/monitor.c +1 -3
@@ -843,20 +842,40 @@ static void dom_data_exit(struct rdt_resource *r)
 	mutex_unlock(&rdtgroup_mutex);
 }
 
-static struct mon_evt llc_occupancy_event = {
-	.name		= "llc_occupancy",
-	.evtid		= QOS_L3_OCCUP_EVENT_ID,
+static struct mon_evt all_events[QOS_NUM_EVENTS] = {

"all_events" is very generic for a global. How about "mon_event_all" (placing the "_all" at end to match, for example, resctrl_schema_all).

Quoted 11 lines
fs/resctrl/monitor.c +11 -0
+	[QOS_L3_OCCUP_EVENT_ID] = {
+		.name	= "llc_occupancy",
+		.evtid	= QOS_L3_OCCUP_EVENT_ID,
+	},
+	[QOS_L3_MBM_TOTAL_EVENT_ID] = {
+		.name	= "mbm_total_bytes",
+		.evtid	= QOS_L3_MBM_TOTAL_EVENT_ID,
+	},
+	[QOS_L3_MBM_LOCAL_EVENT_ID] = {
+		.name	= "mbm_local_bytes",
+		.evtid	= QOS_L3_MBM_LOCAL_EVENT_ID,

fs/resctrl/monitor.c static void dom_data_exit(struct rdt_resource *r) · line 862

Quoted 10 lines
fs/resctrl/monitor.c +4 -4
+	},
 };
 
-static struct mon_evt mbm_total_event = {
-	.name		= "mbm_total_bytes",
-	.evtid		= QOS_L3_MBM_TOTAL_EVENT_ID,
-};
+int rdt_lookup_evtid_by_name(char *name)
+{
+	int evt;

Since this is resctrl fs code, please replace "rdt" with "resctrl".

fs/resctrl/monitor.c static void dom_data_exit(struct rdt_resource *r) · line 867

Quoted 9 lines
fs/resctrl/monitor.c +4 -4
-static struct mon_evt mbm_local_event = {
-	.name		= "mbm_local_bytes",
-	.evtid		= QOS_L3_MBM_LOCAL_EVENT_ID,
-};
+	for_each_set_bit(evt, rdt_mon_features, QOS_NUM_EVENTS) {
+		if (!strcmp(name, all_events[evt].name))
+			return evt;
+	}

* This is resctrl fs code. rdt_mon_features should be private to x86 so resctrl fs should not be accessing it directly. Perhaps there can be a new arch helper that can be used by resctrl to query if event is enabled? Similar to resctrl_arch_is_llc_occupancy_enabled() and friend but where the event ID is parameter and arch code can use rdt_mon_features.
* While the function name is "lookup_evtid_by_name" this function does not just look up the event id by name but also ensures that the event is enabled. The caller then uses "lookup_evtid_by_name" as a proxy for "is this event enabled". I think the code will be easier to understand if the functions do not have such hidden "features". resctrl fs could first use new arch helper to determine if
event is enabled and then use a fs helper that reads name directly from the event array.
* For a function returning event id the return type is expected to be enum resctrl_event_id.

fs/resctrl/monitor.c static void dom_data_exit(struct rdt_resource *r) · line 878

Quoted 11 lines
fs/resctrl/monitor.c +11 -0
+
+	return -EINVAL;
+}
+
+char *rdt_event_name(enum resctrl_event_id evt)
+{
+	if (!test_bit(evt, rdt_mon_features))
+		return "unknown";
+
+	return all_events[evt].name;
+}

Same comments as rdt_lookup_evtid_by_name()

Also please watch for rdt_mon_features in rest of resctrl fs changes.

Reinette

author maintainer
#5 lore raw

PATCH v3 3/26 fs/resctrl: Change how events are initialized

In reply to Tony Luck

New monitor events break some assumptions:

1) New events can be in resources other than L3.
2) Enumeration of events may not be complete during early boot.

Prepare for events in other resources.

Delay building the event lists until first mount of the resctrl file system.

---
 fs/resctrl/internal.h |  3 +++
 fs/resctrl/monitor.c  | 30 +++++++++++++++++++-----------
 fs/resctrl/rdtgroup.c |  2 ++
 3 files changed, 24 insertions(+), 11 deletions(-)
fs/resctrl/internal.h +3 -0
index 7a65ea02d442..08dbf89939ac 100644
@@ -76,6 +76,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
  */
 struct mon_evt {
 	enum resctrl_event_id	evtid;
+	enum resctrl_res_level	rid;
 	char			*name;
 	bool			configurable;
 	struct list_head	list;
@@ -390,6 +391,8 @@ int rdt_lookup_evtid_by_name(char *name);
 
 char *rdt_event_name(enum resctrl_event_id evt);
 
+void resctrl_init_mon_events(void);
+
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
 
fs/resctrl/monitor.c +19 -11
index 66e613906f3e..472754d082cb 100644
@@ -846,14 +846,17 @@ static struct mon_evt all_events[QOS_NUM_EVENTS] = {
 	[QOS_L3_OCCUP_EVENT_ID] = {
 		.name	= "llc_occupancy",
 		.evtid	= QOS_L3_OCCUP_EVENT_ID,
+		.rid	= RDT_RESOURCE_L3,
 	},
 	[QOS_L3_MBM_TOTAL_EVENT_ID] = {
 		.name	= "mbm_total_bytes",
 		.evtid	= QOS_L3_MBM_TOTAL_EVENT_ID,
+		.rid	= RDT_RESOURCE_L3,
 	},
 	[QOS_L3_MBM_LOCAL_EVENT_ID] = {
 		.name	= "mbm_local_bytes",
 		.evtid	= QOS_L3_MBM_LOCAL_EVENT_ID,
+		.rid	= RDT_RESOURCE_L3,
 	},
 };
 
@@ -878,22 +881,29 @@ char *rdt_event_name(enum resctrl_event_id evt)
 }
 
 /*
- * Initialize the event list for the resource.
+ * Initialize the event list for all mon_capable resources.
  *
- * Note that MBM events are also part of RDT_RESOURCE_L3 resource
- * because as per the SDM the total and local memory bandwidth
- * are enumerated as part of L3 monitoring.
- *
- * mon_put_default_kn_priv_all() also assumes monitor events are only supported
- * on the L3 resource.
+ * Called on each mount of the resctrl file system when all
+ * events have been enumerated. Only needs to build the per-resource
+ * event lists once.
  */
-static void l3_mon_evt_init(struct rdt_resource *r)
+void resctrl_init_mon_events(void)
 {
+	struct rdt_resource *r;
+	static bool only_once;
 	int evt;
 
-	INIT_LIST_HEAD(&r->evt_list);
+	if (only_once)
+		return;
+	only_once = true;
+
+	for_each_mon_capable_rdt_resource(r)
+		INIT_LIST_HEAD(&r->evt_list);
 
 	for_each_set_bit(evt, rdt_mon_features, QOS_NUM_EVENTS) {
+		r = resctrl_arch_get_resource(all_events[evt].rid);
+		if (!r->mon_capable)
+			continue;
 		list_add_tail(&all_events[evt].list, &r->evt_list);
 	}
 }
@@ -922,8 +932,6 @@ int resctrl_mon_resource_init(void)
 	if (ret)
 		return ret;
 
-	l3_mon_evt_init(r);
-
 	if (resctrl_arch_is_evt_configurable(QOS_L3_MBM_TOTAL_EVENT_ID)) {
 		all_events[QOS_L3_MBM_TOTAL_EVENT_ID].configurable = true;
 		resctrl_file_fflags_init("mbm_total_bytes_config",
fs/resctrl/rdtgroup.c +2 -0
index 8d15d53fae76..1433fc098a90 100644
@@ -2574,6 +2574,8 @@ static int rdt_get_tree(struct fs_context *fc)
 		goto out;
 	}
 
+	resctrl_init_mon_events();
+
 	ret = rdtgroup_setup_root(ctx);
 	if (ret)
 		goto out;
Signature · 1 line
2.48.1
maintainer
#6 lore raw

Re: [PATCH v3 03/26] fs/resctrl: Change how events are initialized

Hi Tony,

Quoted 1 line · Tony Luck

New monitor events break some assumptions:

This changelog jumps into a problem without any context.

Please follow changelog guidance from maintainer-tip.rst. Specifically, "A good structure is to explain the context, the problem and the solution in separate paragraphs and this order."

Quoted 6 lines

1) New events can be in resources other than L3.
2) Enumeration of events may not be complete during early boot.

Prepare for events in other resources.

Please include what this preparation involves.

Quoted 3 lines

Delay building the event lists until first mount of the resctrl file system.

Please include in context what is meant by "event lists".

But ... previous patch reminded reader about all the event state that is allocated during domain online, which usually happens *before* mount of resctrl. This work thus goes from "use enumeration of events during boot to allocate necessary event state" in one patch to "enumeration of events are not complete during boot so build event lists on resctrl mount" in the next patch. This is a big contradiction to me.

I think it is clear that not all events can be treated equally but this implementation pretends to treat them equally when convenient (this patch) and relies on code flow assumptions (previous patch that only allocated state for L3 events during domain online) for things to "work out" in the end.

Quoted 13 lines
---
 fs/resctrl/internal.h |  3 +++
 fs/resctrl/monitor.c  | 30 +++++++++++++++++++-----------
 fs/resctrl/rdtgroup.c |  2 ++
 3 files changed, 24 insertions(+), 11 deletions(-)
fs/resctrl/internal.h +0 -0
index 7a65ea02d442..08dbf89939ac 100644

fs/resctrl/internal.h static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc) · line 79

Quoted 4 lines
fs/resctrl/internal.h +1 -0
@@ -76,6 +76,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
  */
 struct mon_evt {
 	enum resctrl_event_id	evtid;
+	enum resctrl_res_level	rid;

This structure has some kernel-doc that is not visible in this hunk but also needs a change when adding a new member.

Quoted 57 lines
fs/resctrl/internal.h +2 -0
 	char			*name;
 	bool			configurable;
 	struct list_head	list;
@@ -390,6 +391,8 @@ int rdt_lookup_evtid_by_name(char *name);
 
 char *rdt_event_name(enum resctrl_event_id evt);
 
+void resctrl_init_mon_events(void);
+
 #ifdef CONFIG_RESCTRL_FS_PSEUDO_LOCK
 int rdtgroup_locksetup_enter(struct rdtgroup *rdtgrp);
 
fs/resctrl/monitor.c +10 -8
index 66e613906f3e..472754d082cb 100644
@@ -846,14 +846,17 @@ static struct mon_evt all_events[QOS_NUM_EVENTS] = {
 	[QOS_L3_OCCUP_EVENT_ID] = {
 		.name	= "llc_occupancy",
 		.evtid	= QOS_L3_OCCUP_EVENT_ID,
+		.rid	= RDT_RESOURCE_L3,
 	},
 	[QOS_L3_MBM_TOTAL_EVENT_ID] = {
 		.name	= "mbm_total_bytes",
 		.evtid	= QOS_L3_MBM_TOTAL_EVENT_ID,
+		.rid	= RDT_RESOURCE_L3,
 	},
 	[QOS_L3_MBM_LOCAL_EVENT_ID] = {
 		.name	= "mbm_local_bytes",
 		.evtid	= QOS_L3_MBM_LOCAL_EVENT_ID,
+		.rid	= RDT_RESOURCE_L3,
 	},
 };
 
@@ -878,22 +881,29 @@ char *rdt_event_name(enum resctrl_event_id evt)
 }
 
 /*
- * Initialize the event list for the resource.
+ * Initialize the event list for all mon_capable resources.
  *
- * Note that MBM events are also part of RDT_RESOURCE_L3 resource
- * because as per the SDM the total and local memory bandwidth
- * are enumerated as part of L3 monitoring.
- *
- * mon_put_default_kn_priv_all() also assumes monitor events are only supported
- * on the L3 resource.
+ * Called on each mount of the resctrl file system when all
+ * events have been enumerated. Only needs to build the per-resource
+ * event lists once.
  */
-static void l3_mon_evt_init(struct rdt_resource *r)
+void resctrl_init_mon_events(void)
 {
+	struct rdt_resource *r;
+	static bool only_once;
 	int evt;

fs/resctrl/monitor.c char *rdt_event_name(enum resctrl_event_id evt) · line 903

Quoted 10 lines
fs/resctrl/monitor.c +6 -1
 
-	INIT_LIST_HEAD(&r->evt_list);
+	if (only_once)
+		return;
+	only_once = true;
+
+	for_each_mon_capable_rdt_resource(r)
+		INIT_LIST_HEAD(&r->evt_list);
 
 	for_each_set_bit(evt, rdt_mon_features, QOS_NUM_EVENTS) {

This is fs code so this needs to be done without peeking into rdt_mon_features.

Quoted 20 lines
fs/resctrl/monitor.c +3 -2
+		r = resctrl_arch_get_resource(all_events[evt].rid);
+		if (!r->mon_capable)
+			continue;
 		list_add_tail(&all_events[evt].list, &r->evt_list);
 	}
 }
@@ -922,8 +932,6 @@ int resctrl_mon_resource_init(void)
 	if (ret)
 		return ret;
 
-	l3_mon_evt_init(r);
-
 	if (resctrl_arch_is_evt_configurable(QOS_L3_MBM_TOTAL_EVENT_ID)) {
 		all_events[QOS_L3_MBM_TOTAL_EVENT_ID].configurable = true;
 		resctrl_file_fflags_init("mbm_total_bytes_config",
fs/resctrl/rdtgroup.c +0 -0
index 8d15d53fae76..1433fc098a90 100644

fs/resctrl/rdtgroup.c static int rdt_get_tree(struct fs_context *fc) · line 2581

Quoted 8 lines
fs/resctrl/rdtgroup.c +2 -0
@@ -2574,6 +2574,8 @@ static int rdt_get_tree(struct fs_context *fc)
 		goto out;
 	}
 
+	resctrl_init_mon_events();
+
 	ret = rdtgroup_setup_root(ctx);
 	if (ret)
 		goto out;

Reinette

author maintainer
#7 lore raw

PATCH v3 4/26 fs/resctrl: Set up Kconfig options for telemetry events

In reply to Tony Luck

Enumeration support is provided by the Intel PMT subsystem.

arch/x86 selects this option based on:

X86_64: Counter registers are in MMIO space. There is no readq() function on 32-bit. Emulation is possible with readl(), but there are races. Running 32-bit kernels on systems that support this feature seems pointless.

CPU_SUP_INTEL: It is an Intel specific feature.

---
 arch/x86/Kconfig                       | 1 +
 drivers/platform/x86/intel/pmt/Kconfig | 6 ++++++
 2 files changed, 7 insertions(+)
arch/x86/Kconfig +1 -0
index ea29d22a621f..44a195ee7a42 100644
@@ -504,6 +504,7 @@ config X86_CPU_RESCTRL
 	bool "x86 CPU resource control support"
 	depends on X86 && (CPU_SUP_INTEL || CPU_SUP_AMD)
 	depends on MISC_FILESYSTEMS
+	select INTEL_AET_RESCTRL if (X86_64 && CPU_SUP_INTEL)
 	select ARCH_HAS_CPU_RESCTRL
 	select RESCTRL_FS
 	select RESCTRL_FS_PSEUDO_LOCK
drivers/platform/x86/intel/pmt/Kconfig +6 -0
index e916fc966221..b282910b49ef 100644
@@ -38,3 +38,9 @@ config INTEL_PMT_CRASHLOG
 
 	  To compile this driver as a module, choose M here: the module
 	  will be called intel_pmt_crashlog.
+
+config INTEL_AET_RESCTRL
+	bool
+	help
+	  Architecture config should "select" this option to enable
+	  support for RMID telemtry events in the resctrl file system.
Signature · 1 line
2.48.1
maintainer
#8 lore raw

Re: [PATCH v3 04/26] fs/resctrl: Set up Kconfig options for telemetry events

Hi Tony,

Quoted 1 line · Tony Luck

Enumeration support is provided by the Intel PMT subsystem.

Needs context before jumping in with solution.

Quoted 33 lines

arch/x86 selects this option based on:

X86_64: Counter registers are in MMIO space. There is no readq() function on 32-bit. Emulation is possible with readl(), but there are races. Running 32-bit kernels on systems that support this feature seems pointless.

CPU_SUP_INTEL: It is an Intel specific feature.

---
 arch/x86/Kconfig                       | 1 +
 drivers/platform/x86/intel/pmt/Kconfig | 6 ++++++
 2 files changed, 7 insertions(+)
arch/x86/Kconfig +1 -0
index ea29d22a621f..44a195ee7a42 100644
@@ -504,6 +504,7 @@ config X86_CPU_RESCTRL
 	bool "x86 CPU resource control support"
 	depends on X86 && (CPU_SUP_INTEL || CPU_SUP_AMD)
 	depends on MISC_FILESYSTEMS
+	select INTEL_AET_RESCTRL if (X86_64 && CPU_SUP_INTEL)
 	select ARCH_HAS_CPU_RESCTRL
 	select RESCTRL_FS
 	select RESCTRL_FS_PSEUDO_LOCK
drivers/platform/x86/intel/pmt/Kconfig +0 -0
index e916fc966221..b282910b49ef 100644

drivers/platform/x86/intel/pmt/Kconfig config INTEL_PMT_CRASHLOG · line 43

Quoted 6 lines
drivers/platform/x86/intel/pmt/Kconfig +3 -0
@@ -38,3 +38,9 @@ config INTEL_PMT_CRASHLOG
 
 	  To compile this driver as a module, choose M here: the module
 	  will be called intel_pmt_crashlog.
+
+config INTEL_AET_RESCTRL
+	bool

Usually these settings come with some "depends on" ... no dependencies needed?

drivers/platform/x86/intel/pmt/Kconfig config INTEL_PMT_CRASHLOG · line 46

Quoted 3 lines
drivers/platform/x86/intel/pmt/Kconfig +3 -0
+	help
+	  Architecture config should "select" this option to enable
+	  support for RMID telemtry events in the resctrl file system.

telemtry -> telemetry

Please define what is meant with "RMID telemetry events" before this statement.

Reinette

author maintainer
#9 lore raw

PATCH v3 5/26 x86/rectrl: Fake OOBMSM interface

In reply to Tony Luck

Real version is coming soon ... this is here so the remaining parts will build (and run ... assuming a 2 socket system that supports RDT monitoring ... only missing part is that the event counters just report fixed values).

Just for ease of testing and RFC discussion.

---
 .../cpu/resctrl/fake_intel_aet_features.h     | 73 ++++++++++++++++
 .../cpu/resctrl/fake_intel_aet_features.c     | 87 +++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/Makefile          |  1 +
 3 files changed, 161 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.h
 create mode 100644 arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.h new file +73 -0
new file mode 100644
index 000000000000..c835c4108abc
@@ -0,0 +1,73 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/* Bits stolen from OOBMSM VSEC discovery code */
+
+enum pmt_feature_id {
+	FEATURE_INVALID			= 0x0,
+	FEATURE_PER_CORE_PERF_TELEM	= 0x1,
+	FEATURE_PER_CORE_ENV_TELEM	= 0x2,
+	FEATURE_PER_RMID_PERF_TELEM	= 0x3,
+	FEATURE_ACCEL_TELEM		= 0x4,
+	FEATURE_UNCORE_TELEM		= 0x5,
+	FEATURE_CRASH_LOG		= 0x6,
+	FEATURE_PETE_LOG		= 0x7,
+	FEATURE_TPMI_CTRL		= 0x8,
+	FEATURE_RESERVED		= 0x9,
+	FEATURE_TRACING			= 0xA,
+	FEATURE_PER_RMID_ENERGY_TELEM	= 0xB,
+	FEATURE_MAX			= 0xB,
+};
+
+/**
+ * struct oobmsm_plat_info - Platform information for a device instance
+ * @cdie_mask:       Mask of all compute dies in the partition
+ * @package_id:      CPU Package id
+ * @partition:       Package partition id when multiple VSEC PCI devices per package
+ * @segment:         PCI segment ID
+ * @bus_number:      PCI bus number
+ * @device_number:   PCI device number
+ * @function_number: PCI function number
+ *
+ * Structure to store platform data for a OOBMSM device instance.
+ */
+struct oobmsm_plat_info {
+	u16 cdie_mask;
+	u8 package_id;
+	u8 partition;
+	u8 segment;
+	u8 bus_number;
+	u8 device_number;
+	u8 function_number;
+};
+
+enum oobmsm_supplier_type {
+	OOBMSM_SUP_PLAT_INFO,
+	OOBMSM_SUP_DISC_INFO,
+	OOBMSM_SUP_S3M_SIMICS,
+	OOBMSM_SUP_TYPE_MAX
+};
+
+struct oobmsm_mapping_supplier {
+	struct device *supplier_dev[OOBMSM_SUP_TYPE_MAX];
+	struct oobmsm_plat_info plat_info;
+	unsigned long features;
+};
+
+struct telemetry_region {
+	struct oobmsm_plat_info	plat_info;
+	void __iomem		*addr;
+	size_t			size;
+	u32			guid;
+	u32			num_rmids;
+};
+
+struct pmt_feature_group {
+	enum pmt_feature_id	id;
+	int			count;
+	struct kref		kref;
+	struct telemetry_region	regions[];
+};
+
+struct pmt_feature_group *intel_pmt_get_regions_by_feature(enum pmt_feature_id id);
+
+void intel_pmt_put_feature_group(struct pmt_feature_group *feature_group);
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c new file +87 -0

87 lines not on this page · raw message

arch/x86/kernel/cpu/resctrl/Makefile +1 -0
index 909be78ec6da..c56d3acf8ac7 100644
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_X86_CPU_RESCTRL)		+= core.o rdtgroup.o monitor.o
 obj-$(CONFIG_X86_CPU_RESCTRL)		+= ctrlmondata.o
+obj-$(CONFIG_INTEL_AET_RESCTRL)		+= fake_intel_aet_features.o
 obj-$(CONFIG_RESCTRL_FS_PSEUDO_LOCK)	+= pseudo_lock.o
 
 # To allow define_trace.h's recursive include:
Signature · 1 line
2.48.1
maintainer
#10 lore raw

Re: [PATCH v3 05/26] x86/rectrl: Fake OOBMSM interface

Hi Tony:

(deja vu ... lets try this comment again ... x86/rectrl -> x86/resctrl)

On 4/7/25 4:40 PM, Tony Luck wrote:

...

Quoted 11 lines
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c new file +5 -0
new file mode 100644
index 000000000000..5a16db67c7b8
@@ -0,0 +1,87 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/cleanup.h>
+#include <linux/minmax.h>
+#include <linux/slab.h>
+#include "fake_intel_aet_features.h"

arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c line 15

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c new file +10 -0
+#include <linux/intel_vsec.h>
+#include <linux/resctrl.h>
+
+#include "internal.h"
+
+/* Amount of memory for each fake MMIO space */
+#define ENERGY_QWORDS	((576 * 2) + 3)
+#define ENERGY_SIZE	(ENERGY_QWORDS * 8)
+#define PERF_QWORDS	((576 * 7) + 3)
+#define PERF_SIZE	(PERF_QWORDS * 8)

Could you please add explanations for the magic numbers? For example, why are both energy and perf using 576?

Quoted 54 lines
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c new file +54 -0
+
+static long pg[4 * ENERGY_QWORDS + 2 * PERF_QWORDS];
+
+/*
+ * Fill the fake MMIO space with all different values,
+ * all with BIT(63) set to indicate valid entries.
+ */
+static int __init fill(void)
+{
+	u64 val = 0;
+
+	for (int i = 0; i < sizeof(pg); i += sizeof(val)) {
+		pg[i / sizeof(val)] = BIT_ULL(63) + val;
+		val++;
+	}
+	return 0;
+}
+device_initcall(fill);
+
+#define PKG_REGION(_entry, _guid, _addr, _size, _pkg, _num_rmids)	\
+	[_entry] = { .guid = _guid, .addr = (void __iomem *)_addr, \
+		     .num_rmids = _num_rmids, \
+		     .size = _size, .plat_info = { .package_id = _pkg }}
+
+/*
+ * Set up a fake return for call to:
+ *   intel_pmt_get_regions_by_feature(FEATURE_PER_RMID_ENERGY_TELEM);
+ * Pretend there are two aggregators on each of the sockets to test
+ * the code that sums over multiple aggregators.
+ */
+static struct pmt_feature_group fake_energy = {
+	.count = 4,
+	.regions = {
+		PKG_REGION(0, 0x26696143, &pg[0 * ENERGY_QWORDS], ENERGY_SIZE, 0, 64),
+		PKG_REGION(1, 0x26696143, &pg[1 * ENERGY_QWORDS], ENERGY_SIZE, 0, 64),
+		PKG_REGION(2, 0x26696143, &pg[2 * ENERGY_QWORDS], ENERGY_SIZE, 1, 64),
+		PKG_REGION(3, 0x26696143, &pg[3 * ENERGY_QWORDS], ENERGY_SIZE, 1, 64)
+	}
+};
+
+/*
+ * Fake return for:
+ *   intel_pmt_get_regions_by_feature(FEATURE_PER_RMID_PERF_TELEM);
+ */
+static struct pmt_feature_group fake_perf = {
+	.count = 2,
+	.regions = {
+		PKG_REGION(0, 0x26557651, &pg[4 * ENERGY_QWORDS + 0 * PERF_QWORDS], PERF_SIZE, 0, 576),
+		PKG_REGION(1, 0x26557651, &pg[4 * ENERGY_QWORDS + 1 * PERF_QWORDS], PERF_SIZE, 1, 576)
+	}
+};
+
+struct pmt_feature_group *
+intel_pmt_get_regions_by_feature(enum pmt_feature_id id)

arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c line 79

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c new file +10 -0
+{
+	switch (id) {
+	case FEATURE_PER_RMID_ENERGY_TELEM:
+		return &fake_energy;
+	case FEATURE_PER_RMID_PERF_TELEM:
+		return &fake_perf;
+	default:
+		return ERR_PTR(-ENOENT);
+	}
+	return ERR_PTR(-ENOENT);

Not reachable.

Quoted 13 lines
arch/x86/kernel/cpu/resctrl/fake_intel_aet_features.c new file +8 -0
+}
+
+/*
+ * Nothing needed for the "put" function.
+ */
+void intel_pmt_put_feature_group(struct pmt_feature_group *feature_group)
+{
+}
arch/x86/kernel/cpu/resctrl/Makefile +0 -0
index 909be78ec6da..c56d3acf8ac7 100644

arch/x86/kernel/cpu/resctrl/Makefile line 7

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/Makefile +1 -0
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_X86_CPU_RESCTRL)		+= core.o rdtgroup.o monitor.o
 obj-$(CONFIG_X86_CPU_RESCTRL)		+= ctrlmondata.o
+obj-$(CONFIG_INTEL_AET_RESCTRL)		+= fake_intel_aet_features.o
 obj-$(CONFIG_RESCTRL_FS_PSEUDO_LOCK)	+= pseudo_lock.o
 
 # To allow define_trace.h's recursive include:

Reinette

author maintainer
#11 lore raw

PATCH v3 6/26 fs-x86/rectrl: Improve domain type checking

In reply to Tony Luck

The rdt_domain_hdr structure is used in both control and monitor domain structures to provide common methods for operations such as adding a CPU to a domain, removing a CPU from a domain, accessing the mask of all CPUs in a domain.

The "type" field provides a simple check whether a domain is a control or monitor domain so that programming errors operating on domains will be quickly caught.

To prepare for additional domain types that depend on the rdt_resource to which they are connected add the resource id into the type.

---
 include/linux/resctrl.h            | 19 ++++++++++---------
 arch/x86/kernel/cpu/resctrl/core.c | 12 ++++++------
 fs/resctrl/ctrlmondata.c           |  2 +-
 3 files changed, 17 insertions(+), 16 deletions(-)
include/linux/resctrl.h +10 -9
index d6a926b6fc0e..177f9879bae1 100644
@@ -111,11 +111,6 @@ struct resctrl_staged_config {
 	bool			have_new_ctrl;
 };
 
-enum resctrl_domain_type {
-	RESCTRL_CTRL_DOMAIN,
-	RESCTRL_MON_DOMAIN,
-};
-
 /**
  * struct rdt_domain_hdr - common header for different domain types
  * @list:		all instances of this resource
@@ -124,12 +119,18 @@ enum resctrl_domain_type {
  * @cpu_mask:		which CPUs share this resource
  */
 struct rdt_domain_hdr {
-	struct list_head		list;
-	int				id;
-	enum resctrl_domain_type	type;
-	struct cpumask			cpu_mask;
+	struct list_head	list;
+	int			id;
+	u32			type;
+	struct cpumask		cpu_mask;
 };
 
+/* Bitfields in rdt_domain_hdr.type */
+#define	DOMTYPE_RID		GENMASK(8, 0)
+#define DOMTYPE_CTRL		BIT(9)
+#define DOMTYPE_MON		BIT(10)
+#define DOMTYPE(rid, type)	(((rid) & DOMTYPE_RID) | (type))
+
 /**
  * struct rdt_ctrl_domain - group of CPUs sharing a resctrl control resource
  * @hdr:		common header for different domain types
arch/x86/kernel/cpu/resctrl/core.c +6 -6
index 6f4a3bd02a42..d82a4a2db699 100644
@@ -457,7 +457,7 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r)
 
 	hdr = resctrl_find_domain(&r->ctrl_domains, id, &add_pos);
 	if (hdr) {
-		if (WARN_ON_ONCE(hdr->type != RESCTRL_CTRL_DOMAIN))
+		if (WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_CTRL)))
 			return;
 		d = container_of(hdr, struct rdt_ctrl_domain, hdr);
 
@@ -473,7 +473,7 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r)
 
 	d = &hw_dom->d_resctrl;
 	d->hdr.id = id;
-	d->hdr.type = RESCTRL_CTRL_DOMAIN;
+	d->hdr.type = DOMTYPE(r->rid, DOMTYPE_CTRL);
 	cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
 
 	rdt_domain_reconfigure_cdp(r);
@@ -512,7 +512,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 
 	hdr = resctrl_find_domain(&r->mon_domains, id, &add_pos);
 	if (hdr) {
-		if (WARN_ON_ONCE(hdr->type != RESCTRL_MON_DOMAIN))
+		if (WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_MON)))
 			return;
 		d = container_of(hdr, struct rdt_mon_domain, hdr);
 
@@ -526,7 +526,7 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 
 	d = &hw_dom->d_resctrl;
 	d->hdr.id = id;
-	d->hdr.type = RESCTRL_MON_DOMAIN;
+	d->hdr.type = DOMTYPE(r->rid, DOMTYPE_MON);
 	d->ci = get_cpu_cacheinfo_level(cpu, RESCTRL_L3_CACHE);
 	if (!d->ci) {
 		pr_warn_once("Can't find L3 cache for CPU:%d resource %s\n", cpu, r->name);
@@ -582,7 +582,7 @@ static void domain_remove_cpu_ctrl(int cpu, struct rdt_resource *r)
 		return;
 	}
 
-	if (WARN_ON_ONCE(hdr->type != RESCTRL_CTRL_DOMAIN))
+	if (WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_CTRL)))
 		return;
 
 	d = container_of(hdr, struct rdt_ctrl_domain, hdr);
@@ -628,7 +628,7 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
 		return;
 	}
 
-	if (WARN_ON_ONCE(hdr->type != RESCTRL_MON_DOMAIN))
+	if (WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_MON)))
 		return;
 
 	d = container_of(hdr, struct rdt_mon_domain, hdr);
fs/resctrl/ctrlmondata.c +1 -1
index ce02e961a6c3..0c245af0ff42 100644
@@ -614,7 +614,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 		 * the resource to find the domain with "domid".
 		 */
 		hdr = resctrl_find_domain(&r->mon_domains, domid, NULL);
-		if (!hdr || WARN_ON_ONCE(hdr->type != RESCTRL_MON_DOMAIN)) {
+		if (!hdr || WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_MON))) {
 			ret = -ENOENT;
 			goto out;
 		}
Signature · 1 line
2.48.1
maintainer
#12 lore raw

Re: [PATCH v3 06/26] fs-x86/rectrl: Improve domain type checking

Hi Tony,

Quoted 45 lines · Tony Luck

The rdt_domain_hdr structure is used in both control and monitor domain structures to provide common methods for operations such as adding a CPU to a domain, removing a CPU from a domain, accessing the mask of all CPUs in a domain.

The "type" field provides a simple check whether a domain is a control or monitor domain so that programming errors operating on domains will be quickly caught.

To prepare for additional domain types that depend on the rdt_resource to which they are connected add the resource id into the type.

---
 include/linux/resctrl.h            | 19 ++++++++++---------
 arch/x86/kernel/cpu/resctrl/core.c | 12 ++++++------
 fs/resctrl/ctrlmondata.c           |  2 +-
 3 files changed, 17 insertions(+), 16 deletions(-)
include/linux/resctrl.h +2 -9
index d6a926b6fc0e..177f9879bae1 100644
@@ -111,11 +111,6 @@ struct resctrl_staged_config {
 	bool			have_new_ctrl;
 };
 
-enum resctrl_domain_type {
-	RESCTRL_CTRL_DOMAIN,
-	RESCTRL_MON_DOMAIN,
-};
-
 /**
  * struct rdt_domain_hdr - common header for different domain types
  * @list:		all instances of this resource
@@ -124,12 +119,18 @@ enum resctrl_domain_type {
  * @cpu_mask:		which CPUs share this resource
  */
 struct rdt_domain_hdr {
-	struct list_head		list;
-	int				id;
-	enum resctrl_domain_type	type;
-	struct cpumask			cpu_mask;
+	struct list_head	list;
+	int			id;

include/linux/resctrl.h enum resctrl_domain_type { · line 133

Quoted 10 lines
include/linux/resctrl.h +8 -0
+	u32			type;
+	struct cpumask		cpu_mask;
 };
 
+/* Bitfields in rdt_domain_hdr.type */
+#define	DOMTYPE_RID		GENMASK(8, 0)
+#define DOMTYPE_CTRL		BIT(9)
+#define DOMTYPE_MON		BIT(10)
+#define DOMTYPE(rid, type)	(((rid) & DOMTYPE_RID) | (type))
+

This seems unnecessarily complicated to me. Why not just add the resource id to the domain header?

Reinette

author maintainer
#13 lore raw

PATCH v3 7/26 x86/resctrl: Move L3 initialization out of domain_add_cpu_mon()

In reply to Tony Luck

To prepare for additional types of monitoring domains, move all the L3 specific initialization into a helper function.

---
 arch/x86/kernel/cpu/resctrl/core.c | 57 ++++++++++++++++++------------
 1 file changed, 35 insertions(+), 22 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +35 -22
index d82a4a2db699..703423b0be0e 100644
@@ -493,33 +493,12 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r)
 	}
 }
 
-static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
+static void setup_l3_mon_domain(int cpu, int id, struct rdt_resource *r, struct list_head *add_pos)
 {
-	int id = get_domain_id_from_scope(cpu, r->mon_scope);
-	struct list_head *add_pos = NULL;
 	struct rdt_hw_mon_domain *hw_dom;
-	struct rdt_domain_hdr *hdr;
 	struct rdt_mon_domain *d;
 	int err;
 
-	lockdep_assert_held(&domain_list_lock);
-
-	if (id < 0) {
-		pr_warn_once("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n",
-			     cpu, r->mon_scope, r->name);
-		return;
-	}
-
-	hdr = resctrl_find_domain(&r->mon_domains, id, &add_pos);
-	if (hdr) {
-		if (WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_MON)))
-			return;
-		d = container_of(hdr, struct rdt_mon_domain, hdr);
-
-		cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
-		return;
-	}
-
 	hw_dom = kzalloc_node(sizeof(*hw_dom), GFP_KERNEL, cpu_to_node(cpu));
 	if (!hw_dom)
 		return;
@@ -552,6 +531,40 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 	}
 }
 
+static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
+{
+	int id = get_domain_id_from_scope(cpu, r->mon_scope);
+	struct list_head *add_pos = NULL;
+	struct rdt_domain_hdr *hdr;
+	struct rdt_mon_domain *d;
+
+	lockdep_assert_held(&domain_list_lock);
+
+	if (id < 0) {
+		pr_warn_once("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n",
+			     cpu, r->mon_scope, r->name);
+		return;
+	}
+
+	hdr = resctrl_find_domain(&r->mon_domains, id, &add_pos);
+	if (hdr) {
+		if (WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_MON)))
+			return;
+		d = container_of(hdr, struct rdt_mon_domain, hdr);
+
+		cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
+		return;
+	}
+
+	switch (r->rid) {
+	case RDT_RESOURCE_L3:
+		setup_l3_mon_domain(cpu, id, r, add_pos);
+		break;
+	default:
+		WARN_ON_ONCE(1);
+	}
+}
+
 static void domain_add_cpu(int cpu, struct rdt_resource *r)
 {
 	if (r->alloc_capable)
Signature · 1 line
2.48.1
maintainer
#14 lore raw

Re: [PATCH v3 07/26] x86/resctrl: Move L3 initialization out of domain_add_cpu_mon()

Hi Tony,

Quoted 34 lines · Tony Luck

To prepare for additional types of monitoring domains, move all the L3 specific initialization into a helper function.

---
 arch/x86/kernel/cpu/resctrl/core.c | 57 ++++++++++++++++++------------
 1 file changed, 35 insertions(+), 22 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +1 -12
index d82a4a2db699..703423b0be0e 100644
@@ -493,33 +493,12 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r)
 	}
 }
 
-static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
+static void setup_l3_mon_domain(int cpu, int id, struct rdt_resource *r, struct list_head *add_pos)
 {
-	int id = get_domain_id_from_scope(cpu, r->mon_scope);
-	struct list_head *add_pos = NULL;
 	struct rdt_hw_mon_domain *hw_dom;
-	struct rdt_domain_hdr *hdr;
 	struct rdt_mon_domain *d;
 	int err;
 
-	lockdep_assert_held(&domain_list_lock);
-
-	if (id < 0) {
-		pr_warn_once("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n",
-			     cpu, r->mon_scope, r->name);
-		return;
-	}
-

arch/x86/kernel/cpu/resctrl/core.c static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) · line 522

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/core.c +0 -10
-	hdr = resctrl_find_domain(&r->mon_domains, id, &add_pos);
-	if (hdr) {
-		if (WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_MON)))
-			return;
-		d = container_of(hdr, struct rdt_mon_domain, hdr);
-
-		cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
-		return;
-	}
-

Many functions called by this new "setup_l3_mon_domain()" are specific to L3 resource but since L3 resource has so far been the only one supported the naming has been generic. Now that this function is made resource specific I think it will help make the code clear if the L3 specific functions called by it are also renamed. For example, mon_domain_free() can be renamed to free_l3_mon_domain() to match the "setup_l3_mon_domain()" introduced here. Also arch_mon_domain_online() -> arch_l3_mon_domain_online().
Seems like resctrl_online_mon_domain() is only temporarily specific to L3 in this series (would be helpful if such details are explained in changelog) ... to start it could include a check that ensures it is only called on L3 resource and then it will help to clarify what this patch does and how following work builds on it.

arch/x86/kernel/cpu/resctrl/core.c static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) · line 533

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/core.c +0 -0
 	hw_dom = kzalloc_node(sizeof(*hw_dom), GFP_KERNEL, cpu_to_node(cpu));
 	if (!hw_dom)
 		return;
@@ -552,6 +531,40 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 	}
 }
 

Reinette

author maintainer
#15 lore raw

Re: [PATCH v3 07/26] x86/resctrl: Move L3 initialization out of domain_add_cpu_mon()

Quoted 46 lines · Reinette Chatre

Hi Tony,

Quoted 43 lines · Tony Luck

To prepare for additional types of monitoring domains, move all the L3 specific initialization into a helper function.

---
 arch/x86/kernel/cpu/resctrl/core.c | 57 ++++++++++++++++++------------
 1 file changed, 35 insertions(+), 22 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +1 -21
index d82a4a2db699..703423b0be0e 100644
@@ -493,33 +493,12 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r)
 	}
 }
 
-static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
+static void setup_l3_mon_domain(int cpu, int id, struct rdt_resource *r, struct list_head *add_pos)
 {
-	int id = get_domain_id_from_scope(cpu, r->mon_scope);
-	struct list_head *add_pos = NULL;
 	struct rdt_hw_mon_domain *hw_dom;
-	struct rdt_domain_hdr *hdr;
 	struct rdt_mon_domain *d;
 	int err;
 
-	lockdep_assert_held(&domain_list_lock);
-
-	if (id < 0) {
-		pr_warn_once("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n",
-			     cpu, r->mon_scope, r->name);
-		return;
-	}
-
-	hdr = resctrl_find_domain(&r->mon_domains, id, &add_pos);
-	if (hdr) {
-		if (WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_MON)))
-			return;
-		d = container_of(hdr, struct rdt_mon_domain, hdr);
-
-		cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
-		return;
-	}
Quoted 8 lines

arch/x86/kernel/cpu/resctrl/core.c static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) · line 522

Quoted 1 line
arch/x86/kernel/cpu/resctrl/core.c +0 -1
-

Many functions called by this new "setup_l3_mon_domain()" are specific to L3 resource but since L3 resource has so far been the only one supported the naming has been generic. Now that this function is made resource specific I think it will help make the code clear if the L3 specific functions called by it are also renamed. For example, mon_domain_free() can be renamed to free_l3_mon_domain() to match the "setup_l3_mon_domain()" introduced here. Also arch_mon_domain_online() -> arch_l3_mon_domain_online().

What about "struct rdt_mon_domain"? It is now specific to L3. Should I change that to rdt_l3_mon_domain" as well (60 lines affected)?

Ditto for rdt_hw_mon_domain (but only 12 lines for this one).

Quoted 14 lines

Seems like resctrl_online_mon_domain() is only temporarily specific to L3 in this series (would be helpful if such details are explained in changelog) ... to start it could include a check that ensures it is only called on L3 resource and then it will help to clarify what this patch does and how following work builds on it.

arch/x86/kernel/cpu/resctrl/core.c static void domain_add_cpu_mon(int cpu, struct rdt_resource *r) · line 533

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/core.c +0 -0
 	hw_dom = kzalloc_node(sizeof(*hw_dom), GFP_KERNEL, cpu_to_node(cpu));
 	if (!hw_dom)
 		return;
@@ -552,6 +531,40 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 	}
 }
 

Reinette

maintainer
#16 lore raw

Re: [PATCH v3 07/26] x86/resctrl: Move L3 initialization out of domain_add_cpu_mon()

Hi Tony,

Quoted 56 lines · Tony
Quoted 46 lines · Reinette Chatre

Hi Tony,

Quoted 43 lines · Tony Luck

To prepare for additional types of monitoring domains, move all the L3 specific initialization into a helper function.

---
 arch/x86/kernel/cpu/resctrl/core.c | 57 ++++++++++++++++++------------
 1 file changed, 35 insertions(+), 22 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +1 -21
index d82a4a2db699..703423b0be0e 100644
@@ -493,33 +493,12 @@ static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r)
 	}
 }
 
-static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
+static void setup_l3_mon_domain(int cpu, int id, struct rdt_resource *r, struct list_head *add_pos)
 {
-	int id = get_domain_id_from_scope(cpu, r->mon_scope);
-	struct list_head *add_pos = NULL;
 	struct rdt_hw_mon_domain *hw_dom;
-	struct rdt_domain_hdr *hdr;
 	struct rdt_mon_domain *d;
 	int err;
 
-	lockdep_assert_held(&domain_list_lock);
-
-	if (id < 0) {
-		pr_warn_once("Can't find monitor domain id for CPU:%d scope:%d for resource %s\n",
-			     cpu, r->mon_scope, r->name);
-		return;
-	}
-
-	hdr = resctrl_find_domain(&r->mon_domains, id, &add_pos);
-	if (hdr) {
-		if (WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_MON)))
-			return;
-		d = container_of(hdr, struct rdt_mon_domain, hdr);
-
-		cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
-		return;
-	}
Quoted 8 lines

arch/x86/kernel/cpu/resctrl/core.c static void domain_add_cpu_ctrl(int cpu, struct rdt_resource *r) · line 522

Quoted 1 line
arch/x86/kernel/cpu/resctrl/core.c +0 -1
-

Many functions called by this new "setup_l3_mon_domain()" are specific to L3 resource but since L3 resource has so far been the only one supported the naming has been generic. Now that this function is made resource specific I think it will help make the code clear if the L3 specific functions called by it are also renamed. For example, mon_domain_free() can be renamed to free_l3_mon_domain() to match the "setup_l3_mon_domain()" introduced here. Also arch_mon_domain_online() -> arch_l3_mon_domain_online().

Quoted 4 lines

What about "struct rdt_mon_domain"? It is now specific to L3. Should I change that to rdt_l3_mon_domain" as well (60 lines affected)?

Ditto for rdt_hw_mon_domain (but only 12 lines for this one).

Thank you for considering this. My vote would be "yes" for both. I think it will help understand and maintain the code if naming helps to make obvious what data/code applies to. Monitoring has been synonymous with L3 monitoring for so long that there may be many instances of this implicit assumption.

Reinette

author maintainer
#17 lore raw

PATCH v3 8/26 x86/resctrl: Refactor domain_remove_cpu_mon() ready for new domain types

In reply to Tony Luck

Different types of domains require different actions when the last CPU in the domain is removed.

Refactor to make it easy to add new actions for new types of domains.

---
 arch/x86/kernel/cpu/resctrl/core.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/resctrl/core.c +8 -6
index 703423b0be0e..7080447028b0 100644
@@ -644,17 +644,19 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
 	if (WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_MON)))
 		return;
 
-	d = container_of(hdr, struct rdt_mon_domain, hdr);
-	hw_dom = resctrl_to_arch_mon_dom(d);
+	cpumask_clear_cpu(cpu, &hdr->cpu_mask);
+	if (!cpumask_empty(&hdr->cpu_mask))
+		return;
 
-	cpumask_clear_cpu(cpu, &d->hdr.cpu_mask);
-	if (cpumask_empty(&d->hdr.cpu_mask)) {
+	switch (r->rid) {
+	case RDT_RESOURCE_L3:
+		d = container_of(hdr, struct rdt_mon_domain, hdr);
+		hw_dom = resctrl_to_arch_mon_dom(d);
 		resctrl_offline_mon_domain(r, d);
 		list_del_rcu(&d->hdr.list);
 		synchronize_rcu();
 		mon_domain_free(hw_dom);
-
-		return;
+		break;
 	}
 }
 
Signature · 1 line
2.48.1
maintainer
#18 lore raw

Re: [PATCH v3 08/26] x86/resctrl: Refactor domain_remove_cpu_mon() ready for new domain types

Hi Tony,

Quoted 1 line · Tony Luck

Different types of domains require different actions when the last

Please expand/explain what is meant with "different types of domains".

Quoted 8 lines

CPU in the domain is removed.

Refactor to make it easy to add new actions for new types of domains.

---

Reinette

author maintainer
#19 lore raw

PATCH v3 9/26 x86/resctrl: Change generic monitor functions to use struct rdt_domain_hdr

In reply to Tony Luck

Functions that don't need the internal details of the rdt_mon_domain can operate on just the rdt_domain_hdr.

Add sanity checks where container_of() is used to find the surrounding domain structure that hdr has the expected type.

---
 include/linux/resctrl.h            |  2 +-
 arch/x86/kernel/cpu/resctrl/core.c |  2 +-
 fs/resctrl/rdtgroup.c              | 66 +++++++++++++++++++++---------
 3 files changed, 48 insertions(+), 22 deletions(-)
include/linux/resctrl.h +1 -1
index 177f9879bae1..0fce626605b9 100644
@@ -412,7 +412,7 @@ int resctrl_arch_update_one(struct rdt_resource *r, struct rdt_ctrl_domain *d,
 u32 resctrl_arch_get_config(struct rdt_resource *r, struct rdt_ctrl_domain *d,
 			    u32 closid, enum resctrl_conf_type type);
 int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d);
-int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d);
+int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr);
 void resctrl_offline_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d);
 void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d);
 void resctrl_online_cpu(unsigned int cpu);
arch/x86/kernel/cpu/resctrl/core.c +1 -1
index 7080447028b0..59844fd7105f 100644
@@ -523,7 +523,7 @@ static void setup_l3_mon_domain(int cpu, int id, struct rdt_resource *r, struct
 
 	list_add_tail_rcu(&d->hdr.list, add_pos);
 
-	err = resctrl_online_mon_domain(r, d);
+	err = resctrl_online_mon_domain(r, &d->hdr);
 	if (err) {
 		list_del_rcu(&d->hdr.list);
 		synchronize_rcu();
fs/resctrl/rdtgroup.c +46 -20

177 lines not on this page · raw message

Signature · 1 line
2.48.1
maintainer
#20 lore raw

Re: [PATCH v3 09/26] x86/resctrl: Change generic monitor functions to use struct rdt_domain_hdr

Hi Tony,

fs/resctrl/rdtgroup.c int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d · line 4107

Quoted 13 lines · Tony Luck
fs/resctrl/rdtgroup.c +4 -1
@@ -4074,10 +4097,13 @@ int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d
 	return err;
 }
 
-int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d)
+int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr)
 {
+	struct rdt_mon_domain *d;
 	int err;
 
+	WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_MON));
+	d = container_of(hdr, struct rdt_mon_domain, hdr);
 	mutex_lock(&rdtgroup_mutex);

It is a bit unexpected to see code added outside of the mutex_lock(). This looks fine since it is accessing domain list from the hotplug handlers but since it is fs code that is jumped to by arch code and fs code relies on arch code for the locking I'd like to suggest a:

	lockdep_assert_cpus_held();

Reinette

author maintainer
#21 lore raw

PATCH v3 10/26 fs/resctrl: Improve handling for events that can be read from any CPU

In reply to Tony Luck

Add a flag to each instance of struct mon_event to indicate that there is no need for cross-processor interrupts to read this event from a CPU in a specific rdt_mon_domain.

The flag is copied to struct mon_data for ease of access when a user reads an event file invoking rdtgroup_mondata_show().

Copied again into struct rmid_read in mon_event_read() for use by sanity checks in __mon_event_count().

When the flag is set allow choice from cpu_online_mask. This makes the smp_call*() functions default to the current CPU.

---
 fs/resctrl/internal.h    |  8 +++++++-
 fs/resctrl/ctrlmondata.c | 10 +++++++---
 fs/resctrl/monitor.c     |  4 ++--
 fs/resctrl/rdtgroup.c    |  1 +
 4 files changed, 17 insertions(+), 6 deletions(-)
fs/resctrl/internal.h +7 -1
index 08dbf89939ac..74a77794364d 100644
@@ -72,6 +72,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
  * @evtid:		event id
  * @name:		name of the event
  * @configurable:	true if the event is configurable
+ * @any_cpu:		true if this event can be read from any CPU
  * @list:		entry in &rdt_resource->evt_list
  */
 struct mon_evt {
@@ -79,6 +80,7 @@ struct mon_evt {
 	enum resctrl_res_level	rid;
 	char			*name;
 	bool			configurable;
+	bool			any_cpu;
 	struct list_head	list;
 };
 
@@ -93,6 +95,7 @@ struct mon_evt {
  *                   the event file belongs. When @sum is one this
  *                   is the id of the L3 cache that all domains to be
  *                   summed share.
+ * @any_cpu:         true if this event can be read from any CPU
  *
  * Stored in the kernfs kn->priv field, readers and writers must hold
  * rdtgroup_mutex.
@@ -103,6 +106,7 @@ struct mon_data {
 	enum resctrl_event_id evtid;
 	unsigned int sum;
 	unsigned int domid;
+	bool any_cpu;
 };
 
 /**
@@ -115,6 +119,7 @@ struct mon_data {
  *	   domains in @r sharing L3 @ci.id
  * @evtid: Which monitor event to read.
  * @first: Initialize MBM counter when true.
+ * @any_cpu: When true read can be executed on any CPU.
  * @ci:    Cacheinfo for L3. Only set when @d is NULL. Used when summing domains.
  * @err:   Error encountered when reading counter.
  * @val:   Returned value of event counter. If @rgrp is a parent resource group,
@@ -129,6 +134,7 @@ struct rmid_read {
 	struct rdt_mon_domain	*d;
 	enum resctrl_event_id	evtid;
 	bool			first;
+	bool			any_cpu;
 	struct cacheinfo	*ci;
 	int			err;
 	u64			val;
@@ -358,7 +364,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg);
 
 void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 		    struct rdt_mon_domain *d, struct rdtgroup *rdtgrp,
-		    cpumask_t *cpumask, int evtid, int first);
+		    const cpumask_t *cpumask, int evtid, int first);
 
 int resctrl_mon_resource_init(void);
 
fs/resctrl/ctrlmondata.c +7 -3
index 0c245af0ff42..cd77960657f0 100644
@@ -525,7 +525,7 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,
 
 void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 		    struct rdt_mon_domain *d, struct rdtgroup *rdtgrp,
-		    cpumask_t *cpumask, int evtid, int first)
+		    const cpumask_t *cpumask, int evtid, int first)
 {
 	int cpu;
 
@@ -571,6 +571,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	u32 resid, evtid, domid;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
+	const cpumask_t *mask;
 	struct mon_data *md;
 	int ret = 0;
 
@@ -589,6 +590,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	resid = md->rid;
 	domid = md->domid;
 	evtid = md->evtid;
+	rr.any_cpu = md->any_cpu;
 	r = resctrl_arch_get_resource(resid);
 
 	if (md->sum) {
@@ -601,8 +603,9 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 		list_for_each_entry(d, &r->mon_domains, hdr.list) {
 			if (d->ci->id == domid) {
 				rr.ci = d->ci;
+				mask = md->any_cpu ? cpu_online_mask : &d->ci->shared_cpu_map;
 				mon_event_read(&rr, r, NULL, rdtgrp,
-					       &d->ci->shared_cpu_map, evtid, false);
+					       mask, evtid, false);
 				goto checkresult;
 			}
 		}
@@ -619,7 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 			goto out;
 		}
 		d = container_of(hdr, struct rdt_mon_domain, hdr);
-		mon_event_read(&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false);
+		mask = md->any_cpu ? cpu_online_mask : &d->hdr.cpu_mask;
+		mon_event_read(&rr, r, d, rdtgrp, mask, evtid, false);
 	}
 
 checkresult:
fs/resctrl/monitor.c +2 -2
index 472754d082cb..1cf0b085e07a 100644
@@ -375,7 +375,7 @@ static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr)
 
 	if (rr->d) {
 		/* Reading a single domain, must be on a CPU in that domain. */
-		if (!cpumask_test_cpu(cpu, &rr->d->hdr.cpu_mask))
+		if (!rr->any_cpu && !cpumask_test_cpu(cpu, &rr->d->hdr.cpu_mask))
 			return -EINVAL;
 		rr->err = resctrl_arch_rmid_read(rr->r, rr->d, closid, rmid,
 						 rr->evtid, &tval, rr->arch_mon_ctx);
@@ -388,7 +388,7 @@ static int __mon_event_count(u32 closid, u32 rmid, struct rmid_read *rr)
 	}
 
 	/* Summing domains that share a cache, must be on a CPU for that cache. */
-	if (!cpumask_test_cpu(cpu, &rr->ci->shared_cpu_map))
+	if (!rr->any_cpu && !cpumask_test_cpu(cpu, &rr->ci->shared_cpu_map))
 		return -EINVAL;
 
 	/*
fs/resctrl/rdtgroup.c +1 -0
index 5011e404798a..97c2ba8af930 100644
@@ -2926,6 +2926,7 @@ static struct mon_data *mon_get_kn_priv(int rid, int domid, struct mon_evt *mevt
 	priv->domid = domid;
 	priv->sum = do_sum;
 	priv->evtid = mevt->evtid;
+	priv->any_cpu = mevt->any_cpu;
 	list_add_tail(&priv->list, &kn_priv_list);
 
 	return priv;
Signature · 1 line
2.48.1
maintainer
#22 lore raw

Re: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU

Hi Tony,

Quoted 5 lines · Tony Luck

Add a flag to each instance of struct mon_event to indicate that there is no need for cross-processor interrupts to read this event from a CPU in a specific rdt_mon_domain.

The flag is copied to struct mon_data for ease of access when a user

Copy the flag ...

Quoted 7 lines

reads an event file invoking rdtgroup_mondata_show().

Copied again into struct rmid_read in mon_event_read() for use by sanity checks in __mon_event_count().

When the flag is set allow choice from cpu_online_mask. This makes the smp_call*() functions default to the current CPU.

Please use imperative tone.

Quoted 55 lines
---
 fs/resctrl/internal.h    |  8 +++++++-
 fs/resctrl/ctrlmondata.c | 10 +++++++---
 fs/resctrl/monitor.c     |  4 ++--
 fs/resctrl/rdtgroup.c    |  1 +
 4 files changed, 17 insertions(+), 6 deletions(-)
fs/resctrl/internal.h +5 -0
index 08dbf89939ac..74a77794364d 100644
@@ -72,6 +72,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
  * @evtid:		event id
  * @name:		name of the event
  * @configurable:	true if the event is configurable
+ * @any_cpu:		true if this event can be read from any CPU
  * @list:		entry in &rdt_resource->evt_list
  */
 struct mon_evt {
@@ -79,6 +80,7 @@ struct mon_evt {
 	enum resctrl_res_level	rid;
 	char			*name;
 	bool			configurable;
+	bool			any_cpu;
 	struct list_head	list;
 };
 
@@ -93,6 +95,7 @@ struct mon_evt {
  *                   the event file belongs. When @sum is one this
  *                   is the id of the L3 cache that all domains to be
  *                   summed share.
+ * @any_cpu:         true if this event can be read from any CPU
  *
  * Stored in the kernfs kn->priv field, readers and writers must hold
  * rdtgroup_mutex.
@@ -103,6 +106,7 @@ struct mon_data {
 	enum resctrl_event_id evtid;
 	unsigned int sum;
 	unsigned int domid;
+	bool any_cpu;
 };
 
 /**
@@ -115,6 +119,7 @@ struct mon_data {
  *	   domains in @r sharing L3 @ci.id
  * @evtid: Which monitor event to read.
  * @first: Initialize MBM counter when true.
+ * @any_cpu: When true read can be executed on any CPU.
  * @ci:    Cacheinfo for L3. Only set when @d is NULL. Used when summing domains.
  * @err:   Error encountered when reading counter.
  * @val:   Returned value of event counter. If @rgrp is a parent resource group,

fs/resctrl/internal.h struct rmid_read { · line 140

Quoted 7 lines
fs/resctrl/internal.h +1 -0
@@ -129,6 +134,7 @@ struct rmid_read {
 	struct rdt_mon_domain	*d;
 	enum resctrl_event_id	evtid;
 	bool			first;
+	bool			any_cpu;
 	struct cacheinfo	*ci;
 	int			err;
 	u64			val;

Duplicating the same property across three structures does not look right. It looks to me that struct mon_evt should be the "source of truth" for any event and that these other structures can point to it instead of copying the data?

Quoted 50 lines
fs/resctrl/internal.h +1 -1
@@ -358,7 +364,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg);
 
 void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 		    struct rdt_mon_domain *d, struct rdtgroup *rdtgrp,
-		    cpumask_t *cpumask, int evtid, int first);
+		    const cpumask_t *cpumask, int evtid, int first);
 
 int resctrl_mon_resource_init(void);
 
fs/resctrl/ctrlmondata.c +5 -2
index 0c245af0ff42..cd77960657f0 100644
@@ -525,7 +525,7 @@ struct rdt_domain_hdr *resctrl_find_domain(struct list_head *h, int id,
 
 void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 		    struct rdt_mon_domain *d, struct rdtgroup *rdtgrp,
-		    cpumask_t *cpumask, int evtid, int first)
+		    const cpumask_t *cpumask, int evtid, int first)
 {
 	int cpu;
 
@@ -571,6 +571,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	u32 resid, evtid, domid;
 	struct rdtgroup *rdtgrp;
 	struct rdt_resource *r;
+	const cpumask_t *mask;
 	struct mon_data *md;
 	int ret = 0;
 
@@ -589,6 +590,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	resid = md->rid;
 	domid = md->domid;
 	evtid = md->evtid;
+	rr.any_cpu = md->any_cpu;
 	r = resctrl_arch_get_resource(resid);
 
 	if (md->sum) {
@@ -601,8 +603,9 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 		list_for_each_entry(d, &r->mon_domains, hdr.list) {
 			if (d->ci->id == domid) {
 				rr.ci = d->ci;
+				mask = md->any_cpu ? cpu_online_mask : &d->ci->shared_cpu_map;
 				mon_event_read(&rr, r, NULL, rdtgrp,
-					       &d->ci->shared_cpu_map, evtid, false);
+					       mask, evtid, false);
 				goto checkresult;
 			}
 		}

fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 626

Quoted 6 lines
fs/resctrl/ctrlmondata.c +2 -1
@@ -619,7 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 			goto out;
 		}
 		d = container_of(hdr, struct rdt_mon_domain, hdr);
-		mon_event_read(&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false);
+		mask = md->any_cpu ? cpu_online_mask : &d->hdr.cpu_mask;
+		mon_event_read(&rr, r, d, rdtgrp, mask, evtid, false);

I do not think this accomplishes the goal of this patch. Looking at mon_event_read() it calls cpumask_any_housekeeping(cpumask, RESCTRL_PICK_ANY_CPU) before any of the smp_*() calls.

	cpumask_any_housekeeping()
	{
		...
		if (exclude_cpu == RESCTRL_PICK_ANY_CPU)
			cpu = cpumask_any(mask);
		...
	}

cpumask_any() is just cpumask_first() so it will pick the first CPU in the online mask that may not be the current CPU.

fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/

Reinette

author maintainer
#23 lore raw

Re: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU

Quoted 19 lines · Reinette Chatre

fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 626

Quoted 7 lines
fs/resctrl/ctrlmondata.c +2 -1
@@ -619,7 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 			goto out;
 		}
 		d = container_of(hdr, struct rdt_mon_domain, hdr);
-		mon_event_read(&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false);
+		mask = md->any_cpu ? cpu_online_mask : &d->hdr.cpu_mask;
+		mon_event_read(&rr, r, d, rdtgrp, mask, evtid, false);

I do not think this accomplishes the goal of this patch. Looking at mon_event_read() it calls cpumask_any_housekeeping(cpumask, RESCTRL_PICK_ANY_CPU) before any of the smp_*() calls.

	cpumask_any_housekeeping()
	{
		...
		if (exclude_cpu == RESCTRL_PICK_ANY_CPU)
			cpu = cpumask_any(mask);
		...
	}
Quoted 5 lines

cpumask_any() is just cpumask_first() so it will pick the first CPU in the online mask that may not be the current CPU.

fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/

I remember Peter complaining[1] about extra context switches when cpumask_any_housekeeping() was introduced, but it seems that the discussion died with no fix applied.

The blocking problem is that ARM may not be able to read a counter on a tick_nohz CPU because it may need to sleep.

Do we need more options for events:

1) Must be read on a CPU in the right domain // Legacy
2) Can be read from any CPU // My addtion
3) Must be read on a "housekeeping" CPU // James' code in upstream
4) Cannot be read on a tick_nohz CPU // Could be combined with 1 or 2?

Quoted 1 line

Reinette

[1] https://lore.kernel.org/all/[email protected]/

Quoted 1 line
maintainer
#24 lore raw

Re: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU

Hi Tony,

Quoted 23 lines · Tony
Quoted 22 lines · Reinette Chatre

fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 626

Quoted 7 lines
fs/resctrl/ctrlmondata.c +2 -1
@@ -619,7 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 			goto out;
 		}
 		d = container_of(hdr, struct rdt_mon_domain, hdr);
-		mon_event_read(&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false);
+		mask = md->any_cpu ? cpu_online_mask : &d->hdr.cpu_mask;
+		mon_event_read(&rr, r, d, rdtgrp, mask, evtid, false);

I do not think this accomplishes the goal of this patch. Looking at mon_event_read() it calls cpumask_any_housekeeping(cpumask, RESCTRL_PICK_ANY_CPU) before any of the smp_*() calls.

	cpumask_any_housekeeping()
	{
		...
		if (exclude_cpu == RESCTRL_PICK_ANY_CPU)
			cpu = cpumask_any(mask);
		...
	}

cpumask_any() is just cpumask_first() so it will pick the first CPU in the online mask that may not be the current CPU.

Quoted 6 lines
Quoted 2 lines

fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/

I remember Peter complaining[1] about extra context switches when cpumask_any_housekeeping() was introduced, but it seems that the discussion died with no fix applied.

The initial complaint was indeed that reading individual events is slower.

The issue is that the intended use case read from many files at frequent intervals and thus becomes vulnerable to any changes in this area that really is already a slow path (reading from a file ... taking a mutex ...).

Instead of working on shaving cycles off this path the discussion transitioned to resctrl providing better support for the underlying use case. I understood that this is being experimented with [2] and last I heard it looks promising.

Quoted 10 lines

The blocking problem is that ARM may not be able to read a counter on a tick_nohz CPU because it may need to sleep.

Do we need more options for events:

1) Must be read on a CPU in the right domain // Legacy
2) Can be read from any CPU // My addtion
3) Must be read on a "housekeeping" CPU // James' code in upstream
4) Cannot be read on a tick_nohz CPU // Could be combined with 1 or 2?

I do not see needing additional complexity here. I think it will be simpler
to just replace use of cpumask_any_housekeeping() in mon_event_read() with
open code that supports the particular usage. As I understand it is prohibited for all CPUs to be in tick_nohz_full_mask so it looks to me as though the existing "if (tick_nohz_full_cpu(cpu))" should never be true (since no CPU is being excluded). Also, since mon_event_read() has no need to exclude CPUs, just a cpumask_andnot() should suffice to determine what remains of given mask after accounting for all the NO_HZ CPUs if tick_nohz_full_enabled().

Reinette

Quoted 5 lines
Quoted 2 lines

Reinette

[1] https://lore.kernel.org/all/[email protected]/

Quoted 1 line

[2] https://lore.kernel.org/lkml/CALPaoCgpnVORZfbKVLXDFUZvv8jhpShHPzB3cwdLTZQH1o9ULw@mail.gmail.com/

author maintainer
#25 lore raw

RE: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU

Quoted 32 lines
Quoted 9 lines
Quoted 5 lines

cpumask_any() is just cpumask_first() so it will pick the first CPU in the online mask that may not be the current CPU.

fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/

I remember Peter complaining[1] about extra context switches when cpumask_any_housekeeping() was introduced, but it seems that the discussion died with no fix applied.

The initial complaint was indeed that reading individual events is slower.

The issue is that the intended use case read from many files at frequent intervals and thus becomes vulnerable to any changes in this area that really is already a slow path (reading from a file ... taking a mutex ...).

Instead of working on shaving cycles off this path the discussion transitioned to resctrl providing better support for the underlying use case. I understood that this is being experimented with [2] and last I heard it looks promising.

Quoted 10 lines

The blocking problem is that ARM may not be able to read a counter on a tick_nohz CPU because it may need to sleep.

Do we need more options for events:

1) Must be read on a CPU in the right domain // Legacy
2) Can be read from any CPU // My addtion
3) Must be read on a "housekeeping" CPU // James' code in upstream
4) Cannot be read on a tick_nohz CPU // Could be combined with 1 or 2?

Quoted 8 lines

I do not see needing additional complexity here. I think it will be simpler
to just replace use of cpumask_any_housekeeping() in mon_event_read() with
open code that supports the particular usage. As I understand it is prohibited for all CPUs to be in tick_nohz_full_mask so it looks to me as though the existing "if (tick_nohz_full_cpu(cpu))" should never be true (since no CPU is being excluded). Also, since mon_event_read() has no need to exclude CPUs, just a cpumask_andnot() should suffice to determine what remains of given mask after accounting for all the NO_HZ CPUs if tick_nohz_full_enabled().

Maybe there isn’t much complexity to make this "read one counter" better on systems where reading from any CPU is possible. Taking your advice from the earlier review the filesystem code can set a flag in the mon_evt structure. struct mon_data and struct rmid_read can change from holding the event id to holding a pointer to the mon_evt (as the source of truth).

Then mon_event_read() can just have a simple direct call to mon_event_count() just before the call to cpumask_any_housekeeping() like this:

		if (evt->any_cpu) {
			mon_event_count(rr);
			goto done;
		}

The "goto done" jumps to the resctrl_arch_mon_ctx_free(r, evt->evtid, rr->arch_mon_ctx); at the end of mon_event_read()

Folks can still pursue the bulk read of many counters (though I expect you might want one file per domain, rather than a single file to report everything).

Quoted 10 lines

Reinette

Quoted 5 lines
Quoted 2 lines

Reinette

[1] https://lore.kernel.org/all/[email protected]/

Quoted 1 line

[2] https://lore.kernel.org/lkml/CALPaoCgpnVORZfbKVLXDFUZvv8jhpShHPzB3cwdLTZQH1o9ULw@mail.gmail.com/

-Tony

maintainer
#26 lore raw

Re: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU

Hi Tony,

Quoted 50 lines · Tony
Quoted 32 lines
Quoted 9 lines
Quoted 5 lines

cpumask_any() is just cpumask_first() so it will pick the first CPU in the online mask that may not be the current CPU.

fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/

I remember Peter complaining[1] about extra context switches when cpumask_any_housekeeping() was introduced, but it seems that the discussion died with no fix applied.

The initial complaint was indeed that reading individual events is slower.

The issue is that the intended use case read from many files at frequent intervals and thus becomes vulnerable to any changes in this area that really is already a slow path (reading from a file ... taking a mutex ...).

Instead of working on shaving cycles off this path the discussion transitioned to resctrl providing better support for the underlying use case. I understood that this is being experimented with [2] and last I heard it looks promising.

Quoted 10 lines

The blocking problem is that ARM may not be able to read a counter on a tick_nohz CPU because it may need to sleep.

Do we need more options for events:

1) Must be read on a CPU in the right domain // Legacy
2) Can be read from any CPU // My addtion
3) Must be read on a "housekeeping" CPU // James' code in upstream
4) Cannot be read on a tick_nohz CPU // Could be combined with 1 or 2?

Quoted 8 lines

I do not see needing additional complexity here. I think it will be simpler
to just replace use of cpumask_any_housekeeping() in mon_event_read() with
open code that supports the particular usage. As I understand it is prohibited for all CPUs to be in tick_nohz_full_mask so it looks to me as though the existing "if (tick_nohz_full_cpu(cpu))" should never be true (since no CPU is being excluded). Also, since mon_event_read() has no need to exclude CPUs, just a cpumask_andnot() should suffice to determine what remains of given mask after accounting for all the NO_HZ CPUs if tick_nohz_full_enabled().

Maybe there isn’t much complexity to make this "read one counter" better on systems where reading from any CPU is possible. Taking your advice from the earlier review the filesystem code can set a flag in the mon_evt structure. struct mon_data and struct rmid_read can change from holding the event id to holding a pointer to the mon_evt (as the source of truth).

Then mon_event_read() can just have a simple direct call to mon_event_count() just before the call to cpumask_any_housekeeping() like this:

Quoted 7 lines
		if (evt->any_cpu) {
			mon_event_count(rr);
			goto done;
		}

The "goto done" jumps to the resctrl_arch_mon_ctx_free(r, evt->evtid, rr->arch_mon_ctx); at the end of mon_event_read()

Thanks, this looks great.

Quoted 3 lines

Folks can still pursue the bulk read of many counters (though I expect you might want one file per domain, rather than a single file to report everything).

I will have to re-read that thread to refresh myself on the discussion but scanning the thread I did find a summary of points (end of [3]) and this was there.

Quoted 5 lines
Quoted 5 lines

Reinette

Quoted 1 line
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Reinette

[1] https://lore.kernel.org/all/[email protected]/

Quoted 1 line

[2] https://lore.kernel.org/lkml/CALPaoCgpnVORZfbKVLXDFUZvv8jhpShHPzB3cwdLTZQH1o9ULw@mail.gmail.com/

[3] https://lore.kernel.org/lkml/[email protected]/

Reinette

#27 lore raw

Re: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU

In reply to Reinette Chatre

Hi Reinette,

Quoted 36 lines · Reinette Chatre

Hi Tony,

Quoted 23 lines · Tony
Quoted 22 lines · Reinette Chatre

fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 626

Quoted 7 lines
fs/resctrl/ctrlmondata.c +2 -1
@@ -619,7 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
                    goto out;
            }
            d = container_of(hdr, struct rdt_mon_domain, hdr);
-           mon_event_read(&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false);
+           mask = md->any_cpu ? cpu_online_mask : &d->hdr.cpu_mask;
+           mon_event_read(&rr, r, d, rdtgrp, mask, evtid, false);

I do not think this accomplishes the goal of this patch. Looking at mon_event_read() it calls cpumask_any_housekeeping(cpumask, RESCTRL_PICK_ANY_CPU) before any of the smp_*() calls.

     cpumask_any_housekeeping()
     {
             ...
             if (exclude_cpu == RESCTRL_PICK_ANY_CPU)
                     cpu = cpumask_any(mask);
             ...
     }

cpumask_any() is just cpumask_first() so it will pick the first CPU in the online mask that may not be the current CPU.

Quoted 6 lines
Quoted 2 lines

fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/

I remember Peter complaining[1] about extra context switches when cpumask_any_housekeeping() was introduced, but it seems that the discussion died with no fix applied.

The initial complaint was indeed that reading individual events is slower.

Quoted 12 lines

The issue is that the intended use case read from many files at frequent intervals and thus becomes vulnerable to any changes in this area that really is already a slow path (reading from a file ... taking a mutex ...).

Instead of working on shaving cycles off this path the discussion transitioned to resctrl providing better support for the underlying use case. I understood that this is being experimented with [2] and last I heard it looks promising.

Quoted 3 lines

The blocking problem is that ARM may not be able to read a counter on a tick_nohz CPU because it may need to sleep.

If I hadn't already turned my attention to optimizing bulk counter reads, I might have mentioned that the change Tony referred to is broken on MPAM implementations because the MPAM resctrl_arch_rmid_read() cannot wait for its internal mutex with preemption disabled.

Quoted 8 lines
Quoted 7 lines

Do we need more options for events:

1) Must be read on a CPU in the right domain // Legacy
2) Can be read from any CPU // My addtion
3) Must be read on a "housekeeping" CPU // James' code in upstream
4) Cannot be read on a tick_nohz CPU // Could be combined with 1 or 2?

Quoted 8 lines

I do not see needing additional complexity here. I think it will be simpler
to just replace use of cpumask_any_housekeeping() in mon_event_read() with
open code that supports the particular usage. As I understand it is prohibited for all CPUs to be in tick_nohz_full_mask so it looks to me as though the existing "if (tick_nohz_full_cpu(cpu))" should never be true (since no CPU is being excluded). Also, since mon_event_read() has no need to exclude CPUs, just a cpumask_andnot() should suffice to determine what remains of given mask after accounting for all the NO_HZ CPUs if tick_nohz_full_enabled().

Can you clarify what you mean by "all CPUs"? It's not difficult for all CPUs in an L3 domain to be in tick_nohz_full_mask on AMD implementations, where there are many small L3 domains (~8 CPUs each) in a socket.

Google makes use of isolation along this domain boundary on AMD platforms in some products and these users prefer to read counters using IPIs because they are concerned about introducing context switches to the isolated part of the system. In these configurations, there is typically only one RMID in that domain, so few of these IPIs are needed. (Note that these are different users from the ones I had described before who spawn large numbers of containers not limited to any domains and want to read the MBM counters for all the RMIDs on all the domains frequently.)

Thanks,
-Peter

maintainer
#28 lore raw

Re: [PATCH v3 10/26] fs/resctrl: Improve handling for events that can be read from any CPU

Hi Peter,

Quoted 76 lines · Peter Newman

Hi Reinette,

Quoted 36 lines · Reinette Chatre

Hi Tony,

Quoted 23 lines · Tony
Quoted 22 lines · Reinette Chatre

fs/resctrl/ctrlmondata.c int rdtgroup_mondata_show(struct seq_file *m, void *arg) · line 626

Quoted 7 lines
fs/resctrl/ctrlmondata.c +2 -1
@@ -619,7 +622,8 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
                    goto out;
            }
            d = container_of(hdr, struct rdt_mon_domain, hdr);
-           mon_event_read(&rr, r, d, rdtgrp, &d->hdr.cpu_mask, evtid, false);
+           mask = md->any_cpu ? cpu_online_mask : &d->hdr.cpu_mask;
+           mon_event_read(&rr, r, d, rdtgrp, mask, evtid, false);

I do not think this accomplishes the goal of this patch. Looking at mon_event_read() it calls cpumask_any_housekeeping(cpumask, RESCTRL_PICK_ANY_CPU) before any of the smp_*() calls.

     cpumask_any_housekeeping()
     {
             ...
             if (exclude_cpu == RESCTRL_PICK_ANY_CPU)
                     cpu = cpumask_any(mask);
             ...
     }

cpumask_any() is just cpumask_first() so it will pick the first CPU in the online mask that may not be the current CPU.

Quoted 6 lines
Quoted 2 lines

fwiw ... there are some optimizations planned in this area that I have not yet studied:
https://lore.kernel.org/lkml/[email protected]/

I remember Peter complaining[1] about extra context switches when cpumask_any_housekeeping() was introduced, but it seems that the discussion died with no fix applied.

The initial complaint was indeed that reading individual events is slower.

Quoted 12 lines

The issue is that the intended use case read from many files at frequent intervals and thus becomes vulnerable to any changes in this area that really is already a slow path (reading from a file ... taking a mutex ...).

Instead of working on shaving cycles off this path the discussion transitioned to resctrl providing better support for the underlying use case. I understood that this is being experimented with [2] and last I heard it looks promising.

Quoted 3 lines

The blocking problem is that ARM may not be able to read a counter on a tick_nohz CPU because it may need to sleep.

If I hadn't already turned my attention to optimizing bulk counter reads, I might have mentioned that the change Tony referred to is broken on MPAM implementations because the MPAM resctrl_arch_rmid_read() cannot wait for its internal mutex with preemption disabled.

Quoted 8 lines
Quoted 7 lines

Do we need more options for events:

1) Must be read on a CPU in the right domain // Legacy
2) Can be read from any CPU // My addtion
3) Must be read on a "housekeeping" CPU // James' code in upstream
4) Cannot be read on a tick_nohz CPU // Could be combined with 1 or 2?

Quoted 8 lines

I do not see needing additional complexity here. I think it will be simpler
to just replace use of cpumask_any_housekeeping() in mon_event_read() with
open code that supports the particular usage. As I understand it is prohibited for all CPUs to be in tick_nohz_full_mask so it looks to me as though the existing "if (tick_nohz_full_cpu(cpu))" should never be true (since no CPU is being excluded). Also, since mon_event_read() has no need to exclude CPUs, just a cpumask_andnot() should suffice to determine what remains of given mask after accounting for all the NO_HZ CPUs if tick_nohz_full_enabled().

Quoted 1 line

Can you clarify what you mean by "all CPUs"? It's not difficult for

I mentioned this in the context of this patch that adds support for events that can be ready from *any* CPU. The CPU reading the event data need not be in the domain for which data is being read so all CPUs on the system are available to the flow supporting these events. Since all CPUs on the system cannot be in tick_nohz_full_mask there will always be a CPU available to read this type of event that can be read from any CPU.

I made it way too complicated with this though. Tony proposed something much better and simpler [1].

Quoted 4 lines

all CPUs in an L3 domain to be in tick_nohz_full_mask on AMD implementations, where there are many small L3 domains (~8 CPUs each) in a socket.

Quoted 10 lines

Google makes use of isolation along this domain boundary on AMD platforms in some products and these users prefer to read counters using IPIs because they are concerned about introducing context switches to the isolated part of the system. In these configurations, there is typically only one RMID in that domain, so few of these IPIs are needed. (Note that these are different users from the ones I had described before who spawn large numbers of containers not limited to any domains and want to read the MBM counters for all the RMIDs on all the domains frequently.)

Thank you for this insight. There is no change planned for reading event counters for those events that need to be read from their domain. Tony's recent proposal [1] moves the handling of these new style of events to a separate branch.

Reinette

[1] https://lore.kernel.org/lkml/DS7PR11MB607763D8B912A60A3574D2BAFCBA2@DS7PR11MB6077.namprd11.prod.outlook.com/

author maintainer
#29 lore raw

PATCH v3 11/26 fs/resctrl: Add support for additional monitor event display formats

In reply to Tony Luck

Add a type field to both the mon_evt and mon_data structures.

Legacy monitor events are still all displayed as an unsigned decimal 64-bit integer.

Add an additional format of fixed-point with 18 binary places displayed as a floating point value with six decimal places.

---
 include/linux/resctrl_types.h |  8 ++++++++
 fs/resctrl/internal.h         |  4 ++++
 fs/resctrl/ctrlmondata.c      | 23 ++++++++++++++++++++++-
 fs/resctrl/monitor.c          |  3 +++
 fs/resctrl/rdtgroup.c         |  1 +
 5 files changed, 38 insertions(+), 1 deletion(-)
include/linux/resctrl_types.h +8 -0
index 898068a99ef7..fbd4b55c41aa 100644
@@ -58,6 +58,14 @@ enum resctrl_event_id {
 #define QOS_NUM_MBM_EVENTS	(QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1)
 #define MBM_EVENT_IDX(evt)	((evt) - QOS_L3_MBM_TOTAL_EVENT_ID)
 
+/*
+ * Event value display types
+ */
+enum resctrl_event_type {
+	EVT_TYPE_U64,
+	EVT_TYPE_U46_18,
+};
+
 static inline bool resctrl_is_mbm_event(int e)
 {
 	return (e >= QOS_L3_MBM_TOTAL_EVENT_ID &&
fs/resctrl/internal.h +4 -0
index 74a77794364d..4a840e683e96 100644
@@ -71,6 +71,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
  * struct mon_evt - Entry in the event list of a resource
  * @evtid:		event id
  * @name:		name of the event
+ * @type:		format for display to user
  * @configurable:	true if the event is configurable
  * @any_cpu:		true if this event can be read from any CPU
  * @list:		entry in &rdt_resource->evt_list
@@ -79,6 +80,7 @@ struct mon_evt {
 	enum resctrl_event_id	evtid;
 	enum resctrl_res_level	rid;
 	char			*name;
+	enum resctrl_event_type	type;
 	bool			configurable;
 	bool			any_cpu;
 	struct list_head	list;
@@ -89,6 +91,7 @@ struct mon_evt {
  * @list:            List of all allocated structures.
  * @rid:             Resource id associated with the event file.
  * @evtid:           Event id associated with the event file.
+ * @type:            Format for display to user
  * @sum:             Set when event must be summed across multiple
  *                   domains.
  * @domid:           When @sum is zero this is the domain to which
@@ -104,6 +107,7 @@ struct mon_data {
 	struct list_head list;
 	unsigned int rid;
 	enum resctrl_event_id evtid;
+	enum resctrl_event_type type;
 	unsigned int sum;
 	unsigned int domid;
 	bool any_cpu;
fs/resctrl/ctrlmondata.c +22 -1
index cd77960657f0..5ea8113c96ac 100644
@@ -562,6 +562,27 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 	resctrl_arch_mon_ctx_free(r, evtid, rr->arch_mon_ctx);
 }
 
+#define NUM_FRAC_BITS	18
+#define FRAC_MASK	GENMASK(NUM_FRAC_BITS - 1, 0)
+
+static void show_value(struct seq_file *m, enum resctrl_event_type type, u64 val)
+{
+	u64 frac;
+
+	switch (type) {
+	case EVT_TYPE_U64:
+		seq_printf(m, "%llu\n", val);
+		break;
+	case EVT_TYPE_U46_18:
+		frac = val & FRAC_MASK;
+		frac = frac * 1000000;
+		frac += 1ul << (NUM_FRAC_BITS - 1);
+		frac >>= NUM_FRAC_BITS;
+		seq_printf(m, "%llu.%06llu\n", val >> NUM_FRAC_BITS, frac);
+		break;
+	}
+}
+
 int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 {
 	struct kernfs_open_file *of = m->private;
@@ -633,7 +654,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	else if (rr.err == -EINVAL)
 		seq_puts(m, "Unavailable\n");
 	else
-		seq_printf(m, "%llu\n", rr.val);
+		show_value(m, md->type, rr.val);
 
 out:
 	rdtgroup_kn_unlock(of->kn);
fs/resctrl/monitor.c +3 -0
index 1cf0b085e07a..1efad57d1d85 100644
@@ -847,16 +847,19 @@ static struct mon_evt all_events[QOS_NUM_EVENTS] = {
 		.name	= "llc_occupancy",
 		.evtid	= QOS_L3_OCCUP_EVENT_ID,
 		.rid	= RDT_RESOURCE_L3,
+		.type	= EVT_TYPE_U64,
 	},
 	[QOS_L3_MBM_TOTAL_EVENT_ID] = {
 		.name	= "mbm_total_bytes",
 		.evtid	= QOS_L3_MBM_TOTAL_EVENT_ID,
 		.rid	= RDT_RESOURCE_L3,
+		.type	= EVT_TYPE_U64,
 	},
 	[QOS_L3_MBM_LOCAL_EVENT_ID] = {
 		.name	= "mbm_local_bytes",
 		.evtid	= QOS_L3_MBM_LOCAL_EVENT_ID,
 		.rid	= RDT_RESOURCE_L3,
+		.type	= EVT_TYPE_U64,
 	},
 };
 
fs/resctrl/rdtgroup.c +1 -0
index 97c2ba8af930..bd41f7a0f416 100644
@@ -2927,6 +2927,7 @@ static struct mon_data *mon_get_kn_priv(int rid, int domid, struct mon_evt *mevt
 	priv->sum = do_sum;
 	priv->evtid = mevt->evtid;
 	priv->any_cpu = mevt->any_cpu;
+	priv->type = mevt->type;
 	list_add_tail(&priv->list, &kn_priv_list);
 
 	return priv;
Signature · 1 line
2.48.1
maintainer
#30 lore raw

Re: [PATCH v3 11/26] fs/resctrl: Add support for additional monitor event display formats

Hi Tony,

Please add context.

Quoted 1 line · Tony Luck

Add a type field to both the mon_evt and mon_data structures.

This is getting too much? How about mon_data pointing to mon_evt?

include/linux/resctrl_types.h

Quoted 20 lines

Legacy monitor events are still all displayed as an unsigned decimal 64-bit integer.

Add an additional format of fixed-point with 18 binary places displayed as a floating point value with six decimal places.

---
 include/linux/resctrl_types.h |  8 ++++++++
 fs/resctrl/internal.h         |  4 ++++
 fs/resctrl/ctrlmondata.c      | 23 ++++++++++++++++++++++-
 fs/resctrl/monitor.c          |  3 +++
 fs/resctrl/rdtgroup.c         |  1 +
 5 files changed, 38 insertions(+), 1 deletion(-)
include/linux/resctrl_types.h +0 -0
index 898068a99ef7..fbd4b55c41aa 100644

Why is this exposed to architecture? There are no users in arch code. changelog gives no hints on how to interpret this.

include/linux/resctrl_types.h enum resctrl_event_id { · line 62

Quoted 6 lines
include/linux/resctrl_types.h +2 -0
@@ -58,6 +58,14 @@ enum resctrl_event_id {
 #define QOS_NUM_MBM_EVENTS	(QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1)
 #define MBM_EVENT_IDX(evt)	((evt) - QOS_L3_MBM_TOTAL_EVENT_ID)
 
+/*
+ * Event value display types

How about "Event value display formats"?

include/linux/resctrl_types.h enum resctrl_event_id { · line 64

Quoted 2 lines
include/linux/resctrl_types.h +2 -0
+ */
+enum resctrl_event_type {

resctrl_event_format/resctrl_event_fmt?

include/linux/resctrl_types.h enum resctrl_event_id { · line 65

Quoted 1 line
include/linux/resctrl_types.h +1 -0
+	EVT_TYPE_U64,

EVT_FMT_U64/EVT_FORMAT_U64?

Quoted 47 lines
include/linux/resctrl_types.h +3 -0
+	EVT_TYPE_U46_18,
+};
+
 static inline bool resctrl_is_mbm_event(int e)
 {
 	return (e >= QOS_L3_MBM_TOTAL_EVENT_ID &&
fs/resctrl/internal.h +4 -0
index 74a77794364d..4a840e683e96 100644
@@ -71,6 +71,7 @@ static inline struct rdt_fs_context *rdt_fc2context(struct fs_context *fc)
  * struct mon_evt - Entry in the event list of a resource
  * @evtid:		event id
  * @name:		name of the event
+ * @type:		format for display to user
  * @configurable:	true if the event is configurable
  * @any_cpu:		true if this event can be read from any CPU
  * @list:		entry in &rdt_resource->evt_list
@@ -79,6 +80,7 @@ struct mon_evt {
 	enum resctrl_event_id	evtid;
 	enum resctrl_res_level	rid;
 	char			*name;
+	enum resctrl_event_type	type;
 	bool			configurable;
 	bool			any_cpu;
 	struct list_head	list;
@@ -89,6 +91,7 @@ struct mon_evt {
  * @list:            List of all allocated structures.
  * @rid:             Resource id associated with the event file.
  * @evtid:           Event id associated with the event file.
+ * @type:            Format for display to user
  * @sum:             Set when event must be summed across multiple
  *                   domains.
  * @domid:           When @sum is zero this is the domain to which
@@ -104,6 +107,7 @@ struct mon_data {
 	struct list_head list;
 	unsigned int rid;
 	enum resctrl_event_id evtid;
+	enum resctrl_event_type type;
 	unsigned int sum;
 	unsigned int domid;
 	bool any_cpu;
fs/resctrl/ctrlmondata.c +0 -0
index cd77960657f0..5ea8113c96ac 100644

fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 568

Quoted 7 lines
fs/resctrl/ctrlmondata.c +4 -0
@@ -562,6 +562,27 @@ void mon_event_read(struct rmid_read *rr, struct rdt_resource *r,
 	resctrl_arch_mon_ctx_free(r, evtid, rr->arch_mon_ctx);
 }
 
+#define NUM_FRAC_BITS	18
+#define FRAC_MASK	GENMASK(NUM_FRAC_BITS - 1, 0)
+
+static void show_value(struct seq_file *m, enum resctrl_event_type type, u64 val)

show_value() is a bit vague ... print_event_value() ?

fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 579

Quoted 11 lines
fs/resctrl/ctrlmondata.c +11 -0
+{
+	u64 frac;
+
+	switch (type) {
+	case EVT_TYPE_U64:
+		seq_printf(m, "%llu\n", val);
+		break;
+	case EVT_TYPE_U46_18:
+		frac = val & FRAC_MASK;
+		frac = frac * 1000000;
+		frac += 1ul << (NUM_FRAC_BITS - 1);

Could you please help me understand why above line is needed? Seems like shift below will just undo it?

Quoted 47 lines
fs/resctrl/ctrlmondata.c +7 -1
+		frac >>= NUM_FRAC_BITS;
+		seq_printf(m, "%llu.%06llu\n", val >> NUM_FRAC_BITS, frac);
+		break;
+	}
+}
+
 int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 {
 	struct kernfs_open_file *of = m->private;
@@ -633,7 +654,7 @@ int rdtgroup_mondata_show(struct seq_file *m, void *arg)
 	else if (rr.err == -EINVAL)
 		seq_puts(m, "Unavailable\n");
 	else
-		seq_printf(m, "%llu\n", rr.val);
+		show_value(m, md->type, rr.val);
 
 out:
 	rdtgroup_kn_unlock(of->kn);
fs/resctrl/monitor.c +3 -0
index 1cf0b085e07a..1efad57d1d85 100644
@@ -847,16 +847,19 @@ static struct mon_evt all_events[QOS_NUM_EVENTS] = {
 		.name	= "llc_occupancy",
 		.evtid	= QOS_L3_OCCUP_EVENT_ID,
 		.rid	= RDT_RESOURCE_L3,
+		.type	= EVT_TYPE_U64,
 	},
 	[QOS_L3_MBM_TOTAL_EVENT_ID] = {
 		.name	= "mbm_total_bytes",
 		.evtid	= QOS_L3_MBM_TOTAL_EVENT_ID,
 		.rid	= RDT_RESOURCE_L3,
+		.type	= EVT_TYPE_U64,
 	},
 	[QOS_L3_MBM_LOCAL_EVENT_ID] = {
 		.name	= "mbm_local_bytes",
 		.evtid	= QOS_L3_MBM_LOCAL_EVENT_ID,
 		.rid	= RDT_RESOURCE_L3,
+		.type	= EVT_TYPE_U64,
 	},
 };
 
fs/resctrl/rdtgroup.c +0 -0
index 97c2ba8af930..bd41f7a0f416 100644

fs/resctrl/rdtgroup.c static struct mon_data *mon_get_kn_priv(int rid, int domid, struct mon_evt *mevt · line 2933

Quoted 7 lines
fs/resctrl/rdtgroup.c +1 -0
@@ -2927,6 +2927,7 @@ static struct mon_data *mon_get_kn_priv(int rid, int domid, struct mon_evt *mevt
 	priv->sum = do_sum;
 	priv->evtid = mevt->evtid;
 	priv->any_cpu = mevt->any_cpu;
+	priv->type = mevt->type;
 	list_add_tail(&priv->list, &kn_priv_list);
 
 	return priv;

Reinette

author maintainer
#31 lore raw

Re: [PATCH v3 11/26] fs/resctrl: Add support for additional monitor event display formats

Quoted 8 lines · Reinette Chatre

fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 579

Quoted 4 lines
fs/resctrl/ctrlmondata.c +4 -0
+	case EVT_TYPE_U46_18:
+		frac = val & FRAC_MASK;
+		frac = frac * 1000000;
+		frac += 1ul << (NUM_FRAC_BITS - 1);

Could you please help me understand why above line is needed? Seems like shift below will just undo it?

Quoted 4 lines

fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 583

Quoted 4 lines
fs/resctrl/ctrlmondata.c +4 -0
+		frac >>= NUM_FRAC_BITS;
+		seq_printf(m, "%llu.%06llu\n", val >> NUM_FRAC_BITS, frac);
+		break;
+	}

The extra addtion is to round the value to nearest decimal supported value.

E.g. take the case where val == 1 This is a 1 in the 18th binary place, so the precise decimal representation is:

	1 / 2^18 = 0.000003814697265625

rounding to six decimal places should give: 0.000004

If you run the above code without the small addition you get:

	frac = val & FRAC_MASK; // frac == 1
	frac = frac * 1000000;	// frac == 1000000
	frac >>= NUM_FRAC_BITS;	// frac == 3

So the output with be the truncated, not rounded, 0.000003

The addition will have a "carry bit" in to the upper bits. That isn't lost when shifting right. The value added is as if there was a "1" in the 19th binary place.

-Tony

maintainer
#32 lore raw

Re: [PATCH v3 11/26] fs/resctrl: Add support for additional monitor event display formats

Hi Tony,

Quoted 30 lines · Tony
Quoted 8 lines · Reinette Chatre

fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 579

Quoted 4 lines
fs/resctrl/ctrlmondata.c +4 -0
+	case EVT_TYPE_U46_18:
+		frac = val & FRAC_MASK;
+		frac = frac * 1000000;
+		frac += 1ul << (NUM_FRAC_BITS - 1);

Could you please help me understand why above line is needed? Seems like shift below will just undo it?

Quoted 4 lines

fs/resctrl/ctrlmondata.c void mon_event_read(struct rmid_read *rr, struct rdt_resource *r, · line 583

Quoted 4 lines
fs/resctrl/ctrlmondata.c +4 -0
+		frac >>= NUM_FRAC_BITS;
+		seq_printf(m, "%llu.%06llu\n", val >> NUM_FRAC_BITS, frac);
+		break;
+	}

The extra addtion is to round the value to nearest decimal supported value.

E.g. take the case where val == 1 This is a 1 in the 18th binary place, so the precise decimal representation is:

	1 / 2^18 = 0.000003814697265625

rounding to six decimal places should give: 0.000004

If you run the above code without the small addition you get:

	frac = val & FRAC_MASK; // frac == 1
	frac = frac * 1000000;	// frac == 1000000
	frac >>= NUM_FRAC_BITS;	// frac == 3
Quoted 6 lines

So the output with be the truncated, not rounded, 0.000003

The addition will have a "carry bit" in to the upper bits. That isn't lost when shifting right. The value added is as if there was a "1" in the 19th binary place.

Thank you very much.

Reinette

author maintainer
#33 lore raw

PATCH v3 12/26 fs/resctrl: Add hook for architecture code to set monitor event attributes

In reply to Tony Luck

Architecture code knows whether an event can be read from any CPU, or from a CPU on a specific domain. It also knows what format to use when printing each event value.

Add a hook to set mon_event.any_cpu and mon_event.type.

---
 include/linux/resctrl.h |  3 +++
 fs/resctrl/monitor.c    | 12 ++++++++++++
 2 files changed, 15 insertions(+)
include/linux/resctrl.h +3 -0
index 0fce626605b9..8ac77b738de5 100644
@@ -418,6 +418,9 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d
 void resctrl_online_cpu(unsigned int cpu);
 void resctrl_offline_cpu(unsigned int cpu);
 
+int resctrl_set_event_attributes(enum resctrl_event_id evt,
+				 enum resctrl_event_type type, bool any_cpu);
+
 /**
  * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid
  *			      for this resource and domain.
fs/resctrl/monitor.c +12 -0
index 1efad57d1d85..5846a13c631a 100644
@@ -863,6 +863,18 @@ static struct mon_evt all_events[QOS_NUM_EVENTS] = {
 	},
 };
 
+int resctrl_set_event_attributes(enum resctrl_event_id evt,
+				 enum resctrl_event_type type, bool any_cpu)
+{
+	if (evt >= QOS_NUM_EVENTS)
+		return -ENOENT;
+
+	all_events[evt].type = type;
+	all_events[evt].any_cpu = any_cpu;
+
+	return 0;
+}
+
 int rdt_lookup_evtid_by_name(char *name)
 {
 	int evt;
Signature · 1 line
2.48.1
maintainer
#34 lore raw

Re: [PATCH v3 12/26] fs/resctrl: Add hook for architecture code to set monitor event attributes

Hi Tony,

Quoted 5 lines · Tony Luck

Architecture code knows whether an event can be read from any CPU, or from a CPU on a specific domain. It also knows what format to use when printing each event value.

Add a hook to set mon_event.any_cpu and mon_event.type.

If the architecture modifies the output format then the values exposed to user space will look different between architectures. User space will need to know how to parse the data. We do not want user space to need to know which architecture it is running on to determine how to interact with user space so this makes me think that this change needs to be accompanied with a change that exposes the event format to user space.

Quoted 26 lines
---
 include/linux/resctrl.h |  3 +++
 fs/resctrl/monitor.c    | 12 ++++++++++++
 2 files changed, 15 insertions(+)
include/linux/resctrl.h +3 -0
index 0fce626605b9..8ac77b738de5 100644
@@ -418,6 +418,9 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d
 void resctrl_online_cpu(unsigned int cpu);
 void resctrl_offline_cpu(unsigned int cpu);
 
+int resctrl_set_event_attributes(enum resctrl_event_id evt,
+				 enum resctrl_event_type type, bool any_cpu);
+
 /**
  * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid
  *			      for this resource and domain.
fs/resctrl/monitor.c +0 -0
index 1efad57d1d85..5846a13c631a 100644

fs/resctrl/monitor.c static struct mon_evt all_events[QOS_NUM_EVENTS] = { · line 867

Quoted 5 lines
fs/resctrl/monitor.c +2 -0
@@ -863,6 +863,18 @@ static struct mon_evt all_events[QOS_NUM_EVENTS] = {
 	},
 };
 
+int resctrl_set_event_attributes(enum resctrl_event_id evt,
+				 enum resctrl_event_type type, bool any_cpu)

So this is not actually a hook (in the resctrl_arch sense) but a direct interface for arch code to change resctrl fs internals ... this needs to be done with care.

fs/resctrl/monitor.c static struct mon_evt all_events[QOS_NUM_EVENTS] = { · line 880

Quoted 13 lines
fs/resctrl/monitor.c +10 -0
+{
+	if (evt >= QOS_NUM_EVENTS)
+		return -ENOENT;
+
+	all_events[evt].type = type;
+	all_events[evt].any_cpu = any_cpu;
+
+	return 0;
+}
+
 int rdt_lookup_evtid_by_name(char *name)
 {
 	int evt;

Reinette

author maintainer
#35 lore raw

Re: [PATCH v3 12/26] fs/resctrl: Add hook for architecture code to set monitor event attributes

Quoted 7 lines · Reinette Chatre

Hi Tony,

Quoted 4 lines · Tony Luck

Architecture code knows whether an event can be read from any CPU, or from a CPU on a specific domain. It also knows what format to use when printing each event value.

Quoted 8 lines
Quoted 1 line

Add a hook to set mon_event.any_cpu and mon_event.type.

If the architecture modifies the output format then the values exposed to user space will look different between architectures. User space will need to know how to parse the data. We do not want user space to need to know which architecture it is running on to determine how to interact with user space so this makes me think that this change needs to be accompanied with a change that exposes the event format to user space.

Would it be enough to include this in Documentation? I.e. add specific entries for "core_energy" to say that it is reported as a floating point value with unit Joules, and "activity" is reported as a floating point value with unit of Farads.

Alternatively the filesystem code could convert the fixed point values to integer values of micro-Joules and micro-Farads. Then the filesystem code can print with "%llu" just like every other event. Would still need the Documentation entries to explain the units. This has the limitation that some future implementation that measures in greater precision would be forced to round to nearest micro-{Joule,Farad}.

File system code controls the formating options available, so options for architecture code to break the user interface are limited.

-Tony

maintainer
#36 lore raw

Re: [PATCH v3 12/26] fs/resctrl: Add hook for architecture code to set monitor event attributes

Hi Tony,

Quoted 17 lines · Tony
Quoted 7 lines · Reinette Chatre

Hi Tony,

Quoted 4 lines · Tony Luck

Architecture code knows whether an event can be read from any CPU, or from a CPU on a specific domain. It also knows what format to use when printing each event value.

Quoted 8 lines
Quoted 1 line

Add a hook to set mon_event.any_cpu and mon_event.type.

If the architecture modifies the output format then the values exposed to user space will look different between architectures. User space will need to know how to parse the data. We do not want user space to need to know which architecture it is running on to determine how to interact with user space so this makes me think that this change needs to be accompanied with a change that exposes the event format to user space.

Quoted 4 lines

Would it be enough to include this in Documentation? I.e. add specific entries for "core_energy" to say that it is reported as a floating point value with unit Joules, and "activity" is reported as a floating point value with unit of Farads.

If it is always a floating point, yes, but this patch enables an architecture to modify output to be different.

Quoted 7 lines

Alternatively the filesystem code could convert the fixed point values to integer values of micro-Joules and micro-Farads. Then the filesystem code can print with "%llu" just like every other event. Would still need the Documentation entries to explain the units. This has the limitation that some future implementation that measures in greater precision would be forced to round to nearest micro-{Joule,Farad}.

I do not think we are talking about the same issue.

Quoted 3 lines

File system code controls the formating options available, so options for architecture code to break the user interface are limited.

File system gives away the control with this patch, no?

This patch enables the architecture to change the output format. On one architecture the event may thus be presented as floating point while on another architecture the event will not be floating point.

Reinette

author maintainer
#37 lore raw

PATCH v3 13/26 fs/resctrl: Add an architectural hook called for each mount

In reply to Tony Luck

Enumeration of Intel telemetry events is not complete when the resctrl "late_init" code is executed.

Add a hook at the beginning of the mount code that will be used to check for telemetry events and initialize if any are found.

The hook is called on every mount. But expectations are that most actions (like enumeration) will only need to be performed on the first call.

---
 include/linux/resctrl.h            | 3 +++
 arch/x86/kernel/cpu/resctrl/core.c | 9 +++++++++
 fs/resctrl/rdtgroup.c              | 2 ++
 3 files changed, 14 insertions(+)
include/linux/resctrl.h +3 -0
index 8ac77b738de5..25f51a57b0b7 100644
@@ -421,6 +421,9 @@ void resctrl_offline_cpu(unsigned int cpu);
 int resctrl_set_event_attributes(enum resctrl_event_id evt,
 				 enum resctrl_event_type type, bool any_cpu);
 
+/* Architecture hook called for each file system mount */
+void resctrl_arch_mount(void);
+
 /**
  * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid
  *			      for this resource and domain.
arch/x86/kernel/cpu/resctrl/core.c +9 -0
index 59844fd7105f..a066a9c54a1f 100644
@@ -711,6 +711,15 @@ static int resctrl_arch_offline_cpu(unsigned int cpu)
 	return 0;
 }
 
+void resctrl_arch_mount(void)
+{
+	static bool only_once;
+
+	if (only_once)
+		return;
+	only_once = true;
+}
+
 enum {
 	RDT_FLAG_CMT,
 	RDT_FLAG_MBM_TOTAL,
fs/resctrl/rdtgroup.c +2 -0
index bd41f7a0f416..5ca6de6a6e5c 100644
@@ -2564,6 +2564,8 @@ static int rdt_get_tree(struct fs_context *fc)
 	struct rdt_resource *r;
 	int ret;
 
+	resctrl_arch_mount();
+
 	cpus_read_lock();
 	mutex_lock(&rdtgroup_mutex);
 	/*
Signature · 1 line
2.48.1
maintainer
#38 lore raw

Re: [PATCH v3 13/26] fs/resctrl: Add an architectural hook called for each mount

Hi Tony,

Quoted 22 lines · Tony Luck

Enumeration of Intel telemetry events is not complete when the resctrl "late_init" code is executed.

Add a hook at the beginning of the mount code that will be used to check for telemetry events and initialize if any are found.

The hook is called on every mount. But expectations are that most actions (like enumeration) will only need to be performed on the first call.

---
 include/linux/resctrl.h            | 3 +++
 arch/x86/kernel/cpu/resctrl/core.c | 9 +++++++++
 fs/resctrl/rdtgroup.c              | 2 ++
 3 files changed, 14 insertions(+)
include/linux/resctrl.h +0 -0
index 8ac77b738de5..25f51a57b0b7 100644

include/linux/resctrl.h void resctrl_offline_cpu(unsigned int cpu); · line 424

Quoted 4 lines
include/linux/resctrl.h +1 -0
@@ -421,6 +421,9 @@ void resctrl_offline_cpu(unsigned int cpu);
 int resctrl_set_event_attributes(enum resctrl_event_id evt,
 				 enum resctrl_event_type type, bool any_cpu);
 
+/* Architecture hook called for each file system mount */

Please add some description of what architecture could use it for as well as more specific detail of when it is called during mount. I think it is important to highlight and make it part of agreement that resctrl fs calls this on mount before any resctrl fs actions. Considering this, perhaps resctrl_arch_pre_mount()?

It is also worth highlighting in the API doc that fs does not actually call resctrl_arch_mount() on every mount but every mount *attempt* (resctrl may already be mounted) so it is up for arch to maintain any needed state.

(Also see later comment about locking)

Quoted 30 lines
include/linux/resctrl.h +2 -0
+void resctrl_arch_mount(void);
+
 /**
  * resctrl_arch_rmid_read() - Read the eventid counter corresponding to rmid
  *			      for this resource and domain.
arch/x86/kernel/cpu/resctrl/core.c +9 -0
index 59844fd7105f..a066a9c54a1f 100644
@@ -711,6 +711,15 @@ static int resctrl_arch_offline_cpu(unsigned int cpu)
 	return 0;
 }
 
+void resctrl_arch_mount(void)
+{
+	static bool only_once;
+
+	if (only_once)
+		return;
+	only_once = true;
+}
+
 enum {
 	RDT_FLAG_CMT,
 	RDT_FLAG_MBM_TOTAL,
fs/resctrl/rdtgroup.c +0 -0
index bd41f7a0f416..5ca6de6a6e5c 100644

fs/resctrl/rdtgroup.c static int rdt_get_tree(struct fs_context *fc) · line 2571

Quoted 8 lines
fs/resctrl/rdtgroup.c +2 -0
@@ -2564,6 +2564,8 @@ static int rdt_get_tree(struct fs_context *fc)
 	struct rdt_resource *r;
 	int ret;
 
+	resctrl_arch_mount();
+
 	cpus_read_lock();
 	mutex_lock(&rdtgroup_mutex);
 	/*

Could you please elaborate on the locking requirements here? Worth a mention of these expectations in changelog also. That it is called without any locks held and arch is responsible for all locking should be documented as part of API in include/linux/resctrl.h

Reinette

author maintainer
#39 lore raw

PATCH v3 14/26 x86/resctrl: Add first part of telemetry event enumeration

In reply to Tony Luck

The OOBMSM driver provides an interface to discover any RMID based events for "energy" and "perf" classes.

Hold onto references to any pmt_feature_groups that resctrl uses until resctrl exit.

---
 arch/x86/kernel/cpu/resctrl/internal.h  |  8 ++++
 arch/x86/kernel/cpu/resctrl/core.c      |  5 ++
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 62 +++++++++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/Makefile    |  1 +
 4 files changed, 76 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/resctrl/intel_aet.c
arch/x86/kernel/cpu/resctrl/internal.h +8 -0
index 45eabc7919c6..70b63bbc429d 100644
@@ -172,4 +172,12 @@ void __init intel_rdt_mbm_apply_quirk(void);
 
 void rdt_domain_reconfigure_cdp(struct rdt_resource *r);
 
+#ifdef CONFIG_INTEL_AET_RESCTRL
+bool intel_aet_get_events(void);
+void __exit intel_aet_exit(void);
+#else
+static inline bool intel_aet_get_events(void) { return false; }
+static inline void intel_aet_exit(void) { };
+#endif
+
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/core.c +5 -0
index a066a9c54a1f..f0f256a5ac66 100644
@@ -718,6 +718,9 @@ void resctrl_arch_mount(void)
 	if (only_once)
 		return;
 	only_once = true;
+
+	if (!intel_aet_get_events())
+		return;
 }
 
 enum {
@@ -1063,6 +1066,8 @@ late_initcall(resctrl_arch_late_init);
 
 static void __exit resctrl_arch_exit(void)
 {
+	intel_aet_exit();
+
 	cpuhp_remove_state(rdt_online);
 
 	resctrl_exit();
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +62 -0

62 lines not on this page · raw message

arch/x86/kernel/cpu/resctrl/Makefile +1 -0
index c56d3acf8ac7..74c3b2333dde 100644
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_X86_CPU_RESCTRL)		+= core.o rdtgroup.o monitor.o
 obj-$(CONFIG_X86_CPU_RESCTRL)		+= ctrlmondata.o
+obj-$(CONFIG_INTEL_AET_RESCTRL)		+= intel_aet.o
 obj-$(CONFIG_INTEL_AET_RESCTRL)		+= fake_intel_aet_features.o
 obj-$(CONFIG_RESCTRL_FS_PSEUDO_LOCK)	+= pseudo_lock.o
 
Signature · 1 line
2.48.1
maintainer
#40 lore raw

Re: [PATCH v3 14/26] x86/resctrl: Add first part of telemetry event enumeration

Hi Tony,

The following three patches progress as follows:
x86/resctrl: Add first part ...
x86/resctrl: Second stage ...
x86/resctrl: Third phase ...

Could you please make the language consistent?

Quoted 2 lines · Tony Luck

The OOBMSM driver provides an interface to discover any RMID based events for "energy" and "perf" classes.

Please add context about what an RMID based event is.

Quoted 2 lines

Hold onto references to any pmt_feature_groups that resctrl

Please add context about what a "pmt_feature_groups" (intended to be pmt_feature_group?) is.

Quoted 33 lines

uses until resctrl exit.

---
 arch/x86/kernel/cpu/resctrl/internal.h  |  8 ++++
 arch/x86/kernel/cpu/resctrl/core.c      |  5 ++
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 62 +++++++++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/Makefile    |  1 +
 4 files changed, 76 insertions(+)
 create mode 100644 arch/x86/kernel/cpu/resctrl/intel_aet.c
arch/x86/kernel/cpu/resctrl/internal.h +8 -0
index 45eabc7919c6..70b63bbc429d 100644
@@ -172,4 +172,12 @@ void __init intel_rdt_mbm_apply_quirk(void);
 
 void rdt_domain_reconfigure_cdp(struct rdt_resource *r);
 
+#ifdef CONFIG_INTEL_AET_RESCTRL
+bool intel_aet_get_events(void);
+void __exit intel_aet_exit(void);
+#else
+static inline bool intel_aet_get_events(void) { return false; }
+static inline void intel_aet_exit(void) { };
+#endif
+
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/core.c +0 -0
index a066a9c54a1f..f0f256a5ac66 100644

arch/x86/kernel/cpu/resctrl/core.c void resctrl_arch_mount(void) · line 722

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/core.c +2 -0
@@ -718,6 +718,9 @@ void resctrl_arch_mount(void)
 	if (only_once)
 		return;
 	only_once = true;
+
+	if (!intel_aet_get_events())

hmmm ... keep in mind that this is called without any locking and thus there may be risk of parallel calls?

Please document how/if this is relying on some fs features to ensure there are not two instances running at same time.

Quoted 25 lines
arch/x86/kernel/cpu/resctrl/core.c +3 -0
+		return;
 }
 
 enum {
@@ -1063,6 +1066,8 @@ late_initcall(resctrl_arch_late_init);
 
 static void __exit resctrl_arch_exit(void)
 {
+	intel_aet_exit();
+
 	cpuhp_remove_state(rdt_online);
 
 	resctrl_exit();
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +6 -0
new file mode 100644
index 000000000000..8e531ad279b5
@@ -0,0 +1,62 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Resource Director Technology(RDT)
+ * - Intel Application Energy Telemetry
+ *
+ * Copyright (C) 2025 Intel Corporation

arch/x86/kernel/cpu/resctrl/intel_aet.c line 16

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +10 -0
+ *
+ * Author:
+ *    Tony Luck <[email protected]>
+ */
+
+#define pr_fmt(fmt)   "resctrl: " fmt
+
+#include <linux/cpu.h>
+#include <linux/cleanup.h>
+#include "fake_intel_aet_features.h"

This include can be marked as "Temporary" to highlight that it will not stay.

Please separate headers into blocks and sort alphabetically.

arch/x86/kernel/cpu/resctrl/intel_aet.c line 19

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +3 -0
+#include <linux/intel_vsec.h>
+#include <linux/resctrl.h>
+#include <linux/slab.h>

Are all these headers used in code below?

Quoted 22 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +22 -0
+
+#include "internal.h"
+
+static struct pmt_feature_group *feat_energy;
+static struct pmt_feature_group *feat_perf;
+
+DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *,	\
+	if (!IS_ERR_OR_NULL(_T))					\
+		intel_pmt_put_feature_group(_T))
+
+/*
+ * Ask OOBMSM discovery driver for all the RMID based telemetry groups
+ * that it supports.
+ */
+bool intel_aet_get_events(void)
+{
+	struct pmt_feature_group *p1 __free(intel_pmt_put_feature_group) = NULL;
+	struct pmt_feature_group *p2 __free(intel_pmt_put_feature_group) = NULL;
+	bool use_p1, use_p2;
+
+	p1 = intel_pmt_get_regions_by_feature(FEATURE_PER_RMID_ENERGY_TELEM);
+	p2 = intel_pmt_get_regions_by_feature(FEATURE_PER_RMID_PERF_TELEM);

arch/x86/kernel/cpu/resctrl/intel_aet.c line 51

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +10 -0
+	use_p1 = !IS_ERR_OR_NULL(p1);
+	use_p2 = !IS_ERR_OR_NULL(p2);
+
+	if (!use_p1 && !use_p2)
+		return false;
+
+	if (use_p1)
+		feat_energy = no_free_ptr(p1);
+	if (use_p2)
+		feat_perf = no_free_ptr(p2);

This reminds me of something I read recently .... "There's a rule in computer programming that objects appear zero, once, or many times. So code accordingly." Not expecting a change .... just finding it amusing.

Quoted 16 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c new file +11 -0
+
+	return true;
+}
+
+void __exit intel_aet_exit(void)
+{
+	if (feat_energy)
+		intel_pmt_put_feature_group(feat_energy);
+	if (feat_perf)
+		intel_pmt_put_feature_group(feat_perf);
+}
arch/x86/kernel/cpu/resctrl/Makefile +0 -0
index c56d3acf8ac7..74c3b2333dde 100644

arch/x86/kernel/cpu/resctrl/Makefile line 7

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/Makefile +1 -0
@@ -1,6 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_X86_CPU_RESCTRL)		+= core.o rdtgroup.o monitor.o
 obj-$(CONFIG_X86_CPU_RESCTRL)		+= ctrlmondata.o
+obj-$(CONFIG_INTEL_AET_RESCTRL)		+= intel_aet.o
 obj-$(CONFIG_INTEL_AET_RESCTRL)		+= fake_intel_aet_features.o
 obj-$(CONFIG_RESCTRL_FS_PSEUDO_LOCK)	+= pseudo_lock.o
 

Reinette

author maintainer
#41 lore raw

PATCH v3 15/26 x86/resctrl: Second stage of telemetry event enumeration

In reply to Tony Luck

Scan the telemetry_region structures looking for recognised guid values. Count how many are found in each package.

Note that telemetry support depends on at least one of the original RDT monitoring features being enabled (so that the CPU hotplug notifiers for resctrl are running).

---
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 112 +++++++++++++++++++++++-
 1 file changed, 110 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/intel_aet.c +110 -2
index 8e531ad279b5..9d414dd40f8b 100644
@@ -23,10 +23,100 @@
 static struct pmt_feature_group *feat_energy;
 static struct pmt_feature_group *feat_perf;
 
+/* Per-package event groups active on this machine */
+static struct pkg_info {
+	int			count;
+	struct telemetry_region	*regions;
+} *pkg_info;
+
+/**
+ * struct pmt_event - Telemetry event.
+ * @evtid:	Resctrl event id
+ * @evt_offset:	MMIO offset of counter
+ * @type:	Type for format user display of event value
+ */
+struct pmt_event {
+	enum resctrl_event_id	evtid;
+	int			evt_offset;
+	enum resctrl_event_type	type;
+};
+
+/**
+ * struct telem_entry - Summarized form from XML telemetry description
+ * @name:			Name for this group of events
+ * @guid:			Unique ID for this group
+ * @size:			Size of MMIO mapped counter registers
+ * @num_rmids:			Number of RMIDS supported
+ * @overflow_counter_off:	Offset of overflow count
+ * @last_overflow_tstamp_off:	Offset of overflow timestamp
+ * @last_update_tstamp_off:	Offset of last update timestamp
+ * @active:			Marks this group as active on this system
+ * @num_events:			Size of @evts array
+ * @evts:			Telemetry events in this group
+ */
+struct telem_entry {
+	char	*name;
+	int	guid;
+	int	size;
+	int	num_rmids;
+	int	overflow_counter_off;
+	int	last_overflow_tstamp_off;
+	int	last_update_tstamp_off;
+	bool	active;
+	int	num_events;
+	struct pmt_event evts[];
+};
+
+/* All known telemetry event groups */
+static struct telem_entry *telem_entry[] = {
+	NULL
+};
+
+/*
+ * Scan a feature group looking for guids recognized
+ * and update the per-package counts of known groups.
+ */
+static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_group *p)
+{
+	struct telem_entry **tentry;
+	bool found = false;
+
+	if (IS_ERR_OR_NULL(p))
+		return false;
+
+	for (int i = 0; i < p->count; i++) {
+		struct telemetry_region *tr = &p->regions[i];
+
+		for (tentry = telem_entry; *tentry; tentry++) {
+			if (tr->guid == (*tentry)->guid) {
+				if (tr->plat_info.package_id > max_pkgs) {
+					pr_warn_once("Bad package %d\n", tr->plat_info.package_id);
+					continue;
+				}
+				if (tr->size > (*tentry)->size) {
+					pr_warn_once("MMIO region for guid 0x%x too small\n", tr->guid);
+					continue;
+				}
+				found = true;
+				(*tentry)->active = true;
+				pkg[tr->plat_info.package_id].count++;
+				break;
+			}
+		}
+	}
+
+	return found;
+}
+
 DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *,	\
 	if (!IS_ERR_OR_NULL(_T))					\
 		intel_pmt_put_feature_group(_T))
 
+DEFINE_FREE(free_pkg_info, struct pkg_info *,				\
+	if (_T)								\
+		for (int i = 0; i < topology_max_packages(); i++)	\
+			kfree(_T[i].regions);				\
+	kfree(_T))
 /*
  * Ask OOBMSM discovery driver for all the RMID based telemetry groups
  * that it supports.
@@ -35,20 +125,32 @@ bool intel_aet_get_events(void)
 {
 	struct pmt_feature_group *p1 __free(intel_pmt_put_feature_group) = NULL;
 	struct pmt_feature_group *p2 __free(intel_pmt_put_feature_group) = NULL;
+	struct pkg_info *pkg __free(free_pkg_info) = NULL;
+	int num_pkgs = topology_max_packages();
 	bool use_p1, use_p2;
 
+	pkg = kcalloc(num_pkgs, sizeof(*pkg_info), GFP_KERNEL);
+	if (!pkg)
+		return false;
+
 	p1 = intel_pmt_get_regions_by_feature(FEATURE_PER_RMID_ENERGY_TELEM);
 	p2 = intel_pmt_get_regions_by_feature(FEATURE_PER_RMID_PERF_TELEM);
-	use_p1 = !IS_ERR_OR_NULL(p1);
-	use_p2 = !IS_ERR_OR_NULL(p2);
+	use_p1 = count_events(pkg, num_pkgs, p1);
+	use_p2 = count_events(pkg, num_pkgs, p2);
 
 	if (!use_p1 && !use_p2)
 		return false;
 
+	if (!resctrl_arch_mon_capable()) {
+		pr_info("Telemetry available but monitor support disabled\n");
+		return false;
+	}
+
 	if (use_p1)
 		feat_energy = no_free_ptr(p1);
 	if (use_p2)
 		feat_perf = no_free_ptr(p2);
+	pkg_info = no_free_ptr(pkg);
 
 	return true;
 }
@@ -59,4 +161,10 @@ void __exit intel_aet_exit(void)
 		intel_pmt_put_feature_group(feat_energy);
 	if (feat_perf)
 		intel_pmt_put_feature_group(feat_perf);
+
+	if (pkg_info) {
+		for (int i = 0; i < topology_max_packages(); i++)
+			kfree(pkg_info[i].regions);
+	}
+	kfree(pkg_info);
 }
Signature · 1 line
2.48.1
maintainer
#42 lore raw

Re: [PATCH v3 15/26] x86/resctrl: Second stage of telemetry event enumeration

Hi Tony,

Quoted 1 line · Tony Luck

Scan the telemetry_region structures looking for recognised guid

Please add context before description of what patch does.

Also, please pick British or American English and stick with it.

Quoted 17 lines

values. Count how many are found in each package.

Note that telemetry support depends on at least one of the original RDT monitoring features being enabled (so that the CPU hotplug notifiers for resctrl are running).

---
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 112 +++++++++++++++++++++++-
 1 file changed, 110 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/resctrl/intel_aet.c +0 -0
index 8e531ad279b5..9d414dd40f8b 100644
@@ -23,10 +23,100 @@
 static struct pmt_feature_group *feat_energy;

arch/x86/kernel/cpu/resctrl/intel_aet.c line 33

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +8 -0
 static struct pmt_feature_group *feat_perf;
 
+/* Per-package event groups active on this machine */
+static struct pkg_info {
+	int			count;
+	struct telemetry_region	*regions;
+} *pkg_info;
+
+/**
+ * struct pmt_event - Telemetry event.

Why does it need "pmt" prefix? Can it be "telem_event" to match telem_entry?

arch/x86/kernel/cpu/resctrl/intel_aet.c line 36

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +3 -0
+ * @evtid:	Resctrl event id
+ * @evt_offset:	MMIO offset of counter
+ * @type:	Type for format user display of event value

I cannot make sense of "Type for format user display of event value"

arch/x86/kernel/cpu/resctrl/intel_aet.c line 45

Quoted 9 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +9 -0
+ */
+struct pmt_event {
+	enum resctrl_event_id	evtid;
+	int			evt_offset;
+	enum resctrl_event_type	type;
+};
+
+/**
+ * struct telem_entry - Summarized form from XML telemetry description

Copying from v2 review:
"It is not clear to me how useful it is to document that this is "Summarized form from XML telemetry description". Either more detail should be added to help reader understand what XML is being talked about or the description should be a summary of what this data structure represents."

arch/x86/kernel/cpu/resctrl/intel_aet.c line 50

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +5 -0
+ * @name:			Name for this group of events
+ * @guid:			Unique ID for this group
+ * @size:			Size of MMIO mapped counter registers
+ * @num_rmids:			Number of RMIDS supported
+ * @overflow_counter_off:	Offset of overflow count

Description just rewrites member name and changes "counter" to "count". Could description have more details about what is represented by this? What overflowed?

arch/x86/kernel/cpu/resctrl/intel_aet.c line 51

Quoted 1 line
arch/x86/kernel/cpu/resctrl/intel_aet.c +1 -0
+ * @last_overflow_tstamp_off:	Offset of overflow timestamp

What overflowed at this timestamp?

arch/x86/kernel/cpu/resctrl/intel_aet.c line 52

Quoted 1 line
arch/x86/kernel/cpu/resctrl/intel_aet.c +1 -0
+ * @last_update_tstamp_off:	Offset of last update timestamp

What was updated at this timestamp?

arch/x86/kernel/cpu/resctrl/intel_aet.c line 53

Quoted 1 line
arch/x86/kernel/cpu/resctrl/intel_aet.c +1 -0
+ * @active:			Marks this group as active on this system

What does it mean when a group is "active"?

arch/x86/kernel/cpu/resctrl/intel_aet.c line 54

Quoted 1 line
arch/x86/kernel/cpu/resctrl/intel_aet.c +1 -0
+ * @num_events:			Size of @evts array

Would __counted_by() be useful?

arch/x86/kernel/cpu/resctrl/intel_aet.c line 64

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +10 -0
+ * @evts:			Telemetry events in this group
+ */
+struct telem_entry {
+	char	*name;
+	int	guid;
+	int	size;
+	int	num_rmids;
+	int	overflow_counter_off;
+	int	last_overflow_tstamp_off;
+	int	last_update_tstamp_off;

Most of types are "int" ... I do not expect many of these types to be negative so I would like to check if int is most appropriate for all? Usually size_t is used for size and off_t/loff_t is available for offsets.

arch/x86/kernel/cpu/resctrl/intel_aet.c line 67

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +3 -0
+	bool	active;
+	int	num_events;
+	struct pmt_event evts[];

(missing tab)

arch/x86/kernel/cpu/resctrl/intel_aet.c line 70

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +3 -0
+};
+
+/* All known telemetry event groups */

This is more useful by not being "All known Summarized form from XML telemetry description".

arch/x86/kernel/cpu/resctrl/intel_aet.c line 76

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6 -0
+static struct telem_entry *telem_entry[] = {
+	NULL
+};
+
+/*
+ * Scan a feature group looking for guids recognized

Switch from British to American English in same patch.

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6 -0
+ * and update the per-package counts of known groups.
+ */
+static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_group *p)
+{
+	struct telem_entry **tentry;
+	bool found = false;

arch/x86/kernel/cpu/resctrl/intel_aet.c line 92

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +10 -0
+
+	if (IS_ERR_OR_NULL(p))
+		return false;
+
+	for (int i = 0; i < p->count; i++) {
+		struct telemetry_region *tr = &p->regions[i];
+
+		for (tentry = telem_entry; *tentry; tentry++) {
+			if (tr->guid == (*tentry)->guid) {
+				if (tr->plat_info.package_id > max_pkgs) {

Should this be >=?

Quoted 31 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +23 -0
+					pr_warn_once("Bad package %d\n", tr->plat_info.package_id);
+					continue;
+				}
+				if (tr->size > (*tentry)->size) {
+					pr_warn_once("MMIO region for guid 0x%x too small\n", tr->guid);
+					continue;
+				}
+				found = true;
+				(*tentry)->active = true;
+				pkg[tr->plat_info.package_id].count++;
+				break;
+			}
+		}
+	}
+
+	return found;
+}
+
 DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *,	\
 	if (!IS_ERR_OR_NULL(_T))					\
 		intel_pmt_put_feature_group(_T))
 
+DEFINE_FREE(free_pkg_info, struct pkg_info *,				\
+	if (_T)								\
+		for (int i = 0; i < topology_max_packages(); i++)	\
+			kfree(_T[i].regions);				\
+	kfree(_T))
 /*
  * Ask OOBMSM discovery driver for all the RMID based telemetry groups
  * that it supports.

arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 132

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +3 -0
@@ -35,20 +125,32 @@ bool intel_aet_get_events(void)
 {
 	struct pmt_feature_group *p1 __free(intel_pmt_put_feature_group) = NULL;
 	struct pmt_feature_group *p2 __free(intel_pmt_put_feature_group) = NULL;
+	struct pkg_info *pkg __free(free_pkg_info) = NULL;
+	int num_pkgs = topology_max_packages();
 	bool use_p1, use_p2;
 
+	pkg = kcalloc(num_pkgs, sizeof(*pkg_info), GFP_KERNEL);

sizeof(*pkg)?

Reinette

author maintainer
#43 lore raw

PATCH v3 16/26 x86/resctrl: Third phase of telemetry event enumeration

In reply to Tony Luck

Allocate per-package arrays for the known telemetry_regions and initialize with pointers to the structures acquired from the intel_pmt_get_regions_by_feature() calls.

---
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 38 +++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_aet.c +38 -0
index 9d414dd40f8b..fb03f2e76306 100644
@@ -108,6 +108,30 @@ static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_
 	return found;
 }
 
+/*
+ * Copy the pointers to telemetry regions associated with a given package
+ * and with known guids over to the pkg_info structure for that package.
+ */
+static int setup(struct pkg_info *pkg, int pkgnum, struct pmt_feature_group *p, int slot)
+{
+	struct telem_entry **tentry;
+
+	for (int i = 0; i < p->count; i++) {
+		for (tentry = telem_entry; *tentry; tentry++) {
+			if (!(*tentry)->active)
+				continue;
+			if (pkgnum != p->regions[i].plat_info.package_id)
+				continue;
+			if (p->regions[i].guid != (*tentry)->guid)
+				continue;
+
+			pkg[pkgnum].regions[slot++] =  p->regions[i];
+		}
+	}
+
+	return slot;
+}
+
 DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *,	\
 	if (!IS_ERR_OR_NULL(_T))					\
 		intel_pmt_put_feature_group(_T))
@@ -128,6 +152,7 @@ bool intel_aet_get_events(void)
 	struct pkg_info *pkg __free(free_pkg_info) = NULL;
 	int num_pkgs = topology_max_packages();
 	bool use_p1, use_p2;
+	int slot;
 
 	pkg = kcalloc(num_pkgs, sizeof(*pkg_info), GFP_KERNEL);
 	if (!pkg)
@@ -146,6 +171,19 @@ bool intel_aet_get_events(void)
 		return false;
 	}
 
+	for (int i = 0; i < num_pkgs; i++) {
+		if (!pkg[i].count)
+			continue;
+		pkg[i].regions = kmalloc_array(pkg[i].count, sizeof(*pkg[i].regions), GFP_KERNEL);
+		if (!pkg[i].regions)
+			return false;
+		slot = 0;
+		if (use_p1)
+			slot = setup(pkg, i, p1, slot);
+		if (use_p2)
+			slot = setup(pkg, i, p2, slot);
+	}
+
 	if (use_p1)
 		feat_energy = no_free_ptr(p1);
 	if (use_p2)
Signature · 1 line
2.48.1
maintainer
#44 lore raw

Re: [PATCH v3 16/26] x86/resctrl: Third phase of telemetry event enumeration

Hi Tony,

Quoted 3 lines · Tony Luck

Allocate per-package arrays for the known telemetry_regions and initialize with pointers to the structures acquired from the intel_pmt_get_regions_by_feature() calls.

Why?

Quoted 11 lines
---
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 38 +++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_aet.c +0 -0
index 9d414dd40f8b..fb03f2e76306 100644

arch/x86/kernel/cpu/resctrl/intel_aet.c static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_ · line 115

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +5 -0
@@ -108,6 +108,30 @@ static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_
 	return found;
 }
 
+/*
+ * Copy the pointers to telemetry regions associated with a given package
+ * and with known guids over to the pkg_info structure for that package.
+ */
+static int setup(struct pkg_info *pkg, int pkgnum, struct pmt_feature_group *p, int slot)

setup() is very generic. I do not have a suggestion since it is not yet clear what this does (more below).

arch/x86/kernel/cpu/resctrl/intel_aet.c static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_ · line 128

Quoted 13 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +13 -0
+{
+	struct telem_entry **tentry;
+
+	for (int i = 0; i < p->count; i++) {
+		for (tentry = telem_entry; *tentry; tentry++) {
+			if (!(*tentry)->active)
+				continue;
+			if (pkgnum != p->regions[i].plat_info.package_id)
+				continue;
+			if (p->regions[i].guid != (*tentry)->guid)
+				continue;
+
+			pkg[pkgnum].regions[slot++] =  p->regions[i];

(please fix spacing)

hmmm .. this looks like one structure copied to another, not the pointer copy the comment mentions.

Quoted 18 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +7 -0
+		}
+	}
+
+	return slot;
+}
+
 DEFINE_FREE(intel_pmt_put_feature_group, struct pmt_feature_group *,	\
 	if (!IS_ERR_OR_NULL(_T))					\
 		intel_pmt_put_feature_group(_T))
@@ -128,6 +152,7 @@ bool intel_aet_get_events(void)
 	struct pkg_info *pkg __free(free_pkg_info) = NULL;
 	int num_pkgs = topology_max_packages();
 	bool use_p1, use_p2;
+	int slot;
 
 	pkg = kcalloc(num_pkgs, sizeof(*pkg_info), GFP_KERNEL);
 	if (!pkg)

arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 177

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +4 -0
@@ -146,6 +171,19 @@ bool intel_aet_get_events(void)
 		return false;
 	}
 
+	for (int i = 0; i < num_pkgs; i++) {
+		if (!pkg[i].count)
+			continue;
+		pkg[i].regions = kmalloc_array(pkg[i].count, sizeof(*pkg[i].regions), GFP_KERNEL);

As I understand it sizeof(*pkg[i].regions) would be the size of a struct telemetry_region that the code in setup() initializes by copying the data from the struct pmt_feature_group.

The changelog and comments creates impression that resctrl's initialization consists of adding pointers to the data in struct pmt_feature_group but as I read it the data is copied instead. Am I reading it wrong?

arch/x86/kernel/cpu/resctrl/intel_aet.c bool intel_aet_get_events(void) · line 189

Quoted 12 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +9 -0
+		if (!pkg[i].regions)
+			return false;
+		slot = 0;
+		if (use_p1)
+			slot = setup(pkg, i, p1, slot);
+		if (use_p2)
+			slot = setup(pkg, i, p2, slot);
+	}
+
 	if (use_p1)
 		feat_energy = no_free_ptr(p1);
 	if (use_p2)

Reinette

author maintainer
#45 lore raw

PATCH v3 17/26 x86/resctrl: Build a lookup table for each resctrl event id

In reply to Tony Luck

User requests to read events arrive from the file system layer with a domain id, RMID, and resctrl event id. Responding to those requests needs information from various structures.

Build a quick lookup table indexed by resctrl event id.

---
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_aet.c +18 -0
index fb03f2e76306..44d2fe747ed8 100644
@@ -67,6 +67,12 @@ struct telem_entry {
 	struct pmt_event evts[];
 };
 
+/* Lookup table to get from resctrl event id to useful structures */
+static struct evtinfo {
+	struct telem_entry	*telem_entry;
+	struct pmt_event	*pmt_event;
+} evtinfo[QOS_NUM_EVENTS];
+
 /* All known telemetry event groups */
 static struct telem_entry *telem_entry[] = {
 	NULL
@@ -151,6 +157,7 @@ bool intel_aet_get_events(void)
 	struct pmt_feature_group *p2 __free(intel_pmt_put_feature_group) = NULL;
 	struct pkg_info *pkg __free(free_pkg_info) = NULL;
 	int num_pkgs = topology_max_packages();
+	struct telem_entry **tentry;
 	bool use_p1, use_p2;
 	int slot;
 
@@ -184,6 +191,17 @@ bool intel_aet_get_events(void)
 			slot = setup(pkg, i, p2, slot);
 	}
 
+	for (tentry = telem_entry; *tentry; tentry++) {
+		if (!(*tentry)->active)
+			continue;
+		for (int i = 0; i < (*tentry)->num_events; i++) {
+			enum resctrl_event_id evtid = (*tentry)->evts[i].evtid;
+
+			evtinfo[evtid].telem_entry = *tentry;
+			evtinfo[evtid].pmt_event = &(*tentry)->evts[i];
+		}
+	}
+
 	if (use_p1)
 		feat_energy = no_free_ptr(p1);
 	if (use_p2)
Signature · 1 line
2.48.1
maintainer
#46 lore raw

Re: [PATCH v3 17/26] x86/resctrl: Build a lookup table for each resctrl event id

Hi Tony,

Quoted 5 lines · Tony Luck

User requests to read events arrive from the file system layer with a domain id, RMID, and resctrl event id. Responding to those requests needs information from various structures.

Build a quick lookup table indexed by resctrl event id.

Why is a second lookup table needed? What about the lookup table ("all_events") that patch #2 introduced?

Perhaps struct mon_evt could get a "void *priv" or related member that points to event's private data that varies by type of event?

Reinette

author maintainer
#47 lore raw

PATCH v3 18/26 x86/resctrl: Add code to read core telemetry events

In reply to Tony Luck

The new telemetry events will be part of a new resctrl resource. Add the RDT_RESOURCE_PERF_PKG to enum resctrl_res_level.

Add hook resctrl_arch_rmid_read() to pass reads on this resource to the telemetry code.

There may be multiple devices tracking each package, so scan all of them and add up counters.

---
 include/linux/resctrl_types.h           |  1 +
 arch/x86/kernel/cpu/resctrl/internal.h  |  5 +++
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 58 +++++++++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/monitor.c   |  6 +++
 4 files changed, 70 insertions(+)
include/linux/resctrl_types.h +1 -0
index fbd4b55c41aa..3354f21e82ad 100644
@@ -39,6 +39,7 @@ enum resctrl_res_level {
 	RDT_RESOURCE_L2,
 	RDT_RESOURCE_MBA,
 	RDT_RESOURCE_SMBA,
+	RDT_RESOURCE_PERF_PKG,
 
 	/* Must be the last */
 	RDT_NUM_RESOURCES,
arch/x86/kernel/cpu/resctrl/internal.h +5 -0
index 70b63bbc429d..1b1cbb948a9a 100644
@@ -175,9 +175,14 @@ void rdt_domain_reconfigure_cdp(struct rdt_resource *r);
 #ifdef CONFIG_INTEL_AET_RESCTRL
 bool intel_aet_get_events(void);
 void __exit intel_aet_exit(void);
+int intel_aet_read_event(int domid, int rmid, int evtid, u64 *val);
 #else
 static inline bool intel_aet_get_events(void) { return false; }
 static inline void intel_aet_exit(void) { };
+static inline int intel_aet_read_event(int domid, int rmid, int evtid, u64 *val)
+{
+	return -EINVAL;
+}
 #endif
 
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/intel_aet.c +58 -0

67 lines not on this page · raw message

arch/x86/kernel/cpu/resctrl/monitor.c +6 -0
index 06623d51d006..4fa297d463ba 100644
@@ -236,6 +236,12 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
 	u32 prmid;
 	int ret;
 
+	if (r->rid == RDT_RESOURCE_PERF_PKG) {
+		ret = intel_aet_read_event(d->hdr.id, rmid, eventid, val);
+
+		return ret ? ret : 0;
+	}
+
 	resctrl_arch_rmid_read_context_check();
 
 	prmid = logical_rmid_to_physical_rmid(cpu, rmid);
Signature · 1 line
2.48.1
maintainer
#48 lore raw

Re: [PATCH v3 18/26] x86/resctrl: Add code to read core telemetry events

Hi Tony,

(deja vu ... "Add code to" can be dropped)

Quoted 2 lines · Tony Luck

The new telemetry events will be part of a new resctrl resource. Add the RDT_RESOURCE_PERF_PKG to enum resctrl_res_level.

Please follow tip changelog structure custom throughout this series.

Quoted 32 lines

Add hook resctrl_arch_rmid_read() to pass reads on this resource to the telemetry code.

There may be multiple devices tracking each package, so scan all of them and add up counters.

---
 include/linux/resctrl_types.h           |  1 +
 arch/x86/kernel/cpu/resctrl/internal.h  |  5 +++
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 58 +++++++++++++++++++++++++
 arch/x86/kernel/cpu/resctrl/monitor.c   |  6 +++
 4 files changed, 70 insertions(+)
include/linux/resctrl_types.h +1 -0
index fbd4b55c41aa..3354f21e82ad 100644
@@ -39,6 +39,7 @@ enum resctrl_res_level {
 	RDT_RESOURCE_L2,
 	RDT_RESOURCE_MBA,
 	RDT_RESOURCE_SMBA,
+	RDT_RESOURCE_PERF_PKG,
 
 	/* Must be the last */
 	RDT_NUM_RESOURCES,
arch/x86/kernel/cpu/resctrl/internal.h +0 -0
index 70b63bbc429d..1b1cbb948a9a 100644

arch/x86/kernel/cpu/resctrl/internal.h void rdt_domain_reconfigure_cdp(struct rdt_resource *r); · line 178

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/internal.h +1 -0
@@ -175,9 +175,14 @@ void rdt_domain_reconfigure_cdp(struct rdt_resource *r);
 #ifdef CONFIG_INTEL_AET_RESCTRL
 bool intel_aet_get_events(void);
 void __exit intel_aet_exit(void);
+int intel_aet_read_event(int domid, int rmid, int evtid, u64 *val);

This can use enum resctrl_event_id for evtid?

Quoted 15 lines
arch/x86/kernel/cpu/resctrl/internal.h +4 -0
 #else
 static inline bool intel_aet_get_events(void) { return false; }
 static inline void intel_aet_exit(void) { };
+static inline int intel_aet_read_event(int domid, int rmid, int evtid, u64 *val)
+{
+	return -EINVAL;
+}
 #endif
 
 #endif /* _ASM_X86_RESCTRL_INTERNAL_H */
arch/x86/kernel/cpu/resctrl/intel_aet.c +0 -0
index 44d2fe747ed8..67a1245858dc 100644

arch/x86/kernel/cpu/resctrl/intel_aet.c static struct evtinfo { · line 80

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +5 -0
@@ -73,6 +73,12 @@ static struct evtinfo {
 	struct pmt_event	*pmt_event;
 } evtinfo[QOS_NUM_EVENTS];
 
+#define EVT_NUM_RMIDS(evtid)	(evtinfo[evtid].telem_entry->num_rmids)
+#define EVT_NUM_EVENTS(evtid)	(evtinfo[evtid].telem_entry->num_events)
+#define EVT_GUID(evtid)		(evtinfo[evtid].telem_entry->guid)
+
+#define EVT_OFFSET(evtid)	(evtinfo[evtid].pmt_event->evt_offset)

Please open code these or use functions if you need to.

Quoted 14 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +7 -0
+
 /* All known telemetry event groups */
 static struct telem_entry *telem_entry[] = {
 	NULL
@@ -224,3 +230,55 @@ void __exit intel_aet_exit(void)
 	}
 	kfree(pkg_info);
 }
+
+#define VALID_BIT	BIT_ULL(63)
+#define DATA_BITS	GENMASK_ULL(62, 0)
+
+/*
+ * Walk the array of telemetry groups on a specific package.

arch/x86/kernel/cpu/resctrl/intel_aet.c void __exit intel_aet_exit(void) · line 248

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +10 -0
+ * Read and sum values for a specific counter (described by
+ * guid and offset).
+ * Return failure (~0x0ull) if any counter isn't valid.
+ */
+static u64 scan_pmt_devs(int package, int guid, int offset)
+{
+	u64 rval, val;
+	int ndev = 0;
+
+	rval = 0;

This can be done as part of definition.

Quoted 41 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +36 -0
+
+	for (int i = 0; i < pkg_info[package].count; i++) {
+		if (pkg_info[package].regions[i].guid != guid)
+			continue;
+		ndev++;
+		val = readq(pkg_info[package].regions[i].addr + offset);
+
+		if (!(val & VALID_BIT))
+			return ~0ull;
+		rval += val & DATA_BITS;
+	}
+
+	return ndev ? rval : ~0ull;
+}
+
+/*
+ * Read counter for an event on a domain (summing all aggregators
+ * on the domain).
+ */
+int intel_aet_read_event(int domid, int rmid, int evtid, u64 *val)
+{
+	u64 evtcount;
+	int offset;
+
+	if (rmid >= EVT_NUM_RMIDS(evtid))
+		return -ENOENT;
+
+	offset = rmid * EVT_NUM_EVENTS(evtid) * sizeof(u64);
+	offset += EVT_OFFSET(evtid);
+	evtcount = scan_pmt_devs(domid, EVT_GUID(evtid), offset);
+
+	if (evtcount != ~0ull || *val == 0)
+		*val += evtcount;
+
+	return evtcount != ~0ull ? 0 : -EINVAL;
+}
arch/x86/kernel/cpu/resctrl/monitor.c +0 -0
index 06623d51d006..4fa297d463ba 100644

arch/x86/kernel/cpu/resctrl/monitor.c int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d, · line 243

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/monitor.c +5 -0
@@ -236,6 +236,12 @@ int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d,
 	u32 prmid;
 	int ret;
 
+	if (r->rid == RDT_RESOURCE_PERF_PKG) {
+		ret = intel_aet_read_event(d->hdr.id, rmid, eventid, val);
+
+		return ret ? ret : 0;
+	}

Not sure if I am missing something at this stage but it looks like, since resctrl_arch_rmid_read() can now return ENOENT, and rmid_read::err obtain value of ENOENT, that there may be an
issue when this error is returned since rdtgroup_mondata_show()'s "checkresult" does not have handling for ENOENT and will attempt to print data to user space.

arch/x86/kernel/cpu/resctrl/monitor.c int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d, · line 245

Quoted 2 lines
arch/x86/kernel/cpu/resctrl/monitor.c +1 -0
+
 	resctrl_arch_rmid_read_context_check();

Please keep this context check at top of function.

arch/x86/kernel/cpu/resctrl/monitor.c int resctrl_arch_rmid_read(struct rdt_resource *r, struct rdt_mon_domain *d, · line 247

Quoted 2 lines
arch/x86/kernel/cpu/resctrl/monitor.c +0 -0
 	prmid = logical_rmid_to_physical_rmid(cpu, rmid);

Reinette

author maintainer
#49 lore raw

PATCH v3 19/26 x86/resctrl: Sanity check telemetry RMID values

In reply to Tony Luck

There are three values of interest:
1) The number of RMIDs supported by the CPU core. This is enumerated by CPUID leaf 0xF. Linux saves the value in boot_cpu_data.x86_cache_max_rmid.
2) The number of counter registers in each telemetry region. This is described in the XML file for the region. Linux hard codes it into the struct telem_entry..num_rmids field.
3) The maximum number of RMIDs that can be tracked simultaneously for a telemetry region. This is provided in the structures received from the intel_pmt_get_regions_by_feature() calls.

Print appropriate warnings if these values do not match.

TODO: Need a better UI. The number of implemented counters can be different per telemetry region.

---
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 31 +++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_aet.c +31 -0
index 67a1245858dc..0bcbac326bee 100644
@@ -13,6 +13,7 @@
 
 #include <linux/cpu.h>
 #include <linux/cleanup.h>
+#include <linux/minmax.h>
 #include "fake_intel_aet_features.h"
 #include <linux/intel_vsec.h>
 #include <linux/resctrl.h>
@@ -51,6 +52,7 @@ struct pmt_event {
  * @last_overflow_tstamp_off:	Offset of overflow timestamp
  * @last_update_tstamp_off:	Offset of last update timestamp
  * @active:			Marks this group as active on this system
+ * @rmid_warned:		Set to stop multiple rmid sanity warnings
  * @num_events:			Size of @evts array
  * @evts:			Telemetry events in this group
  */
@@ -63,6 +65,7 @@ struct telem_entry {
 	int	last_overflow_tstamp_off;
 	int	last_update_tstamp_off;
 	bool	active;
+	bool	rmid_warned;
 	int	num_events;
 	struct pmt_event evts[];
 };
@@ -84,6 +87,33 @@ static struct telem_entry *telem_entry[] = {
 	NULL
 };
 
+static void rmid_sanity_check(struct telemetry_region *tr, struct telem_entry *tentry)
+{
+	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG].r_resctrl;
+	int system_rmids = boot_cpu_data.x86_cache_max_rmid + 1;
+
+	if (tentry->rmid_warned)
+		return;
+
+	if (tentry->num_rmids != system_rmids) {
+		pr_info("Telemetry region %s has %d RMIDs system supports %d\n",
+			tentry->name, tentry->num_rmids, system_rmids);
+		tentry->rmid_warned = true;
+	}
+
+	if (tr->num_rmids < tentry->num_rmids) {
+		pr_info("Telemetry region %s only supports %d simultaneous RMIDS\n",
+			tentry->name, tr->num_rmids);
+		tentry->rmid_warned = true;
+	}
+
+	/* info/PKG_PERF_MON/num_rmids reports number of guaranteed counters */
+	if (!r->num_rmid)
+		r->num_rmid = tr->num_rmids;
+	else
+		r->num_rmid = min((u32)r->num_rmid, tr->num_rmids);
+}
+
 /*
  * Scan a feature group looking for guids recognized
  * and update the per-package counts of known groups.
@@ -109,6 +139,7 @@ static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_
 					pr_warn_once("MMIO region for guid 0x%x too small\n", tr->guid);
 					continue;
 				}
+				rmid_sanity_check(tr, *tentry);
 				found = true;
 				(*tentry)->active = true;
 				pkg[tr->plat_info.package_id].count++;
Signature · 1 line
2.48.1
maintainer
#50 lore raw

Re: [PATCH v3 19/26] x86/resctrl: Sanity check telemetry RMID values

Hi Tony,

Quoted 6 lines · Tony Luck

There are three values of interest:
1) The number of RMIDs supported by the CPU core. This is enumerated by CPUID leaf 0xF. Linux saves the value in boot_cpu_data.x86_cache_max_rmid.
2) The number of counter registers in each telemetry region. This is described in the XML file for the region. Linux hard codes it into the struct telem_entry..num_rmids field.

Syntax telem_entry::num_rmids can be used for a member.

Quoted 3 lines

3) The maximum number of RMIDs that can be tracked simultaneously for a telemetry region. This is provided in the structures received from the intel_pmt_get_regions_by_feature() calls.

Is (2) and (3) not required to be the same? If not, how does resctrl know which counter/RMID is being tracked?

Quoted 2 lines

Print appropriate warnings if these values do not match.

As mentioned in cover letter I do not think that just printing a warning is sufficient. It really becomes a trial-and-error guessing game for user space to know which monitor group supports telemetry events.

Quoted 14 lines

TODO: Need a better UI. The number of implemented counters can be different per telemetry region.

---
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 31 +++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_aet.c +0 -0
index 67a1245858dc..0bcbac326bee 100644

arch/x86/kernel/cpu/resctrl/intel_aet.c line 16

Quoted 4 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +1 -0
@@ -13,6 +13,7 @@
 
 #include <linux/cpu.h>
 #include <linux/cleanup.h>
+#include <linux/minmax.h>

Please sort includes alphabetically.

arch/x86/kernel/cpu/resctrl/intel_aet.c struct pmt_event { · line 55

Quoted 8 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +1 -0
 #include "fake_intel_aet_features.h"
 #include <linux/intel_vsec.h>
 #include <linux/resctrl.h>
@@ -51,6 +52,7 @@ struct pmt_event {
  * @last_overflow_tstamp_off:	Offset of overflow timestamp
  * @last_update_tstamp_off:	Offset of last update timestamp
  * @active:			Marks this group as active on this system
+ * @rmid_warned:		Set to stop multiple rmid sanity warnings

rmid -> RMID.

I find the description unclear on how to interact with this member. How about something like:
True if user space have been warned about number of RMIDs used by different resources not matching.

Quoted 12 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +1 -0
  * @num_events:			Size of @evts array
  * @evts:			Telemetry events in this group
  */
@@ -63,6 +65,7 @@ struct telem_entry {
 	int	last_overflow_tstamp_off;
 	int	last_update_tstamp_off;
 	bool	active;
+	bool	rmid_warned;
 	int	num_events;
 	struct pmt_event evts[];
 };

arch/x86/kernel/cpu/resctrl/intel_aet.c static struct telem_entry *telem_entry[] = { · line 93

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +4 -0
@@ -84,6 +87,33 @@ static struct telem_entry *telem_entry[] = {
 	NULL
 };
 
+static void rmid_sanity_check(struct telemetry_region *tr, struct telem_entry *tentry)
+{
+	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG].r_resctrl;
+	int system_rmids = boot_cpu_data.x86_cache_max_rmid + 1;

It is not clear what "system_rmids" should represent here. Is it, as changelog states, maximum supported by CPU core, or is it maximum supported by L3 resource, which is the maximum number of monitor groups that can be created.

We see in rdt_get_mon_l3_config() that:

	r->num_rmid = (boot_cpu_data.x86_cache_max_rmid + 1) / snc_nodes_per_l3_cache;

This makes me wonder how this feature behaves on SNC systems?

arch/x86/kernel/cpu/resctrl/intel_aet.c static struct telem_entry *telem_entry[] = { · line 99

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6 -0
+
+	if (tentry->rmid_warned)
+		return;
+
+	if (tentry->num_rmids != system_rmids) {
+		pr_info("Telemetry region %s has %d RMIDs system supports %d\n",

Is pr_info() intended to be pr_warn()?
The message self could do with a comma?

arch/x86/kernel/cpu/resctrl/intel_aet.c static struct telem_entry *telem_entry[] = { · line 102

Quoted 3 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +3 -0
+			tentry->name, tentry->num_rmids, system_rmids);
+		tentry->rmid_warned = true;
+	}

Could you please add comments about consequences of when this is encountered?

arch/x86/kernel/cpu/resctrl/intel_aet.c static struct telem_entry *telem_entry[] = { · line 108

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +6 -0
+
+	if (tr->num_rmids < tentry->num_rmids) {
+		pr_info("Telemetry region %s only supports %d simultaneous RMIDS\n",
+			tentry->name, tr->num_rmids);
+		tentry->rmid_warned = true;
+	}

I am still trying to get used to all the data structures. From what I can tell, the offset of counter is obtained from struct telem_entry. If struct telem_entry thus thinks there are more RMIDs than what the region supports, would this not cause memory reads to exceed what region supports?

Could you please add comments about consequences of when this is encountered?

arch/x86/kernel/cpu/resctrl/intel_aet.c static struct telem_entry *telem_entry[] = { · line 115

Quoted 7 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +7 -0
+
+	/* info/PKG_PERF_MON/num_rmids reports number of guaranteed counters */
+	if (!r->num_rmid)
+		r->num_rmid = tr->num_rmids;
+	else
+		r->num_rmid = min((u32)r->num_rmid, tr->num_rmids);
+}

As I mentioned in response to previous version it may be possible to move resctrl_mon_resource_init() to rdt_get_tree() to be done after these RMID counts are discovered. When doing so it is possible to size the available RMIDs used on system to be supported by all resources.

arch/x86/kernel/cpu/resctrl/intel_aet.c static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_ · line 145

Quoted 12 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +2 -0
+
 /*
  * Scan a feature group looking for guids recognized
  * and update the per-package counts of known groups.
@@ -109,6 +139,7 @@ static bool count_events(struct pkg_info *pkg, int max_pkgs, struct pmt_feature_
 					pr_warn_once("MMIO region for guid 0x%x too small\n", tr->guid);
 					continue;
 				}
+				rmid_sanity_check(tr, *tentry);
 				found = true;
 				(*tentry)->active = true;
 				pkg[tr->plat_info.package_id].count++;

Reinette

author maintainer
#51 lore raw

PATCH v3 20/26 x86/resctrl: Add and initialize rdt_resource for package scope core monitor

In reply to Tony Luck

Counts for each Intel telemetry event are periodically sent to one or more aggregators on each package where accumulated totals are made available in MMIO registers.

Add a new resource for monitoring these events with code to build domains at the package granularity.

---
 include/linux/resctrl.h            |  1 +
 arch/x86/kernel/cpu/resctrl/core.c | 11 +++++++++++
 2 files changed, 12 insertions(+)
include/linux/resctrl.h +1 -0
index 25f51a57b0b7..c03e7dc1f009 100644
@@ -232,6 +232,7 @@ enum resctrl_scope {
 	RESCTRL_L2_CACHE = 2,
 	RESCTRL_L3_CACHE = 3,
 	RESCTRL_L3_NODE,
+	RESCTRL_PACKAGE,
 };
 
 /**
arch/x86/kernel/cpu/resctrl/core.c +11 -0
index f0f256a5ac66..9578d9c7260c 100644
@@ -99,6 +99,15 @@ struct rdt_hw_resource rdt_resources_all[RDT_NUM_RESOURCES] = {
 			.schema_fmt		= RESCTRL_SCHEMA_RANGE,
 		},
 	},
+	[RDT_RESOURCE_PERF_PKG] =
+	{
+		.r_resctrl = {
+			.rid			= RDT_RESOURCE_PERF_PKG,
+			.name			= "PERF_PKG",
+			.mon_scope		= RESCTRL_PACKAGE,
+			.mon_domains		= mon_domain_init(RDT_RESOURCE_PERF_PKG),
+		},
+	},
 };
 
 u32 resctrl_arch_system_num_rmid_idx(void)
@@ -431,6 +440,8 @@ static int get_domain_id_from_scope(int cpu, enum resctrl_scope scope)
 		return get_cpu_cacheinfo_id(cpu, scope);
 	case RESCTRL_L3_NODE:
 		return cpu_to_node(cpu);
+	case RESCTRL_PACKAGE:
+		return topology_physical_package_id(cpu);
 	default:
 		break;
 	}
Signature · 1 line
2.48.1
author maintainer
#52 lore raw

PATCH v3 21/26 fs-x86/resctrl: Handle RDT_RESOURCE_PERF_PKG in domain create/delete

In reply to Tony Luck

Add a new rdt_perf_pkg_mon_domain structure. This only consists of the common rdt_domain_hdr as there is no need for any per-domain data structures.

Use as much as possible of the existing domain setup and tear down infrastructure. In many cases the RDT_RESOURCE_PERF_PKG uses the same functions but just skips over the pieces it does not need.

---
 include/linux/resctrl.h            |  8 ++++++++
 arch/x86/kernel/cpu/resctrl/core.c | 32 ++++++++++++++++++++++++++++++
 fs/resctrl/rdtgroup.c              | 11 ++++++++--
 3 files changed, 49 insertions(+), 2 deletions(-)
include/linux/resctrl.h +8 -0
index c03e7dc1f009..6f598a64b192 100644
@@ -169,6 +169,14 @@ struct rdt_mon_domain {
 	int				cqm_work_cpu;
 };
 
+/**
+ * struct rdt_perf_pkg_mon_domain - CPUs sharing an Intel-PMT-scoped resctrl monitor resource
+ * @hdr:               common header for different domain types
+ */
+struct rdt_perf_pkg_mon_domain {
+	struct rdt_domain_hdr   hdr;
+};
+
 /**
  * struct resctrl_cache - Cache allocation related data
  * @cbm_len:		Length of the cache bit mask
arch/x86/kernel/cpu/resctrl/core.c +32 -0
index 9578d9c7260c..6f5d52a8219b 100644
@@ -542,6 +542,29 @@ static void setup_l3_mon_domain(int cpu, int id, struct rdt_resource *r, struct
 	}
 }
 
+static void setup_intel_aet_mon_domain(int cpu, int id, struct rdt_resource *r,
+				       struct list_head *add_pos)
+{
+	struct rdt_perf_pkg_mon_domain *d;
+	int err;
+
+	d = kzalloc_node(sizeof(*d), GFP_KERNEL, cpu_to_node(cpu));
+	if (!d)
+		return;
+
+	d->hdr.id = id;
+	d->hdr.type = DOMTYPE(r->rid, DOMTYPE_MON);
+	cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
+	list_add_tail_rcu(&d->hdr.list, add_pos);
+
+	err = resctrl_online_mon_domain(r, &d->hdr);
+	if (err) {
+		list_del_rcu(&d->hdr.list);
+		synchronize_rcu();
+		kfree(d);
+	}
+}
+
 static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 {
 	int id = get_domain_id_from_scope(cpu, r->mon_scope);
@@ -571,6 +594,9 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 	case RDT_RESOURCE_L3:
 		setup_l3_mon_domain(cpu, id, r, add_pos);
 		break;
+	case RDT_RESOURCE_PERF_PKG:
+		setup_intel_aet_mon_domain(cpu, id, r, add_pos);
+		break;
 	default:
 		WARN_ON_ONCE(1);
 	}
@@ -668,6 +694,12 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
 		synchronize_rcu();
 		mon_domain_free(hw_dom);
 		break;
+	case RDT_RESOURCE_PERF_PKG:
+		resctrl_offline_mon_domain(r, d);
+		list_del_rcu(&hdr->list);
+		synchronize_rcu();
+		kfree(container_of(hdr, struct rdt_perf_pkg_mon_domain, hdr));
+		break;
 	}
 }
 
fs/resctrl/rdtgroup.c +9 -2
index 5ca6de6a6e5c..34fcd20f8dd7 100644
@@ -4020,6 +4020,9 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d
 	if (resctrl_mounted && resctrl_arch_mon_capable())
 		rmdir_mondata_subdir_allrdtgrp(r, &d->hdr);
 
+	if (r->rid == RDT_RESOURCE_PERF_PKG)
+		goto done;
+
 	if (resctrl_is_mbm_enabled())
 		cancel_delayed_work(&d->mbm_over);
 	if (resctrl_arch_is_llc_occupancy_enabled() && has_busy_rmid(d)) {
@@ -4036,7 +4039,7 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d
 	}
 
 	domain_destroy_mon_state(d);
-
+done:
 	mutex_unlock(&rdtgroup_mutex);
 }
 
@@ -4104,12 +4107,15 @@ int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d
 int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr)
 {
 	struct rdt_mon_domain *d;
-	int err;
+	int err = 0;
 
 	WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_MON));
 	d = container_of(hdr, struct rdt_mon_domain, hdr);
 	mutex_lock(&rdtgroup_mutex);
 
+	if (r->rid == RDT_RESOURCE_PERF_PKG)
+		goto do_mkdir;
+
 	err = domain_setup_mon_state(r, d);
 	if (err)
 		goto out_unlock;
@@ -4123,6 +4129,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr
 	if (resctrl_arch_is_llc_occupancy_enabled())
 		INIT_DELAYED_WORK(&d->cqm_limbo, cqm_handle_limbo);
 
+do_mkdir:
 	/*
 	 * If the filesystem is not mounted then only the default resource group
 	 * exists. Creation of its directories is deferred until mount time
Signature · 1 line
2.48.1
maintainer
#53 lore raw

Re: [PATCH v3 21/26] fs-x86/resctrl: Handle RDT_RESOURCE_PERF_PKG in domain create/delete

Hi Tony,

Quoted 20 lines · Tony Luck

Add a new rdt_perf_pkg_mon_domain structure. This only consists of the common rdt_domain_hdr as there is no need for any per-domain data structures.

Use as much as possible of the existing domain setup and tear down infrastructure. In many cases the RDT_RESOURCE_PERF_PKG uses the same functions but just skips over the pieces it does not need.

---
 include/linux/resctrl.h            |  8 ++++++++
 arch/x86/kernel/cpu/resctrl/core.c | 32 ++++++++++++++++++++++++++++++
 fs/resctrl/rdtgroup.c              | 11 ++++++++--
 3 files changed, 49 insertions(+), 2 deletions(-)
include/linux/resctrl.h +0 -0
index c03e7dc1f009..6f598a64b192 100644

include/linux/resctrl.h struct rdt_mon_domain { · line 173

Quoted 5 lines
include/linux/resctrl.h +2 -0
@@ -169,6 +169,14 @@ struct rdt_mon_domain {
 	int				cqm_work_cpu;
 };
 
+/**
+ * struct rdt_perf_pkg_mon_domain - CPUs sharing an Intel-PMT-scoped resctrl monitor resource

It is a red flag when architecture specific things ("rdt" and "Intel-PMT-scoped") land in include/linux/resctrl.h

Quoted 54 lines
include/linux/resctrl.h +6 -0
+ * @hdr:               common header for different domain types
+ */
+struct rdt_perf_pkg_mon_domain {
+	struct rdt_domain_hdr   hdr;
+};
+
 /**
  * struct resctrl_cache - Cache allocation related data
  * @cbm_len:		Length of the cache bit mask
arch/x86/kernel/cpu/resctrl/core.c +26 -0
index 9578d9c7260c..6f5d52a8219b 100644
@@ -542,6 +542,29 @@ static void setup_l3_mon_domain(int cpu, int id, struct rdt_resource *r, struct
 	}
 }
 
+static void setup_intel_aet_mon_domain(int cpu, int id, struct rdt_resource *r,
+				       struct list_head *add_pos)
+{
+	struct rdt_perf_pkg_mon_domain *d;
+	int err;
+
+	d = kzalloc_node(sizeof(*d), GFP_KERNEL, cpu_to_node(cpu));
+	if (!d)
+		return;
+
+	d->hdr.id = id;
+	d->hdr.type = DOMTYPE(r->rid, DOMTYPE_MON);
+	cpumask_set_cpu(cpu, &d->hdr.cpu_mask);
+	list_add_tail_rcu(&d->hdr.list, add_pos);
+
+	err = resctrl_online_mon_domain(r, &d->hdr);
+	if (err) {
+		list_del_rcu(&d->hdr.list);
+		synchronize_rcu();
+		kfree(d);
+	}
+}
+
 static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 {
 	int id = get_domain_id_from_scope(cpu, r->mon_scope);
@@ -571,6 +594,9 @@ static void domain_add_cpu_mon(int cpu, struct rdt_resource *r)
 	case RDT_RESOURCE_L3:
 		setup_l3_mon_domain(cpu, id, r, add_pos);
 		break;
+	case RDT_RESOURCE_PERF_PKG:
+		setup_intel_aet_mon_domain(cpu, id, r, add_pos);
+		break;
 	default:
 		WARN_ON_ONCE(1);
 	}

arch/x86/kernel/cpu/resctrl/core.c static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r) · line 698

Quoted 5 lines
arch/x86/kernel/cpu/resctrl/core.c +2 -0
@@ -668,6 +694,12 @@ static void domain_remove_cpu_mon(int cpu, struct rdt_resource *r)
 		synchronize_rcu();
 		mon_domain_free(hw_dom);
 		break;
+	case RDT_RESOURCE_PERF_PKG:
+		resctrl_offline_mon_domain(r, d);

Something should have complained about using an uninitialized variable here (d).

Quoted 12 lines
arch/x86/kernel/cpu/resctrl/core.c +4 -0
+		list_del_rcu(&hdr->list);
+		synchronize_rcu();
+		kfree(container_of(hdr, struct rdt_perf_pkg_mon_domain, hdr));
+		break;
 	}
 }
 
fs/resctrl/rdtgroup.c +0 -0
index 5ca6de6a6e5c..34fcd20f8dd7 100644

fs/resctrl/rdtgroup.c void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d · line 4024

Quoted 5 lines
fs/resctrl/rdtgroup.c +2 -0
@@ -4020,6 +4020,9 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d
 	if (resctrl_mounted && resctrl_arch_mon_capable())
 		rmdir_mondata_subdir_allrdtgrp(r, &d->hdr);
 
+	if (r->rid == RDT_RESOURCE_PERF_PKG)
+		goto done;

Could you please change this test to

	if (r->rid != RDT_RESOURCE_L3)

this makes it clear about what the code that follows supports as opposed to one resource that it does not support (and possibly cause issue in future when adding another monitoring resource).

Quoted 16 lines
fs/resctrl/rdtgroup.c +2 -1
+
 	if (resctrl_is_mbm_enabled())
 		cancel_delayed_work(&d->mbm_over);
 	if (resctrl_arch_is_llc_occupancy_enabled() && has_busy_rmid(d)) {
@@ -4036,7 +4039,7 @@ void resctrl_offline_mon_domain(struct rdt_resource *r, struct rdt_mon_domain *d
 	}
 
 	domain_destroy_mon_state(d);
-
+done:
 	mutex_unlock(&rdtgroup_mutex);
 }
 
@@ -4104,12 +4107,15 @@ int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d
 int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr)
 {

fs/resctrl/rdtgroup.c int resctrl_online_ctrl_domain(struct rdt_resource *r, struct rdt_ctrl_domain *d · line 4117

Quoted 10 lines
fs/resctrl/rdtgroup.c +3 -1
 	struct rdt_mon_domain *d;
-	int err;
+	int err = 0;
 
 	WARN_ON_ONCE(hdr->type != DOMTYPE(r->rid, DOMTYPE_MON));
 	d = container_of(hdr, struct rdt_mon_domain, hdr);
 	mutex_lock(&rdtgroup_mutex);
 
+	if (r->rid == RDT_RESOURCE_PERF_PKG)
+		goto do_mkdir;

same

fs/resctrl/rdtgroup.c int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr · line 4135

Quoted 12 lines
fs/resctrl/rdtgroup.c +2 -0
+
 	err = domain_setup_mon_state(r, d);
 	if (err)
 		goto out_unlock;
@@ -4123,6 +4129,7 @@ int resctrl_online_mon_domain(struct rdt_resource *r, struct rdt_domain_hdr *hdr
 	if (resctrl_arch_is_llc_occupancy_enabled())
 		INIT_DELAYED_WORK(&d->cqm_limbo, cqm_handle_limbo);
 
+do_mkdir:
 	/*
 	 * If the filesystem is not mounted then only the default resource group
 	 * exists. Creation of its directories is deferred until mount time

Reinette

author maintainer
#54 lore raw

PATCH v3 22/26 fs/resctrl: Add type define for PERF_PKG files

In reply to Tony Luck

Creation of the default info file for monitor resources requires an RFTYPE_RES_ define and mapping from the resource id.

Add the define and case in fflags_from_resource().

---
 fs/resctrl/internal.h | 2 ++
 fs/resctrl/rdtgroup.c | 2 ++
 2 files changed, 4 insertions(+)
fs/resctrl/internal.h +2 -0
index 4a840e683e96..b7bc820da726 100644
@@ -253,6 +253,8 @@ struct rdtgroup {
 
 #define RFTYPE_DEBUG			BIT(10)
 
+#define RFTYPE_RES_PERF_PKG		BIT(11)
+
 #define RFTYPE_CTRL_INFO		(RFTYPE_INFO | RFTYPE_CTRL)
 
 #define RFTYPE_MON_INFO			(RFTYPE_INFO | RFTYPE_MON)
fs/resctrl/rdtgroup.c +2 -0
index 34fcd20f8dd7..cae68e8b9f86 100644
@@ -2175,6 +2175,8 @@ static unsigned long fflags_from_resource(struct rdt_resource *r)
 	case RDT_RESOURCE_MBA:
 	case RDT_RESOURCE_SMBA:
 		return RFTYPE_RES_MB;
+	case RDT_RESOURCE_PERF_PKG:
+		return RFTYPE_RES_PERF_PKG;
 	}
 
 	return WARN_ON_ONCE(1);
Signature · 1 line
2.48.1
author maintainer
#55 lore raw

PATCH v3 23/26 fs/resctrl: Add new telemetry event id and structures

In reply to Tony Luck

Nine new events for energy and perf monitoring per-RMID.

---
 include/linux/resctrl_types.h | 19 ++++++++++++---
 fs/resctrl/monitor.c          | 45 +++++++++++++++++++++++++++++++++++
 2 files changed, 61 insertions(+), 3 deletions(-)
include/linux/resctrl_types.h +16 -3
index 3354f21e82ad..2c959e7233dd 100644
@@ -46,16 +46,29 @@ enum resctrl_res_level {
 };
 
 /*
- * Event IDs, the values match those used to program IA32_QM_EVTSEL before
- * reading IA32_QM_CTR on RDT systems.
+ * Event IDs
  */
 enum resctrl_event_id {
+	/* Legacy events. Values must match X86 IA32_QM_EVTSEL usage */
 	QOS_L3_OCCUP_EVENT_ID		= 0x01,
 	QOS_L3_MBM_TOTAL_EVENT_ID	= 0x02,
 	QOS_L3_MBM_LOCAL_EVENT_ID	= 0x03,
+
+	/* Intel Telemetry Events */
+	PMT_EVENT_ENERGY,
+	PMT_EVENT_ACTIVITY,
+	PMT_EVENT_STALLS_LLC_HIT,
+	PMT_EVENT_C1_RES,
+	PMT_EVENT_UNHALTED_CORE_CYCLES,
+	PMT_EVENT_STALLS_LLC_MISS,
+	PMT_EVENT_AUTO_C6_RES,
+	PMT_EVENT_UNHALTED_REF_CYCLES,
+	PMT_EVENT_UOPS_RETIRED,
+
+	/* Must be the last */
+	QOS_NUM_EVENTS
 };
 
-#define QOS_NUM_EVENTS		(QOS_L3_MBM_LOCAL_EVENT_ID + 1)
 #define QOS_NUM_MBM_EVENTS	(QOS_L3_MBM_LOCAL_EVENT_ID - QOS_L3_MBM_TOTAL_EVENT_ID + 1)
 #define MBM_EVENT_IDX(evt)	((evt) - QOS_L3_MBM_TOTAL_EVENT_ID)
 
fs/resctrl/monitor.c +45 -0

51 lines not on this page · raw message

Signature · 1 line
2.48.1
author maintainer
#56 lore raw

PATCH v3 24/26 x86/resctrl: Final steps to enable RDT_RESOURCE_PERF_PKG

In reply to Tony Luck

For each of the discovered telemetry events:
Mark as enabled in the rdt_mon_features bitmap.
Set the value display type.
Mark that the event can be read from any CPU.

Because the resource was not marked as enabled during early initialization no domain discovery and allocation was done. Do that in the architecture first mount hook.

---
 arch/x86/kernel/cpu/resctrl/core.c      | 14 ++++++++++++++
 arch/x86/kernel/cpu/resctrl/intel_aet.c |  5 +++++
 2 files changed, 19 insertions(+)
arch/x86/kernel/cpu/resctrl/core.c +14 -0
index 6f5d52a8219b..83da63b24f45 100644
@@ -756,7 +756,9 @@ static int resctrl_arch_offline_cpu(unsigned int cpu)
 
 void resctrl_arch_mount(void)
 {
+	struct rdt_resource *r = &rdt_resources_all[RDT_RESOURCE_PERF_PKG].r_resctrl;
 	static bool only_once;
+	int cpu;
 
 	if (only_once)
 		return;
@@ -764,6 +766,18 @@ void resctrl_arch_mount(void)
 
 	if (!intel_aet_get_events())
 		return;
+
+	/*
+	 * Late discovery of telemetry events means the domains for the
+	 * resource were not built. Do that now.
+	 */
+	cpus_read_lock();
+	mutex_lock(&domain_list_lock);
+	r->mon_capable = true;
+	for_each_online_cpu(cpu)
+		domain_add_cpu_mon(cpu, r);
+	mutex_unlock(&domain_list_lock);
+	cpus_read_unlock();
 }
 
 enum {
arch/x86/kernel/cpu/resctrl/intel_aet.c +5 -0
index 0bcbac326bee..529f6d49e3a3 100644
@@ -233,9 +233,14 @@ bool intel_aet_get_events(void)
 			continue;
 		for (int i = 0; i < (*tentry)->num_events; i++) {
 			enum resctrl_event_id evtid = (*tentry)->evts[i].evtid;
+			enum resctrl_event_type type;
 
 			evtinfo[evtid].telem_entry = *tentry;
 			evtinfo[evtid].pmt_event = &(*tentry)->evts[i];
+
+			__set_bit(evtid, rdt_mon_features);
+			type = (*tentry)->evts[i].type;
+			resctrl_set_event_attributes(evtid, type, true);
 		}
 	}
 
Signature · 1 line
2.48.1
author maintainer
#57 lore raw

PATCH v3 25/26 fs-x86/resctrl: Add detailed descriptions for Clearwater Forest events

In reply to Tony Luck

There are two event groups one for energy reporting and another for "perf" events.

See the XML description files in https://github.com/intel/Intel-PMT
in the xml/CWF/OOBMSM/{RMID-ENERGY,RMID-PERF}/ for the detailed descriptions that were used to derive these descriptions.

---
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 57 +++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_aet.c +57 -0
index 529f6d49e3a3..e1097767009e 100644
@@ -42,6 +42,8 @@ struct pmt_event {
 	enum resctrl_event_type	type;
 };
 
+#define EVT(id, offset, _type) { .evtid = id, .evt_offset = offset, .type = _type }
+
 /**
  * struct telem_entry - Summarized form from XML telemetry description
  * @name:			Name for this group of events
@@ -82,8 +84,63 @@ static struct evtinfo {
 
 #define EVT_OFFSET(evtid)	(evtinfo[evtid].pmt_event->evt_offset)
 
+/*
+ * https://github.com/intel/Intel-PMT
+ * xml/CWF/OOBMSM/RMID-ENERGY *.xml
+ */
+#define NUM_RMIDS_0x26696143	576
+#define GUID_0x26696143		0x26696143
+#define NUM_EVENTS_0x26696143	2
+#define EVT_BYTES_0x26696143	(NUM_RMIDS_0x26696143 * NUM_EVENTS_0x26696143 * sizeof(u64))
+
+static struct telem_entry energy_0x26696143 = {
+	.name				= "energy",
+	.guid				= GUID_0x26696143,
+	.size				= EVT_BYTES_0x26696143 + sizeof(u64) * 3,
+	.num_rmids			= NUM_RMIDS_0x26696143,
+	.overflow_counter_off		= EVT_BYTES_0x26696143 + sizeof(u64) * 0,
+	.last_overflow_tstamp_off	= EVT_BYTES_0x26696143 + sizeof(u64) * 1,
+	.last_update_tstamp_off		= EVT_BYTES_0x26696143 + sizeof(u64) * 2,
+	.num_events			= NUM_EVENTS_0x26696143,
+	.evts				= {
+		EVT(PMT_EVENT_ENERGY, 0x0, EVT_TYPE_U46_18),
+		EVT(PMT_EVENT_ACTIVITY, 0x8, EVT_TYPE_U46_18),
+	}
+};
+
+/*
+ * https://github.com/intel/Intel-PMT
+ * xml/CWF/OOBMSM/RMID-PERF *.xml
+ */
+#define NUM_RMIDS_0x26557651	576
+#define GUID_0x26557651		0x26557651
+#define NUM_EVENTS_0x26557651	7
+#define EVT_BYTES_0x26557651	(NUM_RMIDS_0x26557651 * NUM_EVENTS_0x26557651 * sizeof(u64))
+
+static struct telem_entry perf_0x26557651 = {
+	.name				= "perf",
+	.guid				= GUID_0x26557651,
+	.size				= EVT_BYTES_0x26557651 + sizeof(u64) * 3,
+	.num_rmids			= NUM_RMIDS_0x26557651,
+	.overflow_counter_off		= EVT_BYTES_0x26557651 + sizeof(u64) * 0,
+	.last_overflow_tstamp_off	= EVT_BYTES_0x26557651 + sizeof(u64) * 1,
+	.last_update_tstamp_off		= EVT_BYTES_0x26557651 + sizeof(u64) * 2,
+	.num_events			= NUM_EVENTS_0x26557651,
+	.evts				= {
+		EVT(PMT_EVENT_STALLS_LLC_HIT, 0x0, EVT_TYPE_U64),
+		EVT(PMT_EVENT_C1_RES, 0x8, EVT_TYPE_U64),
+		EVT(PMT_EVENT_UNHALTED_CORE_CYCLES, 0x10, EVT_TYPE_U64),
+		EVT(PMT_EVENT_STALLS_LLC_MISS, 0x18, EVT_TYPE_U64),
+		EVT(PMT_EVENT_AUTO_C6_RES, 0x20, EVT_TYPE_U64),
+		EVT(PMT_EVENT_UNHALTED_REF_CYCLES, 0x28, EVT_TYPE_U64),
+		EVT(PMT_EVENT_UOPS_RETIRED, 0x30, EVT_TYPE_U64),
+	}
+};
+
 /* All known telemetry event groups */
 static struct telem_entry *telem_entry[] = {
+	&energy_0x26696143,
+	&perf_0x26557651,
 	NULL
 };
 
Signature · 1 line
2.48.1
maintainer
#58 lore raw

Re: [PATCH v3 25/26] fs-x86/resctrl: Add detailed descriptions for Clearwater Forest events

Hi Tony,

Quoted 2 lines · Tony Luck

There are two event groups one for energy reporting and another for "perf" events.

Please add context.

Quoted 6 lines

See the XML description files in https://github.com/intel/Intel-PMT
in the xml/CWF/OOBMSM/{RMID-ENERGY,RMID-PERF}/ for the detailed descriptions that were used to derive these descriptions.

The url in text can be a "Link:" here.

Quoted 18 lines
---
 arch/x86/kernel/cpu/resctrl/intel_aet.c | 57 +++++++++++++++++++++++++
 1 file changed, 57 insertions(+)
arch/x86/kernel/cpu/resctrl/intel_aet.c +2 -0
index 529f6d49e3a3..e1097767009e 100644
@@ -42,6 +42,8 @@ struct pmt_event {
 	enum resctrl_event_type	type;
 };
 
+#define EVT(id, offset, _type) { .evtid = id, .evt_offset = offset, .type = _type }
+
 /**
  * struct telem_entry - Summarized form from XML telemetry description
  * @name:			Name for this group of events

arch/x86/kernel/cpu/resctrl/intel_aet.c static struct evtinfo { · line 89

Quoted 6 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +3 -0
@@ -82,8 +84,63 @@ static struct evtinfo {
 
 #define EVT_OFFSET(evtid)	(evtinfo[evtid].pmt_event->evt_offset)
 
+/*
+ * https://github.com/intel/Intel-PMT
+ * xml/CWF/OOBMSM/RMID-ENERGY *.xml

This can be one line.

Quoted 14 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +14 -0
+ */
+#define NUM_RMIDS_0x26696143	576
+#define GUID_0x26696143		0x26696143
+#define NUM_EVENTS_0x26696143	2
+#define EVT_BYTES_0x26696143	(NUM_RMIDS_0x26696143 * NUM_EVENTS_0x26696143 * sizeof(u64))
+
+static struct telem_entry energy_0x26696143 = {
+	.name				= "energy",
+	.guid				= GUID_0x26696143,
+	.size				= EVT_BYTES_0x26696143 + sizeof(u64) * 3,
+	.num_rmids			= NUM_RMIDS_0x26696143,
+	.overflow_counter_off		= EVT_BYTES_0x26696143 + sizeof(u64) * 0,
+	.last_overflow_tstamp_off	= EVT_BYTES_0x26696143 + sizeof(u64) * 1,
+	.last_update_tstamp_off		= EVT_BYTES_0x26696143 + sizeof(u64) * 2,

arch/x86/kernel/cpu/resctrl/intel_aet.c static struct evtinfo { · line 113

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +10 -0
+	.num_events			= NUM_EVENTS_0x26696143,
+	.evts				= {
+		EVT(PMT_EVENT_ENERGY, 0x0, EVT_TYPE_U46_18),
+		EVT(PMT_EVENT_ACTIVITY, 0x8, EVT_TYPE_U46_18),
+	}
+};
+
+/*
+ * https://github.com/intel/Intel-PMT
+ * xml/CWF/OOBMSM/RMID-PERF *.xml

This can be one line.

Quoted 21 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +21 -0
+ */
+#define NUM_RMIDS_0x26557651	576
+#define GUID_0x26557651		0x26557651
+#define NUM_EVENTS_0x26557651	7
+#define EVT_BYTES_0x26557651	(NUM_RMIDS_0x26557651 * NUM_EVENTS_0x26557651 * sizeof(u64))
+
+static struct telem_entry perf_0x26557651 = {
+	.name				= "perf",
+	.guid				= GUID_0x26557651,
+	.size				= EVT_BYTES_0x26557651 + sizeof(u64) * 3,
+	.num_rmids			= NUM_RMIDS_0x26557651,
+	.overflow_counter_off		= EVT_BYTES_0x26557651 + sizeof(u64) * 0,
+	.last_overflow_tstamp_off	= EVT_BYTES_0x26557651 + sizeof(u64) * 1,
+	.last_update_tstamp_off		= EVT_BYTES_0x26557651 + sizeof(u64) * 2,
+	.num_events			= NUM_EVENTS_0x26557651,
+	.evts				= {
+		EVT(PMT_EVENT_STALLS_LLC_HIT, 0x0, EVT_TYPE_U64),
+		EVT(PMT_EVENT_C1_RES, 0x8, EVT_TYPE_U64),
+		EVT(PMT_EVENT_UNHALTED_CORE_CYCLES, 0x10, EVT_TYPE_U64),
+		EVT(PMT_EVENT_STALLS_LLC_MISS, 0x18, EVT_TYPE_U64),
+		EVT(PMT_EVENT_AUTO_C6_RES, 0x20, EVT_TYPE_U64),

arch/x86/kernel/cpu/resctrl/intel_aet.c static struct evtinfo { · line 144

Quoted 10 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +7 -0
+		EVT(PMT_EVENT_UNHALTED_REF_CYCLES, 0x28, EVT_TYPE_U64),
+		EVT(PMT_EVENT_UOPS_RETIRED, 0x30, EVT_TYPE_U64),
+	}
+};
+
 /* All known telemetry event groups */
 static struct telem_entry *telem_entry[] = {
+	&energy_0x26696143,
+	&perf_0x26557651,
 	NULL

Looks like a change from previous design to use telem_entry::num_events instead of NULL entry. The NULL entry can thus be removed?

arch/x86/kernel/cpu/resctrl/intel_aet.c static struct evtinfo { · line 146

Quoted 2 lines
arch/x86/kernel/cpu/resctrl/intel_aet.c +0 -0
 };
 

Reinette

author maintainer
#59 lore raw

PATCH v3 26/26 x86/resctrl: Update Documentation for package events

In reply to Tony Luck

Each "mon_data" directory is now divided between L3 events and package events.

The "info/PERF_PKG_MON" directory contains parameters for perf events.

---
 Documentation/filesystems/resctrl.rst | 38 ++++++++++++++++++++-------
 1 file changed, 28 insertions(+), 10 deletions(-)
Documentation/filesystems/resctrl.rst +28 -10
index 6768fc1fad16..b89a188b0321 100644
@@ -167,7 +167,7 @@ with respect to allocation:
 			bandwidth percentages are directly applied to
 			the threads running on the core
 
-If RDT monitoring is available there will be an "L3_MON" directory
+If RDT L3 monitoring is available there will be an "L3_MON" directory
 with the following files:
 
 "num_rmids":
@@ -261,6 +261,17 @@ with the following files:
 		bytes) at which a previously used LLC_occupancy
 		counter can be considered for re-use.
 
+If RDT PERF monitoring is available there will be an "L3_PERF_PKG" directory
+with the following files:
+
+"num_rmids":
+		The guaranteed number of hardware countes supporting RMIDs.
+		If more "CTRL_MON" + "MON" groups than this number are created,
+		the system may report that counters are "unavailable" when read.
+
+"mon_features":
+		Lists the perf monitoring events that are enabled on this system.
+
 Finally, in the top level of the "info" directory there is a file
 named "last_cmd_status". This is reset with every "command" issued
 via the file system (making new directories or writing to any of the
@@ -366,15 +377,22 @@ When control is enabled all CTRL_MON groups will also contain:
 When monitoring is enabled all MON groups will also contain:
 
 "mon_data":
-	This contains a set of files organized by L3 domain and by
-	RDT event. E.g. on a system with two L3 domains there will
-	be subdirectories "mon_L3_00" and "mon_L3_01".	Each of these
-	directories have one file per event (e.g. "llc_occupancy",
-	"mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
-	files provide a read out of the current value of the event for
-	all tasks in the group. In CTRL_MON groups these files provide
-	the sum for all tasks in the CTRL_MON group and all tasks in
-	MON groups. Please see example section for more details on usage.
+	This contains a set of directories, one for each instance
+	of an L3 cache, or of a processor package. The L3 cache
+	directories are named "mon_L3_00", "mon_L3_01" etc. The
+	package directories "mon_PERF_PKG_00", "mon_PERF_PKG_01" etc.
+
+	Within each directory there is one file per event. In
+	the L3 directories: "llc_occupancy", "mbm_total_bytes",
+	and "mbm_local_bytes".	In the PERF_PKG directories: "core_energy",
+	"activity", etc.
+
+	In a MON group these files provide a read out of the current
+	value of the event for all tasks in the group. In CTRL_MON groups
+	these files provide the sum for all tasks in the CTRL_MON group
+	and all tasks in MON groups. Please see example section for more
+	details on usage.
+
 	On systems with Sub-NUMA Cluster (SNC) enabled there are extra
 	directories for each node (located within the "mon_L3_XX" directory
 	for the L3 cache they occupy). These are named "mon_sub_L3_YY"
Signature · 1 line
2.48.1
maintainer
#60 lore raw

Re: [PATCH v3 26/26] x86/resctrl: Update Documentation for package events

Hi Tony,

Quoted 15 lines · Tony Luck

Each "mon_data" directory is now divided between L3 events and package events.

The "info/PERF_PKG_MON" directory contains parameters for perf events.

---
 Documentation/filesystems/resctrl.rst | 38 ++++++++++++++++++++-------
 1 file changed, 28 insertions(+), 10 deletions(-)
Documentation/filesystems/resctrl.rst +0 -0
index 6768fc1fad16..b89a188b0321 100644

Documentation/filesystems/resctrl.rst with respect to allocation: · line 170

Quoted 5 lines
Documentation/filesystems/resctrl.rst +1 -1
@@ -167,7 +167,7 @@ with respect to allocation:
 			bandwidth percentages are directly applied to
 			the threads running on the core
 
-If RDT monitoring is available there will be an "L3_MON" directory
+If RDT L3 monitoring is available there will be an "L3_MON" directory

I think "RDT" can just be dropped.

Documentation/filesystems/resctrl.rst with the following files: · line 264

Quoted 8 lines
Documentation/filesystems/resctrl.rst +1 -0
 with the following files:
 
 "num_rmids":
@@ -261,6 +261,17 @@ with the following files:
 		bytes) at which a previously used LLC_occupancy
 		counter can be considered for re-use.
 
+If RDT PERF monitoring is available there will be an "L3_PERF_PKG" directory

"L3_PERF_PKG" -> "PERF_PKG_MON" ?

I understand that the existing L3 documentation contains this term but I do not see a reason why the documentation should make this new monitoring Intel/RDT specific.

Also, I do not think user can be expected to know what "perf monitoring" is.

Documentation/filesystems/resctrl.rst with the following files: · line 268

Quoted 4 lines
Documentation/filesystems/resctrl.rst +4 -0
+with the following files:
+
+"num_rmids":
+		The guaranteed number of hardware countes supporting RMIDs.

countes -> counters?

The use of "hardware counters" is a bit unexpected ... the series did not mention this or I must have missed this.

Documentation/filesystems/resctrl.rst with the following files: · line 270

Quoted 2 lines
Documentation/filesystems/resctrl.rst +2 -0
+		If more "CTRL_MON" + "MON" groups than this number are created,
+		the system may report that counters are "unavailable" when read.

To be precise it is "Unavailable" ... but I do not think that is a good interface.

Documentation/filesystems/resctrl.rst with the following files: · line 273

Quoted 3 lines
Documentation/filesystems/resctrl.rst +3 -0
+
+"mon_features":
+		Lists the perf monitoring events that are enabled on this system.

"PERF" (all caps) at top and "perf" lower case here?

Quoted 15 lines
Documentation/filesystems/resctrl.rst +1 -7
+
 Finally, in the top level of the "info" directory there is a file
 named "last_cmd_status". This is reset with every "command" issued
 via the file system (making new directories or writing to any of the
@@ -366,15 +377,22 @@ When control is enabled all CTRL_MON groups will also contain:
 When monitoring is enabled all MON groups will also contain:
 
 "mon_data":
-	This contains a set of files organized by L3 domain and by
-	RDT event. E.g. on a system with two L3 domains there will
-	be subdirectories "mon_L3_00" and "mon_L3_01".	Each of these
-	directories have one file per event (e.g. "llc_occupancy",
-	"mbm_total_bytes", and "mbm_local_bytes"). In a MON group these
-	files provide a read out of the current value of the event for
-	all tasks in the group. In CTRL_MON groups these files provide

Documentation/filesystems/resctrl.rst When control is enabled all CTRL_MON groups will also contain: · line 387

Quoted 10 lines
Documentation/filesystems/resctrl.rst +8 -2
-	the sum for all tasks in the CTRL_MON group and all tasks in
-	MON groups. Please see example section for more details on usage.
+	This contains a set of directories, one for each instance
+	of an L3 cache, or of a processor package. The L3 cache
+	directories are named "mon_L3_00", "mon_L3_01" etc. The
+	package directories "mon_PERF_PKG_00", "mon_PERF_PKG_01" etc.
+
+	Within each directory there is one file per event. In
+	the L3 directories: "llc_occupancy", "mbm_total_bytes",
+	and "mbm_local_bytes".	In the PERF_PKG directories: "core_energy",

stray tab here

Documentation/filesystems/resctrl.rst When control is enabled all CTRL_MON groups will also contain: · line 398

Quoted 11 lines
Documentation/filesystems/resctrl.rst +8 -0
+	"activity", etc.
+
+	In a MON group these files provide a read out of the current
+	value of the event for all tasks in the group. In CTRL_MON groups
+	these files provide the sum for all tasks in the CTRL_MON group
+	and all tasks in MON groups. Please see example section for more
+	details on usage.
+
 	On systems with Sub-NUMA Cluster (SNC) enabled there are extra
 	directories for each node (located within the "mon_L3_XX" directory
 	for the L3 cache they occupy). These are named "mon_sub_L3_YY"

Reinette

maintainer
#61 lore raw
In reply to Tony Luck

Hi Tony,

Quoted 4 lines · Tony Luck

Previous version here:
https://lore.kernel.org/all/[email protected]/

This series is based on James Morse's "fs/resctrl/" snapshot.

Would be helpful to provide link to snapshot used to avoid any uncertainty about what base to use.

Quoted 10 lines

Background

Telemetry features are being implemented in conjunction with the IA32_PQR_ASSOC.RMID value on each logical CPU. This is used to send counts for various events to a collector in a nearby OOMMSM device to be accumulated with counts for each <RMID, event> pair received from other CPUs. Cores send event counts when the RMID value changes, or after each 2ms elapsed time.

Quoted 4 lines

Each OOBMSM device may implement multiple event collectors with each servicing a subset of the logical CPUs on a package. In the initial hardware implementation, there are two categories of events:

(missing the two categories of events)

Quoted 21 lines

The counters are arranged in groups in MMIO space of the OOBMSM device. E.g. for the energy counters the layout is:

Offset: Counter
0x00 core energy for RMID 0
0x08 core activity for RMID 0
0x10 core energy for RMID 1
0x18 core activity for RMID 1

1) Energy - Two counters
core_energy: This is an estimate of Joules consumed by each core. It is calculated based on the types of instructions executed, not from a power meter. This counter is useful to understand how much energy a workload is consuming.

activity: This measures "accumulated dynamic capacitance". Users who want to optimize energy consumption for a workload may use this rather than core_energy because it provides consistent results independent of any frequency or voltage changes that may occur during the runtime of the application (e.g. entry/exit from turbo mode).

Quoted 3 lines

2) Performance - Seven counters
These are similar events to those available via the Linux "perf" tool, but collected in a way with mush lower overhead (no need to collect data

"mush" -> "much"

Quoted 37 lines

on every context switch).

stalls_llc_hit - Counts the total number of unhalted core clock cycles when the core is stalled due to a demand load miss which hit in the LLC

c1_res - Counts the total C1 residency across all cores. The underlying counter increments on 100MHz clock ticks

unhalted_core_cycles - Counts the total number of unhalted core clock cycles

stalls_llc_miss - Counts the total number of unhalted core clock cycles when the core is stalled due to a demand load miss which missed all the local caches

c6_res - Counts the total C6 residency. The underlying counter increments on crystal clock (25MHz) ticks

unhalted_ref_cycles - Counts the total number of unhalted reference clock (TSC) cycles

uops_retired - Counts the total number of uops retired

Enumeration

The only CPUID based enumeration for this feature is the legacy CPUID(eax=7,ecx=0).ebx{12} that indicates the presence of the IA32_PQR_ASSOC MSR and the RMID field within it.

The OOBMSM driver discovers which features are present via PCIe VSEC capabilities. Each feature is tagged with a unique identifier. These identifiers indicate which XML description file from
https://github.com/intel/Intel-PMT describes which event counters are available and their layout within the MMIO BAR space of the OOBMSM device.

Resctrl User Interface

Quoted 8 lines

Because there may be multiple OOBMSM collection agents per processor package, resctrl accumulates event counts from all agents on a package and presents a single value to users. This will provide a consistent user interface on future platforms that vary the number of collectors, or the mappings from logical CPUs to collectors.

Users will see the legacy monitoring files in the "L3" directories and the telemetry files in "PKG" directories (with each file

Now PERF_PKG?

Quoted 34 lines

providing the aggregated value from all OOBMSM collectors on that package).

$ tree /sys/fs/resctrl/mon_data/
/sys/fs/resctrl/mon_data/
├── mon_L3_00
│   ├── llc_occupancy
│   ├── mbm_local_bytes
│   └── mbm_total_bytes
├── mon_L3_01
│   ├── llc_occupancy
│   ├── mbm_local_bytes
│   └── mbm_total_bytes
├── mon_PKG_00
│   ├── activity
│   ├── c1_res
│   ├── c6_res
│   ├── core_energy
│   ├── stalls_llc_hit
│   ├── stalls_llc_miss
│   ├── unhalted_core_cycles
│   ├── unhalted_ref_cycles
│   └── uops_retired
└── mon_PKG_01
    ├── activity
    ├── c1_res
    ├── c6_res
    ├── core_energy
    ├── stalls_llc_hit
    ├── stalls_llc_miss
    ├── unhalted_core_cycles
    ├── unhalted_ref_cycles
    └── uops_retired
Quoted 3 lines

Resctrl Implementation

The OOBMSM driver exposes a function "intel_pmt_get_regions_by_feature()"

(nit: no need to use "a function" if using ())

Quoted 26 lines

that returns an array of structures describing the per-RMID groups it found from the VSEC enumeration. Linux looks at the unique identifiers for each group and enables resctrl for all groups with known unique identifiers.

The memory map for the counters for each <RMID, event> pair is described by the XML file. This is too unwieldy to use in the Linux kernel, so a simplified representation is built into the resctrl code. Note that the counters are in MMIO space instead of accessed using the IA32_QM_EVTSEL and IA32_QM_CTR MSRs. This means there is no need for cross-processor calls to read counters from a CPU in a specific domain. The counters can be read from any CPU.

High level description of code changes:

1) New scope RESCTRL_PACKAGE
2) New struct rdt_resource RDT_RESOURCE_INTEL_PMT
3) Refactor monitor code paths to split existing L3 paths from new ones. In some cases this ends up with:

        switch (r->rid) {
        case RDT_RESOURCE_L3:
                helper for L3
                break;
        case RDT_RESOURCE_INTEL_PMT:
                helper for PKG
                break;
        }
Quoted 1 line

4) New source code file "intel_pmt.c" for the code to enumerate, configure, and report event counts.

Needs an update to match new version of this work.

Quoted 8 lines

With only one platform providing this feature, it's tricky to tell exactly where it is going to go. I've made the event definitions platform specific (based on the unique ID from the VSEC enumeration). It seems possible/likely that the list of events may change from generation to generation.

I've picked names for events based on the descriptions in the XML file.

One aspect that is only hinted to in the final documentation patch is how users are expected to use this feature. As I understand the number of monitor groups supported by resctrl is still guided by the number of RMIDs supported by L3 monitoring. This work hints that the telemetry feature may not match that number of RMIDs and a monitor group may thus exist but when a user attempts to ready any of these perf files it will return "unavailable".

The series attempts to address it by placing the number of RMIDs available for this feature in a "num_rmids" file, but since the RMID assigned to a monitor group is not exposed to user space (unless debugging enabled) the user does not know if a monitor group will support this feature or not. This seems awkward to me. Why not limit the number of monitor groups that can be created to the minimum number of RMIDs across these resources like what is done for CLOSid?

Reinette

author maintainer
#62 lore raw
Quoted 8 lines · Reinette Chatre

One aspect that is only hinted to in the final documentation patch is how users are expected to use this feature. As I understand the number of monitor groups supported by resctrl is still guided by the number of RMIDs supported by L3 monitoring. This work hints that the telemetry feature may not match that number of RMIDs and a monitor group may thus exist but when a user attempts to ready any of these perf files it will return "unavailable".

Quoted 6 lines

The series attempts to address it by placing the number of RMIDs available for this feature in a "num_rmids" file, but since the RMID assigned to a monitor group is not exposed to user space (unless debugging enabled) the user does not know if a monitor group will support this feature or not. This seems awkward to me. Why not limit the number of monitor groups that can be created to the minimum number of RMIDs across these resources like what is done for CLOSid?

Reinette,

The mismatch between number of RMIDs supported by different components is a thorny one, and may keep repeating since it feels like systems are composed of a bunch of lego-like bricks snapped together from a box of parts available to the h/w architect.

In this case we have three meanings for "number of RMIDs":

1) The number for legacy features enumerated by CPUID leaf 0xF.

2) The number of registers in MMIO space for each event. This is enumerated in the XML files and is the value I placed into telem_entry::num_rmids.

3) The number of "h/w counters" (this isn't a strictly accurate description of how things work, but serves as a useful analogy that does describe the limitations) feeding to those MMIO registers. This is enumerated in telemetry_region::num_rmids returned from the call to intel_pmt_get_regions_by_feature()

If "1" is the smallest of these values, the OS will be limited in which values can be written to the IA32_PQR_ASSOC MSR. Existing code will do the right thing by limiting RMID allocation to this value.

If "2" is greater than "1", then the extra MMIO registers will sit unused.

If "2" is less than "1" my v3 returns the (problematic) -ENOENT This can't happen in the CPU that debuts this feature, but the check is there to prevent running past the end of the MMIO space in case this does occur some day. I'll fix error path in next version to make sure this end up with "Unavailable".

If "3" is less than "2" then the system will attach "h/w counters" to MMIO registers in a "most recently used" algorithm. So if the number of active RMIDs in some time interval is less than "3" the user will get good values. But if the number of active RMIDs rises above "3" then the user will see "Unavailable" returns as "h/w counters" are reassigned to different RMIDs (making the feature really hard to use).

In the debut CPU the "energy" feature has sufficient "energy" counters to avoid this. But not enough "perf" counters. I've pushed and the next CPU with the feature will have enough "h/w counters".

My proposal for v4:

Add new options to the "rdt=" kernel boot parameter for "energy" and "perf".

Treat the case where there are not enough "h/w counters" as an erratum and do not enable the feature. User can override with "rdt=perf" if they want the counters for some special case where they limit the number of simultaneous active RMIDs.

User can use "rdt=!energy,!perf" if they don't want to see the clutter of all the new files in each mon_data directory.

I'll maybe look at moving resctrl_mon_resource_init() to rdt_get_tree() and add a "take min of all RMID limits". But since this is a "can't happen" scenario I may skip this if it starts to get complicated.

Which leaves what should be in info/PERF_PKG_MON/num_rmids? It's possible that some CPU implementation will have different MMIO register counts for "perf" and "energy". It's more than possible that number of "h/w counters" will be different. But they share the same info file. My v3 code reports the minimum of the number of "h/w counters" which is the most conservative option. It tells the user not to make more mon_data directories than this if they want usable counters across *both* perf and energy. Though they will actually keep getting good "energy" results even if then go past this limit.

-Tony

Quoted 1 line
maintainer
#63 lore raw

Hi Tony,

Quoted 16 lines · Tony
Quoted 8 lines · Reinette Chatre

One aspect that is only hinted to in the final documentation patch is how users are expected to use this feature. As I understand the number of monitor groups supported by resctrl is still guided by the number of RMIDs supported by L3 monitoring. This work hints that the telemetry feature may not match that number of RMIDs and a monitor group may thus exist but when a user attempts to ready any of these perf files it will return "unavailable".

Quoted 6 lines

The series attempts to address it by placing the number of RMIDs available for this feature in a "num_rmids" file, but since the RMID assigned to a monitor group is not exposed to user space (unless debugging enabled) the user does not know if a monitor group will support this feature or not. This seems awkward to me. Why not limit the number of monitor groups that can be created to the minimum number of RMIDs across these resources like what is done for CLOSid?

Quoted 6 lines

Reinette,

The mismatch between number of RMIDs supported by different components is a thorny one, and may keep repeating since it feels like systems are composed of a bunch of lego-like bricks snapped together from a box of parts available to the h/w architect.

With resctrl needing to support multiple architectures' way of doing things, needing to support variety within an architecture just seems like another step.

Quoted 5 lines

In this case we have three meanings for "number of RMIDs":

1) The number for legacy features enumerated by CPUID leaf 0xF.

Quoted 8 lines

2) The number of registers in MMIO space for each event. This is enumerated in the XML files and is the value I placed into telem_entry::num_rmids.

3) The number of "h/w counters" (this isn't a strictly accurate description of how things work, but serves as a useful analogy that does describe the limitations) feeding to those MMIO registers. This is enumerated in telemetry_region::num_rmids returned from the call to intel_pmt_get_regions_by_feature()

Thank you for explaining this. This was not clear to me.

Quoted 8 lines

If "1" is the smallest of these values, the OS will be limited in which values can be written to the IA32_PQR_ASSOC MSR. Existing code will do the right thing by limiting RMID allocation to this value.

If "2" is greater than "1", then the extra MMIO registers will sit unused.

This is also an issue with this implementation, no? resctrl will not allow creating more monitor groups than "1".

Quoted 5 lines

If "2" is less than "1" my v3 returns the (problematic) -ENOENT This can't happen in the CPU that debuts this feature, but the check is there to prevent running past the end of the MMIO space in case this does occur some day. I'll fix error path in next version to make sure this end up with "Unavailable".

This is a concern since this means the interface becomes a "try and see" for user space. As I understand a later statement the idea is that "2" should be used by user space to know how many "mon_groups" directories should be created to get telemetry support. To me this looks to be a space that will create a lot of confusion. The moment user space creates "2" + 1 "mon_groups" directories it becomes a guessing game of what any new monitor group actually supports. After crossing that threshold I do not see a good way for going back since if user space removes one "mon_data" directory it does get back to "2" but then needs to rely on resctrl internals or debugging to know for sure what the new monitor group supports.

Quoted 7 lines

If "3" is less than "2" then the system will attach "h/w counters" to MMIO registers in a "most recently used" algorithm. So if the number of active RMIDs in some time interval is less than "3" the user will get good values. But if the number of active RMIDs rises above "3" then the user will see "Unavailable" returns as "h/w counters" are reassigned to different RMIDs (making the feature really hard to use).

Could the next step be for the architecture to allow user space to specify which hardware counters need to be assigned? With a new user interface being created for such capability it may be worthwhile to consider how it could be used/adapted for this feature. [1]

Quoted 7 lines

In the debut CPU the "energy" feature has sufficient "energy" counters to avoid this. But not enough "perf" counters. I've pushed and the next CPU with the feature will have enough "h/w counters".

My proposal for v4:

Quoted 7 lines

Add new options to the "rdt=" kernel boot parameter for "energy" and "perf".

Treat the case where there are not enough "h/w counters" as an erratum and do not enable the feature. User can override with "rdt=perf" if they want the counters for some special case where they limit the number of simultaneous active RMIDs.

This only seems to address the "3" is less than "2" issue. It is not so obvious to me that it should be treated as an erratum. Although, I could not tell from your description how obvious this issue will be to user space. For example, is it clear that if user space
gets *any* value then it is "good" and "Unavailable" means ... "Unavailable", or could a returned value mean "this is partial data that was collected during timeframe with hardware counter re-assigned at some point"?

Quoted 7 lines

User can use "rdt=!energy,!perf" if they don't want to see the clutter of all the new files in each mon_data directory.

I'll maybe look at moving resctrl_mon_resource_init() to rdt_get_tree() and add a "take min of all RMID limits". But since this is a "can't happen" scenario I may skip this if it starts to get complicated.

I do not think that the "2" is less than "1" scenario should be ignored for reasons stated above and in review of this version.

What if we enhance resctrl's RMID assignment (setting aside for a moment PMG assignment) to be directed by user space?

Below is an idea of an interface that can give user space control over what monitor groups are monitoring. This is very likely not the ideal interface but I would like to present it as a start for better ideas.

For example, monitor groups are by default created with most abundant (and thus supporting fewest features on fewest resources) RMID.
The user is then presented with a new file (within each monitor group) that lists all available features and which one(s) are active. For example, let's consider hypothetical example where PERF_PKG perf has x RMID, PERF_PKG energy has y RMID, and L3_MON has z RMID, with x < y < z. By default when user space creates a monitor group resctrl will pick "abundant" RMID from range y + 1 to z that only supports L3 monitoring:

# cat /sys/fs/resctrl/mon_groups/m1/new_file_mon_resource_and_features
[L3]
PERF_PKG:energy
PERF_PKG:perf

In above case there will be *no* mon_PERF_PKG_XX directories in /sys/fs/resctrl/mon_groups/m1/mon_data.

*If* user space wants perf/energy telemetry for this monitor group then they can enable needed feature with clear understanding that it is disruptive to all ongoing monitoring since a new RMID will be assigned. For example, if user wants PERF_PKG:energy and PERF_PKG:perf then user can do so with:

# echo PERF_PKG:perf > /sys/fs/resctrl/mon_groups/m1/new_file_mon_resource_and_features
# cat /sys/fs/resctrl/mon_groups/m1/new_file_mon_resource_and_features
[L3]
[PERF_PKG:energy]
[PERF_PKG:perf]

After the above all energy and perf files will appear in new mon_PERF_PKG_XX directories.

User space can then have full control of what is monitored by which monitoring group. If no RMIDs are available in a particular pool then user space can get an "out of space" error and be the one to decide how it should be managed.

This also could be a way in which the "2" is larger than "1" scenario can be addressed.

Quoted 10 lines

Which leaves what should be in info/PERF_PKG_MON/num_rmids? It's possible that some CPU implementation will have different MMIO register counts for "perf" and "energy". It's more than possible that number of "h/w counters" will be different. But they share the same info file. My v3 code reports the minimum of the number of "h/w counters" which is the most conservative option. It tells the user not to make more mon_data directories than this if they want usable counters across *both* perf and energy. Though they will actually keep getting good "energy" results even if then go past this limit.

num_rmids is a source of complications. It does not have a good equivalent for MPAM and there has been a few attempts at proposing alternatives that may be worth keeping in mind while making changes here:
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

Quoted 3 lines

-Tony

Reinette

[1] https://lore.kernel.org/lkml/[email protected]/

ps. I needed to go back an re-read the original cover-letter a couple of times, while doing so I noticed one typo in the Background section: OOMMSM -> OOBMSM.

author maintainer
#64 lore raw
Quoted 51 lines · Reinette Chatre

Hi Tony,

Quoted 16 lines · Tony
Quoted 8 lines · Reinette Chatre

One aspect that is only hinted to in the final documentation patch is how users are expected to use this feature. As I understand the number of monitor groups supported by resctrl is still guided by the number of RMIDs supported by L3 monitoring. This work hints that the telemetry feature may not match that number of RMIDs and a monitor group may thus exist but when a user attempts to ready any of these perf files it will return "unavailable".

Quoted 6 lines

The series attempts to address it by placing the number of RMIDs available for this feature in a "num_rmids" file, but since the RMID assigned to a monitor group is not exposed to user space (unless debugging enabled) the user does not know if a monitor group will support this feature or not. This seems awkward to me. Why not limit the number of monitor groups that can be created to the minimum number of RMIDs across these resources like what is done for CLOSid?

Quoted 6 lines

Reinette,

The mismatch between number of RMIDs supported by different components is a thorny one, and may keep repeating since it feels like systems are composed of a bunch of lego-like bricks snapped together from a box of parts available to the h/w architect.

With resctrl needing to support multiple architectures' way of doing things, needing to support variety within an architecture just seems like another step.

Quoted 5 lines

In this case we have three meanings for "number of RMIDs":

1) The number for legacy features enumerated by CPUID leaf 0xF.

Quoted 8 lines

2) The number of registers in MMIO space for each event. This is enumerated in the XML files and is the value I placed into telem_entry::num_rmids.

3) The number of "h/w counters" (this isn't a strictly accurate description of how things work, but serves as a useful analogy that does describe the limitations) feeding to those MMIO registers. This is enumerated in telemetry_region::num_rmids returned from the call to intel_pmt_get_regions_by_feature()

Thank you for explaining this. This was not clear to me.

Quoted 6 lines

If "1" is the smallest of these values, the OS will be limited in which values can be written to the IA32_PQR_ASSOC MSR. Existing code will do the right thing by limiting RMID allocation to this value.

Quoted 5 lines
Quoted 2 lines

If "2" is greater than "1", then the extra MMIO registers will sit unused.

This is also an issue with this implementation, no? resctrl will not allow creating more monitor groups than "1".

On Intel there is no point in creating more groups than "1" allows. You can't make use of any RMID above that limit because you will get a #GP fault trying to write to the IA32_PQR_ASSOC MSR.

You could read the extra MMIO registers provided by "2", but they will always be zero since no execution occurred with an RMID in the range "1" ... "2".

The "2" is greater than "1" may be relatively common since the h/w for the telemetry counters is common for SKUs with different numbers of cores, and thus different values of "1". So low core count systems will see more telemetry counters than they can actually make use of. I will make sure not to print a message for this case.

Quoted 6 lines
Quoted 5 lines

If "2" is less than "1" my v3 returns the (problematic) -ENOENT This can't happen in the CPU that debuts this feature, but the check is there to prevent running past the end of the MMIO space in case this does occur some day. I'll fix error path in next version to make sure this end up with "Unavailable".

Quoted 11 lines

This is a concern since this means the interface becomes a "try and see" for user space. As I understand a later statement the idea is that "2" should be used by user space to know how many "mon_groups" directories should be created to get telemetry support. To me this looks to be a space that will create a lot of confusion. The moment user space creates "2" + 1 "mon_groups" directories it becomes a guessing game of what any new monitor group actually supports. After crossing that threshold I do not see a good way for going back since if user space removes one "mon_data" directory it does get back to "2" but then needs to rely on resctrl internals or debugging to know for sure what the new monitor group supports.

But I assert that it is a "can't happen" concern. "2" will be >= "1". See below. I will look at addressing this, unless it gets crazy complex because of the different enumeration timeline. Delaying calculation of number of RMIDs until rdt_get_tree() as you have suggested may be the right thing to do.

"3" is the real problem

Quoted 28 lines
Quoted 7 lines

If "3" is less than "2" then the system will attach "h/w counters" to MMIO registers in a "most recently used" algorithm. So if the number of active RMIDs in some time interval is less than "3" the user will get good values. But if the number of active RMIDs rises above "3" then the user will see "Unavailable" returns as "h/w counters" are reassigned to different RMIDs (making the feature really hard to use).

Could the next step be for the architecture to allow user space to specify which hardware counters need to be assigned? With a new user interface being created for such capability it may be worthwhile to consider how it could be used/adapted for this feature. [1]

Quoted 7 lines

In the debut CPU the "energy" feature has sufficient "energy" counters to avoid this. But not enough "perf" counters. I've pushed and the next CPU with the feature will have enough "h/w counters".

My proposal for v4:

Quoted 7 lines

Add new options to the "rdt=" kernel boot parameter for "energy" and "perf".

Treat the case where there are not enough "h/w counters" as an erratum and do not enable the feature. User can override with "rdt=perf" if they want the counters for some special case where they limit the number of simultaneous active RMIDs.

Quoted 7 lines

This only seems to address the "3" is less than "2" issue. It is not so obvious to me that it should be treated as an erratum. Although, I could not tell from your description how obvious this issue will be to user space. For example, is it clear that if user space
gets *any* value then it is "good" and "Unavailable" means ... "Unavailable", or could a returned value mean "this is partial data that was collected during timeframe with hardware counter re-assigned at some point"?

When running jobs with more distinct RMIDs than "3" users are at the mercy of the h/w replacement algorithm. Resctrl use cases for monitoring are all "read an event counter; wait for some time; re-read the event counter; compute the rate". With "h/w counter" reassignment the second read may get "Unavailable", or worse the "h/w counter" may have been taken, and the returned so a value will be provided to the user, but it won't provide the count of events since the first read.

That's why I consider this an erratum. There's just false hope that you can get a pair of meaningful event counts and no sure indication that you didn't get garbage.

Quoted 8 lines
Quoted 7 lines

User can use "rdt=!energy,!perf" if they don't want to see the clutter of all the new files in each mon_data directory.

I'll maybe look at moving resctrl_mon_resource_init() to rdt_get_tree() and add a "take min of all RMID limits". But since this is a "can't happen" scenario I may skip this if it starts to get complicated.

Quoted 5 lines

I do not think that the "2" is less than "1" scenario should be ignored for reasons stated above and in review of this version.

What if we enhance resctrl's RMID assignment (setting aside for a moment PMG assignment) to be directed by user space?

I'll take a look at reducing user reported num_rmids to the minimum of the "1" and "2" values.

Quoted 5 lines

Below is an idea of an interface that can give user space control over what monitor groups are monitoring. This is very likely not the ideal interface but I would like to present it as a start for better ideas.

Quoted 8 lines

For example, monitor groups are by default created with most abundant (and thus supporting fewest features on fewest resources) RMID.
The user is then presented with a new file (within each monitor group) that lists all available features and which one(s) are active. For example, let's consider hypothetical example where PERF_PKG perf has x RMID, PERF_PKG energy has y RMID, and L3_MON has z RMID, with x < y < z. By default when user space creates a monitor group resctrl will pick "abundant" RMID from range y + 1 to z that only supports L3 monitoring:

There is no way for s/w to control the reallocation of "h/w counters" when "3" is too small. So there is no set of RMIDs that support many events vs. fewer events. AMD is solving this similar problem with their scheme to pin h/w counters to specific RMIDs. I discussed such an option for the "3" case, but it wasn't practical to apply to the upcoming CPU that has this problem. The long term solution is to ensure that "3" is always large enough that all RMIDs have equal monitoring capabilities.

Quoted 51 lines
# cat /sys/fs/resctrl/mon_groups/m1/new_file_mon_resource_and_features
[L3]
PERF_PKG:energy
PERF_PKG:perf

In above case there will be *no* mon_PERF_PKG_XX directories in /sys/fs/resctrl/mon_groups/m1/mon_data.

*If* user space wants perf/energy telemetry for this monitor group then they can enable needed feature with clear understanding that it is disruptive to all ongoing monitoring since a new RMID will be assigned. For example, if user wants PERF_PKG:energy and PERF_PKG:perf then user can do so with:

# echo PERF_PKG:perf > /sys/fs/resctrl/mon_groups/m1/new_file_mon_resource_and_features
# cat /sys/fs/resctrl/mon_groups/m1/new_file_mon_resource_and_features
[L3]
[PERF_PKG:energy]
[PERF_PKG:perf]

After the above all energy and perf files will appear in new mon_PERF_PKG_XX directories.

User space can then have full control of what is monitored by which monitoring group. If no RMIDs are available in a particular pool then user space can get an "out of space" error and be the one to decide how it should be managed.

This also could be a way in which the "2" is larger than "1" scenario can be addressed.

Quoted 10 lines

Which leaves what should be in info/PERF_PKG_MON/num_rmids? It's possible that some CPU implementation will have different MMIO register counts for "perf" and "energy". It's more than possible that number of "h/w counters" will be different. But they share the same info file. My v3 code reports the minimum of the number of "h/w counters" which is the most conservative option. It tells the user not to make more mon_data directories than this if they want usable counters across *both* perf and energy. Though they will actually keep getting good "energy" results even if then go past this limit.

num_rmids is a source of complications. It does not have a good equivalent for MPAM and there has been a few attempts at proposing alternatives that may be worth keeping in mind while making changes here:
https://lore.kernel.org/all/[email protected]/
https://lore.kernel.org/lkml/[email protected]/

Quoted 3 lines

-Tony

Quoted 7 lines

Reinette

[1] https://lore.kernel.org/lkml/[email protected]/

ps. I needed to go back an re-read the original cover-letter a couple of times, while doing so I noticed one typo in the Background section: OOMMSM -> OOBMSM.

Noted. Will fix.

-Tony

maintainer
#65 lore raw

Hi Tony,

Quoted 66 lines · Tony
Quoted 51 lines · Reinette Chatre

Hi Tony,

Quoted 16 lines · Tony
Quoted 8 lines · Reinette Chatre

One aspect that is only hinted to in the final documentation patch is how users are expected to use this feature. As I understand the number of monitor groups supported by resctrl is still guided by the number of RMIDs supported by L3 monitoring. This work hints that the telemetry feature may not match that number of RMIDs and a monitor group may thus exist but when a user attempts to ready any of these perf files it will return "unavailable".

Quoted 6 lines

The series attempts to address it by placing the number of RMIDs available for this feature in a "num_rmids" file, but since the RMID assigned to a monitor group is not exposed to user space (unless debugging enabled) the user does not know if a monitor group will support this feature or not. This seems awkward to me. Why not limit the number of monitor groups that can be created to the minimum number of RMIDs across these resources like what is done for CLOSid?

Quoted 6 lines

Reinette,

The mismatch between number of RMIDs supported by different components is a thorny one, and may keep repeating since it feels like systems are composed of a bunch of lego-like bricks snapped together from a box of parts available to the h/w architect.

With resctrl needing to support multiple architectures' way of doing things, needing to support variety within an architecture just seems like another step.

Quoted 5 lines

In this case we have three meanings for "number of RMIDs":

1) The number for legacy features enumerated by CPUID leaf 0xF.

Quoted 8 lines

2) The number of registers in MMIO space for each event. This is enumerated in the XML files and is the value I placed into telem_entry::num_rmids.

3) The number of "h/w counters" (this isn't a strictly accurate description of how things work, but serves as a useful analogy that does describe the limitations) feeding to those MMIO registers. This is enumerated in telemetry_region::num_rmids returned from the call to intel_pmt_get_regions_by_feature()

Thank you for explaining this. This was not clear to me.

Quoted 6 lines

If "1" is the smallest of these values, the OS will be limited in which values can be written to the IA32_PQR_ASSOC MSR. Existing code will do the right thing by limiting RMID allocation to this value.

Quoted 5 lines
Quoted 2 lines

If "2" is greater than "1", then the extra MMIO registers will sit unused.

This is also an issue with this implementation, no? resctrl will not allow creating more monitor groups than "1".

On Intel there is no point in creating more groups than "1" allows. You can't make use of any RMID above that limit because you will get a #GP fault trying to write to the IA32_PQR_ASSOC MSR.

You could read the extra MMIO registers provided by "2", but they will always be zero since no execution occurred with an RMID in the range "1" ... "2".

Quoted 5 lines

The "2" is greater than "1" may be relatively common since the h/w for the telemetry counters is common for SKUs with different numbers of cores, and thus different values of "1". So low core count systems will see more telemetry counters than they can actually make use of. I will make sure not to print a message for this case.

I see, thank you.

Quoted 19 lines
Quoted 7 lines
Quoted 6 lines

If "2" is less than "1" my v3 returns the (problematic) -ENOENT This can't happen in the CPU that debuts this feature, but the check is there to prevent running past the end of the MMIO space in case this does occur some day. I'll fix error path in next version to make sure this end up with "Unavailable".

Quoted 11 lines

This is a concern since this means the interface becomes a "try and see" for user space. As I understand a later statement the idea is that "2" should be used by user space to know how many "mon_groups" directories should be created to get telemetry support. To me this looks to be a space that will create a lot of confusion. The moment user space creates "2" + 1 "mon_groups" directories it becomes a guessing game of what any new monitor group actually supports. After crossing that threshold I do not see a good way for going back since if user space removes one "mon_data" directory it does get back to "2" but then needs to rely on resctrl internals or debugging to know for sure what the new monitor group supports.

Quoted 5 lines

But I assert that it is a "can't happen" concern. "2" will be >= "1". See below. I will look at addressing this, unless it gets crazy complex because of the different enumeration timeline. Delaying calculation of number of RMIDs until rdt_get_tree() as you have suggested may be the right thing to do.

As you explain it, it does not sound as though calculating how many RMIDs can be supported is required to be done in rdt_get_tree(), but doing so would make the implementation more robust since doing so does not rely on assumptions about what hardware can and will support.

Quoted 30 lines

"3" is the real problem

Quoted 27 lines
Quoted 7 lines

If "3" is less than "2" then the system will attach "h/w counters" to MMIO registers in a "most recently used" algorithm. So if the number of active RMIDs in some time interval is less than "3" the user will get good values. But if the number of active RMIDs rises above "3" then the user will see "Unavailable" returns as "h/w counters" are reassigned to different RMIDs (making the feature really hard to use).

Could the next step be for the architecture to allow user space to specify which hardware counters need to be assigned? With a new user interface being created for such capability it may be worthwhile to consider how it could be used/adapted for this feature. [1]

Quoted 14 lines

In the debut CPU the "energy" feature has sufficient "energy" counters to avoid this. But not enough "perf" counters. I've pushed and the next CPU with the feature will have enough "h/w counters".

My proposal for v4:

Add new options to the "rdt=" kernel boot parameter for "energy" and "perf".

Treat the case where there are not enough "h/w counters" as an erratum and do not enable the feature. User can override with "rdt=perf" if they want the counters for some special case where they limit the number of simultaneous active RMIDs.

I get this now. This will require rework of the kernel command line parsing support since current implementation is so closely integrated with the X86_FEATURE_* flags (and is perhaps an unexpected architecture specific portion of resctrl).
What if "rdt=perf" means that "3" is also included in the computation of how many monitor groups are supported? That would help users to not need to limit the number of simultaneous active RMIDs.

Quoted 29 lines
Quoted 8 lines

This only seems to address the "3" is less than "2" issue. It is not so obvious to me that it should be treated as an erratum. Although, I could not tell from your description how obvious this issue will be to user space. For example, is it clear that if user space
gets *any* value then it is "good" and "Unavailable" means ... "Unavailable", or could a returned value mean "this is partial data that was collected during timeframe with hardware counter re-assigned at some point"?

When running jobs with more distinct RMIDs than "3" users are at the mercy of the h/w replacement algorithm. Resctrl use cases for monitoring are all "read an event counter; wait for some time; re-read the event counter; compute the rate". With "h/w counter" reassignment the second read may get "Unavailable", or worse the "h/w counter" may have been taken, and the returned so a value will be provided to the user, but it won't provide the count of events since the first read.

That's why I consider this an erratum. There's just false hope that you can get a pair of meaningful event counts and no sure indication that you didn't get garbage.

Quoted 8 lines
Quoted 7 lines

User can use "rdt=!energy,!perf" if they don't want to see the clutter of all the new files in each mon_data directory.

I'll maybe look at moving resctrl_mon_resource_init() to rdt_get_tree() and add a "take min of all RMID limits". But since this is a "can't happen" scenario I may skip this if it starts to get complicated.

Quoted 8 lines
Quoted 5 lines

I do not think that the "2" is less than "1" scenario should be ignored for reasons stated above and in review of this version.

What if we enhance resctrl's RMID assignment (setting aside for a moment PMG assignment) to be directed by user space?

I'll take a look at reducing user reported num_rmids to the minimum of the "1" and "2" values.

When comparing to "num_closids" the expectation may be that "num_rmids" would be accurate for particular resource with understanding that the minimum among all resources guides the number of monitor groups. This seems close enough to existing interface to not use this as moment to move to a new "num_mon_hw_id" or such that works for MPAM also.

Quoted 15 lines
Quoted 6 lines

Below is an idea of an interface that can give user space control over what monitor groups are monitoring. This is very likely not the ideal interface but I would like to present it as a start for better ideas.

Quoted 8 lines

For example, monitor groups are by default created with most abundant (and thus supporting fewest features on fewest resources) RMID.
The user is then presented with a new file (within each monitor group) that lists all available features and which one(s) are active. For example, let's consider hypothetical example where PERF_PKG perf has x RMID, PERF_PKG energy has y RMID, and L3_MON has z RMID, with x < y < z. By default when user space creates a monitor group resctrl will pick "abundant" RMID from range y + 1 to z that only supports L3 monitoring:

Quoted 7 lines

There is no way for s/w to control the reallocation of "h/w counters" when "3" is too small. So there is no set of RMIDs that support many events vs. fewer events. AMD is solving this similar problem with their scheme to pin h/w counters to specific RMIDs. I discussed such an option for the "3" case, but it wasn't practical to apply to the upcoming CPU that has this problem. The long term solution is to ensure that "3" is always large enough that all RMIDs have equal monitoring capabilities.

ack.

Reinette

maintainer
#66 lore raw
In reply to Tony Luck

Hi Tony,

Just noticed ... could you please include x86 maintainers (X86 ARCHITECTURE (32-BIT AND 64-BIT) in MAINTAINERS) in your submission?

Thank you

Reinette